Subject: [lmap] Reporting of service parameters (WGLC comments on draft-ietf-lmap-framework-03)

From: <[email-masked]>
Date: Wed, 19 Feb 2014 17:35:14 +0000

One issue that was raised by Jason is that it may be good to include 'service parameters' in the Report - I presume this is things like the subscribed rate and type of access).

Currently we say in S5.4
<<The primary purpose of the Report Protocol is to allow a Measurement
   Agent to report its Measurement Results to a Collector, and the
   context in which they were obtained.
...
   The Report contains:

   o  the MA-ID or a Group-ID (to anonymise results)

   o  the actual Measurement Results, including the time they were
      measured

   o  the details of the Measurement Task (to avoid the Collector having
      to ask the Controller for this information later)
I propose we add another bullet
   o  the MA also includes the Subscribed service parameters, such as the subscribed rate and type of access, if inclusion of such information has been configured for the Report Channel and is known by the MA

this would imply that we should include something in the Instruction - I think as part of the configuration of each Report Channel:
      *  a configuration option that the MA includes information about any Subscribed service parameters that it knows (by default, not included)

I'm assuming that this will just be a free list of service parameters, ie the information model doesn't have to worry about defining the details (perhaps the information model should say something about it being a list of name-value pairs? For instance  "subscribed rate: 80 Mbs up 20 Mbs down ; access type : FTTC")

What do you reckon?

thanks
phil
--

Original comment from Jason:

<<
2) Reporting of Service Parameters by MA if available
The Charter includes this sentence on the topic of Service Parameters: "However, if the service parameters are available to the MAs, they could be reported
with the measurement results in the Report Protocol"

Discovery of Service Parameters is out of scope per the Charter, but I believe there has been some discussion on the list that if the MA has the information it may be included in the Reporting Protocol. It is stated as a 'could' but I think it would be useful to state this somewhere in the discussion of the Report Protocol. There currently is this bullet which seems to imply that only the Results are to be delivered to the collector and not the Service Parameters but I may be reading too much into this bullet:

   o  a Report Protocol, which delivers a Report from the MA to a
      Collector.  The Report contains the Measurement Results.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 19 Feb 2014 23:26:06 +0100

On Wed, Feb 19, 2014 at 05:35:14PM +0000, [email-masked] wrote:

> One issue that was raised by Jason is that it may be good to include
> 'service parameters' in the Report - I presume this is things like
> the subscribed rate and type of access).

While it might be convenient for processing results to have service
parameters available, it feels wrong to decorate every Report with
service parameters.

> I'm assuming that this will just be a free list of service
> parameters, ie the information model doesn't have to worry about
> defining the details (perhaps the information model should say
> something about it being a list of name-value pairs? For instance
> "subscribed rate: 80 Mbs up 20 Mbs down ; access type : FTTC")

This makes it kind of non-interoperable by definition. So why define
it then?

Another way of dealing with this is to define a measurement task that
simply reports the service parameters. This way, you can schedule how
frequently service parameters are shipped to collectors.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 19 Feb 2014 18:09:23 -0500

Although it is permitted in the LMAP charter, there's no need to include the 
service parameters in the ISP use case.  A third party could get the service
info per participating subscriber from the service provider separately, probably more
efficient than having the MA do it. 

Al
________________________________________
From: lmap [[email-masked]] On Behalf Of Juergen Schoenwaelder [[email-masked]]
Sent: Wednesday, February 19, 2014 5:26 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [lmap] Reporting of service parameters (WGLC comments on draft-ietf-lmap-framework-03)

/js

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: "Weil, Jason" <[email-masked]>
Date: Wed, 19 Feb 2014 21:42:12 -0500

I was thinking more of a may operation. If the data is available you may
want to send it up with the measurement info in the Report Protocol. The
way it reads now this is not supported at all. If this is the case we
should explicitly state that as well.

Jason

>efficient than having the MA do it.

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 20 Feb 2014 09:11:41 +0000

I concur - If they are available and defined and contextually relevant to the test (in fact saying % of Terms of Service throughput) then they are required.

We have found that some terms of service literally change over time of day, 
Others change based on "state of the service" passed a cap and get throttle  - more of a wireless thing today but growing on the other technologies
Walled gardens from people getting hacked and not understanding it..

The bigger issues here-
   If you have 2 people out of 10 in a service tier with a "mis understood service state" - publishing results that don't accurate reflect facts would be highly problematic to say the least.

So a small bit of work here goes a long way.

Regards,
Mike

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Weil, Jason
Sent: Wednesday, February 19, 2014 8:42 PM
To: MORTON, ALFRED C (AL); Juergen Schoenwaelder; [email-masked]
Cc: [email-masked]
Subject: Re: [lmap] Reporting of service parameters (WGLC comments on draft-ietf-lmap-framework-03)

I was thinking more of a may operation. If the data is available you may want to send it up with the measurement info in the Report Protocol. The way it reads now this is not supported at all. If this is the case we should explicitly state that as well.

Jason

On 2/19/14 6:09 PM, "MORTON, ALFRED C (AL)" <[email-masked]> wrote:

>Although it is permitted in the LMAP charter, there's no need to 
>include the service parameters in the ISP use case.  A third party 
>could get the service info per participating subscriber from the 
>service provider separately, probably more efficient than having the MA 
>do it.
>
>Al
>This makes it kind of non-interoperable by definition. So why define it 
>then?
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 20 Feb 2014 10:21:44 +0100

Sure, but for me this is just another 'measurement task' reporting
service parameters. You want the flexibility to control and schedule
this like any other measurement task. The current LMAP framework and
information model provide you the means to do this and hence I believe
there is nothing technical to be done to address this requirement.

I do not mind if text gets added to the framework document pointing
out that service parameters can be reported by a special 'service
parameters measurement task' if that adds clarity how one can address
the "However, if the service parameters are available to the MAs, they
could be reported with the measurement results in the Report
Protocol." phrase of the charter.

/js

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 20 Feb 2014 10:24:35 +0000

Great! - So I think we have agreement that we can add a "test" step that includes hitting the ISP interface for Service Attributes.
So a "service attributes" "dip" should be included if the MA was configured or discovered an ISP Service interface... COOL!

We still have to resolve the "detecting" transit traffic - the interface to grab utilization information...
	Could be the same interface, or different one...

Any druthers ???

Mike

-----Original Message-----
From: Juergen Schoenwaelder [mailto:[email-masked]] 
Sent: Thursday, February 20, 2014 3:22 AM
To: Bugenhagen, Michael K
Cc: 'Weil, Jason'; 'MORTON, ALFRED C (AL)'; [email-masked]'; [email-masked]'
Subject: Re: [lmap] Reporting of service parameters (WGLC comments on draft-ietf-lmap-framework-03)

Sure, but for me this is just another 'measurement task' reporting service parameters. You want the flexibility to control and schedule this like any other measurement task. The current LMAP framework and information model provide you the means to do this and hence I believe there is nothing technical to be done to address this requirement.

I do not mind if text gets added to the framework document pointing out that service parameters can be reported by a special 'service parameters measurement task' if that adds clarity how one can address the "However, if the service parameters are available to the MAs, they could be reported with the measurement results in the Report Protocol." phrase of the charter.

/js

On Thu, Feb 20, 2014 at 09:11:41AM +0000, Bugenhagen, Michael K wrote:
> We have found that some terms of service literally change over time of 
> day, Others change based on "state of the service" passed a cap and 
> get throttle  - more of a wireless thing today but growing on the other technologies Walled gardens from people getting hacked and not understanding it..
> 
> The bigger issues here-
> >service provider separately, probably more efficient than having the 
> >MA do it.
> >
> >Al
> >> One issue that was raised by Jason is that it may be good to 
> >> include 'service parameters' in the Report - I presume this is 
> >> things like the subscribed rate and type of access).
> >

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 21 Feb 2014 14:03:54 +0000

> I was thinking more of a may operation. If the data is available you may want
> to send it up with the measurement info in the Report Protocol. The way it
> reads now this is not supported at all. If this is the case we should explicitly
> state that as well.
> 
> Jason

Going back to what Jason said...
The framework already says:

"5.4.  Report Protocol

   The primary purpose of the Report Protocol is to allow a Measurement
   Agent to report its Measurement Results to a Collector, and the
   context in which they were obtained."

It then has a bulleted list about what the Report contains. I think if we added another bullet that said something like:
 - the context in which the Measurement Results were obtained, where the details of which (if any) context to include are dependent on implementation and Measurement Method/Task

And then later in the same section (maybe at the end) if we added a paragraph that said something like:
The context in which Measurement Results are obtained may include attributes related to an end user's service (e.g., subscribed data rate) as well as information related to the conditions that existed when the Measurement Task took place (e.g., resource utilization on the physical device where the MA resides). How such information may be obtained is outside the scope of the LMAP WG. Which contextual information to include in a Report is specific to the MA implementation and the Measurement Methods / Tasks.

--------------
Information Model
I think there still needs to be a discussion regarding what to include in the Information Model related to this contextual information. But I don't feel ready to have that conversation here now. I think I'll be ready in about a month.

Barbara

---

From: <[email-masked]>
Date: Fri, 21 Feb 2014 14:45:22 +0000

Now I engage my brain, I remember (I think) that a while back we agreed that enhancing measurement data with subscriber info was a post collector activity. 

However, Barbara's text works for me. I think it's also worth adding a note that subscriber /line information can be enhanced post collection - in the Repository or Data collection tools (in Figure 1 of the framework).
Framework-03 almost says this already:  
A results repository records all Measurement Results in an equivalent
   form, for example an SQL database, so that they can easily be
   accessed by the data analysis tools.  The data analysis tools also
   need to understand the Subscriber's service information, for example
   the broadband contract.

phil

> From: STARK, BARBARA H [mailto:[email-masked]]
> Sent: 21 February 2014 14:04
> To: Weil, Jason; MORTON JR., AL; Juergen Schoenwaelder;
> Eardley,PL,Philip,TUB8 R
> Cc: [email-masked]
> Subject: RE: [lmap] Reporting of service parameters (WGLC comments on
> draft-ietf-lmap-framework-03)
> 
> > I was thinking more of a may operation. If the data is available you
> > may want to send it up with the measurement info in the Report
> > Protocol. The way it reads now this is not supported at all. If this
> > is the case we should explicitly state that as well.
> >
> > Jason
> 
> It then has a bulleted list about what the Report contains. I think if
> we added another bullet that said something like:
>  - the context in which the Measurement Results were obtained, where
> the details of which (if any) context to include are dependent on
> implementation and Measurement Method/Task
> 
> The context in which Measurement Results are obtained may include
> attributes related to an end user's service (e.g., subscribed data
> rate) as well as information related to the conditions that existed
> when the Measurement Task took place (e.g., resource utilization on the
> physical device where the MA resides). How such information may be
> obtained is outside the scope of the LMAP WG. Which contextual
> information to include in a Report is specific to the MA implementation
> and the Measurement Methods / Tasks.
> 
> --------------
> Information Model
> I think there still needs to be a discussion regarding what to include
> in the Information Model related to this contextual information. But I
> don't feel ready to have that conversation here now. I think I'll be
> ready in about a month.
> 
> Barbara

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 21 Feb 2014 14:56:19 +0000

> Now I engage my brain, I remember (I think) that a while back we agreed
> that enhancing measurement data with subscriber info was a post collector
> activity.

Minor nit to this remembrance...
My remembrance was that we agreed that this *could* be a post collector activity, or it *could* be done by the MA and attached to reported results. And that lmap would not try to preclude or mandate how this would be accomplished, or define any method for acquiring the subscriber info. But I think that the Report does need to *allow* for inclusion of such information, or else it would effectively be precluding the MA-attachment model.

Since you said you were ok with my suggested text, and my suggestion was to allow for this inclusion, I'm hoping we're ok.
Barbara

---

From: "Weil, Jason" <[email-masked]>
Date: Fri, 21 Feb 2014 10:06:55 -0500

I am inline with Barbara's thinking on this point so if the current text
works then I am ok on this point as well.

Jason

On 2/21/14 9:56 AM, "STARK, BARBARA H" <[email-masked]> wrote:

>My remembrance was that we agreed that this *could* be a post collector
>activity, or it *could* be done by the MA and attached to reported
>results. And that lmap would not try to preclude or mandate how this
>would be accomplished, or define any method for acquiring the subscriber
>info. But I think that the Report does need to *allow* for inclusion of
>such information, or else it would effectively be precluding the
>MA-attachment model.
>
>Since you said you were ok with my suggested text, and my suggestion was
>to allow for this inclusion, I'm hoping we're ok.
>Barbara
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 21 Feb 2014 16:15:06 +0100

> My remembrance was that we agreed that this *could* be a post collector activity, or it *could* be done by the MA and attached to reported results. And that lmap would not try to preclude or mandat…

I am worried about the word 'attached' or 'attachment'. I would see
this as separate result records generated by a subscriber info
measurement task.

/js

---

From: <[email-masked]>
Date: Mon, 24 Feb 2014 09:59:23 +0000

Certainly the information model allows it. In the report header there is an optional section for context (e.g. line characteristics) and of course each measurement is also free to report such parameters in the result data columns.

Trevor.

Trevor Burbridge
Network Infrastructure & Innovation | BT Innovate & Design
Tel: 01473 645115
Fax: 01473 640929

This email contains BT information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you.
We monitor our email system, and may record your emails. 
British Telecommunications plc
Registered office: 81 Newgate Street London EC1A 7AJ
Registered in England no: 1800000 

>From: lmap [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
>Sent: 21 February 2014 14:56
>To: Eardley,PL,Philip,TUB8 R; [email-masked]; MORTON JR., AL;
>[email-masked]
>Cc: [email-masked]
>> Now I engage my brain, I remember (I think) that a while back we
>> agreed that enhancing measurement data with subscriber info was a post
>> collector activity.
>
>activity, or it *could* be done by the MA and attached to reported results. And
>that lmap would not try to preclude or mandate how this would be accomplished,
>or define any method for acquiring the subscriber info. But I think that the Report
>does need to *allow* for inclusion of such information, or else it would
>effectively be precluding the MA-attachment model.
>
>Since you said you were ok with my suggested text, and my suggestion was to
>allow for this inclusion, I'm hoping we're ok.
>Barbara
>

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Mon, 24 Feb 2014 10:15:52 +0000

I also think that this is important for a qualified test.   If one doesn't have the known service speed profile the test result shouldn't arbitrarily be compared to what one thinks the speed is - it should be compared to what the service said it is at the time of the test.   That way there is no variation in any test result conducted by two parties.   I.e.  A customer running a test vs. a ISP should result in the same answer.

This is why all those test environment attributes are so critical in making a reproducible test.   Without those recorded at the time of the test we we are left with having to guess what went wrong.   Keep in mind they don't weaken any test, but they do make the test result much stronger.

Mike

Sent from my iPhone

> On Feb 21, 2014, at 4:07 PM, "Weil, Jason" <[email-masked]> wrote:
>

---

From: <[email-masked]>
Date: Mon, 24 Feb 2014 10:27:25 +0000

Mike - re your comment - Just as a side note, please remember the extensive discussion we had many months ago about this point. However, it is not something I want to re-start.

> From: Bugenhagen, Michael K
> [mailto:[email-masked]]
> Sent: 24 February 2014 10:16
> To: Weil, Jason
> Cc: STARK, BARBARA H; Eardley,PL,Philip,TUB8 R; MORTON JR., AL;
> [email-masked]; [email-masked]
> I also think that this is important for a qualified test.   If one
> doesn't have the known service speed profile the test result shouldn't
> arbitrarily be compared to what one thinks the speed is - it should be
> compared to what the service said it is at the time of the test.   That
> way there is no variation in any test result conducted by two parties.
> I.e.  A customer running a test vs. a ISP should result in the same
> answer.
> 
> This is why all those test environment attributes are so critical in
> making a reproducible test.   Without those recorded at the time of the
> test we we are left with having to guess what went wrong.   Keep in
> mind they don't weaken any test, but they do make the test result much
> stronger.
> 
> Mike
> 
> Sent from my iPhone
> 
> > I am inline with Barbara's thinking on this point so if the current
> > text works then I am ok on this point as well.
> >
> > Jason
> >
> >>> agreed that enhancing measurement data with subscriber info was a
> >>> post collector activity.
> >>
> >> My remembrance was that we agreed that this *could* be a post
> >> collector activity, or it *could* be done by the MA and attached to
> >> reported results. And that lmap would not try to preclude or mandate
> >> how this would be accomplished, or define any method for acquiring
> >> the subscriber info. But I think that the Report does need to
> *allow*
> >> for inclusion of such information, or else it would effectively be
> >> precluding the MA-attachment model.
> >>
> >> Since you said you were ok with my suggested text, and my suggestion
> >> was to allow for this inclusion, I'm hoping we're ok.
> >> Barbara
> >>
> > This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or subject
> to copyright belonging to Time Warner Cable. This E-mail is intended
> solely for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient of this E-mail, you
> are hereby notified that any dissemination, distribution, copying, or
> action taken in relation to the contents of and attachments to this E-
> mail is strictly prohibited and may be unlawful. If you have received
> this E-mail in error, please notify the sender immediately and
> permanently delete the original and any copy of this E-mail and any
> printout.
> >

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Thu, 27 Feb 2014 13:00:36 +0000

So is the result that we have alignment between the BBF and IETF on the reporting of the test environment in the results  ?

I couldn't tell from your reply.

Mike

Sent from my iPhone

> On Feb 24, 2014, at 4:27 AM, "[email-masked]" <[email-masked]> wrote:
>