Subject: [lmap] FW: New Version Notification for draft-ietf-lmap-framework-03.txt

From: <[email-masked]>
Date: Tue, 21 Jan 2014 15:00:16 +0000

We submitted a new version of the framework document.

http://tools.ietf.org/html/draft-ietf-lmap-framework 

Here's a summary of the changes:
   o  alignment with the Information Model
      [I-D.burbridge-lmap-information-model] as this is agreed as a WG
      document

   o  One-off and periodic Measurement Schedules are kept separate, so
      that they can be updated independently

   o  Measurement Suppression in a separate sub-section.  Can now
      optionally include particular Measurement Tasks &/or Schedules to
      suppress, and start/stop time

   o  for clarity, concept of Channel split into Control, Report and MA-
      to-Controller Channels

   o  numerous editorial changes, mainly arising from a very detailed
      review by Charles Cook

best wishes,
phil, al, marcelo, trevor, paul, aamer

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 22 Jan 2014 10:32:52 +0000

Phil and co-authors,

Thanks for the submission and for the good work until now. 

Question - is this version ready for WGLC? If not, what's left to be done? 

Everybody's opinions are welcome. 

Regards,

Dan

> -----Original Message-----
> From: lmap [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: יום ג 21 ינואר 2014 17:00
> To: [email-masked]
> Subject: [lmap] FW: New Version Notification for draft-ietf-lmap-framework-
> 03.txt
> 
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: marcelo bagnulo braun <[email-masked]>
Date: Wed, 22 Jan 2014 11:57:46 +0100

IMO, this is ready for WGLC
Regards, marcelo

---

From: <[email-masked]>
Date: Wed, 22 Jan 2014 11:50:03 +0000

I agree

> From: lmap [mailto:[email-masked]] On Behalf Of marcelo bagnulo
> braun
> Sent: 22 January 2014 10:58
> To: [email-masked]
> Subject: Re: [lmap] FW: New Version Notification for draft-ietf-lmap-
> framework-03.txt
> 
> >> Subject: [lmap] FW: New Version Notification for
> >> draft-ietf-lmap-framework- 03.txt
> >>

---

From: <[email-masked]>
Date: Wed, 22 Jan 2014 11:54:27 +0000

Yes fine for me.

Trevor Burbridge

>From: lmap [mailto:[email-masked]] On Behalf Of [email-masked]
>Sent: 22 January 2014 11:50
>To: [email-masked]
>Subject: Re: [lmap] FW: New Version Notification for draft-ietf-lmap-framework-
>03.txt
>
>I agree
>
>> >>        [I-D.burbridge-lmap-information-model] as this is agreed as
>> >> a
>> WG
>> >>        document
>> >>
>> >>     o  One-off and periodic Measurement Schedules are kept
>> >> separate,
>> so

---

From: "Weil, Jason" <[email-masked]>
Date: Wed, 22 Jan 2014 14:37:19 -0500

Phil et al,

Overall I think the document is coming together nicely. I have 3 comments
after reading the latest version with chair hat removed:

Section 5.1 - It is not clear what role and scope the Initial Controller
plays. Is this a new device type/function - does it need to be described
somewhere other than a cursory glance that is presented here - or does
this represent a device that is part of the bootstrap operation that is
out of scope? It is not very clear here. Maybe rename to Bootstrap
Controller instead?

Section 5.2.1 & 5.3 These sections seem to contradict each other slightly.
Suppression seems to imply that the Controller is capable of actively
managing the MA. This includes the challenges stated in Section 6.2.2. I
am wondering if we should state optional or if required contingent on
transport protocol operation? Section 5.3 then goes on to say that the MA
acts autonomously once it receives the Measurement and Report Schedules
from the Controller. This sentence seems to contradict 5.2.1 operation.

Section 6.2 - "A single site (home, branch office etc.) that is
participating in a measurement could make use of one or multiple
Measurement Agents in a single measurement."
This concept needs more details if it is going to be included I believe.
How does a measurement task employ multiple MAs for a single test? Which
MA is responsible for reporting the results? Is there some kind of
master/slave function implied here or is Group-ID tie these devices
together? Would all the MAs participating in this test group need to be
associated with the same controller and does that controller need to be
aware of the group? Does the Collector need to be aware of the test group
as well?

Thanks,

Jason

>phil, al, marcelo, trevor, paul, aamer

This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.

---

From: <[email-masked]>
Date: Fri, 24 Jan 2014 11:46:08 +0000

Jason,

Thanks for your comments and careful reading
In-line
phil

> From: Weil, Jason [mailto:[email-masked]]
> Sent: 22 January 2014 19:37
> To: Eardley,PL,Philip,TUB8 R; [email-masked]
> Overall I think the document is coming together nicely. I have 3
> comments after reading the latest version with chair hat removed:
> 
> Section 5.1 - It is not clear what role and scope the Initial
> Controller plays. Is this a new device type/function - does it need to
> be described somewhere other than a cursory glance that is presented
> here - or does this represent a device that is part of the bootstrap
> operation that is out of scope? It is not very clear here. Maybe rename
> to Bootstrap Controller instead?

I think you're right that this section isn't very clear. 
To my mind, we're simply trying to say that the bootstrap process isn't one we're going to define but it needs to get the MA-id, maybe a group-id and details of the various channels. And here are some options about how to get the info on the MA. i think it would be better to scrap the picture - as it will be read as implying we're defining things that we aren't.

One other possibility that we should mention:
The bootstrapping information might be configured on the MA. 
(Marcelo mentioned this during our editing, sorry I forgot to include it) 

> managing the MA. This includes the challenges stated in Section 6.2.2.
> I am wondering if we should state optional or if required contingent on
> transport protocol operation? Section 5.3 then goes on to say that the
> MA acts autonomously once it receives the Measurement and Report
> Schedules from the Controller. This sentence seems to contradict 5.2.1
> operation.

You're right that the MA "acts autonomously" should have a supplement "until it gets a new Instruction (including perhaps an Instruction about Measurement Suppression)" 

I don't think suppression should be optional (to implement). It seems a very useful capability. 

I agree that if the MA is behind a NAT, then that presents issues for how lmap operates over the transport protocol - but I don't think they're different for suppression & the ordinary instruction. 

> How does a measurement task employ multiple MAs for a single test?
> Which MA is responsible for reporting the results? Is there some kind
> of master/slave function implied here or is Group-ID tie these devices
> aware of the group? Does the Collector need to be aware of the test
> group as well?

I think that's a mistake. It shouldn't mention the possibility of multiple MAs. 
If I remember a discussion quite a while ago, we agreed that if you want a multi agent test, then it would be up to you to define two Measurement Tasks and schedule them at the same time - but that wouldn't be a concern of lmap.

Thanks!
phil

> This E-mail and any of its attachments may contain Time Warner Cable
> proprietary information, which is privileged, confidential, or subject
> to copyright belonging to Time Warner Cable. This E-mail is intended
> solely for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient of this E-mail, you
> are hereby notified that any dissemination, distribution, copying, or
> action taken in relation to the contents of and attachments to this E-
> mail is strictly prohibited and may be unlawful. If you have received
> this E-mail in error, please notify the sender immediately and
> permanently delete the original and any copy of this E-mail and any
> printout.

---

From: Charles Cook <[email-masked]>
Date: Mon, 27 Jan 2014 13:32:52 -0700

Phil,

I am in the process of reviewing the draft.  I like how many things have
been improved.  Thanks for the good work.

That said, I need some clarity of certain terms and how they relate to
each other, because I am getting confused as I read through the draft.

The terms I am concerned about are:

  * Registry
  * Metric
  * Measurement Task
  * Measurement Method

It appears that Measurement Task and Measurement Method are sometimes
used synonymously, and sometimes they appear to be very different. 
Sometimes a Measurement Method appears to be a Metric with its
associated parameters not populated.  Sometimes Measurement Method seems
to refer to a Measurement Task with variables that have not been
populated.  Generally the term "method" is a synonym with "procedure",
which is generally how I interpret the term "method".  Maybe the use of
"method" rather than "procedure" is causing part of my confusion.  Maybe
part of my problem is that the Registry, its role, and what is stored in
the Registry is not well described. 

Here is what I thought was going on:

 1. A registry exists somewhere that contains a list of the standardised
    metrics of interest.
 2. The metric may be composed of some parameters that can be varied. 
    For example, a metric called "Throughput" may have parameters like,
    originating end point, terminating endpoint, traffic type (UDP/TCP),
    etc.
 3. A Measurement Task is the result of the Controller populating the
    variables of the standardised metric.
 4. A Measurement Method would define particular constraints that the
    measurements must be performed under in order for the Measurement
    Results to be comparable with other Measurement Results of the same
    Measurement Task performed at different times and/or locations. 
    This could be verification that there are sufficient resources to
    perform the Measurement Task (the MA has sufficient processing
    power), the path being measured is a direct path versus a derived
    path, a result that requires multiple iterations averaged together
    has confirmation that the same path is used for each iteration,
    etc.  Note that is it not enough for two Measurement Tasks to be
    comparable simply because their parameters are the same.  Certain
    external conditions must also be the same in order for the results
    to be comparable.

Here is what the draft appears to say:

 1. A registry exists somewhere  that contains a list of standardised
    Measurement Methods.
 2. Measurement Methods consist of specific actions that have parameters
    associated with them that have not been populated.
 3. A Measurement Task is the result of the Controller populating the
    parameters of the standardised Measurement Method.
 4. No discussion of what mechanism is used to ensure that Measurement
    Tasks conducted at different times and places can be comparable.

Is there a gap?  After we clarify what the intent is, I can re-read the
draft and suggest clarifying comments.

Thanks,
Charles

-- 

Charles Cook 
Principal Architect
Network
5325 Zuni Street; Suite 224
Denver, CO  80221
Tel:  303.992.8952  Fax:  925.281.0662
[email-masked]

---

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 27 Jan 2014 21:43:18 +0100

Hi Charles,

thanks for the comments.

In section 3 of the draft the following defintions are included:

    Measurement Method: The process for assessing the value of a Metric;
    the process of measuring some performance or reliability parameter;
    the generalisation of a Measurement Task.

    Measurement Task: The act that yields a single Measurement Result;
    the act consisting of the (single) operation of the Measurement
    Method at a particular time and with all its parameters set to
    specific values.

    Measurement Result: The output of a single Measurement Task (the
    value obtained for the parameter of interest or Metric).

    Metric: The quantity related to the performance and reliability of
    the network that we'd like to know the value of, and that is
    carefully specified.

Registry is not defined, nor metric registry nor entries of the registry.

So, my question is, do you find the above definitions confusing/inaccurate/incomplete, or do you believe that the terms are used with a different meaning in the draft or else?

I mean, i wonder if we should focus our effort in improving the defintion or how the terms are used in the body of the draft.

Regards, marcelo

El 27/01/14 21:32, Charles Cook escribió:
> Phil,
>
> I am in the process of reviewing the draft.  I like how many things 
> have been improved.  Thanks for the good work.
>
> associated parameters not populated.  Sometimes Measurement Method 
> seems to refer to a Measurement Task with variables that have not been 
> which is generally how I interpret the term "method".  Maybe the use 
> of "method" rather than "procedure" is causing part of my confusion.  
> Maybe part of my problem is that the Registry, its role, and what is 
> stored in the Registry is not well described.
>
>  1. A registry exists somewhere that contains a list of the
>     standardised metrics of interest.
>     For example, a metric called "Throughput" may have parameters
>     like, originating end point, terminating endpoint, traffic type
>     (UDP/TCP), etc.
>     Results to be comparable with other Measurement Results of the
>     same Measurement Task performed at different times and/or
>     locations.  This could be verification that there are sufficient
>     resources to perform the Measurement Task (the MA has sufficient
>     processing power), the path being measured is a direct path versus
>     a derived path, a result that requires multiple iterations
>     averaged together has confirmation that the same path is used for
>     each iteration, etc.  Note that is it not enough for two
>     Measurement Tasks to be comparable simply because their parameters
>     are the same.  Certain external conditions must also be the same
>     in order for the results to be comparable.
>
>  2. Measurement Methods consist of specific actions that have
>     parameters associated with them that have not been populated.
> Is there a gap?  After we clarify what the intent is, I can re-read 
> the draft and suggest clarifying comments.
>
> Thanks,
> Charles
>
>
>
>
>

---

From: Charles Cook <[email-masked]>
Date: Mon, 27 Jan 2014 13:53:48 -0700

Marcelo,

For the most part, I am finding that the terms are used with a different
meanings in the draft.  I also find use of the term "generalisation" to
be confusing.  However, I would focus more on how the terms are used in
the draft. 

Charles

On 1/27/2014 1:43 PM, marcelo bagnulo braun wrote:
> So, my question is, do you find the above definitions
> confusing/inaccurate/incomplete, or do you believe that the terms are
> used with a different meaning in the draft or else?
>
> I mean, i wonder if we should focus our effort in improving the
> defintion or how the terms are used in the body of the draft.
>
> Regards, marcelo
>
>
>
>
>
>> That said, I need some clarity of certain terms and how they relate
>> to each other, because I am getting confused as I read through the
>> draft.
>>
>> seems to refer to a Measurement Task with variables that have not
>> been populated.  Generally the term "method" is a synonym with
>> "procedure", which is generally how I interpret the term "method". 
>> Maybe the use of "method" rather than "procedure" is causing part of
>> my confusion.  Maybe part of my problem is that the Registry, its
>> role, and what is stored in the Registry is not well described.
>>

-- 

Charles Cook 
Principal Architect
Network

---

From: <[email-masked]>
Date: Tue, 28 Jan 2014 12:24:31 +0000

Charles,
Thanks for the comments!

> >>     Tasks conducted at different times and places can be comparable.

I think the above is basically correct.

> >> used synonymously, 
[phil] such uses are wrong

> >> and sometimes they appear to be very different.
> >> associated parameters not populated.  
[phil] such a use is wrong. A Measurement Method is how the value of a Metric is measured.

> >> Sometimes Measurement Method
[phil] I can sympathise with this. We're working on an overall draft describing a Registry for performance metrics. This will appear in IPPM WG soon (hopefully). 
This will be a registry of IETF registered metrics. An entry in the registry defines the Measurement Method.
However (at least in my mind) the operator of the measurement system may define their own registry with their own Measurement Methods. 

> >>     standardised metrics of interest.
[phil] the ietf-iana registry has the list, yes.

> >>     like, originating end point, terminating endpoint, traffic type
[phil] the Measurement Method has some variable parameters that don't affect the fundamental nature of the test. End points are good examples. Traffic type is not.
The Metric is the thing measured by the Measurement Method - the thing we want to know the value of. 

So if the Metric is "UDP One-way-packet delay variation 95% poisson", the Method might be "sections 2.4 and 3.4 of [RFC3393] with 200 bit packet, dscp=0, poisson distribution with lambda =1sec, output 95%-ile, <other details>"
	
> >>     variables of the standardised metric.
[phil] the Measurement Task is a particular measurement made using the Measurement Method with its Parameters set and at a particular time. Yes, the Controller tells the MA the Measurement Method, Parameters & the Schedule. 

> >>     locations.  
[phil] the Measurement Method should be well enough defined that Measurement Tasks (ie particular measurements that use the Measurement Method) produce Measurement Results that it is reasonable to compare

> >>     This could be verification that there are sufficient
[phil] it depends on how the Measurement Method is defined.
The Method could include a check that the end user isn't active, the path followed is the one expected etc, and if it fails then the Task doesn't send Active Measurement Traffic. Then the Result is "No Active Measurement Traffic sent due to xy"
On the other hand...	
The Method could include a check that the end user isn't active, the path followed is the one expected etc, but (whatever the result of the check) Active Measurement Traffic is sent. Then the Result is "Metric measured as 47, but caution - measurement made under xy conditions so Result may be suspect"

In both cases, it is reasonable to compare the Measurement Results, but in the second case the analysis needs to exercise more caution.

Of course, comparability may need some extra work. For example, a regulator comparing two operators would need to make sure that measurements were made at similar times (not busy hour vs off peak), on similar access lines (not ones next to exchange vs furthest point) etc. 

Re verification that the MA has sufficient processing power. I'd put this in a different category to your other checks. It should be reported as "Task not run - failed due to lack of MA processing power"; there is no Measurement Result. The operator of the measurement system should know the capabilities of the device with the MA, and shouldn't be anywhere near stretching its capabilities, unless there's some failure condition like the device is in the middle of crashing. 

Thanks
phil
> From: lmap [mailto:[email-masked]] On Behalf Of Charles Cook
> Sent: 27 January 2014 20:54
> To: marcelo bagnulo braun
> Cc: [email-masked]
> For the most part, I am finding that the terms are used with a
> different meanings in the draft.  I also find use of the term
> "generalisation" to be confusing.  However, I would focus more on how
> the terms are used in the draft.
> 
> Charles
> 
> a WG
> separate, so
> >> Tel:  303.992.8952  Fax:  925.281.0662 [email-masked]
> >>
> >>

---

From: Charles Cook <[email-masked]>
Date: Tue, 28 Jan 2014 07:02:11 -0700

Phil,

Thanks for the clarifications.  They are helpful.  I plan to re-read the
draft based on these clarifications.

Charles

On 1/28/2014 5:24 AM, [email-masked] wrote:
>>>> > >> It appears that Measurement Task and Measurement Method are

-- 

Charles Cook 
Principal Architect
Network