Subject: [lmap] Measurement Methods and Tasks

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 6 Jun 2014 13:48:47 +0000

Now that it appears we may have resolved the LMAP active/passive debates (by getting rid of the words) and ambiguity around "which tasks get suppressed by default", maybe we could agree on what Measurement Method and Measurement Task are?

I'd like to propose:
   Measurement Method: The process for assessing the value of a Metric;
   the process of measuring some performance or reliability parameter
   associated with the transfer of traffic; where this process involves multiple MAs or MPs, each may perform different roles.

   Measurement Task: The act that consists of the single operation of
   a Measurement Method role at a particular time and with all its Input
   Parameters set to specific values.

Measurement Method role is not a defined term, but can be used to express the capability implemented within a MA (or MP) that allows it to participate in a Measurement Method. I see that various ippm RFCs use the word "role" in this way, including OWAMP and TWAMP.

If there is a need to talk about an instance of executing a Measurement Method, this could be called a Measurement Method instance. Again, without definition as a formal term.

There exist several instances of "Measurement Method" that would need to become "Measurement Method role". For example:
   "A Measurement Task is a specific instantiation of a Measurement
   Method role."

   Capabilities: Information about the performance measurement
   capabilities of the MA, in particular the Measurement Method roles that it
   can perform, and the device hosting the MA, for example its interface
   type and speed, but not dynamic information.

"the Measurement Method roles that the MA supports"

But most discussion of Measurement Method in framework is of the Measurement Method and not the Measurement Method role. 

There is one instance where "Measurement Task capability" is used in the same sense as a Measurement Method role: "...Measurement Agents may come from
      different vendors, be in wired and wireless networks, have
      different Measurement Task capabilities and be on devices with
      IPv4 or IPv6 addresses."

I found one instance where I question the use of "Measurement Task" and wonder if it should be "Measurement Method":
   "Some Measurement Tasks involve several MAs acting in a coordinated
   fashion." 

In my scan of framework, I couldn't find a place where I thought Measurement Task was used to mean a Measurement Method instance.
Barbara

---

From: Sharam Hakimi <[email-masked]>
Date: Fri, 6 Jun 2014 14:50:58 +0000

Barbara,

I understand the intention but I do not think adding another definition will help in this case. Also a measurement task can be a single or a sequence of methods to achieve a single result(even though we have not defied a sequence yet). 

Sharam

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of STARK, BARBARA H
Sent: Friday, June 06, 2014 9:49 AM
To: [email-masked]
Subject: [lmap] Measurement Methods and Tasks

I'd like to propose:

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Charles Cook <[email-masked]>
Date: Fri, 06 Jun 2014 08:54:26 -0600

I agree with Barbara that we need to ensure that we have a clear
distinction between Measurement Method and Measurement Task. 

My view of the Measurement Method is that the Measurement Method is what
is included in a Registry somewhere.  The Measurement Method is
essentially a template of a particular type of measurement with a set of
parameters that can be specified.  A Measurement Task is an instance of
the Measurement Method where specific values have been specified for
each of the parameters in the Measurement Method.  Those that are closer
to defining what a Measurement Method is and what a Measurement Task is
can hopefully confirm or clarify as appropriate.

An MA will need to be able to report to the Controller that it is
registered to what capabilities it has relative to what Measurement
Methods it can support.  If a MA indicates it can support a particular
Measurement Method, that implies the MA can support the full range of
values that can be assigned by the Controller to the parameters of the
Measurement Method in the creation of a Measurement Task.  If the MA
cannot support the full range of values, then the MA cannot claim to
support the Measurement Method.

Even though it may not be within the scope of LMAP, I believe it would
be helpful if someone can propose a sample Measurement Method with all
its parameters as an example that we can use to validate whether the
LMAP framework is adequately complete.  Maybe a sample Measurement
Method can be derived from work in ippm (I have not participated in ippm
so I don't know if this can be done). 

A sample Measurement Method to exercise the LMAP framework should help
us to:
- determine whether all the bases have been covered in terms of what
functions the LMAP framework needs to support to ensure the execution of
meaningful Measurement Tasks, and
- ensure that a Measurement Method is sufficiently specified to enable
results from multiple executions of a Measurement Task generated from
the Measurement Method to be compared and analyzed in post processing in
a meaningful way.

Charles

-- 

Charles Cook 
Principal Architect
Network
5325 Zuni Street; Suite 224
Denver, CO  80221
Tel:  303.992.8952  Fax:  925.281.0662
[email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 6 Jun 2014 17:34:09 +0200

I vote for not using 'Measurement Method' anywhere in the LMAP
documents. If there is concensus this can't be done, then I think
Barabara's proposal is a start but not good enough for a definition.
In particular, I find the definition of 'Measurement Task' rather
unclear (what is a 'single operation or a .. role'? - this does not
help me understand what LMAP measurement tasks (as they are used in
the information model) really are.

But as said above, my preference is to leave 'Measurement Method'
reserved for IPPM folks. ;-)

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: KEN KO <[email-masked]>
Date: Fri, 6 Jun 2014 16:52:29 +0000

I support Barbara's proposal. IMO, if we define a Measurement Task as what is configured and executed by an MA - that is, dependent upon not only Measurement Method but the MA's role in implementing that Method - the language throughout the framework and information model documents becomes more consistent and unambiguous. Otherwise, a Measurement Task and a Measurement Task Configuration have different scopes and that seems confusing.

With regard to Juergen's preference that references to Measurement Method be deleted from the framework, I can't go that far. I think Measurement Methods as a concept are specified in many RFCs and (hopefully soon to be) referenced via registry entries, and that the framework should acknowledge this.

Ken

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
Sent: Friday, June 06, 2014 11:34 AM
To: STARK, BARBARA H
Cc: [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

I vote for not using 'Measurement Method' anywhere in the LMAP documents. If there is concensus this can't be done, then I think Barabara's proposal is a start but not good enough for a definition.
In particular, I find the definition of 'Measurement Task' rather unclear (what is a 'single operation or a .. role'? - this does not help me understand what LMAP measurement tasks (as they are used in the information model) really are.

/js

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 6 Jun 2014 16:55:39 +0000

> Barbara,
> 
> I understand the intention but I do not think adding another definition will
> help in this case. Also a measurement task can be a single or a sequence of
> methods to achieve a single result(even though we have not defied a
> sequence yet).
> 
> Sharam

I did not suggest adding another definition. I suggested changing the existing definition to better match the way the terms are used most of the time in lmap drafts, and not to use those terms in ways that do not match the definition.

The current information model draft version points to exactly one registry entry for any particular task. This implies that any sequence of events that needs to occur would need to be defined by the registry entry's method. Alternately, a schedule could have multiple tasks that are sequenced. Your view of a task that can be a sequence of methods is not supported by the current information model. I think the current information model is fine in this regard.
Barbara

> Now that it appears we may have resolved the LMAP active/passive debates
> (by getting rid of the words) and ambiguity around "which tasks get
> suppressed by default", maybe we could agree on what Measurement
> Method and Measurement Task are?
> 
> I'd like to propose:
>    associated with the transfer of traffic; where this process involves multiple
> MAs or MPs, each may perform different roles.
> 
> Measurement Method role is not a defined term, but can be used to express
> the capability implemented within a MA (or MP) that allows it to participate
> in a Measurement Method. I see that various ippm RFCs use the word "role"
> in this way, including OWAMP and TWAMP.
> 
> If there is a need to talk about an instance of executing a Measurement
> Method, this could be called a Measurement Method instance. Again,
> without definition as a formal term.
> 
> There exist several instances of "Measurement Method" that would need to
> become "Measurement Method role". For example:
> But most discussion of Measurement Method in framework is of the
> Measurement Method and not the Measurement Method role.
> 
> There is one instance where "Measurement Task capability" is used in the
> same sense as a Measurement Method role: "...Measurement Agents may
> come from
> I found one instance where I question the use of "Measurement Task" and
> wonder if it should be "Measurement Method":
> In my scan of framework, I couldn't find a place where I thought
> Measurement Task was used to mean a Measurement Method instance.
> Barbara
>

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 6 Jun 2014 17:21:01 +0000

Hi Charles,
I provided a UDP Echo example a week ago. Here is my breakdown of that example:
Measurement Method = UDP Echo
  role 1 = UDP Echo Server
     inputs = a port to listen on and a source IP address (MA1 will only respond to UDP packets from this source IP address)
     outputs = packets and bytes received, packets and bytes sent
  role 2 = UDP Echo Client
     inputs = packet size, destination IP address, time interval between UDP packets, and number of UDP packets
     outputs = average/max elapsed time between a sent UDP packet and the response, jitter, etc.

If a MA reported its capability to the Controller as "UDP Echo", this would be ambiguous, because it would mean the MA supported client, server, or both and the Controller has no way of knowing which. 

If the registry simply listed UDP Echo input parameters as listening port, source IP address, packet size, destination IP address, time interval between UDP packets, and number of UDP packets, then the Controller (who has no information as to which inputs belong to which role or even which role the MA supports) would need to send values for all of these input parameters to the MA and hope the MA can pick out the right ones? The Controller would have to tell the MA which role it needed the MA to perform and hope the MA is capable of that particular role.

I propose that the MA reported capabilities indicate the supported role(s) of a particular method, and that the registry entries indicate inputs and outputs per role.
An MA does *not* support UDP Echo. It supports UDP Echo Client or UDP Echo Server. If both, then the MA reports them to the Controller as separate capabilities.
Barbara

> From: Charles Cook [mailto:[email-masked]]
> Sent: Friday, June 06, 2014 10:54 AM
> To: STARK, BARBARA H; [email-masked]
> I agree with Barbara that we need to ensure that we have a clear distinction
> between Measurement Method and Measurement Task.
> 
> My view of the Measurement Method is that the Measurement Method is
> what is included in a Registry somewhere.  The Measurement Method is
> parameters that can be specified.  A Measurement Task is an instance of the
> Measurement Method where specific values have been specified for each of
> the parameters in the Measurement Method.  Those that are closer to
> defining what a Measurement Method is and what a Measurement Task is
> An MA will need to be able to report to the Controller that it is registered to
> what capabilities it has relative to what Measurement Methods it can
> support.  If a MA indicates it can support a particular Measurement Method,
> that implies the MA can support the full range of values that can be assigned
> by the Controller to the parameters of the Measurement Method in the
> creation of a Measurement Task.  If the MA cannot support the full range of
> values, then the MA cannot claim to support the Measurement Method.
> 
> Even though it may not be within the scope of LMAP, I believe it would be
> helpful if someone can propose a sample Measurement Method with all its
> parameters as an example that we can use to validate whether the LMAP
> framework is adequately complete.  Maybe a sample Measurement Method
> can be derived from work in ippm (I have not participated in ippm so I don't
> know if this can be done).
> 
> A sample Measurement Method to exercise the LMAP framework should
> help us to:
> results from multiple executions of a Measurement Task generated from the
> Measurement Method to be compared and analyzed in post processing in a
> meaningful way.
> 
> Charles
> 
> 
> 
> 
> > Now that it appears we may have resolved the LMAP active/passive
> debates (by getting rid of the words) and ambiguity around "which tasks get
> >    associated with the transfer of traffic; where this process involves
> multiple MAs or MPs, each may perform different roles.
> >
> > Measurement Method role is not a defined term, but can be used to
> express the capability implemented within a MA (or MP) that allows it to
> participate in a Measurement Method. I see that various ippm RFCs use the
> word "role" in this way, including OWAMP and TWAMP.
> >
> > There exist several instances of "Measurement Method" that would need
> to become "Measurement Method role". For example:

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Fri, 6 Jun 2014 17:38:14 +0000

> With regard to Juergen's preference that references to Measurement
> Method be deleted from the framework, I can't go that far. I think
> Measurement Methods as a concept are specified in many RFCs and
> (hopefully soon to be) referenced via registry entries, and that the
> framework should acknowledge this.
> 
> Ken

I would also prefer that framework continue to talk about Measurement Method. I do agree with Juergen that it should really be an ippm term. I would prefer if it were documented in one of the ippm drafts and just pointed to and used by framework. But that dependency might delay framework's publication, so that may not be a practical solution.
Barbara

---

From: Sharam Hakimi <[email-masked]>
Date: Fri, 6 Jun 2014 17:53:21 +0000

A measurement method could include multiple tasks and a task could include multiple methods. Therefore a single definition would be more appropriate. Also an MA only understands tasks and the method would only be applicable at the controller where tasks are defined and sent to the MA in my opinion.

Sharam

---

From: KEN KO <[email-masked]>
Date: Fri, 6 Jun 2014 18:04:21 +0000

First - I don't understand the statement "a task could include multiple methods." Can you provide a concrete example?

Second - How is the second sentence ("Therefore a single definition would be more appropriate.") justified by the first? Even if you accept the first sentence as a whole, I don't see how it leads to the second.

-----Original Message-----
From: Sharam Hakimi [mailto:[email-masked]] 
Sent: Friday, June 06, 2014 1:53 PM
To: STARK, BARBARA H; KEN KO; Juergen Schoenwaelder
Cc: [email-masked]
Subject: RE: [lmap] Measurement Methods and Tasks

Sharam

---

From: Sharam Hakimi <[email-masked]>
Date: Fri, 6 Jun 2014 18:21:39 +0000

A task can be a sequence and one could measure throughput for example with multiple methods to compare the results. 

Sharam

-----Original Message-----
From: Sharam Hakimi [mailto:[email-masked]] 
Sent: Friday, June 06, 2014 1:53 PM
To: STARK, BARBARA H; KEN KO; Juergen Schoenwaelder
Cc: [email-masked]
Subject: RE: [lmap] Measurement Methods and Tasks

Sharam

---

From: Charles Cook <[email-masked]>
Date: Fri, 06 Jun 2014 13:59:47 -0600

Barbara,

Thanks.  This is a good example, and I think I agree with you.  I agree
that there needs to be enough information to ensure the MA knows exactly
how to run a Measurement Task, and that Measurement Methods and
Measurement Tasks need to be sufficiently well defined to avoid any
ambiguity. 

Using "roles", as you suggest in the example, is one way it could be
accomplished.  Another way it could be accomplished would be to have two
Measurement Methods.   In your UDP Echo example you could have both a
UDP Echo Server Measurement Method, and a UDP Echo Client Measurement
Method. 

However, I think in this case because the UDP Echo Measurement Method
has two closely related but different components that need to be
orchestrated, I like your concept of "roles".  Maybe answer is if a
Measurement Method involves more than one MA or MP, then roles need to
be identified in the Measurement Method.  If the Measurement Method
specifies each role and the parameters for each role, then the
Controller can specify which MA is to perform which role when it
generates a Measurement Task.  This requires that the MA needs to report
to the Controller which roles it is capable of supporting. 

All,
This is exactly why I would like to see some sample Measurement Methods
to exercise the LMAP framework for completeness.   I think Barbara's UDP
Echo example alone has highlighted some ambiguity that needs to be
addressed in the LMAP framework that I did not recognized before.

Charles

-- 

Charles Cook 
Principal Architect
Network

---

From: Charles Cook <[email-masked]>
Date: Fri, 06 Jun 2014 14:03:12 -0600

Sharam,

My view is that Measurement Methods are the building blocks.  Generally
a Measurement Task is simply a Measurement Method where the parameters
have been filled in.  I do see the possibility of creating a Measurement
Task that is the combination of more than one Measurement Method.  But I
do not see the case where a Measurement Method can be composed of
Measurement Tasks.  Rather, I see a Controller creating multiple
Measurement Tasks from the Measurement Methods, and then chaining the
Measurement Tasks to be run in the proper order at the proper time.

Regarding a Measurement Task composed of multiple Measurement Methods, I
do not recall anywhere in the LMAP framework any discussion on how a
Measurement Task can be built out of multiple Measurement Methods.  The
way I interpret the LMAP framework is that if I need to run a
measurement that involves two or more Measurement Methods, I would need
to create a Measurement Task for each Measurement Method.  If we want to
be more complex than that, then additional work needs to be done
regarding how multiple Measurement Methods can be combined into a single
Measurement Task in such a way that the MA will have the capability to
run the new complex Measurement Task. 

Basically what I am saying is that I believe that an MA may have the
resources to run a Measurement Task generated from Measurement Method A
by itself and the MA may have the resources to run a Measurement Task
generated from Measurement Method B by itself, but the MA may not have
the resources to run a Measurement Task that combines Measurement Method
A and Measurement Method B. 

I'm not opposed to expanding the LMAP framework to address more complex
Measurement Tasks, but unless we have more definition on how multiple
Measurement Methods can be combined into a single Measurement Task
without exceeding the capabilities of the MA, my preference would be to
define additional Measurement Methods and maintain a one-to-one mapping
between Measurement Methods and Measurement Tasks.

Charles

-- 

Charles Cook 
Principal Architect
Network

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 6 Jun 2014 13:20:35 -0700

Hi Charles,
though discussion of what level of support of particular Measurement Method
by an MA is acceptable seems premature and more appropriate to discussion
of Control Protocol, I'd like to note that I cannot agree with your
suggestion: "If the MA
cannot support the full range of values, then the MA cannot claim to
support the Measurement Method."

Regards,
Greg

On Fri, Jun 6, 2014 at 7:54 AM, Charles Cook <[email-masked]>
wrote:

> suppressed by default", maybe we could agree on what Measurement Method and
> Measurement Task are?
> >
> > I'd like to propose:
> Method, this could be called a Measurement Method instance. Again, without
> definition as a formal term.
> >
> same sense as a Measurement Method role: "...Measurement Agents may come
> from

---

From: Charles Cook <[email-masked]>
Date: Fri, 06 Jun 2014 14:54:38 -0600

Greg,

It may be a question of what level of granularity does a MA need to
report its capabilities.  The simplest would be a scenario where the MA
reports a list of Measurement Methods that it supports upon registration
with the Controller.  If the MA is able to also report supported
parameter ranges of each supported Measurement Method, then this can be
effectively addressed.  Perhaps the LMAP framework can provide more
information on what level of detail the MA needs to report.

It may also depend on whether the MA simply informs the Controller of
its capabilities as a list, or whether there is a negotiation protocol
where the Controller can query the MA for additional details of its
capabilities.  For example, upon registration the MA can provide a list
of supported Measurement Methods.  Then at some later point, the
Controller through some protocol can query the MA relative to the range
that can be supported for each parameter in the Measurement Method of
interest. 

Because I am of the opinion that if MAs are to be implementable in
devices that could include modems and residential gateways at the
customer premises, cost becomes an issue.  In other words, I want the
ability to add MA functionality to a modem or residential gateway to be
a simple as possible, and the cost to be as low as possible.   So my
thoughts will generally come from the perspective of minimizing protocol
complexity.  Consequently, I had the list reporting approach upon
registration in mind.  But now that you mention it, I can see how the
LMAP framework can be interpreted to allow an MA to report that it can
support a given Measurement Method even though it may not support the
full range of valid values for each parameter. 

That said, I can back off of my original statement of, "If the MA cannot
support the full range of values, then the MA cannot claim to support
the Measurement Method."  However, I would like to see some text in LMAP
to elaborate on this topic.  I think this discussion also highlights the
need for the protocol work to be done so that LMAP can point to it.

Charles

On 6/6/2014 2:20 PM, Greg Mirsky wrote:
> Hi Charles,
> though discussion of what level of support of particular Measurement
> Method by an MA is acceptable seems premature and more appropriate to
> discussion of Control Protocol, I'd like to note that I cannot agree
> with your suggestion: "If the MA
> On Fri, Jun 6, 2014 at 7:54 AM, Charles Cook
> <[email-masked] <mailto:[email-masked]>>
> wrote:
>
>     debates (by getting rid of the words) and ambiguity around "which
>     tasks get suppressed by default", maybe we could agree on what
>     Measurement Method and Measurement Task are?
>     >
>     > I'd like to propose:
>     >    associated with the transfer of traffic; where this process
>     involves multiple MAs or MPs, each may perform different roles.
>     >
>     > Measurement Method role is not a defined term, but can be used
>     to express the capability implemented within a MA (or MP) that
>     allows it to participate in a Measurement Method. I see that
>     various ippm RFCs use the word "role" in this way, including OWAMP
>     and TWAMP.
>     >
>     > If there is a need to talk about an instance of executing a
>     Measurement Method, this could be called a Measurement Method
>     instance. Again, without definition as a formal term.
>     >
>     > There exist several instances of "Measurement Method" that would
>     need to become "Measurement Method role". For example:
>     > There is one instance where "Measurement Task capability" is
>     used in the same sense as a Measurement Method role:
>     "...Measurement Agents may come from
>     > I found one instance where I question the use of "Measurement
>     Task" and wonder if it should be "Measurement Method":
>     > [email-masked] <mailto:[email-masked]>
>     Tel:  303.992.8952 <tel:303.992.8952>  Fax:  925.281.0662
>     <tel:925.281.0662>
>     [email-masked] <mailto:[email-masked]>
>
>     [email-masked] <mailto:[email-masked]>

-- 

Charles Cook 
Principal Architect
Network

---

From: Greg Mirsky <[email-masked]>
Date: Fri, 6 Jun 2014 14:14:46 -0700

Hi Charles,
" I think this discussion also highlights the need for the protocol work to
be done so that LMAP can point to it."
wholeheartedly agree and enthusiastically support.

Regards,
Greg

On Fri, Jun 6, 2014 at 1:54 PM, Charles Cook <[email-masked]>
wrote:

>  Greg,
>
> It may be a question of what level of granularity does a MA need to report
> its capabilities.  The simplest would be a scenario where the MA reports a
> list of Measurement Methods that it supports upon registration with the
> Controller.  If the MA is able to also report supported parameter ranges of
> each supported Measurement Method, then this can be effectively addressed.
> Perhaps the LMAP framework can provide more information on what level of
> detail the MA needs to report.
>
> It may also depend on whether the MA simply informs the Controller of its
> capabilities as a list, or whether there is a negotiation protocol where
> the Controller can query the MA for additional details of its
> capabilities.  For example, upon registration the MA can provide a list of
> supported Measurement Methods.  Then at some later point, the Controller
> through some protocol can query the MA relative to the range that can be
> supported for each parameter in the Measurement Method of interest.
>
> Because I am of the opinion that if MAs are to be implementable in devices
> that could include modems and residential gateways at the customer
> premises, cost becomes an issue.  In other words, I want the ability to add
> MA functionality to a modem or residential gateway to be a simple as
> possible, and the cost to be as low as possible.   So my thoughts will
> generally come from the perspective of minimizing protocol complexity.
> Consequently, I had the list reporting approach upon registration in mind.
> But now that you mention it, I can see how the LMAP framework can be
> interpreted to allow an MA to report that it can support a given
> Measurement Method even though it may not support the full range of valid
> values for each parameter.
>
> support the full range of values, then the MA cannot claim to support the
> Measurement Method."  However, I would like to see some text in LMAP to
> elaborate on this topic.  I think this discussion also highlights the need
> for the protocol work to be done so that LMAP can point to it.
>
> Charles
>
>
>
> discussion of Control Protocol, I'd like to note that I cannot agree with
> your suggestion: "If the MA
> On Fri, Jun 6, 2014 at 7:54 AM, Charles Cook <
> [email-masked]> wrote:
>
>> > There is one instance where "Measurement Task capability" is used in
>> the same sense as a Measurement Method role: "...Measurement Agents may
>> come from
> Tel:  303.992.8952  Fax:  [email-masked]
>
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sat, 7 Jun 2014 09:52:48 -0400

Sorry to join in late, flying all day Friday...

I tend to agree that there are example scenarios where the 
"role" communicated to an MA resolves ambiguity.

I imagine a task where the role is TWAMP server and responder, and
the task is to open the well-know ports and listen for duration X, etc.
Of course, this MA and device may take on different roles in other tasks
if equipped to do so, and "role" specification resolves that ambiguity.

OTOH, another MA would be assigned to fill the roles of 
TWAMP Control-Client and Sender, and to initiate sessions with the
responder tasked above. Without roles, you could have two responders
listening for each other.

It may help to consider that the MA will likely coordinate with the 
local measurement control in its device, and measurement control
subsequently coordinates with local measurement functions to 
measure the registered metrics (and that's where the registry has
relevance, after roles are established for the measurement protocol).

What I wonder about at the moment is how the role might be expressed
in general, yet unambiguous to any measurement protocol available at the MA,
or whether the Controller would need to know the measurement protocols
available, specifying the one to use and the protocol-specific role.

Al
________________________________________
From: lmap [[email-masked]] On Behalf Of STARK, BARBARA H
Sent: Friday, June 06, 2014 1:21 PM
To: [email-masked]; [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

*** Security Advisory: This Message Originated Outside of AT&T ***.
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Hi Charles,

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 9 Jun 2014 12:35:53 +0000

> Now let me take the same UDP echo example with two "UDP echo clients"
> MAs, MA1 and MA2 simultaneously executing a UDP echo measurement
> with the same "UDP echo server" MA0.
> 
> 
> MA1 <---  UDP echo ------> | MA0 | <------ UDP echo ---------> MA 2
> 
> Here, we have two instances of the same measurement method.
> In each instance, the MA0 is playing the server role.
> 
> From the following, what is the correct answer:
> 
> a-  MA0 is executing two different tasks.
> 
> b- MA0 is executing only one global server task. The task consists in starting
> the UDP-
> echo server according to the controller scheduler.
> 
> c- MA0 is executing two instances of the same task.
> 
> 
> Answer a is not practical since MA0 has to have a different task for each
> possible
> client.
> 
> Answer b is the most suitable with the current lmap model, since a task is
> instructed by
> the controller and started according to some schedule. But this is not
> compatible with
> my understanding of "Barbara's tasks" (Here, it is kind of a Meta-task)
> 
> Answer c, means that MA0 has only one task that has been started twice by
> MA1 and MA2.
> This means that servers tasks cannot be scheduled like client tasks and need
> specific
> treatment.

Since Source IP Address is one of the task/role configured inputs, and MA1 and MA2 presumably have different source IP addresses, I think (a) is actually the right answer in a UDP Echo context. Note that the UDP Echo Server is expected to be on a CPE device, and the UDP Echo Client is expected to be somewhere in the network. UDP Echo Server is not something left continuously running that any old UDP Echo Client can test against. This is a tightly coordinated test.

But if we generalize this question, it's certainly possible to envision holistic measurement techniques where (b) would be the right answer -- for example something like an ICMP Echo (ping) server that is continuously running (i.e., the schedule has no end time) and tracks and reports statistics related to its ICMP Echo server activities. But where (b) is the right answer, I wouldn't expect there to be coordination between the <X> Echo Server and <X> Echo Client Measurement Tasks. They would likely be communicating with different Controllers. In fact the <X> Echo Server might even be in a MP and not MA. But I don't think it follows that this server activity isn't consistent with the way I conceptualize a Measurement Task. I think it would still be possible to configure this continuously running server as a single Measurement Task. There just wouldn't be any correlation between the Client and Server Measurement Tasks, and the role entries in the Registry would be very different.

I don't think I understand (c).
Barbara

---

From: <[email-masked]>
Date: Mon, 9 Jun 2014 17:50:36 +0100

So in the framework we say
<< The Instruction defines information with the following aims
   ([I-D.ietf-lmap-information-model] defines the consequent list of
   information elements):

   o  the Measurement Task configurations, each of which needs:

      *  the Metric, specified as a URI to a registry entry; it includes
         the specification of a Measurement Method.

Should this bullet refer to Measurement Method or Measurement Method role? Do we define a registry of Methods or Method roles?

Either is possible.
I'm not sure, but it's probably more consistent with ippm for it to be a registry of Methods. So then the Method needs an Input Parameter (or some other kind of configuration option) to choose between different roles (client, server...)

Thoughts?

> From: lmap [mailto:[email-masked]] On Behalf Of MORTON, ALFRED C
> (AL)
> Sent: 07 June 2014 14:53
> To: STARK, BARBARA H; [email-masked]; [email-masked]
> I tend to agree that there are example scenarios where the "role"
> communicated to an MA resolves ambiguity.
> 
> I imagine a task where the role is TWAMP server and responder, and the
> task is to open the well-know ports and listen for duration X, etc.
> Of course, this MA and device may take on different roles in other
> tasks if equipped to do so, and "role" specification resolves that
> ambiguity.
> 
> OTOH, another MA would be assigned to fill the roles of TWAMP Control-
> Client and Sender, and to initiate sessions with the responder tasked
> above. Without roles, you could have two responders listening for each
> other.
> 
> subsequently coordinates with local measurement functions to measure
> the registered metrics (and that's where the registry has relevance,
> after roles are established for the measurement protocol).
> 
> What I wonder about at the moment is how the role might be expressed in
> general, yet unambiguous to any measurement protocol available at the
> MA, or whether the Controller would need to know the measurement
> protocols available, specifying the one to use and the protocol-
> specific role.
> 
> Al
>      inputs = a port to listen on and a source IP address (MA1 will
> only respond to UDP packets from this source IP address)
>      inputs = packet size, destination IP address, time interval
> between UDP packets, and number of UDP packets
> If a MA reported its capability to the Controller as "UDP Echo", this
> would be ambiguous, because it would mean the MA supported client,
> server, or both and the Controller has no way of knowing which.
> 
> If the registry simply listed UDP Echo input parameters as listening
> port, source IP address, packet size, destination IP address, time
> interval between UDP packets, and number of UDP packets, then the
> Controller (who has no information as to which inputs belong to which
> role or even which role the MA supports) would need to send values for
> all of these input parameters to the MA and hope the MA can pick out
> the right ones? The Controller would have to tell the MA which role it
> needed the MA to perform and hope the MA is capable of that particular
> role.
> 
> I propose that the MA reported capabilities indicate the supported
> role(s) of a particular method, and that the registry entries indicate
> inputs and outputs per role.
> An MA does *not* support UDP Echo. It supports UDP Echo Client or UDP
> Echo Server. If both, then the MA reports them to the Controller as
> separate capabilities.
> Barbara
> 
> > essentially a template of a particular type of measurement with a set
> > of parameters that can be specified.  A Measurement Task is an
> > instance of the Measurement Method where specific values have been
> > specified for each of the parameters in the Measurement Method.
> Those
> > that are closer to defining what a Measurement Method is and what a
> > Measurement Task is can hopefully confirm or clarify as appropriate.
> >
> > Method can be derived from work in ippm (I have not participated in
> > ippm so I don't know if this can be done).
> >
> help
> > functions the LMAP framework needs to support to ensure the execution
> > of meaningful Measurement Tasks, and
> > the Measurement Method to be compared and analyzed in post processing
> > in a meaningful way.
> >
> > Charles
> >
> >
> >
> >
> > > involves
> > express the capability implemented within a MA (or MP) that allows it
> > to participate in a Measurement Method. I see that various ippm RFCs
> > use the word "role" in this way, including OWAMP and TWAMP.
> > >
> > > Measurement
> > > need
> > > the
> > Tel:  303.992.8952  Fax:  925.281.0662 [email-masked]
>

---

From: KEN KO <[email-masked]>
Date: Mon, 9 Jun 2014 16:56:17 +0000

> ... it's probably more consistent with ippm for it to be a registry of Methods. So then the ***Measurement Task Configuration*** needs an Input Parameter (or some other kind of configuration option) to choose between different roles (client, server...)

With that one (hopefully minor) tweak, +1

---

From: Charles Cook <[email-masked]>
Date: Mon, 09 Jun 2014 11:49:59 -0600

If I understand correctly, the Measurement Method defines all the
possible roles.  The Measurement Task generated by the Control sets an
Input parameter that selects the role the MA is to perform.

Charles

-- 

Charles Cook 
Principal Architect
Network

---

From: KEN KO <[email-masked]>
Date: Mon, 9 Jun 2014 18:12:28 +0000

Yes, that is how I interpret it.

-----Original Message-----
From: Charles Cook [mailto:[email-masked]] 
Sent: Monday, June 09, 2014 1:50 PM
To: KEN KO; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

If I understand correctly, the Measurement Method defines all the possible roles.  The Measurement Task generated by the Control sets an Input parameter that selects the role the MA is to perform.

Charles

On 6/9/2014 10:56 AM, KEN KO wrote:
>> ... it's probably more consistent with ippm for it to be a registry 
>> of Methods. So then the ***Measurement Task Configuration*** needs an 
>> Input Parameter (or some other kind of configuration option) to 
>> choose between different roles (client, server...)

-- 

Charles Cook
Principal Architect
Network

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 9 Jun 2014 17:34:25 -0400

Charles,

When you say, 
"...the Measurement Method defines all possible roles."
it may be true for the details of making measurements.
But I think there are other roles which are equally 
important to establish - measurement control protocol roles
and test session protocol roles.

One of the example figures in the Framework covers OWAMP:

      +----------------+    OWAMP     +----------------+    ^
      | OWAMP          |<--control--->| MP:            |    |
      | control-client |>test-traffic>| OWAMP server & |   IPPM
      | fetch-client & |<----fetch----| session-rec'ver|  Scope
      | session-sender |              |                |    |
      |                |              +----------------+    |
   ...|................|....................................v...
      | LMAP interface |                                    ^
      +----------------+                                    |

The protocol roles, such as control-client <---> server,
need to be specified and communicated from the LMAP MA
to the control-client interface, the device on the left
above is not a session-sender by default, for example.

In the Measurement Methods written by IPPM, there will 
typically be the roles of Source and Destination, and these
are the only set of roles that would normally appear.
This is the material that will appear in the Registry for
Performance metrics - but there won't be any discussion
of OWAMP-specifics, just packet measurement processes.

So,  both Protocol roles and Measurement roles are needed,
and only one can be found in the registry.

regards,
Al

> Sent: Monday, June 09, 2014 2:12 PM
> To: [email-masked]; [email-masked]; MORTON, ALFRED C
> (AL); STARK, BARBARA H; [email-masked]
> If I understand correctly, the Measurement Method defines all the possible
> roles.  The Measurement Task generated by the Control sets an Input
> parameter that selects the role the MA is to perform.
> 
> Charles
>

---

From: Charles Cook <[email-masked]>
Date: Mon, 09 Jun 2014 17:02:53 -0600

I agree. 

My comment was only addressing the details of making measurements.

Charles

-- 

Charles Cook 
Principal Architect
Network

---

From: <[email-masked]>
Date: Tue, 10 Jun 2014 09:44:17 +0100

So in terms of the text in the section about Control Protocol - Instruction.
Finding it quite tricky to get the right words...
Al, haven't managed to capture your point about protocol roles.

<<
  The Instruction defines information with the following aims
   ([I-D.ietf-lmap-information-model] defines the consequent list of
   information elements):

      *  the Metric, specified as a URI to a registry entry; it includes
         the specification of a Measurement Method.  The registry could
         be defined by the IETF [I-D.manyfolks-ippm-metric-registry],
         locally by the operator of the measurement system or perhaps by
         another standards organisation.

[new] *  the Measurement Method role. For some Measurement Methods, different parties play different roles; for example (figure A3 in the Appendix) an iperf sender and receiver. Depending on how the registry is defined, the different roles could be different Measurement Methods or the same Measurement Method with different Input Parameters. 

      *  any Input Parameters that need to be set for the Measurement
         Method, such as the address of a Measurement Peer (or other
         Measurement Agent) that may be involved in a Measurement
         Task
[above bullet tweaked to remove 'Active']

      *  if the device with the MA has multiple interfaces, then the
         interface to use (if not defined, then the default interface is
         used)

In the appendix, add to the para about figure A3
<<The example also illustrates that there may be different roles involved in a Measurement Method, since MA1 and MA2 operate different Measurement Tasks. 

Maybe something could be added about A2, covering different protocol roles? (although note that the A2 example involves only one MA, so less clear it really involves two Measurement Method roles)

Thanks
phil

> From: MORTON, ALFRED C (AL) [mailto:[email-masked]]
> Sent: 09 June 2014 22:34
> To: KEN KO; [email-masked]; Eardley,PL,Philip,TUB8 R;
> STARK, BARBARA H; [email-masked]
> But I think there are other roles which are equally important to
> establish - measurement control protocol roles and test session
> protocol roles.
> 
> The protocol roles, such as control-client <---> server, need to be
> specified and communicated from the LMAP MA to the control-client
> interface, the device on the left above is not a session-sender by
> default, for example.
> 
> In the Measurement Methods written by IPPM, there will typically be the
> roles of Source and Destination, and these are the only set of roles
> that would normally appear.
> This is the material that will appear in the Registry for Performance
> metrics - but there won't be any discussion of OWAMP-specifics, just
> packet measurement processes.
> 
> So,  both Protocol roles and Measurement roles are needed, and only one
> can be found in the registry.
> 
> regards,
> Al
> 
> > To: [email-masked]; [email-masked]; MORTON,
> > ALFRED C (AL); STARK, BARBARA H; [email-masked]
> > >> of Methods. So then the ***Measurement Task Configuration*** needs
> > >> an Input Parameter (or some other kind of configuration option) to

---

From: <[email-masked]>
Date: Tue, 10 Jun 2014 13:54:14 +0100

Can I suggest a mod:
Measurement Task: The action performed by a particular Measurement Agent that consists of the single operation of the Measurement Method at a particular time and with all its Input Parameters set to specific values.  

Ie if a Method involves two MAs then a particular Task is what one MA does (and the other MA needs to be instructed with another Task)

Ps Barbara - thanks for these clarifications

phil 
      
> Sent: 06 June 2014 14:49
> debates (by getting rid of the words) and ambiguity around "which tasks
> get suppressed by default", maybe we could agree on what Measurement

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 10 Jun 2014 13:01:22 +0000

I agree as well.

    In between this we have the "MA behavior to environment" as well that won't be covered by the IPPM test registry.
I think that needs to be discussed in the next f2f meeting on both sides of the fence.

I haven't seen anyone propose how to close that issue in a "standard protocol" method or approach, or a testing standard state approach beyond "watch for cross traffic" wording.

-M

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Cook, Charles
Sent: Monday, June 09, 2014 6:03 PM
To: MORTON, ALFRED C (AL); KEN KO; [email-masked]; STARK, BARBARA H; [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

I agree. 

Charles

On 6/9/2014 3:34 PM, MORTON, ALFRED C (AL) wrote:
> In the Measurement Methods written by IPPM, there will typically be 
> the roles of Source and Destination, and these are the only set of 
> roles that would normally appear.
> So,  both Protocol roles and Measurement roles are needed, and only 
> one can be found in the registry.
>
> regards,
> Al
>
>> possible roles.  The Measurement Task generated by the Control sets 
>> an Input parameter that selects the role the MA is to perform.
>>
>> Charles
>>

-- 

Charles Cook
Principal Architect
Network

---

From: KEN KO <[email-masked]>
Date: Tue, 10 Jun 2014 13:14:32 +0000

Mike,

You bring up an interesting point. 

One approach would be to include environmental options in the formal definition of the Measurement Method.

Another approach would be to define a set of environmental options independent of Measurement Method (another registry???) that can be set as optional input parameters in any Measurement Task Configuration.

I hadn't thought about this before, but it seems to me that the second approach has merit. Comments?

Ken

-----Original Message-----
From: Bugenhagen, Michael K [mailto:[email-masked]] 
Sent: Tuesday, June 10, 2014 9:01 AM
To: Cook, Charles; MORTON, ALFRED C (AL); KEN KO; [email-masked]; STARK, BARBARA H; [email-masked]
Subject: RE: [lmap] Measurement Methods and Tasks

I agree as well.

-M

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Cook, Charles
Sent: Monday, June 09, 2014 6:03 PM
To: MORTON, ALFRED C (AL); KEN KO; [email-masked]; STARK, BARBARA H; [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

I agree. 

Charles

-- 

Charles Cook
Principal Architect
Network

---

From: KEN KO <[email-masked]>
Date: Tue, 10 Jun 2014 13:18:21 +0000

Sorry if this is also a duplicate - I had to reset my membership in the lmap list.

Mike,

Ken

-----Original Message-----
From: Bugenhagen, Michael K [mailto:[email-masked]]
Sent: Tuesday, June 10, 2014 9:01 AM
To: Cook, Charles; MORTON, ALFRED C (AL); KEN KO; [email-masked]; STARK, BARBARA H; [email-masked]
Subject: RE: [lmap] Measurement Methods and Tasks

I agree as well.

-M

-----Original Message-----
From: lmap [mailto:[email-masked]] On Behalf Of Cook, Charles
Sent: Monday, June 09, 2014 6:03 PM
To: MORTON, ALFRED C (AL); KEN KO; [email-masked]; STARK, BARBARA H; [email-masked]
Subject: Re: [lmap] Measurement Methods and Tasks

I agree. 

Charles

-- 

Charles Cook
Principal Architect
Network

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 10 Jun 2014 13:41:28 +0000

> Can I suggest a mod:
> Measurement Task: The action performed by a particular Measurement
> Agent that consists of the single operation of the Measurement Method at a
> particular time and with all its Input Parameters set to specific values.
> 
> Ie if a Method involves two MAs then a particular Task is what one MA does
> (and the other MA needs to be instructed with another Task)

I find the proposed definition confusing. I'm assuming that you're using Measurement Method to identify a holistic technique (because Charles had suggested it could be used to identify a specific role of a holistic technique)? A single operation of a holistic technique that has multiple roles would involve all necessary roles. So, IMO,  
"the action that consists of the single operation of the Measurement Method at a particular time" would involve all necessary roles (all MAs participating in this action).
It's just not possible for a single MA to perform an action that equates to a single operation of a multi-MA Measurement Method.

I agree with your 2nd sentence, but the proposed definition doesn't say this at all.

Is there a reason why the word "role" is being avoided? I see this word used in several ippm RFCs. 

I would prefer:
Measurement Task: The action performed by a particular Measurement
Agent that consists of the single operation of a Measurement Method role at a
particular time and with all of the role's Input Parameters set to specific values.

Alternately, we could go with Charles' approach and define "Measurement Method" as "the role of a participating point in a holistic measurement technique". Then the proposed definition would be accurate. Personally, I think this may add to confusion, because so many people already consider "Measurement Method" to be the holistic technique. 
Barbara

---

From: KEN KO <[email-masked]>
Date: Tue, 10 Jun 2014 15:20:09 +0000

> Agent that consists of the single operation of a Measurement Method
> role at a particular time and with all of the role's Input Parameters set
>  to specific values.

+1 to the above.

> Alternately, we could go with Charles' approach and define
> "Measurement Method" as "the role of a participating point in a
> holistic measurement technique". Then the proposed definition
> would be accurate. Personally, I think this may add to confusion,
> because so many people already consider "Measurement Method"
> to be the holistic technique. 

I agree that this terminology would be confusing. I prefer the first alternative above.

Ken

---

From: Charles Cook <[email-masked]>
Date: Tue, 10 Jun 2014 14:07:37 -0600

+1

-- 

Charles Cook 
Principal Architect
Network

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 11 Jun 2014 18:44:23 -0700

Hi Phil,
not sure whether in our discussion of Active/Passive Measurement methods in
LMAP Framework we've looked at how proposed changes get reflected in Figure
1 in Section 2. If we have agreed that LMAP Framework should be transparent
whether Measurement Method/Task is Active or Passive, then I'd suggest to
remove 'Active' from 'Active Measurement Traffic'. I think that
'Measurement Traffic' will correctly characterize data flow between MA and
MP.

Regards,
Greg

On Tue, Jun 10, 2014 at 1:07 PM, Charles Cook <[email-masked]