Subject: [lmap] starting work description discussion

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 16 Oct 2012 07:57:27 +0200

Hi,

Irrespectively whether this work takes place in a new WG, a new RG or in 
the IPPM WG, i guess we should start the discussion on the work description.
This is my input on how the work description should look like, based on 
the draft and the discussion so far.

Comments are welcome.

Regards, marcelo

LMAP Work description

Over the last few years, we have witnessed the deployment of large 
measurement platforms that enable measurements from a vast set of 
vantage points. Examples of these platforms include SamKnows and RIPE 
ATLAS, to name a few. Such platforms integrate a large number (in the 
orders of tens of thousands or more) of measurement agents. These 
measurement agents are a piece of code that is executed either dedicated 
hardware or in the end user equipment (e.g. computer or mobile device). 
A large fraction of these measurement agents are usually located in the 
end-user premises and they can run measurements against other user 
agents that are located in strategic locations according to the desired 
measurements to be performed (typically servers in the core of the network).
Thanks to the large number of measurement agents, these platforms can 
provide data about key indicators of the network performance from the 
end user perspective. These data is useful to network operators to 
improve their operations as well to regulators and to end users themselves.

In addition to the measurement agents, there are other elements in these 
large measurement platforms, including (but not limited to):
- The Controller that informs the measurement agents about what 
measurements to perform and when to do them.
- The Data collector that collects the results of the measurements 
performed by the measurement agents (along with additional information 
such as measurement agent identification and time stamp).

Currently deployed platforms use proprietary protocols to exchange 
information between the different parts. As these platforms grow to 
become an important tool to understand network performance, it is 
important to standardize the protocols between the different elements of 
the platform.

The primary work items are:

- An architecture document (informational document) that describes the 
different components of the measurement platform and their interactions.

- A use cases document (informational) that describe the main use cases 
for these large scale measurement.

- One or more protocol documents (Standard track) that specify at least 
the following interactions: (i) The protocol between the measurement 
agent and the controller, where the controller can specify the 
measurement to be executed by the agent and their schedule. (ii) the 
protocol between the measurement agent and the data collector through 
which the agent can report the results of the measurements performed and 
provide additional data (agent identification, time stamp). The 
protocols used to actually perform the measurements are out of the scope 
of this effort and can be done in the IPPM WG (this should disappear if 
this becomes part of the IPPM WG)

- A document describing data ownership and privacy guidelines. As these 
platforms collect a large amount of data, data management is a critical 
element.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Tue, 16 Oct 2012 08:32:05 +0200

Marcelo,

let me play devil's advocate a bit here...

> important to standardize the protocols between the different
> elements of the platform.

[...]

> - One or more protocol documents (Standard track) that specify at
> least the following interactions: (i) The protocol between the
> measurement agent and the controller, where the controller can
> specify the measurement to be executed by the agent and their
> schedule. (ii) the protocol between the measurement agent and the
> data collector through which the agent can report the results of the
> measurements performed and provide additional data (agent
> identification, time stamp). The protocols used to actually perform
> the measurements are out of the scope of this effort and can be done
> in the IPPM WG (this should disappear if this becomes part of the
> IPPM WG)

Taking RIPE Atlas and Sam Knows as examples, I wonder whether we
really see the need for a standard allowing RIPE probes to interwork
with Sam Knows and vice versa. Is this a realisitic target?  How many
such large measurement platforms do we expect to be implemented, do we
expect an ecosystem of probe suppliers that provide hardware for
different large scale measurement platforms? And will an attempt to
standardize the protocol between the probes and the
controller/collector not potentially hinder evolution?

Perhaps it makes more sense to look at standards for protocols that
allow systems to access the collected measurement data, taking into
account the need to protect data and perhaps to inject additional
measurement requests into the control plane of measurement
networks. Having an open interface to access data collected by large
scale measurement systems may open up new opportunities to use the
data by systems that can influence (directly or indirectly) how
networks are operated.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 16 Oct 2012 08:50:25 +0200

Hi Juergen,

below...

> controller/collector not potentially hinder evolution?

My view on this is the following: I would expect that the user agents 
are either embedded in the CPE (home router) from the vendor, or it can 
be dowloaded to the end user device (e.ge standard linux app or whatever).

Suppose that home router have this measurement agent already built in. 
then deployment of very large platforms would be much easier, you should 
simply rely on this existent capability.

So, I do believe there is value on standardizing this and i do believe 
this would make easier to deploy these platforms.

> data by systems that can influence (directly or indirectly) how
Right, i agree with tis (actually i had written in the initial version 
of the DoW a work item on this, but then i felt it maight be better to 
have a more narrow scope.

I mean, i find that relevant and interesting, i am concerned about 
having a too broad/large amount of work.

Regards, marcelo

---

From: Robert Kisteleki <[email-masked]>
Date: Tue, 16 Oct 2012 09:38:29 +0200

Hi,

> controller/collector not potentially hinder evolution?

This is spot on. I cannot imagine how one could realistically manage
multiple control infrastructures for the same set of measurement devices.
Speaking for Atlas only: if there was a potential second (third, ...) source
of measurement tasks, we would lose our ability to manage measurement
capacities, bandwidth, scheduling on the measurement nodes.

> data by systems that can influence (directly or indirectly) how

I think this is a much more interesting approach. It focuses on how to
interact with such a system and how to transform the results into a common
format. The combination of the two allows for users to leverage the power of
the different measurement networks, yet leaves the implementation details to
those networks.

It's also relatively simple to implement: one only needs to translate the
control messages and result data to/from the common format.

Robert
(who is somewhat involved in the making of RIPE Atlas)

---

From: Brian Trammell <[email-masked]>
Date: Tue, 16 Oct 2012 11:00:36 +0200

Hi, all,

catching up on the backlog, so I'm following a couple of threads at once here, inline. first from last week:

On Oct 10, 2012, at 6:05 PM, Mirja Kühlewind wrote:

> 3. There might also be some kind of hierarchy between data collectors. If I 
> use more than one data collector for load balancing reasons. I guess those 
> single collectors could do some kind of preprocessing but then I still need 
> one entity to correlate all data to get a view on my whole network. Thus 
> maybe a protocol to query the data should be regarded as well. Btw. there can 
> already be some kind of preprocessing at the measurements endpoint 
> (client/server) as well. 

Yep. I presume you'll probably end up with mediators in most large scale deployments, either to handle load from widely distributed measurements at collectors, or to consolidate interdomain data exchange (i.e. each operator has a first-tier collector, which then sends results to a central collector after applying any data inspection/aggregation/etc necessary.) 

One question in this case would then be one of scope: would the interactions behind the mediator then be subject to control by LMAP, or does LMAP's responsibility stop with the first tier collector? Is it a protocol for controlling simple measurements, or for producing answers by controlling a hierarchy of devices end to end?

> -> All over we might only need two protocols. One to request or initiate 
> measurements and one to request or exchange measurement data. 

+1.

I'd go further and say it probably makes sense to try and handle measurement data exchange (from measurement point to collector) using existing protocols, when they exist -- if I'm doing passive measurement based on flows, for example, and I have an LMAP-enabled flow meter that speaks IPFIX and a collector that speaks IPFIX, it makes sense to just have them talk to each other using the protocol they already know, as opposed to doing transcoding on both ends. Same goes for OWAMP/TWAMP, though they do control and data export over the same protocol, so that might need to be split somehow to fit LMAP as specified.

In cases for supporting measurement for which there's no existing export protocol, it's not clear what to do, though.

> The first one can be used between the controller and measurement endpoint, as 
> well as between measurement endpoints and between controllers in a 
> hierarchical structure.

Not to do design before we know all the requirements... :) but it seems to me like this protocol would just be a way to pass "measurement specifications" around, with all the parameters required: what to measure, how to measure it, and who to send the results to, with a . Controllers could split these up and send them down to subordinate controllers as they see fit.

As I noted in an earlier comment, I think it makes sense for the protocol for exchanging these specifications be initiator-independent -- it makes sense to have (some subset of) measurement points that can do on demand active measurement, without waiting for a polling interval, for cause analysis or differentiation during troubleshooting. These would probably be owned by the operator, as compared to measurement points closer to the edge, who might be owned by customers and therefore should pull their measurement specifications, both to allow customer control of the measurement as well as to work around NAT. It also makes sense, if we support a hierarchy of controllers, to allow superordinate controllers to push specifications to their subordinates.

> The second one can be used between the measurement point and the data 
> collector (where the measurement might also be able to store and evaluate the 
> data) and as well between measurement points and data collectors and also 
> between the controller and the data collector/measurement endpoints (as the 
> controller might need previous results to configure new measurement runs).

> Both protocols will also need some kind of capacity negotiation as well as 
> some kind of authentication and integrity check.

Hm... what do you mean by capacity negotiation here? In terms of the measurement load on the network, the number of measurements a measurement point is willing to handle concurrently, the amount of data the collector is willing to accept? Actually -- all of these will need to be handled, I think...

and now for this morning's thread:

> capacities, bandwidth, scheduling on the measurement nodes.

First, I think (as noted above) that the measurement point -> collector protocol is out of scope; there are already protocols for doing this, each meeting the requirements of particular measurements, and there's no need either to reinvent the wheel or to put a lot of effort into writing transcoders for data that will never leave the probe->collector pipe.

I also don't think the right approach is to build multiple control infrastructures; instead, if we consider a "measurement specification" as the unit of LMAP control (also as above), these would be an additional source of instructions, that could be translated into instructions and configuration for the existing control interface, subject to local constraints, such as measurement traffic capacity, or even restrictions of the functions available via LMAP as opposed to via the "native" control interface.

>> data by systems that can influence (directly or indirectly) how

A measurement specification could also be seen as a query, of sorts. I.e., you'd send an LMAP measurement specification to a measurement point (or it would retrieve a specification, in the pull model) to start a running measurement, and you'd send the same specification (perhaps with a different temporal scope) to a collector to retrieve information already collected. 

> format.

+1.

I think there are two separate parts to this common format: one would be the semantic definition of what each measurement of set of measurements means, and the other would be the actual representation. I think we should probably aim to separate the data model from the formatting, both to keep from getting caught up in a JSON-vs-XML-vs-ASN.1 argument (as fun as those are), and because the data model is the more interesting problem, or rather, the one which is most critical to get right.

> The combination of the two allows for users to leverage the power of

Once the hard decisions are made about the semantic meaning of the data (and how data from one or another measurement system might need to be transformed to meet that definition), I agree. For basic measurement, the IPPM metrics are an obvious choice as a starting point (as, indeed, mentioned in the lmap draft), though we'll need comparable methods for passive measurement of these metrics; see http://tools.ietf.org/html/draft-trammell-ippm-hybrid-ps-00.html for the very beginning of a problem statement on the subject.

Best regards,

Brian

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 16 Oct 2012 14:24:47 +0200

below....

> Yep. I presume you'll probably end up with mediators in most large scale deployments, either to handle load from widely distributed measurements at collectors, or to consolidate interdomain data ex…

MB> that makes sense. It is not obvious to me whether the protocol needs 
to be different for the measurement clietns and the mediators though. 
Probably at this point we should leave it open and make sure that the 
work description allows for both.

> One question in this case would then be one of scope: would the interactions behind the mediator then be subject to control by LMAP, or does LMAP's responsibility stop with the first tier collector…

MB> see above. I mean, i guess this should be left for discussion for 
the actual work. It is not obvious to me how significantly different 
should these two protocol be.

> In cases for supporting measurement for which there's no existing export protocol, it's not clear what to do, though.

MB> So, my understanding is that current platforms support an extensive 
set of active measruements that go beyond OWAMP/TWAMP. So, i think more 
than these two protocols will be needed if we want to support this.
I guess the question at this point is how generic a protocol for 
exchanging measurement results can be when it is not tied to a specfic 
measurement.
I mean, we could define a protocol (or reuse a protocol) that is a 
general envelope to exchange results from the measurement agent to the 
data collector, that is generic irrespectively to the actual test being 
reported. Then for each specific tests, we should add the specific data 
fields that the report should contain.
the generic envelope would carry information about the measurement agent 
identification, time stamp, and additional security data. I am uncertain 
whether this has value, but i guess it may.
the test specific part shoudl be defined for each test whose results is 
being reported and it should contain specific fields for the specific test.
I agree that we should reuse existing protocols. Do you know an exisitng 
protocol that could serve as envelope protocol?

> Not to do design before we know all the requirements... :) but it seems to me like this protocol would just be a way to pass "measurement specifications" around, with all the parameters required: w…

MB> I think this is one place where the IETF could add value. We should 
define a registry for measurements that can be perfoemed by the clients 
and for each of them to have a code identifying the tests, and the set 
of parameters that are needed as input and the oputput that is needs to 
be reported.
then this protocol should convey to the measurement agent the code for 
the measurement as defined in the registry as well as the input parameters.
(I have seen that the IPPM metrcis registry, that probably looks a bit 
like this has been deprecated, I wonder if there is something to learn 
from that experience...)

...
> First, I think (as noted above) that the measurement point -> collector protocol is out of scope; there are already protocols for doing this, each meeting the requirements of particular measurement…

MB> see above about the envelope protocol. Besides, you also point that 
there are some cases where another protocol is needed, so i am kind of 
confused by your summary here.
My understanding is that we could either define one specific protocol 
for each type of measurement we are going to support (i understand this 
is what OWAMP and TWAMP are), but we probably support more tests.
I believe this envelope protocol would provide an extensible framework 
to define then the specific tests.

>
> I also don't think the right approach is to build multiple control infrastructures;

MB> as i mentioned to my reply to Juergen, i think there is much more 
value in standardizing the interfaces other than supporting multiple 
control infrastrucutres. For example, the possibility that of the shelf 
equipment already supports the measurements and can then be integrated 
into a measurment platform without requiring additional code (e.g. 
suppose the home routers already support LMAP, they could then be 
integrated into a platform without additional deployment burden)

Regards, marcelo

---

From: Brian Trammell <[email-masked]>
Date: Wed, 17 Oct 2012 15:54:43 +0200

hi, Marcelo, all,

inline per tradition...

On Oct 16, 2012, at 2:24 PM, marcelo bagnulo braun wrote:

> MB> that makes sense. It is not obvious to me whether the protocol needs to be different for the measurement clietns and the mediators though. Probably at this point we should leave it open and make sure that the work description allows for both.
> 
> MB> see above. I mean, i guess this should be left for discussion for the actual work. It is not obvious to me how significantly different should these two protocol be.

Agreed on both these points... actually, if we think in terms of measurement specifications, the distinction of the position of a particular entity within a measurement infrastructure -- whether it's a mediator, or a final collector, or a measurement point, or whatever -- goes away; there are just sources and sinks of data...

But yes, this should be decided as a question of scope after the work description...

> MB> So, my understanding is that current platforms support an extensive set of active measruements that go beyond OWAMP/TWAMP. So, i think more than these two protocols will be needed if we want to support this.

I'm thinking as well specifically about passive measurement tools, and all the information that might be available over IPFIX or SNMP.

> I guess the question at this point is how generic a protocol for exchanging measurement results can be when it is not tied to a specfic measurement.
> I mean, we could define a protocol (or reuse a protocol) that is a general envelope to exchange results from the measurement agent to the data collector, that is generic irrespectively to the actual test being reported. Then for each specific tests, we should add the specific data fields that the report should contain.
> the generic envelope would carry information about the measurement agent identification, time stamp, and additional security data. I am uncertain whether this has value, but i guess it may.
> the test specific part shoudl be defined for each test whose results is being reported and it should contain specific fields for the specific test.
> I agree that we should reuse existing protocols. Do you know an exisitng protocol that could serve as envelope protocol?

IPFIX might be a good fit for this envelope, if the data about the measurements is relatively uniform in structure, if it acceptable for the measurement-point-to-collector communication to be (1) unidirectional and (2) initiated by the measurement point. It's self-describing without being inefficient, it has a rich set of information elements for representing timestamps and information about measurement points, and has bindings to TLS and DTLS. New information elements for specific metrics and metric parameters can be easily added.

As defined, it's not a particularly good fit for architectures where the information is pulled in chunks, although you could place results in an IPFIX File and transport that over any of the various protocols we've defined for moving files around. It's also not a good fit for applications where each record has a wildly different structure.

> MB> I think this is one place where the IETF could add value. We should define a registry for measurements that can be perfoemed by the clients and for each of them to have a code identifying the tests, and the set of parameters that are needed as input and the oputput that is needs to be reported.
> then this protocol should convey to the measurement agent the code for the measurement as defined in the registry as well as the input parameters.

Absolutely.

> (I have seen that the IPPM metrcis registry, that probably looks a bit like this has been deprecated, I wonder if there is something to learn from that experience...)

From what I understand, this was due to the combinatoric nature of metric definitions in IPPM, which led to a lack of use. (I'm just a long-time-lurker in IPPM, though, so if I've got this wrong, IPPM people please correct me.) I would presume that an IPPM measurement registry would register classes of measurements, where the specific parameters of the measurement would not be captured within the type information of the metric definition, and would rather be captured in the configuration parameters in a measurement specification / control protocol interaction.

> MB> see above about the envelope protocol. Besides, you also point that there are some cases where another protocol is needed, so i am kind of confused by your summary here.

Sorry, I didn't express that very well... What I mean is that where we have an existing data export protocol, we should probably use it; where we do not, it probably doesn't make much sense to define a new data export protocol for every new measurement use case, but it might also be difficult to define a one-size-fits-all protocol.

> My understanding is that we could either define one specific protocol for each type of measurement we are going to support (i understand this is what OWAMP and TWAMP are), but we probably support more tests.
> I believe this envelope protocol would provide an extensible framework to define then the specific tests.
> 
> 
> 
>> 
> MB> as i mentioned to my reply to Juergen, i think there is much more value in standardizing the interfaces other than supporting multiple control infrastrucutres. For example, the possibility that of the shelf equipment already supports the measurements and can then be integrated into a measurment platform without requiring additional code (e.g. suppose the home routers already support LMAP, they could then be integrated into a platform without additional deployment burden)

+n for very large n. 

I wasn't trying to contradict this point; instead, I meant that standardized interfaces could be supported _inline_ with existing control for existing measurement infrastructures through translation, as opposed to a completely alternate source of control. Maybe this is a too fine a distinction...

It is indeed the off-the-shelf-equipment argument that makes LMAP so interesting for me; it's not about existing infrastructures, but existing tools from which new infrastructures can be built.

Cheers,

Brian

---

From: Al Morton <[email-masked]>
Date: Thu, 18 Oct 2012 19:52:44 -0400

Marcelo, Juergen, and Robert,

please see below:
At 02:50 AM 10/16/2012, marcelo bagnulo braun wrote:
>>...Marcelo wrote:
   Juergen wrote:
>Marcelo wrote:
>My view on this is the following: I would expect 
>that the user agents are either embedded in the 
>CPE (home router) from the vendor, or it can be 
>dowloaded to the end user device (e.ge standard linux app or whatever).
>
>Suppose that home router have this measurement 
>agent already built in. then deployment of very 
>large platforms would be much easier, you should 
>So, I do believe there is value on standardizing 
>this and i do believe this would make easier to deploy these platforms.
I agree with Marcelo, the LMAP requirements indicate that the desired
architecture for "...scales drawing on thousands or millions of nodes."
will benefit from standards for interaction between all components.
The greatest gains from standardization come from reduction in "complexity"
at the network edge, where the scale is the largest.

At the same time, (addressing Robert's point to some extent)
deployed architectures need not adopt standards at
all the way down to the measurement agents as long as they join the
standard architecture at a reasonable interface, and perform the
measurements to collect the standard metrics in a compliant fashion.

This is probably a good time to point-out the rest of (what might
be) IPPM's work related to the LMAP initiative:

- "defining any additional performance metrics as needed." [LMAP-req-00]
   (this is in-progress, see
    http://tools.ietf.org/html/draft-ietf-ippm-rate-problem-00  )

- Narrowing down the parameters for existing metrics, and refine metrics
   as needed for the "wired or wireless networks" scope of [LMAP-req-00]

- Providing network characterization and study guidance

- Repeat all the above for *passive* and hybrid passive/active metrics

- Other related work proposed on IPPM list and LMAP list
http://www.ietf.org/mail-archive/web/ippm/current/msg02945.html
http://www.ietf.org/mail-archive/web/lmap/current/msg00042.html

regards,
Al

---

From: Al Morton <[email-masked]>
Date: Thu, 18 Oct 2012 20:09:34 -0400

At 08:24 AM 10/16/2012, marcelo bagnulo braun wrote:
>MB> I think this is one place where the IETF could add value. We 
>should define a registry for measurements that can be perfoemed by 
>the clients and for each of them to have a code identifying the 
>tests, and the set of parameters that are needed as input and the 
>oputput that is needs to be reported.
>then this protocol should convey to the measurement agent the code 
>for the measurement as defined in the registry as well as the input parameters.
>(I have seen that the IPPM metrcis registry, that probably looks a 
>bit like this has been deprecated, I wonder if there is something to 
>learn from that experience...)
Hi Marcelo,

(As Axe-man for the deprecation of the IPPM Registry,)
I can summarize that we (all IPPM authors) were
wasting time assigning codes to metrics because the registry itself
had insufficient detail: it had no way to indicate the settings of
critical parameters of the metrics, so the metric codes were meaningless.
The morbidly curious can read the whole indictment here:
http://tools.ietf.org/html/rfc6248
and there are definitely lessons learned, plus some around who still
remember them. We *can* get this right the second time.

regards,
Al

---

From: Al Morton <[email-masked]>
Date: Thu, 18 Oct 2012 20:12:48 -0400

And Brian's summary is also correct, somehow this escaped my
previous scan...

At 09:54 AM 10/17/2012, Brian Trammell wrote:
> From what I understand, this was due to the combinatoric nature of 
> metric definitions in IPPM, which led to a lack of use. (I'm just a 
> long-time-lurker in IPPM, though, so if I've got this wrong, IPPM 
> people please correct me.) I would presume that an IPPM measurement 
> registry would register classes of measurements, where the specific 
> parameters of the measurement would not be captured within the type 
> information of the metric definition, and would rather be captured 
> in the configuration parameters in a measurement specification / 
> control protocol interaction.

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 19 Oct 2012 07:51:03 +0200

Hi Al,

Just to complete this list, i think an important contribution from IPPM 
for this work, is to create a new meaningfull registry for metrics, so 
that LMAP can refer to them. I guess you had this in mind, but to make 
it explicit.

Regards, marcelo