Subject: [lmap] Proposed LMAP Charter

From: Marc Linsner <[email-masked]>
Date: Fri, 18 Jan 2013 15:47:46 -0500

Let's discuss possible charter language for the proposed LMAP work.  At this
point, IMO, we should propose a charter for the work that could either be
added to an existing WG's charter, or standalone as a WG charter.  That's an
AD decision.  A follow-on discussion to this would obviously be the output
documents and milestones.

Here is a strawman to start with -

"
Measuring broadband service on a large scale is important for network
diagnostics by providers and users, as well for public policy. To conduct
service measurements, user networks gather data, either on their own
initiative or instructed by a measurement controller, and then upload the
measurement results to a designated measurement server. There is a need to
standardize a logical architecture and describe key requirements for
protocols to connect the components.  A large-scale measurement system needs
to support residential and small-enterprise networks, using either wired or
wireless networks. The architecture needs to consider the management of the
active and passive measurements.
 
The LMAP work will include describing various use cases considered to be
Large Scale Performance Measurement systems, define the requirements of the
management of these systems, and finally define the management communication
architecture and protocols.
 
This work will also include discussions about the privacy and security
concerns and will address them within its documents.
"

Fire away!

-Marc Linsner-

---

From: <[email-masked]>
Date: Wed, 23 Jan 2013 09:54:39 +0000

Thanks Marc. We're very interested in discussing a potential charter.
One thing we think is worth trying to do is to simplify the proposed work as much as possible, as the potential scope is big. Of course this could be done entirely as part of the WG's work, but doing this beforehand would enable a more rapid start. Below is our attempt to suggest some simplifying assumptions.
Each of the simplifying assumptions  has some penalty, so it would be great to hear whether people think they're reasonable, or over-restrict the use case that you have in mind. If it's helpful we could write up the architectural assumptions in an I-D - co-authors very welcome.
So a key question is whether the prospective WG should start with a phase of reaching consensus on architecture and choosing starting protocols, or whether we can reach enough consensus before the next IETF to move straight to standards work. I'm not sure that we can decide this right now, as it depends on how the discussion goes over the next month or so. There are some work items below that might apply in the latter case.
--
Measuring broadband service on a large scale is important for network diagnostics by providers and users, as well for public policy.  {Comment: operators may want measurement to help network planning and network mgt like fault identification; not sure whether this needs explicitly noting.}  To conduct service measurements, Measurement Agents (MAs) on user networks gather data, either on their own initiative or instructed by a Controller, and then upload the measurement results to a designated Collector. Currently measurement platforms contain up to a few thousand MAs. However, the vision is that MAs could potentially be embedded in every home hub, cable modem, enterprise edge router, set top box, smartphone and so on. Standards will help this capability be more pervasive, manageable and directly comparable.
RFCs from the IPPM WG define how to measure metrics - requests to define new tests should be handled by IPPM. These tests effectively define the communications between a MA and Test Server. {Comment: there are likely new metrics that need to be defined (eg latency under load) or tests under new conditions (eg current test for speed may not scale well to high line speeds).} . In a few cases the IPPM definition may need to be extended to move results data from the Test Server to the MA for later reporting.
The LMAP WG makes the following architectural assumptions - in order to make rapid progress with an acceptable loss of generality:-

*         We assume that Measurement Agent, tests Controller and results Collector are all under control of same organisation (eg Samknows, FCC, BT). Reason: Inter-organisation interactions (for example to share results) is better handled by business-level negotiation than a control protocol.

*         We assume the MA initiates a measurement. Reasons: having the test schedule on MA avoids it having to check frequently with the Controller; it is easier if the MA is behind a NAT; and the MA knows whether the user is active and therefore whether to run test or delay it.

*         We assume measurement results are sent from MA, and not from test server. Reason-1: It is easier to secure the MA to Collector communication. One implication is that for an upload test measured by the test server, the result is reported back to the MA which reports it in turn. Reason-2: Test servers may be controlled by a third party ('load time for IETF home page').

*         We assume no negotiation is needed between Controller and MA. The Controller already knows the characteristics of the MA, for instance via TR069 for Broadband Forum devices. The Controller simply instructs (securely) the MA "Measure this"; the MA replies "OK /error".

*         Similarly we assume no negotiation is needed between the MA and Collector. The protocol simply delivers (securely) the measurement results from the MA to the Collector, and returns an ack or error code.

*         We assume each MA is controlled by only one Controller. Reason: to avoid different Controllers giving the MA conflicting schedules. Note: a measurement platform may have several Controllers, with each Controller controlling a subset of the MAs, for example one controlling wireline MAs and one mobile MAs.

*         We assume the raw measurement results are reported; filtering /averaging of measuremnts by the MA is not in the initial scope, except as defined by IPPM as part of the metric definition.

*         We assume that uploading a new test capability to the MA is out of scope. This can be done as a firmware upgrade for home hub, or new app for PC, etc.

*         We assume as out of scope a network parameter server (which stores nominal values such as the contractual data rate). The functional role of the Network Paraemeter Server may be fulfilled by existing OSS (such as network or home gateway management systems). Interfaces to the NPS are pre-measurement (used by the Controller to decide which tests to schedule) or post-measurement (used to analyse the measurement data).

*         We assume data filtering is out of the initial scope. Filtering includes: removal of outlier measurements, archiving (curating) data for scientific research, and so on. There are certainly some aspects where best practice would be useful but data filtering is not a first priority.

*         We assume that data privacy issues are out of scope although security is considered. The MA, Controller and Collector are all under control of the same organisation. IPPM considers the implications of any passive measurement tests that it defines.
The LMAP WG standardises the following items:

1.      The data model that defines the test schedule, which includes:

a.      The IPPM metric to be tested and values for its parameters. {Question: is the restriction to IPPM OK? Ie non-IPPM tests are out of scope}

b.      Test schedule: when the test should be done and how often repeated ('once an hour at xx.03'). A test may also be a one-off, done immediately on demand.

c.       Output parameters: some IPPM metrics may define a choice of outputs

d.      Test Server: which server or servers should be used for the test

e.      Environmental conditions: such as 'no user traffic' which means a particular instance of the test is delayed or abandoned.

f.        (perhaps) Metadata: such as a mobile's location when the test should be made, the line's nominal rate etc. There may be privacy implications for the scenario where the measurement plaform is not controlled by the operator; the initial scope will exclude this scenario.

g.      (perhaps) Metadat to report: for example, the MA may be able to measure Layer 1 o 2 charcteristics such as DSL sync speed and interleaving; the actual test is out of scope. As for the previous bullet, there may be privacy implications for the general scenario, which we exclude from the initial scope.

h.      (perhaps) Priority: the Controller decides which tests are done when. However a test might get delayed until it overlaps with another scheduled test - which should be done (or done first)? Or should it simply be abandoned?

2.       The data model that defines how an MA should report test results. This may be combined with (1) above or specified as a separate reporting schedule. This includes:

a.       Where to report results to (one or more Collectors)

b.      How often to report results for the different tests

c.       What to do when Collector is not available (fail-over, try again)

3.      The data model that defines the report, which includes:

a.      The metric that was measured

b.      The time of the test

c.       Whether the test is scheduled, one-off triggered by the Controller, or user-initiated

d.      The measurement result - usually several results are batched together; the result of a one-off test may be wanted immediately

e.      (perhaps) Metadata, such as the mobile's location

f.        Comment: it would make sense to use the same language as for 1, for instance XML, YANG...

4.      The application level protocol for the (secure) delivery of the test and reporting schedule (#1, #2)

a.      A simple instruction - response protocol, with error codes defined

b.      LMAP could specify how it operates over one existing protocol (REST style HTTP(s), Netconf...)

5.      The application level protocol for the (secure) delivery of the report (#3)

a.      Same comments as for #4

6.      A document explaining the simplifying architectural assumptions in more detail. This document could also define some terminology and explain the use cases.
Looking forward to the discussion,
Best wishes,
Trevor and Phil
From: [email-masked] [mailto:[email-masked]] On Behalf Of Marc Linsner
Sent: 18 January 2013 20:48
To: [email-masked]
Subject: [lmap] Proposed LMAP Charter

Let's discuss possible charter language for the proposed LMAP work.  At this point, IMO, we should propose a charter for the work that could either be added to an existing WG's charter, or standalone as a WG charter.  That's an AD decision.  A follow-on discussion to this would obviously be the output documents and milestones.

"
Measuring broadband service on a large scale is important for network diagnostics by providers and users, as well for public policy.  To conduct service measurements, user networks gather data, either on their own initiative or instructed by a measurement controller, and then upload the measurement results to a designated measurement server. There is a need to standardize a logical architecture and describe key requirements for protocols to connect the components.  A large-scale measurement system needs to support residential and small-enterprise networks, using either wired or wireless networks. The architecture needs to consider the management of the active and passive measurements.

The LMAP work will include describing various use cases considered to be Large Scale Performance Measurement systems, define the requirements of the management of these systems, and finally define the management communication architecture and protocols.

This work will also include discussions about the privacy and security concerns and will address them within its documents.
"

Fire away!

-Marc Linsner-

---

From: Matthew Ford <[email-masked]>
Date: Wed, 23 Jan 2013 12:07:37 +0000

Phil,

Nice post, which I think I completely agree with.

On 23 Jan 2013, at 09:54, [email-masked] wrote:

> ·         We assume that Measurement Agent, tests Controller and results Collector are all under control of same organisation (eg Samknows, FCC, BT). Reason: Inter-organisation interactions (for example to share results) is better handled by business-level negotiation than a control protocol.

I think we also need an assumption that nothing will preclude having collectors, for example, be under control of third-parties. I think it is worth explicitly stating that, unless there is disagreement on this point?

Mat

---

From: Brian Trammell <[email-masked]>
Date: Wed, 23 Jan 2013 13:15:51 +0100

hi Mat, Phil, all,

I agree as well, though other comments on finer points may follow. One inline below.

> I think we also need an assumption that nothing will preclude having collectors, for example, be under control of third-parties. I think it is worth explicitly stating that, unless there is disagre…

+1 on this point. I'd suggest that we leave the component ownership/control issues completely aside; i.e., we don't necessarily want to preclude organization(s) from agreeing to use a measurement infrastructure in place (which uses this control protocol) for sharing of intermediate or final measurement results (in support of an existing business-level agreement). 

What's _really_ important about this assumption is that it makes the problems of inter-domain measurement control and exchange explicitly out of scope for the charter, and simplifies issues of trust among the components. I think it should probably be explicitly phrased as an issue of scope rather than an assumption about deployment.

Cheers,

Brian

---

From: KEN KO <[email-masked]>
Date: Wed, 23 Jan 2013 13:30:51 +0000

Phil, Trevor and Marc,

Great, detailed proposal. A lot of good information here.

One additional item to consider is the association of measurement results with service attributes such as uplink and downlink speed. In recent trials, these attributes have provided a useful baseline to which results are compared, but the process of collecting and validating the attributes has been largely manual and is not scalable. Associating applicable service attributes with the data model sent from the MA to the Collector - either by reference or by directly including them as metadata - would eliminate any need for a manual process and facilitate synchronization of attributes (which occasionally change due to upgrades and service redefinitions) and measurement results.
Thanks,
Ken Ko

---

From: <[email-masked]>
Date: Wed, 23 Jan 2013 16:48:56 +0000

Our position is that nothing should preclude 3rd parties operating separate components of the measurement architecture. Of course data protection would have to be considered/observed but we will not need anything explicit in the technical work of LMAP to do so.

What we want to avoid explicitly is two or more controllers connecting to a single Measurement Agent and having to manage such conflicts. There might, however, be two or more controllers for disjoint subsets of measurement agents (e.g. one for mobile devices and one for home gateways). There of course will be multiple test servers (to test multiple network paths) and could also be multiple collectors (where a MA replicates or partitions its results).

Trevor.

Trevor Burbridge
Network Infrastructure & Innovation | BT Innovate & Design
Tel: 01473 645115
Fax: 01473 640929

This email contains BT information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you.
We monitor our email system, and may record your emails. 
British Telecommunications plc
Registered office: 81 Newgate Street London EC1A 7AJ
Registered in England no: 1800000 

>-----Original Message-----
>From: Matthew Ford [mailto:[email-masked]]
>Sent: 23 January 2013 12:08
>To: Eardley,PL,Philip,TUB8 R
>Cc: [email-masked]; [email-masked]; Burbridge,T,Trevor,TUB8 R
>Subject: Re: [lmap] Proposed LMAP Charter
>
>Phil,
>
>> *         We assume that Measurement Agent, tests Controller and results
>Collector are all under control of same organisation (eg Samknows, FCC, BT).
>Reason: Inter-organisation interactions (for example to share results) is
>better handled by business-level negotiation than a control protocol.
>
>I think we also need an assumption that nothing will preclude having
>collectors, for example, be under control of third-parties. I think it is worth
>explicitly stating that, unless there is disagreement on this point?
>
>Mat
>

---

From: <[email-masked]>
Date: Wed, 23 Jan 2013 16:53:02 +0000

Yes - good point. We certainly do need to consider how best to enhance the measurement data with line/customer/CPE data. One question would be if we need to define this data within LMAP or simply define a mechanism for retrieving/carrying it (e.g. between the MA and Collector) whatever it may be.

Trevor.

Trevor Burbridge

From: KEN KO [mailto:[email-masked]]
Sent: 23 January 2013 13:31
To: Eardley,PL,Philip,TUB8 R; [email-masked]; [email-masked]; Burbridge,T,Trevor,TUB8 R
Subject: RE: Proposed LMAP Charter

Phil, Trevor and Marc,

Thanks,
Ken Ko

---

From: Julien Moutinho <[email-masked]>
Date: Thu, 24 Jan 2013 11:26:26 +0100

Hi everyone,

Interesting post Trevor and Phil!
I've made below a few humble comments derived from my (still young) experience
in developping the Grenouille system.

Le mer. 23 janv. 2013 à 09:54:39 +0000, [email-masked] a écrit :
> The LMAP WG makes the following architectural assumptions - in order to
> make rapid progress with an acceptable loss of generality:-
> 
> *         We assume that Measurement Agent, tests Controller and
> results Collector are all under control of same organisation (eg
> Samknows, FCC, BT). Reason: Inter-organisation interactions (for
> example to share results) is better handled by business-level
> negotiation than a control protocol.
agreed for an initial scope.
though, what is the precise meaning of "control" here?
because shouldn't the MA be "under control" of anyone
but the owner of the system it runs on?
which for instance would mean free/open source software,
and being able to precisely see and filter what is sent to whom and when.

> *         We assume the MA initiates a measurement. Reasons: having the
> test schedule on MA avoids it having to check frequently with the
> Controller; it is easier if the MA is behind a NAT; and the MA knows
> whether the user is active and therefore whether to run test or delay it.
agreed.

> *         We assume measurement results are sent from MA, and not from
> test server. Reason-1: It is easier to secure the MA to Collector
> communication. One implication is that for an upload test measured by
> the test server, the result is reported back to the MA which reports it
> in turn. Reason-2: Test servers may be controlled by a third party
> ('load time for IETF home page').
agreed. and we should also keep in mind that a Test server
(wouldn't be Landmark a better name?)
could run on the same system than the MA
so as to have MA to MA measurements.

> *         We assume no negotiation is needed between Controller and MA.
> The Controller already knows the characteristics of the MA, for
> instance via TR069 for Broadband Forum devices. The Controller simply
> instructs (securely) the MA "Measure this"; the MA replies "OK /error".
agreed for an initial scope, but I think there should eventually be
a more complex MA<>Controller negotiation,
for instance to synchronise several MA, a Controller could tell
each MA to recontact it in a given amount of time,
until it has accumulated a given number of MA to trigger the probing.

also, some characteristics may be volatile, like the geolocation,
and may be used by the Controller to filter its MA,
therefore I think a mechanism to keep track of the MA's characteristics
should be part of this negotiation; be it via TR-069
if it is flexible and supported enough or else.

moreover as it is a botnet which is designed here, preventing usurpation is critical,
thus maybe a level of mutual authentification should have to be inserted here,
for instance I think that using an horizontal public key infrastructure
like OpenPGP's should be considered: each MA/Controller/Collector
having its own private key (self-generated with possibly a designed revoker).

> *         Similarly we assume no negotiation is needed between the MA
> and Collector. The protocol simply delivers (securely) the measurement
> results from the MA to the Collector, and returns an ack or error code.
agreed for an initial scope.
though mutual authentification should eventually be done here too.

> *         We assume each MA is controlled by only one Controller.
> Reason: to avoid different Controllers giving the MA conflicting
> schedules. Note: a measurement platform may have several Controllers,
> with each Controller controlling a subset of the MAs, for example one
> controlling wireline MAs and one mobile MAs.
agreed for an initial scope.
though I do not really see how this would be enforced at this point;
at least I think that each instance of a MA should choose an initial Controller,
which in turn may suggest the MA to contact an other Controller.

> *         We assume the raw measurement results are reported; filtering
> /averaging of measuremnts by the MA is not in the initial scope, except
> as defined by IPPM as part of the metric definition.
what is the need for this assumption?
I think the MA should just be able to perform computation
when the concept of the probing allows it,
for instance for certain kinds of local anomalies detection
having the MA doing some computation is cheap on the MA
and saves a lot of computer resources when data mining the Collector.

> *         We assume that uploading a new test capability to the MA is
> out of scope. This can be done as a firmware upgrade for home hub, or
> new app for PC, etc.
agreed for an initial scope.
and not exactly that but related: I've been suggested
that some kind of sandboxing of a minimal language (à la google App Engine)
should be supported so that the casual user can develop and deploy its own probe
while keeping it safe (with several levels of safety)
for the MA running the probe; having such possibility
may also further the adoption/deployment of the MA.

> *         We assume as out of scope a network parameter server (which
> stores nominal values such as the contractual data rate). The
> functional role of the Network Paraemeter Server may be fulfilled by
> existing OSS (such as network or home gateway management systems).
> Interfaces to the NPS are pre-measurement (used by the Controller to
> decide which tests to schedule) or post-measurement (used to analyse
> the measurement data).
agreed for an initial scope.

> *         We assume data filtering is out of the initial scope.
> Filtering includes: removal of outlier measurements, archiving
> (curating) data for scientific research, and so on. There are certainly
> some aspects where best practice would be useful but data filtering is
> not a first priority.
agreed for an initial scope.
though best practices for the archiving part would be very welcome.

> *         We assume that data privacy issues are out of scope although
> security is considered. The MA, Controller and Collector are all under
> control of the same organisation. IPPM considers the implications of
> any passive measurement tests that it defines.
what is the need for this frightening assumption?
does it just means that privacy concerns must be specified within IPPM RFC?

>   a.      The IPPM metric to be tested and values for its parameters.
>   {Question: is the restriction to IPPM OK? Ie non-IPPM tests are out of scope}
what is the need for this assumption?
I think metrics should be easily extensible according to a common schema
(for instance specifying a hierarchical namespace for probes
gathering them by <kind of metric>/<kind of protocol>/<kind of underlying software>/..).

>   b.      Test schedule: when the test should be done and how often
>   repeated ('once an hour at xx.03'). A test may also be a one-off, done
>   immediately on demand.
I think that a variety of common temporal series should be considered
(like, raw-list, periodic, exponential-based, gamma-based, uniform-random-based, ..);
for instance some tomography probes require these kind of distributions.

an maybe that these series could be reused to vary fields of network protocols,
for instance some probes discovering the network topology accross load-balancers use them.

>   e.      Environmental conditions: such as 'no user traffic' which means
>   a particular instance of the test is delayed or abandoned.
other environmental conditions could include:
 - cheap network connection available (not 3G at per consumption rate for instance)
 - stable power supply available

>   f.        (perhaps) Metadata: such as a mobile's location when the test
>   should be made, the line's nominal rate etc. There may be privacy
>   implications for the scenario where the measurement plaform is not
>   controlled by the operator; the initial scope will exclude this
>   scenario.
>   g.      (perhaps) Metadata to report: for example, the MA may be able to
>   measure Layer 1 o 2 charcteristics such as DSL sync speed and
>   interleaving; the actual test is out of scope. As for the previous
>   bullet, there may be privacy implications for the general scenario,
>   which we exclude from the initial scope.
>   h.      (perhaps) Priority: the Controller decides which tests are done
>   when. However a test might get delayed until it overlaps with another
>   scheduled test - which should be done (or done first)? Or should it
>   simply be abandoned?

moreover I think versioning the test models (I called them configurations)
and report models (I called them samples) should also be envisaged.

> 2.       The data model that defines how an MA should report test
> results. This may be combined with (1) above or specified as a separate
> reporting schedule. This includes:
> 
I think there should be a simple local queue, filesystem-based for instance,
or possibly a local mail server could be used to post results
(just an idea, or at least a source of inspiration),

moreover environmental conditions could also intervene here to delay the report.

>   d.      The measurement result - usually several results are batched
>   together; the result of a one-off test may be wanted immediately
I would recommend to use a language designed to be a data interchange format,
which for instance means more JSON than XML which is rather a document standard.
the few MA I happen to know use JSON.

>   b.      LMAP could specify how it operates over one existing protocol (REST style HTTP(s), Netconf...)
yes, I think that ReST HTTP should be used,
which AFAIK would boil down to specifying:
 - URI schemes for identifying resources;
 - representations of the states of these resources,
   transferred between components.

> 6.      A document explaining the simplifying architectural assumptions
> in more detail. This document could also define some terminology and
> explain the use cases.

looking forward to the discussion,
julm

---

From: Arturo Servin <[email-masked]>
Date: Thu, 24 Jan 2013 08:50:31 -0200

Hi al,

	I was going to ask something similar.

	For example, we have a project where we have done some measurements but those are confined to a specific region of the world. It would be interesting to share that data among other organizations doing the same but today we can't because there is not a standard (that I asume we are intent to produce) and because privacy implications (that I am not sure that we are addressing in the chapter).

	So, is a WG goal to address the problem of sharing and anomyzing data (or address any privacy concern)?

Regards,
as

On 23 Jan 2013, at 10:15, Brian Trammell wrote:

> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap

---

From: <[email-masked]>
Date: Thu, 24 Jan 2013 11:45:47 +0000

I would argue that the scope of LMAP is to define the overall measurement framework. If this framework is used to run the same (IPPM defined) tests then it will yield comparable results.

How to process these results into equivalent statistics is post-measurement and beyond the scope of LMAP (and would probably fall in the remit of IPPM in any case). Similar argument for data sharing - data can be anonymised/aggregated (if necessary) and shared after collection.

Trevor.

>From: Arturo Servin [mailto:[email-masked]]
>Sent: 24 January 2013 10:51
>To: [email-masked]
>Cc: Matthew Ford; Eardley,PL,Philip,TUB8 R; Burbridge,T,Trevor,TUB8 R;
>[email-masked]; Brian Trammell
>	For example, we have a project where we have done some
>measurements but those are confined to a specific region of the world. It
>would be interesting to share that data among other organizations doing the
>same but today we can't because there is not a standard (that I asume we
>are intent to produce) and because privacy implications (that I am not sure
>that we are addressing in the chapter).
>
>> +1 on this point. I'd suggest that we leave the component
>ownership/control issues completely aside; i.e., we don't necessarily want to
>preclude organization(s) from agreeing to use a measurement infrastructure
>in place (which uses this control protocol) for sharing of intermediate or final
>measurement results (in support of an existing business-level agreement).
>>
>> What's _really_ important about this assumption is that it makes the
>problems of inter-domain measurement control and exchange explicitly out
>of scope for the charter, and simplifies issues of trust among the
>components. I think it should probably be explicitly phrased as an issue of
>scope rather than an assumption about deployment.
>>
>> Cheers,
>>
>> Brian
>>
>>

---

From: <[email-masked]>
Date: Thu, 24 Jan 2013 12:34:18 +0000

thanks Julien for your detailed and intersting comments, some responses in-line
phil

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Julien Moutinho [[email-masked]]
Sent: 24 January 2013 10:26
To: [email-masked]
Subject: Re: [lmap] Proposed LMAP Charter

Hi everyone,

[phil] "under control" - I'd say this is the party who owns /manages the MA. For a hardware device this is "the owner of the systme it runs on" - for a software MA then it's the owner of the app (not necessarily the owner of the system it runs on). 
the Controller instructs the MA what test server(s) to use and what Collectors to send results to. 

agreed.

[phil] I think it's possible that the device with the MA functionality will also have test server functionality. 
personally I prefer the term test server to Landmark.

[phil] please can you explain a bit more, I don't follow.

[phil]i agree that it will learn these charcteristics via TR-069 or something else outside lmap. i think the implication for lmap is that no negoiation is needed in lmap.

[phil] let me see if i get your scenario. we have mobiles with MA functionality. the Controller wants tests to be run by mobiles in Cambridge.
from an lmap point of view, i think the Controller would liaise with the mobile operator's location database to select mobiles that are near Cambridge, and then send an instruction to those mobiles "for the next hour, if you're in cambridge then run this test". an hour later it would re-check the location DB to re-tweak the set of mobiles and re-send the instruction. i dont think this needs lmap negotiation.

[phil] agree that security is needed for the MA/controller/Collector comms.

[phil] i agree there is some initialisation activity needed. 
i guess this is part of the Collector's job. would be interesting to hear people's views about how to do this and what we want in prospective WG's scope.

[phil] the reason for the assumption is to simplify the defintion of lmap's Reporting data model. 
reporting the raw measurements would, i tihnk, be the most usual case.
I agree that there'll be some cases where the MA wants to do some averaging (or other data manipulation) rather than leaving all data manipulation to the Collector - but I think it is Ok to leave this out of the initial scope.

[phil] sorry, didn't phrase this bullet very well. We meant: as a consequence of the previous assumptions, therefore data privacy can be excluded from the scope (becuase MA, controller and collector under control fo same organisation). 
of course the Collector may send some anonymised version of the collected data to other parties - this would clearly have privcay implications. but this is out of scope -- the range of things that the Collected data could be used for are large - for instance how could the operator use it for their network mgt? this requires interaction with the OSS, and i think this is out of scope of lmap.
Basic rule of thumb therefore is that lmap avoids thinking about "how to use the collected measurements" - its scope is "how to control and collect the measurments"

[phil] thanks - interested to hear people's views about this

[phil] sounds like a good idea.

[phil] ok

[phil] for this and your JSON comment it would be interesting to hear people's views (to see if consensus can be reached in the next month or so, or whether the (propspective) WG needs to do the selection.

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Thu, 24 Jan 2013 07:53:33 -0500

Hi Trevor,

You are right that post-singleton measurement processing is part of
IPPM's work (singletons, samples, statistics). But LMAP could easily 
use what has been specified in IPPM RFCs to specify some form of
results format. So I tend to agree with Arturo.

For me, the simplest LMAP framework is the one which ensures
the same measurements are performed according to the same sampling plan
(schedule) and reported the same way by each organization in their network.
Thus, there is freedom to choose the best implementation within a
network.

regards,
Al

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Thursday, January 24, 2013 6:46 AM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked];
> [email-masked]
> I would argue that the scope of LMAP is to define the overall measurement
> framework. If this framework is used to run the same (IPPM defined) tests
> then it will yield comparable results.
> 
> How to process these results into equivalent statistics is post-
> measurement and beyond the scope of LMAP (and would probably fall in the
> remit of IPPM in any case). Similar argument for data sharing - data can
> be anonymised/aggregated (if necessary) and shared after collection.
> 
> Trevor.
> 
> results

---

From: <[email-masked]>
Date: Thu, 24 Jan 2013 13:26:52 +0000

I was thinking really of statistics produced across a set of MAs - this could not be done on the MA and is beyond the scope of the LMAP measurement control and collection.

Statistics for a single MA, I guess could be just another test that the probe could operate. One more question for this group would then be if this statistic just looks the same as a singleton measurement to the LMAP framework, or whether LMAP contains support for constructing statistics in different ways from singleton metrics. Obviously the former simplifies our work considerably but means different statistics have to be coded as a single test and may not be able to operate in conjunction with other statistics based upon the same singleton test. The other problem is that streams or statistics will overlap - contention of tests needs to be done on the singleton level.

Options:
a) only allow singleton measures - streams and statistics are constructed after collection. Obviously not good if the singleton test is a ping every second.
b) allow streams and statistics to be coded as tests but only allow scheduling/contention to be managed at the test level within LMAP. If streams/statistics overlap they may affect each other. Contention management for sub-test elements (e.g. singleton ping tests within an average latency stream) would be coded into the test.
c) allow streams and statistics to be constructed on the MA but manage contention at the singleton level. In effect same as (a) but with processing capabilities on the MA. Could get very complicated.

I guess I would advocate (b).

Trevor.

>From: MORTON JR., ALFRED (AL) [mailto:[email-masked]]
>Sent: 24 January 2013 12:54
>To: Burbridge,T,Trevor,TUB8 R; [email-masked]; [email-masked]
>Cc: Eardley,PL,Philip,TUB8 R; [email-masked]; [email-masked];
>[email-masked]
>Subject: RE: [lmap] Proposed LMAP Charter
>
>Hi Trevor,
>

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Thu, 24 Jan 2013 19:08:39 -0500

I agree with option b) below, given those three as choices.

I wasn't thinking of cross MA or full-study statistics as part of LMAP,
or any standards-setting effort for that matter. Analysis is partly art.

While the key IPPM metrics mandate the collection and storage of singletons,
their value is sometimes very limited (the loss ratio is zero, no need to see
the arrival status of each packet) and has storage implications in large scale.
It has made sense in some circumstances to store singletons (e.g., diagnostics), 
but not forever.

Al

> Sent: Thursday, January 24, 2013 8:27 AM
> To: MORTON JR., ALFRED (AL); [email-masked]; [email-masked]
> I was thinking really of statistics produced across a set of MAs - this
> could not be done on the MA and is beyond the scope of the LMAP
> measurement control and collection.
> 
> Statistics for a single MA, I guess could be just another test that the
> probe could operate. One more question for this group would then be if
> this statistic just looks the same as a singleton measurement to the LMAP
> framework, or whether LMAP contains support for constructing statistics in
> different ways from singleton metrics. Obviously the former simplifies our
> work considerably but means different statistics have to be coded as a
> single test and may not be able to operate in conjunction with other
> statistics based upon the same singleton test. The other problem is that
> streams or statistics will overlap - contention of tests needs to be done
> on the singleton level.
> 
> Options:
> a) only allow singleton measures - streams and statistics are constructed
> after collection. Obviously not good if the singleton test is a ping every
> second.
> b) allow streams and statistics to be coded as tests but only allow
> scheduling/contention to be managed at the test level within LMAP. If
> streams/statistics overlap they may affect each other. Contention
> management for sub-test elements (e.g. singleton ping tests within an
> average latency stream) would be coded into the test.
> c) allow streams and statistics to be constructed on the MA but manage
> contention at the singleton level. In effect same as (a) but with
> processing capabilities on the MA. Could get very complicated.
> 
> >> >would be interesting to share that data among other organizations
> doing
> >> the
> >> >> I agree as well, though other comments on finer points may follow.
> One
> >> >inline below.
> >> >>
> >> >Collector are all under control of same organisation (eg Samknows,
> FCC,
> >> BT).
> >> >collectors, for example, be under control of third-parties. I think it
> is
> >> worth
> >> >in place (which uses this control protocol) for sharing of
> intermediate
> >> or final
> >> >components. I think it should probably be explicitly phrased as an
> issue
> >> of

---

From: Julien Moutinho <[email-masked]>
Date: Fri, 25 Jan 2013 12:09:55 +0100

Hi Phil! some comments of mine inlined.

[phil] "under control" - I'd say this is the party who owns
/manages the MA. For a hardware device this is "the owner of
the system it runs on" - for a software MA then it's the owner
of the app (not necessarily the owner of the system it runs on).
the Controller instructs the MA what test server(s) to
use and what Collectors to send results to.

[julm] what I just think is that there should be
an MA-local policy configuration that is authoritative
over test configurations sent by Controllers,
and whoever on the system happens to be able to edit this MA-local policy
is said the owner of the MA, so that it is not necessarily
the same owner than the Controller's or Collector's.
I agree to left it aside for an initial scope.

[julm] for instance in the context of simple-loss tomography
we want each MA within a set large enough, to send one ping
to a common Test server, all at the same time.
I have been suggested that one way to coordinate the MA
could be to have the Controller accumulating MA
by telling them to recontact it later
or just keep waiting on that established connexion,
and once it considers the set of MA large enough,
to tell each MA to send the ping.
There certainly are several algorithms/methods
to reach certain kinds of temporal synchronisation amongst MA,
and maybe this is a property useful in enough uses cases
so that a variety of them could be considered in LMAP.
I agree to left it aside for an initial scope.

[phil] i agree that it will learn these characteristics via
TR-069 or something else outside lmap. i think the implication
for lmap is that no negotiation is needed in lmap.

[phil] let me see if i get your scenario. we have mobiles with
MA functionality. the Controller wants tests to be run by
mobiles in Cambridge.  from an lmap point of view, i think the
Controller would liaise with the mobile operator's location
database to select mobiles that are near Cambridge, and then
send an instruction to those mobiles "for the next hour, if
you're in cambridge then run this test". an hour later it
would re-check the location DB to re-tweak the set of mobiles
and re-send the instruction. i dont think this needs lmap
negotiation.

[julm] what I simply think is that there should be a way
for the Controller to probe the MA's capabilities
to select the MA it wants to use before sending it a test configuration.
When speaking about geolocation I was referring to probe
the embedded GPS or the GSM tower identifier;
not contacting an operator's location database.
Maybe the tracking of capabilities could rather be
a feedback from the Collector to the Controller.
but I don't know/haven't thought about it more at this point.

[phil] the reason for the assumption is to simplify the
definition of lmap's Reporting data model.
reporting the raw measurements would, i think, be the most usual case.
I agree that there'll be some cases where the MA wants to do
some averaging (or other data manipulation) rather than leaving
all data manipulation to the Collector - but I think it is Ok to
leave this out of the initial scope.

[julm] agreed for an initial scope.

what is the need for this frightening assumption?
does it just mean that privacy concerns must be specified within IPPM RFC?

[phil] sorry, didn't phrase this bullet very well. We meant:
as a consequence of the previous assumptions, therefore data
privacy can be excluded from the scope (because MA, controller
and collector are under control of the same organisation).
of course the Collector may send some anonymised version of the
collected data to other parties - this would clearly have
privacy implications. but this is out of scope -- the range of
things that the Collected data could be used for are large -
for instance how could the operator use it for their network
mgt? this requires interaction with the OSS, and i think this
is out of scope of lmap.  Basic rule of thumb therefore is
that lmap avoids thinking about "how to use the collected
measurements" - its scope is "how to control and collect the
measurements"

[julm] well, my concern here is not on how to use the collected measurements,
but rather on how to give the user running a MA
a language to express a policy specifying what they allow their MA to do,
which for instance could express things like:
 - allow to receive configurations from such Controller;
 - allow to send samples to such Collector;
 - allow such category of probes to run during
   such time periods/system parameters/network parameters;
 - allow such category of probes to use
   such system devices/system ressources/network interfaces.

[julm] and this hierarchy could be replicated
in the test configuration and report sample models.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Fri, 25 Jan 2013 17:10:32 +0000

Trevor,

    Just observing something here -- Given there is no standard definition of what "3rd party" is, and the definition varies depending on if we discuss business roles, vs. vendor roles, vs. operator the undefined use of "3rd party" without some contextual qualifier is pretty dangerous in terms of causing confusion.

  So if the words "3rd party" is used anywhere it really has to be accompanied by some type of category or qualifier.

Regards,
Mike 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Wednesday, January 23, 2013 10:49 AM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: Re: [lmap] Proposed LMAP Charter

Trevor.

Trevor Burbridge

This email contains BT information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you.
We monitor our email system, and may record your emails. 
British Telecommunications plc
Registered office: 81 Newgate Street London EC1A 7AJ Registered in England no: 1800000 

>Reason: Inter-organisation interactions (for example to share results) 
>is better handled by business-level negotiation than a control protocol.
>
>is worth explicitly stating that, unless there is disagreement on this point?
>
>Mat
>

---

From: Marc Linsner <[email-masked]>
Date: Fri, 25 Jan 2013 12:55:46 -0500

Mike,

How about 'different administrative domain'?

-Marc-

-----Original Message-----
From: "Bugenhagen, Michael K" <[email-masked]>
Date: Friday, January 25, 2013 12:10 PM
To: "[email-masked]'" <[email-masked]>, "[email-masked]"
<[email-masked]>, "[email-masked]" <[email-masked]>
Cc: Marc Linsner <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: RE: [lmap] Proposed LMAP Charter

>Trevor,
>
>    Just observing something here -- Given there is no standard
>definition of what "3rd party" is, and the definition varies depending on
>if we discuss business roles, vs. vendor roles, vs. operator the
>undefined use of "3rd party" without some contextual qualifier is pretty
>dangerous in terms of causing confusion.
>
>  So if the words "3rd party" is used anywhere it really has to be
>accompanied by some type of category or qualifier.
>
>Regards,
>Mike 
>
>
>
>
>Our position is that nothing should preclude 3rd parties operating
>separate components of the measurement architecture. Of course data
>protection would have to be considered/observed but we will not need
>anything explicit in the technical work of LMAP to do so.
>
>What we want to avoid explicitly is two or more controllers connecting to
>a single Measurement Agent and having to manage such conflicts. There
>might, however, be two or more controllers for disjoint subsets of
>measurement agents (e.g. one for mobile devices and one for home
>gateways). There of course will be multiple test servers (to test
>multiple network paths) and could also be multiple collectors (where a MA
>replicates or partitions its results).
>
>
>Trevor.
>
>Trevor Burbridge
>This email contains BT information, which may be privileged or
>confidential. It's meant only for the individual(s) or entity named
>above. If you're not the intended recipient, note that disclosing,
>copying, distributing or using this information is prohibited. If you've
>received this email in error, please let me know immediately on the email
>address above. Thank you.

---

From: James Miller <[email-masked]>
Date: Fri, 25 Jan 2013 13:28:45 -0500

I'm not sure I fully understand the difference of the three options, but b)
looks good from my perspective.

I think Al's point from a few days ago is important to remember.  A key
point from my perspective is that the value of the collected results may be
measured in the public policy context based on the quality of the sample
plan, ability of the managers to monitor anomalies in the environments and
interpret the results appropriately, and the post-collection statistics
work on the results.  The protocols should be able to accept other layers
of management for selection of MA's and execution of pre-defined test
schedules, and the ability to transfer results in a way that supports
post-processing.

Also wanted to tag again that the b) should be able to absorb security
features imposed to support differing privacy policies for collected
results being securely transferred to select entities for post-collection
data processing.

The opinions expressed are those of the author and do not necessarily
represent the views of the Federal Communications Commission or the United
States Government; The Maureen and Mike Mansfield Foundation; or any
Japanese Ministry or the Government of Japan.

> >Thus, there is freedom to choose the best implementation within a

On Thu, Jan 24, 2013 at 7:08 PM, MORTON JR., ALFRED (AL)
<[email-masked]>wrote:

> > >> >measurements but those are confined to a specific region of the
> world.
> > It
> > >> >are intent to produce) and because privacy implications (that I am
> not
> > >> sure

---

From: James Miller <[email-masked]>
Date: Fri, 25 Jan 2013 13:39:58 -0500

I think the third-party point was introduced from Matt Ford and +1'd but I
am having trouble seeing where there was an edit to your original charter
text.

Where would this go?  Below is placeholder sentence..

To conduct service measurements, user networks gather data, either on their
own initiative or instructed by a measurement controller, and then upload
the measurement results to a designated measurement server.
[ These functions in some cases may be executed by different entities that
may or may not share clearly defined organizational or legal relationships.
]

I think the sentence should cover the many of the architectural use cases
we have in the field today such as:

1) Contractual relationships between arms-length parties, e.g. Carriers and
Solutions providers, Regulators and Solutions providers,
2) Relationships based on shared interests lacking legal privitiy and legal
obligations, e.g. Regulators and loose collaborative groups, Carriers and
research entities without contractual agreements.

One thing I'd considered early was including these and other features at
the data schema level to help people evaluate the results of LMAP projects.
 For example, results of a roughly coded research project without
administrative support and scrutiny might be used in different ways than a
contracted solution with clearly defined and legally enforceable service
obligations.

Maybe not crucial for the charter text but I do think the data schemas
should include description of the kinds of administrative and operational
features implementable with the protocols.

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Fri, 25 Jan 2013 19:02:32 +0000

Marc,

A suggestion on an language for "3rd party" is to use the Agency distinction- 

   Personally I think a more appropriate term that reflects an privacy relationship between the customers (and their private identification / information) is the word "non-affiliated agency" vs. "3rd party".

The reasoning here -

   When an operator "outsources" testing to a "3rd party" they are an agent of the operator and have the same responsibilities of the operator.
When a non-operator affiliated agent is doing testing - they have the responsibility of maintaining the relationship with the customer in terms of privacy.  So non-operator affiliated agency would be an more accurate term in my opinion, and it reflects more closely the role of customer privacy than simply stating 3rd party.... 

Another dimension that some standards bodies will observer --- 

a) Independent certified lab  vs. affiliated
	- Is the agency affiliated with another organization or fully an Certified Independent lab - this is an international standard ...

b) Test results qualifiers That dimensionally relate to the Tester agent being able to validate information with a Operator -
Non-Validated tests = the tester doesn't know / can't guarantee (So you have validated and non-validated results)
	1) Provisioned speed tier
	2) What the "sold speed" of the operator is - and these can and do change
	3) That the customer actually put the device on the line they said they are on (not a relatives line)
	4) That they are on the correct operator

Regards 
-Mike

-----Original Message-----
From: Marc Linsner [mailto:[email-masked]] 
Sent: Friday, January 25, 2013 11:56 AM
To: Bugenhagen, Michael K; [email-masked]'; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [lmap] Proposed LMAP Charter

Mike,

-Marc-

-----Original Message-----
From: "Bugenhagen, Michael K" <[email-masked]>
Date: Friday, January 25, 2013 12:10 PM
To: "[email-masked]'" <[email-masked]>, "[email-masked]"

>definition of what "3rd party" is, and the definition varies depending 
>on if we discuss business roles, vs. vendor roles, vs. operator the 
>undefined use of "3rd party" without some contextual qualifier is 
>pretty dangerous in terms of causing confusion.
>
>What we want to avoid explicitly is two or more controllers connecting 
>to a single Measurement Agent and having to manage such conflicts. 
>There might, however, be two or more controllers for disjoint subsets 
>of measurement agents (e.g. one for mobile devices and one for home 
>multiple network paths) and could also be multiple collectors (where a 
>MA replicates or partitions its results).
>
>
>Trevor.
>
>Trevor Burbridge
>copying, distributing or using this information is prohibited. If 
>you've received this email in error, please let me know immediately on 
>the email address above. Thank you.
>>FCC, BT).

---

From: James Miller <[email-masked]>
Date: Fri, 25 Jan 2013 14:15:12 -0500

Mike,

I'd comment that "agent" would be preferable to "agency" in your text.

On Fri, Jan 25, 2013 at 2:02 PM, Bugenhagen, Michael K <
[email-masked]> wrote:

> uage for "3rd party" is to use the Agent

Also use of agency alone to describe the scope of possible relationships is
incomplete.  Pasting back a post w/ edits that I think I missed you, my
text is broader.  The relationships must include both agency or
contractual (legal agency) and non-contractual (extra-legal) relationships.
 For example, M-Lab does not appear to be a 501(c)(3) non-profit but is a
"collaborative effort" with a steering committee that may not satisfy an
agency or contractural definition in how they support current LMAP efforts.

I think replacing "agency" below would be fine but note that agency is a
different area of law from contract and so a more appropriate term that
captures the points is probably "legal".  They taught me in law school that
agency law has been largely subsumed by contract law, but the main point is
there are legally enforceable obligations in some cases.. and in others not.

1) Contractual or other legal relationships between arms-length parties,
e.g. Carriers and Solutions providers, Regulators and Solutions providers,
2) Relationships based on shared interests lacking legal privity and legal
obligations, e.g. Regulators and loose collaborative groups, Carriers and
research entities without contractual agreements.

---

From: Julien Moutinho <[email-masked]>
Date: Sat, 26 Jan 2013 10:26:13 +0100

> in the test configuration and report sample models.

[julm] so I see that you all have started to work
on this registry and its organisation :
	http://www.ietf.org/mail-archive/web/ippm/current/msg03034.html

it's very welcome :)

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Sat, 26 Jan 2013 10:48:47 -0500

Hi Julien, please see reply below:

> Julien Moutinho
> Sent: Saturday, January 26, 2013 4:26 AM
> To: [email-masked]

Glad to hear it, and if you take a look at the drafts, please provide comments
on the ippm-list. We need more opinions to make a decision between
the two alternatives, hierarchical and independent registries, 
as Marcelo described them. This is a critical but not very exciting
topic, so even an expression of support for the work would be useful!

thanks and regards,
Al

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Sun, 27 Jan 2013 19:28:01 +0000

James,

   Unfortunately the word "agent" is already in use in different forums as the "probe" or test client,  agency however hadn't been used that I know of.   I was thinking "agent" as we'll but them realized the two definitions for the same word should not be used for different meanings in the same document.  Those were my thoughts anyway.

Regards
Mike

Sent from my iPhone

On Jan 25, 2013, at 1:15 PM, "James Miller" <[email-masked]<mailto:[email-masked]>> wrote:

Mike,

On Fri, Jan 25, 2013 at 2:02 PM, Bugenhagen, Michael K <[email-masked]<mailto:[email-masked]>> wrote:
uage for "3rd party" is to use the Agent

Also use of agency alone to describe the scope of possible relationships is incomplete.  Pasting back a post w/ edits that I think I missed you, my text is broader.  The relationships must include both agency or contractual (legal agency) and non-contractual (extra-legal) relationships.  For example, M-Lab does not appear to be a 501(c)(3) non-profit but is a "collaborative effort" with a steering committee that may not satisfy an agency or contractural definition in how they support current LMAP efforts.

I think replacing "agency" below would be fine but note that agency is a different area of law from contract and so a more appropriate term that captures the points is probably "legal".  They taught me in law school that agency law has been largely subsumed by contract law, but the main point is there are legally enforceable obligations in some cases.. and in others not.

[ These functions in some cases may be executed by different entities that may or may not share clearly defined organizational or legal relationships. ]

I think the sentence should cover the many of the architectural use cases we have in the field today such as:

1) Contractual or other legal relationships between arms-length parties, e.g. Carriers and Solutions providers, Regulators and Solutions providers,
2) Relationships based on shared interests lacking legal privity and legal obligations, e.g. Regulators and loose collaborative groups, Carriers and research entities without contractual agreements.

_______________________________________________
lmap mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lmap

---

From: Brian Trammell <[email-masked]>
Date: Mon, 28 Jan 2013 12:13:40 +0100

Greetings, all,

I'm behind the curve a bit on this thread, so I thought I'd sum up from my point of view... Please note I'm not wearing my IPPM co-chair hat here; comments on what belongs where below are personal opinion only. 

Executive summary: I broadly agree with Phil and Trevor's proposal.

There seem to be a few areas of work to build something like the union of the visions of LMAP I've seen on this list; in determining how to charter a WG, the question for each of these is does it already have a home in another WG (here I'm thinking IPPM), does it fit into a charter simple enough to make LMAP likely to complete its initial milestones in a timely fashion, or should it be declared out of scope? 

(1) A common definition of singleton metrics to be produced by a single measurement agents. In my opinion, this is key to ensuring interoperability, and is pretty clearly in the scope of IPPM; new development required by LMAP should probably be done by IPPM. As you pointed out, there are two variants of a new registry of these under consideration in IPPM, as well: draft-bagnulo-ippm-new-registry and draft-bagnulo-ippm-new-registry-independent. I repeat your call for those interested in LMAP to have a look at these and provide comments to the IPPM list.

(2) A common definition of sampling/statistics over singleton metrics to be produced by a single measurement agent. This is also pretty clearly in the scope of IPPM.

(3) A common definition of statistics over singleton metrics to be produced by multiple measurement agents. Some specific cases, especially as regards the derivation of whole-path metrics from subpath metrics or whole-interval metrics from sub-interval metrics, have already been covered by IPPM (RFC 5835 defines a Framework for Metric Composition, and RFC 6049, Spatial Composition of Metrics). As for other statistics, IPPM could provide a home for this work, depending on how close to the assumptions of RFC 5835 those statistics hold.

(4) A set of reference sample plans referencing these metrics and statistics in (1)-(3); this provides the basis for comparison among multiple tests run by different test providers. 

(5) A definition of a control protocol for setting up measurements at measurement agents, with standard representations for the sampling plans in (4), based on metrics and statistics in (1)-(3). Key for interoperability in a multi-vendor environment (as you're likely to have with measurement agents on user end systems or CPE), and more so in a multi-measurement-operator environment (even if this group decides it's out of scope). Such work also implies the definition of a reference architecture so everyone can be sure what the terminology referenced by the protocol means. This seems to be the focus of much of the energy in LMAP to date, and probably makes sense to put this central in the charter of a new working group should one be formed. This is broadly in line with the charter proposal under discussion, so, so far, so good.

(6) A definition (or selection) of a data export protocol for moving measurements from agents to collectors. If the data export protocol is part of the measurement specification the controller gives to the agent, this is perhaps less crucial (though then you do need to make sure your agent and collector protocols match at deployment time). Otherwise, this must be defined along with (5), or you have no common representation for interoperability between collectors and agents.

(7) Cross-domain reporting of results in (6) above. This adds anonymization/pseudonymization for privacy preservation and authorization to receive reports to the issues that must be addressed. 

(8) Cross-domain control of measurement agents to support (7) above. As above, plus you have a whole host of new AAA problems to solve, too. 

(9) Best practices in the management of relationships among the entities managing each piece of an LMAP infrastructure. This touches on the legal conversation on this subject, which is somewhat outside of my area of expertise, but an applicability statement is probably a good starting point here.

(10) Definition of best practices for data processing and archiving (e.g., outliner removal from statistics, specification of metadata, etc). There is some work in this area under consideration for IPPM (draft-mathis-ippm-data-curation), but no plan as yet to address this point comprehensively. 

I'm not sure where the line between IPPM and not-IPPM should be for composition of metrics across multiple measurement agents as in (3). I'm inclined to say that operation on IPPM metrics that doesn't fit into anything that we need that's not covered in existing IPPM frameworks should probably be done as new work in IPPM.  Similarly, (10) gets into the management of the data at the semantic level; as such, it's kind of tied to the definitions of the metrics (1) - (3). I'm inclined to say this is IPPM work, but maybe that's only because there's already a draft on the subject up for consideration in IPPM.

Recommendations for sample plans (4) should follow from the LMAP use cases, and as such are the first thing "up the stack" that, in my mind, is not clearly IPPM work. Definitions are only necessary to provided a basis for comparison of tests from multiple providers, but would be useful as recommendations in any case.

I think focusing on (4), then, plus the control protocol (5), and determining to what extent we can reuse/extend existing protocols for data export (6) is probably a good starting point for an LMAP charter. Cross-domain issues (7) and (8) are of interest to some potential users of the control protocol (myself included), but are probably best considered after the core is done. Work on (9) might be necessary in any case other than single-owner, single-operator infrastructures (i.e., even if you don't need interdomain control you may need to think about who's responsible for what in order to ensure the protocol supports the ownership structures you require), so that should probably appear in an applicability statement, as well. 

Best regards,

Brian

---

From: "Sharam Hakimi" <[email-masked]>
Date: Mon, 28 Jan 2013 09:22:00 -0500

Mike,
I agree with the notion that a third party has different meaning in
different situations as you have described. I think that the language
should be that the architecture shall not prevent multiple test devices
that perform the same test on the defined network, subnet, etc. 

I do not believe that going down the path of having multiple controlling
agents controlling the same device is not a good idea in the same
operator's network and certainly not acceptable especially when that
device is owned and maintained by someone else.

The current proposed architecture does not prevent multiple test
agents/devices, each controlled by its deployer, to perform the same
test (that provides information from the users/customers point of view)
at the same test points and I think that provides enough flexibility to
arrive at the required/needed information.

Sharam

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Bugenhagen, Michael K
Sent: Friday, January 25, 2013 2:03 PM
To: 'Marc Linsner'; [email-masked]'; [email-masked];
[email-masked]
Cc: [email-masked]
Subject: Re: [lmap] Proposed LMAP Charter

Marc,

A suggestion on an language for "3rd party" is to use the Agency
distinction- 

   Personally I think a more appropriate term that reflects an privacy
relationship between the customers (and their private identification /
information) is the word "non-affiliated agency" vs. "3rd party".

The reasoning here -

   When an operator "outsources" testing to a "3rd party" they are an
agent of the operator and have the same responsibilities of the
operator.
When a non-operator affiliated agent is doing testing - they have the
responsibility of maintaining the relationship with the customer in
terms of privacy.  So non-operator affiliated agency would be an more
accurate term in my opinion, and it reflects more closely the role of
customer privacy than simply stating 3rd party.... 

a) Independent certified lab  vs. affiliated
	- Is the agency affiliated with another organization or fully an
Certified Independent lab - this is an international standard ...

b) Test results qualifiers That dimensionally relate to the Tester agent
being able to validate information with a Operator -
Non-Validated tests = the tester doesn't know / can't guarantee (So you
have validated and non-validated results)
	1) Provisioned speed tier
	2) What the "sold speed" of the operator is - and these can and
do change
	3) That the customer actually put the device on the line they
said they are on (not a relatives line)
	4) That they are on the correct operator

Regards 
-Mike

-----Original Message-----
From: Marc Linsner [mailto:[email-masked]] 
Sent: Friday, January 25, 2013 11:56 AM
To: Bugenhagen, Michael K; [email-masked]'; [email-masked];
[email-masked]
Cc: [email-masked]

Mike,

-Marc-

-----Original Message-----
From: "Bugenhagen, Michael K" <[email-masked]>
Date: Friday, January 25, 2013 12:10 PM
To: "[email-masked]'" <[email-masked]>,
"[email-masked]"

>>is worth explicitly stating that, unless there is disagreement on this

>>point?

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 28 Jan 2013 18:05:59 +0100

Hi everybody,

thanks Phil for this really good proposal. I agreed that it is import to scope 
down the initial work to a well defined scenario and focus on basic protocol 
functionality.

Regarding the text below and also following the discussion on data processing, 
I would like to add one comment and maybe can some feedback from people how 
already have experience with large scale measurements:

For me the proposal seem to produce a huge amount of data. We envision a MA in 
basically every home network, there are frequent test scheduled by one or 
more 'parties' and there is all the raw data which needs to be transmitted to 
the collector. Thinking about this I afraid that most of our future Internet 
traffic will be the measurement (and collection) traffic itself. Maybe this 
issue is out of scope for the actual work group focus, but I guess it's worth 
think about this when defining the architecture/scenarios and protocols.

Mirja

On Wednesday 23 January 2013 10:54:39 [email-masked] wrote:
> One thing we think is worth trying to do is to simplify the proposed work
> as much as possible, as the potential scope is big. Of course this could be
> done entirely as part of the WG's work, but doing this beforehand would
> enable a more rapid start. Below is our attempt to suggest some simplifying
> assumptions. Each of the simplifying assumptions  has some penalty, so it
> would be great to hear whether people think they're reasonable, or
> over-restrict the use case that you have in mind. If it's helpful we could
> write up the architectural assumptions in an I-D - co-authors very welcome.
> So a key question is whether the prospective WG should start with a phase
> of reaching consensus on architecture and choosing starting protocols, or
> whether we can reach enough consensus before the next IETF to move straight
> to standards work. I'm not sure that we can decide this right now, as it
> depends on how the discussion goes over the next month or so. There are
> some work items below that might apply in the latter case. --
> diagnostics by providers and users, as well for public policy.  {Comment:
> operators may want measurement to help network planning and network mgt
> like fault identification; not sure whether this needs explicitly noting.} 
> To conduct service measurements, Measurement Agents (MAs) on user networks
> gather data, either on their own initiative or instructed by a Controller,
> and then upload the measurement results to a designated Collector.
> Currently measurement platforms contain up to a few thousand MAs. However,
> the vision is that MAs could potentially be embedded in every home hub,
> cable modem, enterprise edge router, set top box, smartphone and so on.
> Standards will help this capability be more pervasive, manageable and
> directly comparable. RFCs from the IPPM WG define how to measure metrics -
> requests to define new tests should be handled by IPPM. These tests
> effectively define the communications between a MA and Test Server.
> {Comment: there are likely new metrics that need to be defined (eg latency
> under load) or tests under new conditions (eg current test for speed may
> not scale well to high line speeds).} . In a few cases the IPPM definition
> may need to be extended to move results data from the Test Server to the MA
> for later reporting. The LMAP WG makes the following architectural
> assumptions - in order to make rapid progress with an acceptable loss of
> generality:-
>
> Collector are all under control of same organisation (eg Samknows, FCC,
> BT). Reason: Inter-organisation interactions (for example to share results)
> *         We assume measurement results are sent from MA, and not from test
> server. Reason-1: It is easier to secure the MA to Collector communication.
> One implication is that for an upload test measured by the test server, the
> result is reported back to the MA which reports it in turn. Reason-2: Test
> servers may be controlled by a third party ('load time for IETF home
> page').
>
> *         We assume no negotiation is needed between Controller and MA. The
> Controller already knows the characteristics of the MA, for instance via
> TR069 for Broadband Forum devices. The Controller simply instructs
> (securely) the MA "Measure this"; the MA replies "OK /error".
>
> *         Similarly we assume no negotiation is needed between the MA and
> Collector. The protocol simply delivers (securely) the measurement results
> from the MA to the Collector, and returns an ack or error code.
>
> *         We assume each MA is controlled by only one Controller. Reason:
> to avoid different Controllers giving the MA conflicting schedules. Note: a
> measurement platform may have several Controllers, with each Controller
> controlling a subset of the MAs, for example one controlling wireline MAs
> and one mobile MAs.
>
> /averaging of measuremnts by the MA is not in the initial scope, except as
> defined by IPPM as part of the metric definition.
>
> *         We assume that uploading a new test capability to the MA is out
> of scope. This can be done as a firmware upgrade for home hub, or new app
> for PC, etc.
>
> stores nominal values such as the contractual data rate). The functional
> role of the Network Paraemeter Server may be fulfilled by existing OSS
> (such as network or home gateway management systems). Interfaces to the NPS
> are pre-measurement (used by the Controller to decide which tests to
> schedule) or post-measurement (used to analyse the measurement data).
>
> *         We assume data filtering is out of the initial scope. Filtering
> includes: removal of outlier measurements, archiving (curating) data for
> scientific research, and so on. There are certainly some aspects where best
> practice would be useful but data filtering is not a first priority.
>
> control of the same organisation. IPPM considers the implications of any
> passive measurement tests that it defines. The LMAP WG standardises the
> following items:
>
> b.      Test schedule: when the test should be done and how often repeated
> ('once an hour at xx.03'). A test may also be a one-off, done immediately
> on demand.
>
> e.      Environmental conditions: such as 'no user traffic' which means a
> particular instance of the test is delayed or abandoned.
>
> controlled by the operator; the initial scope will exclude this scenario.
>
> g.      (perhaps) Metadat to report: for example, the MA may be able to
> measure Layer 1 o 2 charcteristics such as DSL sync speed and interleaving;
> the actual test is out of scope. As for the previous bullet, there may be
> privacy implications for the general scenario, which we exclude from the
> initial scope.
>
> scheduled test - which should be done (or done first)? Or should it simply
> be abandoned?
>
> 2.       The data model that defines how an MA should report test results.
> This may be combined with (1) above or specified as a separate reporting
> schedule. This includes:
>
> 6.      A document explaining the simplifying architectural assumptions in
> more detail. This document could also define some terminology and explain
> the use cases. Looking forward to the discussion,
> Best wishes,
> Trevor and Phil
> Marc Linsner Sent: 18 January 2013 20:48
> To: [email-masked]
> Let's discuss possible charter language for the proposed LMAP work.  At
> this point, IMO, we should propose a charter for the work that could either
> be added to an existing WG's charter, or standalone as a WG charter. 
> That's an AD decision.  A follow-on discussion to this would obviously be
> the output documents and milestones.
>
> protocols to connect the components.  A large-scale measurement system
> needs to support residential and small-enterprise networks, using either
> wired or wireless networks. The architecture needs to consider the
> management of the active and passive measurements.
>
> management of these systems, and finally define the management
> communication architecture and protocols.
>
> concerns and will address them within its documents. "
>
> Fire away!
>
> -Marc Linsner-

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: <[email-masked]>
Date: Mon, 28 Jan 2013 17:16:10 +0000

This is a good point.
Most usually the Controller will select a subset of the Measurement Agents, and instruct them to collect measurements according to some schedule ('measure download speed once an hour'). The other MAs won't be making measurements - otherwise as you say the internet will be dominated by measurement traffic! This is fine where the operator or regulator is interested in averages-across-lines (eg to compare one ISP against another)

Because (hopefully) all home hubs etc include MA functionality, if a customer reports problems, then the Controller can tell the relevant MA to run some specific tests. This is the new bit that would be great.

Also, think the default case is to report the raw measurement. But eg some IPPM metrics define a "statistic" as well as a "singleton" measurement result - so that might be reported instead. 

phil 

-----Original Message-----
From: Mirja Kuehlewind [mailto:[email-masked]] 
Sent: 28 January 2013 17:06
To: [email-masked]
Cc: Eardley,PL,Philip,TUB8 R; Burbridge,T,Trevor,TUB8 R
Subject: measurement traffic load [was: Re: Proposed LMAP Charter]

Hi everybody,

thanks Phil for this really good proposal. I agreed that it is import to scope down the initial work to a well defined scenario and focus on basic protocol functionality.

Regarding the text below and also following the discussion on data processing, I would like to add one comment and maybe can some feedback from people how already have experience with large scale measurements:

For me the proposal seem to produce a huge amount of data. We envision a MA in basically every home network, there are frequent test scheduled by one or more 'parties' and there is all the raw data which needs to be transmitted to the collector. Thinking about this I afraid that most of our future Internet traffic will be the measurement (and collection) traffic itself. Maybe this issue is out of scope for the actual work group focus, but I guess it's worth think about this when defining the architecture/scenarios and protocols.

Mirja

tel: +49(0)711/685-67973

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Mon, 28 Jan 2013 12:24:02 -0500

Hi Mirja, see below:

> Mirja Kuehlewind
> Sent: Monday, January 28, 2013 12:06 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: [lmap] measurement traffic load [was: Re: Proposed LMAP Charter]
> 
...
> think about this when defining the architecture/scenarios and protocols.

This is a good point to raise again - it's something we briefly discussed last 
November and I doubt the traffic implications have escaped the mind of operators.
The authors of the IPPM Framework were on top of this issue for measurement
traffic, but the same goes for the results as you said.

At 06:01 PM 11/14/2012, Al Morton wrote:
It is an important tenet of the IPPM framework RFC 2330, sec6.2:
   ...
   Finally, some measurement methodologies may be 'conservative' in the
   sense that the act of measurement does not modify, or only slightly
   modifies, the value of the performance metric the methodology
   attempts to measure.  {Comment: for example, in a wide-area high-speed
   network under modest load, a test using several small 'ping' packets
   to measure delay would likely not interfere (much) with the delay
   properties of that network as observed by others.  The corresponding
   statement about tests using a large flow to measure flow capacity
   would likely fail.}

There are many reasons to prefer a "conservative" methodology,
especially if a reasonable projection or model based on fundamental
metrics is possible/available.
regards,
Al

---

From: Brian Trammell <[email-masked]>
Date: Mon, 28 Jan 2013 18:35:40 +0100

Hi, all,

> metrics is possible/available.

Good point. However, scaling up and out might make the distance between "doesn't affect the measurement" and "doesn't affect the Internet" wider than we think about in the core network measurement case. Tens of thousands of MAs on tens of thousands of home routers looking at tens of thousands of links means that even very conservative measurements, in aggregate, can lead both to significant test and reporting traffic. 

If the definition of recommended / defined test plans falls into the LMAP charter, then a requirement of those test plans should definitely include bounds on measurement and reporting traffic. And I think issues like this point to it being a good idea to define test plans, separate from the singleton and metrics they're based on.

Best regards,

Brian

---

From: "Guangqing Deng" <[email-masked]>
Date: Tue, 29 Jan 2013 09:40:07 +0800

Thanks for Mirja's point. Usually, the bottleneck links in internet are those cross-ISP links. If the collector and the MA are in different ISPs, the measurement traffic from MAs to collectors will bring huge pressure to the already busy cross-ISP links. So how to reduce the cross-ISP measurement traffic while not reducing the measurement accuracy may be a challenging problem that needs more consideration. 

Guangqing Deng
CNNIC 

From: [email-masked]
Date: 2013-01-29 01:16
To: [email-masked]; [email-masked]
CC: trevor.burbridge
Subject: Re: [lmap] measurement traffic load [was: Re: Proposed LMAP Charter]
This is a good point.
Most usually the Controller will select a subset of the Measurement Agents, and instruct them to collect measurements according to some schedule ('measure download speed once an hour'). The other MAs won't be making measurements - otherwise as you say the internet will be dominated by measurement traffic! This is fine where the operator or regulator is interested in averages-across-lines (eg to compare one ISP against another)

phil 

-----Original Message-----
From: Mirja Kuehlewind [mailto:[email-masked]] 
Sent: 28 January 2013 17:06
To: [email-masked]
Cc: Eardley,PL,Philip,TUB8 R; Burbridge,T,Trevor,TUB8 R
Subject: measurement traffic load [was: Re: Proposed LMAP Charter]

Hi everybody,

Mirja

tel: +49(0)711/685-67973