Subject: Re: [lmap] LMAP: situation: Q1-Use Cases

From: Jérôme Benoit <[email-masked]>
Date: Thu, 2 May 2013 17:50:28 +0200

On Fri, 19 Apr 2013 11:29:40 +0100 in
<[email-masked]>,
[email-masked] <[email-masked]> wrote:

> 
> > Question 1. Use cases
> > Do we agree that we want to focus on the Internet Service Provider and
> > the End User Network Diagnostics use cases for now? As a phase 2, we
> > could focus on the 3rd party use case: multi-provider, regulator, over
> > the top
> > Question: in the End User Network Diagnostics use case, I'm unclear if
> > the user initiated measurement is included?
> 
> Here's a proposed answer:-
> 
> In order to simplify the security and policy considerations, we propose that LMAP phase 1 should make the assumption that the measurement system is under the control of a single organisation. 'Measurement system' refers to the Measurement Agent, Controller and Collector. There is likely to be interest in LMAP phase 2 in scenarios where the measurement system is not under the control of a single organisation.
> 
> Note that the Measurement Peer may be under another organisation's control (for example, the Measurement Peer might be in example.com, with the Measurement Task to download example.com's homepage or to send a request to its DNS server). However, this doesn't affect LMAP's 'single organisation' assumption since the job of defining Measurement Tasks is left to IPPM.

I'm not sure the single org approach will solve the lack of coherency
between each logical component inside the measurement system. inter or
intra do not matter (I still disagree on the single org that will be
responsible of the system but I understand that it's only a phase) : the
measurement system need to keep track of the relevant state,
characteristics and capabilities of each logical component. It's a
mandatory component to do CAC. 

Use case : 10 min of UDP stream to a landmark run to a single
landmark. Can the landmark handle the network load ? Can the MA infer
cross-traffic (via UPnP if runt behind the CPE, via an other
measurement method) ?  

> 
> Discussion has revealed one new capability that may be required. Before a Measurement Task actually runs, the Measurement Agent may need to check that the Peer can carry out the Task. If one considers this as an initial stage of the Task, then it would be something for IPPM to define.

The load is just one aspect of the problem. The real problem is that
some measurement method need to know about the states in the measurement
system : do I have enough landmark ? do I have enough MA (free slot) ?
do I have enough residual bandwidth ? 

Before thinking about on how to implement the decision tree that will
trigger a measurement task, the first task it the have a component
that will permit to build the decision tree : the states,
characteristics and capabilities tracker of each logical component. 

Regards,  

-- 
Jérôme Benoit aka fraggle
La Météo du Net - http://grenouille.com
OpenPGP Key ID : 9FE9161D
Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D

---

From: Benoit Claise <[email-masked]>
Date: Sun, 05 May 2013 16:45:15 +0100

Philip,
> Since Benoit probably won't read all 44 emails in this thread, I tried to draft an answer to his original question.
I read them all, but with a little bit of delay, as you can see.
Your summary is very much appreciated
See in line.
> Is it a fair summary of the consensus?
>
>> Question 1. Use cases
> So in terms of Benoit's questions:
> - The Internet Service Provider use case is in scope, assuming that the ISP controls the functions for the Measurement Agent, Controller and Collector
> - The regulator use case is in scope, assuming that the regulator controls the functions for the Measurement Agent, Controller and Collector
I'm wondering whether the previous two points are not  incompatible?
Most ISPs have, or will have, a type of LMAP architecture anyway. When I 
call my provider (home network), they can already do the type of LMAP 
tests on my router. So what use case is left of the regulator (knowing 
the assumption that a MA is under the administration of a single entity)?

Regards, Benoit
> - The End User Network Diagnostics use case is not directly in scope. However, the end user can get its MA to run Measurement Tasks if it wants and report results to whoever it wants; LMAP doesn't try to stop this happening.
>
>
>
> On separate topics (but should be mentioned somewhere in reply to Benoit)
>
> We have also discussed about the bootstrapping of a Measurement Agent (for example, about what Controller to contact). Here the consensus is to define the process but not a protocol. Because bootstrapping could be done in different ways, depending on the device and the measurement system, for instance: loaded at manufacture, updated locally via USB port, or orchestrated via a protocol.
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap
>
>

---

From: <[email-masked]>
Date: Tue, 7 May 2013 15:41:00 +0100

> > - The Internet Service Provider use case is in scope, assuming that
> > the ISP controls the functions for the Measurement Agent, Controller
> > and Collector
> > - The regulator use case is in scope, assuming that the regulator
> > controls the functions for the Measurement Agent, Controller and
> > Collector
> Most ISPs have, or will have, a type of LMAP architecture anyway. When
> I call my provider (home network), they can already do the type of LMAP

Not sure I get your point - anyway, I don't think they're incompatible.

The MAs, controller and collector(s) are all under common control (single measurement system). 
>From a commercial point of view this looks the same as existing systems - either samknows style measurement systems operated on behalf of a regulator - or (as you say) where an ISP has measurement functions in a router. 
Lmap would enable this sort of system to scale better, allow multi-vendor operation, etc

Thanks
phil

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Tue, 7 May 2013 13:08:26 -0400

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Tuesday, May 07, 2013 10:41 AM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; j.schoenwaelder@jacobs-
> university.de; [email-masked];
> [email-masked]; [email-masked]; STARK, BARBARA H
> Subject: Re: [lmap] LMAP: situation: Q1-Use Cases
> 
> From a commercial point of view this looks the same as existing systems -
> either samknows style measurement systems operated on behalf of a
> regulator - or (as you say) where an ISP has measurement functions in a
> router.

I think, when an ISP has a measurement system, and can provide the required
standardized measurements in the regulator's desired format, then the regulator's
need to deploy their own system would be obviated. If this is what Juergen 
meant by incompatible, I get it. Mutually exclusive might be a way express it.
(but see the next message)

Al

---

From: Benoit Claise <[email-masked]>
Date: Mon, 13 May 2013 18:38:11 +0200

> Not sure I get your point - anyway, I don't think they're incompatible.
Let me more explicit.
Example: A home, with a single router and a single PC.
ISP use case: the MA is on the router, under ISP supervision.

Then what's left for the regulator use case?
- The ideal regulator situation is to get a MA on the router. But the 
router is under the ISP supervision in this case. That breaks the 
assumption that a MA is under a single supervision.
-  So it has to be on my personal device? Do you envision the regulator 
forcing me to install something on my personal device because the router 
is under the ISP supervision?
- Or do you envision the regulator forcing me to install an extra 
probe/MA box in my network?
- Or, because the router is taken, and I don't want to install anything 
else in my network, the regulator should get the test reports from the ISP?

Regards, Benoit

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Mon, 13 May 2013 16:17:07 -0400

Hi Benoit,

I appreciate that one organization's implementation of LMAP
may constrain another's, especially if they both intend to measure
almost exactly the same portion of the network. An alternative when
the ISP is using an MA on their own gateway/router is
that the regulator asks users to volunteer to make measurements,
and that's been done many times already with a "white-box". It seems
this could be extended to an embedded client with reasonable 
qualifications, especially connectivity, listening for competing user traffic 
(that's simple in your single case with a single PC)
and perhaps others. An environment with some overlapping measurement coverage
seems very likely (to several people on the list).

> else in my network, the regulator should get the test reports from the

I suspect that the agreed measurements and desired results, collected at large scale, 
and reported in the way a regulator wants, would go a long way to support information gathering
without the complexity of maintaining a measurement system. 

Al

> Benoit Claise
> Sent: Monday, May 13, 2013 12:38 PM
> To: [email-masked]
> > >From a commercial point of view this looks the same as existing systems
> - either samknows style measurement systems operated on behalf of a

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Mon, 13 May 2013 20:47:20 +0000

> - The ideal regulator situation is to get a MA on the router. But the router is
> under the ISP supervision in this case. That breaks the assumption that a MA
> is under a single supervision.

I disagree that this breaks that assumption. The regulator negotiates (out of band) with the ISP as to what measurements get run, measurement peers, etc. The ISP manages the router, manages running of the measurements, and collects the data. The ISP then provides the regulator with the agreed-upon data (which will have been associated with network parameters and scrubbed for privacy, as deemed appropriate).

> forcing me to install something on my personal device because the router is
> under the ISP supervision?
> - Or do you envision the regulator forcing me to install an extra probe/MA
> box in my network?

Forcing is a harsh word. I'm only aware of programs involving voluntary participation, to date. If you wish to participate, then to do so, you may find (in the case where the ISP router doesn't support the regulator-desired MA function) that you are required to have the extra box or install something on a personal device. If you don't want to do this, don't volunteer to participate.

> - Or, because the router is taken, and I don't want to install anything else in
> my network, the regulator should get the test reports from the ISP?

What's wrong with the regulator getting test reports from the ISP? It appears to me that this is exactly what some regulators *want* to do. That is, it seems that some regulators would prefer to get the data from the ISP. If that isn't possible (ISP doesn't provide a router, router is unmanaged, router is too old and can't be updated to support the desired measurements, etc.), then there needs to also be the possibility of the 3rd party MA, with control and data collection done by that same 3rd party (which could be the regulator, but more likely is someone the regulator contracted with -- but all parts under one operational domain). And if the router can't do the needed measurements and you don't want to install anything, then you don't participate in the regulatory measurement collection program.

3rd party regulator MAs (in separate boxes or personal device apps) are much more about the case where the router can't do the measurements, than they are about the regulator not wanting or not being able to get measurements that an ISP has collected from an ISP router. [Note: Many people don't have a router provided or managed by the ISP. That includes me. Neither of my 2 ISPs offers me a router.]
Barbara

---

From: "Sharam Hakimi" <[email-masked]>
Date: Mon, 13 May 2013 16:49:31 -0400

Benoit,

I don't think that the regulator can force you to install any MA on your
PC. You might want to if you are unhappy with the results that the ISP
provides, as you might not agree with their results. Then installing a
MA that is not under an ISP control might help in resolving the issue.

One problem with testing from the MA in an ISP router is that the
customer interface(connected to the customer network) and capabilities
of the ISP router will not be tested. This is because tests are
generated within the ISP router with the MA and then sent on the
outbound interface to the ISP. Therefore if there are any issues from
the customer network side of the ISP router it will not be seen. Having
an MA on the PC that you mentioned will test the entire path including
the customer interface of the ISP router.

It is possible to have other MA agents under ISP control in devices at
the same connection point as the PC mentioned below that would be able
to test the entire path.

Regards,
Sharam

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
MORTON JR., ALFRED C (AL)
Sent: Monday, May 13, 2013 4:17 PM
To: Benoit Claise; [email-masked]
Cc: [email-masked]; [email-masked];
[email-masked];
[email-masked]; [email-masked];
[email-masked]; STARK,BARBARA H
Subject: Re: [lmap] LMAP: situation: Q1-Use Cases

Hi Benoit,

qualifications, especially connectivity, listening for competing user
traffic 
and perhaps others. An environment with some overlapping measurement
coverage

> else in my network, the regulator should get the test reports from the

I suspect that the agreed measurements and desired results, collected at
large scale, 
and reported in the way a regulator wants, would go a long way to
support information gathering
without the complexity of maintaining a measurement system. 

Al

---

From: Benoit Claise <[email-masked]>
Date: Tue, 14 May 2013 12:26:24 +0200

On 13/05/2013 22:47, STARK, BARBARA H wrote:
> What's wrong with the regulator getting test reports from the ISP?
There is nothing wrong. I'm trying to understand the different use 
cases, and, as a consequence, which problem(s) this group want to solve.

Regards, Benoit
> It appears to me that this is exactly what some regulators *want* to do. That is, it seems that some regulators would prefer to get the data from the ISP. If that isn't possible (ISP doesn't provide a router, router is unmanaged, router is too old and can't be updated to support the desired measurements, etc.), then there needs to also be the possibility of the 3rd party MA, with control and data collection done by that same 3rd party (which could be the regulator, but more likely is someone the regulator contracted with -- but all parts under one operational domain). And if the router can't do the needed measurements and you don't want to install anything, then you don't participate in the regulatory measurement collection program.
>

---

From: Benoit Claise <[email-masked]>
Date: Tue, 14 May 2013 12:28:24 +0200

Philip,
> Benoit
> Thanks for clarifying. Basically agree with Al and Sharam's points.
>
> I think there are several ways this could happen.
> - regulator has MAs with different subset of customers (as today)
> - regulator installs MA in parallel with ISP's (so some customers have MA from both regulator and ISP)
> - ISP lends some MAs to the regulator (which get Instructions from the regulator's Controller and report to the regulator's Collector) (The "lending protocol" is out of scope)
> - ISP gives regulator some results (per MA or aggregated) (ie from ISP's Collector)
> - etc
>
> Personally I think the LMAP work is unaffected - we simply do the control protocol and report protocol.
Thanks for the clarification.

Regards, Benoit
>> From: Sharam Hakimi [mailto:[email-masked]]
>> Sent: 13 May 2013 21:50
>> To: MORTON JR., ALFRED C (AL); Benoit Claise; Eardley,PL,Philip,TUB8 R
>> Cc: [email-masked]; Burbridge,T,Trevor,TUB8 R; j.schoenwaelder@jacobs-
>> [email-masked]; [email-masked]; STARK,BARBARA H
>> Subject: RE: [lmap] LMAP: situation: Q1-Use Cases
>>
>> Benoit,
>>
>> I don't think that the regulator can force you to install any MA on
>> your PC. You might want to if you are unhappy with the results that the
>> ISP provides, as you might not agree with their results. Then
>> installing a MA that is not under an ISP control might help in
>> resolving the issue.
>>
>> university.de;
>> I appreciate that one organization's implementation of LMAP may
>> constrain another's, especially if they both intend to measure almost
>> exactly the same portion of the network. An alternative when the ISP is
>> using an MA on their own gateway/router is that the regulator asks
>> users to volunteer to make measurements, and that's been done many
>> times already with a "white-box". It seems this could be extended to an
>> embedded client with reasonable qualifications, especially
>> connectivity, listening for competing user traffic (that's simple in
>> your single case with a single PC) and perhaps others. An environment
>> with some overlapping measurement coverage seems very likely (to
>> several people on the list).
>>
>>> else in my network, the regulator should get the test reports from
>> the
>>> ISP?
>> I suspect that the agreed measurements and desired results, collected
>> at large scale, and reported in the way a regulator wants, would go a
>> long way to support information gathering without the complexity of
>> maintaining a measurement system.
>>
>> Al
>>
>>> else in my network, the regulator should get the test reports from
>> the
>>> ISP?
>>>
>>> Regards, Benoit
>>>

---

From: <[email-masked]>
Date: Tue, 14 May 2013 08:39:05 +0100

Benoit

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 14 May 2013 13:21:16 +0000

A few points -

1) A single measurement controller sounds great however when considering multiple technology domains (wireless vs. wireline) and separate operational domains such as regional network management op's domains - it's simply not practical. Also the fact that a controller may also be used for traffic study which is much more intensive than a simple performance study makes regional domain controllers much more flexible, and desirable.
  - therefore I suggest we not constrict to any "single controller" 

2) Probe ownership isn't the problem - The Complexity of sold speed is - it requires validation, which requires a protocol...
	Having done some analysis on the results data from the Measuring Broadband America myself and others have found that it's critical to know the "promised speed" vs. assuming you know the speed.   Terms of service have changed over time to include use cap's that can change your expected speed, and if we're truly international in observing service definition, the expected speeds can change with the time of day.
   To this point, if the results are going to have solid scientific methods they almost certainly require a time of day expected or promised speed number for the ISP.   

    - I recommend part of the scope to include a protocol to obtain that expected speed from the ISP.
	 Placing the correct information in the hands of the customer, ISP and regulators is good - so the data needs to be as solid as possible.

		Personally I don't recommend "splicing" the information after the fact, I think a test protocol that retrieves expected speed at the time of 		the test is the way to go.

Thanks,
Mike

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]] 
Sent: Tuesday, May 14, 2013 5:28 AM
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; Bugenhagen, Michael K; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP: situation: Q1-Use Cases

Philip,
> - ISP lends some MAs to the regulator (which get Instructions from the 
> regulator's Controller and report to the regulator's Collector) (The 
> "lending protocol" is out of scope)

Regards, Benoit
>> your PC. You might want to if you are unhappy with the results that 
>> the ISP provides, as you might not agree with their results. Then 
>> customer interface(connected to the customer network) and 
>> capabilities of the ISP router will not be tested. This is because 
>> tests are generated within the ISP router with the MA and then sent 
>> on the outbound interface to the ISP. Therefore if there are any 
>> issues from the customer network side of the ISP router it will not 
>> be seen. Having an MA on the PC that you mentioned will test the 
>> entire path including the customer interface of the ISP router.
>>
>> It is possible to have other MA agents under ISP control in devices 
>> at the same connection point as the PC mentioned below that would be 
>> able to test the entire path.
>>
>> Regards,
>> Sharam
>>
>>
>> From: [email-masked] [mailto:[email-masked]] On Behalf 
>> Of MORTON JR., ALFRED C (AL)
>> exactly the same portion of the network. An alternative when the ISP 
>> is using an MA on their own gateway/router is that the regulator asks 
>> times already with a "white-box". It seems this could be extended to 
>> an embedded client with reasonable qualifications, especially 
>>> - The ideal regulator situation is to get a MA on the router. But 
>>> the router is under the ISP supervision in this case. That breaks 
>>> the assumption that a MA is under a single supervision.
>>> regulator - or (as you say) where an ISP has measurement functions 
>>> in
>> a
>>> router.

---

From: <[email-masked]>
Date: Tue, 14 May 2013 15:27:00 +0100

>From: Bugenhagen, Michael K
>[mailto:[email-masked]]
>Sent: 14 May 2013 14:21
>To: 'Benoit Claise'; Eardley,PL,Philip,TUB8 R
>Cc: [email-masked]; [email-masked]; [email-masked];
>Burbridge,T,Trevor,TUB8 R; [email-masked];
>[email-masked]; [email-masked]; [email-masked]
>1) A single measurement controller sounds great however when considering
>multiple technology domains (wireless vs. wireline) and separate operational
>domains such as regional network management op's domains - it's simply
>not practical. Also the fact that a controller may also be used for traffic study
>which is much more intensive than a simple performance study makes
>regional domain controllers much more flexible, and desirable.

We were suggesting a single controller at any point in time for a single MA to avoid any negotiation between MA and controllers over resources. Of course you can have multiple disjoint sets of MAs under multiple controllers with this restriction.

I'm not sure but I hope you're not suggesting franchised geographic controllers. You can't force someone to control their tests through another party.

>2) Probe ownership isn't the problem - The Complexity of sold speed is - it
>requires validation, which requires a protocol...
>	Having done some analysis on the results data from the Measuring
>Broadband America myself and others have found that it's critical to know
>the "promised speed" vs. assuming you know the speed.   Terms of service
>have changed over time to include use cap's that can change your expected
>speed, and if we're truly international in observing service definition, the
>expected speeds can change with the time of day.
>   To this point, if the results are going to have solid scientific methods they
>almost certainly require a time of day expected or promised speed number
>for the ISP.
>
>	 Placing the correct information in the hands of the customer, ISP
>and regulators is good - so the data needs to be as solid as possible.
>
>		Personally I don't recommend "splicing" the information
>after the fact, I think a test protocol that retrieves expected speed at the
>time of 		the test is the way to go.

Splicing extra information into the measurement data "after the fact" _will_ occur. There is a mass of operational data that I use to analyse the performance of my lines that cannot possibly be shared with the MA (because there's too much of it, I don't know what it is at the time, it's not available at the time of testing or I don't trust the MA with it).

The question more appropriately is should a subset of this information be shared with the MA to report back along with the measurement data? If this is for the end-user I also don't know that the measurement results and/or the line characteristics couldn't be shared via a web portal. Is there really a need to give this to the MA in a standard way?

Trevor.