Subject: [lmap] user-initiated testing

From: Alissa Cooper <[email-masked]>
Date: Tue, 30 Jul 2013 10:02:37 +0200

Just to clarify from the discussion at the mic:

I don't think the eventual LMAP use cases document should say that user-initiated measurements are out of scope. I think they are in scope.

With that said, at this point it's not clear that user-initiated tests impose any additional requirements beyond the ISP use case and the regulator use case. User-initiated tests could use, at a minimum, the same information model that will be defined to support the other two use cases. User-initiated tests could also potentially make use of the reporting protocol if the user wants to report the results to a collector.

Alissa

---

From: Benoit Claise <[email-masked]>
Date: Tue, 30 Jul 2013 10:12:38 +0200

Exactly my thoughts.
Thanks Alissa

Regards, Benoit
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap
>
>

---

From: Brian Trammell <[email-masked]>
Date: Tue, 30 Jul 2013 10:16:53 +0200

hi Alissa, all,

> I don't think the eventual LMAP use cases document should say that user-initiated measurements are out of scope. I think they are in scope.

+1

> With that said, at this point it's not clear that user-initiated tests impose any additional requirements beyond the ISP use case and the regulator use case. User-initiated tests could use, at a mi…

Indeed, and to clarify my point from the mic, we should make a distinction between user-requested (i.e., user sends a request to the LMAP infrastructure, which initiates a measurement on the user's behalf) and user-run (controller runs on the user device, or measurement agent runs on the user device without the controller's involvement). The scalability issues that Paul and Phil were worried about only affect the latter case (and I personally agree it's probably a bad idea to try to accommodate it).

Cheers,

Brian

---

From: Manner Jukka <[email-masked]>
Date: Tue, 30 Jul 2013 08:38:30 +0000

Hi,

A comment to the latter item, i.e., scalability of user-run measurements, load balancing and load control solves the issue quite well. 

Our "Netradar" (available on iOS, WP, Android, Meego, Symbian, and data at netradar.org) mobile network measurements service is using end-user run measurements. We have measurement points globally distributed on three continents and control the number of users to each individual measurement server. So far has worked like a charm, for over a year. If the load, i.e. number of active MAs, grows, we can power up more measurement points and balance the load on them. So, user-run is not a problem if one plans for it.

I'm here in Berlin up to Wednesday to chat more if needed.

cheers,
Jukka

---

From: "MORTON JR., ALFRED C (AL)" <[email-masked]>
Date: Tue, 30 Jul 2013 07:10:12 -0400

On User Initiated tests: Time was short and I couldn't make this point at the critical time.

Yesterday, we (Paul and Phil and I)  summed-up the related discussion with the term "User Requested",
as-in users can initiate measurement taks from the LMAP infrastructure with the help of 
the infrastructure, and then other desirable features follow (such as selection of the best MAs or
MPs and the ability to avoid inaccurate results due to overloaded test devices).

User initiated measurement requests are important. The evidence shows that almost everybody
wants to run tests, but they also want accurate and meaningful results, and we won't really
achieve LMAP's intent without delivering both.

Al

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Brian Trammell [[email-masked]]
...
Indeed, and to clarify my point from the mic, we should make a distinction between user-requested (i.e., user sends a request to the LMAP infrastructure, which initiates a measurement on the user's behalf) and user-run (controller runs on the user device, or measurement agent runs on the user device without the controller's involvement). The scalability issues that Paul and Phil were worried about only affect the latter case (and I personally agree it's probably a bad idea to try to accommodate it).

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Tue, 30 Jul 2013 14:46:35 +0100

+1

If an end-user wants to initiate a test, they must somehow contact a 
controller.

eg, even if the user has an MA app on their smart phone, the "test now" 
button must somehow indicate the test request to the cellular provider's 
controller, which then uses the LMAP framework to initiate the test. We 
do not envisage a combined controller/MA device.

This is important for several reasons which include:

     * only the controller has a "big picture" view
         eg if 1000's of users request a bandwidth test about the same 
time, the bandwidth server could be overloaded and give the wrong results.

     * the controller can ensure appropriate conditions for the test to 
be run
         eg if a test requires quiescent conditions on the measurement 
peer, only the controller can ensure that no other tests are scheduled 
against that peer at the same time.

     * individual user's measurements may have little meaning on their 
own, ie there's no baseline or context to be compared against.
         the controller can schedule several tests from similar nodes 
for comparison.

     * the collector would not be expecting the reports
         the collector may not currently have resources or ability to 
process the test results

     * combined controller/MA device puts the controller under control 
of the end users rather than within the control of a single organisation.
         so it's outwith the LMAP scope.

So end-users _can_ request tests using the LMAP infra in the framework, 
and not by some other means.

However, end-users aren't under control of the organisation, so they're 
out of scope :-)

P.

---

From: [email-masked]
Date: Tue, 30 Jul 2013 10:41:57 -0400

> De : Paul Aitken <[email-masked]>
>

Following this from the side lines (I am not in Berlin), I'd like to voice 

some support to Alissa's point of vue. I believe there is some value in 
having user-initiated measurements in-scope, as long as the software used
(either an app or a web-based interface) is consistent and is using a 
different collector than the ISP-owned MAs. 

As mentioned in the use-case draft, reliable statistics can't be produced
from such a self-selecting group. But for an ISP, there is some value in 
seeing measurements from end-users and gathering statistics from their 
point of view. This is a rather rough measurement of the user's quality
of experience, but it does give an idea of how that experience measures
up to an ISP's advertised services. 

Videotron (my employer) actually already performs such measurements by 
running its own speedtest service and gathering data correlated to the 
end-user tier. It gives us a high-level view of users "happiness" levels 
with regards to their services. (see http://testvitesse.videotron.ca and
http://ipv6.testvitesse.videotron.ca). Such data is desirable for an 
ISP and I believe should be in-scope. 

An ISP could, for example, publish an LMAP-compliant app for its users 
to test their speed and gather the results in a separate collector. 

/JF

---

From: "Guangqing Deng" <[email-masked]>
Date: Wed, 31 Jul 2013 01:28:20 +0800

Totally agree. What's more, different users may care about different metrics. Such users who like watching online videos may care about download speed very much; while those who prefer online games may care network delay more. So, different users may perform measurements for different metrics, if user-initiated measurements are supported. For ISPs, they can learn more about what users are paying attention to through such measurements (initiated by users) and then accordingly adjust their networks to meet the demand of different users. So it seems that having user-initiated measurement in-scope is beneficial to both users and ISPs. 

Guangqing Deng
cnnic 

From: Jean-Francois.TremblayING
Date: 2013-07-30 22:41
To: [email-masked]
Subject: Re: [lmap] user-initiated testing

/JF

---

From: "Sharam Hakimi" <[email-masked]>
Date: Tue, 30 Jul 2013 14:10:44 -0400

Agree with capability for some user initiated tests but it does not have
to involve the Controller necessarily. A Resource Authorization function
that would authorize a test being done can exist in a Controller but it
could also be a function outside the controller. With large numbers of
possible users a distributed test authorization function would work
better in my opinion.

Sharam Hakimi,  

EXFO

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Guangqing Deng
Sent: Tuesday, July 30, 2013 1:28 PM
To: Jean-Francois.TremblayING
Cc: [email-masked]
Subject: Re: [lmap] user-initiated testing

Totally agree. What's more, different users may care about different
metrics. Such users who like watching online videos may care about
download speed very much; while those who prefer online games may care
network delay more. So, different users may perform measurements for
different metrics, if user-initiated measurements are supported. For
ISPs, they can learn more about what users are paying attention to
through such measurements (initiated by users) and then accordingly
adjust their networks to meet the demand of different users. So it seems
that having user-initiated measurement in-scope is beneficial to both
users and ISPs. 

________________________________

Guangqing Deng
cnnic 

From: Jean-Francois.TremblayING
<mailto:[email-masked]> 

Date: 2013-07-30 22:41

To: [email-masked]

> out of scope :-)

Following this from the side lines (I am not in Berlin), I'd like to
voice 

having user-initiated measurements in-scope, as long as the software
used

As mentioned in the use-case draft, reliable statistics can't be
produced

/JF

lmap mailing list

[email-masked]

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 30 Jul 2013 18:30:18 +0000

With user "initiated" we really need to contextualize if we mean

1)      User testing their own Home network

Vs.

2)      User hooks into an ISP test system that executes a test for them.
Of course the key issue is who ownes the Measurement point at the User interface (DSL or Cable modem)..  These are often multi-interface, so the only aggregated interface / access test point is generally located on the ISP side of the NID vs. the customers side.
Regards,
Mike

From: [email-masked] [mailto:[email-masked]] On Behalf Of Sharam Hakimi
Sent: Tuesday, July 30, 2013 1:11 PM
To: dengguangqing; Jean-Francois.TremblayING
Cc: [email-masked]
Subject: Re: [lmap] user-initiated testing

Agree with capability for some user initiated tests but it does not have to involve the Controller necessarily. A Resource Authorization function that would authorize a test being done can exist in a Controller but it could also be a function outside the controller. With large numbers of possible users a distributed test authorization function would work better in my opinion.
Sharam Hakimi,
EXFO

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Guangqing Deng
Sent: Tuesday, July 30, 2013 1:28 PM
To: Jean-Francois.TremblayING
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [lmap] user-initiated testing

From: Jean-Francois.TremblayING<mailto:[email-masked]>
Date: 2013-07-30 22:41
To: [email-masked]<mailto:[email-masked]>
Subject: Re: [lmap] user-initiated testing
> De : Paul Aitken <[email-masked]<mailto:[email-masked]>>
>

/JF

_______________________________________________
lmap mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lmap

---

From: "Sharam Hakimi" <[email-masked]>
Date: Tue, 30 Jul 2013 14:51:08 -0400

Mike ,

I think there is one more case:

3)    User uses an ISP test system to perform a test for the user itself

Regards,

Sharam

From: Bugenhagen, Michael K
[mailto:[email-masked]] 
Sent: Tuesday, July 30, 2013 2:30 PM
To: Sharam Hakimi; dengguangqing; Jean-Francois.TremblayING
Cc: [email-masked]
Subject: RE: [lmap] user-initiated testing

Vs.

2)      User hooks into an ISP test system that executes a test for
them.

Of course the key issue is who ownes the Measurement point at the User
interface (DSL or Cable modem)..  These are often multi-interface, so
the only aggregated interface / access test point is generally located
on the ISP side of the NID vs. the customers side.

Regards,

Mike

Sharam Hakimi

Sharam Hakimi,  

EXFO

Guangqing Deng
cnnic 

Date: 2013-07-30 22:41

To: [email-masked]

/JF

lmap mailing list

[email-masked]

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 30 Jul 2013 18:52:49 +0000

Good catch....
   I'd call in a "2b" user case,,, but yes the  ISP system test from the  Broadband Access NID aggregation device to the Internet, or to the customer...

From: Sharam Hakimi [mailto:[email-masked]]
Sent: Tuesday, July 30, 2013 1:51 PM
To: Bugenhagen, Michael K; dengguangqing; Jean-Francois.TremblayING
Cc: [email-masked]
Subject: RE: [lmap] user-initiated testing

Mike ,

Regards,

Sharam

From: Bugenhagen, Michael K [mailto:[email-masked]]

Vs.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Sharam Hakimi

/JF

---

From: "Guangqing Deng" <[email-masked]>
Date: Wed, 31 Jul 2013 09:18:06 +0800

I think the ISPs' LMAP measurement infrastructure should have the ability to perform the user-initiated measurement since the users are the largest group of participants of all kinds of networks and they have the requirement that the user-initiated measurement can be performed at any time when users have a measurement requirement (for instance, when there is something wrong with users' home network). 

Guangqing Deng
cnnic 

From: Sharam Hakimi
Date: 2013-07-31 02:51
CC: lmap

With user “initiated” we really need to contextualize if we mean
1)      User testing their own Home network 
Vs.
2)      User hooks into an ISP test system that executes a test for them.
Of course the key issue is who ownes the Measurement point at the User interface (DSL or Cable modem)..  These are often multi-interface, so the only aggregated interface / access test point is generally located on the ISP side of the NID vs. the customers side.
Regards,
Mike

 
From: [email-masked] [mailto:[email-masked]] On Behalf Of Guangqing Deng
Sent: Tuesday, July 30, 2013 1:28 PM
To: Jean-Francois.TremblayING
Cc: [email-masked]
Subject: Re: [lmap] user-initiated testing

Guangqing Deng
cnnic 

 

 
 
/JF

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 31 Jul 2013 08:16:19 +0200

> Of course the key issue is who ownes the Measurement point at the User interface (DSL or Cable modem)..  These are often multi-interface, so the only aggregated interface / access test point is gen…

I think there is no reason to restrict 1) to the user's home network. I
think the cases are:

A) A user runs his own LMAP system (e.g. she owns a home router
   including an MA and he runs his own controller/collector and data
   analysis application).

B) A user hooks into a 3rd party provided LMAP measurement system and
   then uses that system to run tests and gain insights into the
   network.

With the assumption that the LMAP measurement system is under the
control of a single organization, I do not yet see that user-initiated
measurements lead to any additional requirements.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 31 Jul 2013 08:18:14 +0000

I agree with recent comments that suggest the phrase "user-initiated" has no assumptions attached to it as to (a) domain that controls MA/controller and (b) whether collected data gets reported to anyone other than the user. Collected data does not necessarily have to end up in the hands of regulators or ISPs or be used for regulatory purposes, even when it is reported to someone other than the user.

Summary of my opinion: MA and controller in ISP/measurement provider/regulator domain creates no new work items or scope expansion. If we want to consider MA and controller in the user domain, the work not currently included in the scope would be to figure out how to allow user-controlled MA access (authentication/authorization) to desired Measurement Peers, how to supply service attributes to the user domain, and how to include service attributes in reported data.

Details of my opinion:
(1) For (a) Domain = ISP, (b) don't care whether collected data reported to anyone other than user
Where domain of control is the ISP, I think posters have correctly suggested that the functionality needed to make this happen, that are above and beyond the work elements described in the current charter, should probably remain out of scope at this point. That would include elements like how to interface with the user, how the user interface gets to the controller for initiation, and how the resulting data gets to the user interface. Whether resulting data gets used/stored/collected by the service provider or reported as part of a regulatory program I believe is "don't care". The chartered work will allow for data to be reported anywhere, but doesn't insist that data be reported to regulators or stored by ISPs. There's nothing I would like to see added to currently-defined scope of work to more completely support this case. I also don't see impacts to the framework or information model.

(2) For (a) Domain = regulator or other measurement provider, (b) don't care
I think my comments for domain = ISP apply here as well.

(3a) For (a) Domain = user, (b) no data reported to anyone other than user
I think that for this case the controller is likely also to be inside the home network and may even be collocated with the MA. If user wants to test against the same Measurement Peers as get used by ISP or regulator, then I think there may be a problem with credentials, depending on whether any of the Measurement Peers require some form of authentication prior to participating in a test with the MA. There is also no means for the user to get service attributes. Currently neither of these is in scope. Personally, I consider this to be an important use case, and would be happy if these were in scope. I've floated proposals for getting service attributes to the user, and think this would be incredibly easy to do.

(3b) For (a) Domain = user, (b) user supplies data to another entity
Same issues as for (3a), but I think it would be necessary to better identify the recommended reporting protocol used to supply the data in this case, since this is a cross-domain transfer of info (which requires greater standardization than intra-domain protocols). To me this suggests that this case should be considered when selecting a reporting protocol. Also, this case implies the service attributes need to be supplied in the reporting information model. I think that including service attributes in the report information model in support of this case would be a good idea. Note that there is nothing implied in this case as to whether the other entity receiving the data has anything to do with a regulator, or that the supplied data will be used for regulatory purposes. Personally, I would object to assuming that this data would be used for regulatory purposes. The only assumption I'm making is that the receiving entity is not the end user.

Barbara

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 11:39:33 +0100

Juergen,

>     then uses that system to run tests and gain insights into the

Scenario (B) could work, since it looks something like this:

      isp   .                   ispCollector
         \  .                  /
          \ .                 /
            controller ---- MA -- {test peers}
          / .                 \
         /  .                  \
     user   .                   userCollector
          LMAP
        boundary

This is subtly, though crucially, different from the scenario in my 
previous email, because the system now falls under a single Controller.

The ISP could instruct the controller to initiate some tests reporting 
to the ispCollector, while the user instructs the controller to initiate 
tests reporting to the userCollector.

With this architecture,  everything within the LMAP boundary is in 
accordance with the WG charter and the proposed framework.

P.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 31 Jul 2013 12:07:28 +0100

Michael, Sharam,

I've no objections to this case since it's the architecture I just 
described like so:

      isp   .                   ispCollector
         \  .                  /
          \ .                 /

Note that:
     * the MA may be present on the user's device
     * the mechanism for the user to request that the controller 
initiate the test is outwith the current LMAP scope.
     * this supposes that a single collector is not programmed into the 
MA, but is indicated per test.

P.

On 30/07/13 19:52, Bugenhagen, Michael K wrote:
>
> Good catch....
>
>    I'd call in a "2b" user case,,, but yes the  ISP system test from 
> the  Broadband Access NID aggregation device to the Internet, or to 
> the customer...
>
> *From:*Sharam Hakimi [mailto:[email-masked]]
> *Sent:* Tuesday, July 30, 2013 1:51 PM
> *To:* Bugenhagen, Michael K; dengguangqing; Jean-Francois.TremblayING
> *Cc:* [email-masked]
> *Subject:* RE: [lmap] user-initiated testing
>
> Mike ,
>
> *From:*Bugenhagen, Michael K 
> *Sent:* Tuesday, July 30, 2013 2:30 PM
> *To:* Sharam Hakimi; dengguangqing; Jean-Francois.TremblayING
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Subject:* RE: [lmap] user-initiated testing
>
> 1)User testing their own Home network
>
> Vs.
>
> 2)User hooks into an ISP test system that executes a test for them.
>
> *From:[email-masked] <mailto:[email-masked]> 
> [mailto:[email-masked]] *On Behalf Of *Sharam Hakimi
> *Sent:* Tuesday, July 30, 2013 1:11 PM
> *To:* dengguangqing; Jean-Francois.TremblayING
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Subject:* Re: [lmap] user-initiated testing
>
> Agree with capability for some user initiated tests but it does not 
> have to involve the Controller necessarily. A Resource Authorization 
> function that would authorize a test being done can exist in a 
> Controller but it could also be a function outside the controller. 
> With large numbers of possible users a distributed test authorization 
> function would work better in my opinion.
>
> Sharam Hakimi,
>
> EXFO
>
> *From:[email-masked] <mailto:[email-masked]> 
> [mailto:[email-masked]] *On Behalf Of *Guangqing Deng
> *Sent:* Tuesday, July 30, 2013 1:28 PM
> *To:* Jean-Francois.TremblayING
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Subject:* Re: [lmap] user-initiated testing
>
> adjust their networks to meet the demand of different users. So it 
> seems that having user-initiated measurement in-scope is beneficial to 
> both users and ISPs.
>
> ------------------------------------------------------------------------
>
> Guangqing Deng
> cnnic
>
> *From:*Jean-Francois.TremblayING 
> *Date:* 2013-07-30 22:41
> *To:[email-masked] <mailto:[email-masked]>
>
> *Subject:* Re: [lmap] user-initiated testing
>
> > De : Paul Aitken <[email-masked] <mailto:[email-masked]>>
>
> >
>
> [email-masked] <mailto:[email-masked]>
>