Subject: Re: [lmap] LMAP regulator use case

From: <[email-masked]>
Date: Wed, 6 Nov 2013 02:42:10 +0000

Frode
thanks very much for the comments - and it is excellent to have detailed text suggestions.

2.2
I think i'm basically OK with this replacing the current S2.2 (including the subsections)
I suggest that it should mention comparability in the final para (as well as in the penultimate one) - personally I think this is the most important characteristic from a regulator's point of view, so that it's fair to compare measurements they make across the different ISPs.
the one thing in the current S2.2 (actually in S2.2.3) that I think is worth keeping is a mention of fixed and mobile. I think this could be handled by adding to the Intro (S1) something like: "Measurements will take place on both fixed and mobile access"

Happy to add a new chapter 4, but comments about some of your sub use cases

4.1 Promoting competition through transparency
>>For competition to successfully discipline operators' behaviour in the interests of their customers, end users must be fully aware of the characteristics of the ISPs' access offers.
"must be fully aware" over-states things.

>>The system can be operated by a regulator or a measurement provider.
add "on behalf of the regulator"

>>  However, the quality of an Internet access service is also determined by the connectivity to the rest of the Internet. Therefore test configurations which measure beyond the ISP's own network may also be relevant in some cases.
true. perhaps worth mentioning that the ISPs can influence this to some extent - how well they're connected to the Internet, how they cache popular content etc.

4.2 Promoting end user empowerment
i think this can be deleted. this is the end user use case, not the regulator use case.

4.3 Monitoring overall market development
good for me

4.4 Detecting degradation of the Internet access
4.5 Detecting degradation of specific applications

I don't like these sections about testing for whether the ISP is being net neutral.
I think it would be very hard to design tests.
In part this is because net neutrality is more of a political concept than technical one (in my view)

i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market development issue (the internet has got slower, and doesnt meet the policy objective of Superfast broadband by Year 2015 - or whatever)-

best wishes
phil

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Sørensen, Frode [[email-masked]]
Sent: 30 October 2013 14:55
To: [email-masked]
Subject: [lmap] LMAP regulator use case

Marc, Phil, Trevor, all,

I have prepared a proposed description of a regulator use case for the LMAP use case ID. I noticed the "advertisement" for further input from regulators for the use case a couple of weeks ago. Finally I have managed to find sufficient time to contribute to the ID, supported by a couple of "regulator friends" from the Net Neutrality Working Group of the BEREC organization, representing European national regulators. The proposal consists of two parts; a modification of section 2.2 "Regulators", plus a new chapter after chapter 3 "Details of ISP use case", titled "Details of regulator use case".

Thanks,
Frode

Proposed update:

2.2 Regulators
-------------------
Regulators in jurisdictions around the world are responding to consumers' adoption of Internet access services for traditional telecommunications and media services by promoting competition among providers of electronic communications, to ensure that users derive maximum benefit in terms of choice, price, and quality.

Some jurisdictions have responded to a need for greater information about Internet access service performance in the development of regulatory policies and approaches for broadband technologies by developing large-scale measurement programs. Programs such as the U.S. Federal Communications Commission's Measuring Broadband America, European Commission's Quality of Broadband Services in the EU reports and a growing list of other programs employ a diverse set of operational and technical approaches to gathering data to perform analysis and reporting on diverse aspects of broadband performance.

While each jurisdiction responds to distinct consumer, industry, and regulatory concerns, much commonality exists in the need to produce datasets that are able to compare multiple Internet access service providers, diverse technical solutions, geographic and regional distributions, and marketed and provisioned levels and combinations of broadband Internet access services. In some jurisdictions, the role of measuring is provided by a measurement provider.

Measurement providers measure network performance from users towards multiple content and application providers, included dedicated test measurement servers, to show a performance of the actual Internet access service provided by different ISPs. Users need to know the performance that are achieving from their own ISP. In addition, they need to know the performance of other ISPs of same location as background information for selecting their ISP. Measurement providers will provide measurement results with associated measurement methods and measurement metrics.

>From a consumer perspective, the differentiation between fixed and mobile (cellular) Internet access services is blurring as the applications used are very similar. Hence, similar measurements will take place on both fixed and mobile Internet access services.

Regulators role in the development and enforcement of broadband Internet access service policies also require that the measurement approaches meet a high level of verifiability, accuracy and provider-independence to support valid and meaningful comparisons of Internet access service performance

LMAP standards could answer regulators shared needs by providing scalable, cost-effective, scientifically robust solutions to the measurement and collection of broadband Internet access service performance information.

Proposed new chapter:

4 Details of regulator use case
---------------------------------------
4.1 Promoting competition through transparency
---------------------------------------------------------------
Competition plays a vital role in regulation of the electronic communications markets. For competition to successfully discipline operators' behaviour in the interests of their customers, end users must be fully aware of the characteristics of the ISPs' access offers. In some jurisdictions regulators mandate transparent information made available about service offers.

End users need effective transparency to be able to make informed choices throughout the different stages of their relationship with ISPs, when selecting Internet access service offers, and when considering switching service offer within an ISP or to an alternative ISP. Quality information about service offers can be provided based on performance measurements of the services provided in the market.

A set of measurement parameters and associated measurement methods are used over time, e.g. speed, delay, and jitter. Then the measurement raw data are collected and go through statistical post-processing before the results can be published in an Internet access service quality index to facilitate end users' choice of service provider and offer.

In order to make information on offerings more meaningful and comparable to end users, common frames of reference on Internet access service quality metrics are essential. An example would be that in relation to access speed, the actual download and upload speeds are measured and published, not only the maximum speed.

A measurement system that monitor Internet access services and collect quality information can typically consist of a number of measurement probes and one or more test servers located at peering points. The system can be operated by a regulator or a measurement provider.  Number and distribution of probes follows specific requirements depending on the scope and the desired statistical reliability of the measurement campaign.

In many test configurations, the measurement results will only reflect the performance of ISP's own network. However, the quality of an Internet access service is also determined by the connectivity to the rest of the Internet. Therefore test configurations which measure beyond the ISP's own network may also be relevant in some cases.

4.2 Promoting end user empowerment
--------------------------------------------------
In addition to the advantages that end users receive from the competition in the market, through availability of a plethora of ISPs and Internet access service offers, facilitated by transparent quality information about these offers at a general level, end users would further leverage on information about specific services they subscribe to, or which they are considering to subscribe to.

Performance of Internet access service offers vary based on where the service is provided (geographical and topological aspects), how it is provided (e.g. access technology and ISP's interconnection to the Internet) and when it is provided (variation through the day and over the week). Therefore there is a need to obtain more detailed quality information about specific service offers.

Quality measurement tools can be made available for end users to monitor their Internet access service, giving information about the speed and other quality metrics of the subscribed service. Speed meters and similar tools can be provided by regulators or other third parties, but also directly by the ISP for the benefit of the subscriber.

Such measurement tools assist end users in verifying the performance of the Internet access service provided, comparing the measurement results achieved with the information given in the contract. If the contracted service quality level is not met, end users can use these results when complaining to the ISP to achieve improvement and compensation.

It may also help end users to assess whether they have chosen the package which best suits their specific needs. With their performance results at hand, they can benefit from quality evaluation guides, which offer comparisons based on their specific usage profiles. This may enable them to check whether their current subscription is the most appropriate, and assist them to switch package or provider if it is in their best interest.

When provided by regulators and other third parties, such quality measurement tools can also be used to compare performance between ISPs. It should, though, be recognised that measurement results can be affected by factors outside the ISP's control, related for instance with local network connection (e.g. Wi-Fi), equipment and software being used.

4.3 Monitoring overall market development
---------------------------------------------------------
Governments sometimes set strategic goals for high-speed Internet access penetration as an important component of the economic, cultural and social development of the society. To evaluate the effect of the stimulated growth over time, broadband Internet access take-up and penetration of high-speed access can be monitored through measurement campaigns.

An example of such an initiative is the "Digital Agenda for Europe" which was adopted in 2010, to achieve universal broadband access. The goal is to achieve by 2020, access for all Europeans to Internet access speeds of 30 Mbps or above, and 50% or more of European households subscribing to Internet connections above 100 Mbps.

To monitor actual broadband Internet access performance in a specific country or a region, extensive measurement campaigns are needed. A panel can be built based on operators and packages in the market, spread over urban, suburban and rural areas. Probes can then be distributed to the participants of the campaign.

Periodic tests running on the probes can for example measure actual speed at peak and off-peak hours, but also other detailed quality metrics like delay and jitter. Collected data goes afterwards through statistical analysis, deriving estimates for the whole population which can then be presented and published regularly.

Using a harmonized or standardised measurement methodology, or even a common quality measurement platform, measurement results could also be used for benchmarking of providers and/or countries.

4.4 Detecting degradation of the Internet access
--------------------------------------------------------------
Regulation related to net neutrality and the open Internet has been introduced in some jurisdictions' Internet policy, and monitoring methods for detection of potential net neutrality breaches is needed to follow up such regulations with concrete actions.

Net neutrality is typically related to equal treatment of traffic transmitted over the Internet access service, while other traffic transmitted in parallel over the end user's broadband connection e.g. to provide IPTV in IP networks separated from the Internet (also referred to as "closed IP networks") can be exempted from net neutrality regulation. Services provided in parallel to the Internet access services are referred to as specialised services (ref. BEREC and FCC).

In order for such regulatory service architecture to protect net neutrality and the open Internet, it is essential that the specialised services are not provided at the expense of the Internet access service. Therefore regulators may need to measure performance of the Internet access service over time, and thereby detect whether this service is becoming degraded.

When regulators monitor this development to ensure net neutrality and the open Internet, they can check e.g. whether Internet access service performance and level of quality reflect advances in technology and that this is not impaired by specialised services. Comparison between ISPs or between different countries may also be relevant for this kind of evaluation.

Based on such Internet access service quality monitoring, regulators will typically report results and findings regularly, e.g. on an annual basis. In some jurisdictions regulators may also eventually impose minimum quality of service requirements on ISPs in order to prevent degradation of Internet access services.

4.5 Detecting degradation of specific applications
---------------------------------------------------------------
In the regulatory context related to net neutrality and the open Internet, in addition to the monitoring to detect potential degradation of the access service as a whole described above, it is essential to perform monitoring to detect potential degradation of individual applications using the Internet access.

Since net neutrality relates to equal treatment of traffic transmitted over the Internet access service, the treatment of different applications' traffic is in the core of net neutrality. When traffic originating from different applications are treated independent of which application each IP packet belongs to, this is referred to as application agnosticism.

Examples of departure from application agnosticism are blocking or throttling of traffic from specific applications, but also preferential treatment of specific applications. If traffic is forwarded at different priority levels, traffic having higher priority level implies that other traffic has lower priority. Detection of such traffic management practices does not necessarily imply breaches to net neutrality, but these observations can be used as an input to the regulatory evaluation of the practice.

When regulators monitor traffic management practices to ensure net neutrality and the open Internet, they can check compliance with criteria defining what is considered reasonable traffic management. Measurement systems designed to monitor this kind of effects need to send application-specific traffic and then measure in detail the behaviour of the different packets receive when transferred over the Internet access service.

Today there exist measurement tools that can detect differentiated treatment of individual applications. Another method that can be used is to measure performance at application layer, e.g. assisted by content and application providers, but also passive measurements are foreseen as a promising approach for monitoring of application-specific treatment.

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Nieminen Klaus <[email-masked]>
Date: Wed, 6 Nov 2013 05:19:16 +0000

Hi Philip, Frode and all,

As one of the  regulator friends that Frode mentioned, I would like to tell how we see these issues in Finland as I'm working for the Finnish telco regulator.

I would like to see this chapter to be included. As said publishing transparent and comparable information is important but so it's also end user's ability to verify the performance of the Internet access service provided, comparing the measurement results achieved with the information given in the contract. If the contracted service quality level is not met, end users can use these results when complaining to the ISP to achieve improvement and compensation.

If you want to include it to end user requirements it is ok for me to reference it from the regulator part. However I do not want it to be totally deleted as it is also our requirement not only end user one.

Yes I know that this is a hard problem how to measure NN. However it is a clear regulatory requirement at least in the EU. See Directive 2009/136/EC http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:337:0011:01:EN:HTML 

According to Art 22(3), “in order to prevent the degradation of service and the hindering or slowing down of traffic over networks, Member States shall ensure that national regulatory authorities are able to set minimum quality of service requirements on an undertaking or undertakings providing public communications networks.” The reach of this regulatory tool has been extensively examined by BEREC, which sees it as the third and last step of the regulatory approach to net neutrality breaches

So we do need tools for this purpose as Frode has described.

best regards,

- Klaus -
___________________________________________
KLAUS NIEMINEN
Communications Network Specialist
Finnish Communications Regulatory Authority
P.O. Box 313
FIN-00181 HELSINKI, FINLAND
tel.: +358 29 5390 528
fax: +358 9 6966 873
e-mail: [email-masked] 
www: http://www.ficora.fi  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: 6. marraskuuta 2013 4:42
To: [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP regulator use case

Frode

2.2
I think i'm basically OK with this replacing the current S2.2 (including the subsections) I suggest that it should mention comparability in the final para (as well as in the penultimate one) - personally I think this is the most important characteristic from a regulator's point of view, so that it's fair to compare measurements they make across the different ISPs.
the one thing in the current S2.2 (actually in S2.2.3) that I think is worth keeping is a mention of fixed and mobile. I think this could be handled by adding to the Intro (S1) something like: "Measurements will take place on both fixed and mobile access"

4.3 Monitoring overall market development good for me

best wishes
phil

Marc, Phil, Trevor, all,

Thanks,
Frode

Proposed update:

2.2 Regulators
-------------------

---

From: James Miller <[email-masked]>
Date: Wed, 6 Nov 2013 01:47:26 -0500

I would propose adding in section 2 or 4 some text describing a regulators
need to ensure that policies intended to protect the privacy interests of a
potential tester can be enforced using features available in the system.
 Recognizing that
http://www.ietf.org/id/draft-folks-lmap-framework-00.txtand in the
lists there's a view that a single entity will be overseeing a
large scale measurement, there is a clear need for a test infrastructure to
include tools to enforce the privacy policies of the managing entity and
coordinate collaborative activity.  A desire to protect the privacy of
testers contributing to our mobile measurement effort lead our work on the
project, and our decision to adopt changes to our fixed technical
infrastructure and data model to support a totally anonymous collection at
the MA and communications between controllers, and through the collectors
and the test results collected.  I understand there has been discussion on
this point in the past and we had included this in the Section 7 Security
Considerations of our original I-D.  In any event I think the regulator use
case benefits from calling the point out specifically.
http://tools.ietf.org/id/draft-schulzrinne-lmap-requirements-00.txt

Proposed addition in Section 2.2:
:: Regulators role in the development and enforcement of broadband Internet
access service policies also require that the measurement approaches meet a
high level of verifiability, accuracy and provider-independence to support
valid and meaningful comparisons of Internet access service
performance.   Regulators
also require that policies intended to protect the privacy interests of
potential testers can be enforced using features available in the system.

Naturally any use case is going to have some variation among the specific
practices of the parties.  On Klaus and Frode's comments (and excellent
draft contributions thanks!) I would chime in to add that "Transparency" is
a fundamental feature of the rules intended to protect the Open Internet
(or network neutrality in other parlance).  Our rule is intended to enable
"consumers to make informed choices regarding use of such services" but
also "for content, application, service, and device providers to develop,
market, and maintain Internet offerings." 47 CFR 8.3 - Transparency.

Whether protecting consumers is moved I think you find the thought and
regulatory use case solidly grounded in the US approach as well.  I would
also suggest adding some text to emphasize that Transparency is also
important to enabling development in the ecosystem.

Proposed addition in Section 4.1:
:: End users need effective transparency to be able to make informed
choices throughout the different stages of their relationship with ISPs,
when selecting Internet access service offers, and when considering
switching service offer within an ISP or to an alternative ISP.
 Transparency in Internet performance information is also important to
content, application, service, and device providers' to ensure they are
able to develop, market, and maintain Internet offerings.  Quality
information about service offering can be provided based on performance
measurements of the services provided in the market.

On Klaus' comments on what in US jurisdictions might be understand as
"blocking" or "discriminatory" conduct, I'd add that US law includes
similar provisions. 47 C.F.R.§§ 8.5, 8.7.  Recognizing that these topics
including "Quality of Service" are evolving and differ in many respects
across jurisdictions, a high level description focused on the value of
performance information to regulators in this regard may be helpful in the
section.

I would also breach another topic for future discussion on assessing
quality on the test network itself.  To maintain comparability and quality
of the collected data, the architecture itself must be monitored with
auditing of whether the infrastructure is functioning correctly, e.g.
server health and connectivity. MA side is much easier to keep a handle on
with special purpose probes (such as with our fixed "whitebox" approach)
but server side checks on and between controllers is also a very important
issue to keep an eye on.  I don't have a specific proposal how to address
this in an I-D right now but I'd note it as something any large scale
measurement manager will need to address.
--
[email-masked]

"Wovon man nicht sprechen kann, darüber muß man schweigen."
(What we cannot speak about we must pass over in silence.  )
Ludwig Josef Johann Wittgenstein

On Wed, Nov 6, 2013 at 12:19 AM, Nieminen Klaus <[email-masked]>wrote:

> As one of the  regulator friends that Frode mentioned, I would like to
> tell how we see these issues in Finland as I'm working for the Finnish
> telco regulator.
>
> I would like to see this chapter to be included. As said publishing
> transparent and comparable information is important but so it's also end
> user's ability to verify the performance of the Internet access service
> provided, comparing the measurement results achieved with the information
> given in the contract. If the contracted service quality level is not met,
> end users can use these results when complaining to the ISP to achieve
> improvement and compensation.
>
> If you want to include it to end user requirements it is ok for me to
> reference it from the regulator part. However I do not want it to be
> totally deleted as it is also our requirement not only end user one.
>
> Yes I know that this is a hard problem how to measure NN. However it is a
> clear regulatory requirement at least in the EU. See Directive 2009/136/EC
> http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2009:337:0011:01:EN:HTML
>
> According to Art 22(3), “in order to prevent the degradation of service
> and the hindering or slowing down of traffic over networks, Member States
> shall ensure that national regulatory authorities are able to set minimum
> quality of service requirements on an undertaking or undertakings providing
> public communications networks.” The reach of this regulatory tool has been
> extensively examined by BEREC, which sees it as the third and last step of
> the regulatory approach to net neutrality breaches
>
> I think i'm basically OK with this replacing the current S2.2 (including
> the subsections) I suggest that it should mention comparability in the
> final para (as well as in the penultimate one) - personally I think this is
> the most important characteristic from a regulator's point of view, so that
> it's fair to compare measurements they make across the different ISPs.
> the one thing in the current S2.2 (actually in S2.2.3) that I think is
> worth keeping is a mention of fixed and mobile. I think this could be
> handled by adding to the Intro (S1) something like: "Measurements will take
> place on both fixed and mobile access"
>
> >>For competition to successfully discipline operators' behaviour in the
> interests of their customers, end users must be fully aware of the
> characteristics of the ISPs' access offers.
> >>  However, the quality of an Internet access service is also determined
> by the connectivity to the rest of the Internet. Therefore test
> configurations which measure beyond the ISP's own network may also be
> relevant in some cases.
> true. perhaps worth mentioning that the ISPs can influence this to some
> extent - how well they're connected to the Internet, how they cache popular
> content etc.
>
> i tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market
> development issue (the internet has got slower, and doesnt meet the policy
> objective of Superfast broadband by Year 2015 - or whatever)-
>
> best wishes
> phil
>
> I have prepared a proposed description of a regulator use case for the
> LMAP use case ID. I noticed the "advertisement" for further input from
> regulators for the use case a couple of weeks ago. Finally I have managed
> to find sufficient time to contribute to the ID, supported by a couple of
> "regulator friends" from the Net Neutrality Working Group of the BEREC
> organization, representing European national regulators. The proposal
> consists of two parts; a modification of section 2.2 "Regulators", plus a
> new chapter after chapter 3 "Details of ISP use case", titled "Details of
> regulator use case".
>
> Thanks,
> Frode
>
>
> Proposed update:
>
> 2.2 Regulators
> -------------------
> Regulators in jurisdictions around the world are responding to consumers'
> adoption of Internet access services for traditional telecommunications and
> media services by promoting competition among providers of electronic
> communications, to ensure that users derive maximum benefit in terms of
> choice, price, and quality.
>
> Some jurisdictions have responded to a need for greater information about
> Internet access service performance in the development of regulatory
> policies and approaches for broadband technologies by developing
> large-scale measurement programs. Programs such as the U.S. Federal
> Communications Commission's Measuring Broadband America, European
> Commission's Quality of Broadband Services in the EU reports and a growing
> list of other programs employ a diverse set of operational and technical
> approaches to gathering data to perform analysis and reporting on diverse
> aspects of broadband performance.
>
> While each jurisdiction responds to distinct consumer, industry, and
> regulatory concerns, much commonality exists in the need to produce
> datasets that are able to compare multiple Internet access service
> providers, diverse technical solutions, geographic and regional
> distributions, and marketed and provisioned levels and combinations of
> broadband Internet access services. In some jurisdictions, the role of
> measuring is provided by a measurement provider.
>
> Measurement providers measure network performance from users towards
> multiple content and application providers, included dedicated test
> measurement servers, to show a performance of the actual Internet access
> service provided by different ISPs. Users need to know the performance that
> are achieving from their own ISP. In addition, they need to know the
> performance of other ISPs of same location as background information for
> selecting their ISP. Measurement providers will provide measurement results
> with associated measurement methods and measurement metrics.
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 6 Nov 2013 08:35:35 -0500

Hi Frode,

I read your mail and particularly agree with Phil's comments excerpted below:

> I tihnk some of the stuff in S4.4 is really part of 4.3, as it's a market

If one form of the absence of neutrality is censorship, even
censorship is hard to define and can include performance degradation
according to Nick Feamster's presentation at the IRTF open meeting:
http://www.ietf.org/proceedings/88/slides/slides-88-irtfopen-3.pptx

I think it's fair to say this sort of detection testing is still a research topic,
Al

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Tuesday, November 05, 2013 9:42 PM
> the subsections)
> I suggest that it should mention comparability in the final para (as well
> as in the penultimate one) - personally I think this is the most important
> characteristic from a regulator's point of view, so that it's fair to
> compare measurements they make across the different ISPs.
> handled by adding to the Intro (S1) something like: "Measurements will
> take place on both fixed and mobile access"
>
>
> extent - how well they're connected to the Internet, how they cache
> popular content etc.
>
> adoption of Internet access services for traditional telecommunications
> and media services by promoting competition among providers of electronic
> policies and approaches for broadband technologies by developing large-
> scale measurement programs. Programs such as the U.S. Federal
> service provided by different ISPs. Users need to know the performance
> that are achieving from their own ISP. In addition, they need to know the
> selecting their ISP. Measurement providers will provide measurement
> results with associated measurement methods and measurement metrics.
>
> From a consumer perspective, the differentiation between fixed and mobile
> (cellular) Internet access services is blurring as the applications used
> are very similar. Hence, similar measurements will take place on both
> fixed and mobile Internet access services.
>
> Regulators role in the development and enforcement of broadband Internet
> access service policies also require that the measurement approaches meet
> a high level of verifiability, accuracy and provider-independence to
> support valid and meaningful comparisons of Internet access service
> performance
>
> LMAP standards could answer regulators shared needs by providing scalable,
> cost-effective, scientifically robust solutions to the measurement and
> collection of broadband Internet access service performance information.
>
>
> Proposed new chapter:
>
> Competition plays a vital role in regulation of the electronic
> communications markets. For competition to successfully discipline
> operators' behaviour in the interests of their customers, end users must
> be fully aware of the characteristics of the ISPs' access offers. In some
> jurisdictions regulators mandate transparent information made available
> about service offers.
>
> End users need effective transparency to be able to make informed choices
> throughout the different stages of their relationship with ISPs, when
> selecting Internet access service offers, and when considering switching
> service offer within an ISP or to an alternative ISP. Quality information
> about service offers can be provided based on performance measurements of
> the services provided in the market.
>
> A set of measurement parameters and associated measurement methods are
> used over time, e.g. speed, delay, and jitter. Then the measurement raw
> data are collected and go through statistical post-processing before the
> results can be published in an Internet access service quality index to
> facilitate end users' choice of service provider and offer.
>
> In order to make information on offerings more meaningful and comparable
> to end users, common frames of reference on Internet access service
> quality metrics are essential. An example would be that in relation to
> access speed, the actual download and upload speeds are measured and
> published, not only the maximum speed.
>
> A measurement system that monitor Internet access services and collect
> quality information can typically consist of a number of measurement
> probes and one or more test servers located at peering points. The system
> can be operated by a regulator or a measurement provider.  Number and
> distribution of probes follows specific requirements depending on the
> scope and the desired statistical reliability of the measurement campaign.
>
> In many test configurations, the measurement results will only reflect the
> performance of ISP's own network. However, the quality of an Internet
> access service is also determined by the connectivity to the rest of the
> Internet. Therefore test configurations which measure beyond the ISP's own
> network may also be relevant in some cases.
>
>
> In addition to the advantages that end users receive from the competition
> in the market, through availability of a plethora of ISPs and Internet
> access service offers, facilitated by transparent quality information
> about these offers at a general level, end users would further leverage on
> information about specific services they subscribe to, or which they are
> considering to subscribe to.
>
> Performance of Internet access service offers vary based on where the
> service is provided (geographical and topological aspects), how it is
> provided (e.g. access technology and ISP's interconnection to the
> Internet) and when it is provided (variation through the day and over the
> week). Therefore there is a need to obtain more detailed quality
> information about specific service offers.
>
> Quality measurement tools can be made available for end users to monitor
> their Internet access service, giving information about the speed and
> other quality metrics of the subscribed service. Speed meters and similar
> tools can be provided by regulators or other third parties, but also
> directly by the ISP for the benefit of the subscriber.
>
> Such measurement tools assist end users in verifying the performance of
> the Internet access service provided, comparing the measurement results
> achieved with the information given in the contract. If the contracted
> service quality level is not met, end users can use these results when
> complaining to the ISP to achieve improvement and compensation.
>
> It may also help end users to assess whether they have chosen the package
> which best suits their specific needs. With their performance results at
> hand, they can benefit from quality evaluation guides, which offer
> comparisons based on their specific usage profiles. This may enable them
> to check whether their current subscription is the most appropriate, and
> assist them to switch package or provider if it is in their best interest.
>
> When provided by regulators and other third parties, such quality
> measurement tools can also be used to compare performance between ISPs. It
> should, though, be recognised that measurement results can be affected by
> factors outside the ISP's control, related for instance with local network
> connection (e.g. Wi-Fi), equipment and software being used.
>
>
> Governments sometimes set strategic goals for high-speed Internet access
> penetration as an important component of the economic, cultural and social
> development of the society. To evaluate the effect of the stimulated
> growth over time, broadband Internet access take-up and penetration of
> high-speed access can be monitored through measurement campaigns.
>
> An example of such an initiative is the "Digital Agenda for Europe" which
> was adopted in 2010, to achieve universal broadband access. The goal is to
> achieve by 2020, access for all Europeans to Internet access speeds of 30
> Mbps or above, and 50% or more of European households subscribing to
> Internet connections above 100 Mbps.
>
> To monitor actual broadband Internet access performance in a specific
> country or a region, extensive measurement campaigns are needed. A panel
> can be built based on operators and packages in the market, spread over
> urban, suburban and rural areas. Probes can then be distributed to the
> participants of the campaign.
>
> Periodic tests running on the probes can for example measure actual speed
> at peak and off-peak hours, but also other detailed quality metrics like
> delay and jitter. Collected data goes afterwards through statistical
> analysis, deriving estimates for the whole population which can then be
> presented and published regularly.
>
> Using a harmonized or standardised measurement methodology, or even a
> common quality measurement platform, measurement results could also be
> used for benchmarking of providers and/or countries.
>
>
> Regulation related to net neutrality and the open Internet has been
> introduced in some jurisdictions' Internet policy, and monitoring methods
> for detection of potential net neutrality breaches is needed to follow up
> such regulations with concrete actions.
>
>
> Net neutrality is typically related to equal treatment of traffic
> transmitted over the Internet access service, while other traffic
> transmitted in parallel over the end user's broadband connection e.g. to
> provide IPTV in IP networks separated from the Internet (also referred to
> as "closed IP networks") can be exempted from net neutrality regulation.
> Services provided in parallel to the Internet access services are referred
> to as specialised services (ref. BEREC and FCC).
>
> In order for such regulatory service architecture to protect net
> neutrality and the open Internet, it is essential that the specialised
> services are not provided at the expense of the Internet access service.
> Therefore regulators may need to measure performance of the Internet
> access service over time, and thereby detect whether this service is
> becoming degraded.
>
> When regulators monitor this development to ensure net neutrality and the
> open Internet, they can check e.g. whether Internet access service
> performance and level of quality reflect advances in technology and that
> this is not impaired by specialised services. Comparison between ISPs or
> between different countries may also be relevant for this kind of
> evaluation.
>
> Based on such Internet access service quality monitoring, regulators will
> typically report results and findings regularly, e.g. on an annual basis.
> In some jurisdictions regulators may also eventually impose minimum
> quality of service requirements on ISPs in order to prevent degradation of
> Internet access services.
>
>
> In the regulatory context related to net neutrality and the open Internet,
> in addition to the monitoring to detect potential degradation of the
> access service as a whole described above, it is essential to perform
> monitoring to detect potential degradation of individual applications
> using the Internet access.
>
> Since net neutrality relates to equal treatment of traffic transmitted
> over the Internet access service, the treatment of different applications'
> traffic is in the core of net neutrality. When traffic originating from
> different applications are treated independent of which application each
> IP packet belongs to, this is referred to as application agnosticism.
>
> Examples of departure from application agnosticism are blocking or
> throttling of traffic from specific applications, but also preferential
> treatment of specific applications. If traffic is forwarded at different
> priority levels, traffic having higher priority level implies that other
> traffic has lower priority. Detection of such traffic management practices
> does not necessarily imply breaches to net neutrality, but these
> observations can be used as an input to the regulatory evaluation of the
> practice.
>
> When regulators monitor traffic management practices to ensure net
> neutrality and the open Internet, they can check compliance with criteria
> defining what is considered reasonable traffic management. Measurement
> systems designed to monitor this kind of effects need to send application-
> specific traffic and then measure in detail the behaviour of the different
> packets receive when transferred over the Internet access service.
>
> Today there exist measurement tools that can detect differentiated
> treatment of individual applications. Another method that can be used is
> to measure performance at application layer, e.g. assisted by content and
> application providers, but also passive measurements are foreseen as a
> promising approach for monitoring of application-specific treatment.
>

---

From: James Miller <[email-masked]>
Date: Wed, 6 Nov 2013 09:42:34 -0500

I agree with Frode's comments on the 2.2 points.  I would add the quality
of the comparability of data is crucial for a regulators use and extends to
technical neutrality across the access link technologies as well, eg
ability to compare docsis, DSL, fiber, fixed wireless or other solutions.
Most studies I am familiar with include present and post processing
techniques to support comparisons across major broadband technologies that
include out of scope topics.

With 4.4 or 4.5 I agree there are a mix of views but would suggest there
are production quality tools as well as more research level quality
topics.  I believe many are likely to extend to discussion of ippm
metrics.  In addition to the tools mentioned, nick weaver's netalizer suite
is an example that tests for easy things (like port blocking) and harder
more subtle effects (like throttling, caching and proxy effects), and would
also mention work by bitag and other groups trying to reach technical
consensus on description of many traffic management practices.  It is an
evolving space but something relevant for regulators use case description.
 On Nov 6, 2013 8:36 AM, "MORTON, ALFRED C (AL)" <[email-masked]> wrote:

> > I don't like these sections about testing for whether the ISP is being
> net
> > neutral.
> > In part this is because net neutrality is more of a political concept
> than
> > technical one (in my view)
> >
> > I don't like these sections about testing for whether the ISP is being
> net
> > neutral.
> > In part this is because net neutrality is more of a political concept
> than
> > technical one (in my view)
> >
> > From: [email-masked] [[email-masked]] On Behalf Of
> Sørensen,
> > Frode [[email-masked]]

---

From: Nieminen Klaus <[email-masked]>
Date: Wed, 6 Nov 2013 15:30:13 +0000

Dear all,

I would also like to add that as far as I have understood, LMAP WG don't standardize performance metrics. Therefore I think that we don't have to worry that much about how hard it is to measure degradation of the Internet access or specific applications. In addition as James and Frode mentioned there are already some tools available for this purpose. 

I fully support James' last remark that "it is an evolving space but something relevant for regulators use case description."

- Klaus

-----Original Message-----
From: James Miller [mailto:[email-masked]] 
Sent: 6. marraskuuta 2013 16:43
To: MORTON JR. ALFRED, (AL)
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP regulator use case

I agree with Frode's comments on the 2.2 points.  I would add the quality of the comparability of data is crucial for a regulators use and extends to technical neutrality across the access link technologies as well, eg ability to compare docsis, DSL, fiber, fixed wireless or other solutions.  Most studies I am familiar with include present and post processing techniques to support comparisons across major broadband technologies that include out of scope topics.

With 4.4 or 4.5 I agree there are a mix of views but would suggest there are production quality tools as well as more research level quality topics.  I believe many are likely to extend to discussion of ippm metrics.  In addition to the tools mentioned, nick weaver's netalizer suite is an example that tests for easy things (like port blocking) and harder more subtle effects (like throttling, caching and proxy effects), and would also mention work by bitag and other groups trying to reach technical consensus on description of many traffic management practices.  It is an evolving space but something relevant for regulators use case description.

	Hi Frode,

---

From: Sørensen, Frode <[email-masked]>
Date: Thu, 7 Nov 2013 07:16:09 +0000

Thanks for your comments,

Yes, I agree that these use cases are complex and challenging. Although a complete surveillance of such matters is a research topic, sample tests are performed today based on exiting measurement tools. As far as I understood, FCC investigated manipulation of BitTorrent traffic a while ago, I am not sure to what extent they measured in that case, but this shows anyway the relevance of the use case. I believe regulators are in a good position to explain what the regulator use case should look like, but of course interest and comments from other stakeholders are appreciated and welcome.

Best,
Frode

-----Opprinnelig melding-----
Fra: Nieminen Klaus [mailto:[email-masked]] 
Sendt: 6. november 2013 16:30
Til: James Miller; MORTON JR. ALFRED, (AL)
Kopi: [email-masked]; [email-masked]; Sørensen, Frode
Emne: RE: [lmap] LMAP regulator use case

Dear all,

- Klaus

-----Original Message-----
From: James Miller [mailto:[email-masked]] 
Sent: 6. marraskuuta 2013 16:43
To: MORTON JR. ALFRED, (AL)
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [lmap] LMAP regulator use case

	Hi Frode,

---

From: James Miller <[email-masked]>
Date: Thu, 7 Nov 2013 10:01:36 -0500

I am very much overloaded right now with.. some news to follow but would be
happy to do a webex or email chain discussing the "Special Studies" we have
been contemplating on this point.

On Thu, Nov 7, 2013 at 2:16 AM, Sørensen, Frode <[email-masked]>wrote:

> Yes, I agree that these use cases are complex and challenging. Although a
> complete surveillance of such matters is a research topic, sample tests are
> performed today based on exiting measurement tools. As far as I understood,
> FCC investigated manipulation of BitTorrent traffic a while ago, I am not
> sure to what extent they measured in that case, but this shows anyway the
> relevance of the use case. I believe regulators are in a good position to
> explain what the regulator use case should look like, but of course
> interest and comments from other stakeholders are appreciated and welcome.
>
> Best,
> Frode
>
>
> I would also like to add that as far as I have understood, LMAP WG don't
> standardize performance metrics. Therefore I think that we don't have to
> worry that much about how hard it is to measure degradation of the Internet
> access or specific applications. In addition as James and Frode mentioned
> there are already some tools available for this purpose.
>
> I fully support James' last remark that "it is an evolving space but
> something relevant for regulators use case description."
>
> - Klaus
>
> are production quality tools as well as more research level quality topics.
>  I believe many are likely to extend to discussion of ippm metrics.  In
> addition to the tools mentioned, nick weaver's netalizer suite is an
> example that tests for easy things (like port blocking) and harder more
> subtle effects (like throttling, caching and proxy effects), and would also
> mention work by bitag and other groups trying to reach technical consensus
> on description of many traffic management practices.  It is an evolving
> space but something relevant for regulators use case description.
>
>
>         > I don't like these sections about testing for whether the ISP is
> being net
>         > neutral.
>         > In part this is because net neutrality is more of a political
> concept than
>         > thanks very much for the comments - and it is excellent to have
> detailed
>         > text suggestions.
>         >
>         > 2.2
>         > Happy to add a new chapter 4, but comments about some of your
> sub use
>         > cases
>         >
>         > i think this can be deleted. this is the end user use case, not
> the
>         > regulator use case.
>         >
>         > I don't like these sections about testing for whether the ISP is
> being net
>         > neutral.
>         > In part this is because net neutrality is more of a political
> concept than
>         > From: [email-masked] [[email-masked]] On Behalf
> Of Sørensen,
> large-

---

From: <[email-masked]>
Date: Tue, 26 Nov 2013 09:52:42 +0000

James,

I agree that privacy (of the end user who has the MA) is important - in the ISP use case as well as the regulator one

http://tools.ietf.org/html/draft-ietf-lmap-framework-01#section-8 discusses privacy (data protection). It would be great to get your feedback on this section. One idea that's been raised is a Group-ID (section 8.7) - would be very interested to hear your opinion and also other ideas for improving privacy (personally am not convinced by the Group-ID, as seems a direct trade-off with security)

thanks
phil

From: James Miller [mailto:[email-masked]]
Sent: 06 November 2013 06:47
To: Nieminen Klaus
Cc: Eardley,PL,Philip,TUB8 R; [email-masked]; [email-masked]; Marc Linsner; Henning Schulzrinne
Subject: Re: [lmap] LMAP regulator use case

I would propose adding in section 2 or 4 some text describing a regulators need to ensure that policies intended to protect the privacy interests of a potential tester can be enforced using features available in the system.  Recognizing that http://www.ietf.org/id/draft-folks-lmap-framework-00.txt and in the lists there's a view that a single entity will be overseeing a large scale measurement, there is a clear need for a test infrastructure to include tools to enforce the privacy policies of the managing entity and coordinate collaborative activity.  A desire to protect the privacy of testers contributing to our mobile measurement effort lead our work on the project, and our decision to adopt changes to our fixed technical infrastructure and data model to support a totally anonymous collection at the MA and communications between controllers, and through the collectors and the test results collected.  I understand there has been discussion on this point in the past and we had included this in the Section 7 Security Considerations of our original I-D.  In any event I think the regulator use case benefits from calling the point out specifically.
http://tools.ietf.org/id/draft-schulzrinne-lmap-requirements-00.txt

Proposed addition in Section 2.2:
:: Regulators role in the development and enforcement of broadband Internet access service policies also require that the measurement approaches meet a high level of verifiability, accuracy and provider-independence to support valid and meaningful comparisons of Internet access service performance.   Regulators also require that policies intended to protect the privacy interests of potential testers can be enforced using features available in the system.

...