Subject: [lmap] AD review of draft-ietf-lmap-use-cases-03

From: Benoit Claise <[email-masked]>
Date: Thu, 24 Jul 2014 10:30:11 -0400

Dear all,

Here is my AD review

- OLD

Abstract

    Measuring broadband performance on a large scale is important for
    network diagnostics by providers and users, as well as for public
    policy.  Understanding the various scenarios and users of measuring
    broadband performance is essential to development of the framework,
    information model and protocol. This document details two use cases
    that can assist to developing that framework.  The details of the
    measurement metrics themselves are beyond the scope of this document.

NEW:

Abstract

    Measuring broadband performance on a large scale is important for
    network diagnostics by providers and users, as well as for public
    policy.  Understanding the various scenarios and users of measuring
    broadband performance is essential to development of the_Large-scale Measurement
    of Broadband Performance (LMAP)_  framework,
    information model and protocol. This document details two use cases
    that can assist to developing that framework.  The details of the
    measurement metrics themselves are beyond the scope of this document.

-  Section 2.1

    An ISP, or indeed another network operator, needs to understand the
    performance of their networks, the performance of the suppliers
    (downstream and upstream networks), the performance of services, and
    the impact that such performance has on the experience of their
    customers.

"indeed _another _network operator"?

- Section 2.1

       Identifying, isolating and fixing problems in the network,
       services or with CPE and end user equipment.

I can't parse the sentence

-  Section 2.1

This sentence speaks about "services"

       o Identifying, isolating and fixing problems in the network,
       services or with CPE and end user equipment.

Section 3.2 speaks about new services.
So, this paragraph also includes "services":

       o Understanding the impact and operation of new devices and
       technology. As a new product is deployed, or a new technology
       introduced into the network, it is essential that its operation
       and impact on other services is measured. This also helps to
       quantify the advantage that the new technology is bringing and
       support the business case for larger roll-out.

NEW:

       o Understanding the impact and operation of new devices,
       technology,_or services._  As a new product is deployed, or a new technology
       introduced into the network, it is essential that its operation
       and impact on other services is measured. This also helps to
       quantify the advantage that the new technology is bringing and
       support the business case for larger roll-out.

- I'm confused by:

  2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2>   Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
      2.1  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.1>  Internet Service Provider (ISP) Use Case . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
      2.2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.2>  Regulators . . . . . . . . . . . . . . . . . . . . . . . . .4  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-4>
      2.3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.3>  Implementation options . . . . . . . . . . . . . . . . . . .5  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-5>

What is the link between the "Implementation options" subsection and use 
cases?
Should "implementation options" have its own section?

-

    Another approach involves implementing the measurement capability as
    a webpage or an "app" that end users are encouraged to download onto
    their mobile phone or computing device. Measurements are triggered by
    the end user, for example the user interface may have a button to
    "test my broadband now". Compared with the previous approach, the
    system is much more loosely controlled, as the panel of end users and
    the schedule of tests are determined by the end users themselves
    rather than the measurement system. It would be easier to get large-
    scale, however it is harder to get comparable benchmarks as the
    measurements are affected by the home network and also the population
    is self-selecting and so potentially biased towards those who think
    they have a problem. This could be alleviated by stimulating
    widespread downloading of the app and careful post-processing of the
    results to reduce biases.

This end-user triggered measurement provides a huge advantage: it 
reflects the end-user QoE, as opposed to the broadband device QoE (*).  
In combination with LMAP measurements in the broadband device, the 
end-user triggered measurement might prove that the home network is the 
source of performance degradations.
You might want to add those points either in section "Implementation 
options" or around the section 3.1  3rd paragraph.
This point is partly covered in last paragraph of section 3.5

(*) except if the end-user triggered measurement is executed directly on 
the broadband device ... For example, downloading a page, or initiating 
a test from the web server on the broadband device, as you mentioned:

    There are several other possibilities. For example, as a variant on
    the first approach, the measurement capability could be implemented
    as software embedded in the home gateway, which would make it more
    viable to have the capability on every user line. As a variant on the
    second approach, the end user could initiate measurements in response
    to a request from the measurement system.

- In connection with the previous point,  I was confused while reading 
the document.
I don't see"end-user triggered measurement" in section 2.X next to the 
ISP and operators use case. fine, but it's still discussed throughout 
the draft. For example

    Measurements are triggered by
    the end user, for example the user interface may have a button to
    "test my broadband now".

    OR

    As a variant on the
    second approach, the end user could initiate measurements in response
    to a request from the measurement system.

    OR

    The operator really wants to understand the end-to-end service
    experience. However, the home network (Ethernet, WiFi, powerline) is
    highly variable and outside its control. To date, operators (and
    regulators) have instead measured performance from the home gateway.
    However, mobile operators clearly must include the wireless link in
    the measurement.

    OR

    The operator would like the
    end-to-end view of the service, rather than (say) just the access
    portion

So, I've been wondering whether this end-user triggered measurement was 
in scope or not.
I only found my answer in the conclusion section

    There are
    other use cases that are not the focus of the initial LMAP charter
    (although it is expected that the mechanisms developed would be
    readily applied), for example end users would like to use
    measurements to help identify problems in their home network and to
    monitor the performance of their broadband provider.

You should make it very clear, upfront, that the end-user triggered 
measurement is not in scope.
Something like section 5.6.1 End-user-controlled measurement systemin 
the framework document

- Section 3.1

    Operators want to understand the quality of experience (QoE) of their
    broadband customers.

There is a QoE reference in RFC 5481

- Section 3.5

    One example was described in [IETF85-Plenary]. The operator was
    running a measurement panel for reasons discussed in sub use case #1.

Should we all review [IETF85-Plenary] to understand what sub use case # 
1 is? ;-)

- Section 4.1

    The published information needs to be:

       o  Accurate - the measurement results must be correct and not
       influenced by errors or side effects. The results should be
       reproducible and consistent over time.

       o  Comparable - common metrics should be used across different
       ISPs and service offerings so that measurement results can be
       compared.

       o  Meaningful - the metrics used for measurements need to reflect
       what end users value about their broadband Internet access service

       o  Reliable - the number and distribution of measurement agents,
       and the statistical processing of the raw measurement raw data,
       needs to be appropriate

Not sure if this was discussed, but one goal behind the regulator use 
case is to strongly suggest the ISPs to use the same metrics in their 
SLC (Service Level Contracts).
If you ever tried to compare SLC between ISPs, well, go luck.
This would be a nice additional 4.x section

- Pervasive Monitoring, RFC 7528. This RFC should at least be 
referenced, for example around this sentence:

        The other approach is passive measurements
        on the customer's ordinary traffic; the advantage is that it measures
        what the customer actually does, but it creates extra variability
        (different traffic mixes give different results) and especially it
        raises privacy concerns.

Regards, Benoit