Subject: Re: [ippm] [lmap] Focus on Tests, Not Architectures...

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 30 Nov 2012 23:53:08 +0000

One thing that I have learned through the MBA experience that it is hard to find a single metric that addresses all use cases. For example, consumers (and national bodies charged with consumer protection) might want to know whether advertised rates agree with actual rates. Unless ISPs are advertising the model-based metric, it may not be as useful. For other applications, such as VoIP, latency and packet loss for specific traffic types matter. Having a metric that is RTT-independent and uses a different traffic pattern is not helpful in that case, as much as it might be otherwise.

On a more general note, from a consumer perspective, there are probably three kinds of metrics that are useful:

(a) Comparison between advertised and real-life metric, to allow fair comparison of different providers.

(b) Relative metrics: I don't much care what the number is (it could be expressed in some number between 0 and 100, rather than Mb/s or ms), but it should preserve relative ordering. If provider A is significantly better according to the metric X, I should expect my application (related to that metric) to perform significantly better, even if I don't get the exact same performance. A rough example for that is cellular throughput: It is very unlikely that an individual consumer will get exactly the average throughput measured by whatever entity, given geographic and device variations, but they should still be able to pick a better-performing carrier based on that metric. That's similar to Al's gas mileage example, "your mileage may vary", but any Prius is still likely to get better gas mileage than a Lincoln Navigator.

(c) Threshold metrics: The metric should tell me whether a particular service is likely to be suitable for my need. Latency is the classical example: If my access link has a round-trip latency of 400 ms, I may die prematurely in a first-person-shooter game. Absolute accuracy isn't that important (say, 400 vs. 420 ms), but consistent definition of the path component and other attributes is.

Some metrics might serve all three roles, but the emphasis differs.

I would be interested in comments on the metrics that the FCC Measurement Broadband America project uses.

(a) Which ones need standardization first? (Some are already based on IPPM RFCs, but most aren't.)

(b) Which ones might be less valuable and/or should be substantially modified?

See http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf (pg. 22) for the current list.

Henning

________________________________
From: [email-masked] [[email-masked]] on behalf of Al Morton [[email-masked]]
Sent: Monday, November 26, 2012 12:20 AM
To: Matt Mathis
Cc: [email-masked]; Nicholas Weaver; [email-masked]; Brian Trammell
Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures...

Matt,

I think we agree that better metrics are needed for LMAP,
and that ideally a metric can be applied at any point or pair of points.

RFC 2330 and most IPPM metrics have a decidedly Source-to-Destination
host construction. Frankly, we did a better job with arbitrary
measurement points in Rec Y.1540. It's worth a look:
http://www.itu.int/rec/T-REC-Y.1540/en

There are many goals the new metrics could serve:
 - externally observable
 - measurable by users
 - easily related to user application performance
 - service verification
 - support consumer choice of ???
 - etc.
We need to agree on the set that are needed.

While there may be some information in a measurement that covers
a combination of networks, measuring more than one network
is only useful when the combined network path delivers
the target performance - the sunny day outcome. Otherwise,
they are inconclusive, as you said in your memo. Some target
performance levels only apply in the scope where they are offered.

We don't really have the milk fat scenario here, unless the producer,
the truck driver, the convenience store, and your home refrigerator
can all add some unknown amount of fat to the milk. ;-)

When considering this problem, I've been thinking of the fuel
efficiency ratings provided for new cars in the US and Europe (at least).
The specs illustrate very well that the quantity measured is variable
depending on the conditions of the measurement (in the US, the miles per
gallon for city and highway driving), and they imply the
range of values that normal users can experience (a fairly wide range).
These are some aspects that would be useful to reproduce in our
metrics, I think.

regards, and YMMV,
Al

---

From: Al Morton <[email-masked]>
Date: Sun, 02 Dec 2012 22:02:31 -0500

Regarding "advertised rates":

- I think there's variation among the underlying units of measure for
   the numbers advertised today - this is the point I tried to make
   during Henning's Transport Area presentation. It would certainly help
   with service verification if the parameters describing service were
   clearly defined for those performing the verification - I appreciate that
   most consumers won't care much about the units of measure, but verification
   should be scientific (e.g., conducted at the same layer as the spec).

- The model-based metrics have appeal to me, in that they are derived
   from fundamental metrics that we have a reasonable hope of specifying
   sufficiently such that independent implementations and different
   hardware/software platforms will be able to make equivalent measurements.
   It may even turn-out that similar methods can provide a basis for
   model-based estimates of multiple (application?) metrics (Matt's
   memo focuses on TCP, which is certainly enough for now).

Changing topics slightly, in the list:
http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf 
(pg. 22)

I'd prefer to pare this down to start. Even where we have RFCs to refer to,
the details of the test stream (for active measurement without user traffic,
or active/hybrid measurement during user traffic) need to be agreed,
and even the question of whether tests will be conducted during user
activity or not needs discussion, maybe this is the first question to
address.

Al

At 06:53 PM 11/30/2012, Henning Schulzrinne wrote:
>One thing that I have learned through the MBA experience that it is 
>hard to find a single metric that addresses all use cases. For 
>example, consumers (and national bodies charged with consumer 
>protection) might want to know whether advertised rates agree with 
>actual rates. Unless ISPs are advertising the model-based metric, it 
>may not be as useful. For other applications, such as VoIP, latency 
>and packet loss for specific traffic types matter. Having a metric 
>that is RTT-independent and uses a different traffic pattern is not 
>helpful in that case, as much as it might be otherwise.
>
>On a more general note, from a consumer perspective, there are 
>probably three kinds of metrics that are useful:
>
>(b) Relative metrics: I don't much care what the number is (it could 
>be expressed in some number between 0 and 100, rather than Mb/s or 
>ms), but it should preserve relative ordering. If provider A is 
>significantly better according to the metric X, I should expect my 
>application (related to that metric) to perform significantly 
>better, even if I don't get the exact same performance. A rough 
>example for that is cellular throughput: It is very unlikely that an 
>individual consumer will get exactly the average throughput measured 
>by whatever entity, given geographic and device variations, but they 
>should still be able to pick a better-performing carrier based on 
>that metric. That's similar to Al's gas mileage example, "your 
>mileage may vary", but any Prius is still likely to get better gas 
>mileage than a Lincoln Navigator.
>
>(c) Threshold metrics: The metric should tell me whether a 
>particular service is likely to be suitable for my need. Latency is 
>the classical example: If my access link has a round-trip latency of 
>400 ms, I may die prematurely in a first-person-shooter game. 
>Absolute accuracy isn't that important (say, 400 vs. 420 ms), but 
>consistent definition of the path component and other attributes is.
>
>I would be interested in comments on the metrics that the FCC 
>Measurement Broadband America project uses.
>
>See 
>http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf 
>(pg. 22) for the current list.
>
>Henning
>