Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures...

From: Al Morton <[email-masked]>
Date: Sun, 25 Nov 2012 09:12:36 -0500

LMAP and IPPM,

After catching-up on many messages, I would like to add a key

consideration and area for discussion and agreement for

development of standardized tests:

- For each test we eventually specify, it's necessary to specify

the equivalent measurement points along the user's packet
transfer

path/architecture of each different access technology.

This effort certainly needs folks with standards experience in all of
the

access architectures to complete the task, because working out

measurement point details will be a critical aspect of conducting
comparable

measurements. So, for anyone who read Nick's and Brian's messages

below (which I mostly agree with) and was about to un-subscribe or add
"lmap"

to your spam filter, don't.  There's plenty to do here that's

architecture-related. The various measurement points needed depend

on the list of tests of course, and that list needs discussion,
too.

As an example needing some discussion:

All packet-transfer services have a demarcation point where the
customer

connects their own equipment to the network. It is valuable from both
the

diagnostic and service verification points of view to have a
measurement

point very close to the demarcation point. But direct access to this

point is often impossible for a measurement agent - where can we agree

are the functionally equivalent alternatives in each access
architecture?

(don't forget wireless access)

regards,

Al

At 11:59 AM 11/20/2012, Brian Trammell wrote:

Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion
will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:

> This focus on architecture is deeply misguided.  As someone
who's built one such architecture (Netalyzr), working on two others
(Fathom, a browser extension, and techniques using GuruPlugs/Raspberry
Pis), the architecture is the easy part.

> In all this work, having some "standard" way to
communicate between my measurement clients and servers is useless.
I just use HTTP, HTTPS, or SSH depending on whether quick & easy or
server side or mutual authentication is required, with whatever
load-balancing is needed handled by random assignment of remote test
node.

> The internal data store is whatever is appropriate for the project
(eg. for Netalyzr its python pickled objects for storage on our web
server, data in a Postgress database for analysis).

> Updates are handled appropriately (Netalyzr is updateless, its an
applet so its always up-to-date.  Our android version will just use
the app store update mechanism.  Fathom we use Firefox's update
mechanism.  The plugs are a hack using SSH, but its a fair amount of
custom goop, and they update nightly).

> And all three end up having rather different architectural
requirements.

Architectures, I would submit (and as I've said here before), are useful
to hang terminology off of: you need boxes and lines so everyone
understands (roughly) the same thing when you say "client" and
"server" and "controller" with reference to a set of
measurements you define.

But yes, specifying a way to specify a measurement is largely
arbitrary. I'd even advocate separating the representation from the
transport, especially as the general problem of measurement
interoperability has data sources operating and wildly different scales,
but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both
in the abstract and within the API constraints of the target
platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I
can run Java, so therefore I can just "Run Netalyzr" for all
Netalyzr tests.  But the same isn't the case for the Ripe Atlas or
Bismark platform.

> And Marc's list is actually just a start: you also have generic
proxy presence & location detection, application specific proxy
detection and analysis, packet injection detection, differential
application behavior detection, application agnostic traffic shaping
detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for
all of the previous, and a ton of others.  Netalyzr is currently
composed of >100 distinct tests.  And this amount just continues
to grow.

> If the IETF actually wants to have an impact in this area, don't
bother focusing on protocols for communication, focus on tests with given
attributes.  Because people like me who actually build these things
are going to ignore an architectural focus because that has no
value.  But having ACCEPTED STANDARD tests would be very
valuable.

+ 1e<large number>. Stated another way, the protocol gives you the
grammar. What you need to make this at all interoperable is the
vocabulary.

IPPM has defined a several of these at a low level, with a focus on
active measurement methods that produce comparable results. There appears
to be interest in IPPM both in adding passive methods for the existing
metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions
of more complex metrics based in part on the simple ones (see
draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in
Flash (so TCP/UDP connections only to cooperating servers) would be
great.  Our current one in Netalyzr is only OK: we believe that one
could do a lot better.

> If you could then figure out WHERE the bottleneck is (using a server
that can also do simultaneous traceroutes), this would be even more
valuable.

(This seems to make the assumption that traceroute-measurable latency is
strongly correlated with application latency at a given hop, but I get
your point...)

The efforts of IPPM to date have been focused on simply describable,
easily repeatable metrics... there's not a lot of specification of
arbitrarily complex or unknown preconditions (as would be useful in the
latency-under-load example you give), because these tend to be counter to
the goals of maximizing repeatability and minimizing uncertainty. There's
also a focus on active measurement for the same reason. On the other
hand, a lot of work in distributed measurement systems is focused on
measuring the network as it is, with implicit or explicit passive
components (again, returning to "latency under load", the load
traffic will be made up of generated as well as existing background
load).

So, what might be useful would be to open another line of work within
IPPM on interoperable descriptions of some of these existing tests as
empirically specified metrics, perhaps with relaxed compliance with the
IPPM framework as appropriate. This could be used to provide a more
complex vocabulary to LMAP, but could also be useful independently --
if LMAP is eventually defined to apply only to a restricted set of use
cases (i.e. specifically mandated tests for regulatory compliance, but
not research or troubleshooting), comparable metrics with incompatible
architecture and protocols could still lead to wider interoperability in
the measurement space, the need for protocol shims (which are after
all easy to write if the number inside has the same meaning)
aside.

Best regards,

Brian

_______________________________________________

ippm mailing list

[email-masked]

https://www.ietf.org/mailman/listinfo/ippm

---

From: Matt Mathis <[email-masked]>
Date: Sun, 25 Nov 2012 13:20:50 -0800

I would like to disagree here.   Another missing requirement from 2330 is
that measurement vantage point should not matter.   Consider the standards
for milk: milk fat is a parameter that anybody can measure (producer,
consumer, independent lab, school classroom, etc).  It can be process
controlled by the manufacturer (tweaking centrifuge settings), and is
useful to the consumer.  As an extremely robust metric it stabilizes the
industry and brings sanity to the marketplace.

A metric that can only be measured by the producer and does not agree with
the users intuition as to the value they are getting for their money will
always be perceived as crooked, even if it is not.   lmap is a logical step
down this path.....

What we really need are better metrics.

The Model Based Metrics ID is my first attempt here, although the words in
the draft fall far short of telling the right story.   I have started a
"Model Based Metrics working doc" at
https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#
  I believe that the "Sustained burst test" at the bottom is (close to) the
metric that we need.  It has (is expected to have) a whole list of cool
properties, including actionable by the ISPs, useful to the users,
RTT insensitivity, and consequently vantage insensitivity.

BTW: I enable "public comment" on the doc, so all of you can add margin
notes.

I would like to request  an extended slot in IPPM, to present this.  It
should be in a proper ID by that time.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Sun, Nov 25, 2012 at 6:12 AM, Al Morton <[email-masked]> wrote:

>   the equivalent measurement points along the user's packet transfer
> This effort certainly needs folks with standards experience in all of the
> on the list of tests of course, and that list needs discussion, too.
>
> All packet-transfer services have a demarcation point where the customer
> connects their own equipment to the network. It is valuable from both the
> diagnostic and service verification points of view to have a measurement
> are the functionally equivalent alternatives in each access architecture?
> > This focus on architecture is deeply misguided.  As someone who's built
> one such architecture (Netalyzr), working on two others (Fathom, a browser
> extension, and techniques using GuruPlugs/Raspberry Pis), the architecture
> is the easy part.
> >
> > In all this work, having some "standard" way to communicate between my
> measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH
> depending on whether quick & easy or server side or mutual authentication
> is required, with whatever load-balancing is needed handled by random
> assignment of remote test node.
> >
> > The internal data store is whatever is appropriate for the project (eg.
> for Netalyzr its python pickled objects for storage on our web server, data
> in a Postgress database for analysis).
> >
> > Updates are handled appropriately (Netalyzr is updateless, its an applet
> so its always up-to-date.  Our android version will just use the app store
> update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are
> a hack using SSH, but its a fair amount of custom goop, and they update
> nightly).
> >
> > And all three end up having rather different architectural requirements.
>
> understands (roughly) the same thing when you say "client" and "server" and
> "controller" with reference to a set of measurements you define.
>
> *But yes, specifying a way to specify a measurement is largely arbitrary.
> I'd even advocate separating the representation from the transport,
> especially as the general problem of measurement interoperability has data
> sources operating and wildly different scales, but as you say deciding this
> is the easy part.
> *
> > The hard part is developing the test metrics and measurements, both in
> the abstract and within the API constraints of the target platform.  E.g.
> why I like the GuruPlugs and Raspberry Pi is that I can run Java, so
> therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same
> isn't the case for the Ripe Atlas or Bismark platform.
> >
> > And Marc's list is actually just a start: you also have generic proxy
> presence & location detection, application specific proxy detection and
> analysis, packet injection detection, differential application behavior
> detection, application agnostic traffic shaping detection, DNSSEC support,
> DNS manipulation detection, path MTU, IPv6 for all of the previous, and a
> ton of others.  Netalyzr is currently composed of >100 distinct tests.  And
> this amount just continues to grow.
> >
> >
> > If the IETF actually wants to have an impact in this area, don't bother
> focusing on protocols for communication, focus on tests with given
> attributes.  Because people like me who actually build these things are
> going to ignore an architectural focus because that has no value.  But
> having ACCEPTED STANDARD tests would be very valuable.
>
> grammar. What you need to make this at all interoperable is the vocabulary.
>
> IPPM has defined a several of these at a low level, with a focus on active
> measurement methods that produce comparable results. There appears to be
> interest in IPPM both in adding passive methods for the existing metrics
> (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more
> complex metrics based in part on the simple ones (see
> Flash (so TCP/UDP connections only to cooperating servers) would be great.
> Our current one in Netalyzr is only OK: we believe that one could do a lot
> better.
> >
> strongly correlated with application latency at a given hop, but I get your
> point...)
>
> also a focus on active measurement for the same reason. *On the other
> components (again, returning to "latency under load", the load traffic will
> be made up of generated as well as existing background load).
> *
> *So, what might be useful would be to open another line of work within
> IPPM framework as appropriate.* This could be used to provide a more
> complex vocabulary to LMAP, but could also be useful independently -- *if
> LMAP is eventually defined to apply only to a restricted set of use cases
> (i.e. specifically mandated tests for regulatory compliance, but not
> research or troubleshooting), comparable metrics with incompatible
> the measurement space*, the need for protocol shims (which are after all
> easy to write if the number inside has the same meaning) aside.
>
> Best regards,
>
> Brian

---

From: Al Morton <[email-masked]>
Date: Mon, 26 Nov 2012 00:20:44 -0500

Matt,

I think we agree that better metrics are needed for LMAP,

and that ideally a metric can be applied at any point or pair of
points.

RFC 2330 and most IPPM metrics have a decidedly Source-to-Destination

host construction. Frankly, we did a better job with arbitrary

measurement points in Rec Y.1540. It's worth a look:

http://www.itu.int/rec/T-REC-Y.1540/en

There are many goals the new metrics could serve:

- externally observable

- measurable by users

- easily related to user application performance

- service verification

- support consumer choice of ???

- etc.

We need to agree on the set that are needed.

While there may be some information in a measurement that covers

a combination of networks, measuring more than one network

is only useful when the combined network path delivers

the target performance - the sunny day outcome. Otherwise,

they are inconclusive, as you said in your memo. Some target

performance levels only apply in the scope where they are
offered.

We don't really have the milk fat scenario here, unless the
producer,

the truck driver, the convenience store, and your home refrigerator

can all add some unknown amount of fat to the milk. ;-)

When considering this problem, I've been thinking of the fuel

efficiency ratings provided for new cars in the US and Europe (at
least).

The specs illustrate very well that the quantity measured is
variable

depending on the conditions of the measurement (in the US, the miles per

gallon for city and highway driving), and they imply the

range of values that normal users can experience (a fairly wide
range).

These are some aspects that would be useful to reproduce in our

metrics, I think.

regards, and YMMV,

Al

At 04:20 PM 11/25/2012, Matt Mathis wrote:

I would like to disagree
here.   Another missing requirement from 2330 is that
measurement vantage point should not matter.   Consider the
standards for milk: milk fat is a parameter that anybody can measure
(producer, consumer, independent lab, school classroom, etc).  It
can be process controlled by the manufacturer (tweaking centrifuge
settings), and is useful to the consumer.  As an extremely robust
metric it stabilizes the industry and brings sanity to the
marketplace.

A metric that can only be measured by the producer and does not agree
with the users intuition as to the value they are getting for their money
will always be perceived as crooked, even if it is not.   lmap
is a logical step down this path.....

The Model Based Metrics ID is my first attempt here, although the words
in the draft fall far short of telling the right story.   I
have started a "Model Based Metrics working doc" at

https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#
I believe that the "Sustained burst test" at
the bottom is (close to) the metric that we need.  It has (is
expected to have) a whole list of cool properties, including actionable
by the ISPs, useful to the users, RTT insensitivity, and consequently
vantage insensitivity.

BTW: I enable "public comment" on the doc, so all of you can
add margin notes.

I would like to request  an extended slot in IPPM, to present
this.  It should be in a proper ID by that time.

Thanks,

--MM--

The best way to predict the future is to create it.  - Alan
Kay

Privacy matters!  We know from recent events that people are using
our services to speak in defiance of unjust governments.   We
treat privacy and security as matters of life and death, because for some
users, they are.

On Sun, Nov 25, 2012 at 6:12 AM, Al Morton
<[email-masked]>
wrote:

LMAP and IPPM,

path/architecture of each different access
technology.

measurements. So, for anyone who read Nick's and Brian's
messages

below (which I mostly agree with) and was about to un-subscribe or
add "lmap"

architecture-related. The various measurement points needed
depend

point very close to the demarcation point. But direct access to
this

point is often impossible for a measurement agent - where can we
agree

regards,

Al

Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics
discussion will be interesting there too).

> The internal data store is whatever is appropriate for the
project (eg. for Netalyzr its python pickled objects for storage on our
web server, data in a Postgress database for analysis).

> Updates are handled appropriately (Netalyzr is updateless, its
an applet so its always up-to-date.  Our android version will just
use the app store update mechanism.  Fathom we use Firefox's update
mechanism.  The plugs are a hack using SSH, but its a fair amount of
custom goop, and they update nightly).

requirements.

Architectures, I would submit (and as I've said here before), are
useful to hang terminology off of: you need boxes and lines so everyone
understands (roughly) the same thing when you say "client" and
"server" and "controller" with reference to a set of
measurements you define.

> The hard part is developing the test metrics and measurements,
both in the abstract and within the API constraints of the target
platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I
can run Java, so therefore I can just "Run Netalyzr" for all
Netalyzr tests.  But the same isn't the case for the Ripe Atlas or
Bismark platform.

+ 1e<large number>. Stated another way, the protocol gives you
the grammar. What you need to make this at all interoperable is the
vocabulary.

> E.g. a good, standard latency-under-load test that can be
written in Flash (so TCP/UDP connections only to cooperating servers)
would be great.  Our current one in Netalyzr is only OK: we believe
that one could do a lot better.

> If you could then figure out WHERE the bottleneck is (using a
server that can also do simultaneous traceroutes), this would be even
more valuable.

(This seems to make the assumption that traceroute-measurable latency
is strongly correlated with application latency at a given hop, but I get
your point...)

complex vocabulary to LMAP, but could also be useful independently -- if

Best regards,

Brian

ippm mailing list

[email-masked]

ippm mailing list

[email-masked]

---

From: <[email-masked]>
Date: Mon, 26 Nov 2012 10:26:22 +0100

Matt, Al,

one can measure accross a terminal device, home gateway, provider access system,
provider policy management node, provider gateway, central lmap capture system.

>From what I've seen and done, an undesired measurement result only tells me,
I have a problem. But it doesn't explain a black box, it doesn't explain, whether
and why a reference I compare an actual result against is relevant, what are
the differences and so on.

I doubt that all issues can be solved by a better metric alone.

Regards,

Rüdiger

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Matt Mathis
Sent: Sunday, November 25, 2012 10:21 PM
To: Al Morton
Cc: [email-masked]; [email-masked]
Subject: Re: [ippm] [lmap] Focus on Tests, Not Architectures...

I would like to disagree here.   Another missing requirement from 2330 is that measurement vantage point should not matter.   Consider the standards for milk: milk fat is a parameter that anybody can measure (producer, consumer, independent lab, school classroom, etc).  It can be process controlled by the manufacturer (tweaking centrifuge settings), and is useful to the consumer.  As an extremely robust metric it stabilizes the industry and brings sanity to the marketplace.

A metric that can only be measured by the producer and does not agree with the users intuition as to the value they are getting for their money will always be perceived as crooked, even if it is not.   lmap is a logical step down this path.....

The Model Based Metrics ID is my first attempt here, although the words in the draft fall far short of telling the right story.   I have started a "Model Based Metrics working doc" at https://docs.google.com/document/d/1KDJMuZ2cWfXbnWjgwNDpN4w5smvZLfCL-gTk6Xgx1mI/edit#    I believe that the "Sustained burst test" at the bottom is (close to) the metric that we need.  It has (is expected to have) a whole list of cool properties, including actionable by the ISPs, useful to the users, RTT insensitivity, and consequently vantage insensitivity.

BTW: I enable "public comment" on the doc, so all of you can add margin notes.

I would like to request  an extended slot in IPPM, to present this.  It should be in a proper ID by that time.

Thanks,
--MM--

Privacy matters!  We know from recent events that people are using our services to speak in defiance of unjust governments.   We treat privacy and security as matters of life and death, because for some users, they are.

On Sun, Nov 25, 2012 at 6:12 AM, Al Morton <[email-masked]<mailto:[email-masked]>> wrote:
LMAP and IPPM,

This effort certainly needs folks with standards experience in all of the
access architectures to complete the task, because working out
measurement point details will be a critical aspect of conducting comparable
measurements. So, for anyone who read Nick's and Brian's messages
below (which I mostly agree with) and was about to un-subscribe or add "lmap"
to your spam filter, don't.  There's plenty to do here that's
architecture-related. The various measurement points needed depend
on the list of tests of course, and that list needs discussion, too.

regards,
Al

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> This focus on architecture is deeply misguided.  As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.
>
> In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.
>
> The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis).
>
> Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.  Our android version will just use the app store update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly).
>

Architectures, I would submit (and as I've said here before), are useful to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same isn't the case for the Ripe Atlas or Bismark platform.
>
> And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.  Netalyzr is currently composed of >100 distinct tests.  And this amount just continues to grow.
>
>
> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build these things are going to ignore an architectural focus because that has no value.  But having ACCEPTED STANDARD tests would be very valuable.

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary.

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.  Our current one in Netalyzr is only OK: we believe that one could do a lot better.
>
> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load).

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian
_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 30 Nov 2012 23:53:08 +0000

One thing that I have learned through the MBA experience that it is hard to find a single metric that addresses all use cases. For example, consumers (and national bodies charged with consumer protection) might want to know whether advertised rates agree with actual rates. Unless ISPs are advertising the model-based metric, it may not be as useful. For other applications, such as VoIP, latency and packet loss for specific traffic types matter. Having a metric that is RTT-independent and uses a different traffic pattern is not helpful in that case, as much as it might be otherwise.

On a more general note, from a consumer perspective, there are probably three kinds of metrics that are useful:

(a) Comparison between advertised and real-life metric, to allow fair comparison of different providers.

(b) Relative metrics: I don't much care what the number is (it could be expressed in some number between 0 and 100, rather than Mb/s or ms), but it should preserve relative ordering. If provider A is significantly better according to the metric X, I should expect my application (related to that metric) to perform significantly better, even if I don't get the exact same performance. A rough example for that is cellular throughput: It is very unlikely that an individual consumer will get exactly the average throughput measured by whatever entity, given geographic and device variations, but they should still be able to pick a better-performing carrier based on that metric. That's similar to Al's gas mileage example, "your mileage may vary", but any Prius is still likely to get better gas mileage than a Lincoln Navigator.

(c) Threshold metrics: The metric should tell me whether a particular service is likely to be suitable for my need. Latency is the classical example: If my access link has a round-trip latency of 400 ms, I may die prematurely in a first-person-shooter game. Absolute accuracy isn't that important (say, 400 vs. 420 ms), but consistent definition of the path component and other attributes is.

Some metrics might serve all three roles, but the emphasis differs.

I would be interested in comments on the metrics that the FCC Measurement Broadband America project uses.

(a) Which ones need standardization first? (Some are already based on IPPM RFCs, but most aren't.)

(b) Which ones might be less valuable and/or should be substantially modified?

See http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf (pg. 22) for the current list.

Henning

________________________________
From: [email-masked] [[email-masked]] on behalf of Al Morton [[email-masked]]
Sent: Monday, November 26, 2012 12:20 AM
To: Matt Mathis
Cc: [email-masked]; Nicholas Weaver; [email-masked]; Brian Trammell
Subject: Re: [lmap] [ippm] Focus on Tests, Not Architectures...

Matt,

I think we agree that better metrics are needed for LMAP,
and that ideally a metric can be applied at any point or pair of points.

performance levels only apply in the scope where they are offered.

We don't really have the milk fat scenario here, unless the producer,
the truck driver, the convenience store, and your home refrigerator
can all add some unknown amount of fat to the milk. ;-)

When considering this problem, I've been thinking of the fuel
efficiency ratings provided for new cars in the US and Europe (at least).
The specs illustrate very well that the quantity measured is variable
depending on the conditions of the measurement (in the US, the miles per
gallon for city and highway driving), and they imply the
range of values that normal users can experience (a fairly wide range).
These are some aspects that would be useful to reproduce in our
metrics, I think.

regards, and YMMV,
Al