Subject: [lmap] List of metrics

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 3 Dec 2012 04:14:48 +0000

Currently, all the active metrics assume an idle link, as the goal is to measure the link performance without background traffic. (We can leave out the only passive metric, the total byte count, as that's probably not all that interesting for this discussion.)

At least from our experience, removing samples due to user activity is not a major problem, i.e., it removes only a marginal number of busy-hour samples.

I don't see how any other measurement policy can yield useful results for the current metrics.

Henning

________________________________________
From: [email-masked] [[email-masked]] on behalf of Al Morton [[email-masked]]
Sent: Sunday, December 02, 2012 10:02 PM
To: Henning Schulzrinne; Matt Mathis
Cc: [email-masked]; Nicholas Weaver; Brian Trammell; [email-masked]
Subject: Re: [lmap] [ippm]  Focus on Tests, Not Architectures...

Changing topics slightly, in the list:
http://transition.fcc.gov/cgb/measuringbroadbandreport/2012/Technical-Appendix.pdf
(pg. 22)

I'd prefer to pare this down to start. Even where we have RFCs to refer to,
the details of the test stream (for active measurement without user traffic,
or active/hybrid measurement during user traffic) need to be agreed,
and even the question of whether tests will be conducted during user
activity or not needs discussion, maybe this is the first question to
address.

Al

---

From: Al Morton <[email-masked]>
Date: Mon, 03 Dec 2012 09:21:55 -0500

Ok, then the UDP latency and UDP packet loss tests were suspended
on detecting user activity, and the Latency under Load test
used other active tests to provide a known load.

I think it would be worthwhile to discuss hybrid active+passive
testing, at least. This is what Brian had proposed in
http://tools.ietf.org/html/draft-trammell-ippm-hybrid-ps-00
Such tests may tend toward use in diagnostics rather than consumer info,
and the value may be tightly linked with the passive characterization
of user traffic (appropriately anonymized and privatized, of course),
IOW it may not be useful if the load is overly summarized.

And some of the Technical Appendix page 22 list also seem diagnostic
oriented and somewhat overlapping (ICMP latency, UDP Latency).
It's probably good to think of usage categories as we proceed.

Al

At 11:14 PM 12/2/2012, Henning Schulzrinne wrote:
>Currently, all the active metrics assume an idle link, as the goal 
>is to measure the link performance without background traffic. (We 
>can leave out the only passive metric, the total byte count, as 
>that's probably not all that interesting for this discussion.)
>
>At least from our experience, removing samples due to user activity 
>is not a major problem, i.e., it removes only a marginal number of 
>busy-hour samples.
>
>_______________________________________________
>ippm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/ippm

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 3 Dec 2012 18:25:18 +0000

Yes, for all active tests, tests are suspended when user activity is detected. Besides measurement accuracy, the perception was that if users (= volunteers) got the notion that participating in this project would slow down their Internet connection, we'd lose them pretty quickly.

The UDP and ICMP latency tests aren't meant to be diagnostic - they do highlight important technology differences. As you can see from the published reports (and the GaTech SIGCOMM paper), DSL tends to impose significantly higher delays than cable, due to interleaving and other factors.

I agree that hybrid tests could be useful and would likely be seen as more diagnostic, given their inherent higher variability. I certainly wouldn't want to exclude them from consideration, and don't see much difficulty of integrating them into the LMAP framework, but they are probably of lower priority from a public policy perspective.

________________________________________
From: Al Morton [[email-masked]]
Sent: Monday, December 03, 2012 9:21 AM
To: Henning Schulzrinne; Matt Mathis
Cc: [email-masked]; [email-masked]
Subject: Re: [ippm] List of metrics

Al