Subject: [lmap] My reflections on draft-schulzrinne-lmap-requirements-00

From: Nicholas Weaver <[email-masked]>
Date: Mon, 5 Nov 2012 09:58:47 -0800

It appears to be a good document, but I feel that it is putting the cart before the horse.

The difficult problem is coming up with meaningful tests, especially for a given level of system privilege, and ensuring that these tests are meaningful and well calibrated.

Even something "simple" like bandwidth testing becomes a huge can of worms: look at the effort that Ookla has gone through in developing a bandwidth test that is capable of operating across a wide variety of OSs (including OS-limitations such as the lack of window scaling) in the limited privilege context of a Flash applet.

Then the question becomes how to turn a bandwidth test into a realistic latency-under-load test (reliably measuring the bufferboat problem).

Then the question becomes identifying the bottleneck which is the source of the latency-under-load...

And all ideally in a world where the client is running with the permissions of Flash or HTML5 (although the server can be full custom)...

In comparison, the infrastructure for running measurements is trivia: one-and-done browser measurements work just fine by simply reporting the data back to the web server, and platforms for persistent measurements are more often defined by the hardware cost and installation on the client side, rather than the control software.  

And the nature of such systems having "one-off infrastructure" (e.g. SamKnows uses different infrastructure than BisMark than Ripe Atlas than Netalyzr) isn't that big of a problem compared with making sure the tests are sound and correct, within various client capability restrictions.

Overall, where I'd like to see this group go is focus on developing and standardizing tests, of the form:

Given a client with a given set of capabilities and a given persistence model, can one construct a test to measure X in an efficient and reliable manner?

Because there are a lot of tests that need to be done, and it is the tests themselves that is the problem area.