Subject: Re: [ippm] [lmap] Focus on Tests, Not Architectures...

From: Brian Trammell <[email-masked]>
Date: Tue, 20 Nov 2012 17:59:25 +0100

Hi, Nick,

(Copying this over to IPPM as well, as the focus-on-metrics discussion will be interesting there too).

On 20 Nov 2012, at 17:01 , Nicholas Weaver wrote:

> I've said it once, I'll say it one final time:
> 
> This focus on architecture is deeply misguided.  As someone who's built one such architecture (Netalyzr), working on two others (Fathom, a browser extension, and techniques using GuruPlugs/Raspberry Pis), the architecture is the easy part.
> 
> In all this work, having some "standard" way to communicate between my measurement clients and servers is useless.  I just use HTTP, HTTPS, or SSH depending on whether quick & easy or server side or mutual authentication is required, with whatever load-balancing is needed handled by random assignment of remote test node.   
> 
> The internal data store is whatever is appropriate for the project (eg. for Netalyzr its python pickled objects for storage on our web server, data in a Postgress database for analysis).  
> 
> Updates are handled appropriately (Netalyzr is updateless, its an applet so its always up-to-date.  Our android version will just use the app store update mechanism.  Fathom we use Firefox's update mechanism.  The plugs are a hack using SSH, but its a fair amount of custom goop, and they update nightly).  
> 
> And all three end up having rather different architectural requirements.

Architectures, I would submit (and as I've said here before), are useful to hang terminology off of: you need boxes and lines so everyone understands (roughly) the same thing when you say "client" and "server" and "controller" with reference to a set of measurements you define.

But yes, specifying a way to specify a measurement is largely arbitrary. I'd even advocate separating the representation from the transport, especially as the general problem of measurement interoperability has data sources operating and wildly different scales, but as you say deciding this is the easy part.

> The hard part is developing the test metrics and measurements, both in the abstract and within the API constraints of the target platform.  E.g. why I like the GuruPlugs and Raspberry Pi is that I can run Java, so therefore I can just "Run Netalyzr" for all Netalyzr tests.  But the same isn't the case for the Ripe Atlas or Bismark platform.
> 
> And Marc's list is actually just a start: you also have generic proxy presence & location detection, application specific proxy detection and analysis, packet injection detection, differential application behavior detection, application agnostic traffic shaping detection, DNSSEC support, DNS manipulation detection, path MTU, IPv6 for all of the previous, and a ton of others.  Netalyzr is currently composed of >100 distinct tests.  And this amount just continues to grow.
> 
> 
> If the IETF actually wants to have an impact in this area, don't bother focusing on protocols for communication, focus on tests with given attributes.  Because people like me who actually build these things are going to ignore an architectural focus because that has no value.  But having ACCEPTED STANDARD tests would be very valuable.  

+ 1e<large number>. Stated another way, the protocol gives you the grammar. What you need to make this at all interoperable is the vocabulary. 

IPPM has defined a several of these at a low level, with a focus on active measurement methods that produce comparable results. There appears to be interest in IPPM both in adding passive methods for the existing metrics (see draft-trammell-ippm-hybrid-ps) as well as adding definitions of more complex metrics based in part on the simple ones (see draft-mathis-ippm-model-based-metrics).

> E.g. a good, standard latency-under-load test that can be written in Flash (so TCP/UDP connections only to cooperating servers) would be great.  Our current one in Netalyzr is only OK: we believe that one could do a lot better.
> 
> If you could then figure out WHERE the bottleneck is (using a server that can also do simultaneous traceroutes), this would be even more valuable.

(This seems to make the assumption that traceroute-measurable latency is strongly correlated with application latency at a given hop, but I get your point...)

The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be useful in the latency-under-load example you give), because these tend to be counter to the goals of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement for the same reason. On the other hand, a lot of work in distributed measurement systems is focused on measuring the network as it is, with implicit or explicit passive components (again, returning to "latency under load", the load traffic will be made up of generated as well as existing background load). 

So, what might be useful would be to open another line of work within IPPM on interoperable descriptions of some of these existing tests as empirically specified metrics, perhaps with relaxed compliance with the IPPM framework as appropriate. This could be used to provide a more complex vocabulary to LMAP, but could also be useful independently -- if LMAP is eventually defined to apply only to a restricted set of use cases (i.e. specifically mandated tests for regulatory compliance, but not research or troubleshooting), comparable metrics with incompatible architecture and protocols could still lead to wider interoperability in the measurement space, the need for protocol shims (which are after all easy to write if the number inside has the same meaning) aside.

Best regards,

Brian

---

From: KEN KO <[email-masked]>
Date: Tue, 20 Nov 2012 17:53:19 +0000

>Since client and controller are deployed by the same entity, it should be up to that entity to figure out their control architecture, and that entity should have full say in how to do their own control architecture.

I don't think it's a universally held assumption that the client and controller are deployed by the same entity. For instance, a regulator use case spanning multiple operators could use a controller function (and, a data collection function) that communicates with clients deployed by each of those operators. So I don't think we should ignore architecture.

That said, I do believe that focusing on tests is an excellent idea and that that work could progress in ippm while other lmap issues are still being discussed.

---

From: Shane Amante <[email-masked]>
Date: Mon, 26 Nov 2012 21:40:02 -0700

On Nov 20, 2012, at 10:53 AM, KEN KO <[email-masked]> wrote:
> I don't think it's a universally held assumption that the client and controller are deployed by the same entity.

+1.

> For instance, a regulator use case spanning multiple operators could use a controller function (and, a data collection function) that communicates with clients deployed by each of those operators. So I don't think we should ignore architecture.

Agreed.  I also want to stress that there are other applicable, non-regulatory use cases, as well, where (for example) a single controller speaks to LMAP measurement clients deployed across a multitude of operators.

> That said, I do believe that focusing on tests is an excellent idea and that that work could progress in ippm while other lmap issues are still being discussed.

Agreed.

I'd also add that there may be two solution spaces emerging wrt "metrics" (in IPPM) that we discover.  First, as Brian Trammell pointed out in his e-mail on this thread:
> The efforts of IPPM to date have been focused on simply describable, easily repeatable metrics... there's not a lot of specification of arbitrarily complex or unknown preconditions (as would be
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> useful in the latency-under-load example you give), because these tend to be counter to the goals
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> of maximizing repeatability and minimizing uncertainty. There's also a focus on active measurement
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  for the same reason.

... these may be applicable for an extremely large number of (or, all?) "LMAP Measurement Clients" and provide 'basic' measurement capabilities, e.g.: UDP packet loss, UDP latency, etc.

On the other end of the spectrum, I think this is where the research community is still (?) in the midst of exploring + defining "new" metrics that can then be used in small to medium-size experiments, (e.g.: on a smaller base of "LMAP Measurement Clients"), whose scientific results can ultimately inform which of these tests may get formalized into standards by the IETF and, if appropriate, incorporated as "new" tests or replace one or more existing tests (completely).  Obviously, how those tests ultimately get rolled out to the field (potentially through firmware updates or only when new generations of HW + firmware are shipped), will be determined by the extent of the change, etc.

While I have an interest in following the new(er) metrics being explored, in the 2nd paragraph, I am more interested in and do not want us to lose sight of those basic metrics, in the first paragraph, as I believe a lot of valuable data can still be gleaned from them, today.

-shane