Subject: [lmap] Comments on the current round of I-Ds, before the meeting tomorrow

From: Steve Miller <[email-masked]>
Date: Tue, 5 Nov 2013 23:11:26 -0500

Hi, I'm Steve Miller, and (despite the email address (-: ) I work for Verisign.  I've been lurking on the lmap list for quite a while, and while I have some background in working on systems such as the ones being discussed here, I haven't had a chance until the flight out to Vancouver to really dig into the current crop of I-Ds.  I figured I'd send my comments out here, as while I have a fair number of them, it's not clear to me that any of them are necessarily worth my tying up a microphone during the WG meeting.

   I'll start by saying that I'm excited to see a group tackling this sort of large-scale performance-measurement problem.  I think this sort of thing has huge potential to be useful to ISPs, to content providers, and (directly or indirectly) to the end users.

   And I'll say that the answer "go read the mail archive, we've covered all this" is also acceptable. (-:  I'm in that unenviable position where I feel like I'm behind on all that's transpired, so while I feel like I've done a lot of my homework I doubt I've done it all.  I hope some of what I say makes sense and that the places where I screw up, I don't screw up too much...

   Before I work through the drafts, I had some general notes about time.  Time handling seems to be one of those things that shouldn't be all that hard, but given how many implementors I see get it wrong, I'm thinking it'd be good if we can stack the odds a bit in our favor.  I'd think very seriously about changing all the timestamp stuff in here to be seconds and microseconds since midnight 1970 UTC (the usual Unix-like timestamp, though of course going to nanoseconds wouldn't be tragic).  I'd also think about enforcing the idea that all the MAs and the collectors (and probably the controllers) all express and interpret all times as UTC, as otherwise -- at least in a system where more than one organization may be using the same set of MAs, or where the set of MAs is made up of systems controlled by more than one organization -- we'll end up with half the stuff in UTC and a quarter in EST and some in every other timezone out there.

   Finally, it might be worth calling out that all these systems need to have their clocks synced using NTP or GPS signals or something roughly equivalent, or the users of such a system will tear their hair out trying to cross-correlate results from different MAs.  It is possible that this sort of scenario is responsible for some of my own hair loss.

   Now, working through the drafts in order from the PDF in the agenda...

LMAP Protocol (draft-bagnulo-lmap-http-01)
------------------------------------------
I see several places in the examples, at least, where the MA is POSTing its IP address: e.g., one where it's posting that as part of a dump of interface information, and one where it's posting it as part of the results for the sample UDP latency check.  For a MA that's sitting at the boundary of a network (e.g., the router in some end-user's house), that's not too bad to do, but for a MA that's further inside such a network, it might be difficult for it to know what "real" (non-RFC-1918) address it's using.

I could definitely see how having it publish the real address that it used would be useful to the collector but if we need that we'll have to figure out how to have the MA discover the real address it'd be using.  I'm not sure how useful it is to just have the RFC1918 address, though.

It occurred to me, too, thinking more about the case where the MA is inside a network rather than at its boundary, saying stuff about the interface speed should be taken with a grain of salt.  After all, if the MA says it's connected at GigE, that's great, but if it's GigE up to the DSL line, we'll see some possibly-surprising test results.

When it comes to handling communications failures (6.3.1), there's all the usual stuff to worry about in terms of what to do about a more prolonged communications failure.  I could see arguments why the Controller would want all the resultsbut "all" can be a dangerous word.  I think we could also make an argument that the results of measurement X are not as interesting once the results for measurement X+1 of the same type are available.  Should we issue any guidance?  Or just assume that this is the sort of detail that the implementors will get right?  There's a fair amount of discussion about exponential backoff but not anything to say when to stop. (-:

FWIW, with respect to not doing tests when there's cross-traffic, or at least when it comes to indicating that there is cross traffic, in some similar things I've worked on I've relied on tracking the results on a per-MA basis as a hint whether the suitability of a given MA has been compromised.  That is, if the performance for a given MA for a related set of checks has been at a certain level and then changes to a different level (more loss, higher latency, etc.), that's a strong hint as to the performance of the network in or around the MA.  For that matter, there are many things that can happen between the MA and what it's testing, that could make the results from that MA look worse, so to some extent it'd be easy to look at the "no cross traffic, all awesome!" indicator and end up with a false sense of security.

A lot of that comes down to the age-old problem in this problem space, when device X says there's a problem with Y, if Y is actually broken or if X is broken.

All that being said, given that the MA may not be executing measurements solely on behalf of a single requestor I'm not sure that I see how to cross-correlate the results from a single MA.

Information Model (draft-burbridge-lmap-information-model-01)
-------------------------------------------------------------
About the security model (not really specific to this I-D, admittedly): the idea of using the group ID as a security measure, to make it harder to associate the results of a measurement with an individual MA, seems optimistic.  Not that I have a better idea, mind you, but I did feel that I should say that.

About periodic timing: why specify the interval in milliseconds and not, say, microseconds, or even just fractional seconds (e.g., 0.2)?  Sure, I think the ability to launch tests with microsecond precision is a bit questionable, but I don't see how specifying greater precision really hurts us so long as people understand that the MA's ability to hit that precisely -- even on a millisecond basis -- is going to be somewhat limited.

I'm not sure that I saw anything call out what a MA should do if it can't execute a measurement at the specified time (e.g., it's too busy with other work to do so).  Another classic failure of simplistic implementations of systems like the MA is to assume that they can complete all work in the time allowed, and when inevitably that doesn't happen (e.g., every test takes much longer than usual because they all time out because the device has lost connectivity) they just get further and further behind.  I think in that sort of scenario the MA should simply not execute any measurements that it can't do on time, alerting to its manager that it has an issue and if possible reporting to the relevant collector for each measurement it can't do that it couldn't do that measurement.

Metrics Registry (draft-mornulo-ippm-registry-00)
(and draft-mornulo-ippm-registry-columns-01)
-------------------------------------------------
It might be good to have the idea captured somewhere, somehow that we could have measurements that return a histogram of their results or some other indication as to their distribution (e.g., a histogram of latencies in the "go do this 100 times and then tell me the results" case).

It's important to express the concept of null values as well (which might be covered, I wasn't sure).  I've seen lots of people try to use sentinel values to represent things like the latency in the case of 100% loss, and that tends to end up destroying information ("it's not zero, it's undefined!") and ultimately often ends up getting averaged inadvertantly with real data ("hey, this had 99% loss, but the latency is freakishly low").

It seemed like this was calling out the idea of expressing the registry as columns in what seems logically like a database or spreadsheet, but I like the idea of expressing it in JSON or YAML or something more like that.  I may be reading this too literally here.

LMAP Framework (draft-ietf-lmap-framework-01)
---------------------------------------------
In general, the model here always has the MA connecting to the collector to submit the results of a measurement.  I've run into situations before where it's been considered preferable to have the collector connect outbound to the MAs to get the relevant results.  Do we have a way to address that?  (Admittedly that's more difficult given that many MAs are likely to be behind NATs; we'd have to address that as well, even if we just pointed to suggestions as to how the MA could keep a line of communication open.  I know that's more or less a solved problem even without resorting to uPNP or whatever, because I know that some of the VOIP vendors do it, even though the solutions are all kinda heedious.)

This may be a little controversial to say, but for a system of significant scale it seems unlikely that using a SQL database for the Results Database will work as well as one might hope (ask me how I know).  It might be worth calling that out, though that's a minor tweak.

In 4.2, there's discussion about having just one controller at a time, but I thought the HTTP draft said or implied that a MA might have more than one controller telling it what to do.

p. 14, at the top, discussion about what to do if the controller fails: I'm not sure that's all that important to consider here.  It seems like the MA can just do another DNS lookup to locate the controller (and the owner of the controller can change the IP address returned for the controller if needed), or the owner of the controller can just stand up another host at the same IP address -- in short, that seems like a solved problem and in this case I'm not sure it's worth calling out the solutions.

LMAP Use Cases (draft-ietf-lmap-use-cases-00)
---------------------------------------------
There's a specific use case in which I have quite a lot of interest, and while the text that's here already certainly doesn't preclude that use case, I don't know if it'd be worth laying out the use case I envision, or if it's too special-case.

I think there's an unspoken assumption here that a measurement is from point A to point B, when for certain types of measurements -- DNS being the first one that comes to my mind! (-: -- are from point A to one of potentially many points B, because B is anycasted.  That's also not too different from the case where A finds B via a DNS lookup, but B has more than one IP address.

Most, if not all, of [a-m].root-servers.net are anycasted, as are several of the com/net service addresses, and certainly Verisign's peers and competitors do the same sorts of things, they're standard operating procedure in the DNS world.

Something that I think all operators have to deal with is that with heavily anycasted infrastructure, it's easy to end up in a situation where some users have poor connectivity to some service instance, but there's no good way for the operator of the infrastructure to *know* that some users are having issues.  For example, if someone connected to an ISP in Cairo is having issues hitting the j.root-servers.net instance in Cairo, I don't have a good way to know that unless I can also test from within that user's ISP.

So having the ability to perform a small number of tests from a very broad range of MAs with wide IP-topologic scope would be massively useful.  Going along with that, we'd need to have some way as part of the measurement being performed to have the MA identify the instance it's reached -- including the result of a "chaos null hostname.bind" query, a traceroute, or somesuch, would be needed.

Anyway, we might be weird, but since I think we have a lot of company, I thought it was worth calling out this use case...

There's also a lot of emphasis in this and the other docs about backing off when the link is in use, but I think there's a lot to be learned from performing relatively low-bandwidth tests all the time, even when the link is otherwise in use, small enough that no real backoff approach is needed.  You can learn a lot from pinging something every 10 seconds or so, without burning a lot of bandwidth.  I agree that at least in certain scenarios we'd have to worry about counting against someone's data allowance.

There's some stuff in here about aggregation, and in particular about using aggregation in part to help preserve user privacy.  I'm not sure that I'm personally all that keen on aggregation; it seems like if you do short-term aggregation, inevitably you say "hey this thing is busted!" and then you want the details, which of course by then are gone.  In the longer term, I also think it's easy to get into situations where there's something odd with the data, or you need to lookat it in a different way, and with aggregated data it can be difficult to track down the source of (for example) data-collection problems that happened months ago.  I understand the desire to aggregate, from both a privacy standpoint and because it'd be nice not to buy disks by the truckload, but I wanted to call this out as well.

	-Steve

---

From: <[email-masked]>
Date: Mon, 25 Nov 2013 16:55:24 +0000

Steve
Thanks for all the comments
Replying first on use cases

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Steve Miller
> Sent: 06 November 2013 04:11
> To: [email-masked]
> Subject: [lmap] Comments on the current round of I-Ds, before the
> meeting tomorrow
> 
.... 
> There's a specific use case in which I have quite a lot of interest,
> and while the text that's here already certainly doesn't preclude that
> use case, I don't know if it'd be worth laying out the use case I
> envision, or if it's too special-case.
> 
> I think there's an unspoken assumption here that a measurement is from
> point A to point B, when for certain types of measurements -- DNS being
> the first one that comes to my mind! (-: -- are from point A to one of
> potentially many points B, because B is anycasted.  That's also not too
> different from the case where A finds B via a DNS lookup, but B has
> more than one IP address.
> 
> Most, if not all, of [a-m].root-servers.net are anycasted, as are
> several of the com/net service addresses, and certainly Verisign's
> peers and competitors do the same sorts of things, they're standard
> operating procedure in the DNS world.
> 
> Something that I think all operators have to deal with is that with
> heavily anycasted infrastructure, it's easy to end up in a situation
> where some users have poor connectivity to some service instance, but
> there's no good way for the operator of the infrastructure to *know*
> that some users are having issues.  For example, if someone connected
> to an ISP in Cairo is having issues hitting the j.root-servers.net
> instance in Cairo, I don't have a good way to know that unless I can
> also test from within that user's ISP.
> 
> So having the ability to perform a small number of tests from a very
> broad range of MAs with wide IP-topologic scope would be massively
> useful.  Going along with that, we'd need to have some way as part of
> the measurement being performed to have the MA identify the instance
> it's reached -- including the result of a "chaos null hostname.bind"
> query, a traceroute, or somesuch, would be needed.
> 
> Anyway, we might be weird, but since I think we have a lot of company,
> I thought it was worth calling out this use case...
> 
> There's also a lot of emphasis in this and the other docs about backing
> off when the link is in use, but I think there's a lot to be learned
> from performing relatively low-bandwidth tests all the time, even when
> the link is otherwise in use, small enough that no real backoff
> approach is needed.  You can learn a lot from pinging something every
> 10 seconds or so, without burning a lot of bandwidth.  I agree that at
> least in certain scenarios we'd have to worry about counting against
> someone's data allowance.

I think this could be worth mentioning more directly in the use cases &/or framework.
Will add something in the next iteration

Will also mention about the value of low bandwidth tests, completely agree with you on this
Thanks
phil

---

From: <[email-masked]>
Date: Tue, 26 Nov 2013 16:05:12 +0000

Steve,
thanks for your careful reading - following up on the framework

> In general, the model here always has the MA connecting to the
> collector to submit the results of a measurement.  I've run into
> situations before where it's been considered preferable to have the
> collector connect outbound to the MAs to get the relevant results.  Do
> we have a way to address that?  (Admittedly that's more difficult given
> that many MAs are likely to be behind NATs; we'd have to address that
> as well, even if we just pointed to suggestions as to how the MA could
> keep a line of communication open.  I know that's more or less a solved
> problem even without resorting to uPNP or whatever, because I know that
> some of the VOIP vendors do it, even though the solutions are all kinda
> heedious.)

[phil] ok, I'll try something like: the information flow is MA to Collector, but it's possible the Collector initiates communications
(similarly, the Control Protocol part is supposed to say that the information flow is Controller to MA, but most likely the MA initiates communications as behind a NAT. )

> 
> This may be a little controversial to say, but for a system of
> significant scale it seems unlikely that using a SQL database for the
> Results Database will work as well as one might hope (ask me how I
> know).  It might be worth calling that out, though that's a minor
> tweak.

[phil] I think the merits of sql, hadoop etc is a bit too detailed.

> 
> In 4.2, there's discussion about having just one controller at a time,
> but I thought the HTTP draft said or implied that a MA might have more
> than one controller telling it what to do.

[phil] that's a comment for the http draft!

> 
> p. 14, at the top, discussion about what to do if the controller fails:
> I'm not sure that's all that important to consider here.  It seems like
> the MA can just do another DNS lookup to locate the controller (and the
> owner of the controller can change the IP address returned for the
> controller if needed), or the owner of the controller can just stand up
> another host at the same IP address -- in short, that seems like a
> solved problem and in this case I'm not sure it's worth calling out the
> solutions.

[phil] agree, will delete this. 

Thanks!
phil