Subject: [lmap] Comments on LMAP architecture

From: Jérôme Benoit <[email-masked]>
Date: Thu, 27 Dec 2012 00:25:31 +0100

Hello, 

I've already stated that the logical architecture is overloaded. We
currently implementing a measurement system that will fit most of the
requirement needed to run large scale measurement from the network
edge and I still think that the thin functional separation should not
be exposed as a pre-requirement : 
- Why do you separate measurement result data from measurement
  definition and metadata ? when you want to dig data, you need the
  surrounding information that permit to aggregate data without the
  need to contact anyone. 
- How will you perform end to end measurement active measurement if the
  measurement client (I prefer agent) do not embed a measurement
  server ? 
- Are you sure that ISP will implement a network parameter server (I
  prefer network characteristics server) ? We plan to collect the
  information either via measurements or via the end user. And how
  will you trust the information given by this server ? 

I've really a bunch of comments to write (and probably RFCs when the
deployment of our measurement system will be on beta stage in France)
but I think it's enough to start a fruitful discussion. 

Cheers.   

-- 
Jérôme Benoit aka fraggle
La Météo du Net - http://grenouille.com
OpenPGP Key ID : 9FE9161D
Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 30 Dec 2012 04:14:56 +0000

Quick comments:

1) I'm not sure where you infer the separation of the two. You certainly don't want to include a complete measurement description with each individual measurement sample, so there will be some notion of a label that refers to the description. Can you describe your concern in more detail?

I'm envisioning something similar to Netflow, if that helps to make the discussion a bit more concrete.

2) I'm afraid I don't understand. I agree that 'agent' is a more generic term (and I will substitute this in the next version of the draft), but the idea is that the infrastructure agent and the in-home agent exchange active measurement traffic, as specified for each metric. In general, traffic will flow in both directions. In many practical scenarios, NAT considerations will dictate that the in-home agent initiates the (TCP) session or sends the ICMP request, but that's not a fundamental part of the architecture.

3) The parameter server is an optional component. Not every infrastructure will have one, but our experience is that ISPs are interested in providing accurate information and spend a fair amount of manual effort doing so in our current measurement project. They have an interest in providing accurate information since errors are likely to hurt their performance numbers. The parameter server helps in automating the process and avoids errors, dealing with spreadsheets and the like.

Unfortunately, it's not always possible to easily measure what the "nominal" performance is, particularly for more complicated technologies, such as cable with PowerBoost (temporary speed increases).

So far, we have found no reason not to trust the information provided by the ISP. If you want to get legal about it, false advertising laws in most countries, as well as competitors, would discourage any intentional fudging by the ISP. Also, the measurements provide a sanity check - if the ISP declared 100 Mb/s, but your measurements show 50, this would likely either cause the measurement sample to be discarded, or a manual investigation.

Henning

________________________________________
From: [email-masked] [[email-masked]] on behalf of Jérôme Benoit [[email-masked]]
Sent: Wednesday, December 26, 2012 6:25 PM
To: [email-masked]
Subject: [lmap] Comments on LMAP architecture

Hello,

Cheers.

---

From: Jérôme Benoit <[email-masked]>
Date: Mon, 31 Dec 2012 15:25:16 +0100

On Sun, 30 Dec 2012 04:14:56 +0000
Henning Schulzrinne <[email-masked]> wrote:

> Quick comments:
> 
> 1) I'm not sure where you infer the separation of the two. 

I infer because of the logical separation of the data collector and the
measurement server.

> You certainly don't want to include a complete measurement description with each individual measurement sample, 

Not in every sample, but before, I think I have to explain a bit how
your measurement agent work : grenouille_probe receive a measurement
definition expressed in a specific language that say for example "do
this and every ten seconds collect measurement results samples". To be
concrete : do FTP download of a 600 MB file from
ftp://pipo.com/iso_file_path/iso_file_name and every ten seconds,
give me the instant bitrate, the instant bitrate is what I call the
samples. The complete measurement result contains all timestamped
instant bitrate + the measurement definition + the access link
+ other relevant meta-data . The measurement collector do
serialization and data mining is easier.   

> so there will be some notion of a label that refers to the description.
> Can you describe your concern in more detail?

You can do it also with keeping an external and unique reference to the
measurement definition + access link characteristics + whatever is
relevant. But that will not really ease the data mining. The data
mining will trigger traffic to measurement controller and network
parameter server, which I think is not a good idea.  

> I'm envisioning something similar to Netflow, if that helps to make the discussion a bit more concrete.

Keep in mind that IPFIX architecture is relevant to passive
measurement. A network element raw flow (as IPFIX define them) contain
already almost all the information needed to do troubleshooting, the
filtering permit to isolate what you want to troubleshoot. 
active measurement results do not contain all the information needed,
only what you want to measure : timestamped TCP throughput from A to B
sample every 5 seconds, RTT from a forged packets train that aim to
trigger the delay from several landmarks, etc. all this numbers are
meaningless without measurement definition.   

> 2) I'm afraid I don't understand. I agree that 'agent' is a more generic term (and I will substitute this in the next version of the draft), but the idea is that the infrastructure agent and the in…

In practice, network usage include end to end traffic (between 1ton
NATed home networks). If you want to measure this kind of usage, the
measurement agent have to include the measurement server. Typical
measurement are P2P protocols metrics. There's other use case but
none of them are really ready for production (this mean that a
complete overlay thought end to end that permit to do most of the
diagnostic with a few overhead is not ready). 
In the future, the grenouille_landmark will be a parametrisable
socket servers shipped is the measurement agent. The agent will send
message status to a to be defined protocol that will help to keep
track of agent state, capabilities and home network functionalities
(UPnP on/off, etc.)
. 
> Unfortunately, it's not always possible to easily measure what the "nominal" performance is, particularly for more complicated technologies, such as cable with PowerBoost (temporary speed increases…

If ISPs care to collaborate, that fine. But if they do not want,
can't and that happen when you're not a backed by a national agency,
they tend to trick the measurement system but that's another story.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 31 Dec 2012 14:51:54 +0000

Let me try to amplify: The data collector is logically separate, but can obviously be in the same box. This is necessary for the cases where the agent on the home/enterprise side generates the measurements, as would be the case for most latency measurements, for example.

To summarize: Two measurement agents, one in the home and one in the network somewhere, exchange measurement traffic, such as, say, ICMP echo requests or a TCP download. The agent at home takes the metric (throughput, latency) plus identifying information (metric, time, endpoint identifier) and sends it to the data collector. The transmission format is likely different from the format made available to users of the measurement data. One proposal was to use IPFIX as a way to encode the data, as IPFIX is just a container format for records. (Obviously, this is only about the transmission format, not the other components of IPFIX, which are indeed not applicable.)

The data collector can then transform, combine and annotate data in any format, or multiple formats, for appropriate usage by authorized entities. It might make data available in an SQL database, a downloadable file or via an API, for example.

I'm not sure I understand why you want them to be the same entity. I admit I couldn't quite follow your IPFIX objection, but maybe you can explain, based on the summary above, where you see the problem.

Henning
________________________________________
From: Jérôme Benoit [[email-masked]]
Sent: Monday, December 31, 2012 9:25 AM
To: Henning Schulzrinne
Cc: [email-masked]
Subject: Re: [lmap] Comments on LMAP architecture

---

From: Jérôme Benoit <[email-masked]>
Date: Tue, 1 Jan 2013 06:45:24 +0100

On Mon, 31 Dec 2012 14:51:54 +0000
Henning Schulzrinne <[email-masked]> wrote:

Ok, don't get me wrong. 

> Let me try to amplify: The data collector is logically separate, but can obviously be in the same box. This is necessary for the cases where the agent on the home/enterprise side generates the meas…

Which is fine, just state it on the RFC (amplify the fact that the
logical separation can also be a implemented on the agent). 

> I'm not sure I understand why you want them to be the same entity. I admit I couldn't quite follow your IPFIX objection, but maybe you can explain, based on the summary above, where you see the pro…

My IPFIX objection is based a an real implementation of a well know
active measurement (with extras) and whatever how drunk is I am is, the
IPFIX data format do not respect Shannon's theoretical information
system standard speed derivation (the IPFIX standard is fine, but do
not include octet/time derivative unit) and that's a pity for active
measurement. Just state that fact in the RFC and I will never complain 
about IPFIX standard anymore (maybe, IPFIX is really a bunch of
crappy point in time metrics from my point of view, but I will launch
that delighted troll on an other mailing list :))

++