Subject: Re: [lmap] Comments on LMAP architecture

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 2 Jan 2013 03:23:09 +0000

I'm not suggesting RFC 5655 as an output file format, although there might be elements in the RFC that are worth looking at.

I'm more talking about something like
http://en.wikipedia.org/wiki/IP_Flow_Information_Export

I admit that I lost you at how Shannon got involved in packet formats.

However, this is already straying into solution space, so probably premature. I'm just trying to make things a bit more concrete, since I'm finding abstract discussions less helpful and prone to talking past-each-other.

Henning

________________________________________
From: Jérôme Benoit [[email-masked]]
Sent: Tuesday, January 01, 2013 12:45 AM
To: Henning Schulzrinne
Cc: [email-masked]
Subject: Re: [lmap] Comments on LMAP architecture

On Mon, 31 Dec 2012 14:51:54 +0000
Henning Schulzrinne <[email-masked]> wrote:

>
> To summarize: Two measurement agents, one in the home and one in the network somewhere, exchange measurement traffic, such as, say, ICMP echo requests or a TCP download. The agent at home takes the metric (throughput, latency) plus identifying information (metric, time, endpoint identifier) and sends it to the data collector. The transmission format is likely different from the format made available to users of the measurement data. One proposal was to use IPFIX as a way to encode the data, as IPFIX is just a container format for records. (Obviously, this is only about the transmission format, not the other components of IPFIX, which are indeed not applicable.)
>
> The data collector can then transform, combine and annotate data in any format, or multiple formats, for appropriate usage by authorized entities. It might make data available in an SQL database, a downloadable file or via an API, for example.
>
> I'm not sure I understand why you want them to be the same entity. I admit I couldn't quite follow your IPFIX objection, but maybe you can explain, based on the summary above, where you see the problem.
>

My IPFIX objection is based a an real implementation of a well know
active measurement (with extras) and whatever how drunk is I am is, the
IPFIX data format do not respect Shannon's theoretical information
system standard speed derivation (the IPFIX standard is fine, but do
not include octet/time derivative unit) and that's a pity for active
measurement. Just state that fact in the RFC and I will never complain
about IPFIX standard anymore (maybe, IPFIX is really a bunch of
crappy point in time metrics from my point of view, but I will launch
that delighted troll on an other mailing list :))

++

--
Jérôme Benoit aka fraggle
La Météo du Net - http://grenouille.com
OpenPGP Key ID : 9FE9161D
Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D

---

From: Jérôme Benoit <[email-masked]>
Date: Wed, 2 Jan 2013 13:53:14 +0100

On Wed, 2 Jan 2013 03:23:09 +0000

> http://en.wikipedia.org/wiki/IP_Flow_Information_Export

I will read it when time permit. 

> However, this is already straying into solution space, so probably premature. I'm just trying to make things a bit more concrete, since I'm finding abstract discussions less helpful and prone to ta…

I admit I shouldn't have post my last email (don't post when drunk :p). 

But before I find time to dig further, have the FCC already started an
implementation of an measurement system ?

Happy new year.

---

From: James Miller <[email-masked]>
Date: Wed, 2 Jan 2013 09:14:13 -0500

Jerome,

I'm not sure if you are asking if the FCC has done anything with IPFlow
Info Export or a measurement system in general.  We have been running a
fixed program for two years with a network of 8000 some probes and are
preparing to release our third report in the coming weeks.  We are also
currently on schedule to begin mobile performance testing on smartphones
early this year.

Note our interest is to offer the work we've done as a helpful reference
and do not have an expectation that any IETF work would be bound to the
particular solution we have.

We do implement in our program a push model akin to the IPFIX and the
observation points (clients in our parlance) do deliver test results to
collectors that are run by our contractor.

The significant difference from IPFIX is that our program is an active
testing environment so we are fundamentally not doing passive observations.
 We have collected a limited set of byte count data from some of the probes
but it is not implemented uniformly across the platform.  I will discuss
rfc3917 IPFIX with our contractor again.

Happy to answer any other deep dive questions you might have as you are
thinking about the draft and ongoing work.

On Wed, Jan 2, 2013 at 7:53 AM, Jérôme Benoit
<[email-masked]>wrote:

> > I'm not suggesting RFC 5655 as an output file format, although there
> might be elements in the RFC that are worth looking at.
> >
> > However, this is already straying into solution space, so probably
> premature. I'm just trying to make things a bit more concrete, since I'm
> finding abstract discussions less helpful and prone to talking
> past-each-other.
>
> _______________________________________________
> lmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lmap
>
>

---

From: "Michael De Leo (mdeleo)" <[email-masked]>
Date: Wed, 2 Jan 2013 16:03:05 +0000

I'm thinking we might want to detail more the use cases for measurement. In the case of having a server and collector, there are many things that can be tested beyond bandwidth IMHO that might be valuable to the consumer as well as the ISP. I think this would be important to most users, but keeping the architecture open to other test for future flexibility (without overcomplicating).

Some work like ICSI Analyzer from Berkeley (I have no relation to the project), do elements at the IP layer level. Some important measurements are port filtering, DNS caching, because we are in transition IPv6 support and NAT detection.

Maybe an XML schema with test definitions might be more flexible?

Mike

From: [email-masked] [mailto:[email-masked]] On Behalf Of James Miller
Sent: Wednesday, January 02, 2013 9:14 AM
To: Jérôme Benoit
Cc: [email-masked]; Henning Schulzrinne
Subject: Re: [lmap] Comments on LMAP architecture

Jerome,

I'm not sure if you are asking if the FCC has done anything with IPFlow Info Export or a measurement system in general.  We have been running a fixed program for two years with a network of 8000 some probes and are preparing to release our third report in the coming weeks.  We are also currently on schedule to begin mobile performance testing on smartphones early this year.

Note our interest is to offer the work we've done as a helpful reference and do not have an expectation that any IETF work would be bound to the particular solution we have.

We do implement in our program a push model akin to the IPFIX and the observation points (clients in our parlance) do deliver test results to collectors that are run by our contractor.

The significant difference from IPFIX is that our program is an active testing environment so we are fundamentally not doing passive observations.  We have collected a limited set of byte count data from some of the probes but it is not implemented uniformly across the platform.  I will discuss rfc3917 IPFIX with our contractor again.

Happy to answer any other deep dive questions you might have as you are thinking about the draft and ongoing work.

On Wed, Jan 2, 2013 at 7:53 AM, Jérôme Benoit <[email-masked]<mailto:[email-masked]>> wrote:
On Wed, 2 Jan 2013 03:23:09 +0000

Happy new year.

_______________________________________________
lmap mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/lmap

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 3 Jan 2013 22:55:10 +0000

I agree that we don't want to be too specific as to the end points and throughput/delay measurements. But all measurements have a set of features in common:
- they have one or more endpoints, with "two" being a common case (the other end point may not be fixed, as for the DNS example you mention)
- they are either a single event or repeat
- they generate results (tuples)

Just to make this more concrete again, imagine a simple request-response protocol that has the following functionality:

"Hi, this is your controller speaking. Dear measurement agent, please start the X(5060) metric (= port blocking test for port 5060) and the Y(example.com) metric (= DNS caching test with parameter example.com), run it once, and deposit the results at collector Z. Thank you."

"Hi controller. I can't do X, but I'm happy to do Y."

I suspect that this functionality can be expressed in XML, JSON or even ASN.1. I wager that the latter will be the less likely outcome.

Henning

________________________________
From: [email-masked] [[email-masked]] on behalf of Michael De Leo (mdeleo) [[email-masked]]
Sent: Wednesday, January 02, 2013 11:03 AM
To: [email-masked]
Subject: Re: [lmap] Comments on LMAP architecture

I’m thinking we might want to detail more the use cases for measurement. In the case of having a server and collector, there are many things that can be tested beyond bandwidth IMHO that might be valuable to the consumer as well as the ISP. I think this would be important to most users, but keeping the architecture open to other test for future flexibility (without overcomplicating).

Mike

Jerome,

Happy new year.

---

From: Jérôme Benoit <[email-masked]>
Date: Sat, 5 Jan 2013 01:04:41 +0100

On Wed, 2 Jan 2013 09:14:13 -0500
James Miller <[email-masked]> wrote:

> Info Export or a measurement system in general.

A measurement system for switched network in general. 

> We have been running a

We plan to do the same (+ FOSS implementation). I think we should
continue to discuss all the glory details that we have encountered.  

> collectors that are run by our contractor.

measurement agents push measurement results, and pull measurements
definition. You can't do otherwise when you are behind a CPE.  

> rfc3917 IPFIX with our contractor again.

I very well understand that the proposed architecture mimic the IPFIX
one. But again, IPFIX is thought for passive measurement. 

To make this sentence clearer : 

* when we push active measurement
results to collectors, the results do not fit in the data model
that IPFIX propose without cheating around the model, we have for
example instant bandwidth, we have RTT between fixed hops that 
might fellow very precise time series. Try to express a traceroute
result in the IPFIX data model, it's possible but not the right
solution to do so. But I'm OK to take part of the IPFIX data model
and extend it to fit active measurement requirements.   

* when we need to coordinate n measurement agents, there's no logical
component that permit to trigger such a coherence in the measurement
system. I have a measurement that need i agents behind the same
edge router as a requirement and to achieve this kind of active
measurement requirement, we have introduced a measurement agents
states and capabilities tracker logical component. This component is
also responsible for not overloading the network for intrusive
measurement. I think this component is a must have in a large scale
measurement system. It can be added to measurement controller
functionalities. We do not have worked on the coherency and
scheduling side so much so far but we plan to start to work on a
syntax that will permit to express measurement dependencies and
trigger coherence to the measurement system. We do not know yet
where we will put this kind of intelligence, WIP. The active and
passive measurement intelligence is on the measurement agents (datagram
filtering, datagram forge, datagram injection, sampling results, etc.).

I hope I've made myself clearer.

---

From: Jérôme Benoit <[email-masked]>
Date: Sat, 5 Jan 2013 01:40:00 +0100

On Thu, 3 Jan 2013 22:55:10 +0000

> - they have one or more endpoints, with "two" being a common case (the other end point may not be fixed, as for the DNS example you mention)

They also can have dependencies, the most important one is connectivity
for all active measurements.

> - they are either a single event or repeat

This two points go in the measurement definition : a way to express what
the measurement will do, how and when. Some measurement inject packets
fellowing time series, some open data streams, etc. You will need a
deeply flexible syntax to cover all the needs. 

The other point I really want to talk about, is the runtime bias : you
express the measurement definition in whatever language you choose and
your measurement need a very precise injection process. When the agent
will execute the measurement, you'll have to check that all packets
seen on the wire have : 
- been injected; 
- followed the wished time series to a tolerated unprecision. 
otherwise, you'll have to return an error code to the controller
with the result (keeping the biased results is important also).   

> - they generate results (tuples)

yep (tuples or not tuples) 

> I suspect that this functionality can be expressed in XML, JSON or even ASN.1. I wager that the latter will be the less likely outcome.

the exchange protocol and data format is a detail at the moment. The
logical components matter most at the moment :)