Subject: Re: [lmap] measurement traffic load [was: Re: Proposed LMAP Charter]

From: Jérôme Benoit <[email-masked]>
Date: Mon, 4 Feb 2013 22:16:51 +0100

On Tue, 29 Jan 2013 09:40:07 +0800
"Guangqing Deng" <[email-masked]> wrote:

> Thanks for Mirja's point. Usually, the bottleneck links in internet are those cross-ISP links. If the collector and the MA are in different ISPs, the measurement traffic from MAs to collectors will bring huge pressure to the already busy cross-ISP links. So how to reduce the cross-ISP measurement traffic while not reducing the measurement accuracy may be a challenging problem that needs more consideration. 

Indeed. 

First, you can't have a stateful protocol to achieve this goal (stateful
at layer 3, OK, not at layer 7).
Second, you can't have a binary protocol to structure the measurements
results, it generate too much packets and compression in the payload
will then be a pain in the ass.
Third, I do not know what is the status of the logical component
responsible to synchronize every single MA in the measurement system,
but I still think it's a must have and the synchronization algo and
proto must also be thought in a way to not overload the network AND the
hosting infrastructure resources. 

-- 
Jérôme Benoit aka fraggle
La Météo du Net - http://grenouille.com
OpenPGP Key ID : 9FE9161D
Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D

---

From: "Guangqing Deng" <[email-masked]>
Date: Tue, 5 Feb 2013 10:19:02 +0800

Hi,jerome, comments are inline below.

Guangqing Deng
CNNIC 

From: Jérôme Benoit
Date: 2013-02-05 05:16
To: lmap
Subject: Re: [lmap] measurement traffic load [was: Re: Proposed LMAP Charter]
On Tue, 29 Jan 2013 09:40:07 +0800

Indeed. 

The packet sent by one MA can be generally divided into two kinds: one is those sent to the measurement server for triggering a measurement; the other is those sent to data collector server to report the measurement result. Here, the signaling traffic (between MA and measurement controller, for instance) is omitted. The former kind of packet usually cannot be compressed; while the latter kind of packet may be compressed before sending it to the data collector server. Through actual measurement, it is found that Internet traffic volume from 8 p.m. to 11 p.m. everyday is much larger than that from 1 a.m. to 5 a.m. So another way to reduce the effect on network infrastructure is to let the MA send measurement results when the network is not so busy. 

Usually, the synchronization between the MA and the measurement server may depend on the chosen performance metric and RFCs in IPPM work group may be helpful. While for the synchronization between the MA and the measurement controller (or the data collector), it seem that the “heartbeat” mechanism can be taken into consideration. 

_______________________________________________
lmap mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/lmap

---

From: Brian Trammell <[email-masked]>
Date: Tue, 5 Feb 2013 10:24:11 +0100

Hi, Jerome, all,

> at layer 3, OK, not at layer 7).

Clearly. Even just the normal connectivity churn of so many machines under someone else's physical control rules this out.

> will then be a pain in the ass.

I'm not sure I follow this point.

> hosting infrastructure resources. 

There's two points to synchronization: making sure each MA knows what to do (control sync), and making sure it does that at the right time (clock sync). As long as your measurement plan doesn't (1) change often or (2) involve receiving results in anything like real time, you can split your synchronization in two. Let something like NTP sync clocks (not good enough for lining up multiple observations of the same packet on a high-speed link on its own, but it should do for scheduling of broadband measurements). Since what we're basically building is a legitimate volunteer botnet, we can probably take some inspiration from successful botnet C&C: our goal is to reduce measurement control load to increase scalability, while the goal of a botnet is to evade detection. Both involve treading lightly on the network. So control sync happens via pull from a known CDN or redirect thereto, P2P flooding, etc., etc.

Best regards,

Brian

---

From: Jérôme Benoit <[email-masked]>
Date: Tue, 5 Feb 2013 13:24:39 +0100

On Tue, 5 Feb 2013 10:24:11 +0100

> I'm not sure I follow this point.

For information exchange between each logical components in the system.
If you choose binary protocol that structure de data exchanged, you
will have more packets than say RESTful protocol. 

> There's two points to synchronization: making sure each MA knows what to do (control sync), 

It's the hard point that need to trigger a high convergence time in the
system to avoid out of sync measurement run with overloading the
network. Really tricky to design.  

> and making sure it does that at the right time (clock sync). 

That's a prerequisite for the former.

---

From: Jérôme Benoit <[email-masked]>
Date: Tue, 5 Feb 2013 13:40:57 +0100

On Tue, 5 Feb 2013 10:19:02 +0800
"Guangqing Deng" <[email-masked]> wrote:

>
> The packet sent by one MA can be generally divided into two kinds: one is those sent to the measurement server for triggering a measurement; the other is those sent to data collector server to report the measurement result. 

Two main kinds I think : the probing traffic and all the information
exchange inside the measurement system : control traffic, collector
traffic and so on. 

My main interest is for now the information exchange inside the
measurement system.    

>Here, the signaling traffic (between MA and measurement controller, for instance) is omitted. The former kind of packet usually cannot be compressed; while the latter kind of packet may be compressed before sending it to the data collector server. Through actual measurement, it is found that Internet traffic volume from 8 p.m. to 11 p.m. everyday is much larger than that from 1 a.m. to 5 a.m. So another way to reduce the effect on network infrastructure is to let the MA send measurement results when the network is not so busy. 

Agreed. The MA will need to buffer them.  

> Usually, the synchronization between the MA and the measurement server may depend on the chosen performance metric and RFCs in IPPM work group may be helpful. While for the synchronization between…

Hearbeat is OK as a mechanism for signalling states of the measurement
agents. What about capabilities ?

---

From: Brian Trammell <[email-masked]>
Date: Tue, 5 Feb 2013 14:34:19 +0100

Hi, Jerome, all,

> will have more packets than say RESTful protocol. 

Okay, sorry, I understand what you're saying, but I don't see any correlation between data encoding and packet count. Compressibility, similarly, is a function of the entropy in the data being represented first, and is only secondarily related to the encoding and framing, unless the encoding and framing are gratuitously inefficient.

> network. Really tricky to design.  

I'm not understanding your point here... I think we might be making different assumptions about the arrangement of the MAs...

> That's a prerequisite for the former. 

Well, if you want it to work, yes, it helps if the clocks are synced. :) But my point was that clock synchronization and control distribution are separate functions of the control protocol(s) -- and further, that clock synchronization with the accuracy required by LMAP would seem already to be solved by NTP.

The assumption I'm making here is that each measurement specification has a time at which it should run, and a time at which it should report results, as part of the specification, and that the MA compares this with its system clock (which should be synchronized with the system clocks of other MAs, of course) to decide when to do what. Without a common time reference or an allowance for the time it takes a measurement specification to make it to the MA, yes, synchronization is a whole lot harder.

Best regards,

Brian

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Tue, 5 Feb 2013 14:41:01 +0000

> Well, if you want it to work, yes, it helps if the clocks are synced. :) But my
> point was that clock synchronization and control distribution are separate
> functions of the control protocol(s) -- and further, that clock synchronization
> with the accuracy required by LMAP would seem already to be solved by
> NTP.

I'm really hoping that this is a consensus conclusion that we can reach. This assumption (that NTP would be sufficient for the needs of LMAP) would make the overall architecture much less complex than if we were to try to introduce more precise clock synchronization. Lower complexity is a wonderful goal to strive for. I'm very much in favor of aiming our sights at "good enough" rather than "100% perfect". I suspect that the end customers will be much better served by an architecture that is "good enough", rather than one that is over-engineered to achieve a level of precision that those end customers aren't asking for.
Barbara

---

From: "Bugenhagen, Michael K" <[email-masked]>
Date: Tue, 5 Feb 2013 15:32:53 +0000

I think we should consider test validation criteria as part of the output of the test..

1) Did it have good "time and date".. 
2) Does the tester "know" the provisioned speed - (I don't mean assumed...)
3) Was the test conducted successfully 
	- No customer traffic interrupted it
	- No abnormal test termination occurred
	- was the there any abnormal conditions (CPU high use, ...) that could have impacted the test

These are far too often left out of pre-test checks.

Regards,
Mike

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Tuesday, February 05, 2013 7:34 AM
To: Jérôme Benoit
Cc: [email-masked]
Subject: Re: [lmap] measurement traffic load [was: Re: Proposed LMAP Charter]

Hi, Jerome, all,

On 5 Feb 2013, at 13:24 , Jérôme Benoit wrote:

>>> Second, you can't have a binary protocol to structure the 
>>> measurements results, it generate too much packets and compression 
>>> in the payload will then be a pain in the ass.
>> 

>>> responsible to synchronize every single MA in the measurement 
>>> system, but I still think it's a must have and the synchronization 
>>> algo and proto must also be thought in a way to not overload the 
>>> network AND the hosting infrastructure resources.
>> 
> It's the hard point that need to trigger a high convergence time in 
> the system to avoid out of sync measurement run with overloading the 

Best regards,

Brian

> La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key 
> fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D

---

From: Jérôme Benoit <[email-masked]>
Date: Tue, 5 Feb 2013 23:28:03 +0100

On Tue, 5 Feb 2013 14:34:19 +0100

> Okay, sorry, I understand what you're saying, but I don't see any correlation between data encoding and packet count. Compressibility, similarly, is a function of the entropy in the data being repr…

Ok, but can we just state that all network procotols inside the
measurement system must really take care of efficiency regarding
number keepalived and state-full of connexion, data size exchanged to
begin with ?  

> > system to avoid out of sync measurement run with overloading the
						    ^out
> I'm not understanding your point here... I think we might be making different assumptions about the arrangement of the MAs...

The time synchronization is one part. I'm talking about another type of
synchronisation : the one that permit to run measurement with
topological information, the one that permit to run a measurement only
if more than say N MAs are online and support the feature X. I do have
that kind of measurements scenarios. It's what I call measurement
system coherence. Time synchronization is a good feature, really. But
certain measurements do not care that much about time synchronization
in the whole measurement system (they care much about local clock
stability) but they care about the number N of MAs online behind the
same network link and the features on the N MAs online. 

> Well, if you want it to work, yes, it helps if the clocks are synced. :) But my point was that clock synchronization and control distribution are separate functions of the control protocol(s) -- an…

It will depends ... some scenario require ms accuracy (or even ns, but
not in real world :)), some do not care. 

> The assumption I'm making here is that each measurement specification has a time at which it should run, and a time at which it should report results, as part of the specification, and that the MA…

I hope this mail make my point clearer : I want to talk about a global
coherence in the measurement system. Time synchronization is one type
of coherence. I think the LMAP WG should care about other type of
coherency such as : 

- Keeping track of MA capabilities
- Keeping track of MA states
- Keeping track of MA localisation - geo and topo - 
- Keeping track of MA time synchronization 

Cheers.

---

From: Brian Trammell <[email-masked]>
Date: Wed, 6 Feb 2013 09:02:16 +0100

Hi, Jerome,

> number keepalived and state-full of connexion, data size exchanged to

Absolutely. And the point that efficiency isn't just about how many bytes sent, but in how many packets, and when, is well-taken, too.

> system coherence.

Ahhhhh. Got it. Yeah, that's a hard problem.

> Time synchronization is a good feature, really. But

Indeed. ms level should be good enough if you're using time-based coordination.

> - Keeping track of MA states

For these first two, a RESTful protocol seems a natural fit. 

> - Keeping track of MA localisation - geo and topo - 

I tend to see localization as a capability as well: "I can see/send packets (on this AS/subnet/behind or through this piece of infrastructure/in this geographic region/etc.)" Geoloc gets into privacy concerns, of course. Fortunately, we've already got a working group for that, too. :)

> - Keeping track of MA time synchronization 

Again, I think applying NTP here is probably advisable.

But in general, I agree -- and thanks for clearing up the coherency issue for me, it makes sense now.

Cheers,

Brian