Subject: [ippm] Advanced Stream and Sampling Framework for IPPM

From: Joachim Fabini <[email-masked]>
Date: Mon, 08 Oct 2012 17:47:38 +0200

Dear group,

Following an initial contact on RFC 6703 (while in draft status) Al 
Morton and me have started to isolate and summarize some shortcomings of 
RFC 2330 with respect to active measurements (metrics, methodologies) in 
reactive and/or time-slotted networks. As it turned out during 
discussions, today's access networks implement optimized on-demand 
capacity allocation, such that several methodology properties (in 
particular continuity and repeatability, but also extensibility) stated 
in RFC 2330, section 6.2, are no longer satified. Our measurement 
results confirm these findings.

We believe that the current status can be improved by defining an 
advanced stream sampling framework. There is an increased interest in 
these aspects (IEEE P802.16.3 project, US FCC, European Commission, 
National Regulators in EU) and such a stream sampling framework can help.

Al has submitted the first version of the document, available at
<https://datatracker.ietf.org/doc/draft-morton-ippm-2330-update/>

Any feedback to the mailing list is warmly welcome. In case you do not 
have access to the referenced publications please contact me by email.

best regards
Joachim

-- 
---------------------------------------
Dr. Joachim Fabini
Institute of Telecommunications
Vienna University of Technology
Favoritenstraße 9/389
A-1040 Vienna, Austria

Tel:    +43 1 58801-38813
Fax:    +43 1 58801-38898
mailto: [email-masked]
---------------------------------------

---

From: Matt Mathis <[email-masked]>
Date: Thu, 11 Oct 2012 19:35:04 -0700

I think this is a really good start.  You might want to add a general
notion of "pre-load" which would be traffic sent before a test to
cause the network to transition into an active or higher load state.
Also perhaps a general notion of "test load bound" which defines the
point at which a test designed to measure low power states might be at
risk of causing a transition into a higher power state.

One other item for 2330: in the list of metric criteria, we failed to
include "actionable" - some of the existing metrics don't provide any
guidance for somebody who is unhappy with the results.   Although
"actionable" might be considered to be a sub item of "meaningful",
some existing metrics don't do so well, even though they have well
defined meanings.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat
privacy and security as matters of life and death, because for some
users, they are.

On Mon, Oct 8, 2012 at 8:47 AM, Joachim Fabini
<[email-masked]> wrote:
> Dear group,
>
> Following an initial contact on RFC 6703 (while in draft status) Al Morton
> and me have started to isolate and summarize some shortcomings of RFC 2330
> with respect to active measurements (metrics, methodologies) in reactive
> and/or time-slotted networks. As it turned out during discussions, today's
> access networks implement optimized on-demand capacity allocation, such that
> several methodology properties (in particular continuity and repeatability,
> but also extensibility) stated in RFC 2330, section 6.2, are no longer
> satified. Our measurement results confirm these findings.
>
> We believe that the current status can be improved by defining an advanced
> stream sampling framework. There is an increased interest in these aspects
> (IEEE P802.16.3 project, US FCC, European Commission, National Regulators in
> EU) and such a stream sampling framework can help.
>
> Any feedback to the mailing list is warmly welcome. In case you do not have
> access to the referenced publications please contact me by email.
>
> best regards
> Joachim
>
> --
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: Joachim Fabini <[email-masked]>
Date: Fri, 12 Oct 2012 08:38:08 +0200

On 12.10.2012 04:35, Matt Mathis wrote:
> I think this is a really good start.

Thank you.

> You might want to add a general

The new stream and sampling framework is supposed to assess and define 
parameter restrictions which are likely to impact on (or improve) 
measurement repeatability in reactive networks (or other types of 
network which differ from the deterministic wire model which RFC 2330 
silently assumes). Pre-load, a technique commonly used in today's 3G 
network measurements, is definitely one aspect. We have generalized this 
in the draft by the term "recent history" (alternatively precondition). 
There might be other preconditions which influence on results, too. Our 
main aim is to have the framework as general as possible.

> some existing metrics don't do so well, even though they have well

RFC 2330 defines extensibility, which (combined with repeatability) imo 
fits pretty much what you describe as "actionable". The solution which 
we are currently preferring is to find/define a set of metrics which can 
assess a) repeatability and b) extensibility of a measurement 
methodology. E.g., using measurement samples of one or two subsequent 
measurement scenarios (metric & methodology) as input, these metrics 
should provide hints if 1) there is a likelihood of repeatability and 2) 
we can extend measurement results to other scenarios/parameters - in the 
sense of metric linearity for a specific methodology.
Perhaps even more important and easier to realize is a metric output 
that properties 1) and 2) are NOT fulfilled.

We could circumscribe such meta-metrics as QoMM, "quality of metric and 
methodology". This is basically what the framework is about, assess the 
quality of measurements and propose ways how to improve it.

regards
Joachim

---

From: <[email-masked]>
Date: Fri, 12 Oct 2012 09:10:48 +0200

Joachim, Matt,

I've copied in a section of RFC2330 below, which deals with metric
design and result interpretation in the case of dynamic network
behaviour.

>From what I've read of Matt, his interpretion of Joachims work is
to find load or usage conditions, under which network behaviour
changes. The metric then is a load (and may be a RTT or other
parameters)? One could of course characterise the
perfromance by delay, loss and so on before that behaviour shift
and afterwards. But the aim of these new metrics is to change
network behaviour?

I'm not an expert on wireless access networks. To me, repeatability
of metric measurements is a very important aspect. IPPM should pick
up this work, if the number of parameters having an impact on the
metric is small enough to enable useful results. If however temporal
background load, local weather conditions or other factors which
aren't captured by the metric, have a signigicant impact, I wonder
about the interpretation of the results.

Regards, Rüdiger

---------Extract of RFC2330----------------------------

   Note further that, in practice, it may not be practical to know (or
   be able to quantify) the conditions relevant to a measurement at a
   given time.  For example, since the instantaneous load (in packets to
   be served) at a given router in a high-speed wide-area network can
   vary widely over relatively brief periods and will be very hard for
   an external observer to quantify, various statistics of a given
   metric may be more repeatable, or may better exhibit continuity.  In
   that case those particular statistics should be specified when the
   metric is specified.

   Finally, some measurement methodologies may be 'conservative' in the
   sense that the act of measurement does not modify, or only slightly
   modifies, the value of the performance metric the methodology
   attempts to measure.  {Comment: for example, in a wide-are high-speed
   network under modest load, a test using several small 'ping' packets
   to measure delay would likely not interfere (much) with the delay
   properties of that network as observed by others.  The corresponding
   statement about tests using a large flow to measure flow capacity
   would likely fail.}

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joachim Fabini
Sent: Friday, October 12, 2012 8:38 AM
To: Matt Mathis
Cc: Al Morton; [email-masked]
Subject: Re: [ippm] Advanced Stream and Sampling Framework for IPPM

Thank you.

regards
Joachim

---

From: Al Morton <[email-masked]>
Date: Fri, 12 Oct 2012 09:43:25 -0400

Rüdiger and Matt,

Thanks for your review and comments so far.

I'll just add two points below (one to help define

"actionable" in the way that I think of it).

regards,

Al

At 02:38 AM 10/12/2012, Joachim Fabini wrote:

On 12.10.2012 04:35, Matt Mathis
wrote:

I think this is a really good
start.

Thank you.
+1, we've had this in-progress for months, great
to get feedback now.

...

One other item for 2330: in the
list of metric criteria, we failed to

include "actionable" - some of the existing metrics don't
provide any

guidance for somebody who is unhappy with the results.
Although

"actionable" might be considered to be a sub item of
"meaningful",

defined meanings.

RFC 2330 defines extensibility, which (combined with repeatability) imo
fits pretty much what you describe as "actionable". The
solution which we are currently preferring is to find/define a set of
metrics which can assess a) repeatability and b) extensibility of a
measurement methodology. E.g., using measurement samples of one or two
subsequent measurement scenarios (metric & methodology) as input,
these metrics should provide hints if 1) there is a likelihood of
repeatability and 2) we can extend measurement results to other
scenarios/parameters - in the sense of metric linearity for a specific
methodology.

Perhaps even more important and easier to realize is a metric output that
properties 1) and 2) are NOT fulfilled.

We (network operators) always have "actionable" as a goal,

meaning that the measurement results combined with some
interpretation

should lead the design or maintenance forces to their next steps

(to improve performance or repair a problem). We move a long

way toward the goal by selecting the right metrics from the
start.

All of our/user packet streams are influenced by a growing number of

reactive network components, maintaining and changing state.

It's matter-of-fact for most user flows.  In reactive nets we can
use

discovery, characterization, and with on-going measurements,
expectation,

to help determine if there is a fault or other undesirable condition

and what network elements may be involved. If the measurements
really

help to narrow-down the problem space, they are considered
"actionable".

So, I agree this is a useful consideration to add-in/update RFC
2330.

As Joachim mentioned, this expansion of the stream framework is

intended to address various forms of reactive networks, and
certainly

includes wired networks today. The concept of pre-load (as Matt called
it)

is important to any test path that includes a firewall. This

aspect is an important consideration in the LMAP requirements:

http://tools.ietf.org/html/draft-schulzrinne-lmap-requirements-00

and probably also  the project described in IEEE 802.16
Liaison:

https://datatracker.ietf.org/liaison/1195/

There is almost certainly some overlap between our draft and

the intent of latter project.

---

From: <[email-masked]>
Date: Mon, 22 Oct 2012 12:38:37 +0200

Al, Joachim and Matt,

it just came to my mind, whether we need to narrow down "reactive network behaviour" or whether
it stays undefined.

reactive could a network behave:
- based on the history of and current local load conditions
- on user behaviour
  # in terms of load in general
  # in terms of visited online content
  # in terms of payload contents

To be more precise, active bandwidth throtteling or flow suppression may be regarded as "reactive
network behaviour" too.

Regards,

Rüdiger

________________________________
From: Al Morton [mailto:[email-masked]]
Sent: Friday, October 12, 2012 3:43 PM
To: Joachim Fabini; Matt Mathis; Geib, Rüdiger
Cc: [email-masked]
Subject: Re: [ippm] Advanced Stream and Sampling Framework for IPPM

Rüdiger and Matt,

regards,
Al

Thank you.
+1, we've had this in-progress for months, great to get feedback now.

...

RFC 2330 defines extensibility, which (combined with repeatability) imo fits pretty much what you describe as "actionable". The solution which we are currently preferring is to find/define a set of metrics which can assess a) repeatability and b) extensibility of a measurement methodology. E.g., using measurement samples of one or two subsequent measurement scenarios (metric & methodology) as input, these metrics should provide hints if 1) there is a likelihood of repeatability and 2) we can extend measurement results to other scenarios/parameters - in the sense of metric linearity for a specific methodology.
Perhaps even more important and easier to realize is a metric output that properties 1) and 2) are NOT fulfilled.

We (network operators) always have "actionable" as a goal,
meaning that the measurement results combined with some interpretation
should lead the design or maintenance forces to their next steps
(to improve performance or repair a problem). We move a long
way toward the goal by selecting the right metrics from the start.

All of our/user packet streams are influenced by a growing number of
reactive network components, maintaining and changing state.
It's matter-of-fact for most user flows.  In reactive nets we can use
discovery, characterization, and with on-going measurements, expectation,
to help determine if there is a fault or other undesirable condition
and what network elements may be involved. If the measurements really
help to narrow-down the problem space, they are considered "actionable".

So, I agree this is a useful consideration to add-in/update RFC 2330.

As Joachim mentioned, this expansion of the stream framework is
intended to address various forms of reactive networks, and certainly
includes wired networks today. The concept of pre-load (as Matt called it)
is important to any test path that includes a firewall. This
aspect is an important consideration in the LMAP requirements:
http://tools.ietf.org/html/draft-schulzrinne-lmap-requirements-00
and probably also  the project described in IEEE 802.16 Liaison:
https://datatracker.ietf.org/liaison/1195/
There is almost certainly some overlap between our draft and
the intent of latter project.

---

From: Matt Mathis <[email-masked]>
Date: Mon, 22 Oct 2012 09:08:01 -0700

Good question.  I hadn't considered poor performance as a possible
penalty for "bad surfing behavior", but it certainly is.   I think we
can keep the longer timescale issues in mind, but the real point is to
improve measurement repeatability at shorter timescales e.g. session
history.

A slightly different question is over what timescale do things shift?
  If I transition from idle to heavy use, how long does it take to
transition into a "power user" policy bin?  (Put another way, what is
the averaging timescale?)

Thanks,
--MM--

On Mon, Oct 22, 2012 at 3:38 AM,  <[email-masked]> wrote:
> RFC 2330 defines extensibility, which (combined with repeatability) imo fits
> pretty much what you describe as "actionable". The solution which we are
> currently preferring is to find/define a set of metrics which can assess a)
> repeatability and b) extensibility of a measurement methodology. E.g., using
> measurement samples of one or two subsequent measurement scenarios (metric &
> methodology) as input, these metrics should provide hints if 1) there is a
> likelihood of repeatability and 2) we can extend measurement results to
> other scenarios/parameters - in the sense of metric linearity for a specific
> methodology.

---

From: Joachim Fabini <[email-masked]>
Date: Mon, 22 Oct 2012 23:02:44 +0200

Rüdiger,

thank you, very good point. Straight ahead I'd prefer to include an 
exact definition of what we mean by "reactive network behavior" to 
delimit the document's scope - unless this turns out to be a huge 
limiting factor (with respect to framework generality or applicability) 
in the long run.
However I fully agree with you that it is challenging to draw a clear 
line between reactive and "normal" behavior. And perhaps we can isolate 
other properties which make networks behave different from wires with 
respect to measurements? Common examples for reactive network behavior 
include on-demand capacity allocation or flow capacity which depends on 
some layer 2 or 1 flow state (history). A delicate topic in this context 
is (content-specific?) traffic shaping which might lead to reactive 
behavior in some cases. However, radio channel modifications or content 
compression in a wireless network segment might also generate effects 
which are reactive by nature.

Trying to not lose focus (and still doing so), "reactive behavior" is 
imho very tightly coupled to measurement repeatability - which, as you 
have pointed out, in its philosophical dimension is difficult to handle 
for real measurements. My opinion is that today's networks (systems) are 
too complex to guarantee repeatability.

So perhaps it is worth to consider replacing this purely theoretical 
concept of RFC2330-repeatability by a weakened term, e.g., 
"p-repeatability", which is conditioned by a set of internal and 
external parameters (load, history, etc.). In this context RFC2330's 
Packet-Type-P would be replaced by a Stream-Type-P configuring specific 
stream constraints (referencing a set of Packet-Type-P's with their 
temporal ordering) or a Measurement-Type-P, adding some explicit 
measurement parameter settings to stream definitions.

Becoming philosophical, too, we could use systems theory (networks are 
complex systems) to differentiate between system-internal and 
system-external measurement parameters (and system/flow state) and 
integrate the relevant ones into the Type-P-stream definition. The huge 
benefit of such a solution is to make explicit the amount of uncertainty 
which is associated with a measurement sample set.

What I perceive as highly useful in practice is a set of explicit 
parameters (internal stream constraints, methodologies, external 
parameters), which can help measurement engineers to obtain repeatable 
measurements (or, at least, a high likelihood of repeatability) for 
specific metrics and link types. The new framework could support this, 
e.g., by also defining metrics and methodologies to test two sample sets 
on repeatability for specific measurement parameters. Going even 
further, we could try to develop meta-metrics to test sample sets on 
other properties like extensibility.

So I fully agree with you that the term "reactive behavior" must be 
clarified. But my feeling is that one main target we should aim at is to 
act on "repeatability" as of RFC 2330. Perhaps we can collect a useful 
list of network link properties (like "reactive") and propose metrics 
and methodologies to test sample sets on these properties?

Any feedback is warmly welcome!

Thanks again,
regards
Joachim

Am 22.10.2012 12:38, schrieb [email-masked]:
> ------------------------------------------------------------------------
> *From:* Al Morton [mailto:[email-masked]]
> *Sent:* Friday, October 12, 2012 3:43 PM
> *To:* Joachim Fabini; Matt Mathis; Geib, Rüdiger
> *Cc:* [email-masked]
> *Subject:* Re: [ippm] Advanced Stream and Sampling Framework for IPPM
>
> Rüdiger and Matt,
>
>> RFC 2330 defines extensibility, which (combined with repeatability)
>> imo fits pretty much what you describe as "actionable". The solution
>> which we are currently preferring is to find/define a set of metrics
>> which can assess a) repeatability and b) extensibility of a

---

From: Joachim Fabini <[email-masked]>
Date: Mon, 22 Oct 2012 23:42:41 +0200

Matt,

if I get you right, this relates somehow to the point-of-view topic (RFC 
6703). Developing this scenario: consider a VoIP media connection over 
mobile networks. When measuring high end-to-end delay, VoIP clients 
might infer on network overload and re-negotiate their codec to reduce 
bandwidth. In networks with on-demand capacity allocation this is the 
worst decision, as it likely reduces allocated capacity and increases 
delay (and voice quality) even more. Again, it's the question of user 
view and perception vs. system view, optimization and configuration.

Concerning your specific question about transition duration: in some 
evaluated HSPA networks, depending on the specific state and 
precondition, it takes some hundreds of miliseconds (it's basically a 
matter of one or several round-trips). As an example, we have done some 
measurements with respect to VoIP SIP signaling: using 5-10 seconds 
inter-arrival times, effective HSPA round-trip delay (400-800 bytes 
payload) increases from an optimum of 40-50ms (at much higher data rate) 
to 800ms and stays there.

thanks,
regards
Joachim

---

From: Matt Mathis <[email-masked]>
Date: Mon, 22 Oct 2012 15:39:52 -0700

I was thinking about two cases very different than yours...

Probably important to include in scope: Systems based on RFC 6057,
which in a general sense enforces average fair share over a longer
time interval.  It has the property that for "small" users, the
network looks under loaded (user get "full rate" for a short time),
then as users transition to a "High Consumption State" they get 1/N of
the capacity allocated to the big users.   My impression is that the
thresholds are units of Mbytes and tens of seconds.  Note that in this
case the performance shift is in the opposite direction than it is for
HSPA.

For both HSPA and RFC 6057 it is easy to test the "high utilization"
state if you know how hard you have to pre-load the network to get it
there.   A harder problem is measuring the low utilization state, when
it may not be easy to tell when you trigger the state transition.   We
really should to be able to address this case too.  (e.g. Must perform
measurement in under X bytes/packets/etc)

Note that it might also make sense to run 6057 over HSPA....

Probably not in scope:  With non-encrypted traffic to some geo
locations, typing certain queries into search engines causes
"interesting" changes in network properties.
http://www.nytimes.com/2012/06/02/world/asia/google-to-alert-users-to-chinese-censorship.html

But the point remains: *arbitrary* events in the past can influence
measured performance over a whole range of timescales.  We want to
manage a bunch of them as best we can, and to think carefully about
picking the largest possible tractable scope.

(And my apologies to our Chinese subscribers, who probably will not
see this message).

Thanks,
--MM--

On Mon, Oct 22, 2012 at 2:42 PM, Joachim Fabini
<[email-masked]> wrote:
> mobile networks. When measuring high end-to-end delay, VoIP clients might
> infer on network overload and re-negotiate their codec to reduce bandwidth.
> In networks with on-demand capacity allocation this is the worst decision,
> as it likely reduces allocated capacity and increases delay (and voice
> quality) even more. Again, it's the question of user view and perception vs.
> system view, optimization and configuration.
>
> evaluated HSPA networks, depending on the specific state and precondition,
> it takes some hundreds of miliseconds (it's basically a matter of one or
> several round-trips). As an example, we have done some measurements with
> respect to VoIP SIP signaling: using 5-10 seconds inter-arrival times,
> effective HSPA round-trip delay (400-800 bytes payload) increases from an
> optimum of 40-50ms (at much higher data rate) to 800ms and stays there.
>
> thanks,
> regards
> Joachim
>
>
>
>>> To be more precise, active bandwidth throtteling or flow suppression may
>>> be
>>> regarded as "reactive
>>> network behaviour" too.
>>>
>>> Regards,
>>>
>>> Rüdiger
>>>
>>> fits

---

From: <[email-masked]>
Date: Tue, 23 Oct 2012 08:59:54 +0200

Matt,

whether or not the behaviour I've heard of is caused by a power user,
I don't know. I think, families having access to the Internet may
be regarded as power users, if consumption of streaming content
grows. And that's a present trend for sure.

I quote from a blog:

Reactive part 1:
Since a few days, the customer of a cable provider sees a packet-burst
and then nothing for a second (TCP dump). The author claims neither
to be a file sharer, nor to be consuming more than the daily maximum
byte budget given by the contract.

Reactive part 2:
The author plays around, once hitting a URL containing "/speedtest/".
It's a 100 MB file download. While this runs, a different parallel
download increases from 300 kB/s to 1,2 MB/s. Once the speedtest
finishes, the parallel download is back to 300 kB/s again.

The author further claims to have seen reports from other
companies behaving similar.

End of quote.

I'm not interested in metrics to detect the above reactive behaviour.
I'm interested in metrics capturing the standard performance as
perceived by end-systems. That seems to be a reasonable way to
compare things. A consequence, if any, may be to avoid
easy detection of measurements.

Regards,

Rüdiger

-----Original Message-----
From: Matt Mathis [mailto:[email-masked]]

Thanks,
--MM--

---

From: Matt Mathis <[email-masked]>
Date: Mon, 29 Oct 2012 18:53:00 -0700

Can you provide more information about your Item #2?    I am aware of
a small number of examples of technical problems (i.e. bugs)  that
causes this sort of symptom, but it could also be premeditated white
listing.

(Consider device that tries to batch packets but has an inappropriate
dwell timer when there are not enough for a full batch.   Adding a
little more traffic can cause a phase transition from one (short)
batch on each timer tick to a more continuous streaming mode.   I bet
there are some DOCIS failure modes that resemble this as well. )

Yes your point is true: more stealthy active measurement would be
good, as would ongoing passive measurement of real content delivery.
This is why I work so hard on RFC 4898.

Thanks,
--MM--

---

From: <[email-masked]>
Date: Tue, 30 Oct 2012 08:46:52 +0100

Matt,

the information was translated from a German Blog (the author is
not related to me personally). There's little to add:

The blog starts with "since a few days my download rates broke down".

The author performed the speedtest with a speed limit on it (10 kB/s
and 100 kB/s), but the download bandwidth increase of the parallel
download was less significant then.

The blog is about one cable provider. A second one is mentioned
and a third company, which is a city provider. I don't know whether
the latter two behave exactly like the former. I also don't know
whether the city provider runs DOCSIS.

All in all, the blog has just 20 lines.

Regards, Rüdiger

-----Original Message-----
From: Matt Mathis [mailto:[email-masked]]
Sent: Tuesday, October 30, 2012 2:53 AM
To: Geib, Rüdiger
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [ippm] Advanced Stream and Sampling Framework for IPPM

Thanks,
--MM--