Subject: Re: [ippm] Advanced Stream and Sampling Framework for IPPM

From: <[email-masked]>
Date: Thu, 11 Oct 2012 09:53:01 +0200

Joachim,

after reading your and Al's draft, it is not quite clear to me,
what's the aim of the document as related to IPPM.

RFC 2330 defines the following metric definition criteria:
 +    The metrics must be concrete and well-defined,
 +    A methodology for a metric should have the property that it is
      repeatable: if the methodology is used multiple times under
      identical conditions, the same measurements should result in the
      same measurements.
 +    The metrics must exhibit no bias for IP clouds implemented with
      identical technology,
 +    The metrics must exhibit understood and fair bias for IP clouds
      implemented with non-identical technology,
 +    The metrics must be useful to users and providers in understanding
      the performance they experience or provide,
 +    The metrics must avoid inducing artificial performance goals.

I'd expect repeatability of measurements under otherwise identical
conditions. If this is not given, measurements don't make sense.

A measurement obviously captures performance only for the flow tested. Your
proposal seems to be to add more randomisations in measurement flows. So
far, IPPM recommends randomised inter packet gaps for some metrics.

Does your document propose to add more randomisations, like eg. packet
size or payload? You mention addresses as another parameter impacting
measurement results. Further, the access to the network has an impact
on results.

Approaches to characterise expectable performance could be to apply
different type-p streams and change chracteristics of them step by step.

Or one randomises type-p streams more heavily (randomised inter packet
gaps, packet sizes, payload, addresses, access type and so on).

The latter will at some stage result in repeatable results only over
sufficiently long time intervals (that depends on the number of
randomised parameters).

Could you clarify how you propose to acount for variability of the
Type-P properties having an impact on performance?

Regards,

Rüdiger

---

From: <[email-masked]>
Date: Fri, 12 Oct 2012 12:14:10 +0200

Joachim,

thanks, I got most parts of your intents right, I guess.

I'm having an issue only with the wording may be. Under
otherwise identical conditions, a result must be reproducible.

No matter which preload etc. there is, this must hold. If the
same flow sees different metric results, it only indicates
that conditions have not been identical. It's a pholosophical
discussion, which events form part of "condition", one of
which may be load history.

What is new is indeed, that you propose to influence network
conditions by a test stream. This is justified, if the measurement
flow is not deviating significantly from any ordinary customer
network flow, I think.

Allright, I'll drop out of discussion for some time now.

Regards,

Rüdiger

-----Original Message-----
From: Joachim Fabini [mailto:[email-masked]]
Sent: Friday, October 12, 2012 11:40 AM
To: Geib, Rüdiger
Cc: [email-masked]
Subject: Re: [ippm] Advanced Stream and Sampling Framework for IPPM

Ruediger,

thank you for your comments. The point is that RFC 2330 does neither
consider reactive networks nor preconditions (link history). When it was
written we had deterministic networks (wires), but with on-demand
ressource allocation performance of today's links depends to a large
extent on flow state and history which is typically transparent to the
user, being maintained at layers below IP.
Router queuing is a topic in RFC2330 - but its impact on performance is
minor compared to what current access links do - see below. Main
difference is imo that user-generated traffic typically does not
influence on router queueing strategies. But it does exhibit a huge bias
on measurements in reactive networks, being main reason for huge
performance variations.

Please find some more detailed responses to your topics inline.

> conditions. If this is not given, measurements don't make sense.

That's the statement of RFC 2330 and from a theory point of view I fully
agree with you. But defining repeatability imo boils down to defining
"identical conditions". My believe after some years in large-scale
mobile network measurements is that in some networks types it is
straight-forward impossible to create identical (pre)conditions.
In particular this applies to measurements in reactive (e.g., mobile 3G)
networks where the impact of external factors (radio provisioning, users
in a cell, instantaneous own traffic and traffic history) outweighs most
of the factors mentioned in RFC 2330 by orders of magnitude. E.g.,
depending exclusively on recent load history, the identical measurement
packet can have a more than 100-times variation in one-way delay (20ms
to 5s).

 From a theoretical point of view we could therefore conclude RFC 2330:
measurements in today's 3G or other wireless networks do _never_ make
sense because we can never reproduce identical conditions (except in a
fully isolated environment). Don't think that the LAMP group is happy
with this finding.

Al and me have started work on this draft considering that it is useful
to relax instead the "identical" condition of RFC 2330 which we can not
satisfy, anyhow.
Our feeling is that it is more important for black-box measurements to
a) determine whether the network under test behaves deterministic or
reactive, b) propose metrics for testing on irregularities which Matt
has questioned (e.g., multi-modal sample distributions are a first
indication of potential reactive network behavior) and c) provide
guidelines and constraints that can help to improve measurement
representativeness and repeatibility.
I think that it is also a highly valuable result to confirm that results
for specific measurement methodologies on a specific link do _not_
satisfy particular RFC 2330 properties and applications should be
prepared to face non-deterministic link behavior.

> A measurement obviously captures performance only for the flow tested.

Yes. But nevertheless most measurements (consider LAMP) intend to obtain
"representative" measurements - whatever this means. My understanding of
LAMP is that they want representative measurements with respect to i)
repeatability (from _user_ perspective this means: comparable results
when a user maintains the parameters constant which he can influence on,
e.g., measures the same flow at the same location using identical
devices), ii) extensibility (e.g.: can (or can't) the user infer from
his existing measurement results on the performance for a packet size
which has not been tested).

 > Your

That's not really a proposal. In my measurements I have used orthogonal
components of randomization in an attempt to capture a broadest
possible, representative image of a network link's behavior. It's the
point where we can start from in restricting parameters to increase
determinism.

> measurement results. Further, the access to the network has an impact

More randomisation might be useful to cover a wider range of possible
network link behavior. However, the document is rather about defining
stream constraints, reasoning about the meaningfulness of measurement
results and discussing parameters which can influence on the result.

Addresses have been mentioned in the draft to emphasize that we must be
prepared to handle technology changes beyond what's covered in RFC 2330,
which are transparent to the measurements. While in most cases such an
access technology change should be detectable through IP adresss changes
(case in which we could revert to distinct points of
attachment/interfaces as discussed by RFC 2330), it is theoretically
possible for a mobile operator to provide the same IP address after a
device has switched from 2G to 3G or after it has changed the cell. Such
a handover is transparent to the measurement process and significant
effort is required to detect it (AT modem commands typically can help).

> Type-P properties having an impact on performance?

For giving people focused on core network measurements an impression of
what orders of magnitude we are talking about, please have a look at the
attached x-y scatter plot which depicts HSPA uplink for a continuous
35-hour end-to-end measurement session: same location, same device, same
session. (Just a kind of worst-case example, don't want to restrict
discussion to mobile networks, although I consider the challenges in
mobile networks to be higher than in wired ones). The measurement uses a
stream based on two independent, uniform distributed random variables,
one for delay (100ms-5000ms) and one for payload size (64 bytes - 1400
bytes): a random-sized packet is sent at random send time. Any dot in
the figure corresponds to one measurement sample (50K in total).
Depicted are 80% of all samples, the CDF reveals that 20% have delays
higher than 200ms.
To test repeatability we pre-compute and store the so-called measurement
scenario in advance, i.e., we are able to measure the identical data
stream again at a later point in time.

The figure confirms that the link/network uses _massive_ flow state at
lower layers. The proposed framework should give an overview and
guidelines on how to tailor various stream parameters (which, as you
propose, we plan to constrain incrementally). And if it is not possible
to constrain sufficient parameters (e.g., because the network operator
deploys a time-of-day depending scheduling) to be repeatable then the
draft should provide metrics which can evaluate the results and warn.
Factors which we have noticed so far to improve measurement determinism
and repeatability are, e.g., constant (instantaneous) data rate and
increased data rate. There might be many more for other technologies.

Summarizing: I consider the requirement for "identical" conditions to be
a major limitation of RFC 2330 with respect to real-life
appropriateness. While being theoretically sound (and avoiding many
complex discussions), this a-priori prevents measurement repeatability
or extensibility in shared access networks with potential on-demand
ressource allocation. Taken verbatim, the term "identical" conditions
restricts the applicability of RFC 2330 metrics, methodologies and
measurements to fully isolated, deterministic systems.
Moreover, we could extend this to inter-metric dependency which goes far
beyond what we expect to happen in wired networks: if the network drops
a packet, the instantaneous data rate decreases, possibly resulting in
the scheduler to reduce the capacity allocated to the flow. Therefore
the delay for all subsequent packets my increase substantially.

So we think that it's time to relax the "identical" condition and
provide a framework which guides on meaningful stream parameter
selection and constraints. Moreover, the framework should point out
potential pitfalls and ideally propose tests (metrics) that can assess
the quality of measurement results with respect to metric and
methodology properties stated in RFC 2330.
Finally, development of a stream and sampling framework can help
real-time applications to perform better in reactive networks but, most
important, beneficially supports some activities which are currently in
the focus of attention (LAMP, 802.16.3).

Regards
Joachim

---

From: Joachim Fabini <[email-masked]>
Date: Fri, 12 Oct 2012 12:34:50 +0200

Rüdiger,

full agree. One short note on influence: the ideal measurement scenario 
in reactive networks is to have a real/prototypical application traffic 
and measure its performance on a specific network. Ideally you can then 
evaluate metric and methodology properties (repeatability, ...) and 
infer on network characteristics.

Still, the question is (how) can we define prototypical/representative 
traffic and how sound is it to extend measurement results to other 
scenarios. The former one is imo not so much an IPPM question although 
performance depends on it (perhaps focus of LAMP?).

regards
Joachim

---

From: Joachim Fabini <[email-masked]>
Date: Fri, 12 Oct 2012 11:39:30 +0200

Ruediger,

Regards
Joachim