Subject: [ippm] packet reordering measuerments - comments

From: Michal Przybylski <[email-masked]>
Date: Tue, 06 Jul 2004 11:02:02 +0200

Dear all,

On 7/5/2004 8:52 PM, Henk Uijterwaal (RIPE NCC) wrote:
 > Michal, Bartosz,
 >
 > The IPPM group wants to finalize the IPPM reordering draft(s) in the near
 > future. As you implemented the metrics, we were wondering if you had any
 > comments on the text of the drafts.  What is clear, what is unclear, what
 > should be added to the text, which metric is best suited for which
 > purpose, etc.  The more comments, the better :-)
 >

Definitely there are some thoughts and comments - I will try to summarise them below 
(probably some of them were already discussed within TF-NGN mailing list, but at least 
I'll try to bring them together.

  We have performed the measurements for 2 different reordering metrics: 
draft-ietf-ippm-reordering-05.txt (IPPM)  and draft-jayasumana-reorder-density-02.txt (RD) 
  each of them measuring reordering differently.

The one most important comment is the approach to the measurement:

1. The draft should suggest not only definition of metrics, but also the way how it is 
measured (there is also a comment about this from editors in the draft).

a) sending (with a maximum link speed) a burst of longest packets followed by a burst of 
shortest packets (both longest and shortest packets had to fit into single 1500 bytes 
frame). Such a measurement gave the reordering extent of 10 and more, for many links where 
reordering might have occured. However this is the worst case, and rather unrealistic, 
since we din't see that pattern in any of analysed application. We consider this kind of 
measurement useful for testing i.e. single network device (such as ethernet switch, 
router) or a complete network.
There may be also variations of this scenario (especially with packet patterns, one may 
use only short packets or long packets only). General idea is to send with the maxiumum 
possible speed./
This kind of test answers the question - "what is the worst reordering I can get from my 
network?"

b) re-sending captured packet stream emulating given application - we think that this is 
best way to see how given application copes with the network, and how impaired network 
influences such application. We can use IPPM metrics to see the necessary buffer size for 
such application. So the question here is "what is the de-reordering buffer length I need 
in my application for this network"

We think that the networks should in general be checked against reordering (method "a"), 
but for given application one shall use method "b" for measuring influence of impaired 
network.

2. The important issue is with the packet loss
You can look at
http://ngn.man.poznan.pl/en/research/reordering/results-html/out.401-CZ-RD.html
and
http://ngn.man.poznan.pl/en/research/reordering/results-html/out.401-CZ-IPPM.html

We have received only 3 packets of the stream:

ID=401 SEQ=684 SRC=193.1.200.241:3001 DST=195.113.147.23:5000
                   Tx=1086343531.959051 Rx=1086343531.557416 SIZE=20

ID=401 SEQ=685 SRC=193.1.200.241:3001 DST=195.113.147.23:5000
                    Tx=1086343531.959241 Rx=1086343531.557444 SIZE=20

ID=401 SEQ=686 SRC=193.1.200.241:3001 DST=195.113.147.23:5000
                    Tx=1086343531.959431 Rx=1086343531.557492 SIZE=20

There is no reordering, but there is packet loss. If one wants to cope with reordering, he 
has to implement buffers. If application gets packet with the SEQ number higher that the 
last one by more than 1 it considers this packet as "early packet" and will store it in 
the buffer untill some time period, when it will notice packet loss. In this scope IPPM 
metric is right and wrong ;-). Right because there is no reordering but packet loss, wrong 
because it says that the buffer size is 0. In fact the receiving application, willing to 
cope with possible reordering (it got the higher SEQ packet than the lower one!) will use 
de-reorder buffers up to a certain time limit. So the buffers are there.

We find here the Reorder Density (defined in draft-jayasumana-reorder-density-02.txt) 
metric very useful as a metric of usage of the buffer. The buffer is always used when 
received packet is not the expected one (received packet is stored in buffer).
Algorytm doesn't differentiate between the lost packets and the late packets.
So this algorithm can be used as an extention for IPPM reordering metrics for scenarios 
with packet loss.

We think that the only accurate metric should be focusing on given application traffic 
pattern and shall also consider packet loss (perhaps combination of IPPM and jayasumana 
drafts)

All the best,

Michal

-- 
______________________________________________________________________
Michal Przybylski        Network Research and Development
[email-masked]    Poznan Supercomputing and Networking Center
+48 61 858 20 27         www.man.poznan.pl
______________________________________________________________________

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: Al Morton <[email-masked]>
Date: Wed, 07 Jul 2004 09:47:49 -0400

Hi Michal,

Thanks for your efforts to implement the IPPM reordering metrics
and commenting on your experiences.

Please see responses below,
Al

At 05:02 AM 07/06/2004, Michal Przybylski wrote:
>...
> draft-ietf-ippm-reordering-05.txt (IPPM)  and 
> draft-jayasumana-reorder-density-02.txt (RD)  each of them measuring 
> reordering differently.

Yes, the differences are intentional, since RD includes the effects of
packet loss, and ippm-reordering-05 does not.

>1. The draft should suggest not only definition of metrics, but also the 
>way how it is measured (there is also a comment about this from editors in 
>the draft).

We have a section on measurement considerations, and we have expanded
the coverage of implementation issues thanks to comments and suggestions
from David Newman (see ippm-reordering-06 soon).  However, this is the
draft where we define the metrics, and the IPPM charter says that measurement
details are in the scope of specific Applicability Statement (AS) documents.

(from the charter)
      Each AS document will discuss the performance
      characteristics that are pertinent to a specified service class;
      clearly identify the set of metrics that aid in the description of
      those characteristics; specify the methodologies required to collect
      said metrics; and lastly, present the requirements for the common,
      unambiguous reporting of testing results. Specific topics of these
      AS documents must be approved by the Area Directors as charter
      additions.

>a) sending (with a maximum link speed) a burst of longest packets followed 
>by ...
>This kind of test answers the question - "what is the worst reordering I 
>can get from my network?"

I'll note this in section 6.

>b) re-sending captured packet stream emulating given application - we 
>think that this is best way to see how given application copes with the 
>network, and how impaired network influences such application. We can use 
>IPPM metrics to see the necessary buffer size for such application. So the 
>question here is "what is the de-reordering buffer length I need in my 
>application for this network"

But if the application modifies its stream in response to reordering
conditions, the captured stream would not take this into account.
We can still estimate buffer sizes required to sort out reordering.

>We think that the networks should in general be checked against reordering 
>(method "a"), but for given application one shall use method "b" for 
>measuring influence of impaired network.
>

Nice reporting formats!

>There is no reordering, but there is packet loss. If one wants to cope 
>with reordering, he has to implement buffers. If application gets packet 
>with the SEQ number higher that the last one by more than 1 it considers 
>this packet as "early packet" and will store it in the buffer untill some 
>time period, when it will notice packet loss. In this scope IPPM metric is 
>right and wrong ;-). Right because there is no reordering but packet loss, 
>wrong because it says that the buffer size is 0.

I would say the IPPM metric is right, because the buffer is needed
only to deal with packet loss, not reordering.
The size of the buffer needed to deal with reordering is zero,
because there was no reordering :-) That's orthogonal metrics at work.

>In fact the receiving application, willing to cope with possible 
>reordering (it got the higher SEQ packet than the lower one!) will use 
>We find here the Reorder Density (defined in 
>draft-jayasumana-reorder-density-02.txt) metric very useful as a metric of 
>usage of the buffer. The buffer is always used when received packet is not 
>the expected one (received packet is stored in buffer).

Yes, this is a key point.  We have very carefully maintained this
distinction in ippm-reordering-05. This is not to say that the metrics
are completely immune to loss - all IPPM metrics that rely on packet
arrivals are influenced to some extent by non-arrivals.

>So this algorithm can be used as an extention for IPPM reordering metrics 
>for scenarios with packet loss.
>
>We think that the only accurate metric should be focusing on given 
>application traffic pattern and shall also consider packet loss (perhaps 
>combination of IPPM and jayasumana drafts)

I agree that assessment of receiver buffer size to compensate for
Loss and Reordering *is* a useful metric, but it is not a metric of
reordering alone, so it is beyond the scope of ippm-reordering-05.

I think that one way forward might be to rename "Reorder Density"
as "Receiver Buffer Density", or something similar.
This would be a "Derived Metric", like IPDV,
using the definitions for loss and reordering, (and IPDV too?)
to estimate the receiver buffer occupation/density over time.

With loss and hopefully reordering metrics now well-defined,
the work on a derived metric has a decent foundation.

best regards,
Al

---

From: Michal Przybylski <[email-masked]>
Date: Wed, 07 Jul 2004 16:13:32 +0200

Hello Al,

Thank you for such exhaustive comments to my mail.

On 7/7/2004 3:47 PM, Al Morton wrote:

> 
>> a) sending (with a maximum link speed) a burst of longest packets 
>> followed by ...
>> This kind of test answers the question - "what is the worst reordering 
>> I can get from my network?"
> 
> 

Just keeping in mind interpretation - this is reordering caused by parallel processing. I 
was trying to guess, however how to explain reordering in the scenario where only long 
packets of the same size are being sent - reordering here would indicate some parallel 
processing with different processing speed in each queue - but I guess this is out of 
scope for metric definition.

> to estimate the receiver buffer occupation/density over time.

I find base metrics not always very useful, as interpreting several metrics and converting 
them to "expected application beahvior" might be quite difficult. Therefore I am in favor 
of this "derived metric", which could be called "Receiver Buffer". It is pretty 
straightforward and explaining - especially for application designers.
> the work on a derived metric has a decent foundation.

We will watch closely and surely test it ;-)

All the best,

Michal