Subject: Re: WG input needed on "Periodic Streams"

From: [email-masked]
Date: Mon, 20 Mar 2000 02:10:11 EST

In a message dated 3/17/00 3:30:15 PM Eastern Standard Time, 
[email-masked] writes:

>  1) ...While the previous metrics have stressed Poisson traffic
>     to avoid synchronizing with network activity, this draft seems to
>     present a well-motivated reason to use periodic streams that is
>     potentially important to a significant class of applications.
>  
Although the packets in the streams are periodic to simulate those sent by 
the application(s) of interest, and such periods are likely to be very 
dissimilar to Poisson inter-arrival intervals, there is no reason why the 
streams themselves could not be sent at Poisson intervals to gain the 
advantages of Poisson sampling. To converge to an estimate of the network's 
performance over various useful time periods, it will be necessary to send 
more than a single stream anyway, (unless the test stream is sent 
continuously).

>  2) Is the proposed metric itself well-defined and useful?
>      Does it give the right information to be useful for these streams?
>      Are there better alternatives?
>  
The periodic packet interval, incT, should be better defined i.e., is it from 
the first bit of packet 1 to first bit of packet 2, or from the last bit of 
packet 1 to first bit of packet 2?

The threshold for delay equivalent to loss, dTloss, is listed as an 
"optional" parameter of the metric. Although it may not be included in an 
application, how can it be excluded from the metric? Without some value of 
dTloss, how will a lost packet be counted or a delay value of "infinity" 
returned? Sect. 4.8 first mentions the "threshold for delay equivalent to 
loss *(if any)*" - again optional - but Sect. 4.8.2 says it MUST be reported. 
I think further clarification of this parameter is required. Also, it could 
be stated that if this parameter is present in a given application, the same 
value can be used in the metric.

It should be stated that unlike Delay, DJit(i) can be either zero, positive 
or negative or alternately, expressed as: |Tstamp(Dst)[i]-Tstamp(Dst)[i-1] - 
(Tstamp(Src)[i]-Tstamp(Src)[i-1]|.

>  3) If the class of stream is important and the proposed metric useful,
>      how should the draft be improved?  One specific concern we have already
>      forwarded to the authors is that the metric can be better mapped
>      to the IPPM metric framework.  In particular, the progression between
>      singleton to sample is muddied here.
>  
The metric is currently defined as a sample metric. Since this is an 
application level metric and the streams are used to simulate application 
traffic, perhaps it would be useful to consider the measurement results for 
each instance of a packet stream (as opposed to each instance of a packet) as 
a singleton metric (based on measurements of the delays of the individual 
packets in the stream). I believe this is supported by RFC 2330 Sect. 11.

The sample metric would be composed of measurement data from several streams, 
separated in time by Poisson intervals as suggested above.

Application level statistics could then be generated from the sample. These 
would complement the statistics proposed in Sect. 4.9 of the draft.

E.g., if the application is a VoIP call and each stream represents one call, 
or the packets sent in one direction of a call, one could generate a 
statistic stating that n% of the *calls* made via the network meet or exceed 
some performance criteria such as those shown in Sect. 4.9. Another statistic 
might say that 90% of the calls actually meet some other measured criteria. 
Either way, it is the performance of the application via the network (the 
calls) - not just the packets - that is being reported in these statistics.

Regards,
Howard Stanislevic
HSComms Network Engineering and Consulting

---

From: [email-masked]
Date: Tue, 21 Mar 2000 14:24:10 +0200

Hello Howard,

thanks for the comments.

Your point about tests performed at Poisson distributed intervals vs.
network time scales is a good one. The present draft was formulated in a
hurry and more effort should be invested in clarifying issues such as this
one.

As to singleton/sample metrics, the proposed metric was originally called
"sample metric" to be consistent with the parlance of one-way and RT
metrics. This way, it is possible to refer to "singleton metrics" in
previous IPPM docs for related caveats &c. My original idea was that
definition of statistics based on sample metrics might define Poisson
distributed intervals between sample tests. I'm open to suggestions and
comments, however. List, comments?

In summary, I think that you raised important points and I hope that the
sample vs. singleton issue and definition of statistics (or sample metric in
the alternative parlance) will be discussed in Adelaide. Moreover, the
utility of using a percentage of "acceptable" calls vs. network behaviour
time scales would probably need further attention. Unfortunately, I can't
attend the meeting but will read the minutes with interest :).

	BR,
		Vilho

> -----Original Message-----
> From: EXT [email-masked] [mailto:[email-masked]]
> Sent: 20. March 2000 9:10
> To: [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: WG input needed on "Periodic Streams"
> 
>

---

From: Grotefeld Glenn-cecl03 <[email-masked]>
Date: Tue, 21 Mar 2000 13:20:34 -0600

Vilho and I appreciate Howard's comments (and certainly Will's!) and I make
specific responses within (noted as GG)

Glenn Grotefeld

Motorola, Inc.
[email-masked]
US  847 576-5992
      847 538-7455  FAX

GG:  I agree that BETWEEN samples could be modeled as Poisson inter-arrival
interval.  In the context of ippm, more than one sample would represent a
"sample of samples".  Contributions on text to describe this would be
appreciated.

>      Are there better alternatives?
The periodic packet interval, incT, should be better defined i.e., is it
from 
the first bit of packet 1 to first bit of packet 2, or from the last bit of 
packet 1 to first bit of packet 2?

GG:  I agree that more text is needed.  I bogged down over the question of
how to describe the potential variability of incT.

loss *(if any)*" - again optional - but Sect. 4.8.2 says it MUST be
reported. 

GG:  I left this open-ended, thinking that there may be an application that
does not care about the delay that a packet experiences.  Now that I think
more, what application could stand waiting for 24 hours for a packet?  

GG:  The authors have different views on whether jitter should be defined at
all.

each instance of a packet stream (as opposed to each instance of a packet)
as 

The sample metric would be composed of measurement data from several
streams, 
separated in time by Poisson intervals as suggested above.

some performance criteria such as those shown in Sect. 4.9. Another
statistic 
might say that 90% of the calls actually meet some other measured criteria. 
Either way, it is the performance of the application via the network (the 
calls) - not just the packets - that is being reported in these statistics.

<<GG:  I did not give sufficient attention to defining a singleton because a
number of items of interest (out of sequence, duplicate) CANNOT be defined
for a singleton.  I will provide/accept additional text to "unmuddy" the
progression.  As I stated in an earlier section, I would not define a stream
as a singleton, but define a number of streams as a "sample of samples." >>

Regards,
Howard Stanislevic

---

From: Al Morton <[email-masked]>
Date: Wed, 22 Mar 2000 16:11:28 -0500

After reading the Draft and responses from Howard and the authors,
let me add the following to Will's questions for the WG:

At 01:55 PM 3/17/00 -0500, Will E. Leland wrote:
>The key questions for the working group are:
>
>1) Is this an area in which a metric would be useful?
Yes. A small group here has been defining measurement methods
for this application class, and agree with the author's reasoning.

>    Are there better alternatives?
I'd like to know if the WG will take up this work before typing up 
several suggestions.  But let me add to the discussion on incT, since
Howard Stanislevic pointed out a potential source of ambiguity and Glenn
agreed to address it.  IMO, the logical boundaries of the periodic
packet interval are first bit of packet n to first bit of packet n+1.
Now incT corresponds to the duration of real-time media contained in
the packets, and avoids the variability of last bit-first bit or
last bit-last bit intervals when using non-uniform packet lengths.

Also, measuring delay variation has been an important part of our
small group's deliberations, so I found Glenn's comment intriguing
>GG:  The authors have different views on whether jitter should be defined at
I'd like to hear the view on why jitter is unnecessary.

>    singleton to sample is muddied here.

One suggestion would be to specify several alternatives for
the Statistics reporting section. The alternatives correspond
to typical application receivers (e.g., one where delay variation 
is critical, another where substantial delay variation and even 
out-of-sequence packets can be tolerated). These alternative
"receivers" would ideally be few in number, but cover the majority
of applications with periodic packet streams.

>
>If we do not see some interest from the Working Group, we'll view that
>as a clear statement that this metric is not needed.
>
We're willing to review, comment, and contribute as necessary to the
development of this metric.  If for some reason ippm chooses not
to take up this work, we'll follow the authors to a 
different venue and help with development there.

Enjoy Adelaide,
Al Morton

---

From: Grotefeld Glenn-cecl03 <[email-masked]>
Date: Mon, 27 Mar 2000 09:06:00 -0600

~GG:  Responses supplied within the section.

Glenn Grotefeld

Motorola, Inc.
[email-masked]
US  847 576-5992
      847 538-7455  FAX

On Fri, 17 Mar 2000, Will E. Leland wrote:

> 
> As Chairs, we need discussion and insight from the IPPM list on the
> new I-D on periodic streams. We do not want to encourage a wide range
> of specific (and not widely useful) metrics, but rather a small set
> of generally useful metrics.  We think the current draft represents
> a class of traffic not well captured by the existing metrics.  However,
> we seek your input.
> 
>    This draft represents the convergence of two independent proposals,
>    which leads us to believe there's a good chance the metric would
>    be used.  While the previous metrics have stressed Poisson traffic

I find these questions hard, if not impossible to answer from just the
current draft. Before finalizing the draft, I think that there should be
at least a preliminary implementation plus some practical experience to
show that this actually measures something useful.

Then, paragraphs 4.2.1 to 4.2.4 confuse me: If I understand it correctly,
the metric itself consists of global definitions (4.2.1) and (4.2.4). To
extract the data one has to do measurements at SRC and DST (4.2.2 and
4.2.3).  Wouldn't it make more sense to define the submetrics first, then
merge 4.2.1 and 4.2.4 into 1, with some discussion on how to get there
from the indivisual measurements?
-----START GG-------
GG: The titling of the subparagraphs could be sharpened up to eliminate the
potential for confusion (hint: alternate text welcome).  
+ 4.2.1 relates to items that apply to all three following sections.
+ 4.2.1 also applies to parameters that are known prior to (deep breath,
follow the protocol) the collection of singletons* that form the sample*.
The other sections (especially 4.2.4) are only known AFTER the sample.
+ 4.2.2 and 4.2.3 are items collected at a local level (Src and Dest).  
+ "4.2.4 Metrics resulting when metrics collected at MP(Src) and MP(Dst)are
merged":  I'm sorry, that looks pretty clear to me.
-----END GG/ START GG------------
Finally, there is no paragrpaph 4.3 in the draft, is this correct?

GG:  Yes, paragraph renumbering was missed as we were pushing around
sections.
-----START  GG------

2 minor comments:

----START GG------
GG:  See Matt's separate message
----END GG --------

4.5  5th + sign ("Depending on..")
    I think the choice of GPS or NTP is an implementation detail. The
    metric should be reported with an experimental error, and this will
    be larger for a setup with just NTP. It's up to the user to determine
    if the error is still acceptable.

4.7.3.  Shouldn't the + signs be +/- ? (in reported value is true value
    +/- random error).  If one can deterimen the random error, it's not
random anymore.

Henk

----------------------------------------------------------------------------
--
Henk Uijterwaal                    Email: [email-masked]
RIPE Network Coordination Centre     WWW: http://www.ripe.net/home/henk
Singel 258                         Phone: +31.20.535-4414,  Fax -4445
1016 AB Amsterdam                   Home: +31.20.4195305
The Netherlands                   Mobile: +31.6.55861746  
----------------------------------------------------------------------------
--

The Committee (...) was unable to reach a consensus that substantial merit
was
lacking. Thus, the appeal was deemed meritorious.          (Orlando NABC
#19).

~