Subject: Re: [ippm] draft-ietf-ippm-owdp-08.txt

From: stanislav shalunov <[email-masked]>
Date: 19 Jul 2004 00:02:07 -0400

Henk,

Thank you very much for the excellent comments.

I'm rolling the changes made in response to this set of comments and a
few more minor changes made between -08 and now into a new version, -09.

"Henk Uijterwaal (RIPE NCC)" <[email-masked]> writes:

> * General: The document should specify that the implementation MUST have
>   rate limitations in unauthenticated mode (to avoid that we end up with
>   the same discussion as the requirements doc).

I appended the following text to the end of the security
considerations section.  I think it strikes about the right balance
between the need to require the implementor to include useful policy
mechanisms and the desire to give the implementor freedom in giving
the administrator tools to express local policy.  I believe that this
is an area where implementors can be allowed to compete.  If the group
thinks that ``one way'' (from the third paragraph) should be changed
to ``The Way,'' I would not object; the third paragraph would then
need to be peppered with MUST/SHOULD language.  But I think the second
paragraph is sufficient to satisfy the requirements of RFC 3763
without creating a whole Tao of Quota Allocation and Resource
Management.

#                        Resource Use Limitations
# 
# An OWAMP server can consume resources of various kinds.  The two most
# important kinds of researces are network capacity and memory (primary
# or secondary) for storing test results.
# 
# Any implemenation of OWAMP server MUST include technical mechanisms to
# limit the use of network capacity and memory.  Mechanisms for managing
# the researces consumed by unauthenticated users and users
# authenticated with a username and passphrase SHOULD be separate.  The
# default configuration of an implementation MUST enable these
# meachnisms and set the resource use limits to conservatively low
# values.
# 
# One way to design the resource limitation meachanisms is as follows:
# assign each session to a user class.  User classes are partially
# ordered with ``includes'' relation, with one class (``all users'')
# that is always present and that includes any other class.  The
# assignment of a session to a user class can be based on the presence
# of authentication of the session, the user name, IP address range,
# time of day, and, perhaps, other factors.  Each user class would have
# a limit for usage of network capacity (specified in units of
# bit/second) and memory for storing test results (specified in units of
# octets).  Along with the limits for resource use, current use would be
# tracked by the server.  When a session is requested by a user in a
# specific user class, the resources needed for this session are
# computed: the average network capacity use (based on the sending
# schedule) and the maximum memory use (based on the number of packets
# and number of octets each packet would need to be stored internally --
# note that outgoing sessions would not require any memory use).  These
# resource use numbers are added to the current resource use numbers for
# the given user class; if such addition would take the resource use
# outside of the limits for the given user class, the session is
# rejected.  When resources are reclaimed, corresponding measures are
# subtracted from the current use.  Network capacity is reclaimed as
# soon as the session ends.  Memory is reclaimed when the data is
# deleted.  For unauthenticated sessions, memory consumed by an
# OWAMP-Test session SHOULD be reclaimed after the OWAMP-Control
# connection that initiated the session is closed (gracefully or
# otherwise).  For authenticated sessions, the administrator who
# configures the service should be able to decide the exact policy, but
# useful policy mechanisms that MAY be implemented are the ability to
# automatically reclaim memory when the data is retrieved and the
# ability to reclaim memory after a certain configurable (based on user
# class) period of time passes after the OWAMP-Test session terminates.

> * Page 4, 2nd drawing. Why is the fetch_client sitting in the sending
>   box.  I'd think that results are typically requested by a machine
>   outside the pair sender and receiver.

The more general case is represented in the first diagram.  The second
diagram is just what it says it is: an example.

> * Section 5.1. Server Greeting: Why does one send 12 unused octets
>   in the server greeting first?

To align on the 16-octet block boundary so that CBC mode does not lose
its authentication properties.

> * Page 10: I cannot find the definition of MBZ anywhere.

Inserted the following text after the first occurrence of MBZ.  (I was
assuming that the abbreviation would be familiar to an RFC reader, but
this is probably assuming too much.)

# MBZ (MUST be zero) fields here and hereafter have the same semantics:
# the party that sends the message MUST set the field to a string of
# zero bits; the party that interprets the message MUST ignore the
# value.  (This way the field could be used for future extensions.)

> * 5.4. First sentence:  this is not correct, the receiver only knows
>   that packets were sent _if_ it knows that sending machine did send
>   what it was supposed to send.  The receiver cannot tell from the
>   measurements if the sending machine stopped sending or all packets
>   were lost.

Changed the sentence to read (unidiff change format):

 This way, when packets are lost, the receiver knows when they were
+supposed to be
 sent.

Note that since there's a graceful termination procedure at the end of
the session, a peer always knows whether the other peer crashed or
stayed up until the end of the session.

>   Same paragraph: are 0 and 1: poisson and periodic the only values
>   that can be used here, or is this a point for future extension?

This could be used for future extensions, obviously, but I don't see
need for that: we only need to compress well common schedules --
should a need for an uncommon schedule arise, an arbitrary one can be
specified using this mechanism.

> * Page 16: 'The server should start ...'.
>   Suppose the start time is specified as t=0 with a rate of 100 packets.
>   For some reason, the server takes more time to start, so the first
>   packet cannot be sent before t=1.  What should be done with the 100
>   packets that had to be sent between t=0 and t=1?
> 
>    - Send as fast as possible?  Possibly overloading the network,
>      with the risk of making a complete run invalid.
>    - Send with a (minimum) gap?
>    - Skip

The way it is written now, the implication is that they should be
skipped.  The send schedule is rigid and reproducible, with exact
timestamps attached to every packet to be sent.  If some packets are
in the past, they would have to be skipped.

Is this reasonable?

The alternative approach would be to have the sender communicate the
actual start time via OWAMP-Control.  Is it worth doing this way?

(I've marked this as an open issue.)

> * Page 18-19: 'if a receiver ... invalidated'
>   Doesn't this mean that if the server crashes near the end, all results
>   are suddenly invalid, even though they are perfectly good measurements?

They're not perfectly good.  The crashing computer that doesn't close
the session gracefully leaves it unknown whether packets were lost or
simply not sent.  The protocol includes this provision to make the
measurement results more reliable.  Note that not all results are
invalidated, one can still keep the part known to be good.  Quoting
existing text:

# If the OWAMP-Control connection associated with an OWAMP-Test
# receiver receives the (0xFFFFFFFF) special value for the Session
# Packets Sent, or if the OWAMP-Control connection breaks when the
# Stop-Sessions command is sent, the receiver MAY not completely
# invalidate the session results.  It MUST discard any records of lost
# packets that follow (in other words, have greater sequence number
# than) the last packet that was actually received.  This will help
# differentiate between packet losses that occurred in the network and
# the sender crashing.  When the results of such an OWAMP-Test session
# or an OWAMP-Test session that was prematurely aborted successfully
# (with confirmation) are later fetched using Fetch-Session, the
# original number of packets MUST be supplied in the reproduction of
# the Request-Session command.

>   (We typically run measurements for weeks, and the chances that they
>   are terminated by an outside source are pretty close to 1)

The way continuous measurements are handled for Abilene is that they
are split into relatively short sessions.

> * Page 21: the test session data.  Typically, the error estimates and
>   time stamp will be in 4 variables in the code:
> 
>     Send-error-estimate    16 bits
>     Receive time stamp     32 bits (seconds), 32 bits (microseconds)
>     Receive error estimate 16 bits
> 
>   To store this in the  format described here one has to:
> 
>    - Multiply the SEE by 2^16
>    - Take the seconds word of the RTS and split it into 2 16 bit words
>    - Divide the top half by 2^16 and add to the SEE
>    - Multiply the bottom half by 2^16, store
>    - Take the microsecond word, split in two times 16 bits
>    - Divide the top half by 2^16 and add to the bottom half of the RTS
>    - Multiply the bottom half by 2^16 and add to the REE.
> 
>   Then do exactly the opposite on the receiving side.  Wouldn't it make
>   more sense to use the first 2 word for the timestamp, then the
>   next one for the EE.  That way, most of this will be a straight copy,
>   one only has to manipulate the SEE bits.

This assumes an extremely naive implementation using a specific
internal system time format that is, indeed, native to Unix-like OSes
but that may not be identical on other OSes or in hardware
implementations.

Regarding error estimates: Firstly, there is no need to generate the
error estimate in the protocol format for every packet.  The error
estimate changes infrequently.  Secondly, the possible values of error
estimate are often quantized in such a way that only a few are
actually produced; if the results of conversion are cached (say, only
for the last value), the vast majority of conversions is eliminated.
Thirdly, there is no need to do the conversion of the error estimate
between fetching the timestamp and sending the packet; only the actual
timestamp needs to be converted in this crucial short time.  The
implementation at http://owamp.internet2.edu/ uses these
considerations and any other implementation can and should.

Regarding timestamp values: we use what NTP uses.

> * Page 23: Is there a technical reason to start sequence numbers with
>   0 and not 1?

No special technical reason to start it with any particular number.
Given that 0 is the smallest natural number it seemed like an
acceptable place to start.

> * Page 23: I've made this comment before, but I think this way to
>   express the EE introduces a lot of complexity at a point where one
>   wants to use the minimum number of CPU cycles.

See above (``regarding error estimates'').

>   When timestamping a packet, one wants to extract a timestamp just before
>   the packet is sent, insert it in the packet, then write the packet to
>   the socket.  Any CPU cycle between reading the time stamp and writing
>   the packet means introducing a delay and making the measurement less
>   accurate.  Using this method, one has to do a lot of calculations to
>   file the word, if we'd use RFC 1305 timestamps for everything, the
>   operation would be reduced to a simple copy.

I don't think so.  The system internal representation (seconds and
``microseconds'' [1] or ``nanoseconds'') is not in the NTP format.

>   Also, implementations of the metrics where the ethernet card inserts
>   the timestamp have been suggested.  In this case, it is impossible to
>   use this method for inserting errors.

Suppose one wanted to build a NIC that implemented (some portion of)
OWAMP internally -- in cooperation with the host software, of course.
I don't see how the chosen formats are any better or worse than
others.  First, in a sane implementation the NIC would only fill in
the timestamp itself (and modify the UDP checksum or use the first two
bytes of padding to make the checksum match the value that was in the
checksum field before).  The IP and MAC addresses, port numbers,
OWAMP-Test sequence number, and the error estimate would already be
filled.  The NIC would need some source of time (probably an internal
oscillator) and some way to convert it to UTC.  The latter part is not
trivial and would require the NIC, in essence, to allow the host to
run a little NTP server on its behalf that would periodically compare
the time as the NIC thinks it is with some source of UTC (e.g., the
system clock) and use the huff-puff algorithm or a different algorithm
to undate the two conversion parameters.  The complexity of the
conversion to UTC would dwarf the complexity of converting the UTC
timestamp in any reasonable internal format to any reasonable wire
format.  That the wire format happens to be NTP would enable the
implementor to gain efficiency if it was desired to put part of an NTP
implementation into the NIC.

For what it's worth, note that the NTP cakebox implementations to
date, at least the ones that I know the design for, do not keep this
extra oscillator and extra conversion coefficients in the NIC, so this
talk about putting OWAMP in the NIC is somewhat theoretical.

Again, thanks for the careful reading of the document and your
excellent and stimulating comments.

                                                --Stas

[1] Scare quotes, because tv_usec can, according to the interface
    specification, be greater than or equal to 1000000.  This actually
    happened now and then on FreeBSD 3.x.  I just checked and it
    didn't happen in any of 100,000,000 tries on this FreeBSD 5.x box,
    but this can't be counted on, unfortunately [2].

[2] Unfortunately, because the design that *typically*, but not in a
    guaranteed way, will normalize microseconds in the kernel to be
    smaller than 1000000, will introduce an extra memory fetch and a
    conditional instruction into the typical operation of getting a
    timestamp.  But given how inefficient getting timestamps is
    through gettimeofday() [3], this is really a rather minor issue
    compared to two inherent context switches [4].

[3] 4.5us per timestamp on this 2-GHz CPU, for crying out loud.

[4] One can get rid of the context switches by using the TSC register
    on Pentium CPUs.  This still requires NTP-like maintenance of the
    two conversion coefficients.

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

Just my 0.086g of Ag.

_______________________________________________
ippm mailing list
[email-masked] 
https://www1.ietf.org/mailman/listinfo/ippm

---

From: "Henk Uijterwaal (RIPE NCC)" <[email-masked]>
Date: Tue, 27 Jul 2004 14:44:48 +0200 (CEST)

Stas,

> in the past, they would have to be skipped.

Perfectly reasonable, this is just so we know how to implement this, I
don't think that there is a need for anything more complex.

> but that may not be identical on other OSes or in hardware

Well, yes, I know it can be done somewhat more efficient, but whatever you
do, you will end up splitting 2 16-bits and 2 32-bits numbers over 3 32
bit words.  If the order is changed to 32 bits for the 2 timestamps, then
1 word for the 2 16 bit EE, you have to put 2x16 bits into one word (and
this requires some manipulation) but the rest are straight copies. This is
code that is sitting in the most time-critical part of your application,
anything we can do to reduce the number of operations there, will improve
the measurements.

If there is a techical argument for this, then please tell me, if not, I
don't see why the order cannot be changed.

> estimate changes infrequently.

I tend to disagree.   Also, if one decides that error estimate is only
checked and recalculated once every N seconds, then there will be 2
classes of packets: one where the error estimation code was called and one
where it was not.  In the former case, the systematic errors will be
different than in the latter, but the user has no way to tell.

>  Secondly, the possible values of error

The set of natural numbers excluding 0 is smaller but still contains
sufficient numbers for all packets :-)

Henk

------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

Process and Procedure are the last hiding place of people without the wit
and wisdom to do their job properly.                          (David Brent).