Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Sun, 30 Nov 2003 21:35:20 -0800

Hi,

  The most widely supported (in terms of number of platforms) representation
of time is that used by both J2SE and J2ME, defined by the java.util.Date
object.

  Specifically the methods get and setTime() available in both environments
interprets a 64-bit integer value as the number of milliseconds since
January 1, 1970, 00:00:00 GMT.  For thorough documentation of behavior see:

  http://java.sun.com/j2se/1.4.1/docs/api/java/util/Date.html  - and the
introduction describing GMT vs. UTC concerns (i.e. leap seconds) and
getTime() and setTime() methods describe the usage of the 64-bit quantity.

  This integer value proves quite handy for doing direct millisecond
granularity math, i.e. duration_msec = end_time_msec - start_time_msec.
Compare this simple model which has no roll over for 10,000 years, with
alternatives at second resolution or finer resolution.  I think msec is the
most practical.

  So, I would argue that this time format is the optimal for rollover free
64-bit time.

  I would agree that NTP format optimizes capacity of 64-bits for a wide
variety of time, providing resolution as fine as .232 nanoseconds.
 
  I question the need for microsecond based resolution, but for completeness
using timeval format for 64-bit microsecond resolution would be my
recommended encoding.

  So I would propose encoding formats of:

   - dateTime (second resolution, using 32-bits representing unsigned
seconds since 1970 EPOCH)
   - dateTimeMsec (msec resolution, using 64-bits represening unsinged msecs
since 1970 EPOCH)
   - dateTimeUsec (usec resolution using 2 32-bit quantities representing
seconds since 1970 EPOCH and another for the number of usec, i.e. range of
0-.999999)
   - dateTimeNsec (nsec resolution, using 2 32-bit quantities representing
seconds since 1900 (not 1970) EPOCH and another 32-bit for the fractional
component.  I.e. the 32-bit value 0x00000001 = 1*2**-32.  And the 32-bit
value of 0xFFFFFFFF = (2**32 - 1) * 2**-32, assuming standard C language
operator precedence.

  I still believe there is a desire among some WG participants to leverage
per "flow group" compacting via the use of integral offsets from some time
stamp stored in a common format in the header.  E.g. header has dateTimeUsec
format of some reference time and flows in that group specify times as
32-bit offset + or - from this value.

  If the group feels this tradeoff in complexity vs. compactness is
worthwile (it may be) then I'm OK.  However some consensus on default time
encodings and addressing all four of the time resolutions shown above is
necessary.

Regards,

  Jeff Meyer

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Sebastian Zander
Sent: Wednesday, November 26, 2003 6:21 AM
To: Falko Dressler
Cc: [email-masked]
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

Hi,

common time formats are:

- time_t: seconds since epoch (4 byte)
- struct timeval: seconds, microseconds since epoch (8 byte)
- struct timespec (POSIX): seconds, nanoseconds since epoch (8 byte)
- NTP: seconds, 1/2^32 fraction since epoch (8 byte)

Unix epoch is 1970 whereas NTP epoch is 1900. Therefore some conversion is
needed to convert from NTP to one of the others.

The size of the first 3 depend on the architecture. On my machine (Intel)
they
have the sizes above but on other platforms this may be different AFAIK on
Alpha
a long is 8 bytes.

I'd like to see a timestamp format with second resolution which can be used
where a higher granularity is not required (e.g. accounting) and saves some
bytes on the wire. Another type with at least microsecond resolution should
be available for cases where a higher granularity is required (e.g. delay
measurement)

NTP makes the most of the bits (in terms of resolution ;-) and is an IETF
standard. Coversion to time_t etc. is simple.

Cheers,

Sebastian

Falko Dressler wrote:
> Issue: INFO-24  Encoding options for high-resolution timestamps
> Description:
> http://ipfix.doit.wisc.edu/archive/2214.html
> 
> Various encoding options for high-resolution timestamps for recommended 
> or optional usage.
> 
> Available types:
> 
> * NTP format (8 byte)
>   2 Parts: seconds (4 byte) / 1/2**32 fraction of seconds)
> 
> * Posix timeval
>   2 Parts: seconds (4 byte) / nanoseconds (4 byte)
> 
> * Milliseconds sinch EPOCH (8 byte)
> 
> * Micro/Nanoseconds since EPOCH (8 byte, rollover!)
> 
> 
> 
> 
> -- 
> Help        mailto:[email-masked] and say "help" in message 
> body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

-- 
Sebastian Zander                         E-mail: [email-masked]
Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany
www.fokus.fraunhofer.de/usr/sebastian.zander

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Nevil Brownlee <[email-masked]>
Date: Mon, 8 Dec 2003 17:36:23 +1300

Hi Jeff:

Thanks for your description of the Java 64-bit ms since 1 Jan 1970,
that does seem like a good compromise - to me, ms for flow times
is about right (I need us for packets, but not flows), and I like its
'no rollover' feature.

> seconds since 1970 EPOCH and another for the number of usec, i.e. range of

I agree we should support the first two, the others seem less useful to me.

> 32-bit offset + or - from this value.

I can see that it would save space to do this.  Would it be enough to
have a dateTime (seconds) in the header, then 32-bit ms offsets from that
in the flow groups?

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/

---

From: Maurizio Molina <[email-masked]>
Date: Mon, 08 Dec 2003 16:30:23 +0100

>I agree we should support the first two, the others seem less useful to me.
Hi Nevil,
I think we should be careful with excluding encodings for resolutions 
higher than the millisecond.
As Sebastian pointed out in some previous mail,
"Another type with at least microsecond resolution should
be available for cases where a higher granularity is required (e.g. delay
measurement)"
Delay measurements are actually considered in the IPFIX applicability 
draft. Moreover, PSAMP is considering of  using IPFIX as a transport, 
and PSAMP based applications are likely to include delay measurements.
Also, at the last meeting there was a hummmm in favour of microseconds.

I think this sentence from Jeff is the one that best captures the issue:

  - dateTimeUsec and Nsec - micro and nano second resolution are too exotic
for the majority of my customers, but there appears to be interest from the group.

All that said, I'd be in favour of also keeping the support of encodings 
for high time resolutions.
Cheers,
Maurizio

---

From: Falko Dressler <[email-masked]>
Date: Mon, 08 Dec 2003 20:03:00 +0100

I think there *is* a consensus for keeping high resolution time stamps. 
The remaining question is about its encodings.
Do we really need "a number of" different encodings or is it possible to 
find a consensus on a particular encoding?

My favorite is still NTP ;-)

Cheers,
Falko.

-- 
Dr.-Ing. Falko Dressler
Computer Networks and Internet
Wilhelm-Schickard-Institute for Computer Science
University of Tuebingen, Germany
Phone: +49 7071 29-70522 / Fax: +49 7071 29-5220
EMail: [email-masked] / [email-masked]
http://net.informatik.uni-tuebingen.de/members/dressler/

---

From: Mark Fullmer <[email-masked]>
Date: Tue, 9 Dec 2003 00:37:10 -0500

> have a dateTime (seconds) in the header, then 32-bit ms offsets from that

That's an interesting twist on the Netflow implementation.  You don't
need to deal with rollovers anymore this way.

mark