Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 9 Dec 2003 00:48:24 -0500

Mark,

  I see two different concerns in this disucssion.  One is information
modeling
and the other is encoding mechanisms.

  Your suggestion, which sounds plausible to me, is to only define two
encoding
models for IPFIX.  The first model, which is "split time" encoded requires a
32-bit
header and a 32-bit msec offset value.  The second encoding model is also
"split
time" in that the 32-bit header is seconds, but now seconds since 1900 (not
1970)
and the offset is NTP style fractional component (i.e. n/2**-32, where n is
the 
encoded 32-bit unsigned value).

  If this were the case, than IPFIX information models should likely only
care
about information elements which are dateTimeMsec and dateTimeNsec.  Other
transfer
formats or protocols may identify timestamps with different reolutions, but
there is 
a clear model for transforming from one type to the other.

  Given my preference for simplicity, I would support this proposal:

    - dateTimeMsec -> split 32-bit integer seconds since 1970 carried in 
                    IPFIX flow header and 32-bit integer count of (signed)
                    microseconds offset from the header.  **This would
                    imply that all events for a flow set using this
optimization 
                    would need to occur w/in about 2000 seconds before or
                    after the sentinel time in the header. [+/- 2000 seconds
                    is +/-2,000,000,000 usecs, around the signed int32
limit]

    - dateTimeNsec -> split 32-bit time in NTP format with 32-bits of
seconds 
                     since 1900 (NOT 1970!) stored in the IPFIX header.  And
32-bit 
                     fraction of a second (n*2**-32) component.  This
implies
                     that all timestamps must fall begin with the same
second
                     offset in a flow set.

-- Jeff

-----Original Message-----
From: Mark Fullmer [mailto:[email-masked]]
Sent: Monday, December 08, 2003 9:31 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

Do we really need four ways to represent time?  How about one 32 bit
format and one 64 bit format?

The existing NetFlow 32 bit msec Start/End time stamps + header
real time offset gives you msec real time resolution.  Implementations
that can not support / don't need msec resolution can just leave the low
order bits 0.  So this covers dateTime and dateTimeMsec with one 32
bit encoding.

64 bit NTP format format with nanosecond resolution covers microsecond
resolution implementations.  Again, implementations that want microsecond
just leave the low order fractional bits 0.

mark

On Mon, Dec 08, 2003 at 10:57:32PM -0500, MEYER,JEFFREY D (HP-Cupertino,ex1)
wrote:
> Resending to full mail list...
> 
> -- Jeff
> 
> -----Original Message-----
> From: MEYER,JEFFREY D (HP-Cupertino,ex1) 
> Sent: Monday, December 08, 2003 3:53 PM
> To: 'Nevil Brownlee'
> Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding
> 
> 
> Hi,
> 
>   Here's a proposal for encoding:
> 
>    - dateTime -> 32-bit integer seconds since 1970
>    - dateTimeMsec -> 64-bit integer msec since 1970 (Java format)
>    - dateTimeUsec -> split 32-bit integer seconds since 1970 carried in 
>                    IPFIX flow header and 32-bit integer count of (signed)
>                    microseconds offset from the header.  **This would
>                    imply that all events for a flow set using this
> optimization 
>                    would need to occur w/in about 2000 seconds before or
>                    after the sentinel time in the header. [+/- 2000
seconds
>                    is +/-2,000,000,000 usecs, around the signed int32
limit]
>    - dateTimeNsec -> 64-bit time in NTP format with 32-bits of seconds
since
>                    1900 (NOT 1970!) and 32-bit fraction of a second
> component.
>                    Both values occur in the flow record.
> 
>  
>   Note a separate issue is what type should the basic flowCreationTime
> and flowEndTime information elements should have?  Currently Info-02 has
> them
> as dateTime, but should probably be dateTimeMsec to reflect current
> information
> content of deployed netflow, or perhaps dateTimeUsec, if folks prefer to
> only use 32-bits to achieve some size reduction when passing many
> timestamps.
> 
>   My preference would be dateTimeMsec.
> 
> -- Jeff
> 
> > -----Original Message-----
> > From: majordomo listserver 
> > [mailto:[email-masked]]On Behalf
> > Of Nevil Brownlee
> > Sent: Monday, December 08, 2003 2:07 PM
> > To: [email-masked]
> > Cc: [email-masked]
> > Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding
> > 
> > 
> > Hello Carter:
> > 
> > >    This is an interesting question, as it assumes that
> > > all the records are coming from a single IPFIX data
> > > source.  What happens if IPFIX is transporting data
> > > from multiple sources as they are received, but the
> > > timestamps in the various records are not synchronized?
> > 
> > Sounds like a good reason not to do it.  
> > That would simply mean that timestamps in an IPFIX record would all 
> > be in whichever of the four encodings the meter was configured to 
> > send, right?
> > 
> > Cheers, Nevil
> > 
> > --------------------------------------------------------------
> > ---------
> >    Nevil Brownlee                   Director, Technology Development
> >    Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
> >    FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand
> > 
> > 
> > -------------------------------------------------
> > This mail sent through University of Auckland
> > http://www.auckland.ac.nz/
> > 
> > --
> > Help        mailto:[email-masked] and say "help" 
> > in message body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> > 
> 
> --
> Help        mailto:[email-masked] and say "help" in message
body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Tue, 9 Dec 2003 01:15:51 -0500

Yes.  So what document would this get described in?  Effectively the
protocol is doing an encoding optimization for certain types of timestamps.

When I replied I didn't read Nevil's message describing header offsets
vs the existing header synchronization timestamps.  I'd like to think
about this for a day or so, but his idea sounds better than what we
have.

I'm personally okay with using microsecond offsets instead of millisecond,
but I'd like to let others chime in.

Effectively in the protocol we have

  32 bit timestamps + header time aka "split time" gets you microsecond
  resolution start and end times.  The lifetime of a flow is limited
  to 4294 seconds with this encoding.

  64 bit timestamps + no header time gets you nanosecond resolution
  start and end times.  Flow lifetime is limited to 136 years.

In the information model

  dateTimeMsec (microsecond resolution)

  dateTimeNsec (nanosecond resolution)

mark