Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

From: "Carter Bullard" <[email-masked]>
Date: Mon, 8 Dec 2003 18:00:04 -0500

I would hope so.
Carter

-----Original Message-----
From: Nevil Brownlee [mailto:[email-masked]]
Sent: Monday, December 08, 2003 5:07 PM
To: [email-masked]
Cc: [email-masked]
Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

Hello Carter:

>    This is an interesting question, as it assumes that
> all the records are coming from a single IPFIX data
> source.  What happens if IPFIX is transporting data
> from multiple sources as they are received, but the
> timestamps in the various records are not synchronized?

Sounds like a good reason not to do it.
That would simply mean that timestamps in an IPFIX record would all
be in whichever of the four encodings the meter was configured to
send, right?

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Mon, 8 Dec 2003 21:57:14 -0800

Mark,

  A correction on my previous e-mail.  In rereading your suggestion I think
you are recommending the following encoding model and information set
restriction:

    - dateTimeMsec -> split 32-bit integer seconds since 1970 carried in 
                    IPFIX flow header and 32-bit integer count of (signed)
                    microseconds offset from the header.  **This would
                    imply that all events for a flow set using this
optimization 
                    would need to occur w/in about 2000 seconds before or
                    after the sentinel time in the header. [+/- 2000 seconds
                    is +/-2,000,000,000 usecs, around the signed int32
limit]

    - dateTimeNsec -> 64-bit time in NTP format with 32-bits of seconds
since
                    1900 (NOT 1970!) and 32-bit fraction of a second
component.
                    Both values occur in the flow record.

  i.e. the second high res format is fully encoded in the individual flow
record
and is not dependent on some timestamp in the flow header.

-- Jeff

-----Original Message-----
From: MEYER,JEFFREY D (HP-Cupertino,ex1) 
Sent: Monday, December 08, 2003 9:48 PM
To: 'Mark Fullmer'; MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Ipfix Wg' (E-mail)
Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

Mark,

  I see two different concerns in this disucssion.  One is information
modeling
and the other is encoding mechanisms.

  Your suggestion, which sounds plausible to me, is to only define two
encoding
models for IPFIX.  The first model, which is "split time" encoded requires a
32-bit
header and a 32-bit msec offset value.  The second encoding model is also
"split
time" in that the 32-bit header is seconds, but now seconds since 1900 (not
1970)
and the offset is NTP style fractional component (i.e. n/2**-32, where n is
the 
encoded 32-bit unsigned value).

  If this were the case, than IPFIX information models should likely only
care
about information elements which are dateTimeMsec and dateTimeNsec.  Other
transfer
formats or protocols may identify timestamps with different reolutions, but
there is 
a clear model for transforming from one type to the other.

  Given my preference for simplicity, I would support this proposal:

    - dateTimeMsec -> split 32-bit integer seconds since 1970 carried in 
                    IPFIX flow header and 32-bit integer count of (signed)
                    microseconds offset from the header.  **This would
                    imply that all events for a flow set using this
optimization 
                    would need to occur w/in about 2000 seconds before or
                    after the sentinel time in the header. [+/- 2000 seconds
                    is +/-2,000,000,000 usecs, around the signed int32
limit]

    - dateTimeNsec -> split 32-bit time in NTP format with 32-bits of
seconds 
                     since 1900 (NOT 1970!) stored in the IPFIX header.  And
32-bit 
                     fraction of a second (n*2**-32) component.  This
implies
                     that all timestamps must fall begin with the same
second
                     offset in a flow set.

-- Jeff

-----Original Message-----
From: Mark Fullmer [mailto:[email-masked]]
Sent: Monday, December 08, 2003 9:31 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

Do we really need four ways to represent time?  How about one 32 bit
format and one 64 bit format?

The existing NetFlow 32 bit msec Start/End time stamps + header
real time offset gives you msec real time resolution.  Implementations
that can not support / don't need msec resolution can just leave the low
order bits 0.  So this covers dateTime and dateTimeMsec with one 32
bit encoding.

64 bit NTP format format with nanosecond resolution covers microsecond
resolution implementations.  Again, implementations that want microsecond
just leave the low order fractional bits 0.

mark

On Mon, Dec 08, 2003 at 10:57:32PM -0500, MEYER,JEFFREY D (HP-Cupertino,ex1)
wrote:
> Resending to full mail list...
> 
> -- Jeff
> 
> From: MEYER,JEFFREY D (HP-Cupertino,ex1) 
> Sent: Monday, December 08, 2003 3:53 PM
> To: 'Nevil Brownlee'
> Hi,
>   Here's a proposal for encoding:
> 
>    - dateTime -> 32-bit integer seconds since 1970
>    - dateTimeMsec -> 64-bit integer msec since 1970 (Java format)
>    - dateTimeUsec -> split 32-bit integer seconds since 1970 carried in 
>                    IPFIX flow header and 32-bit integer count of (signed)
>                    microseconds offset from the header.  **This would
>                    imply that all events for a flow set using this
> optimization 
>                    would need to occur w/in about 2000 seconds before or
>                    after the sentinel time in the header. [+/- 2000
seconds
>                    is +/-2,000,000,000 usecs, around the signed int32
limit]
>    - dateTimeNsec -> 64-bit time in NTP format with 32-bits of seconds
since
>                    1900 (NOT 1970!) and 32-bit fraction of a second
> component.
>                    Both values occur in the flow record.
> 
>  
>   Note a separate issue is what type should the basic flowCreationTime
> and flowEndTime information elements should have?  Currently Info-02 has
> them
> as dateTime, but should probably be dateTimeMsec to reflect current
> information
> content of deployed netflow, or perhaps dateTimeUsec, if folks prefer to
> only use 32-bits to achieve some size reduction when passing many
> timestamps.
> 
>   My preference would be dateTimeMsec.
> 
> -- Jeff
> 
> > From: majordomo listserver 
> > [mailto:[email-masked]]On Behalf
> > Of Nevil Brownlee
> > Sent: Monday, December 08, 2003 2:07 PM
> > To: [email-masked]

---

From: "Carter Bullard" <[email-masked]>
Date: Tue, 9 Dec 2003 09:58:31 -0500

Hey Mark,
   IPFIX really does need to be able to support transport
of records from many sources at the same time.

Carter

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]] On Behalf Of
Mark Fullmer
Sent: Tuesday, December 09, 2003 1:27 AM
To: Carter Bullard
Cc: 'Nevil Brownlee'; 'Maurizio Molina'; [email-masked]
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

It depends on how smart the exporter is...A simple way to do this
would be to limit flows from each source to a single message.  Note, The
existing draft uses the term "packet" where it it should be message.

There appear to be some deeper problems with having multiple
data sources in a single IPFIX session though.  For example there
is currently no mechanism defined to describe where the data in a
template or multiple templates representing the same data source
come from.

The protocol can move the data but reporting tools that work with
it don't know how its related.

mark

On Mon, Dec 08, 2003 at 02:27:43PM -0500, Carter Bullard wrote:
> Hey Nevil,
> Sent: Monday, December 08, 2003 1:34 PM
> To: [email-masked]; 'Maurizio Molina'
> Hi Carter and Maurizio
> >    I have customers today that are using uSec granularity
> > exclusively and we're moving to nSec granularity for some
> > applications shortly.  So, these resolutions are not exotic
> > for my customers, they are the norm.
>
> OK, by all means let's keep all four of Jeff's timestamp formats.
>
> Now what about the notion of having a dateTimeUsec timestamp in the
> header and 32-bit offsets from that for timestamps in flow groups?
> Would that save enough space to be useful?
>
> CHeers, Nevil
>

--
Help        mailto:[email-masked] and say "help" in message
body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "Carter Bullard" <[email-masked]>
Date: Tue, 9 Dec 2003 10:04:52 -0500

Gentle People,
There will always be problems with using offsets,
because there are flows that live longer than the offset
rollover period, no matter what the offset scheme is.

You will need to support both Absolute and Relative
timestamps for IPFIX.

You should be able to support Unix timeval timestamps,
simply because tcpdump() is a significant source of
raw data for flow modelers.  Why waste the time
converting from timevals, that are in the packet stream
to begin with?

Carter

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]] On Behalf Of
MEYER,JEFFREY D (HP-Cupertino,ex1)
Sent: Tuesday, December 09, 2003 1:58 AM
To: 'Mark Fullmer'; MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: ''Ipfix Wg' (E-mail)'
Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

Hi,

  My preference would be milliseconds with a signed quantity.

  This gives you about +/- 2147 seconds on either side of the sentinel time.
If you do not use a signed offset it forces a flow record to be adjusted
"backward" so that all time offsets are positive (could be tricky for
FlowCreationTime).

  Using microseconds would yield only +/- 2.14 seconds which is a pretty
narrow window for grouping flows.

-- Jeff

-----Original Message-----
From: Mark Fullmer [mailto:[email-masked]]
Sent: Monday, December 08, 2003 10:18 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: ''Ipfix Wg' (E-mail)'
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

Microsecond might be okay...

mark

On Mon, Dec 08, 2003 at 10:04:48PM -0800, MEYER,JEFFREY D (HP-Cupertino,ex1)
wrote:
> OK,
>
>   I'll get it right yet...
>
>   It should say for dateTimeMsec   "...32-bit integer count of (signed)
>   millisecond offset... [not microsecond]"

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 9 Dec 2003 14:03:45 -0500

I would say "ClockPrecision" is another "Option Information Element",
which could
be transmitted by an exporter in a option flow record, rather than try to
infer
it from precision.  

  If we treat the 32 bit quantity as signed usecs, then if you only have
msec clock accuracy, your values would always be in increments of 1000.
(e.g. 32000, -27000, 1000...).  If you only have second precision you
increment/decrement in units of 1 million.

  Alternatively you specify time with 64-bit NTP style.  Again populating to
the available precision of the system.  I.e. if you only have a second
resolution clock, then the 32-bit fraction would always be 0.

-- Jeff

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Maurizio Molina
Sent: Tuesday, December 09, 2003 4:32 AM
To: Mark Fullmer
Cc: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

>just leave the low order fractional bits 0.
I think this opens an interesting issue:
Should IPFIX explicitly define a method for letting the exporter 
indicating the precision of its clock, e.g. via this zero-padding you 
mention?
In this case, how do we deal with the case the exporter has, for 
example, millisecond resolution but it has by chance  to fill a record 
exacly with three zeros in the fractional part, i.e. 12345.000? Should 
IPFIX specify that it has to force it 12345.001? Or should this 
precision indication be sent once and for all, e.g. in an option record? 
Or shouldn't it be sent at all?
Maurizio

>
>
>mark
>
>On Mon, Dec 08, 2003 at 10:57:32PM -0500, MEYER,JEFFREY D
seconds
limit]
since
body
body

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 9 Dec 2003 14:56:21 -0500

To the rest of the list.

-----Original Message-----
From: MEYER,JEFFREY D (HP-Cupertino,ex1) 
Sent: Tuesday, December 09, 2003 11:45 AM
To: 'Mark Fullmer'
Subject: RE: [ipfix] [issue] INFO-24 High-res time encoding

Mark,

  Here's some pseudo code for how this might work.

-- Jeff

//  pseudo code for microsecond offset in IPFIX encoded flow records.
//

   struct flow{
      uint32  tv_sec;
      uint32  tv_usec;
      uint32  numbytes;
      ...  // other information elements...
   };
   
   struct flow flowtable [MAX_TABLE_SIZE];
   int lastflowindex = -1;

   writeflows() {
   
      if (lastflowindex < 0) return;
      
      // simply take the second field from the first available flow
      // and make this the base time for this collection of flows.
      uint32  base_sec = flowtable[0].tv_sec;
      
      writeheaderToSocket(base_sec); // put 32-bit second value in header
      
      for (int i=0; i<=lastflowindex; i++){
      
         int32 offset = (flowtable[i].tv_sec - base_sec) * 1000000 +
flowtable[i].tv_usec;
         writeint32ToSocket(offset);  // put the 32-bit time offset in the
record.
         
         // write other information elements...
      }
   }

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Mark Fullmer
Sent: Tuesday, December 09, 2003 10:53 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

How do you envision an exporter implementing "split time?"

Foreach flow the metering could use a unix timeval structure internally,
ie tv_sec and tv_usec.  Then when a collection of flows is ready for
export the exporter would need to run down the list of flows, pick
a suitable "header time" then adjust all the flow Start and End times
to the correct usec offset.

This seems like a lot of extra work compared to the existing NetFlow
method where the flow times are not offsets and no special work
needs to be done during export.

Do you see an easier way to implement this?

mark

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 9 Dec 2003 15:11:54 -0500

Sebastian,

  Comments inline <Jeff></Jeff>

-- Jeff

-----Original Message-----
From: Sebastian Zander [mailto:[email-masked]]
Sent: Tuesday, December 09, 2003 3:52 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

Jeff,

some questions on the delta timestamps:

Why should the microsecond deltas be _signed_?

<Jeff>
If I have a collection of flows which have both FlowCreation and FlowEnd
time,
then rather than scanning the list for the "lowest value", you simply pick
the
first time value.  Signed allows you to avoid finding the lowest timestamp.

Calculating the offset time is done by looking at the difference between
the current time stamps second value and the sentinel second value in 
the header, multiply the SIGNED difference by 1000000 and add to the 
current tv_usec value.
</Jeff>

What happens when the flow is longer than 2147 seconds? Is a new flow
started? Is there a timestamp refreshment e.g. every x export includes
an absolute (sentinel) timestamp or is there an absolute timestamp
if the exporters detects the duration is > 2147?

<Jeff>
My experience with netflow is that there are active flow timers which
"chop" flows up for long duration sessions.  I think typical values are
30 minutes = 180 seconds.

So this common behavior falls easily within the window.
</Jeff>

What happens in case of unreliable transport when packets with
sentinel time get lost?
<Jeff>
The sentinel time is sent in the header of ALL flowrecord sets, so this
should not be an issue.
</Jeff>

In my view delta timestamp support seems to be a bit more than just simple
encoding and I don't fully understand how it works now.
<Jeff>
Maybe this made it clearer.  I agree that split time stamps are harder to
work with than having the full info available in one place.  And if you
split there are a couple of ways to do it.  I think the offset method I
outlined may be simplest, alternatively, something more like the current
NFv9 time encoding may be preferred, but it currently only has msec
resolution.
</Jeff>

At the moment I tend to agree in keeping all four encodings.
<Jeff>
I think so as well.  It's just a question of the details of each encoding.
I would recommend simple encoding for dateTime and dateTimeMsec (i.e. 32-bit
or 64-bit count of seconds or msecs since 1970), and leave the dateTimeUsec
and dateTimeNsec for whatever the group hammers out.
</Jeff>

Cheers,

Sebastian

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> Resend to list...
> Sent: Monday, December 08, 2003 11:57 PM
> To: 'Mark Fullmer'
>   You're right.  It's too late at night and microseconds give you a
> reasonable window.
> 
>   So the two IPFIX information types would be:
> 
>   dateTimeUsec - signed microsecond reslolution, yielding a +/- 2147
second
> window in IPFIX encoding model.
> 
>   dateTimeNsec - nanosecond resolution, encoded using 64-bit NTP format
with
> about 136 year rollover.
> 
>   Both formats have a relatively narrow (on a geologic scale) time window.

> 
>   Java's 64-bit dateTimeMsec format increases the 64-bit rollover interval
> to +/- 292M years approximately.  I.e. A range of 2**64 msec, treating
this
> as a signed quantity means it should roll over in  +/-
> (10**-3)*(2**63)/60/60/24/365 ~= 292,471,208.
> 
>   So as a general purpose time representation, dateTimeMsec has some
> desirable properties.  But it may be that time for IPFIX is typically sent
> in dateTimeUsec or dateTimeNsec, but is persisted or transformed to
> dateTimeMsec for further processing, dependent on application.  It still
may
> make sense to define dateTimeMsec encoding according to Java time.  
> 
>   I would agree that basic dateTime (second resolution) could logically be
> transmitted using the dateTimeUsec encoding and simply stripping off the
> subsecond resolution.  So there may be no need to define an encoding for
> this in IPFIX, but it would satisfy completeness.
> 
>   How about recommending the use of dateTimeUsec or dateTimeNsec, but
> defining encodings for all four resolutions?
> 
>   In particular, the default flowCreationTime and flowEndTime would be
> redefined in the current information model as dateTimeUsec.
> 
> 
> -- Jeff
> 
> 
> Sent: Monday, December 08, 2003 11:19 PM
> I think you're off by one SI unit, or I'm sending e-mail too late at
> night...
> 
> Millisecond : 10^-3 seconds
> Microsecond : 10^-6 seconds
> Nanosecond  : 10^-9 seconds
> 
> 2^32 * 10^-3 = 4294967 seconds
> 2^32 * 10^-6 = 4294 seconds
> 2^32 * 10^-9 = 4 seconds
> 
> I'm not sure running down the list of flows per export packet to fix
> the timestamps is that big a deal, but it probably doesn't matter.  An
> implementation that wants the full range can put the header/sentinel
> time in the middle of the range.
> 
> mark
> 
> On Tue, Dec 09, 2003 at 01:58:04AM -0500, MEYER,JEFFREY D
(HP-Cupertino,ex1)
>>  This gives you about +/- 2147 seconds on either side of the sentinel
> 
> time.
> 
>>On Mon, Dec 08, 2003 at 10:04:48PM -0800, MEYER,JEFFREY D
> 
> (HP-Cupertino,ex1)
> 
>>wrote:
>>
>>>OK,
>>>
>>>  I'll get it right yet...
>>>
body

-- 
Sebastian Zander                         E-mail: [email-masked]
Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany
www.fokus.fraunhofer.de/usr/sebastian.zander

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 10 Dec 2003 10:06:56 -0800

Maurizio,

  Thanks for reviewing the threads to date.

  At this point, I believe there is an acknowledgement that all four types
may be used.  However for the default flow record one of these four needs to
be chosen.  

  I think there is also agreement that two of the types should be encoded as

follows:

   dateTime - 32-bit unsigned seconds since 1970
   dateTimeNsec - NTP format time which occupies 64-bits.  High order
32-bits
     represent seconds since 1900 (not 1970!) and low order 32-bits
represent
     the fractional seconds (fraction ranging from 0 - (2**32-1)/2**32.

  I would recommend that the following encoding also be used:

   dateTimeMsec - 64-bit unsigned milliseconds since 1970

  Finally for dateTimeUsec, I would recommend that some form of split time 
be used, where base time information is stored in a flow header and some
offset is stored in 32-bits.

  Also making dateTimeUsec the default time will provide more compact 
flow records with high resolution time.

  I've proposed a specific model for dateTimeUsec, based on a signed
microsecond component, but I don't think there is consensus on this 
point, although I'm not sure what the specific proposed alternatives
are (perhaps using the NFv9 model?).  The one issue I have w/ NFv9 is that
I don't think it achieves the optimal space efficiency and time
resolution tradeoff compared to the microsecond proposal.

Regards,

  Jeff Meyer

-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Maurizio Molina
Sent: Wednesday, December 10, 2003 7:59 AM
To: 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] [issue] INFO-24 High-res time encoding

> For the other 3 "high-res" 64 bit discussed encoding schemes, the 
> "desirable" characteristics I saw discussed so far were
> 1) Allowed maximum resolution
> 2) possibility to "split" into 32bit (header) + 32 bit (flow record) 
> so that to save space in export msg
> 3) maximum flow lifetime
>
> According to these 3 creteria dateTimeNsec is certainly better than 
> dateTimeUsec.

I went through this thread once again, and I realize that this my 
statement is clearly incorrect, as dateTimeNsec (NTP encoding) cannot be 
split  between the header and the flow record. Then it is a good reason 
to support all 4 types of encoding.
Maurizio