Subject: Re: [IPFIX] timestamps, exporters, and other animals (fwd)

From: Josh Bailey <[email-masked]>
Date: Wed, 12 Oct 2011 00:47:27 -0700 (PDT)

Hi Juergen;

I agree with your statement about SNMP not being designed for subsecond 
precision. And I definitely agree that there are things you can do to 
minimise potential delays and that often those delays are implementation 
specific.

I don't agree though that sysUpTime is when the packet left the agent; 
especially if the agent is running slowly for whatever reason, or if the 
sysUpTime reply happens to be part of a packet containing replies for 
other variables (perhaps from different CPUs in a typical distributed CPU 
router).

I also don't agree that cycles are usually measured in minutes. In my 
experience cycles are very much shorter on average.

As you say having a timestamp per counter while costly is ideal in some 
sense. However I am looking to explore different tradeoffs; for example, a 
timestamp followed by a list of OIDs all current at that same timestamp.

I am aware of counter implementations that could do such a "snapshot" 
operation for example.

Thanks,

>>
>> Hi Juergen;
>>
>> I assert that polling the sysUpTime does not take care of the core
>> problem, because you cannot know when sysUpTime itself was sampled
>> (eg. you received it after 1.5s because of network delay and because
>> the packet was queued in the control plane kernel because the
>> control plane CPU was busy).
>
> The SNMP agents controls sysUpTime - so sysUpTime is pretty much the
> time the packet left the agent. If you talk about delays in the
> instrumentation, that is the delay to read the register, then you are
> correct. But once again, SNMP polling cycles are usually counted in
> minutes.
>
>> I don't mind so much that the reply was delayed (though I like fast
>> replies!), but I do mind that there can be a very large uncertainty.
>
> Very large is relative to the precision you want. SNMP was not
> designed with subsecond precision in mind.
>
>> There is also the overhead of requesting sysUpTime itself, not to
>> mention when walking a table, you may time slew over the course of
>> walking the table.
>
> For any counter, you need discontinuity detection. The way you walk a
> table has indeed a big impact in terms of data consistency - the
> recommendation here is to use getbulk and to walk the columns of
> interest concurrently.
>
>> I understand definitely re caching hardware counters, etc, and I
>> think that's fine (and necessary in a system with multiple NMSes),
>> and I don't mind that the counters are cached, but I do want to know
>> when they were cached for me to recover maximum temporal
>> information.
>
> SNMP agents typically do not help you with that - caching is rather
> something implementation specific. And yes, operators have complained
> about SNMP counters on some boxes being less "precise" than what the
> CLI shows. I am no way saying SNMP is a model to choose - I just
> wanted to help making it clear what SNMP does and what not.
>
> If you want high accuracy, then every counter reading must be
> timestamped exactly when the counter is read. This of course has a
> price.
>
> /js
>
> -- 
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
> Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
>
>
>

--
Josh Bailey

---

From: Brian Trammell <[email-masked]>
Date: Wed, 12 Oct 2011 10:13:32 +0200

Hi, Josh, all,

As I read it, one of the wins of MIB Variable Export is that you get the whole timestamp machinery of IPFIX for free, including absolute timestamps wherein individual records can have observation timestamps down to nanosecond precision completely independent of the export timestamps. Relative timestamps can be expressed either as negative offsets to the export time (which can be a little tricky to implement, mind) or as positive offsets to the system initialization time (itself exported via options)

Now, as you'll see elsewhere on the list, the flexible timestamps need a little work -- the microsecond and nanosecond types in particular are confusingly defined in a way that has led to (not-yet-verified) interoperability failures; the discussion about fixing this in 5101bis/5102bis is elsewhere on the ipfix list.

Simply because an exporter has the _tools_ to export accurate timestamps, of course, does not mean that it can (e.g. due to implementation constraints) or will (e.g. due to faults or simple laziness). And system initialization based relative timestamps can have their own subtle pitfalls as in NetFlow V9 (on which see our paper in PAM this year, "Peeling Away Timing Error in NetFlow Data", author copy at ftp://ftp.tik.ee.ethz.ch/pub/people/martibur/Netflow-Timing-PAM2011.pdf -- the cyclic error from lost export time precision are V9-specific, but particularly the drift and delay components of the error there are universally applicable to any such arrangement, I think)

But the protocol here should give you the room to do what you need with respect to timing at the implementation level... WRT impacts on documents, would it be helpful to have a more thorough handling of IPFIX timestamping in the MIB Export draft (i.e., as an introduction for SNMP people)?

Cheers,

Brian

On Oct 12, 2011, at 9:47 AM, Josh Bailey wrote:

> 
> Hi Juergen;
> 
> I agree with your statement about SNMP not being designed for subsecond precision. And I definitely agree that there are things you can do to minimise potential delays and that often those delays are implementation specific.
> 
> I don't agree though that sysUpTime is when the packet left the agent; especially if the agent is running slowly for whatever reason, or if the sysUpTime reply happens to be part of a packet containing replies for other variables (perhaps from different CPUs in a typical distributed CPU router).
> 
> I also don't agree that cycles are usually measured in minutes. In my experience cycles are very much shorter on average.
> 
> As you say having a timestamp per counter while costly is ideal in some sense. However I am looking to explore different tradeoffs; for example, a timestamp followed by a list of OIDs all current at that same timestamp.
> 
> I am aware of counter implementations that could do such a "snapshot" operation for example.
> 
> Thanks,
> 
>>> 
>>> Hi Juergen;
>>> 
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix