Subject: Re: [IPFIX] draft-ietf-ipfix-protocol-rfc5101bis-03

From: Andrew Feren <[email-masked]>
Date: Tue, 04 Dec 2012 13:22:57 -0500

Hi All, Atsushi,

On 11/25/2012 08:41 AM, ATSUSHI KOBAYASHI wrote:
> Dear all, Nevil,
>
> I checked the draft, I found some comments and editorial issue.
> Please see in-line.
>
> Regards, Atsushi
[ snip ]
> Comment#3:
>
> I could not understand the following paragraph. If you want to describe
> another way, please explain it in more detail.
I think Atsushi makes a good point here.
>>     Put another way, a Template only describes Records contained in IPFIX
>>     Messages with the same Export Time as the IPFIX Message containing
>>     Template Record, or a subsequent export time. Likewise, a Template
>>     Withdrawal is only in effect for IPFIX Messages with the same Export
>>     Time as the Template Withdrawal, or a subsequent Export Time.
Perhaps something like...

    Put another way, a Template only describes Data Records contained
    in IPFIX Messages with the an Export Time greater than or equal to
    the Export Time of the IPFIX Message containing Template Record.
    Or until a Template Withdrawal is received. Likewise, a Template
    Withdrawal is only in effect for IPFIX Messages with the an Export
    Time greater than or equal to the Export Time of the IPFIX Message
    containing Template Withdrawal.  Or until a new Template with the
    withdrawn Template ID is received.

-Andrew

---

From: Paul Aitken <[email-masked]>
Date: Tue, 04 Dec 2012 20:11:49 +0000

Andrew, all,

There's a loophole in both definitions: if a collector receives data 
records with export time = T2, then a Template with export time = T1 
(where T1 <= T2), the Template applies to the previously-received data 
records. This could be due to UDP out of order, or delivery in different 
SCTP streams.

However when the collector receives data records at T1 it doesn't know 
that a template is coming at T2, so its behaviour is unclear, especially 
if it already has a template for those data records. That could be an 
earlier template which no longer applies.

So the definition might benefit from some words about reception order.

P.

On 04/12/12 18:22, Andrew Feren wrote:
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Wed, 5 Dec 2012 11:59:52 +0100

hi, Paul, Andrew, all,

Agreed, though it is not clear to me how to really account for time at receiver here. Actually, I think this makes more sense to explain in terms of "template lifetime" as opposed to "time of effect of a template or withdrawal"... How about the following?

NEW:

  Put another way, a Template describes Data Records contained in IPFIX Messages
  with an Export Time between the Export Time of the IPFIX Message containing the
  Template Record and the end of the Transport Session, inclusive, if that
  Template is not subsequently withdrawn.

  If a Template is subsequently withdrawn, then that Template describes Data
  Records contained in IPFIX Messages with an Export Time between the Export Time
  of the IPFIX Message containing the Template Record and the Export Time of the
  IPFIX Message containing the Template Withdrawal, inclusive.

Stepping back further, the real issue here is that you can only have two of the following three things:

(1) template identifier reuse,

(2) template transmission without reliable ordering (i.e., templates in multiple SCTP streams, or UDP as a transport protocol),

(3) verifiably correct interpretation of data records.

I strongly suspect this is why we have seen no interest in five years of interoperability testing in (1), and interest in (2) only because IPFIX is envisioned for deployment to replace UDP transport of older versions of NetFlow, where the network is either provisioned to minimize loss, or the processes around the collection and processing of the data have been adjusted to the fact that UDP is simply not reliable.

And it looks we fell into the trap again, just like we did in 2007 with 5101, of trying to pretend this is a fixable problem. So I think in addition we probably need a warning label on template reuse -- even if you follow all these rules, a stream of IPFIX messages compliant with the protocol may come out the other side of the network in an order which makes it difficult to determine what the exporter meant, so practically speaking reuse needs various knobs that admins can tune to determine timing delays for reuse, and those admins need to be conservative when tweaking them, or tolerant of lost or corrupted data. I'd replace the final paragraph of section 8.2 with an actual warning; something like:

NEW:

    Note that, even if sent in-order, IPFIX Messages containing Template
    management actions could arrive at the Collecting Process out-of-order,
    i.e. if sent via UDP or via different SCTP streams. Template Withdrawals
    and subsequent reuse of Template IDs, in particular, can significantly
    complicate the problem of determining Template lifetimes at the Collecting
    Process. Therefore, Collecting Processes MAY implement a buffer to handle
    out-of-order Template management events; this buffer, if implemented,
    SHOULD be configurable to impart a delay on the order of the maximum
    reordering delay experienced at the Collecting Process.

This, I think, handles the reception order problem, as well.

Thoughts?

Best regards,

Brian

On 4 Dec 2012, at 21:11, Paul Aitken <[email-masked]> wrote:

> Andrew, all,
> 
> There's a loophole in both definitions: if a collector receives data records with export time = T2, then a Template with export time = T1 (where T1 <= T2), the Template applies to the previously-received data records. This could be due to UDP out of order, or delivery in different SCTP streams.
> 
> However when the collector receives data records at T1 it doesn't know that a template is coming at T2, so its behaviour is unclear, especially if it already has a template for those data records. That could be an earlier template which no longer applies.
>

---

From: Andrew Feren <[email-masked]>
Date: Fri, 07 Dec 2012 17:54:19 -0500

Hi Brian, all,

That all makes sense.  I particularly like the addition of a disclaimer.

-Andrew