Subject: [IPFIX] template lifetime

From: Paul Aitken <[email-masked]>
Date: Thu, 18 Aug 2011 10:34:10 +0100

Dear IPFIX experts,

RFC5102, section 10.3.7 ("Collecting Process") says:

    The Collecting Process MUST associate a lifetime with each Template
    (or another definition of an identifier considered unique within the
    Transport Session) received via UDP.  Templates (and similar
    definitions) not refreshed by the Exporting Process within the
    lifetime are expired at the Collecting Process.

    ...

    The Template lifetime at the Collecting Process MUST be at least 3
    times higher than the Template refresh timeout configured on the
    Exporting Process.

How does a collector determine the correct lifetime to associate with 
each Template, and how does it know "the Template refresh timeout 
configured on the Exporting Process." ?

What are the default and acceptable range of lifetime values?

Should we have a mechanism which allows an Exporting Process to report 
Template lifetimes to the Collecting Process?
eg, by exporting an option of { scope = templateId, lifetime = 
lifeTimeUnits }, where lifeTimeUnits = u32 milliseconds - which allows 
1ms to 49.7 days, with 0 = infinite?

Thanks,
P.

---

From: Brian Trammell <[email-masked]>
Date: Thu, 18 Aug 2011 13:08:53 +0200

Hi, Paul,

My personal opinion is that, if we're not going to deprecate UDP with prejudice (which I realize to my sorrow we cannot do), we should jettison all this template lifetime nonsense. We spent an inordinate amount of time on 5101/5102 the first time around trying to come up with sensible guidance, then gave up and basically said "template lifetimes are configured out of band according to operational environment and application constraints", which is essentially RFCese for "here, you make this work." Which is actually sensible: there is no default that covers edge-network, core-network, and IPFIX-as-SNMP-replacement-on-the-moon use cases.

Half a decade later, there has never been a successful interop test to my knowledge demonstrating the template management (expiry, withdrawal, reissue) features in IPFIX.

Most export applications use one, two, twenty, thirty templates, and that's all they'll ever use, because those templates reflect the core data model in use by the application, and the export app wants to waste as little effort in export transcoding as possible. Sure, collectors in the wild have to handle more templates than that, but it doesn't take long as a collector operator (or even implementor) to realize that practically speaking, nobody is ever going to reuse an ID on you without dropping a session, and just setting your template expiry time to something practically infinite . You interpret templates as they come in, you replace old with new, and everything works unless you find yourself in an insanely rare corner case -- and if you do, you write something to the log and wake up the human.

Trying to export template lifetime information from EP to CP adds complexity while pretending that the system as specified works better than an infinite-lifetime-at-collector with silent replacement, which IMO is not the case.

I'd propose we use the RFC5655 File Reader template rules (accept anything that would be legal on any transport in any combination), but I don't know the right way to do this in 5101bis: perhaps filing technical errata on all the "MUST drop the session" language on the grounds that 1. a receiver cannot always reliably signal session failure to a transmitter and 2. it violates the "be liberal in what you accept" principle of protocol design.

Cheers,

Brian

On Aug 18, 2011, at 11:34 AM, Paul Aitken wrote:

> How does a collector determine the correct lifetime to associate with each Template, and how does it know "the Template refresh timeout configured on the Exporting Process." ?
> 
> Should we have a mechanism which allows an Exporting Process to report Template lifetimes to the Collecting Process?
> eg, by exporting an option of { scope = templateId, lifetime = lifeTimeUnits }, where lifeTimeUnits = u32 milliseconds - which allows 1ms to 49.7 days, with 0 = infinite?
> 
> Thanks,
> P.
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Gerhard Muenz <[email-masked]>
Date: Thu, 18 Aug 2011 21:15:38 +0200

Hi Paul,

As Brian says, this problem is quite an old story.

I agree with you: at a certain point, every careful reader of the IPFIX 
documents will recognize that Option Templates are defined for a lot of 
things, but strangely not for the Template refresh timeout, although 
this parameter seems so essential. I do not know the details why this 
has not been added to the standard.

I agree with Brian: in practice, the knowledge of the Template refresh 
timeout is usually not essential because Collectors can just operate 
with a very long or even infinite lifetime (RFC 5101 does not seem to 
disallow this).

A default minimum lifetime is indirectly defined by RFC 5101:
"The default value for the frequency of the (Options) Template 
transmission is 10 minutes."
This means that, by default, the lifetime at the collector is 30 minutes 
or longer.

Regards,
Gerhard

On 18.08.2011 13:08, Brian Trammell wrote:
> Hi, Paul,
>
> My personal opinion is that, if we're not going to deprecate UDP with
> prejudice (which I realize to my sorrow we cannot do), we should
> jettison all this template lifetime nonsense. We spent an inordinate
> amount of time on 5101/5102 the first time around trying to come up
> with sensible guidance, then gave up and basically said "template
> lifetimes are configured out of band according to operational
> environment and application constraints", which is essentially RFCese
> for "here, you make this work." Which is actually sensible: there is
> no default that covers edge-network, core-network, and
> IPFIX-as-SNMP-replacement-on-the-moon use cases.
>
> Half a decade later, there has never been a successful interop test
> to my knowledge demonstrating the template management (expiry,
> withdrawal, reissue) features in IPFIX.
>
> Most export applications use one, two, twenty, thirty templates, and
> that's all they'll ever use, because those templates reflect the core
> data model in use by the application, and the export app wants to
> waste as little effort in export transcoding as possible. Sure,
> collectors in the wild have to handle more templates than that, but
> it doesn't take long as a collector operator (or even implementor) to
> realize that practically speaking, nobody is ever going to reuse an
> ID on you without dropping a session, and just setting your template
> expiry time to something practically infinite . You interpret
> templates as they come in, you replace old with new, and everything
> works unless you find yourself in an insanely rare corner case -- and
> if you do, you write something to the log and wake up the human.
>
> Trying to export template lifetime information from EP to CP adds
> complexity while pretending that the system as specified works better
> than an infinite-lifetime-at-collector with silent replacement, which
> IMO is not the case.
>
> I'd propose we use the RFC5655 File Reader template rules (accept
> anything that would be legal on any transport in any combination),
> but I don't know the right way to do this in 5101bis: perhaps filing
> technical errata on all the "MUST drop the session" language on the
> grounds that 1. a receiver cannot always reliably signal session
> failure to a transmitter and 2. it violates the "be liberal in what
> you accept" principle of protocol design.
>
> Cheers,
>
> Brian
>
>> The Collecting Process MUST associate a lifetime with each
>> Template (or another definition of an identifier considered unique
>> within the Transport Session) received via UDP.  Templates (and
>> similar definitions) not refreshed by the Exporting Process within
>> the lifetime are expired at the Collecting Process.
>>
>> ...
>>
>> How does a collector determine the correct lifetime to associate
>> with each Template, and how does it know "the Template refresh
>> timeout configured on the Exporting Process." ?
>>
>> Should we have a mechanism which allows an Exporting Process to
>> report Template lifetimes to the Collecting Process? eg, by
>> exporting an option of { scope = templateId, lifetime =
>> lifeTimeUnits }, where lifeTimeUnits = u32 milliseconds - which
>> allows 1ms to 49.7 days, with 0 = infinite?
>>
>> Thanks, P. _______________________________________________ IPFIX
>> mailing list [email-masked]
> _______________________________________________ IPFIX mailing list
> [email-masked] https://www.ietf.org/mailman/listinfo/ipfix

---

From: Andrew Feren <[email-masked]>
Date: Thu, 18 Aug 2011 15:28:36 -0400

I agree with Brian.

You could export an options template with template lifetime information, 
but I don't think it solves any problems.  The exporter still can't know 
when it is safe to reuse template ids since there is no upper bound on 
the lifetime that the collector MUST associate with a template.

The way I see it this means either

1) the exporters also need to be notified of the lifetime used by the 
collector (how?)

2) template lifetime is a collector implementation detail and templates 
are silently replaced, logging problems as necessary

Option 2 is a lot simpler and works well in practice.

-Andrew

---

From: Paul Aitken <[email-masked]>
Date: Thu, 18 Aug 2011 20:36:36 +0100

Brian, Gerhard,

Thanks for your replies. I know it's an old problem, but someone asked 
what the default refresh interval should be.

So it seems we should either define a mechanism for Exporters to inform 
Collectors what their default refresh interval is (eg, define a new 
standard options template), or re-work (or even remove?) the template 
timeout sections.

Currently this is a gap which allows an Exporting Process and a 
Collecting Process to both claim to be IPFIX compliant, while being 
quite un-interoperable.

P.

---

From: Paul Aitken <[email-masked]>
Date: Thu, 18 Aug 2011 22:46:32 +0100

Andrew,

> I agree with Brian.
>
> You could export an options template with template lifetime 
> information, but I don't think it solves any problems.  The exporter 
> still can't know when it is safe to reuse template ids since there is 
> no upper bound on the lifetime that the collector MUST associate with 
> a template.
>

Out of band configuration.

However this implies that one size must fit all, which is probably not 
the case. Exporters only need to re-send their templates in proportion 
to the data loss they can tolerate. For some, that may mean a template 
in every data packet (though they should clearly be using SCTP). For 
others, once in 24 hours may be quite sufficient.

> 2) template lifetime is a collector implementation detail and 
> templates are silently replaced, logging problems as necessary

I think the idea of template lifetime is to ensure that templates expire 
and are deleted before they're ever re-used - so there's never any 
redefinition.

> Option 2 is a lot simpler and works well in practice.

If we accept silent re-definition then lifetime is a non-issue.

Thanks,
P.