Subject: [IPFIX] RFC 5101bis: CP MUST Close and Simplified Template Management

From: Brian Trammell <[email-masked]>
Date: Mon, 2 Apr 2012 13:33:53 +0200

Greetings, all,

Following up on the discussion at the WG meeting in Paris on Thursday, I'd like to clarify the issues with CP must close and proposed simplified template management in RFC5101bis, and start the discussion thereon.

The _main_ problem IMO with RFC5101 template management is the language stating the Collecting Process MUST close the session on the following conditions:

1. Receiving a malformed message. (What a "malformed message" is is not defined, though it is explicitly noted that non-zero Set padding does not constitute a malformed message.)

2. Receiving a Template for a given Template ID which has been previously defined without having been withdrawn (on SCTP and TCP).

3. Receiving a Template Withdrawal for a Template ID which is not presently defined (on SCTP and TCP).

The problems with this arrangement are as follows:

1. There is no way for a UDP collector to effectively terminate a session, as over UDP, IPFIX is completely unidirectional. A CP can certainly shutdown whenever it wants, but aside from ICMP Port Unreachable or Host Unreachable messages, the EP cannot detect this.

2. The "error" signal is useless to the EP, because the EP cannot distinguish CP close due to error versus CP close due to CP shutdown.

3. After having the session terminated by the CP, there is no effective remedy at the EP to restart, as all the conditions leading to a CP close are effectively "badly implemented EP which doesn't follow the protocol on three specific points enumerated above.

So, I propose removing the "CP MUST close" language, and replacing it with "CP MUST ignore / SHOULD log" for malformed messages or other anomalies. 

The other issue with template management in RFC5101 is that it is unnecessarily complex, especially on UDP, and that template management operates completely differently over UDP than over TCP/SCTP, and SCTP has some timing issues related to templates on multiple streams. UDP template management, in particular, requires hand tuning of timeouts to requirements, and will generally either "just work" (on properly provisioned links) or not work at _all_ due to packet loss in transit or insufficiently sized buffers at the endpoints.

I'd like to fix this as well in 5101bis, especially as any template management scheme other than "define templates at session start and export them" has never been successfully interop'd. 

The changes are defined in Section 8 of RFC5101bis, and were presented at the WG meeting. Essentially, this is taken from RFC 5655 (IPFIX files), and states that on all transports:

1. Templates MAY be resent at any time on any transport; they MUST be sent on UDP.

2. UDP Collecting Processes MAY expire Templates not resent within a configurable delay to save resources.

3. Template IDs SHOULD be withdrawn using a Template ID when they will no longer be used. 

4. Template Withdrawals MAY be sent on UDP.

5. Template IDs MAY be reused without withdrawal or expiration by simply sending a new Template.

6. Template management actions MUST be sequenced in IPFIX Message Header Export Time order.

This sixth point is (perhaps poorly) addressed in Section 8.3 of 5101bis. To clarify:

A. an EP MUST ensure when defining a Template that it exports no messages containing data sets described by the new Template with export times _earlier_ than the message containing the template set.

B. an EP MUST ensure when withdrawing a Template ID that it exports no messages containing data sets described by the withdrawn template with export times _later_ than the message containing the withdrawal.

C. an EP MUST ensure when reusing a Template ID that the following actions are uniquely sequenceable in the following order by the Export Time of the containing messages:

   i. last export of a data set described by the old template
   ii. withdrawal (if present)
   iii. definition of new template for reused template ID
   iv. first export of a data set described by the new template

D. Sequencing actions within a Message counts (i.e., a message could contain a template followed by data sets described thereby); it is not yet clear to me how to account for (UDP) reordering of Messages with the same Export Time.

The intent in the new section 8 is that the SHOULD language is compatible with the old, complex template management in 5101, while 5101bis EPs MAY use the simplifications to ease multi-transport implementation. Note that points 1 (on SCTP, TCP), 4, and 5 would cause 5101 CPs which actually implement MUST close to do so.

There is a pending question before our new AD as to whether this is acceptably interoperable with respect to 5101bis advancing IPFIX to full standard.

Other questions, comments, concerns, flames, please, the floor is open. :)

Best regards,

Brian

---

From: Gerhard Muenz <[email-masked]>
Date: Mon, 02 Apr 2012 22:58:09 +0200

Brian,

See inline.

> 3. After having the session terminated by the CP, there is no effective remedy at the EP to restart, as all the conditions leading to a CP close are effectively "badly implemented EP which doesn't…

I do not see the link to conditions 1 to 3.

The three problems concern UDP. Conditions 2 and 3 do not affect UDP.

Regarding condition 1, I would assume that a message containing Data 
Sets for a Template which is unknown to the CP is not "malformed" in the 
sense of RFC5101. This is the only problem which can happen if UDP 
collector crashes. Due to the retransmission, it is not a problem.

> So, I propose removing the "CP MUST close" language, and replacing it with "CP MUST ignore / SHOULD log" for malformed messages or other anomalies.

This is ok. Collectors should be tolerant in what they receive.

> I'd like to fix this as well in 5101bis, especially as any template management scheme other than "define templates at session start and export them" has never been successfully interop'd.

I do not see that the changes proposed below solve any of the UDP 
issues. IPFIX over UDP will still depend on proper timeout configuration 
and good luck. Therefore, UDP exporters and collectors will still differ 
from TCP and SCTP exporters and collectors.

I think that it is a good idea to relax the UDP transport specification 
and to remove restrictions which require a distinct behavior from TCP 
and SCTP. However, I do not see any advantage in changing the current 
TCP and SCTP transport specifications for IPFIX.

I would prefer keeping changes small (e.g. restrict changes to UDP 
transport) in order to increase interoperability between RFC5101 and 
~bis. Changing the protocol just to make it easier for a few people to 
implement all transports is not a good argument because most transports 
are optional anyway.

> 1. Templates MAY be resent at any time on any transport; they MUST be sent on UDP.

I do not see how this helps UDP. I do not see any advantages for TCP and 
SCTP. I would keep the original spec and allow resending for UDP only.

> 2. UDP Collecting Processes MAY expire Templates not resent within a configurable delay to save resources.

Ok. This clarifies that infinite expiration timeouts are allowed.

> 3. Template IDs SHOULD be withdrawn using a Template ID when they will no longer be used.

I would keep the MUST for TCP and SCTP. I do not see any advantage in 
removing this rule.

> 4. Template Withdrawals MAY be sent on UDP.

Ok. Template Withdrawal does not hurt in case of UDP. Also, it can be 
used by the exporter to "close" the session.

> 5. Template IDs MAY be reused without withdrawal or expiration by simply sending a new Template.

This causes incompatibility with RFC5101 without offering any 
advantages. I would keep the mandatory withdrawal for TCP and SCTP.

> 6. Template management actions MUST be sequenced in IPFIX Message Header Export Time order.

This is a reasonable clarification.

> A. an EP MUST ensure when defining a Template that it exports no messages containing data sets described by the new Template with export times _earlier_ than the message containing the template set.

How can the EP ensure this in case of SCTP with multiple streams?
It is difficult for an application to obtain information from the SCTP 
socket when a message has left the interface card.
It's a question of when the exporter can assume that a message is exported.

> B. an EP MUST ensure when withdrawing a Template ID that it exports no messages containing data sets described by the withdrawn template with export times _later_ than the message containing the wi…

Same as above.

>     iv. first export of a data set described by the new template

Good idea.

Thanks,
Gerhard

> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Brian Trammell <[email-masked]>
Date: Tue, 3 Apr 2012 15:35:11 +0200

Hi, Gerhard,

Thanks for the comments; replies inline as per tradition...

> The three problems concern UDP. Conditions 2 and 3 do not affect UDP.

Indeed, it's unclear what CP must close means on UDP.

> Regarding condition 1, I would assume that a message containing Data Sets for a Template which is unknown to the CP is not "malformed" in the sense of RFC5101. This is the only problem which can happen if UDP collector crashes. Due to the retransmission, it is not a problem.

Correct. (Indeed, this is IMO further evidence that "malformed message" doesn't mean anything either. Of course, a CP has to protect itself from a malicious or clearly broken EP, but we do this in section 11.4, more or less.)

> I do not see that the changes proposed below solve any of the UDP issues. IPFIX over UDP will still depend on proper timeout configuration and good luck. Therefore, UDP exporters and collectors will still differ from TCP and SCTP exporters and collectors.

Indeed. However, it would be nice if the documents acknowledged this; the current 5101 template management seems based on rules of thumb and give the impression that UDP template management can be made to work if these are followed; in reality it's up to the provisioning of the network (and good luck). If your pipes and buffers are big enough it works, otherwise no amount of timeout and retransmit will give you acceptable performance. (If you're willing to lose more than, say, 10% of your flows, I'd submit you don't actually want to be collecting unsampled flows in the first place.)

> I think that it is a good idea to relax the UDP transport specification and to remove restrictions which require a distinct behavior from TCP and SCTP. However, I do not see any advantage in changing the current TCP and SCTP transport specifications for IPFIX.
> 
> I would prefer keeping changes small (e.g. restrict changes to UDP transport) in order to increase interoperability between RFC5101 and ~bis. Changing the protocol just to make it easier for a few people to implement all transports is not a good argument because most transports are optional anyway.

This is a very good point. I'll work up an alternate proposal based on these comments to minimize changes. (These should also have the effect of minimizing the possible incompatibility with 5101...)

> I do not see how this helps UDP. I do not see any advantages for TCP and SCTP. I would keep the original spec and allow resending for UDP only.

Good point, noted.

> Ok. This clarifies that infinite expiration timeouts are allowed.

Another key here is that the protocol shouldn't prevent the CP from keeping state that it might have another reason to.

> I would keep the MUST for TCP and SCTP. I do not see any advantage in removing this rule.

Other than using nc or some other "dumb" EP to send a file via TCP that was itself received via UDP. But yes, that looks pretty much like the definition of "corner case". :)

> Ok. Template Withdrawal does not hurt in case of UDP. Also, it can be used by the exporter to "close" the session.

It also wouldn't lead to a UDP session reset, because there is no such thing.

> This causes incompatibility with RFC5101 without offering any advantages. I would keep the mandatory withdrawal for TCP and SCTP.

As above, good point... However, this does basically mean we're not simplifying template management (we still have "reliable" and "unreliable" template management protocols)... though there may still be some benefit in simplifying the specification of them.

> It is difficult for an application to obtain information from the SCTP socket when a message has left the interface card.

The EP can ensure this by setting the Export Time in the header; all sequencing is done via this header (as opposed to actual time the packet leaves the EP or arrives at the CP).

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Tue, 03 Apr 2012 16:03:00 +0100

Brian, Gerhard,

> Ok. Template Withdrawal does not hurt in case of UDP.

UDP doesn't guarantee in-order delivery. Therefore not waiting long 
enough after sending a TWM before sending a new template definition 
using the same template ID, could result in the new template definition 
arriving first, then being withdrawn.

ie, "Template N¹, TWM N, Template N²" becomes, "Template N¹, Template 
N², TWM N".

Of course, this can be avoided by checking the header timestamps, and 
ignoring the TWM if it originated prior to the current template.

Brian's reply later in this thread:

I wonder how many collectors process data in time order versus order of 
receipt? Since 5101 has nothing to say on the matter, some interesting 
interop test cases could be devised around this question.

P.

---

From: Brian Trammell <[email-masked]>
Date: Tue, 3 Apr 2012 17:31:36 +0200

Hi, Paul,

Inline...

On Apr 3, 2012, at 5:03 PM, Paul Aitken wrote:

> UDP doesn't guarantee in-order delivery. Therefore not waiting long enough after sending a TWM before sending a new template definition using the same template ID, could result in the new template definition arriving first, then being withdrawn.
> 
> ie, "Template N¹, TWM N, Template N²" becomes, "Template N¹, Template N², TWM N".
> 
> Of course, this can be avoided by checking the header timestamps, and ignoring the TWM if it originated prior to the current template.

You can also stick them all in the same message, no?

> I wonder how many collectors process data in time order versus order of receipt? Since 5101 has nothing to say on the matter, some interesting interop test cases could be devised around this question.

I presume most CPs process data in order of receipt, and for the most part things should run as such. However, if a collector detects that something is not right (in circumstances in which it can do so, which we addressed in the hallway once but which are now less clear to me than they were then) then it can use the export time ordering to try and recover.

(Or it can scream at the admin and have the admin go tweak some timeouts and/or fetch a bigger pipe. Again, IPFIX over UDP is a game for people with very-well-provisioned measurement networks and/or a penchant for gambling.)

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Tue, 03 Apr 2012 16:55:59 +0100

Brian,

> You can also stick them all in the same message, no?

That violates 5101, which specifically excludes TWM over UDP.

However, if the restriction is removed, then inter-message timestamps 
and intra-message order must both be honoured. eg, if your collector 
sub-divides messages and processes them in multiple threads.

> I presume most CPs process data in order of receipt,

+1

However, they should then reject out-of-order messages according to the 
header timestamp.

- except that this introduces a potential DoS vector, where I can send 
my ISP a (possibly null) message with a header timestamp of "7th Jan 
2106" and block all future messages...

- except that collectors should also reject messages which are far in 
the future (while allowing for collectors in remote timezones and DST 
changes).

> and for the most part things should run as such. However, if a collector detects that something is not right (in circumstances in which it can do so, which we addressed in the hallway once but which are now less clear to me than they were then)

+1

P.

>   then it can use the export time ordering to try and recover.
>