Subject: RE: [ipfix] Option templates issues - Consensus?

From: "Meyer, Jeffrey D (http://usage.fc.hp.c)" <[email-masked]>
Date: Fri, 9 Jan 2004 14:57:20 -0800

Hi,
 
  I agree that there is a distinction between protocol extensions (e.g. a hello message) and special (options) information element extensions.
 
  So I'm OK with this proposal.  The mechanism for header extensions I think needs some more details.
 
  The current protocol spec seems to define exactly one message type with four possible payload subtypes (template, option template,
  data, option data).  These payloads may be intermixed.
 
  The mechanism for adding extensions would seem to be based on the reserved range of 0-255 for templateId's.  Currently 0 and 1
  are used for flow templates and option templates.  Flow data and option data are distinguished solely based on previous id assignments
  by the respective templates.
 
  So I guess the proposal would be a hello message or other header extensions would utilize other values in the range 2-255.  Is this
  correct?
 
Regards,
 
  Jeff Meyer

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]]
Sent: Thursday, January 08, 2004 4:19 AM
To: Meyer, Jeffrey D (http://usage.fc.hp.c)
Cc: Mark Fullmer; 'Ipfix Wg' (E-mail)
Subject: Re: [ipfix] Option templates issues - Consensus?

Hi,

In order to progress with the protocol draft... is there a consensus for the following concepts?

There are two issues here 

  1) How to provide extra information about the flow data and metering 
     process. 

  2) How to allow for protocol extensions. 

1) How to provide extra information about the flow data and metering process. 
Typical example: the meterstats.
An option template will be used
The protocol draft will specify what the minimal set of data types is required
A new data type (referred to ipfixOption by Jeff) would be used to identify specific options templates
This would help the collector answering the question:
switch (ipfixOption) 
  case METER_STATS
    do that. 
  case ... 
    do something else. 

2) How to allow for protocol extensions. 
Typical example: hello, exporter ID, authentication, etc...
No options template: an extension to the header will be used.

Regards, Benoit.

Mark,

  Not having seen requirements or discussion for "proxies", I'm having a hard time conceptualizing this new requirement.

  Since option template Id's and flow record template Id's are currently dynamically determined, I guess what you are arguing is that options should somehow be fixed.

  Since a "hello" option has not even been defined to date, and your use case seems to be centered around reliable failover, I think there are a lot of tangled concerns going on here.  Could you elaborate a bit more on this scenario?

-- Jeff

-----Original Message-----

From: Mark Fullmer [ mailto:[email-masked]]

Sent: Tuesday, January 06, 2004 10:36 PM

To: Meyer, Jeffrey D ( http://usage.fc.hp.c)

Cc: 'Ipfix Wg' (E-mail)

Subject: Re: [ipfix] Option templates issues

Consider a proxy.  If the proxy needs to use an option ID to say 

maintain

HELLO's for a reliable fail-over mechanism it will have to pick a 

template ID

that potentially clashes with a future one used by an exporter.  So now

a proxy needs to track all inbound template ID's, map them to outbound 

ID's,

and rewrite fields in the messages that refer to the template ID's.

mark

On Dec 23, 2003, at 2:12 PM, Meyer, Jeffrey D ( http://usage.fc.hp.c) 

wrote:

Mark,

  Agreed, it is a choice, and both will work, it is a questions of 

anticipated extensibility.  Fewer fields in the header and most 

information elements defined in a consistent manner seems to me to 

address the approach of "Make it as simple as possible and no 

simpler".

  Since the requirement appears to be addressable with a single 

mechanism, i.e. a flag distinguishing options from flow elements, and 

reusing the information modeling, I still don't see the benefit in 

introducing more mechanisms which need to be implemented and tested on 

both ends.

-- Jeff

-----Original Message-----

From: Mark Fullmer [ mailto:[email-masked]]

Sent: Tuesday, December 23, 2003 10:11 AM

To: Meyer, Jeffrey D ( http://usage.fc.hp.c)

Cc:  [email-masked]; Benoit Claise

Subject: Re: [ipfix] Option templates issues

There are two issues here

   1) How to provide extra information about the flow data and metering

      process.

   2) How to allow for protocol extensions.

If we add a required "key" or "command" field to the options templates

then I think what we have will work given the proper text

describing what IE's MAY, SHOULD, and MUST be present in an options

template with a specific key.  We may want to even have a separate

IE namespace for the options....

The protocol extension I'm still of the opinion should be handled

with an extensible header.  Examples of protocol extensions could be

    Reliable fail-over.

    Authentication.

    The optional time stamp for the split-time model.

    Hooks for proxies.

    Future unanticipated extensions.

And yes, all of this could be stuffed in an options template but the

same

argument could be used to do away with the protocol header entirely.

mark

On Dec 22, 2003, at 11:17 AM, Meyer, Jeffrey D ( http://usage.fc.hp.c)

wrote:

Hi,

  Again, I'd point out, that as Benoit indicates, there are likely to

be different things we want to send as "options".

  Rather than inventing an entirely new model, just make option a

flag on the template and REUSE what we already have for data records.

It's that simple!

-- Jeff

-----Original Message-----

From: majordomo listserver [ mailto:[email-masked]]On

Behalf

Of Benoit Claise

Sent: Monday, December 22, 2003 4:04 AM

To: Mark Fullmer

Cc:  [email-masked]

Subject: Re: [ipfix] Option templates issues

Mark,

[sorry for the delay, i've been absent from the list for a while]

I understand the concern but let me ask a few questions.

What if the metering process can't "meter" one of them? Ex: flows not

exported due to resource starvation

What if we would like to export some extra data types? Ex: IPv6 

packets

and bytes dropped

So what if the METERSTAT struct is slightly modified?

Ganesh had a good point I think:

    If the group can come up with fixed structs that you mentioned

below,

    then I think what you propose can be adopted.

Aren't we going to spend a long time to determine what MUST be 

exported

in this/those structure(s)?

What you defined in your METERSTAT makes perfectly from a collector

point of view but will not always be possible to collect on the

exporter?

And it does perfectly sense now, but what about in 1 year? So would we

have a new data type with a new slightly different structure?

I'm just wondering what we should say in the protocol draft:

    1. SHOULD export a METERSTAT data type(as described in your 

email)?

What if one data type can't be exported, we don't send any metering

stats "message"?

    2. MUST export a metering statistics Options Templates that SHOULD

contain lost_flows, lost_flows_pkts, lost_flows_bytes, lost_pkts,

lost_bytes, time.

    3. MUST export a metering statistics Options Templates that MUST

contain X, Y, and SHOULD contain Z

    4. MUST export a metering statistics Options Templates that 

contain

at least X, Y

Actually, I think the flexibility of the Options Template is 

benefitial

for any data types.

Now, this is a different story for the Hello, Failover, etc... types 

of

messages that you spoke about, as the goal is totally different!

Regards, Benoit.

I'd like to propose an alternative mechanism for sending auxiliary

data such

as time synch messages, protocol hellos (needed for fail-over),

metering

process loss statistics (ie packets/flows lost due to resource

starvation),

etc.

The existing framework allows "other" data to be sent in options

templates.

Take for example a metering process stat message.  Lets say the

metering

process will periodically report statistics such as

  struct METERSTAT {

    u_int32_t lost_flows;       /* flows not exported due to resource

starvation */

    u_int32_t lost_flows_pkts;  /* packets in the lost flows */

    u_int32_t lost_flows_bytes; /* bytes in the lost flows */

    u_int32_t lost_pkts;   /* packets dropped by metering process */

    u_int32_t lost_bytes;  /* bytes dropped by metering process */

    time_t    now;         /* when this record was generated */

  };

  The question becomes how to encode this with an options template.

  Each field in the record could have a type (6 types), then the

options template

  would list each of the 6 types.  The problem here is during

decoding.  What

  will the collector do if only 5 of these items show up.  The packet

decode

  process is also not very straightforward when just data is sent

(other option

  fields could be in the same template) because it will have to 

figure

out

  what to do based on the field type and there's no way to group

fields.

  The other option is to call METERSTAT a type (of length 24).  Now

the

  decode process can key off of the type and all the fields will

always be

  there because METERSTAT is fixed and defined in the protocol

document just

  like other fields like PACKETS, BYTES, PROTOCOL, etc.  The issue I

have here

  is why are we bothering with a template/data model for this.  Why

not just

  use traditional TLV's.

  So what I'd like to see (I think) is to take one of the existing

reserved

  flowset ID's, lets say 3 and use it for generic TLV data encodings.

Then

  define structured data in the protocol such as METERSTAT.  IMHO 

this

allows us

  to address a number of other issues including how to add HELLO's 

for

the

  (optional) reliable failover, how to add aux flags like "this is

potentially

  a duplicate PDU because I failed over", etc.

  The TLV encodings would only be used for the low bit-rate protocol

messages,

  NOT the high bandwidth flow data.  I'm also not proposing the 

option

templates

  go away, just that we use a more traditional method for encoding 

the

non flow

  data.

mark

-- 

Help         mailto:[email-masked] and say "help" in

message body

Unsubscribe  mailto:[email-masked] and say

"unsubscribe ipfix" in message body

Archive      http://ipfix.doit.wisc.edu/archive/

--

Help         mailto:[email-masked] and say "help" in

message body

Unsubscribe  mailto:[email-masked] and say

"unsubscribe ipfix" in message body

Archive      http://ipfix.doit.wisc.edu/archive/

--

Help         mailto:[email-masked] and say "help" in message body

Unsubscribe  mailto:[email-masked] and say

"unsubscribe ipfix" in message body

Archive      http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Sun, 11 Jan 2004 20:47:14 -0500

I think that's where we're at.  Ganesh had suggested the extensible 
header
idea after my proposal for using one of the unused FlowSet ID's.

Benoit have discussed this offline and (I think) we're both leaning 
towards
using the existing mechanism with FlowSet ID's.  The only real change
this may mean for IPFIX is a better term than "FlowSet".

mark

On Jan 9, 2004, at 5:57 PM, Meyer, Jeffrey D (http://usage.fc.hp.c) 
wrote:

>   The mechanism for adding extensions would seem to be based on the 
> reserved range of 0-255 for templateId's.  Currently 0 and 1
>   are used for flow templates and option templates.  Flow data and 
> option data are distinguished solely based on previous id assignments
>   So I guess the proposal would be a hello message or other header 
> extensions would utilize other values in the range 2-255.  Is this
>   correct?
>  
> Regards,
>  
>   Jeff Meyer
> From:Benoit Claise [mailto:[email-masked]]
> Sent:Thursday, January 08, 2004 4:19 AM
> To:Meyer, Jeffrey D (http://usage.fc.hp.c)
> Cc:Mark Fullmer; 'Ipfix Wg' (E-mail)
> Subject:Re: [ipfix] Option templates issues - Consensus?
>
> Hi,
>
>   Not having seen requirements or discussion for "proxies", I'm having 
> a hard time conceptualizing this new requirement.
>
>   Since option template Id's and flow record template Id's are 
> currently dynamically determined, I guess what you are arguing is that 
> options should somehow be fixed.
>
>   Since a "hello" option has not even been defined to date, and your 
> use case seems to be centered around reliable failover, I think there 
> are a lot of tangled concerns going on here.  Could you elaborate a 
> bit more on this scenario?
>
> -- Jeff
>
> From: Mark Fullmer [mailto:[email-masked]]
> On Dec 23, 2003, at 2:12 PM, Meyer, Jeffrey D (http://usage.fc.hp.c)
> wrote:
>
>
> Mark,
>
> From: Mark Fullmer [mailto:[email-masked]]
> On Dec 22, 2003, at 11:17 AM, Meyer, Jeffrey D (http://usage.fc.hp.c)
> wrote:
>
>
> Hi,
>
> From: majordomo listserver [mailto:[email-masked]]On
> Behalf
> Of Benoit Claise

---

From: Benoit Claise <[email-masked]>
Date: Mon, 19 Jan 2004 15:28:22 +0100

> this may mean for IPFIX is a better term than "FlowSet". 

Yes we are.

Regards, Benoit.

>>   Not having seen requirements or discussion for "proxies", I'm 
>> having a hard time conceptualizing this new requirement.
>>
>> currently dynamically determined, I guess what you are arguing is 
>> that options should somehow be fixed.
>>