Subject: [ipfix] Option templates issues

From: Mark Fullmer <[email-masked]>
Date: Tue, 18 Nov 2003 19:13:41 -0500

I'd like to propose an alternative mechanism for sending auxiliary data 
such
as time synch messages, protocol hellos (needed for fail-over), metering
process loss statistics (ie packets/flows lost due to resource 
starvation),
etc.

The existing framework allows "other" data to be sent in options 
templates.
Take for example a metering process stat message.  Lets say the metering
process will periodically report statistics such as

   struct METERSTAT {
     u_int32_t lost_flows;       /* flows not exported due to resource 
starvation */
     u_int32_t lost_flows_pkts;  /* packets in the lost flows */
     u_int32_t lost_flows_bytes; /* bytes in the lost flows */
     u_int32_t lost_pkts;   /* packets dropped by metering process */
     u_int32_t lost_bytes;  /* bytes dropped by metering process */
     time_t    now;         /* when this record was generated */
   };

   The question becomes how to encode this with an options template.

   Each field in the record could have a type (6 types), then the 
options template
   would list each of the 6 types.  The problem here is during decoding. 
  What
   will the collector do if only 5 of these items show up.  The packet 
decode
   process is also not very straightforward when just data is sent 
(other option
   fields could be in the same template) because it will have to figure 
out
   what to do based on the field type and there's no way to group fields.

   The other option is to call METERSTAT a type (of length 24).  Now the
   decode process can key off of the type and all the fields will always 
be
   there because METERSTAT is fixed and defined in the protocol document 
just
   like other fields like PACKETS, BYTES, PROTOCOL, etc.  The issue I 
have here
   is why are we bothering with a template/data model for this.  Why not 
just
   use traditional TLV's.

   So what I'd like to see (I think) is to take one of the existing 
reserved
   flowset ID's, lets say 3 and use it for generic TLV data encodings.  
Then
   define structured data in the protocol such as METERSTAT.  IMHO this 
allows us
   to address a number of other issues including how to add HELLO's for 
the
   (optional) reliable failover, how to add aux flags like "this is 
potentially
   a duplicate PDU because I failed over", etc.

   The TLV encodings would only be used for the low bit-rate protocol 
messages,
   NOT the high bandwidth flow data.  I'm also not proposing the option 
templates
   go away, just that we use a more traditional method for encoding the 
non flow
   data.

mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 18 Nov 2003 19:03:13 -0800

Mark,

  See inline..

Mark Fullmer wrote:

>
> I'd like to propose an alternative mechanism for sending auxiliary 
> data such
>   would list each of the 6 types.  The problem here is during 
> decoding.  What

Why will only 5 show up if the template is defined as containing 6 
fields and the
flowset is confined to one packet?
The channel for contol information being reliable (is it not?),  the 
option template
for control information need to be send just once for a connection. Also how
then are we going to convey the scope information pertaining to the Types?
One possibility is that if different of these control fields are desired 
to be
exported with different periodicity.With the above METERSTAT example
say "lost_flows" needs to be send with a different periodicity as compared
to "lost_flows_pkts" and we do not need to create many templates from the
exporter for the various combinations.
I agree that if  the exporter needs to send some un-structured
information which pertain to the explicit scope which is the observation
domain, then TLV could be handy (though slightly inefficient) in some
cases.
But I feel that it is upto the exporter to make a decision
to encode a field as TLV flowset or otherwise rather than the
protocol dictating this.

>   The other option is to call METERSTAT a type (of length 24).  

How is this different from option templates?

> Now the
>   decode process can key off of the type and all the fields will 
> always be
>   there because METERSTAT is fixed and defined in the protocol 
> document just
>   is why are we bothering with a template/data model for this.  Why 
> not just
>   use traditional TLV's. 

>   a duplicate PDU because I failed over", etc. 

Can't these be handled by using an extended header which is encoded as  TLV?

Thanks
Ganesh

---

From: Mark Fullmer <[email-masked]>
Date: Tue, 18 Nov 2003 22:39:36 -0500

> flowset is confined to one packet?

The issues is the exporter is free to define a template with one, all
or some of the fields.

> One possibility is that if different of these control fields are 
> desired to be

If you need a scope then the record can contain a scope field.

> exporter for the various combinations.

I'm not sure I see any value in providing this level of granularity for
counters like this.

> to encode a field as TLV flowset or otherwise rather than the

My concern is that unstructured data presented to the collector like
this is going to add a lot of complexity.

>>   fields could be in the same template) because it will have to 
>> figure out

The option template doesn't have a "type" field to group data into 
structures.

>>   define structured data in the protocol such as METERSTAT.  IMHO 
>> this allows us
>>   to address a number of other issues including how to add HELLO's 
>> for the

The 10,000 foot view I have of IPFIX is

  <fixed header>
  TLV
  ...
  TLV

Where
   T=0 is an template flowset (V has the templates)
   T=1 is an options template flowset (V has the templates)
   T=2..255 are reserved flowsets.
   T>255 are data flowsets. (V has the data)

What I'm proposing is use one of the reserved flowset ID's and use the
V portion to further encode TLV's.

mark

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 18 Nov 2003 20:01:04 -0800

Mark,

Mark Fullmer wrote:

>>>   will the collector do if only 5 of these items show up.  The 
>>> packet decode
>>
>>

Ok. I guess your idea is to have predefined structs for control information
that most of the collectors would be interested in.

> If you need a scope then the record can contain a scope field. 

Then it has to be TLSV .

> I'm not sure I see any value in providing this level of granularity for

I agree. This was just cited as an example ( a bad one -:)

> this is going to add a lot of complexity.

If the group can come up with fixed structs that you mentioned below,
then I think what you propose ca be adopted.

>>>   flowset ID's, lets say 3 and use it for generic TLV data 
>>> encodings.  Then

What I meant in the above statement is <fixed hdr, extensions flag> 
<Extensions TLVs>.
This way the protocol specific information (like Hello) can be carried 
without packing
it into a flowset. I feel this would be easier to decode at the 
 collector end.

Thanks
Ganesh

---

From: Mark Fullmer <[email-masked]>
Date: Wed, 19 Nov 2003 07:14:38 -0500

> that most of the collectors would be interested in.

Yes.

>> If you need a scope then the record can contain a scope field.

I think we're saying the same thing.

> then I think what you propose ca be adopted.

I'm willing to at least start the process.  Trying to write up
how a SYNCH (packets/flows transmitted) message would work is
what prompted this.

> it into a flowset. I feel this would be easier to decode at the 

That's fine.  I'll start a header changes thread since the "extensions"
area is something else we needed to address.

mark