Subject: Re: [ipfix] Option templates issues

From: Mark Fullmer <[email-masked]>
Date: Tue, 18 Nov 2003 20:43:15 -0500

On one level I would agree if the difference was only the namespace, but
options templates also define a scope for the option data fields.

Think of protocol messages like a HELLO.  With the template/data model
first you have to send a HELLO template then send the HELLO data 
message.
This just seems a bit silly and potentially complex when you have many
many different ways a HELLO could be sent (single options template, vs
mixing it with other data).

The template model is very useful for the flow data because of the 
overhead
it saves, I'm just not sure why we would not want to stick to 
traditional
TLV's with fixed record formats for the other messages.

Another way to put this is if we just use the field type, someone is 
going
to have to write up a "what if" scenario for every bit of option data.
For example in the METERSTAT example what if the collector only receives
a template with lost_bytes and no other data.  What does it do?  Ignore 
it
because it's incomplete, do the best it can by adding a local timestamp
and assume the other stats are zero?  What if it gets two templates one
with lost_bytes and lost_pkts and the other with just a lost_bytes, what
then?

mark

On Nov 18, 2003, at 7:34 PM, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

> Mark,
>
>   I'd actually argue in the opposite direction.  Why do we need a whole
> different model for option templates vs. flow templates?
>
>   Both represent sets of structured information which are sent 
> repeatedly
> (hence using templates).  Both also may be extended over time as new 
> types
> of information are determined relevant for IPFIX.
>
>   The only distinction I see is that some collectors may simply want to
> ignore or quickly segregate "real flow info" from "other meta info" 
> like
> stats.  So I could simply picture a single flag on a template to
> differentiate between the two types.  Flow template/Flow Data vs. 
> Option
> template/Option data (perhaps simply reserve the high order template 
> id bit
> to be "0" for flow data and "1" for options).
>
>   Introducing even more encoding/decoding strategies would seem to me 
> to
> make the code base all that more complex, not simpler.
>
> Regards,
>
>   Jeff Meyer
>
>> -----Original Message-----
>> From: majordomo listserver
>> [mailto:[email-masked]]On Behalf
>> Of Mark Fullmer
>> Sent: Tuesday, November 18, 2003 4:14 PM
>> To: [email-masked]
>> Subject: [ipfix] Option templates issues
>>
>>
>>
>> I'd like to propose an alternative mechanism for sending
>> auxiliary data
>> such
>> as time synch messages, protocol hellos (needed for
>> fail-over), metering
>> process loss statistics (ie packets/flows lost due to resource
>> starvation),
>> etc.
>>
>> The existing framework allows "other" data to be sent in options
>> templates.
>> Take for example a metering process stat message.  Lets say
>> the metering
>> process will periodically report statistics such as
>>
>>    struct METERSTAT {
>>      u_int32_t lost_flows;       /* flows not exported due to
>> resource
>> starvation */
>>      u_int32_t lost_flows_pkts;  /* packets in the lost flows */
>>      u_int32_t lost_flows_bytes; /* bytes in the lost flows */
>>      u_int32_t lost_pkts;   /* packets dropped by metering process */
>>      u_int32_t lost_bytes;  /* bytes dropped by metering process */
>>      time_t    now;         /* when this record was generated */
>>    };
>>
>>    The question becomes how to encode this with an options template.
>>
>>    Each field in the record could have a type (6 types), then the
>> options template
>>    would list each of the 6 types.  The problem here is
>> during decoding.
>>   What
>>    will the collector do if only 5 of these items show up.
>> The packet
>> decode
>>    process is also not very straightforward when just data is sent
>> (other option
>>    fields could be in the same template) because it will have
>> to figure
>> out
>>    what to do based on the field type and there's no way to
>> group fields.
>>
>>    The other option is to call METERSTAT a type (of length
>> 24).  Now the
>>    decode process can key off of the type and all the fields
>> will always
>> be
>>    there because METERSTAT is fixed and defined in the
>> protocol document
>> just
>>    like other fields like PACKETS, BYTES, PROTOCOL, etc.  The issue I
>> have here
>>    is why are we bothering with a template/data model for
>> this.  Why not
>> just
>>    use traditional TLV's.
>>
>>    So what I'd like to see (I think) is to take one of the existing
>> reserved
>>    flowset ID's, lets say 3 and use it for generic TLV data
>> encodings.
>> Then
>>    define structured data in the protocol such as METERSTAT.
>> IMHO this
>> allows us
>>    to address a number of other issues including how to add
>> HELLO's for
>> the
>>    (optional) reliable failover, how to add aux flags like "this is
>> potentially
>>    a duplicate PDU because I failed over", etc.
>>
>>    The TLV encodings would only be used for the low bit-rate protocol
>> messages,
>>    NOT the high bandwidth flow data.  I'm also not proposing
>> the option
>> templates
>>    go away, just that we use a more traditional method for
>> encoding the
>> non flow
>>    data.
>>
>> mark
>>
>>
>>
>> --
>> Help        mailto:[email-masked] and say "help"
>> in message body
>> Unsubscribe mailto:[email-masked] and say
>> "unsubscribe ipfix" in message body
>> Archive     http://ipfix.doit.wisc.edu/archive/
>>

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 19 Nov 2003 09:19:30 -0800

Whoops,

  I only sent this to Mark originally...

-- Jeff

> From: MEYER,JEFFREY D (HP-Cupertino,ex1) 
> Sent: Tuesday, November 18, 2003 4:34 PM
> To: 'Mark Fullmer'
> Subject: RE: [ipfix] Option templates issues
> 
> 
> Mark,
> 
>   I'd actually argue in the opposite direction.  Why do we 
> need a whole different model for option templates vs. flow templates?
> 
>   Both represent sets of structured information which are 
> sent repeatedly (hence using templates).  Both also may be 
> extended over time as new types of information are determined 
> relevant for IPFIX.
> 
>   The only distinction I see is that some collectors may 
> simply want to ignore or quickly segregate "real flow info" 
> from "other meta info" like stats.  So I could simply picture 
> a single flag on a template to differentiate between the two 
> types.  Flow template/Flow Data vs. Option template/Option 
> data (perhaps simply reserve the high order template id bit 
>   Introducing even more encoding/decoding strategies would 
> seem to me to make the code base all that more complex, not simpler.
> 
> Regards,
> 
>   Jeff Meyer
>