Subject: [ipfix] Option templates issues

From: Mark Fullmer <[email-masked]>
Date: Wed, 19 Nov 2003 14:01:00 -0500

A few more examples where TLV's seem a more natural approach:

   The exporter needs to present it's ID, which is usually the source
   IP address inside the packet so that clients that connect via proxies
   get the correct exporter ID.  This is an item that needs to be sent
   once right after connection setup.

   A proxy might want to indicate to the collector it's inline or maybe 
that
   it's doing proxy aggregation.  With TLV's this is easy. With templates
   the proxy now has to construct a template before sending any data.
   It has no way of knowing that the template ID will not be used by
   the collector in the future so now it has to provide template ID
   mappings.  This could get really ugly if template ID's get encoded
   in fields later.  Proxies that provide a fanout (receive from the
   exporter, send to many collectors) are common in existing NetFlow
   deployments.

   I'm also looking at how to add the reliable fail-over option which
   also requires protocol messages that don't benefit from the template
   style encodings.

mark

On Nov 18, 2003, at 8:43 PM, Mark Fullmer wrote:

>
> On one level I would agree if the difference was only the namespace, 
> but
> options templates also define a scope for the option data fields.
>
> Think of protocol messages like a HELLO.  With the template/data model
> first you have to send a HELLO template then send the HELLO data 
> message.
> This just seems a bit silly and potentially complex when you have many
> many different ways a HELLO could be sent (single options template, vs
> mixing it with other data).
>
> The template model is very useful for the flow data because of the 
> overhead
> it saves, I'm just not sure why we would not want to stick to 
> traditional
> TLV's with fixed record formats for the other messages.
>
> Another way to put this is if we just use the field type, someone is 
> going
> to have to write up a "what if" scenario for every bit of option data.
> For example in the METERSTAT example what if the collector only 
> receives
> a template with lost_bytes and no other data.  What does it do?  
> Ignore it
> because it's incomplete, do the best it can by adding a local timestamp
> and assume the other stats are zero?  What if it gets two templates one
> with lost_bytes and lost_pkts and the other with just a lost_bytes, 
> what
> then?
>
> mark

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Tue, 25 Nov 2003 13:10:25 -0500

Mark,

  Actually it is quite easy.  Define an information element ipfixOption,
which is an identifier.

--------------
6.XX ipfixOption

Description:

The option identifier indicates that this flow record contains meta
information about the exporters
behavior.  Specific ipfixOption values indicate what information items may
be expected.

Type: unsignedInt.

Semantics: identifier.

Field Id: XX
---------------

  In your example it has three values TIME_SYNCH, LOSS_STATS and HELLO.
Require that this 
IE be present in the different specific structures.  And voila, you've
reused what we've
already built!

-- Jeff

-----Original Message-----
From: Mark Fullmer [mailto:[email-masked]]
Sent: Monday, November 24, 2003 7:33 PM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: [email-masked]
Subject: Re: [ipfix] Option templates issues

I think you're missing the problem.

With a template mechanism there is no key to tell the collector
what this option means.  Template ID's have absolutely no
meaning outside the protocol.

A collector needs to infer what the exporter means when
a template shows up with some special fields, like for
example a "lost bytes" field.

In other words there's no easy way to code something like

switch (ipfix_option)

   case IPFIX_TIME_SYNC:
     do this

   case IPFIX_LOSS_STATS:
     do that.

   case IPFIX_HELLO:
     do something else.

mark

On Nov 24, 2003, at 4:43 PM, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:

> Mark,
>
>   AVP's are less efficient than templates for parsing records, be they
> actual flows or other meta information.  All a template is is a dynamic
> "fixed structure" over the life of a conversation.
>
>   Nothing says that certain records in IPFIX cannot have some required
> minimal set of fields.  This is done by Diameter today for key message
> types.
>
>   IPFIX alreay has a mechanism accomodating this type of transport. 
> Based on
>
> my experience collecting from dozens of different protocols and 
> hundreds of
> different devices, the model you propose is simply more work.  No 
> efficiency
> gains.  It should result in less code to maintain on both the exporter 
> and
> collector.  Less is more.
>
> -- Jeff
>
>> -----Original Message-----
>> From: Mark Fullmer [mailto:[email-masked]]
>> Sent: Thursday, November 20, 2003 12:28 PM
>> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
>> Cc: [email-masked]
>> Subject: Re: [ipfix] Option templates issues
>>
>>
>> These aren't hypothetical.  A robust collector needs to be able to
>> deal with anything the exporter throws at it.  There are messages
>> from the exporter to the collector that have a fixed format, it's
>> much much easier for the collector to deal with these using TLV's.
>>
>> There needs to be a well defined standard for certain types of
>> messages, including loss stats.  Vendors may choose to augment this
>> with their own messages, including an aggregated flow which has
>> interface based loss information.
>>
>> mark
>>
>> On Nov 19, 2003, at 3:06 PM, MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
>>
>>> Mark,
>>>
>>>   I think this discussion may be wandering off into hypotheticals
>>> versus
>>> specific use cases.
>>>
>>>   In the current (pre-v9) netflow implementations, there is no
>>> additional
>>> metadata being sent, and this works fine.  So the analogy
>> to HELLO's
>>> seems a
>>> bit overstated (i.e. I would picture an exporter only having 1 or 2
>>> different statistics blocks it may send [if any])
>>>
>>>   You've cited a specific example of metering statistics which
>>> summarize the
>>> behavior of the exporter at intermittent points in time.
>> This sounds
>>> worthwile, but I would imagine that like the flow data itself,
>>> different
>>> vendors may chose to send varying sets of information
>> fields in their
>>> statistics records.
>>>
>>>   Scoping issues sound to me like it is just a matter of having the
>>> appropriate identifier transferred as part of the record.
>> I.e. if I
>>> want
>>> statistics on a per interface basis, then the record may be:
>>>
>>>   ingressPort
>>>   totalFlows
>>>   lostFlows
>>>   lostPackets
>>>    .. whatever
>>>
>>>   I don't really see how this differs from general flow
>> data.  Or why
>>> you
>>> wouldn't want to use the same Information modeling (and encoding)
>>> techniques
>>> hammered out so far.  In fact it seems almost like a
>> specialized form
>>> of
>>> aggregation.  Introducing different models for largely similar
>>> information
>>> introduces complexity (and probably inconsistency in the long term).
>>>
>>>   So, I'd really like to see as few mechanisms as possible
>> for moving
>>> data,
>>> unless we simply can't get by with the set available.  In
>> this case I
>>> think
>>> we can.
>>>
>>> Regards,
>>>
>>>   Jeff Meyer
>>>
>>>> -----Original Message-----
>>>> From: majordomo listserver
>>>> [mailto:[email-masked]]On Behalf
>>>> Of Mark Fullmer
>>>> Sent: Wednesday, November 19, 2003 11:01 AM
>>>> To: [email-masked]
>>>> Subject: [ipfix] Option templates issues
>>>>
>>>>
>>>>