Subject: RE: [ipfix] extensibility of the ipfix protocol

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 11 Jun 2003 12:45:37 -0400

Benoit,
 
  So it sounds like "scope" is more or less a way to pack multiple discrete
records into a single template.  This means that templates are no longer
necessarily in first normal form as some attributes may repeat.
 
  In general, I still think this is a lot of complexity to address a use
case which could be addressed with other mechanisms already present in NFv9,
i.e. treat all attributes as equals from the protocol perspective, only have
one template model, distinguish between "option records" and "flow records".
 
  Since I believe that the information transfer you describe can be
accomplished with fewer moving parts in the protocol, and since the
"simplicity" of netflow was cited as a key reason for its selection, can we
make this simplification?
 
Regards,
 
  Jeff Meyer

-----Original Message-----
From: Benoit Claise [mailto:[email-masked]]
Sent: Wednesday, June 11, 2003 4:35 AM
To: Sebastian Zander
Cc: MEYER,JEFFREY D (HP-Cupertino,ex1); [email-masked]
Subject: Re: [ipfix] extensibility of the ipfix protocol

Hi Jeff and Sebastian,

Jeff,

comments below 

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote: 

Sebastian, 

  I'd agree that making the extensibility model more explicit is desireable.

    I'm still unclear on when or why one would extend information into the 
header 
vs. extending the data template sent itself.  I view the Option template as 
flagging "pseudo-flow records" which have a set of attributes which carry 
information other than actual flow details, e.g. configuration info or 
counters. 

  But isn't the combination of having Extensible Flow Records (for "real" 
data) and Extensible Option Records (for "other" data) sufficient to move 
any form of data the exporter wants to export? 

  I guess this is what Mark was saying.  So I'd agree to that. 

I also agree 

  I would see the same "Information Model" and extension model being used 
for both Flow Record and Options. 

  So to clarify, the header is fixed.  There is a distinction between flow 
records and "other" data transmitted by the exporter.  And a common way 
of defining how attributes which appear in either flow or option records 
will be used. 

  I'm not sure on why the "Scope" options should be treated differently 
than any other attribute which may appear in an Option Record (Section 6.1) 
in NFv9 specification.  Seems like things would be simpler and just as 
effective if there were just a "Scope" attribute which could appear in 
Option records.  In this case an Options template would be identical 
to a Flow Template except it would have a FlowSet id of 1 vs. 0. 

  Does that work? 

Jeff,
Yes it would work.
One argument for treating the scope differently is that the scope must be
applied to all options data records.
And the Scope is an important data type. If you don't know about it, you
can't apply the "action" that is associated with the options flow record.
For example: if the scope is the line card X and the sampling rate Y and if
the collector doesn't know abouth the scope for the line card, the exporter
doesn't know that the number of bytes should be multipled by the sampling
rate.
Ok, that would be almost the same for an unknow data types in the flow data
record, but that one could just be reported, without taking an "action"

I must admit that i don't 100% understand how scope is supposed to work. 
The example in 10.4 and 10.5 seems somewhat flawed. The draft says there are

N scope fields and N option fields. Must the number of scope fields be equal

to the number of option fields? If yes this seems to waste lots of space if
the 
scope does not change (and it is not the case in the example). If not
(probably 
the authors meant M and N) the question is how to map the scope fields to
the 
options fields when there is more than 1 scope field (seems impossibe)? 

Sebastian, you are right that the intention is to have M and N. This has
been corrected in the latest draft.
How to map the scope fields to the options fields when there is more than 1
scope field?
The scope can be limited further by listing multiple scopes which all have
to match at the same time.

Making scope another attribute sounds somewhat good. However has scope some 
kind of grouping functionality 

Yes exactly. See the previous answer to Jeff

or must it explicitly appear before each option? 
Can scopes be combined such as e.g. this option is per interface vs. this
option 
is per template per interface?

Exactly.

Regards, Benoit.

I think my questions are about grouping because 
scope seems to have some kind of grouping function... 

Cheers, 

Sebastian 

Regards, 

  Jeff Meyer 

-----Original Message----- 
From: Sebastian Zander [ mailto:[email-masked]
<mailto:[email-masked]> ] 
Sent: Wednesday, May 21, 2003 5:19 PM 
To: MEYER,JEFFREY D (HP-Cupertino,ex1) 
Cc: [email-masked] <mailto:[email-masked]>  
Subject: Re: [ipfix] extensibility of the ipfix protocol 

Jeff, 

MEYER,JEFFREY D (HP-Cupertino,ex1) wrote: 

Sebastian, 

 One of the issues of talking about extensibility in the generic sense is 
that it creates the temptation to specify generality without any driving 
use case.  "Someone may want to ..." seems like a slippery slope in 

bounding 

requirements. 

 In particular the case (1) extending fields in the header sounds 

dubious. 

 Since IPFIX is about getting flow records from the exporter to the 
collector, 
what is the purpose of allowing arbitrary header extensions? 

The purpose is to send information per packet which is associated to or 
necessary for using the rest of the information contained in the packet. For

me its pretty obvious that you need such an extension mechanism. And as Mark

pointed out this is already supported by the use of option templates. 

netflow certainly meets the extensibility reqs but the netflow drafts do 
*not* 
a good job IMO in explaining its extensibility compared to e.g. what 
Diameter 
does. Instead of letting the reader guess the ipfix draft should have more 
verbage. 

  In case (2), I'd certainly recommend option (b), making an explicit 
separation of namespaces (in this case integer values) by using unique 
vendor ids as independent "namespace authorities" seems like common sense. 
SNMP's been doing this for years, and Diameter is also following this 
convention as you point out. 

 As for case (3), my understanding is that explicit type information in 
the payload/template itself did not achieve consensus.   This does mean 
that the numeric id (and optional vendorId) needs some concept of type, 
in a supporting document.  Having each attribute specify its type by 
reference to a type space defined in the base IPFIX Information Model 
is the way to go here (in my opinion). 

Yeah, but the protocol draft must be aligned somehow to the information 
model draft e.g. where are the types defined, where is the encoding 
defined... 

Cheers, 

Sebastian 

 When a vendor extends the set of attributes, chosing a well defined 
type would be preferred to inventing a new type.  If the vendor cannot 
resist the overwhelming temptation to create new types, it can be defined in
their own information model. 

Regards, 

 Jeff Meyer 

-----Original Message----- 
From: Sebastian Zander [ mailto:[email-masked]
<mailto:[email-masked]> ] 
Sent: Tuesday, May 06, 2003 6:55 PM 
To: [email-masked] <mailto:[email-masked]>  
Subject: [ipfix] extensibility of the ipfix protocol 

Hi, 

although extensibility is mandatory by the requirements there hasn't 
been a lot discussion on how to do it and the chosen candidate protocol 
doesn't say much about extensibility. So here are my thoughts about 
extensibility. 

The protocol specification must define how the protocol can be extended. 

I see the following (but there may be more) ways of extending the 
protocol: 

1. Extending the fixed packet header 
This would allow to add new header fields without changing the 
protocol specification. 

A bit (X) could be stolen from version to indicate the presence of 
extension header(s). Extension header(s) follow the fixed header if 
X=1. Extension header(s) consists of type, length and value. 

2. Extending the attribute/field space 
This is certainly needed. It could be done at least in two ways: 
a) A flat number space where numbers are specified in extension 
   RFCs and registered by IANA. A nice thing would a dynamic 
   space where numbers are not defined by IANA but could be negotiated 
   out of band (see payload type in the RTP specification). Such a 
   space could also be used as safe playground. 
b) A number space which is defined in extension RFCs (IANA) and the 
   possibility of using vendor specific attributes by having an optional 
   vendor ID field in the template specification. The vendor ID would 
   be IANA assigned but the vendor then could use its private field ID 
   space (see the Diameter Base protocol specification). Adds more 
   complexity and overhead to the protocol but could be very useful 
   when the protocol becomes widely adopted and a lot of extensions 
   are done. 

3. Extending the type space (if there are types at all) 
Probably there won't be much need to extent this because the protocol 
specification could do a good job of specifying all the basic types. 
If this really needs to be extended it would be probably done with a 
revised protocol specification. 

Comments? Opinions? 

Cheers, 

Sebastian

---

From: Benoit Claise <[email-masked]>
Date: Thu, 12 Jun 2003 11:50:40 +0200

Jeff,

>  
>   So it sounds like "scope" is more or less a way to pack multiple 
> discrete records into a single template. 

I would say: to group multiple data types that belong to the same scope; 
the scope could be
               0x0001 System  
               0x0002 Interface 
               0x0003 Line Card 
               0x0004 Cache 
               0x0005 Template
For example (example section in the draft):
   - total number of Export Packets and total number of exported Flows 
exported by each line card
   So scope = line card, data types = Export Packets and Exported Flows

There was a previous discussion about exporting the number of flow 
records per template. This is also possible
So scope: = template, data type = Exported Flows

> This means that templates are no longer necessarily in first normal 
> form as some attributes may repeat.

Can you please elaborate.

>  
>   In general, I still think this is a lot of complexity to address a 
> use case which could be addressed with other mechanisms already 
> present in NFv9, i.e. treat all attributes as equals from the protocol 
> perspective, only have one template model, distinguish between "option 
> records" and "flow records".

Let me take 3 examples

1. I export flow data records composed of src IP address, dst IP 
address, Fragmented, number of packets, number of bytes
where Fragmented is just 0 or 1.
But the collecting process doesn't know about this Fragmented data type; 
it has got 2 choices.
- either it discards all the information because it doesn't understand 
one of the data types
- either it just sum up the number of packets and bytes per IP 
addresses, regardless of the Fragmented, to at least report something 
meaningfull. So a default action (summing up) is possible.

2. I export an options data record composed of scope =  line card X and 
data type = the sampling rate Y.
But the collecting process doesn't know about the the line card scope; 
it can't even take a default action because it doesn't know where to 
apply it to. The exporter doesn't know what to do with the sampling 
rate, even if it understands the sampling rate data type.

3. To follow up on the example 2, I export _flow _data records composed 
of  number of packets, number of bytes, input interface, line card, 
sampling rate, exported flows, template ID. So data types are equals.
Let's assume that the collecting process knows about all data types.
What should the sampling rate be applied to? Interface, line card, or 
template?
What should the export packet be applied to? Interface, line card, or 
template?

> "simplicity" of netflow was cited as a key reason for its selection, 
> can we make this simplification?

Can we make this simplification? Well, as explained during the last IETF 
meeting, this new IPFIX protocol belongs to the IETF. So it will become 
what the WG consensus will decide it to be!
 
Nevertheless, I'm in favor of keeping the scope because of :
0. in flow data records, we don't always what is the scope
1. the scope is used as grouping factor (Sebastian's term)
2. with flow data record (without scope) we can take a default action
3. the scope is a more important data type to know about

Regards, Benoit.

>     *From:* Benoit Claise [mailto:[email-masked]]
>     *Sent:* Wednesday, June 11, 2003 4:35 AM
>     *To:* Sebastian Zander
>     *Cc:* MEYER,JEFFREY D (HP-Cupertino,ex1); [email-masked]
>     *Subject:* Re: [ipfix] extensibility of the ipfix protocol
>
>     Hi Jeff and Sebastian,
>
>>     Jeff,
>>
>>     comments below
>>
>     One argument for treating the scope differently is that the scope
>     must be applied to all options data records.
>     And the Scope is an important data type. If you don't know about
>     it, you can't apply the "action" that is associated with the
>     options flow record.
>     For example: if the scope is the line card X and the sampling rate
>     Y and if the collector doesn't know abouth the scope for the line
>     card, the exporter doesn't know that the number of bytes should be
>     multipled by the sampling rate.
>     Ok, that would be almost the same for an unknow data types in the
>     flow data record, but that one could just be reported, without
>     taking an "action"
>
>>>
>>
>>     to the number of option fields? If yes this seems to waste lots
>>     of space if the
>>     scope does not change (and it is not the case in the example). If
>>     not (probably
>>     the authors meant M and N) the question is how to map the scope
>>     fields to the
>     Sebastian, you are right that the intention is to have M and N.
>     This has been corrected in the latest draft.
>     How to map the scope fields to the options fields when there is
>     more than 1 scope field?
>     The scope can be limited further by listing multiple scopes which
>     all have to match at the same time.
>
>>
>>
>>     Can scopes be combined such as e.g. this option is per interface
>>     vs. this option
>>>     From: Sebastian Zander [mailto:[email-masked]]
>>>     Cc: [email-masked]
>>>>     resist the overwhelming temptation to create new types, it can
>>>>     be defined in their own information model.
>>>>
>>>>
>>>>     Regards,
>>>>
>>>>      Jeff Meyer
>>>>
>>>>     From: Sebastian Zander [mailto:[email-masked]]
>>>>     To: [email-masked]