Subject: [IPFIX] Changing the definition of IE #37 commonPropertiesId

From: Juergen Quittek <[email-masked]>
Date: Thu, 04 Jan 2007 13:24:03 +0100

Dear all,

I wish all of you a very happy new year 2007!

IPFIX-INFO was evaluated by the IESG in November and we have to
react on the feedback we got.  You'll find it at
<https://datatracker.ietf.org/public/pidtracker.cgi?command=print_ballot&ballot_id=1870&filename=draft-ietf-ipfix-info>

Collecting also open issues from the mailing list I found a request to
change the definition of Information Element #37 commonPropertiesId.
If I got the discussion right, the request is to change the definition
as follows.

OLD
      An identifier of a set of common properties that is unique per
      Observation Domain.  Typically, this Information Element is used
      to link to information reported in separate records.
NEW
      An identifier of a set of common properties that is locally unique
      per Observation Domain and Transport Session.  Typically, this
      Information Element is used to link to information reported in
      separate Data Records.

Is this really what has been requested?
Would anybody have objections against this change?

Thanks,

    Juergen

--On 08.12.06 17:07 +0000 Andrew Johnson wrote:

> Benoit Claise wrote:
>>   Andrew,
>>> Benoit Claise wrote:
>>>> Andrew,
>>> [SNIP]
>>>>> My concern wasn't really directed at the draft itself, but just the
>>>>> name "common properties".  From IPFIX-PROTO:
>>>>>  "A Flow is defined as a set of IP packets passing an Observation
>>>>> Point in the network during a certain time interval.  All packets
>>>>> belonging to a particular Flow have a set of common properties."
>>>>>
>>>>> And then from IPFIX-INFO:
>>>>>   Description of commonPropertiesId:
>>>>>      An identifier of a set of common properties that is unique per
>>>>>      Observation Domain.  Typically, this Information Element is used
>>>>>      to link to information reported in separate records.
>>>>>
>>>>> The terminology is almost identical.  The commonPropertiesId refers
>>>>> to "a set of common properties" that is quite different from ones
>>>>> that determine a Flow.
>>>>
>>>> What about something such as?
>>>> Description of commonPropertiesId:
>>>>      An identifier of a set of common properties between different
>>>> Flows.
>>>>      The that is unique per Observation Domain.      Typically, this
>>>> Information Element is used
>>>>      to link to information reported in separate records.
>>>
>>> I suggested something quite similar to Elisa:
>>>  An identifier of a set of properties that is common across more
>>>  that one Flow.  Typically, this Information Element is used to
>>>  link to information reported in separate records.  The identifier
>>>  is unique per Transport Session and Observation Domain.
>>
>> Agreed. I actually saw Elisa's email after I proposed this :(
>> One small correction though, after re-reading the Flow and Flow Record
>> definitions.
>> We should be using
>>
>>     The commonPropertiesID is an identifier of a set of common properties
>>     that is locally unique per Observation Domain and Transport Session.
>>     Typically, this Information Element is used to link to information
>>     reported in separate  Data Records .
>
> This looks fine.
>
>
>>>>>>> Is there any need to define a new term for "Specific
>>>>>>> Properties"? These are just part of a normal Data Record.
>>>>>>> For example:
>>>>>>> "... The rows are then referenced by the Specific Properties
>>>>>>> by using the appropriate value for the Common Properties
>>>>>>> identifier."
>>>>>>>
>>>>>>> could simply be:
>>>>>>> "... The rows are then referenced by Data Records by
>>>>>>> using the appropriate value for the Common Properties
>>>>>>> identifier."
>>>>>>
>>>>>> In your example, the "Data Records" doesn't mean the full data
>>>>>> records before applying the redundancy reduction mechanism but only
>>>>>> a part of it.
>>>>>> So it might be confusing... That's the trick we found.
>>>>>>
>>>>>> However, I could agree that the draft would work without a formal
>>>>>> (upper case) definition.... So do you have the same idea for the
>>>>>> Common Properties?
>>>>>
>>>>> Considering there is a commonPropertiesID, I am quite happy with you
>>>>> referring to Common Properties.
>>>>
>>>> Ok. So let's see how the draft looks like if we remove the "Specific
>>>> Properties" formal definition but if we keep the "Common Properties"
>>>> one.
>>>
>>> Sounds good.
>> I went through the process of trying "Specific Properties" into
>> "specific properties" and I have to admit that some of the sentences and
>> figures look strange and even sometimes misleading. For example, "Common
>> and specific Properties" is strange. So I have to agree with Elisa with
>> keeping the definition in order to clarify the document. Anyway you
>> replied in your last email "I just don't think it is a useful
>> definition, but I don't feel that strongly.". So I think we can close
>> this one.
>
> Yes, I think the draft _could_ be written without out but if it's a lot
> of work then it's probably not worth the effort.
>
>
>>>>> There seems to be a bit of a conceptual difference here though.  You
>>>>> seem to imagine that an Exporting Process has a set of IPFIX Data
>>>>> Records which it then applies some processing to to get some new
>>>>> special IPFIX Data Records and some Common Property options.  The
>>>>> original set of Data Records may never actually exist, depending on
>>>>> how the scheme is implemented.
>>>>
>>>> Sure, this is an implementation detail, but conceptually, we want to
>>>> reduce "something". How do you call this "something"?
>>>
>>> There are already several names for the something.  Most common is Flows
>>> or data.  Sometimes even "Flow Record" but I think that term is too
>>> easily confused with "Data Record".  Data Record has a very specific
>>> meaning and is not what is meant here.  A Data Record is defined by
>>> an IPFIX Template.  The Flows that are sent from the Metering Process
>>> to the Exporting Process have no set format.
>>
>> Acknowledged.
>> In the draft, I choose Flow Record, as the definitions say "The Metering
>> Process generates Flow Records"
>> I checked every occurrence of Data Record and Flow Record in the draft
>> and you're right there are some minor corrections to be done.
>>
>> OLD
>>    Regardless of the fields content, flow records with common
>>    properties export the same fields in every single flow record.
>> NEW
>>    Regardless of the fields content, flow records with common
>>    properties export the same fields in every single data record.
>>
>> OLD
>>    The Common Properties Flow Records MUST be sent on a reliable SCTP
>>    stream.
>> NEW
>>    The Common Properties Data Records MUST be sent on a reliable SCTP
>>    stream.
>>
>> OLD
>>    The Collecting Process can either store the Flow Records as they
>>    arrive, without reconstructing the initial Flow Record, or
>>    reconstruct the initial Flow Record.
>> NEW
>>    The Collecting Process can either store the Data Records as they
>>    arrive, without reconstructing the initial Flow Record, or
>>    reconstruct the initial Flow Record.
>>
>> OLD
>>    In addition there
>>    might be less storage capacity required at the Collector side if the
>>    Collector decides to store the Flow Records as they arrive, without
>>    reconstructing the initial Flow Record.
>> NEW
>>    In addition there
>>    might be less storage capacity required at the Collector side if the
>>    Collector decides to store the Data Records as they arrive, without
>>    reconstructing the initial Flow Record.
>>
>> OLD:
>>    The assignment of Flow Records to common attributes could be done
>>    via the commonPropertiesID and alternatively via the templateID
>>    Information Element.
>> NEW:
>>    The assignment of common attributes could be done
>>    via the commonPropertiesID and alternatively via the templateID
>>    Information Element.
>>
>> OLD
>>    On the other hand, these methods require additional resources on
>>    both the Exporter and the Collector.  The Exporter has to manage
>>    Common Properties information and to assign commonPropertiesId
>>    values to Flow Records.
>> NEW
>>    On the other hand, these methods require additional resources on
>>    both the Exporter and the Collector.  The Exporter has to manage
>>    Common Properties information and to assign commonPropertiesId
>>    values.
>>
>> If you see some more required changes, let us know.
>
> These changes all look fine.
>
>
>> For figures 2 to 6, I think the "record" should be "Data Record".
>>    Figure 2: Common and Specific Properties exported in the same record
>>    Figure 3: Common and Specific Properties exported in different records
>>    Figure 4: Common and Specific Properties exported in the same record
>>    Figure 5: Multiple Common (above) and Specific Properties (below)
>>    exported in different records
>>    Figure 6: Common and Specific Properties represented in one record
>> Alternatively, if we want to avoid the term record or Data Record in figures, let's use the trick of figure 7:
>>    Figure 7: Common and Specific (packet) Properties exported
>>    separately
>> So let's use "exported together" versus "exported separately"
>
> OK.
>
>
>> [SNIP]
>>>>>>> Are they meaningful in IPFIX? i.e. per the PROTO will an option
>>>>>>> with scope but no data be deemed invalidated by the Collector.
>>>>>>> Scope without data is the same as never having recieved the
>>>>>>> mapping in the first place. I think that it would be up to the
>>>>>>> implementation but it seems like a good idea so if we can't add
>>>>>>> it to the PROTO can we put it in the implementation guidelines?
>>>>>>>
>>>>>>> How do you propose to withdraw common properties when they are
>>>>>>> associated with a template rather than the Common Properties index?
>>>>>>
>>>>>> I don't ;) This is the reason why the use the Common Properties
>>>>>> index is a cleaner design.
>>>>>
>>>>> I think withdrawal messages should work in general for options
>>>>> data, so why not send an empty record with the template as scope?
>>>> I guess that you wanted to say:
>>>>     I think withdrawal messages should work in general for options
>>>>     data, so why not send an empty record with the _common property_
>>>> as scope?
>>>> Well, I prefer to have a clear design so that this draft is really
>>>> independent, with its withdrawal mechanism.
>>>
>>> Then using the template ID to identify Common Properties can be
>>> mentioned in passing, which I guess it is, in 8.3, however it is
>>> also discussed in a little detail in section 4.1.1 (and figure 9),
>>> and section 7.  It seems that these details should be removed.
>>
>> I understand the issue that we put at the same level the common
>> properties with commonPropertiesID and with templateID, while we don't
>> speak about the template withdrawal in this draft.
>> Here is a proposal:
>> We remove the templateID from section 3 and 4, we put it into section
>> "8.3 commonPropertiesID vs. TemplateID scope"
>> And we mention in that paragraph that:
>> - the template withdrawal, as specified in [IPFIX-PROTO], can be used to
>> withdraw Template
>> - the template ID could be used for common properties if the common
>> properties remain constant for the live of the template.
>>   Otherwise we enter into some problems such as: what's happening when
>> we want to change the common properties in the template.
>> - that we don't need template withdrawal in this is case, as the
>> template withdrawal is covered by [IPFIX-PROTO] already.
>> - If the Template is withdrawn then the associated data is invalid too.
>
> I think your suggestion is a good one, but I think using the Template ID
> may be a bad idea for Common Properties and perhaps it shouldn't be
> suggested at all in the draft.  I will start a new thread to specifically
> discuss this issue.
>
>
>>>> Otherwise, you're telling in this I-RFC the standard track IPFIX
>>>> protocol should consider an empty record as a withdrawal. This could
>>>> be true a true assumption, but this is not clearly explained/specified!
>>>
>>> Not at all.  Either withdrawing options in this way is valid in IPFIX,
>>> in which case Reducing Redundancy can use it, or it a valid technique
>>> for anybody.  Now I'm not declaring that this is a technique for all
>>> is within the scope of the draft.  If red-red is going to use this
>>> technique though we should decide in the WG that it is valid and say
>>> so in the implementation guidelines or the "Managing Scope" draft.
>>>
>>>
>>>>>>> 4. Specifications for bandwidth saving information export
>>>>>>>
>>>>>>> What is being specified here that isn't true for ALL options as
>>>>>>> described in IPFIX-PROTO?
>>>>>>
>>>>>> What do you suggest?
>>>>>
>>>>> Well it seems that section three really explains almost the same
>>>>> thing 3 times.  The section 4 explains most of it again (with a
>>>>> bit more info) another 3 times.
>>>>>
>>>>> I would prefer section 3 to simply explain the technique.  Walk
>>>>> through how the information from a Flow is split into two parts,
>>>>> and then add the bit from section 4 how these two parts are
>>>>> exported.
>>>>>
>>>>> Using more than one CPid in a record should be part of the
>>>>> advanced techniques once the basics are made clear.  Per packet
>>>>> should just be a detailed example at the end.
>>>>>
>>>>>>> I think section 3 does a better job of explaining how all this
>>>>>>> works. I'm really not sure what section 4 adds.
>>>>>>
>>>>>> The idea was to put the specs in section 4 while the section 3 was
>>>>>> the high level solution.
>>>>>
>>>>> Rereading it, I see that now, but there seems to be a fair amount
>>>>> of repetition.
>>>>
>>>> Let me ask the question differently. Do you see this repetition as
>>>> problem?
>>>
>>> Yes.  I think the draft would be cleared if how options work was left
>>> to IPFIX-PROTO.  red-red should reference that, then build describe
>>> exactly how the basic Common Properties work for the simple case, from
>>> concept to execution.
>>
>> You should be speaking about the beginning of section 4.
>> As many clarifications to the scope concept had to be made on the
>> mailing list, we thought that adding this extra paragraph of
>> explanations would benefit the readers...
>
> I think you can still refer to section 3.4.2 of [IPFIX-PROTO], but
> by all means clarify what you need to.
>
>
>>> The other variations on this can then build upon the proper foundation
>>> of the simple case.  This way the reader can understand how the system
>>> works before you introduce the concept of including more than one
>>> commonPropertiesId in a Template/Data Record.
>>>
>>>
>>> [SNIP]
>>>>>>> The text reads:
>>>>>>> Common Properties IDs are unique per SCTP association and per
>>>>>>> Observation Domain. If the Collecting Process receives a common
>>>>>>> Properties ID which has already been received but which has not
>>>>>>> previously been withdrawn (i.e. a commonPropertiesID from the same
>>>>>>> Exporter Observation Domain with the same Template ID received on
>>>>>>> the SCTP association), then the Collecting Process MUST shutdown
>>>>>>> the association.
>>>>>> This is similar to [IPFIX-PROTO]
>>>>>>
>>>>>>  Template IDs are unique per SCTP association and per Observation
>>>>>>  Domain.  If the Collecting Process receives a Template which has
>>>>>>  already been received but which has not previously been
>>>>>> withdrawn    (i.e. a Template Record from the same Exporter
>>>>>> Observation Domain     with the same Template ID received on the
>>>>>> SCTP association), then    the Collecting Process MUST shutdown the
>>>>>> association.
>>>>>>
>>>>>>>
>>>>>>> and:
>>>>>>> If the Collecting Process receives a Common Properties Withdrawal
>>>>>>> message for a Common Properties that it has not received before on
>>>>>>> this SCTP assocation, it MUST reset the SCTP association, discard
>>>>>>> the IPFIX Message, and SHOULD log the error as it does for
>>>>>>> malformed IPFIX Messages.
>>>>>>> I think this might go against IPFIX-PROTO.
>>>>>> This is similar to [IPFIX-PROTO]
>>>>>
>>>>> These are similar to what IPFIX-PROTO says about templates, but that
>>>>> doesn't mean the text can just be used to talk about Data Records sent
>>>>> for Options Templates.
>>>>>
>>>>> Option data withdrawal messages are not mentioned in IPFIX-PROTO,
>>>>> although I think they can work.  However, in general options data with
>>>>> the same scope as an earlier record will just overwrite the older
>>>>> record.
>>>>
>>>> See my previous remark about an independent draft.
>>>
>>> What?
>>>
>>> AFAIK red-red is an information draft aimed at explaining how a technique
>>> you can use with an IPFIX compliant Exporter/Collector.
>>> IPFIX-PROTO says
>>> nothing about withdrawing options data and if this draft says:
>>>  If the Collecting Process receives a common Properties ID which has
>>>  already been received but which has not previously been withdrawn ...
>>>  then the Collecting Process MUST shutdown the association.
>>> Then it is expanding on what IPFIX-PROTO says and is no longer just a
>>> way of using IPFIX but an extension to it.
>>> If we accept that withdrawing
>>> options is part of IPFIX today then this is about the only extension
>>> being asked for.
>>>
>>> I can see why you want it, but the problem you are trying to solve here
>>> is the same for ALL options, not just Common Property definitions.
>>
>> This draft, even if this will be an Informational RFC, is an extension
>> on how to use IPFIX in this particular case of redundancy reduction.
>> So, yes, this draft can be more specific. We're trying to solve the only
>> case of the Common Property.
>
> Yes, I think I'll have to agree that some extensions will be needed,
> even if some of the techniques aren't really prohibited in IPFIX-PROTO,
> and fall into a grey area.
>
> Please detail the extensions in an Appendix as most of the methods used
> in the draft are from IPFIX-PROTO and there are only a few areas that
> you expand upon.  It would be good for the implementer of an IPFIX
> compliant device to clearly see what the differences are.
>
>
>>> [SNIP]
>>>> In this draft, we require a mechanism such as the following:
>>>> "If the Common Properties are not refreshed before that lifetime
>>>> has   expired, the Collecting Process MUST discard the
>>>> corresponding   definition of the commonPropertiesID and any current
>>>> and future   associated Data Records. In which case, an alarm MUST be
>>>> logged. "
>>>
>>> Why is it required for common properties and not other options?
>> There are differences between the data records sent by option data record.
>> Example1: NetFlow V9 exports the ifIndex, IfDescr matchings in Option
>> Template Records
>> Example2: We could think to export the refresh timeout in Option
>> Template Records
>> Example3: The Common Properties ID for redundancy reduction.
>> Even if the 3 examples use Option Template Records, the fundamental
>> difference is in the meaning of the I.E.s:
>>     - example 1 and 2 contain I.E.s that are self explaining: they mean
>> something by themselves. We don't need those for the data records to be
>> meaningfull
>>     - example 3: the Common Properties ID MUST be present in order to
>> decode the initial Flow Record
>> This implies that:
>>     1. we can't treat all options template record the same way.
>>     2. "If the Common Properties are not refreshed before that lifetime
>> has expired, the Collecting Process MUST discard the corresponding
>> definition of the commonPropertiesID and any current and future
>> associated Data Records. In which case, an alarm MUST be logged. "
>>
>>> Why is it ok to use a mechanism that is not part of the protocol?
>> Because this draft is an extension.
>> If we didn't have to specify some more specifications compared to
>> [IPFIX-PROTO], then an extra example in [IPFIX-AS] would be enough.
>>>
>>>> Because there is no withdraw message with UDP:
>>>> " If UDP is selected as the transport protocol, the commonPropertiesID
>>>>   Template Withdraw Messages MUST not be used, as this method is
>>>>   inefficient due to the unreliable nature of UDP."
>>>> So I'm wondering why we are discussing this?
>>>
>>> Because you're trying to extend the protocol because you don't think
>>> it's good enough.  While I agree that the protocol could do a better
>>> job of handling options, I think this extension is out of scope of
>>> the draft.
>>>
>>>
>>>
>>>
>>>
>>>> Q: Furthermore you shouldn't dictate the the record is discarded,
>>>> just because you
>>>> can't tell a piece of potentially trivial information about a flow
>>>> doesn't mean it's meaningless.
>>>> A: without the Common Properties, we can't decode the records. So we
>>>> must discard them
>>>
>>> Surely that depends on what the Common Properties show.  Perhaps the
>>> particular properties that are common aren't important.  Similarly,
>>> if a normal full Data Record was received without something the
>>> exporter thought was necessary, it could be discarded.
>> Exactly, this is the reason why I made a distinction between the
>> different examples in my previous comment.
>>>
>>>
>>>> Q: The Common Properties in question could be anything.
>>>> A: I think we should limit this draft to Common Properties
>>>
>>> It wasn't a question and your answer makes no sense.  I'm saying
>>> that the properties defined by a particular cpID could be something
>>> that is not important to the Collector.  Perhaps it's the
>>> classOfService field and the collector doesn't use it.
>> We agree, aren't we? See my previous remark (example 1 and 2)
>
> We agree.  With an identifier such as cpID, if it's not defined then
> you have no idea what you are missing.  With data associated with
> an IE that is valid by itself (ifIndex for example) then you don't
> need the definition, and in fact wouldn't know you were missing it.
>
> regards
>
> Andrew

-- 
Juergen Quittek        [email-masked]       Tel: +49 6221 4342-115
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 4342-155
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.netlab.nec.de

_______________________________________________
IPFIX mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ipfix

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 04 Jan 2007 13:24:03 +0100

Dear all,

OLD

Thanks,

    Juergen

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 04 Jan 2007 14:32:57 +0000

Jeurgen

This change looks fine to me.

Andrew

Juergen Quittek wrote:
>>> Alternatively, if we want to avoid the term record or Data Record in 
>>> figures, let's use the trick of figure 7:

---

From: Andrew Johnson <[email-masked]>
Date: Thu, 04 Jan 2007 14:32:57 +0000

Jeurgen

Andrew

---

From: Brian Trammell <[email-masked]>
Date: Thu, 04 Jan 2007 10:01:28 -0500

This change looks good to me. Indeed, I thought it had already been made.

Regards, and happy 2007 to all,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 04 Jan 2007 10:01:28 -0500

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 04 Jan 2007 17:23:10 -0500

Juergen and All,

A couple points on the IESG discusses (may have others in the future,
these are just the ones on which I know I have a definite opinion on
beyond simple agreement, and a proposed solution for).

> Magnus Westerlund:
> 
> Discuss [2006-11-15]:
> Section 3.1.12:
> macAddress
> 
>    The type "macAddress" represents a string of 6 octets.
> 
> How is this field going to be able to handle IEEE EUI-64 (64 bits) MACs? I do
> suggest that you either define two types of MAC or expand it to 64 bit.

I'd suggest a second EUI-64 macAddress data type and EUI-64 IEs to match
, as this is an address size issue much like IPv4/IPv6 and should be
treated as such.

> Section 5.5.10:
> 
> The tcpWindowSize is useless unless you capture and store the WSCALE option
> that appears on the SYN [RFC1323]. The document needs to specify inforamtion
> element and capture method for this information. 

I'd prefer to define this as the actual window size (wsize * wscale), as
opposed to separate export of size and scale, unless there's a specific
use case for creating another IE here. Doing so would require changing
the size of this IE to unsigned32.

Regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Thu, 04 Jan 2007 17:23:10 -0500

Juergen and All,

Regards,

Brian