Subject: Re: [Fwd: [ipfix-protocol] PROTO-21: metering process statistics option template]

From: Maurizio Molina <[email-masked]>
Date: Thu, 21 Oct 2004 23:03:21 +0200 (CEST)

Benoit,

some comments on your proposal.

First, since the Exporter Reliability Statistics options template, as you say "specifies lack of reliability in the Metering and Exporting process" I think its name is not appropriate. A possibility would be to call it "Meter and Exporter Reliability Statistics options template", but I don’t like it either, as it mixes things up. My suggestion would be to call it "Meter Reliability Statistics options template", and move the "dropped flows" Information element into another options template (see later). Also, I would insert into this "Meter Reliability Statistics options template"two other information elements giving the time of the first and the last dropped packet. This time information is easy to get and update (if the metering process drops a packet and can update the droppedFUByteCount, it can also easily update a timestamp of this event: probably the timestamp is in the same data structure giving the number of bytes of the dropped packet….). The time information is useful to
 understand the dynamic of the aggregated flow of all the packets dropped by the metering process. The value of the ipfixOption field should become "METER_RELIABILITY_STATS"

Second, as said, I would put the "dropped flows" information element in another option template, call it "Exporter Reliability Statistics Option template", and give "EXPORTER_RELIABILITY_STATS" as a value to its ipfixOption field. This option template should contain at least the ipfixOption, the droppedFlows and the time information elements (with the same meaning as in the Meter Reliability Statistics options template). My suggestion, however, would be to add other four information elements to it, and namely:

-droppedFAPacketCount packets in the dropped flows

-droppedFAByteCount bytes in the dropped flows

-timeFirstFADropped time of the first packet received in one of the dropped flows

-timeLastFADropped time of the last packet received in one of the dropped flows

These are, once again, information easy to get and to update: each time you decide to drop a flow you have the full flow record, so you know how many packets and bytes you’re dropping, and you also have the time of the first and last packet of the record: you can compare them with the timeFirstFADropped and timeLastFADropped and update them if necessary. In this way you have the full dynamic of the aggregated flow resulting from all the drop in the exporting process.

This would be more in line with my original suggestion, whose intent was to give the collector an idea of the amount and dynamic of the LOST traffic, so that it can compare it with the REPORTED traffic.

A final note: a flow could be dropped not only because of starvation of the exporting process, but also because of policies of the flow recording process. Perhaps the wording can be adjusted to say this….

Regards,

Maurizio

---------------------------- Original Message ----------------------------
Subject: [ipfix-protocol] PROTO-21: metering process statistics option
template From: "Benoit Claise" 
Date: Wed, October 20, 2004 13:06
To: [email-masked]
--------------------------------------------------------------------------

Dear all,

The IPFIX protocol version 5 lists this remaining issue.
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-protocol-05.txt

PROTO-21: Do we need to define some mandatory content of the
metering process statistics option template?
- Maurizio suggested text on the mailing list
- proposal after IETF60: look at what is required in [IPFIX-
REQ], and come up with a minimum set of data types.
Note: the ipfixOption is not yet defined in the [IPFIX-INFO]; needed 
for the metering process statistics

The text in section "8.1 The Metering Process Statistics Option Template"
is not complete:

*8.1 The Metering Process Statistics Option Template

The Metering Process Statistics Option Template defines the Metering 
Process Statistics with the export of the following Information 
Elements [IPFIX-INFO]:
ipfixOption The value MUST be METER_STATS
observationDomain Source ID
lostFlows Flows not exported due to resource
starvation
lostFlowsPackets Packets in the lost Flows
lostFlowsBytes Bytes in the lost Flows
droppedPacketCount Packets dropped by Metering Process
at the Observation Point
droppedByteCount Bytes dropped by Metering Process at the
Observation Domain
time; When this record was generated

The minimum set of Information Element in the Metering Process 
Statistics Option Template is: ipfixOption, observationDomain, 
lostFlows, time *

<>**I think this version comes from discussion between Maurizio and I, 
but I'm not too sure anymore, as this is pretty old. Maurizio proposed 
some new text at http://ipfix.doit.wisc.edu/archive/2407.html

We should discuss whether we need some mandatory information element 
exported. If yes, which one? I reviewed the IPFIX requirement draft 
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-reqs-16.txt, and the
applicable sections are:

5.1. Reliability

The metering process must either be reliable or the absence of
reliability must be known and indicated. The metering process is
reliable if each packet passing the observation point is metered
according to the configuration of the metering process. If, e.g. due
to some overload, not all passing packets can be included into the
metering process, then the metering process must be able to detect
this failure and to report it.

6.5. Regular Reporting Interval

The exporting process should be capable of reporting measured traffic
data regularly according to a given interval length.

6.3.2 -> discusses also the data transfer reliability: somehow related.

A couple of points to Maurizio's proposal, that leads to the simplified
proposal below: - I don't think we need to export the Observation Domain
ID, as the scope field will determine whether the scope is the interface,
the Observation Domain ID, etc... - There are 2 different requirements in
RFC3917, so we should have 2 different options template records - I think
we should remove the MUST, SHOULD, MAY requirement in the flow expiration
section because the requirements imposed constraints on the metering
process, and we are standardizing the IP Flow Information eXport.
As a consequence, we should not put MUST, SHOULD, MAY requirements on
the Metering Process in order to export those Option Templates.
- If you have a metering process resource starvation how can you know how
many flows have been lost? It is impossible to determine how many packets
are in that flow because you can not keep flow information. All you can do
is keep the count of how many packets and bytes that did not fit into the
flow table.

Here is the text I propose.

* 8. Specific Reporting Requirements

Some specific Options Templates and Options Templates Records are 
necessary to provide extra information about the Flow Records and 
about the Metering Process.

The ipfixOption Field [IPFIX-INFO], always included in these
specific Options Templates, defines the type of information sent in 
the Option Template / Option Template Record pair. For example, if 
the ipfixOption [IPFIX-INFO] value is METER_STATS, then the Option 
Template will specify information about the Metering Process
statistics.

The Option Template and Option Template Records defined in these
sub-sections are not mandatory to implement as they impose some
constraints about the Metering Process implementation: this document
specifies the protocol to export the records, not the Metering Process
implementation. However, if the specific Option Templates are
implemented, they should implemented as specified in these
sub-sections. In any case, if the ipfixOption Information Element is
present, it MUST always be the first Information Element in the Option
Template so that the Collector can quickly determine which specific
Option Template Record is received.

The minimum set of Information Elements is always specified in these 
Specific IPFIX Options Templates. Nevertheless, extra Information 
Elements may be used in these specific Options Templates. **

8.1 The Metering Process Statistics Option Template

The Metering Process Statistics Option Template specifies the Metering 
Process Statistics. It contains the following Information
Elements [IPFIX-INFO]
ipfixOption The value is METERING_STATS
exportedOctetCount The number of all octets reported
by the Exporting Process to the
Collecting Process.
exportedPacketCount The number of all packets reported
by the exporting process to the
Collecting Process.
** exportedFlowCount The number of all flows records
reported by the Exporting Process
to the Collecting Process.** *
* time The time at which the record was generated

The Exporting Process should export the Metering Process Statistics Option
Template Record on a regular basis or based on some export policy. This
periodicity or export policy should be configurable. The **Metering 
Process Statistics Option Template could be extended with other
Information Elements.***
*
* *8.2 The Exporter Reliability Statistics Option Template **

The Exporter Reliability Option Template specifies information
about lack of reliability in the Metering and Exporting process.
**It contains the following Information Elements [IPFIX-INFO]:*
*
ipfixOption The value is RELIABILITY_STATS
** droppedFUPacketCount Packets dropped by Metering Process ** 
droppedFUOctetCount Bytes dropped by Metering Process** * * 
droppedFlows Number of flow records lost due to resources
starvation at Exporting Process
** time The time at which the record was generated
** *
*The Exporting Process should export the Exporter Reliability Statistics
Option Template Record on a regular basis or based on some export policy.
This periodicity or export policy should be configurable. **The **Exporter
Reliability Statistics **Option Template could be extended with other
Information Elements.

*Note that the ipfixOption, time, droppedFUPacketCount,
droppedFUByteCount, and droppedFlows are not yet specified in [IPFIX-INFO]

Regards, Benoit.
*
*

Dear all,

The IPFIX protocol version 5 lists this remaining issue.http://www.ietf.org/internet-drafts/draft-ietf-ipfix-protocol-05.txt PROTO-21: Do we need to define some mandatory content of the    metering process statistics option template?        - Maurizio suggested text on the mailing list        - proposal after IETF60: look at what is required in [IPFIX-   REQ], and come up with a minimum set of data types.    Note: the ipfixOption is not yet defined in the [IPFIX-INFO]; needed    for the metering process statisticsThe text in section "8.1 The Metering Process Statistics Option Template" is not complete:

 8.1      The Metering Process Statistics Option Template     The Metering Process Statistics Option Template defines the Metering    Process Statistics with the export of the following Information    Elements [IPFIX-INFO]:        ipfixOption             The value MUST be METER_STATS        observationDomain       Source ID        lostFlows               Flows not exported due to resource                                  starvation        lostFlowsPackets        Packets in the lost Flows        lostFlowsBytes          Bytes in the lost Flows         droppedPacketCount      Packets dropped by Metering Process                             at the Observation Point    droppedByteCount        Bytes dropped by Metering Process at the                                  Observation Domain         time;                   When this record was generated     The minimum set of Information Element in the Metering Process    Statistics Option Template is: ipfixOption, observationDomain,    lostFlows,
 time 
<>I think this version comes from discussion between Maurizio and I, but I'm not too sure anymore, as this is pretty old. Maurizio proposed some new text at http://ipfix.doit.wisc.edu/archive/2407.html 

We should discuss whether we need some mandatory information element exported. If yes, which one? I reviewed the IPFIX requirement draft http://www.ietf.org/internet-drafts/draft-ietf-ipfix-reqs-16.txt, and the applicable sections are: 5.1. Reliability

The metering process must either be reliable or the absence of
reliability must be known and indicated. The metering process is
reliable if each packet passing the observation point is metered
according to the configuration of the metering process. If, e.g. due
to some overload, not all passing packets can be included into the
metering process, then the metering process must be able to detect
this failure and to report it.
6.5. Regular Reporting Interval

The exporting process should be capable of reporting measured traffic
data regularly according to a given interval length.

6.3.2 -> discusses also the data transfer reliability: somehow related.

A couple of points to Maurizio's proposal, that leads to the simplified proposal below:- I don't think we need to export the Observation Domain ID, as the scope field will determine whether the scope is the interface, the Observation Domain ID, etc...- There are 2 different requirements in RFC3917, so we should have 2 different options template records- I think we should remove the MUST, SHOULD, MAY requirement in the flow expiration section because the requirements imposed constraints on the metering process, and we are standardizing the IP Flow Information eXport.  As a consequence, we should not put MUST, SHOULD, MAY requirements on the Metering Process in order to export those Option Templates.- If you have a metering process resource starvation how can you know how many flows have been lost? It is impossible to determine how many packets arein that flow because you can not keep flow information. All you can do is keep the count of how many packets and bytes that did not fit into
 theflow table.Here is the text I propose. 8.     Specific Reporting Requirements        Some specific Options Templates and Options Templates Records are    necessary to provide extra information about the Flow Records and    about the Metering Process.          The ipfixOption Field [IPFIX-INFO], always included in these    specific Options Templates, defines the type of information sent in    the Option Template / Option Template Record pair.  For example, if    the ipfixOption [IPFIX-INFO] value is METER_STATS, then the Option    Template will specify information about the Metering Process    statistics.    The Option Template and Option Template Records defined in these sub-sections    are not mandatory to implement as they impose some constraints about    the Metering Process implementation: this document specifies the protocol   to export the records, not the Metering Process implementation.   However, if the specific Option Templates are implemented, they should   
 implemented as specified in these sub-sections.  In any case, if the    ipfixOption Information Element is present, it MUST always be the first    Information Element in the Option Template so that the Collector   can quickly determine which specific Option Template Record is received.        The minimum set of Information Elements is always specified in these    Specific IPFIX Options Templates.  Nevertheless, extra Information    Elements may be used in these specific Options Templates.    8.1      The Metering Process Statistics Option Template  The Metering Process Statistics Option Template specifies the Metering Process Statistics. It contains the following Information Elements [IPFIX-INFO]      ipfixOption             The value is METERING_STATS      exportedOctetCount      The number of all octets reported                              by the Exporting Process to the                             Collecting Process.     exportedPacketCount     The number of all packets reported
                              by the exporting process to the                             Collecting Process.     exportedFlowCount       The number of all flows records                              reported by the Exporting Process                             to the Collecting Process.      time                    The time at which the record was generated  The Exporting Process should export the Metering Process Statistics Option Template Record on a regular basis or based on some export policy. This periodicity or export policy should be configurable. The Metering Process Statistics Option Template could be extended with other Information Elements. 8.2      The Exporter Reliability Statistics Option Template The Exporter Reliability Option Template specifies information about lack of reliability in the Metering and Exporting process.It contains the following Information Elements [IPFIX-INFO]:     ipfixOption             The value is RELIABILITY_STATS     droppedFUPacketCount   
 Packets dropped by Metering Process     droppedFUOctetCount     Bytes dropped by Metering Process      droppedFlows            Number of flow records lost due to resources                              starvation at Exporting Process     time                    The time at which the record was generated The Exporting Process should export the Exporter Reliability Statistics Option Template Record on a regular basis or based on some export policy. This periodicity orexport policy should be configurable. The Exporter Reliability Statistics Option Template could be extended with other Information Elements.Note that the ipfixOption, time, droppedFUPacketCount, droppedFUByteCount, and droppedFlows are not yet specified in [IPFIX-INFO]Regards, Benoit.

---------------------------------
Nuovo Yahoo! Messenger E' molto più divertente: Audibles, Avatar, Webcam, Giochi, Rubrica… Scaricalo ora!

---

From: Benoit Claise <[email-masked]>
Date: Fri, 22 Oct 2004 12:32:38 +0200

Maurizio,

I inserted all your proposals below in the next version of [IPFIX-PROTO] 
I will post today, for the sake of continuing the discussion on some 
written text.

Regards, Benoit.

>     First, since the Exporter Reliability Statistics options template,
>     as you say "specifies lack of reliability in the Metering and
>     Exporting process" I think its name is not appropriate. A
>     possibility would be to call it "Meter and Exporter Reliability
>     Statistics options template", but I don't like it either, as it
>     mixes things up. My suggestion would be to call it "Meter
>     Reliability Statistics options template", and move the "dropped
>     flows" Information element into another options template (see
>     later). Also, I would insert into this "Meter Reliability
>     Statistics options template"two other information elements giving
>     the time of the first and the last dropped packet. This time
>     information is easy to get and update (if the metering process
>     drops a packet and can update the droppedFUByteCount, it can also
>     easily update a timestamp of this event: probably the timestamp is
>     in the same data structure giving the number of bytes of the
>     dropped packet....). The time information is useful to understand
>     the dynamic of the aggregated flow of all the packets dropped by
>     the metering process. The value of the ipfixOption field should
>     become "METER_RELIABILITY_STATS"
>
>     Second, as said, I would put the "dropped flows" information
>     element in another option template, call it "Exporter Reliability
>     Statistics Option template", and give "EXPORTER_RELIABILITY_STATS"
>     as a value to its ipfixOption field. This option template should
>     contain at least the ipfixOption, the droppedFlows and the time
>     information elements (with the same meaning as in the Meter
>     Reliability Statistics options template). My suggestion, however,
>     would be to add other four information elements to it, and namely:
>
>     These are, once again, information easy to get and to update: each
>     time you decide to drop a flow you have the full flow record, so
>     you know how many packets and bytes you're dropping, and you also
>     have the time of the first and last packet of the record: you can
>     compare them with the timeFirstFADropped and timeLastFADropped and
>     update them if necessary. In this way you have the full dynamic of
>     the aggregated flow resulting from all the drop in the exporting
>     process.
>
>     This would be more in line with my original suggestion, whose
>     intent was to give the collector an idea of the amount and dynamic
>     of the LOST traffic, so that it can compare it with the REPORTED
>     traffic.
>
>     A final note: a flow could be dropped not only because of
>     starvation of the exporting process, but also because of policies
>     of the flow recording process. Perhaps the wording can be adjusted
>     to say this....
>
>     Regards,
>
>     Maurizio
>
>The text in section "8.1 The Metering Process Statistics Option Template" is not complete:
>
>    
>
>     <>I think this version comes from discussion between Maurizio and
>     I, but I'm not too sure anymore, as this is pretty old. Maurizio
>     proposed some new text at
>     http://ipfix.doit.wisc.edu/archive/2407.html
>
>     We should discuss whether we need some mandatory information
>     element exported. If yes, which one? I reviewed the IPFIX
>     requirement draft
>     http://www.ietf.org/internet-drafts/draft-ietf-ipfix-reqs-16.txt,
>     and the applicable sections are:
>
>         5.1. Reliability
>
>A couple of points to Maurizio's proposal, that leads to the simplified proposal below:
>- I don't think we need to export the Observation Domain ID, as the scope field will determine whether the scope is the interface, the Observation Domain ID, etc...
>- There are 2 different requirements in RFC3917, so we should have 2 different options template records
>- I think we should remove the MUST, SHOULD, MAY requirement in the flow expiration section because the requirements imposed constraints on the metering process, and we are standardizing the IP Flow Information eXport.
>  As a consequence, we should not put MUST, SHOULD, MAY requirements on the Metering Process in order to export those Option Templates.
>- If you have a metering process resource starvation how can you know how many flows have been lost? It is impossible to determine how many packets are
>in that flow because you can not keep flow information. All you can do is keep the count of how many packets and bytes that did not fit into the
>flow table.
>
>
>   The Option Template and Option Template Records defined in these sub-sections 
>   are not mandatory to implement as they impose some constraints about 
>   the Metering Process implementation: this document specifies the protocol
>   to export the records, not the Metering Process implementation.
>   However, if the specific Option Templates are implemented, they should 
>   implemented as specified in these sub-sections.  In any case, if the 
>   ipfixOption Information Element is present, it MUST always be the first 
>   Information Element in the Option Template so that the Collector
>   can quickly determine which specific Option Template Record is received.
>     
>Template Record on a regular basis or based on some export policy. 
>This periodicity or export policy should be configurable. The **Metering 
>Process Statistics Option Template could be extended with other Information 
>Elements.***
>*
>**     droppedFUPacketCount    Packets dropped by Metering Process
>**     droppedFUOctetCount     Bytes dropped by Metering Process** *
>*     droppedFlows            Number of flow records lost due to resources 
>**     time                    The time at which the record was generated **
>*
>*The Exporting Process should export the Exporter Reliability Statistics Option 
>Template Record on a regular basis or based on some export policy. This periodicity or
>export policy should be configurable. **The **Exporter Reliability Statistics **Option 
>Template could be extended with other Information Elements.
>
>*Note that the ipfixOption, time, droppedFUPacketCount, droppedFUByteCount, and droppedFlows are not yet specified in [IPFIX-INFO]
>
>Regards, Benoit.
>*
>*
>
>
>
>    
>
> ------------------------------------------------------------------------
> *Nuovo Yahoo! Messenger* 
> <http://it.rd.yahoo.com/mail/taglines/*http://it.messenger.yahoo.com> 
> E' molto più divertente: Audibles, Avatar, Webcam, Giochi, Rubrica... 
> Scaricalo ora!