Subject: [IPFIX] I-D Action: draft-ietf-ipfix-flow-selection-tech-06.txt

From: [email-masked]
Date: Mon, 23 May 2011 02:21:28 -0700

A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the IP Flow Information Export Working Group of the IETF.

	Title           : Flow Selection Techniques
	Author(s)       : Salvatore D&#39;Antonio
                          Tanja Zseby
                          Christian Henke
                          Lorenzo Peluso
	Filename        : draft-ietf-ipfix-flow-selection-tech-06.txt
	Pages           : 25
	Date            : 2011-05-23

   Flow selection is the process of selecting a subset of flows from all
   flows observed at an observation point.  Flow selection reduces the
   effort of post-processing flow data and transferring flow records.
   This document describes motivations for flow selection and presents
   flow selection techniques.  It provides an information model for
   configuring flow selection techniques and discusses what information
   about a flow selection process should be exported.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-flow-selection-tech-06.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-ipfix-flow-selection-tech-06.txt

---

From: "Zseby, Tanja" <[email-masked]>
Date: Mon, 23 May 2011 11:30:09 +0200

Hi all,

we worked on a major revision of the flow selection draft and just submitted a new version (see below). Among other changes we now provide a much improved classification of methods, which is more consistent with the PSAMP packet selection documents. 
Many thanks to all who provided comments. 

Changes:
 - Flow recording process removed
 - Clarification of difference between flow selection and packet selection
 - Distinguished flow filtering and flow sampling similar to PSAMP
 - Flow selection in the metering process either before aggregation or after aggregation
 - Integrated Flow-state dependent packet selection
 - Supporting arbitrary  key space subsets with property match flow filtering
 - Removed timestamp IEs for reporting 
 - Mediator integrated in first picture
 - Configuration parameters and IEs redefined and re-named
 - Many rewording, shortened several paragraphs to improve readability

Kind regards
Tanja

-----Ursprüngliche Nachricht-----
Von: [email-masked] [mailto:[email-masked]] Im Auftrag von [email-masked]
Gesendet: Montag, 23. Mai 2011 11:21
An: [email-masked]
Cc: [email-masked]
Betreff: [IPFIX] I-D Action: draft-ietf-ipfix-flow-selection-tech-06.txt

This Internet-Draft can be retrieved at:
ftp://ftp.ietf.org/internet-drafts/draft-ietf-ipfix-flow-selection-tech-06.txt
_______________________________________________
IPFIX mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipfix

---

From: Nevil Brownlee <[email-masked]>
Date: Wed, 25 May 2011 09:27:35 +1200

Hi all:

Thanks very much to all the authors of the flow-selection draft,
it does indeed look much better.  Since there's so much new material
in it, we need to run a new WG LC on it - that start's today, and
will finish on Friday, 10 June.  Do please read it anew, and send
comments to the list.  Brief comments such as "yes, this looks fine
now" are welcome, of course!

Cheers, Nevil

-- 
---------------------------------------------------------------------
  Nevil Brownlee                    Computer Science Department | ITS
  Phone: +64 9 373 7599 x88941             The University of Auckland
  FAX: +64 9 373 7453   Private Bag 92019, Auckland 1142, New Zealand

---

From: Brian Trammell <[email-masked]>
Date: Thu, 26 May 2011 16:20:35 +0200

Hi, Nevil, all,

The following is intended as a WGLC comment on flow-selection-tech. It will not be my only one; however, it identifies an issue I've seen in a cursory review of the document which I believe could benefit from some discussion:

Is this document intended to update RFC 5470? I'm not convinced this is necessary; e.g. RFC 6235 defines a flow-modifying intermediate process, and shows how it could be integrated into various stages within the IPFIX Architecture, without introducing a new model of how data is handled in the architecture. If each additional intermediate process definition updates the architecture, the architecture becomes confusing, and subsequently useless.

If the authors and the WG believe it is necessary to update 5470 to support flow selection, I would _strongly_ suggest reviewing all new terminology, as the proposed terms collide with existing terms of art in network measurement. Specifically, there are two serious problems here. First, I'm not aware of any use of the word "classification" with respect to the associating of packets with flows; classification instead seems primarily to be used when associating a label with a packet of a flow (e.g., application classification). Second, while packets are indeed aggregated into flows, the word "aggregation" in this draft is used in a completely different way than in either -ipfix-a9n or in RFCs 5982 or 6183. This could lead to confusion.

More detailed comments to follow.

Best regards,

Brian

---

From: Benoit Claise <[email-masked]>
Date: Fri, 27 May 2011 12:38:24 +0200

Dear Tanja,

One quick remark (a thorough review should follow) regarding this I.E.

7.5.  fsFlowEntryTotalCount

    Description:

       This Information Element specifies the current number of all Flow
       Entries that form the parent population as input to the Flow
       Selection Process.

    Abstract Data Type: unsigned64

    ElementId: TBD5

    Status: Proposed

    Units: Flow Entries

In Cisco, years ago, we have defined the I.E. #3 with the same concept. 
It is used when we do router-based aggregation, which is exported with 
NetFlow version 8. It represents the number of flow that we're given as 
input to the router-based aggregation process.
See 
http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9_ps6601_Products_White_Paper.html 
(which btw, we're busy updating)

So this TBD5 should be 3.  Does it make sense?

Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Fri, 27 May 2011 13:11:59 +0200

Hi, Benoit,

I disagree, I think, for two reasons.

First, if IE3 is defined with parallel semantics to IE1 and IE2 (octet and packet delta count), it should provide _delta_ count, not _total_ count, semantics (i.e., subsequent exports are a delta from last export, not a replacement value).

Second, this is an IE with a much more restricted semantic meaning; it refers to Flow Entries (a new bit of flow selection terminology, the difference between which and "Flow" and "Flow Record" is still a bit unclear to me) that are given as _input_ to a (undefined but I presume contextually resolvable) Flow Selection Process. 

This is somewhat different than the Flow analog of Packet and Octet Delta Count.

(However, if IE3 was _not_ defined by Cisco with this in mind, and really means "the total number of flows seen at the (Metering Process or analogue) _before_ flow processing", objection withdrawn.)

Best regards,

Brian

On May 27, 2011, at 12:38 PM, Benoit Claise wrote:

> In Cisco, years ago, we have defined the I.E. #3 with the same concept. It is used when we do router-based aggregation, which is exported with NetFlow version 8. It represents the number of flow that we're given as input to the router-based aggregation process.
> See http://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9_ps6601_Products_White_Paper.html (which btw, we're busy updating)
>

---

From: Paul Aitken <[email-masked]>
Date: Fri, 27 May 2011 13:51:50 +0100

Brian,

The flow count in field #3 follows the same semantics as the octet count 
in field #1 and the packet count in field #2.

We use it internally in our "top talkers" feature to count the number of 
aggregated flows, though we don't currently export it.

As to the terminology issue: a "flow" is the set of packets which are 
observed at the input to the metering process, while a "flow record" is 
the output from the metering process. I look forward to finding out what 
"Flow Entries" are :-)

P.

---

From: Benoit Claise <[email-masked]>
Date: Fri, 27 May 2011 14:42:31 +0200

Hi Brian,
> (However, if IE3 was _not_ defined by Cisco with this in mind, and really means "the total number of flows seen at the (Metering Process or analogue) _before_ flow processing", objection withdrawn.)
See RFC5101

  Metering Process

       The Metering Process generates Flow Records.  Inputs to the
       process are packet headers and characteristics observed at an
       Observation Point, and packet treatment at the Observation Point
       (for example, the selected output interface).

Considering that there are no flows as input to the Metering Process, ...

	"the total number of flows seen at the (Metering Process or analogue) _before_ flow processing"

That gives

	"the total number of flows seen at the Flow Selection Process before flow processing"

Which is equivalent to

Description:

And yes, this is how we used this I.E. in Cisco. See Figure D, RFC 6183

                      IPFIX (Data Records)
                                ^ ^
    +---------------------------|-|------------------------+
    | Original Exporter         | |                        |
    |                           | |                        |
    |     .---------------------|-+-------------------.    |
    |    .----------------------+--------------------.|    |
    |    |           Exporting Process(es)           |'    |
    |    '----------------------^--------------------'     |
    |                           | |                        |
    |     .---------------------|-+-------------------.    |
    |    .----------------------+--------------------.|    |
    |    |          Intermediate Process(es)         |'    |
    |    '---------^-----------------------^---------'     |
    |       (**)   |      Data Records     |   (**)        |
    |   .----------+---------.   .---------+----------.    |
    |   | Metering Process 1 |...| Metering Process N |    |
    |   '----------^---------'   '---------^----------'    |
    |              |                       |               |
    |  .-----------+---------.   .---------+-----------.   |
    |  | Observation Point 1 |...| Observation Point N |   |
    |  '-----------^---------'   '---------^-----------'   |
    +--------------|-----------------------|---------------+
                   |                       |
             Packets coming into Observation Points

(**) this is where value is metered.

Btw, Tanja, this is one of my important WG-LAST points.
http://tools.ietf.org/html/draft-ietf-ipfix-flow-selection-tech-06, 
Figure 1, should build on the top of the RFC6183, and clearly explains 
where is the Flow Intermediate Process.

Regards, Benoit.

---

From: Brian Trammell <[email-masked]>
Date: Fri, 27 May 2011 15:41:36 +0200

Hi, all,

The following are the balance of my WGLC comments on the flow selection draft.

Summary: The document has made _significant_ progress in the right direction, in my opinion, but requires at least one more revision before it is ready for the IESG. The issues are as follows:

1. As Benoit said, and in refinement of my earlier message, the model which the document uses should be based on an Intermediate (Flow) Selection Process (IFSP) as in RFC 6183, instead of adding new concepts to 5470. The case in section 4.1 is a bit of a special one, to address an uncommon requirement; in addition, it isn't really a flow selection process at all, but rather a packet selection process that selects packets on the same criteria that by which packets are separated into flows by the flow key. Therefore it can be noted here, I think, that the _function_ run within an IFSP can run _as a packet selection process_ before flow generation within an MP without having to hack new processes into the 5470 model. Section 6 will have to be updated a bit to fit this change as well.

2. The document should use a different term for "aggregation" (of packets into flows) to avoid collision with terminology related to aggregation in RFCs 5982 and 6183.

3. The document should use a different term for "classification" to avoid confusion with the use of this term with respect to "flow classification" and labeling. I realize 5470 uses the verb "classify" to refer to this process; I really wish it didn't. 

Aggregation and classification _together_ comprise the "assignment" of packets to flows, and flow "generation". I'm afraid I don't have any better suggestions at this time.

4. Terminology based on terminology borrowed by analogy or reference to the packet selection techniques draft is a good idea; however, these terms should be defined in section . Additionally, terminology through the entire document should be reviewed for proper capitalization.

5. Within section 6, it is not clear how to apply this configuration data model. Should this be harmonized with ipfix-configuration-model?

6. In section 7, I would suggest unifying Paul and Benoit's comments on the subject of fsFlowRecordTotalCount; additionally, the IE should be named flowDeltaCount to keep parallel with IEs 1 and 2. Consider renaming the IEs in general to drop the "fs" prefix. perhaps "postSelection" or something similar (as with the pre/post IEs). Review other IEs to ensure they are not covered by existing IEs (especially those "pre-selection", as with IE 3)

7. What is a Flow Entry? Should this be defined in this document, or referenced as terminology from another? If this is what I think it is (reporting flow cache size inline), I'm not certain it belongs here in flow selection (perhaps in a document on metering process monitoring?).

Best regards,

Brian

---

From: Brian Trammell <[email-masked]>
Date: Fri, 27 May 2011 15:41:59 +0200

Hi, Benoit,

Understood. Okay, this makes sense, explained like this... Thanks.

(See WGLC in the other email)

Best regards,

Brian

On May 27, 2011, at 2:42 PM, Benoit Claise wrote:

> http://tools.ietf.org/html/draft-ietf-ipfix-flow-selection-tech-06, Figure 1, should build on the top of the RFC6183, and clearly explains where is the Flow Intermediate Process.
> 
> Regards, Benoit.
>> Best regards,
>> 
>> Brian
>>

---

From: Paul Aitken <[email-masked]>
Date: Fri, 27 May 2011 15:12:07 +0100

Benoit,

> Considering that there are no flows as input to the Metering Process, ...

A mediator may receive packets containing flow records from other exporters.

The number of flows contained in those those packets counts as a "packet 
characteristic" per the RFC 5101 definition above.

If the mediator wanted to report the number of bytes, packets, and flows 
which it observed, I'd expect it to use fields #1, #2 and #3.

P.

---

From: Benoit Claise <[email-masked]>
Date: Fri, 27 May 2011 16:16:39 +0200

> A mediator may receive packets containing flow records from other 
A mediator, so an Intermediate Process.
We're in synch.

Regards, Benoit
>
> The number of flows contained in those those packets counts as a 
> "packet characteristic" per the RFC 5101 definition above.
>
> If the mediator wanted to report the number of bytes, packets, and 
> flows which it observed, I'd expect it to use fields #1, #2 and #3.
>
> P.