Subject: RE: [ipfix] Middle box functions

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Wed, 11 Jun 2003 13:12:39 -0400

Ganesh,

  I think I see your point.  In rereading the Middlebox specification:

    http://www.ietf.org/rfc/rfc3234.txt

  They provide a list (in section 2) of 22 different functional activities
of various
categories of middleboxes.

  They also provide a list (in section 2.23) of functions which are NOT
considered to be middlebox activities.

  In this non-middlebox list they have the following text:

     Wiretaps and snoopers in general - if they are working correctly,
     they have no impact on traffic, so do not require analysis.

  In effect they are saying a Middlebox by definition ALTERS the default
flow of IP packets.  It would seem to me that IPFIX does not specify any
such behavior.  The observer is simply watching, not altering the flow.

  Hence, I would recommend REMOVING references to Middleboxes.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: Ganesh Sadasivan [mailto:[email-masked]]
> Sent: Tuesday, June 10, 2003 5:03 PM
> To: Tal Givoly
> Cc: [email-masked]
> Subject: Re: [ipfix] Middle box functions
> 
> 
> Sorry for the brevity.
> There is a line in the architecture spec. on the function of 
> Observation Domain.
> 
> "Perform appropriate middle-box functions to translate the
> flow information."
> 
> 1. This is not a function of the Observation Domain as it is 
> a logical block
> rather
> than a functional block.
> 2.It would be moslty a IPFIX protocol function in which case 
> I would like to put
> 
> in some more words. So if someone can provide a few sentences 
> stating what is
> intended by "Middle box functions" I can put it into the arch. spec.
> 
> Thanks
> Ganesh
> 
> Tal Givoly wrote:
> 
> > Ganesh,
> >
> > I personally didn't quite understand the question, perhaps 
> you would like to
> > elaborate or provide some more context for us to clarify/respond.
> >
> > Thanks,
> >
> > Tal
> >
> > -----Original Message-----
> > From: majordomo listserver 
> [mailto:[email-masked]]On Behalf
> > Of Ganesh Sadasivan
> > Sent: Tuesday, June 10, 2003 1:49 PM
> > To: [email-masked]
> > Subject: [ipfix] Middle box functions
> >
> > Can someone restate what  "middle-box functions" need to be done
> > by an IPFIX device?
> >
> > Thanks
> > Ganesh
> >
> > --
> > Help        mailto:[email-masked] and say 
> "help" in message
> > body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> 
> --
> Help        mailto:[email-masked] and say "help" 
> in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 11 Jun 2003 11:58:20 -0700

Hi,
   I want to add this figure into the architecture spec. Please go through
this send
   send any comments/corrections.

                    Packet(s) coming into Observation Point(s)
                    |                                   |
                    |                                   |
                    v                                   v
  +-----------------+-------------------------+   +-----+------+
  |           Metering Process on an          |   |            |
  |              Observation Point            |   |            |
  |  packet header capturing                  |   |            |
  |         |                                 |   | Metering   |
  |    timestamping                           |   | Process    |
  |         |                                 |   | on an      |
  |         v                                 |   | Observation|
  |  +------+                                 |   | Point      |
  |  |      |                                 |   |            |
  |  |   sampling Si (1:1 in case of no       |   |            |
  |  |      |          sampling)              |   |            |
  |  | classifying Fi (NULL when No criteria) |   |            |
  |  |      |                                 |   |            |
  |  +------+                                 |   |            |
  |         |                                 |   |            |
  +---------+---------------------------------+   +-----+------+
            |                                           |
          Flows (identified by observation domain)    Flows
            +----...                                    +--------+...
            |                                           v
            |     +-------------------------------------+----------------+
            |     |             Flow Recording Process(*)                |
            |     | +----------------------+     +------------------+    |
            |     | | Flow data base       |<----|Provide non-flow  |    |
            |     | | (includes flows      |     | information (Eg. |    |
            |     | | from all obs.        |     | router state)    |    |
            |     | | points in an obs.    |     +------------------+
|.....
            |     | | domain)              |                             |
            |     | |                      |     +------------------+    |
            |     | |                      |<----|Maintain aggregate|    |
            |     | |                      |     | statistics       |    |
            |     | +----------------------+     +------------------+    |
            |     +-------------------------------+----------------------+
            |                                     |
            |        Flow Database (identified by observation domain)
            |
+------------------------....
            v                                     v
+-----------+-------------------------------------+--------------------------+

|           |            Export Process           |                          |

|           |                                     +----------------------... |

| +---------+------------------------------------------------------------+   |

| |         v                  IPFIX Protocol                            |   |

| |+---------------------------------+  +-------------------------------+|   |

| || Rules for                       |  |Functions                      ||   |

| || - Picking & sending templates   |  |- Packetize selected control   ||   |

| || - Picking & sending data records|->|  & data information into      ||   |

| || - Timing out flows              |  |  IPFIX export packet.         ||...|

| || - Encoding template & data      |  |- Handle export errors         ||   |

| || - Selecting flows for export(*) |  |- Handle timeouts & overloads  ||   |

| |+---------------------------------+  +-------------------------------+|   |

| |                                                                      |   |

| +-----------------------------+----------------------------------------+   |

|                               |                                            |

|                               +-----------------------------------------...|

|                     IPFIX exported packet                                  |

|                               |                                            |

| +-----------------------------+----------------------------------------+   |

| |                        Transport  Protocol                           |   |

| +-----------------------------+----------------------------------------+   |

|                               |                                            |

+-------------------------------+--------------------------------------------+

                                |
                                v
                        IPFIX export packet to collector.

(*) indicates that the block is optional.

Thanks
Ganesh

---

From: "Tal Givoly" <[email-masked]>
Date: Wed, 11 Jun 2003 21:49:53 -0700

Ganesh,

Usually, diagrams are useful for illustration of concepts, but aren't ideal
for specification. If you would like this to be a specification of the inner
workings of the metering process and exporting process, I would recommend
you add the following:
- Identification of all modules and interconnects
- Text that describes each of these modules and interconnects to
disambiguate it for the reader.

Regarding the content itself - I believe aggregation is not clear. Also
annonymization location isn't clear.

It seems to me to make more sense to have two distinct diagrams - one at the
high level showing MP (many to 1) -> EP. Then it would be possible to blow
up the MP and the EP independently and discuss them independently. The EP
depends primarily on the protocol features while the MP would have the
support for most of the capabilities such as timestamping, and others that
you discuss.

Tal
-----Original Message-----
From: majordomo listserver [mailto:[email-masked]]On Behalf Of
Ganesh Sadasivan
Sent: Wednesday, June 11, 2003 11:58 AM
To: [email-masked]
Subject: [ipfix] IPFIX-ARCH : Functional flow diagram.

  Hi,

+-----------+-------------------------------------+-------------------------
-+
  |           |            Export Process           |
|
  |           |
+----------------------... |
  | +---------+------------------------------------------------------------+
|
  | |         v                  IPFIX Protocol                            |
|
  | |+---------------------------------+  +-------------------------------+|
|
  | || Rules for                       |  |Functions                      ||
|
  | || - Picking & sending templates   |  |- Packetize selected control   ||
|
  | || - Picking & sending data records|->|  & data information into      ||
|
  | || - Timing out flows              |  |  IPFIX export packet.
||...|
  | || - Encoding template & data      |  |- Handle export errors         ||
|
  | || - Selecting flows for export(*) |  |- Handle timeouts & overloads  ||
|
  | |+---------------------------------+  +-------------------------------+|
|
  | |                                                                      |
|
  | +-----------------------------+----------------------------------------+
|
  |                               |
|
  |
+-----------------------------------------...|
  |                     IPFIX exported packet
|
  |                               |
|
  | +-----------------------------+----------------------------------------+
|
  | |                        Transport  Protocol                           |
|
  | +-----------------------------+----------------------------------------+
|
  |                               |
|

+-------------------------------+-------------------------------------------
-+
                                  |
                                  v
                          IPFIX export packet to collector.
  (*) indicates that the block is optional.

  Thanks
  Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 12 Jun 2003 07:20:08 -0700

Tal,

   I agree with you that the diagram does not show the complete
interconnects.
   Sorry, I did not mention that this diagram is in addition to the
already existing
   diagram in the arch spec that shows the interconnects which needs
some
   modification.
   I was planning to just put the important functions in the diagram and
the detail
   text for each module would  in the different sections. But then maybe
we need
   to still modify the text in the blocks.
   What did you mean by aggregation? Is it of flows ? There is text in
"flow recording
   process" which mentions about aggregate statistics. But that has
nothing to do with
   aggregation itself.
   Anonymization IMO is a protocol function and we need to add this to
the rule and
   function.

Thanks
Ganesh

Tal Givoly wrote:

>  Ganesh,Usually, diagrams are useful for illustration of concepts, but
> aren't ideal for specification. If you would like this to be a
> specification of the inner workings of the metering process and
> exporting process, I would recommend you add the following:-
> Identification of all modules and interconnects- Text that describes
> each of these modules and interconnects to disambiguate it for the
> reader.Regarding the content itself - I believe aggregation is not
> clear. Also annonymization location isn't clear.It seems to me to make
> more sense to have two distinct diagrams - one at the high level
> showing MP (many to 1) -> EP. Then it would be possible to blow up the
> MP and the EP independently and discuss them independently. The EP
> support for most of the capabilities such as timestamping, and others
> that you discuss.Tal-----Original Message-----
> From: majordomo listserver [mailto:[email-masked]]On
> Behalf Of Ganesh Sadasivan
>         I want to add this figure into the architecture spec.
>      Please go through this send
>        |              Observation Point            |
>      Metering   |
>        |         |                                 |   | on
>      an      |
>        |         v                                 |   |
>      Observation|
>        |  +------+                                 |   |
>      Point      |
>        |  |      |          sampling)              |
>        |  +------+                                 |
>                  +----...
>      +--------+...
>      +-------------------------------------+----------------+
>                  |     |             Flow Recording
>      Process(*)                |
>                  |     | | points in an obs.    |
>      +------------------+    |.....
>                  |     | | domain)
>      |                             |
>                  |     | |                      |
>      +------------------+    |
>                  |     | |                      |<----|Maintain
>      aggregate|    |
>                  |     | |                      |     |
>      statistics       |    |
>      +-------------------------------+----------------------+
>                  |                                     |
>      -----------+-------------------------------------+--------------------------+
>
>      |           |            Export Process
>      |                          |
>      |           |
>      +---------+------------------------------------------------------------+
>      |
>      | |         v                  IPFIX
>      Protocol                            |   |
>      +-----------------------------+----------------------------------------+
>      |
>      |
>      |                                            |
>      |
>      |                     IPFIX exported
>      packet                                  |
>      +-----------------------------+----------------------------------------+
>      |
>      | |                        Transport
>      Protocol                           |   |
>      |
>      +-----------------------------+----------------------------------------+
>      |
>      |
>      |                                            |
>
>      -------------------------------+--------------------------------------------+
>
>                                      |
>                                      v

---

From: "Tal Givoly" <[email-masked]>
Date: Thu, 12 Jun 2003 15:41:50 -0700

I probably wasn't clear enough... that happens.
   What did you mean by aggregation? Is it of flows ?
Yes, aggregation of flows based on some policy.

   Anonymization IMO is a protocol function and we need to add this to the
rule and
   function.

I cannot see how anonymization is part of the protocol (or the Export
Process). The EP should be abstracted from this level of detail, as it
should from aggregation, sampling, etc.

Are the rest of the points I made clearer?

Tal

  -----Original Message-----
  From: Ganesh Sadasivan [mailto:[email-masked]]
  Sent: Thursday, June 12, 2003 7:20 AM
  To: Tal Givoly
  Cc: [email-masked]
  Subject: Re: [ipfix] IPFIX-ARCH : Functional flow diagram.

  Tal,
     I agree with you that the diagram does not show the complete
interconnects.
     Sorry, I did not mention that this diagram is in addition to the
already existing
     diagram in the arch spec that shows the interconnects which needs some
     modification.
     I was planning to just put the important functions in the diagram and
the detail
     text for each module would  in the different sections. But then maybe
we need
     to still modify the text in the blocks.
     What did you mean by aggregation? Is it of flows ? There is text in
"flow recording
     process" which mentions about aggregate statistics. But that has
nothing to do with
     aggregation itself.
     Anonymization IMO is a protocol function and we need to add this to the
rule and
     function.

  Thanks
  Ganesh

  Tal Givoly wrote:

     Ganesh,Usually, diagrams are useful for illustration of concepts, but
aren't ideal for specification. If you would like this to be a specification
of the inner workings of the metering process and exporting process, I would
recommend you add the following:- Identification of all modules and
interconnects- Text that describes each of these modules and interconnects
to disambiguate it for the reader.Regarding the content itself - I believe
aggregation is not clear. Also annonymization location isn't clear.It seems
to me to make more sense to have two distinct diagrams - one at the high
level showing MP (many to 1) -> EP. Then it would be possible to blow up the
MP and the EP independently and discuss them independently. The EP depends
primarily on the protocol features while the MP would have the support for
most of the capabilities such as timestamping, and others that you
discuss.Tal-----Original Message-----
    From: majordomo listserver [mailto:[email-masked]]On Behalf
Of Ganesh Sadasivan
    Sent: Wednesday, June 11, 2003 11:58 AM
    To: [email-masked]
    Subject: [ipfix] IPFIX-ARCH : Functional flow diagram.

      Hi,
         I want to add this figure into the architecture spec. Please go
through this send
         send any comments/corrections.
                          Packet(s) coming into Observation Point(s)
                          |                                   |
                          |                                   |
                          v                                   v
        +-----------------+-------------------------+   +-----+------+
        |           Metering Process on an          |   |            |
        |              Observation Point            |   |            |
        |  packet header capturing                  |   |            |
        |         |                                 |   | Metering   |
        |    timestamping                           |   | Process    |
        |         |                                 |   | on an      |
        |         v                                 |   | Observation|
        |  +------+                                 |   | Point      |
        |  |      |                                 |   |            |
        |  |   sampling Si (1:1 in case of no       |   |            |
        |  |      |          sampling)              |   |            |
        |  | classifying Fi (NULL when No criteria) |   |            |
        |  |      |                                 |   |            |
        |  +------+                                 |   |            |
        |         |                                 |   |            |
        +---------+---------------------------------+   +-----+------+
                  |                                           |
                Flows (identified by observation domain)    Flows
                  +----...                                    +--------+...
                  |                                           v
                  |
+-------------------------------------+----------------+
                  |     |             Flow Recording Process(*)
|
                  |     | +----------------------+     +------------------+
|
                  |     | | Flow data base       |<----|Provide non-flow  |
|
                  |     | | (includes flows      |     | information (Eg. |
|
                  |     | | from all obs.        |     | router state)    |
|
                  |     | | points in an obs.    |     +------------------+
|.....
                  |     | | domain)              |
|
                  |     | |                      |     +------------------+
|
                  |     | |                      |<----|Maintain aggregate|
|
                  |     | |                      |     | statistics       |
|
                  |     | +----------------------+     +------------------+
|
                  |
+-------------------------------+----------------------+
                  |                                     |
                  |        Flow Database (identified by observation domain)
                  |
+------------------------....
                  v                                     v

+-----------+-------------------------------------+-------------------------
-+
      |           |            Export Process           |
|
      |           |
+----------------------... |
      |
+---------+------------------------------------------------------------+   |
      | |         v                  IPFIX Protocol
|   |
      | |+---------------------------------+
+-------------------------------+|   |
      | || Rules for                       |  |Functions
||   |
      | || - Picking & sending templates   |  |- Packetize selected control
||   |
      | || - Picking & sending data records|->|  & data information into
||   |
      | || - Timing out flows              |  |  IPFIX export packet.
||...|
      | || - Encoding template & data      |  |- Handle export errors
||   |
      | || - Selecting flows for export(*) |  |- Handle timeouts & overloads
||   |
      | |+---------------------------------+
+-------------------------------+|   |
      | |
|   |
      |
+-----------------------------+----------------------------------------+   |
      |                               |
|
      |
+-----------------------------------------...|
      |                     IPFIX exported packet
|
      |                               |
|
      |
+-----------------------------+----------------------------------------+   |
      | |                        Transport  Protocol
|   |
      |
+-----------------------------+----------------------------------------+   |
      |                               |
|

      Thanks
      Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Fri, 13 Jun 2003 12:20:28 -0700

Tal,

Aggregation imo should happen in the flow recording proces. But some
rules
of aggreation should also be available to the protocol also for purposes
of
export.

If you look at the EP , it is an abstract entrity that includes the
transport
protocol also. Anonymization is specific to the application and should
be carrier out
in the protocol unless we are dependent on other standard ways to doing
the same.
If we are using some standard ways it should appear between IPFIX
protocol and
transport protocol. Is this correct?
If so I can modify the figure slightly.

Thanks
Ganesh

Tal Givoly wrote:

>  I probably wasn't clear enough... that happens.   What did you mean
> by aggregation? Is it of flows ? Yes, aggregation of flows based on
> some policy.
>    function.I cannot see how anonymization is part of the protocol (or
> the Export Process). The EP should be abstracted from this level of
> detail, as it should from aggregation, sampling, etc.Are the rest of
> the points I made clearer?Tal
>
>         I agree with you that the diagram does not show the
>      complete interconnects.
>         Sorry, I did not mention that this diagram is in addition
>      to the already existing
>         I was planning to just put the important functions in the
>      diagram and the detail
>         text for each module would  in the different sections.
>      But then maybe we need
>         What did you mean by aggregation? Is it of flows ? There
>      is text in "flow recording
>         process" which mentions about aggregate statistics. But
>      that has nothing to do with
>         aggregation itself.
>         Anonymization IMO is a protocol function and we need to
>      add this to the rule and
>         function.
>
>      Thanks
>      Ganesh
>
>
>      Tal Givoly wrote:
>
>     > Ganesh,Usually, diagrams are useful for illustration of
>     > concepts, but aren't ideal for specification. If you would
>     > like this to be a specification of the inner workings of
>     > the metering process and exporting process, I would
>     > recommend you add the following:- Identification of all
>     > modules and interconnects- Text that describes each of
>     > these modules and interconnects to disambiguate it for the
>     > reader.Regarding the content itself - I believe
>     > aggregation is not clear. Also annonymization location
>     > isn't clear.It seems to me to make more sense to have two
>     > distinct diagrams - one at the high level showing MP (many
>     > to 1) -> EP. Then it would be possible to blow up the MP
>     > and the EP independently and discuss them independently.
>     > The EP depends primarily on the protocol features while
>     > the MP would have the support for most of the capabilities
>     > such as timestamping, and others that you
>     > [mailto:[email-masked]]On Behalf Of Ganesh
>     > Sadasivan
>     >         I want to add this figure into the
>     >      architecture spec. Please go through this send
>     >        |              Observation Point
>     >      |   | Metering   |
>     >      |   | on an      |
>     >      |   | Observation|
>     >        |  +------+
>     >      |   | Point      |
>     >        |  |      |          sampling)
>     >        |  +------+
>     >      |<----|Maintain aggregate|    |
>     >                  |     | |                      |
>     >      | statistics       |    |
>     >      |           |            Export
>     >      Process           |                          |

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 16 Jun 2003 12:35:51 -0700

Hi,

   The following text describes the architecture for exporting control
   information. The protocol should have it own section on this topic.
   Please let me know your thoughts on the paragraphs below.

Exporting Control Information

   The Control Information is used by the collector to :
   - decode and interpret flow records.
   - understand the exporter state.

   As such sending control information from exporter in a timely and
   reliable manner is critical to the proper functionality of the IPFIX
   collector. The following approaches MAY be taken for the export of
   control information.

     1. Send all the control information pertaining to flow records
        prior to sending the flow records themselves. This includes any
        incremental changes that happens to the definition of the flow
        records.

     2. Notify on a near real time basis the state of the exporter to
        the collector. This includes all the changes like a
        configuration change that affects the flow behavior, changes to
        exporter resources that affects export rates, etc that which the

        collector needs to be aware of.

     3. For the reasons 1. and 2. it is preferred that the export of
        Control Information be done over a reliable connection. This MAY

        be done through a reliable transport or a partially reliable
        transport when the underlying network is reliable.

Thanks
Ganesh

---

From: "Reinaldo Penno" <[email-masked]>
Date: Tue, 17 Jun 2003 00:14:35 -0400

Hello Ganesh,

I guess everything looks okay except number 3...Some comments

You say that is it preferred that the export de done over a reliable
connection. Do you mean IPfix connection? If it the "preferred" method I
guess a SHOULD would be more appropriated.

Using terms like "partially reliable" could be opening a can of worms...What
is a partially reliable transport? And what you mean that the network is
reliable?

Anyway, my overall suggestion is to rewrite number 3 to make it less
ambiguous and open to interpretation.

regards,

Reinaldo

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 17 Jun 2003 12:45:25 -0700

Hi Reinaldo,

   I agree that the statements are fuzzy. How about this one?

  For the reasons 1. and 2., the export of the control information
  SHOULD be done such that the it reaches the collector reliably.
  One way to achieve this would be  to send the control
  information over a reliable transport.

Thanks
Ganesh