Subject: Re: Observation Domain [ Was Re: [ipfix] Changes to architecture doc.]

From: Juergen Quittek <[email-masked]>
Date: Mon, 02 Jun 2003 10:55:57 +0200

Ganesh,

Thank you for drawing the figure below. Although you say that there are
only slight modifications, to me they are important. I fully agree to
the new figure. Now, the observation domain really is a functional block
not including observation points, but associated with a set of
observation points.

I still do not like the terminology, because we call a functional block
a 'domain'. This is a minor issue, but anyway here is a suggestion:

In your figure below, the box called 'observation domain' is a
functional block. What about calling it 'flow recording process'
or similarly. Then you could declare the 'observation domain' of
of a flow recording process to be the set of observation points
associated with this flow recording process.

Cheers,

    Juergen
-- 
Juergen Quittek        [email-masked]        Tel: +49 6221 90511-15
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 90511-55
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.ccrle.nec.de

-- Ganesh Sadasivan wrote on 30 May 2003 08:31 -0700:

> Juergen,
>
>
> Juergen Quittek wrote:
>
>> Ganesh
>>
>> -- Ganesh Sadasivan wrote on 16 May 2003 14:35 -0700:
>>
>> > * Section 4.3 Observation Domain
>> > Functions of Observation domain.
>> > " * Encoding the flow records and sending them to the export process.
>> >   * Encoding the control information into templates and sending them to
>> >   the export process."
>> > The above 2 are already covered by IPFIX protocol.
>> >
>> > "  * Deciding which flow records/control information to export using
>> >      rules based on time, thresholds, configuration events etc.
>> >    * Aggregating flow records generated by one or more metering
>> >      processes."
>> > The above 2 are also IPFIX protocol function. So also is flow
>> > expiration.
>> >
>> > Observation point is a logical block which maintains flows for a
>>
>> Here you mean observation domain, not observation point.
>
> Yes. sorry for the confusion.
>
>>
>>
>> > a set of {observation point, metering process} (in a database form)
>> > and maintains aggregate statistics. All the other flow related
>> > functions should be handled by IPFIX protocol and metering process.
>> >
>>
>> I have a problem with the concept of an observation domain.  You say it
>> is a 'functional block', but in the figure it includes observation points
>> which are locations in the network and no functional blocks.  So the
>> concept of the observation domain is not a clean one.  I see an observation
>> domain as a useful concept for implementation design of IPFIX on a
>> device with multiple line cards, but not useful for the general IPFIX
>> architecture.
>
> The idea is to present a single identity for a group of observation points
> to the exporter. That is why it is mentioned as a logical block which
> encompasses a set of observation points. It does not matter whether the
> observation points are in a single board or different board or even different
> nodes. It is a question as to how the collector identifies a group of observation
>
> points.
> Ofcourse a degenrate case is a single observation point in an observation
> domain.
>
>>
>>
>> Also I do not see a reason, why the general architecture should define
>> an internal function block for "sending them [flow records] from [the
>> metering process] to the exporting process".  Why sending them, if they
>> are co-located?  Whether or not you locate metering process and exporting
>> process on different boards and need to send them over an internal bus
>> is a detail of an implementation design, but IMHO should not be part
>> of the general IPFIX architecture. And we are definitely not defining
>> a netwrok protocol between metering process and exporting process.
>>
>
> I am copying the figure down once again with slight modifications for clarity
> and please explain how this restricts to a class of implementtations.
>
>    +-----------------------------+-------------------------------------+
>    |              packet header capturing                              |
>    |                             |               METERING PROCESS      |
>    |                        timestamping         on an                 |
>    |                             |               observation point     |
>    |                             v                                     |
>    |                      +----->+                                     |
>    |                      |      |                                     |
>    |                      |   sampling Si (1:1 in case of no sampling) |
>    |                      |      |                                     |
>    |                      | classifying Fi (NULL when No criteria)     |
>    |                      |      |                                     |
>    |                      +------+                                     |
>    |                             |                                     |
>    +-----------------------------+-------------------------------------+
>                                  |
>                                  v                                      .
>                                Flows                                    .
>                                  +--------------------------------------+...
>                                  |
>                                  |
>    +-----------------------------+----------------------------------------+
>    |                             V    OBSERVATION DOMAIN                  |
>    |             +----------------------+         +------------------+    |
>    |             | Flow data base       |<--------|Provide non-flow  |    |
>    |             | (includes flows      |         | information (Eg. |    |
>    |             | from all obs.        |         | router state)    |    |
>    |             | points in an obs.    |         +------------------+    |
>    |             | domain)              |                                 |
>    |             |                      |         +------------------+    |
>    |             |                      |<--------|Maintain aggregate|    |
>    |             |                      |         | statistics       |    |
>    |             +----------------------+         +------------------+    |
>    |                                                                      |
>    +-----------------------------+----------------------------------------+
>                                  |
>                             Flow Database (identified by observation domain)
>    +-----------------------------+----------------------------------------+
>    |                             V    IPFIX PROTOCOL                      |
>    |             +----------------------+                                 |
>    |             | Rules for 1. picking&|                                 |
>    |             | sending templates    |                                 |
>    |             | 2.picking& sending   |                                 |
>    |             | data records 3. timi-|                                 |
>    |             | out flows 4. Encoding|                                 |
>    |             | template & data.     |                                 |
>    |             |                      |                                 |
>    |             |                      |                                 |
>    |             |                      |                                 |
>    |             +----------------------+                                 |
>    |                                                                      |
>    +----------------------------------------------------------------------+
>
>>
>> The requirements do intentionally not discuss how flow records are
>> shared between metering process and exporting process, because there
>> are many different potential hardware and software device architectures
>> that require different ways of communication between these processes.
>> I do not think the IPFIX architecture should request that the there
>> is an instance sending flow records from one process to the other,
>> because there are several alternatives which would become excluded
>> without need, for example shared memory (including remote memory access
>> from the exporter to a metering card).
>>
>
>  The idea is that any  function mentioned in the figure above
> as part of an observation domain is optional, but if such a functionality exists,
>
> it exists in the context of an observation domain. It is like pieces in a
> metering
> process. Not every one would want to do sampling and classifying. But they
> are still listed as part of metering.
>
>>
>> In the requirements document we mapped requirements concerning the
>> reports to the collecting process (for example the reporting period)
>> to the section of requirements for the exporting process. Consequently,
>> I would see several of the functions you listed for the observation
>> records as part of the exporting process (for example generating templates).
>
> If you are refering to section 4.3 of the arch. spec, yes they specify functions
> which
> are not done by the observation domain. I had mentioned in my first e-mail
> on this topic
> "> * Section 4.3 Observation Domain
>   > Functions of Observation domain.
>   > " * Encoding the flow records and sending them to the export process.
>   >   * Encoding the control information into templates and sending them to
>   >   the export process."
>   > The above 2 are already covered by IPFIX protocol.
>   > "  * Deciding which flow records/control information to export using
>   >      rules based on time, thresholds, configuration events etc.
>   >    * Aggregating flow records generated by one or more metering
>   >      processes."
>   > The above 2 are also IPFIX protocol function. So also is flow
>   > expiration. "
> But then we have to mention that IPFIX protocol happens in the context of
> export process.
>
>>
>>
>> For all these reasons I suggest to use the following maping of functions
>> to metering process and exporting process.
>>
>> The metering process is a process triggered by packet arrival. It performs
>> time critical wire-speed processing of packets and creating/updating of
>> the corresponding flow records.
>>
>> Compared to the metering process, the export process is a rather low speed
>> process triggered by flow timeouts and reporting interval timers (and
>> probably by some other events). It encodes flow records into packets,
>> generates templates and sends everything securely to the collecting process.
>>
>
> Agreed.
>
>>
>> Now, coming back to the point that probably was a motivation for inventing
>> the observation domain:  If you want to split the exporting process in
>> a portion that acts on router line cards (generating templetes and reports
>> and a part acting on the central processor board of a router (securely
>> exporting all reports generated on different line cards), then I suggest
>> to split the exporting process (as defined in the requirements) into a
>> reporting process and sending process (you might like to change terms).
>
> If observation domain were to do functions in 4.3 yes then there a good reason.
> But IMO is not the function of observation domain. As far the export process
> is concerened (which includes the IPFIX protocol), itcan be split the way
> you mentioned above. But not mentioning this explicitly does not limit the
> implementation on distributed systems (or does it?) .
>
> Thanks
> Ganesh
>
>>
>
>>
>>
>> Then the chain of observation point, metering process and exporting
>> process
>>
>>   O - M - E
>>
>> would then be replaced by the chain observation point, metering process,
>> reporitng process and sending process
>>
>>   O - M - R - S
>>
>> Advantages:
>>   - it is in line with the requirements
>>   - it is consistent (not merging functions and locations)
>>   - it still allows you to place parts of the exporting
>>     process on line cards, while the rest remains central
>>   - it is much in line with the PSAMP architecture
>>     increasing synergy between both WGs
>>
>> Cheers,
>>
>>     Juergen
>> --
>> Juergen Quittek        [email-masked]        Tel: +49 6221 90511-15
>> NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 90511-55
>> Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.ccrle.nec.de
>>
>> --
>> Help        mailto:[email-masked] and say "help" in message body
>> Unsubscribe mailto:[email-masked] and say
>> "unsubscribe ipfix" in message body
>> Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Mon, 02 Jun 2003 07:35:42 -0700

Juergen,

  I am fine with the name "flow recording process" for the functional
  block.

  Probably we should show 2 different figures in the arch. spec.
  1. A functional block model (as drawn below in my previous e-mail)
  2. Modify the figure which is in the arch spec to show the association
     between observation point, observation domain & metering.

 Thanks
 Ganesh

---

From: Juergen Quittek <[email-masked]>
Date: Mon, 02 Jun 2003 18:15:40 +0200

Ganesh,

>      between observation point, observation domain & metering.

I support your suggestion.

Cheers,

    Juergen

>> >    |             |                      | <--------|Maintain aggregate|    |
>> >    |             |                      |         | statistics       |    |