Subject: [ipfix] Changes to architecture doc.

From: Ganesh Sadasivan <[email-masked]>
Date: Fri, 16 May 2003 14:35:19 -0700

Hi,

In going through the architecture spec. I find  that the following
topics
needs to be modified. I'll list all of them in one e-mail and then on a
topic-by-topic we can have separate e-mail threads.

BTW I could not get a copy of the arch. spec from the link
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-architecture-02.txt

This has been deleted from the internet drafts database because it was
not updated for more than 6 months. I guess the same is true with data
model
document.

The alternate link that I got it from is :
http://www.ietf.org/proceedings/02nov/I-D/draft-ietf-ipfix-architecture-02.txt

* Now that we have chosen the IPFIX protocol, is a separate document
needed for architecture and protocols?

IMHO yes. The architecture discusses the basic terminology (like flow
definition) components of IPFIX framework, different models and
association between these components which are fairly static in nature.
More over the chosen protocol may not fit into all the models described
in the arch. spec. The protocol on the other hand discusses message
etc which could keep evolving to newer versions.

* Section 2 : Scope
There are a few points mentioned here :

" * Define the criteria to select the IPFIX Protocol."
We have already selected the protocol. Do we still need
to keep this?

" * Specify the control/data message formats and handshaking details
  to pass the IP flow information. "
This is really protocol specific and should not be discussed in this
document.

* Section 4: IPFIX Device figure
A device could have multiple exporters and each could have 1:n
mapping with observation domain. This is not well captured in the
figure.

* Section 4.3 Observation Domain
Functions of Observation domain.
" * Encoding the flow records and sending them to the export process.
  * Encoding the control information into templates and sending them to
  the export process."
The above 2 are already covered by IPFIX protocol.

"  * Deciding which flow records/control information to export using
     rules based on time, thresholds, configuration events etc.
   * Aggregating flow records generated by one or more metering
     processes."
The above 2 are also IPFIX protocol function. So also is flow
expiration.

Observation point is a logical block which maintains flows for a
a set of {observation point, metering process} (in a database form)
and maintains aggregate statistics. All the other flow related
functions should be handled by IPFIX protocol and metering process.

* Section 4.5 :
"  There MAY be additional rules defined within the observation domain
   so that only certain flows records are picked up for export. "
Though rules are defined w.r.t observation domain,
the entity that applies these rules is the IPFIX protocol.

* Section 5.1 Selection Criteria for IPFIX Protocol
The first paragraph of this section is not required now. But the
properties (5.1.1 and 5.1.2) still hold good.

* Section 5.1.3 IMO we can remove this.

* Section 5.1.4 Remove the line
"Once the selection is made from the set of candidate protocols, this
 section would be replaced by the chosen protocol."

* We should add a some description on netflow v9 in section 5 not
getting into too much details.

* Section 6.2 IPFIX end point authentication
Do we really need this section. Doesn't section 6.1.x take care of
what is mentioned in 6.2? Is there any plan now or in the future to
achieve this through netflow v9?

* Section 6.3 Denial of service (DoS) attack prevention
This section is quite superficial and I'd suggest we either write
one or more concrete ways to achieve this or remove this section.

* Section 7 Flow Expiration
This should appear as a subsection in IPFIX protocol.

Thanks
Ganesh

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Maurizio Molina <[email-masked]>
Date: Wed, 21 May 2003 12:01:00 +0200

> http://www.ietf.org/proceedings/02nov/I-D/draft-ietf-ipfix-architecture-02.txt

...........

> the entity that applies these rules is the IPFIX protocol.

Hi,
I'd like to discuss the type of packet sampling/flow selection that seems to be
considered by IPFIX so far.
Re-reading section 4.5 of the arch. spec., and the previous ones (4.4.2, 4.4.3,
4.4.4), I got the impression that only the following types of sampling/slection at
the packet or flow level are considered:
1) Sampling of packets within a flow (sec. 4.4.4): Packets undergo a filter to see
if they match a flow to be monitored; if they do, they are (possibly) sampled (to
alleviate METER processing load).
2) Selection of flows to export (sec. 4.5): all defined flows are monitored
(possibly packet sampling within flows as defined above is applied), then a
decisions is taken about whether to export the entire pool of monitored flows or
just a subset of it. This, with the goal to alleviate EXPORTER/COLLECTOR load.

Imo, what's not explicitly considered is a third case, where there is a packet
sampling but "oriented" to flow selection, i.e. packets undergo a filter match but
then they're not "blindly" (i.e. n out of N or similar) sampled, but rather
sampled or discarded on the basis of some property of the flow they matched. The
property may be static (e.g. "flows with high monitoring priority") or dynamically
built  (e.g. flows carrying more traffic than others->sometime called "elephant"
flows). This type of sampling can alleviate both METER and EXPORTER/COLLECTOR
processing load.
The interest for this type of "sampling/selection" (actually, it's both packet
sampling and flow selection) is that for some applications it may be preferred to
receive reliable figures of  the volume of a subset of flows rather that less
reliable estimates of the volume of ALL the flows (as sampling of type 1 would
provide). Furthermore, differently from flow selection of type 2), it's not
required to keep a flow record for ALL the flows before deciding whether to export
them or not, as the decision  whether to open a new flow record is taken "on the
fly" when packets arrive. Then it's potentially more scalable.

That said, what are the issues that a standard body should consider? Imo, they are
mainly two:
- the definition of policies for seclecting packets on the basis the property of
flow they match (i.e. high priority monitoring flow definition, or flows carrying
high traffic volumes, or other....). This, without dwelling into the details of
algorithms may implement this "on line" detection of  properties.
- the type of information that has to be exported: e.g. which percentage of
traffic is being discarded? Otherwise a collector would only get the report about
the monitored flows (high priority or high volume, or other...), but it's
different to know that they compose the 99% oe the 10% of the traffic....

In the requirement draft,
http://www.ietf.org/internet-drafts/draft-ietf-ipfix-reqs-09.txt, in sec. 5.2
(sampling) and 5.3 (overload behaviour) there's already some text that partially,
but not completely and not exactly, supports the type of sampling/selection I've
described  above.
I'd be glad to know whether people in the group share this view that this type of
sampling/selection has to be more explicitly considered. The next step would then
be to discuss if/what needs to be standardized to support it.

Regards,
Maurizio

---

From: Ganesh Sadasivan <[email-masked]>
Date: Wed, 21 May 2003 08:51:40 -0700

Hi Maurizio,

    I can understand that such an information is useful for some applications.
    But in IPFIX we are focusing on flow collection and export. Flow discard
    is something out of scope of  IPFIX discussion. That does not preclude
    a vendor from implementing what you mentioned below and use a vendor
    specific template to report this to the collector.

Thanks
Ganesh

---

From: Nevil Brownlee <[email-masked]>
Date: Thu, 29 May 2003 13:43:17 +1200

> etc which could keep evolving to newer versions.

Agreed.  *Architecture* defines terms (including 'IPFIX flow), and explains
how the overall system works.  *Information Model*  explains how IPFIX flow
data is made up, i.e. what data elements may be part of a flow, what they 
mean, how they are represented, and how they are combined into a flow's 
data structure.  *Protocol* specifies how IPFIX flow data and control 
information is carried (via a congestion-aware transport protocol) from 
IPFIX exporter to IPFIX collector. [If you thought IPFIX is only about
how to transport flow data, have another look at the WG charter.]

> We have already selected the protocol. Do we still need

We put that phrase into the Architecture document as a placeholder
until we'd selected a protocol as the starting point for IPFIX.
It's time to take it out.

> This is really protocol specific and should not be discussed in this

Agreed.

> mapping with observation domain. This is not well captured in the

There are two diagrams in the introduction to section 4.  Both seem 
fairly clear, but I think that they both would benefit from a paragraph
or two of explanatory text.  I'll write some, real soon now ..

Also, there's a comment in the text about "interfaces shown as <~~>,"
but I can't see any like that on the diagrams ???

> functions should be handled by IPFIX protocol and metering process.

No.  Section 3 defines 'Observation Point' as "a location in the network
where IP packets can be observed," and that seems fine to me, let's
leave it like that.  As for section 4.3's list of 'typical functions
of an observation domain,' I think it's helpful to keep the list there -
but we should also add a reference to the *Protocol* document.

[Come to think of it, we should probably refer to the *Protocol* and
*Information Model* documents in quite a few other sections through
the *Architecture* document.]

> the entity that applies these rules is the IPFIX protocol.

Surely it's the metering process that applies selection rules?  After
all, if a flow is not going to be exported, there's no point in a meter
wasting resources on it??

The metering process, i.e. deciding how to actually build flows by
executing a sequence of functions (with specified parameters) on 
the observed packets, is surely part of the Information model.  Or is
it?  Maybe it fits better here in the *Architecture* document.
It certainly needs to be defined somewhere, it's not something which
can be left floating as an 'implementation' detail!

> properties (5.1.1 and 5.1.2) still hold good.

Agreed

> * Section 5.1.3 IMO we can remove this.

Agreed

> getting into too much details.

Yes indeed.  Let's ask Benoit to provide some text for that (remembering
that we only need an outline, and a reference to the *Protocol* document).

> achieve this through netflow v9?

Hmm.  6.2 does seem to be covered by 6.1.2, maybe we should move its
text to there.  6.1.2 says that authentication could be achieved by
transport-layer means such as IP Authentication Header, so there's
not really any point in IPFIX doing this for itself!

> one or more concrete ways to achieve this or remove this section.

Yes, it is superficial.  But it's better than nothing.  It ends with
the remark "to be discussed on the general (i.e. IPFIX) list," so
does anyone have any suitable text we could re-use for this?

> This should appear as a subsection in IPFIX protocol.

This seems very similar to section 4.5 (see above) - it's part of the
metering process.  How about combining 4.5 and 7 into a new section
called 'The Metering Process;' it would go between the current sections
4 and 5 ????

Cheers, Nevil

-----------------------------------------------------------------------
   Nevil Brownlee                   Director, Technology Development
   Phone: +64 9 373 7599 x88941     ITSS, The University of Auckland
   FAX: +64 9 373 7021      Private Bag 92019, Auckland, New Zealand

-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 May 2003 10:40:32 -0700

Hi Nevil,
   See inline.

> but I can't see any like that on the diagrams ???

Between "Collector [1]" and "[Application1] .. [Application n]".

> but we should also add a reference to the *Protocol* document.

Oops that was a typing mistake . I meant "Observation Domain".

> the *Architecture* document.]

Yes.

> wasting resources on it??

I was refering to the post-flow selection and pre-export phase. This is
optional and may be implemented in some cases only.

> can be left floating as an 'implementation' detail!

The figure after 4.4.4 in the architecture doc provides some details of this.
Do we want to go into more details?

> that we only need an outline, and a reference to the *Protocol* document).

I can write this up.

> called 'The Metering Process;' it would go between the current sections

I am a little confused here.
Here are my thoughts on Metering, Observation Domain & IPFIX protocol
and please correct me if I'm wrong.

   +-----------------------------+----------------------------------------+
   |              packet header capturing                                 |
   |                             |               METERING PROCESS         |
   |                        timestamping                                  |
   |                             |                                        |
   |                             v                                        |
   |                      +----->+                                        |
   |                      |      |                                        |
   |                      |   sampling Si (1:1 in case of no sampling)    |
   |                      |      |                                        |
   |                      | classifying Fi (NULL when No criteria)        |
   |                      |      |                                        |
   |                      +------+                                        |
   |                             |                                        |
   +-----------------------------+----------------------------------------+
                                 |
                                 v
                               Flows
                                 |
                                 |
                                 |
   +-----------------------------+----------------------------------------+
   |                             V    OBSERVATION DOMAIN                  |
   |             +----------------------+         +------------------+    |
   |             | Flow data base       |<--------|Provide non-flow  |    |
   |             | (includes flows      |         | information (Eg. |    |
   |             | from all obs.        |         | router state)    |    |
   |             | points in an obs.    |         +------------------+    |
   |             | domain)              |                                 |
   |             |                      |         +------------------+    |
   |             |                      |<--------|Maintain aggregate|    |
   |             |                      |         | statistics       |    |
   |             +----------------------+         +------------------+    |
   |                                                                      |
   +-----------------------------+----------------------------------------+
                                 |
                            Flow Database
   +-----------------------------+----------------------------------------+
   |                             V    IPFIX PROTOCOL                      |
   |             +----------------------+                                 |
   |             | Rules for 1. sending |                                 |
   |             | templates 2. sending |                                 |
   |             | data records 3. timi-|                                 |
   |             | out flows 4. Encoding|                                 |
   |             | template & data.     |                                 |
   |             |                      |                                 |
   |             |                      |                                 |
   |             |                      |                                 |
   |             +----------------------+                                 |
   |                                                                      |
   +----------------------------------------------------------------------+

Thanks
Ganesh

---

From: Juergen Quittek <[email-masked]>
Date: Fri, 30 May 2003 15:39:43 +0200

Ganesh

> Observation point is a logical block which maintains flows for a

Here you mean observation domain, not observation point.

> functions should be handled by IPFIX protocol and metering process.

I have a problem with the concept of an observation domain.  You say it
is a 'functional block', but in the figure it includes observation points
which are locations in the network and no functional blocks.  So the
concept of the observation domain is not a clean one.  I see an observation
domain as a useful concept for implementation design of IPFIX on a
device with multiple line cards, but not useful for the general IPFIX
architecture.

Also I do not see a reason, why the general architecture should define
an internal function block for "sending them [flow records] from [the
metering process] to the exporting process".  Why sending them, if they
are co-located?  Whether or not you locate metering process and exporting
process on different boards and need to send them over an internal bus
is a detail of an implementation design, but IMHO should not be part
of the general IPFIX architecture. And we are definitely not defining
a netwrok protocol between metering process and exporting process.

The requirements do intentionally not discuss how flow records are
shared between metering process and exporting process, because there
are many different potential hardware and software device architectures
that require different ways of communication between these processes.
I do not think the IPFIX architecture should request that the there
is an instance sending flow records from one process to the other,
because there are several alternatives which would become excluded
without need, for example shared memory (including remote memory access
from the exporter to a metering card).

In the requirements document we mapped requirements concerning the
reports to the collecting process (for example the reporting period)
to the section of requirements for the exporting process. Consequently,
I would see several of the functions you listed for the observation
records as part of the exporting process (for example generating templates).

For all these reasons I suggest to use the following maping of functions
to metering process and exporting process.

The metering process is a process triggered by packet arrival. It performs
time critical wire-speed processing of packets and creating/updating of
the corresponding flow records.

Compared to the metering process, the export process is a rather low speed
process triggered by flow timeouts and reporting interval timers (and
probably by some other events). It encodes flow records into packets,
generates templates and sends everything securely to the collecting process.

Now, coming back to the point that probably was a motivation for inventing
the observation domain:  If you want to split the exporting process in
a portion that acts on router line cards (generating templetes and reports
and a part acting on the central processor board of a router (securely
exporting all reports generated on different line cards), then I suggest
to split the exporting process (as defined in the requirements) into a
reporting process and sending process (you might like to change terms).

Then the chain of observation point, metering process and exporting
process

  O - M - E

would then be replaced by the chain observation point, metering process,
reporitng process and sending process

  O - M - R - S

Advantages:
  - it is in line with the requirements
  - it is consistent (not merging functions and locations)
  - it still allows you to place parts of the exporting
    process on line cards, while the rest remains central
  - it is much in line with the PSAMP architecture
    increasing synergy between both WGs

Cheers,

    Juergen
-- 
Juergen Quittek        [email-masked]        Tel: +49 6221 90511-15
NEC Europe Ltd.,       Network Laboratories        Fax: +49 6221 90511-55
Kurfuersten-Anlage 36, 69115 Heidelberg, Germany   http://www.ccrle.nec.de

---

From: Ganesh Sadasivan <[email-masked]>
Date: Fri, 30 May 2003 08:31:49 -0700

Juergen,

> Here you mean observation domain, not observation point.

Yes. sorry for the confusion.

> device with multiple line cards, but not useful for the general IPFIX

The idea is to present a single identity for a group of observation points
to the exporter. That is why it is mentioned as a logical block which
encompasses a set of observation points. It does not matter whether the
observation points are in a single board or different board or even different
nodes. It is a question as to how the collector identifies a group of observation

points.
Ofcourse a degenrate case is a single observation point in an observation
domain.

> a netwrok protocol between metering process and exporting process.

I am copying the figure down once again with slight modifications for clarity
and please explain how this restricts to a class of implementtations.

   +-----------------------------+-------------------------------------+
   |              packet header capturing                              |
   |                             |               METERING PROCESS      |
   |                        timestamping         on an                 |
   |                             |               observation point     |
   |                             v                                     |
   |                      +----->+                                     |
   |                      |      |                                     |
   |                      |   sampling Si (1:1 in case of no sampling) |
   |                      |      |                                     |
   |                      | classifying Fi (NULL when No criteria)     |
   |                      |      |                                     |
   |                      +------+                                     |
   |                             |                                     |
   +-----------------------------+-------------------------------------+
                                 |
                                 v                                      .
                               Flows                                    .
                                 +--------------------------------------+...
                                 |
                                 |
   +-----------------------------+----------------------------------------+
   |                             V    OBSERVATION DOMAIN                  |
   |             +----------------------+         +------------------+    |
   |             | Flow data base       |<--------|Provide non-flow  |    |
   |             | (includes flows      |         | information (Eg. |    |
   |             | from all obs.        |         | router state)    |    |
   |             | points in an obs.    |         +------------------+    |
   |             | domain)              |                                 |
   |             |                      |         +------------------+    |
   |             |                      |<--------|Maintain aggregate|    |
   |             |                      |         | statistics       |    |
   |             +----------------------+         +------------------+    |
   |                                                                      |
   +-----------------------------+----------------------------------------+
                                 |
                            Flow Database (identified by observation domain)
   +-----------------------------+----------------------------------------+
   |                             V    IPFIX PROTOCOL                      |
   |             +----------------------+                                 |
   |             | Rules for 1. picking&|                                 |
   |             | sending templates    |                                 |
   |             | 2.picking& sending   |                                 |
   |             | data records 3. timi-|                                 |
   |             | out flows 4. Encoding|                                 |
   |             | template & data.     |                                 |
   |             |                      |                                 |
   |             |                      |                                 |
   |             |                      |                                 |
   |             +----------------------+                                 |
   |                                                                      |
   +----------------------------------------------------------------------+

> from the exporter to a metering card).

 The idea is that any  function mentioned in the figure above
as part of an observation domain is optional, but if such a functionality exists,

it exists in the context of an observation domain. It is like pieces in a
metering
process. Not every one would want to do sampling and classifying. But they
are still listed as part of metering.

> records as part of the exporting process (for example generating templates).

If you are refering to section 4.3 of the arch. spec, yes they specify functions
which
are not done by the observation domain. I had mentioned in my first e-mail
on this topic
"> * Section 4.3 Observation Domain
  > expiration. "
But then we have to mention that IPFIX protocol happens in the context of
export process.

Agreed.

> reporting process and sending process (you might like to change terms).

If observation domain were to do functions in 4.3 yes then there a good reason.
But IMO is not the function of observation domain. As far the export process
is concerened (which includes the IPFIX protocol), itcan be split the way
you mentioned above. But not mentioning this explicitly does not limit the
implementation on distributed systems (or does it?) .

Thanks
Ganesh