Subject: [ipfix-req] IPFIX Requirements Questions

From: [email-masked]
Date: Tue, 27 Aug 2002 14:49:17 -0400

As I work on the advocates document some questions
have come to mind on the requirements document.

4.3 Distinguishing flows using Transport Header Fields

	In the case of fragmented packets there are
	no port numbers. Are we saying flow state information
	MUST be maintained? In general, it is not clear from 
	the document what the requirements are for fragmented 
	packets.

4.4 MPLS Label

	In the case of dynamic LSP's the label is not useful.
	Why require distinguishing flows based on label.
	What can be done with that information?

5.3 Overload Behavior

	This section needs to be re-worked a little. Some
	of the MUST's impose undue burden on the device.

	1. Must distinguish flow records before and after the
	   behavior change. 

	   	In the case where the behavior is to drop
		overflow flows, why do we need to distinguish
		flows. Perhaps I misunderstood something.

	2. All flows from previous behavior MUST be terminated.

		Again, if I simply dropped excess flows why terminate
		all existing ones. This will likely cause more overflow
		as the get reestablished. This also seems to go to
		far towards implementation.

	3. Meeting process MUST NOT merge previous records.

        I think what we are trying to say is flows with a different
	definition MUST be distinguishable. How that is done is not
	part of the requirements doc.

5.4 Timestamps

	Time stamps to not always mapped to the first packet and last
	packet. In many cases, the end timestamp is merely when
	the flow timed out and has nothing to do with when the
	last packet was seen. Allowing both semantics may be useful.
	A re-wording like this...

   		The metering process MUST be able to generate timestamps for the
		start and end of a flow. The metering process MAY also provide 
		timestamps for the first and the last observed packet 
		of a flow. The timestamp resolution MUST be at least the one of 
		the sysUpTime [RFC1213], which is one centisecond.

	Note - this does not mandate 2 timestamp fields for a flow. You
	could have one element for each meaning (e.g. Flow-start-time, 
	first-packet-time, flow-start-and-first-packet-time) and use
	the one with the desired meaning. 

6.1 information Model

	9  Packet Counter
	10 Byte counter

		As mentioned earlier, for fragments this
		requires state information? If there is no
		state info, then what flow are the counted
		towards?

	13 BGP AS number

		Is this the AS number for the observation point, the
		source/destination address in the packet or some
		combination?

	14. MPLS Label

		As stated earlier, in the case of dynamic LSP's the
		label has no meaning.

	21 multicast replication number

		The document stated this can change over the life
		of the flow. But no mention of what happens when
		it changes. Is that a new flow? If not, what meaning
		does it have when reported? What can I do with it?

6.2 Data Model

	What about allowing Vendors to add information independently?
	Some data to be transported may be vendor specific and never
	make it into the IPFIX spec. 

8.3 Several Collecting Processes

	No mention is made when reporting to several devices
	of ensuring the duplicate data does not cause a double
	counting problem.

9. Special Device Considerations

	I'm not sure of the meaning for the following...

	...

   Please note that here, the observation
   point of a single flow cannot exceed the set of most fine-granular
   observation points linked to a single metering process, because only
   the metering process can merge packets observed at different fine-
   granular observation points to a joint flow.

	...

   Also the
   locations of metering processes are not of any relevance for this
   document (in contrast to the locations of observation points and the
   exporting processes).

Paul

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Juergen Quittek <[email-masked]>
Date: Tue, 27 Aug 2002 21:29:06 +0200

> 	MUST be maintained? In general, it is not clear from

No, the requirements document does not say so.

> 	the document what the requirements are for fragmented

I think the requirements should not include a requirement for
keeping state of fragmented packets. Would you like to see
an explicit statement on that? This would be a NO NEED FOR
statement :-)

> 	What can be done with that information?

Well, for each attribute there are situations where it does not make
sense to use it for distinguishing flows. But being able to distinguish
flows by label seems to be useful to me in general.

> 		far towards implementation.

I agree.

> 	part of the requirements doc.

Also agreed. However, it is not just different definition, but also
different measurement technique, different sampling rate, etc.

We need to express this intended requirement it in a better way.

> 	the one with the desired meaning.

You just added a MAY requirement for the timeout timestamp. We can do this,
but everything you mention was already possible without this additional
requirement. It was just required that the metering process can provide
these timestamps (if configured so). Whether or not they are exported
and whether or not other timestamps are exported was already left open
by the requirements.

> 		state info, then what flow are the counted

Do you suggest to include a requirement for keeping fragment state info?

> 		source/destination address in the packet or some

This is already revised in the upcoming version. This MUST requirement
is removed and replaced by a MAY requirement on source address AS#,
destination address AS#, and next hop AS#.

> 		As stated earlier, in the case of dynamic LSP's the

Yes, but here the general ability of reporting it is required.

> 		does it have when reported? What can I do with it?

Whether or not it is a new flow when the factor changes depends on the
used flow definition. In the revision we added a phrase that computation
of the factor must be clearly define (factor of last observed packet,
mean value, ...). However, from the applications we could not derive a
clear preference for one or the other definition. Therefore, the precise
definiton is left to the concrete protocol.

> 	make it into the IPFIX spec.

I don't see how the requirements forbid this.

> 	of ensuring the duplicate data does not cause a double

We discussed this issue, but did not find a clear requirement for
avoiding this or for providing means for avoiding this. One problem
is that the collecting process is completely out of scope.
Do you have an idea?

>    granular observation points to a joint flow.

When merging flow records already exported from the first level metering
systems, you might create new aggregated flows for which also the "observation
point" gets aggregated. The resulting one might include all first level
observation points.

>    document (in contrast to the locations of observation points and the

Well what's the question here. We care about where packets are observed,
not about where they are processed and converted into fow records.

    Juergen

---

From: [email-masked]
Date: Tue, 27 Aug 2002 20:26:11 -0400

> an explicit statement on that? This would be a NO NEED FOR

	I think we need to be explicit. Most people will think
	of fragements as being counted in the same flow as the
	first packet not as 2 differernt flows.

> flows by label seems to be useful to me in general.

	Then why not include things like VPI/VCI for ATM
	or other L2.5 tunnels? Why does MPLS get special
	consideration?

	Also, I may be wrong here but I beleive the primary
	use of MPLS will be with dynamic LSP's so the label
	has little meaning the majority of time. At best
	it would be a MAY requirement, not a MUST.

	If we are going to go this way, which I am
	against, then we should also include FEC. At least
	that would be more meaningful information in the
	case of dynamic LSPs.

> We need to express this intended requirement it in a better way.

	Agreed.

	What this points out to me is that we do not have a term
	that refers to the set of information that is the flow
	definition. We touch on it in section 2.1 when discussing
	the flow definition itself, in section 2.3 when discussing 
	classifying and then again in section 4 when discussing 
	distinguishing flows.

	What about a new term "Distinguishing Properties" or
	"Flow Properties"? Its definition would be

		The set of fields, functions and parameters (e.g. sampling
		rate) used to map packets to flows.

	Then we can say each flow MUST be able to be mapped to its 
	"Distinguishing properties".

	This would cover changing behavior due to overload, configuration
	changes, etc...

	Thoughts?

> and whether or not other timestamps are exported was already left open

	I disagree. I'm not talking about what is exported, I talking
	about semantics. The req doc currently ties the timestamp event
	to packet observation. All metering processes will know
	when they created and deleted a flow. They may or may not
	know when the first or last packet was observed.

	The typical example of this is last packet. In many platforms
	the first packet is processed by the CPU and creates a hardware entry.
	The rest of the packets are processed and counted in hardware
	and thus there is no timestamp available, just a count. All
	that is known is when the flow was artifically timed out.

	A weaker argument can be made for first packet observed. There
	may be a known significant delay from packet observation to timestamp.
	In this case the metering process can report the time at which it
	created the flow but not first packet time. Weak I know. 

> Do you suggest to include a requirement for keeping fragment state info?

	A MAY is as far as I would go. There may certainly be probes
	that have this capability. We should have a requirement that
	says fragment counting MUST be clearly defined.

> destination address AS#, and next hop AS#.

	Great.

> definiton is left to the concrete protocol.

	Agreed.

> I don't see how the requirements forbid this.

	The only reason I'm calling this out is "extensible protocol"
	usually means future changes to the protocol can be added
	without breaking existing applications. The vendor 
	specific extensions allow extenstions without modifying the
	protocol, without breaking existing server and without stepping 
	on other verndor's extensions.

> avoiding this or for providing means for avoiding this.

	I don't understand. Double counting would surely cause
	problems for many if not all of the applications. 

> One problem

	We are opening the door by allowing data to be sent to
	multiple Collectors. I think it would be difficult and 
	perhaps even impossible, without protocol support. But 
	I'm open to ideas on how it could be accomplished without 
	protocol support. 

> Do you have an idea?

	For example, in LFAP a Flow ID, timestamp and message ID
	uniquely identifies flow information. So the Collector could
	use that to weed out duplicate data. How and when would be
	out of scope. But the means is defined by the protocol.

> point" gets aggregated. The resulting one might include all first level

	I still don't get it. Let me think about it.

> not about where they are processed and converted into fow records.

	Ahh. Now I get it.

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 09:45:14 +0200

Hi Paul,

> 	first packet not as 2 differernt flows.

I see. Waht about adding a new subsection to Section "5. Metering
Process":

5.8.  Packet Fragmentation

   In case of IP packet fragmentation, only one fragment of a single
   packet might contain sufficient information for classifying the
   packet correctly. The metering process MAY keep state of
   IP packet fragmentation in order to map fragments that do not
   contain sufficient header information correctly to flows.

 [...]

> 	says fragment counting MUST be clearly defined.

OK. Currently, we have

      9. packet counter
         If a packet is fragmented, each fragment is counted as an
         individual packet.
     10. byte counter
         Which bytes of a packet are counted MUST be defined exactly.

What about appending

         The behavior of the byte counter in case of IP packet
         fragmentation MUST be clearly defined.
?

Please note that we also have in the MAY attributes section

     26. fragmented packet counter
         counter of all packets for which the fragmented bit is set in
         the IP header

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 10:27:52 +0200

Hi Paul,

[...]

> 	or other L2.5 tunnels? Why does MPLS get special

In general I see your point, although MPLS appears to be closer
related to IP than plain ATM.

> 	it would be a MAY requirement, not a MUST.

For me, metering on a per-LSP basis is highly benefitial for the
operation of MPLS networks, even in the case of dynamic label
assignment. The LSR MIB already offers this kind of performance
information, but I think it also fits well into IPFIX.

> 	that would be more meaningful information in the

This could be useful, but the FEC is a higher level information
that is not available in many cases. Also we would need a good model
for FEC information.

    Juergen

---

From: Sebastian Zander <[email-masked]>
Date: Wed, 28 Aug 2002 12:21:23 +0200

Hi Paul,

sorry for dropping in but I have some comments below.

> 	first packet not as 2 differernt flows.

Good point! According to the current draft the fragments after the
first would not be counted at all (in case you match on >L3) unless
there would be a more general rule matching the fragments and then
they appear in a different flow. The current draft requires a packet
counter which counts all fragments per flow so it seems to require
keeping flow state which is not so easy I guess. In case there is
no state the notion of the packet counter is that if we filter on
L3 we can count L3 packets (including fragments) and if we filter on
above that we can only count L4 packets (first fragments). That also
impacts the byte count because if we filter on above L4 we won't get
the correct byte count for the flow.

> 	created the flow but not first packet time. Weak I know. 

I agree that we have to look at current practice but I have the feeling that
we get more and more vague just to support any existing thing. The requirements
are driven by applications. Lets look at accounting. I am wondering how IPFIX
information can be used for accounting when

- there are no correct stop timestamps (there are no requirements on

  the flow timeout)

- there are no requirements on the report time (when are reports send)
   (e.g. it is not required to export information at/after flow end)
- there is no timestamp required in interim flow records
- byte counters may not be correct (in case of port filtering and fragmentation)

Well forget about the last one but with the first three we basically do not
comply to the req of the AAA WG for accounting.

If the MUST requirement of the current draft is a problem for a lot of current
hardware then at least the requirement should be reduced to a SHOULD and not
a MAY.

> 	on other verndor's extensions.

I don't think this belongs to the req draft. But certainly in the
spec.

> 	out of scope. But the means is defined by the protocol.
  
I support to include duplicate detection as a requirement in the draft 

but without suggestion a specific solution. That should be left for

the spec.

Cheers,

Sebastian

-- 
Sebastian Zander                         E-mail: [email-masked]
FhI FOKUS / Global Networking (GloNe)    Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany                  www.fokus.fhg.de/usr/sebastian.zander

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 13:52:36 +0200

Hi Paul,

 [...]

> 	This would cover changing behavior due to overload, configuration

I think the mistake in the current version of the text is that it
asks on separating flows before and after the metering process was
modified. It is indepenedent of whether the individual flows are
affected or not. I think it is sufficient to fix this.

The current text is

   Overload behavior is not restricted to the four options listed above.
   But in case the overload behavior has an impact on the metering
   process or the exporting process, the overload behavior MUST be
   clearly defined and the collecting process MUST be able to
   distinguish the flow records exported before and after the metering
   process behavior change: in case of any change of the meter's
   behavior, all flow records metered by the previous behavior MUST be
   terminated and exported according to the configuration of the
   exporting process. The metering process MUST not merge the flow
   records generated with the new behavior with the flow records
   generated with the previous behavior.

I suggest to replace it by

   Overload behavior is not restricted to the four options listed above.
   But in case the overload behavior induces a change of the behavior of
   metering process and/or the exporting process, the following requirements
   must be met for all flows affected by the change of behavior:
     - The overload behavior MUST be clearly defined.
     - For each flows affected by the change, its record MUST be terminated
       at the time of change.
     - For each flow affected by the change, the collecting process MUST be
       able to determine whether the corresponding exported flow record was
       created before or after the change of behavior.

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 14:10:36 +0200

Sebastian,

> but without suggestion a specific solution. That should be left for

Could you (or anyone else) suggest where to extend the requirements
document and what text to insert?

It looks like this would be a requirement for the exporting process.

    Juergen

---

From: [email-masked]
Date: Wed, 28 Aug 2002 09:06:45 -0400

>    contain sufficient header information correctly to flows.

  How about a slight re-wording...

    In case of IP datagram fragmentation, only the first fragment might
    contain sufficient information for classifying the packet correctly. 
    The metering process MAY keep state of IP fragmentation in order to 
    map fragments without sufficient header information to the same
    flow as the first fragmented packet.
	
>          fragmentation MUST be clearly defined.

	With the new section on fragmentation I think we can 
	eliminate any mention of it on the counters.

>          counter of all packets for which the fragmented bit is set in

	I'm not sure why this was added. If you want that info,
	make the IP fragment bit part of the flow definition.

---

From: [email-masked]
Date: Wed, 28 Aug 2002 09:18:22 -0400

> related to IP than plain ATM.

	Why? I can take and ATM PVC and use it as a tunnel for
	IP traffic just like MPLS.

> information, but I think it also fits well into IPFIX.

	Are you talking about metering LSP's themselves or
	metering IP flows traveling through an LSP?

	Assuming you are talking about the latter, what can be done 
	with the label information? If the flow F1 had label 57
	and F2 had label 57 you don't even know if it is the
	same LSP.

> that is not available in many cases. Also we would need a good model

	If the device knows the label it knows the FEC. There are
	some concrete FEC definitions now and we can always add more
	as they materialize.

---

From: [email-masked]
Date: Wed, 28 Aug 2002 09:37:38 -0400

> - there are no correct stop timestamps (there are no requirements on

	I don't think that is the case. If you report a flow end
	and the flow timeout period is 10 seconds, that will be
	close enough for most applications, even accounting.

>    (e.g. it is not required to export information at/after flow end)

	Good point. Maybe we need some text on report time. 

		The exporting process MUST be able to report on flow
		start and end within a configurable time limit. 

> - there is no timestamp required in interim flow records

	I'm not sure I follow here. There is a SHOULD requirement
	for regular reporting. 

> hardware then at least the requirement should be reduced to a SHOULD and not

	In this particular case I believe the definition is the
	problem, not the current hardware. I can't imagine any 
	hardware ever timestamping every packet.

	Agreed.

---

From: [email-masked]
Date: Wed, 28 Aug 2002 09:55:38 -0400

> affected or not. I think it is sufficient to fix this.

	You side stepped my main 2 issues here. 

		1. We need a definition for the set of functions,
		   fields and parameters use to map packets to flows.
		   If it is there and I missed it, let me know.

		2. We need a requirement that allows the Collecting
		   device to map flows to the set of functions, 
		   fields and parameters used to create the flow.	

	Agreed.

>      - For each flows affected by the change, its record MUST be terminated

	This seems like implementation detail. Maybe I want to leave
	them in tact and when the overload situation is resolved I can
	start adding to the counter again.

>        created before or after the change of behavior.

	I believe that if we address the 2 issues discussed earlier
	these would no longer be needed.

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 16:20:43 +0200

Hi Paul,

>     flow as the first fragmented packet.

Initially I also wanted to phrase like this. But it's IP. And fragments
might get re-ordered. The first fragment of a packet that is observed,
is not necessarily the fragment containing the TCP/UDP header.

Now, if you say "first", it is not clear whether you mean the fragment
with the first bytes of the IP payload or the first fragment observed
at the observation point. Your last sentence seems to assume that the
first observed packet is the one with the first bytes of the payload.

> 	eliminate any mention of it on the counters.

The new section of the fragmentation is a MAY requirement. But for
understamding the behavior of the byte counter you MUST know whether
or not the option is implemented.

> 	make the IP fragment bit part of the flow definition.

Do you suggest to remove it? Anyone else?

     Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 16:30:12 +0200

Hi Paul,

> 	IP traffic just like MPLS.

Yes, but MPLS integrates with IP forwarcing when using CR-LDP
or RSVP-TE. Also you can use MPLS as "glue" between IP and ATM.

> 	metering IP flows traveling through an LSP?

I'm talking about metering LSPs and about observing which flow maps
into which LSP.

> 	and F2 had label 57 you don't even know if it is the

You are right. I would also need interface information.

> 	some concrete FEC definitions now and we can always add more

All you need for an LSP are entries in insegment table, outsegment table
and switching table. How does the device know anything about the FEC if the
LSP was explicitly setup by a management system that added entries to these
tables.

    Juergen

---

From: Juergen Quittek <[email-masked]>
Date: Wed, 28 Aug 2002 16:39:49 +0200

Hi Paul,

> 		   fields and parameters used to create the flow.	

My intention was not to side step them, but to answer on the portion
of your comment on which I already made up my mind already. On this issue
I still have to think some more.

> 	start adding to the counter again.

I don't think this would be a good idea, because it might not be clear
to the collector that you ignored all packets for some time. This seems
to be a very unreliable behavior.

However, you still might be right. Very often it is hard to distinguish
between the real requirement and a particular implementation meeting
that requirement. In this case I am not sure.

> 	these would no longer be needed.

Would be fine.

    Juergen

---

From: [email-masked]
Date: Wed, 28 Aug 2002 10:52:10 -0400

> or RSVP-TE. Also you can use MPLS as "glue" between IP and ATM.

	I don't understand your argument here. Both technologies
	are viable solutions and exist in the market today.
	If one is included, so should the other. 

> I'm talking about metering LSPs and about observing which flow maps

	I'm a little confused. I thought we were only doing IP 
	flows. Are we now including MPLS flows (LSPs)?

> You are right. I would also need interface information.

	OK. So now I have F1, I1 and L57 and then F2, I1 and L57.
	I still don't know if it is the same LSP.

> LSP was explicitly setup by a management system that added entries to these

	Let me make sure I understand. Some external management 
	station received the FEC and programmed the device? If 
	that is the case then you are correct. However, wont many 
	deployments have the 2 functions on one device? If so, 
	then reporting FEC is at least a MAY requirement.

---

From: David Moore <[email-masked]>
Date: Wed, 28 Aug 2002 09:20:46 -0700

> 	that is known is when the flow was artifically timed out.

Is this because there is no clock available to the hardware?

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 11:54:13 +0200

Paul,

[let me take the latest email on this thread]

>>> >> >               flows. Perhaps I misunderstood something. 
An example (somehow already discussed below) is:
1. you do sampling 1/100.
2. for whatever reason, you drop overflow flows (or you decide to change 
the sampling rate)
3. the new sampling rate is now 1/200
4. the collector must know that.

>>> >> >       part of the requirements doc. 
Agreed. It was actually the goal of the sentence "the overload behavior 
MUST be
   clearly defined and the collecting process MUST be able to
   distinguish the flow records exported before and after the metering
   process behavior change"
Because, in my view, dropping the overflows flows is changing the 
"definition" of the flow, as you wrote Paul wrote it in "flows with a 
different  definition MUST be distinguishable"

> to be a very unreliable behavior. 

Agreed with Juergen on that one.
In my example:
1. you do sampling 1/100.
2. for whatever reason, you drop overflow flows (or you decide to change 
the sampling rate)
3. the new sampling rate is now 1/200
4. the collector must know that.

How would the collector be able to distinguish the 2 series of flow 
records. If for example you want to multiply the first serie usage by 
100 and the second serie by 200?

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 12:15:13 +0200

Paul and Juergen,

> first observed packet is the one with the first bytes of the payload. 

Why not merge your 2 texts:

   In case of IP packet fragmentation, only one fragment of the initial
   packet  might contain sufficient information for classifying the
   packet correctly. Note that this fragment is the first one generated
   by the router imposing the fragmentation, but might not be the first
   one observed by the IPFIX device, due reordering reasons.
   The metering process MAY keep state of  IP packet fragmentation
   in order to map fragments that do not contain sufficient header
   information correctly to flows.

> or not the option is implemented. 

So, haven't we enough with "The behavior of the byte and packet counters 
in case of IP packet
 fragmentation MUST be clearly defined."?
 We need the packet counter in the sentence above: to know if the flow 
state information is maintained for fragemented packets

>>     make the IP fragment bit part of the flow definition. 
It makes sense.

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 13:10:34 +0200

Hi,

>	then reporting FEC is at least a MAY requirement.
I have to admit that I haven't been thinking too much in the draft about 
MPLS.
I agree now that  it makes more sense to report the FEC than the MPLS 
label, since the label has got a local significance

But this is not in contradiction with what is in the draft!

4.4.  MPLS Label

   If the observation point is located at a device supporting
   Multiprotocol Label Switching (MPLS, see [RFC3031]) then the metering
   process MUST be able to separate flows by the MPLS label.

And SHOU:LD report the FEC

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 16:23:32 +0200

All,

[taking the latest email on this thread ... hopefully because there are 
so many emails]

>>>      that is known is when the flow was artifically timed out.
The timestamp of the last packet observed is what we need for accounting 
and not when the flow expires from the cache.
In our implementation, we create the timestamp of the last packet 
observed in the flow. We do this in hardware and obvioulsy in software 
on the smaller platforms.

Regards, Benoit.

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 29 Aug 2002 16:35:35 +0200

Hi Benoit,

-- Benoit Claise wrote on 29 August 2002 13:10 +0200:

> I have to admit that I haven't been thinking too much in the draft about MPLS.
> I agree now that  it makes more sense to report the FEC than the MPLS label, since the label has got a local significance
>

Here in Section 4 we are talking about separating flows.
Do we want to separate flows by the FEC also?
Or are you suggesting to add the FEC to the list of SHOULD
attributes in Section 6.1?

    Juergen

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 17:18:52 +0200

> Do we want to separate flows by the FEC also? 

As there is a matching Label -> FEC, we can still separate by Label

> attributes in Section 6.1? 

And yes: SHOULD report the FEC in Section 6.1

Regards, Benoit.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 15:34:16 -0400

> 4. the collector must know that.

	I agree with your example. However, I have a different example
	where it is not necessary. But, according to the document
	I MUST distinguish the flow records. The requirements go
	to far in mandating behavior for all overload solutions.

	I think Juergen agreed on this particular point. How to 
	rectify it is still unclear. I chose one way and Juergen
	another. We still need further discussion.

> different  definition MUST be distinguishable"

	Dropping is not a change for all flow definitions. In fact 
	sampling is the only definition I can think of which is 
	affected by dropping.

> 100 and the second serie by 200?

	I agree sampling is a special case where the sampled pool is
	a piece of key information and thus can be considered part
	of the definition.

	In the example I gave suppose I switch to a coarse grained
	flow definition but there are still a few thousand of them. Then
	the overload situation is over and I go back to my fine grained
	definition. Overload happens again and now I have to set up
	all those flows again at the worst possible time. 

	Paul

---

From: [email-masked]
Date: Thu, 29 Aug 2002 15:37:29 -0400

>    information correctly to flows.

	Juergen suggested a merged text. Are you suggesting
	an alternative?

---

From: [email-masked]
Date: Thu, 29 Aug 2002 15:43:40 -0400

> But this is not in contradiction with what is in the draft!

	I still don't understand how the label can be used.

	What about ATM-AAL5?

> And SHOU:LD report the FEC

	Did you mean "report" or "separate"? I think for
	it to be useful it must be able to separate flows
	by it.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 15:51:27 -0400

> As there is a matching Label -> FEC, we can still separate by Label

	I thought there could be multiple FEC's per label.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 15:49:26 -0400

> on the smaller platforms.

	The hardware timestamps every packet?

	In any case, I'm not saying it is one or the other.
	I'm saying we need to provide both definitions. Not
	all platforms will timestamp every packet. The timeout
	time is still useful. If the timeout period is say 10 seconds
	you know the last packet time within 10 seconds.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 12:55:03 -0700

> flows by label seems to be useful to me in general.

I agree especially if label to prefix mapping can be provided by some
out-of-scope means -:)

> >               far towards implementation.

It may be a good idea :
* as a basic operation indicate this condition to the collector.
* if there is a per flow/per observation point action taken like
  change of sampling rate, then it make sens to terminate & recreate
  the flows. But I am not comfortable with a MUST here.

> Do you suggest to include a requirement for keeping fragment state info?

I guess this is a a requirement quite difficult to meet ubiquitously.
Thanks
Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 13:03:04 -0700

>          If a packet is fragmented, each fragment is counted as an

But it may be counted into a wrong or a coarser flow. So it depends
on the device capability & traffic pattern.

>          fragmentation MUST be clearly defined.

Had a basic question here. Whe we say byte counter, does it
exclude upto L4 header?
Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 13:19:15 -0700

> related to IP than plain ATM.

I'll have to agree with both of you:) I think there has been more
voices for MPLS compared to ATM though I do not see a reson
for it not to be included.

> information, but I think it also fits well into IPFIX.

Though dynamic in nature, the label meaning can be obtained from
the edge device at different levels. So I think this information
is very useful.
-ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 13:26:04 -0700

> >       the first packet is processed by the CPU and creates a hardware entry.

This may not be the first packet in the flow but it is the first packet seen by
this
observation point.

> >       that is known is when the flow was artifically timed out.

Same holds good here. It may not be last packet in the flow But we are
forced to flush the cache because of being full or whatever. In either case
if we can register the time at which the first & last packet for this packet came
in at this observation point (with tolerances based on implementation) is that
not sufficiently broad? Some of  the h/w implemetation do a periodic cache
flushing. But the time is still closest to the last packet..
-ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 12:45:13 -0700

>         the document what the requirements are for fragmented

I do not see a way to distinguishing flows for fragmented packets
and not all fragments pass through the same router.

>         What can be done with that information?

If we can get a label to prefix mapping (from the control plane)
then it would be be very useful information. So I think this defn.
still holds good.

>         1. Must distinguish flow records before and after the

Probably the intention is that it is one of the flow expiration
reasons. But I am not sure whether it needs to be tied with
a flow is it is the device's behavior. So keeping a MUST here
does not seem to be required.

>                 far towards implementation.

I do not understand why. The reason you stated above is
good enough for this not to be done.

>         part of the requirements doc.

What I understand here is that 3 different flow records be cretaed
* prior to overload
* overload
* post overload
While I can udnderstand this from an export process perspective,
I am not sure how much sense it makes to go per flow on overload.

>         A re-wording like this...

While there may be a minor delay in setting the timestamp after the
last packet is seen, I am not sure by the above statement that
"has nothing to do with when the last packet was seen." Is it not the
case that the flow was alive at this observation point only till it saw the
last packet?

>                 source/destination address in the packet or some

Correct me if I'm wrong - usually a set of prefixes are associated with an AS.
An observation point is too vague a definition to associate to an AS. Say
it is a router LC, there could be multiple ASs within it.

>                 does it have when reported? What can I do with it?

It would be useful if the collector is going to calculate the
replication factor over a period of time.

>         of ensuring the duplicate data does not cause a double

I must have forgotten. Can you elaborate a little more?

Thanks
Ganesh

---

From: [email-masked]
Date: Thu, 29 Aug 2002 16:41:21 -0400

> and not all fragments pass through the same router.

	Right. Keeping state is no guarentee.  

> then it would be be very useful information. So I think this defn.

	Who is "we"? If you mean the Collecting Process, then
	querying the device after the fact has problems since
	LSPs are dynamic.

> good enough for this not to be done.

	I'm not sure I follow. The cost of terminating and
	re-establishing millions of flow is not reason enough?

> last packet is seen, 

	If the hardware stamps all packets, no problem. If the
	software has to do it, how do you know what the last packet
	is?

> I am not sure by the above statement that

	In the case of timeout, it is the time at which the device
	noticed that there were no more packets since the last time
	it checked. It does not know the time of the last packet
	observed.

> it is a router LC, there could be multiple ASs within it.

	But suppose the entire router is in a single AS. Should
	we report it?

	In your example, maybe we report a list. 

> I must have forgotten. Can you elaborate a little more?

	Depends on why you are reporting to 2 devices. If
	it is for completely different applications, no problem.
	But if there is a sharing of data for back up purposes
	or any other reason then ensuring you don't double
	count comes into play.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 16:55:26 -0400

> the edge device at different levels. So I think this information

	Are you saying the Collecting process needs to get additional
	info from the device in order for field reported by IPFIX
	to be useful? Not to mention that since the labels are dynamic in 
	nature, asking about label 57 in 2 consecutive requests may 
	come back with different answers.

	Help me out here, I'm struggling to see the usefulness
	of this piece of data.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 17:02:20 -0400

> This may not be the first packet in the flow but it is the first packet seen by

	Not according to the IPFIX definition of a flow. Flow and
	observation point are tied together. You are using the term
	flow at a higher level.

> flushing. But the time is still closest to the last packet..

	If the timestamping is hardware based I'm OK with
	ignoring minor skews in time due to hw implementation.

	But when it is software that uses a configurable interval
	for inactivity period, I think it is better to have that
	represented in the data.

---

From: Benoit Claise <[email-masked]>
Date: Thu, 29 Aug 2002 23:23:28 +0200

>	to far in mandating behavior for all overload solutions.
Let me take then a different example. Let's consider you use IPFIX for 
billing. So you're not doing sampling.
For whatever reason, you drop the overflow flows. I think that the 
collector MUST know about it.
For example:
 From 8.00 to 10.30 -> correct billing
 From 10.30 to 10.45 -> I lost some flow records -> I must take an 
action (for example a correction on the bill or whatever)
 From 10.45 -> back to normal

>>>>>>>>              Again, if I simply dropped excess flows why
>>>>>>>>                
>>>>>>>>
>>>>>terminate
>>>>>          
>>>>>
>>>>>>>>              all existing ones. This will likely cause more
>>>>>>>>                
>>>>>>>>
>>>>>overflow
>>>>>          
>>>>>
>>>>>>>>        I think what we are trying to say is flows with a
>>>>>>>>                
>>>>>>>>
>>>>>different
>>>>>          
>>>>>
>	Dropping is not a change for all flow definitions. 
>
See my previous remark.

>In fact 
>>>>>>              The set of fields, functions and parameters (e.g.
>>>>>>            
>>>>>>
>>>>>sampling
>>>>>          
>>>>>
>>>>>>      This would cover changing behavior due to overload,
>>>>>>            
>>>>>>
>>>>>configuration
>>>>>          
>>>>>
>>>>>>      changes, etc...
>>>>>>
>>>>>>      Thoughts?
>>>>>>            
>>>>>>
I agree there is a  risk of overload flapping in your specific example.
But anyway you can not do anything with flow records on the collector if 
you don't know the "distinguishing properties" (term that you introduced).
As a consequence, you have to stop/expire/export all flow records with 
the coarse grained flow definition.
Otherwise, you mismatch both flow record types on the collector -> you 
can not use the data

Regards, Benoit.

---

From: [email-masked]
Date: Thu, 29 Aug 2002 17:55:27 -0400

>  From 10.45 -> back to normal

	We must be getting close because we're getting to 
	nitty gritty details :-)

	First, I have no problem reporting the overload situation
	along with any parameters (time, number of packets dropped,
	etc...)

	My issue is with terminating all existing flows.

	Suppose I have room for 100 flows. I'm using SRC/DST IP as
	my "Distinguishing Properties". 100 flows have now
	been established and I'm continuing to increment the counter				as the
packets for those flows arrive. Now at 10:30 101th flow 
	comes in and so I drop it. 

	Why do I need to terminate the 100 existing flows that
	are happily incrementing their counters? The fact that
	I dropped the flow has not altered the "Distinguishing 
	Properties" which are in affect.

> Otherwise, you mismatch both flow record types on the collector -> you

	Yes I can :-) Lets assume under normal conditions a device
	wants to use multiple "distinguishing property" sets. 
	Nothing unusual here. But it does mean there is a need
	to map a flow to a "distinguishing property" set, right?
	
	If we still agree, then my overload flows have one
	set of "distinguishing property" while the other flows
	have a different "distinguishing property" set, no problem.

	Please keep in mind I am not advocating this as an
	overload behavior. But I am against the requirements
	ruling it or other similar ones out.

	Paul

---

From: Benoit Claise <[email-masked]>
Date: Fri, 30 Aug 2002 00:42:16 +0200

Paul,

>	I thought there could be multiple FEC's per label.
I doublechecked with some of our experts.

    label: FEC is a 1:1 relationship
    FEC: label is a 1:n relationship

In conclusion, what we need in the req. draft is:

4.4.  MPLS Label

6.1.  Information Model
   The exporting process SHOULD be able to report the following
   attributes for each metered flow:
   if MPLS is supported at the observation point: the Forwarding Equivalent Class (relative to the top label)

Why a SHOULD and not a MUST?
Because in case of static label whose setup is done by a NMS, we don't need to export the labels.

And if we want to be complete...

6.1.  Information Model
   The exporting process MAY be able to report the following
   attributes for each metered flow:
   if MPLS is supported at the observation point: the underlying labels (second bottom label and below)

Why? the report of the MPLS/VPN label for example

Regards, Benoit.

---

From: Benoit Claise <[email-masked]>
Date: Fri, 30 Aug 2002 01:01:52 +0200

Paul, Juergen and all,

>	Juergen suggested a merged text. Are you suggesting
To summarize:

Juergen proposed:
  In case of IP datagram fragmentation, for many typical
  classification schemes, only one fragment contains
  sufficient information to classify the packet.

I proposed above:

I think both of them express the same idea.
The second one maybe contains some more clarifications.

So I'm not really in favor of one or the other. Choose the one that you want to.

I'm not sure what were your conclusions for fragmentation and section "6.1 Information Model"

1. Have something like (implicitely talking about fragmentation):
      9. packet counter
         Which packets are counted MUST be defined exactly.
     10. byte counter
         Which bytes of a packet are counted MUST be defined exactly.

OR 

2. 
     9. packet counter
         Which packets are counted MUST be defined exactly.
     10. byte counter
         Which bytes of a packet are counted MUST be defined exactly.
And an extra sentence about fragmentation.
"The behavior of the byte and packet counters in case of IP packet fragmentation MUST be clearly defined."

Regards, Benoit.

>>>>>>>Do you suggest to include a requirement for keeping fragment
>>>>>>>              
>>>>>>>
>>>>>state info?
>>>>>          
>>>>>

---

From: Ganesh Sadasivan <[email-masked]>
Date: Thu, 29 Aug 2002 16:52:21 -0700

>         querying the device after the fact has problems since

The association of {incoming label, FEC} is available at the
exporter (referred to as "we") and application that it points to.
I am not sure how dynamic this information is -I guess it depends
on the label type (TE-tunnel, BGP label, IGP label, VPN label etc).
These associations when made available to the collector with
timestamping, it can do the mapping from "label in flow record" ->
FEC/application/etc. Infact there are vendors interested in getting
this kind of information.

>         re-establishing millions of flow is not reason enough?

I am in sync with you.

>         software has to do it, how do you know what the last packet

I'm confused. If there is a mechanism to detect the flow end by
means of packet or otherwise, then it is possible to get
a fine grained granularity. If otherwise, there could be some
periodic mechanisms (operating in msec periodicity)
that poll of counter deltas for each flow & update the timestamp.

>         But suppose the entire router is in a single AS. Should

Why not?

>         or any other reason then ensuring you don't double

Ok, I understand the application. Probably then it is a good idea
to associate the flows with the latest timestamp which is put
on the export packet when it is sent out. Just a thought.
-ganesh

---

From: [email-masked]
Date: Fri, 30 Aug 2002 10:13:30 -0400

>   sufficient information to classify the packet.
					   ^^^^^^
					   fragment

	With the above change I think this is clear and easy to read. 

>    information correctly to flows.

	This one is OK but goes into explanation I'm not sure 
	is necessary if you assume familiarity with IP. 

	Also, there are 3 terms "initial packet", "packet"
	and "fragment". So I get a bit lost.

	But either way will suffice as far as I'm concerned.

>          Which bytes of a packet are counted MUST be defined exactly.

	Which bytes are counted is a legitimate issue. Do we include
	ethernet headers, IP headers, CRC, etc...In other words some
	bytes are never counted under ANY circumstances.

	I don't see similar issues for the packet counter.

	As soon as you start talking about which flow a packet gets
	counted against, it is a "Distinguishing Properties" issue.
	If you talk about what flows get counted it is a 
	configuration issue.

	The question on fragment is are they distinguished by the
	full header information or just a subset.

---

From: [email-masked]
Date: Fri, 30 Aug 2002 10:23:45 -0400

>     FEC: label is a 1:n relationship

	I got a different answer from our guys. They said
	our current implementation is 1:1 for label->FEC but
	specifications indicate 1:n is possible.

	Maybe we can get hold of someone in the MPLS WG to
	help clear this up.

>    process MUST be able to separate flows by the MPLS label.

	My comment is LSP's are too dynamic for labels to
	be useful. 

	Are you saying you believe labels are stable enough to
	be useful?

	Again, I'm no MPLS expert so I'm just posing the 
	questions based on my limited understanding.

> Because in case of static label whose setup is done by a NMS, we don't need to export the labels.

	It is worded like this, it can remain a MUST.

	if dynamic MPLS LSPs are supported at the observation point: 
	the Forwarding Equivalent Class (relative to the top label)

---

From: [email-masked]
Date: Fri, 30 Aug 2002 10:48:57 -0400

> document and what text to insert?

	Since no one else took a stab...

	In draft-ietf-ipfix-reqs-05.txt section 6.3.2.  Reliability

	In the event Flow Records are sent more than once (to the same
	or separate Collectors), the records MUST be uniquely identifiable.

---

From: [email-masked]
Date: Fri, 30 Aug 2002 11:11:34 -0400

> exporter (referred to as "we") and application that it points to.

	Agreed.  if the exporter does the FEC look up at the time it 
	records the flow, that should be good enough.

> on the label type (TE-tunnel, BGP label, IGP label, VPN label etc).

	How dynamic the label is, is probably the critical
	piece of knowledge I'm not totally sure of. Hopefully
	we can get some clarity on this.

> this kind of information.

	Are you  saying that some other process is collecting
	and time stamping label->FEC/application/etc. information?

> >         re-establishing millions of flow is not reason enough?

	Sorry, I miss read your response. I guess I'm just
	used to getting beaten up lately :-)

> that poll of counter deltas for each flow & update the timestamp.

	How do you poll millions of flows in msec?

> >         But suppose the entire router is in a single AS. Should

	Agreed. But that was not clear from the req doc.

> on the export packet when it is sent out. Just a thought.

	Might work depending on the stamp stamp granularity.

	Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Fri, 30 Aug 2002 10:17:17 -0700

>         we can get some clarity on this.

I'll try to poll some experts  in this area and get a better idea.

>         and time stamping label->FEC/application/etc. information?

Right now I'll have to assume the above unless we are exploring means to
describe FEC through the export protocol. The short answer is that label
is an interesting field for collectors and they can use it as an ID to do
whatever post-processing they want to.
Thanks
Ganesh

>         How do you poll millions of flows in msec?

This is an implementation detail. I was suggesting the above as an idea -
that's all. Probably handling millions of flows would have some
kind of h/w assist or is that not the case? Well there may be other
means to implement the same.
But in an extreme case if there are no resources to handle such a
situation, the implementation may have to adopt flushing the cache
every 'n' seconds or so with updating the timestamp while doing the
flush. I am not sure if this is a common practice or not...