Subject: [IPFIX] unobserved fields

From: Andrew Feren <[email-masked]>
Date: Fri, 07 Dec 2012 13:11:24 -0500

Hi all,

My earlier email about Extended Field Specifiers reminded me of an other 
draft (draft-aitken-ipfix-unobserved-fields) that doesn't seem to have 
gone anywhere, but that I would love to move forward. Having a standard 
way to know if  field is unobserved/NULL for a given flow would be very 
valuable.

Any other interest?

-Andrew

---

From: Paul Aitken <[email-masked]>
Date: Fri, 07 Dec 2012 19:01:43 +0000

Andrew,

> My earlier email about Extended Field Specifiers reminded me of an 
> other draft (draft-aitken-ipfix-unobserved-fields) that doesn't seem 
> to have gone anywhere, but that I would love to move forward. Having a 
> standard way to know if  field is unobserved/NULL for a given flow 
> would be very valuable.

It's not that this draft didn't go anywhere. Rather, I've written all I 
can for now, yet the WG isn't in a position to adopt the work. However 
it's a real issue that needs to be solved -in fact, we're already using 
some of the techniques in this document - so in my mind it's only "on 
hold" until I think of something to add and/or the WG can adopt it.

You're right, Extended Field Specifiers would allow us to tag each field 
with an "Observed" attribute with advantages:

     - we wouldn't need to change fields to varlen just to report length 
= 0.

     - we could have a many-valued reason rather than a boolean.
         ie, we can report why the value is not available or not applicable.

P.

---

From: Brian Trammell <[email-masked]>
Date: Sun, 9 Dec 2012 10:32:15 +0100

Hi, Paul, Andrew,

I'm not sure I understand the proposed mechanism here...

Observed/unobserved is a per-field/per-record attribute, while extended field specifiers modify IEs per-column: every instance of a given IE in a Template is "decorated" with its extension.
 
As I understand the requirements for -unobserved-fields, we want a way to distinguish, from an MP: "I am capable of observing this attribute, but it was not present in the observation" versus "I am not capable of observing this attribute". We have the second already: simply leave the IE out of the template. And, indeed, you can also use leaving things out of the template to signify the first: the presence of a field in some templates from an EP/MP combination implies that it's observable, its absence in the other implies it's not observed. FWIW, YAF does this for those things that haven't moved behind the 6313 curtain: no reverse elements if it's not a biflow, for instance.

Given that, how would you use a per-IE specifier to signal a per-record attribute? What am I missing?

Cheers,

Brian

On 7 Dec 2012, at 20:01, Paul Aitken <[email-masked]> wrote:

> Andrew,
> 
>> My earlier email about Extended Field Specifiers reminded me of an other draft (draft-aitken-ipfix-unobserved-fields) that doesn't seem to have gone anywhere, but that I would love to move forward. Having a standard way to know if  field is unobserved/NULL for a given flow would be very valuable.
> 
> It's not that this draft didn't go anywhere. Rather, I've written all I can for now, yet the WG isn't in a position to adopt the work. However it's a real issue that needs to be solved -in fact, we're already using some of the techniques in this document - so in my mind it's only "on hold" until I think of something to add and/or the WG can adopt it.
> 
> You're right, Extended Field Specifiers would allow us to tag each field with an "Observed" attribute with advantages:
> 
>    - we wouldn't need to change fields to varlen just to report length = 0.
> 
> _______________________________________________
> IPFIX mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipfix

---

From: Paul Aitken <[email-masked]>
Date: Sun, 09 Dec 2012 18:01:39 +0000

Brian,

> Observed/unobserved is a per-field/per-record attribute, while extended field specifiers modify IEs per-column: every instance of a given IE in a Template is "decorated" with its extension.

Actually each EFS can contribute additional data within the template 
(ie, applying per column) and/or additional info within data records 
(ie, applying per row).

Look at slide 7 here: 
http://tools.ietf.org/agenda/85/slides/slides-85-ipfix-2.pdf

eg, within the template we'd want to indicate which OID is being 
exported, while we want to indicate indices per data-record.

> As I understand the requirements for -unobserved-fields, we want a way to distinguish, from an MP: "I am capable of observing this attribute, but it was not present in the observation"

Or, I don't (yet) have sufficient data to calculate a derived metric, eg 
average / min / max, count or time based metrics.

> versus "I am not capable of observing this attribute". We have the second already: simply leave the IE out of the template.

No. Leaving the metric out of the template does not actively indicate 
that the MP is not capable of measuring it. A CP cannot make any 
assumptions about metrics which the MP isn't reporting. Rather than this 
*passive* method, we're missing a way for the MP to *actively* inform 
the CP that although it's capable of measuring some metric, it's 
currently not able to provide any further information about the metric.

> And, indeed, you can also use leaving things out of the template to signify the first: the presence of a field in some templates from an EP/MP combination implies that it's observable, its absence in the other implies it's not observed.

No it doesn't. That's the whole point of -unobserved-fields! The MP 
could have observed it and between the MP and EP, decided not to export 
it. eg, certain IP addresses, user names, URLs, could be filtered from 
export stream for privacy or security. The absence of a field in the 
template isn't directly related to the field's observability.

> FWIW, YAF does this for those things that haven't moved behind the 6313 curtain: no reverse elements if it's not a biflow, for instance.
>

EFS are also per-record, eg OID index. See the MIB draft.

P.

---

From: Brian Trammell <[email-masked]>
Date: Mon, 10 Dec 2012 10:18:25 +0100

hi, Paul, all,

Ahhhh -- okay, so you'd be proposing using an indexing mechanism, as with MIB objects, to explicitly link one IE in a record with another to allow the "observation index" IE to expose the observation state of each IE?

I think an example illustrating this -- how MIB-style EFS would support -unobserved-fields -- in the next rev of -unobserved-fields would be very useful.

A couple of other points inline...

On 9 Dec 2012, at 19:01, Paul Aitken <[email-masked]> wrote:

> Or, I don't (yet) have sufficient data to calculate a derived metric, eg average / min / max, count or time based metrics.
> 
> 
> No. Leaving the metric out of the template does not actively indicate that the MP is not capable of measuring it. A CP cannot make any assumptions about metrics which the MP isn't reporting. Rather than this *passive* method, we're missing a way for the MP to *actively* inform the CP that although it's capable of measuring some metric, it's currently not able to provide any further information about the metric.

Absolutely agree; I'm pointing out the "way to do this today" -- which is ambiguous.

> No it doesn't. That's the whole point of -unobserved-fields! The MP could have observed it and between the MP and EP, decided not to export it. eg, certain IP addresses, user names, URLs, could be filtered from export stream for privacy or security. The absence of a field in the template isn't directly related to the field's observability.

As above. I'm not saying this is the right way to do things; I'm saying it's the de facto approach.

> EFS are also per-record, eg OID index. See the MIB draft.

Okay, so EFS -- specificially, any extension applying an "indexing" feature -- allows specifying that one IE in a Template is present to modify another, so the per-record information still appears in the record; it's just that EFS has allowed an explicit linkage between the IEs, where before we'd have only an implicit linkage (bad, because it's very easy to interpret incorrectly) or an explicit linkage keyed to a specific function in the definition of the IE, e.g. flowKeyIndicator (less bad but still bad, because continuing along these lines you run out of IEs pretty quickly)...correct?

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Mon, 10 Dec 2012 09:53:37 +0000

Brian,

> Ahhhh -- okay, so you'd be proposing using an indexing mechanism, as with MIB objects, to explicitly link one IE in a record with another to allow the "observation index" IE to expose the observati…

No. There's only one IE.

In MIB export, the template effectively says "here is a MIB, with two 
extensions; the first defines the OID (in the template); the second 
defines the index (in the data records)". Appropriate sizes are given, 
and the template and data records are extended accordingly. So instead 
of getting a 4-byte value in the data record, there might be 6 bytes: 
the 4 bytes value followed by a 2-bytes index. However the index has no 
IE of its own, so each new extension does not require a new IE.

In the unobserved fields scenario, the template would says "here is an 
IE, with an extension (in the data records)", and that extension would 
indicate the observability of the IE. So instead of 4 bytes of RTT time, 
there might be 5 bytes: 4 bytes of RTT (being zero, presumably), and 1 
byte reporting "insufficient time".

> I think an example illustrating this -- how MIB-style EFS would support -unobserved-fields -- in the next rev of -unobserved-fields would be very useful.

Are MIBS ever unobserved?

> Absolutely agree; I'm pointing out the "way to do this today" -- which is ambiguous.

Great, we agree.

> As above. I'm not saying this is the right way to do things; I'm saying it's the de facto approach.

Which has this huge gap...

> Okay, so EFS -- specificially, any extension applying an "indexing" feature -- allows specifying that one IE in a Template is present to modify another, so the per-record information still appears…

Sort of, except there's only one IE with additional data. I'm not 
proposing new IEs for every new attribute specified in an EFS.

P.

---

From: Brian Trammell <[email-masked]>
Date: Mon, 10 Dec 2012 11:21:12 +0100

>> Ahhhh -- okay, so you'd be proposing using an indexing mechanism, as with MIB objects, to explicitly link one IE in a record with another to allow the "observation index" IE to expose the observat…

Ah. Okay. Got it. I'd missed something in my reading of the MIB draft, then (or, rather, I'd paid the most attention to the IE-index-based indexing, which looks the "most IPFIXish", to me anyway, and sort of assumed the others looked the same...)

> In MIB export, the template effectively says "here is a MIB, with two extensions; the first defines the OID (in the template); the second defines the index (in the data records)". Appropriate sizes are given, and the template and data records are extended accordingly. So instead of getting a 4-byte value in the data record, there might be 6 bytes: the 4 bytes value followed by a 2-bytes index. However the index has no IE of its own, so each new extension does not require a new IE.
> 
> In the unobserved fields scenario, the template would says "here is an IE, with an extension (in the data records)", and that extension would indicate the observability of the IE. So instead of 4 bytes of RTT time, there might be 5 bytes: 4 bytes of RTT (being zero, presumably), and 1 byte reporting "insufficient time".

thinking aloud... so, the "insufficient time" not-IE is an entry in a registry of "in-record IE decorator", like MIB index; there's an extension for "in-record IE decorator" which points to this registry entry; presumably, this decorator is "generic" enough to apply to most or all IEs? (actually, I'd call it just "insufficient data"; maybe it makes sense to combine this with other flags for per-record attributes, for the sake of efficiency...)

> Are MIBS ever unobserved?

I'm not enough of an SMI geek to answer that question. I'd seen MIB export as simply another source of information elements, so would ask "why not?" but there may be reasons why not that I don't appreciate.

(we've agreed to agree on the rest of the points, so <snip>)

Cheers,

Brian

---

From: Paul Aitken <[email-masked]>
Date: Mon, 10 Dec 2012 10:46:39 +0000

Brian,

> thinking aloud... so, the "insufficient time" not-IE is an entry in a registry of "in-record IE decorator", like MIB index; there's an extension for "in-record IE decorator" which points to this re…

Yes, there's a 16-bit type field in the EFS which specifies what the 
field attribute is (eg, OID, index, unobserved, ...). Actually I've been 
wonder whether there are any EFS which require data in both the template 
and the data record; if not then the topmost type bit could indicate 
this, so there'd only need to be one length field (rather than one for 
template info + one for data record info).

Since the "unobserved" field needs to be at least 8 bits (since IPFIX 
deals with bytes rather than bits), we can run to 255 "unobserved" 
reasons (plus zero for "observed").
So although we could start with just a few generic reasons, we can 
afford to call out exactly why no data was available. eg, "insufficient" 
could be due to insufficient time (eg, RTT or 1-minute average) or 
insufficient data (eg, average / mean).

P.