Subject: Re: [ipfix] drawing the line between protocol and data model document

From: [email-masked]
Date: Thu, 01 May 2003 10:51:49 -0400

It sounds to me like the issue is at what level is the protocol
aware of the data which is being passed (i.e 4 byte value
vs IP address).

Jeff mentioned that it would be handly to know the value of some
unknown type is say an IP address. My question is what can you do with
that since you don't know what is the IP address of?

I'll add my 2 cents on where to draw the line between protocol
and data model.

The data model defines the semantics or meaning of the data along
with the valid range of values. Where appropriate we should reference
other standards. It also defines the meaing of multiple fields taken
together (e.g. this set of fields is the flow key).

It would be up to the protocol to define how the data is actaully
represented.

For example, the "Flow Octet Counter". The data model would specify
how/when the counter is incrememted.  And the data model would say 
it has has a valid range of 0 to 2^64. Yes, you could argue that a 
counter theoretically has no upper limit.

The richness of the protocol encoding specification is another
matter. The protocol would reference each element in the data
model and define exaclty how that element is encoded. In the above 
example, the protocol is free to encode that as an unsigned byte 
if the value is < 256. 

As another example, lets take Source IP address. The data model
would define that it is the value taken from the source address
slot as defined in rfc 793. It may also define how NAT addresses
are handled (example only, I'm not saying we do or don't handle NAT)

The protocol would define how it is encoded. For example
as a 4 byte unsigned value, or if it has the notion as an
IP address value. It could also define it in more general terms
as an address family, address length and address value. There are 
pros and cons in all three encodings.

Note that the base protocol (aka netflow) ties semantics and
value encoding together. The TYPE defines both the sematnics
of the data and how it is encoded.

Paul

"MEYER,JEFFREY D (HP-Cupertino,ex1)" wrote:
> 
> Mark,
> 
>   I'll try to clarify:
> 
>   Concept #1:  The information model and the encoding are separate.
>    A well defined mapping determines how an information model is
>    to be encoded for a specific use (e.g. IPFIX).  The encoding
>    may take whatever approach allows both ends to do something
>    useful with the data.  As you point out there is some useful
>    stuff which can be done with the length alone.
> 
>   Concept #2:  Derived typing.  By defining a base set of types,
>    which must be understood, derived types can be defined in terms
>    of these base types.  For instance Diameter does this.  This
>    enables a core set of types to be defined and coded to, but
>    also enables richer types later on with an appropriate fallback
>    for systems which aren't aware of the extensions.
> 
>   The benefit I see of explicit typing is that instead of seeing
> that a field is of length "4 bytes" something which will be seen
> A LOT.  Is that we can instead differentiate between IPAddr, Integer,
> unsigned integer, and float.  This seems handy to me.
> 
>   Your argument seems centered around the long term potential of
> adding lots of fixed length fields whose base length is not 4,8
> or 16 bytes long, for instance MACAddr (although if we agree on
> going the type route, I'd recommend adding MACAddr and then you
> have 1,2,4,6,8 and 16 bytes covered.  I guess I don't foresee
> a lot more fixed length fields coming down the pipe in the IPFIX
> domain.  Hence my desire to make it more explicit for the types
> which do occur all the time and are likely points of extension.
> WHY?  Because then a tool could at least present these in a meaningful
> way.  As opposed to incorrectly formatting a float type as an int.
> 
>   But hey, either way will work.  If the goal is to limit the
> amount of changes, I'll (grudgingly) go along.  I haven't heard
> any opinions from anyone but you in either direction.  So,
> at this point I don't see a quorom either way...
> 
> Regards,
> 
>   Jeff Meyer
> 
> > -----Original Message-----
> > From: Mark Fullmer [mailto:[email-masked]]
> > Sent: Tuesday, April 29, 2003 2:53 PM
> > To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> > Cc: [email-masked]
> > Subject: Re: [ipfix] drawing the line between protocol and data model
> > docu ment
> >
> >
> > IMHO crippling the protocol to serve the needs of a documentation tool
> > is just silly.
> >
> > It would be possible to use both a Type name and a Length instead of
> > the existing Length, which would allow a collector to process
> > unknown Types.
> >
> > With the existing v9 protocol a collector can ignore an
> > unknown attribute,
> > or possibly just pass it on to post processing tools which
> > might understand
> > it.
> >
> > Your suggested changes no longer allow a collector to ignore unknown
> > attributes if the Type is also unknown.  Resorting to encoding all
> > new Types after the initial protocol spec as octet strings is (IMHO)
> > not acceptable.
> >
> > I'm still trying to understand what the benefits to explicit
> > Types sent
> > in templates are.
> >
> > mark
> >
> > On Tue, Apr 29, 2003 at 02:50:02PM -0400, MEYER,JEFFREY D
> > (HP-Cupertino,ex1) wrote:
> > > Juergen,
> > >
> > >   I agree that the data model should have a well defined type
> > > system.   And that that typing system should reflect the
> > requirements.
> > >
> > >   This is in part why I was hoping that the protocol itself would
> > > be more explicit here.  (See my mail
> > > http://ipfix.doit.wisc.edu/archive/1604.html)
> > >
> > >   There is also a type list in that mail.  However, I would suggest
> > > using the names from XML-Schema part 2 Data types, as this would
> > > facilitate creating a formal/normative definition, versus
> > reinventing
> > > names for the types.
> > >
> > >   Here are the type names from XML-Schema/IPDR:
> > >
> > >      boolean      - true/false (may be encoded in byte)
> > >      byte
> > >      unsignedByte
> > >      short
> > >      unsignedShort
> > >      int          - 32-bit signed
> > >      long         - 64-bit signed
> > >      unsignedInt
> > >      unsignedLong
> > >      float
> > >      double
> > >      dateTime     - 32-bit integer representing seconds since EPOCH
> > > (1/1/1970 0:00 GMT)
> > >      string       - UTF-8 formatted text
> > >      ipdr:dateTimeMsec - 64-bit integer representing msecs
> > since EPOCH
> > >      ipdr:ipV4Addr
> > >      ipdr:ipV6Addr
> > >      ipdr:UUID
> > >      hexBinary    - arbitrary sequence of octets
> > >
> > >   If the protocol itself has a basic "octetString" (or from
> > XML, hexBinary)
> > > payload,
> > > then this can be used to carry new classes of information
> > in the future w/o
> > > rejiggering the protocol itself.
> > >
> > >   In addition other "derived" types may be created from the
> > base types.  For
> > > instance in IPDR, the ipdr:dateTimeMsec is derived from long.
> > >
> > > Regards,
> > >
> > >   Jeff Meyer
> > >
> > > > -----Original Message-----
> > > > From: Juergen Quittek [mailto:[email-masked]]
> > > > Sent: Monday, April 28, 2003 2:06 PM
> > > > To: [email-masked]
> > > > Subject: [ipfix] drawing the line between protocol and data model
> > > > document
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > When shaping the planned documents, we have to draw a line
> > > > between the protocol and the data model.
> > > >
> > > > Definitely, the data model document defines all field types,
> > > > but does it also define basic data types, such as '16 bit
> > > > unsigned integer'?
> > > >
> > > > Maybe we should have a set of basic data types defined in
> > > > the protocol document and restrict the data model doc to
> > these types?
> > > > This would further structure the data model.
> > > >
> > > > Any thoughts?
> > > >
> > > >     Juergen
> > > >
> > > > --
> > > > Help        mailto:[email-masked] and say "help"
> > > > in message body
> > > > Unsubscribe mailto:[email-masked] and say
> > > > "unsubscribe ipfix" in message body
> > > > Archive     http://ipfix.doit.wisc.edu/archive/
> > > >
> > >
> > > --
> > > Help        mailto:[email-masked] and say
> > "help" in message body
> > > Unsubscribe mailto:[email-masked] and say
> > > "unsubscribe ipfix" in message body
> > > Archive     http://ipfix.doit.wisc.edu/archive/
> >
> 
> --
> Help        mailto:[email-masked] and say "help" in message body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 1 May 2003 11:38:32 -0400

So the difference boils down to a tool being able to display

<UNKNOWN,ID=0x1234> = 0x01020304
  vs
<UNKNOWN,ID=0x1234> = 1.2.3.4

In this situation someone will either grab a new version of the
software or an updated data definition to resolve the unknown attribute,
which would also define it's type.

mark

---

From: Juergen Quittek <[email-masked]>
Date: Thu, 01 May 2003 17:32:35 +0200

Paul,

Do I understand you right, that you suggest, that for each flow attribute
the info model doc defines the semantics and the protocol document defines
the data encoding?

What about the following:
The protocol document defines a set of basic types (counter64, IP address, ...)
including their binary representation (big endian, ...).
And the info model document defines the the list of all attributes, defines
their semantics and selects a basic type from the protocol doc for each attribute.
This would have the advantage, that the protocol doc must not contain a list of
all attributes. Also extending the attributes can be done by just updating the
info model doc (assuming that the set of basic data types is sufficient for the
extension).

    Juergen

---

From: [email-masked]
Date: Thu, 01 May 2003 11:53:55 -0400

> which would also define it's type.

	I don't see much value in that.

	Paul

---

From: Mark Fullmer <[email-masked]>
Date: Thu, 1 May 2003 16:21:16 -0400

On Thu, May 01, 2003 at 10:51:49AM -0400, [email-masked] wrote:

<snip>

> example, the protocol is free to encode that as an unsigned byte 

The ramifications of doing this with a template based system are a bit
ugly.  Consider a flow with just 2 counters -- octets, packets.
Each could have a 1 to 8 byte encoding.  That's a potential of 64
different templates for representing 1 type of flow.

This (at first glance) looks like it adds considerable cycles to the
encode / decode process to save a few bytes in the transport.

The current v9 spec appears to allow this, but only for IN_BYTES and
IN_PKTS.

>                                           counter with length
>    IN_BYTES                 1       N     N x 8 bits for bytes
>                                           associated with an IP Flow

Is this true?  Benoit?

mark

---

From: [email-masked]
Date: Tue, 06 May 2003 09:51:11 -0400

Just a note on this. I'm not at this point advocating any
encoding scheme. Merely saying that the data model is
independent of the encoding scheme. Whether you encode it
as 8 bytes or 1 byte doesn't matter to the data model as
the semantics remain the same.

However, that said, I still need to think more about this
issue from a practical standpoint. Theory doesn't always
line up nicely with reality :-)

Paul