Subject: RE: [ipfix] drawing the line between protocol and data model docu ment

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Sat, 10 May 2003 00:10:46 -0400

Paul,

Thanks for putting forward a good example to talk around.

I'd like to clarify the concept of typing of attributes.  In particular the
concept 
of "base types" and "derived types".   Your dismissal of option II, implies
to me that
you are not making the distinction between these two types of types.

If we assume that extending the typing system is a good thing, then there
still exists
the possibility of defining a set of "base types" which are defined ONCE in
the base
protocol and then defining a set of "derived types" which have additional
semantics, 
but are encoded in the format of one of the base types.

This concept is not new.  Look at any programming language or database
schema.
Or look at the Diameter base protocol sections 4.2 and 4.3.

  http://www.diameter.org/drafts/latest/draft-ietf-aaa-diameter-17.txt

Section 4.2 is titled. "Basic AVP Data Formats". Section 4.3 is titled
"Derived AVP 
Data Formats"

Section 4.2 defines the following set of base data types:

      OctetString
      Integer32
      Integer64
      Unsigned32
      Unsigned64
      Float32
      Float64

In addition, Diameter defines "Grouped" which is structured type, which is
not relevant
to IPFIX, in my opinion.

My proposal had a few additional "base types" indicated by "*":

  *boolean      - true/false (may be encoded in byte)
  *byte
  *unsignedByte
  *short
  *unsignedShort
  int          - 32-bit signed
  long         - 64-bit signed
  unsignedInt
  unsignedLong
  float
  double
  *dateTime     - 32-bit integer representing seconds
  *string       - UTF-8 formatted text
  *ipdr:dateTimeMsec - 64-bit integer representing msecs
  *ipdr:ipV4Addr
  *ipdr:ipV6Addr
  *ipdr:UUID
  hexBinary    - arbitrary sequence of octets

Derived types have a relationship to a base type.  I.e. when defining a
derived
type you reference the unique base type from which it is derived.

So based on the discussion and the notation you introduced there are 4
pieces of
information a collector should know, either explicitly or indirectly:

  - The attribute id
  - The derived attribute type
  - The base attribute type
  - The length of the encoded attribute

More details on each:

  - The "attribute id" identifies a unique attribute which may appear in
    an IPFIX record.  The attribute ids  are assigned as part of the
information
    model.  Extension of the information model, implies adding new
attributes,
    with new attribute ids.  This implies some sort of managed namespace
    (or namespaces).  Managing the namespace implies that a registry of
defined
    attribute ids is maintained, and that additional attribute ids can be
incorporated 
    through some process at a later date.
    
  - The "attribute type" indicates the type of information held in the
attribute.
    E.g. is it an integer, a string, a floating point value, etc.  For
practical
    reasons, types may reflect some encoding or machine representation
information.
    E.g. for a human we could simply indicate whether a particular attribute
is
    numeric, but for practical reasons, distinguishing between shorts,
longs,
    floats, etc. is also included.
    
    To enable extension, new attribute types may be introduced later.
However, 
    these new types should be considered as derived from some base type, to 
    enable legacy implementations to do some basic parsing of the field.
The base 
    type should be capable of completely holding the data represented by the
derived 
    type, so that no information loss occurs.  Since one of the base types
is hexBinary 
    (same as OctetString in Diameter), any new type can always be completely
held
    by this base type.
    
  - The length is the number of bytes occupied in the the encoded IPFIX
record
    for a given attribute.

If an implementation was guaranteed access to the information model(s) used
by
a given IPFIX expoter, then (in theory) a collector would ONLY need the 
attribute ID in a template.  Since all other information could be inferred
from the information model (assuming it was machine readable).

For practical reasons, it is best to assume that the information model may
not be available.  And to assume the collector may not be in synch with any
recent
changes to the information model, including the introduction of new derived
types.

In this case the collector can completely parse the record with either:

  a. (id, encoded length)   [Paul's option I]
  b. (id, base type).  

It will NOT work if we provide the collector with ONLY the following
information 
(id, derived type) [Paul's option II], because the collector 
may not be able to determine how many bytes to skip.

One might chose to provide additional information in the template such as:

  c. (id, derived type, length) [Paul's option III]
  d. (id, base type, length)
  e. (id, derived type, base type)
  f. (id, derived type, base type, length)

Since compactness is a concern, and since there is clearly confusion around
the use of base types vs. derived types in the encoding, I'm inclined to
throw
in the towel and say let's continue with option a, which is what NFv9 does
today.

However, I hope is also clear to folks now, that by using the concept of
base types,
length does NOT need to be in the template to enable parsing of unknown 
attributes.

Regards,

  Jeff Meyer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, May 06, 2003 8:45 AM
To: MEYER,JEFFREY D (HP-Cupertino,ex1)
Cc: 'Sebastian Zander'; 'Mark Fullmer'; [email-masked]
Subject: Re: [ipfix] drawing the line between protocol and data model
document

OK. I'm confused. Let me run through an example and people can correct 
me where I'm wrong.

Lets take "Source IP v4 address". Lets also hold off on variable
length fields for now.

All options need some kind of type value to indicate what the value
semantically means. Lets pick 57 == Source IP v4 address. So what we 
need next is info on the value part so we can decode it and it should 
be backward compatible (i.e. older versions can at least skip unknown 
elements.

Option I: Element and Length

	57, 4

Any software understanding element 57 knows it is the Source
IP address and can display/handle it correctly. Any one else would
skip over the value in the flow set.

Option II: Element and data type

Lets say data type 22 == IP V4 Address

	57, 22

Any software understanding element 57 knows it is the Source
IP address. But what if it doesn't know what data type 22 is?
How can I skip over it?

Option III: Element and data type and length

	57, 22, 4

Any software which understands 57 and 22 can handle it correclty.
If it only understands 22, it can ONLY display the value in
some rational way, nothing else can be done. If I don't understand 22
I can at least skip over it by using the length.

Correct away!

Paul

"MEYER,JEFFREY D (HP-Cupertino,ex1)" wrote:
> 
> Sebastian,
> 
>   Thanks for speaking up!  I agree with your restatement of the
> relationship between typing and semantics.
> 
>   Regarding types like UTF8 or OctetString, as I read the proposal
> from Benoit, the "Length Encoding" model would use a special value
> of 0xFFFF instead of length in the template.  This implies that
> the encoding of the "variable length" field would be of the form
> (length) (value), versus fixed length types which can be encoded by
> just placing their (value) in the payload.
> 
>   (see http://ipfix.doit.wisc.edu/archive/1602.html)
> 
>   This same model would apply in a type based scheme, if the type
> were string or hexBinary (indicated by the appropriate integer
> identifier in the template), then the encoded value would still be
> of the form (length) (value) in the payload.
> 
>   Regardless of what template scheme is chosen, however, I think it
> is important that the protocol spec speak in terms of how types are
> encoded, not how each individual element declared in the information model
> is encoded.  This makes the inevitable extension of the information
> model easier.
> 
> Regards,
> 
>   Jeff Meyer
> 
> > -----Original Message-----
> > From: Sebastian Zander [mailto:[email-masked]]
> > Sent: Thursday, May 01, 2003 5:57 PM
> > To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> > Cc: 'Mark Fullmer'; [email-masked]
> > Subject: Re: [ipfix] drawing the line between protocol and data model
> > docu ment
> >
> >
> > Jeff,
> >
> > i support the idea of typing. I think there is some
> > misunderstanding here
> > by some people who confuse types with semantics. A type does
> > not define the
> > semantics. In my opinion typing has the advantage that even
> > if i don't know
> > the semantics by the time of the decoding i still can
> > correctly decode the
> > value and present it. It also has the advantage that for a
> > well known type
> > the encoding is well defined. If that is not the case people
> > may encode
> > attributes of the same type differently just making
> > interoperability harder.
> > I would define these basic data types in the protocol document.
> >
> > I am not so sure if both alternatives take the same space
> > because there
> > are some types which which do not have an intrinsic length
> > definition. An
> > utf or octet string could have an arbitrary length. Without
> > typing everything
> > would be basically an octetstring of some length ;-). However
> > i though the
> > protocol is supossed to use templates and so it would only affect the
> > template exchange wouldn't it?
> >
> > Cheers,
> >
> > Sebastian
> >
> > MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> > > vs.
> > >
> > > <UNKNOWN,ID=0x1234> = 16909060 (integer)
> > >
> > > vs.
> > >
> > > <UNKNOWN,ID=0x1234> = whatever the heck 0x01020304 represents
> > >   in IEEE Floating Point notation.  (something around 4.06)
> > >
> > >
> > > So, yes that's all it buys you.  Since both alternatives take
> > > the same amount of space, I'd prefer the one with more
> > > intrinsic information.  But, I guess I'm in the minority :-(
> > >
> > >
> > > Regards,
> > >
> > >   Jeff Meyer
> > >
> > >
> > >
> > >>-----Original Message-----
> > >>From: Mark Fullmer [mailto:[email-masked]]
> > >>Sent: Thursday, May 01, 2003 8:39 AM
> > >>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> > >>Cc: [email-masked]
> > >>Subject: Re: [ipfix] drawing the line between protocol and
> > data model
> > >>docu ment
> > >>
> > >>
> > >>So the difference boils down to a tool being able to display
> > >>
> > >><UNKNOWN,ID=0x1234> = 0x01020304
> > >>  vs
> > >><UNKNOWN,ID=0x1234> = 1.2.3.4
> > >>
> > >>In this situation someone will either grab a new version of the
> > >>software or an updated data definition to resolve the unknown
> > >>attribute,
> > >>which would also define it's type.
> > >>
> > >>mark
> > >>
> > >>On Tue, Apr 29, 2003 at 06:25:11PM -0400, MEYER,JEFFREY D
> > >>(HP-Cupertino,ex1) wrote:
> > >>
> > >>>Mark,
> > >>>
> > >>>  I'll try to clarify:
> > >>>
> > >>>  Concept #1:  The information model and the encoding are separate.
> > >>>   A well defined mapping determines how an information model is
> > >>>   to be encoded for a specific use (e.g. IPFIX).  The encoding
> > >>>   may take whatever approach allows both ends to do something
> > >>>   useful with the data.  As you point out there is some useful
> > >>>   stuff which can be done with the length alone.
> > >>>
> > >>>  Concept #2:  Derived typing.  By defining a base set of types,
> > >>>   which must be understood, derived types can be defined in terms
> > >>>   of these base types.  For instance Diameter does this.  This
> > >>>   enables a core set of types to be defined and coded to, but
> > >>>   also enables richer types later on with an appropriate fallback
> > >>>   for systems which aren't aware of the extensions.
> > >>>
> > >>>  The benefit I see of explicit typing is that instead of seeing
> > >>>that a field is of length "4 bytes" something which will be seen
> > >>>A LOT.  Is that we can instead differentiate between
> > >>
> > >>IPAddr, Integer,
> > >>
> > >>>unsigned integer, and float.  This seems handy to me.
> > >>>
> > >>>  Your argument seems centered around the long term potential of
> > >>>adding lots of fixed length fields whose base length is not 4,8
> > >>>or 16 bytes long, for instance MACAddr (although if we agree on
> > >>>going the type route, I'd recommend adding MACAddr and then you
> > >>>have 1,2,4,6,8 and 16 bytes covered.  I guess I don't foresee
> > >>>a lot more fixed length fields coming down the pipe in the IPFIX
> > >>>domain.  Hence my desire to make it more explicit for the types
> > >>>which do occur all the time and are likely points of extension.
> > >>>WHY?  Because then a tool could at least present these in a
> > >>
> > >>meaningful
> > >>
> > >>>way.  As opposed to incorrectly formatting a float type as an int.
> > >>>
> > >>>  But hey, either way will work.  If the goal is to limit the
> > >>>amount of changes, I'll (grudgingly) go along.  I haven't heard
> > >>>any opinions from anyone but you in either direction.  So,
> > >>>at this point I don't see a quorom either way...
> > >>>
> > >>>Regards,
> > >>>
> > >>>  Jeff Meyer
> > >>>
> > >>>
> > >>>
> > >>>>-----Original Message-----
> > >>>>From: Mark Fullmer [mailto:[email-masked]]
> > >>>>Sent: Tuesday, April 29, 2003 2:53 PM
> > >>>>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> > >>>>Cc: [email-masked]
> > >>>>Subject: Re: [ipfix] drawing the line between protocol
> > >>>
> > >>and data model
> > >>
> > >>>>docu ment
> > >>>>
> > >>>>
> > >>>>IMHO crippling the protocol to serve the needs of a
> > >>>
> > >>documentation tool
> > >>
> > >>>>is just silly.
> > >>>>
> > >>>>It would be possible to use both a Type name and a Length
> > >>>
> > >>instead of
> > >>
> > >>>>the existing Length, which would allow a collector to process
> > >>>>unknown Types.
> > >>>>
> > >>>>With the existing v9 protocol a collector can ignore an
> > >>>>unknown attribute,
> > >>>>or possibly just pass it on to post processing tools which
> > >>>>might understand
> > >>>>it.
> > >>>>
> > >>>>Your suggested changes no longer allow a collector to
> > >>>
> > >>ignore unknown
> > >>
> > >>>>attributes if the Type is also unknown.  Resorting to encoding all
> > >>>>new Types after the initial protocol spec as octet
> > >>>
> > >>strings is (IMHO)
> > >>
> > >>>>not acceptable.
> > >>>>
> > >>>>I'm still trying to understand what the benefits to explicit
> > >>>>Types sent
> > >>>>in templates are.
> > >>>>
> > >>>>mark
> > >>>>
> > >>>>On Tue, Apr 29, 2003 at 02:50:02PM -0400, MEYER,JEFFREY D
> > >>>>(HP-Cupertino,ex1) wrote:
> > >>>>
> > >>>>>Juergen,
> > >>>>>
> > >>>>>  I agree that the data model should have a well defined type
> > >>>>>system.   And that that typing system should reflect the
> > >>>>
> > >>>>requirements.
> > >>>>
> > >>>>>  This is in part why I was hoping that the protocol
> > >>>>
> > >>itself would
> > >>
> > >>>>>be more explicit here.  (See my mail
> > >>>>>http://ipfix.doit.wisc.edu/archive/1604.html)
> > >>>>>
> > >>>>>  There is also a type list in that mail.  However, I
> > >>>>
> > >>would suggest
> > >>
> > >>>>>using the names from XML-Schema part 2 Data types, as this would
> > >>>>>facilitate creating a formal/normative definition, versus
> > >>>>
> > >>>>reinventing
> > >>>>
> > >>>>>names for the types.
> > >>>>>
> > >>>>>  Here are the type names from XML-Schema/IPDR:
> > >>>>>
> > >>>>>     boolean      - true/false (may be encoded in byte)
> > >>>>>     byte
> > >>>>>     unsignedByte
> > >>>>>     short
> > >>>>>     unsignedShort
> > >>>>>     int          - 32-bit signed
> > >>>>>     long         - 64-bit signed
> > >>>>>     unsignedInt
> > >>>>>     unsignedLong
> > >>>>>     float
> > >>>>>     double
> > >>>>>     dateTime     - 32-bit integer representing seconds
> > >>>>
> > >>since EPOCH
> > >>
> > >>>>>(1/1/1970 0:00 GMT)
> > >>>>>     string       - UTF-8 formatted text
> > >>>>>     ipdr:dateTimeMsec - 64-bit integer representing msecs
> > >>>>
> > >>>>since EPOCH
> > >>>>
> > >>>>>     ipdr:ipV4Addr
> > >>>>>     ipdr:ipV6Addr
> > >>>>>     ipdr:UUID
> > >>>>>     hexBinary    - arbitrary sequence of octets
> > >>>>>
> > >>>>>  If the protocol itself has a basic "octetString" (or from
> > >>>>
> > >>>>XML, hexBinary)
> > >>>>
> > >>>>>payload,
> > >>>>>then this can be used to carry new classes of information
> > >>>>
> > >>>>in the future w/o
> > >>>>
> > >>>>>rejiggering the protocol itself.
> > >>>>>
> > >>>>>  In addition other "derived" types may be created from the
> > >>>>
> > >>>>base types.  For
> > >>>>
> > >>>>>instance in IPDR, the ipdr:dateTimeMsec is derived from long.
> > >>>>>
> > >>>>>Regards,
> > >>>>>
> > >>>>>  Jeff Meyer
> > >>>>>
> > >>>>>
> > >>>>>>-----Original Message-----
> > >>>>>>From: Juergen Quittek [mailto:[email-masked]]
> > >>>>>>Sent: Monday, April 28, 2003 2:06 PM
> > >>>>>>To: [email-masked]
> > >>>>>>Subject: [ipfix] drawing the line between protocol
> > >>>>>
> > >>and data model
> > >>
> > >>>>>>document
> > >>>>>>
> > >>>>>>
> > >>>>>>Hi all,
> > >>>>>>
> > >>>>>>When shaping the planned documents, we have to draw a line
> > >>>>>>between the protocol and the data model.
> > >>>>>>
> > >>>>>>Definitely, the data model document defines all field types,
> > >>>>>>but does it also define basic data types, such as '16 bit
> > >>>>>>unsigned integer'?
> > >>>>>>
> > >>>>>>Maybe we should have a set of basic data types defined in
> > >>>>>>the protocol document and restrict the data model doc to
> > >>>>>
> > >>>>these types?
> > >>>>
> > >>>>>>This would further structure the data model.
> > >>>>>>
> > >>>>>>Any thoughts?
> > >>>>>>
> > >>>>>>    Juergen
> > >>>>>>
> > >>>>>>--
> > >>>>>>Help        mailto:[email-masked] and say "help"
> > >>>>>>in message body
> > >>>>>>Unsubscribe mailto:[email-masked] and say
> > >>>>>>"unsubscribe ipfix" in message body
> > >>>>>>Archive     http://ipfix.doit.wisc.edu/archive/
> > >>>>>>
> > >>>>>
> > >>>>>--
> > >>>>>Help        mailto:[email-masked] and say
> > >>>>
> > >>>>"help" in message body
> > >>>>
> > >>>>>Unsubscribe mailto:[email-masked] and say
> > >>>>>"unsubscribe ipfix" in message body
> > >>>>>Archive     http://ipfix.doit.wisc.edu/archive/
> > >>>>
> > >>>--
> > >>>Help        mailto:[email-masked] and say
> > >>
> > >>"help" in message body
> > >>
> > >>>Unsubscribe mailto:[email-masked] and say
> > >>>"unsubscribe ipfix" in message body
> > >>>Archive     http://ipfix.doit.wisc.edu/archive/
> > >>
> > >
> > > --
> > > Help        mailto:[email-masked] and say
> > "help" in message body
> > > Unsubscribe mailto:[email-masked] and say
> > > "unsubscribe ipfix" in message body
> > > Archive     http://ipfix.doit.wisc.edu/archive/
> > >
> >
> >
> > --
> > Sebastian Zander                         E-mail:
> > [email-masked]
> > Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
> > Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
> > D-10589 Berlin, Germany
> > www.fokus.fraunhofer.de/usr/sebastian.zander
> >
> >
> >
> >
> 
> --
> Help        mailto:[email-masked] and say "help" in message
body
> Unsubscribe mailto:[email-masked] and say
> "unsubscribe ipfix" in message body
> Archive     http://ipfix.doit.wisc.edu/archive/

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/