Subject: RE: [ipfix] drawing the line between protocol and data model docu ment

From: "MEYER,JEFFREY D (HP-Cupertino,ex1)" <[email-masked]>
Date: Thu, 1 May 2003 22:36:17 -0400

Sebastian,

  Thanks for speaking up!  I agree with your restatement of the
relationship between typing and semantics.

  Regarding types like UTF8 or OctetString, as I read the proposal
from Benoit, the "Length Encoding" model would use a special value
of 0xFFFF instead of length in the template.  This implies that 
the encoding of the "variable length" field would be of the form  
(length) (value), versus fixed length types which can be encoded by 
just placing their (value) in the payload.

  (see http://ipfix.doit.wisc.edu/archive/1602.html)

  This same model would apply in a type based scheme, if the type
were string or hexBinary (indicated by the appropriate integer
identifier in the template), then the encoded value would still be 
of the form (length) (value) in the payload.

  Regardless of what template scheme is chosen, however, I think it 
is important that the protocol spec speak in terms of how types are 
encoded, not how each individual element declared in the information model
is encoded.  This makes the inevitable extension of the information
model easier.

Regards,

  Jeff Meyer

> -----Original Message-----
> From: Sebastian Zander [mailto:[email-masked]]
> Sent: Thursday, May 01, 2003 5:57 PM
> To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> Cc: 'Mark Fullmer'; [email-masked]
> Subject: Re: [ipfix] drawing the line between protocol and data model
> docu ment
> 
> 
> Jeff,
> 
> i support the idea of typing. I think there is some 
> misunderstanding here
> by some people who confuse types with semantics. A type does 
> not define the
> semantics. In my opinion typing has the advantage that even 
> if i don't know
> the semantics by the time of the decoding i still can 
> correctly decode the
> value and present it. It also has the advantage that for a 
> well known type
> the encoding is well defined. If that is not the case people 
> may encode
> attributes of the same type differently just making 
> interoperability harder.
> I would define these basic data types in the protocol document.
> 
> I am not so sure if both alternatives take the same space 
> because there
> are some types which which do not have an intrinsic length 
> definition. An
> utf or octet string could have an arbitrary length. Without 
> typing everything
> would be basically an octetstring of some length ;-). However 
> i though the
> protocol is supossed to use templates and so it would only affect the
> template exchange wouldn't it?
> 
> Cheers,
> 
> Sebastian
> 
> MEYER,JEFFREY D (HP-Cupertino,ex1) wrote:
> > vs.
> > 
> > <UNKNOWN,ID=0x1234> = 16909060 (integer)
> > 
> > vs.
> > 
> > <UNKNOWN,ID=0x1234> = whatever the heck 0x01020304 represents
> >   in IEEE Floating Point notation.  (something around 4.06)
> > 
> > 
> > So, yes that's all it buys you.  Since both alternatives take
> > the same amount of space, I'd prefer the one with more
> > intrinsic information.  But, I guess I'm in the minority :-(
> > 
> > 
> > Regards,
> > 
> >   Jeff Meyer
> > 
> > 
> > 
> >>-----Original Message-----
> >>From: Mark Fullmer [mailto:[email-masked]]
> >>Sent: Thursday, May 01, 2003 8:39 AM
> >>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> >>Cc: [email-masked]
> >>Subject: Re: [ipfix] drawing the line between protocol and 
> data model
> >>docu ment
> >>
> >>
> >>So the difference boils down to a tool being able to display
> >>
> >><UNKNOWN,ID=0x1234> = 0x01020304
> >>  vs
> >><UNKNOWN,ID=0x1234> = 1.2.3.4
> >>
> >>In this situation someone will either grab a new version of the
> >>software or an updated data definition to resolve the unknown 
> >>attribute,
> >>which would also define it's type.
> >>
> >>mark
> >>
> >>On Tue, Apr 29, 2003 at 06:25:11PM -0400, MEYER,JEFFREY D 
> >>(HP-Cupertino,ex1) wrote:
> >>
> >>>Mark,
> >>>
> >>>  I'll try to clarify:
> >>>
> >>>  Concept #1:  The information model and the encoding are separate.
> >>>   A well defined mapping determines how an information model is
> >>>   to be encoded for a specific use (e.g. IPFIX).  The encoding
> >>>   may take whatever approach allows both ends to do something
> >>>   useful with the data.  As you point out there is some useful
> >>>   stuff which can be done with the length alone.
> >>>
> >>>  Concept #2:  Derived typing.  By defining a base set of types,
> >>>   which must be understood, derived types can be defined in terms
> >>>   of these base types.  For instance Diameter does this.  This
> >>>   enables a core set of types to be defined and coded to, but
> >>>   also enables richer types later on with an appropriate fallback
> >>>   for systems which aren't aware of the extensions.
> >>>
> >>>  The benefit I see of explicit typing is that instead of seeing
> >>>that a field is of length "4 bytes" something which will be seen
> >>>A LOT.  Is that we can instead differentiate between 
> >>
> >>IPAddr, Integer,
> >>
> >>>unsigned integer, and float.  This seems handy to me.
> >>>
> >>>  Your argument seems centered around the long term potential of
> >>>adding lots of fixed length fields whose base length is not 4,8
> >>>or 16 bytes long, for instance MACAddr (although if we agree on
> >>>going the type route, I'd recommend adding MACAddr and then you
> >>>have 1,2,4,6,8 and 16 bytes covered.  I guess I don't foresee
> >>>a lot more fixed length fields coming down the pipe in the IPFIX 
> >>>domain.  Hence my desire to make it more explicit for the types 
> >>>which do occur all the time and are likely points of extension.  
> >>>WHY?  Because then a tool could at least present these in a 
> >>
> >>meaningful
> >>
> >>>way.  As opposed to incorrectly formatting a float type as an int.
> >>>
> >>>  But hey, either way will work.  If the goal is to limit the
> >>>amount of changes, I'll (grudgingly) go along.  I haven't heard
> >>>any opinions from anyone but you in either direction.  So,
> >>>at this point I don't see a quorom either way...
> >>>
> >>>Regards,
> >>>
> >>>  Jeff Meyer
> >>>
> >>>
> >>>
> >>>>-----Original Message-----
> >>>>From: Mark Fullmer [mailto:[email-masked]]
> >>>>Sent: Tuesday, April 29, 2003 2:53 PM
> >>>>To: MEYER,JEFFREY D (HP-Cupertino,ex1)
> >>>>Cc: [email-masked]
> >>>>Subject: Re: [ipfix] drawing the line between protocol 
> >>>
> >>and data model
> >>
> >>>>docu ment
> >>>>
> >>>>
> >>>>IMHO crippling the protocol to serve the needs of a 
> >>>
> >>documentation tool
> >>
> >>>>is just silly.
> >>>>
> >>>>It would be possible to use both a Type name and a Length 
> >>>
> >>instead of
> >>
> >>>>the existing Length, which would allow a collector to process
> >>>>unknown Types.
> >>>>
> >>>>With the existing v9 protocol a collector can ignore an 
> >>>>unknown attribute,
> >>>>or possibly just pass it on to post processing tools which 
> >>>>might understand
> >>>>it.
> >>>>
> >>>>Your suggested changes no longer allow a collector to 
> >>>
> >>ignore unknown
> >>
> >>>>attributes if the Type is also unknown.  Resorting to encoding all
> >>>>new Types after the initial protocol spec as octet 
> >>>
> >>strings is (IMHO)
> >>
> >>>>not acceptable.
> >>>>
> >>>>I'm still trying to understand what the benefits to explicit 
> >>>>Types sent
> >>>>in templates are.
> >>>>
> >>>>mark
> >>>>
> >>>>On Tue, Apr 29, 2003 at 02:50:02PM -0400, MEYER,JEFFREY D 
> >>>>(HP-Cupertino,ex1) wrote:
> >>>>
> >>>>>Juergen,
> >>>>>
> >>>>>  I agree that the data model should have a well defined type
> >>>>>system.   And that that typing system should reflect the 
> >>>>
> >>>>requirements.
> >>>>
> >>>>>  This is in part why I was hoping that the protocol 
> >>>>
> >>itself would
> >>
> >>>>>be more explicit here.  (See my mail
> >>>>>http://ipfix.doit.wisc.edu/archive/1604.html)
> >>>>>
> >>>>>  There is also a type list in that mail.  However, I 
> >>>>
> >>would suggest
> >>
> >>>>>using the names from XML-Schema part 2 Data types, as this would
> >>>>>facilitate creating a formal/normative definition, versus 
> >>>>
> >>>>reinventing
> >>>>
> >>>>>names for the types.
> >>>>>
> >>>>>  Here are the type names from XML-Schema/IPDR:
> >>>>>
> >>>>>     boolean      - true/false (may be encoded in byte)
> >>>>>     byte
> >>>>>     unsignedByte
> >>>>>     short
> >>>>>     unsignedShort
> >>>>>     int          - 32-bit signed
> >>>>>     long         - 64-bit signed
> >>>>>     unsignedInt
> >>>>>     unsignedLong
> >>>>>     float
> >>>>>     double
> >>>>>     dateTime     - 32-bit integer representing seconds 
> >>>>
> >>since EPOCH
> >>
> >>>>>(1/1/1970 0:00 GMT)
> >>>>>     string       - UTF-8 formatted text
> >>>>>     ipdr:dateTimeMsec - 64-bit integer representing msecs 
> >>>>
> >>>>since EPOCH
> >>>>
> >>>>>     ipdr:ipV4Addr
> >>>>>     ipdr:ipV6Addr
> >>>>>     ipdr:UUID
> >>>>>     hexBinary    - arbitrary sequence of octets
> >>>>>   
> >>>>>  If the protocol itself has a basic "octetString" (or from 
> >>>>
> >>>>XML, hexBinary)
> >>>>
> >>>>>payload,
> >>>>>then this can be used to carry new classes of information 
> >>>>
> >>>>in the future w/o
> >>>>
> >>>>>rejiggering the protocol itself.
> >>>>>
> >>>>>  In addition other "derived" types may be created from the 
> >>>>
> >>>>base types.  For
> >>>>
> >>>>>instance in IPDR, the ipdr:dateTimeMsec is derived from long.
> >>>>>
> >>>>>Regards,
> >>>>>
> >>>>>  Jeff Meyer
> >>>>>
> >>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: Juergen Quittek [mailto:[email-masked]]
> >>>>>>Sent: Monday, April 28, 2003 2:06 PM
> >>>>>>To: [email-masked]
> >>>>>>Subject: [ipfix] drawing the line between protocol 
> >>>>>
> >>and data model
> >>
> >>>>>>document
> >>>>>>
> >>>>>>
> >>>>>>Hi all,
> >>>>>>
> >>>>>>When shaping the planned documents, we have to draw a line
> >>>>>>between the protocol and the data model.
> >>>>>>
> >>>>>>Definitely, the data model document defines all field types,
> >>>>>>but does it also define basic data types, such as '16 bit
> >>>>>>unsigned integer'?
> >>>>>>
> >>>>>>Maybe we should have a set of basic data types defined in
> >>>>>>the protocol document and restrict the data model doc to 
> >>>>>
> >>>>these types?
> >>>>
> >>>>>>This would further structure the data model.
> >>>>>>
> >>>>>>Any thoughts?
> >>>>>>
> >>>>>>    Juergen
> >>>>>>
> >>>>>>--
> >>>>>>Help        mailto:[email-masked] and say "help" 
> >>>>>>in message body
> >>>>>>Unsubscribe mailto:[email-masked] and say
> >>>>>>"unsubscribe ipfix" in message body
> >>>>>>Archive     http://ipfix.doit.wisc.edu/archive/
> >>>>>>
> >>>>>
> >>>>>--
> >>>>>Help        mailto:[email-masked] and say 
> >>>>
> >>>>"help" in message body
> >>>>
> >>>>>Unsubscribe mailto:[email-masked] and say
> >>>>>"unsubscribe ipfix" in message body
> >>>>>Archive     http://ipfix.doit.wisc.edu/archive/
> >>>>
> >>>--
> >>>Help        mailto:[email-masked] and say 
> >>
> >>"help" in message body
> >>
> >>>Unsubscribe mailto:[email-masked] and say
> >>>"unsubscribe ipfix" in message body
> >>>Archive     http://ipfix.doit.wisc.edu/archive/
> >>
> > 
> > --
> > Help        mailto:[email-masked] and say 
> "help" in message body
> > Unsubscribe mailto:[email-masked] and say
> > "unsubscribe ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> > 
> 
> 
> -- 
> Sebastian Zander                         E-mail: 
> [email-masked]
> Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
> Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
> D-10589 Berlin, Germany                  
> www.fokus.fraunhofer.de/usr/sebastian.zander
> 
> 
> 
> 

--
Help        mailto:[email-masked] and say "help" in message body
Unsubscribe mailto:[email-masked] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/

---

From: [email-masked]
Date: Tue, 06 May 2003 11:44:37 -0400

OK. I'm confused. Let me run through an example and people can correct 
me where I'm wrong.

Lets take "Source IP v4 address". Lets also hold off on variable
length fields for now.

All options need some kind of type value to indicate what the value
semantically means. Lets pick 57 == Source IP v4 address. So what we 
need next is info on the value part so we can decode it and it should 
be backward compatible (i.e. older versions can at least skip unknown 
elements.

Option I: Element and Length

	57, 4

Any software understanding element 57 knows it is the Source
IP address and can display/handle it correctly. Any one else would
skip over the value in the flow set.

Option II: Element and data type

Lets say data type 22 == IP V4 Address

	57, 22

Any software understanding element 57 knows it is the Source
IP address. But what if it doesn't know what data type 22 is?
How can I skip over it?

Option III: Element and data type and length

	57, 22, 4

Any software which understands 57 and 22 can handle it correclty.
If it only understands 22, it can ONLY display the value in
some rational way, nothing else can be done. If I don't understand 22
I can at least skip over it by using the length.

Correct away!

Paul

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 06 May 2003 16:35:09 -0700

> I can at least skip over it by using the length.

If  protocol were to do the encoding, does it not depend on the
encoding (regardless of the type) the way a field is intrepreted/displayed
by the collector software ?  Eg. One could encode IP address as a 15 byte
string or a 4 byte unsigned.
On the otherhand if 57 always maps to 22, is there a need to
specify the type to decode this in s/w (agreed that this has to be
specified in the document)?
Is display the only advantage behind defining the data (encoding) type?
What other benefits do we get out of option III?

Ganesh

---

From: Ganesh Sadasivan <[email-masked]>
Date: Tue, 06 May 2003 17:30:34 -0700

I agree to the statement that typing and semantics are
indepenedent of each other and need to be documented.
Also encoding for the types should be in a standard
manner.
The benefits that I understood from the e-mail thread is
a better presentation. Is that alone a  very strong
argument go with this approach in an implementation (.i.e
specify the data types also in the templates)?
I am not sure how much this would be useful if the
semantics is not known.

Thanks
Gansh

---

From: Sebastian Zander <[email-masked]>
Date: Wed, 07 May 2003 02:48:51 +0200

> IP address and can display/handle it correctly. Any one else would

No it can't necessarily handle it correctly because you have not
specified how it is encoded.

> IP address. But what if it doesn't know what data type 22 is?

I don't thing this option is a good idea

> I can at least skip over it by using the length.

In my opinion the ability to display a value correctly even if i do
not know the semantics is quite nice. I can learn the semantics later
without having to store the attribute in its binary form. I even might
be able to learn the semantics just by looking at the value. What's the
disadvantage? templates become bigger but ideally they are not exchanged
too often anyway...

Also everyone who knows 22 knows exactly knows how to encode 57.

In my opinion the later is mandatory for the protocol. So even if the type
is not send to the collector to save bandwidth we must define the encoding.
Instead of defining the encoding for each attribute/field again and again
i propose to define some basic types and then map the attributes to that
basic types.

Having a specification like the following will only cause problems because
some people might not know how to encode ip adresses or might mess up host
and network byte order etc.
 > IP_SRC_ADDR              8       N     Source IP Address
 >                                           IPv4 have N=4
 >                                           IPv6 have N=16

Just my 0.2 cents,

Sebastian

-- 
Sebastian Zander                         E-mail: [email-masked]
Fraunhofer FOKUS / METEOR                Tel: +49-30-3463-7287
Kaiserin-Augusta-Allee 31                Fax: +49-30-3463-8287
D-10589 Berlin, Germany                  www.fokus.fraunhofer.de/usr/sebastian.zander

---

From: Mark Fullmer <[email-masked]>
Date: Wed, 7 May 2003 00:00:08 -0400

<cc list trimmed>

This is a good summary.

After thinking about this for a few days one other alternative would
be to allow the exporter to upload a Attribute -> Type mapping using
the options template mechanism.

Either way, now you have the ability to display (for an unknown Attribute)

<99 UNKNOWN> = 1.2.3.4

When to be really useful you want

<CounterFoo> = 1.2.3.4

And this is the type of information that could be put in a MIB, along with
similar tables such as ifName, ifDescr, ifSpeed which can be just as useful
when post processing flows.

I'm going to stand by my opinion that this doesn't belong in the IPFIX
protocol because it's not something that's needed to glue the exporter
and collector together.

mark