Subject: RE: [Forces-protocol] issue 1: packet encoding

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sat, 6 Mar 2004 14:37:57 -0800

Jamal,

Before I give my comments, I would like to say that I appreciate the
fact that you did not copy/paste the netlink header from your draft over
here! That is a very positive sign to me, lets try to stick to the best
technical solution rather than pushing our own protocol implementations.
That's the best way to make progress on this merger!!
 
Here are my views on this issue, particularly the common header:

Command: Agree this is needed. We think 8-12 bits will be more than
sufficient for this depending on whether it is subdivided. We have
sub-divided this field into Class/type...this is an OO approach, seemed
pretty neat. However, I am personally fine with having a single field
for this, no problem with that.

Length: Agree needed and agree with size.

xIDs: Agree this is needed. More detailed comments on length, semantics
in my previous email.

Sequence No: Agree needed and agree with size.

Flags/Priority: We need 2-3 bits for priority in every message. I don't
think we need any other flags in all messages.

Typeid: This is one field I don't think is needed in all messages. I am
not sure why this is needed at all cause, we will be using the Model and
LFB IDs to address processing functional blocks on the FEs.

Missing field: Version (2-4 bits): This is present in all protocols, and
is generally included in most IETF protocols. Its usually the 1st field.

General comment: Lets try to be stingy on the size of the header and its
fields since this will be included in all ForCES messages.

Jamal's text.......
The Forces Message Header: 

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                     0               1               2             3
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |             Command         |               Length            |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                          Source xID                           |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        Destination xID                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        Sequence Number                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |             flags           |             typeid              |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

XML encoding:
XML encoding may be useful in capabilities definition
(i.e slow path) but not in the configuration of data path tables.
This is because of its nature to require higher bandwith and
CPU computational needs.

HK Comment: I think we should stick with a single encoding type, namely
TLV. This is also what some of the model team folks have mentioned
before for the protocol. Is there a reason why capabilities cannot be
expressed using TLVs?

Thanks
Hormuzd

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Jamal Hadi Salim
Sent: Friday, March 05, 2004 2:00 PM
To: [email-masked]
Subject: [Forces-protocol] issue 1: packet encoding

Attached is text for stimulating discussion on issue 1.

cheers,
jamal

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 06 Mar 2004 22:10:11 -0500

> That's the best way to make progress on this merger!!

We actually have learnt a few things while implementing ;->
The thinking right now is to learn from mistakes (and from good things
of course).

> for this, no problem with that.

I think we could resolve this later; i am not seeing a lot of commands.
On the outset i can only see ADD/DEL/GET

> think we need any other flags in all messages.

How do you propose we do things like 2 phase commits; atomic
transactions etc?
Maybe what you called "class" in the command is a flag then?
Also in regards to priority; do you see this being used all the time?
and would an underlying transport priority be insufficient
(example diffserv, 802.1p etc)?

> LFB IDs to address processing functional blocks on the FEs.

I am not sure i see this. If you could explain more mayeb it would help.
Like i said the only value is in being able to look at the outer header
and recognizing for example the target where the message is being sent
to (example LFB class). If not for anything else, debugging (via
sniffing) is a good reason - this way i could dump the rest of the
message by recognizing what the TLVs mean.

> is generally included in most IETF protocols. Its usually the 1st field.

Agreed. 4 bits is what i have seen normally.

> fields since this will be included in all ForCES messages.

Agreed - the main thing is that if something needs to be used in all
messages then it should go on this header (since its cheaper than a
TLV).

cheers,
jamal

---

From: [email-masked]
Date: Mon, 8 Mar 2004 17:17:37 +0900

Hi,

I have just caught up with the messages on this list.

One general comment, I think the way you are all working through the 
issues is great - fills me with hope.

Specifically to the header, and I don't know how this fits into the 
issue numbering scheme:

- I think you should consider adding an error indication to the header. 
  It can either be along the lines included in GRMP or otherwise.  I 
have found having this in the header a very useful thing.

- I agree that you need to include version and preferably subversion 
number.

- I also think that 8 bits is enough for message type.

best regards
a.

(avoiding commenting on topics not yet under discussion - i think)

---

From: Jamal Hadi Salim <[email-masked]>
Date: 08 Mar 2004 08:36:27 -0500

Hi Avri,

> have found having this in the header a very useful thing.

Good point.
This is in responses only i take it? 
We implemented a NACK response when things went wrong as opposed
to an ACK. Not sure if that is what you are refering to.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Mon, 8 Mar 2004 21:55:25 +0800

Hi Avri, Jamal, Hormuzd, and all,

Please allow me to begin to address this issue by trying to propose some basic rules that we 
(GRMP team) think when we design the protocol header as well as the body:
1. As Jamal mentioned, a very important rule is to put things that surely every protocol
 messge need to the protocol header. 
2. When the same goal can be made, a saving bits scheme should be absolutely selected. 
This is expecially for ForCES protocol. Just imaging hundreds of EFs working  together 
and exchanging things may only use a quite limited transport resources, we have no
reason to spend one extra bit in the header.

Then some comments:

1. The "command" field
This is really a field that confuses me. The confusion is:
1) If it is a field only with command "Add/Del/Get", I suppose we can just obsolete it. 
The command can be surely be expressed in the followed body by using TLVs.
2) If we keep it, I'm afraid it's meaning should be far from these. For example, how 
an event report can be represented, and how a command bundle is represented then, 
and many others? 
We just think to have a field  with insufficient functions really makes confusions.

As a result, I suppose two solution schemes for this problem:
Scheme 1: Define this as "message type", to have the message body to express the 
operation not just based on TLV format. This means things like "Flags" "operating fields" 
are explicitly expressed according to the "message type" operation requirements. Remember
 that by using this scheme, we have already taken the whole ForCES message as one TLV.
Scheme 2: Not define anything in the header on the "command" or "message type", leaving 
things defined in the followed body, while the body is expressed by a set of TLVs, each TLV 
representing a type of operation. 
The advantage for Scheme1 is it is very clear in semantics and saves bits quite a lot. 
The disadvantage is it has to define a specific message type for command bundle. 
The advantage for Scheme 2 is it is easy to implement command bundle, but the disadvantage
 is it costs more bits and makes protocol header less to do.

2. The Checksum field and other error control mechanism
One essential function a protocol header takes is for a reliable information exchange, therefore, 
the common error control mechanism should be included in the header instead in the body. 

It seems no argue that a checksum should be provided as an option. Only difference between 
the candidate protocols is how it should be put. GRMP puts it at the protocol message end, 
Netlink2 has it as a TLV, I suppose FACT also intends the same. I'm thinking the purpose to 
have it as a TLV is mainly trying to reach a uniform TLV format. What we argue is to put 
checksum in a TLV may get more loss for the checking function, because it then will be in a risk 
that the TLV structure itself may be erred, which may result in the invalidation of the checksum. 
While in former case, only the protocol message length change will make checksum invalid. 
Actually checksum in many protocols are directly put in the message surface instead of being 
incapsulated deeply, like UDP. 

GRMP reuses the idea "result" and "code" from GSMP, I'm sure this error control function should be 
included in the protocol. As the protocol header takes the tasks for a reliable information exchange, I
strongly suggest it is put in the header. Could other candidate team show your opinions or solutions to 
this?

I'm afraid is already too long. I'l put other comments in other emails.

Cheers,
Weiming

----- Original Message ----- 
From: <[email-masked]>
To: <[email-masked]>
Sent: Monday, March 08, 2004 4:17 PM
Subject: Re: [Forces-protocol] issue 1: packet encoding

> > The Forces Message Header:
¢·²šè¶‡(–f¢–)à–+-ŠÜzÊk¢Ú¢Xžµú+‚m¦ÏÿÃ5‰ë_¢¸?™¨¥™©ÿ–+-Šwèýú+që)®‹hr‰

---

From: Alex Audu <[email-masked]>
Date: Mon, 08 Mar 2004 11:32:25 -0600

I don't know what the flags could be used for. The TypeID could be used for
siganling the type of data encoding explicitly,  (i.e. XML, ASN.1 etc). 

My preference will be the following.

     0                   1                   2                   3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
                     0               1               2             3
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |vers   |eType  | prio| reservd |  msgClass     |   msgType     |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                       message length                          |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |          Source xID           | Destination xID               |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |                        Sequence Number                        |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

1) version # indicates the version of the protocol.
2) the type of encoding comes next.
3) the priority of the message comes next.
4) the command has been divided into message class and type to ease 
management and handling.
5) the source and destination are now both 16 bits each. This allows up 
to 64K FEs. That is cool!
6) the length is now 32 bits long. This will accommodate flow 
measurement messages which
    tend to be huge,  from FE to CE.  It will also accommodate large 
table dumps from CE to
    FE.
7) then comes the sequence number.

Any comments?

Regards,
Alex.

---

From: Alex Audu <[email-masked]>
Date: Mon, 08 Mar 2004 13:22:27 -0600

Hello Avri,

I think error indication is actually another type of data to be sent 
between FE and CE.
So we should carry it in the message itself (as an error message type).

Regards,
Alex.

[email-masked] wrote:

> - I think you should consider adding an error indication to the 
> header.  It can either be along the lines included in GRMP or 
> otherwise.  I have found having this in the header a very useful thing.
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: 08 Mar 2004 14:43:28 -0500

Weimimg,
Please make sure you wrap your emails around 80 characters.

On Mon, 2004-03-08 at 08:55, Wang,Weiming wrote:
[..]

> The command can be surely be expressed in the followed body by using TLVs.

Every message will have a command. For that reason it belongs to the
main header.

> an event report can be represented, and how a command bundle is represented 

I will explain how we implemented events; note, this is different from
the current command structure but shoudl give you a view:
- An "add route" from CE->FE means to configure something on one or more
FEs that are addressed.
- An "add route" from FE->CE is an event notifying that a route was
added.

An alternative is to reserve one bit for saying its an event or 
configuration then you dont have to intepret directions.

>  is it costs more bits and makes protocol header less to do.

I prefer to have the headers with flags; flags were usable on every
message.

> invalidation of the checksum.

Netlink2 has the concept of several shades of reliability.
Some messages dont need to be validated and therefore dont
need a checksum. This is also the path we are following now,
for that reason checksum should stay an optional field.

> show your opinions or solutions to 

We had the concept of an ACK which returns results.
Depending on the result, this could be considered a NACK.
The ACK is requested per message using flagd and so sometimes may never
be sent. Again this has to do with levels of reliability.
In other words while the result is useful in the header,
it is only useful when the result is requested.
This is described in the section " The ACK Netlink2 Message".
I think this is similar to the way you do it in GRMP.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 9 Mar 2004 21:17:27 +0800

Hi Jamal,

Could you just add some tag to indicate your reply? Or else, I have to
help to add it so that others can see your reply.
Some comments as below.

Thank you.

Weiming

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>
To: "Wang,Weiming" <[email-masked]>

> > 1) If it is a field only with command "Add/Del/Get", I suppose we
can just
> > obsolete it. The command can be surely be expressed in the followed
body
> > by using TLVs.

>[Jamal Re:
> main header.]
[Wang Re: I would say it is also possible to have a message contain more
than
one command. But I don't want push this too hard so that we may be
quicker
to reach an agreement.]
>
> > 2) If we keep it, I'm afraid its meaning should be far from these.
For
> > example, how an event report can be represented, and how a command
bundle
> > is represented then, and many others?

>[Jamal Re:
> configuration then you dont have to intepret directions.]

[Wang Re:
Why not just let us use an "Event report message" instead of using such
a "add route" along with some bit to say it is an event, while leaving
the "command" field from being fully used.]

> > We just think to have a field  with insufficient functions really
makes
> > confusions.
> > Scheme 1: Define this as "message type", to have the message body to
express
> > the operation not just based on TLV format. This means things like
"Flags"
> > "operating fields" are explicitly expressed according to the
"message type"
> > operation requirements. Remember that by using this scheme, we have
already
> > taken the whole ForCES message as one TLV.
> >
> > Scheme 2: Not define anything in the header on the "command" or
> > "message type", leaving things defined in the followed body, while
the body
> > is expressed by a set of TLVs, each TLV representing a type of
operation.
> >
> > The advantage for Scheme1 is it is very clear in semantics and saves
bits
> > quite a lot.
> > The advantage for Scheme 2 is it is easy to implement command
bundle,
> > but the disadvantage is it costs more bits and makes protocol header
less
> > to do.

>{Jamal Re:
> message. ]

[Wang Re: not very sure how your reply maps what I said.]

> > One essential function a protocol header takes is for a reliable
> > information exchange, therefore, the common error control mechanism
should
> > be included in the header instead in the body.
> >
> > It seems no argue that a checksum should be provided as an option.
> > Only difference between the candidate protocols is how it should be
put.
> > GRMP puts it at the protocol message end, Netlink2 has it as a TLV,
I
> > suppose FACT also intends the same. I'm thinking the purpose to have
it as
> > a TLV is mainly trying to reach a uniform TLV format. What we argue
is to
> > put checksum in a TLV may get more loss for the checking function,
because
> > it then will be in a risk that the TLV structure itself may be
erred,
> > which may result in the invalidation of the checksum.
>
>[Jamal Re:

[Wang Re: checksum in GRMP is also optional, my opinion is it has better
performance to optionally put it at the message end, just the same as
UDP.]

> ]
>
> > GRMP reuses the idea "result" and "code" from GSMP, I'm sure this
> > error control function should be included in the protocol. As the
protocol
> > header takes the tasks for a reliable information exchange, I
> > strongly suggest it is put in the header. Could other candidate team
> > show your opinions or solutions to this?
>
> [Jamal Re:
> ]

---

From: "Wang,Weiming" <[email-masked]>
Date: Tue, 9 Mar 2004 21:40:50 +0800

Hi Jamal,

Could you just add some tag to indicate your reply? Or else, I have to help to
add it so that others can see your reply. Some comments as below.

Thank you.

Weiming

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>
To: "Wang,Weiming" <[email-masked]>

> > 1) If it is a field only with command "Add/Del/Get", I suppose we can just
> > obsolete it. The command can be surely be expressed in the followed body
> > by using TLVs.

> Every message will have a command. For that reason it belongs to the

[Wang Re: I would say it is also possible to have a message contain more than
one command. But I don't want push this too hard so that we may be quicker to
reach an agreement.]

>
> > 2) If we keep it, I'm afraid its meaning should be far from these. For
> > example, how an event report can be represented, and how a command bundle

> configuration then you dont have to intepret directions.]

[Wang Re:
Why not just let us use an "Event report message" instead of using such a "add
route" along with some bit to say it is an event, while leaving the "command"
field from being fully used.]

> > Scheme 1: Define this as "message type", to have the message body to express
> > the operation not just based on TLV format. This means things like "Flags"
> > "operating fields" are explicitly expressed according to the "message type"
> > operation requirements. Remember that by using this scheme, we have already
> > "message type", leaving things defined in the followed body, while the body
> > is expressed by a set of TLVs, each TLV representing a type of operation.
> >
> > The advantage for Scheme 2 is it is easy to implement command bundle,
> > but the disadvantage is it costs more bits and makes protocol header less
> > to do.

> > information exchange, therefore, the common error control mechanism should
> > Only difference between the candidate protocols is how it should be put.
> > GRMP puts it at the protocol message end, Netlink2 has it as a TLV, I
> > suppose FACT also intends the same. I'm thinking the purpose to have it as
> > a TLV is mainly trying to reach a uniform TLV format. What we argue is to
> > put checksum in a TLV may get more loss for the checking function, because
> > it then will be in a risk that the TLV structure itself may be erred,
> for that reason checksum should stay an optional field.]

[Wang Re: checksum in GRMP is also optional, my opinion is it has better
performance to optionally put it at the message end, just the same as UDP.]

> > error control function should be included in the protocol. As the protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 09 Mar 2004 09:27:43 -0500

Hi Weiming,

> one command. But I don't want push this too hard so that we may be quicker to

Now i see your reasoning. 
I may be biased by the way we did things in netlink2; but i am willing
to consider things differently if it makes sense. In netlink2 we batched
several messages in one TCP or UDP PDU. The overhead of this is
much smaller than having a command as a TLV. Essentially a 8 bit command
will now take 64 bits space (8 bytes).
I think we agree that there will always be a command on each message,
correct? Given that scenario, i think that for a single command
putting the command in the TLV space instead of header is more waste
of bits. The breakeven point for UDP is around 5 commands batched
together. Around 7 for TCP and lower for running directly over ethernet.
So we may have to weight out the options and come to some conclusion.
You can respond to this point but lets consider it closed. 

> field from being fully used.]

Theres a few ways to skin this cat (with apologies to people who own
cats). We can discuss later.
I also think this is resolvable.
Lets close it.

> [Wang Re: not very sure how your reply maps what I said.]

I meant that flags would further refine the headers. I dont wanna go
into a lot of details because i think this is further resolvable.

> performance to optionally put it at the message end, just the same as UDP.]

Nod. Same with authentication headers etc.

Weiming, Please keep the momentum going so we can close these
issues. 

Shall we close issue 1? I think any leftovers are resolvable.

Hormuzd, are you working on issue 2's text? If you have it please post
it. This way we have two topics in the pipeline all the time.
Somebody should volunteer for writing up topic 3; the timeout is 24
hours, if no volunteers, i will write it up.

cheers,
jamal

---

From: [email-masked]
Date: Tue, 9 Mar 2004 13:39:02 +0900

> So we should carry it in the message itself (as an error message type).

I think that could work.  Though it might still be helpful to mark as 
message as being/containing an error message in the header.

If one of the specific message types that can be sent is error, then it 
could work, but then you would be restricted to error payload in a 
message.  If there was a flag  (and agree with the comment about not 
shortchanging the header in terms of flags.) indicating the message 
contained error information that would be helpful too.

I have a general question about headers, is there any feeling in this 
group that a common header could not be arrived at.  If not then we 
could move on to other issues.  The reason I ask is that one can spend 
a whole lot of time fine tuning a header to be just right.  If there 
are as of yet unresolved issues on the header, can anyone list them?

a.