Subject: Re: [Forces-protocol] topic 4c: Common Header

From: Jamal Hadi Salim <[email-masked]>
Date: 02 May 2004 11:58:37 -0400

BTW, there are a few things we seem to be skipping over - all of which
are important to the header:

- Transactions, definition and scope
- Atomicity of transactions
- batching of messages (maybe transactions)

and this last one i am not one 100% sure, but do we need to have
windowing as well for pipelining? or is batching sufficient?
In other words can we have only a single transaction active at a time or
can we have more running in parallel.

Responses to your email below:

On Fri, 2004-04-30 at 22:18, Khosravi, Hormuzd M wrote:
> Here is a summary of our discussion so far on the common header...
> The fields in the common header that there was consensus around are as
> follows,
> 
> IDs: CE ID, FE ID (Used to address a single, group or all CEs/FEs
> respectively)
> 
> Length: 16-32 bits...TBD
> 
> Version/sub-ver: 4 bits
> 
> Sequence No. : 32 bits

Once we answer the question on transactions, this could have slightly
different semantics.

> Length : 16 bits
> 
> Command Type : 8 bits (semantics - TBD)

I am still trying to recover from my zone of last week; the commands
will be limited to those 6 or so you posted earlier?

> Flags: 16 bits, including 1-3 bits for Priority. (The semantics/length
> of Flags and Priority fields is TBD)
> 
> LFB TypeID: TBD, no consensus on whether it is needed or not in the
> header

I think you somehow need to be able to inspect the message and say
clearly who (FE/CE) and what (LFBID) that the message is directed at.
If not for any other reason, this is useful for debugging in the field.
I would like to be able to run a tcpdump session in a box where all
forces messages are mirrored to and debug a race condition for example.
My view would be to have this field to have may be one bit to represent
direction and the rest of the bits to represent they type of LFB being
targetted (set to zero or all 1s if message is not LFB specific)

> 
> ------------------
> Based on our latest discussion, I think we have decided to use Src ID,
> Dst ID fields in the common header instead of CEID, FEID, where a
> SrcID can be either CEID or FEID, same for Dst ID.
> The only other outstanding issue remaining is LFB Type ID. Could team
> members pls post their thoughts on this ??

I think we may have overlooked the other requirements like atomicity;
better catch them now.

I will share the netlink2 experience maybe we can borrow something from
it:

Batching:
1) We had a flag which indicates "more messages for this transaction are
on their way"
2) We had a command (not flag) which said "all messages are done for
this transaction"

A batch will start with one or more of #1 and end with #2.

Atomicity:
We had a speacial flag which indicated that a transaction or parts of it
are atomic.

The sequence number in our case was a strictly per message value.
A more interesting approach would be to break it into a transaction:msg
sequence.

We also had a window of 1. i.e at any point theres only one transaction
outstanding. 

cheers,
jamal

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Sun, 2 May 2004 21:04:50 -0700

Resending this since I didn't see it for a long time..

-----Original Message-----
From: Khosravi, Hormuzd M 
Sent: Sunday, May 02, 2004 8:32 PM
To: [email-masked]'
Cc: [email-masked]'
Subject: RE: [Forces-protocol] topic 4c: Common Header

Hey Jamal,

Glad to see you're back in action ! Also, its good that you brought up
these issues (below), I agree on most points. See my comments below.

-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 

[HK] Yes, I agree these are important and was hoping you will bring this
up :) The Flags field in the header will cover some of these, if needed.
We needed to complete the discussion on the Flags, that we had before
(during Protocol Analysis team).

I am still trying to recover from my zone of last week; the commands
will be limited to those 6 or so you posted earlier? [yes]

[HK] Robert had pointed out that he would like to use this field to
address a message to a group of LFBs in different FEs (multicast based
on LFB Type ID). Do you agree with this ? That might be a more
compelling reason to me for having this field. I don't say that
debugging is not important, but we do have sophisticated tools these
days, which can easily look beyond headers into protocol messages these
days. (I believe even tcpdump can do this quite easily.)

Batching:

Atomicity:

[HK] We had similar flags/approach for batching/atomicity. The one
question that I have is if this is something which will be needed for
all messages or only for Config message ? If you think we need
Batching/Atomicity for all messages then we need these Flags in the
common header, otherwise we can just have it in the Config message. What
do you think ?

Regards
Hormuzd

---

From: [email-masked]
Date: Tue, 4 May 2004 14:14:56 -0400

Hello Robert,
 
Comments are inline.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of ext Robert Haas
Sent: Tuesday, May 04, 2004 1:49 PM
To: Weiming Wang
Cc: [email-masked]
Subject: Re: [Forces-protocol] topic 4c: Common Header

Weiming, and all,

Weiming Wang wrote:

Hi All,

Sorry be off  for a few days owing to a holiday in China.

Firstly,  on the msg direction indicator in the header, I actually originally

have two thoughts for it.  One is to define CE id and FE id in a different

space, that is, e.g., using a bit in the space head (mst significent bit) to

mark the different space, another is to  put the 1 bit flag along with the

transaction ID, that is, by viewing the ID we then know where the msg is

generated. One more advantage for latter approach is it may supply some extra

information for FE or CE to trace the transactions,  and may be of some help to

sth like transaction atomicity?

Frankly, I can't see how adding a flag in the message to indicate its direction (i.e., FE to CE, or CE to FE, and later maybe CE to CE and FE to FE ?) adds further information to help tracing transactions. Please explain. For simplicity I'd rather stick to the usual IP header semantics with a dst field and a src field.

Splitting the ID address space in two equal subspaces (which would be the case if we choose the MSB of the ID to distinguish among CE and FE ID) would be an overkill for CE space. I guesstimate the ratio of FEs vs CEs in an NE to be in the order of 100 to 1000.

Who has a proposal how to best split the address space ?

[<Ramg>]  Instead of splitting in two halves you can reserve a portion of region for CE and portion for FE.   This will make better utlization of address space.
 
Regards
Ramg
  Regards,
-Robert

Seems we begin to discuss how the writing task is to be assigned.  Hornestly, my

team ( I and Ligang) more prefer to take the task of protocol msg and common

header writhing, because we think we may be more fit for sth more technique

instead of beautiful literature oriented, .  Actually protocol msg is a big

task, I just suggest we assign the 6 msgs to two individuals (1st 3, and last 3)

for draft starter writing.  For common header, if Jamal is interested, I would

like to have a close co-work with you for this part.  I may also write the

definition part, for it seems mostly copy work. I hightly suggest  Avri write

the introduction as well as acknowledge and ref, and I also highly suggest Jamal

you write the TML requirement part, because I think you may have a more

understanding of this than us (at least me) anyway.

I also agree to continue our discussion on the remained issue even when we have

parallelly begun the writing.

Sincerely,

Weiming

----- Original Message -----

From:   <mailto:[email-masked]> <[email-masked]>

HI,

On 4 maj 2004, at 01.04, Khosravi, Hormuzd M wrote:

Avri,

Glad to have you back! As I said in my previous reply to Jamal, I am

fine with this (below) as well. I think any special nos. etc which are

needed for addressing can directly be defined by the person writing the

section on Common header and we can all review that, once its done.

Seems a reasonable approach.

Couple of questions...

In your previous email, you had sent a .txt file for the draft. Is

there

also some XML file which we need to use first to write the draft, which

later gets converted to a .txt file ?

Yes, there is a set of XML files, one for each section, and I can break

it down smaller if folks want.  I can send those out, but I was waiting

for the CVS setup so that they could be started there.

And yes, we then put all the xml sections through the xml2rfc tcl

script to get the txt output.

As I said, I am willing to play the role of assembling the doc from the

various xml sections people write, icnluding debugging the xml

formatting as needed.

Unless someone wants to assign me a section to author, I would prefer

to keep myself in a neutral copy-editor role so that I can't be seen as

preferring my words over anyone else's. But if there is something that

someone wants written I can.  I can certainly take care of sections

like abstract, maybe an introduction, references, etc.

Did you want to post any other thoughts that you have on peer-to-peer

model (topic 6), or do you think we already covered this topic in our

protocol messages discussion ?

I think we are actually subsuming it into the other discussions.  In

brief, in pure master-slave model, the FE would only respond when

requested to do so, with the exception of alarms etc..  As we progress

toward a peer-peer model, each of the peers has its role and can

initiate a transaction depending on its own needs.

As time goes on, and FE hierarchy and FE-FE communications become a

reality, then I can envision occasions where the FE would notify the CE

of changes in the FE set that are initiated by the FE based on some

policy or other.  I am also envisioning that there will be cases,

though not 'permitted' where an FE will be directly manipulated by some

process or operator other then the CE, where the FE will notify the CE

of changed state or function.  And while this is not currently an item

for discussion, I can see the FE as being instrumental in the neighbor

discovery process (I tend to divide routing into 3 not 2 sub-processes)

a.

ps. in terms of timing, I will be spending May in Sweden, so my

response times will be skewed by that.

[email-masked]

[email-masked]

  

-- 

Robert Haas

IBM Zurich Research Laboratory

Säumerstrasse 4

CH-8803 Rüschlikon/Switzerland

phone +41-1-724-8698  fax +41-1-724-8578   http://www.zurich.ibm.com/~rha

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 4 May 2004 20:29:02 -0700

Jamal,

-----Original Message-----
> I think we should define it as a message sequence. 
> 
> [HK] Sure, I thought we already decided to call it Sequence no. Not
sure
> why Weiming used a different name ? BTW, do you agree with having the
> same Sequence no in Req/Resp to match them ? I think you do, but just
> wanted to confirm.

Yes i do. [Thanks]

> 
> Next big question (I asked this earlier) how many PL level messages
can
> we have in flight? If more than one, do we need a windowing syntax in
> the common header?
> 
> [HK] We can have more than one PL msg in flight, but I don't think we
> need any windowing syntax in the common header. This will be provided
by
> the TML layer if needed. We will just use the Sequence no to match the
> request/response msgs.

I thinks its harder than that. 
Pipeling is known to improve throughput; basically for a given b/width
and RTT, you should be able to send in the range of BW*RTT bytes without
getting an ACK that the first message you sent. Windowing just keeps
track of how many such messages you have delivered to the network.
Imagine that given a 10Gige pipe, you can only using 100Mbps. 
Now i agree that adding windowing complicates things, but theres no way
you can pass the pipelining responsibility to the TML. The TML will send
as much data as you give it.
Batching may be sufficient - dont know.

[HK] Yes when you send multiple msgs in a single transaction, batching
should be sufficient. As long as you get responses/acks back for all
your msgs (atleast the important control msgs), you don't need
windowing...also assuming the TML will provide strict reliability for
control msgs atleast.

---

From: [email-masked]
Date: Wed, 5 May 2004 04:47:30 -0400

Hello Hormuzd,Jamal and others,

Comments are inline.

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of ext Khosravi,
> Hormuzd M
> Sent: Tuesday, May 04, 2004 10:59 PM
> To: [email-masked]
> Cc: Weiming Wang; [email-masked]
> > [HK] I agree. The Transaction ID or Sequence No. should be 
> the same in
> > both Request and Response msgs e.g. Config/Config Resp to 
> denote that
> > they are part of the same transaction. That was the reason 
> for having
> a
> > Sequence no to begin with.
> 
> If you call that 32 bit value a transaction # it may be a little
> misleading; example a transaction may take several messages 
> not just one
> (Consider a two phase commit transaction where you send 
> several messages
> followed by a commit to complete the transaction). 

<RamG>

One more id requirement....

If we use command bundling (mulitple queries or configs in one) then CE needs to match not
only the Seqno but also each command response against some other identifier???.

Each command should be self contained with some unique id to match the response.

<RamG>

> [HK] Sure, I thought we already decided to call it Sequence 
> no. Not sure
> Next big question (I asked this earlier) how many PL level 
> messages can
> need any windowing syntax in the common header. This will be 
> provided by
> Thanks

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 5 May 2004 19:30:17 -0700

Weiming, All

>
> > There will also be times when i wont be interested in
> > the response (and this is not just in the case of events).
> >
> > [HK] Most of the Control Plane Apps such as routing/switching
protocols
> > that I have worked with need responses for any commands such as
Config
> > or Query msgs. Could you give some examples of what msgs you don't
need
> > responses for ? Also, how do we satisfy the timeliness requirement
which
> > without protocol level responses or acks ?
>
> I dont wanna speculate about others - i think the possibilty is
> certainly there, but a simple example is a PL heartbeat scheme where i
> can occasionaly request for an ACK. We have used this and found it
> useful.
>
[Weiming]I think the scheme as 'Pls let me know when you encounter
difficulties,
otherwise pls don't bother' should be very useful for ForCES
architecture, where
we can think of the heavy burden of FE resources and lack of b/w
resouces for
CE-FE interconnection. E.g., (as I mentioned in my earlier post), to add
huge
routes, I would prefer to let FE send back responses when it finds some
difficulty to do that. App level response is different from low level
responses.
Still take add huge routes as e.g., App level may only get a response by
low
level (not necessariy from FE) to tell that the huge routes (as one
task) has
been implemented sucessfully (or stoped owing to an error). It's not
always the
case the app. level needs the huge responses.

[HK] I just responded to this in context to Jamal's email. I am fine
with an ACK flag in the header, if you guys find this useful. It hasn't
been our experience but that's fine.

Hope that helps,
Hormuzd

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Wed, 5 May 2004 19:37:20 -0700

-----Original Message-----

> [HK] So just to clarify, you would like to have an ACK flag in the
> header to indicate whether you need a Response/Ack for a particular
msg
> or not...correct ? If that is the case, then I agree that this is not
> very costly, just 1 bit in the header...I can live with this if you
> think this is something your customers want. I will always have this
> turned on in my code, cause all our customers want responses for all
the
> control msgs. Let me know if this one is resolved, then ?

I think the ACK flag would be sufficient to close this. 

[HK] Consider this closed from my side (assuming you agree with the
description for the Ack flag above)

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 6 May 2004 19:17:20 -0700

Hi Weiming,

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Weiming Wang

I also think a ACK flag to give customers more flexibility will be
benificial
for our protocol being futher more widely used.

One thing now remained is we may now need to decide if we put ACK as a
specific
msg and included in the msgs, or just let others such as common header
section
do it.

[HK] I don't think we need an ACK msg, we already have Response messages
for all the messages that we need a response/ack for. As I had explained
before, you can think of Response as being a superset of ACK, i.e. it
does convey ack and them some more stuff. Also, look at my email on Msg
Types for more clarification.

Hope this helps,

Thanks
Hormuzd

---

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Thu, 6 May 2004 19:26:56 -0700

Jamal,

-----Original Message-----
> 2. Jamal, I have a feeling that you seem very careful not to put too
many flags
> in msg bodies. I wonder if there are some technical disadvantage or
barrier
> behind to do like this according to your implementation experience?
Hornestly,
> I'm short of experiences on this.

I am not picky. I think flags in the main header are for extending the
message. A lot of of protocols i know keep them there - thats where my
bias is coming from. I have no problem with putting flags in the
submsg;however, since you will have flags in the main header, why not
use them? Note we can also have flags in the TLVs T part. I think
diameter does this.

[HK] For some cases in our case, such as 2-phase commit, I think it
makes sense to have flags only in the message body or the TLV that we
define for that msg rather than the msg header. But I don't want to be
too picky about this either. Let us know if you're fine with this
(having flags in msg body) ? 

BTW, we don't need to have 16 bits for Flags in the common header if we
don't need them. We can just do with 4-5 bits for example, if that is
sufficient.

Thanks
Hormuzd