Subject: RE: [Forces-protocol] topic 4c: Common Header

From: "Khosravi, Hormuzd M" <[email-masked]>
Date: Tue, 4 May 2004 20:05:03 -0700

Jamal,
-----Original Message-----
From: Jamal Hadi Salim [mailto:[email-masked]] 
> 
> [HK] Let me try to clarify this question about ACK vs Response. I
> thought we had already discussed and closed on this before (for common
> header in Protocol Analysis team), but may be I am wrong. Anyways we
> need an Ack or Response for almost all messages (except control packet
> Redirect msgs & Events/HBs) to meet the timeliness requirement
specified
> in the Requirements rfc. So we don't really need an ACK Flag in the
> header as I see it. The difference between an ACK msg and Response msg
> in my mind is that, an ACK only provides an acknowledgement from the
> receiver, that it received the message that was sent to it. A Response
> msg provides bit more information e.g. if a Config was sent by the CE
to
> FE, the Config Resp msg says whether Config was successful or not on
the
> FE, if partially successful then which config parameters failed.
Another
> example would be a Query Resp which would give the results of the
Query
> such as some FE or LFB attributes, FE Caps, etc. So to answer your
> question Jamal, Yes I think we need both kinds of Replies and the
> Response messages will address both of these replies. Do you agree ?

I will have to wait and see what the response message looks like.
To me theres no ambiguity in the desire to have responses when things go
wrong or right. 
[Agreed]

There will also be times when i wont be interested in
the response (and this is not just in the case of events).  

[HK] Most of the Control Plane Apps such as routing/switching protocols
that I have worked with need responses for any commands such as Config
or Query msgs. Could you give some examples of what msgs you don't need
responses for ? Also, how do we satisfy the timeliness requirement which
without protocol level responses or acks ?

Thanks
Hormuzd

_______________________________________________
Forces-protocol mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/forces-protocol

---

From: Jamal Hadi Salim <[email-masked]>
Date: 04 May 2004 23:14:42 -0400

> without protocol level responses or acks ?

I dont wanna speculate about others - i think the possibilty is
certainly there, but a simple example is a PL heartbeat scheme where i
can occasionaly request for an ACK. We have used this and found it
useful.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 5 May 2004 13:32:08 +0800

----- Original Message -----
From: "Jamal Hadi Salim" <[email-masked]>
To: "Khosravi, Hormuzd M" <[email-masked]>
Cc: "Weiming Wang" <[email-masked]>; <[email-masked]>
Sent: Wednesday, May 05, 2004 11:14 AM
Subject: RE: [Forces-protocol] topic 4c: Common Header

> can occasionaly request for an ACK. We have used this and found it
[Weiming]I think the scheme as 'Pls let me know when you encounter difficulties,
otherwise pls don't bother' should be very useful for ForCES architecture, where
we can think of the heavy burden of FE resources and lack of b/w resouces for
CE-FE interconnection. E.g., (as I mentioned in my earlier post), to add huge
routes, I would prefer to let FE send back responses when it finds some
difficulty to do that. App level response is different from low level responses.
Still take add huge routes as e.g., App level may only get a response by low
level (not necessariy from FE) to tell that the huge routes (as one task) has
been implemented sucessfully (or stoped owing to an error). It's not always the
case the app. level needs the huge responses.

---

From: Robert Haas <[email-masked]>
Date: Wed, 05 May 2004 10:50:54 +0200

Weiming,
I think the approach of  'Pls let me know when you encounter 
difficulties, otherwise pls don't bother' is indeed useful. Let me 
elaborate a bit on this. Sorry for the rather long message. I'll devote 
another note to windowing.

 Batching is used to group commands together. Each command possibly has 
its own sequence number (or transaction number), or the group has its 
own number with subnumbers for each message (this remains to be 
decided.) The main use of batching is to perform atomicity: all commands 
in the group will be performed at the same time or none will be 
performed. This is where 2-phase commit kicks in: all commands are first 
transmitted to the FE, and unless the FE reports an error, the CE can 
ultimately issue a commit command so all these commands are executed in 
an atomic operation at the FE.

We investigated the idea of packing multiple commands in the same ForCES 
message (it saves quite some headers and the associated processing). As 
a motivation, think of adding 100'000 routes on an FE: should we use 
100'000 ForCES messages, or only 1000 (say 15 bytes per route update 
command to be generous, with a 1500 bytes MTU size, or even less if we 
have Jumbo frames) ?

Note that having many commands in the same ForCES message results in 
slighlty more complex error/ack reporting. Take this example: two 
commands A and B in the same ForCES message issued by a CE. FE executes 
both of  them, but only A succeeds. It should report this somehow to the 
CE. A possible solution is to return the original message trimmed down 
such that it contains exactly two responses, a positive and a negative one.

I think we should focus our effort on defining the format and state 
machine of the ForCES protocol as seen on the wire (and define a few 
encapsulation methods for ForCES over UDP, TCP, etc, with specific 
reserved port numbers). So I don't think we need to worry about the way 
the Apps get notifications from the PL and so on, and if a 
single/multiple signal/s is/are used to confirm that a batch of messages 
has been successfully executed. This will remain vendor specific. Do you 
have different views ?

Cheers,
-Robert

-- 
Robert Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-1-724-8698  fax +41-1-724-8578  http://www.zurich.ibm.com/~rha

---

From: Robert Haas <[email-masked]>
Date: Wed, 05 May 2004 17:04:13 +0200

All,

Let me try to briefly tackle the windowing issue now:

First, we should definitely not shoot for a TCP-like flow-control 
solution at the PL level.

Instead, we could define a high-priority/control message that an FE can 
send in case of overruns. Overruns occur when the FE is too slow at 
processing the incoming CE messages, and its receive buffers are full. 
The reaction of the CE to the overrun message can be left vendor 
specific. It the TML is TCP, then the TCP flow-control will 
automatically take care of this and overruns will not be generated by 
the FE. Other TMLs might not provide such capability, hence the need for 
this (extremely simple) windowing scheme at the PL level using overrun 
messages.

Let me know what other approaches you would consider. The approach above 
fits in the 'Pls let me know when you encounter difficulties, otherwise 
pls don't bother' category Weiming already suggested.

Regards,
-Robert

---

From: "Weiming Wang" <[email-masked]>
Date: Thu, 6 May 2004 13:34:51 +0800

Robert, Jamal, and all,

----- Original Message ----- 
  From: Robert Haas 
  To: Wang,Weiming 
  Cc: [email-masked] 
  Sent: Wednesday, May 05, 2004 4:50 PM
  Subject: Re: [Forces-protocol] topic 4c: Common Header: batching

  Weiming,
  I think the approach of  'Pls let me know when you encounter difficulties, otherwise pls don't bother' is indeed useful. Let me elaborate a bit on this. Sorry for the rather long message. I'll devote another note to windowing.

   Batching is used to group commands together. Each command possibly has its own sequence number (or transaction number), 
  [Weiming]No problem with the name sequence number, I also like it.
  or the group has its own number with subnumbers for each message (this remains to be decided.) The main use of batching is to perform atomicity: 
  [Weiming]I agree this, though I was a little confused before with the notions 'batching, atomicity, 2-phase commit, all or nothing'. Can we understand that they are all targeting the same thing: all or nothing? Or we have other purpose for batching such as just for packetizing?

  all commands in the group will be performed at the same time or none will be performed. This is where 2-phase commit kicks in: all commands are first transmitted to the FE, and unless the FE reports an error, the CE can ultimately issue a commit command so all these commands are executed in an atomic operation at the FE.
  [Weiming]Yes.
  We investigated the idea of packing multiple commands in the same ForCES message (it saves quite some headers and the associated processing
  [Weiming]Here, I think we need to evaluate if the headers are worth there or not. My original idea was it was not worth it (during GRMP time), but now I change some. Just check the possible header fields, we may find that most of them are worth there, and the reward is great, allowing per msg ACK, and with much less complexity for protocol state trace.

  ). As a motivation, think of adding 100'000 routes on an FE: should we use 100'000 ForCES messages, or only 1000 (say 15 bytes per route update command to be generous, with a 1500 bytes MTU size, or even less if we have Jumbo frames) ? 

  Note that having many commands in the same ForCES message results in slighlty more complex error/ack reporting. Take this example: two commands A and B in the same ForCES message issued by a CE. FE executes both of  them, but only A succeeds. It should report this somehow to the CE. A possible solution is to return the original message trimmed down such that it contains exactly two responses, a positive and a negative one.
  [Weiming]Yes, I think you are also trying to see which is better among the following two possible batching schemes:
  1. Batched msgs in one packet or even keeping the msgs separately as its origin, while with some flags such as Atomicity and/or Done Flag to indicate the begin and the end.
  2. Many commands in one msg as you mentioned above. 
  I a little more incline to #1 scheme, as the reason I mentioned above. But there is a remained issue for successful atomicity, that is, it does not support multipul transactions if we use different sequece number for different msgs in a batch, and if we use same sequence number for all the msgs, then it may result in ambiguity for msg meaning and hard for ACK. Therefore, I try to propose to use a sub msg number to see if it is feasible for the purpose. Pls let me try to summarize sth we'v already got (Netlink2?) along with the batch msg # thought as follows:

  Batching msg as:

  Msg1:Msg2:...MsgN (either one packet or separated packets)

  In headers of all the msgs,

  Sequence Numbers: all the same, that means a batch is treated as a transaction.
  ATM flag: mark the end of batching (e.g.,0 for MsgN, and also a signal to excute the batch, 1 for others, and also as a signal to reserve it)
  ACK flag: indicate if this msg need an ACK
  BatchMsgNum: mark the batehed msg sequence number, 0 for Msg1, ... (N-1) for MsgN. Any ACK should include BatchMsgNum as well as others.

  Any of the Msg in the batch still can be fragmented if it exceeds the MTU, a FRG flag is used to treat it, with the thinking that the packets will not be reordered.
  Rob: can you try to present a summary regarding the #2 scheme, so that we can carefully compare and evaluate it?

  I think we should focus our effort on defining the format and state machine of the ForCES protocol as seen on the wire 
  [Weiming] Yes, agreed. The state transfer diagram is important.

  (and define a few encapsulation methods for ForCES over UDP, TCP, etc, with specific reserved port numbers
  [Weiming]I'm not sure if TML will do this or not.
  ). So I don't think we need to worry about the way the Apps get notifications from the PL and so on, and if a single/multiple signal/s is/are used to confirm that a batch of messages has been successfully executed. This will remain vendor specific. Do you have different views ?
  [Weiming]Agreed.

  Cheers,
  -Robert

  Cheers,
  Weiming

---

From: "Weiming Wang" <[email-masked]>
Date: Thu, 6 May 2004 13:43:37 +0800

Hi Robert,
  ----- Original Message ----- 
  From: Robert Haas 
  To: Wang,Weiming 
  Cc: [email-masked] 
  Sent: Wednesday, May 05, 2004 11:04 PM
  Subject: Re: [Forces-protocol] topic 4c: Common Header: windowing

  All,

  First, we should definitely not shoot for a TCP-like flow-control solution at the PL level.

  Instead, we could define a high-priority/control message that an FE can send in case of overruns. 
  [Weming]Could you explain a little more on this. Sorry, I just can not catch up with it well. Is the priority the same as that we defined in the header?

  Overruns occur when the FE is too slow at processing the incoming CE messages, and its receive buffers are full. The reaction of the CE to the overrun message can be left vendor specific. It the TML is TCP, then the TCP flow-control will automatically take care of this and overruns will not be generated by the FE. Other TMLs might not provide such capability, hence the need for this (extremely simple) windowing scheme at the PL level using overrun messages.
  [Weiming] Is it a specific msg walking between CE and FE to manage the overruns?

  Let me know what other approaches you would consider. The approach above fits in the 'Pls let me know when you encounter difficulties, otherwise pls don't bother' category Weiming already suggested.

  Regards,
  -Robert

  Cheers,
  Weiming

---

From: Robert Haas <[email-masked]>
Date: Thu, 06 May 2004 17:27:32 +0200

Weiming,

Weiming Wang wrote:

>     From: Robert Haas <mailto:[email-masked]>
>     To: Wang,Weiming <mailto:[email-masked]>
>     Cc: [email-masked] <mailto:[email-masked]>
>     [Weming]Could you explain a little more on this. Sorry, I just can
>     not catch up with it well. Is the priority the same as that we
>     defined in the header?
>
Yes

>      
>     Overruns occur when the FE is too slow at processing the incoming
>     CE messages, and its receive buffers are full. The reaction of the
>     CE to the overrun message can be left vendor specific. It the TML
>     is TCP, then the TCP flow-control will automatically take care of
>     this and overruns will not be generated by the FE. Other TMLs
>     might not provide such capability, hence the need for this
>     (extremely simple) windowing scheme at the PL level using overrun
>     messages.
Yes. In fact, it could be an event generated by the FE LFB that I 
proposed in the note I sent a few minutes ago. The CE could configure 
the FE LFB it and decide whether it is interested to listen to the 
overrun event.

Regards,
-Robert

>
>
>     Let me know what other approaches you would consider. The approach
>     above fits in the 'Pls let me know when you encounter
>     difficulties, otherwise pls don't bother' category Weiming already
>     suggested.
>
>     Regards,
>     -Robert
>
>     Cheers,
>     Weiming
>      
>

---

From: Robert Haas <[email-masked]>
Date: Thu, 06 May 2004 18:03:59 +0200

Weiming,

Weiming Wang wrote:

>     elaborate a bit on this. Sorry for the rather long message. I'll
>     devote another note to windowing.
>
>      Batching is used to group commands together. Each command
>     possibly has its own sequence number (or transaction number),
>     or the group has its own number with subnumbers for each message
>     (this remains to be decided.) The main use of batching is to
>     perform atomicity:
>     [Weiming]I agree this, though I was a little confused before with
>     the notions 'batching, atomicity, 2-phase commit, all or nothing'.
>     Can we understand that they are all targeting the same thing: all
>     or nothing? Or we have other purpose for batching such as just for
>     packetizing?
>

The purpose of batching, bundling, grouping, atomicity, 2-phase-commit, 
is for the same purpose: execute a serie of commands in such a way that 
their effect is the same as if it had been a single command.

If the number of commands/TLVs to include on a ForCES Msg exceeds the 
maximum length of the ForCES message, then I would simply use another 
ForCES Msg, and only set batching ON if I wanted atomicity across the 
commands in the two separate ForCES Msgs. Does that sound reasonable ?

>      
>     all commands in the group will be performed at the same time or
>     none will be performed. This is where 2-phase commit kicks in: all
>     commands are first transmitted to the FE, and unless the FE
>     reports an error, the CE can ultimately issue a commit command so
>     all these commands are executed in an atomic operation at the FE.
>     [Weiming]Yes.
>     We investigated the idea of packing multiple commands in the same
>     ForCES message (it saves quite some headers and the associated
>     processing
>     [Weiming]Here, I think we need to evaluate if the headers are
>     worth there or not. My original idea was it was not worth it
>     (during GRMP time), but now I change some. Just check the possible
>     header fields, we may find that most of them are worth there, and
>     the reward is great, allowing per msg ACK, and with much less
>     complexity for protocol state trace.
>      
>     ). As a motivation, think of adding 100'000 routes on an FE:
>     should we use 100'000 ForCES messages, or only 1000 (say 15 bytes
>     per route update command to be generous, with a 1500 bytes MTU
>     size, or even less if we have Jumbo frames) ?
>
>     Note that having many commands in the same ForCES message results
>     in slighlty more complex error/ack reporting. Take this example:
>     two commands A and B in the same ForCES message issued by a CE. FE
>     executes both of  them, but only A succeeds. It should report this
>     somehow to the CE. A possible solution is to return the original
>     message trimmed down such that it contains exactly two responses,
>     a positive and a negative one.
>     [Weiming]Yes, I think you are also trying to see which is better
>     among the following two possible batching schemes:
>     1. Batched msgs in one packet or even keeping the msgs separately
>     as its origin, while with some flags such as Atomicity and/or Done
>     Flag to indicate the begin and the end.
>     I a little more incline to #1 scheme, as the reason I mentioned
>     above. But there is a remained issue for successful atomicity,
>     that is, it does not support multipul transactions if we use
>     different sequece number for different msgs in a batch, and if we
>     use same sequence number for all the msgs, then it may result in
>     ambiguity for msg meaning and hard for ACK. Therefore, I try to
>     propose to use a sub msg number to see if it is feasible for the
>     purpose. Pls let me try to summarize sth we'v already got
>     (Netlink2?) along with the batch msg # thought as follows:
>      
>     Batching msg as:
>      
>     ATM flag: mark the end of batching (e.g.,0 for MsgN, and also a
>     signal to excute the batch, 1 for others, and also as a signal to
>     reserve it)
>     BatchMsgNum: mark the batehed msg sequence number, 0 for Msg1, ...
>     (N-1) for MsgN. Any ACK should include BatchMsgNum as well as others.
>      
>     Any of the Msg in the batch still can be fragmented if it exceeds
>     the MTU, a FRG flag is used to treat it, with the thinking that
>     the packets will not be reordered.
>
>     Rob: can you try to present a summary regarding the #2 scheme, so
>     that we can carefully compare and evaluate it?
>
Let me try:

One single ForCES Msg, with N TLVs (TLVs are basically the command 
destined to LFBs). If the batch spans more than one single ForCES Msg 
the ATM flag will have to be set accordingly. The resulting ACK Msg can 
be for the entire ForCES Msg (using the sequence number). Or the ACK Msg 
can contain the same TLVs as the original Message, with a flag in each 
TLV to indicate if that particular TLV succeeded or not. If the Atomic 
flag was set with the 2-phase commit, then all TLVs succeed or none.

The sequence number must be incremented for each new ForCES Msg. If the 
batch of TLVs/commands (TLVs/commands that logically belong to the same 
transaction) are split into two ForCES Msgs, then there will be two 
sequence numbers. I am reluctant to keep the sequence number the same. 
Acknowledging the highest number will mean that all previous Msg were 
received and processed OK. Do you need to have another number (such as 
transaction number or submsg number) necessarily ?

If someone wants to do a transaction and ensure that only a single 
sequence number is associated to it, then it must pack all the TLVs into 
the same ForCES message. Would this be satisfactory ?

I realize that in my previous note I confused command and ForCEs Msg: 
there can be one or more commands (=TLVs) in a ForCES Msg. The sequence 
number is associated with a ForCES Msg.

Regards,
-Robert

>      
>     I think we should focus our effort on defining the format and
>     state machine of the ForCES protocol as seen on the wire
>     ). So I don't think we need to worry about the way the Apps get
>     notifications from the PL and so on, and if a single/multiple
>     signal/s is/are used to confirm that a batch of messages has been
>     successfully executed. This will remain vendor specific. Do you
>     have different views ?
>     [Weiming]Agreed.
>
>     Cheers,
>     -Robert
>     Cheers,
>     Weiming
>

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 7 May 2004 20:57:27 +0800

Hi robert, and Jamal as well,

      Weiming,

  The purpose of batching, bundling, grouping, atomicity, 2-phase-commit, is for the same purpose: execute a serie of commands in such a way that their effect is the same as if it had been a single command.

  If the number of commands/TLVs to include on a ForCES Msg exceeds the maximum length of the ForCES message, then I would simply use another ForCES Msg, and only set batching ON if I wanted atomicity across the commands in the two separate ForCES Msgs. Does that sound reasonable ?

      Batching msg as:

  One single ForCES Msg, with N TLVs (TLVs are basically the command destined to LFBs). If the batch spans more than one single ForCES Msg the ATM flag will have to be set accordingly. The resulting ACK Msg can be for the entire ForCES Msg (using the sequence number). Or the ACK Msg can contain the same TLVs as the original Message, with a flag in each TLV to indicate if that particular TLV succeeded or not. If the Atomic flag was set with the 2-phase commit, then all TLVs succeed or none.
  [Weiming] I just try to think of it into a little more detail. So, will the TLVs themselves include things as ACK flag, and a number to indicate the place of the TLV in the whole big msg?  

  The sequence number must be incremented for each new ForCES Msg. If the batch of TLVs/commands (TLVs/commands that logically belong to the same transaction) are split into two ForCES Msgs, then there will be two sequence numbers. I am reluctant to keep the sequence number the same. Acknowledging the highest number will mean that all previous Msg were received and processed OK. Do you need to have another number (such as transaction number or submsg number) necessarily ?
  [Weiming] My thought to use a uniform Sequence Number for all batched ForCES msg and use BatchMsgNum to indicate the msg inside the batch is mainly for supporting of multipul transactions working in parallel. Consider a case where we have had two batch msgs interleavingly been sent to FE as:

  B1Msg1, B2Msg1, B1Msg2, B2Msg2,.....
  In this case, if we do not use a uniform sequece number for Batch1 and Batch2, It will be hard to correctly do atomicity, because we then at a loss which msgs belong to Batch1 and which belong to Batch2. 

  Hi Jamal, could you also join us in this discussion? I think we have not addressed the batch and the multi-sessions issues before though you have proposed the multi sessions problem. I'm afraid we have to solve it before we begin the header composing.
  [/Weiming]

  If someone wants to do a transaction and ensure that only a single sequence number is associated to it, then it must pack all the TLVs into the same ForCES message. Would this be satisfactory ?
  [Weiming] That's just in the case of scheme #2 (with many TLVs). I agree in this case, we do not have the problem as I mentioned above.

  I realize that in my previous note I confused command and ForCEs Msg: there can be one or more commands (=TLVs) in a ForCES Msg. The sequence number is associated with a ForCES Msg.
  [Weiming] Yes, I actually understand it. Just we are proposing two schemes. The first scheme is supposed not to use command TLVs, instead just use ForCES msgs for batching purposes. I think Netlink2 also has this scheme, right?

  Regards,
  -Robert

  BR,
  Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: 07 May 2004 14:45:05 -0400

Ok, I just deleted everything that was said and i am going to summarize
with a twist of my opinions. (I hope i read all the discussions
correctly). Robert/Weiming please double check. Hopefully, this summary
will allow other people to pitch in.

Why do we batch?
1) to pipeline and improve throughput
2) to achieve atomic operations spanning multiple packets. 

Note that #1 may not be necessarily be atomic operation.

PL message/transaction exceeding MTU:
I think the PL should carefully ensure that the messages going across
are each at least going to fit in the MTU size packet. I think if we can
avoid fragmentation of a command, then we should. 

Types of batching:

1) several PL messages in one TML level PDU. PL level header SOT and EOT
indicate start and end of a train of these messages.
Is this responsibility of TML or PL to put them in one PDU or that of
PL? I feel it is the responsibility of PL.

2) Within one message several TLVs which require that each "atomic" unit
have its own subsequence number (and treat the top header sequence
number as a flat transaction number).

My preference is using #1 mostly because now we have to look for a trick
to stash a subsequence number. too complex. You can already pack many
routes for example in one message - so no need to be clever. 
Each message will have a unique sequence number. Start of transaction
gets marked by SoT and end of transaction gets marked by EoT. And any
middle messages probably marked with MoT?
Weiming points to a concern about the common header introducing
ovrehead. My opinion is the overhead is so small that it is irrelevant
against the benefits of keeping track of subsequence numbers.

What to do in case of failures?
Lets say you sent an atomic batch of A) a del route, B) an add route and
C) add route

Techniques (listed by Joel Halpern once).

1) All or nothing:
if B fails A should also fail; if A was done then you undo

2) Fail upto:
If B fails but A passes, then we just inform CE of Bs failure and dont
execute C.

3) I cant remember it, but i think there was a third technique

I think the most interesting this is what happens if FE1 executes A
successfuly and FE2 both A and B but not C. You need to make sure things
are synchronized

I hope i summarized everuthing that was said.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Sat, 8 May 2004 12:13:01 +0800

Jamal,

Thank you for the post, which makes thing much clearer.

----- Original Message -----
> Note that #1 may not be necessarily be atomic operation.
[Weiming]Agreed. Then you mean batch is for 1) or 1)AND 2), right?  How about
just for 2) then, it will not be called batching?

> avoid fragmentation of a command, then we should.
[Weiming]I just a little worry if the pipeline and the MTU requirement will have
some contradiction with each other.

> PL? I feel it is the responsibility of PL.
[Weiming] I also incline for PL to define the pinelining format. Then, we need
to have a batch format (not necessarily a batch msg) to be defined in the
protocol text. Where do you think to put it, in the protocol overviw, common
header, or in the protocol messages?
[/Weiming]

> number as a flat transaction number).
[Weiming]In this case, we need to define a batching msg, which I think should be
defined by protocol msgs.

> middle messages probably marked with MoT?

[Weiming]Can we summarize that we may need two flags, one for atomicity, and
another for pipelining batch?

> against the benefits of keeping track of subsequence numbers.

[Weiming]Actually I more inclide to #1 also (pls see my previous reply to
robert), with the same idea that the header may waste less. Note that in TLV
mode (#2 mode), we also have to have many in the TLV, such as ACK flag,
subsequece number, etc.
Actually I was trying to propose another approach other than #1 and #2, which is
mainly for atomicity. The approach is to let a bunch of msgs that need atomicity
still be separatedly sent via TML instead of pipelining, while only using the
Atomic flag to trace the start and the end of atomiciy. Do you think we need
this? Moreover, I was considering how we can support multipul atomicity
transactions in this case.
[/Weiming]
> if B fails A should also fail; if A was done then you undo
[Weiming]My understanding of atomicity is just for this.
> If B fails but A passes, then we just inform CE of Bs failure and dont
[Weiming]I'm afraid this fail model may only apply to batching msg(s).

> 3) I cant remember it, but i think there was a third technique
[Weiming]Is it the most general one, that is, to do whatever it can do, and fail
whatever it cannt do?

> successfuly and FE2 both A and B but not C. You need to make sure things
[Weiming]I think in broadcast case, we may have to choose to use a simple fail
scheme such as all or nothing, or else, I'm just afraid it may make things much
complex.
Cheers,
Weiming

---

From: [email-masked]
Date: Sat, 8 May 2004 15:10:57 +0200

> avoid fragmentation of a command, then we should.

Yes, in fact some sort of MTU discovery might be in order.  I think 
avoiding fragmentation is of positive value.  This is probably a TML 
responsibility.

> PL? I feel it is the responsibility of PL.

Well it sort of depends on which entity is responsible for 
non-fragmentations and hence knowing what MTU size to use.  It is 
certainly the PL's responsibility to know which msgs are batched and 
especially  which are atomically batched.

> successfuly and FE2 both A and B but not C. You need to make sure 

when I think of the two kinds of batching, atomic and non, I make the 
distinction that in non-atomic, each goes on and you get a separate 
ack/error code back on each of the component messages.  With atomic, I 
tend to support option 1 above.

Option 2, the do until fail, would seem to me to be a different sort of 
batching and I am not sure how useful, or safe, a variant it is.

> I hope i summarized everuthing that was said.

I like the summaries. I get confused in the back and forth 
conversations and don't really know how to jump in with a response.  So 
thanks.

a.

---

From: "Ligang Dong" <[email-masked]>
Date: Sun, 9 May 2004 16:18:47 +0800

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>
To: "Wang,Weiming" <[email-masked]>
Cc: "Robert Haas" <[email-masked]>; <[email-masked]>
Sent: Saturday, May 08, 2004 2:45 AM
Subject: Re: [Forces-protocol] topic 4c: Common Header: batching

> 2) to achieve atomic operations spanning multiple packets. 
[Dong Ligang] You actually describe two kind of batching. "Fail upto" can be included in the first type.

> avoid fragmentation of a command, then we should. 

[Dong Ligang] I think that MTU is TML's responsibility. We need not consider it in PL. 

> Lets say you sent an atomic batch of A) a del route, B) an add route and
[Dong Ligang] 
1) Batched PL messages are obviously in one PL level PDU. However, it is not necessary that batched PL messages are in one TML level PDU. 
2) PL level header SOT and EOT are not necessary, as the placement order in one PL PDU can implicitly determine the sequence of batched messages. 
3) A field (e.g., BATCH field) in the PL messages header is used to indicate whether or not this message contain multiple batchted messages. Also, this field can be used to indicate we are batching independent messages or transaction messages. 
 4) For batching of independent messages, the responses are also independent; for batching of transaction messages, the response is single. 

> https://www1.ietf.org/mailman/listinfo/forces-protocol����趇(�f��)��+-��z�k���X���+�m����5��_��?�������+-�w���+q�)��hr�

---

From: Jamal Hadi Salim <[email-masked]>
Date: 10 May 2004 08:54:26 -0400

On Sat, 2004-05-08 at 00:13, Wang,Weiming wrote:
> [Weiming]Agreed. Then you mean batch is for 1) or 1)AND 2), right?  

Probably both - 2) is atomic.

> How about just for 2) then, it will not be called batching?

Hrm. I would probably still wanna call it batching although it has other
connotations. The simplest standalone atomic transaction is probably a
single packet for a single transaction such as "add route X". Now lets
say such a simple message wont fit in one packet, then you break it into
several messages unified by the atomicity flag. Same applies when you
group a "batch" of such unified transactions across multiple train of
packets. So i would say, for simplicty sake lets just call it a batch.

> avoiding fragmentation is of positive value.  This is probably a TML 

Agreed TML should be responsible. My worry is more when the TML is stream
oriented (such as in the case of TCP) and there are no distinct message
boundaries at the TML level. Someone needs to feed the PL complete PL messages.
For this reason it may be valuable for the PL to be pMTU aware. Thoughts?

> especially  which are atomically batched.

Ok, shall we say then that if the PL is pMTU aware it will do a better 
job?

> defined by protocol msgs.

Note, my view is we may not need this approach since it is too complex.

> another for pipelining batch?

3 flags:
1. Atomicity
2. SoT (Start of Transaction)
3. EoT (End of Transaction)

A fourth one maybe MoT (Middle of Transaction). Not sure if it is
needed; maybe it can be represented by Sot = 0 and Eot = 0
A transaction that fits within one message will always have SoT and EoT
set to 1.
A two phase commit will at minimal have two message. Something like:
msg 1 := {Atomic, SoT, EoT} = { 1,1,0}
msg ..:= { 1,0,0}
msg ..:= { 1,0,0}
msg ..:= { 1,0,0}
msg last := {1,0,1}

> mode (#2 mode), we also have to have many in the TLV, such as ACK flag,

Ok.

> transactions in this case.

refer

> included in the first type.

The difference is that in 2) you dont have to undo. But they are
probably the same thing.

> [Weiming]Is it the most general one, that is, to do whatever it can do, and fail

I think so.

[Dong Ligang] 
> 1) Batched PL messages are obviously in one PL level PDU. However, it is not 
> necessary that batched PL messages are in one TML level PDU. 

Refer to my comments above about having TML not aware of message boundaries.
Also to my comment that the PL being aware of the MTU would help.

> 2) PL level header SOT and EOT are not necessary, as the placement order in 
> one PL PDU can implicitly determine the sequence of batched messages. 

What about two phase commits?

> 3) A field (e.g., BATCH field) in the PL messages header is used to indicate 
> whether or not this message contain multiple batchted messages. Also, this field 
> can be used to indicate we are batching independent messages or transaction messages. 

Refer to my comments above on {Atomic, SoT, EoT} then lets discuss if you disagree.

>  4) For batching of independent messages, the responses are also independent; for 
> batching of transaction messages, the response is single. 

Yes, this is why it is preferable to have multiple PL level messages because
each has its own sequence number that can be responded to.

> scheme such as all or nothing, or else, I'm just afraid it may make things much

in the case of 3 above, you may not care (atomic flag = 0).

> batching and I am not sure how useful, or safe, a variant it is.

So should we then have two choices? One with all-or-none when atomic
flag is set and the other with try-all mode when atomic flag is not set?
If we are going to add the rhird one i think we need an extra flag.

> conversations and don't really know how to jump in with a response.  So 

I have come to the conclusion the reason that discussions keep getting repeated
is because theres no good summaries so far i.e insufficient meat. This is my first 
attempt. I am hoping we make fastr progress with this approach.
Who is in charge of writting this section? I could rewrite the details with the provided 
comments or whoever is in charge of it could.

cheers,
jamal

---

From: "Weiming Wang" <[email-masked]>
Date: Mon, 10 May 2004 23:40:45 +0800

Jamal,

I'v just thought of one thing. I think if it may be better to consider the batch
and the atomicity as two separate issues. Based on your summary, I describe my
thought as follows,

1. Batching
PL appearance: msg chains.
Flag used: BatchFlag
BatchFlag = 1 - used by first msg and all middle msgs in the batch
          = 0 - used by last of the batch msg, indicating the end of the batch
When in non-batch mode, msg is always set to BatchFlag = 0.

Default fail mode: to do whatever it can and fail those it cannot.

ACK mode: according to ACK flag in individual msg headers.

The BatchFlag is actually used as a signal for TML pinepining implementation.
TML is responsible for things like MTU check and fragment. I think TML should be
also responsible for the check of PL msg boundaries, because it seems in our
current msg chain mode for batch, TML is inevitable to check the msg headers.
But hornestly, I'm not sure if this is out of the scope of TML or not.

2. Atomicity
PL appearance: msg chains.
Flag used: AtomicFlag
AtomicFlag = 1 - used by first msg and all middle msgs in the atomic
           = 0 - used by last msg in the atomic, indicating the end of the
atomic and also as a signal for a receiver to excecute the atomic.
When in non-atomic mode, msg is always set to AtomicFlag = 0

Default fail mode: all or nothing

ACK mode: according to ACK flag in individual msg headers.

3. Atomicity in batching
Just operate separately in the same way as described above except that the fail
mode for atomicity has previlege over that of batching, that is, if in an
atomic, fail mode must be all or nothing.
A batch may contain more than one atomic.

Just want to see if it helps more or less for our discussion.

Cheers,
Weiming

---

From: "Weiming Wang" <[email-masked]>
Date: Mon, 10 May 2004 23:51:54 +0800

Jamal,

> comments or whoever is in charge of it could.
[Weiming]I think it is great if you can write the section. Actually I think
Common Header section contains a lot. We may need to subdivide it into
subsections like Data format, Batch, Atomicity, and ACK. Thoughts?

Cheers,
Weiming

---

From: "Weiming Wang" <[email-masked]>
Date: Tue, 11 May 2004 23:58:26 +0800

Jamal, Robert, Ligang, and Hormuzd,

One more thing for batch and atomicity I'v just thought of.
Consider a scenario, in which a CE is configuring a FE using batch or atomicity.
We used to consider that, during the batch or atomicity process, there might be
no other messages except the batch or atomicity messages sent from CE to FE, but
I'v just thought of that this may be not always true. For e.g., there may be
heartbeat messages sent from CE to FE during the process, because they will be
sent automatically and periodically. If we only use flags to indicate the batch
or atomicity messages, such heartbeat messages may intervene into the batch or
atomicity messages, making the batch or atomicity marked by flags incorrectly
understood by CE TML or by FE PL.

Therefore, we may need to limit more for batch or atomicity. For e.g., we may
need to define that only messages for config, query, and state maintenance are
allowed to be batched or atomically executed, that is, messages other than that
are automatically excluded from batch or atomicity.

A more extended thinking of this is, if we need to support parallel and multipul
atomicity at the same time, that is, if we need to allow CE to send messages
that interleavingly belong to different atoms? I think Jamal has raised this
question before.

Cheers,
Weiming

----- Original Message -----
From: "Weiming Wang" <[email-masked]>
To: <[email-masked]>; <[email-masked]>
Sent: Monday, May 10, 2004 11:40 PM
Subject: Re: [Forces-protocol] topic 4c: Common Header: batching

> > > [Weiming]I just a little worry if the pipeline and the MTU requirement
will
> have
> > > [Dong Ligang] You actually describe two kind of batching. "Fail upto" can
be
> > > included in the first type.
> >
> > > [Weiming]Is it the most general one, that is, to do whatever it can do,
and
> fail
> > > can be used to indicate we are batching independent messages or
transaction