Subject: [tcpm] Consenus on adding a new TCPM charter milestone for accurate feedback

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 2 Aug 2012 22:45:16 +0200

Hi all,

There has been broad support for a new TCPM charter milestone on accurate ECN feedback in TCP.

We therefore want to confirm that working group consensus is to add the following milestone to the TCPM carter:

   "Sep 2013 	Submit document on accurate ECN feedback in TCP to the IESG for publication as an Experimental RFC"

The chairs believe that the consensus is not to adopt any specific document as of now. However, the working group should converge on the mechanism by March 2013 latest.

Please feel free to comment if you have any additional thoughts, within the next two weeks.

Thanks

Michael (on behalf of the TCPM co-chairs)

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 2 Aug 2012 14:13:26 -0700

is accurate ECN part of "maintenance and minor extension"?

On Thu, Aug 2, 2012 at 1:45 PM, Scharf, Michael (Michael)
<[email-masked]> wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 2 Aug 2012 14:17:04 -0700

> is accurate ECN part of "maintenance and minor extension"?
sorry got cut-off.

I second that to add acc-ECN to tcpm, but often times I am confused
the roles between iccrg and tcpm. To me there should be a tcp-dev to
discuss all TCP changes. Other protocols often just apply the
principles of CC in TCP.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 3 Aug 2012 01:10:31 +0200

> > is accurate ECN part of "maintenance and minor extension"?

Do you argue that it is a major change?

For instance, just transporting a more accurate encoding does not imply that CC is changed at all.

> I second that to add acc-ECN to tcpm, but often times I am 
> confused the roles between iccrg and tcpm. To me there should 
> be a tcp-dev to discuss all TCP changes. Other protocols 
> often just apply the principles of CC in TCP.

ICCRG is reseach... Is more research needed to find for an experimental encoding that allows more than one feedback per RTT?

Anybody who thinks that this topic would be too "disruptive" for TCPM: Please speak up now.

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 3 Aug 2012 00:06:11 +0000

Yuchung,

I believe that this work is definitely in charter for TCPM.

AccECN is about a semantic/signaling change of the (existing) wire-protocol bits.

We are not proposing to make incompatible, or major changes to the wire-protocol.

There are other, standards track WGs that actually have a documented need for a better congestion feedback scheme than exists today in TCP (conex). The only feedback scheme meeting these requirements is currently DCCP (Ack Vector ECN-marked), SCTP (draft only), and RTP (draft only).

The problem you are perhaps observing is that as TCP is the de-facto standard protocol for the majority of transport needs (and we probably all agree that it's often not the best choice technically, but the only choice for deployability), the limitations of TCP become apparent and people are still interested in addressing those within the given confines... 
 
Best regards,

Richard Scheffenegger

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Scharf, Michael (Michael)
> Sent: Donnerstag, 02. August 2012 16:11
> To: Yuchung Cheng
> Cc: [email-masked]
> Subject: Re: [tcpm] Consenus on adding a new TCPM charter milestone for
> accurate feedback
> 
> ICCRG is reseach... Is more research needed to find for an experimental
> encoding that allows more than one feedback per RTT?
>

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 2 Aug 2012 17:59:37 -0700

> Anybody who thinks that this topic would be too "disruptive" for TCPM: Please speak up now.

with the proposed milestone, we'll add more signals to ECN then ?
discuss what to do about them in iccrg or other group?
what's the plan for better ECN is my question. sorry if this sounds dumb.

---

From: John Leslie <[email-masked]>
Date: Thu, 2 Aug 2012 21:28:30 -0400

Scharf, Michael (Michael) <[email-masked]> wrote:
> 
> We therefore want to confirm that working group consensus is to add
> the following milestone to the TCPM carter:
> 
>    "Sep 2013 	Submit document on accurate ECN feedback in TCP to the
>     IESG for publication as an Experimental RFC"
> 
> The chairs believe that the consensus is not to adopt any specific
> document as of now. However, the working group should converge on
> the mechanism by March 2013 latest.

   I find myself worried by the name "accurate ECN feedback".

   That is a reasonable name for the draft we've seen; but I don't
think it's the right name for the _output_ of this WG.

   The issue is a bit complicated, alas: there's very good reason
to want a signal coming back to the sender which carries more
precision than the current signal (ECN-Echo in an unreliable ACK).
Quoting RFC 3168 Section 6.1:
] 
] * An ECT codepoint is set in packets transmitted by the sender to
]   indicate that ECN is supported by the transport entities for
]   these packets.
]
] * An ECN-capable router detects impending congestion and detects
]   that an ECT codepoint is set in the packet it is about to drop.
]   Instead of dropping the packet, the router chooses to set the CE
]   codepoint in the IP header and forwards the packet.
]
] * The receiver receives the packet with the CE codepoint set, and
]   sets the ECN-Echo flag in its next TCP ACK sent to the sender.
]
] * The sender receives the TCP ACK with ECN-Echo set, and reacts to
]   the congestion as if a packet had been dropped.
]
] * The sender sets the CWR flag in the TCP header of the next
]   packet sent to the receiver to acknowledge its receipt of and
]   reaction to the ECN-Echo flag.

   In fact, due to the unreliable nature of the ACK, the receiver
continues to set ECN-Echo (ECE) until it sees a CWR. From 6.1.3:
] 
] To provide robustness against the possibility of a dropped ACK packet
] carrying an ECN-Echo flag, the TCP receiver sets the ECN-Echo flag in
] a series of ACK packets sent subsequently.  The TCP receiver uses the
] CWR flag received from the TCP sender to determine when to stop
] setting the ECN-Echo flag.

   This is messy! and it is only acceptable for trying to match the
window-reduction of a packet loss.

   So there is very good reason to come up with a mechanism to provide
better reliability and precision of informing the sender about ECT
codepoints seen by the reciver.

   I entirely support such an effort, but I think "greater precision"
would be a better term than "accurate".

--
John Leslie <john@jlcnet>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 3 Aug 2012 16:40:06 +0000

Hi John,

Am I right in assuming that with "greater precision" you imply that this doesn't necessarily mean "exact" feedback?

(Originally we discussed using "more accurate" in the title of the draft;  that meant lower precision than "exact", but the "more" got lost along the way, leaving "accurate").

I would argue that the current ECE/CWR mechanism is perfectly reliable - up to a certain precision limited by the number of bits set aside for the signal back from the receiver to the sender. What would be more reliable?

(ECE is a counter that can only increase, and remains at the maximum value until being reset; CWR provides the reset signal).

OTOH, if the feedback signal is encoded with enough bits, reliability can be brought arbitrarily close to "perfect", even without an explicit forward signal (CWR).

As soon as the counter (and signal for it) cover all the packets individually in a window, the ECN feedback can be exact; The question becomes, can the reset mechanism be abandoned when "enough" bits are available for the signal, or must the principal architecture (a counter increasing up to some point, until it is reset) be maintained...

BTW: I found that ECN for SCTP also failed to discuss this, but there a 32 bit counter is used (plus a Reset signal/CWR).

To jump forward a bit, I suspect the delineation is somewhere around 5 bits (2.3 bits in the codepoint draft; 3 bits in reECN; 8 bits in the TCP Option draft). Using 5 bits or less, a handshare mechanism is required; using more than 5, and a simple counter suffices to maintain exactness (until TCP suffers from loss of the ACK clock etc).

The signaling is another question; Perhaps a certain loss of granularity is acceptable for the currently discussed protocols (i.e. Conex; slight overestimating congestion is not as bad as underestimating). But other (future) uses (e.g. DCTCP) would probably require the full precision - with other words, "exact" signaling...

However, we (authors), while discussing this internally, would also be glad of any opinion regarding this!

Richard Scheffenegger

> John Leslie
> Sent: Donnerstag, 02. August 2012 18:29
> To: Scharf, Michael (Michael)
> Cc: [email-masked]

---

From: John Leslie <[email-masked]>
Date: Fri, 3 Aug 2012 14:34:21 -0400

Scheffenegger, Richard <[email-masked]> wrote:
> 
> Hi John,
> 
> Am I right in assuming that with "greater precision" you imply that
> this doesn't necessarily mean "exact" feedback?

   Yes, but... This is mainly to avoid arguing what "exact" might mean.

> (Originally we discussed using "more accurate" in the title of the draft;
> that meant lower precision than "exact", but the "more" got lost along
> the way, leaving "accurate").

   (I must admit I'm at least as anxious to avoid arguing what "accurate"
might mean.)

> I would argue that the current ECE/CWR mechanism is perfectly reliable

   I might take the other side of that argument...

> - up to a certain precision limited by the number of bits set aside
> for the signal back from the receiver to the sender. What would be more
> reliable?

   "Reliable" in TCP means that everything not acknowledged will be
retransmitted. That would imply repeating each ECE until that specific
ECE is acknowledged -- and clarifying to the sender how that CWR was
understood.

   But I'm not trying to argue either way whether "reliable" would be
"better" than "unreliable".

> (ECE is a counter that can only increase, and remains at the maximumi
> value until being reset; CWR provides the reset signal).

   Umm... "remains at maximum" implies some loss of precision...

> OTOH, if the feedback signal is encoded with enough bits, reliability
> can be brought arbitrarily close to "perfect", even without an explicit
> forward signal (CWR).

   Indeed, one could argue "reliability" by unacknowledged repetition.
(I'd prefer some other term, but it certainly would serve for the
"precision" I was seeking.)

> As soon as the counter (and signal for it) cover all the packets
> individually in a window, the ECN feedback can be exact;

   Actually, there are two counters, packets and (payload) bytes.

> The question becomes, can the reset mechanism be abandoned when
> "enough" bits are available for the signal, or must the principal
> architecture (a counter increasing up to some point, until it is
> reset) be maintained...

   I don't choose to express an opinion on that yet.

> BTW: I found that ECN for SCTP also failed to discuss this, but
> there a 32 bit counter is used (plus a Reset signal/CWR).

   :^)

> To jump forward a bit, I suspect the delineation is somewhere around
> 5 bits (2.3 bits in the codepoint draft; 3 bits in reECN; 8 bits in
> the TCP Option draft). Using 5 bits or less, a handshare mechanism is
> required; using more than 5, and a simple counter suffices to maintain
> exactness (until TCP suffers from loss of the ACK clock etc).

   No comment (yet)...

> The signaling is another question; Perhaps a certain loss of
> granularity is acceptable for the currently discussed protocols
> (i.e. Conex; slight overestimating congestion is not as bad as
> underestimating). But other (future) uses (e.g. DCTCP) would probably
> require the full precision - with other words, "exact" signaling...

   "Exact" _is_ a reasonable goal...

> be glad of any opinion regarding this!

   I will note that current ECN processing in TCP is awfully specialized:
I can imagine congestion-control algorithms it would serve poorly. :^(
But leaving that aside, ECN processing for other transport protocols
almost certainly deserves a different treatment.

   It would be nice if our "greater precision" effort could serve the
needs of yet-unseen future transport protocols.

--
John Leslie <[email-masked]>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 3 Aug 2012 20:58:18 +0000

Hi John,

Thanks for your comments so far!

>    I might take the other side of that argument...

Could you please explain? (I should have perhaps added, RFC3168 is perfectly reliable for at least, and up to, one CE mark per RTT; The sender gets the information that at least one CE mark was received in the RTT under all circumstances).

>    Actually, there are two counters, packets and (payload) bytes.

Correct; but as soon as not every data segment is ACKed (and the ACK not lost), the delineation (for figuring out which payload bytes got marked) becomes fuzzy... Even when the sender keeps state as to the segment boundaries, it's not clear which of multiple segments were marked (except in the two extreme cases), when delayed ACKs (or even ACK compression) are in play... 

Which brings up another question that I haven't put in this forum:

Will there be a conceivable benefit when the sender knows the exact order or markings?

The way the receiver ECN state-machine of DCTCP is described allows the sender to reconstruct the exact order of markings (the "CE stream"), provided no (relevant) ACKs are lost. With the complication that the sender rarely knows, when and if a relevant ACK got dropped...

This level of precision is lost, when the receiver uses some kind of summary signal; But making such a signal reliable requires even more bits, and some more or less complex encoding scheme...

But I am not sure if such an extreme precision will be necessary to TCP at least (and other transports do provide means to encode that...)

To have a proper counter (congested bytes), I think that would need to be explicit; Mirja did some experiments using implicit ACKs, but to get them work properly, you end up with sending out more ACKs than strictly necessary (like what DCTCP receivers do), and single ACK loss can still result in wrong signals at the sender....

Richard Scheffenegger

> From: John Leslie [mailto:[email-masked]]
> Sent: Freitag, 03. August 2012 11:34
> To: Scheffenegger, Richard
> Cc: [email-masked]
> > (Originally we discussed using "more accurate" in the title of the
> > draft; that meant lower precision than "exact", but the "more" got
> > lost along the way, leaving "accurate").
> 
> > for the signal back from the receiver to the sender. What would be
> > more reliable?
> 
> retransmitted. That would imply repeating each ECE until that specific ECE
> is acknowledged -- and clarifying to the sender how that CWR was
> understood.
> 
> > can be brought arbitrarily close to "perfect", even without an
> > explicit forward signal (CWR).
> 
> > BTW: I found that ECN for SCTP also failed to discuss this, but there
> > a 32 bit counter is used (plus a Reset signal/CWR).
> 
>    :^)
> 
> > granularity is acceptable for the currently discussed protocols (i.e.
> > Conex; slight overestimating congestion is not as bad as
> I can imagine congestion-control algorithms it would serve poorly. :^( But
> leaving that aside, ECN processing for other transport protocols almost
> certainly deserves a different treatment.
>

---

From: Matt Mathis <[email-masked]>
Date: Sun, 5 Aug 2012 13:02:07 -0700

I agree that "accurate" is a little bit too strong of term.  Perhaps
"counted", "proportional", "quantified", "measured", etc.

And yes, TCPM is probably the right place, since all CC standards
track docs are explicitly in scope, even as applied to other
protocols.

If we separate signalling from action, some proposed actions clearly
need to mature in ICCRG for a bit.   Here is a tough question though:
does the choice of alternate actions effect the design of the
signalling?  If so, it may be premature to commit on the signalling.
(e.g. for low threshold ECN (aka DCTCP) isolated ECN marks would be
extremely unlikely and it would probably be ok to miss a few.),

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

---

From: Yuchung Cheng <[email-masked]>
Date: Sun, 5 Aug 2012 13:17:15 -0700

> extremely unlikely and it would probably be ok to miss a few.),
That's exactly my question ... thanks for presenting it in a much
better wording.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 7 Aug 2012 16:40:20 +0000

Hi Matt,

> does the choice of alternate actions effect the design of the signalling?
> If so, it may be premature to commit on the signalling.

I believe that any signaling would still have to cater for legacy style, low-density ECN marking (RED) for some time. 

Further I suspect that the way how the sender handles segments (segment-oriented or bytestream-oriented TCP) can also have an indirect implication as to the quality of a feedback signal, if only a few bits can be used for it.

I asked this question a couple days ago, if the exact ordering in which CE marks are received on the receiver need to be conveyed back to the sender. That would allow the sender to infer exactly which packet was marked. One argument in favor of this which was brought forward was that this could allow the sender to know how many "congestion bytes" were marked. However, that particular aspect can also be addressed with a byte-counting feedback signal, and the full CE "bit stream" is not required then.

The last question I still wonder is, would there be any benefit to the network to know when the sender did react to a control signal...

When using Conex, CWR really is redundant (but it does mark the exact segment when TCP CC reacted; conex signals don't necessarily have that, but the draft is still in flux there).

Today, CWR is also (primarily) used as a "reset" signal to the receiver, as much as a signal that the sender TCP CC reacted.

If we could find enough bits (as mentioned my experiments point to slightly over 5 bits as being sufficient), a very simple, straight forward signaling scheme should cater even unknown future needs (except for the exact CE bitstream case mentioned above).

Best regards,

Richard Scheffenegger

> From: Matt Mathis [mailto:[email-masked]]
> Sent: Sonntag, 05. August 2012 13:02
> Cc: John Leslie; [email-masked]
> And yes, TCPM is probably the right place, since all CC standards track
> docs are explicitly in scope, even as applied to other protocols.
> 
> does the choice of alternate actions effect the design of the signalling?
> If so, it may be premature to commit on the signalling.
> > Could you please explain? (I should have perhaps added, RFC3168 is
> perfectly reliable for at least, and up to, one CE mark per RTT; The
> sender gets the information that at least one CE mark was received in the
> RTT under all circumstances).
> >
> >
> > Correct; but as soon as not every data segment is ACKed (and the ACK not
> lost), the delineation (for figuring out which payload bytes got marked)
> becomes fuzzy... Even when the sender keeps state as to the segment
> boundaries, it's not clear which of multiple segments were marked (except
> in the two extreme cases), when delayed ACKs (or even ACK compression) are
> in play...
> >
> > The way the receiver ECN state-machine of DCTCP is described allows the
> sender to reconstruct the exact order of markings (the "CE stream"),
> provided no (relevant) ACKs are lost. With the complication that the
> sender rarely knows, when and if a relevant ACK got dropped...
> >
> > This level of precision is lost, when the receiver uses some kind of
> summary signal; But making such a signal reliable requires even more bits,
> and some more or less complex encoding scheme...
> >
> >
> > But I am not sure if such an extreme precision will be necessary to
> > TCP at least (and other transports do provide means to encode that...)
> >
> >
> > To have a proper counter (congested bytes), I think that would need to
> be explicit; Mirja did some experiments using implicit ACKs, but to get
> them work properly, you end up with sending out more ACKs than strictly
> necessary (like what DCTCP receivers do), and single ACK loss can still
> result in wrong signals at the sender....
> >
> >
> >
> >
> > Richard Scheffenegger
> >
> >
> >
> >
> >> Subject: Re: [tcpm] Consenus on adding a new TCPM charter milestone
> >> for accurate feedback
> >>
> >> retransmitted. That would imply repeating each ECE until that
> >> specific ECE is acknowledged -- and clarifying to the sender how that
> >> CWR was understood.
> >>
> >> > (ECE is a counter that can only increase, and remains at the
> >> > maximumi value until being reset; CWR provides the reset signal).
> >>
> >> > OTOH, if the feedback signal is encoded with enough bits,
> >> > reliability can be brought arbitrarily close to "perfect", even
> >> > without an explicit forward signal (CWR).
> >>
> >> > the TCP Option draft). Using 5 bits or less, a handshare mechanism
> >> > is required; using more than 5, and a simple counter suffices to
> >> > maintain exactness (until TCP suffers from loss of the ACK clock
> etc).
> >>
> >>    No comment (yet)...
> >>
> >> > underestimating). But other (future) uses (e.g. DCTCP) would
> >> > probably require the full precision - with other words, "exact"
> signaling...
> >>
> >> I can imagine congestion-control algorithms it would serve poorly.
> >> :^( But leaving that aside, ECN processing for other transport
> >> protocols almost certainly deserves a different treatment.
> >>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 28 Aug 2012 09:27:06 +0200

So, let's discuss wordsmithing. What about simply saying "more detailed"?

 "Sep 2013 	Submit document on more detailed ECN feedback in TCP to the IESG for publication as an Experimental RFC"

Comments are welcome.

Michael

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Matt Mathis
> Sent: Sunday, August 05, 2012 10:02 PM
> Subject: Re: [tcpm] Consenus on adding a new TCPM charter 
> milestone for accurate feedback
> 
> I agree that "accurate" is a little bit too strong of term.  
> Perhaps "counted", "proportional", "quantified", "measured", etc.
> 
> And yes, TCPM is probably the right place, since all CC 
> standards track docs are explicitly in scope, even as applied 
> to other protocols.
> 
> (e.g. for low threshold ECN (aka DCTCP) isolated ECN marks 
> would be extremely unlikely and it would probably be ok to 
> miss a few.),
> 
> Thanks,
> --MM--
> > Could you please explain? (I should have perhaps added, 
> RFC3168 is perfectly reliable for at least, and up to, one CE 
> mark per RTT; The sender gets the information that at least 
> one CE mark was received in the RTT under all circumstances).
> >
> >
> > Correct; but as soon as not every data segment is ACKed 
> (and the ACK not lost), the delineation (for figuring out 
> which payload bytes got marked) becomes fuzzy... Even when 
> the sender keeps state as to the segment boundaries, it's not 
> clear which of multiple segments were marked (except in the 
> two extreme cases), when delayed ACKs (or even ACK 
> compression) are in play...
> >
> > The way the receiver ECN state-machine of DCTCP is 
> described allows the sender to reconstruct the exact order of 
> markings (the "CE stream"), provided no (relevant) ACKs are 
> lost. With the complication that the sender rarely knows, 
> when and if a relevant ACK got dropped...
> >
> > This level of precision is lost, when the receiver uses 
> some kind of summary signal; But making such a signal 
> reliable requires even more bits, and some more or less 
> complex encoding scheme...
> >
> >
> > To have a proper counter (congested bytes), I think that 
> would need to be explicit; Mirja did some experiments using 
> implicit ACKs, but to get them work properly, you end up with 
> sending out more ACKs than strictly necessary (like what 
> DCTCP receivers do), and single ACK loss can still result in 
> wrong signals at the sender....
> >
> >
> >
> >
> > Richard Scheffenegger
> >
> >
> >
> >
> >> > maintain exactness (until TCP suffers from loss of the 
> ACK clock etc).
> >>
> >>    No comment (yet)...
> >>
> >> > probably require the full precision - with other words, 
> "exact" signaling...
> >>
> >> > However, we (authors), while discussing this internally, 
> would also 
> >> > be glad of any opinion regarding this!
> >>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 28 Aug 2012 09:38:39 +0200

I like it!

Michael

---

From: John Leslie <[email-masked]>
Date: Tue, 28 Aug 2012 10:37:24 -0400

>  "Sep 2013 	Submit document on more detailed ECN feedback in TCP to the IESG for publication as an Experimental RFC"

   Sounds better to me...

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 28 Aug 2012 08:55:20 -0700

>  "Sep 2013      Submit document on more detailed ECN feedback in TCP to the IESG for publication as an Experimental RFC"
+1