Subject: Re: [tcpm] Working group acceptance call draft-bagnulo-tcpm-generalized-ecn-04

From: Padma Bhooma <[email-masked]>
Date: Sat, 17 Jun 2017 09:27:24 -0700

Hi Marcelo and Bob,

I have reviewed the document draft-bagnulo-tcpm-generalized-ecn-04 and I am sending my comments below. Overall, I think there is value in working on these additions or enhancements to RFC 3168 in tcpm working group. I am willing to review and provide feedback as needed.

Section 3.2.1.3 - If the SYN-ACK does not support AccECN, TCP must conservatively reduce its IW and SHOULD reduce it to 1 SMSS. This clause is very conservative because AccECN support can be deployed on servers slowly. Even if both end hosts support AccECN, there are networks with performance proxies that will not allow the feedback to propagate correctly. On those networks, TCP connections with ECT on SYN will always start with 1MSS of IW. I think this is too conservative and might add multiple RTT to grow the window. I agree with the value in adding ECT marking to SYN and SYN-ACK in terms of the latency benefits that might bring to an end user. I think “pessimistic ECT and cache successes” strategy described in 4.2.1 sounds like a reasonable one.

Section 3.2.3 ECT marking on Pure Acks - It felt like the document suggested further discussion on setting ECT on pure Acks, please correct me if I am wrong here. Setting ECT on pure Acks can increase the load on an AQM that might already be under load. This is because the number of Acks can be large and the information conveyed to the peer is redundant because asks are cumulative. TCP acknowledgements also do not solicit a response from the peer most of the times. So the chances of not echoing CE on an Ack is high. It is also difficult to detect if an Ack is dropped by an incompatible middle box because of ECT marking on it since we do not retransmit Acks.. So the fallback mechanisms will be limited. In Section 4.4.1 (Cwnd response to CE-marking on Pure Acks), the document says that we can echo a CE received on a pure ack on a data segment that will be sent later. In general, I think any congestion notification is relevant only for a short period of time probably in the order of a typical RTT. Once that time window passes, it may not be correct to respond to a delayed congestion notification. So echoing CE from a pure Ack on a data segment sent much later may not be the correct thing to do.

Section 3.2.4 Window Probe - Adding ECT to window probes is really good because this traffic is not too large on any connection and it requires the peer to respond so that there is a way to echo the CE marking. One modification that might be worth discussing is to require congestion response to CE marking on a window probe only when the receive window is opened. I will try to explain why this is reasonable. If congestion at the bottleneck link is transient, then lowering the congestion window in response to CE on a window probe that does not open the receive window might not be necessary because there won’t be any data sent afterwards. This can actually happen multiple times on a connection because a connection can go on for a long time sending multiple window probes when the peer does not open the window. So reducing congestion window only when the receive window is opened as a response to CE seems to be accurate in terms of timing and reaction to congestion notification.

Section 3.2.7 Retransmissions - If a connection doesn’t set ECT on a SYN because it does not support AccECN feedback (due to fallback mechanisms or something else), then the first data packet will be the first packet on a connection with ECT marking on it. If it is dropped, sending retransmissions without ECT might make it go through if there is a middle box that incorrectly drops packets with ECT marking. I think, this detection and fallback mechanism must be somehow included in the proposal to mark retransmissions with ECT.

Another point to note is that a connection should be already in recovery and must have adjusted it’s congestion window before sending a retransmission because it detected loss. If it further detects that there was CE marking on a retransmitted packet then it is not clear if it needs to adjust the congestion window again. Adding some clarification here will be good.

Tail loss probe - I think it will also be useful to include some text about CE marking on probe packets that a connection might send in response to tail loss (draft-dukkipati-tcpm-tcp-loss-probe-01.txt).

Implementation or deployment related comments - The document refers to other drafts or experimental RFCs as possible actions that can be taken in several scenarios. For example AccECN, RFC 5690, RFC 7567 etc,. Even though implementing and deploying these specifications is possible, they may not be implemented unless these specifications are tightly bound together. A casual reader might understand that implementing multiple of these RFCS will make the system work well. But for someone to implement and deploy these specifications, it is not clear if some or all of these implementations will need to be supported. It will be good to make a recommendation for the minimum set of specifications that need to be implemented — may be after some more discussion.

Fallback mechanisms - The document also discusses several fallback mechanisms to handle incompatibilities in the network. If we want implementations to consider these cases, it will be useful to formalize these detection and fallback mechanisms. I found that some of these mechanisms are being discussed in section 4 which was tagged as informative.

Overall, the document is written well and discusses multiple aspects of the proposed changes both on the end hosts and in the network. It is useful to have discussions on the proposed changes and to drive measurements that are needed to collect data to support or validate some of the assumptions made.

Thanks,
Padma

> 
> 
>   1. Working group acceptance call
>      draft-bagnulo-tcpm-generalized-ecn-04
>      (Scharf, Michael (Nokia - DE/Stuttgart))
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Fri, 16 Jun 2017 06:06:43 +0000
> From: "Scharf, Michael (Nokia - DE/Stuttgart)"
> 	<[email-masked]>
> To: "[email-masked]" <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> Subject: [tcpm] Working group acceptance call
> 	draft-bagnulo-tcpm-generalized-ecn-04
> Message-ID:
> 	<[email-masked]>
> 	
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi all,
> 
> As requested by the authors, this e-mail starts a working group acceptance call for draft-bagnulo-tcpm-generalized-ecn-04, which will run on the list until June 30.
> 
> The proposal is to add a new milestone to the  TCPM charter
> 
>  Submit document on adding Explicit Congestion Notification (ECN) to TCP Control Packets to the IESG for publication as Experimental RFC
> 
> and to use draft-bagnulo-tcpm-generalized-ecn as a starting point.
> 
> If you believe that TCPM should work on a document in this space, and in particular if you are willing to review it, please reply to this e-mails. Please also speak up if you have any concerns or objections.
> 
> Thanks
> 
> Michael
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://mailarchive.ietf.org/arch/browse/tcpm/attachments/20170616/151f93e9/attachment.html>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
> 
> 
> ------------------------------
> 
> End of tcpm Digest, Vol 158, Issue 7
> ************************************

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Jun 2017 17:56:19 +0100

Padma,

On 17/06/17 17:27, Padma Bhooma wrote:
> Hi Marcelo and Bob,
>
> I have reviewed the document draft-bagnulo-tcpm-generalized-ecn-04 and 
> I am sending my comments below. Overall, I think there is value in 
> working on these additions or enhancements to RFC 3168 in tcpm working 
> group. I am willing to review and provide feedback as needed.
>
>   * *Section 3.2.1.3 *- If the SYN-ACK does not support AccECN, TCP
>     must conservatively reduce its IW and SHOULD reduce it to 1 SMSS.
>     This clause is very conservative because AccECN support can be
>     deployed on servers slowly. Even if both end hosts support AccECN,
>     there are networks with performance proxies that will not allow
>     the feedback to propagate correctly. On those networks, TCP
>     connections with ECT on SYN will always start with 1MSS of IW. I
>     think this is too conservative and might add multiple RTT to grow
>     the window. I agree with the value in adding ECT marking to SYN
>     and SYN-ACK in terms of the latency benefits that might bring to
>     an end user.
>
[BB] We have structured the document to just recommend the preferred 
approach in the normative spec (section 3), then discuss the rationale 
and possible alternatives in section 4. The sections just before 3.2.1.3 
recommend:
   - 3.2.1.1: the client uses ECT optimistically
   - 3.2.1.2: but the client caches servers that don't recognize ECT.
This is the "optimistic ECT and cache failures" strategy discussed in 
4.2.1 (tagged as 'S2B').

The requirement to reduce IW to 1SMSS only applies if the client is 
adopting this strategy and discovers the server does not support AccECN. 
Perhaps we didn't make that clear?

We have given 3 reasons for why this conservatism will rarely harm 
performance:

1) With this strategy, IW=1SMSS will only be needed the first time a 
client accesses a non-AccECN proxy or server.

2) During transition, this conservative IW=1SMSS will only ever be 
necessary for the client to server half-connection, not server to client.
According to Yuchung Cheng, client Web requests are rarely greater than 
1 segment (no data openly available). Do you have any data to 
corroborate (or contradict) this? I think in non-Web apps, the initial 
client request would often fit within 1 packet too (e.g. email, ftp, 
ssh, scp, etc).

3) Fall-back to 1SMSS is written as SHOULD rather than MUST. And refers 
to text later that says under what conditions you can use a higher IW.

>   * I think “pessimistic ECT and cache successes” strategy described
>     in 4.2.1 sounds like a reasonable one.
>
[BB] What's your reasoning for preferring this one? We recommended 
"optimistic ECT and cache failures".

>
>   * *Section 3.2.3 ECT marking on Pure Acks* - It felt like the
>     document suggested further discussion on setting ECT on pure Acks,
>     please correct me if I am wrong here. Setting ECT on pure Acks can
>     increase the load on an AQM that might already be under load. This
>     is because the number of Acks can be large and the information
>     conveyed to the peer is redundant because asks are cumulative. TCP
>     acknowledgements also do not solicit a response from the peer most
>     of the times. So the chances of not echoing CE on an Ack is high.
>
[BB] Let's take these points one at a time:
PB: Setting ECT on pure ACKs could increase the load on an AQM
BB: An AQM processes packets of all sizes anyway. If pure ACKs are not 
ECT, it still has to decide whether to drop each pure ACK. Marking 
instead of dropping doesn't make any more work for the AQM.

PB: The information conveyed to the peer is redundant because ACKs are 
cumulative.
BB: Just because the ACKnumber info is superseded by the next ACK does 
not mean the ECN marking is superseded.

PB: TCP acknowledgements do not solicit a response from the peer most of 
the times. So the chances of not echoing CE on an Ack is high.
BB: Just because pure ACKs do not solicit a specific response (no ACK of 
an ACK) does not mean the data sender is not continuing to send data 
packets, on which ECE can be set (if RFC3168 feedback) or on which the 
CE counter can be increased (AccECN feedback).

RFC3168 actually says this:

    (One simple possibility would be for the sender to reduce its
    congestion window when it receives a pure ACK packet with the CE
    codepoint set).

Consider this pure ACK example with data solely in the S->C direction, 
where S is the server.
   * Say 3 in 1000 pure ACKs in the C->S direction are CE-marked, 
meaning the C->S marking probability is 0.3%
   * This might be because 0.3% of data packets in other flows (e.g. a 
YouTube upload) are being CE-marked as well.
   * So S needs to tell C that it has seen 3 CE marked packets (which it 
will do on the data packets it is sending, whether with RFC3168 feedback 
or AccECN feedback).
   * C is not (currently) sending any data to S, but it will reduce its 
cwnd for if it does start sending data.

>   * It is also difficult to detect if an Ack is dropped by an
>     incompatible middle box because of ECT marking on it since we do
>     not retransmit Acks.. So the fallback mechanisms will be limited.
>
[BB] You're correct in all respects here. Nonetheless, I have three 
responses:
   * We have been doing millions of measurements over fixed and mobile 
networks, using a machine we control on both sides and checking a sent 
ECT packet is also received. So far, we have not seen any difference in 
the treatment of any control packet if set to ECT vs not-ECT.
   * Section 6.1.4 of RFC3168 
<https://tools.ietf.org/html/rfc3168#section-6.1.4> hinted that in 
future pure ACKs might be set to ECT. So if a security middlebox is 
trying to enforce RFC3168, it still might allow ECT pure ACKs to pass.
   * Nonetheless, if there were a middlebox that dropped ECT pure ACKs, 
you are right that it could black-hole all ECT pure ACKs and stall a 
connection. And there would be no easy way to detect the specific cause 
of the problem and stop setting ECT on pure ACKs.
   * We should at least say that if fall-back has to be used because ECT 
SYNs or SYN-ACKs are not getting through, then ECT ought not to be used 
on pure ACKs either.

This is a good reason for why this protocol is experimental. However, 
given we have not so far found this problem in practice, I think it's OK 
to go ahead. But I agree that we should note that this is an open issue.

>   * In Section 4.4.1 (Cwnd response to CE-marking on Pure Acks), the
>     document says that we can echo a CE received on a pure ack on a
>     data segment that will be sent later. In general, I think any
>     congestion notification is relevant only for a short period of
>     time probably in the order of a typical RTT. Once that time window
>     passes, it may not be correct to respond to a delayed congestion
>     notification. So echoing CE from a pure Ack on a data segment sent
>     much later may not be the correct thing to do.
>
[BB] This is true.
However, this point applies generally to idle periods, not just when 
pure ACKs are marked. TCP can always enter an idle period just after a 
loss or CE mark is received, then if later the sender has something to 
send, cwnd will be lower. Cwnd is already reduced exponentially during 
an idle period.

This is all part of the fairly arbitrary rules on how to evolve cwnd 
during an idle period. Nonetheless, while idling, I think it is sensible 
to reduce cwnd more, if more of any pure ACKs sent during the idle 
period are picking up congestion markings. I don't think we should 
recommend the converse: i.e. if the congestion was a long time ago, we 
should not recommend that cwnd can be increased again.

However, this is an area where I think implementers should be allowed to 
experiment, and I think the specs allow enough flexibility to do so.

>
>   * *Section 3.2.4 Window Probe* - Adding ECT to window probes is
>     really good because this traffic is not too large on any
>     connection and it requires the peer to respond so that there is a
>     way to echo the CE marking. One modification that might be worth
>     discussing is to require congestion response to CE marking on a
>     window probe only when the receive window is opened. I will try to
>     explain why this is reasonable. If congestion at the bottleneck
>     link is transient, then lowering the congestion window in response
>     to CE on a window probe that does not open the receive window
>     might not be necessary because there won’t be any data sent
>     afterwards. This can actually happen multiple times on a
>     connection because a connection can go on for a long time sending
>     multiple window probes when the peer does not open the window. So
>     reducing congestion window only when the receive window is opened
>     as a response to CE seems to be accurate in terms of timing and
>     reaction to congestion notification.
>
[BB] On the other hand, if the window was constrained solely because the 
receive window was low and there had been a CE mark on a window probe 
many round trips ago, but not on more recent window probes, it would not 
be correct to reduce cwnd such a long time after the congestion had 
occurred.

I think it would be better to reduce cwnd immediately there is a CE on a 
window probe, increase cwnd as normal for every non-CE-marked window 
probe (even if rwnd is still the limiting factor). But also rely on cwnd 
validation so that cwnd cannot grow excessively if it is not being used.

>
>   * *Section 3.2.7 Retransmissions *- If a connection doesn’t set ECT
>     on a SYN because it does not support AccECN feedback (due to
>     fallback mechanisms or something else), then the first data packet
>     will be the first packet on a connection with ECT marking on it.
>     If it is dropped, sending retransmissions without ECT might make
>     it go through if there is a middle box that incorrectly drops
>     packets with ECT marking. I think, this detection and fallback
>     mechanism must be somehow included in the proposal to mark
>     retransmissions with ECT.
>
[BB] Good point. I had figured that fall-back only needed to be 
considered for the SYN or SYN-ACK. But you're right that, if the SYN is 
not-ECT, fall-back might be needed later (altho, as already mentioned, 
we have not seen a need in measurements so far). This will generally 
only apply to the C->S half-connection.

This is a general point about fall-back that happens to be noticed when 
both data and retransmissions fail to get through. So I think we ought 
to add a general point about fall-back as a new subsection of section 3, 
rather than writing this in the retransmissions subjection. I propose 
this text:

    3.2.8 General Fall-Back

    Consider a host that has sent the first packet on a half-connection
    (SYN or SYN/ACK) without ECT set and it has been acknowledged. If
    this host's retransmission timer expires more than once for all
    subsequent packet(s) it sends with ECT set, it SHOULD retransmit the
    packet(s) with not-ECT set. If these not-ECT packet(s) are
    acknowledged without any further losses, it is RECOMMENDED that the
    host disables setting ECT on all subsequent packets of the
    half-connection. It could also cache this failure.

>
>     Another point to note is that a connection should be already in
>     recovery and must have adjusted it’s congestion window before
>     sending a retransmission because it detected loss. If it further
>     detects that there was CE marking on a retransmitted packet then
>     it is not clear if it needs to adjust the congestion window again.
>     Adding some clarification here will be good.
>
[BB] Yes. We already say:

    If the TCP sender receives feedback that a retransmitted packet was
    CE-marked, it will react as it would to any feedback of CE-marking on
    a data packet.

Do you have a suggestion for clearer text? Perhaps:

    If the TCP sender receives feedback that a retransmitted packet was
    CE-marked, it will react as it would to any feedback of CE-marking on
    a data packet, in addition to its original reaction to the loss that
    caused the retransmission.

RFC5681 already says:

    Loss in two successive windows of data, or the loss of a
    retransmission, should be taken as two indications of congestion and,
    therefore, cwnd (and ssthresh) MUST be lowered twice in this case.

It's hard to be more specific because we didn't want to contradict other 
congestion control schemes (e.g. some respond to at least one mark per 
RTT, others respond to the extent of CE marking).

>
>   * *Tail loss probe *- I think it will also be useful to include some
>     text about CE marking on probe packets that a connection might
>     send in response to tail loss
>     (draft-dukkipati-tcpm-tcp-loss-probe-01.txt).
>
[BB] Well, I would like to resist mission creep. The scope of our draft 
is intended to be about setting ECT on packets where it was previously 
prohibited.

We have included interactions with SCTP, TFO and IW10 in "5. Interaction 
with popular variants or derivatives of TCP 
<https://tools.ietf.org/html/draft-bagnulo-tcpm-generalized-ecn-04#section-5>" 
because they have been published and can no longer be easily changed.

For TLP, it's not even adopted in TCP yet (although I'm sure it is 
widely used). So I think it's up to the authors of this draft to say 
whether ECT should be set on a TLP and what the response to a CE-marked 
TLP should be.

>
>   * *Implementation or deployment related comments - *The document
>     refers to other drafts or experimental RFCs as possible actions
>     that can be taken in several scenarios. For example AccECN, RFC
>     5690, RFC 7567 etc,. Even though implementing and deploying these
>     specifications is possible, they may not be implemented unless
>     these specifications are tightly bound together. A casual reader
>     might understand that implementing multiple of these RFCS will
>     make the system work well. But for someone to implement and deploy
>     these specifications, it is not clear if some or all of these
>     implementations will need to be supported. It will be good to make
>     a recommendation for the minimum set of specifications that need
>     to be implemented — may be after some more discussion.
>
[BB] If my co-author agrees, we can do this. I would recommend the 
following complements (both normatively referenced):
* AccECN
* RFC5691 (robustness to blind in-window attacks).

As a reciprocal recommendation, currently, the Intro to the AccECN draft 
says:

    It is likely (but not required) that the AccECN protocol will be
    implemented along with the following experimental additions to the
    TCP-ECN protocol: ECN-capable TCP control packets and retransmissions
    [I-D.bagnulo-tcpm-generalized-ecn 
<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-03#ref-I-D.bagnulo-tcpm-generalized-ecn>], which includes the ECN-capable
    SYN-ACK experiment [RFC5562 <https://tools.ietf.org/html/rfc5562>]; and testing receiver non-compliance
    [I-D.moncaster-tcpm-rcv-cheat 
<https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-03#ref-I-D.moncaster-tcpm-rcv-cheat>].

The following are good, but I don't think they qualify as complementary 
to ECN++:
* RFC5690 (ACKcc). It is merely optional and "not incompatible".
* Similarly for TFO and IW10. Optional but not incompatible.
* RFC7567 is about AQM in the network, so I don't it would help an 
implementer to say it is complementary, as host implementers cannot 
control where their implementation is used.
* Originally we were going to recommend RFC5562 (ECN+ i.e. ECT on 
SYN-ACK packets), but once we realized it was over-conservative, we 
wrote a less conservative alternative directly into this ECN++ draft.

More generally, I know some people are starting to get really confused 
over how to put together all the piecemeal modifications to ECN. I think 
that would require an informational "ECN roadmap" draft. No promises, 
but I guess I am someone who would be well-placed to know what to write 
in such a draft.

>
>  *
>     *Fallback mechanisms *- The document also discusses several
>     fallback mechanisms to handle incompatibilities in the network. If
>     we want implementations to consider these cases, it will be useful
>     to formalize these detection and fallback mechanisms. I found that
>     some of these mechanisms are being discussed in section 4 which
>     was tagged as informative.
>
[BB] I have just checked, and I cannot find any case where there is 
fall-back text in section 4 that ought to be normative. If you found 
any, pls say where. We tried to ensure that the recommended approach was 
always in section 3, even if section 4 discussed (then rejected) 
alternatives.

> Overall, the document is written well and discusses multiple aspects 
> of the proposed changes both on the end hosts and in the network. It 
> is useful to have discussions on the proposed changes and to drive 
> measurements that are needed to collect data to support or validate 
> some of the assumptions made.

[BB] Thanks.

In summary, as this conversation progresses, we might agree further 
changes. However, so far I have queued up the following list of changes 
to the text to address your points:
* Intro: Recommend complementary RFCs or drafts
* 3.2.1.3: Clarify that using IW=1SMSS only applies when using the 
strategies recommended in the previous two sections.
* When fall-back due to ECT on SYN (3.2.1.4) or SYN-ACK (3.2.2.3), 
consider disabling ECT on other control packets (esp. pure ACKs).
* Add detection of black-holing of pure ACKs as an open issue (hence why 
we have to take the approach in the previous bullet)
* Add extra section on general fall-back if no ECT packet has ever been 
ack'd
* And we will certainly acknowledge your contribution, and I hope you 
will continue to track changes to this draft carefully and let us know 
if you have implementation experience - thank you.

Bob

>> <[email-masked] <mailto:[email-masked]>>
>> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] 
>> <mailto:[email-masked]>>
>> Cc: "[email-masked] <mailto:[email-masked]>" 
>> <[email-masked] <mailto:[email-masked]>>
>> <[email-masked] 
>> <mailto:[email-masked]>>
>>
>> As requested by the authors, this e-mail starts a working group 
>> acceptance call for draft-bagnulo-tcpm-generalized-ecn-04, which will 
>> run on the list until June 30.
>>
>>  Submit document on adding Explicit Congestion Notification (ECN) to 
>> TCP Control Packets to the IESG for publication as Experimental RFC
>>
>> If you believe that TCPM should work on a document in this space, and 
>> in particular if you are willing to review it, please reply to this 
>> e-mails. Please also speak up if you have any concerns or objections.
>>
>> Thanks
>>
>> Michael
>> URL: 
>> <https://mailarchive.ietf.org/arch/browse/tcpm/attachments/20170616/151f93e9/attachment.html>
>>
>> [email-masked] <mailto:[email-masked]>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 19 Jun 2017 19:38:43 +0100

Padma, [re-sending - somehow my email client missed you off the distr.]

[BB] This is true.

    3.2.8 General Fall-Back

RFC5681 already says:

[BB] Thanks.

Bob

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/

---

From: Padma Bhooma <[email-masked]>
Date: Wed, 28 Jun 2017 07:53:52 -0700

Hi Bob

Thanks for your responses. Please see inline.

> On Jun 19, 2017, at 11:38 AM, Bob Briscoe <[email-masked]> wrote:
> 
>> Section 3.2.1.3 - If the SYN-ACK does not support AccECN, TCP must conservatively reduce its IW and SHOULD reduce it to 1 SMSS. This clause is very conservative because AccECN support can be deployed on servers slowly. Even if both end hosts support AccECN, there are networks with performance proxies that will not allow the feedback to propagate correctly. On those networks, TCP connections with ECT on SYN will always start with 1MSS of IW. I think this is too conservative and might add multiple RTT to grow the window. I agree with the value in adding ECT marking to SYN and SYN-ACK in terms of the latency benefits that might bring to an end user.
> [BB] We have structured the document to just recommend the preferred approach in the normative spec (section 3), then discuss the rationale and possible alternatives in section 4. The sections just before 3.2.1.3 recommend:
> This is the "optimistic ECT and cache failures" strategy discussed in 4.2.1 (tagged as 'S2B').
> 
> The requirement to reduce IW to 1SMSS only applies if the client is adopting this strategy and discovers the server does not support AccECN. Perhaps we didn't make that clear?

[PB] No I think this point was clear.

> 
> We have given 3 reasons for why this conservatism will rarely harm performance:
> 
> 1) With this strategy, IW=1SMSS will only be needed the first time a client accesses a non-AccECN proxy or server.
> 
> 2) During transition, this conservative IW=1SMSS will only ever be necessary for the client to server half-connection, not server to client. 
> According to Yuchung Cheng, client Web requests are rarely greater than 1 segment (no data openly available). Do you have any data to corroborate (or contradict) this? I think in non-Web apps, the initial client request would often fit within 1 packet too (e.g. email, ftp, ssh, scp, etc).
> 
> 3) Fall-back to 1SMSS is written as SHOULD rather than MUST. And refers to text later that says under what conditions you can use a higher IW.
> 
>> I think “pessimistic ECT and cache successes” strategy described in 4.2.1 sounds like a reasonable one.
> [BB] What's your reasoning for preferring this one? We recommended "optimistic ECT and cache failures”.

[PB] I think, 1MSS may not be enough for clients that are not web based. This assumption may not apply to connections that multiplex data on a single connection. Starting at 1MSS will make the client need more RTTs to reach the congestion window that is needed if the client wants to send more than an HTTP get. From the client side, the number of servers they connect to can be large (one-to-many) and using 1MSS on every one of them can effect performance. When there are performance proxies or other kind of middle boxes in the network that will not make AccECN work end-to-end, trying optimistic ECT can potentially have a performance hit.

> 
>> 
>> Section 3.2.3 ECT marking on Pure Acks - It felt like the document suggested further discussion on setting ECT on pure Acks, please correct me if I am wrong here. Setting ECT on pure Acks can increase the load on an AQM that might already be under load. This is because the number of Acks can be large and the information conveyed to the peer is redundant because asks are cumulative. TCP acknowledgements also do not solicit a response from the peer most of the times. So the chances of not echoing CE on an Ack is high. 
> BB: An AQM processes packets of all sizes anyway. If pure ACKs are not ECT, it still has to decide whether to drop each pure ACK. Marking instead of dropping doesn't make any more work for the AQM.

[PB] Here by "load" I meant that the queue sizes can grow and increase the probability of a drop for all traffic. I agree that if the queue sizes are manageable it will be great to not drop Acks also. 

> 
> PB: The information conveyed to the peer is redundant because ACKs are cumulative.
> BB: Just because the ACKnumber info is superseded by the next ACK does not mean the ECN marking is superseded.

[PB] Yes, agreed. That is also true for other information like SACK that can be on an ACK.
> 
> PB: TCP acknowledgements do not solicit a response from the peer most of the times. So the chances of not echoing CE on an Ack is high.
> BB: Just because pure ACKs do not solicit a specific response (no ACK of an ACK) does not mean the data sender is not continuing to send data packets, on which ECE can be set (if RFC3168 feedback) or on which the CE counter can be increased (AccECN feedback).
> 
> Consider this pure ACK example with data solely in the S->C direction, where S is the server.
>   * Say 3 in 1000 pure ACKs in the C->S direction are CE-marked, meaning the C->S marking probability is 0.3%
>   * This might be because 0.3% of data packets in other flows (e.g. a YouTube upload) are being CE-marked as well.
>   * So S needs to tell C that it has seen 3 CE marked packets (which it will do on the data packets it is sending, whether with RFC3168 feedback or AccECN feedback).
>   * C is not (currently) sending any data to S, but it will reduce its cwnd for if it does start sending data.

[PB] I think, it will be good to make AccECN feedback a strong requirement for propagating CE marking on Acks also. Imagine a bottleneck link that is congested due to a large flow and a few other flows sending Acks alone. Then propagating CE marking on Acks can make flows that will never send respond to congestion unnecessarily. 

> 
>> It is also difficult to detect if an Ack is dropped by an incompatible middle box because of ECT marking on it since we do not retransmit Acks.. So the fallback mechanisms will be limited. 
> [BB] You're correct in all respects here. Nonetheless, I have three responses:
>   * We have been doing millions of measurements over fixed and mobile networks, using a machine we control on both sides and checking a sent ECT packet is also received. So far, we have not seen any difference in the treatment of any control packet if set to ECT vs not-ECT. 
>   * Section 6.1.4 of RFC3168 <https://tools.ietf.org/html/rfc3168#section-6.1.4> hinted that in future pure ACKs might be set to ECT. So if a security middlebox is trying to enforce RFC3168, it still might allow ECT pure ACKs to pass.
>   * Nonetheless, if there were a middlebox that dropped ECT pure ACKs, you are right that it could black-hole all ECT pure ACKs and stall a connection. And there would be no easy way to detect the specific cause of the problem and stop setting ECT on pure ACKs. 
>   * We should at least say that if fall-back has to be used because ECT SYNs or SYN-ACKs are not getting through, then ECT ought not to be used on pure ACKs either.

[PB] Yes, that will be a good addition to the fallback mechanisms.

> 
> This is a good reason for why this protocol is experimental. However, given we have not so far found this problem in practice, I think it's OK to go ahead. But I agree that we should note that this is an open issue. 
> 
>> In Section 4.4.1 (Cwnd response to CE-marking on Pure Acks), the document says that we can echo a CE received on a pure ack on a data segment that will be sent later. In general, I think any congestion notification is relevant only for a short period of time probably in the order of a typical RTT. Once that time window passes, it may not be correct to respond to a delayed congestion notification. So echoing CE from a pure Ack on a data segment sent much later may not be the correct thing to do.
> [BB] This is true. 
> However, this point applies generally to idle periods, not just when pure ACKs are marked. TCP can always enter an idle period just after a loss or CE mark is received, then if later the sender has something to send, cwnd will be lower. Cwnd is already reduced exponentially during an idle period. 
> 
> This is all part of the fairly arbitrary rules on how to evolve cwnd during an idle period. Nonetheless, while idling, I think it is sensible to reduce cwnd more, if more of any pure ACKs sent during the idle period are picking up congestion markings. I don't think we should recommend the converse: i.e. if the congestion was a long time ago, we should not recommend that cwnd can be increased again.

[PB] Yes, I agree. It might be  a good idea to refer to work on idle periods here because this connection was not obvious to me :)

> 
> However, this is an area where I think implementers should be allowed to experiment, and I think the specs allow enough flexibility to do so.
> 
> 
>> 
> [BB] On the other hand, if the window was constrained solely because the receive window was low and there had been a CE mark on a window probe many round trips ago, but not on more recent window probes, it would not be correct to reduce cwnd such a long time after the congestion had occurred.
> 
> I think it would be better to reduce cwnd immediately there is a CE on a window probe, increase cwnd as normal for every non-CE-marked window probe (even if rwnd is still the limiting factor). But also rely on cwnd validation so that cwnd cannot grow excessively if it is not being used.

[PB] I think, it will be good to have some clarifying text here. Both the points that I made above (about Section 3.2.4 and Section 4.4.1) are kind of related to connections that see CE marking on control packets but do not have a lot of data to send or they are limited by receive window. The mechanisms for maintaining congestion window correctly when a CE marking is received in these scenarios is not well documented in RFC 3168 or in this draft. I agree that there are congestion window validation mechanisms (RFC 7661) that might apply here. So it will be good to give some recommendation here.

> [BB] Good point. I had figured that fall-back only needed to be considered for the SYN or SYN-ACK. But you're right that, if the SYN is not-ECT, fall-back might be needed later (altho, as already mentioned, we have not seen a need in measurements so far). This will generally only apply to the C->S half-connection. 
> 
> This is a general point about fall-back that happens to be noticed when both data and retransmissions fail to get through. So I think we ought to add a general point about fall-back as a new subsection of section 3, rather than writing this in the retransmissions subjection. I propose this text:
> 3.2.8 General Fall-Back
> 
> Consider a host that has sent the first packet on a half-connection (SYN or SYN/ACK) without ECT set and it has been acknowledged. If this host's retransmission timer expires more than once for all subsequent packet(s) it sends with ECT set, it SHOULD retransmit the packet(s) with not-ECT set. If these not-ECT packet(s) are acknowledged without any further losses, it is RECOMMENDED that the host disables setting ECT on all subsequent packets of the half-connection. It could also cache this failure.
> 

[PB] Yes, this is good.

> It's hard to be more specific because we didn't want to contradict other congestion control schemes (e.g. some respond to at least one mark per RTT, others respond to the extent of CE marking).

[PB] I agree. I did not fully grasp that text. I think this is good.

> [BB] Well, I would like to resist mission creep. The scope of our draft is intended to be about setting ECT on packets where it was previously prohibited.
> 
> We have included interactions with SCTP, TFO and IW10 in "5.  Interaction with popular variants or derivatives of TCP <https://tools.ietf.org/html/draft-bagnulo-tcpm-generalized-ecn-04#section-5>" because they have been published and can no longer be easily changed.
> 
> For TLP, it's not even adopted in TCP yet (although I'm sure it is widely used). So I think it's up to the authors of this draft to say whether ECT should be set on a TLP and what the response to a CE-marked TLP should be.

[PB] Ok, sounds good.

> [BB] If my co-author agrees, we can do this. I would recommend the following complements (both normatively referenced): 
> * AccECN 
> As a reciprocal recommendation, currently, the Intro to the AccECN draft says:
>    [I-D.bagnulo-tcpm-generalized-ecn <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-03#ref-I-D.bagnulo-tcpm-generalized-ecn>], which includes the ECN-capable
>    [I-D.moncaster-tcpm-rcv-cheat <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-03#ref-I-D.moncaster-tcpm-rcv-cheat>].
> 
> 
> The following are good, but I don't think they qualify as complementary to ECN++:
> * RFC7567 is about AQM in the network, so I don't it would help an implementer to say it is complementary, as host implementers cannot control where their implementation is used.
> * Originally we were going to recommend RFC5562 (ECN+ i.e. ECT on SYN-ACK packets), but once we realized it was over-conservative, we wrote a less conservative alternative directly into this ECN++ draft.

[PB] Are there any changes needed on the AQM side in the network to support CE marking on control packets? I do not think so but it will be good to get a clarification.

> 
> 
> More generally, I know some people are starting to get really confused over how to put together all the piecemeal modifications to ECN. I think that would require an informational "ECN roadmap" draft. No promises, but I guess I am someone who would be well-placed to know what to write in such a draft.
> 

[PB] Yes, this is true, especially with the latest experimentation around RFC 3168. It will be good to have an ECN roadmap.

> [BB] I have just checked, and I cannot find any case where there is fall-back text in section 4 that ought to be normative. If you found any, pls say where. We tried to ensure that the recommended approach was always in section 3, even if section 4 discussed (then rejected) alternatives.

[PB] Ok. Some measurements in the field will help to support the need for these fallback mechanisms. I wonder if there is a way to make them as "strong recommendations” so that the implementors do not think of them as optional.

> In summary, as this conversation progresses, we might agree further changes. However, so far I have queued up the following list of changes to the text to address your points:
> * 3.2.1.3: Clarify that using IW=1SMSS only applies when using the strategies recommended in the previous two sections.
> * When fall-back due to ECT on SYN (3.2.1.4) or SYN-ACK (3.2.2.3), consider disabling ECT on other control packets (esp. pure ACKs).
> * Add detection of black-holing of pure ACKs as an open issue (hence why we have to take the approach in the previous bullet)
> * Add extra section on general fall-back if no ECT packet has ever been ack'd
> * And we will certainly acknowledge your contribution, and I hope you will continue to track changes to this draft carefully and let us know if you have implementation experience - thank you.

Sounds good. Thank you!

Padma

>>> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
>>> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
>>> 	<[email-masked] <mailto:[email-masked]>>
>>> 	
>>> URL: <https://mailarchive.ietf.org/arch/browse/tcpm/attachments/20170616/151f93e9/attachment.html <https://mailarchive.ietf.org/arch/browse/tcpm/attachments/20170616/151f93e9/attachment.html>>
>>> 
>>> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>
>>> 
>>> 
>> https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>
> 
> -- 
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>