Subject: [tcpm] ECN++: Whether to allow ECT/CE marking of pure ACKs and what response to recommend to feedback reporting that a pure ACK was CE marked.

From: marcelo bagnulo braun <[email-masked]>
Date: Mon, 9 Oct 2017 12:34:02 +0200

Hi,

Regarding, draft-ietf-tcpm-generalized-ecn, we still have the open issue 
regarding whether to allow ECT/CE marking in pure acks and what response 
to reccomend if a pure ack encountered congestion.

After considering the discussion in the last meeting and in the ml, we 
propose the following text in the draft:

---------------------

For the experiments proposed here, the TCP implementation will set
ECT on pure ACKs.  It can ignore the requirement in section 6.1.4 of
RFC 3168 to set not-ECT on a pure ACK.

A host that sets ECT on pure ACKs SHOULD respond to the congestion 
signal resulting from pure ACKs being marked with the CE codepoint. The 
specific response will need to be defined as an update to each 
congestion control specification. Possible responses to congestion 
feedback include regulating the pure ACK rate (see AckCC [RFC5690])  or 
reducing the congestion window (CWND).

However, if the congestion response implemented reduces the CWND upon 
the report of CE marking of Pure ACKs, then the congestion control 
algorithm must also increase the CWND upon delivery of pure ACKs without 
a CE mark, to avoid a bias in the congestion control algorithm.

(We can also include some of the description of how to do this based on 
the text below for RFC3168 and AccECN in an appendix.)

---------------------

Some background about this proposal:

First of all, it's worth reminding that without ECN++, TCP currently is 
not required to detect loss of pure ACKs and is not required to react to 
them.
The current draft states that it is allowed to ECT/CE mark the pure ACKs 
and that:
"   A host that sets ECT on pure ACKs MUST reduce its congestion window
    in response to any congestion feedback, in order to regulate any data
    segments it might be sending amongst the pure ACKs.  It MAY also
    implement AckCC [RFC5690] to regulate the pure ACK rate, but this is
    not required. "

The feedback we received in the Prague meeting was:

- It is valuable to allow ECT/CE marking of pure ACK to avoid high drop 
rate when there is congestion. There is a strong case for ECN-capable 
pure ACKs as they are critical for connection performance.

- The sender of the pure acks must respond (somehow) to the congestion 
signal resulting from getting CE marks in pure ACKs. Recommending not to 
respond to a received congestion signal seem like a bad recommendation 
to make in general and setting a bad precedent.

There are two possible responses: reduce CWND (somehow) and/or ACK 
congestion control (ACK CC) (as defined in RFC5690).

- The feedback regarding ACK CC is that it is too complex and with too 
many corner cases to mandate its implementation, so we should not 
mandate it.

- Regarding reducing the CWND, there were several arguments against 
doing this. A very compelling argument is that since we do not increase 
the CWND when pure ACKs are successfully delivered, if we reduce the 
CWND when they are CE marked, then we have a strong bias towards 
reducing the CWND. In particular, in the case the one endpoint is only 
sending pure ACKs, the likely result is that the CWND will be 1 after a 
while.

It is possible to address this issue by allowing the increase of CWND in 
case there are no CE marks in the pure ACKs.

A possible approach for this would be the following:

- if ECN++ is being used with RFC3168 ECN, then we simply do what is 
done with data packets, i.e. if during a RTT no ECE is received, the 
sender increases its CWND and if the sender receives one or more ECE 
marks, then it reduces its CWND. The only difference is that we include 
pure ACKs as well.

- if ECN++ is used with AccECN, the situation is slightly more complex 
because the sender of the pure ACKs needs to keep track of the pure ACKs 
that got CE marked and those that didnt. But the sender of the pure ACKs 
knows how many pure ACKs it has sent. The other endpoint includes pure 
ACKs in the packet count of CE marked packets that is reported using 
AccECN, then the sender of the pure ACKs can figure out how many pure 
ACKs have been CE marked and how many were not. It then can feed this 
information to the congestion control algorithm.

(The limitation of this approach is that lost pure ACKs are accounted as 
delivered pure ACKs without any CE marks, meaning that in the case of 
severe congestion, when pure ACKs are being dropped, the congestion 
control algorithm will include those lost pure ACK packets as delivered 
and increase the CWND the corresponding share. Since pure ACKs are small 
packets, it is not clear how much of a problem is this. Similarly (but 
worse), with 3168 feedback, even if detecting CE on pure ACKs, if pure 
ACKs are lost and there's no CE on a pure ACK within an RTT, cwnd will 
continue to increase. However, in both cases, it is still better with 
ECN++, because without ECN++ there was no response to either dropped or 
CE-marked pure ACKs.)

It might be useful for the congestion response to be proportionate to an 
estimate of the bytes marked rather than packets marked, by adding an 
assumed header size on the wire to the size of every segment (otherwise 
pure ACKs would have zero size and not require any response). This would 
be up to the implementation.

Note that the response to the congestion signal is outside the scope of 
draft-ietf-generalized-ecn, because it depends on the specific CC, e.g. 
Reno, Cubic, Compound, etc)

That being said, we should include some guidance of what type of 
response is sensible, particularly for the standards track CC [RFC5681].

thoughts?

Bob & Marcelo

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 9 Oct 2017 13:57:55 +0200

Hi Marcelo,

sorry for top-posting but I’m actually not sure where exactly to reply below because I think the situation is less complicated than you described it below.

First of all I think pure ACKs should not be ECT marked if AccECN is not supported. Losing an ACK is less bad than losing a data packet but you should not mark packets as ECT if there is no way to provide feedback to the sender if congestion was experienced. 

Then if you have AccECN it is actually not that much important if a pure ACK or a data packet was marked. If there is a congestion marking that means the link is congested and you should react to it. Now the question is what’s the right reaction, and there multiple cases here:

1) You send pure ACKs interleaved with data packets that means your congestion window might be large and any indicate of congestion should trigger a congestion control reaction accordingly.

2) You only send pure ACKs; in this case your congestion window should reduce to min_win sooner or later anyway; if there is congestion this might happen a bit quicker but if there is congestion and you already know about this, you should also not just resume sending with your previous rate.  Then if you are at min_cwnd you cannot reduce it any further, therefore you MAY consider using AckCC in this case, or more generally speaking apply some algorithm increase your delayed ack factor.

I really don’t think that increasing the congestion window based on ACK is even a possible option. You don’t get ACKs for ACKs so you never know if an ACK was successfully received. That means you don’t even have any information that would allow you to react on.

Mirja

> Am 09.10.2017 um 12:34 schrieb marcelo bagnulo braun <[email-masked]>:
> 
> Hi,
> 
> Regarding, draft-ietf-tcpm-generalized-ecn, we still have the open issue regarding whether to allow ECT/CE marking in pure acks and what response to reccomend if a pure ack encountered congestion.
> 
> After considering the discussion in the last meeting and in the ml, we propose the following text in the draft:
> 
> ---------------------
> 
> A host that sets ECT on pure ACKs SHOULD respond to the congestion signal resulting from pure ACKs being marked with the CE codepoint. The specific response will need to be defined as an update to each congestion control specification. Possible responses to congestion feedback include regulating the pure ACK rate (see AckCC [RFC5690])  or reducing the congestion window (CWND).
> 
> However, if the congestion response implemented reduces the CWND upon the report of CE marking of Pure ACKs, then the congestion control algorithm must also increase the CWND upon delivery of pure ACKs without a CE mark, to avoid a bias in the congestion control algorithm.
> 
> (We can also include some of the description of how to do this based on the text below for RFC3168 and AccECN in an appendix.)
> 
> ---------------------
> 
> First of all, it's worth reminding that without ECN++, TCP currently is not required to detect loss of pure ACKs and is not required to react to them.
> The current draft states that it is allowed to ECT/CE mark the pure ACKs and that:
> - It is valuable to allow ECT/CE marking of pure ACK to avoid high drop rate when there is congestion. There is a strong case for ECN-capable pure ACKs as they are critical for connection performance.
> 
> - The sender of the pure acks must respond (somehow) to the congestion signal resulting from getting CE marks in pure ACKs. Recommending not to respond to a received congestion signal seem like a bad recommendation to make in general and setting a bad precedent.
> 
> There are two possible responses: reduce CWND (somehow) and/or ACK congestion control (ACK CC) (as defined in RFC5690).
> 
> - The feedback regarding ACK CC is that it is too complex and with too many corner cases to mandate its implementation, so we should not mandate it.
> 
> - Regarding reducing the CWND, there were several arguments against doing this. A very compelling argument is that since we do not increase the CWND when pure ACKs are successfully delivered, if we reduce the CWND when they are CE marked, then we have a strong bias towards reducing the CWND. In particular, in the case the one endpoint is only sending pure ACKs, the likely result is that the CWND will be 1 after a while.
> 
> It is possible to address this issue by allowing the increase of CWND in case there are no CE marks in the pure ACKs.
> 
> - if ECN++ is being used with RFC3168 ECN, then we simply do what is done with data packets, i.e. if during a RTT no ECE is received, the sender increases its CWND and if the sender receives one or more ECE marks, then it reduces its CWND. The only difference is that we include pure ACKs as well.
> 
> - if ECN++ is used with AccECN, the situation is slightly more complex because the sender of the pure ACKs needs to keep track of the pure ACKs that got CE marked and those that didnt. But the sender of the pure ACKs knows how many pure ACKs it has sent. The other endpoint includes pure ACKs in the packet count of CE marked packets that is reported using AccECN, then the sender of the pure ACKs can figure out how many pure ACKs have been CE marked and how many were not. It then can feed this information to the congestion control algorithm.
> 
> (The limitation of this approach is that lost pure ACKs are accounted as delivered pure ACKs without any CE marks, meaning that in the case of severe congestion, when pure ACKs are being dropped, the congestion control algorithm will include those lost pure ACK packets as delivered and increase the CWND the corresponding share. Since pure ACKs are small packets, it is not clear how much of a problem is this. Similarly (but worse), with 3168 feedback, even if detecting CE on pure ACKs, if pure ACKs are lost and there's no CE on a pure ACK within an RTT, cwnd will continue to increase. However, in both cases, it is still better with ECN++, because without ECN++ there was no response to either dropped or CE-marked pure ACKs.)
> 
> It might be useful for the congestion response to be proportionate to an estimate of the bytes marked rather than packets marked, by adding an assumed header size on the wire to the size of every segment (otherwise pure ACKs would have zero size and not require any response). This would be up to the implementation.
> 
> 
> Note that the response to the congestion signal is outside the scope of draft-ietf-generalized-ecn, because it depends on the specific CC, e.g. Reno, Cubic, Compound, etc)
> 
> That being said, we should include some guidance of what type of response is sensible, particularly for the standards track CC [RFC5681].
> 
> thoughts?
> 
> Bob & Marcelo
>

---

From: marcelo bagnulo braun <[email-masked]>
Date: Tue, 10 Oct 2017 10:25:43 +0200

Hi Mirja,

Thanks for your reply.

I understand that what you are proposing is that we only allow ECT 
marking of pure ACKs when AccECN is negotiated and that upon the 
reception of a signal informing that a pure ACK was CE marked, the 
sender must reduce its CWND as with any other packet, is that so?

some further questions below....

> First of all I think pure ACKs should not be ECT marked if AccECN is not supported. Losing an ACK is less bad than losing a data packet but you should not mark packets as ECT if there is no way to…

I am not sure i understand why you say that when AccECN is not 
supported, there is not mean to provide feedback to the sender. Wouldnt 
the receiver simply set the ECE flag in packets going in the other 
direction upon the reception of a pure ACK with the CE mark?

> Then if you have AccECN it is actually not that much important if a pure ACK or a data packet was marked.

Actually, people make the argument that there is a difference between 
data packets and pure ACKs.

The concern people expressed it that reducing the CWND based on CE marks 
on pure ACK introduces a bias, since the sender reduces the CWND when 
Pure ACKs experience congestion but the sender does not increase the 
CWND when the pure ACK does not experience congestion. This is a 
fundamental concern, i believe and it is irrespective whether there is 
data traffic along with pure ACKs, i think.
In AccECN, the bias can be less, i think, because since the pure ACK has 
not data, the amount of bits that experienced congestion is small, so 
the bias is small. This is the reason why i think it is more acceptable 
to have this behaviour with AccECN than with no AccECN.

So, my understanding is that the options are
- not ECN marking of pure ACKs, which results in higher loss probability
- marking pure ACKs but not responding to congestion, bad precedent
- marking pure ACKs and responding to congestion, bias towards 
decreasing the CWND
- marking pure ACKs, responding to congestion and adding some way to 
increase the CWND, complicated and inaccurate.

So, from my side, I think your proposal of only supporting ECN marking 
of pure ACK when AccECN is supported seems a good trade-off, while not 
perfect, may be a good way forward

> If there is a congestion marking that means the link is congested and you should react to it. Now the question is what’s the right reaction, and there multiple cases here:
>

See above, the concern here is the biased introduced towards reducing 
the CWND.

> I really don’t think that increasing the congestion window based on ACK is even a possible option. You don’t get ACKs for ACKs so you never know if an ACK was successfully received. That means you…

You do get some idea of what ACKs got through, but i agree that given 
the cumulative nature of ACKs, some acks can get lost undetected.
I agree this is not perfect, but not of the options are.
That being said, I also think that this option introduces complexity 
with little benefit, so if all solutions are imperfect, we should at 
least choose one that is at least simple.

Regards, marcelo

---

From: Jeremy Harris <[email-masked]>
Date: Tue, 10 Oct 2017 12:40:47 +0100

> increase the CWND, complicated and inaccurate.

- marking pure ACKs and responding to congestion via CWND (bias towards
  decreasing the CWND) plus ACKcc

-- 
Jeremy

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 13 Oct 2017 12:44:01 +0200

Hi Marcelo,

see inline.

> direction upon the reception of a pure ACK with the CE mark?

Actually this is a good question and not well-specified in RFC3168. So my 
guess it that this is a no for most implementations but more research is needed.

> data traffic along with pure ACKs, i think.

There is huge difference here because if you only send pure ACKs your 
congestion would decrease to min-cwnd sooner or later anyway and you should 
not increase that just because you send ACKs continuously. In this scenario, 
increasing the congestion window without actually using it, is a bad idea.

However, usually you don't get any feedback that tells you that an ACK was 
received or not, so I don't even understand how you would want to increase 
the cwnd for ACKs...?

Anyway we seem to agree below on using ECN for pure ACKs only with AccECN, right?

Mirja

---

From: marcelo bagnulo braun <[email-masked]>
Date: Fri, 13 Oct 2017 13:42:14 +0200

> Anyway we seem to agree below on using ECN for pure ACKs only with 

I am ok with that, other peopel feel this is a good way forward?

Regards, marcelo

>>> If there is a congestion marking that means the link is congested 
>>> and you should react to it. Now the question is what’s the right 
>>> reaction, and there multiple cases here:
>>>
>>> 1) You send pure ACKs interleaved with data packets that means your 
>>> congestion window might be large and any indicate of congestion 
>>> should trigger a congestion control reaction accordingly.
>>>
>>> 2) You only send pure ACKs; in this case your congestion window 
>>> should reduce to min_win sooner or later anyway; if there is 
>>> congestion this might happen a bit quicker but if there is 
>>> congestion and you already know about this, you should also not just 
>>> resume sending with your previous rate. Then if you are at min_cwnd 
>>> you cannot reduce it any further, therefore you MAY consider using 
>>> AckCC in this case, or more generally speaking apply some algorithm 
>>> increase your delayed ack factor.
>>
>>> I really don’t think that increasing the congestion window based on 
>>> ACK is even a possible option. You don’t get ACKs for ACKs so you 
>>> never know if an ACK was successfully received. That means you don’t 
>>> even have any information that would allow you to react on.
>>
>>>> Regarding, draft-ietf-tcpm-generalized-ecn, we still have the open 
>>>> issue regarding whether to allow ECT/CE marking in pure acks and 
>>>> what response to reccomend if a pure ack encountered congestion.
>>>>
>>>> After considering the discussion in the last meeting and in the ml, 
>>>> we propose the following text in the draft:
>>>>
>>>> ---------------------
>>>>
>>>> signal resulting from pure ACKs being marked with the CE codepoint. 
>>>> The specific response will need to be defined as an update to each 
>>>> feedback include regulating the pure ACK rate (see AckCC [RFC5690]) 
>>>> or reducing the congestion window (CWND).
>>>>
>>>> However, if the congestion response implemented reduces the CWND 
>>>> upon the report of CE marking of Pure ACKs, then the congestion 
>>>> control algorithm must also increase the CWND upon delivery of pure 
>>>> ACKs without a CE mark, to avoid a bias in the congestion control 
>>>> algorithm.
>>>>
>>>> (We can also include some of the description of how to do this 
>>>> based on the text below for RFC3168 and AccECN in an appendix.)
>>>>
>>>> ---------------------
>>>>
>>>> First of all, it's worth reminding that without ECN++, TCP 
>>>> currently is not required to detect loss of pure ACKs and is not 
>>>> required to react to them.
>>>> - It is valuable to allow ECT/CE marking of pure ACK to avoid high 
>>>> drop rate when there is congestion. There is a strong case for 
>>>> ECN-capable pure ACKs as they are critical for connection performance.
>>>>
>>>> - The sender of the pure acks must respond (somehow) to the 
>>>> congestion signal resulting from getting CE marks in pure ACKs. 
>>>> Recommending not to respond to a received congestion signal seem 
>>>> like a bad recommendation to make in general and setting a bad 
>>>> precedent.
>>>>
>>>> - The feedback regarding ACK CC is that it is too complex and with 
>>>> too many corner cases to mandate its implementation, so we should 
>>>> not mandate it.
>>>>
>>>> doing this. A very compelling argument is that since we do not 
>>>> increase the CWND when pure ACKs are successfully delivered, if we 
>>>> reduce the CWND when they are CE marked, then we have a strong bias 
>>>> towards reducing the CWND. In particular, in the case the one 
>>>> endpoint is only sending pure ACKs, the likely result is that the 
>>>> CWND will be 1 after a while.
>>>>
>>>> It is possible to address this issue by allowing the increase of 
>>>> CWND in case there are no CE marks in the pure ACKs.
>>>>
>>>> - if ECN++ is being used with RFC3168 ECN, then we simply do what 
>>>> is done with data packets, i.e. if during a RTT no ECE is received, 
>>>> the sender increases its CWND and if the sender receives one or 
>>>> more ECE marks, then it reduces its CWND. The only difference is 
>>>> that we include pure ACKs as well.
>>>>
>>>> - if ECN++ is used with AccECN, the situation is slightly more 
>>>> complex because the sender of the pure ACKs needs to keep track of 
>>>> the pure ACKs that got CE marked and those that didnt. But the 
>>>> sender of the pure ACKs knows how many pure ACKs it has sent. The 
>>>> other endpoint includes pure ACKs in the packet count of CE marked 
>>>> packets that is reported using AccECN, then the sender of the pure 
>>>> ACKs can figure out how many pure ACKs have been CE marked and how 
>>>> many were not. It then can feed this information to the congestion 
>>>> control algorithm.
>>>>
>>>> (The limitation of this approach is that lost pure ACKs are 
>>>> accounted as delivered pure ACKs without any CE marks, meaning that 
>>>> in the case of severe congestion, when pure ACKs are being dropped, 
>>>> the congestion control algorithm will include those lost pure ACK 
>>>> packets as delivered and increase the CWND the corresponding share. 
>>>> Since pure ACKs are small packets, it is not clear how much of a 
>>>> problem is this. Similarly (but worse), with 3168 feedback, even if 
>>>> detecting CE on pure ACKs, if pure ACKs are lost and there's no CE 
>>>> on a pure ACK within an RTT, cwnd will continue to increase. 
>>>> However, in both cases, it is still better with ECN++, because 
>>>> without ECN++ there was no response to either dropped or CE-marked 
>>>> pure ACKs.)
>>>>
>>>> It might be useful for the congestion response to be proportionate 
>>>> to an estimate of the bytes marked rather than packets marked, by 
>>>> adding an assumed header size on the wire to the size of every 
>>>> segment (otherwise pure ACKs would have zero size and not require 
>>>> any response). This would be up to the implementation.
>>>>
>>>>
>>>> Note that the response to the congestion signal is outside the 
>>>> scope of draft-ietf-generalized-ecn, because it depends on the 
>>>> specific CC, e.g. Reno, Cubic, Compound, etc)
>>>>