From nobody Thu Jul 1 11:25:24 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E772D3A1406; Thu, 1 Jul 2021 11:25:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fj1ucvVQt18N; Thu, 1 Jul 2021 11:25:17 -0700 (PDT) Received: from mail-io1-xd33.google.com (mail-io1-xd33.google.com [IPv6:2607:f8b0:4864:20::d33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4A36D3A1407; Thu, 1 Jul 2021 11:24:59 -0700 (PDT) Received: by mail-io1-xd33.google.com with SMTP id l5so8674159iok.7; Thu, 01 Jul 2021 11:24:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=l2SkDBPpEhGBIPvBwQQSDJFeN33B/sJVdx0DupISQcM=; b=dqogZ3OOQq9NRwp7p9QddiYkn4xzEmIvNeJcCPbSHLbCheSB2aPcI3z1M6b9MZduiC XeyEZZLVlEF3nY156iBluqw5hwKZ9VKh5vdOEQbug91cMAT93yqkbC0+u1t1wTHLYdiE Lp690ku64XCykeqUiVz8MOFl7bOwbJanQHdYgQ6Dh2psOBAXTIBRVYRAsry3hW0S/4jO QyQI30eQJpHXYhdpRaPOjD41VNQWY+j5te5NG8gUznQHyh9+51DCDmxugFEDjN1+wetz 6oqlqV8BH5avo/2OYD4bzs4ANgyE97eYps2Q+skfwAmLmvN+QfRHH1M8qGLzKEkmZmIE 4fBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=l2SkDBPpEhGBIPvBwQQSDJFeN33B/sJVdx0DupISQcM=; b=csFQkQaGCVa4YrxvfxaQGOk22aSWJvWftonvBvcpEAXjes0xuiYH7svfcwy0hs5eVL GBfCZdC06elgD0+uToL0pHBFr1EXbh0HO6nkkW8l/Lzlqcr49yfmej89a0ZIOJevzd6i gWVpGhyXSbpkVC3fzSDDsbVk9HB0zJJZyM42iz/DDjsxhhVA/BFScSiLX2x/zScTjD2L hSIXhbrqfoKDJjwV6ZxIf8EPgQGAyX/x+TdRL0fWf1jxaDo21K6G+CnT8Jdu2P1IpSRD aWNvbmOWgjmXEOhFGxPy756tJxZ+eE6tnkfswQ8Z+F/TyuI81QdspvFoglsIGXsvDBga kQQg== X-Gm-Message-State: AOAM533e+ltHoqct0qUbIPTReXZm+HluUYX7XAYiLedZxf6p4z51sD7g YtfHPTKGhw0WVfA/qXAm0igC4QXI9j6JbRGc2BGRE2MWblQ= X-Google-Smtp-Source: ABdhPJwpAwP/XZPTVym8/jyDZTMnxVHadIAsnDpM08w/kSWpA0Bo3bTePmk9tFJd5YQJUoGgyV8H1SUN+Re3bgE6/iE= X-Received: by 2002:a02:3705:: with SMTP id r5mr1029021jar.144.1625163897356; Thu, 01 Jul 2021 11:24:57 -0700 (PDT) MIME-Version: 1.0 References: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> In-Reply-To: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> From: Martin Duke Date: Thu, 1 Jul 2021 11:24:47 -0700 Message-ID: To: Wesley Eddy Cc: draft-ietf-tcpm-rfc793bis.all@ietf.org, "tcpm@ietf.org Extensions" Content-Type: multipart/alternative; boundary="000000000000cdc56805c613f422" Archived-At: Subject: Re: [tcpm] RFC793bis AD review X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Jul 2021 18:25:22 -0000 --000000000000cdc56805c613f422 Content-Type: text/plain; charset="UTF-8" On Wed, Jun 30, 2021 at 2:50 PM Wesley Eddy wrote: > Here are my replies on each finding, with proposed changes to make sure > you agree, before cutting another revision of the document: > > > On 6/24/2021 4:12 PM, Martin Duke wrote: > > > (Document Header) Please update the Intended Status to "Internet Standard". > > There doesn't seem to be a way to force this in xml2rfc3; the 'category' > attribute is set to 'std' (Standards Track) and there's no distinction > (this looks consistent with other recent Internet Standard RFCs). > > So, I think no change is needed on this. > OK > > (3.8.3) R1 and R2 can be measured in retransmissions or seconds; the > proposed bounds are in retransmissions for R1 and seconds for R2. So if I > have a time-based R1, should that be 3? If I have a retransmission based > R2, should I compare that to the value of RTO somehow? > > This is heritage from RFC 1122. The text on R1 mentions "at the current > RTO", so I think RO is definitely the intended conversion factor, but since > you asked, maybe it's not clear enough? Do you think maybe in part (a) of > 3.8.3 we could just change: > > R1 and R2 might be measured in time units or as a count of > retransmissions. > > to: > > R1 and R2 might be measured in time units or as a count of > retransmissions (with the current RTO as a conversion factor, if needed). > > I like that, thanks. > > (3.8.5) This section is unclear as to whether the sender can cause the > Urgent Pointer to recede (move to the left), and if so, if that MUST be > reported to the application like a pointer advance. > > Receding looks the same as advancing so much that it wraps. So, if not > already in urgent mode, that would seem to cause a change to urgent mode, > while if not already in urgent mode, that would fall under the existing > rule of: > > If the urgent pointer is updated while the user is in "urgent mode", > the update will be invisible to the user. > > So ... also given that the use of urgent pointer is deprecated anyways and > we don't want to cause changes to TCP in this doc, I'm proposing we do > nothing to clarify this, if you agree? > I don't think SEQ 5000 URG 3000 SEQ 5001 URG 1500 would have a common-sense interpretation that the urg ptr was now 7501 + 2^32. I share your grumpiness about sweating the useless urgent pointer, but it would appear there is a gap in the spec and that in principle these recedes could exist in the wild. This spec mostly matters for new implementations; maybe the safest thing is for those new implementations to prepare to handle a receding pointer, but that senders MUST NOT recede. I would welcome a quick question to the WG if that logic seems spurious to you. Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have If the urgent pointer is updated while the user is in "urgent mode", the update will be invisible to the user. A TCP implementation MUST (MUST-32) inform the application layer asynchronously whenever it receives an Urgent pointer and there was previously no pending urgent data, or whenever the Urgent pointer advances in the data stream. If the latter text is correct, AND we decide to allow receding, we should decide what to do with recedes here. > (3.8.6.1) > > The sending TCP peer must be prepared to accept from the user and > send at least one octet of new data even if the send window is zero. > The sending TCP peer must regularly retransmit to the receiving TCP > peer even when the window is zero > > Please clarify that these probes only happen when there is new data, if > that's the intent. > > The "accept from the user" part means that new data was written already, > right? If the user doesn't write data into the socket, there's nothing to > be sent. > I have been messing with QUIC too long and may have forgotten how this works. But IIRC, this text kind of confuses a few different kinds of transmissions with zero window probes. It might be helpful to actually define "zero window probe" -- a segment of one byte of data designed to elicit an ack with a window update. The retransmission discussion is related but somewhat confusing as currently written. There are four zero-window scenarios that I can see. 1) If there is outstanding in-window data, we will get ACKs so there is no need for a ZWP. 2) No send buffer data, but the application generates data while in zero-window. Clearly, the spec says TCP accepts a byte and sends it. 3) TCP has previously refused application data due to window restrictions, and now we're in zero-window. TCP needs to wake the app to send again so that we have material for ZWP. 4) TCP accepted some data, held it for Nagle, and then the receiver reneged; or all in-flight data is now out of window due to reneging. This data ought to constitute a ZWP whether or not it is strictly a "retransmission". If you believe all of my statements correct, I am happy to propose alternate text that is not as wordy as the above., (3.8.6.1) the first paragraph says Two minutes is recommended for the retransmission interval when the window is zero. but the last one says The transmitting host SHOULD send the first zero-window probe when a zero window has existed for the retransmission timeout period (SHLD- 29) (Section 3.8.1 ), and SHOULD increase exponentially the interval between successive probes (SHLD-30). so is it 2 min or the RTO? Good question ... The first sentence comes from 793 and the other from 1122. I think we should erase the first sentence, since the 1122 intent seems to be updating this (replacing the hard-coded 2 minute heuristic with the RTO one which adapts to reality). Do you agree? Yes, I agree. > > (3.8.6.2.1) If I read this algorithm correctly, an un-pushed send buffer of < MSS, if D < Fs*Max(SND.WND), will sit there indefinitely without > being transmitted; is this intentional? > > This comes from 1122. I think the key to preventing what you mention is > this part of the preceding paragraph: > > To avoid a resulting deadlock, it is necessary to have a > timeout to force transmission of data, overriding the SWS avoidance > algorithm. > > So, SWS avoidance is not supposed to keep a sender waiting forever for the > receiver to possibly advertise a bigger window; the sender is supposed to > wake up and just send eventually. > Yes, but the override timeout is embedded in step (4), which applies PUSH as a condition. Maybe that PUSH condition is incorrect? > > (3.8.2) and (3.8.6.3) Should RFC 5681 be a normative reference, given the > congestion control requirement? If not, (3.8.6.3) shouldn't require the > reader to go there to fully implement delayed ACKs. I suggest > > s/an ACK for at least every second segment/an ACK for least every 2*MSS > bytes of data > > to fully state what is in 5681. > > Yeah, I think 5681 should be normative (since we also rely on the for > descriptions of slow start, etc. that we didn't try to duplicate within > 793bis), but we should also fix the "second segment" sentence to match more > exactly the text in 5681 about 2*RMSS. > OK, there's currently the pointer to 5681 about sub-MSS stuff, so it's technically correct, but really clunky to force people to go there when it's so easy to state correctly. I think we agree. > > > > > (3.10, OPEN/LISTEN) > > If active and the remote socket is specified, then change the > connection from passive to active, > > I think you mean "If passive"? > > Actually, I think to be more clear this should say: > > "If the OPEN call is active ...", since it's referring to the case where > an active OPEN is called when the socket is in LISTEN (passive). > > It's technically correct currently (just not very clear), if you read the > "If active" as referring to the OPEN call rather than the socket (and it > must be that way since a socket in the LISTEN state is the result of a > passive OPEN). > Yes, let's clarify > (3.10, SEGMENT ARRIVES/LISTEN) Step 4 mentions RST but those have already > been handled in Step 1. > > Yeah, you're definitely right, and in both cases the action is to ignore > it, so this part of step 4 is irrelevant, but at least consistent. > > It looks like this is an errata to file on 793; should we do that quickly > and then fix this doc accordingly? > If that is better for your process, then sure. > *NITS:* > > > > > (3.1, Options) " > > Options: [TCP Option]; Options#Size == (DOffset-5)*32; present only > when DOffset > 5. > > What is this notation? > > It is a little odd, but comes from reference [61] for > draft-mcquistin-augmented-ascii-diagrams-08. > OK > > > I'm good with changing all of these above. > > Thank you for the detailed review! > --000000000000cdc56805c613f422 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Jun 30, 2021 at 2:50 PM Wesle= y Eddy <wes@mti-systems.com&g= t; wrote:
=20 =20 =20
Here are my replies on each finding, with proposed changes to make sure you agree, before cutting another revision of the document:


On 6/24/2021 4:12 PM, Martin Duke wrote:
=20

(Document Header) Please update the Intended Status to "Internet Standard".

There doesn't seem to be a way to force this in xml2rfc3; the 'category' attribute is set to 'std' (Standards Track= ) and there's no distinction (this looks consistent with other recent Internet Standard RFCs).

So, I think no change is needed on this.

<= div>
OK
=C2=A0


(3.8.3) R1 and R2 can be measured in retransmissions or seconds; the proposed bounds are in retransmissions for R1 and seconds for R2. So if I have a time-based R1, should that be 3? If I have a retransmission based R2, should I compare that to the value of RTO somehow?

This is heritage from RFC 1122.=C2=A0 The text on R1 mentions "= at the current RTO", so I think RO is definitely the intended conversio= n factor, but since you asked, maybe it's not clear enough?=C2=A0 D= o you think maybe in part (a) of 3.8.3 we could just change:

=C2=A0=C2=A0=C2=A0 R1 and R2 might be measured in time units or as a= count of retransmissions.

to:

=C2=A0=C2=A0=C2=A0 R1 and R2 might be measured in time units or as a= count of retransmissions (with the current RTO as a conversion factor, if needed).

I like that, thanks.
= =C2=A0


(3.8.5)=C2=A0 This section is unclear as to whether the send= er can cause the Urgent Pointer to recede (move to the left), and if so, if that MUST be reported to the application like a pointer advance.

Receding looks the same as advancing so much that it wraps.=C2=A0 So= , if not already in urgent mode, that would seem to cause a change to urgent mode, while if not already in urgent mode, that would fall under the existing rule of:

=C2=A0=C2=A0=C2=A0 If the urgent pointer is updated while the user i= s in "urgent mode", the update will be invisible to the user.

So ... also given that the use of urgent pointer is deprecated anyways and we don't want to cause changes to TCP in this doc, I&= #39;m proposing we do nothing to clarify this, if you agree?

<= /blockquote>
I don't think

SEQ 5000 URG 30= 00
SEQ 5001 URG 1500

would have a common= -sense interpretation that the urg ptr was now 7501 + 2^32. I share your gr= umpiness about sweating the useless urgent pointer, but it would appear the= re is a gap in the spec and that in principle these recedes could exist in = the wild. This spec mostly matters for new implementations; maybe the safes= t thing is for those new implementations to prepare to handle a receding po= inter, but that senders MUST NOT recede. I would welcome a quick question t= o the WG if that logic seems spurious to you.

= Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have= 

If the urgent pointer is updated while the use=
r is in "urgent mode", the
   update will be invisible to the user.
A TCP implementation MUST (MUST-32) inform the=
 application layer
   asynchronously whenever it receives an Urgent pointer and there was
   previously no pending urgent data, or whenever the Urgent pointer
   advances in the data stream.
=C2=A0If the latter text is correct, AND we decide to allow rece= ding, we should decide what to do with recedes here.

(3.8.6.1)
The sending TCP peer must be prepared to accept from the use=
r and
   send at least one octet of new data even if the send window is zero.
   The sending TCP peer must regularly retransmit to the receiving TCP
   peer even when the window is zero
Please clarify that these probes only happen when there is new data, if that's the intent.

The "accept from the user" part means that new data was wr= itten already, right?=C2=A0 If the user doesn't write data into the soc= ket, there's nothing to be sent.

I have= been messing with QUIC too long and may have forgotten how this works. But= IIRC, this text kind of confuses a few different kinds of transmissions wi= th zero window probes. It might be helpful to actually define "zero wi= ndow probe" -- a segment of one byte of data designed to elicit an ack= with a window update. The retransmission discussion is related but somewha= t confusing as currently written. There are four zero-window scenarios that= I can see.

1) If there is outstanding in-wind= ow data, we will get ACKs so there is no need for a ZWP.

2) No send buffer data, but the application generates data while in = zero-window. Clearly, the spec says TCP accepts a byte and sends it.

3) TCP has previously refused application data due t= o window restrictions, and now we're in zero-window. TCP needs to wake = the app to send again so that we have material for ZWP.

4) TCP accepted some data, held it for Nagle, and then the receiver r= eneged; or all in-flight data is now out of window due to reneging. This da= ta ought to constitute a ZWP whether or not it is strictly a "retransm= ission".

If you believe all of my stateme= nts correct, I am happy to propose alternate text that is not as wordy as t= he above.,


(3.8.6.1) the first paragraph says
Two minutes is recommended for the retransmission interval w=
hen the
   window is zero.
but the last one says
The transmitting host SHOULD send the first zero-window prob=
e when a
   zero window has existed for the retransmission timeout period (SHLD-
   29) (Section 3.8.1), and SHOULD =
increase exponentially the interval
   between successive probes (SHLD-30).

so is it 2 min or the RTO?

Good question ... The first sentence come= s from 793 and the other from 1122.=C2=A0 I think we should erase the first sentence, since th= e 1122 intent seems to be updating this (replacing the hard-coded 2 minute heuristic with the RTO one which adapts to reality).=C2=A0 Do you agree?

Yes, I agree. 


(3.8.6.2.1) If I read this a=
lgorithm correctly, an un-pushed send buffer of < MSS, if D < Fs*Max(=
SND.WND), will sit there indefinitely without
being transmitted; is this intentional?

This comes from 1122.=C2=A0 I think = the key to preventing what you mention is this part of the preceding paragraph:

=C2=A0 To avoid a resulting deadlock= , it is necessary to have a
=C2=A0=C2=A0 timeout to force transmission of data, overriding the = SWS avoidance
=C2=A0=C2=A0 algorithm.

So, SWS avoidance is not supposed to keep a sender waiting forever for the receiver to possibly advertise a bigger window; the sender is supposed to wake up and just send eventually.


Yes, but the override timeout is embedded in step (4), which applie= s PUSH as a condition. Maybe that PUSH condition is incorrect?


(3.8.2) and (3.8.6.3) Should RFC 5681 be a normative reference, given the congestion control requirement? If not, (3.8.6.3) shouldn't require the reader to go there t= o fully implement delayed ACKs. I suggest

s/an ACK for at least every second segment/an ACK for least every 2*MSS bytes of data

to fully state what is in 5681.

Yeah, I think 5681 should be normative (since we also rely on the for descriptions of slow start, etc. that we didn't try to duplicate within 793bis), but we should also fix the "second segment" sentence to match more exactly the text in 5681 about 2*RMSS.


OK, there's= currently the pointer to 5681 about sub-MSS stuff, so it's technically= correct, but really clunky to force people to go there when it's so ea= sy to state correctly. I think we agree.


<snip>


(3.10, OPEN/LISTEN) 
If active and the remote socket is specified, then change th=
e
         connection from passive to active,
I think you mean "If passive"?

Actually, I think to be more clear this should say:

"If the OPEN call is active ...", since it's referring= to the case where an active OPEN is called when the socket is in LISTEN (passive).

It's technically correct currently (just not very clear), if you read the "If active" as referring to the OPEN call rather t= han the socket (and it must be that way since a socket in the LISTEN state is the result of a passive OPEN).


=
Yes, let's clarify
=C2=A0

(3.10, SEGMENT ARRIVES/LISTEN) Step 4 mentions RST but those have already been handled in Step 1.

Yeah, you're definitely right, and in both cases the action is t= o ignore it, so this part of step 4 is irrelevant, but at least consistent.

It looks like this is an errata to file on 793; should we do that quickly and then fix this doc accordingly?

=
If that is better for your process, then sure.
=C2=A0

NITS:
<snip>

(3.1, Options) " 
   Options: [TCP Option]; Options#Size =3D=3D (DOffset-5)*32=
; present only
   when DOffset > 5.
What is this notation?
It is a little odd, but comes from reference [61] for draft-mcquistin-augmented-ascii-diagrams-08.

OK


<snip>

I'm good with changing all of these above.

Thank you for the detailed review!

--000000000000cdc56805c613f422-- From nobody Thu Jul 1 18:15:51 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 025E73A1292 for ; Thu, 1 Jul 2021 18:15:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -17.6 X-Spam-Level: X-Spam-Status: No, score=-17.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KVYVz7XVSHtr for ; Thu, 1 Jul 2021 18:15:44 -0700 (PDT) Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 32F383A128C for ; Thu, 1 Jul 2021 18:15:44 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id l1so5631220wme.4 for ; Thu, 01 Jul 2021 18:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=xmzfJqFxQVQAedPLtYPOi4qeqQwkDMeRHWqAepq+LN8=; b=UHKXZWA2sg6lXMMZ1Z+ZL3kx+UnGC7tA4r7RVTw30XaIJnE0yHU481zD/JbUx0SFiu DUtiwkS+iYgp0ZeX8vwrmtpdZmMQkxd8C+k2AhFtuR1CjlhscbHfaUncZbT5fqdg49vL kO45OAB4akbxHlpr0wNTTynxdRaUBWSQKuFmoKcpiy9FaRN1ykhSzTMxD1wEqQ90fcTv SL4/yRr8ztMlLqvSj9WMekIUBVziMmS7+rPMSAtsiNwS/GjYHPlGz1EzqPzjJ4pOKuZ4 T6TcjzAFXiiAK9sD3yb0f44yXwOH1kSnE05qYEx8AeqREoDYolGwOpScqiaIkuiNEZXB LToQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=xmzfJqFxQVQAedPLtYPOi4qeqQwkDMeRHWqAepq+LN8=; b=C29gbiL8voiFZHZP6daCpkOHt2+c9sYE8oJUwTRUqGTkHT1nurfYCnbLkE/Ejx1T5F 7LqiL2rTBDZO2rkyYn05zOJHzheERcRQ4gvlouWM+JCo8+k0HfPOLvr3HwEBkU1QE2Lb jo2Inqi0hhDtr4/pj7gReQuB15XChko6+sKUb4WgDuut1TYXVgt6a4JeZLzt55H9lacd LE96ag3A4/KAIsCY4kS14fWsDefe9IZg2wosu7GUF5b3H/WNmq/epkr/OmdxKj3EzqOH Tj1UEtANUBdPaLbSUbRduSw9rmaxu5PV96GQ2v6EyAcyN8zYTlEjg/9PzuQ4/iliAQqX WFoQ== X-Gm-Message-State: AOAM531ASz8jXx8rBCZS6feRBorNAYn2f/MorZcRhLssxRWNM99IqFbY YgR6xe7NmBvUZJ1RW7pk8xB4Oj7CtJjC0G/WRqlnEw== X-Google-Smtp-Source: ABdhPJxwTw4iOPpqNKiwPY7Kd7N31fGDsdWefZi1xl3qiJN9DwEhGpdc6DoXRYzBovBuGGiR8FAkVwZxYdG8CKEvaOI= X-Received: by 2002:a1c:7204:: with SMTP id n4mr13815520wmc.7.1625188537337; Thu, 01 Jul 2021 18:15:37 -0700 (PDT) MIME-Version: 1.0 References: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> In-Reply-To: From: Yuchung Cheng Date: Thu, 1 Jul 2021 18:15:00 -0700 Message-ID: To: Martin Duke Cc: Wesley Eddy , "tcpm@ietf.org Extensions" , draft-ietf-tcpm-rfc793bis.all@ietf.org, Neal Cardwell Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Archived-At: Subject: Re: [tcpm] RFC793bis AD review X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2021 01:15:49 -0000 On Thu, Jul 1, 2021 at 11:25 AM Martin Duke wrote= : > > > > On Wed, Jun 30, 2021 at 2:50 PM Wesley Eddy wrote: >> >> Here are my replies on each finding, with proposed changes to make sure = you agree, before cutting another revision of the document: >> >> >> On 6/24/2021 4:12 PM, Martin Duke wrote: >> >> >> (Document Header) Please update the Intended Status to "Internet Standar= d". >> >> There doesn't seem to be a way to force this in xml2rfc3; the 'category'= attribute is set to 'std' (Standards Track) and there's no distinction (th= is looks consistent with other recent Internet Standard RFCs). >> >> So, I think no change is needed on this. > > > OK > >> >> >> (3.8.3) R1 and R2 can be measured in retransmissions or seconds; the pro= posed bounds are in retransmissions for R1 and seconds for R2. So if I have= a time-based R1, should that be 3? If I have a retransmission based R2, sh= ould I compare that to the value of RTO somehow? >> >> This is heritage from RFC 1122. The text on R1 mentions "at the current= RTO", so I think RO is definitely the intended conversion factor, but sinc= e you asked, maybe it's not clear enough? Do you think maybe in part (a) o= f 3.8.3 we could just change: >> >> R1 and R2 might be measured in time units or as a count of retransmi= ssions. >> >> to: >> >> R1 and R2 might be measured in time units or as a count of retransmi= ssions (with the current RTO as a conversion factor, if needed). >> > I like that, thanks. hmm but some would convert to time by simply * w/o factoring exponential backoffs. how about "R1 and R2 might be measured in time units or as a count of retransmissions (with the current RTO and corresponding backoffs as a conversion factor, if needed)." > >> >> >> (3.8.5) This section is unclear as to whether the sender can cause the = Urgent Pointer to recede (move to the left), and if so, if that MUST be rep= orted to the application like a pointer advance. >> >> Receding looks the same as advancing so much that it wraps. So, if not = already in urgent mode, that would seem to cause a change to urgent mode, w= hile if not already in urgent mode, that would fall under the existing rule= of: >> >> If the urgent pointer is updated while the user is in "urgent mode",= the update will be invisible to the user. >> >> So ... also given that the use of urgent pointer is deprecated anyways a= nd we don't want to cause changes to TCP in this doc, I'm proposing we do n= othing to clarify this, if you agree? > > I don't think > > SEQ 5000 URG 3000 > SEQ 5001 URG 1500 > > would have a common-sense interpretation that the urg ptr was now 7501 + = 2^32. I share your grumpiness about sweating the useless urgent pointer, bu= t it would appear there is a gap in the spec and that in principle these re= cedes could exist in the wild. This spec mostly matters for new implementat= ions; maybe the safest thing is for those new implementations to prepare to= handle a receding pointer, but that senders MUST NOT recede. I would welco= me a quick question to the WG if that logic seems spurious to you. > > Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have > > If the urgent pointer is updated while the user is in "urgent mode", the > update will be invisible to the user. > > A TCP implementation MUST (MUST-32) inform the application layer > asynchronously whenever it receives an Urgent pointer and there was > previously no pending urgent data, or whenever the Urgent pointer > advances in the data stream. > > If the latter text is correct, AND we decide to allow receding, we shoul= d decide what to do with recedes here. >> >> (3.8.6.1) >> >> The sending TCP peer must be prepared to accept from the user and >> send at least one octet of new data even if the send window is zero. >> The sending TCP peer must regularly retransmit to the receiving TCP >> peer even when the window is zero >> >> Please clarify that these probes only happen when there is new data, if = that's the intent. >> >> The "accept from the user" part means that new data was written already,= right? If the user doesn't write data into the socket, there's nothing to= be sent. > > I have been messing with QUIC too long and may have forgotten how this wo= rks. But IIRC, this text kind of confuses a few different kinds of transmis= sions with zero window probes. It might be helpful to actually define "zero= window probe" -- a segment of one byte of data designed to elicit an ack w= ith a window update. The retransmission discussion is related but somewhat = confusing as currently written. There are four zero-window scenarios that I= can see. > > 1) If there is outstanding in-window data, we will get ACKs so there is n= o need for a ZWP. > > 2) No send buffer data, but the application generates data while in zero-= window. Clearly, the spec says TCP accepts a byte and sends it. > > 3) TCP has previously refused application data due to window restrictions= , and now we're in zero-window. TCP needs to wake the app to send again so = that we have material for ZWP. > > 4) TCP accepted some data, held it for Nagle, and then the receiver reneg= ed; or all in-flight data is now out of window due to reneging. This data o= ught to constitute a ZWP whether or not it is strictly a "retransmission". > > If you believe all of my statements correct, I am happy to propose altern= ate text that is not as wordy as the above., Waking up the application to take one more byte to probe the zero window does not seem necessary since sending an older sequence will do the job. Further, allowing such over-commit if the send buffer is full seems extra complexity w/o practical benefit. How about: "The sending TCP peer must regularly transmit new data (if available) or retransmit to the receiving TCP peer even when the window is zero". Let the implementation decide the best strategy as long as the functionality works. > > > (3.8.6.1) the first paragraph says > > Two minutes is recommended for the retransmission interval when the > window is zero. > > but the last one says > > The transmitting host SHOULD send the first zero-window probe when a > zero window has existed for the retransmission timeout period (SHLD- > 29) (Section 3.8.1), and SHOULD increase exponentially the interval > between successive probes (SHLD-30). > > so is it 2 min or the RTO? > > Good question ... The first sentence comes from 793 and the other from 11= 22. I think we should erase the first sentence, since the 1122 intent seem= s to be updating this (replacing the hard-coded 2 minute heuristic with the= RTO one which adapts to reality). Do you agree? > > Yes, I agree. >> >> >> (3.8.6.2.1) If I read this algorithm correctly, an un-pushed send buffer= of < MSS, if D < Fs*Max(SND.WND), will sit there indefinitely without >> being transmitted; is this intentional? >> >> This comes from 1122. I think the key to preventing what you mention is= this part of the preceding paragraph: >> >> To avoid a resulting deadlock, it is necessary to have a >> timeout to force transmission of data, overriding the SWS avoidance >> algorithm. >> >> So, SWS avoidance is not supposed to keep a sender waiting forever for t= he receiver to possibly advertise a bigger window; the sender is supposed t= o wake up and just send eventually. > > > Yes, but the override timeout is embedded in step (4), which applies PUSH= as a condition. Maybe that PUSH condition is incorrect? >> >> >> (3.8.2) and (3.8.6.3) Should RFC 5681 be a normative reference, given th= e congestion control requirement? If not, (3.8.6.3) shouldn't require the r= eader to go there to fully implement delayed ACKs. I suggest >> >> s/an ACK for at least every second segment/an ACK for least every 2*MSS = bytes of data >> >> to fully state what is in 5681. >> >> Yeah, I think 5681 should be normative (since we also rely on the for de= scriptions of slow start, etc. that we didn't try to duplicate within 793bi= s), but we should also fix the "second segment" sentence to match more exac= tly the text in 5681 about 2*RMSS. > > > OK, there's currently the pointer to 5681 about sub-MSS stuff, so it's te= chnically correct, but really clunky to force people to go there when it's = so easy to state correctly. I think we agree. >> >> >> >> >> >> (3.10, OPEN/LISTEN) >> >> If active and the remote socket is specified, then change the >> connection from passive to active, >> >> I think you mean "If passive"? >> >> Actually, I think to be more clear this should say: >> >> "If the OPEN call is active ...", since it's referring to the case where= an active OPEN is called when the socket is in LISTEN (passive). >> >> It's technically correct currently (just not very clear), if you read th= e "If active" as referring to the OPEN call rather than the socket (and it = must be that way since a socket in the LISTEN state is the result of a pass= ive OPEN). > > > Yes, let's clarify > >> >> (3.10, SEGMENT ARRIVES/LISTEN) Step 4 mentions RST but those have alread= y been handled in Step 1. >> >> Yeah, you're definitely right, and in both cases the action is to ignore= it, so this part of step 4 is irrelevant, but at least consistent. >> >> It looks like this is an errata to file on 793; should we do that quickl= y and then fix this doc accordingly? > > If that is better for your process, then sure. > >> >> NITS: >> >> >> >> >> (3.1, Options) " >> >> Options: [TCP Option]; Options#Size =3D=3D (DOffset-5)*32; present on= ly >> when DOffset > 5. >> >> What is this notation? >> >> It is a little odd, but comes from reference [61] for draft-mcquistin-au= gmented-ascii-diagrams-08. > > > OK > >> >> >> >> I'm good with changing all of these above. >> >> Thank you for the detailed review! > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm From nobody Fri Jul 2 15:03:23 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 98B983A08CE; Fri, 2 Jul 2021 15:02:50 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: "\"IETF Secretariat\"" To: , Cc: martin.h.duke@gmail.com, tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <162526337060.26814.3972872489510047622@ietfa.amsl.com> Date: Fri, 02 Jul 2021 15:02:50 -0700 Archived-At: Subject: [tcpm] tcpm - Requested session has been scheduled for IETF 111 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jul 2021 22:02:51 -0000 Dear Michael Scharf, The session(s) that you have requested have been scheduled. Below is the scheduled session information followed by the original request. tcpm Session 1 (2:00 requested) Tuesday, 27 July 2021, Session III 1600-1800 Room Name: Room 8 size: 508 --------------------------------------------- iCalendar: https://datatracker.ietf.org/meeting/111/sessions/tcpm.ics Request Information: --------------------------------------------------------- Working Group Name: TCP Maintenance and Minor Extensions Area Name: Transport Area Session Requester: Michael Scharf Number of Sessions: 1 Length of Session(s): 2 Hours Number of Attendees: 60 Conflicts to Avoid: People who must be present: Martin Duke Michael Scharf Michael Tuexen Yoshifumi Nishida Resources Requested: Special Requests: --------------------------------------------------------- From nobody Sun Jul 4 00:39:30 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 136493A3B39; Sun, 4 Jul 2021 00:39:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.482 X-Spam-Level: X-Spam-Status: No, score=-1.482 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.398, T_SPF_HELO_PERMERROR=0.01, T_SPF_PERMERROR=0.01] autolearn=no autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f7YFu05e01UE; Sun, 4 Jul 2021 00:39:25 -0700 (PDT) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AFD333A3B37; Sun, 4 Jul 2021 00:39:24 -0700 (PDT) Received: from smtpclient.apple (ip1f100e9c.dynamic.kabel-deutschland.de [31.16.14.156]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 6EB907220BFBB; Sun, 4 Jul 2021 09:39:15 +0200 (CEST) From: tuexen@fh-muenster.de Content-Type: multipart/signed; boundary="Apple-Mail=_0AE2D4B9-0CDB-442E-8B0A-08A80CBADB56"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: <858D0EE1-6CEF-44C2-9B7E-FD6F8D2B2CC8@fh-muenster.de> Date: Sun, 4 Jul 2021 09:39:14 +0200 Cc: tcpm-chairs To: tcpm IETF list X-Mailer: Apple Mail (2.3654.100.0.2.22) Archived-At: Subject: [tcpm] Agenda requests for TCPM@IETF111 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jul 2021 07:39:28 -0000 --Apple-Mail=_0AE2D4B9-0CDB-442E-8B0A-08A80CBADB56 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Dear all, according to the final schedule for IETF 111 (online), TCPM is scheduled for Tuesday, July 27th, 2021, from 23:00 UTC to 01:00 UTC of the next day. If you would like to present during this meeting, please provide by the end of Friday, July 9th, 2021: * Title of the presentation * Name of the presenter * Name of the Internet Draft * Time required (including Q/A) Please send us the slides for your presentations by Friday, July 23rd, 2021, latest. Best regards Michael --Apple-Mail=_0AE2D4B9-0CDB-442E-8B0A-08A80CBADB56 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEKow ggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYTAkRFMSsw KQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYDVQQLDBZULVN5 c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFsUm9vdCBDbGFzcyAy MB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNVBAYTAkRFMUUwQwYDVQQK EzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMg ZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlv biBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMtg1/9moUHN0vqH l4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZsFVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8F XRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0peQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+Ba L2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qL NupOkSk9s1FcragMvp0049ENF4N1xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz 9AkH4wKGMUZrAcUQDBHHWekCAwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU k+PYMiba1fFKpZFK4OpL4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYD VR0TAQH/BAgwBgEB/wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGC LB4wCAYGZ4EMAQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUv cmwvVGVsZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYB BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5jZXIw DQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4eTizDnS6 dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/MOaZ/SLick0+ hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3SPXez7vTXTf/D6OWS T1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc22CzeIs2LgtjZeOJVEqM7 h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bPZYoaorVyGTkwggWsMIIElKAD AgECAgcbY7rQHiw9MA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJERTFFMEMGA1UEChM8VmVy ZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYu MRAwDgYDVQQLEwdERk4tUEtJMS0wKwYDVQQDEyRERk4tVmVyZWluIENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IDIwHhcNMTYwNTI0MTEzODQwWhcNMzEwMjIyMjM1OTU5WjCBjTELMAkGA1UEBhMCREUx RTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5lcyBEZXV0c2NoZW4gRm9yc2NodW5n c25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMGA1UEAwwcREZOLVZlcmVpbiBHbG9i YWwgSXNzdWluZyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ07eRxH3h+Gy8Zp 1xCeOdfZojDbchwFfylfS2jxrRnWTOFrG7ELf6Gr4HuLi9gtzm6IOhDuV+UefwRRNuu6cG1joL6W LkDh0YNMZj0cZGnlm6Stcq5oOVGHecwX064vXWNxSzl660Knl5BpBb+Q/6RAcL0D57+eGIgfn5mI TQ5HjUhfZZkQ0tkqSe3BuS0dnxLLFdM/fx5ULzquk1enfnjK1UriGuXtQX1TX8izKvWKMKztFwUk P7agCwf9TRqaA1KgNpzeJIdl5Of6x5ZzJBTN0OgbaJ4YWa52fvfRCng8h0uwN89Tyjo4EPPLR22M ZD08WkVKusqAfLjz56dMTM0CAwEAAaOCAgUwggIBMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0P AQH/BAQDAgEGMCkGA1UdIAQiMCAwDQYLKwYBBAGBrSGCLB4wDwYNKwYBBAGBrSGCLAEBBDAdBgNV HQ4EFgQUazqYi/nyU4na4K2yMh4JH+iqO3QwHwYDVR0jBBgwFoAUk+PYMiba1fFKpZFK4OpL4qIM z+EwgY8GA1UdHwSBhzCBhDBAoD6gPIY6aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9v dC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDBAoD6gPIY6aHR0cDovL2NkcDIucGNhLmRmbi5kZS9n bG9iYWwtcm9vdC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDCB3QYIKwYBBQUHAQEEgdAwgc0wMwYI KwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBKBggrBgEF BQcwAoY+aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1jYS9wdWIvY2FjZXJ0 L2NhY2VydC5jcnQwSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFsLXJv b3QtZzItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCBeEWkTqR/ DlXwCbFqPnjMaDWpHPOVnj/z+N9rOHeJLI21rT7H8pTNoAauusyosa0zCLYkhmI2THhuUPDVbmCN T1IxQ5dGdfBi5G5mUcFCMWdQ5UnnOR7Ln8qGSN4IFP8VSytmm6A4nwDO/afr0X9XLchMX9wQEZc+ lgQCXISoKTlslPwQkgZ7nu7YRrQbtQMMONncsKk/cQYLsgMHM8KNSGMlJTx6e1du94oFOO+4oK4v 9NsH1VuEGMGpuEvObJAaguS5Pfp38dIfMwK/U+d2+dwmJUFvL6Yb+qQTkPp8ftkLYF3sv8pBoGH7 EUkp2KgtdRXYShjqFu9VNCIaE40GMIIF4DCCBMigAwIBAgIMIRX9tDE2QqO3mVLXMA0GCSqGSIb3 DQEBCwUAMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1ciBGb2VyZGVydW5nIGVp bmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYDVQQLDAdERk4tUEtJMSUw IwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBMB4XDTE5MDYwNDE0MjkxMFoXDTIy MDYwMzE0MjkxMFowfDELMAkGA1UEBhMCREUxIDAeBgNVBAoMF0ZhY2hob2Noc2NodWxlIE11ZW5z dGVyMTIwMAYDVQQLDClGYWNoYmVyZWljaCBFbGVrdHJvdGVjaG5payB1bmQgSW5mb3JtYXRpazEX MBUGA1UEAwwOTWljaGFlbCBUdWV4ZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM r8qQcPxLFCxzPtXvRyM9KeQaxyMA8gwUNc7IIiATs6mRQFe5ib/mvwT9nc0bAa+94go6HJDiD3FJ NkTo4u8aBsIcTt5pJtdBQLm88PLakbe3+fp/00//n7xxbTh7mAtFVCf25LxDCKkrdGk/+jglRq/R VdwhZZ3VpYCrx8YfI/hIzdRL3+4I4z/mnQ8K0X8d2MVVPG+9nBEngdnYGez5f/8wIVOgEYYBc21k yvMnVXLu2Ing+LwBd0gDG9Vasv87MNHCOZfJTNBlLhI2UDei/uNg9Zd4ynlMpPWZ7v0oiDWvmv8E OuD4oric8heyD0OYK2FL4qcVC4dn4pnyulfHAgMBAAGjggJOMIICSjA+BgNVHSAENzA1MA8GDSsG AQQBga0hgiwBAQQwEAYOKwYBBAGBrSGCLAEBBAQwEAYOKwYBBAGBrSGCLAIBBAQwCQYDVR0TBAIw ADAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQW BBTxiodBVL/lA4p5iNesIsJRhhgd6zAfBgNVHSMEGDAWgBRrOpiL+fJTidrgrbIyHgkf6Ko7dDAg BgNVHREEGTAXgRV0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0 cDovL2NkcDEucGNhLmRmbi5kZS9kZm4tY2EtZ2xvYmFsLWcyL3B1Yi9jcmwvY2FjcmwuY3JsMD+g PaA7hjlodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2Rmbi1jYS1nbG9iYWwtZzIvcHViL2NybC9jYWNy bC5jcmwwgdsGCCsGAQUFBwEBBIHOMIHLMDMGCCsGAQUFBzABhidodHRwOi8vb2NzcC5wY2EuZGZu LmRlL09DU1AtU2VydmVyL09DU1AwSQYIKwYBBQUHMAKGPWh0dHA6Ly9jZHAxLnBjYS5kZm4uZGUv ZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwSQYIKwYBBQUHMAKGPWh0dHA6 Ly9jZHAyLnBjYS5kZm4uZGUvZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQw DQYJKoZIhvcNAQELBQADggEBABs3VlmIZ1VF3HkaQdjInZYmamRabbdgJ7cz6m6o/agKL7+Vhwx7 1BaaYs2gMa5Nu/GJ3YfdqIsUlYtKdl58Yhp/89E6xBfJkItS+rE8RFdf2XgklGlx7GmsvdA3tId5 b6K6r9a5wpVN0epxY6K8wwpzEib6fJLcHylybQXZ7JSOaLRLp6WU3QPoyTT7FpvAe/0b2MSCbPX4 fc53PCn2aGgXuRFVQeCn1SP1BF3QW1ppkFhcF6G+0JcUxYFAXE6r/71WZBvUHqoG/th2hAwQnS+Y KhUYh4SZQH3/ursXXJYXQ5vyIhkN1FZlmtWA8+ofdCnoqSTbiSX2Aa/kKbTapwgxggOdMIIDmQIB ATCBnjCBjTELMAkGA1UEBhMCREUxRTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5l cyBEZXV0c2NoZW4gRm9yc2NodW5nc25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMG A1UEAwwcREZOLVZlcmVpbiBHbG9iYWwgSXNzdWluZyBDQQIMIRX9tDE2QqO3mVLXMA0GCWCGSAFl AwQCAQUAoIIBzzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMTA3 MDQwNzM5MTRaMC8GCSqGSIb3DQEJBDEiBCAGDmYZWTJ9gXGidrGVo41ImqD1a6oc3FvDZG29amRg oTCBrwYJKwYBBAGCNxAEMYGhMIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1 ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYD VQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBAgwhFf20 MTZCo7eZUtcwgbEGCyqGSIb3DQEJEAILMYGhoIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8 VmVyZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUu IFYuMRAwDgYDVQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5n IENBAgwhFf20MTZCo7eZUtcwDQYJKoZIhvcNAQELBQAEggEAR2BC9Z9MVPh6q11nKZRETvW1gj5f 7FdwgmAnRA0ZIeVgy5gXpsX7+SLyE1Ipe3VLu0TY0Z32HNWQsV+LZKlA/TrQLaPFZWczIkznKf9S r0247lltoQmbysBQIjtuGxOuH/YxBg9/j+HQVKNPsnPMHOqQS1w1YUjCoy1FcmeESNT9jnWufEtY kn+/rb9ctoUnKo7HtA8mrvv0AauWiHqPKOYDLkn4Qc7+dKR/P/JYNOZ4YNhvCF32eRiXGpnEnvN2 AA9bXix4WS6LdkoE2thoTT4p4sYxHKBGIKW0nHy3myKQbgmsVpNtnyroYO2IWV4hnSUjJ8oIMwuG 8AnCY/f58gAAAAAAAA== --Apple-Mail=_0AE2D4B9-0CDB-442E-8B0A-08A80CBADB56-- From nobody Sun Jul 4 08:37:26 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id F215A3A2254 for ; Sun, 4 Jul 2021 08:37:24 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit To: X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <162541304497.16840.13348546857101026402@ietfa.amsl.com> Date: Sun, 04 Jul 2021 08:37:24 -0700 From: IETF Secretariat Archived-At: Subject: [tcpm] Milestones changed for tcpm WG X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Jul 2021 15:37:25 -0000 Changed milestone "Submit documents on TCP-AO Test Vectors as an Informational RFC", added draft-ietf-tcpm-ao-test-vectors to milestone. URL: https://datatracker.ietf.org/wg/tcpm/about/ From nobody Tue Jul 6 13:05:04 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C8E053A33C7; Tue, 6 Jul 2021 13:05:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FO2QtSdpmtgu; Tue, 6 Jul 2021 13:04:56 -0700 (PDT) Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3DF5C3A33C3; Tue, 6 Jul 2021 13:04:56 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id D494B25A1A; Tue, 6 Jul 2021 22:04:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1625601893; bh=P2Sv76ioNJvPguS+Jswo3IN/LskUmvf0TRl+e4kXxiI=; h=From:To:CC:Subject:Date:From; b=KuIB8dVMbNMxTHKog8wtbnPY6m93JHfAwGbRRI7/ts0tRZvDBdAHzlXKDghyeep3b ZZAfC9R1S6K/P05XCHCq+Eum5nbwdiOKeGtSRTubpZWnKDickVu/gHGjGmyXwVot/j eo8TcPed1DXzaVYsnFRr20a58eBICEQsu/svDkHE= X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4NryEymiqaLy; Tue, 6 Jul 2021 22:04:52 +0200 (CEST) Received: from rznt8201.rznt.rzdir.fht-esslingen.de (rznt8201.hs-esslingen.de [134.108.48.164]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Tue, 6 Jul 2021 22:04:52 +0200 (CEST) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8201.rznt.rzdir.fht-esslingen.de (134.108.48.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Tue, 6 Jul 2021 22:04:52 +0200 Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.2176.014; Tue, 6 Jul 2021 22:04:52 +0200 From: "Scharf, Michael" To: "tcpm@ietf.org" CC: "tcpm-chairs@ietf.org" Thread-Topic: Moving forward draft-touch-sne on independent stream? Thread-Index: AddyoYAExwdNYisyQIax/m998gxedQ== Date: Tue, 6 Jul 2021 20:04:52 +0000 Message-ID: <8de93237209d4fc7ab062bbc24217999@hs-esslingen.de> Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [134.108.140.248] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Archived-At: Subject: [tcpm] Moving forward draft-touch-sne on independent stream? X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2021 20:05:03 -0000 Hi all, Joe has submitted draft-touch-sne to the ISE asking for publication of an R= FC on the Independent Stream. (Background: An RFC on Independent Stream is = not an IETF standard.) Publication on independent stream would be possible provided that the docum= ent does not conflict with TCPM or other IETF work. An obvious alternative = would be homing the document in TCPM (or TSVWG) as an official WG item - bu= t this is not what is currently proposed. A link to the document is: https://datatracker.ietf.org/doc/html/draft-touch-sne Note that the current header field "TCPM" would have to be replaced in the = further publication process, as documents on the independent stream are not= the outcome of an IETF WG. The TCPM chairs have reviewed draft-touch-sne. Our current understanding th= at there would be no harm caused by such an independent stream publication.= As far as RFC 5925 is affected, we assume that an Errata would be possible= and sufficient. Before we get back to the ISE regarding this document, we would like to cro= ss-check whether our understanding is indeed the TCPM consensus: Are there any concerns inside the TCPM community regarding a publication of= draft-touch-sne draft-touch-sne on independent stream? If so, please speak= up! Thanks Michael on behalf of all chairs From nobody Tue Jul 6 16:44:17 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 8D7113A0D0F; Tue, 6 Jul 2021 16:44:11 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162561505151.16988.15150608973071393254@ietfa.amsl.com> Date: Tue, 06 Jul 2021 16:44:11 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-yang-tcp-02.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Jul 2021 23:44:12 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : YANG Model for Transmission Control Protocol (TCP) Configuration Authors : Michael Scharf Vishal Murgai Mahesh Jethanandani Filename : draft-ietf-tcpm-yang-tcp-02.txt Pages : 22 Date : 2021-07-06 Abstract: This document specifies a YANG model for TCP on devices that are configured by network management protocols. The YANG model defines a container for all TCP connections and groupings of some of the parameters that can be imported and used in TCP implementations or by other models that need to configure TCP parameters. The model includes definitions from YANG Groupings for TCP Client and TCP Servers (I-D.ietf-netconf-tcp-client-server). The model is NMDA (RFC 8342) compliant. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-yang-tcp/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-yang-tcp-02 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-yang-tcp-02 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Tue Jul 6 22:25:23 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8B81C3A0989; Tue, 6 Jul 2021 22:25:18 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.318 X-Spam-Level: X-Spam-Status: No, score=-1.318 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=strayalpha.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KtVmfKUqaztG; Tue, 6 Jul 2021 22:25:13 -0700 (PDT) Received: from server217-3.web-hosting.com (server217-3.web-hosting.com [198.54.115.226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A4DC83A098A; Tue, 6 Jul 2021 22:25:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=strayalpha.com; s=default; h=To:References:Message-Id:Cc:Date:In-Reply-To: From:Subject:Mime-Version:Content-Type:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=+ofxRVBj+s/bqlSRo5heqedTOwHVFVtID3P3z7qkdtA=; b=Xh9PZpgH3TXTzz7/p7Z1zK++XR /NJXtvB8YArrRtXUDNhw465Z/VzRCPqS19uRtL8B843kbY6FfEDlo5G5nmsVr16eiFfRDdIUkwYt9 rXWreK4PyhH3M9JmLXyWj0Erd6dd2ww5X8wgDD1MGQyIQUuSM9X39R52GPriS4WoKe4dOx1hrAX1D pNw6XpGgm6sm193Lbf+wRjjUbZaia+opsox6JCSYb7VQkfjj5ok2EzJGJdQOrwYmdOGzSQqCWPKQS 5zl7lLS9oXVnuO6sqrOE2Y8niLoIHTAjOdXv6PsR4QqufQp1l8bGc5WGV46gWwiET0BAZbMY3fZ+V iMxpmFBA==; Received: from cpe-172-250-225-198.socal.res.rr.com ([172.250.225.198]:62658 helo=smtpclient.apple) by server217.web-hosting.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1m103Y-003eX3-5x; Wed, 07 Jul 2021 01:25:12 -0400 Content-Type: multipart/alternative; boundary="Apple-Mail=_9D0D47F5-DFA2-4EA0-9D13-3B3EB054FA60" Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) From: Joseph Touch In-Reply-To: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> Date: Tue, 6 Jul 2021 22:25:04 -0700 Cc: Martin Duke , draft-ietf-tcpm-rfc793bis.all@ietf.org, "tcpm@ietf.org Extensions" Message-Id: References: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> To: Wes Eddy X-Mailer: Apple Mail (2.3654.100.0.2.22) X-OutGoing-Spam-Status: No, score=-1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server217.web-hosting.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - strayalpha.com X-Get-Message-Sender-Via: server217.web-hosting.com: authenticated_id: touch@strayalpha.com X-Authenticated-Sender: server217.web-hosting.com: touch@strayalpha.com X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched Archived-At: Subject: Re: [tcpm] RFC793bis AD review X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2021 05:25:19 -0000 --Apple-Mail=_9D0D47F5-DFA2-4EA0-9D13-3B3EB054FA60 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, all, > On Jun 30, 2021, at 2:50 PM, Wesley Eddy wrote: >=20 >> (3.7.2) In the first paragraph it says >> RFC 1122 recommends = an IP-layer default >> effective MTU of less than or equal to 576 for destinations not >> directly connected >> In the last it says >> RFC 1191 discusses = this implication of many older TCP >> implementations setting MSS to 536 for non-local destinations, = rather >> than deriving it from the MTUs of connected interfaces as >> recommended. >> Could we unify these comments in the same location? Implementers = browsing the spec should see the old recommendation and its drawback in = the same place. > Good idea; I think we can. >=20 Two things that might be useful to include - first, these are intended to refer to the same number; one is an IP = MTU, the other a TCP MSS - for the second, it=E2=80=99s useful to refer to RFC 6691 as well as to = note that this value is for TCP over IPv4 Joe --Apple-Mail=_9D0D47F5-DFA2-4EA0-9D13-3B3EB054FA60 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi, = all,

On Jun 30, 2021, at 2:50 PM, Wesley Eddy = <wes@mti-systems.com> wrote:

(3.7.2) In the first paragraph it says
RFC 1122 =
recommends an IP-layer default
   effective MTU of less than or equal to 576 for destinations not
   directly connected
In the last it = says
RFC 1191 discusses this =
implication of many older TCP
   implementations setting MSS to 536 for non-local destinations, rather
   than deriving it from the MTUs of connected interfaces as
   recommended.
Could we unify these = comments in the same location? Implementers browsing the spec should see = the old recommendation and its drawback in the same = place.

Good idea; I think we = can.

Two things that might be useful = to include

- first, = these are intended to refer to the same number; one is an IP MTU, the = other a TCP MSS
- for the second, it=E2=80=99s = useful to refer to RFC 6691 as = well as to note that this value is for TCP over = IPv4

Joe


= --Apple-Mail=_9D0D47F5-DFA2-4EA0-9D13-3B3EB054FA60-- From nobody Sat Jul 10 02:10:49 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2E3D53A1DFF; Sat, 10 Jul 2021 02:10:46 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.433 X-Spam-Level: X-Spam-Status: No, score=-2.433 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.338, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YxeolTe4D7US; Sat, 10 Jul 2021 02:10:41 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 123FE3A1DFC; Sat, 10 Jul 2021 02:10:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=iv6dKTOMQ7pL0thgV+Qv+UfCFpkyuasE4T6AUpkbvIc=; b=M0BeKoBstuBrZvBcgne+JjL6a4 CO0zzrGHAJ6X9z+6eCwONWfco8yAXnBwbkDG03joBUAaIm0X5QgBS2cE1LFHxTwR8BFnJTkLJsEBS RLijbe4USPqouA0xyNowprp1PENOaw0Yyqv1eSI38SBnZz2Slhu6I4LpR1nHU9squgum8mL7uUdby 4YlwJ0ueSQz7VSjRysB/0C2iOPQ06Dt5OERKg+peeiZOrCkaW7SiTGjwqb6mtHEjK89nvMwmmcbgx FmntIh7GVJD5y3iHEGUmDQDggZaJM39Gk33MS5aPpRHeZqxmPgQ3yscOcIPpm+rvAIeYH/JQOrSVD KLD7rMuQ==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:56822 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m290a-0006HF-B3; Sat, 10 Jul 2021 10:10:38 +0100 To: tuexen@fh-muenster.de, tcpm IETF list Cc: tcpm-chairs , Mirja Kuehlewind , Richard Scheffenegger , =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= References: <858D0EE1-6CEF-44C2-9B7E-FD6F8D2B2CC8@fh-muenster.de> From: Bob Briscoe Message-ID: Date: Sat, 10 Jul 2021 10:10:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <858D0EE1-6CEF-44C2-9B7E-FD6F8D2B2CC8@fh-muenster.de> Content-Type: multipart/alternative; boundary="------------891C6BF444D3491DEA1A5764" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] Agenda requests for TCPM@IETF111 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jul 2021 09:10:46 -0000 This is a multi-part message in MIME format. --------------891C6BF444D3491DEA1A5764 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Michael, Hopefully it's still "the end of Friday" somewhere in the world... ;) * More Accurate ECN Feedback in TCP * Bob Briscoe * draft-ietf-tcpm-accurate-ecn (will be -15 by the deadline) * 10 mins As well as presenting deltas to the draft based on recent on-list and off-list discussions (ACKing ACKs and covert channel), we'll present the results of work Ilpo has been doing to check how resilient AccECN is to ACK filtering, which has resulted in some alterations to the draft text. Bob On 04/07/2021 08:39, tuexen@fh-muenster.de wrote: > Dear all, > > according to the final schedule for IETF 111 (online), TCPM is scheduled > for Tuesday, July 27th, 2021, from 23:00 UTC to 01:00 UTC of the next day. > > If you would like to present during this meeting, please provide by the > end of Friday, July 9th, 2021: > > * Title of the presentation > * Name of the presenter > * Name of the Internet Draft > * Time required (including Q/A) > > Please send us the slides for your presentations by > Friday, July 23rd, 2021, latest. > > Best regards > Michael > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------891C6BF444D3491DEA1A5764 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit Michael,

Hopefully it's still "the end of Friday" somewhere in the world...  ;)

* More Accurate ECN Feedback in TCP
* Bob Briscoe
* draft-ietf-tcpm-accurate-ecn (will be -15 by the deadline)
* 10 mins
As well as presenting deltas to the draft based on recent on-list and off-list discussions (ACKing ACKs and covert channel), we'll present the results of work Ilpo has been doing to check how resilient AccECN is to ACK filtering, which has resulted in some alterations to the draft text.



Bob

On 04/07/2021 08:39, tuexen@fh-muenster.de wrote:
Dear all,

according to the final schedule for IETF 111 (online), TCPM is scheduled
for Tuesday, July 27th, 2021, from 23:00 UTC to 01:00 UTC of the next day.

If you would like to present during this meeting, please provide by the
end of Friday, July 9th, 2021:

* Title of the presentation
* Name of the presenter
* Name of the Internet Draft
* Time required (including Q/A)

Please send us the slides for your presentations by
Friday, July 23rd, 2021, latest.

Best regards
Michael



_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------891C6BF444D3491DEA1A5764-- From nobody Sat Jul 10 06:21:00 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B6D163A1ADE; Sat, 10 Jul 2021 06:20:59 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.836 X-Spam-Level: X-Spam-Status: No, score=-0.836 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.398, SPF_SOFTFAIL=0.665, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iLpKCBerIniG; Sat, 10 Jul 2021 06:20:55 -0700 (PDT) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C34553A1ADC; Sat, 10 Jul 2021 06:20:54 -0700 (PDT) Received: from smtpclient.apple (ip1f100e9c.dynamic.kabel-deutschland.de [31.16.14.156]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 104C2721E2807; Sat, 10 Jul 2021 15:20:49 +0200 (CEST) From: tuexen@fh-muenster.de Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_D2324411-ABC2-43A4-BAE2-C01171392264"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Date: Sat, 10 Jul 2021 15:20:48 +0200 In-Reply-To: Cc: tcpm IETF list , tcpm-chairs , Mirja Kuehlewind , Richard Scheffenegger , =?utf-8?Q?Ilpo_J=C3=A4rvinen?= To: Bob Briscoe References: <858D0EE1-6CEF-44C2-9B7E-FD6F8D2B2CC8@fh-muenster.de> X-Mailer: Apple Mail (2.3654.100.0.2.22) Archived-At: Subject: Re: [tcpm] Agenda requests for TCPM@IETF111 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jul 2021 13:21:00 -0000 --Apple-Mail=_D2324411-ABC2-43A4-BAE2-C01171392264 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 10. Jul 2021, at 11:10, Bob Briscoe wrote: >=20 > Michael, >=20 > Hopefully it's still "the end of Friday" somewhere in the world... ;) Hi Bob, sure, no problem. >=20 > * More Accurate ECN Feedback in TCP > * Bob Briscoe > * draft-ietf-tcpm-accurate-ecn (will be -15 by the deadline) > * 10 mins >=20 > As well as presenting deltas to the draft based on recent on-list and = off-list discussions (ACKing ACKs and covert channel), we'll present the = results of work Ilpo has been doing to check how resilient AccECN is to = ACK filtering, which has resulted in some alterations to the draft text.=20= Great, thanks! Best regards Michael >=20 >=20 >=20 > Bob >=20 > On 04/07/2021 08:39, tuexen@fh-muenster.de wrote: >> Dear all, >>=20 >> according to the final schedule for IETF 111 (online), TCPM is = scheduled >> for Tuesday, July 27th, 2021, from 23:00 UTC to 01:00 UTC of the next = day. >>=20 >> If you would like to present during this meeting, please provide by = the >> end of Friday, July 9th, 2021: >>=20 >> * Title of the presentation >> * Name of the presenter >> * Name of the Internet Draft >> * Time required (including Q/A) >>=20 >> Please send us the slides for your presentations by >> Friday, July 23rd, 2021, latest. >>=20 >> Best regards >> Michael >>=20 >>=20 >>=20 >>=20 >> _______________________________________________ >> tcpm mailing list >>=20 >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm >=20 > --=20 > ________________________________________________________________ > Bob Briscoe =20 > http://bobbriscoe.net/ --Apple-Mail=_D2324411-ABC2-43A4-BAE2-C01171392264 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEKow ggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYTAkRFMSsw KQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYDVQQLDBZULVN5 c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFsUm9vdCBDbGFzcyAy MB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNVBAYTAkRFMUUwQwYDVQQK EzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMg ZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlv biBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMtg1/9moUHN0vqH l4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZsFVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8F XRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0peQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+Ba L2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qL NupOkSk9s1FcragMvp0049ENF4N1xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz 9AkH4wKGMUZrAcUQDBHHWekCAwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU k+PYMiba1fFKpZFK4OpL4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYD VR0TAQH/BAgwBgEB/wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGC LB4wCAYGZ4EMAQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUv cmwvVGVsZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYB BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5jZXIw DQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4eTizDnS6 dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/MOaZ/SLick0+ hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3SPXez7vTXTf/D6OWS T1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc22CzeIs2LgtjZeOJVEqM7 h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bPZYoaorVyGTkwggWsMIIElKAD AgECAgcbY7rQHiw9MA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJERTFFMEMGA1UEChM8VmVy ZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYu MRAwDgYDVQQLEwdERk4tUEtJMS0wKwYDVQQDEyRERk4tVmVyZWluIENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IDIwHhcNMTYwNTI0MTEzODQwWhcNMzEwMjIyMjM1OTU5WjCBjTELMAkGA1UEBhMCREUx RTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5lcyBEZXV0c2NoZW4gRm9yc2NodW5n c25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMGA1UEAwwcREZOLVZlcmVpbiBHbG9i YWwgSXNzdWluZyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ07eRxH3h+Gy8Zp 1xCeOdfZojDbchwFfylfS2jxrRnWTOFrG7ELf6Gr4HuLi9gtzm6IOhDuV+UefwRRNuu6cG1joL6W LkDh0YNMZj0cZGnlm6Stcq5oOVGHecwX064vXWNxSzl660Knl5BpBb+Q/6RAcL0D57+eGIgfn5mI TQ5HjUhfZZkQ0tkqSe3BuS0dnxLLFdM/fx5ULzquk1enfnjK1UriGuXtQX1TX8izKvWKMKztFwUk P7agCwf9TRqaA1KgNpzeJIdl5Of6x5ZzJBTN0OgbaJ4YWa52fvfRCng8h0uwN89Tyjo4EPPLR22M ZD08WkVKusqAfLjz56dMTM0CAwEAAaOCAgUwggIBMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0P AQH/BAQDAgEGMCkGA1UdIAQiMCAwDQYLKwYBBAGBrSGCLB4wDwYNKwYBBAGBrSGCLAEBBDAdBgNV HQ4EFgQUazqYi/nyU4na4K2yMh4JH+iqO3QwHwYDVR0jBBgwFoAUk+PYMiba1fFKpZFK4OpL4qIM z+EwgY8GA1UdHwSBhzCBhDBAoD6gPIY6aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9v dC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDBAoD6gPIY6aHR0cDovL2NkcDIucGNhLmRmbi5kZS9n bG9iYWwtcm9vdC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDCB3QYIKwYBBQUHAQEEgdAwgc0wMwYI KwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBKBggrBgEF BQcwAoY+aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1jYS9wdWIvY2FjZXJ0 L2NhY2VydC5jcnQwSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFsLXJv b3QtZzItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCBeEWkTqR/ DlXwCbFqPnjMaDWpHPOVnj/z+N9rOHeJLI21rT7H8pTNoAauusyosa0zCLYkhmI2THhuUPDVbmCN T1IxQ5dGdfBi5G5mUcFCMWdQ5UnnOR7Ln8qGSN4IFP8VSytmm6A4nwDO/afr0X9XLchMX9wQEZc+ lgQCXISoKTlslPwQkgZ7nu7YRrQbtQMMONncsKk/cQYLsgMHM8KNSGMlJTx6e1du94oFOO+4oK4v 9NsH1VuEGMGpuEvObJAaguS5Pfp38dIfMwK/U+d2+dwmJUFvL6Yb+qQTkPp8ftkLYF3sv8pBoGH7 EUkp2KgtdRXYShjqFu9VNCIaE40GMIIF4DCCBMigAwIBAgIMIRX9tDE2QqO3mVLXMA0GCSqGSIb3 DQEBCwUAMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1ciBGb2VyZGVydW5nIGVp bmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYDVQQLDAdERk4tUEtJMSUw IwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBMB4XDTE5MDYwNDE0MjkxMFoXDTIy MDYwMzE0MjkxMFowfDELMAkGA1UEBhMCREUxIDAeBgNVBAoMF0ZhY2hob2Noc2NodWxlIE11ZW5z dGVyMTIwMAYDVQQLDClGYWNoYmVyZWljaCBFbGVrdHJvdGVjaG5payB1bmQgSW5mb3JtYXRpazEX MBUGA1UEAwwOTWljaGFlbCBUdWV4ZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM r8qQcPxLFCxzPtXvRyM9KeQaxyMA8gwUNc7IIiATs6mRQFe5ib/mvwT9nc0bAa+94go6HJDiD3FJ NkTo4u8aBsIcTt5pJtdBQLm88PLakbe3+fp/00//n7xxbTh7mAtFVCf25LxDCKkrdGk/+jglRq/R VdwhZZ3VpYCrx8YfI/hIzdRL3+4I4z/mnQ8K0X8d2MVVPG+9nBEngdnYGez5f/8wIVOgEYYBc21k yvMnVXLu2Ing+LwBd0gDG9Vasv87MNHCOZfJTNBlLhI2UDei/uNg9Zd4ynlMpPWZ7v0oiDWvmv8E OuD4oric8heyD0OYK2FL4qcVC4dn4pnyulfHAgMBAAGjggJOMIICSjA+BgNVHSAENzA1MA8GDSsG AQQBga0hgiwBAQQwEAYOKwYBBAGBrSGCLAEBBAQwEAYOKwYBBAGBrSGCLAIBBAQwCQYDVR0TBAIw ADAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQW BBTxiodBVL/lA4p5iNesIsJRhhgd6zAfBgNVHSMEGDAWgBRrOpiL+fJTidrgrbIyHgkf6Ko7dDAg BgNVHREEGTAXgRV0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0 cDovL2NkcDEucGNhLmRmbi5kZS9kZm4tY2EtZ2xvYmFsLWcyL3B1Yi9jcmwvY2FjcmwuY3JsMD+g PaA7hjlodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2Rmbi1jYS1nbG9iYWwtZzIvcHViL2NybC9jYWNy bC5jcmwwgdsGCCsGAQUFBwEBBIHOMIHLMDMGCCsGAQUFBzABhidodHRwOi8vb2NzcC5wY2EuZGZu LmRlL09DU1AtU2VydmVyL09DU1AwSQYIKwYBBQUHMAKGPWh0dHA6Ly9jZHAxLnBjYS5kZm4uZGUv ZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwSQYIKwYBBQUHMAKGPWh0dHA6 Ly9jZHAyLnBjYS5kZm4uZGUvZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQw DQYJKoZIhvcNAQELBQADggEBABs3VlmIZ1VF3HkaQdjInZYmamRabbdgJ7cz6m6o/agKL7+Vhwx7 1BaaYs2gMa5Nu/GJ3YfdqIsUlYtKdl58Yhp/89E6xBfJkItS+rE8RFdf2XgklGlx7GmsvdA3tId5 b6K6r9a5wpVN0epxY6K8wwpzEib6fJLcHylybQXZ7JSOaLRLp6WU3QPoyTT7FpvAe/0b2MSCbPX4 fc53PCn2aGgXuRFVQeCn1SP1BF3QW1ppkFhcF6G+0JcUxYFAXE6r/71WZBvUHqoG/th2hAwQnS+Y KhUYh4SZQH3/ursXXJYXQ5vyIhkN1FZlmtWA8+ofdCnoqSTbiSX2Aa/kKbTapwgxggOdMIIDmQIB ATCBnjCBjTELMAkGA1UEBhMCREUxRTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5l cyBEZXV0c2NoZW4gRm9yc2NodW5nc25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMG A1UEAwwcREZOLVZlcmVpbiBHbG9iYWwgSXNzdWluZyBDQQIMIRX9tDE2QqO3mVLXMA0GCWCGSAFl AwQCAQUAoIIBzzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMTA3 MTAxMzIwNDhaMC8GCSqGSIb3DQEJBDEiBCBWmgIGvEvHIzI9YlhtxU9CyQwXxbslpZdZ9St1lWZy eDCBrwYJKwYBBAGCNxAEMYGhMIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1 ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYD VQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBAgwhFf20 MTZCo7eZUtcwgbEGCyqGSIb3DQEJEAILMYGhoIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8 VmVyZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUu IFYuMRAwDgYDVQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5n IENBAgwhFf20MTZCo7eZUtcwDQYJKoZIhvcNAQELBQAEggEASeFDQgtGGTteJ8uNcJyb7BS5nH9R 8AKoVaRKlHUbyhaSdg7GGpgGNk0JEw+PT+X4Jrej8vpMSAKmal6YKrQy65PY/+KW3FF1yyoaoy1O SyYPUYdIPcUF1cGDdY29+An2i46LiH+OXwMKurRxrxOQvBsk7fSIAtsWpiCoP64ytoywpOUXwLHW bygoeg9r5bCzoElabVbPyMsVJ3HQYHTxtsCDSQ0rufCuwJbs9dVtJ3evXHMoRVd17ds/xiXv6NeL SP44Fkyvin66MP9Sy41NRu9CmyFt+dmxPrUs36MX/gqf5Rjv3Y6dZIT7WIbzim++pXQYNpbD0Y2b cvYwTmq2egAAAAAAAA== --Apple-Mail=_D2324411-ABC2-43A4-BAE2-C01171392264-- From nobody Sat Jul 10 14:13:58 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C46193A101D for ; Sat, 10 Jul 2021 14:13:55 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.197 X-Spam-Level: X-Spam-Status: No, score=-0.197 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m5Gua0BhfQCQ for ; Sat, 10 Jul 2021 14:13:51 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 07C163A1019 for ; Sat, 10 Jul 2021 14:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=WYk0NIGcNshODQ80mws2We1/RBJ1ux5kfEyLftCxY3U=; b=DrvNOMBSYPeX3fXk6fs3UHLllt SJxg3qSksghGTATP6NB33xiZ7v6Rh4lND2Bz0XbXD40m+MR0cpVIIrRoPTOMm7GbH6OcpHFmZXNuv nbtXL6IZQisJViPhy36old3cN+8hBnl1/+4fPyM/Ls3h4QbKrglAXj6LHbpMsqJs8XCwQrXF9yJj+ 5R+cE+v9KHV5Lxqy9z8SzbKJMpcxnSJQw6vXbS4Xml7wN768BZsPfrwB9xpDWaWupQ6us+FbdUTLl 80+RWhxpagoHOI+arEL2/FDZnF1e1u3aElAJ4GaOQRTcWUcN/MttGCmIuZHwIhWTF+6/48QuI0/fv O+TxeytQ==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:58002 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m2KIR-0007Kd-A7; Sat, 10 Jul 2021 22:13:49 +0100 To: Yoshifumi Nishida , "tcpm@ietf.org" , Mirja Kuehlewind , "rs.ietf@gmx.at" , "ietf@kuehlewind.net" Cc: "nishida@sfc.wide.ad.jp" References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <221e58f3-ada0-c880-db72-d98af84fedb8@gmx.at> <8151fdef-ae78-80f3-adfc-d40db878ac8e@gmx.at> <48c5910d-5340-acd6-8fd9-fff1b7758310@bobbriscoe.net> <99B71B9A-EC9B-47FD-A149-FBEF9DEEC8DC@kuehlewind.net> From: Bob Briscoe Message-ID: Date: Sat, 10 Jul 2021 22:13:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------0D7D0624489156F45105AE18" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Jul 2021 21:13:56 -0000 This is a multi-part message in MIME format. --------------0D7D0624489156F45105AE18 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Yoshi, Richard, Mirja, tcpm list, We came to a good consensus in late March on the conditions to place around ACKing ACKs in order to keep congestion feedback flowing, but to ensure ping-pong ACKs are rapidly damped as well as avoiding false DupACK detection. I have written that up in a rev of the draft, which I will post separately so as not to distract from the question I have here... ...While writing up, I realized we had glossed over a possibly more important point, when we proposed to change the number in the basic rule from 2 to 3, as follows: An AccECN Data Receiver MUST immediately send an ACK once 'n' CE marks have arrived since the previous ACK, where 'n' SHOULD be 3 and MUST be in the range 3 to 6 inclusive. I'd like to suggest an alternative: An AccECN Data Receiver MUST immediately send an ACK once 'n' CE marks have arrived since the previous ACK. If there is new data to acknowledge, 'n' SHOULD be 2. If there is no new data to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3. In either case, 'n' MUST be no greater than 6. Rationale: The data ACKs case shouldn't be compromised by the ACKs of ACKs case. When we were originally only thinking of the data ACKs case (before we realized this rule might trigger ACKs of ACKs), we recommended n=2  because it ensures congestion information is kept fresh, and during runs of congestion it generates more ACKs, which might make it more likely that at least one ACK survives some types of coalescing before the ACE field wraps. Also 2 is the default for the equivalent repetition of DCTCP feedback. Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK ping-pong, so we recommended 3. Without really discussing the pros and cons, we extended 3 to all cases (both data ACKs and ACKs of ACKs). I suspect the main reason was that it is just simpler to have a single rule. But these two cases are likely to be controlled from different parts of the code anyway. I've also realized that, in the data ACKs case, it was wrong to set a lower bound on 'n', let alone a /mandatory/ lower bound. There might be scenarios where it makes sense to trigger an ACK every CE mark; we have no reason to stop implementers doing that if they want. A lower bound could even be perversely read to mean that you're not allowed to immediately ACK data until you've received 'n' CE marks. Thoughts? Bob -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------0D7D0624489156F45105AE18 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Yoshi, Richard, Mirja, tcpm list,

We came to a good consensus in late March on the conditions to place around ACKing ACKs in order to keep congestion feedback flowing, but to ensure ping-pong ACKs are rapidly damped as well as avoiding false DupACK detection. I have written that up in a rev of the draft, which I will post separately so as not to distract from the question I have here...
...While writing up, I realized we had glossed over a possibly more important point, when we proposed to change the number in the basic rule from 2 to 3, as follows:
      An AccECN Data Receiver MUST immediately send an ACK once 'n' 
      CE marks have arrived since the previous ACK, where 'n' SHOULD 
      be 3 and MUST be in the range 3 to 6 inclusive.

I'd like to suggest an alternative:
      An AccECN Data Receiver MUST immediately send an ACK once 'n' 
      CE marks have arrived since the previous ACK. If there is new 
      data to acknowledge, 'n' SHOULD be 2.  If there is no new data 
      to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3. 
      In either case, 'n' MUST be no greater than 6.

Rationale: The data ACKs case shouldn't be compromised by the ACKs of ACKs case.

When we were originally only thinking of the data ACKs case (before we realized this rule might trigger ACKs of ACKs), we recommended n=2  because it ensures congestion information is kept fresh, and during runs of congestion it generates more ACKs, which might make it more likely that at least one ACK survives some types of coalescing before the ACE field wraps. Also 2 is the default for the equivalent repetition of DCTCP feedback.

Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK ping-pong, so we recommended 3. Without really discussing the pros and cons, we extended 3 to all cases (both data ACKs and ACKs of ACKs). I suspect the main reason was that it is just simpler to have a single rule. But these two cases are likely to be controlled from different parts of the code anyway.

I've also realized that, in the data ACKs case, it was wrong to set a lower bound on 'n', let alone a /mandatory/ lower bound. There might be scenarios where it makes sense to trigger an ACK every CE mark; we have no reason to stop implementers doing that if they want. A lower bound could even be perversely read to mean that you're not allowed to immediately ACK data until you've received 'n' CE marks.

Thoughts?



Bob
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------0D7D0624489156F45105AE18-- From nobody Sat Jul 10 22:38:37 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C09623A1EAE for ; Sat, 10 Jul 2021 22:38:35 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.69 X-Spam-Level: X-Spam-Status: No, score=-0.69 tagged_above=-999 required=5 tests=[NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bBIlGXZLvhqf for ; Sat, 10 Jul 2021 22:38:30 -0700 (PDT) Received: from mx36-out10.antispamcloud.com (mx36-out10.antispamcloud.com [209.126.121.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E00373A1EAC for ; Sat, 10 Jul 2021 22:38:29 -0700 (PDT) Received: from xse7.mail2web.com ([66.113.196.7] helo=xse.mail2web.com) by mx135.antispamcloud.com with esmtp (Exim 4.92) (envelope-from ) id 1m2SAb-0016wb-DG for tcpm@ietf.org; Sun, 11 Jul 2021 07:38:26 +0200 Received: from xsmtp22.mail2web.com (unknown [10.100.68.61]) by xse.mail2web.com (Postfix) with ESMTPS id 4GMwfl2lqCzDPZ for ; Sat, 10 Jul 2021 22:38:23 -0700 (PDT) Received: from [10.5.2.49] (helo=xmail11.myhosting.com) by xsmtp22.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1m2SAZ-0001PF-8t for tcpm@ietf.org; Sat, 10 Jul 2021 22:38:23 -0700 Received: (qmail 11763 invoked from network); 11 Jul 2021 05:38:22 -0000 Received: from unknown (HELO [192.168.1.101]) (Authenticated-user:_huitema@huitema.net@[172.58.43.16]) (envelope-sender ) by xmail11.myhosting.com (qmail-ldap-1.03) with ESMTPA for ; 11 Jul 2021 05:38:22 -0000 To: Bob Briscoe , Yoshifumi Nishida , "tcpm@ietf.org" , Mirja Kuehlewind , "rs.ietf@gmx.at" Cc: "nishida@sfc.wide.ad.jp" References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <8151fdef-ae78-80f3-adfc-d40db878ac8e@gmx.at> <48c5910d-5340-acd6-8fd9-fff1b7758310@bobbriscoe.net> <99B71B9A-EC9B-47FD-A149-FBEF9DEEC8DC@kuehlewind.net> From: Christian Huitema Message-ID: Date: Sat, 10 Jul 2021 22:38:21 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Language: en-US X-Originating-IP: 66.113.196.7 X-Spampanel-Domain: xsmtpout.mail2web.com X-Spampanel-Username: 66.113.196.0/24 Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.196.0/24@xsmtpout.mail2web.com X-Spampanel-Outgoing-Class: unsure X-Spampanel-Outgoing-Evidence: Combined (0.15) X-Recommended-Action: accept X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT9WLQux0N3HQm8ltz8rnu+BPUtbdvnXkggZ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5yjTC8Fov82/EJuxz8FihBPKj/EwzSHE5FGYwwjsNRPCCma DytoMWzyPlKVtFE64rTmD6wdmZPcItWbGe10hXJtXL4FsauCVkDjmcYJdU3yWp7KuHNaaKdg7iBE ZefdsNUFWKwa/wzJUjmazeC7Imcapebr0kNyYC289u5HlaNj1BQ6V51u76v35b1wNe/MvdL/hXir I7jpLA3NtNK1rbkD2+J9PgaoF8SQHto3le4zsHTaeQtlKubP6iUTjj6yPARK6buALVaA782LKxg6 vRmng8N1aLhXqdc+jC1RcnVud53D5caUhbVtvqItBqoizkEt9O20UjkwI0v+LOlw05G4BS+iyyNq bT8dUMXMJ4tUCMj6G37ZfAMLceP5aNHPt26RBupu5v1nytoNnc138GfEJRQ2qC7jjynPIHPNqSn4 QTXUjLjYWQt1/5xnQymMoPsgr/U0flMcy2Vi/IcBgY4arPaiJ1W6hAyiRC61jekdwIcXNugoOEbH RyFULpSjm7jZ1h/HfDRQ5Ig8VhPsPE8NlkBmbR1LS6Kx8w5MHqDEE4BBJaBtD32RALQCSg1oDtMl PyREjmB5EAdXVBvzPnf7DRojSVizNl0ce/s7u0P9b9Tml6eOMCV9kYYwkPx6ZsXvIUzTXkDAiiJi mGhLUFuSOTgtjQWHblEKb/bSn512w9qoLbHrX5KHDBoYpFP6SoHZcpPgEJKLbDyaC/LdLvvY/Vzw IcESbxiIxnB7tuHetfpVB9v9zY0h8asEYmbGGsJkWjQ4xyeNtxxq2TXT/AfNgWH7GTOjKdjx/QnC QSBbKRX0XF3qP/dj48psOHFCwviQxKSBCGH0S84CnKX/NUAV3jR5NeVaJQBh0uawl0Cg8nrgSS6D 7FozWS6JHKREtqdEPzY64lXv2dr2sny4a4Sj0cqzvWDlDrFILmNCVZ/264kd2x35zAiBFPp64JaI ysAWfpirH8g1GOR1IFGt5BWm X-Report-Abuse-To: spam@quarantine11.antispamcloud.com Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 05:38:36 -0000 On 7/10/2021 2:13 PM, Bob Briscoe wrote: > Yoshi, Richard, Mirja, tcpm list, > > We came to a good consensus in late March on the conditions to place=20 > around ACKing ACKs in order to keep congestion feedback flowing, but=20 > to ensure ping-pong ACKs are rapidly damped as well as avoiding false=20 > DupACK detection. I have written that up in a rev of the draft, which=20 > I will post separately so as not to distract from the question I have=20 > here... > ...While writing up, I realized we had glossed over a possibly more=20 > important point, when we proposed to change the number in the basic=20 > rule from 2 to 3, as follows: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 An AccECN Data Receiver MUST immediately= send an ACK once 'n' > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CE marks have arrived since the previous= ACK, where 'n' SHOULD > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 be 3 and MUST be in the range 3 to 6 inc= lusive. > > > I'd like to suggest an alternative: > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 An AccECN Data Receiver MUST immediately= send an ACK once 'n' > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CE marks have arrived since the previous= ACK. If there is new > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 data to acknowledge, 'n' SHOULD be 2.=C2= =A0 If there is no new data > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 to acknowledge, 'n' SHOULD be 3 and MUST= be no less than 3. > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 In either case, 'n' MUST be no greater t= han 6. > > > Rationale: The data ACKs case shouldn't be compromised by the ACKs of=20 > ACKs case. > > When we were originally only thinking of the data ACKs case (before we = > realized this rule might trigger ACKs of ACKs), we recommended n=3D2=C2= =A0=20 > because it ensures congestion information is kept fresh, and during=20 > runs of congestion it generates more ACKs, which might make it more=20 > likely that at least one ACK survives some types of coalescing before=20 > the ACE field wraps. Also 2 is the default for the equivalent=20 > repetition of DCTCP feedback. > > Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK=20 > ping-pong, so we recommended 3. Without really discussing the pros and = > cons, we extended 3 to all cases (both data ACKs and ACKs of ACKs). I=20 > suspect the main reason was that it is just simpler to have a single=20 > rule. But these two cases are likely to be controlled from different=20 > parts of the code anyway. > > I've also realized that, in the data ACKs case, it was wrong to set a=20 > lower bound on 'n', let alone a /mandatory/ lower bound. There might=20 > be scenarios where it makes sense to trigger an ACK every CE mark; we=20 > have no reason to stop implementers doing that if they want. A lower=20 > bound could even be perversely read to mean that you're not allowed to = > immediately ACK data until you've received 'n' CE marks. > > Thoughts?=20 I am looking at this from the QUIC point of view. In general, frequent=20 ACKs provide better control, but there are two points of tension:=20 managing low bandwidth return paths; and, reducing the impact of ACK on=20 performance. There are a number of scenarios in which the return path used by ACKs=20 has a much lower bandwidth than the data path. If ACKs are two frequent, = the return path can become congested. ACKing every two or 3 packets is=20 only sustainable if the return path bandwidth is not too small compared=20 to the data path -- 1/10th is generally sustainable, but lower that=20 1/20th and the ACKs have to be spaced. The alternative to spacing would=20 be increased RTT due to congestion on the return path, and random losses = of ACKs. Of course, even with ACK spacing, there are limits to=20 asymmetry: control becomes loose if the transport does not receive=20 multiple ACKs per RTT. ACK frequency has been found a key factor in the performance of QUIC=20 implementations. For example, implementations may achieve 5Gbps with=20 spaced ACK, but the performance drops to a few 100 Mbps if we insist on=20 an ACK for 2 packets. High performance requires sending packets in=20 batches, using API like UDP GSO, and performance seems best when the=20 frequency of ACKs more or less matches the frequency of batches. -- Christian Huitema From nobody Sun Jul 11 04:43:24 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7E58B3A091F for ; Sun, 11 Jul 2021 04:43:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.197 X-Spam-Level: X-Spam-Status: No, score=-0.197 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RKScQQNqS8zu for ; Sun, 11 Jul 2021 04:43:19 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CFA513A091D for ; Sun, 11 Jul 2021 04:43:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=I0HVX2uInLbob16zeCrny+m58LAVdgqMHxC44+W1rY4=; b=vsjUDhTnQtFvJ7M3Nl5uTLW3Sv gln18Yi41qV95TVTVCcIUfYUE8W2g0khkk+viL+7VnpYHY+zBdY+Ullo5BTM0S06yDp5tz4R5gxeL moqP9NOII+po3T6MaKUZAjvX84SPuDScL0Jx8vfUxO/KjppzUkkSQb2lu4dZTV4m+d3KsBp4vZUHu HA5uFhotES2+G+cLAZiX+3nTfg1cs3wkGitkhMGmUjBveBwYRc/BAM+rUU2N+UAJTg1W4mS4FE7yV YTonSp9SstDfj8V+ucYmbhyiN8W+ibbYcMknsBR2Szd1rBQsluCboBxaoHoR2pV7uRv0W8h9XCfrQ wxyXyupA==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:58474 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m2Xrf-0007Sp-Ii; Sun, 11 Jul 2021 12:43:15 +0100 To: Christian Huitema , Yoshifumi Nishida , "tcpm@ietf.org" , Mirja Kuehlewind , "rs.ietf@gmx.at" Cc: "nishida@sfc.wide.ad.jp" References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <8151fdef-ae78-80f3-adfc-d40db878ac8e@gmx.at> <48c5910d-5340-acd6-8fd9-fff1b7758310@bobbriscoe.net> <99B71B9A-EC9B-47FD-A149-FBEF9DEEC8DC@kuehlewind.net> From: Bob Briscoe Message-ID: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> Date: Sun, 11 Jul 2021 12:43:11 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 11:43:24 -0000 Christian, On 11/07/2021 06:38, Christian Huitema wrote: > > On 7/10/2021 2:13 PM, Bob Briscoe wrote: >> Yoshi, Richard, Mirja, tcpm list, >> >> We came to a good consensus in late March on the conditions to place >> around ACKing ACKs in order to keep congestion feedback flowing, but >> to ensure ping-pong ACKs are rapidly damped as well as avoiding false >> DupACK detection. I have written that up in a rev of the draft, which >> I will post separately so as not to distract from the question I have >> here... >> ...While writing up, I realized we had glossed over a possibly more >> important point, when we proposed to change the number in the basic >> rule from 2 to 3, as follows: >> >>       An AccECN Data Receiver MUST immediately send an ACK once 'n' >>       CE marks have arrived since the previous ACK, where 'n' SHOULD >>       be 3 and MUST be in the range 3 to 6 inclusive. >> >> >> I'd like to suggest an alternative: >> >>       An AccECN Data Receiver MUST immediately send an ACK once 'n' >>       CE marks have arrived since the previous ACK. If there is new >>       data to acknowledge, 'n' SHOULD be 2.  If there is no new data >>       to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3. >>       In either case, 'n' MUST be no greater than 6. >> >> >> Rationale: The data ACKs case shouldn't be compromised by the ACKs of >> ACKs case. >> >> When we were originally only thinking of the data ACKs case (before >> we realized this rule might trigger ACKs of ACKs), we recommended >> n=2  because it ensures congestion information is kept fresh, and >> during runs of congestion it generates more ACKs, which might make it >> more likely that at least one ACK survives some types of coalescing >> before the ACE field wraps. Also 2 is the default for the equivalent >> repetition of DCTCP feedback. >> >> Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK >> ping-pong, so we recommended 3. Without really discussing the pros >> and cons, we extended 3 to all cases (both data ACKs and ACKs of >> ACKs). I suspect the main reason was that it is just simpler to have >> a single rule. But these two cases are likely to be controlled from >> different parts of the code anyway. >> >> I've also realized that, in the data ACKs case, it was wrong to set a >> lower bound on 'n', let alone a /mandatory/ lower bound. There might >> be scenarios where it makes sense to trigger an ACK every CE mark; we >> have no reason to stop implementers doing that if they want. A lower >> bound could even be perversely read to mean that you're not allowed >> to immediately ACK data until you've received 'n' CE marks. >> >> Thoughts? > > [CH] I am looking at this from the QUIC point of view. In general, > frequent ACKs provide better control, but there are two points of > tension: managing low bandwidth return paths; and, reducing the impact > of ACK on performance. > > There are a number of scenarios in which the return path used by ACKs > has a much lower bandwidth than the data path. If ACKs are two > frequent, the return path can become congested. ACKing every two or 3 > packets is only sustainable if the return path bandwidth is not too > small compared to the data path -- 1/10th is generally sustainable, > but lower that 1/20th and the ACKs have to be spaced. The alternative > to spacing would be increased RTT due to congestion on the return > path, and random losses of ACKs. Of course, even with ACK spacing, > there are limits to asymmetry: control becomes loose if the transport > does not receive multiple ACKs per RTT. > > ACK frequency has been found a key factor in the performance of QUIC > implementations. For example, implementations may achieve 5Gbps with > spaced ACK, but the performance drops to a few 100 Mbps if we insist > on an ACK for 2 packets. High performance requires sending packets in > batches, using API like UDP GSO, and performance seems best when the > frequency of ACKs more or less matches the frequency of batches. [BB] I think a brief summary of AccECN might be in order here - it seems like you're starting from scratch, but we're aware of all the above. The implementation will have its own ACK ratio - that's out of scope of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bit counter of CE-marks (which in the worst case of 100% marking in the data direction could then induce 1 ACK per 6 packets). This shouldn't limit forward performance, because it only increases the reverse ACK rate if there is heavy congestion in the forward path, when the Data Sender should be reducing the forward rate anyway. AccECN can also use the AccECN TCP Option that provides a much larger counter (24-bit), but AccECN has to work reasonably well with just the 3-bit counter in the main TCP header in case the option can't traverse the path. However, the Data Receiver doesn't know whether the option is reaching the other end, so it has to assume it might not be. My original question asks what the recommended value of 'n' should be for how much the counter of CE-marked data packets should be allowed to increment between ACKs. We don't have to recommend anything, but it helps implementers if we do. AccECN is TCP only. So when we originally recommended 2, we figured that TCP's default ACK ratio is 2. So, even if ECN marking in the forward direction was 100%, a default increment of 2 CE marked packets ('n') wouldn't cause any more frequent ACKs than the default. Nonetheless, if the implementer has chosen a longer ACK ratio (or perhaps allowed the ratio to be controlled by the sender e.g. draft-gomez-tcpm-ack-rate-request) they don't have to use the recommended n=2; the proposed wording allows a range. Don't worry, we certainly wouldn't "insist on an ACK for 2 packets". Also, FYI, AccECN provides feedback about ECN marking on the ACK stream, back to the Data Receiver who is generating the ACKs. It then has a hook on which it can hang feedback control of TCP's ACK stream without waiting for anyone else to deploy anything (it's up to the implementer or the IETF to add this if they want). However, the scope of the AccECN spec is deliberately wire protocol only, so it doesn't say what to do with this feedback. Cheers Bob > > -- Christian Huitema > > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ From nobody Sun Jul 11 08:54:55 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A3BF33A11C5 for ; Sun, 11 Jul 2021 08:54:53 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.013 X-Spam-Level: X-Spam-Status: No, score=0.013 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8AXqkinlWM7W for ; Sun, 11 Jul 2021 08:54:49 -0700 (PDT) Received: from mx36-out10.antispamcloud.com (mx36-out10.antispamcloud.com [209.126.121.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 41CA93A11C3 for ; Sun, 11 Jul 2021 08:54:48 -0700 (PDT) Received: from xse169.mail2web.com ([66.113.196.169] helo=xse.mail2web.com) by mx133.antispamcloud.com with esmtp (Exim 4.92) (envelope-from ) id 1m2bn2-001AcW-Iu for tcpm@ietf.org; Sun, 11 Jul 2021 17:54:45 +0200 Received: from xsmtp21.mail2web.com (unknown [10.100.68.60]) by xse.mail2web.com (Postfix) with ESMTPS id 4GNBKq1Mp1zLNw for ; Sun, 11 Jul 2021 08:54:39 -0700 (PDT) Received: from [10.5.2.49] (helo=xmail11.myhosting.com) by xsmtp21.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1m2bmx-0006lB-2r for tcpm@ietf.org; Sun, 11 Jul 2021 08:54:39 -0700 Received: (qmail 28795 invoked from network); 11 Jul 2021 15:54:38 -0000 Received: from unknown (HELO smtpclient.apple) (Authenticated-user:_huitema@huitema.net@[172.58.43.16]) (envelope-sender ) by xmail11.myhosting.com (qmail-ldap-1.03) with ESMTPA for ; 11 Jul 2021 15:54:38 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Christian Huitema Mime-Version: 1.0 (1.0) Date: Sun, 11 Jul 2021 08:54:37 -0700 Message-Id: <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> Cc: Yoshifumi Nishida , tcpm@ietf.org, Mirja Kuehlewind , rs.ietf@gmx.at, nishida@sfc.wide.ad.jp In-Reply-To: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> To: Bob Briscoe X-Mailer: iPhone Mail (18F72) X-Originating-IP: 66.113.196.169 X-Spampanel-Domain: xsmtpout.mail2web.com X-Spampanel-Username: 66.113.196.0/24 Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.196.0/24@xsmtpout.mail2web.com X-Spampanel-Outgoing-Class: unsure X-Spampanel-Outgoing-Evidence: Combined (0.15) X-Recommended-Action: accept X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT9WLQux0N3HQm8ltz8rnu+BPUtbdvnXkggZ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5yjTC8Fov82/EJuxz8FihBPKj/EwzSHE5FGYwwjsNRPCIf2 C/+5oaXB4LmK7oUXREfmD6wdmZPcItWbGe10hXJtXL4FsauCVkDjmcYJdU3yWp7KuHNaaKdg7iBE ZefdsNUFWKwa/wzJUjmazeC7Imcapebr0kNyYC289u5HlaNj1BQ6V51u76v35b1wNe/MvdKAGdwU TZKlze5ERymXAD3v2+J9PgaoF8SQHto3le4zsHTaeQtlKubP6iUTjj6yPARK6buALVaA782LKxg6 vRmng8N1aLhXqdc+jC1RcnVud53D5caUhbVtvqItBqoizkEt9O20UjkwI0v+LOlw05G4BS+iyyNq bT8dUMXMJ4tUCMj6G37ZfAMLceP5aNHPt26RBupu5v1nytoNnc138GfEJRQ2qC7jjynPIHPNqSn4 QTXUjLjYWQt1/5xnQymMoPsgr/U0flMcy2Vi/IcBgY4arPaiJ1W6hAyiRC61jekdwIcXNugoOEbH RyFULpSjm7jZ1h/HfDRQ5Ig8VhPsPE8NaP2gA77cO7WeI9Ftai6fuoBBJaBtD32RALQCSg1oDtOl fOBg6oyRcc61yKYNvDGqDRojSVizNl0ce/s7u0P9b9Tml6eOMCV9kYYwkPx6ZsXvIUzTXkDAiiJi mGhLUFuS2lhaIetXfCg1JdAVrOwKfG1URGUMPfpOEfupo44Glz9UoFIvD3sIcP1fhJPM6B/8FXX1 YOP/O5KiF8LgVDF1JIEcSSViKbm9TDKvWYmkK3CJ+dym1L8cD17Js0v4cp1M66lXricRzNzTJabt hC1ICDcKVNeVJ9BXyu9+ceCqThTYg2px1fSoqxQCCHnLMo/m9VKh99btUAanjnMCAH2co+fBoeG+ Hs0afhsY/5zhNYWRVYKU9W9tbmVXJBqdHHDmZEKhyNAv1N35kYWaEdgLurFV5oTvAcwA4rM3FkfW 8/1kE/e7sUnsVpINvARNxpFO X-Report-Abuse-To: spam@quarantine11.antispamcloud.com Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 15:54:54 -0000 =20 > On Jul 11, 2021, at 4:42 AM, Bob Briscoe wrote: >=20 > =EF=BB=BFChristian, >=20 >> On 11/07/2021 06:38, Christian Huitema wrote: >>=20 >>> On 7/10/2021 2:13 PM, Bob Briscoe wrote: >>> Yoshi, Richard, Mirja, tcpm list, >>>=20 >>> We came to a good consensus in late March on the conditions to place aro= und ACKing ACKs in order to keep congestion feedback flowing, but to ensure p= ing-pong ACKs are rapidly damped as well as avoiding false DupACK detection.= I have written that up in a rev of the draft, which I will post separately s= o as not to distract from the question I have here... >>> ...While writing up, I realized we had glossed over a possibly more impo= rtant point, when we proposed to change the number in the basic rule from 2 t= o 3, as follows: >>>=20 >>> An AccECN Data Receiver MUST immediately send an ACK once 'n' >>> CE marks have arrived since the previous ACK, where 'n' SHOULD >>> be 3 and MUST be in the range 3 to 6 inclusive. >>>=20 >>>=20 >>> I'd like to suggest an alternative: >>>=20 >>> An AccECN Data Receiver MUST immediately send an ACK once 'n' >>> CE marks have arrived since the previous ACK. If there is new >>> data to acknowledge, 'n' SHOULD be 2. If there is no new data >>> to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3. >>> In either case, 'n' MUST be no greater than 6. >>>=20 >>>=20 >>> Rationale: The data ACKs case shouldn't be compromised by the ACKs of AC= Ks case. >>>=20 >>> When we were originally only thinking of the data ACKs case (before we r= ealized this rule might trigger ACKs of ACKs), we recommended n=3D2 because= it ensures congestion information is kept fresh, and during runs of congest= ion it generates more ACKs, which might make it more likely that at least on= e ACK survives some types of coalescing before the ACE field wraps. Also 2 i= s the default for the equivalent repetition of DCTCP feedback. >>>=20 >>> Then, we noticed the ACKs of ACKs case, and we wanted to damp any ACK pi= ng-pong, so we recommended 3. Without really discussing the pros and cons, w= e extended 3 to all cases (both data ACKs and ACKs of ACKs). I suspect the m= ain reason was that it is just simpler to have a single rule. But these two c= ases are likely to be controlled from different parts of the code anyway. >>>=20 >>> I've also realized that, in the data ACKs case, it was wrong to set a lo= wer bound on 'n', let alone a /mandatory/ lower bound. There might be scenar= ios where it makes sense to trigger an ACK every CE mark; we have no reason t= o stop implementers doing that if they want. A lower bound could even be per= versely read to mean that you're not allowed to immediately ACK data until y= ou've received 'n' CE marks. >>>=20 >>> Thoughts?=20 >>=20 >> [CH] I am looking at this from the QUIC point of view. In general, freque= nt ACKs provide better control, but there are two points of tension: managin= g low bandwidth return paths; and, reducing the impact of ACK on performance= . >>=20 >> There are a number of scenarios in which the return path used by ACKs has= a much lower bandwidth than the data path. If ACKs are two frequent, the re= turn path can become congested. ACKing every two or 3 packets is only sustai= nable if the return path bandwidth is not too small compared to the data pat= h -- 1/10th is generally sustainable, but lower that 1/20th and the ACKs hav= e to be spaced. The alternative to spacing would be increased RTT due to con= gestion on the return path, and random losses of ACKs. Of course, even with A= CK spacing, there are limits to asymmetry: control becomes loose if the tran= sport does not receive multiple ACKs per RTT. >>=20 >> ACK frequency has been found a key factor in the performance of QUIC impl= ementations. For example, implementations may achieve 5Gbps with spaced ACK,= but the performance drops to a few 100 Mbps if we insist on an ACK for 2 pa= ckets. High performance requires sending packets in batches, using API like U= DP GSO, and performance seems best when the frequency of ACKs more or less m= atches the frequency of batches. >=20 > [BB] I think a brief summary of AccECN might be in order here - it seems l= ike you're starting from scratch, but we're aware of all the above. >=20 > The implementation will have its own ACK ratio - that's out of scope of Ac= cECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bi= t counter of CE-marks (which in the worst case of 100% marking in the data d= irection could then induce 1 ACK per 6 packets). This shouldn't limit forwar= d performance, because it only increases the reverse ACK rate if there is he= avy congestion in the forward path, when the Data Sender should be reducing t= he forward rate anyway. 1 ACK per 6 packets would cause performance issues on high speed links. Comm= on setup is "4 to 8 ACK per RTT", which means intervals much larger than 6 p= ackets.=20 >=20 > AccECN can also use the AccECN TCP Option that provides a much larger coun= ter (24-bit), but AccECN has to work reasonably well with just the 3-bit cou= nter in the main TCP header in case the option can't traverse the path. Howe= ver, the Data Receiver doesn't know whether the option is reaching the other= end, so it has to assume it might not be. In QUIC, the ECN counters are 62 bits, so I guess the guidance can be much l= ooser. >=20 > My original question asks what the recommended value of 'n' should be for h= ow much the counter of CE-marked data packets should be allowed to increment= between ACKs. We don't have to recommend anything, but it helps implementer= s if we do. AccECN is TCP only. So when we originally recommended 2, we figu= red that TCP's default ACK ratio is 2. So, even if ECN marking in the forwar= d direction was 100%, a default increment of 2 CE marked packets ('n') would= n't cause any more frequent ACKs than the default. The default ACK ratio in TCP is one thing, the practical ratio is another. O= n many paths, middleboxes perform ACK thinning, pruning ACKs to improve perf= ormance. A three bit counter is not very robust against such thinning. To be= robust, the counter should be wide enough to accumulate an RTT worth of ECN= marks.=20 >=20 > Nonetheless, if the implementer has chosen a longer ACK ratio (or perhaps a= llowed the ratio to be controlled by the sender e.g. draft-gomez-tcpm-ack-ra= te-request) they don't have to use the recommended n=3D2; the proposed wordi= ng allows a range. >=20 > Don't worry, we certainly wouldn't "insist on an ACK for 2 packets". >=20 >=20 > Also, FYI, AccECN provides feedback about ECN marking on the ACK stream, b= ack to the Data Receiver who is generating the ACKs. It then has a hook on w= hich it can hang feedback control of TCP's ACK stream without waiting for an= yone else to deploy anything (it's up to the implementer or the IETF to add t= his if they want). However, the scope of the AccECN spec is deliberately wir= e protocol only, so it doesn't say what to do with this feedback. Yes, using ECN on the ACK stream has potential. There has been very little r= esearch on congestion control for ACK. I wish there would be more.=20 -- Christian Huitema=20= From nobody Sun Jul 11 11:28:54 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E5BA53A189A for ; Sun, 11 Jul 2021 11:28:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -16.197 X-Spam-Level: X-Spam-Status: No, score=-16.197 tagged_above=-999 required=5 tests=[DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fp9arDgzSRFg for ; Sun, 11 Jul 2021 11:28:47 -0700 (PDT) Received: from mail-ua1-x92f.google.com (mail-ua1-x92f.google.com [IPv6:2607:f8b0:4864:20::92f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 33C6E3A1899 for ; Sun, 11 Jul 2021 11:28:47 -0700 (PDT) Received: by mail-ua1-x92f.google.com with SMTP id r9so6150771ual.7 for ; Sun, 11 Jul 2021 11:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XDESUxuxmcXsY8ZalRAEDtUCWgY8jSGl5t5PEWUp1zw=; b=nXSncw0ZUr+xzEXy5Oprzf1pCYu/endIeKUAJsXR+JKIFnP+/qoA0toKQv1OofZYXM qkm6AJ1KmhUMOXOLq7l6SwUitLGrMUyrickbtCUl3yDegDKUBBSCJZshpVSm+cJ2Pux6 qivDWf9T4rXqmkClJhKqnX6MPosXDTz9JzMdlO1Nw76KIirdu4QDAlNJDDHBzMCWfU5A N++gDVGQAssBuEUSEOCPZXHSMsZP6aOV5qoVcTXjvYz0v22n4ivomzvHH4keQ7vNxs9r a9mfoPinR4VPUwJIxGyr6QxfakmT94xnPiBf2MysOBAiw4sTMmT0NAjutRWzSbesqYq2 LfOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XDESUxuxmcXsY8ZalRAEDtUCWgY8jSGl5t5PEWUp1zw=; b=Vf4+aYUNyflpZxdO9SrjMBjdXtcLD8Evt5mjm9hNnKez075jSCglg5FpEaQg+udj1O H5OKFGKLZJraV5P1D+DDu3ad4M4g7mzg7oV1FCjjy68n3BUSkaH2Xko4gzhNWESIFPaI 9d8Sac7DEqQlkTlxxsKIiTy0+k6YUpWourjvjGhJZH7ZzTWCf3IxnsizxYJ4AIK2hp+j Ze1lMy4a69OlmAzHdwFqfLylew/KQ9Cn48oHtb2n4GPNvpXut6BCYtoLrwYaPl8PPEHS 1zqdruP0iXLTpsJAxgc2oCeGX6zrIkHi0g6ovNJLmrsE9xpOCiHuir8CAW0gj1jO7a7n 8woA== X-Gm-Message-State: AOAM531vi2qwcFdJGn8fF1qjiANTF+ziav3XOJhMcTsNG4Lue0E9k9wi ivmSf/98negOFSHujBxcdRqO5gCT7bAgpdKZnNVna0zIumHbzw== X-Google-Smtp-Source: ABdhPJwbQMFgSgE/wEzG7qbRrzICPLp9fWen4RG5aTShEIKqmW0E3LZoQeHCE1DrlABxzH5pxej95+mwuT+clw4zsXc= X-Received: by 2002:ab0:25c5:: with SMTP id y5mr28103653uan.142.1626028125317; Sun, 11 Jul 2021 11:28:45 -0700 (PDT) MIME-Version: 1.0 References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> In-Reply-To: <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> From: Neal Cardwell Date: Sun, 11 Jul 2021 14:28:28 -0400 Message-ID: To: Christian Huitema Cc: Bob Briscoe , tcpm@ietf.org, Mirja Kuehlewind , nishida@sfc.wide.ad.jp, Yuchung Cheng Content-Type: multipart/alternative; boundary="000000000000ce8ea905c6dd2c31" Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Jul 2021 18:28:53 -0000 --000000000000ce8ea905c6dd2c31 Content-Type: text/plain; charset="UTF-8" On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema wrote: > > > > On Jul 11, 2021, at 4:42 AM, Bob Briscoe wrote: > ... > > The implementation will have its own ACK ratio - that's out of scope of > AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the > 3-bit counter of CE-marks (which in the worst case of 100% marking in the > data direction could then induce 1 ACK per 6 packets). This shouldn't limit > forward performance, because it only increases the reverse ACK rate if > there is heavy congestion in the forward path, when the Data Sender should > be reducing the forward rate anyway. > > 1 ACK per 6 packets would cause performance issues on high speed links. > Common setup is "4 to 8 ACK per RTT", which means intervals much larger > than 6 packets. I share this concern about the AccECN requirement of one ACK per 6 CE-marked data segments potentially causing performance issues with high-speed links. Today, with most high-speed last-hop link technologies (wifi, cellular, Ethernet) TCP receivers often receive (from lower layers of the hardware and software networking stack) large aggregates (often up to 64KBytes or around 44 packets), and generate a single ACK for that aggregate. AFAICT changing this scenario to require 1 packet per 6 CE-marked data segments means that during CE-marked scenarios receiving such an aggregate would require ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x. I have two main concerns in that scenario: (1) CPU load: This seems like it would impose much higher CPU load on the data receiver, at just the moment when the receiver may already be under stress due to receiving data near the maximum rate of the link. (2) Congestion and ACK loss: This requirement for generating up to 8 ACKs per aggregate seems like it is likely to produce a tight burst of 8 ACKs, which is going to increase congestion and increase the odds of losing at least one ACK, which is going to cause accuracy problems for "Accurate" ECN. :-) I understand that this proposed "ACK per 6 CE-marked data segments" rule is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is one of the good arguments against including the ACE field in the AccECN design. The other main concerns I have with the ACE field are: o Complexity o Redundancy with respect to the AccECN counter options o Potential problems with middleboxes o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate ECN Linux Implementation: Experiences and Challenges", at the April 2020 TCPM interim meeting) I realize that if AccECN does not have the ACE field feature, then AccECN and TCP L4S will not be usable on paths with middleboxes that strip the AccECN counter options. But IMHO living without the ACE field is preferable. IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or (b) TCP connections where no middleboxes are stripping AccECN options. cheers, neal --000000000000ce8ea905c6dd2c31 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Jul 11, 2021 at 11:55 AM Chri= stian Huitema <huitema@huitema.ne= t> wrote:
=C2=A0

> On Jul 11, 2021, at 4:42 AM, Bob Briscoe <ietf@bobbriscoe.net> wrote:
...=C2=A0
> The implementation will have its own ACK ratio - that's out of sco= pe of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of= the 3-bit counter of CE-marks (which in the worst case of 100% marking in = the data direction could then induce 1 ACK per 6 packets). This shouldn'= ;t limit forward performance, because it only increases the reverse ACK rat= e if there is heavy congestion in the forward path, when the Data Sender sh= ould be reducing the forward rate anyway.

1 ACK per 6 packets would cause performance issues on high speed links. Com= mon setup is "4 to 8 ACK per RTT", which means intervals much lar= ger than 6 packets.

I share this concern ab= out the AccECN requirement of one ACK per 6 CE-marked data segments potenti= ally causing performance issues with high-speed links. Today, with most hig= h-speed last-hop link technologies (wifi, cellular, Ethernet) TCP receivers= often receive (from lower layers of the hardware and software networking s= tack) large aggregates=C2=A0 (often up to 64KBytes or around 44 packets), a= nd generate a single ACK for that aggregate. AFAICT changing this scenario = to require 1 packet per 6 CE-marked data segments means that during CE-mark= ed scenarios receiving such an aggregate would require ceil(44/6) =3D 8 ACK= s, increasing the number of ACKs by up to 8x.=C2=A0

I have two main concerns in that scenario:

(1) C= PU load: This seems like it would impose much higher CPU load on the data r= eceiver,=C2=A0at just the moment when the receiver may already be under str= ess due to receiving data near the maximum rate of the link.

=
(2) Congestion and ACK loss: This requirement for generating up = to 8 ACKs per aggregate seems like it is likely to produce a tight burst of= 8 ACKs, which is going to increase congestion and increase the odds of los= ing at least one ACK, which is going to cause accuracy problems for "A= ccurate" ECN. :-)

I understand that this prop= osed "ACK per 6 CE-marked data segments" rule is necessary to avo= id issues with the 3-bit ACE field wrapping. So IMHO this is one of the goo= d arguments against including the ACE field in the AccECN design.=C2=A0

The other main concerns I have with the ACE field are= :

o Complexity

o Redundan= cy with respect to the AccECN counter options

o Po= tential problems with middleboxes

o Known problems= with NICs and drivers (based on Ilpo's nice talk, "Accurate ECN Linux Implementation: = Experiences and Challenges", at the April 2020 TCPM interim meeting)

<= /span>
I re= alize that if AccECN does not have the ACE field feature, then AccECN and T= CP L4S will not be usable on paths with middleboxes that strip the A= ccECN counter options. But IMHO living without the ACE field is preferable.= IMHO it's acceptable to say that L4S can only be used with (a) QUIC, o= r (b) TCP connections where no middleboxes are stripping AccECN options.

cheers,
neal


=


--000000000000ce8ea905c6dd2c31-- From nobody Mon Jul 12 05:30:29 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8BB763A12E3 for ; Mon, 12 Jul 2021 05:30:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.196 X-Spam-Level: X-Spam-Status: No, score=-0.196 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JAlQdLrU0B99 for ; Mon, 12 Jul 2021 05:30:22 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0EBBD3A12DF for ; Mon, 12 Jul 2021 05:30:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=fYMXPBSNFzfIPm65WmKUWP2gIiHVCnT69bSi5VL4CPk=; b=0eJta7zUVHjv5zOWZCpMASUT01 UUZYxjUtfkH48i87ztliOEnLN9L2OsTyujyoYQhD1fcBAQZnUH80IIIyCyvWw2O2swfgxejwlM3l4 SWTRhIoSfIEHp0chuQe12xTOVd0LkOdVN2dGYiqXlot8s9cp9ihSXPTWxoNm5q+A7RDwMn1OTDljd IvLo1F9Xb2mijBcPHn6p1ofkGwAbQUPwmwh/fWlbLvnMlmjy5izg29c5FdF9vsMXVGquYKywc1UzH 80AettSryiyqivOCehYdL0B2xEgzrVAFE9VI5SV83vyZ34YWXclCel39Acl64fi75GSHG/uBjlT6m mAjstSoQ==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:33826 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m2v4n-0007MR-1R; Mon, 12 Jul 2021 13:30:19 +0100 To: Neal Cardwell Cc: tcpm@ietf.org, Mirja Kuehlewind , Yuchung Cheng , Richard Scheffenegger , Christian Huitema References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> From: Bob Briscoe Message-ID: <6ad1817b-2b58-8aee-2cf5-3654e33b3eda@bobbriscoe.net> Date: Mon, 12 Jul 2021 13:30:17 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------8D5073207E3D337646AE5982" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 12:30:28 -0000 This is a multi-part message in MIME format. --------------8D5073207E3D337646AE5982 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Neal, On 11/07/2021 19:28, Neal Cardwell wrote: > > > On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema > > wrote: > > > > > On Jul 11, 2021, at 4:42 AM, Bob Briscoe > wrote: > > ... > > > The implementation will have its own ACK ratio - that's out of > scope of AccECN, except to set this max of 6 CEs, which is to > mitigate wrap of the 3-bit counter of CE-marks (which in the worst > case of 100% marking in the data direction could then induce 1 ACK > per 6 packets). This shouldn't limit forward performance, because > it only increases the reverse ACK rate if there is heavy > congestion in the forward path, when the Data Sender should be > reducing the forward rate anyway. > > 1 ACK per 6 packets would cause performance issues on high speed > links. Common setup is "4 to 8 ACK per RTT", which means intervals > much larger than 6 packets. > > > I share this concern about the AccECN requirement of one ACK per 6 > CE-marked data segments potentially causing performance issues with > high-speed links. Today, with most high-speed last-hop link > technologies (wifi, cellular, Ethernet) TCP receivers often receive > (from lower layers of the hardware and software networking stack) > large aggregates  (often up to 64KBytes or around 44 packets), and > generate a single ACK for that aggregate. AFAICT changing this > scenario to require 1 packet per 6 CE-marked data segments means that > during CE-marked scenarios receiving such an aggregate would require > ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x. > > I have two main concerns in that scenario: > > (1) CPU load: This seems like it would impose much higher CPU load on > the data receiver, at just the moment when the receiver may already be > under stress due to receiving data near the maximum rate of the link. > > (2) Congestion and ACK loss: This requirement for generating up to 8 > ACKs per aggregate seems like it is likely to produce a tight burst of > 8 ACKs, which is going to increase congestion and increase the odds of > losing at least one ACK, which is going to cause accuracy problems for > "Accurate" ECN. :-) [BB] I hadn't intended this wording to apply in response to a large burst. I'd want to alter the wording to allow just one ACK in this case. > > I understand that this proposed "ACK per 6 CE-marked data segments" > rule is necessary to avoid issues with the 3-bit ACE field wrapping. > So IMHO this is one of the good arguments against including the ACE > field in the AccECN design. [BB] I think the subtext here is a preference for the DCTCP style of feedback vs ACE (when there is no AccECN TCP Option). I don't think this particular issue is any different between the two. With DCTCP feedback, when a large data burst like this arrives, if there are transitions to and from CE marking within the burst, does Linux DCTCP generate an ACK for each transition, and for every n repetitions of CE not ECT, like the RFC says it should? Or does the implementation just pop out one ACK at the end? Ilpo & I have been doing experiments with high levels of ACK coalescing causing the ACE field seen by the Data Sender to wrap multiple times under high congestion. For example, 1 ACK per ~4ms is one common scheme, which would result in about 1 ACK for 34 data packets at just 100Mb/s. > > The other main concerns I have with the ACE field are: > > o Complexity [BB] I understand there could be a tradeoff between complexity and accuracy. But I think the complexity of ACE and DCTCP feedback are pretty much the same but, with even low levels of ACK coalescing, the accuracy of ACE is superior. Nonetheless, I understand that deploying something different to DCTCP when you've already got DCTCP could involve deployment complexity. > > o Redundancy with respect to the AccECN counter options [BB] The redundancy is intended. For cases where the TCP Option doesn't traverse the path, or where TCP option space is limited. > > o Potential problems with middleboxes [BB] Well, we've tested the 3 header bits over millions of paths without problems. But yes, there are billions more to test. > > o Known problems with NICs and drivers (based on Ilpo's nice talk, > "Accurate ECN Linux Implementation: Experiences and Challenges", at > the April 2020 TCPM interim meeting) [BB] I understood that talk as concluding there weren't really problems. What specific problem do you have in mind? > I realize that if AccECN does not have the ACE field feature, then > AccECN and TCP L4S will not be usable on paths with middleboxes that > strip the AccECN counter options. But IMHO living without the ACE > field is preferable. IMHO it's acceptable to say that L4S can only be > used with (a) QUIC, or (b) TCP connections where no middleboxes are > stripping AccECN options. [BB] When you say 'living without the ACE field', alongside the AccECN TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback in its place? I don't think there's any case for using classic ECN feedback within AccECN. I see the competition as between ACE and DCTCP feedback, each optionally with the AccECN TCP Option. If DCTCP feedback were a stop-gap until we could get good traversal of the TCP Option, it might have some merit, but I think DCTCP isn't going to work well with the level of ACK coalescing in the Internet. However, this is very difficult to judge unless we do large scale A-B experiments on the real Internet (and potentially within DCs). I'll think further about enabling some way to do that, possibly within AccECN's negotiation framework. But today, I have to prioritize for the draft deadline. Cheers Bob > > cheers, > neal > > > > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------8D5073207E3D337646AE5982 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Neal,

On 11/07/2021 19:28, Neal Cardwell wrote:


On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema <huitema@huitema.net> wrote:
 

> On Jul 11, 2021, at 4:42 AM, Bob Briscoe <ietf@bobbriscoe.net> wrote:
... 
> The implementation will have its own ACK ratio - that's out of scope of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bit counter of CE-marks (which in the worst case of 100% marking in the data direction could then induce 1 ACK per 6 packets). This shouldn't limit forward performance, because it only increases the reverse ACK rate if there is heavy congestion in the forward path, when the Data Sender should be reducing the forward rate anyway.

1 ACK per 6 packets would cause performance issues on high speed links. Common setup is "4 to 8 ACK per RTT", which means intervals much larger than 6 packets.

I share this concern about the AccECN requirement of one ACK per 6 CE-marked data segments potentially causing performance issues with high-speed links. Today, with most high-speed last-hop link technologies (wifi, cellular, Ethernet) TCP receivers often receive (from lower layers of the hardware and software networking stack) large aggregates  (often up to 64KBytes or around 44 packets), and generate a single ACK for that aggregate. AFAICT changing this scenario to require 1 packet per 6 CE-marked data segments means that during CE-marked scenarios receiving such an aggregate would require ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x. 

I have two main concerns in that scenario:

(1) CPU load: This seems like it would impose much higher CPU load on the data receiver, at just the moment when the receiver may already be under stress due to receiving data near the maximum rate of the link.

(2) Congestion and ACK loss: This requirement for generating up to 8 ACKs per aggregate seems like it is likely to produce a tight burst of 8 ACKs, which is going to increase congestion and increase the odds of losing at least one ACK, which is going to cause accuracy problems for "Accurate" ECN. :-)

[BB] I hadn't intended this wording to apply in response to a large burst. I'd want to alter the wording to allow just one ACK in this case.


I understand that this proposed "ACK per 6 CE-marked data segments" rule is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is one of the good arguments against including the ACE field in the AccECN design.

[BB] I think the subtext here is a preference for the DCTCP style of feedback vs ACE (when there is no AccECN TCP Option). I don't think this particular issue is any different between the two.

With DCTCP feedback, when a large data burst like this arrives, if there are transitions to and from CE marking within the burst, does Linux DCTCP generate an ACK for each transition, and for every n repetitions of CE not ECT, like the RFC says it should? Or does the implementation just pop out one ACK at the end?

Ilpo & I have been doing experiments with high levels of ACK coalescing causing the ACE field seen by the Data Sender to wrap multiple times under high congestion. For example, 1 ACK per ~4ms is one common scheme, which would result in about 1 ACK for 34 data packets at just 100Mb/s.


The other main concerns I have with the ACE field are:

o Complexity

[BB] I understand there could be a tradeoff between complexity and accuracy. But I think the complexity of ACE and DCTCP feedback are pretty much the same but, with even low levels of ACK coalescing, the accuracy of ACE is superior. Nonetheless, I understand that deploying something different to DCTCP when you've already got DCTCP could involve deployment complexity.


o Redundancy with respect to the AccECN counter options

[BB] The redundancy is intended. For cases where the TCP Option doesn't traverse the path, or where TCP option space is limited.


o Potential problems with middleboxes

[BB] Well, we've tested the 3 header bits over millions of paths without problems. But yes, there are billions more to test.


o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate ECN Linux Implementation: Experiences and Challenges", at the April 2020 TCPM interim meeting)

[BB] I understood that talk as concluding there weren't really problems. What specific problem do you have in mind?

I realize that if AccECN does not have the ACE field feature, then AccECN and TCP L4S will not be usable on paths with middleboxes that strip the AccECN counter options. But IMHO living without the ACE field is preferable. IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or (b) TCP connections where no middleboxes are stripping AccECN options.

[BB] When you say 'living without the ACE field', alongside the AccECN TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback in its place?

I don't think there's any case for using classic ECN feedback within AccECN. I see the competition as between ACE and DCTCP feedback, each optionally with the AccECN TCP Option.

If DCTCP feedback were a stop-gap until we could get good traversal of the TCP Option, it might have some merit, but I think DCTCP isn't going to work well with the level of ACK coalescing in the Internet. However, this is very difficult to judge unless we do large scale A-B experiments on the real Internet (and potentially within DCs). I'll think further about enabling some way to do that, possibly within AccECN's negotiation framework. But today, I have to prioritize for the draft deadline.

Cheers



Bob



cheers,
neal





-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------8D5073207E3D337646AE5982-- From nobody Mon Jul 12 07:32:08 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 26B463A1A2B for ; Mon, 12 Jul 2021 07:32:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -0.1 X-Spam-Level: X-Spam-Status: No, score=-0.1 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=cs.helsinki.fi Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aCS_xY44gHQD for ; Mon, 12 Jul 2021 07:31:58 -0700 (PDT) Received: from script.cs.helsinki.fi (script.cs.helsinki.fi [128.214.11.1]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8BBC43A1A24 for ; Mon, 12 Jul 2021 07:31:58 -0700 (PDT) X-DKIM: Courier DKIM Filter v0.50+pk-2017-10-25 mail.cs.helsinki.fi Mon, 12 Jul 2021 17:31:41 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.helsinki.fi; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version:content-type:content-id; s=dkim20130528; bh=zes/5O PC0aYYpBILePSjDjvYtR2pd5A0u927eXr3xOg=; b=chFBmnjFcvwkwydEbNq5xF qZF40cCwX34twsW0oOYcZRhKs6+X8xHldFBhUi0w7fUR6rxoeYnfo8PCGzKNKvMr Xv83nmnir9gufC6obE919x2paX0XgLslrKwEXlAhzQ4v4OUVewBVzRXeN1FIe8HW eAEqlwdOTWVKULI1EKTB8= Received: from whs-18.cs.helsinki.fi (whs-18.cs.helsinki.fi [128.214.166.46]) (TLS: TLSv1/SSLv3,256bits,AES256-GCM-SHA384) by mail.cs.helsinki.fi with ESMTPS; Mon, 12 Jul 2021 17:31:41 +0300 id 00000000005A01CB.0000000060EC524D.000003F5 Date: Mon, 12 Jul 2021 17:31:41 +0300 (EEST) From: "=?ISO-8859-15?Q?Ilpo_J=E4rvinen?=" X-X-Sender: ijjarvin@whs-18.cs.helsinki.fi To: Neal Cardwell cc: Christian Huitema , Mirja Kuehlewind , nishida@sfc.wide.ad.jp, tcpm@ietf.org In-Reply-To: Message-ID: References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> User-Agent: Alpine 2.22 (DEB 394 2020-01-19) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_script-1038-1626100301-0001-2" Content-ID: Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs withgood cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 14:32:06 -0000 This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_script-1038-1626100301-0001-2 Content-Type: text/plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable Content-ID: Hi Neal, I'm trying to understand your scenarios and I fail to see consistency her= e=20 and there (which makes me wonder if I got you right at all). On Sun, 11 Jul 2021, Neal Cardwell wrote: > On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema > wrote: > =A0 >=20 > > On Jul 11, 2021, at 4:42 AM, Bob Briscoe > wrote: >=20 > ...=A0 > > The implementation will have its own ACK ratio - that's out of > scope of AccECN, except to set this max of 6 CEs, which is to > mitigate wrap of the 3-bit counter of CE-marks (which in the > worst case of 100% marking in the data direction could then > induce 1 ACK per 6 packets). This shouldn't limit forward > performance, because it only increases the reverse ACK rate if > there is heavy congestion in the forward path, when the Data > Sender should be reducing the forward rate anyway. >=20 > 1 ACK per 6 packets would cause performance issues on high speed > links. Common setup is "4 to 8 ACK per RTT", which means > intervals much larger than 6 packets. >=20 >=20 > I share this concern about the AccECN requirement of one ACK per 6 CE-m= arked > data segments potentially causing performance issues with high-speed li= nks. > Today, with most high-speed last-hop link technologies (wifi, cellular, > Ethernet) TCP receivers often receive (from lower layers of the hardwar= e and > software networking stack) large aggregates=A0 (often up to 64KBytes or = around > 44 packets), and generate a single ACK for that aggregate. AFAICT chang= ing > this scenario to require 1 packet per 6 CE-marked data segments means t= hat > during CE-marked scenarios receiving such an aggregate would require > ceil(44/6) =3D 8 ACKs, increasing the number of ACKs by up to 8x.=A0 >=20 > I have two main concerns in that scenario: >=20 > (1) CPU load: This seems like it would impose much higher CPU load on t= he > data receiver,=A0at just the moment when the receiver may already be un= der > stress due to receiving data near the maximum rate of the link. Those would be back-to-back ACKs in response to a 44 seg blob? At that=20 point with warm caches & context, essentially just duplicating the same=20 ACK with a few fields varied a little? Is "much higher" perhaps an=20 overstatement? I'd also tend to think link and CPU are not bottleneck at the same time=20 (but I admit I might have dealt with too idealized environments to=20 experience that in practice). This next doesn't relate directly to your concern about CPU load but only to effect of extra ACK on link performance. I remember discussing with=20 Yuchung (perhaps you were there at that time too) some years back in some=20 IETF about making more ACKs appear on the wire using some kind of ACK=20 sending offloading (back then, main motivation for me was to increase SAC= K=20 reduncancy). Yuchung told me that it would be really bad for things like=20 Wifi. I now wonder if it really relates more to the cases where ACKs are=20 roughly evenly spread out over the RTT preventing efficient use of the=20 wireless channel or does that apply also to cases where they're very=20 unevenly spread (back-to-back)? And for the record, sending x ACKs was not the way I interpreted the particular draft text. I took it just a rule when at latest, the ACK should be generated. I just took it as "Since we got 44 CE marks, we MUST send ACK now". (It seems to have be slightly differently worded in -09 where my interpretation originates from). > (2) Congestion and ACK loss: This requirement for generating up to 8 AC= Ks > per aggregate seems like it is likely to produce a tight burst of 8 ACK= s, > which is going to increase congestion and increase the odds of losing a= t > least one ACK, which is going to cause accuracy problems for "Accurate" = ECN. > :-) So instead of 44 CE marks, 44 or 36 are detected (depending on how the=20 sender treats safe/unsafe on that ACK after the loss)? To me more challenging are the cases where ACE delta is zero but a wrap=20 has occurred because zero usually means there's no CE event going on=20 at all. > I understand that this proposed "ACK per 6 CE-marked data segments" rul= e is > necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO th= is is > one of the good arguments against including the ACE field in the AccECN > design.=A0 >=20 > The other main concerns I have with the ACE field are: >=20 > o Complexity I hope you don't really mean ACE field itself is complex? Most of the=20 complexity related to the header bits is due to the negotiation. > o Redundancy with respect to the AccECN counter options I ended up using ACE field to improve byte counter estimation between=20 AccECN option arrivals when AccECN option is not put into every packet. It makes the corrections to sender's estimation of CE (delivered_ce)=20 smoother as the option eventually arrives. It is of course not necessary if AccECN option is always present. > o Potential problems with middleboxes >=20 > o Known problems with NICs and drivers (based on Ilpo's nice talk, "Acc= urate > ECN Linux Implementation: Experiences and Challenges", at the April 202= 0 > TCPM interim meeting) Are you saying that the offloading engine don't have a toggle for the CWR clearing logic? (I didn't claim to know). I've since then thought more about the offloading cases. For sending, the headers remain the same so the feedback doesn't matter. On receiving, I first looked into GRO expecting some impact, however, it=20 turns out that advancing ACK sequence triggers a flush. As no cumulative=20 ACKs get combined, ACE field changes did not trigger additional flushing=20 in practice. What I don't know, is if HW has ACK receiver offloading which combines AC= K=20 with different ACK sequence numbers (in the end, this works same as ACK=20 coalescing). That might be the only case where clearly more ACKs would=20 be passed upwards to processing when ACE field changes for a continuous=20 run of CEs. In randomly spread CEs case, however, DCTCP feedback produces=20 edges that is roughly similar in number so it's hard to see how ACE field=20 would result in unbearable number of ACKs to process compared with DCTCP=20 feedback. > I realize that if AccECN does not have the ACE field feature, then AccE= CN > and TCP L4S will not be usable on paths with middleboxes that strip the > AccECN counter options. But IMHO living without the ACE field is prefer= able. > IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or = (b) > TCP connections where no middleboxes are stripping AccECN options. This is just trading off "potential problems with middleboxes" with=20 other potential problems with middleboxes. I'm not saying here AccECN with option isn't good but it seems that you=20 say you have concern x on ACE and then accept AccECN option as the=20 solution but it has rather similar challenges related to x. --=20 i. --=_script-1038-1626100301-0001-2-- From nobody Mon Jul 12 07:36:06 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6A6AD3A1A2D for ; Mon, 12 Jul 2021 07:36:05 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -16.197 X-Spam-Level: X-Spam-Status: No, score=-16.197 tagged_above=-999 required=5 tests=[DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5brFF1d7nRbU for ; Mon, 12 Jul 2021 07:35:59 -0700 (PDT) Received: from mail-vk1-xa29.google.com (mail-vk1-xa29.google.com [IPv6:2607:f8b0:4864:20::a29]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 65EF13A1A37 for ; Mon, 12 Jul 2021 07:35:57 -0700 (PDT) Received: by mail-vk1-xa29.google.com with SMTP id e15so4036604vkp.13 for ; Mon, 12 Jul 2021 07:35:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=onkJBFQEGdJAgSK4HQ+cLRC7TwetOPjI+1zOUchIG0I=; b=DFIt6J+GE2gTPZA5cJs1BbY76AWUhx+WEhiB2ZAr6ebnOusC6Q5Txhu0beWDLDyVwj 0abJykVxLYwHrlsn+XT0QZ8gHvst5VT3tlhOKvMVLnwVdW+o29k9ZBYz7Yt5kpxU8KAp /HB+qShz9sdwpIXVJv2Yc4uovVd8N5P4Td5BGxh6ZvbE9e7oRD1HTZbqkzrlnhJ6hHFf bHPkzkRPEg+DHxU0NA4q56xNpN257YSi/fy0SE4/jstED5zwvBg2/go3XhpQRrr9La3S YmaKlCre/qFBXpGsvXhLBp5fkGngKhUbl2uxPoVKqEQW2WPbZjFANK7BQJEucjybAOlA n+Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=onkJBFQEGdJAgSK4HQ+cLRC7TwetOPjI+1zOUchIG0I=; b=tLx4UQe5rvTok7D46bdOVERcVuLfZLcFtgTZZqzy0vL/uBWdIJUHEVG2dIhhP7ietS KwBspi8qvXRJc8S9+qKdAHUWkx2B7AIPTdMtOqb8K9I1nk/ALRed9po78/953FUPw+JF xtZJW13XxHwyU3MjjKCFswOJVFt8W04HQGxFOoyqRXe95M4qj04JH1TVgcRbHvdVEn9I KuhIe8QuEqFV0fISGKqgC/v9hOmF5jtYnrtSAPzZYodaw28flJ0dl6G7IXswJAD6njI9 rH6RDrLuM5sWN9uFC3R/DDm9fggugLwHokaJdhN+intDvZwUWSxmdYrF5bg4Vm6jwnzq Arfg== X-Gm-Message-State: AOAM530U7t21bY8lM1ZxX509ZWomII/5oRGXGlcLOZQOl76OWLrKQ6sb Lx6jqVF7AGoe7F142asS0r/M93hU9PNbMQVGZlWl2X4Z1VyLcg== X-Google-Smtp-Source: ABdhPJw4Nj3NX2W09L/FBm1AwuDjz7Zf0N5jJgoq4EtYr7irNRqaBfwcJm65Tx3cbdmAsQdhfpxOaYbrYaZipAPua40= X-Received: by 2002:a1f:1d4d:: with SMTP id d74mr5706930vkd.19.1626100555431; Mon, 12 Jul 2021 07:35:55 -0700 (PDT) MIME-Version: 1.0 References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> <6ad1817b-2b58-8aee-2cf5-3654e33b3eda@bobbriscoe.net> In-Reply-To: <6ad1817b-2b58-8aee-2cf5-3654e33b3eda@bobbriscoe.net> From: Neal Cardwell Date: Mon, 12 Jul 2021 10:35:38 -0400 Message-ID: To: Bob Briscoe Cc: tcpm@ietf.org, Mirja Kuehlewind , Yuchung Cheng , Richard Scheffenegger , Christian Huitema , Ilpo Jarvinen , =?UTF-8?Q?Ilpo_J=C3=A4rvinen?= Content-Type: multipart/alternative; boundary="000000000000fa5c1e05c6ee09cf" Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 14:36:06 -0000 --000000000000fa5c1e05c6ee09cf Content-Type: text/plain; charset="UTF-8" On Mon, Jul 12, 2021 at 8:30 AM Bob Briscoe wrote: > Neal, > > On 11/07/2021 19:28, Neal Cardwell wrote: > > > > On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema > wrote: > >> >> >> > On Jul 11, 2021, at 4:42 AM, Bob Briscoe wrote: >> > ... > >> > The implementation will have its own ACK ratio - that's out of scope of >> AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the >> 3-bit counter of CE-marks (which in the worst case of 100% marking in the >> data direction could then induce 1 ACK per 6 packets). This shouldn't limit >> forward performance, because it only increases the reverse ACK rate if >> there is heavy congestion in the forward path, when the Data Sender should >> be reducing the forward rate anyway. >> >> 1 ACK per 6 packets would cause performance issues on high speed links. >> Common setup is "4 to 8 ACK per RTT", which means intervals much larger >> than 6 packets. > > > I share this concern about the AccECN requirement of one ACK per 6 > CE-marked data segments potentially causing performance issues with > high-speed links. Today, with most high-speed last-hop link technologies > (wifi, cellular, Ethernet) TCP receivers often receive (from lower layers > of the hardware and software networking stack) large aggregates (often up > to 64KBytes or around 44 packets), and generate a single ACK for that > aggregate. AFAICT changing this scenario to require 1 packet per 6 > CE-marked data segments means that during CE-marked scenarios receiving > such an aggregate would require ceil(44/6) = 8 ACKs, increasing the number > of ACKs by up to 8x. > > I have two main concerns in that scenario: > > (1) CPU load: This seems like it would impose much higher CPU load on the > data receiver, at just the moment when the receiver may already be under > stress due to receiving data near the maximum rate of the link. > > (2) Congestion and ACK loss: This requirement for generating up to 8 ACKs > per aggregate seems like it is likely to produce a tight burst of 8 ACKs, > which is going to increase congestion and increase the odds of losing at > least one ACK, which is going to cause accuracy problems for "Accurate" > ECN. :-) > > > [BB] I hadn't intended this wording to apply in response to a large burst. > I'd want to alter the wording to allow just one ACK in this case. > In the case where there is just one ACK for a big CE-marked burst of >= 8 segments, does that mean the result is: (a) the ACE field loses accuracy, or (b) the TCP connection interpreting the ACE field should presume the ACE field probably wrapped and estimate the number of CE-marked segments using the number of SACKed/ACKed segments, rather than the ACE field? > I understand that this proposed "ACK per 6 CE-marked data segments" rule > is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO > this is one of the good arguments against including the ACE field in the > AccECN design. > > > [BB] I think the subtext here is a preference for the DCTCP style of > feedback vs ACE (when there is no AccECN TCP Option). I don't think this > particular issue is any different between the two. > Yes, when the AccECN TCP Option is not present, I'd prefer DCTCP-style feedback rather than the ACE field. > With DCTCP feedback, when a large data burst like this arrives, if there > are transitions to and from CE marking within the burst, does Linux DCTCP > generate an ACK for each transition, and for every n repetitions of CE not > ECT, like the RFC says it should? Or does the implementation just pop out > one ACK at the end? > Yes, with Linux DCTCP (or BBRv2), when there is a big burst, there is an ACK for each CE<->non-CE transition. They also handle the case where there is a CE<->non-CE transition while a delayed ACK is pending. This requires cooperation from a few different layers. (1) the NIC hardware LRO aggregation is supposed to not aggregate across CE<->non-CE transitions. I believe the de facto spec on NIC aggregation with respect to IP fields is the following, which mandates that: https://docs.microsoft.com/en-us/windows-hardware/drivers/network/updating-the-ip-headers-for-coalesced-segments (2) the Linux software GRO aggregation does not aggregate across CE<->non-CE transitions; see inet_gro_receive() checking the ToS bytes: (iph->tos ^ iph2->tos) (3) the DCTCP and BBRv2 code use dctcp_ece_ack_update() to handle the case where there is a CE<->non-CE transition while a delayed ACK is pending > > Ilpo & I have been doing experiments with high levels of ACK coalescing > causing the ACE field seen by the Data Sender to wrap multiple times under > high congestion. For example, 1 ACK per ~4ms is one common scheme, which > would result in about 1 ACK for 34 data packets at just 100Mb/s. > Great. Yes, certainly that level of aggregation is important to test. > > > The other main concerns I have with the ACE field are: > > o Complexity > > > [BB] I understand there could be a tradeoff between complexity and > accuracy. But I think the complexity of ACE and DCTCP feedback are pretty > much the same but, with even low levels of ACK coalescing, the accuracy of > ACE is superior. Nonetheless, I understand that deploying something > different to DCTCP when you've already got DCTCP could involve deployment > complexity. > IMHO the ACE field is more complex, due to: o context-dependent interpretation of the header bits used by ACE o logic to handle wraps in the ACE field and estimate what the real increment to the ACE field was in the case of possible ACK loss o interactions with drivers or hardware that already have assumptions about the header bits used for ACE (e.g. that the CWR bit should be cleared for the first N-1 segments in a TSO burst and only set on the last segment of the TSO burst) > > > > o Redundancy with respect to the AccECN counter options > > > [BB] The redundancy is intended. For cases where the TCP Option doesn't > traverse the path, or where TCP option space is limited. > I realize the redundancy is intentional, but the redundant code does impose a maintenance cost. > > o Potential problems with middleboxes > > > [BB] Well, we've tested the 3 header bits over millions of paths without > problems. But yes, there are billions more to test. > > > o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate > ECN Linux Implementation: Experiences and Challenges", at the April 2020 > TCPM interim meeting) > > > [BB] I understood that talk as concluding there weren't really problems. > What specific problem do you have in mind? > Mainly I'm concerned about: o interactions with drivers or hardware that already have assumptions about the header bits used for ACE (e.g. that the CWR bit should be cleared for the first N-1 segments in a TSO burst and only set on the last segment of the TSO burst) It sounds like since the ACE field redefines the semantics of some long-standing TCP header fields, there will be a long trial-and-error period of finding the drivers and NICs that are incompatible with the ACE field, and working around those. > I realize that if AccECN does not have the ACE field feature, then AccECN > and TCP L4S will not be usable on paths with middleboxes that strip the AccECN > counter options. But IMHO living without the ACE field is preferable. IMHO > it's acceptable to say that L4S can only be used with (a) QUIC, or (b) TCP > connections where no middleboxes are stripping AccECN options. > > > [BB] When you say 'living without the ACE field', alongside the AccECN TCP > Option, would you leave classic ECN feedback? Or put DCTCP feedback in its > place? > When 'living without the ACE field', I mean either using the AccECN TCP Option or DCTCP-style feedback. > I don't think there's any case for using classic ECN feedback within > AccECN. I see the competition as between ACE and DCTCP feedback, each > optionally with the AccECN TCP Option. > Agreed. I agree there's no case for using classic ECN feedback within AccECN. > If DCTCP feedback were a stop-gap until we could get good traversal of the > TCP Option, it might have some merit, but I think DCTCP isn't going to work > well with the level of ACK coalescing in the Internet. However, this is > very difficult to judge unless we do large scale A-B experiments on the > real Internet (and potentially within DCs). I'll think further about > enabling some way to do that, possibly within AccECN's negotiation > framework. But today, I have to prioritize for the draft deadline. > I agree it's unclear whether DCTCP-style feedback will work well over the public Internet, due to ACK loss and the variety of aggregation mechanisms in play. That is why in my previous email in this thread I was mainly presuming that if the AccECN TCP Option is not forwarded by the path then the flow might have to just disable AccECN and L4S. cheers, neal --000000000000fa5c1e05c6ee09cf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Jul 12, 2021 at 8:30 AM Bob B= riscoe <ietf@bobbriscoe.net&g= t; wrote:
=20 =20 =20
Neal,

On 11/07/2021 19:28, Neal Cardwell wrote:
=20


On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema <huitema@huitema.net> wrote:
=C2=A0

> On Jul 11, 2021, at 4:42 AM, Bob Briscoe <ietf@bobbriscoe.net> wro= te:
...=C2=A0
> The implementation will have its own ACK ratio - that'= s out of scope of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bit counter of CE-marks (which in the worst case of 100% marking in the data direction could then induce 1 ACK per 6 packets). This shouldn't limit forward performance, because it only increases the reverse ACK rate if there is heavy congestion in the forward path, when the Data Sender should be reducing the forward rate anyway.

1 ACK per 6 packets would cause performance issues on high speed links. Common setup is "4 to 8 ACK per RTT", wh= ich means intervals much larger than 6 packets.

I share this concern about the AccECN requirement of one ACK per 6 CE-marked data segments potentially causing performance issues with high-speed links. Today, with most high-speed last-hop link technologies (wifi, cellular, Ethernet) TCP receivers often receive (from lower layers of the hardware and software networking stack) large aggregates=C2=A0 (often up to 64KBytes or around 44 packets), a= nd generate a single ACK for that aggregate. AFAICT changing this scenario to require 1 packet per 6 CE-marked data segments means that during CE-marked scenarios receiving such an aggregate would require ceil(44/6) =3D 8 ACKs, increasing the number of ACKs by up to 8x.=C2=A0

I have two main concerns in that scenario:

(1) CPU load: This seems like it would impose much higher CPU load on the data receiver,=C2=A0at just the moment when the receiver may already be under stress due to receiving data near the maximum rate of the link.

(2) Congestion and ACK loss: This requirement for generating up to 8 ACKs per aggregate seems like it is likely to produce a tight burst of 8 ACKs, which is going to increase congestion and increase the odds of losing at least one ACK, which is going to cause accuracy problems for "Accurate" ECN. :-)

[BB] I hadn't intended this wording to apply in response to a large burst. I'd want to alter the wording to allow just one ACK in this case.

In the case where there= is just one ACK for a big CE-marked burst of >=3D 8 segments, does that= mean the result is:

(a) the ACE field loses accur= acy, or

(b) the TCP connection interpreting the AC= E field should presume the ACE field probably wrapped and estimate the numb= er of CE-marked segments using the number of SACKed/ACKed segments, rather = than the ACE field?

=C2=A0
=20
I understand that this proposed "ACK per 6 CE-marked da= ta segments" rule is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is one of the good arguments against including the ACE field in the AccECN design.

[BB] I think the subtext here is a preference for the DCTCP style of feedback vs ACE (when there is no AccECN TCP Option). I don't think this particular issue is any different between the two.

Yes, when the AccECN TCP Option is not present, = I'd prefer DCTCP-style feedback rather than the ACE field.
= =C2=A0
With DCTCP feedback, when a large data burst like this arrives, if there are transitions to and from CE marking within the burst, does Linux DCTCP generate an ACK for each transition, and for every n repetitions of CE not ECT, like the RFC says it should? Or does the implementation just pop out one ACK at the end?
<= div>
Yes, with Linux DCTCP (or BBRv2), when there is a big bu= rst, there is an ACK for each CE<->non-CE transition. They also handl= e the case where there is a CE<->non-CE transition while a delayed AC= K is pending. This requires cooperation from a few different layers.
<= div>
(1) the NIC hardware LRO aggregation is supposed to not = aggregate across CE<->non-CE transitions. I believe the de facto spec= on NIC aggregation with respect to IP fields is the following, which manda= tes that:

(= 2) the Linux software GRO aggregation does not aggregate across CE<->= non-CE transitions; see=C2=A0inet_gro_receive() checking the=C2=A0 ToS byte= s: (iph->tos ^ iph2->tos)=C2=A0

(3) the DCTC= P and BBRv2 code use=C2=A0dctcp_ece_ack_update() to handle the case where t= here is a CE<->non-CE transition while a delayed ACK is pending
=
=C2=A0

Ilpo & I have been doing experiments with high levels of ACK coalescing causing the ACE field seen by the Data Sender to wrap multiple times under high congestion. For example, 1 ACK per ~4ms is one common scheme, which would result in about 1 ACK for 34 data packets at just 100Mb/s.

Grea= t. Yes, certainly that level of aggregation is important to test.
=C2=A0


The other main concerns I have with the ACE field are:

o Complexity

[BB] I understand there could be a tradeoff between complexity and accuracy. But I think the complexity of ACE and DCTCP feedback are pretty much the same but, with even low levels of ACK coalescing, the accuracy of ACE is superior. Nonetheless, I understand that deploying something different to DCTCP when you've already got DCTC= P could involve deployment complexity.

<= div>IMHO the ACE field is more complex, due to:

o = context-dependent interpretation of the header bits used by ACE
<= br>
o logic to handle wraps in the ACE field and estimate what th= e real increment to the ACE field was in the case of possible ACK loss

o interactions with drivers or hardware that already h= ave assumptions about the header bits used for ACE (e.g. that the CWR bit s= hould be cleared for the first N-1 segments in a TSO burst and only set on = the last segment of the TSO burst)

=C2=A0



o Redundancy with respect to the AccECN counter options

[BB] The redundancy is intended. For cases where the TCP Option doesn't traverse the path, or where TCP option space is limited.

I realize the redundancy is intenti= onal, but the redundant code does=C2=A0impose a maintenance cost.
=C2=A0

o Potential problems with middleboxes

[BB] Well, we've tested the 3 header bits over millions of paths without problems. But yes, there are billions more to test.


o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate ECN Linux Implementation: Experiences and Challenges"= , at the April 2020 TCPM interim meeting)

[BB] I understood that talk as concluding there weren't really problems. What specific problem do you have in mind?

Mainly I'm concerned about:

o interactions with drivers or hardware that already have assumptions= about the header bits used for ACE (e.g. that the CWR bit should be cleare= d for the first N-1 segments in a TSO burst and only set on the last segmen= t of the TSO burst)

It sounds like since the ACE f= ield redefines the semantics of some long-standing TCP header fields, there= will be a long trial-and-error period of finding the drivers and NICs that= are incompatible with the ACE field, and working around those.
= =C2=A0
I real= ize that if AccECN does not have the ACE field feature, then AccECN and TCP= L4S will not be usable on paths with middleboxes that strip the Acc= ECN counter options. But IMHO living without the ACE field is preferable. IMHO it's acceptable to say that L4S can only b= e used with (a) QUIC, or (b) TCP connections where no middleboxes are stripping AccECN options.

[BB] When you say 'living without the ACE field', alongside the AccECN TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback in its place?

= When 'living without the ACE field', I mean either using the AccECN= TCP Option or DCTCP-style feedback.
=C2=A0
I don't think there's any case for using classic ECN feedback w= ithin AccECN. I see the competition as between ACE and DCTCP feedback, each optionally with the AccECN TCP Option.
=
Agreed. I agree there's no case for using classic ECN fe= edback within AccECN.
=C2=A0
If DCTCP feedback were a stop-gap until we could get good traversal of the TCP Option, it might have some merit, but I think DCTCP isn'= t going to work well with the level of ACK coalescing in the Internet. However, this is very difficult to judge unless we do large scale A-B experiments on the real Internet (and potentially within DCs). I'll think further about enabling some way to do that, possibly within AccECN's negotiation framework. But today, I have to prioritize for the draft deadline.

I agree it's unclear whether DCTCP-style feedback will work well = over the public Internet, due to ACK loss and the variety of aggregation me= chanisms in play. That is why in my previous email in this thread I was mai= nly presuming that if the AccECN TCP Option is not forwarded by the path th= en the flow might have to just disable AccECN and L4S.

=
cheers,
neal

--000000000000fa5c1e05c6ee09cf-- From nobody Mon Jul 12 08:46:05 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 6FCB93A1FF5; Mon, 12 Jul 2021 08:46:04 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162610476442.30543.4667406094304409800@ietfa.amsl.com> Date: Mon, 12 Jul 2021 08:46:04 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-hystartplusplus-02.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 15:46:05 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : HyStart++: Modified Slow Start for TCP Authors : Praveen Balasubramanian Yi Huang Matt Olson Filename : draft-ietf-tcpm-hystartplusplus-02.txt Pages : 8 Date : 2021-07-12 Abstract: This doument describes HyStart++, a simple modification to the slow start phase of TCP congestion control algorithms. Traditional slow start can cause overshotting of the ideal send rate and cause large packet loss within a round-trip time which results in poor performance. HyStart++ is composed of the delay increase variant of HyStart to prevent overshooting of the ideal sending rate, while also mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Jul 12 11:48:12 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 4A94E3A0E04; Mon, 12 Jul 2021 11:48:10 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162611569026.7615.3785325543750944369@ietfa.amsl.com> Date: Mon, 12 Jul 2021 11:48:10 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-rfc793bis-24.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 18:48:10 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : Transmission Control Protocol (TCP) Specification Author : Wesley M. Eddy Filename : draft-ietf-tcpm-rfc793bis-24.txt Pages : 108 Date : 2021-07-12 Abstract: This document specifies the Transmission Control Protocol (TCP). TCP is an important transport layer protocol in the Internet protocol stack, and has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFC 1122, and should be considered as a replacement for the portions of that document dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. RFC EDITOR NOTE: If approved for publication as an RFC, this should be marked additionally as "STD: 7" and replace RFC 793 in that role. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rfc793bis-24 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc793bis-24 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Jul 12 11:57:46 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4929D3A0C46 for ; Mon, 12 Jul 2021 11:57:45 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=mti-systems-com.20150623.gappssmtp.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ouush44AU7Lb for ; Mon, 12 Jul 2021 11:57:40 -0700 (PDT) Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 170CD3A0F35 for ; Mon, 12 Jul 2021 11:57:39 -0700 (PDT) Received: by mail-qk1-x732.google.com with SMTP id j184so19035204qkd.6 for ; Mon, 12 Jul 2021 11:57:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mti-systems-com.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=s6aRhi+5KrevTLtmGM2pzMM6Z9c7NkDf7yiohriDz58=; b=1ZFdDbnTgUkHSEeznHWuOc8zjWzYzIEoamSP+9ZEf1l7oBYTyTXTlGrpwQTUWCa4yT RKbDw1pNAnFoYsIeiZAXq2x/EYbLPJXHceNc9LsQTI3SYAwssulZtsw1ZKxJv74eLoYs KaxyvpgoxqBVoWub7R2FJXiZ2Q1N3Um1PiM8mYYQcm5Q5lhU3sQXnBfKhsahdEfzXGpk 0kVpfX9RUxlfg8Y5yuPa9wrf7yhKTsGIP/me+ogXcgYcHYFDWLh1iICqhp/9NcOQvsCC dIQJPuuzckh1yddHVhb88N+0D3czgzHTrnxWM3LJHQJpM2jjkkBvHxG+IUZJ+nsWZCTj sdTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=s6aRhi+5KrevTLtmGM2pzMM6Z9c7NkDf7yiohriDz58=; b=AqHEyaxL0adsyywq73WE0Kn3QbRPnsBA7zmkt5Zr40EfhZdgpNpRZmJ+wio0svWfw1 rmQQYwe6rOIXSZbGwyF80K6IfJC1tjsMr1ErsJUiN/1qR2uWw8xsJUIPu4QfddhoSAe9 UEQzv7wTdiqRYyD7+MvC/Ou79qdg+9uf5vZ92ruROTCeb2KDgAODr0i4U6IomKqVQlJ9 uVQQFrXWqLk85pwsD4FgoQEOPQGaIbHeCNpDx3Zoc6lJsxUEOOFj/oFqx6/G3FKQ1WPC 8r1ydEyEE3uVW4N7ErYpWmbiWjjaNhVnky1scQNDYFFuRp0P1ANBKTpdDIcaRo7p7xWm GwoA== X-Gm-Message-State: AOAM532OqLJAErA5Wz0s/LEeLnnSULgZhJCcCztkKE1brxxPy7B6IO1x jWafkJSKN1LMGAUWx2MQyiQeYlSbCtEnZOCP X-Google-Smtp-Source: ABdhPJzb/vS6MnqYmh5Kc5julAurf8mMan4hCTQpjqYBTqkXUtp3LiWJDsLMrvo80DNQSgPOoo24Eg== X-Received: by 2002:a05:620a:22d6:: with SMTP id o22mr168707qki.444.1626116257908; Mon, 12 Jul 2021 11:57:37 -0700 (PDT) Received: from [192.168.1.114] (069-135-001-122.biz.spectrum.com. [69.135.1.122]) by smtp.gmail.com with ESMTPSA id j65sm7106572qkd.17.2021.07.12.11.57.37 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Jul 2021 11:57:37 -0700 (PDT) To: tcpm@ietf.org References: <162611569026.7615.3785325543750944369@ietfa.amsl.com> From: Wesley Eddy Message-ID: <9f310fe4-1e50-4a94-5ac2-c3eeac4feba6@mti-systems.com> Date: Mon, 12 Jul 2021 14:57:35 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <162611569026.7615.3785325543750944369@ietfa.amsl.com> Content-Type: multipart/alternative; boundary="------------1086F537CECF4DF43C7F397B" Content-Language: en-US Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-rfc793bis-24.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 18:57:45 -0000 This is a multi-part message in MIME format. --------------1086F537CECF4DF43C7F397B Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit This revision has changes resulting from Martin's AD review and subsequent comments from Yuchung and Joe. A few things to specifically check over since I wasn't sure if the thread totally converged on them, and they should be confirmed: (1) urgent pointer "receding" I think the key to understand is that the only time an application can set this is when data is written, so it should not "recede" since you can't write backwards.   There was already a paragraph saying: To send an urgent indication the user must also send at least one data octet. I'm not sure if this completely satisfies Martin's comment, but for now, I added this sentence: Note that because changes in the urgent pointer correspond to data being written by a sending application, the urgent pointer can not "recede" in the sequence space, but a TCP receiver should be robust to invalid urgent pointer values. (2) PUSH check in SWS timer expiration I think Martin is right, and if the intention is that if the SWS timer expires, it's okay to send, even if PUSH isn't set.  This: or if data is PUSHed and the override timeout occurs. was changed to: or if the override timeout occurs. (3) unnecessary Step 4 for LISTEN state when SEGMENT ARRIVES Changed: fourth other text or control Any other control or text-bearing segment (not containing SYN) must have an ACK and thus would be discarded by the ACK processing. An incoming RST segment could not be valid, since it could not have been sent in response to anything sent by this incarnation of the connection. So, if this unlikely condition is reached, the correct behavior is to drop the segment and return. to: fourth other data or control This should not be reached. Drop the segment and return. Any other control or data-bearing segment (not containing SYN) must have an ACK and thus would have been discarded by the ACK processing in the second step, unless it was first discarded by RST checking in the first step. Although maybe if we agree that it's totally pointless to have, then this "fourth" step can just be removed entirely. On 7/12/2021 2:48 PM, internet-drafts@ietf.org wrote: > A New Internet-Draft is available from the on-line Internet-Drafts directories. > This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. > > Title : Transmission Control Protocol (TCP) Specification > Author : Wesley M. Eddy > Filename : draft-ietf-tcpm-rfc793bis-24.txt > Pages : 108 > Date : 2021-07-12 > > Abstract: > This document specifies the Transmission Control Protocol (TCP). TCP > is an important transport layer protocol in the Internet protocol > stack, and has continuously evolved over decades of use and growth of > the Internet. Over this time, a number of changes have been made to > TCP as it was specified in RFC 793, though these have only been > documented in a piecemeal fashion. This document collects and brings > those changes together with the protocol specification from RFC 793. > This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, > 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFC > 1122, and should be considered as a replacement for the portions of > that document dealing with TCP requirements. It also updates RFC > 5961 by adding a small clarification in reset handling while in the > SYN-RECEIVED state. The TCP header control bits from RFC 793 have > also been updated based on RFC 3168. > > RFC EDITOR NOTE: If approved for publication as an RFC, this should > be marked additionally as "STD: 7" and replace RFC 793 in that role. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/ > > There is also an htmlized version available at: > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rfc793bis-24 > > A diff from the previous version is available at: > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc793bis-24 > > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm --------------1086F537CECF4DF43C7F397B Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

This revision has changes resulting from Martin's AD review and subsequent comments from Yuchung and Joe.

A few things to specifically check over since I wasn't sure if the thread totally converged on them, and they should be confirmed:

(1) urgent pointer "receding"

I think the key to understand is that the only time an application can set this is when data is written, so it should not "recede" since you can't write backwards.   There was already a paragraph saying: 

   To send an urgent indication the user must also send at least one
   data octet.

I'm not sure if this completely satisfies Martin's comment, but for now, I added this sentence:

  Note that because changes in the urgent pointer correspond
   to data being written by a sending application, the urgent pointer
   can not "recede" in the sequence space, but a TCP receiver should be
   robust to invalid urgent pointer values.

(2) PUSH check in SWS timer expiration

I think Martin is right, and if the intention is that if the SWS timer expires, it's okay to send, even if PUSH isn't set.  This:

   or if data is PUSHed and the override timeout occurs.

was changed to:

   or if the override timeout occurs.

(3) unnecessary Step 4 for LISTEN state when SEGMENT ARRIVES

Changed:

         fourth other text or control

            Any other control or text-bearing segment (not containing
            SYN) must have an ACK and thus would be discarded by the ACK
            processing.  An incoming RST segment could not be valid,
            since it could not have been sent in response to anything
            sent by this incarnation of the connection.  So, if this
            unlikely condition is reached, the correct behavior is to
            drop the segment and return.

to:

      fourth other data or control

         This should not be reached.  Drop the segment and return.  Any
         other control or data-bearing segment (not containing SYN) must
         have an ACK and thus would have been discarded by the ACK
         processing in the second step, unless it was first discarded by
         RST checking in the first step.

Although maybe if we agree that it's totally pointless to have, then this "fourth" step can just be removed entirely.



On 7/12/2021 2:48 PM, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : Transmission Control Protocol (TCP) Specification
        Author          : Wesley M. Eddy
	Filename        : draft-ietf-tcpm-rfc793bis-24.txt
	Pages           : 108
	Date            : 2021-07-12

Abstract:
   This document specifies the Transmission Control Protocol (TCP).  TCP
   is an important transport layer protocol in the Internet protocol
   stack, and has continuously evolved over decades of use and growth of
   the Internet.  Over this time, a number of changes have been made to
   TCP as it was specified in RFC 793, though these have only been
   documented in a piecemeal fashion.  This document collects and brings
   those changes together with the protocol specification from RFC 793.
   This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093,
   6429, 6528, and 6691 that updated parts of RFC 793.  It updates RFC
   1122, and should be considered as a replacement for the portions of
   that document dealing with TCP requirements.  It also updates RFC
   5961 by adding a small clarification in reset handling while in the
   SYN-RECEIVED state.  The TCP header control bits from RFC 793 have
   also been updated based on RFC 3168.

   RFC EDITOR NOTE: If approved for publication as an RFC, this should
   be marked additionally as "STD: 7" and replace RFC 793 in that role.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rfc793bis-24

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc793bis-24


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--------------1086F537CECF4DF43C7F397B-- From nobody Mon Jul 12 12:05:29 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 558F83A1007 for ; Mon, 12 Jul 2021 12:05:27 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.002 X-Spam-Level: X-Spam-Status: No, score=0.002 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=mti-systems-com.20150623.gappssmtp.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oNmMq9Hlh6q4 for ; Mon, 12 Jul 2021 12:05:24 -0700 (PDT) Received: from mail-qv1-xf2b.google.com (mail-qv1-xf2b.google.com [IPv6:2607:f8b0:4864:20::f2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0EAA23A10BA for ; Mon, 12 Jul 2021 12:03:06 -0700 (PDT) Received: by mail-qv1-xf2b.google.com with SMTP id h9so5142937qvs.0 for ; Mon, 12 Jul 2021 12:03:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mti-systems-com.20150623.gappssmtp.com; s=20150623; h=subject:from:to:references:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=naTurUeMONOPz9Znd6zYCJSuesj3PdHb1q8IYmszYjM=; b=tUpuMGbKeRR6jplnV1t6LDKSsz91/7rDUNLru4KnVPNjxSlnCLNVjwMpYPV09IKDyP RuhZJX5+3JAvHgQC7nmecoRjQy4oPTRrZtfA1F33q284oqWgucKO1EPJuA/KaIuotlR6 pdik4S/JjJh/oEi0oOfuXbM5zcm2Br+W2KmM/t47X0RdrWLyziaMhFaSYuh8VA5zcDqk aWQIsRVDcn7vE2OoQRnTmYw2JjK9k+hDwubojkfEllXwMWQKsm/lAfUYHWD1FZFbD5By c+NXarQB5p3JU0WxkYybkDH1Dh3fgXybfIEfOLAzKc2GBFvFr2TdIpR6sGu30OFcVYxl Es3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=naTurUeMONOPz9Znd6zYCJSuesj3PdHb1q8IYmszYjM=; b=SfrzUbfZMoSoAeel+KmFYjqgonfJmhI2fggplseX8BARrSLUPNZmDzWIuHD8pF+8tp TVHAzdhaJ+fSKEnmVCUvjrvhISxzMSCK4wXunARO2Xer+LEb5jS4LzvkNI50hhRmJJ6N sbhumwPU/DQrrfY/XZY8QJZ9ChaaCDSoAdLsABLZq0Fp1LqfIDOfVb52qSrbMdde8+KP TYYCl6xCyyHvWPoOeHdJGY0VkW849+fX9BETGVQ9w9oBMlF4DWrLOTKqSEqujYPXYQMu hPHiDIwYg+maYic/TGTmaWdu4PcZis3oSMlIE20bMcvDjNq/eI40Yb/7EDPI+DvqvAON jK7g== X-Gm-Message-State: AOAM533Cg7IUL+Ir9PGAO5+7ubs+TWaZfXK6mVkBXPg6gv17al5xBqOF TGa+ziTDLsNxQ+asMF7plNzzONqeavoCSAg2 X-Google-Smtp-Source: ABdhPJyhN41wNhKN3NGWdvecK1TJ5BTWXuJtC1OgIt8Uog1yFzR/YbGZr8v757d6WqitQ98ccDkabg== X-Received: by 2002:ad4:58a4:: with SMTP id ea4mr744156qvb.39.1626116584204; Mon, 12 Jul 2021 12:03:04 -0700 (PDT) Received: from [192.168.1.114] (069-135-001-122.biz.spectrum.com. [69.135.1.122]) by smtp.gmail.com with ESMTPSA id v22sm4623915qkf.50.2021.07.12.12.03.03 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 12 Jul 2021 12:03:03 -0700 (PDT) From: Wesley Eddy To: tcpm@ietf.org References: <162611569026.7615.3785325543750944369@ietfa.amsl.com> <9f310fe4-1e50-4a94-5ac2-c3eeac4feba6@mti-systems.com> Message-ID: Date: Mon, 12 Jul 2021 15:03:02 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <9f310fe4-1e50-4a94-5ac2-c3eeac4feba6@mti-systems.com> Content-Type: multipart/alternative; boundary="------------ADEA73B7D6C321FD9C1FF687" Content-Language: en-US Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-rfc793bis-24.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 19:05:27 -0000 This is a multi-part message in MIME format. --------------ADEA73B7D6C321FD9C1FF687 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Sorry, one more thing to ask the working group ... In a couple of places, Martin suggested changing "text" to "data".  I agree, this probably makes more sense to a modern reader and is merely editorial. Should we change that not just in a couple of places Martin noted, but across the board in the document? On 7/12/2021 2:57 PM, Wesley Eddy wrote: > > This revision has changes resulting from Martin's AD review and > subsequent comments from Yuchung and Joe. > > A few things to specifically check over since I wasn't sure if the > thread totally converged on them, and they should be confirmed: > > (1) urgent pointer "receding" > > I think the key to understand is that the only time an application can > set this is when data is written, so it should not "recede" since you > can't write backwards.   There was already a paragraph saying: > > To send an urgent indication the user must also send at least one > data octet. > > I'm not sure if this completely satisfies Martin's comment, but for > now, I added this sentence: > > Note that because changes in the urgent pointer correspond > to data being written by a sending application, the urgent pointer > can not "recede" in the sequence space, but a TCP receiver should be > robust to invalid urgent pointer values. > > (2) PUSH check in SWS timer expiration > > I think Martin is right, and if the intention is that if the SWS timer > expires, it's okay to send, even if PUSH isn't set. This: > > or if data is PUSHed and the override timeout occurs. > > was changed to: > > or if the override timeout occurs. > > (3) unnecessary Step 4 for LISTEN state when SEGMENT ARRIVES > > Changed: > > fourth other text or control > > Any other control or text-bearing segment (not containing > SYN) must have an ACK and thus would be discarded by the ACK > processing. An incoming RST segment could not be valid, > since it could not have been sent in response to anything > sent by this incarnation of the connection. So, if this > unlikely condition is reached, the correct behavior is to > drop the segment and return. > > to: > > fourth other data or control > > This should not be reached. Drop the segment and return. Any > other control or data-bearing segment (not containing SYN) must > have an ACK and thus would have been discarded by the ACK > processing in the second step, unless it was first discarded by > RST checking in the first step. > > Although maybe if we agree that it's totally pointless to have, then > this "fourth" step can just be removed entirely. > > > > On 7/12/2021 2:48 PM, internet-drafts@ietf.org wrote: >> A New Internet-Draft is available from the on-line Internet-Drafts directories. >> This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. >> >> Title : Transmission Control Protocol (TCP) Specification >> Author : Wesley M. Eddy >> Filename : draft-ietf-tcpm-rfc793bis-24.txt >> Pages : 108 >> Date : 2021-07-12 >> >> Abstract: >> This document specifies the Transmission Control Protocol (TCP). TCP >> is an important transport layer protocol in the Internet protocol >> stack, and has continuously evolved over decades of use and growth of >> the Internet. Over this time, a number of changes have been made to >> TCP as it was specified in RFC 793, though these have only been >> documented in a piecemeal fashion. This document collects and brings >> those changes together with the protocol specification from RFC 793. >> This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, >> 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFC >> 1122, and should be considered as a replacement for the portions of >> that document dealing with TCP requirements. It also updates RFC >> 5961 by adding a small clarification in reset handling while in the >> SYN-RECEIVED state. The TCP header control bits from RFC 793 have >> also been updated based on RFC 3168. >> >> RFC EDITOR NOTE: If approved for publication as an RFC, this should >> be marked additionally as "STD: 7" and replace RFC 793 in that role. >> >> >> The IETF datatracker status page for this draft is: >> https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/ >> >> There is also an htmlized version available at: >> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rfc793bis-24 >> >> A diff from the previous version is available at: >> https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc793bis-24 >> >> >> Internet-Drafts are also available by anonymous FTP at: >> ftp://ftp.ietf.org/internet-drafts/ >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm --------------ADEA73B7D6C321FD9C1FF687 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Sorry, one more thing to ask the working group ...

In a couple of places, Martin suggested changing "text" to "data".  I agree, this probably makes more sense to a modern reader and is merely editorial.

Should we change that not just in a couple of places Martin noted, but across the board in the document?


On 7/12/2021 2:57 PM, Wesley Eddy wrote:

This revision has changes resulting from Martin's AD review and subsequent comments from Yuchung and Joe.

A few things to specifically check over since I wasn't sure if the thread totally converged on them, and they should be confirmed:

(1) urgent pointer "receding"

I think the key to understand is that the only time an application can set this is when data is written, so it should not "recede" since you can't write backwards.   There was already a paragraph saying: 

   To send an urgent indication the user must also send at least one
   data octet.

I'm not sure if this completely satisfies Martin's comment, but for now, I added this sentence:

  Note that because changes in the urgent pointer correspond
   to data being written by a sending application, the urgent pointer
   can not "recede" in the sequence space, but a TCP receiver should be
   robust to invalid urgent pointer values.

(2) PUSH check in SWS timer expiration

I think Martin is right, and if the intention is that if the SWS timer expires, it's okay to send, even if PUSH isn't set.  This:

   or if data is PUSHed and the override timeout occurs.

was changed to:

   or if the override timeout occurs.

(3) unnecessary Step 4 for LISTEN state when SEGMENT ARRIVES

Changed:

         fourth other text or control

            Any other control or text-bearing segment (not containing
            SYN) must have an ACK and thus would be discarded by the ACK
            processing.  An incoming RST segment could not be valid,
            since it could not have been sent in response to anything
            sent by this incarnation of the connection.  So, if this
            unlikely condition is reached, the correct behavior is to
            drop the segment and return.

to:

      fourth other data or control

         This should not be reached.  Drop the segment and return.  Any
         other control or data-bearing segment (not containing SYN) must
         have an ACK and thus would have been discarded by the ACK
         processing in the second step, unless it was first discarded by
         RST checking in the first step.

Although maybe if we agree that it's totally pointless to have, then this "fourth" step can just be removed entirely.



On 7/12/2021 2:48 PM, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : Transmission Control Protocol (TCP) Specification
        Author          : Wesley M. Eddy
	Filename        : draft-ietf-tcpm-rfc793bis-24.txt
	Pages           : 108
	Date            : 2021-07-12

Abstract:
   This document specifies the Transmission Control Protocol (TCP).  TCP
   is an important transport layer protocol in the Internet protocol
   stack, and has continuously evolved over decades of use and growth of
   the Internet.  Over this time, a number of changes have been made to
   TCP as it was specified in RFC 793, though these have only been
   documented in a piecemeal fashion.  This document collects and brings
   those changes together with the protocol specification from RFC 793.
   This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093,
   6429, 6528, and 6691 that updated parts of RFC 793.  It updates RFC
   1122, and should be considered as a replacement for the portions of
   that document dealing with TCP requirements.  It also updates RFC
   5961 by adding a small clarification in reset handling while in the
   SYN-RECEIVED state.  The TCP header control bits from RFC 793 have
   also been updated based on RFC 3168.

   RFC EDITOR NOTE: If approved for publication as an RFC, this should
   be marked additionally as "STD: 7" and replace RFC 793 in that role.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rfc793bis-24

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc793bis-24


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--------------ADEA73B7D6C321FD9C1FF687-- From nobody Mon Jul 12 13:47:45 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6F6733A0D35 for ; Mon, 12 Jul 2021 13:47:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nh2A-LpWQhCV for ; Mon, 12 Jul 2021 13:47:38 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 435DB3A0D7B for ; Mon, 12 Jul 2021 13:47:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=FQD8T4arP/B2Q2GNInI70PdJNgRNnegUZpHaGIUKnGo=; b=oXt0xtV9apIyFNibPxfj4lYANx 5974r5hG04v+q1wicCif/HzRh2TfQ+v+5MwWBG2PskMkZDo1umQkLAm1IUmy8YtbDO/hhJZ4DE665 vaI0n/ZDvFVbqJ2gficYwH/GZCmVrf2LPbl7TIsDhBPQHafjUefqnwvGqIvatk9fwfqwtjMfuS6Ns 5UUVoC6756rn3zxb9MDnYcbIUBPgypcw7B3sxN1rrF//qv32VkwJ7uvhH2kf1zimx4TxhN05JyNdj oEWdcjMp2A1DOZ4p91YGi/yxG/sXIS2tvzoYo48gUx/IrjrRk/56YTZbt6UbBVWjZ876GZrGggAEs 44vbuBpw==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:35580 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m32px-0001BL-Bb; Mon, 12 Jul 2021 21:47:32 +0100 To: Neal Cardwell Cc: tcpm@ietf.org, Mirja Kuehlewind , Yuchung Cheng , Richard Scheffenegger , Christian Huitema , Ilpo Jarvinen , =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= References: <2482da71-5b79-1933-f975-e46cd7661c39@bobbriscoe.net> <474DCED0-622E-413F-A4A0-9539548A6377@huitema.net> <6ad1817b-2b58-8aee-2cf5-3654e33b3eda@bobbriscoe.net> From: Bob Briscoe Message-ID: Date: Mon, 12 Jul 2021 21:47:28 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------ABC66D4F0BDD33F68A232F75" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 20:47:44 -0000 This is a multi-part message in MIME format. --------------ABC66D4F0BDD33F68A232F75 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Neal, On 12/07/2021 15:35, Neal Cardwell wrote: > > > On Mon, Jul 12, 2021 at 8:30 AM Bob Briscoe > wrote: > > Neal, > > On 11/07/2021 19:28, Neal Cardwell wrote: >> >> >> On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema >> > wrote: >> >> >> >> > On Jul 11, 2021, at 4:42 AM, Bob Briscoe >> > wrote: >> >> ... >> >> > The implementation will have its own ACK ratio - that's out >> of scope of AccECN, except to set this max of 6 CEs, which is >> to mitigate wrap of the 3-bit counter of CE-marks (which in >> the worst case of 100% marking in the data direction could >> then induce 1 ACK per 6 packets). This shouldn't limit >> forward performance, because it only increases the reverse >> ACK rate if there is heavy congestion in the forward path, >> when the Data Sender should be reducing the forward rate anyway. >> >> 1 ACK per 6 packets would cause performance issues on high >> speed links. Common setup is "4 to 8 ACK per RTT", which >> means intervals much larger than 6 packets. >> >> >> I share this concern about the AccECN requirement of one ACK per >> 6 CE-marked data segments potentially causing performance issues >> with high-speed links. Today, with most high-speed last-hop link >> technologies (wifi, cellular, Ethernet) TCP receivers often >> receive (from lower layers of the hardware and software >> networking stack) large aggregates  (often up to 64KBytes or >> around 44 packets), and generate a single ACK for that aggregate. >> AFAICT changing this scenario to require 1 packet per 6 CE-marked >> data segments means that during CE-marked scenarios receiving >> such an aggregate would require ceil(44/6) = 8 ACKs, increasing >> the number of ACKs by up to 8x. >> >> I have two main concerns in that scenario: >> >> (1) CPU load: This seems like it would impose much higher CPU >> load on the data receiver, at just the moment when the receiver >> may already be under stress due to receiving data near the >> maximum rate of the link. >> >> (2) Congestion and ACK loss: This requirement for generating up >> to 8 ACKs per aggregate seems like it is likely to produce a >> tight burst of 8 ACKs, which is going to increase congestion and >> increase the odds of losing at least one ACK, which is going to >> cause accuracy problems for "Accurate" ECN. :-) > > [BB] I hadn't intended this wording to apply in response to a > large burst. I'd want to alter the wording to allow just one ACK > in this case. > > > In the case where there is just one ACK for a big CE-marked burst of > >= 8 segments, does that mean the result is: > > (a) the ACE field loses accuracy, or > > (b) the TCP connection interpreting the ACE field should presume the > ACE field probably wrapped and estimate the number of CE-marked > segments using the number of SACKed/ACKed segments, rather than the > ACE field? > [BB] Answers: a) Yes, (obviously) less accurate than with the Option b) Sort-of. Just presuming the field wrapped as many times as possible gives v poor performance. Ilpo and I have been developing heuristics to deal with cases where the number of SACKed/ACKed segments is >=8. I'm planning to report the results in the meeting. >> I understand that this proposed "ACK per 6 CE-marked data >> segments" rule is necessary to avoid issues with the 3-bit ACE >> field wrapping. So IMHO this is one of the good arguments against >> including the ACE field in the AccECN design. > > [BB] I think the subtext here is a preference for the DCTCP style > of feedback vs ACE (when there is no AccECN TCP Option). I don't > think this particular issue is any different between the two. > > > Yes, when the AccECN TCP Option is not present, I'd prefer DCTCP-style > feedback rather than the ACE field. > > With DCTCP feedback, when a large data burst like this arrives, if > there are transitions to and from CE marking within the burst, > does Linux DCTCP generate an ACK for each transition, and for > every n repetitions of CE not ECT, like the RFC says it should? Or > does the implementation just pop out one ACK at the end? > > > Yes, with Linux DCTCP (or BBRv2), when there is a big burst, there is > an ACK for each CE<->non-CE transition. They also handle the case > where there is a CE<->non-CE transition while a delayed ACK is > pending. This requires cooperation from a few different layers. > > (1) the NIC hardware LRO aggregation is supposed to not aggregate > across CE<->non-CE transitions. I believe the de facto spec on NIC > aggregation with respect to IP fields is the following, which mandates > that: > https://docs.microsoft.com/en-us/windows-hardware/drivers/network/updating-the-ip-headers-for-coalesced-segments > > > (2) the Linux software GRO aggregation does not aggregate across > CE<->non-CE transitions; see inet_gro_receive() checking the  ToS > bytes: (iph->tos ^ iph2->tos) > > (3) the DCTCP and BBRv2 code use dctcp_ece_ack_update() to handle the > case where there is a CE<->non-CE transition while a delayed ACK is > pending [BB] Understood. So doesn't my point stand that the existing hardware/software combination already often/typically emits numerous ACKs in response to arrival of a large data burst? Indeed, designing the ACE field as a counter was deliberately intended to reduce that. Did you see the text we included in the current rev (-14 in Feb'21) that explains this: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-14#section-3.3.4 It also explains how the new approach can be handled while hardware is not available. Essentially, assuming you start with a subset of users negotiating AccECN, those sessions would switch receive offload to software-only, until hardware became available. But I'd appreciate your thoughts on the whole section. I've updated my local copy of the 'rules' in our proposed new rev of draft-15, to replace 'immediately send' and explain better what it means. Here it is: 3.2.2.5.1. Data Receiver Safety Procedures The following rules define when a Data Receiver in AccECN mode emits an ACK: Change-Triggered ACKs: An AccECN Data Receiver SHOULD emit an ACK whenever a data packet marked CE arrives after the previous packet was not CE. Even though this rule is stated as a "SHOULD", it is important for a transition to trigger an ACK if at all possible, The only valid exception to this rule is given below these bullets. For the avoidance of doubt, this rule is deliberately worded to apply solely when _data_ packets arrive, but the comparison with the previous packet includes any packet, not just data packets. Increment-Triggered ACKs: An AccECN Data Receiver MUST emit an ACK if 'n' CE marks have arrived since the previous ACK. If there is new data to acknowledge, 'n' SHOULD be 2. If there is no new data to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3. In either case, 'n' MUST be no greater than 6. The above rules for when to send an ACK are designed to be complemented by those in Section 3.2.3.3, which concern whether the AccECN TCP Option ought to be included on ACKs. If the arrivals of a number of data packets are all processed as one event, e.g. using large receive offload (LRO) or generic receive offload (GRO), both the above rules SHOULD be interpreted as requiring multiple ACKs to be emitted back-to-back (for each transition and for each repetition by 'n' CE marks). If this is problematic for high performance, either rule can be interpreted as requiring just a single ACK at the end of the whole receive event. Even if a number of data packets do not arrive as one event, the 'Change-Triggered ACKs' rule could sometimes cause the ACK rate to be problematic for high performance (although high performance protocols such as DCTCP already successfully use change-triggered ACKs). The rationale for change-triggered ACKs is so that the Data Sender can rely on them to detect queue growth as soon as possible, particularly at the start of a flow. The approach can lead to some additional ACKs but it feeds back the timing and the order in which ECN marks are received with minimal additional complexity. If CE marks are infrequent, or there are multiple marks in a row, the additional load will be low. Other marking patterns could increase the load significantly. One possible compromise would be for the receiver to heuristically detect whether the sender is in slow-start, then to implement change-triggered ACKs while the sender is in slow-start, and offload otherwise. With ECN-capable pure ACKs [I-D.ietf-tcpm-generalized-ecn], the 'Increment-Triggered ACKs' rule could cause ECN-marked pure ACKs to trigger further ACKs. [...snipped to avoid muddling this thread with the "ACKing ACKs without good cause" thread] > > Ilpo & I have been doing experiments with high levels of ACK > coalescing causing the ACE field seen by the Data Sender to wrap > multiple times under high congestion. For example, 1 ACK per ~4ms > is one common scheme, which would result in about 1 ACK for 34 > data packets at just 100Mb/s. > > > Great. Yes, certainly that level of aggregation is important to test. > > >> >> The other main concerns I have with the ACE field are: >> >> o Complexity > > [BB] I understand there could be a tradeoff between complexity and > accuracy. But I think the complexity of ACE and DCTCP feedback are > pretty much the same but, with even low levels of ACK coalescing, > the accuracy of ACE is superior. Nonetheless, I understand that > deploying something different to DCTCP when you've already got > DCTCP could involve deployment complexity. > > > IMHO the ACE field is more complex, due to: > > o context-dependent interpretation of the header bits used by ACE [BB] That would I think be the same for the DCTCP feedback scheme (it's only not the case now in a DC where you can hard-code rather than negotiate the scheme). > > o logic to handle wraps in the ACE field and estimate what the real > increment to the ACE field was in the case of possible ACK loss [BB] That's slightly true ;) I would say it like this: * With DCTCP feedback, a simple line of code can fill in the missing ACKs: it assumes the marking continued as per the last signal. This is very inaccurate under high levels of ACK coalescing and there is no obvious way to make it more accurate. * With ACE feedback a simple line of code can also fill in the missing ACKs. However, in contrast to DCTCP, you can add code to make ACE more accurate if you are prepared to add the extra complexity (but you don't have to). > > o interactions with drivers or hardware that already have assumptions > about the header bits used for ACE (e.g. that the CWR bit should be > cleared for the first N-1 segments in a TSO burst and only set on the > last segment of the TSO burst) [BB] > > > >> >> o Redundancy with respect to the AccECN counter options > > [BB] The redundancy is intended. For cases where the TCP Option > doesn't traverse the path, or where TCP option space is limited. > > > I realize the redundancy is intentional, but the redundant code > does impose a maintenance cost. > >> >> o Potential problems with middleboxes > > [BB] Well, we've tested the 3 header bits over millions of paths > without problems. But yes, there are billions more to test. > >> >> o Known problems with NICs and drivers (based on Ilpo's nice >> talk, "Accurate ECN Linux Implementation: Experiences and >> Challenges", at the April 2020 TCPM interim meeting) > > [BB] I understood that talk as concluding there weren't really > problems. What specific problem do you have in mind? > > > Mainly I'm concerned about: > > o interactions with drivers or hardware that already have assumptions > about the header bits used for ACE (e.g. that the CWR bit should be > cleared for the first N-1 segments in a TSO burst and only set on the > last segment of the TSO burst) > > It sounds like since the ACE field redefines the semantics of some > long-standing TCP header fields, there will be a long trial-and-error > period of finding the drivers and NICs that are incompatible with the > ACE field, and working around those. [BB] Yes. You can't make an omelette without breaking eggs. Meaning, to get significantly lower delay over the public Internet, we can't expect it all just to fall into place with minimal effort. Given we have worked out a way to deploy pretty accurate ECN feedback without the option, it would be a shame to decide not to, just because there /might/ be problems. There always /might/ be problems, indeed there nearly always are. If DCTCP feedback were used over the public Internet, it would have to be included some form of negotiation framework, probably like AccECN, and that would probably also trip over some hardware incompatibilities. Why make half-hearted changes and still have potential deployment problems? I really think we need to question the prevailing preference for inconsequential changes, when we end up with so much grief making /any/ change. >> I realize that if AccECN does not have the ACE field feature, >> then AccECN and TCP L4S will not be usable on paths with >> middleboxes that strip the AccECN counter options. But IMHO >> living without the ACE field is preferable. IMHO it's acceptable >> to say that L4S can only be used with (a) QUIC, or (b) TCP >> connections where no middleboxes are stripping AccECN options. > > [BB] When you say 'living without the ACE field', alongside the > AccECN TCP Option, would you leave classic ECN feedback? Or put > DCTCP feedback in its place? > > > When 'living without the ACE field', I mean either using the AccECN > TCP Option or DCTCP-style feedback. > > I don't think there's any case for using classic ECN feedback > within AccECN. I see the competition as between ACE and DCTCP > feedback, each optionally with the AccECN TCP Option. > > > Agreed. I agree there's no case for using classic ECN feedback within > AccECN. > > If DCTCP feedback were a stop-gap until we could get good > traversal of the TCP Option, it might have some merit, but I think > DCTCP isn't going to work well with the level of ACK coalescing in > the Internet. However, this is very difficult to judge unless we > do large scale A-B experiments on the real Internet (and > potentially within DCs). I'll think further about enabling some > way to do that, possibly within AccECN's negotiation framework. > But today, I have to prioritize for the draft deadline. > > > I agree it's unclear whether DCTCP-style feedback will work well over > the public Internet, due to ACK loss and the variety of aggregation > mechanisms in play. That is why in my previous email in this thread I > was mainly presuming that if the AccECN TCP Option is not forwarded by > the path then the flow might have to just disable AccECN and L4S. [BB] If you're saying a third possibility would be to have no ECN feedback in the header, only via an option, that seems perverse, given the ECN header bits would then be idle. And, if you consumed the option space for SACK, timestamps, (even MPTCP) etc. you might have no ECN feedback at all. The ability to have wider early deployment is important for the economic motivations of everyone involved, so not to be dismissed lightly. I'll think whether we can find some way to try out both ACE and DCTCP feedback within a realistic protocol so that all the deployment niggles and performance can be tested and compared. But if you also agree that DCTCP probably won't work well over the Internet, it might be a non-starter anyway (but worth checking if not too onerous). Bob > > cheers, > neal > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------ABC66D4F0BDD33F68A232F75 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Neal,

On 12/07/2021 15:35, Neal Cardwell wrote:


On Mon, Jul 12, 2021 at 8:30 AM Bob Briscoe <ietf@bobbriscoe.net> wrote:
Neal,

On 11/07/2021 19:28, Neal Cardwell wrote:


On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema <huitema@huitema.net> wrote:
 

> On Jul 11, 2021, at 4:42 AM, Bob Briscoe <ietf@bobbriscoe.net> wrote:
... 
> The implementation will have its own ACK ratio - that's out of scope of AccECN, except to set this max of 6 CEs, which is to mitigate wrap of the 3-bit counter of CE-marks (which in the worst case of 100% marking in the data direction could then induce 1 ACK per 6 packets). This shouldn't limit forward performance, because it only increases the reverse ACK rate if there is heavy congestion in the forward path, when the Data Sender should be reducing the forward rate anyway.

1 ACK per 6 packets would cause performance issues on high speed links. Common setup is "4 to 8 ACK per RTT", which means intervals much larger than 6 packets.

I share this concern about the AccECN requirement of one ACK per 6 CE-marked data segments potentially causing performance issues with high-speed links. Today, with most high-speed last-hop link technologies (wifi, cellular, Ethernet) TCP receivers often receive (from lower layers of the hardware and software networking stack) large aggregates  (often up to 64KBytes or around 44 packets), and generate a single ACK for that aggregate. AFAICT changing this scenario to require 1 packet per 6 CE-marked data segments means that during CE-marked scenarios receiving such an aggregate would require ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x. 

I have two main concerns in that scenario:

(1) CPU load: This seems like it would impose much higher CPU load on the data receiver, at just the moment when the receiver may already be under stress due to receiving data near the maximum rate of the link.

(2) Congestion and ACK loss: This requirement for generating up to 8 ACKs per aggregate seems like it is likely to produce a tight burst of 8 ACKs, which is going to increase congestion and increase the odds of losing at least one ACK, which is going to cause accuracy problems for "Accurate" ECN. :-)

[BB] I hadn't intended this wording to apply in response to a large burst. I'd want to alter the wording to allow just one ACK in this case.

In the case where there is just one ACK for a big CE-marked burst of >= 8 segments, does that mean the result is:

(a) the ACE field loses accuracy, or

(b) the TCP connection interpreting the ACE field should presume the ACE field probably wrapped and estimate the number of CE-marked segments using the number of SACKed/ACKed segments, rather than the ACE field?


[BB] Answers:
a) Yes, (obviously) less accurate than with the Option
b) Sort-of.
Just presuming the field wrapped as many times as possible gives v poor performance.
Ilpo and I have been developing heuristics to deal with cases where the number of SACKed/ACKed segments is >=8.
I'm planning to report the results in the meeting.


 
I understand that this proposed "ACK per 6 CE-marked data segments" rule is necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is one of the good arguments against including the ACE field in the AccECN design.

[BB] I think the subtext here is a preference for the DCTCP style of feedback vs ACE (when there is no AccECN TCP Option). I don't think this particular issue is any different between the two.

Yes, when the AccECN TCP Option is not present, I'd prefer DCTCP-style feedback rather than the ACE field.
 
With DCTCP feedback, when a large data burst like this arrives, if there are transitions to and from CE marking within the burst, does Linux DCTCP generate an ACK for each transition, and for every n repetitions of CE not ECT, like the RFC says it should? Or does the implementation just pop out one ACK at the end?

Yes, with Linux DCTCP (or BBRv2), when there is a big burst, there is an ACK for each CE<->non-CE transition. They also handle the case where there is a CE<->non-CE transition while a delayed ACK is pending. This requires cooperation from a few different layers.

(1) the NIC hardware LRO aggregation is supposed to not aggregate across CE<->non-CE transitions. I believe the de facto spec on NIC aggregation with respect to IP fields is the following, which mandates that:

(2) the Linux software GRO aggregation does not aggregate across CE<->non-CE transitions; see inet_gro_receive() checking the  ToS bytes: (iph->tos ^ iph2->tos) 

(3) the DCTCP and BBRv2 code use dctcp_ece_ack_update() to handle the case where there is a CE<->non-CE transition while a delayed ACK is pending

[BB] Understood. So doesn't my point stand that the existing hardware/software combination already often/typically emits numerous ACKs in response to arrival of a large data burst? Indeed, designing the ACE field as a counter was deliberately intended to reduce that.

Did you see the text we included in the current rev (-14 in Feb'21) that explains this:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-14#section-3.3.4
It also explains how the new approach can be handled while hardware is not available. Essentially, assuming you start with a subset of users negotiating AccECN, those sessions would switch receive offload to software-only, until hardware became available. But I'd appreciate your thoughts on the whole section.

I've updated my local copy of the 'rules' in our proposed new rev of draft-15, to replace 'immediately send' and explain better what it means. Here it is:

3.2.2.5.1.  Data Receiver Safety Procedures

   The following rules define when a Data Receiver in AccECN mode emits
   an ACK:

   Change-Triggered ACKs:  An AccECN Data Receiver SHOULD emit an ACK
      whenever a data packet marked CE arrives after the previous packet
      was not CE.

      Even though this rule is stated as a "SHOULD", it is important for
      a transition to trigger an ACK if at all possible, The only valid
      exception to this rule is given below these bullets.

      For the avoidance of doubt, this rule is deliberately worded to
      apply solely when _data_ packets arrive, but the comparison with
      the previous packet includes any packet, not just data packets.

   Increment-Triggered ACKs:  An AccECN Data Receiver MUST emit an ACK
      if 'n' CE marks have arrived since the previous ACK.  If there is
      new data to acknowledge, 'n' SHOULD be 2.  If there is no new data
      to acknowledge, 'n' SHOULD be 3 and MUST be no less than 3.  In
      either case, 'n' MUST be no greater than 6.

   The above rules for when to send an ACK are designed to be
   complemented by those in Section 3.2.3.3, which concern whether the
   AccECN TCP Option ought to be included on ACKs.

   If the arrivals of a number of data packets are all processed as one
   event, e.g. using large receive offload (LRO) or generic receive
   offload (GRO), both the above rules SHOULD be interpreted as
   requiring multiple ACKs to be emitted back-to-back (for each
   transition and for each repetition by 'n' CE marks).  If this is
   problematic for high performance, either rule can be interpreted as
   requiring just a single ACK at the end of the whole receive event.

   Even if a number of data packets do not arrive as one event, the
   'Change-Triggered ACKs' rule could sometimes cause the ACK rate to be
   problematic for high performance (although high performance protocols
   such as DCTCP already successfully use change-triggered ACKs).  The
   rationale for change-triggered ACKs is so that the Data Sender can
   rely on them to detect queue growth as soon as possible, particularly
   at the start of a flow.  The approach can lead to some additional
   ACKs but it feeds back the timing and the order in which ECN marks
   are received with minimal additional complexity.  If CE marks are
   infrequent, or there are multiple marks in a row, the additional load
   will be low.  Other marking patterns could increase the load
   significantly.  One possible compromise would be for the receiver to
   heuristically detect whether the sender is in slow-start, then to
   implement change-triggered ACKs while the sender is in slow-start,
   and offload otherwise.

   With ECN-capable pure ACKs [I-D.ietf-tcpm-generalized-ecn], the
   'Increment-Triggered ACKs' rule could cause ECN-marked pure ACKs to
   trigger further ACKs. [...snipped to avoid muddling this thread 
                 with the "ACKing ACKs without good cause" thread]


 

Ilpo & I have been doing experiments with high levels of ACK coalescing causing the ACE field seen by the Data Sender to wrap multiple times under high congestion. For example, 1 ACK per ~4ms is one common scheme, which would result in about 1 ACK for 34 data packets at just 100Mb/s.

Great. Yes, certainly that level of aggregation is important to test.
 


The other main concerns I have with the ACE field are:

o Complexity

[BB] I understand there could be a tradeoff between complexity and accuracy. But I think the complexity of ACE and DCTCP feedback are pretty much the same but, with even low levels of ACK coalescing, the accuracy of ACE is superior. Nonetheless, I understand that deploying something different to DCTCP when you've already got DCTCP could involve deployment complexity.

IMHO the ACE field is more complex, due to:

o context-dependent interpretation of the header bits used by ACE

[BB] That would I think be the same for the DCTCP feedback scheme (it's only not the case now in a DC where you can hard-code rather than negotiate the scheme).


o logic to handle wraps in the ACE field and estimate what the real increment to the ACE field was in the case of possible ACK loss

[BB] That's slightly true ;)

I would say it like this:
* With DCTCP feedback, a simple line of code can fill in the missing ACKs: it assumes the marking continued as per the last signal. This is very inaccurate under high levels of ACK coalescing and there is no obvious way to make it more accurate.
* With ACE feedback a simple line of code can also fill in the missing ACKs. However, in contrast to DCTCP, you can add code to make ACE more accurate if you are prepared to add the extra complexity (but you don't have to).


o interactions with drivers or hardware that already have assumptions about the header bits used for ACE (e.g. that the CWR bit should be cleared for the first N-1 segments in a TSO burst and only set on the last segment of the TSO burst)

[BB]


 



o Redundancy with respect to the AccECN counter options

[BB] The redundancy is intended. For cases where the TCP Option doesn't traverse the path, or where TCP option space is limited.

I realize the redundancy is intentional, but the redundant code does impose a maintenance cost.
 

o Potential problems with middleboxes

[BB] Well, we've tested the 3 header bits over millions of paths without problems. But yes, there are billions more to test.


o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate ECN Linux Implementation: Experiences and Challenges", at the April 2020 TCPM interim meeting)

[BB] I understood that talk as concluding there weren't really problems. What specific problem do you have in mind?

Mainly I'm concerned about:

o interactions with drivers or hardware that already have assumptions about the header bits used for ACE (e.g. that the CWR bit should be cleared for the first N-1 segments in a TSO burst and only set on the last segment of the TSO burst)

It sounds like since the ACE field redefines the semantics of some long-standing TCP header fields, there will be a long trial-and-error period of finding the drivers and NICs that are incompatible with the ACE field, and working around those.

[BB] Yes. You can't make an omelette without breaking eggs. Meaning, to get significantly lower delay over the public Internet, we can't expect it all just to fall into place with minimal effort. Given we have worked out a way to deploy pretty accurate ECN feedback without the option, it would be a shame to decide not to, just because there /might/ be problems. There always /might/ be problems, indeed there nearly always are.

If DCTCP feedback were used over the public Internet, it would have to be included some form of negotiation framework, probably like AccECN, and that would probably also trip over some hardware incompatibilities. Why make half-hearted changes and still have potential deployment problems?

I really think we need to question the prevailing preference for inconsequential changes, when we end up with so much grief making /any/ change.

 
I realize that if AccECN does not have the ACE field feature, then AccECN and TCP L4S will not be usable on paths with middleboxes that strip the AccECN counter options. But IMHO living without the ACE field is preferable. IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or (b) TCP connections where no middleboxes are stripping AccECN options.

[BB] When you say 'living without the ACE field', alongside the AccECN TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback in its place?

When 'living without the ACE field', I mean either using the AccECN TCP Option or DCTCP-style feedback.
 
I don't think there's any case for using classic ECN feedback within AccECN. I see the competition as between ACE and DCTCP feedback, each optionally with the AccECN TCP Option.

Agreed. I agree there's no case for using classic ECN feedback within AccECN.
 
If DCTCP feedback were a stop-gap until we could get good traversal of the TCP Option, it might have some merit, but I think DCTCP isn't going to work well with the level of ACK coalescing in the Internet. However, this is very difficult to judge unless we do large scale A-B experiments on the real Internet (and potentially within DCs). I'll think further about enabling some way to do that, possibly within AccECN's negotiation framework. But today, I have to prioritize for the draft deadline.

I agree it's unclear whether DCTCP-style feedback will work well over the public Internet, due to ACK loss and the variety of aggregation mechanisms in play. That is why in my previous email in this thread I was mainly presuming that if the AccECN TCP Option is not forwarded by the path then the flow might have to just disable AccECN and L4S.

[BB] If you're saying a third possibility would be to have no ECN feedback in the header, only via an option, that seems perverse, given the ECN header bits would then be idle. And, if you consumed the option space for SACK, timestamps, (even MPTCP) etc. you might have no ECN feedback at all.

The ability to have wider early deployment is important for the economic motivations of everyone involved, so not to be dismissed lightly.

I'll think whether we can find some way to try out both ACE and DCTCP feedback within a realistic protocol so that all the deployment niggles and performance can be tested and compared. But if you also agree that DCTCP probably won't work well over the Internet, it might be a non-starter anyway (but worth checking if not too onerous).


Bob


cheers,
neal


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------ABC66D4F0BDD33F68A232F75-- From nobody Mon Jul 12 14:49:08 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id AB2AD3A0EB8; Mon, 12 Jul 2021 14:48:54 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162612653463.2420.8417243673097691082@ietfa.amsl.com> Date: Mon, 12 Jul 2021 14:48:54 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-15.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 21:49:03 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : More Accurate ECN Feedback in TCP Authors : Bob Briscoe Mirja Kühlewind Richard Scheffenegger Filename : draft-ietf-tcpm-accurate-ecn-15.txt Pages : 59 Date : 2021-07-12 Abstract: Explicit Congestion Notification (ECN) is a mechanism where network nodes can mark IP packets instead of dropping them to indicate incipient congestion to the end-points. Receivers with an ECN- capable transport protocol feed back this information to the sender. ECN was originally specified for TCP in such a way that only one feedback signal can be transmitted per Round-Trip Time (RTT). Recent new TCP mechanisms like Congestion Exposure (ConEx), Data Center TCP (DCTCP) or Low Latency Low Loss Scalable Throughput (L4S) need more accurate ECN feedback information whenever more than one marking is received in one RTT. This document specifies a scheme to provide more than one feedback signal per RTT in the TCP header. Given TCP header space is scarce, it allocates a reserved header bit previously assigned to the ECN-Nonce. It also overloads the two existing ECN flags in the TCP header. The resulting extra space is exploited to feed back the IP-ECN field received during the 3-way handshake as well. Supplementary feedback information can optionally be provided in a new TCP option, which is never used on the TCP SYN. The document also specifies the treatment of this updated TCP wire protocol by middleboxes. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-15 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-15 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Jul 12 14:51:04 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 51C153A0E89; Mon, 12 Jul 2021 14:50:54 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rcR0pQrb8IpQ; Mon, 12 Jul 2021 14:50:49 -0700 (PDT) Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6DECA3A106E; Mon, 12 Jul 2021 14:50:36 -0700 (PDT) Received: by mail-il1-x12d.google.com with SMTP id h3so21057207ilc.9; Mon, 12 Jul 2021 14:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5ahkeDQS6b2VXqGf7I8XH3mwP77out6c11VSaI156+E=; b=eCddSckbN2gOJHm45LnWgIYZVZkiqZTdcbC0H3SPNQDn/Htk1GJZHbOusrrTLEVT0V ZDhMs8UU6gyPJOhLZKup2z0L6Vmrf8JdvEXZCyNYzIgWGuvqr68YhNMIK6orh7OO9NSy yjyyZsoPDM1CRnsvmwtm5ghB6ibmOhrkT4Q8a8v5JZg9H9N3VWR9OjEE6TVrYd1Zr+PL 8rDbglHle1cC6xi/bwpax8WWKBXcfLa9hn+K3AjOpoisLdJLEmeNmsSx61SiLU2GMPVQ kD9PLaixi3YtjufnG2F+0SR5j1QLGvNWgjKLRtBGptfZqS7g07pnn1rYSQ3Q1gv31/Ve YO0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5ahkeDQS6b2VXqGf7I8XH3mwP77out6c11VSaI156+E=; b=KiTb4XubmQOLua/oNzz8DdwnfKLZFHuALu0ABOpeH7e/PFghMEX1oiWnSud9z4OJhN JHn5t7zUkN+X1+cpwy6Y10MhqNOvwflnk5Ccn/QYLzUwekWZeqW3EAgMVW4T8Qs5X8Hd 2LOe61iT9y5pG1OTVFzSlxPn8KfD+CIYs4h/pVF8B/BXIalaD9MTHKZEGrOteXMWbxj6 Wqo+xrGoxC4SIM2tOZIPevGlMwoCAPcYdvVpLjBgRtFkG7Fyf9HCPwqkIiHw/YbQuGDn UBqPDc3MWPdOkidM6Nqm8NZYnPODgb0Mrg8uBqDVSULIh085Rw6PQdmukID0pRFaI+0c /Lqw== X-Gm-Message-State: AOAM530VlylqUTuzbQtoi5XmdZVylOdXxzNWzcSPGR8SWvQF+ghUSB0h 39tHh6YBoggqX+lizmR0MBRI0HLqRFI54prF8nQ= X-Google-Smtp-Source: ABdhPJwUDJT2QTOaL3sKZUtncFfOAFdc4+TsJp0v+MD8V7zdW7ErBSBw6IomhjhFSTd+FxCJbTx3o7brq134+OMduTA= X-Received: by 2002:a92:cd8a:: with SMTP id r10mr578638ilb.287.1626126635135; Mon, 12 Jul 2021 14:50:35 -0700 (PDT) MIME-Version: 1.0 References: <59e1d10e-542c-3f1f-7098-4f1d8932da5b@mti-systems.com> In-Reply-To: From: Martin Duke Date: Mon, 12 Jul 2021 14:50:24 -0700 Message-ID: To: Wesley Eddy Cc: draft-ietf-tcpm-rfc793bis.all@ietf.org, "tcpm@ietf.org Extensions" Content-Type: multipart/alternative; boundary="00000000000072782705c6f41c8e" Archived-At: Subject: Re: [tcpm] RFC793bis AD review X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 21:51:02 -0000 --00000000000072782705c6f41c8e Content-Type: text/plain; charset="UTF-8" Wes, It looks like draft-24 of 793bis gets it all, except: Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have > > If the urgent pointer is updated while the user is in "urgent mode", the > update will be invisible to the user. > > A TCP implementation MUST (MUST-32) inform the application layer > asynchronously whenever it receives an Urgent pointer and there was > previously no pending urgent data, or whenever the Urgent pointer > advances in the data stream. > > If the latter text is correct, AND we decide to allow receding, we should > decide what to do with recedes here. > I'm not holding up Last Call over this, but please fix it before IESG Evaluation. Thanks! On Thu, Jul 1, 2021 at 11:24 AM Martin Duke wrote: > > > On Wed, Jun 30, 2021 at 2:50 PM Wesley Eddy wrote: > >> Here are my replies on each finding, with proposed changes to make sure >> you agree, before cutting another revision of the document: >> >> >> On 6/24/2021 4:12 PM, Martin Duke wrote: >> >> >> (Document Header) Please update the Intended Status to "Internet >> Standard". >> >> There doesn't seem to be a way to force this in xml2rfc3; the 'category' >> attribute is set to 'std' (Standards Track) and there's no distinction >> (this looks consistent with other recent Internet Standard RFCs). >> >> So, I think no change is needed on this. >> > > OK > > >> >> (3.8.3) R1 and R2 can be measured in retransmissions or seconds; the >> proposed bounds are in retransmissions for R1 and seconds for R2. So if I >> have a time-based R1, should that be 3? If I have a retransmission based >> R2, should I compare that to the value of RTO somehow? >> >> This is heritage from RFC 1122. The text on R1 mentions "at the current >> RTO", so I think RO is definitely the intended conversion factor, but since >> you asked, maybe it's not clear enough? Do you think maybe in part (a) of >> 3.8.3 we could just change: >> >> R1 and R2 might be measured in time units or as a count of >> retransmissions. >> >> to: >> >> R1 and R2 might be measured in time units or as a count of >> retransmissions (with the current RTO as a conversion factor, if needed). >> >> I like that, thanks. > > >> >> (3.8.5) This section is unclear as to whether the sender can cause the >> Urgent Pointer to recede (move to the left), and if so, if that MUST be >> reported to the application like a pointer advance. >> >> Receding looks the same as advancing so much that it wraps. So, if not >> already in urgent mode, that would seem to cause a change to urgent mode, >> while if not already in urgent mode, that would fall under the existing >> rule of: >> >> If the urgent pointer is updated while the user is in "urgent mode", >> the update will be invisible to the user. >> >> So ... also given that the use of urgent pointer is deprecated anyways >> and we don't want to cause changes to TCP in this doc, I'm proposing we do >> nothing to clarify this, if you agree? >> > I don't think > > SEQ 5000 URG 3000 > SEQ 5001 URG 1500 > > would have a common-sense interpretation that the urg ptr was now 7501 + > 2^32. I share your grumpiness about sweating the useless urgent pointer, > but it would appear there is a gap in the spec and that in principle these > recedes could exist in the wild. This spec mostly matters for new > implementations; maybe the safest thing is for those new implementations to > prepare to handle a receding pointer, but that senders MUST NOT recede. I > would welcome a quick question to the WG if that logic seems spurious to > you. > > Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have > > If the urgent pointer is updated while the user is in "urgent mode", the > update will be invisible to the user. > > A TCP implementation MUST (MUST-32) inform the application layer > asynchronously whenever it receives an Urgent pointer and there was > previously no pending urgent data, or whenever the Urgent pointer > advances in the data stream. > > If the latter text is correct, AND we decide to allow receding, we should > decide what to do with recedes here. > >> (3.8.6.1) >> >> The sending TCP peer must be prepared to accept from the user and >> send at least one octet of new data even if the send window is zero. >> The sending TCP peer must regularly retransmit to the receiving TCP >> peer even when the window is zero >> >> Please clarify that these probes only happen when there is new data, if >> that's the intent. >> >> The "accept from the user" part means that new data was written already, >> right? If the user doesn't write data into the socket, there's nothing to >> be sent. >> > I have been messing with QUIC too long and may have forgotten how this > works. But IIRC, this text kind of confuses a few different kinds of > transmissions with zero window probes. It might be helpful to actually > define "zero window probe" -- a segment of one byte of data designed to > elicit an ack with a window update. The retransmission discussion is > related but somewhat confusing as currently written. There are four > zero-window scenarios that I can see. > > 1) If there is outstanding in-window data, we will get ACKs so there is no > need for a ZWP. > > 2) No send buffer data, but the application generates data while in > zero-window. Clearly, the spec says TCP accepts a byte and sends it. > > 3) TCP has previously refused application data due to window restrictions, > and now we're in zero-window. TCP needs to wake the app to send again so > that we have material for ZWP. > > 4) TCP accepted some data, held it for Nagle, and then the receiver > reneged; or all in-flight data is now out of window due to reneging. This > data ought to constitute a ZWP whether or not it is strictly a > "retransmission". > > If you believe all of my statements correct, I am happy to propose > alternate text that is not as wordy as the above., > > > (3.8.6.1) the first paragraph says > > Two minutes is recommended for the retransmission interval when the > window is zero. > > but the last one says > > The transmitting host SHOULD send the first zero-window probe when a > zero window has existed for the retransmission timeout period (SHLD- > 29) (Section 3.8.1 ), and SHOULD increase exponentially the interval > between successive probes (SHLD-30). > > so is it 2 min or the RTO? > > Good question ... The first sentence comes from 793 and the other from > 1122. I think we should erase the first sentence, since the 1122 intent > seems to be updating this (replacing the hard-coded 2 minute heuristic with > the RTO one which adapts to reality). Do you agree? > Yes, I agree. > >> >> (3.8.6.2.1) If I read this algorithm correctly, an un-pushed send buffer of < MSS, if D < Fs*Max(SND.WND), will sit there indefinitely without >> being transmitted; is this intentional? >> >> This comes from 1122. I think the key to preventing what you mention is >> this part of the preceding paragraph: >> >> To avoid a resulting deadlock, it is necessary to have a >> timeout to force transmission of data, overriding the SWS avoidance >> algorithm. >> >> So, SWS avoidance is not supposed to keep a sender waiting forever for >> the receiver to possibly advertise a bigger window; the sender is supposed >> to wake up and just send eventually. >> > > Yes, but the override timeout is embedded in step (4), which applies PUSH > as a condition. Maybe that PUSH condition is incorrect? > >> >> (3.8.2) and (3.8.6.3) Should RFC 5681 be a normative reference, given the >> congestion control requirement? If not, (3.8.6.3) shouldn't require the >> reader to go there to fully implement delayed ACKs. I suggest >> >> s/an ACK for at least every second segment/an ACK for least every 2*MSS >> bytes of data >> >> to fully state what is in 5681. >> >> Yeah, I think 5681 should be normative (since we also rely on the for >> descriptions of slow start, etc. that we didn't try to duplicate within >> 793bis), but we should also fix the "second segment" sentence to match more >> exactly the text in 5681 about 2*RMSS. >> > > OK, there's currently the pointer to 5681 about sub-MSS stuff, so it's > technically correct, but really clunky to force people to go there when > it's so easy to state correctly. I think we agree. > >> >> >> >> >> (3.10, OPEN/LISTEN) >> >> If active and the remote socket is specified, then change the >> connection from passive to active, >> >> I think you mean "If passive"? >> >> Actually, I think to be more clear this should say: >> >> "If the OPEN call is active ...", since it's referring to the case where >> an active OPEN is called when the socket is in LISTEN (passive). >> >> It's technically correct currently (just not very clear), if you read the >> "If active" as referring to the OPEN call rather than the socket (and it >> must be that way since a socket in the LISTEN state is the result of a >> passive OPEN). >> > > Yes, let's clarify > > >> (3.10, SEGMENT ARRIVES/LISTEN) Step 4 mentions RST but those have already >> been handled in Step 1. >> >> Yeah, you're definitely right, and in both cases the action is to ignore >> it, so this part of step 4 is irrelevant, but at least consistent. >> >> It looks like this is an errata to file on 793; should we do that quickly >> and then fix this doc accordingly? >> > If that is better for your process, then sure. > > >> *NITS:* >> >> >> >> >> (3.1, Options) " >> >> Options: [TCP Option]; Options#Size == (DOffset-5)*32; present only >> when DOffset > 5. >> >> What is this notation? >> >> It is a little odd, but comes from reference [61] for >> draft-mcquistin-augmented-ascii-diagrams-08. >> > > OK > > >> >> >> I'm good with changing all of these above. >> >> Thank you for the detailed review! >> > --00000000000072782705c6f41c8e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Wes,

It looks like draft-24 = of 793bis gets it all, except:

Meanwhile, it appear= s we've uncovered a contradiction! In 3.8.5 we have

If the urgent pointer is updated while the user is in "urgent mod=
e", the
   update will be invisible to the user.
A TCP implementation MUST (MUST-32) inform the application layer
   asynchronously whenever it receives an Urgent pointer and there was
   previously no pending urgent data, or whenever the Urgent pointer
   advances in the data stream.
=C2=A0If the latter text is correct, AND we decide to allow rece= ding, we should decide what to do with recedes here.

I'm not holding up Last Call over this, but please fix = it before IESG Evaluation.

Thanks!
<= br>

On Thu, Jul 1, 2021 at 11:24 AM Martin Duke <martin.h.duke@gmail.com> wrote:


On Wed, Jun 30, 2021 at 2:50 PM Wesley Eddy <wes@mti-systems.com> wrote:
=20 =20 =20
Here are my replies on each finding, with proposed changes to make sure you agree, before cutting another revision of the document:


On 6/24/2021 4:12 PM, Martin Duke wrote:
=20

(Document Header) Please update the Intended Status to "Internet Standard".

There doesn't seem to be a way to force this in xml2rfc3; the 'category' attribute is set to 'std' (Standards Track= ) and there's no distinction (this looks consistent with other recent Internet Standard RFCs).

So, I think no change is needed on this.

<= div>
OK
=C2=A0


(3.8.3) R1 and R2 can be measured in retransmissions or seconds; the proposed bounds are in retransmissions for R1 and seconds for R2. So if I have a time-based R1, should that be 3? If I have a retransmission based R2, should I compare that to the value of RTO somehow?

This is heritage from RFC 1122.=C2=A0 The text on R1 mentions "= at the current RTO", so I think RO is definitely the intended conversio= n factor, but since you asked, maybe it's not clear enough?=C2=A0 D= o you think maybe in part (a) of 3.8.3 we could just change:

=C2=A0=C2=A0=C2=A0 R1 and R2 might be measured in time units or as a= count of retransmissions.

to:

=C2=A0=C2=A0=C2=A0 R1 and R2 might be measured in time units or as a= count of retransmissions (with the current RTO as a conversion factor, if needed).

I like that, thanks.
= =C2=A0


(3.8.5)=C2=A0 This section is unclear as to whether the send= er can cause the Urgent Pointer to recede (move to the left), and if so, if that MUST be reported to the application like a pointer advance.

Receding looks the same as advancing so much that it wraps.=C2=A0 So= , if not already in urgent mode, that would seem to cause a change to urgent mode, while if not already in urgent mode, that would fall under the existing rule of:

=C2=A0=C2=A0=C2=A0 If the urgent pointer is updated while the user i= s in "urgent mode", the update will be invisible to the user.

So ... also given that the use of urgent pointer is deprecated anyways and we don't want to cause changes to TCP in this doc, I&= #39;m proposing we do nothing to clarify this, if you agree?

<= /blockquote>
I don't think

SEQ 5000 URG 30= 00
SEQ 5001 URG 1500

would have a common= -sense interpretation that the urg ptr was now 7501 + 2^32. I share your gr= umpiness about sweating the useless urgent pointer, but it would appear the= re is a gap in the spec and that in principle these recedes could exist in = the wild. This spec mostly matters for new implementations; maybe the safes= t thing is for those new implementations to prepare to handle a receding po= inter, but that senders MUST NOT recede. I would welcome a quick question t= o the WG if that logic seems spurious to you.

= Meanwhile, it appears we've uncovered a contradiction! In 3.8.5 we have= 

If the urgent pointer is updated while the user is in "urgent mod=
e", the
   update will be invisible to the user.
A TCP implementation MUST (MUST-32) inform the application layer
   asynchronously whenever it receives an Urgent pointer and there was
   previously no pending urgent data, or whenever the Urgent pointer
   advances in the data stream.
=C2=A0If the latter text is correct, AND we decide to allow rece= ding, we should decide what to do with recedes here.

(3.8.6.1)
The sending TCP peer must be prepared to accept from the use=
r and
   send at least one octet of new data even if the send window is zero.
   The sending TCP peer must regularly retransmit to the receiving TCP
   peer even when the window is zero
Please clarify that these probes only happen when there is new data, if that's the intent.

The "accept from the user" part means that new data was wr= itten already, right?=C2=A0 If the user doesn't write data into the soc= ket, there's nothing to be sent.

I have= been messing with QUIC too long and may have forgotten how this works. But= IIRC, this text kind of confuses a few different kinds of transmissions wi= th zero window probes. It might be helpful to actually define "zero wi= ndow probe" -- a segment of one byte of data designed to elicit an ack= with a window update. The retransmission discussion is related but somewha= t confusing as currently written. There are four zero-window scenarios that= I can see.

1) If there is outstanding in-wind= ow data, we will get ACKs so there is no need for a ZWP.

2) No send buffer data, but the application generates data while in = zero-window. Clearly, the spec says TCP accepts a byte and sends it.

3) TCP has previously refused application data due t= o window restrictions, and now we're in zero-window. TCP needs to wake = the app to send again so that we have material for ZWP.

4) TCP accepted some data, held it for Nagle, and then the receiver r= eneged; or all in-flight data is now out of window due to reneging. This da= ta ought to constitute a ZWP whether or not it is strictly a "retransm= ission".

If you believe all of my stateme= nts correct, I am happy to propose alternate text that is not as wordy as t= he above.,


(3.8.6.1) the first paragraph says
Two minutes is recommended for the retransmission interval w=
hen the
   window is zero.
but the last one says
The transmitting host SHOULD send the first zero-window prob=
e when a
   zero window has existed for the retransmission timeout period (SHLD-
   29) (Section 3.8.1), and SHOULD =
increase exponentially the interval
   between successive probes (SHLD-30).

so is it 2 min or the RTO?

Good question ... The first sentence come= s from 793 and the other from 1122.=C2=A0 I think we should erase the first sentence, since th= e 1122 intent seems to be updating this (replacing the hard-coded 2 minute heuristic with the RTO one which adapts to reality).=C2=A0 Do you agree?

Yes, I agree. 


(3.8.6.2.1) If I read this a=
lgorithm correctly, an un-pushed send buffer of < MSS, if D < Fs*Max(=
SND.WND), will sit there indefinitely without
being transmitted; is this intentional?

This comes from 1122.=C2=A0 I think = the key to preventing what you mention is this part of the preceding paragraph:

=C2=A0 To avoid a resulting deadlock= , it is necessary to have a
=C2=A0=C2=A0 timeout to force transmission of data, overriding the = SWS avoidance
=C2=A0=C2=A0 algorithm.

So, SWS avoidance is not supposed to keep a sender waiting forever for the receiver to possibly advertise a bigger window; the sender is supposed to wake up and just send eventually.


Yes, but the override timeout is embedded in step (4), which applie= s PUSH as a condition. Maybe that PUSH condition is incorrect?


(3.8.2) and (3.8.6.3) Should RFC 5681 be a normative reference, given the congestion control requirement? If not, (3.8.6.3) shouldn't require the reader to go there t= o fully implement delayed ACKs. I suggest

s/an ACK for at least every second segment/an ACK for least every 2*MSS bytes of data

to fully state what is in 5681.

Yeah, I think 5681 should be normative (since we also rely on the for descriptions of slow start, etc. that we didn't try to duplicate within 793bis), but we should also fix the "second segment" sentence to match more exactly the text in 5681 about 2*RMSS.


OK, there's= currently the pointer to 5681 about sub-MSS stuff, so it's technically= correct, but really clunky to force people to go there when it's so ea= sy to state correctly. I think we agree.


<snip>


(3.10, OPEN/LISTEN) 
If active and the remote socket is specified, then change th=
e
         connection from passive to active,
I think you mean "If passive"?

Actually, I think to be more clear this should say:

"If the OPEN call is active ...", since it's referring= to the case where an active OPEN is called when the socket is in LISTEN (passive).

It's technically correct currently (just not very clear), if you read the "If active" as referring to the OPEN call rather t= han the socket (and it must be that way since a socket in the LISTEN state is the result of a passive OPEN).


=
Yes, let's clarify
=C2=A0

(3.10, SEGMENT ARRIVES/LISTEN) Step 4 mentions RST but those have already been handled in Step 1.

Yeah, you're definitely right, and in both cases the action is t= o ignore it, so this part of step 4 is irrelevant, but at least consistent.

It looks like this is an errata to file on 793; should we do that quickly and then fix this doc accordingly?

=
If that is better for your process, then sure.
=C2=A0

NITS:
<snip>

(3.1, Options) " 
   Options: [TCP Option]; Options#Size =3D=3D (DOffset-5)*32=
; present only
   when DOffset > 5.
What is this notation?
It is a little odd, but comes from reference [61] for draft-mcquistin-augmented-ascii-diagrams-08.

OK


<snip>

I'm good with changing all of these above.

Thank you for the detailed review!

--00000000000072782705c6f41c8e-- From nobody Mon Jul 12 14:58:06 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 1E3BF3A1151; Mon, 12 Jul 2021 14:57:57 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: The IESG To: "IETF-Announce" X-Test-IDTracker: no X-IETF-IDTracker: 7.34.0 Auto-Submitted: auto-generated Precedence: bulk CC: Michael Scharf , draft-ietf-tcpm-rfc793bis@ietf.org, martin.h.duke@gmail.com, michael.scharf@hs-esslingen.de, tcpm-chairs@ietf.org, tcpm@ietf.org Reply-To: last-call@ietf.org Sender: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-ID: <162612707705.15995.3139474754068412997@ietfa.amsl.com> Date: Mon, 12 Jul 2021 14:57:57 -0700 Archived-At: Subject: [tcpm] Last Call: (Transmission Control Protocol (TCP) Specification) to Internet Standard X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 21:58:05 -0000 The IESG has received a request from the TCP Maintenance and Minor Extensions WG (tcpm) to consider the following document: - 'Transmission Control Protocol (TCP) Specification' as Internet Standard The IESG plans to make a decision in the next few weeks, and solicits final comments on this action. Please send substantive comments to the last-call@ietf.org mailing lists by 2021-08-02. Exceptionally, comments may be sent to iesg@ietf.org instead. In either case, please retain the beginning of the Subject line to allow automated sorting. Abstract This document specifies the Transmission Control Protocol (TCP). TCP is an important transport layer protocol in the Internet protocol stack, and has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFC 1122, and should be considered as a replacement for the portions of that document dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. RFC EDITOR NOTE: If approved for publication as an RFC, this should be marked additionally as "STD: 7" and replace RFC 793 in that role. The file can be obtained via https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc793bis/ No IPR declarations have been submitted directly on this I-D. The document contains these normative downward references. See RFC 3967 for additional information: rfc6633: Deprecation of ICMP Source Quench Messages (Proposed Standard - Internet Engineering Task Force (IETF)) rfc6298: Computing TCP's Retransmission Timer (Proposed Standard - Internet Engineering Task Force (IETF)) rfc5681: TCP Congestion Control (Draft Standard - Internet Engineering Task Force (IETF)) rfc3168: The Addition of Explicit Congestion Notification (ECN) to IP (Proposed Standard - Internet Engineering Task Force (IETF)) rfc2675: IPv6 Jumbograms (Proposed Standard - Internet Engineering Task Force (IETF)) rfc2474: Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers (Proposed Standard - Internet Engineering Task Force (IETF)) rfc1191: Path MTU discovery (Draft Standard - Legacy stream) From nobody Mon Jul 12 15:34:11 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1E8993A1264 for ; Mon, 12 Jul 2021 15:34:09 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Dw-_f6SVrSGM for ; Mon, 12 Jul 2021 15:34:04 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1D4523A137D for ; Mon, 12 Jul 2021 15:34:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:Cc:From:References:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=53pLgM5TPpAfxMK84ap3Fa8dUzXSEBF2WAvTQrnjn3k=; b=D926/sZM4jiMLEjDGwFwEcVmuU iTzkolCnegG3duCG5wp4+LWi6snuCImI1kX7sFV3UdTqvzBzxAbxeM29OF1JlnY33truHv3/O/lGN lAZT/NpN//V+9k9u06uXaEAP8aedLTQ+RIEj61nrjg7H9ukNqu3di4HyauSyPRTCXkOm66bx//fWV 0be1+9e6BE2vhgpO01cewaFi73Di8I709FsV3PGvCFFY7PVA0bLFJH9rYu9JI2pbK+gUmiqIj89x6 QoDgTwYU0h8bnskEpGvWUCUwIQYgc5oxulFZXz9D0+WnokQLFyRGihD6ylLcrPQKBigGiv8fip+UK TsRJwiOA==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:35968 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m34V1-0003FE-O0; Mon, 12 Jul 2021 23:34:00 +0100 To: tcpm@ietf.org References: <162612653463.2420.8417243673097691082@ietfa.amsl.com> From: Bob Briscoe Cc: Mirja Kuehlewind , Richard Scheffenegger , "Scheffenegger, Richard" Message-ID: Date: Mon, 12 Jul 2021 23:33:59 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <162612653463.2420.8417243673097691082@ietfa.amsl.com> Content-Type: multipart/alternative; boundary="------------FD8C327A58112FBD06DB2AA6" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-15.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Jul 2021 22:34:09 -0000 This is a multi-part message in MIME format. --------------FD8C327A58112FBD06DB2AA6 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit tcpm, Just posted draft-15. Here's a summary of the main changes wrt draft-14. Technical changes: * Major changes to "3.2.2.5.1. Data Receiver Safety Procedures": o the rules on Change-Triggered ACKs, Increment-Triggered ACKs, including discussion of ACKs of ACKs, o and clarification of what the rules mean when using receive offload o both as discussed at length on the list under the heading "Seeking WG opinions on ACKing ACKs with good cause" * Ignore AccECN TCP Option on a SYN if it does somehow arrive, even tho the sender shouldn't send it. * Major changes to "3.2.3.3.  Usage of the AccECN TCP Option": o recommended the most effective and simplest scheme to always send fields that have changed at some time during the connection on every packet that acks new data, and rearranged the other rules as variants of this o removed the rule to emit an ACK if any byte counter changes (AFAIR, I thought I had removed this when we made the option optional, but obviously not) o using experience from Ilpo's recent implementation and testing (and actually mostly making the text as we had intended it to be, but didn't realize we had left old stuff in there) * Updates to "3.3.2.  Requirements for Transparent Middleboxes and TCP Normalizers" as discussed with Gorry on list some time ago. o Mostly editorial rather than technical changes - made a clear distinction between what's expected from transparent middleboxes and from normalizers Editorial changes: * Clearer distinction between ECN fields in IP and in TCP, particularly in the explanations in the early parts of the draft. * Replaced the ToDo in Security Considerations with a summary of the covert channel discussion o and added a sentencethat should have been saying sender misbehaviour is out of scope (following SECDIR review) * Took out remnants of references to experiments. * Minor updates throughout HTH Bob and co-authors. On 12/07/2021 22:48, internet-drafts@ietf.org wrote: > A New Internet-Draft is available from the on-line Internet-Drafts directories. > This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. > > Title : More Accurate ECN Feedback in TCP > Authors : Bob Briscoe > Mirja Kühlewind > Richard Scheffenegger > Filename : draft-ietf-tcpm-accurate-ecn-15.txt > Pages : 59 > Date : 2021-07-12 > > Abstract: > Explicit Congestion Notification (ECN) is a mechanism where network > nodes can mark IP packets instead of dropping them to indicate > incipient congestion to the end-points. Receivers with an ECN- > capable transport protocol feed back this information to the sender. > ECN was originally specified for TCP in such a way that only one > feedback signal can be transmitted per Round-Trip Time (RTT). Recent > new TCP mechanisms like Congestion Exposure (ConEx), Data Center TCP > (DCTCP) or Low Latency Low Loss Scalable Throughput (L4S) need more > accurate ECN feedback information whenever more than one marking is > received in one RTT. This document specifies a scheme to provide > more than one feedback signal per RTT in the TCP header. Given TCP > header space is scarce, it allocates a reserved header bit previously > assigned to the ECN-Nonce. It also overloads the two existing ECN > flags in the TCP header. The resulting extra space is exploited to > feed back the IP-ECN field received during the 3-way handshake as > well. Supplementary feedback information can optionally be provided > in a new TCP option, which is never used on the TCP SYN. The > document also specifies the treatment of this updated TCP wire > protocol by middleboxes. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ > > There is also an htmlized version available at: > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-15 > > A diff from the previous version is available at: > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-15 > > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------FD8C327A58112FBD06DB2AA6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit tcpm,

Just posted draft-15. Here's a summary of the main changes wrt draft-14.

Technical changes:
  • Major changes to "3.2.2.5.1. Data Receiver Safety Procedures":
    • the rules on Change-Triggered ACKs, Increment-Triggered ACKs, including discussion of ACKs of ACKs,
    • and clarification of what the rules mean when using receive offload
    • both as discussed at length on the list under the heading "Seeking WG opinions on ACKing ACKs with good cause"
  • Ignore AccECN TCP Option on a SYN if it does somehow arrive, even tho the sender shouldn't send it.
  • Major changes to "3.2.3.3.  Usage of the AccECN TCP Option":
    • recommended the most effective and simplest scheme to always send fields that have changed at some time during the connection on every packet that acks new data, and rearranged the other rules as variants of this
    • removed the rule to emit an ACK if any byte counter changes (AFAIR, I thought I had removed this when we made the option optional, but obviously not)
    • using experience from Ilpo's recent implementation and testing (and actually mostly making the text as we had intended it to be, but didn't realize we had left old stuff in there)
  • Updates to "3.3.2.  Requirements for Transparent Middleboxes and TCP Normalizers" as discussed with Gorry on list some time ago.
    • Mostly editorial rather than technical changes - made a clear distinction between what's expected from transparent middleboxes and from normalizers

Editorial changes:
  • Clearer distinction between ECN fields in IP and in TCP, particularly in the explanations in the early parts of the draft.
  • Replaced the ToDo in Security Considerations with a summary of the covert channel discussion
    • and added a sentencethat should have been saying sender misbehaviour is out of scope (following SECDIR review)
  • Took out remnants of references to experiments.
  • Minor updates throughout

HTH


Bob and co-authors.


On 12/07/2021 22:48, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : More Accurate ECN Feedback in TCP
        Authors         : Bob Briscoe
                          Mirja Kühlewind
                          Richard Scheffenegger
	Filename        : draft-ietf-tcpm-accurate-ecn-15.txt
	Pages           : 59
	Date            : 2021-07-12

Abstract:
   Explicit Congestion Notification (ECN) is a mechanism where network
   nodes can mark IP packets instead of dropping them to indicate
   incipient congestion to the end-points.  Receivers with an ECN-
   capable transport protocol feed back this information to the sender.
   ECN was originally specified for TCP in such a way that only one
   feedback signal can be transmitted per Round-Trip Time (RTT).  Recent
   new TCP mechanisms like Congestion Exposure (ConEx), Data Center TCP
   (DCTCP) or Low Latency Low Loss Scalable Throughput (L4S) need more
   accurate ECN feedback information whenever more than one marking is
   received in one RTT.  This document specifies a scheme to provide
   more than one feedback signal per RTT in the TCP header.  Given TCP
   header space is scarce, it allocates a reserved header bit previously
   assigned to the ECN-Nonce.  It also overloads the two existing ECN
   flags in the TCP header.  The resulting extra space is exploited to
   feed back the IP-ECN field received during the 3-way handshake as
   well.  Supplementary feedback information can optionally be provided
   in a new TCP option, which is never used on the TCP SYN.  The
   document also specifies the treatment of this updated TCP wire
   protocol by middleboxes.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-15

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-15


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------FD8C327A58112FBD06DB2AA6-- From nobody Wed Jul 14 02:15:02 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 926563A0045 for ; Wed, 14 Jul 2021 02:15:00 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.233 X-Spam-Level: X-Spam-Status: No, score=-1.233 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_SOFTFAIL=0.665, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id heiHi1pEJhd0 for ; Wed, 14 Jul 2021 02:14:56 -0700 (PDT) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CDBAC3A003E for ; Wed, 14 Jul 2021 02:14:55 -0700 (PDT) Received: from smtpclient.apple (ip1f100e9c.dynamic.kabel-deutschland.de [31.16.14.156]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id CACED721E2825 for ; Wed, 14 Jul 2021 11:14:46 +0200 (CEST) From: tuexen@fh-muenster.de Content-Type: multipart/signed; boundary="Apple-Mail=_F763969B-240F-4D14-8263-80D3DC305669"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Message-Id: Date: Wed, 14 Jul 2021 11:14:45 +0200 To: tcpm IETF list X-Mailer: Apple Mail (2.3654.100.0.2.22) Archived-At: Subject: [tcpm] Draft Agenda online X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jul 2021 09:15:01 -0000 --Apple-Mail=_F763969B-240F-4D14-8263-80D3DC305669 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Dear all, a draft agenda of the upcoming TCPM meeting is online: https://datatracker.ietf.org/meeting/111/materials/agenda-111-tcpm-00 Best regards Michael --Apple-Mail=_F763969B-240F-4D14-8263-80D3DC305669 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEKow ggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYTAkRFMSsw KQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYDVQQLDBZULVN5 c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFsUm9vdCBDbGFzcyAy MB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNVBAYTAkRFMUUwQwYDVQQK EzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMg ZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlv biBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMtg1/9moUHN0vqH l4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZsFVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8F XRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0peQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+Ba L2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qL NupOkSk9s1FcragMvp0049ENF4N1xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz 9AkH4wKGMUZrAcUQDBHHWekCAwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU k+PYMiba1fFKpZFK4OpL4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYD VR0TAQH/BAgwBgEB/wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGC LB4wCAYGZ4EMAQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUv cmwvVGVsZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYB BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5jZXIw DQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4eTizDnS6 dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/MOaZ/SLick0+ hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3SPXez7vTXTf/D6OWS T1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc22CzeIs2LgtjZeOJVEqM7 h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bPZYoaorVyGTkwggWsMIIElKAD AgECAgcbY7rQHiw9MA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJERTFFMEMGA1UEChM8VmVy ZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYu MRAwDgYDVQQLEwdERk4tUEtJMS0wKwYDVQQDEyRERk4tVmVyZWluIENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IDIwHhcNMTYwNTI0MTEzODQwWhcNMzEwMjIyMjM1OTU5WjCBjTELMAkGA1UEBhMCREUx RTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5lcyBEZXV0c2NoZW4gRm9yc2NodW5n c25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMGA1UEAwwcREZOLVZlcmVpbiBHbG9i YWwgSXNzdWluZyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ07eRxH3h+Gy8Zp 1xCeOdfZojDbchwFfylfS2jxrRnWTOFrG7ELf6Gr4HuLi9gtzm6IOhDuV+UefwRRNuu6cG1joL6W LkDh0YNMZj0cZGnlm6Stcq5oOVGHecwX064vXWNxSzl660Knl5BpBb+Q/6RAcL0D57+eGIgfn5mI TQ5HjUhfZZkQ0tkqSe3BuS0dnxLLFdM/fx5ULzquk1enfnjK1UriGuXtQX1TX8izKvWKMKztFwUk P7agCwf9TRqaA1KgNpzeJIdl5Of6x5ZzJBTN0OgbaJ4YWa52fvfRCng8h0uwN89Tyjo4EPPLR22M ZD08WkVKusqAfLjz56dMTM0CAwEAAaOCAgUwggIBMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0P AQH/BAQDAgEGMCkGA1UdIAQiMCAwDQYLKwYBBAGBrSGCLB4wDwYNKwYBBAGBrSGCLAEBBDAdBgNV HQ4EFgQUazqYi/nyU4na4K2yMh4JH+iqO3QwHwYDVR0jBBgwFoAUk+PYMiba1fFKpZFK4OpL4qIM z+EwgY8GA1UdHwSBhzCBhDBAoD6gPIY6aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9v dC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDBAoD6gPIY6aHR0cDovL2NkcDIucGNhLmRmbi5kZS9n bG9iYWwtcm9vdC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDCB3QYIKwYBBQUHAQEEgdAwgc0wMwYI KwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBKBggrBgEF BQcwAoY+aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1jYS9wdWIvY2FjZXJ0 L2NhY2VydC5jcnQwSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFsLXJv b3QtZzItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCBeEWkTqR/ DlXwCbFqPnjMaDWpHPOVnj/z+N9rOHeJLI21rT7H8pTNoAauusyosa0zCLYkhmI2THhuUPDVbmCN T1IxQ5dGdfBi5G5mUcFCMWdQ5UnnOR7Ln8qGSN4IFP8VSytmm6A4nwDO/afr0X9XLchMX9wQEZc+ lgQCXISoKTlslPwQkgZ7nu7YRrQbtQMMONncsKk/cQYLsgMHM8KNSGMlJTx6e1du94oFOO+4oK4v 9NsH1VuEGMGpuEvObJAaguS5Pfp38dIfMwK/U+d2+dwmJUFvL6Yb+qQTkPp8ftkLYF3sv8pBoGH7 EUkp2KgtdRXYShjqFu9VNCIaE40GMIIF4DCCBMigAwIBAgIMIRX9tDE2QqO3mVLXMA0GCSqGSIb3 DQEBCwUAMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1ciBGb2VyZGVydW5nIGVp bmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYDVQQLDAdERk4tUEtJMSUw IwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBMB4XDTE5MDYwNDE0MjkxMFoXDTIy MDYwMzE0MjkxMFowfDELMAkGA1UEBhMCREUxIDAeBgNVBAoMF0ZhY2hob2Noc2NodWxlIE11ZW5z dGVyMTIwMAYDVQQLDClGYWNoYmVyZWljaCBFbGVrdHJvdGVjaG5payB1bmQgSW5mb3JtYXRpazEX MBUGA1UEAwwOTWljaGFlbCBUdWV4ZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM r8qQcPxLFCxzPtXvRyM9KeQaxyMA8gwUNc7IIiATs6mRQFe5ib/mvwT9nc0bAa+94go6HJDiD3FJ NkTo4u8aBsIcTt5pJtdBQLm88PLakbe3+fp/00//n7xxbTh7mAtFVCf25LxDCKkrdGk/+jglRq/R VdwhZZ3VpYCrx8YfI/hIzdRL3+4I4z/mnQ8K0X8d2MVVPG+9nBEngdnYGez5f/8wIVOgEYYBc21k yvMnVXLu2Ing+LwBd0gDG9Vasv87MNHCOZfJTNBlLhI2UDei/uNg9Zd4ynlMpPWZ7v0oiDWvmv8E OuD4oric8heyD0OYK2FL4qcVC4dn4pnyulfHAgMBAAGjggJOMIICSjA+BgNVHSAENzA1MA8GDSsG AQQBga0hgiwBAQQwEAYOKwYBBAGBrSGCLAEBBAQwEAYOKwYBBAGBrSGCLAIBBAQwCQYDVR0TBAIw ADAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQW BBTxiodBVL/lA4p5iNesIsJRhhgd6zAfBgNVHSMEGDAWgBRrOpiL+fJTidrgrbIyHgkf6Ko7dDAg BgNVHREEGTAXgRV0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0 cDovL2NkcDEucGNhLmRmbi5kZS9kZm4tY2EtZ2xvYmFsLWcyL3B1Yi9jcmwvY2FjcmwuY3JsMD+g PaA7hjlodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2Rmbi1jYS1nbG9iYWwtZzIvcHViL2NybC9jYWNy bC5jcmwwgdsGCCsGAQUFBwEBBIHOMIHLMDMGCCsGAQUFBzABhidodHRwOi8vb2NzcC5wY2EuZGZu LmRlL09DU1AtU2VydmVyL09DU1AwSQYIKwYBBQUHMAKGPWh0dHA6Ly9jZHAxLnBjYS5kZm4uZGUv ZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwSQYIKwYBBQUHMAKGPWh0dHA6 Ly9jZHAyLnBjYS5kZm4uZGUvZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQw DQYJKoZIhvcNAQELBQADggEBABs3VlmIZ1VF3HkaQdjInZYmamRabbdgJ7cz6m6o/agKL7+Vhwx7 1BaaYs2gMa5Nu/GJ3YfdqIsUlYtKdl58Yhp/89E6xBfJkItS+rE8RFdf2XgklGlx7GmsvdA3tId5 b6K6r9a5wpVN0epxY6K8wwpzEib6fJLcHylybQXZ7JSOaLRLp6WU3QPoyTT7FpvAe/0b2MSCbPX4 fc53PCn2aGgXuRFVQeCn1SP1BF3QW1ppkFhcF6G+0JcUxYFAXE6r/71WZBvUHqoG/th2hAwQnS+Y KhUYh4SZQH3/ursXXJYXQ5vyIhkN1FZlmtWA8+ofdCnoqSTbiSX2Aa/kKbTapwgxggOdMIIDmQIB ATCBnjCBjTELMAkGA1UEBhMCREUxRTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5l cyBEZXV0c2NoZW4gRm9yc2NodW5nc25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMG A1UEAwwcREZOLVZlcmVpbiBHbG9iYWwgSXNzdWluZyBDQQIMIRX9tDE2QqO3mVLXMA0GCWCGSAFl AwQCAQUAoIIBzzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMTA3 MTQwOTE0NDVaMC8GCSqGSIb3DQEJBDEiBCAH1IvR+D8LWA+BOZQ7VIkI0MWWxaqmTsg3WD9enBIG lTCBrwYJKwYBBAGCNxAEMYGhMIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1 ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYD VQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBAgwhFf20 MTZCo7eZUtcwgbEGCyqGSIb3DQEJEAILMYGhoIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8 VmVyZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUu IFYuMRAwDgYDVQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5n IENBAgwhFf20MTZCo7eZUtcwDQYJKoZIhvcNAQELBQAEggEAvPjsffuysnJSry+5DEyArIRcGZvG qvS1Mi2dJ1/6aqzossnFeInwzGkdoeh0vqaCPvO2OSpKeN7/OJ25yL88ezdeqipZqL0/T3kVexS7 US4bqtgaJh6OybOWpA9/0OC3SPdtytq+B7XTst9F25LtYoGrgx6ps7kqv+drPNl36wjH/tyY8deZ TcMw4qTUrmN+QbkWWLuh18Nnhv6uiIMQ80yj4wAIaMwpGcKdaZuXcQJOx/diE25h2HHOLxG3dPlx rd4XYqebcQbe8UicgzsavYM2zATdulXOLIuPR2NobYGm/6mOS/p0ZhkOnc0o36Rm64YjiwGlvr1u NnsGT/M44AAAAAAAAA== --Apple-Mail=_F763969B-240F-4D14-8263-80D3DC305669-- From nobody Thu Jul 15 12:15:59 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4B0B73A11B0 for ; Thu, 15 Jul 2021 12:15:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xu_v4swCXs40 for ; Thu, 15 Jul 2021 12:15:52 -0700 (PDT) Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8F5633A11AE for ; Thu, 15 Jul 2021 12:15:52 -0700 (PDT) Received: by mail-il1-x130.google.com with SMTP id w1so5975378ilg.10 for ; Thu, 15 Jul 2021 12:15:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PDIeuPViz30AuEPfBY3SFqhDgUm3+JPvxhpKG3iWwuA=; b=IvK1AKGIupbQe/ulav9CCYe9RFaNwH0KstnX9jXPK8ykOkFTjmIXoftaWgB3aTg+RG wwy2OEDY6EROPydUpA2eUzAc4pMgXzXiKVotnNNBK9PJDF04wG9bwl8s2uPzo8w+yc6R Z3QkHDgqKDIfiYDRm4e3Qy6k/V/Ry65C6wwdbYC1eOpfuTXHN4HvmtY+SfWgX5/Lm1WC wYPLiVPW4pW7unQlaFVRC6lNtrDEIH3E7tpyBA4dVOTRLmYnvFJgNf3eIRNJz0p+XtrP EpkU9G3oS8HTKcbuvAw1CdlXSv/eE5vMxFeHBhZ2gZAPst+uH33Dt3IBXJPxBzvvIzQN Sssg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PDIeuPViz30AuEPfBY3SFqhDgUm3+JPvxhpKG3iWwuA=; b=dkk2c6+Xmj4WesFXM6YEJGQH7gvGIlz+K//jpNICe2Xdp61dzdEGiaMItggQzuzF3b oUzrcHzF8xuRPXErwHTNYhj2GmCHq6jD1JW0pCvdsgGWhhjEZh18nQGW9xRL8IVM/u43 kcOwD0z9zfkd08MHHlA7gsiEyS8fzwuxZniG9kJmhY249vw0b/AV37afYQWXsI1TtKkP 0hbBeeYJvasY3q/lb4jpeiUSM5+U0PukjZwgMvWqaU6O4NppcMxvm5gmMWqtiuTaSy0d Wu98APLmkehkKOJIW0xX+IBm7ZEcNJpvUjrZSag9dQwOFimEHvWWX452742CFInWsbK4 f0uw== X-Gm-Message-State: AOAM530wE30p86c2fn8g4viaiOgYAbdhxUFp+T3po6Z77/9NVFfonycf jEtnKGkAO93RQ0fIQxDGsZvlKNU94btoAqdCsRw= X-Google-Smtp-Source: ABdhPJxCKWSAH3w0ad4T/2AG8EbKjrbFemKIQIuZ6rhMTMgGbkLRPhqNWGHVm2hmG5m5bxMpd5sokdCrwuDvsTHYBA8= X-Received: by 2002:a05:6e02:20eb:: with SMTP id q11mr3781888ilv.272.1626376551185; Thu, 15 Jul 2021 12:15:51 -0700 (PDT) MIME-Version: 1.0 References: <162611569026.7615.3785325543750944369@ietfa.amsl.com> <9f310fe4-1e50-4a94-5ac2-c3eeac4feba6@mti-systems.com> In-Reply-To: <9f310fe4-1e50-4a94-5ac2-c3eeac4feba6@mti-systems.com> From: Martin Duke Date: Thu, 15 Jul 2021 12:15:40 -0700 Message-ID: To: Wesley Eddy Cc: "tcpm@ietf.org Extensions" Content-Type: multipart/alternative; boundary="0000000000009ac2d105c72e4c3f" Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-rfc793bis-24.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jul 2021 19:15:58 -0000 --0000000000009ac2d105c72e4c3f Content-Type: text/plain; charset="UTF-8" On Mon, Jul 12, 2021 at 11:57 AM Wesley Eddy wrote: > I'm not sure if this completely satisfies Martin's comment, but for now, I > added this sentence: > > Note that because changes in the urgent pointer correspond > to data being written by a sending application, the urgent pointer > can not "recede" in the sequence space, but a TCP receiver should be > robust to invalid urgent pointer values. > > I am completely satisfied with this text. IMO 793 does not explicitly prohibit receding, but it is a bad idea so we should discourage it, while being robust to receiving it given an *extremely* small chance that some implementation somewhere is doing it. --0000000000009ac2d105c72e4c3f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Mon, Jul 12, 2021 at 11:57 AM Wesl= ey Eddy <wes@mti-systems.com&= gt; wrote:
=20 =20 =20

I'm not sure if this completely satisfies Martin's comme= nt, but for now, I added this sentence:

  Note that because changes=
 in the urgent pointer correspond
   to data being written by a sending application, the urgent pointer
   can not "recede" in the sequence space, but a TCP receiver sho=
uld be
   robust to invalid urgent pointer values.

I am completely satisfied with= this text. IMO 793 does not explicitly prohibit receding, but it is a bad = idea so we should discourage it, while being robust to receiving it given a= n *extremely* small chance that some implementation somewhere is doing it.<= br>
--0000000000009ac2d105c72e4c3f-- From nobody Thu Jul 22 08:42:34 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A8E373A4AC0; Thu, 22 Jul 2021 08:42:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.087 X-Spam-Level: X-Spam-Status: No, score=-2.087 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TsHV8yQIwyr6; Thu, 22 Jul 2021 08:42:25 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8B13B3A4ABC; Thu, 22 Jul 2021 08:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:Cc:From:References:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=z6wFSqRNkrJI1VA2dcsLzmG1AAWUim97yZdLjM+tEvY=; b=gqiO5TR1kzOyugPhUmuoOcJ0N3 skBjVSZylFV4OWcUUHFreQUjI1RQtPh/Z381fY+kVmbRp2xzPtBmam1gCCJjPWCwKUWq5yJk5B2Fj ovQX4rASqcHdcjgpXpSkp1seDTH5TvtMR6O3gV5QDABuTVRsn8GxNZS3ZhTbtwKb3WfAAjPr+cz9X wWVGukt76taYIsR4rX7p2cekhGc8oNy4+Rn9cVW3wgZZV1cnyI/pbCkBO3GoeVHtr15SPNrB3VDlX r18q4SvHVA3OY8Nsb7au3I7kNVSrjMe8nmwRaujj1n8G2V1JPoi14jSiTmWD6kTd7BO5culqnexel MttHBplg==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:48760 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m6aqB-0002K6-A4; Thu, 22 Jul 2021 16:42:20 +0100 To: Praveen Balasubramanian References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> From: Bob Briscoe Cc: tcpm@ietf.org, draft-ietf-tcpm-hystartplusplus@ietf.org Message-ID: <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> Date: Thu, 22 Jul 2021 16:42:18 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <162610476442.30543.4667406094304409800@ietfa.amsl.com> Content-Type: multipart/alternative; boundary="------------674DDBBD62C5B2BDB82A2B6D" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: [tcpm] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jul 2021 15:42:32 -0000 This is a multi-part message in MIME format. --------------674DDBBD62C5B2BDB82A2B6D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Praveen and co-authors. Thank you for hystart++. This one's my technical review. I've sent my editorial review in a separate posting. I actually reviewed an earlier draft (balasubrmanian-02), but never got round to writing it up for you. I've just reviewed the latest (ietf-02) draft, and incorporated my comments about the older revision where still relevant. Cheers Bob ===Technical Comments== 4.2.  Algorithm Details ------------------------------------------------------------------------  Round can be approximated using sequence numbers as follows: ... At the start of each round during normal slow start and CSS: It might be worth stating that using sequence numbers is the /best/ way to measure rounds, not just one way. You want every round to contain at least one of the ping-ponged segments that were grandfathered by the very first segment of the IW (because that will be the segment most likely to have the min RTT). For instance, if you measure rounds in time, you might underestimate the duration of a round. Then, in some rounds, you would risk not seeing one of this ping-pong series at all. ------------------------------------------------------------------------ "where currRTT is the RTT sampled from the incoming ACK" Need to make it clear this RTT is based on the latest ACK'd packet, not the earliest (in contrast to the RTT used for RTO). ------------------------------------------------------------------------ The somewhat opaque clamping of RTTThresh below needs to be justified: For rounds where cwnd is at or higher than LOW_CWND and N_RTT_SAMPLE RTT samples have been obtained, check if delay increase triggers slow start exit ... RttThresh = clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RTT_THRESH) if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh)) cssBaselineMinRtt = currentRoundMinRTT exit slow start and enter CSS To aid comprehension for the benefit of the list, I'll separate the clamp line into two steps and substitute the default constants:             RttThresh = clamp(32ms, lastRoundMinRTT, 128ms)             RttThresh /= 8 To me, it makes good sense without the clamping. In english, that would be just:     "if queuing due to slow start increases the min RTT by more than 12.5% from one round to the next, exit slow start" However, taking the default values of the constants, the clamping expression adds the following conditions: #1) If the base RTT is lower than 32ms, the queue has to be greater than 4ms to exit slow start. #2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms to exit slow start. I.e. at both extremes, the threshold becomes absolute, rather than a fraction of the base RTT. Possible rationales for the lower condition when the base RTT is small (#1): a) an absolute Qdelay threshold is needed because small delay measurements are too noisy. b) a relative amount of Qdelay would become smaller than humans can perceive, so an absolute floor is sufficient. If (a), this needs to be stated, because different min delay measurement techniques can achieve different noise immunity. I don't agree with (b), because unnecessary delay over a sequence of rounds can accumulate to become perceivable. I can't think of a good rationale for the higher condition (#2). I'm sure it was the result of experiments, but it seems to be likely that as base RTT increases beyond 128ms, Hystart++ would become increasingly sensitive and exit slow start earlier and earlier - unnecessarily (?). Whatever, the rationale for clamping RttThresh needs to be stated, so admins know how to set the parameters for other scenarios (for instance, in DCs excessive loss would occur with the given defaults). ------------------------------------------------------------------------ 5.  Deployments and Performance Evaluations It would be useful to see testbed results separated by whether the bottleneck before the flow arrives is empty, full, or part-full of traffic. Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry). ------------------------------------------------------------------------ Regards Bob On 12/07/2021 16:46, internet-drafts@ietf.org wrote: > A New Internet-Draft is available from the on-line Internet-Drafts directories. > This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. > > Title : HyStart++: Modified Slow Start for TCP > Authors : Praveen Balasubramanian > Yi Huang > Matt Olson > Filename : draft-ietf-tcpm-hystartplusplus-02.txt > Pages : 8 > Date : 2021-07-12 > > Abstract: > This doument describes HyStart++, a simple modification to the slow > start phase of TCP congestion control algorithms. Traditional slow > start can cause overshotting of the ideal send rate and cause large > packet loss within a round-trip time which results in poor > performance. HyStart++ is composed of the delay increase variant of > HyStart to prevent overshooting of the ideal sending rate, while also > mitigating poor performance which can result from false positives. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ > > There is also an htmlized version available at: > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 > > A diff from the previous version is available at: > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02 > > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------674DDBBD62C5B2BDB82A2B6D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Praveen and co-authors.

Thank you for hystart++.
This one's my technical review. I've sent my editorial review in a separate posting.

I actually reviewed an earlier draft (balasubrmanian-02), but never got round to writing it up for you. I've just reviewed the latest (ietf-02) draft, and incorporated my comments about the older revision where still relevant.

Cheers


Bob


===Technical Comments==


4.2.  Algorithm Details

 Round can be approximated using sequence numbers as follows:
   ...
   At the start of each round during normal slow start and CSS:
It might be worth stating that using sequence numbers is the /best/ way to measure rounds, not just one way. You want every round to contain at least one of the ping-ponged segments that were grandfathered by the very first segment of the IW (because that will be the segment most likely to have the min RTT). For instance, if you measure rounds in time, you might underestimate the duration of a round. Then, in some rounds, you would risk not seeing one of this ping-pong series at all.


"where currRTT is the RTT sampled from the incoming ACK"
Need to make it clear this RTT is based on the latest ACK'd packet, not the earliest (in contrast to the RTT used for RTO).


The somewhat opaque clamping of RTTThresh below needs to be justified:

      For rounds where cwnd is at or higher than LOW_CWND and
      N_RTT_SAMPLE RTT samples have been obtained, check if delay
      increase triggers slow start exit
      ...
            RttThresh = clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RTT_THRESH)
            if (currentRoundMinRTT >= (lastRoundMinRTT + RttThresh))
               cssBaselineMinRtt = currentRoundMinRTT
               exit slow start and enter CSS

To aid comprehension for the benefit of the list, I'll separate the clamp line into two steps and substitute the default constants:
            RttThresh = clamp(32ms, lastRoundMinRTT, 128ms)
            RttThresh /= 8

To me, it makes good sense without the clamping. In english, that would be just:
    "if queuing due to slow start increases the min RTT by more than 12.5% from one round to the next, exit slow start"

However, taking the default values of the constants, the clamping expression adds the following conditions:
#1) If the base RTT is lower than 32ms, the queue has to be greater than 4ms to exit slow start.
#2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms to exit slow start.
I.e. at both extremes, the threshold becomes absolute, rather than a fraction of the base RTT.

Possible rationales for the lower condition when the base RTT is small (#1):
a) an absolute Qdelay threshold is needed because small delay measurements are too noisy.
b) a relative amount of Qdelay would become smaller than humans can perceive, so an absolute floor is sufficient.
If (a), this needs to be stated, because different min delay measurement techniques can achieve different noise immunity.
I don't agree with (b), because unnecessary delay over a sequence of rounds can accumulate to become perceivable.

I can't think of a good rationale for the higher condition (#2). I'm sure it was the result of experiments, but it seems to be likely that as base RTT increases beyond 128ms, Hystart++ would become increasingly sensitive and exit slow start earlier and earlier - unnecessarily (?).

Whatever, the rationale for clamping RttThresh needs to be stated, so admins know how to set the parameters for other scenarios (for instance, in DCs excessive loss would occur with the given defaults).



5.  Deployments and Performance Evaluations

It would be useful to see testbed results separated by whether the bottleneck before the flow arrives is empty, full, or part-full of traffic.

Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry).



Regards


Bob


On 12/07/2021 16:46, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : HyStart++: Modified Slow Start for TCP
        Authors         : Praveen Balasubramanian
                          Yi Huang
                          Matt Olson
	Filename        : draft-ietf-tcpm-hystartplusplus-02.txt
	Pages           : 8
	Date            : 2021-07-12

Abstract:
   This doument describes HyStart++, a simple modification to the slow
   start phase of TCP congestion control algorithms.  Traditional slow
   start can cause overshotting of the ideal send rate and cause large
   packet loss within a round-trip time which results in poor
   performance.  HyStart++ is composed of the delay increase variant of
   HyStart to prevent overshooting of the ideal sending rate, while also
   mitigating poor performance which can result from false positives.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------674DDBBD62C5B2BDB82A2B6D-- From nobody Thu Jul 22 08:43:22 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3D5C63A4AC9; Thu, 22 Jul 2021 08:43:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S_kqSBbApNkc; Thu, 22 Jul 2021 08:43:14 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 063053A4AC6; Thu, 22 Jul 2021 08:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:Cc:References:To:Subject:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=NrKstjJUVTxCBUu/swiC9P2nTi9Bk6yiGSWdjaoNbX8=; b=McCJDzkXZBv2ni6e584ma2D6Qy BzgsbePBKeb8THgoinwuPUKhMlqY0APNGLEj61BVljxV3bO+++b7/427955cLbj+uzvRV3Hp/8XeG uNCdpqBu5OxioB1bEeXz7q+PIJCzEXG35HZ8e4ewkJYaTTviAX4dti4oSVB8qmHrb7cOLCXD2/xwM 3lX/dXrLeY8kaaY3nJwtil/r9+brr8Tnn9azqhxBLwuqH88zMp0oIe9AKx+9pfDHUH666oIxa81Ei E+SMKnXgEOziuBEW66U1ihr0kcYXnBczmo1sk3ehzLsCZz9IqISm5AuIzRgJ+6wJ10tjLIsttHTKO swhh9SmQ==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:48764 helo=[192.168.1.10]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m6aqw-0002vl-94; Thu, 22 Jul 2021 16:43:07 +0100 From: Bob Briscoe To: Praveen Balasubramanian References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> Cc: tcpm@ietf.org, draft-ietf-tcpm-hystartplusplus@ietf.org Message-ID: <81092896-921b-9563-42b5-e9f412d5782d@bobbriscoe.net> Date: Thu, 22 Jul 2021 16:43:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <162610476442.30543.4667406094304409800@ietfa.amsl.com> Content-Type: multipart/alternative; boundary="------------15980C87736543ED7F9D9E1D" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: [tcpm] Full review of: draft-ietf-tcpm-hystartplusplus-02 (editorial) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jul 2021 15:43:20 -0000 This is a multi-part message in MIME format. --------------15980C87736543ED7F9D9E1D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Praveen and co-authors. Thank you for hystart++. I've sent a separate posting with technical review comments. Cheers Bob ==Editorial comments== Abstract s/doument/document/ s/overshotting/overshooting 1. Intro s/pefrormance/performance/ "...lead to overshooting the ideal sending rate and cause significant packet loss" Also queuing delay? Is reducing queuing delay (for other flows) also a goal of Hystart++? Last para of intro seems to be about Hystart, and it doesn't make it clear how Hystart relates to Hystart++: * "HyStart++ adds delay increase..."    Adds to what? * "After the HyStart delay algorithm finds an exit point, a Conservative Slow Start (CSS) phase is used"   But what about Hystart++? does it do CSS too? Final sentence is not related to the rest of the para about Hystart, so should be a separate para. 3. Definitions It seems odd to define some well-known variables (like cwnd), but not others like SND.UNA, SND.NXT, etc. cwnd: Would be useful to define the units assumed in the pseudocode later (bytes), because some implementations use segments. It's confusing for cwnd to use bytes but LOW_CWND to use segments. 4.2 Algorithm Details     "The choice of value of L is up to the implementation." I suggest:     "The choice of value of L is discussed in RFC 3465." Reason: RFC3465 doesn't say it's completely up to the implementation - it gives recommendations and advice. "normal slow start" I appreciate you were asked to use this word, but... I suggest it's best to define this phrase because, for instance, the normal slow start for Cubic (Hystart) is different from the normal slow start for Reno. It's also a problem finding a good word that means the pre-existing thing. 'Reno slow start' would be more precise than 'normal slow start' (if you mean the one in Reno). Because Cubic is more normal than Reno these days. And saying 'standard' will soon be ambiguous, because Cubic is on its way onto the standards track.     "If congestion is observed anytime during slow start or CSS, enter congestion avoidance." The word 'congestion' is ambiguous, because it could include delay increase. How about:     "If loss or ECN-marking is observed anytime during slow start or CSS, enter congestion avoidance." 5.  Deployments and Performance Evaluations  As of the time of writing, HyStart++ has been default enabled for all TCP connections in Windows for two years. The original Hystart has been default-enabled for all TCP connections in Linux TCP for a decade. I suggest you give the actual deployment times, so these two sentences don't keep having to be updated as the draft ages. Also, in Linux, Hystart is only implemented for Cubic (the default CC module), so Hystart is not default if a non-default CC module is in use. How about: In Linux, the original Hystart has been default enabled for TCP connections using the default congestion control module (Cubic) for a decade. Regards Bob On 12/07/2021 16:46, internet-drafts@ietf.org wrote: > A New Internet-Draft is available from the on-line Internet-Drafts directories. > This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. > > Title : HyStart++: Modified Slow Start for TCP > Authors : Praveen Balasubramanian > Yi Huang > Matt Olson > Filename : draft-ietf-tcpm-hystartplusplus-02.txt > Pages : 8 > Date : 2021-07-12 > > Abstract: > This doument describes HyStart++, a simple modification to the slow > start phase of TCP congestion control algorithms. Traditional slow > start can cause overshotting of the ideal send rate and cause large > packet loss within a round-trip time which results in poor > performance. HyStart++ is composed of the delay increase variant of > HyStart to prevent overshooting of the ideal sending rate, while also > mitigating poor performance which can result from false positives. > > > The IETF datatracker status page for this draft is: > https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ > > There is also an htmlized version available at: > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 > > A diff from the previous version is available at: > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02 > > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------15980C87736543ED7F9D9E1D Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Praveen and co-authors.

Thank you for hystart++.
I've sent a separate posting with technical review comments.

Cheers


Bob

==Editorial comments==


Abstract

s/doument/document/
s/overshotting/overshooting

1. Intro

s/pefrormance/performance/

"...lead to overshooting the ideal sending rate and cause significant packet loss"
Also queuing delay? Is reducing queuing delay (for other flows) also a goal of Hystart++?

Last para of intro seems to be about Hystart, and it doesn't make it clear how Hystart relates to Hystart++:
* "HyStart++ adds delay increase..."
   Adds to what?
* "After the HyStart delay algorithm finds an exit point, a Conservative Slow Start (CSS) phase is used"
  But what about Hystart++? does it do CSS too?

Final sentence is not related to the rest of the para about Hystart, so should be a separate para.

3. Definitions

It seems odd to define some well-known variables (like cwnd), but not others like SND.UNA, SND.NXT, etc.

cwnd: Would be useful to define the units assumed in the pseudocode later (bytes), because some implementations use segments.
It's confusing for cwnd to use bytes but LOW_CWND to use segments.


4.2 Algorithm Details


    "The choice of value of L is up to the implementation."
I suggest:
    "The choice of value of L is discussed in RFC 3465."
Reason: RFC3465 doesn't say it's completely up to the implementation - it gives recommendations and advice.

"normal slow start"
I appreciate you were asked to use this word, but... I suggest it's best to define this phrase because, for instance, the normal slow start for Cubic (Hystart) is different from the normal slow start for Reno. It's also a problem finding a good word that means the pre-existing thing. 'Reno slow start' would be more precise than 'normal slow start' (if you mean the one in Reno). Because Cubic is more normal than Reno these days. And saying 'standard' will soon be ambiguous, because Cubic is on its way onto the standards track.


    "If congestion is observed anytime during slow start or CSS, enter congestion avoidance."
The word 'congestion' is ambiguous, because it could include delay increase. How about:
    "If loss or ECN-marking is observed anytime during slow start or CSS, enter congestion avoidance."


5.  Deployments and Performance Evaluations 

   As of the time of writing, HyStart++ has been default enabled for all
   TCP connections in Windows for two years.  The original Hystart has
   been default-enabled for all TCP connections in Linux TCP for a
   decade.

I suggest you give the actual deployment times, so these two sentences don't keep having to be updated as the draft ages.
Also, in Linux, Hystart is only implemented for Cubic (the default CC module), so Hystart is not default if a non-default CC module is in use. How about:
In Linux, the original Hystart has been default enabled for
TCP connections using the default congestion control module (Cubic)
for a decade.

Regards


Bob


On 12/07/2021 16:46, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : HyStart++: Modified Slow Start for TCP
        Authors         : Praveen Balasubramanian
                          Yi Huang
                          Matt Olson
	Filename        : draft-ietf-tcpm-hystartplusplus-02.txt
	Pages           : 8
	Date            : 2021-07-12

Abstract:
   This doument describes HyStart++, a simple modification to the slow
   start phase of TCP congestion control algorithms.  Traditional slow
   start can cause overshotting of the ideal send rate and cause large
   packet loss within a round-trip time which results in poor
   performance.  HyStart++ is composed of the delay increase variant of
   HyStart to prevent overshooting of the ideal sending rate, while also
   mitigating poor performance which can result from false positives.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm



-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------15980C87736543ED7F9D9E1D-- From nobody Sat Jul 24 03:13:25 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F1D2D3A3560; Sat, 24 Jul 2021 03:13:22 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H14_W3BiN6W8; Sat, 24 Jul 2021 03:13:17 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7CF943A355D; Sat, 24 Jul 2021 03:13:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:References:Cc:To:From:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=YgRBy8LKg+kgZViVbnoSn6Aytf9UaE0wz3u7iSvLzMg=; b=AllB+G0FyR9m0acj/VsMd1+tNQ 2bPPkJofO+/+N6wMfWsTuhCl0g4sVWj61RSJzsZQQtvc8Y111/pgWgJ3NjX6R3nrC5E0ddgH9HC3F TCx8HbANLNuwCVOzDcsYOPWONkCF1k7Ig4SMWXPCnGjT2CaniPrxKVXnyJAUpz7ztVnc+PaB8kZ+Z kpXqFOS6Bsb9XgMZXMN44IrJku3JRH6UyWqpSgnddyA6CkfNLwHFVoZseqDzzFBYXMPWJTbqzvWop o9N4dZuXlwmLXxjPP3uZKWsbIBhR+/RRn9rwA3pcfCFNbS9UY3EEeLIPsxPSgnQCLT6Ri4O6U/f9v ruyRee2A==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:45280 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m7Eeq-0004JM-Vy; Sat, 24 Jul 2021 11:13:14 +0100 From: Bob Briscoe To: Praveen Balasubramanian Cc: tcpm@ietf.org, draft-ietf-tcpm-hystartplusplus@ietf.org References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <81092896-921b-9563-42b5-e9f412d5782d@bobbriscoe.net> Message-ID: Date: Sat, 24 Jul 2021 11:13:13 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <81092896-921b-9563-42b5-e9f412d5782d@bobbriscoe.net> Content-Type: multipart/alternative; boundary="------------82A64B5A4D2E61C5BD82B517" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] Full review of: draft-ietf-tcpm-hystartplusplus-02 (editorial) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Jul 2021 10:13:23 -0000 This is a multi-part message in MIME format. --------------82A64B5A4D2E61C5BD82B517 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Praveen & co-authors, Sry, omitted one more comment about the abstract. Added inline. Bob On 22/07/2021 16:43, Bob Briscoe wrote: > Praveen and co-authors. > > Thank you for hystart++. > I've sent a separate posting with technical review comments. > > Cheers > > > Bob > > > ==Editorial comments== > > > Abstract > s/prevent/mitigate/ Otherwise over-claiming. > s/doument/document/ > s/overshotting/overshooting > > > 1. Intro > > s/pefrormance/performance/ > > "...lead to overshooting the ideal sending rate and cause significant > packet loss" > Also queuing delay? Is reducing queuing delay (for other flows) also a > goal of Hystart++? > > Last para of intro seems to be about Hystart, and it doesn't make it > clear how Hystart relates to Hystart++: > * "HyStart++ adds delay increase..." >    Adds to what? > * "After the HyStart delay algorithm finds an exit point, a > Conservative Slow Start (CSS) phase is used" >   But what about Hystart++? does it do CSS too? > > Final sentence is not related to the rest of the para about Hystart, > so should be a separate para. > > > 3. Definitions > > It seems odd to define some well-known variables (like cwnd), but not > others like SND.UNA, SND.NXT, etc. > > cwnd: Would be useful to define the units assumed in the pseudocode > later (bytes), because some implementations use segments. > It's confusing for cwnd to use bytes but LOW_CWND to use segments. > > > 4.2 Algorithm Details > > >     "The choice of value of L is up to the implementation." > I suggest: >     "The choice of value of L is discussed in RFC 3465." > Reason: RFC3465 doesn't say it's completely up to the implementation - > it gives recommendations and advice. > > "normal slow start" > I appreciate you were asked to use this word, but... I suggest it's > best to define this phrase because, for instance, the normal slow > start for Cubic (Hystart) is different from the normal slow start for > Reno. It's also a problem finding a good word that means the > pre-existing thing. 'Reno slow start' would be more precise than > 'normal slow start' (if you mean the one in Reno). Because Cubic is > more normal than Reno these days. And saying 'standard' will soon be > ambiguous, because Cubic is on its way onto the standards track. > > >     "If congestion is observed anytime during slow start or CSS, enter > congestion avoidance." > The word 'congestion' is ambiguous, because it could include delay > increase. How about: >     "If loss or ECN-marking is observed anytime during slow start or > CSS, enter congestion avoidance." > > > 5.  Deployments and Performance Evaluations > >  As of the time of writing, HyStart++ has been default enabled for all > TCP connections in Windows for two years. The original Hystart has > been default-enabled for all TCP connections in Linux TCP for a > decade. > > I suggest you give the actual deployment times, so these two sentences > don't keep having to be updated as the draft ages. > Also, in Linux, Hystart is only implemented for Cubic (the default CC > module), so Hystart is not default if a non-default CC module is in > use. How about: > > In Linux, the original Hystart has been default enabled for > TCP connections using the default congestion control module (Cubic) > for a decade. > > > Regards > > > Bob > > > On 12/07/2021 16:46, internet-drafts@ietf.org wrote: >> A New Internet-Draft is available from the on-line Internet-Drafts directories. >> This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. >> >> Title : HyStart++: Modified Slow Start for TCP >> Authors : Praveen Balasubramanian >> Yi Huang >> Matt Olson >> Filename : draft-ietf-tcpm-hystartplusplus-02.txt >> Pages : 8 >> Date : 2021-07-12 >> >> Abstract: >> This doument describes HyStart++, a simple modification to the slow >> start phase of TCP congestion control algorithms. Traditional slow >> start can cause overshotting of the ideal send rate and cause large >> packet loss within a round-trip time which results in poor >> performance. HyStart++ is composed of the delay increase variant of >> HyStart to prevent overshooting of the ideal sending rate, while also >> mitigating poor performance which can result from false positives. >> >> >> The IETF datatracker status page for this draft is: >> https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ >> >> There is also an htmlized version available at: >> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 >> >> A diff from the previous version is available at: >> https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02 >> >> >> Internet-Drafts are also available by anonymous FTP at: >> ftp://ftp.ietf.org/internet-drafts/ >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm > > > -- > ________________________________________________________________ > Bob Briscoehttp://bobbriscoe.net/ -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------82A64B5A4D2E61C5BD82B517 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Praveen & co-authors,

Sry, omitted one more comment about the abstract. Added inline.

Bob

On 22/07/2021 16:43, Bob Briscoe wrote:
Praveen and co-authors.

Thank you for hystart++.
I've sent a separate posting with technical review comments.

Cheers


Bob

==Editorial comments==


Abstract


s/prevent/mitigate/
Otherwise over-claiming.

s/doument/document/
s/overshotting/overshooting

1. Intro

s/pefrormance/performance/

"...lead to overshooting the ideal sending rate and cause significant packet loss"
Also queuing delay? Is reducing queuing delay (for other flows) also a goal of Hystart++?

Last para of intro seems to be about Hystart, and it doesn't make it clear how Hystart relates to Hystart++:
* "HyStart++ adds delay increase..."
   Adds to what?
* "After the HyStart delay algorithm finds an exit point, a Conservative Slow Start (CSS) phase is used"
  But what about Hystart++? does it do CSS too?

Final sentence is not related to the rest of the para about Hystart, so should be a separate para.

3. Definitions

It seems odd to define some well-known variables (like cwnd), but not others like SND.UNA, SND.NXT, etc.

cwnd: Would be useful to define the units assumed in the pseudocode later (bytes), because some implementations use segments.
It's confusing for cwnd to use bytes but LOW_CWND to use segments.


4.2 Algorithm Details


    "The choice of value of L is up to the implementation."
I suggest:
    "The choice of value of L is discussed in RFC 3465."
Reason: RFC3465 doesn't say it's completely up to the implementation - it gives recommendations and advice.

"normal slow start"
I appreciate you were asked to use this word, but... I suggest it's best to define this phrase because, for instance, the normal slow start for Cubic (Hystart) is different from the normal slow start for Reno. It's also a problem finding a good word that means the pre-existing thing. 'Reno slow start' would be more precise than 'normal slow start' (if you mean the one in Reno). Because Cubic is more normal than Reno these days. And saying 'standard' will soon be ambiguous, because Cubic is on its way onto the standards track.


    "If congestion is observed anytime during slow start or CSS, enter congestion avoidance."
The word 'congestion' is ambiguous, because it could include delay increase. How about:
    "If loss or ECN-marking is observed anytime during slow start or CSS, enter congestion avoidance."


5.  Deployments and Performance Evaluations 

   As of the time of writing, HyStart++ has been default enabled for all
   TCP connections in Windows for two years.  The original Hystart has
   been default-enabled for all TCP connections in Linux TCP for a
   decade.

I suggest you give the actual deployment times, so these two sentences don't keep having to be updated as the draft ages.
Also, in Linux, Hystart is only implemented for Cubic (the default CC module), so Hystart is not default if a non-default CC module is in use. How about:
In Linux, the original Hystart has been default enabled for
TCP connections using the default congestion control module (Cubic)
for a decade.

Regards


Bob


On 12/07/2021 16:46, internet-drafts@ietf.org wrote:
A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : HyStart++: Modified Slow Start for TCP
        Authors         : Praveen Balasubramanian
                          Yi Huang
                          Matt Olson
	Filename        : draft-ietf-tcpm-hystartplusplus-02.txt
	Pages           : 8
	Date            : 2021-07-12

Abstract:
   This doument describes HyStart++, a simple modification to the slow
   start phase of TCP congestion control algorithms.  Traditional slow
   start can cause overshotting of the ideal send rate and cause large
   packet loss within a round-trip time which results in poor
   performance.  HyStart++ is composed of the delay increase variant of
   HyStart to prevent overshooting of the ideal sending rate, while also
   mitigating poor performance which can result from false positives.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/

There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-02


Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm



-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------82A64B5A4D2E61C5BD82B517-- From nobody Sun Jul 25 12:08:58 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 339033A361A; Sun, 25 Jul 2021 12:08:44 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.35.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162724012416.25845.1161922153629084292@ietfa.amsl.com> Date: Sun, 25 Jul 2021 12:08:44 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-hystartplusplus-03.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jul 2021 19:08:54 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : HyStart++: Modified Slow Start for TCP Authors : Praveen Balasubramanian Yi Huang Matt Olson Filename : draft-ietf-tcpm-hystartplusplus-03.txt Pages : 8 Date : 2021-07-25 Abstract: This doument describes HyStart++, a simple modification to the slow start phase of TCP congestion control algorithms. Traditional slow start can cause overshooting of the ideal send rate and cause large packet loss within a round-trip time which results in poor performance. HyStart++ uses a delay increase heuristic to exit slow start early while also mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-03 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-hystartplusplus-03 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Jul 26 01:31:09 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 00C013A20D2; Mon, 26 Jul 2021 01:31:08 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.35.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <162728826795.26459.12542596346017742913@ietfa.amsl.com> Date: Mon, 26 Jul 2021 01:31:07 -0700 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-rfc8312bis-03.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jul 2021 08:31:08 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : CUBIC for Fast and Long-Distance Networks Authors : Lisong Xu Sangtae Ha Injong Rhee Vidhi Goel Lars Eggert Filename : draft-ietf-tcpm-rfc8312bis-03.txt Pages : 30 Date : 2021-07-26 Abstract: CUBIC is a standard TCP congestion control algorithm that uses a cubic function instead of the linear window increase function on the sender side to improve scalability and stability over fast and long- distance networks. CUBIC has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks. This document updates the specification of CUBIC to include algorithmic improvements based on these implementations and recent academic work. Based on the extensive deployment experience with CUBIC, it also moves the specification to the Standards Track, obsoleting [RFC8312]. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-rfc8312bis/ There is also an HTML version available at: https://www.ietf.org/archive/id/draft-ietf-tcpm-rfc8312bis-03.html A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rfc8312bis-03 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Jul 26 17:36:09 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3FBF63A094F; Mon, 26 Jul 2021 17:36:08 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.451 X-Spam-Level: X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AypWP_cZZcJZ; Mon, 26 Jul 2021 17:36:03 -0700 (PDT) Received: from NAM06-DM3-obe.outbound.protection.outlook.com (mail-eopbgr640128.outbound.protection.outlook.com [40.107.64.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E01B73A094E; Mon, 26 Jul 2021 17:36:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LIb7DKz5oHWAwMAs2dYUUbcepNXZ4dD++FdwTvidPupRe5JPDW/hmkXLvkQAcEpd+XIi6/Ddigw/rj7XLxFmtf9QLqccqGQMweIc44rCim/i0n/7054spvRVhU9YJ0EKfKzyFL868K5euuYDgEqjfm6VOatgK3d/aa9Nytuye8XhaUdO0afuQVxNuirjsLokH6y4H+BGw/6yUD5p86ZoCW2CHHNRH8QSUsoyW2P9zMKzU+jf+e+7YmTsL0dbvnqw6k1jk8NwuNki4yXVsQ2eFnAKuJJsc8SAsbmutGoLSUxhd06OYBdyawOg1772b8b479X7eF2v27kXpig7w7NkEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=duFpQM9qB9UR3lubUnvPYAyEYVr0ADCYHOE1PCLOPzE=; b=gcc/CHjeHIYJA0Dul8GMabgQxpTUR33KUxVwOmdPaqEIzuYsjQ5yDcg+UBd9OJkcHg5c7YtJxgXXXGeB0lz8nOTbdkflwpBixe2HDGCm3vTY4RSGaGuGYtlAy8Ab1y18eYMX+rD6t7aJ932Qua4pr/7cBJm1NSWk32V71LlHqwdERhRprPO40jJPjMWTx/pfIlbkB9bBnh+6XvdntNU7tYNv0mThhoE0KokqNvivmFqEPo6MJXNWSI5E6/OlCvBM0WkRe3Z3hYYwCQE58Ud83KMbU6ScbA1zCKjArVeAcHi0Y7bSXoL5nC+1VWUJqzYXfvQiNNKsW9unl09cvdXgeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=duFpQM9qB9UR3lubUnvPYAyEYVr0ADCYHOE1PCLOPzE=; b=W2YRLqKf3xnzFcaKR99I0FmpUBPyVC79AJ579Z0UMwia1bcXGKR0oRKAYMD/SHMQCoWcH4fN7BH7YEQgiAyhYs57CujrTjo2T1MJVtAD1a8fJffiDoywq+5SQXQ6PFIUuAVTDX7yUWU9vJuh3VECsKBgutR6YOo3puW2Sf0qgjM= Received: from PH0PR00MB1030.namprd00.prod.outlook.com (2603:10b6:510:48::7) by PH0PR00MB1352.namprd00.prod.outlook.com (2603:10b6:510:10f::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4399.0; Tue, 27 Jul 2021 00:33:55 +0000 Received: from PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88]) by PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88%5]) with mapi id 15.20.4408.000; Tue, 27 Jul 2021 00:33:55 +0000 From: Praveen Balasubramanian To: "ietf@bobbriscoe.net" CC: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Thread-Topic: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) Thread-Index: AQHXfxBhDL5MNyxppUKpsFE4e4Tyu6tV/jRg Date: Tue, 27 Jul 2021 00:33:54 +0000 Message-ID: References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> In-Reply-To: <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=e14d1ae9-9b3e-4f51-919f-58e4f79d2f8c; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-27T00:27:22Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: bobbriscoe.net; dkim=none (message not signed) header.d=none;bobbriscoe.net; dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 55622c6f-5f70-4b7a-f920-08d950963749 x-ms-traffictypediagnostic: PH0PR00MB1352: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +gcQ0kQsID960n+Pf3qEE5/OSzYH/RH/GXEEezdZxAZ8a7Wl80LFA3zj47Ndq+ujxxhZg+Yl+NjsIQVNqac1IUmnL5N6Zoa7gwndOEN93U8bRDlQSJ0G93khh5qgy+Lur4zVoP4EBpY4Yxrv9tN2mMcskNFZ5brSeeLhrJWwouhfLZvyBdgQJtAdW49GYfj98Al8Cv+qCY7iDlIiTPYwCs62dnmRAdSsjeF574u2clhpjAki73ghpYWxTj6ZBVzJIyztq6NChFWG4eutSiUy0H6kHVSE61F514ibEwYuAlu0zVeuKgXJoP0RRT3EysW6u2ocRmX92/9Vy7ZfKvdglBqqQEixxm6vh7tfVwxQbATFMhh1gW+8rw9635cs53XzVn8P5dlRKhA6VjUef+cLonK1ecp6Xu2aqCXMz7jy5g5bcm8F848lyAb4K4ZP8IIV/CZB+bGvzmnYx86oKnBGWaDu6JdIwHaPqHNjJwmnZuYeQF3u9lVhpMfPCVRbwXBP9UMvbKHFcJqFwsoYYoHG0b1Pff/88gYZgr9jvUHmelSjW8IcKuVlLDOGaKle8lzjoe2wICwJV2mTBPRCx8risld6e69zzdX0SABDUrlJeHDHcQ/76p2Duu+9G8BcY8/CEyuHOzxvc1pLO1zmcyHXV4fQ6iTOqRi+cq8EiEdzwLJdKsDph5L1b4k0yhbAhYV9zYXOUNKGX1hTuTfjVIw6gUXuuhidgOvuJoQ2/yyUVKXwMZDi5oLVR8dlUgYelAfhgv1jI1Gkgj3/8PaMVSlt89UvQ6I9Ii5j773lmmidsj5KGK/GKLHJqUrtNZ0ULzcSr5QITPXcdwjC0PSSMQ5jcw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR00MB1030.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(186003)(71200400001)(76116006)(82960400001)(8676002)(4326008)(66556008)(508600001)(82950400001)(66946007)(66476007)(64756008)(66446008)(10290500003)(316002)(54906003)(2906002)(8990500004)(122000001)(38100700002)(53546011)(966005)(5660300002)(86362001)(166002)(6506007)(9686003)(6916009)(8936002)(52536014)(66574015)(7696005)(33656002)(55016002)(83380400001)(38070700004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?j9rHMs6sgMErB/DXq+UI2n/Y8XUS1ltoJa794wQlhoAupsx2xBbVJRi8WONo?= =?us-ascii?Q?bSeJfRlEj11bZ9ux9cRG+15w1cbBcZgAm3pbUFNHCGdaldSSxLvF/zbwX5rZ?= =?us-ascii?Q?3qjjYePEkb3BbWEZEs39YvDfr+5VCdZ46OOldV+rdHAXOw3zCwuXRSOUlie5?= =?us-ascii?Q?jxSHY5nevpiQR+ifucKsWwM7akr4TmuNJw+bBU7f8ET0kf3LhadRgd4w9yQf?= =?us-ascii?Q?8ct61VqHBndlIP1FeqStNhwR9yms13mXui1Mlz5IpYll4lWoqj3UD+aKcxBd?= =?us-ascii?Q?7xx/czdBlY6/6SBMBYBWACzgD9nBcfSPYux/GN8f8NhR1lZ4RPPdwV7JqVdO?= =?us-ascii?Q?hOxySIaWH/wb/AE5aAmkQ2qaNwteLZrThiDiVOrKDFZEceFnhdghb357QSvz?= =?us-ascii?Q?Gb29jo8199ZecfQC843nhtJhaciHEvL9gbTAwSz/zlPhyri4xG2dzSKKa1Wk?= =?us-ascii?Q?8OOCTHOShYnbJRsD12gkjcUVuEFUqK0bqi7eFYXdFm8Ox7Pzr8hvq9oh1gRR?= =?us-ascii?Q?6DUrLNWjeFnDDm7eh4asE61IUPO/TUdET2WHUb+F5qnAgExCe3el+QKtOYXc?= =?us-ascii?Q?Z5Ru07N8XwM9PSffyA4AdRoV02KrNc2x2Rhp3b0if1T/Wc0iVcd2qrDBysY0?= =?us-ascii?Q?9luuP5m5Nv/yw+ZGgD6XyU4w6nIVOlQgx6/hxKRMr9w/09j+s4bv7w8TUBl+?= =?us-ascii?Q?xaq2zNIbbt010h63eOjwkZXxjnUP/3h+2IK1ZrcxNqLK3rjDVVjTjwRryoXS?= =?us-ascii?Q?8UWRgANpHx+ALX4RLjtqZ5spZRObXMOJX5qObJyncZ5BX0ADrSc28eGh05QQ?= =?us-ascii?Q?seUEpRAFgzY3h169BI06fVZ1jOkelHGpQxkvwIqLJaTB3EzEsB9dig7BdwFQ?= =?us-ascii?Q?nnF+0lSo81ebY44PjT4Dh+c6Zv98PmgsL38mX8/FC1TqR7PY/8azD+sDJRl0?= =?us-ascii?Q?bSFm8Fz78A1C3hE2emEulC66lAjjr/GjMBQv+bHrPqKtmmn7ww1xF4sB7MiO?= =?us-ascii?Q?Wt8eFBucGD/bwqKr+pNPwpRR8ez6Hlyf69cqMJfbbbip6hqxJwQFDzx7Lf+7?= =?us-ascii?Q?mD6dRpQM0Hl2MGGRmLtxalLotXUf7WkLkhQTKnueC1BOJJY8JVD96tFT5aRi?= =?us-ascii?Q?N/EkWDDth6h9wWfH1M3bNOc2hpAIaFBDbXO2pQFqR79bMy9nFvXyDG34RL/S?= =?us-ascii?Q?XKFwDb971dl6Ww08qN9Te5F24+tnU3Vin+k5Guno98ZXi2gT0J++oWMlt5FW?= =?us-ascii?Q?rRrGkA/Kr4Xz2o5oJryK/L6IB9MBUX2XWnGuexMjKctf7pThFJ5N6A6ky7ej?= =?us-ascii?Q?xSCfIkjR9g/DYnUV8G6LoprLIO6diI0TEjugTxCZD2FSUnkwg/bvWcqkCxev?= =?us-ascii?Q?lBrGt0PghulX+/tlxnYQBD07mXJ/?= x-ms-exchange-transport-forked: True Content-Type: multipart/alternative; boundary="_000_PH0PR00MB1030126A3220BC056A406490B6E99PH0PR00MB1030namp_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR00MB1030.namprd00.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55622c6f-5f70-4b7a-f920-08d950963749 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jul 2021 00:33:54.9729 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: y1knxUtaLwTIXzAXUWQEpDLVSG+Gzu/hUiVWfvUZ4RGRYGrsZGVLvavbsmbXPbv8MO6cMGdXr1ICy0SucTmaRnRNkQ2Kg9i96/STq7fYa8g= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR00MB1352 Archived-At: Subject: Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 00:36:08 -0000 --_000_PH0PR00MB1030126A3220BC056A406490B6E99PH0PR00MB1030namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks Bob for the review. We published draft-03 addressing some of your co= mments. Inline prefixed by >>>> From: Bob Briscoe Sent: Thursday, July 22, 2021 8:42 AM To: Praveen Balasubramanian Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (tec= hnical) Praveen and co-authors. Thank you for hystart++. This one's my technical review. I've sent my editorial review in a separate= posting. I actually reviewed an earlier draft (balasubrmanian-02), but never got rou= nd to writing it up for you. I've just reviewed the latest (ietf-02) draft,= and incorporated my comments about the older revision where still relevant= . Cheers Bob =3D=3D=3DTechnical Comments=3D=3D 4.2. Algorithm Details ________________________________ Round can be approximated using sequence numbers as follows: ... At the start of each round during normal slow start and CSS: It might be worth stating that using sequence numbers is the /best/ way to = measure rounds, not just one way. You want every round to contain at least = one of the ping-ponged segments that were grandfathered by the very first s= egment of the IW (because that will be the segment most likely to have the = min RTT). For instance, if you measure rounds in time, you might underestim= ate the duration of a round. Then, in some rounds, you would risk not seein= g one of this ping-pong series at all. >>>> Added a recommendation for using the sequence number approach. ________________________________ "where currRTT is the RTT sampled from the incoming ACK" Need to make it clear this RTT is based on the latest ACK'd packet, not the= earliest (in contrast to the RTT used for RTO). >>>> Clarified. ________________________________ The somewhat opaque clamping of RTTThresh below needs to be justified: For rounds where cwnd is at or higher than LOW_CWND and N_RTT_SAMPLE RTT samples have been obtained, check if delay increase triggers slow start exit ... RttThresh =3D clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RT= T_THRESH) if (currentRoundMinRTT >=3D (lastRoundMinRTT + RttThresh)) cssBaselineMinRtt =3D currentRoundMinRTT exit slow start and enter CSS To aid comprehension for the benefit of the list, I'll separate the clamp l= ine into two steps and substitute the default constants: RttThresh =3D clamp(32ms, lastRoundMinRTT, 128ms) RttThresh /=3D 8 To me, it makes good sense without the clamping. In english, that would be = just: "if queuing due to slow start increases the min RTT by more than 12.5% = from one round to the next, exit slow start" However, taking the default values of the constants, the clamping expressio= n adds the following conditions: #1) If the base RTT is lower than 32ms, the queue has to be greater than 4m= s to exit slow start. #2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms= to exit slow start. I.e. at both extremes, the threshold becomes absolute, rather than a fracti= on of the base RTT. Possible rationales for the lower condition when the base RTT is small (#1)= : a) an absolute Qdelay threshold is needed because small delay measurements = are too noisy. b) a relative amount of Qdelay would become smaller than humans can perceiv= e, so an absolute floor is sufficient. If (a), this needs to be stated, because different min delay measurement te= chniques can achieve different noise immunity. I don't agree with (b), because unnecessary delay over a sequence of rounds= can accumulate to become perceivable. I can't think of a good rationale for the higher condition (#2). I'm sure i= t was the result of experiments, but it seems to be likely that as base RTT= increases beyond 128ms, Hystart++ would become increasingly sensitive and = exit slow start earlier and earlier - unnecessarily (?). Whatever, the rationale for clamping RttThresh needs to be stated, so admin= s know how to set the parameters for other scenarios (for instance, in DCs = excessive loss would occur with the given defaults). >>>> This is worth discussing in the WG meeting tomorrow time permitting. T= hese are a result of experiments. These are fixed clamps because queue buil= d up at the bottleneck has no direct correlation with the base RTT. I agree= though that stating the justification is useful and we will look into this= for the next revision. We are open to ideas to make these thresholds dynam= ic. ________________________________ 5. Deployments and Performance Evaluations It would be useful to see testbed results separated by whether the bottlene= ck before the flow arrives is empty, full, or part-full of traffic. >>>> This is a good suggestion and can be implemented by staggered start of= multiple flows. We will incorporate these tests. Although Hystart is default enabled in Linux, it is invariably disabled. So= , it's misleading to just say Hystart is default enabled, which implies it'= s widely used, when people clearly find it has problems (which motivates Hy= start++). I found this out through an informal survey I did at the Mar'18 I= ETF in London by asking round the implementers of the most prevalent stacks= (I would name names if I could find the note I later sent to someone or to= some list, but I can't find it - sry). >>>> I'd like some citations on this versus anecdata if possible before I a= dd that caveat to the text. Do large deployments disable this? I haven't co= me across this suggestion in any Linux tuning guides to date. ________________________________ Regards Bob On 12/07/2021 16:46, internet-drafts@ietf.org wrote: A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the TCP Maintenance and Minor Extensions WG of= the IETF. Title : HyStart++: Modified Slow Start for TCP Authors : Praveen Balasubramanian Yi Huang Matt Olson Filename : draft-ietf-tcpm-hystartplusplus-02.txt Pages : 8 Date : 2021-07-12 Abstract: This doument describes HyStart++, a simple modification to the slow start phase of TCP congestion control algorithms. Traditional slow start can cause overshotting of the ideal send rate and cause large packet loss within a round-trip time which results in poor performance. HyStart++ is composed of the delay increase variant of HyStart to prevent overshooting of the ideal sending rate, while also mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hystartplusplus-02 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ _______________________________________________ tcpm mailing list tcpm@ietf.org https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --_000_PH0PR00MB1030126A3220BC056A406490B6E99PH0PR00MB1030namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks Bob for the review. We published draft-03 add= ressing some of your comments.

 

Inline prefixed by >>>>

 

From: Bob Briscoe <ietf@bobbriscoe.net>=
Sent: Thursday, July 22, 2021 8:42 AM
To: Praveen Balasubramanian <pravb@microsoft.com>
Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-= 02 (technical)

 

Praveen and co-author= s.

Thank you for hystart++.
This one's my technical review. I've sent my editorial review in a separate= posting.

I actually reviewed an earlier draft (balasubrmanian-02), but never got rou= nd to writing it up for you. I've just reviewed the latest (ietf-02) draft,= and incorporated my comments about the older revision where still relevant= .

Cheers


Bob

=3D=3D=3DTechnical Comments=3D=3D


4.2.  Algorithm Details

 
 Round can be approximated using sequence numbers as follows:
   ...
   At the start of each round during normal slow start and C=
SS:

It might be worth sta= ting that using sequence numbers is the /best/ way to measure rounds, not j= ust one way. You want every round to contain at least one of the ping-ponge= d segments that were grandfathered by the very first segment of the IW (because that will be the segment most li= kely to have the min RTT). For instance, if you measure rounds in time, you= might underestimate the duration of a round. Then, in some rounds, you wou= ld risk not seeing one of this ping-pong series at all.

 

>>>> Adde= d a recommendation for using the sequence number approach.


"where currRTT is the RTT sampled from the incoming ACK"
Need to make it clear this RTT is based on the latest ACK'd packet, not the= earliest (in contrast to the RTT used for RTO).

 

>>>> Clar= ified.


The somewhat opaque clamping of RTTThresh below needs to be justified:= 

      For rounds where cwnd is at or higher t=
han LOW_CWND and
      N_RTT_SAMPLE RTT samples have been obta=
ined, check if delay
      increase triggers slow start exit<=
/o:p>
      ...
            Rtt=
Thresh =3D clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RTT_THRESH)<=
/o:p>
            if =
(currentRoundMinRTT >=3D (lastRoundMinRTT + RttThresh))
           &nbs=
p;   cssBaselineMinRtt =3D currentRoundMinRTT
           &nbs=
p;   exit slow start and enter CSS


To aid comprehension for the benefit of the list, I'll separate the clamp l= ine into two steps and substitute the default constants:
            RttThres= h =3D clamp(32ms, lastRoundMinRTT, 128ms)
            RttThres= h /=3D 8

To me, it makes good sense without the clamping. In english, that would be = just:
    "if queuing due to slow start increases the min RTT= by more than 12.5% from one round to the next, exit slow start"

However, taking the default values of the constants, the clamping expressio= n adds the following conditions:

#1) If the base RTT is lower than 32ms, the queue ha= s to be greater than 4ms to exit slow start.
#2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms= to exit slow start.

I.e. at both extremes, the threshold becomes absolut= e, rather than a fraction of the base RTT.

Possible rationales for the lower condition when the base RTT is small (#1)= :

a) an absolute Qdelay threshold is needed because sm= all delay measurements are too noisy.
b) a relative amount of Qdelay would become smaller than humans can perceiv= e, so an absolute floor is sufficient.

If (a), this needs to= be stated, because different min delay measurement techniques can achieve = different noise immunity.
I don't agree with (b), because unnecessary delay over a sequence of rounds= can accumulate to become perceivable.

I can't think of a good rationale for the higher condition (#2). I'm sure i= t was the result of experiments, but it seems to be likely that as base RTT= increases beyond 128ms, Hystart++ would become increasingly sensitive and = exit slow start earlier and earlier - unnecessarily (?).

Whatever, the rationale for clamping RttThresh needs to be stated, so admin= s know how to set the parameters for other scenarios (for instance, in DCs = excessive loss would occur with the given defaults).

>>>> This= is worth discussing in the WG meeting tomorrow time permitting. These are = a result of experiments. These are fixed clamps because queue build up at t= he bottleneck has no direct correlation with the base RTT. I agree though that stating the justification is useful and we w= ill look into this for the next revision. We are open to ideas to make thes= e thresholds dynamic.  


5.  Deployments and Performance Evaluations

It would be useful to see testbed results separated by whether the bottlene= ck before the flow arrives is empty, full, or part-full of traffic.

>>>> This= is a good suggestion and can be implemented by staggered start of multiple= flows. We will incorporate these tests.


Although Hystart is default enabled in Linux, it is invariably disabled. So= , it's misleading to just say Hystart is default enabled, which implies it'= s widely used, when people clearly find it has problems (which motivates Hy= start++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the = implementers of the most prevalent stacks (I would name names if I could fi= nd the note I later sent to someone or to some list, but I can't find it - = sry).

>>>> I= 217;d like some citations on this versus anecdata if possible before I add = that caveat to the text. Do large deployments disable this? I haven’t= come across this suggestion in any Linux tuning guides to date.

 


Regards


Bob

On 12/07/2021 16:46, internet-drafts@ietf.org wrote:

 
A New Internet-Draft is available from the on-line Internet-Drafts dir=
ectories.
This draft is a work item of the TCP Maintenance and Minor Extensions =
WG of the IETF.
 
        Title   &nbs=
p;       : HyStart++: Modified Slow Start for=
 TCP
        Authors   &n=
bsp;     : Praveen Balasubramanian
           &nbs=
p;            &=
nbsp; Yi Huang
           &nbs=
p;            &=
nbsp; Matt Olson
  Filename        : draft-ietf=
-tcpm-hystartplusplus-02.txt
  Pages         &nbs=
p; : 8
  Date          =
;  : 2021-07-12
 
Abstract:
   This doument describes HyStart++, a simple modification t=
o the slow
   start phase of TCP congestion control algorithms.  T=
raditional slow
   start can cause overshotting of the ideal send rate and c=
ause large
   packet loss within a round-trip time which results in poo=
r
   performance.  HyStart++ is composed of the delay inc=
rease variant of
   HyStart to prevent overshooting of the ideal sending rate=
, while also
   mitigating poor performance which can result from false p=
ositives.
 
 
The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartpl=
usplus/
 
There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-=
tcpm-hystartplusplus-02
 
A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hys=
tartplusplus-02
 
 
Internet-Drafts are also available by anonymous FTP at:

ftp://ftp.i=
etf.org/internet-drafts/


 


 


_______________________________________________


tcpm mailing list


tcpm@ietf.org


https:/=
/www.ietf.org/mailman/listinfo/tcpm





-- 
________________________________________________________________<=
/o:p>
Bob Briscoe          =
;            &n=
bsp;        http://bobbriscoe.net/
--_000_PH0PR00MB1030126A3220BC056A406490B6E99PH0PR00MB1030namp_-- From nobody Mon Jul 26 17:52:08 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6AC4F3A0BC2; Mon, 26 Jul 2021 17:52:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.554 X-Spam-Level: X-Spam-Status: No, score=-2.554 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kW7RM8ryXcQz; Mon, 26 Jul 2021 17:51:59 -0700 (PDT) Received: from NAM06-DM3-obe.outbound.protection.outlook.com (mail-eopbgr640093.outbound.protection.outlook.com [40.107.64.93]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7C04D3A0BB9; Mon, 26 Jul 2021 17:51:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=l5j+w1OEO+fBuTuAWLx3jtB9VDpzNxFJQBZrQXkSNH6FJiJGf4qKnaPK24YqyDD1J5mULeGgK/5Er7DaPRtLDmtbHs1jgAZYCc81uxxcHW+uVbuwjf4cfHQnUV1CmdgLXsZNov2VRs+RNQoGVPlJ0B4rCfrsDvFPt/e4+QJMi/4Y3I3l975woNPqFbHJmAoy/+ML+cWAbO83FfHod8uPax2RvWEueyHyk44uA5v0CTIKERh/BSfq8Ll/DsgrVrITQAbBA8bbBDfdmGCLeFGmw+SKMB268NaflGL7gk1bfbQfL9gRPRf36s/NS3/bKxTXl9AcyNOqWSsg2yu2nllc9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uyoUhEZPXjG85B1uNSHgNWCFYn3S8KnXc4vSrGsGAy8=; b=dVpMte6fCuM3VxteabM7DJuEl3yJixxx1w0GpRKgka1/YYG3F5dITVYs4XSvv+iE0NCxCqN9dCbltAfAG9SDg1aizXHNelstxwEm5dG3v5/D/TJTdtX7Y94Uhy6xIjf72HzSBrdrOnGdJStlyxBZvnL2DUG7/XVpz4ckOXAwprb7LViQAObF2bSnws9kgTJtv0itJV8r2QkOnKVfsfMzsz1AhOonkKJfUL+fWDYrUZoS2F+3M4FYW/e6ShqWe2jRLXfe/XNioBORJUjqS91BFjU7kSHJxpc6MSGTcmdfl8RjA/Q6caJ43RhjHvzhB5KajTpj9ZPfa0/Vv8pbk/YOOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uyoUhEZPXjG85B1uNSHgNWCFYn3S8KnXc4vSrGsGAy8=; b=hxupl1+ehwBoMHS/CsfZKNGlwhrgdQuKjh3gMLFYDvcx6xcNyhgAQkT1NP0V9LC6i2utpcDvS2xChvks3hhN2sEjdC+efxng7ok280M/yTDzDE13Ob7+VPFN5D8+t2ZuV2ox7baaBu1itybOD253XaXRsyI5qgSNW2HKlnEf4aw= Received: from PH0PR00MB1030.namprd00.prod.outlook.com (2603:10b6:510:48::7) by PH0PR00MB1135.namprd00.prod.outlook.com (2603:10b6:510:9c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4402.0; Tue, 27 Jul 2021 00:50:06 +0000 Received: from PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88]) by PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88%5]) with mapi id 15.20.4408.000; Tue, 27 Jul 2021 00:50:05 +0000 From: Praveen Balasubramanian To: "ietf@bobbriscoe.net" CC: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Thread-Topic: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) Thread-Index: AQHXfxBhDL5MNyxppUKpsFE4e4Tyu6tV/jRggAAGIGA= Date: Tue, 27 Jul 2021 00:50:05 +0000 Message-ID: References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=e14d1ae9-9b3e-4f51-919f-58e4f79d2f8c; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-27T00:27:22Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: bobbriscoe.net; dkim=none (message not signed) header.d=none;bobbriscoe.net; dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4f2393cf-b290-4680-834d-08d9509879eb x-ms-traffictypediagnostic: PH0PR00MB1135: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Ijk3kO+FK5gzOo0A1MGvjdnDdJOC3BFWfzPB5j0tFrQlYWOsjti/Es45++oqgJQM9qOQsfu7+0IZ7W81AwdLPbGK6Z89sboI2QM/51DKgePWZ6QTqoi62LFNyIA1MLDtWUKZ6BjwZ2S+A2vhaPBjV/01DABGo+buMUy1yhBvJ2E828yjxZ1dvNi6eA9ixaXGnFkK3yQaYuKmFWtY8YcVhOAjDdpWZUA/hWqihkii5TyazYmD6bkq7hD5zExd+qCZhEwNVgvRXruOuBPety1mTLkF9HkiniXXD0J8U+sIbSV7Z2Wqc1Y6U/rsHBcaQndRnWqhGtW2UISzKfdRdhrOYJzqJ/6jopGVdrovPZpM0L7Fnj4ZCQDvHg+b72oH/thN4OjvAlEDO4KXavMJZIDsRoiPpw3KbcfXp08uOwc6w8dCJLk1R69QD7RTTWQpf4Nuxst1i7BzXREDBv+kA7aYNdBcc4s/2gdkrgZgWw3w+3ROdek4Co7FEDxKSMVtMfQsEhR/qnbjuK7U+cOG5ZMCNK650mCpfKCDGQ2uPTrQz9ImE8N0QC+c66DQ+ySqDI764bqaSvLF4mDB4Aj3OzXL1T0OP9z0z/UVs/Qui9uQsKijSvCUid/cQvvuXdk+DKtdmftAMQovgionh29IzrjA2IZ22cE3HxxaY1BQblJZGrQ08Ws8bEPj1+2hCiXcR3aiARat6pO5vFTwa8NX1BXpORWuKRtoopMBkmVMWWYyJKXBYbirDHXHGHKafoY6hc1VeF6Uc4KzaDGdYv5TvJhM6IzAI+tkM+FIhwdM8bJc0VFuNdLo8JUt3GUu+Q3mkrMRlWQ+7BLHGGG/LhAIULirEQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR00MB1030.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(66574015)(54906003)(966005)(82960400001)(5660300002)(6506007)(7696005)(64756008)(83380400001)(508600001)(66556008)(66476007)(76116006)(8676002)(82950400001)(122000001)(8990500004)(66946007)(52536014)(55016002)(71200400001)(2906002)(186003)(316002)(38100700002)(86362001)(6916009)(53546011)(33656002)(8936002)(2940100002)(4326008)(66446008)(9686003)(10290500003)(38070700004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?lKSV9OHoB1Z4S1xzsmgD01wgDkFAIZLrtoghijp2U/oemfvUhhoIg+JR0Z?= =?iso-8859-1?Q?hzoCtwGa5G0CMJtj4zVBvfdQERPmZMV2P7Ekb05JQq+UxSGvGXvIU/hsBk?= =?iso-8859-1?Q?WNnVcZOPcUAaO0XyAMSL/PULOlUtnXRBDiZT8nN5m3YSA7wfLXBDhC0LCr?= =?iso-8859-1?Q?PjJTfMOR3AZVjWAMLm07xOfovfESrymWEU5ONbSnO9Q+WOerKFVABP6NrB?= =?iso-8859-1?Q?3/arHb+5vUhQTedQgUdSxiCCeg5OKglgHMrji8EAny+LolWPpZzWhiNcQP?= =?iso-8859-1?Q?1TM0CcBe0sRuyaHoDm77GaK7pIZdVs3lZD57UatnoJ0ZCiz653kneoS9TQ?= =?iso-8859-1?Q?TgcHuyXLQwprU2diw6x6QemJjSQRJ0gDBzEv6FAAwqNpCUgpWeblQTENv4?= =?iso-8859-1?Q?8/KhFiD0WZFWZIb11UV3wKUdB4l+ZOQL2ItU/7qels53tmWAPprDOX+WSa?= =?iso-8859-1?Q?N1/mfoa5VUHUHW6+XywEIyoE310KV8K5JfRUpiOzhXfeJoIVQSEEjMuiPv?= =?iso-8859-1?Q?nCUtOvACrnmk1AK0GS7YNipSvOIZYYK8n1OppD6zN7lJKM7qwPZrQ2TqOY?= =?iso-8859-1?Q?H2FlXOEc3BCcA/EHCo2nH5d8s/2RA9FLZ22LfJielZKcm9u8/uvYc6UekY?= =?iso-8859-1?Q?njS2BwgzaMGc6rtCttCupJwMAYtNCORGJ8+Jy24ZUhhQRyita+e/mmZHs/?= =?iso-8859-1?Q?hSZACXIfs83tC29zwjXx1zdh74x7uSzESbQAlwxVdpWGJP+8lCJ6TFPcWr?= =?iso-8859-1?Q?o+T6FlsbCmnEoZw1Shlza8MDJMHRB1ye4m+pgbXQvJQK3V3ZOGusYLUt1I?= =?iso-8859-1?Q?CvrdKXFfRssIcNmC1t+xqR4JLHFZzH2BLQjKdC6pqb6eWjgdXGe0lhXBiN?= =?iso-8859-1?Q?4Mozp2bwWwiT62FVU0AxqNB+nAaO4y1qHwdPPDI4DNThRhicXxM2SLaUe9?= =?iso-8859-1?Q?VKnJT4sxpB3ujdECIa4cWWT6swlhpfCYBRr1yLCPH5v3IDVIogQd6ZYIB3?= =?iso-8859-1?Q?SDGwZq/LkXwJQqgYgOG0B7u/jSgYl4zxLZHccF/kPeUXMxYe0SufzxyuzH?= =?iso-8859-1?Q?zamiK8ddZ31Y+v38EPxy23nymSKEy2JkWHiJTWAkUNa2FD9ZDXNRUD4btK?= =?iso-8859-1?Q?WhAZUnyVilkUX7dKVW3Ik1LNYHgKoYKFBowBs926ZBxklNuq025UehXYiP?= =?iso-8859-1?Q?GNO3MqDHobQNZ5vuI071tr9Dl9sSETaNtP+S4R6gS1CbnIRBqbBXMuTjWS?= =?iso-8859-1?Q?oHrv4iD1Kgyqdtu2Y62n4sDgm+w/vAnYE6w3VpqvhvxXuBBixklDAoo6jV?= =?iso-8859-1?Q?+FcnbXxfDxCZDdVYXbxDXZgEuXhDLd8ajA2Ij7YStTEU6UoAKgyWIPPJ4a?= =?iso-8859-1?Q?1N7eNsWzKO8uYkh+Wbqb8tnbee9hij9GsomvWtLXv498bhBgXgdI/nxfV+?= =?iso-8859-1?Q?IRrLPocDk5dbMRy1?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR00MB1030.namprd00.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f2393cf-b290-4680-834d-08d9509879eb X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jul 2021 00:50:05.7207 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M2uXjX/M259TuYF5MGtQhbmYv676/O5YCRImN6dULsvjiURf4WqicENFgvjgOSyAFngePgcxwSnXkGygmctzMjbzAglfrBWpCickGS/mRPA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR00MB1135 Archived-At: Subject: Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 00:52:07 -0000 Thanks Bob for the review. We published draft-03 addressing some of your co= mments.=20 Inline prefixed by >>>> From: Bob Briscoe =20 Sent: Thursday, July 22, 2021 8:42 AM To: Praveen Balasubramanian Cc: mailto:tcpm@ietf.org; mailto:draft-ietf-tcpm-hystartplusplus@ietf.org Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (tec= hnical) Praveen and co-authors. Thank you for hystart++.=20 This one's my technical review. I've sent my editorial review in a separate= posting. I actually reviewed an earlier draft (balasubrmanian-02), but never got rou= nd to writing it up for you. I've just reviewed the latest (ietf-02) draft,= and incorporated my comments about the older revision where still relevant= . Cheers Bob =3D=3D=3DTechnical Comments=3D=3D 4.2.=A0 Algorithm Details ________________________________________ =A0Round can be approximated using sequence numbers as follows: =A0=A0 ... =A0=A0 At the start of each round during normal slow start and CSS: It might be worth stating that using sequence numbers is the /best/ way to = measure rounds, not just one way. You want every round to contain at least = one of the ping-ponged segments that were grandfathered by the very first s= egment of the IW (because that will be the segment most likely to have the = min RTT). For instance, if you measure rounds in time, you might underestim= ate the duration of a round. Then, in some rounds, you would risk not seein= g one of this ping-pong series at all.=20 >>>> Added a recommendation for using the sequence number approach.=20 ________________________________________ "where currRTT is the RTT sampled from the incoming ACK" Need to make it clear this RTT is based on the latest ACK'd packet, not the= earliest (in contrast to the RTT used for RTO). >>>> Clarified. ________________________________________ The somewhat opaque clamping of RTTThresh below needs to be justified: =A0=A0=A0=A0=A0 For rounds where cwnd is at or higher than LOW_CWND and =A0=A0=A0=A0=A0 N_RTT_SAMPLE RTT samples have been obtained, check if delay =A0=A0=A0=A0=A0 increase triggers slow start exit =A0=A0=A0=A0=A0 ... =A0=A0=A0=A0 =A0=A0=A0=A0=A0=A0=A0RttThresh =3D clamp(MIN_RTT_THRESH, lastR= oundMinRTT / 8, MAX_RTT_THRESH) =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 if (currentRoundMinRTT >=3D (lastRoundMin= RTT + RttThresh)) =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 cssBaselineMinRtt =3D currentRou= ndMinRTT =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 exit slow start and enter CSS To aid comprehension for the benefit of the list, I'll separate the clamp l= ine into two steps and substitute the default constants: =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 RttThresh =3D clamp(32ms, lastRoundMinRTT= , 128ms) =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 RttThresh /=3D 8 To me, it makes good sense without the clamping. In english, that would be = just: =A0=A0=A0 "if queuing due to slow start increases the min RTT by more than = 12.5% from one round to the next, exit slow start" However, taking the default values of the constants, the clamping expressio= n adds the following conditions: #1) If the base RTT is lower than 32ms, the queue has to be greater than 4m= s to exit slow start.=20 #2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms= to exit slow start. I.e. at both extremes, the threshold becomes absolute, rather than a fracti= on of the base RTT. Possible rationales for the lower condition when the base RTT is small (#1)= : a) an absolute Qdelay threshold is needed because small delay measurements = are too noisy. b) a relative amount of Qdelay would become smaller than humans can perceiv= e, so an absolute floor is sufficient. If (a), this needs to be stated, because different min delay measurement te= chniques can achieve different noise immunity. I don't agree with (b), because unnecessary delay over a sequence of rounds= can accumulate to become perceivable. I can't think of a good rationale for the higher condition (#2). I'm sure i= t was the result of experiments, but it seems to be likely that as base RTT= increases beyond 128ms, Hystart++ would become increasingly sensitive and = exit slow start earlier and earlier - unnecessarily (?). Whatever, the rationale for clamping RttThresh needs to be stated, so admin= s know how to set the parameters for other scenarios (for instance, in DCs = excessive loss would occur with the given defaults). >>>> This is worth discussing in the WG meeting tomorrow time permitting. T= hese are a result of experiments. These are fixed clamps because queue buil= d up at the bottleneck has no direct correlation with the base RTT. I agree= though that stating the justification is useful and we will look into this= for the next revision. We are open to ideas to make these thresholds dynam= ic. =A0 ________________________________________ 5.=A0 Deployments and Performance Evaluations It would be useful to see testbed results separated by whether the bottlene= ck before the flow arrives is empty, full, or part-full of traffic. >>>> This is a good suggestion and can be implemented by staggered start of= multiple flows. We will incorporate these tests.=20 Although Hystart is default enabled in Linux, it is invariably disabled. So= , it's misleading to just say Hystart is default enabled, which implies it'= s widely used, when people clearly find it has problems (which motivates Hy= start++). I found this out through an informal survey I did at the Mar'18 I= ETF in London by asking round the implementers of the most prevalent stacks= (I would name names if I could find the note I later sent to someone or to= some list, but I can't find it - sry). >>>> I'd like some citations on this versus anecdata if possible before I a= dd that caveat to the text. Do large deployments disable this? I haven't co= me across this suggestion in any Linux tuning guides to date.=20 ________________________________________ Regards Bob On 12/07/2021 16:46, mailto:internet-drafts@ietf.org wrote: A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the TCP Maintenance and Minor Extensions WG of= the IETF. =A0=A0=A0=A0=A0=A0=A0 Title=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : HyStart++: Modi= fied Slow Start for TCP =A0=A0=A0=A0=A0=A0=A0 Authors=A0=A0=A0=A0=A0=A0=A0=A0 : Praveen Balasubrama= nian =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= Yi Huang =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= Matt Olson =A0 Filename=A0=A0=A0=A0=A0=A0=A0 : draft-ietf-tcpm-hystartplusplus-02.txt =A0 Pages=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : 8 =A0 Date=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 : 2021-07-12 Abstract: =A0=A0 This doument describes HyStart++, a simple modification to the slow =A0=A0 start phase of TCP congestion control algorithms.=A0 Traditional slo= w =A0=A0 start can cause overshotting of the ideal send rate and cause large =A0=A0 packet loss within a round-trip time which results in poor =A0=A0 performance.=A0 HyStart++ is composed of the delay increase variant = of =A0=A0 HyStart to prevent overshooting of the ideal sending rate, while als= o =A0=A0 mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdatatra= cker.ietf.org%2Fdoc%2Fdraft-ietf-tcpm-hystartplusplus%2F&data=3D04%7C01%7Cp= ravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af9= 1ab2d7cd011db47%7C1%7C0%7C637625654367629970%7CUnknown%7CTWFpbGZsb3d8eyJWIj= oiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata= =3DwnozkJrwBmG2cbtSB39sg0vYr8cP8%2BBqRnn7bNh6lbM%3D&reserved=3D0 There is also an htmlized version available at: https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fdatatra= cker.ietf.org%2Fdoc%2Fhtml%2Fdraft-ietf-tcpm-hystartplusplus-02&data=3D04%7= C01%7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86= f141af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d= 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&= sdata=3DSOKqs1Y%2Bi5cNPnn4rv1jlP7rRR%2Fy1iFs9eFlMtxd2ic%3D&reserved=3D0 A diff from the previous version is available at: https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fwww.iet= f.org%2Frfcdiff%3Furl2%3Ddraft-ietf-tcpm-hystartplusplus-02&data=3D04%7C01%= 7Cpravb%40microsoft.com%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141= af91ab2d7cd011db47%7C1%7C0%7C637625654367639937%7CUnknown%7CTWFpbGZsb3d8eyJ= WIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdat= a=3DGlGu0C5KV8%2BdYbVeoJzOszS3YmDCnqf5L5GNBEK9eWY%3D&reserved=3D0 Internet-Drafts are also available by anonymous FTP at: https://nam06.safelinks.protection.outlook.com/?url=3Dftp%3A%2F%2Fftp.ietf.= org%2Finternet-drafts%2F&data=3D04%7C01%7Cpravb%40microsoft.com%7Cea743d922= a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625= 654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC= JBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=3DENURbF%2FuD14Os8t2xQ9oNQVCDITag= h%2FAfznG30S%2BjTw%3D&reserved=3D0 _______________________________________________ tcpm mailing list mailto:tcpm@ietf.org https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fwww.iet= f.org%2Fmailman%2Flistinfo%2Ftcpm&data=3D04%7C01%7Cpravb%40microsoft.com%7C= ea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0= %7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV= 2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=3DpXpAocP3fib4kNDsuwgDZA= qljB0TUWWJQB4RDWSkT3U%3D&reserved=3D0 --=20 ________________________________________________________________ Bob Briscoe=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 https://nam06.safelinks.protection.outlook.com/= ?url=3Dhttp%3A%2F%2Fbobbriscoe.net%2F&data=3D04%7C01%7Cpravb%40microsoft.co= m%7Cea743d922a3b42d4af8f08d94d274b02%7C72f988bf86f141af91ab2d7cd011db47%7C1= %7C0%7C637625654367649886%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQI= joiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=3DrRojz5MR9vnVizaQBe= qy%2FLdUifSnt%2FjBAD1TBXuukLA%3D&reserved=3D0 From nobody Mon Jul 26 18:25:42 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id ABEB23A0D98; Mon, 26 Jul 2021 18:25:40 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.451 X-Spam-Level: X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CB9eK-I6dN5U; Mon, 26 Jul 2021 18:25:36 -0700 (PDT) Received: from NAM06-DM3-obe.outbound.protection.outlook.com (mail-dm3nam06on0723.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe56::723]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4A30F3A0D97; Mon, 26 Jul 2021 18:25:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=I4Jiqnx+PXTPLOZqR3vsrmfRBZlrPBn4FgLOg1T7nfbLVVVRv0OYSh9zaXbXiP4S8poIryVgusoAPSF8o5hgnm7rp0XiN0gacFTADyMfYgieaLqwHys24wEYArPirOUXF/wrw+P9X+lYSgHzFWuY1EwhqVGekvD6cL1/g/omgCNH9K184LluEOxpt9yIuPEFMOkX5e0hTwkvhfg/I43QTAhbjcu6Hz/0sOQurK4JakdMqS2w7f4uJh+W1W4piTH6JodAM+0hP9YY1F7fiGdOuYgYYe6KC9fZTwxyNALBSVVcGrClivDDozrBBe3BHSYR+40BOnR85xSe84aWaoVkWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XNqwp83mCQqiFEkAA4FIV6pQ1ReiSAOAuV7zG5UdiGM=; b=Sdh9iOaLdTpL8ga0gYwTqaP7iWi5x1y4aaXUCJsSnPkEK+P+5hzFEtNmjphAnmfro2AFmksWNoOxq3nTfWXKe619d3UaPJC7dPhf2JMV8PTSQk8MICaPq1KbyH5FAbKgY8g3JAbTLq4igXyaJKu537W65LY7je4Es2olRK0hfQ9NxXvD80RstRt/9H40eTFe1fD7B/jNGPzNr/98ZSrtO+2uJNlQlv6lhLgGOSkTbOHnb6iJivvw3paZ5eMQs2QEgd9H9GzrP5ZeBKFakg9pgMt/fdKBTCnT6y3rqgyWLl5A3G17CI56R9q1tnCYtmvDE22+60U4M79ipeFxbmOIFA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XNqwp83mCQqiFEkAA4FIV6pQ1ReiSAOAuV7zG5UdiGM=; b=hQXc4tQrVA9JCFSEDxN3SIJm/DEwdn/WIyWcMcsIfNrMGxFeYk2Ula8ZLOyQaizCvdhCQIsbpmfYgJv7XuwYw41wHrt8zJ0f/MaJuDCC1QFfROoM4m7UpKJMN2wdEr5HvChhHdETuVdtvw63YCUjRuO1lkr5MO2wJQsRDkph7t4= Received: from PH0PR00MB1030.namprd00.prod.outlook.com (2603:10b6:510:48::7) by PH0PR00MB1029.namprd00.prod.outlook.com (2603:10b6:510:48::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4408.0; Tue, 27 Jul 2021 01:24:37 +0000 Received: from PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88]) by PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88%5]) with mapi id 15.20.4408.000; Tue, 27 Jul 2021 01:24:37 +0000 From: Praveen Balasubramanian To: "ietf@bobbriscoe.net" CC: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Thread-Topic: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (editorial) Thread-Index: AQHXfxBnggfBtbS3XEyZO+saZFOIL6tWABFQ Date: Tue, 27 Jul 2021 01:24:36 +0000 Message-ID: References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <81092896-921b-9563-42b5-e9f412d5782d@bobbriscoe.net> In-Reply-To: <81092896-921b-9563-42b5-e9f412d5782d@bobbriscoe.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=32baa8c7-012e-4658-bc2a-8c65058faf9d; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-27T00:34:02Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: bobbriscoe.net; dkim=none (message not signed) header.d=none;bobbriscoe.net; dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4aa0c5ae-6b3a-4ea6-eb6c-08d9509d4c70 x-ms-traffictypediagnostic: PH0PR00MB1029: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wle0/bjSrAYd3VgChOwwHFTeWvtIcPayUXN6nRQOJ9H09m2pskk6QwzBZbkZXYQ4RD+CBIt+ySG+Bywlo5/RJ5lnGlfAU1pLvzuYLusVVb8biGYOcbEmDGoJVzdCn4rYXCQ7Kd3qg8CoVerrl21jh6e7CWMhrPK7suKPk22bm6awoXzgwqUEEP7KBjqlGPCN3bNGZnGHlcvs1WVhjpCZjysH95vOGsT2gMLPR68zNW6PB9HpiXdbLgwVskVik73j8ZHdWXLRFAqoK5JYN7GjmcIR5LkLKoqLrjxuj80GsFohAjklWuZjePD76888HfERdgpyxYPrs1F9TGy5kICbUft2p0rbbqpjDM0ixrvTcbmt70TzKIwWvWZkd2dbREHdMjXeLvDQbNh9kkXF44SfzhM8uB8192u8Yerchx2aDqaJuRp2KuheWcaBWV0uRXpRovAZe56qmWrfi4CwSiVKU4bLr09ExsIFHBHxlJoMBvxGzaiQNllr1NIYf2feXTHTcNFAvLMJz8nCT7jNfSsqp3E6YrH3CxmTskwakb7ari0tf0sI6BZN+k9j4ndLXFtPOe4Owtegn6nAhSVwqi2Q5FKofxFnZX3NqHYEcvGboh8tC+LeyUk1tWNT03POUJOcwIMIBo52T/E8Go0nZQQaLHIkr/tDejCoAWJsSeLFmlfNgfuaCQEtPDme15Z8umUjB5/Dq+kw49F+q6N2K3OLygCaUkWVYwkS5G8GRxBs9QwnBzjeqLXl3IqPwZFTi5ZX1z69skTQtWvlsUNrHoMZVESYLLlMjRRho8ExQIQEeQP7LZYrEJadDnGvPTKTznQjiBEgzt1yI+iiH8TF0EyFmQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR00MB1030.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(66556008)(54906003)(64756008)(66476007)(508600001)(66446008)(8676002)(38100700002)(122000001)(86362001)(52536014)(186003)(66946007)(316002)(8990500004)(83380400001)(5660300002)(966005)(33656002)(76116006)(2906002)(166002)(10290500003)(6916009)(6506007)(82960400001)(82950400001)(55016002)(71200400001)(4326008)(9686003)(7696005)(53546011)(8936002)(66574015)(38070700004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?0V/P6a9CWR9E9i6ZXM+TGGvjVx1ZgfdtTaXZA+EpSOhVFFu4kFQPP6wqzBIW?= =?us-ascii?Q?FlDFJxCg2EcVGUdAs+EDepo9ff2IKG0A8lzd++g1mnU1bbd8uNS1yq6vcibx?= =?us-ascii?Q?nt5RRTdiAK5KaKvjzx4uaR6rhNrpIlElWJRROdh8IVT5kihza8OoVQ2ks/U3?= =?us-ascii?Q?2T2xr4of+tbBX6RNbu/YgNZbg+soD5DWYX5nXak6O6sdj6wpN+Jk8a6U7kJ+?= =?us-ascii?Q?FPFXxKCwlRSAvtjKtj9/B8z28bxXqJrx6Je6lZXtx97KqEo6ZZQ0tX4814/7?= =?us-ascii?Q?1cC/uvEYK9tIzsKIqyvFwRTltSsiE0/mnOq4oLM8BmkNlpS/VsZ3G9KDUqW8?= =?us-ascii?Q?P3KdBuk+Ofy4yxZL+ZRP0zTJ0MzwxdlmoD++Ij6DwdmJKOfaqHvt5Us7utRj?= =?us-ascii?Q?IKp7DaBacwLESOY79kdkF08aoT20oQ2TTIToVgSilQoHE5GDZr+EQ3PtHDcH?= =?us-ascii?Q?/GX/J/MAVPxAdARaHPuWjUsjaFu7JE8LY9/sQDSsrjceSRAfOZkwrr54Rjqm?= =?us-ascii?Q?4/ySK93ZXNwoBp14eV8heod4bpnYOSkYbD51Vik6n39GQACyfBI9IaF8W/Ag?= =?us-ascii?Q?SnnqKXj0ONYVeCmPzFSSccbEO3XJjEDaq2j/XRX/h0dneitd6m/7+bGhCnlZ?= =?us-ascii?Q?L/ATf7YbRSZlZxuLIBID02mvVN+Ke3ET8JbdPbK7wDYKSwxce7FI+kMPerv/?= =?us-ascii?Q?Qzp16LLiah2E10Edekrl4r9+qFR/kr6K9w/qURCHNWSbJFp03f8Q7cI8nZVI?= =?us-ascii?Q?PC9405Ddzdno+Nb9TXPb/xI6WGoCek2ZtdncpudN1g0X7d0Xft48wpY900up?= =?us-ascii?Q?kXV7mxK78zbpMczTufJxUFheYUjPUhlfpD3P08cF1PUeAd3kT9YyOnRdvpOA?= =?us-ascii?Q?QUCRJfnR9pvMq4frbS1Y0CbMLx/fFWBwxIGF14BfHSp1I8ryytGV19FFWeOe?= =?us-ascii?Q?tc5JqFEXU2CVK4Cu9worIYGFCE0bvxysh6K2+vy9QnQmajEPDwxpGVlkTC/0?= =?us-ascii?Q?KGYWEZ+I2seAWIFr+aZJsDnejujmVSNCL32NxrWpiMPcCHAYMtX4+sxGIsM0?= =?us-ascii?Q?SnqFXu62g2nj6IJWHjXYM0FZCP+bZ6Le9dMbmTPFrIkD9SLXLYCan8tAQwkc?= =?us-ascii?Q?m9V0JiQQRdRNkLdwny9VBua3Egcb7EqTjaAfVafAOzTOrtNZPFDZg6XTGX6I?= =?us-ascii?Q?QZypZ25o+nqLZSPtBBek1fzF/zWPXGfZSrE69GtnD5Ir2E8V2t5Qrr2U6CD8?= =?us-ascii?Q?GzD54AmnZnrnH0JMtJMq6V0LPJ+f383+LEWPf0RfTX0HqWm1W6h/xHIbOlQW?= =?us-ascii?Q?3RRS/kHK5VU2JzRVM2h8loDFTW7M/XwaurMej0o3YyyyNKHsmqQ3r8Lo5p05?= =?us-ascii?Q?xSWEfjVX9br/Ta3HGjWs5x5WmsmP?= x-ms-exchange-transport-forked: True Content-Type: multipart/alternative; boundary="_000_PH0PR00MB1030985320A58D936DB343E5B6E99PH0PR00MB1030namp_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR00MB1030.namprd00.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4aa0c5ae-6b3a-4ea6-eb6c-08d9509d4c70 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jul 2021 01:24:36.9772 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: l3rfBVt3YrFaWs0OIu1lbd9zFNq1gWLwybkzv1Tc2Wf3kIOVbQE93XvvqbUsPhssUjdVTQ3LESnzq5D/2v45WG07XvVyP6GnksFCJ5wfFeo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR00MB1029 Archived-At: Subject: Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (editorial) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 01:25:41 -0000 --_000_PH0PR00MB1030985320A58D936DB343E5B6E99PH0PR00MB1030namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks Bob for the review. We published draft-03 addressing some of your co= mments. I am having issues replying with inline replies to your email so I am copy = pasting. >> s/doument/document/ Fixed all typos except this one. Will fix in next revision. >> Is reducing queuing delay (for other flows) also a goal of Hystart++? It's not an explicit goal. Since this can be used with a loss based congest= ion controller, we cannot make a blanket statement about reducing queueing = delay. >> Last para of intro seems to be about Hystart, and it doesn't make it cle= ar how Hystart relates to Hystart++: Rewrote most of this text to make it clear. >> It seems odd to define some well-known variables (like cwnd), but not ot= hers like SND.UNA, SND.NXT, etc. RFC 5681 doesn't define them either but uses them. >> cwnd: Would be useful to define the units assumed in the pseudocode late= r (bytes), These are verbatim from RFC 5681. We can add a clarification if it helps in= the pseudocode section. >> I suggest: "The choice of value of L is discussed in RFC 3465." Fixed to convey this. >>"normal slow start" Replaced by standard slow start and a reference to RFC 5681. >> The word 'congestion' is ambiguous, because it could include delay incre= ase. How about:"If loss or ECN-marking is observed anytime during slow star= t or CSS, enter congestion avoidance." Fixed. >> I suggest you give the actual deployment times, so these two sentences d= on't keep having to be updated as the draft ages We have made some edits. Will make this section up to date as we get more d= ata on draft-02. >> Also, in Linux, Hystart is only implemented for Cubic Clarified this and added a reference to RFC 8312. From: Bob Briscoe Sent: Thursday, July 22, 2021 8:43 AM To: Praveen Balasubramanian Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (edi= torial) Praveen and co-authors. Thank you for hystart++. I've sent a separate posting with technical review comments. Cheers Bob =3D=3DEditorial comments=3D=3D Abstract s/doument/document/ s/overshotting/overshooting >>>> Fixed 1. Intro s/pefrormance/performance/ "...lead to overshooting the ideal sending rate and cause significant packe= t loss" Also queuing delay? Is reducing queuing delay (for other flows) also a goal= of Hystart++? Last para of intro seems to be about Hystart, and it doesn't make it clear = how Hystart relates to Hystart++: * "HyStart++ adds delay increase..." Adds to what? * "After the HyStart delay algorithm finds an exit point, a Conservative Sl= ow Start (CSS) phase is used" But what about Hystart++? does it do CSS too? Final sentence is not related to the rest of the para about Hystart, so sho= uld be a separate para. 3. Definitions It seems odd to define some well-known variables (like cwnd), but not other= s like SND.UNA, SND.NXT, etc. cwnd: Would be useful to define the units assumed in the pseudocode later (= bytes), because some implementations use segments. It's confusing for cwnd to use bytes but LOW_CWND to use segments. 4.2 Algorithm Details "The choice of value of L is up to the implementation." I suggest: "The choice of value of L is discussed in RFC 3465." Reason: RFC3465 doesn't say it's completely up to the implementation - it g= ives recommendations and advice. "normal slow start" I appreciate you were asked to use this word, but... I suggest it's best to= define this phrase because, for instance, the normal slow start for Cubic = (Hystart) is different from the normal slow start for Reno. It's also a pro= blem finding a good word that means the pre-existing thing. 'Reno slow star= t' would be more precise than 'normal slow start' (if you mean the one in R= eno). Because Cubic is more normal than Reno these days. And saying 'standa= rd' will soon be ambiguous, because Cubic is on its way onto the standards = track. "If congestion is observed anytime during slow start or CSS, enter cong= estion avoidance." The word 'congestion' is ambiguous, because it could include delay increase= . How about: "If loss or ECN-marking is observed anytime during slow start or CSS, e= nter congestion avoidance." 5. Deployments and Performance Evaluations As of the time of writing, HyStart++ has been default enabled for all TCP connections in Windows for two years. The original Hystart has been default-enabled for all TCP connections in Linux TCP for a decade. I suggest you give the actual deployment times, so these two sentences don'= t keep having to be updated as the draft ages. Also, in Linux, Hystart is only implemented for Cubic (the default CC modul= e), so Hystart is not default if a non-default CC module is in use. How abo= ut: In Linux, the original Hystart has been default enabled for TCP connections using the default congestion control module (Cubic) for a decade. Regards Bob On 12/07/2021 16:46, internet-drafts@ietf.org wrote: A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the TCP Maintenance and Minor Extensions WG of= the IETF. Title : HyStart++: Modified Slow Start for TCP Authors : Praveen Balasubramanian Yi Huang Matt Olson Filename : draft-ietf-tcpm-hystartplusplus-02.txt Pages : 8 Date : 2021-07-12 Abstract: This doument describes HyStart++, a simple modification to the slow start phase of TCP congestion control algorithms. Traditional slow start can cause overshotting of the ideal send rate and cause large packet loss within a round-trip time which results in poor performance. HyStart++ is composed of the delay increase variant of HyStart to prevent overshooting of the ideal sending rate, while also mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hystartplusplus-02 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ _______________________________________________ tcpm mailing list tcpm@ietf.org https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --_000_PH0PR00MB1030985320A58D936DB343E5B6E99PH0PR00MB1030namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks Bob for the review. We published draft-03 add= ressing some of your comments.

 

I am having issues replying with inline replies to y= our email so I am copy pasting.

 

>> s/doument/document/

Fixed all typos except this one. Will fix in next re= vision.

 

>> Is reducing queuing delay (for other flows)= also a goal of Hystart++?

It’s not an explicit goal. Since this can be u= sed with a loss based congestion controller, we cannot make a blanket state= ment about reducing queueing delay.

 

>> Last para of intro seems to be about Hystar= t, and it doesn't make it clear how Hystart relates to Hystart++:

Rewrote most of this text to make it clear.

 

>> It seems odd to define some well-known vari= ables (like cwnd), but not others like SND.UNA, SND.NXT, etc.

RFC 5681 doesn’t define them either but uses t= hem.

 

>> cwnd: Would be useful to define the units a= ssumed in the pseudocode later (bytes),

These are verbatim from RFC 5681. We can add a clari= fication if it helps in the pseudocode section.

 

>> I suggest:     "The cho= ice of value of L is discussed in RFC 3465."

Fixed to convey this.

 

>>"normal slow start"
Replaced by standard slow start and a reference to RFC 5681.

 

>> The word 'congestion' is ambiguous, because= it could include delay increase. How about:"If loss or ECN-marking is= observed anytime during slow start or CSS, enter congestion avoidance.&quo= t;
Fixed.

 

>> I suggest you give the actual deployment ti= mes, so these two sentences don't keep having to be updated as the draft ag= es

We have made some edits. Will make this section up t= o date as we get more data on draft-02.

 

>> Also, in Linux, Hystart is only implemented= for Cubic

Clarified this and added a reference to RFC 8312.

 

From: Bob Briscoe <ietf@bobbriscoe.net>=
Sent: Thursday, July 22, 2021 8:43 AM
To: Praveen Balasubramanian <pravb@microsoft.com>
Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-= 02 (editorial)

 

Praveen and co-author= s.

Thank you for hystart++.
I've sent a separate posting with technical review comments.

Cheers


Bob

=3D=3DEditorial comments=3D=3D

 

Abstract

s/doument/document/ s/overshotting/overshooting

>>>> Fi= xed

 

1. Intro

s/pefrormance/perform= ance/

"...lead to overshooting the ideal sending rate and cause significant = packet loss"
Also queuing delay? Is reducing queuing delay (for other flows) also a goal= of Hystart++?

Last para of intro seems to be about Hystart, and it doesn't make it clear = how Hystart relates to Hystart++:
* "HyStart++ adds delay increase..."
   Adds to what?
* "After the HyStart delay algorithm finds an exit point, a Conservati= ve Slow Start (CSS) phase is used"
  But what about Hystart++? does it do CSS too?

Final sentence is not related to the rest of the para about Hystart, so sho= uld be a separate para.

3. Definitions

It seems odd to defin= e some well-known variables (like cwnd), but not others like SND.UNA, SND.N= XT, etc.

cwnd: Would be useful to define the units assumed in the pseudocode later (= bytes), because some implementations use segments.
It's confusing for cwnd to use bytes but LOW_CWND to use segments.

4.2 Algorithm Details


    "The choice of value of L is up to the implementati= on."
I suggest:
    "The choice of value of L is discussed in RFC 3465.= "
Reason: RFC3465 doesn't say it's completely up to the implementation - it g= ives recommendations and advice.

"normal slow start"
I appreciate you were asked to use this word, but... I suggest it's best to= define this phrase because, for instance, the normal slow start for Cubic = (Hystart) is different from the normal slow start for Reno. It's also a pro= blem finding a good word that means the pre-existing thing. 'Reno slow start' would be more precise than 'norm= al slow start' (if you mean the one in Reno). Because Cubic is more normal = than Reno these days. And saying 'standard' will soon be ambiguous, because= Cubic is on its way onto the standards track.


    "If congestion is observed anytime during slow star= t or CSS, enter congestion avoidance."
The word 'congestion' is ambiguous, because it could include delay increase= . How about:
    "If loss or ECN-marking is observed anytime during = slow start or CSS, enter congestion avoidance."

5.  Deployments and Performance Evaluations 

   As of the time of writing, HyStart++ has been defaul=
t enabled for all
   TCP connections in Windows for two years.  The origi=
nal Hystart has
   been default-enabled for all TCP connections in Linux TCP=
 for a
   decade.


I suggest you give the actual deployment times, so these two sentences don'= t keep having to be updated as the draft ages.
Also, in Linux, Hystart is only implemented for Cubic (the default CC modul= e), so Hystart is not default if a non-default CC module is in use. How abo= ut:

In Linux, the original Hystart has been default enab= led for
TCP connections using the default congestion control module (Cubic)
for a decade.


Regards


Bob

On 12/07/2021 16:46, internet-drafts@ietf.org wrote:

A New Internet-Draft is available from the on-line Internet-Drafts dir=
ectories.
This draft is a work item of the TCP Maintenance and Minor Extensions =
WG of the IETF.
 
        Title   &nbs=
p;       : HyStart++: Modified Slow Start for=
 TCP
        Authors   &n=
bsp;     : Praveen Balasubramanian
           &nbs=
p;            &=
nbsp; Yi Huang
           &nbs=
p;            &=
nbsp; Matt Olson
  Filename        : draft-ietf=
-tcpm-hystartplusplus-02.txt
  Pages         &nbs=
p; : 8
  Date          =
;  : 2021-07-12
 
Abstract:
   This doument describes HyStart++, a simple modification t=
o the slow
   start phase of TCP congestion control algorithms.  T=
raditional slow
   start can cause overshotting of the ideal send rate and c=
ause large
   packet loss within a round-trip time which results in poo=
r
   performance.  HyStart++ is composed of the delay inc=
rease variant of
   HyStart to prevent overshooting of the ideal sending rate=
, while also
   mitigating poor performance which can result from false p=
ositives.
 
 
The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartpl=
usplus/
 
There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm=
-hystartplusplus-02
 
A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hysta=
rtplusplus-02
 
 
Internet-Drafts are also available by anonymous FTP at:

ftp://ftp.ietf.=
org/internet-drafts/


 


 


_______________________________________________


tcpm mailing list


tcpm@ietf.org


htt=
ps://www.ietf.org/mailman/listinfo/tcpm






-- 
________________________________________________________________<=
/o:p>
Bob Briscoe          =
;            &n=
bsp;        http://bobbriscoe.net/
--_000_PH0PR00MB1030985320A58D936DB343E5B6E99PH0PR00MB1030namp_-- From nobody Mon Jul 26 18:31:27 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7BD493A0DF1; Mon, 26 Jul 2021 18:31:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.452 X-Spam-Level: X-Spam-Status: No, score=-2.452 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RJE68Suy23Jk; Mon, 26 Jul 2021 18:31:20 -0700 (PDT) Received: from NAM06-DM3-obe.outbound.protection.outlook.com (mail-eopbgr640104.outbound.protection.outlook.com [40.107.64.104]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 57E873A0DEA; Mon, 26 Jul 2021 18:31:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gtoWQlNivILx7najgBAVl2AUA7OXi1wKRuhlmYsIF+cVM0VkNyY7WSP/wOEK4im3VWmVXdhr0RQbcdKxs0uiVjgR7P7K5vyv1ZOHIgWMnxYNzIzPNdSjGCIZe6UevRMhCnbf3Eab0hlried7gzZ0iA5KM5kAXRpwwx8tbMtoboKvw84YLCbZns8ugTGreWXotL/IBbL8JWScUg/w1WS7vWSetQPfiY2qAtlT+8foEEUZ4gvDC1E2CFyXhaZ1/ealdO4mEaxGA9HjaZFjVUqZW1KgBMUakaywVAvFQIDBRwuXpJz/4VBbA0+F653/wTFhtABzsrydk2iv8QwGe9bwVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H329IWuS25+XNMbNC6jqvRmxhPcLkE69sKylEEK0AZc=; b=jnT70hZrPEQeM55z3cjnZ89e3RqyBShyejgC+tAaeVPZhG8tSo/BcHjyrnkmTS7N85UGmHSLJGcvoq7ohZEhGRkI2vLuBF/EQPYLxmf9UZ9NFiBTh6A8YK4kwSRSEIinm8vLiRB5xrjO6VJ8Nh0EJXl67R1Owk0SZYvWpuxAaCUaUiAFk+bqcuoeeTE/cMJiv3rQLnC36ue4p4MXkR5ZAlBjap1dYX+SR28NPOsp+iDErQevhCtNIRktpS9HWuxSfobWNmGGh7cwDQFzbyRnbCAq1q5tpDGzHVW0btaE6LDqaoNc/Acvp5wqbM1sqM2D+7LXg5IFaB1NRWZVGJUMUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H329IWuS25+XNMbNC6jqvRmxhPcLkE69sKylEEK0AZc=; b=C25EE9KMbnR9qN0ZgIT5n/t0r8qUsE11bhOZpN7ZjC+drmPDkZZK6Bu/lK+SUhn3dEnevH/Dyw4oLML3K283ZRRjgv+3Cif3ylcviJwCluVL3wdpv396NEvKJWQ2tyoGQ33GZWB3pJAUu1cBr+DmX0JEJpQm4MndLnts2yf6DP4= Received: from PH0PR00MB1030.namprd00.prod.outlook.com (2603:10b6:510:48::7) by PH0PR00MB1166.namprd00.prod.outlook.com (2603:10b6:510:9c::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4408.0; Tue, 27 Jul 2021 01:30:00 +0000 Received: from PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88]) by PH0PR00MB1030.namprd00.prod.outlook.com ([fe80::70c6:3116:b603:3f88%5]) with mapi id 15.20.4408.000; Tue, 27 Jul 2021 01:29:59 +0000 From: Praveen Balasubramanian To: "ietf@bobbriscoe.net" CC: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Thread-Topic: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) Thread-Index: AQHXfxBhDL5MNyxppUKpsFE4e4Tyu6tWD5vA Date: Tue, 27 Jul 2021 01:29:59 +0000 Message-ID: References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> In-Reply-To: <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=c084f2ed-e538-4bf3-9751-c1514d38eb12; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-27T01:29:39Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: bobbriscoe.net; dkim=none (message not signed) header.d=none;bobbriscoe.net; dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1fd438b5-1318-41f8-9077-08d9509e0cc3 x-ms-traffictypediagnostic: PH0PR00MB1166: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: njTetniOYCcrVfq0LMDkuYWgAwrV89+z65C6H8wOG9+WyQdpwLWApE9lGuSfo5kybQHwqvbmqDNBPpxckehn60pwyx9keHe3JeSH9o3kqLR5VlQZuANLn+xDSSi5Fcze5NoaySRy93kQDAcVtLwBTLracdeXiNfaEJnJ/98lJ1JPhX/IKCylljq7WPzR/cAm9+VsI3i/8EFeMLjo+c3bkNtB5VkhGsHX+nWsBVfYQUrhulGDUmPuovu/1WafikTioQ1D5VoFDaMXNq10+gSTXVXVSTEpFfsnJeMnS+0oQ+Nem1XsvcCVmvFHKW7/Yzw6HGssY6SWFmIoUleOWHF6TmUOZPgKz21MnawgbKryvdxFR1TKJyVrHbCzd5oGlOz1/xb//cVg3wrxZq9Bjm2USeJ4UFxjETidl9Apqa6Cfyd3mCCtVDA6pXjFrgcb/yXymqVBnenLPPUUkcH4mO+XGoSbR6rz8danPqb/Xdo+1FbhJJtMQAp+jV/PwSnwijwpa3KpoJvYXiy5qi42F9UFMGTdTpKBO7qzpt/smcsOrc2QYjbZO6sTye5bhlbeILFtewJhobj7/FFVmo+bwaxab162Jjj/1iEF5YWA2fx8BMgRnbiGa7xt4NZA74fNuxqda/r1BQRHcsj4JHsxkEnkGQBc5hFJThNN86bD8jF+sx6ApxMF+glIvYbg6wssjoM/6Ue/oWoI9o5E/n7wqAlQg6E6d+426TfqTLpCnAt3asx2qOvOaW8g1Cv+F/A+iABm67dNjk4tjy1LqwUUpljhA9pJOwo/MdAJuvCOnzWdffpOfJOQtGxVuJvfnNasdMe5kmLSud979BRAgCe9lJPSBA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR00MB1030.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(55016002)(71200400001)(9686003)(4326008)(6506007)(82950400001)(82960400001)(8936002)(66574015)(53546011)(7696005)(316002)(52536014)(186003)(66946007)(86362001)(508600001)(66446008)(8676002)(66556008)(54906003)(64756008)(66476007)(122000001)(38100700002)(8990500004)(10290500003)(6916009)(166002)(5660300002)(83380400001)(966005)(76116006)(2906002)(33656002)(38070700004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?z6dw0yR4LWFP8DDMAZo3p2RYeUAsqjfd/66+aOeyJLYJx235hnk5pGRRCPkE?= =?us-ascii?Q?XHEddY4PF4Dg/zoSPx1X7+iQntawIXKLe9FeR6BVmM8rm4Qfy5tzNsnlXsHI?= =?us-ascii?Q?/B+ohhskseIBe2HxqOaM/VJypOwcamqc+fJmMrieH0wzNDyuVsHYl9VOZGHq?= =?us-ascii?Q?pJqELW0uImmNYX7Fa/m0NdP5vFRP0YGu6NyXUoAF/8N/i1WhT0jidlO2lYas?= =?us-ascii?Q?KCWbtmiOlcse7PHR9DWowwNp6+ke4bRa+MUWtPBW/tCpmEdSHeO6knJ/F+eC?= =?us-ascii?Q?gZolkagEBmITqmClFQp2cWMFeDIeW2ZeY1RmA5ODsxqKYCLNdAWlFpiTktgH?= =?us-ascii?Q?p/MWuQDxL63VDoOZs4a2Ny0MINVvXoqV9gjKzKE19K2O8f4V9BydDKAXWcQX?= =?us-ascii?Q?iYQAmHGAx3a+K5wfYXxixPNS1tIHx5fsbca319tzXPRWJN+P+6kGv9MO8j/Y?= =?us-ascii?Q?FIy1g8tXjTam2MVaUqa1q9NZNV3a3x+kY1b6Nlz7YrJcL6bgxUcML+ikXrym?= =?us-ascii?Q?API+ZeZJVdSvSPvYUIpJ/KAXFG6EPePif3Us6egWf+0x7L12BrHMqNo4mP0y?= =?us-ascii?Q?Z5qZ7SxXHcH+6yvhQQQPxzNuEaHP9tG0iT6fProXiz0r+70TQ+0w5en1d0MU?= =?us-ascii?Q?HXsmvo0xLrZzNh5Ltz8fryer9QFKBPGIPFPJwx8lP+4BfPgNirbGu2qf7QOt?= =?us-ascii?Q?EB8QK5TKhygmtzaAwVRd9pHfhQUv7GW4WRyt6pkaP6MhS27/VrXT2f+20ExI?= =?us-ascii?Q?QamoUgRAvtJwvMOgrk5mkp0W7RmHoR/GQJ6Y2yK+DGYJJg7gpU+y3JMHok/M?= =?us-ascii?Q?1oM2N2E4ApTy8lOYvmOGhoH+5Yos/RkBOSw89lhXxYzHv+EN6maf4gbGsewg?= =?us-ascii?Q?8qlNRc6jNElTkaixbL6cI/zdtLkUrvPG6O835UYlIpfim1NojFqR6Lluu05x?= =?us-ascii?Q?1gawtdmEIIFd5UpKzfTy5C5B2f6JF+K7AjUwH6nIQc5HEE7NIfn39RmygQF6?= =?us-ascii?Q?APx5MI4WUOATEmacBepkL1jF/z6sPQny/7fZhiTSyyuFDN37ET7noVZKqvGY?= =?us-ascii?Q?wI9sODfZT4+sn6PH9tx6rY/13kb1tJ/NOiwTIb0ol8zfHt3i1+2TZU1vnP4w?= =?us-ascii?Q?5tpC3bh+Ce8dMyolccBG/xZHqY55pfSHPt4SbnfJdRaF+5r27Ca8f9XPv/S9?= =?us-ascii?Q?gRsxXrysZxODsILFD3zvnOwvu5XYentGtO1uhEI/qizBYJDY+L99AagI1ANQ?= =?us-ascii?Q?UR+EbtHbTxybWWSQdj0/mN4cYhr43nWkOSRIlBG+HH03/55MTJBcPy4Qk9nP?= =?us-ascii?Q?ohuoeq58NC+pTLPmrw7swKAkFfdjmlYt01B/7R1nFl+Xo+zuWzzBvMYWHiwp?= =?us-ascii?Q?ojEu2hlelv5DAuyuCCjr5eBAqk5+?= x-ms-exchange-transport-forked: True Content-Type: multipart/alternative; boundary="_000_PH0PR00MB10305B7C7C9A60BCEBB47D88B6E99PH0PR00MB1030namp_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR00MB1030.namprd00.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1fd438b5-1318-41f8-9077-08d9509e0cc3 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jul 2021 01:29:59.5478 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +2LJzIhuA+I56/ryssavQC+VfBdSaq7lnx1ecK3z2Tl5UwiRfRZ2CER5YuH8EL9Bp7Vgwe1OXN1daKpeqhN9n/12NyfC0qnUClIaXtFB90U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR00MB1166 Archived-At: Subject: Re: [tcpm] [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (technical) X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 01:31:26 -0000 --_000_PH0PR00MB10305B7C7C9A60BCEBB47D88B6E99PH0PR00MB1030namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks Bob for the review. We published draft-03 addressing some of your co= mments. I am having issues replying with inline replies to your email so I am copy = pasting. >> It might be worth stating that using sequence numbers is the /best/ way = to measure rounds Added a recommendation for using the sequence number approach. >> Need to make it clear this RTT is based on the latest ACK'd packet Clarified. >> Whatever, the rationale for clamping RttThresh needs to be stated, so ad= mins know how to set the parameters for other scenarios (for instance, in D= Cs excessive loss would occur with the given defaults). This is worth discussing in the WG meeting tomorrow time permitting. These = are a result of experiments. These are fixed clamps because queue build up = at the bottleneck has no direct correlation with the base RTT. I agree thou= gh that stating the justification is useful and we will look into this for = the next revision. We are open to ideas to make these thresholds dynamic. >> Although Hystart is default enabled in Linux, it is invariably disabled I'd like some citations on this versus anecdata if possible before I add th= at caveat to the text. Do large deployments disable this? I haven't come ac= ross this suggestion in any Linux tuning guides to date. From: Bob Briscoe Sent: Thursday, July 22, 2021 8:42 AM To: Praveen Balasubramanian Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-02 (tec= hnical) Praveen and co-authors. Thank you for hystart++. This one's my technical review. I've sent my editorial review in a separate= posting. I actually reviewed an earlier draft (balasubrmanian-02), but never got rou= nd to writing it up for you. I've just reviewed the latest (ietf-02) draft,= and incorporated my comments about the older revision where still relevant= . Cheers Bob =3D=3D=3DTechnical Comments=3D=3D 4.2. Algorithm Details ________________________________ Round can be approximated using sequence numbers as follows: ... At the start of each round during normal slow start and CSS: It might be worth stating that using sequence numbers is the /best/ way to = measure rounds, not just one way. You want every round to contain at least = one of the ping-ponged segments that were grandfathered by the very first s= egment of the IW (because that will be the segment most likely to have the = min RTT). For instance, if you measure rounds in time, you might underestim= ate the duration of a round. Then, in some rounds, you would risk not seein= g one of this ping-pong series at all. ________________________________ "where currRTT is the RTT sampled from the incoming ACK" Need to make it clear this RTT is based on the latest ACK'd packet, not the= earliest (in contrast to the RTT used for RTO). ________________________________ The somewhat opaque clamping of RTTThresh below needs to be justified: For rounds where cwnd is at or higher than LOW_CWND and N_RTT_SAMPLE RTT samples have been obtained, check if delay increase triggers slow start exit ... RttThresh =3D clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RT= T_THRESH) if (currentRoundMinRTT >=3D (lastRoundMinRTT + RttThresh)) cssBaselineMinRtt =3D currentRoundMinRTT exit slow start and enter CSS To aid comprehension for the benefit of the list, I'll separate the clamp l= ine into two steps and substitute the default constants: RttThresh =3D clamp(32ms, lastRoundMinRTT, 128ms) RttThresh /=3D 8 To me, it makes good sense without the clamping. In english, that would be = just: "if queuing due to slow start increases the min RTT by more than 12.5% = from one round to the next, exit slow start" However, taking the default values of the constants, the clamping expressio= n adds the following conditions: #1) If the base RTT is lower than 32ms, the queue has to be greater than 4m= s to exit slow start. #2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms= to exit slow start. I.e. at both extremes, the threshold becomes absolute, rather than a fracti= on of the base RTT. Possible rationales for the lower condition when the base RTT is small (#1)= : a) an absolute Qdelay threshold is needed because small delay measurements = are too noisy. b) a relative amount of Qdelay would become smaller than humans can perceiv= e, so an absolute floor is sufficient. If (a), this needs to be stated, because different min delay measurement te= chniques can achieve different noise immunity. I don't agree with (b), because unnecessary delay over a sequence of rounds= can accumulate to become perceivable. I can't think of a good rationale for the higher condition (#2). I'm sure i= t was the result of experiments, but it seems to be likely that as base RTT= increases beyond 128ms, Hystart++ would become increasingly sensitive and = exit slow start earlier and earlier - unnecessarily (?). Whatever, the rationale for clamping RttThresh needs to be stated, so admin= s know how to set the parameters for other scenarios (for instance, in DCs = excessive loss would occur with the given defaults). ________________________________ 5. Deployments and Performance Evaluations It would be useful to see testbed results separated by whether the bottlene= ck before the flow arrives is empty, full, or part-full of traffic. Although Hystart is default enabled in Linux, it is invariably disabled. So= , it's misleading to just say Hystart is default enabled, which implies it'= s widely used, when people clearly find it has problems (which motivates Hy= start++). I found this out through an informal survey I did at the Mar'18 I= ETF in London by asking round the implementers of the most prevalent stacks= (I would name names if I could find the note I later sent to someone or to= some list, but I can't find it - sry). ________________________________ Regards Bob On 12/07/2021 16:46, internet-drafts@ietf.org wrote: A New Internet-Draft is available from the on-line Internet-Drafts director= ies. This draft is a work item of the TCP Maintenance and Minor Extensions WG of= the IETF. Title : HyStart++: Modified Slow Start for TCP Authors : Praveen Balasubramanian Yi Huang Matt Olson Filename : draft-ietf-tcpm-hystartplusplus-02.txt Pages : 8 Date : 2021-07-12 Abstract: This doument describes HyStart++, a simple modification to the slow start phase of TCP congestion control algorithms. Traditional slow start can cause overshotting of the ideal send rate and cause large packet loss within a round-trip time which results in poor performance. HyStart++ is composed of the delay increase variant of HyStart to prevent overshooting of the ideal sending rate, while also mitigating poor performance which can result from false positives. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartplusplus/ There is also an htmlized version available at: https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus-02 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hystartplusplus-02 Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ _______________________________________________ tcpm mailing list tcpm@ietf.org https://www.ietf.org/mailman/listinfo/tcpm -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --_000_PH0PR00MB10305B7C7C9A60BCEBB47D88B6E99PH0PR00MB1030namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks Bob for the review. We published draft-03 = addressing some of your comments.

 

I am having issues replying with inline replies t= o your email so I am copy pasting.

 

>> It might be worth stating that using seq= uence numbers is the /best/ way to measure rounds

Added a recommendation for using the sequence num= ber approach.

 

>> Need to make it clear this RTT is based = on the latest ACK'd packet

Clarified.

 

>> Whatever, the rationale for clamping Rtt= Thresh needs to be stated, so admins know how to set the parameters for oth= er scenarios (for instance, in DCs excessive loss would occur with the give= n defaults).

This is worth discussing in the WG meeting tomorr= ow time permitting. These are a result of experiments. These are fixed clam= ps because queue build up at the bottleneck has no direct correlation with = the base RTT. I agree though that stating the justification is useful and we will look into this for the nex= t revision. We are open to ideas to make these thresholds dynamic. 

 

>> Although Hystart is default enabled in L= inux, it is invariably disabled

I’d like some citations on this versus anec= data if possible before I add that caveat to the text. Do large deployments= disable this? I haven’t come across this suggestion in any Linux tun= ing guides to date.

 

 

From: Bob Briscoe <ietf@bobbriscoe.net>=
Sent: Thursday, July 22, 2021 8:42 AM
To: Praveen Balasubramanian <pravb@microsoft.com>
Cc: tcpm@ietf.org; draft-ietf-tcpm-hystartplusplus@ietf.org
Subject: [EXTERNAL] Full review of: draft-ietf-tcpm-hystartplusplus-= 02 (technical)

 

Praveen and co-author= s.

Thank you for hystart++.
This one's my technical review. I've sent my editorial review in a separate= posting.

I actually reviewed an earlier draft (balasubrmanian-02), but never got rou= nd to writing it up for you. I've just reviewed the latest (ietf-02) draft,= and incorporated my comments about the older revision where still relevant= .

Cheers


Bob

=3D=3D=3DTechnical Comments=3D=3D


4.2.  Algorithm Details

 
 Round can be approximated using sequence numbers as follows:
   ...
   At the start of each round during normal slow start and C=
SS:

It might be worth sta= ting that using sequence numbers is the /best/ way to measure rounds, not j= ust one way. You want every round to contain at least one of the ping-ponge= d segments that were grandfathered by the very first segment of the IW (because that will be the segment most li= kely to have the min RTT). For instance, if you measure rounds in time, you= might underestimate the duration of a round. Then, in some rounds, you wou= ld risk not seeing one of this ping-pong series at all.


"where currRTT is the RTT sampled from the incoming ACK"
Need to make it clear this RTT is based on the latest ACK'd packet, not the= earliest (in contrast to the RTT used for RTO).


The somewhat opaque clamping of RTTThresh below needs to be justified:= 

      For rounds where cwnd is at or higher t=
han LOW_CWND and
      N_RTT_SAMPLE RTT samples have been obta=
ined, check if delay
      increase triggers slow start exit<=
/o:p>
      ...
            Rtt=
Thresh =3D clamp(MIN_RTT_THRESH, lastRoundMinRTT / 8, MAX_RTT_THRESH)<=
/o:p>
            if =
(currentRoundMinRTT >=3D (lastRoundMinRTT + RttThresh))
           &nbs=
p;   cssBaselineMinRtt =3D currentRoundMinRTT
           &nbs=
p;   exit slow start and enter CSS


To aid comprehension for the benefit of the list, I'll separate the clamp l= ine into two steps and substitute the default constants:
            RttThres= h =3D clamp(32ms, lastRoundMinRTT, 128ms)
            RttThres= h /=3D 8

To me, it makes good sense without the clamping. In english, that would be = just:
    "if queuing due to slow start increases the min RTT= by more than 12.5% from one round to the next, exit slow start"

However, taking the default values of the constants, the clamping expressio= n adds the following conditions:

#1) If the base RTT is lower than 32ms, the queue ha= s to be greater than 4ms to exit slow start.
#2) If the base RTT is higher than 128ms, the queue only has to exceed 16ms= to exit slow start.

I.e. at both extremes, the threshold becomes absolut= e, rather than a fraction of the base RTT.

Possible rationales for the lower condition when the base RTT is small (#1)= :

a) an absolute Qdelay threshold is needed because sm= all delay measurements are too noisy.
b) a relative amount of Qdelay would become smaller than humans can perceiv= e, so an absolute floor is sufficient.

If (a), this needs to= be stated, because different min delay measurement techniques can achieve = different noise immunity.
I don't agree with (b), because unnecessary delay over a sequence of rounds= can accumulate to become perceivable.

I can't think of a good rationale for the higher condition (#2). I'm sure i= t was the result of experiments, but it seems to be likely that as base RTT= increases beyond 128ms, Hystart++ would become increasingly sensitive and = exit slow start earlier and earlier - unnecessarily (?).

Whatever, the rationale for clamping RttThresh needs to be stated, so admin= s know how to set the parameters for other scenarios (for instance, in DCs = excessive loss would occur with the given defaults).


5.  Deployments and Performance Evaluations

It would be useful to see testbed results separated by whether the bottlene= ck before the flow arrives is empty, full, or part-full of traffic.

Although Hystart is default enabled in Linux, it is invariably disabled. So= , it's misleading to just say Hystart is default enabled, which implies it'= s widely used, when people clearly find it has problems (which motivates Hy= start++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the = implementers of the most prevalent stacks (I would name names if I could fi= nd the note I later sent to someone or to some list, but I can't find it - = sry).


Regards


Bob

On 12/07/2021 16:46, internet-drafts@ietf.org wrote:

 
A New Internet-Draft is available from the on-line Internet-Drafts dir=
ectories.
This draft is a work item of the TCP Maintenance and Minor Extensions =
WG of the IETF.
 
        Title   &nbs=
p;       : HyStart++: Modified Slow Start for=
 TCP
        Authors   &n=
bsp;     : Praveen Balasubramanian
           &nbs=
p;            &=
nbsp; Yi Huang
           &nbs=
p;            &=
nbsp; Matt Olson
  Filename        : draft-ietf=
-tcpm-hystartplusplus-02.txt
  Pages         &nbs=
p; : 8
  Date          &nbs=
p; : 2021-07-12
 
Abstract:
   This doument describes HyStart++, a simple modification t=
o the slow
   start phase of TCP congestion control algorithms.  T=
raditional slow
   start can cause overshotting of the ideal send rate and c=
ause large
   packet loss within a round-trip time which results in poo=
r
   performance.  HyStart++ is composed of the delay inc=
rease variant of
   HyStart to prevent overshooting of the ideal sending rate=
, while also
   mitigating poor performance which can result from false p=
ositives.
 
 
The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-hystartpl=
usplus/
 
There is also an htmlized version available at:
https://datatracker.ietf.org/doc/html/draft-ietf-=
tcpm-hystartplusplus-02
 
A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-hys=
tartplusplus-02
 
 
Internet-Drafts are also available by anonymous FTP at:

ftp://ftp.i=
etf.org/internet-drafts/


 


 


_______________________________________________


tcpm mailing list


tcpm@ietf.org


https:/=
/www.ietf.org/mailman/listinfo/tcpm





-- 
________________________________________________________________<=
/o:p>
Bob Briscoe          =
;            &n=
bsp;        http://bobbriscoe.net/
--_000_PH0PR00MB10305B7C7C9A60BCEBB47D88B6E99PH0PR00MB1030namp_-- From nobody Tue Jul 27 16:37:30 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 78ED93A1024; Tue, 27 Jul 2021 16:37:29 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LpcB_cGehsuA; Tue, 27 Jul 2021 16:37:24 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 98EF83A1022; Tue, 27 Jul 2021 16:37:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=hkcsO16pTTUrRW7aV/Hy0z5vIP68udRez+sXUZonkfE=; b=qVjajz0tgVdH+25jGW2ep7m8+L olKJztcAKy3HYeusOZhIbAVilSKZfJ8B+UZ/c0nAgEXY+8yyOypED8gvsRZmLmzuvV6vqDtpHmfeE ggJIHnhnel6DICJeMWzQLI2nHSmKFTsRcP5asDUmLt0rUkjHgoc9mBKfUafnHMxkurZZejR/ytSzf NZnUkTTDS4WBOJwCjUopK7mBFkU7x760I5JizrPjpYDbItYaWs6fuMqkIT2z/Ac3ixE1kAaK3dlHF OBckjqhMZlH5kKK/NXrMqACN65LNxQQKoDEZDdeuML9Vt7wJts87QBtvLfjibM2MhfR0TB3DCNtTP DcvNundw==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:43224 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m8WdZ-0008Sx-Hr; Wed, 28 Jul 2021 00:37:20 +0100 To: Praveen Balasubramanian Cc: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> From: Bob Briscoe Message-ID: <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> Date: Wed, 28 Jul 2021 00:37:19 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------31D07BD34B80B20F58D3163A" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 23:37:30 -0000 This is a multi-part message in MIME format. --------------31D07BD34B80B20F58D3163A Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Any large server operator out there who are using Cubic, If you're willing to state whether or not your operations disable Hystart, pls do so in reply. Then Praveen can cite this mailing list thread in the Hystart++ draft, as requested below. If you are willing to reply privately, I would be willing to keep your confidences, and provide an anonymized result to the list. Cheers Bob On 27/07/2021 01:50, Praveen Balasubramanian wrote: > Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry). >>>>> I'd like some citations on this versus anecdata if possible before I add that caveat to the text. Do large deployments disable this? I haven't come across this suggestion in any Linux tuning guides to date. -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------31D07BD34B80B20F58D3163A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit Any large server operator out there who are using Cubic,
If you're willing to state whether or not your operations disable Hystart, pls do so in reply.
Then Praveen can cite this mailing list thread in the Hystart++ draft, as requested below.

If you are willing to reply privately, I would be willing to keep your confidences, and provide an anonymized result to the list.

Cheers



Bob

On 27/07/2021 01:50, Praveen Balasubramanian wrote:
Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry).

I'd like some citations on this versus anecdata if possible before I add that caveat to the text. Do large deployments disable this? I haven't come across this suggestion in any Linux tuning guides to date. 


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------31D07BD34B80B20F58D3163A-- From nobody Tue Jul 27 16:47:08 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4D8E73A1083 for ; Tue, 27 Jul 2021 16:47:06 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WIWRZGUTp_oV for ; Tue, 27 Jul 2021 16:47:01 -0700 (PDT) Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 35DD63A107A for ; Tue, 27 Jul 2021 16:47:00 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id q3so352323wrx.0 for ; Tue, 27 Jul 2021 16:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yGydkPAGRgdoFI7AJJq+jyd7TJgw1t7GHayBbJeocRs=; b=XgtMSJPNB6X5dP75pR5Xj9YXHoa9uuTKTsomEyOLrGvdcBIhY9QGgc5XXQ98WEo1Pf KMmKvM+SnZ87nIEKGujWa8JpvGxRSv8reKYjU6V6Fbqa80jHdZXs2qdRkHGE06xxvdyA outoAXQsuhc35yLcCK5mErk5gBXwvFrjfh52Fu72b5JkNuMW6SZkZcQNZAu1xQ3hAX47 cabXElCGNjGBz8/EIj72LNCUefzdm45/LI0dRw4MeEqk5fIAU7D2aXyFlMnkcIRnNMW+ MNnazgUxN5E/Va56/dz+W27znM+Sxajm05MJmaO32gLZkMTteMYQZUxjGBeuDzWtoYio DGBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yGydkPAGRgdoFI7AJJq+jyd7TJgw1t7GHayBbJeocRs=; b=CTG1tbRl6PGnxaGnt1DFAyBSp7AcMPG/PYSoVslKCPxSfA3oiizlUuvMFMLt3KB/G/ fytc+1tgf76Lrapv0+x2K7Q9M969/tnm9+4yEKoFzDrfIkI4lLXNYpn/QqfRgsyFCQEq l4ngtMBHLJmrQvUEZsCKHMUX0QsgQYKPCn9fi9MJRGP4VIgXNkCz1Cnm3b71/C12ZacO PVmTErS0IHTPqRNo8tLQypm9kwInVZ2w0aYr2xOhDXr6tz9Aa7I4dg8IPEM5wn/rahKx iy12em0ylxHrBtWlpS7ZK1PUWieKabqofY3e90wopjEtzUyRU187CdD9gYJFRNNFzbnt dqIA== X-Gm-Message-State: AOAM532P2HvI/7RgU5ddTwOE/l5RleOMEWrdGuA/hF5LtPWQY52wfcHN EVbVjKYj3TsYNuFAB5ZvXKYCBiZUcU8JpskLzvP4xw== X-Google-Smtp-Source: ABdhPJyX3w5FZWxQrUpYRpWtTvYz2tIFCeqjQ4hqqZQ8ahby0M3jXXS/7o+0Pg8DYuKA37bD0GM3Xj9VcDmIklkaoGc= X-Received: by 2002:a5d:4284:: with SMTP id k4mr8216476wrq.407.1627429618543; Tue, 27 Jul 2021 16:46:58 -0700 (PDT) MIME-Version: 1.0 References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> In-Reply-To: <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> From: Yuchung Cheng Date: Tue, 27 Jul 2021 16:46:21 -0700 Message-ID: To: Bob Briscoe Cc: Praveen Balasubramanian , "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" , Neal Cardwell Content-Type: multipart/alternative; boundary="0000000000004fcfca05c8237ceb" Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Jul 2021 23:47:06 -0000 --0000000000004fcfca05c8237ceb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Wait -- how is hystart "invariably disabled" in Linux (cubic)!? What data indicates that On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe wrote: > Any large server operator out there who are using Cubic, > If you're willing to state whether or not your operations disable Hystart= , > pls do so in reply. > Then Praveen can cite this mailing list thread in the Hystart++ draft, as > requested below. > > If you are willing to reply privately, I would be willing to keep your > confidences, and provide an anonymized result to the list. > > Cheers > > > > Bob > > On 27/07/2021 01:50, Praveen Balasubramanian wrote: > > Although Hystart is default enabled in Linux, it is invariably disabled. = So, it's misleading to just say Hystart is default enabled, which implies i= t's widely used, when people clearly find it has problems (which motivates = Hystart++). I found this out through an informal survey I did at the Mar'18= IETF in London by asking round the implementers of the most prevalent stac= ks (I would name names if I could find the note I later sent to someone or = to some list, but I can't find it - sry). > > I'd like some citations on this versus anecdata if possible before I add = that caveat to the text. Do large deployments disable this? I haven't come = across this suggestion in any Linux tuning guides to date. > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --0000000000004fcfca05c8237ceb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Wait -- how is hystart "invariably disabled" in = Linux (cubic)!?

What data indicates that

=
On Tue, Ju= l 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net> wrote:
=20 =20 =20
Any large server operator out there who are using Cubic,
If you're willing to state whether or not your operations disable Hystart, pls do so in reply.
Then Praveen can cite this mailing list thread in the Hystart++ draft, as requested below.

If you are willing to reply privately, I would be willing to keep your confidences, and provide an anonymized result to the list.

Cheers



Bob

On 27/07/2021 01:50, Praveen Balasubramanian wrote:
Although Hystart is default enabled in Linux, it is invariably d=
isabled. So, it's misleading to just say Hystart is default enabled, wh=
ich implies it's widely used, when people clearly find it has problems =
(which motivates Hystart++). I found this out through an informal survey I =
did at the Mar'18 IETF in London by asking round the implementers of th=
e most prevalent stacks (I would name names if I could find the note I late=
r sent to someone or to some list, but I can't find it - sry).

I'd like some citations on this versus anecdata if p=
ossible before I add that caveat to the text. Do large deployments disable =
this? I haven't come across this suggestion in any Linux tuning guides =
to date.=20


--=20
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--0000000000004fcfca05c8237ceb-- From nobody Tue Jul 27 17:54:27 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 315933A1406 for ; Tue, 27 Jul 2021 17:54:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.553 X-Spam-Level: X-Spam-Status: No, score=-2.553 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z-PGOman6N7v for ; Tue, 27 Jul 2021 17:54:21 -0700 (PDT) Received: from NAM06-BL2-obe.outbound.protection.outlook.com (mail-eopbgr650109.outbound.protection.outlook.com [40.107.65.109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2E2D33A1405 for ; Tue, 27 Jul 2021 17:54:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RDF29G5ner6ANbLHV6u7Z1ldDIgrX3/J9cb5ZAclWJwxIlDIMvufhs5q6jIoyCFEtnvnkzAx5OMP14BxfjE5mOyYxbLGoIyW7h9tcb/k3nkD9VzRyJHys6Qgx9/pSVdVZRRei0MPa/eIAZSqoX1iXUBcZiq4gsGEs7IS+hXSI7mBG5EpBJUmod0H7ifX77/s2EjSYQ7+nTID4meF1OOeOKhKkIaYcKgiRjfFTwqNPIud7hFs9vKNh2JeCZOBzI1YDJyFKY+fDl/xq6X4EXsgGjVIXo/7YyX4MkPPs8xxC8GlVpnrtyyGf0Az3dlTsaw9T0d6xfo2hwi74Ydc9rVxdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gP4woNjMTp7XWR8rdqBiOCVQqJpKkvWQk0lO2LpJBn0=; b=lcn4zxcmBI9FqnGYdyLY0ec3j6g2HSlmqmk3nnjfE53HN1lwFE2HIyEQLkwdGCB3NDWkvKvRYBF5/35Frqwc677Ifaeb+cgrxKFQU19km/NsTtSBY1hES6A3Kuomgf9GfSsvHu1+KWZePw9hH+1CrWNgUUfMkM68KTudvy1NWbwo2T5fKHBoEsMudpeCuJjkfLXqvFsfU8gTOcsVQFfmJTWdGQf+b+2drC3Ky30rCk6SDvAjqJaAGuzUG/a9QA8M/eqz422JrOwWvKoZ5pXsE92pL4YGOYxFh3G+ReQi3T//M6m/WoqktLlULQOTSnY1mmCSpn9n0zG622jrPxuicA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gP4woNjMTp7XWR8rdqBiOCVQqJpKkvWQk0lO2LpJBn0=; b=N2BYp3XINC22DAeC7MdMeYw7FtUFJ5xCq9SgEsYKXEudKl7V2KDAZSoCFhMxm9sucODf1+OBPFr72JW2oukesu45unOjcIOtu5lPWaUJQIVAlisdMFL7Ch+WECVNsoQr7Dzzt6MBqt8gHeXDG8stIdBv6iNBia07IGrNREQzgJ0= Received: from SA0PR00MB1036.namprd00.prod.outlook.com (2603:10b6:806:132::21) by SA1PR00MB1343.namprd00.prod.outlook.com (2603:10b6:806:1ff::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4411.0; Wed, 28 Jul 2021 00:54:18 +0000 Received: from SA0PR00MB1036.namprd00.prod.outlook.com ([fe80::3518:fbbc:eea9:d7cd]) by SA0PR00MB1036.namprd00.prod.outlook.com ([fe80::3518:fbbc:eea9:d7cd%4]) with mapi id 15.20.4411.000; Wed, 28 Jul 2021 00:54:18 +0000 From: Praveen Balasubramanian To: "vidhi_goel@apple.com" CC: "tcpm@ietf.org" Thread-Topic: draft-ietf-tcpm-rfc8312bis questions Thread-Index: AdeDSlBOP8Gi7tDSSdGTKD2PGdx06w== Date: Wed, 28 Jul 2021 00:54:17 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=f958dd53-d6df-4cd3-825f-eec6a7865690; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2021-07-28T00:47:33Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: apple.com; dkim=none (message not signed) header.d=none;apple.com; dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fe0a75d3-0107-46ea-a503-08d951623a92 x-ms-traffictypediagnostic: SA1PR00MB1343: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8273; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IaaiwZle0Uc3dYbVAqK4RBBfxY1B6lgQgzrsbqCbBe6WHkBFRHeSZtZUsSCn9jTn2QohyOMP1hYtcmHio4p0wn18N9D+RCzAvpb4NH9LEdLwHiT9it8+9/KuOf4LRZu+1r+CvpckL6ExCnK3SXr12GvD+pfDQnKMA15gcq+YP5wZBcMOmWFuWEurgucqHNFqvEdtdQ6Xx47FqfyBzRl88o6NPH7bsY4peEpfwE7S+00oamgYB7OUwan5vYf5RghQLiWnJKKgE2+rgboaY4pGdf6D9PXn1IcMRBK8VuVB8G+f8Oz656lwxnvJKjlqpkBiaKuo9K0WTAsKLrAsqNUD+2V03NkdASwVZ0kSVIUjxjQskicp6XPNmO82xt2nQhu89TzNZRkehulCB2UySKEiSh1/m3wW3gr8it1Muo0hnWMSMKRUvtjfO6vGJhMuZ7O712m8HOGKaWfNgyFagJmm3MwzbUeEZ2PsOIxp0jpdjHBTtr/HFM46BUTPmoxoSRsVgydnbCkFYK4QCUf7tFHz+ErD9xt6ukWL2KLn/Wuhk3FYUXLhQSZZQwWoqm7N+orh7WEmMvZgdh7zrhzkhZzId0Kh4PE7fWgMSnUOO6EEz5aHKVYZIlMqbFo+dNOmRlJtT+H5jlQLgNCCWbu/DVDmle4UE5RvOCQinKKlMpSkahEmbzFOjJz3XKNt8nQVyxSnvGTCSY7Jgj/iQS/4gz7BvA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA0PR00MB1036.namprd00.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(6506007)(508600001)(7696005)(558084003)(52536014)(122000001)(38100700002)(66946007)(76116006)(66476007)(66556008)(64756008)(66446008)(186003)(5660300002)(4326008)(6916009)(86362001)(316002)(2906002)(33656002)(9686003)(10290500003)(8676002)(8936002)(71200400001)(55016002)(8990500004)(82950400001)(82960400001)(38070700005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?FkWa1dj7hQJxMLAhBhk5uMFL8IVH9BjxBSbM3pCx6M/k7WSac7xxnU4f6R7X?= =?us-ascii?Q?STQsSa81x2zIGxQLQAJC1VPxTHojdY8WfZ2hpLwlMjEVDKIepZ+aE62lQYtk?= =?us-ascii?Q?EjcfFDxaDeuxhZb7J2ZSYuL2l+8FnsnLwccMqzZXBEGi/fX8RCW4IYBnxXcB?= =?us-ascii?Q?zCB0Nir2X+zoeBU+q6nHzl6VOsRHHbtKFQtcOk1dqhake7TV6s28coW6zbQF?= =?us-ascii?Q?HuhnjKhI3iBNthHKviJFyAlesXmgAPjUgaPjvEMi95QzFA/TJQXFJGfxCk/u?= =?us-ascii?Q?oCo03NbRlN/J8/RyvBipennmiy3uEdLvpU3glo9WXzlV5PCkd8H/Ig7p5nff?= =?us-ascii?Q?+21NQHd1GIt59OSdSDSYxCOVDRs23SuXINKuiqn6BueyqI1feDVYnG6mO3Sl?= =?us-ascii?Q?5xxRb6wA7pMAwNi8ymq8nMvyUKXsY8q0p746CW7VuesyFe9++Gzpg9kV/ddj?= =?us-ascii?Q?gNsIH/uz4uQV+0410tnTpVDpzFJQj4/hG7xT+c1Az3hPpod7sTAxXyzWAGm6?= =?us-ascii?Q?yeh8L31vZ1ECHCCYpvMF7nkAV9MjIpcSIX7YIBNIY4MBIYUwWZkTeSz17qro?= =?us-ascii?Q?etcr6LuTeU9LpteQ45ncpperHJ2/qpF8pFKK3bPmCxKsUMhA8BbXOKVezIxm?= =?us-ascii?Q?IZpRoqYSALFzmbU2G1hlC33G402vtDuMzcFRNr1trsiNjZneZfxxAZvvUm+m?= =?us-ascii?Q?2bV/aSQalf5bcrj8BA/2k+wUJ6ex3IE9M4Q+5e7kQF2r4O9fh3mZz/0TVPJm?= =?us-ascii?Q?3RWgDd6XQlOqdQwYOOec56dqe3rwrQWmhvMOdLCri5q3Tw6HoXVLmH8P/2Sv?= =?us-ascii?Q?CzqXYPyk9AjBKEyUkd9gtxD+BH5tIqAVORQWBvctUrQfmEWqD757szyAigdc?= =?us-ascii?Q?odjdUko7ih0zpYCW4qWfyUrFbaB1eRzO9jjFPocGT61nAE6sZI9BHjyVqr5q?= =?us-ascii?Q?SEOA7+AISIvlmyOIWB/hvOABvbc1e3VkmqVq9CBKlj91WGZAfuZgHz/QqE+F?= =?us-ascii?Q?Uoz6L/v27leotwcQ+Vs6XsP2jPb+DNiZ0LS/ozlzn5+sfREa3OsHI9z5dA3P?= =?us-ascii?Q?nK8NkeA74WB9XUku/A9zHCY3+nnbUDF/kowWZ+COYtHTd6++eXKt8fwOHarm?= =?us-ascii?Q?DJQ5ELe4Q3uyG049aQ9X2J2IcuKzZdf6tup4Wxhz6Q0EEP9Hkk+NlpYXTgYO?= =?us-ascii?Q?Cf+NRz3z94IEhi8V0GuD5w+ec/eqpidkbTetsCMBM1U19WB7v45ksOJCVxsN?= =?us-ascii?Q?fmv98Fuz4wgTH5AOzBfOHmavsqalNC4sYlrGDlzySrnnVc1Glw2MnmjaXZZ5?= =?us-ascii?Q?JFfAFytn0rFnVb3MOVEyXtLif9fvdKf+fX1K5PqbP3V7xAXc3ZMlZWubKqdy?= =?us-ascii?Q?G7bmsczYVI97bdsK5hsomF6YBFw6?= x-ms-exchange-transport-forked: True Content-Type: multipart/alternative; boundary="_000_SA0PR00MB103641CC616E1E29A7761528B6EA9SA0PR00MB1036namp_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SA0PR00MB1036.namprd00.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fe0a75d3-0107-46ea-a503-08d951623a92 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jul 2021 00:54:17.9665 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: RTV50XYvefyVF++Fx+kpks9168REPSpFtwlRiCkbRhlQP1cc/ghTO96vSwUrSI/vAoZlvyLkY4cqZtkb6dZIxBjGDvjWHhNRHPLLfP8lwAU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR00MB1343 Archived-At: Subject: [tcpm] draft-ietf-tcpm-rfc8312bis questions X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 00:54:25 -0000 --_000_SA0PR00MB103641CC616E1E29A7761528B6EA9SA0PR00MB1036namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable As discussed in the working group meeting, can you please clarify what beha= vior changes an implementation will see as a result of adopting 8312-bis ve= rsus RFC 8312? Particularly interested to know which changes will result in= improved performance. Thanks! --_000_SA0PR00MB103641CC616E1E29A7761528B6EA9SA0PR00MB1036namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

As discussed in the working group meeting, can you p= lease clarify what behavior changes an implementation will see as a result = of adopting 8312-bis versus RFC 8312? Particularly interested to know which= changes will result in improved performance.

 

Thanks!

 

 

--_000_SA0PR00MB103641CC616E1E29A7761528B6EA9SA0PR00MB1036namp_-- From nobody Tue Jul 27 17:58:42 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 77E603A145F for ; Tue, 27 Jul 2021 17:58:32 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QvGagQbgqijG for ; Tue, 27 Jul 2021 17:58:27 -0700 (PDT) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 460A43A1453 for ; Tue, 27 Jul 2021 17:58:27 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id l11-20020a7bc34b0000b029021f84fcaf75so3132314wmj.1 for ; Tue, 27 Jul 2021 17:58:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hh4uR8HbYrkkCH5n/8ZiCr2u1a5O2K2D+a2npo65YOA=; b=gt9+W43SX4bw6atzTfSUnjJfYYxspdzBs8gVJ7MozNMU424xIuw1tcznqtCP0BFwOC Edy0i9OcfK+Q4Ybgt2k1yga77LzNK28xNqDG4HjwLiPnjTxSDpMxIJDTeUbQ6h2+apyo 7wdglZEBreEhsgxJ0q5P1NMqMOkSEKpZInKR8Pmbb/goRahr7ONbqsiHBzEPUClNyEqU F/oC0wJKO5xK8LPXbOHAYKF7V+3iRewPw1xcdZRxF5JFJFynsBdjdatCeAKuZTQrST7n 6m5WD9oi7j/LoQVarjmdKjAl/eDnZOZxtSqF3UWhEZ+u77p//xMhnItZBpxgtiWM4bkb zbSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hh4uR8HbYrkkCH5n/8ZiCr2u1a5O2K2D+a2npo65YOA=; b=dPlBU7kLmPvFBrsLWK1dqST/oPpP9178dHm7atKKYSWKtAELlyBs1dqmg2QEfgt3iK l80XzLSxL27vNnqa4YP6FOZLTkTb2TjI99J6kD/sXEEycxLqa6C2sRCai/TFTN+mW7U7 8FbzNxcI+Ba9vCMwYCvF2YE6rAWAx4WWaZrt0Oxq+9aSlGtTmj8QZGSBqFmbdjlUPQmv xfrZIONWUuJPnLyZuRWaF+5IeShYW1GevgkURksuJUg+4EB+Q2jPPMQcRaY1fzpDijot fgMwRASFuzN3oeMKHlS4/KT3cpI3/ckI355r4vOmoh7sY7s7SXZA5Q1OeDxc3RpJIWxF MEYQ== X-Gm-Message-State: AOAM531bQZoN7KOT/9u+B+U0U7Y7Tg+c8iP7888ZaKzGQNLwrk8Txy5L 9nBAOUq6oTPqmkZsyDP813MSWokmPAm9niPJfejeBA== X-Google-Smtp-Source: ABdhPJwwIUWR/+ybB/lLhQM91Z0IRcXPPfRKHdWeuyBNUKYxKhCtFofBIXtx06JthM26Wru6BiVNbMF82485jAiaFZk= X-Received: by 2002:a1c:f206:: with SMTP id s6mr24044981wmc.102.1627433904031; Tue, 27 Jul 2021 17:58:24 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yuchung Cheng Date: Tue, 27 Jul 2021 17:57:47 -0700 Message-ID: To: Praveen Balasubramanian Cc: "vidhi_goel@apple.com" , "tcpm@ietf.org" Content-Type: multipart/alternative; boundary="000000000000bf8bcd05c8247b4a" Archived-At: Subject: Re: [tcpm] draft-ietf-tcpm-rfc8312bis questions X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 00:58:39 -0000 --000000000000bf8bcd05c8247b4a Content-Type: text/plain; charset="UTF-8" btw it seems RFC5681 is even more conservative in dealing with stretched acks in terms of ABC adoption. sec 3.1 "During slow start,.... While traditionally TCP implementations have increased cwnd by precisely SMSS bytes upon receipt of an ACK covering new data, we RECOMMEND that TCP implementations increase cwnd, per: cwnd += min (N, SMSS) (2) where N is the number of previously unacknowledged bytes acknowledged in the incoming ACK. This adjustment is part of Appropriate Byte Counting [RFC3465] and provides robustness against misbehaving receivers that may attempt to induce a sender to artificially inflate cwnd using a mechanism known as "ACK Division" [SCWA99]." On Tue, Jul 27, 2021 at 5:54 PM Praveen Balasubramanian wrote: > As discussed in the working group meeting, can you please clarify what > behavior changes an implementation will see as a result of adopting > 8312-bis versus RFC 8312? Particularly interested to know which changes > will result in improved performance. > > > > Thanks! > > > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --000000000000bf8bcd05c8247b4a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
btw it seems RFC5681 is even more conservative in dealing = with stretched acks in terms of ABC adoption.=C2=A0

sec = 3.1

"During slow start,....=C2=A0 While
= =C2=A0 =C2=A0traditionally TCP implementations have increased cwnd by preci= sely
=C2=A0 =C2=A0SMSS bytes upon receipt of an ACK covering new data, w= e RECOMMEND
=C2=A0 =C2=A0that TCP implementations increase cwnd, per:
=C2=A0 =C2=A0 =C2=A0 cwnd +=3D min (N, SMSS) =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(2)

=C2=A0 =C2= =A0where N is the number of previously unacknowledged bytes acknowledged=C2=A0 =C2=A0in the incoming ACK.=C2=A0 This adjustment is part of Appropr= iate Byte
=C2=A0 =C2=A0Counting [RFC3465] and provides robustness agains= t misbehaving
=C2=A0 =C2=A0receivers that may attempt to induce a sender= to artificially inflate
=C2=A0 =C2=A0cwnd using a mechanism known as &q= uot;ACK Division" [SCWA99]."


<= br>
On Tue,= Jul 27, 2021 at 5:54 PM Praveen Balasubramanian <pravb=3D40microsoft.com@dmarc.ietf.org> = wrote:

As discussed in the working group meeting, can you p= lease clarify what behavior changes an implementation will see as a result = of adopting 8312-bis versus RFC 8312? Particularly interested to know which= changes will result in improved performance.

=C2=A0

Thanks!

=C2=A0

=C2=A0

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--000000000000bf8bcd05c8247b4a-- From nobody Tue Jul 27 22:51:47 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 795303A1E52; Tue, 27 Jul 2021 22:51:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S_h2pBK-4rAv; Tue, 27 Jul 2021 22:51:37 -0700 (PDT) Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4F4143A1E53; Tue, 27 Jul 2021 22:51:34 -0700 (PDT) Received: by rfc-editor.org (Postfix, from userid 30) id CAC44F4070A; Tue, 27 Jul 2021 22:51:15 -0700 (PDT) To: ietf-announce@ietf.org, rfc-dist@rfc-editor.org X-PHP-Originating-Script: 1005:ams_util_lib.php From: rfc-editor@rfc-editor.org Cc: rfc-editor@rfc-editor.org, drafts-update-ref@iana.org, tcpm@ietf.org Content-type: text/plain; charset=UTF-8 Message-Id: <20210728055115.CAC44F4070A@rfc-editor.org> Date: Tue, 27 Jul 2021 22:51:15 -0700 (PDT) Archived-At: Subject: [tcpm] =?utf-8?q?RFC_9040_on_TCP_Control_Block_Interdependence?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 05:51:44 -0000 A new Request for Comments is now available in online RFC libraries. RFC 9040 Title: TCP Control Block Interdependence Author: J. Touch, M. Welzl, S. Islam Status: Informational Stream: IETF Date: July 2021 Mailbox: touch@strayalpha.com, michawe@ifi.uio.no, safiquli@ifi.uio.no Pages: 29 Obsoletes: RFC 2140 I-D Tag: draft-ietf-tcpm-2140bis-11.txt URL: https://www.rfc-editor.org/info/rfc9040 DOI: 10.17487/RFC9040 This memo provides guidance to TCP implementers that is intended to help improve connection convergence to steady-state operation without affecting interoperability. It updates and replaces RFC 2140's description of sharing TCP state, as typically represented in TCP Control Blocks, among similar concurrent or consecutive connections. This document is a product of the TCP Maintenance and Minor Extensions Working Group of the IETF. INFORMATIONAL: This memo provides information for the Internet community. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited. This announcement is sent to the IETF-Announce and rfc-dist lists. To subscribe or unsubscribe, see https://www.ietf.org/mailman/listinfo/ietf-announce https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist For searching the RFC series, see https://www.rfc-editor.org/search For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk Requests for special distribution should be addressed to either the author of the RFC in question, or to rfc-editor@rfc-editor.org. Unless specifically noted otherwise on the RFC itself, all RFCs are for unlimited distribution. The RFC Editor Team Association Management Solutions, LLC From nobody Wed Jul 28 00:13:51 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 50C2D3A20FA for ; Wed, 28 Jul 2021 00:13:49 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.55 X-Spam-Level: X-Spam-Status: No, score=-2.55 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KbxF103Jckam for ; Wed, 28 Jul 2021 00:13:47 -0700 (PDT) Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5D4DC3A20F6 for ; Wed, 28 Jul 2021 00:13:46 -0700 (PDT) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16S6vUg0060928; Wed, 28 Jul 2021 00:13:45 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=8CjxSB/LB+uf6X6BLOsDAS9drDlPg6Kw6u4GHolok7g=; b=NkiNbqzLGAnQAKNnYUiWg1VEUwNtsAY0cjcm24QnxFagbM5LhNgZ8hn1F0fjZoL2b8g0 4s4TPXPaYfg7VqKt+qLlVvfWrvRAr2zCAiOa8J4125B0QJr99K3ETQWeIscaytsra6Zz oLz2pqJYtdNOcrjIz9gGygRXBMYmNE3ji6PmKLLJB65WpNTFbWkh4Z2TzRDMNmv/zHFC Wxh10YobY2LSlPlqcfY+LaentUuONnm1kKHWfNx67RThByi1ZvntDIuJXWNMvOMpPsxk EpXOUWAXTtxi9MOhY6LWNe4OQBS/kTE8akzWK2rn0xyDGbPbqr+P3SN3w8B2lAe1dFi+ QA== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 3a234gq0f8-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 28 Jul 2021 00:13:45 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QWY008QZ1EXEWG0@rn-mailsvcp-mta-lapp01.rno.apple.com>; Wed, 28 Jul 2021 00:13:45 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QWY00M000TUN800@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 00:13:45 -0700 (PDT) X-Va-A: X-Va-T-CD: 766d32dce0a9c027bcd9f1f38891080b X-Va-E-CD: a53e180a56842f1755dd6cf39cfb1647 X-Va-R-CD: 22bf1e320b833e94248c9585988464b3 X-Va-CD: 0 X-Va-ID: 09f94e83-5b01-407d-9aa6-369a46526104 X-V-A: X-V-T-CD: 766d32dce0a9c027bcd9f1f38891080b X-V-E-CD: a53e180a56842f1755dd6cf39cfb1647 X-V-R-CD: 22bf1e320b833e94248c9585988464b3 X-V-CD: 0 X-V-ID: 70c2b4a9-f067-4722-98c6-81c0b4b9b753 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_05:2021-07-27, 2021-07-28 signatures=0 Received: from smtpclient.apple (unknown [17.234.31.61]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QWY009301EW8000@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 00:13:45 -0700 (PDT) From: Vidhi Goel Message-id: <012AB76C-115B-41A7-8505-D5993FCE7416@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_905FA06F-DA1A-4D14-A2BC-F389E521016E" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.11\)) Date: Wed, 28 Jul 2021 00:13:44 -0700 In-reply-to: Cc: "tcpm@ietf.org" To: Praveen Balasubramanian References: X-Mailer: Apple Mail (2.3654.100.0.2.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_05:2021-07-27, 2021-07-28 signatures=0 Archived-At: Subject: Re: [tcpm] draft-ietf-tcpm-rfc8312bis questions X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 07:13:49 -0000 --Apple-Mail=_905FA06F-DA1A-4D14-A2BC-F389E521016E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Praveen, Following changes improve the performance of CUBIC: 1. Updated K=E2=80=99s definition to use congestion window at the start = of congestion avoidance. More details at issue = https://github.com/NTAP/rfc8312bis/issues/1 = 2. Adding lower and upper bound for CUBIC target increase - lower bound = is no longer mandatory with the change in 1. The upper bound is more = interesting as it will make sure that the increase function is slower = than slow start. This change averts the risk of overshooting the = congestion window during CA. More details = https://github.com/NTAP/rfc8312bis/issues/14 = 3. Always use a minimum cwnd of 2. = https://github.com/NTAP/rfc8312bis/issues/7 = 4. Change the AIMD friendly equation to use segments acknowledged = instead of using time. This allows more accurate increase for cases such = as rate limited apps, sending after idle etc. It also performs better = with ACK thinning. https://github.com/NTAP/rfc8312bis/issues/20 = 5. When W_est > W_max, we now set alpha to 1. This makes sure that Cubic = doesn=E2=80=99t perform worse than New Reno when we are probing for new = W_max. https://github.com/NTAP/rfc8312bis/issues/2 = 6. Spurious losses result in lowering the congestion window erroneously. = So, we have added guidance for handling spurious losses for CUBIC. = https://github.com/NTAP/rfc8312bis/issues/23 = Let me know if something is unclear or if you need more info. And as = always, feedback is welcome on the current draft. Thanks, Vidhi > On Jul 27, 2021, at 5:54 PM, Praveen Balasubramanian = wrote: >=20 > As discussed in the working group meeting, can you please clarify what = behavior changes an implementation will see as a result of adopting = 8312-bis versus RFC 8312? Particularly interested to know which changes = will result in improved performance. > =20 > Thanks! --Apple-Mail=_905FA06F-DA1A-4D14-A2BC-F389E521016E Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Praveen,
Following changes improve the performance of = CUBIC:

1. = Updated K=E2=80=99s definition to use congestion window at the start of = congestion avoidance. More details at issue https://github.com/NTAP/rfc8312bis/issues/1
2. Adding lower and upper bound for CUBIC target increase - = lower bound is no longer mandatory with the change in 1. The upper bound = is more interesting as it will make sure that the increase function is = slower than slow start. This change averts the risk of overshooting the = congestion window during CA. More details https://github.com/NTAP/rfc8312bis/issues/14
3. Always use a minimum cwnd of 2. https://github.com/NTAP/rfc8312bis/issues/7
4. Change the AIMD friendly equation to use segments = acknowledged instead of using time. This allows more accurate increase = for cases such as rate limited apps, sending after idle etc. It also = performs better with ACK thinning. https://github.com/NTAP/rfc8312bis/issues/20
5. When W_est > W_max, we now set alpha to 1. This makes = sure that Cubic doesn=E2=80=99t perform worse than New Reno when we are = probing for new W_max. https://github.com/NTAP/rfc8312bis/issues/2
6. Spurious losses result in lowering the congestion window = erroneously. So, we have added guidance for handling spurious losses for = CUBIC.  https://github.com/NTAP/rfc8312bis/issues/23

Let me know if something = is unclear or if you need more info. And as always, feedback is welcome = on the current draft.

Thanks,
Vidhi


On Jul 27, 2021, at 5:54 PM, = Praveen Balasubramanian <pravb@microsoft.com> wrote:

As discussed in the working group meeting, can = you please clarify what behavior changes an implementation will see as a = result of adopting 8312-bis versus RFC 8312? Particularly interested to = know which changes will result in improved performance.
 
Thanks!

= --Apple-Mail=_905FA06F-DA1A-4D14-A2BC-F389E521016E-- From nobody Wed Jul 28 00:30:24 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0EE783A1788 for ; Wed, 28 Jul 2021 00:30:23 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.552 X-Spam-Level: X-Spam-Status: No, score=-2.552 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SH3ABVG9-Vqp for ; Wed, 28 Jul 2021 00:30:19 -0700 (PDT) Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D864B3A1800 for ; Wed, 28 Jul 2021 00:30:18 -0700 (PDT) Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16S7ISbX024594; Wed, 28 Jul 2021 00:30:15 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=/6tCBU3Dfs/1cY6bpxju08DQuD79gbVby3BXEuREmFE=; b=OozuK6Kd9dH+FqDVzd2JlSa09vgy1QdqQNBr1djTOEt8T/FUvt1pT2OOyBEbmwsLuZ/X Ru6o24bcAAt6SiBotB29qCfDdy+kQiedEP16n0ccOIvT8aQemeTgs9lWwzCqi/u6lPi3 k1s6VMJi56UDEmljUj+DZMKcZeS2RhoZbZiRSux0XiEN0OJMih5GbJWdhdNC+RXaaA2W vG6uvQpKzNWLPlxg7hmk2WM50HMxPtVTxmRKe41AOhhZsxorO8cRYN940sa3ZerthqwP t8jgnfL9qee3qLrVwCYsSVChA895vAyeeXZ+FpEdiIWMIT4KFlz7cqhY0YfTCzlKjT64 1g== Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 3a236kfj5m-29 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 28 Jul 2021 00:30:15 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QWY001MO26DUYC0@rn-mailsvcp-mta-lapp02.rno.apple.com>; Wed, 28 Jul 2021 00:30:13 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QWY00D0025VYF00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 00:30:13 -0700 (PDT) X-Va-A: X-Va-T-CD: 496dd5144492d37e6d353a65a06de324 X-Va-E-CD: 755c3673a619f35c03849d0dc5e0d415 X-Va-R-CD: 1a94df0043af936a262058355be6bd4c X-Va-CD: 0 X-Va-ID: 28c22983-6201-4382-9d02-b54fd6285842 X-V-A: X-V-T-CD: 496dd5144492d37e6d353a65a06de324 X-V-E-CD: 755c3673a619f35c03849d0dc5e0d415 X-V-R-CD: 1a94df0043af936a262058355be6bd4c X-V-CD: 0 X-V-ID: 0560ba97-1602-477c-af38-bba7ca570023 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_05:2021-07-27, 2021-07-28 signatures=0 Received: from smtpclient.apple (unknown [17.234.31.61]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QWY0074L26C0Q00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 00:30:12 -0700 (PDT) From: Vidhi Goel Message-id: <2FB9F21D-6D8B-40B0-8D73-7D10B2483CD0@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_BDAB24CA-18EC-4EEF-A1D7-BCED5E7C43E6" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.11\)) Date: Wed, 28 Jul 2021 00:30:12 -0700 In-reply-to: Cc: Praveen Balasubramanian , "tcpm@ietf.org" To: Yuchung Cheng References: X-Mailer: Apple Mail (2.3654.100.0.2.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_05:2021-07-27, 2021-07-28 signatures=0 Archived-At: Subject: Re: [tcpm] draft-ietf-tcpm-rfc8312bis questions X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 07:30:23 -0000 --Apple-Mail=_BDAB24CA-18EC-4EEF-A1D7-BCED5E7C43E6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > btw it seems RFC5681 is even more conservative in dealing with = stretched acks in terms of ABC adoption.=20 Thanks Yuchung for double checking. > "During slow start,.... While > traditionally TCP implementations have increased cwnd by precisely > SMSS bytes upon receipt of an ACK covering new data, we RECOMMEND > that TCP implementations increase cwnd, per: >=20 > cwnd +=3D min (N, SMSS) (2) This is mentioned as a recommendation and not a MUST. > where N is the number of previously unacknowledged bytes = acknowledged > in the incoming ACK. This adjustment is part of Appropriate Byte > Counting [RFC3465] and provides robustness against misbehaving > receivers that may attempt to induce a sender to artificially = inflate Later, 5681 references RFC 3465 - although not for any guidance on the = max limit. I am not sure if RFC 5681 needs any update but it would be = good to update 3465 to reflect today=E2=80=99s practices. Thanks, Vidhi > On Jul 27, 2021, at 5:57 PM, Yuchung Cheng wrote: >=20 > btw it seems RFC5681 is even more conservative in dealing with = stretched acks in terms of ABC adoption.=20 >=20 > sec 3.1 >=20 > "During slow start,.... While > traditionally TCP implementations have increased cwnd by precisely > SMSS bytes upon receipt of an ACK covering new data, we RECOMMEND > that TCP implementations increase cwnd, per: >=20 > cwnd +=3D min (N, SMSS) (2) >=20 > where N is the number of previously unacknowledged bytes = acknowledged > in the incoming ACK. This adjustment is part of Appropriate Byte > Counting [RFC3465] and provides robustness against misbehaving > receivers that may attempt to induce a sender to artificially = inflate > cwnd using a mechanism known as "ACK Division" [SCWA99]." >=20 >=20 >=20 > On Tue, Jul 27, 2021 at 5:54 PM Praveen Balasubramanian = > wrote: > As discussed in the working group meeting, can you please clarify what = behavior changes an implementation will see as a result of adopting = 8312-bis versus RFC 8312? Particularly interested to know which changes = will result in improved performance. >=20 > =20 >=20 > Thanks! >=20 > =20 >=20 > =20 >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm = --Apple-Mail=_BDAB24CA-18EC-4EEF-A1D7-BCED5E7C43E6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
btw it seems RFC5681 is even more conservative in dealing = with stretched acks in terms of ABC = adoption. 

Thanks Yuchung for double checking.

"During slow start,....  = While
   traditionally TCP implementations have increased = cwnd by precisely
   SMSS bytes upon receipt of = an ACK covering new data, we RECOMMEND
   that = TCP implementations increase cwnd, per:

      cwnd +=3D min (N, SMSS)     =                 =  (2)

This is = mentioned as a recommendation and not a MUST.

   where N is the number of previously = unacknowledged bytes acknowledged
   in the = incoming ACK.  This adjustment is part of Appropriate Byte
   Counting [RFC3465] and provides robustness = against misbehaving
   receivers that may = attempt to induce a sender to artificially inflate

Later, 5681 references RFC 3465 - = although not for any guidance on the max limit. I am not sure if RFC = 5681 needs any update but it would be good to update 3465 to reflect = today=E2=80=99s practices.

Thanks,
Vidhi

On Jul 27, 2021, at 5:57 PM, Yuchung Cheng <ycheng@google.com> = wrote:

btw it seems RFC5681 is even more conservative in = dealing with stretched acks in terms of ABC adoption. 

sec 3.1

"During slow = start,....  While
   traditionally TCP = implementations have increased cwnd by precisely
  =  SMSS bytes upon receipt of an ACK covering new data, we = RECOMMEND
   that TCP implementations increase = cwnd, per:

      cwnd +=3D = min (N, SMSS)                 =      (2)

   where = N is the number of previously unacknowledged bytes acknowledged
   in the incoming ACK.  This adjustment is = part of Appropriate Byte
   Counting [RFC3465] = and provides robustness against misbehaving
  =  receivers that may attempt to induce a sender to artificially = inflate
   cwnd using a mechanism known as "ACK = Division" [SCWA99]."



On Tue, Jul = 27, 2021 at 5:54 PM Praveen Balasubramanian <pravb=3D40microsoft.com@dmarc.ietf.org> wrote:

As discussed in the working group meeting, can you = please clarify what behavior changes an implementation will see as a = result of adopting 8312-bis versus RFC 8312? Particularly interested to = know which changes will result in improved performance.

 

Thanks!

 

 

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

= --Apple-Mail=_BDAB24CA-18EC-4EEF-A1D7-BCED5E7C43E6-- From nobody Wed Jul 28 03:20:02 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1F2763A2672; Wed, 28 Jul 2021 03:20:01 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bxo-fYIq0gwG; Wed, 28 Jul 2021 03:19:56 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7DC413A2671; Wed, 28 Jul 2021 03:19:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BvejF7yzPPQvnMcPdVfgOMuqTDb6HYspGsrBZACXvUE=; b=TlLYbvXvip+RcnxOjsTJsOQnOJ AyDM00Rn3AYSprx6/PvOqSvPeWyp7O4WjCuMBCn3uL0rgtBgYZWfs6CobW296CMtnmCTn6K8pAXFD oIiYbHC25cMrJX8LtEDoWyWRgjtRlBBg5BTiE3tRXzwEtkAtXX/uobZQxIFhP+kvIz2R53AGgjdjx pAs2SQgDwLXtpk1y+LmM7VHW62OAyEWsEsxvXTGALN+hkVuiRar4UCArhq/U6mITcdvZEzx6lkovv 5TC/5QRmpBmAAJZQ3hHH0ogp25J27vmVdHa1P9RolCElpiw97AJR3GQaPEAacavxatZqBIiwKJakD GTCMhTew==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:44556 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m8gfK-0004RM-0H; Wed, 28 Jul 2021 11:19:48 +0100 To: Yuchung Cheng Cc: Praveen Balasubramanian , "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" , Neal Cardwell References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> From: Bob Briscoe Message-ID: <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> Date: Wed, 28 Jul 2021 11:19:44 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------82B64F67464A4A67ACC1A5AC" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 10:20:01 -0000 This is a multi-part message in MIME format. --------------82B64F67464A4A67ACC1A5AC Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Yuchung, It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work". I can't remember why I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart. If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now. Bob On 28/07/2021 00:46, Yuchung Cheng wrote: > Wait -- how is hystart "invariably disabled" in Linux (cubic)!? > > What data indicates that > > On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe > wrote: > > Any large server operator out there who are using Cubic, > If you're willing to state whether or not your operations disable > Hystart, pls do so in reply. > Then Praveen can cite this mailing list thread in the Hystart++ > draft, as requested below. > > If you are willing to reply privately, I would be willing to keep > your confidences, and provide an anonymized result to the list. > > Cheers > > > > Bob > > On 27/07/2021 01:50, Praveen Balasubramanian wrote: >> >> Although Hystart is default enabled in Linux, it is invariably >> disabled. So, it's misleading to just say Hystart is default >> enabled, which implies it's widely used, when people clearly find >> it has problems (which motivates Hystart++). I found this out >> through an informal survey I did at the Mar'18 IETF in London by >> asking round the implementers of the most prevalent stacks (I >> would name names if I could find the note I later sent to someone >> or to some list, but I can't find it - sry). >>>>>> >>>>>> I'd like some citations on this versus anecdata if possible >>>>>> before I add that caveat to the text. Do large deployments >>>>>> disable this? I haven't come across this suggestion in any >>>>>> Linux tuning guides to date. > > -- > ________________________________________________________________ > Bob Briscoehttp://bobbriscoe.net/ > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------82B64F67464A4A67ACC1A5AC Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work". I can't remember why I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now.


Bob


On 28/07/2021 00:46, Yuchung Cheng wrote:
Wait -- how is hystart "invariably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net> wrote:
Any large server operator out there who are using Cubic,
If you're willing to state whether or not your operations disable Hystart, pls do so in reply.
Then Praveen can cite this mailing list thread in the Hystart++ draft, as requested below.

If you are willing to reply privately, I would be willing to keep your confidences, and provide an anonymized result to the list.

Cheers



Bob

On 27/07/2021 01:50, Praveen Balasubramanian wrote:

Although Hystart is default enabled in Linux, it is invariably disabled. So, it's misleading to just say Hystart is default enabled, which implies it's widely used, when people clearly find it has problems (which motivates Hystart++). I found this out through an informal survey I did at the Mar'18 IETF in London by asking round the implementers of the most prevalent stacks (I would name names if I could find the note I later sent to someone or to some list, but I can't find it - sry).

I'd like some citations on this versus anecdata if possible before I add that caveat to the text. Do large deployments disable this? I haven't come across this suggestion in any Linux tuning guides to date. 

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------82B64F67464A4A67ACC1A5AC-- From nobody Wed Jul 28 09:47:34 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 466AE3A1890 for ; Wed, 28 Jul 2021 09:47:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.888 X-Spam-Level: X-Spam-Status: No, score=-1.888 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xfO5eC7_bUcM for ; Wed, 28 Jul 2021 09:47:21 -0700 (PDT) Received: from mx36-out21.antispamcloud.com (mx36-out21.antispamcloud.com [209.126.121.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C83013A17FB for ; Wed, 28 Jul 2021 09:47:04 -0700 (PDT) Received: from xse109.mail2web.com ([66.113.196.109] helo=xse.mail2web.com) by mx133.antispamcloud.com with esmtp (Exim 4.92) (envelope-from ) id 1m8mhu-000ZSO-5i for tcpm@ietf.org; Wed, 28 Jul 2021 18:47:01 +0200 Received: from xsmtp22.mail2web.com (unknown [10.100.68.61]) by xse.mail2web.com (Postfix) with ESMTPS id 4GZfhF52SXzLDp for ; Wed, 28 Jul 2021 09:46:53 -0700 (PDT) Received: from [10.5.2.14] (helo=xmail04.myhosting.com) by xsmtp22.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1m8mhp-0007Ha-In for tcpm@ietf.org; Wed, 28 Jul 2021 09:46:53 -0700 Received: (qmail 30611 invoked from network); 28 Jul 2021 16:46:52 -0000 Received: from unknown (HELO [192.168.1.104]) (Authenticated-user:_huitema@huitema.net@[172.58.43.253]) (envelope-sender ) by xmail04.myhosting.com (qmail-ldap-1.03) with ESMTPA for ; 28 Jul 2021 16:46:52 -0000 To: Bob Briscoe , Yuchung Cheng Cc: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> From: Christian Huitema Message-ID: <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> Date: Wed, 28 Jul 2021 09:46:52 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.12.0 MIME-Version: 1.0 In-Reply-To: <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> Content-Type: multipart/alternative; boundary="------------1455D1D7C3FC1353F392291E" Content-Language: en-US X-Originating-IP: 66.113.196.109 X-Spampanel-Domain: xsmtpout.mail2web.com X-Spampanel-Username: 66.113.196.0/24 Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.196.0/24@xsmtpout.mail2web.com X-Spampanel-Outgoing-Class: unsure X-Spampanel-Outgoing-Evidence: Combined (0.15) X-Recommended-Action: accept X-Filter-ID: Pt3MvcO5N4iKaDQ5O6lkdGlMVN6RH8bjRMzItlySaT9WLQux0N3HQm8ltz8rnu+BPUtbdvnXkggZ 3YnVId/Y5jcf0yeVQAvfjHznO7+bT5yjTC8Fov82/EJuxz8FihBPKj/EwzSHE5FGYwwjsNRPCNsk 4s3QUE9Ikbu0D7c8RPbmD6wdmZPcItWbGe10hXJtXL4FsauCVkDjmcYJdU3yWp7KuHNaaKdg7iBE ZefdsNUFWKwa/wzJUjmazeC7Imcapebr0kNyYC289u5HlaNj1BQ6V51u76v35b1wNe/MvdKAGdwU TZKlze5ERymXAD3v2+J9PgaoF8SQHto3le4zsHTaeQtlKubP6iUTjj6yPARK6buALVaA782LKxg6 vRmng8N1aLhXqdc+jC1RcnVud53D5caUhbVtvqItBqoizkEt9O20UjkwI0v+LOlw05G4BS+iyyNq bT8dUMXMJ4tUCMj6G37ZfAMLceP5aNHPt26RBupu5v1nytoNnc138GfEJRQ2qC7jjynPIHPNqSn4 QTXUjLjYWQt1/5xnQymMoPsgr/U0flMcy2Vi/IcBgY4arPaiJ1W6hAyiRC61jekdwIcXNugoOEbH RyFULpSjm7jZ1h/HfDRQ5Ig8VhPsPE8NaP2gA77cO7WeI9Ftai6fusDXp7t2nLqCQxvxn0YNWXvz HU+t5Fa1CSbFsnUGqUzJDRojSVizNl0ce/s7u0P9b9Tml6eOMCV9kYYwkPx6ZsXvIUzTXkDAiiJi mGhLUFuS2lhaIetXfCg1JdAVrOwKfDfxVmiprXIZY6X8fH9rAIlUoFIvD3sIcP1fhJPM6B/87s2a Xm69McMIC42KTw7i32Lua3RJ0N4pwp1zcex/7jiJ+dym1L8cD17Js0v4cp1MaDI3VMeXrtWilILQ zgNtADcKVNeVJ9BXyu9+ceCqThTYg2px1fSoqxQCCHnLMo/m9VKh99btUAanjnMCAH2co+fBoeG+ Hs0afhsY/5zhNYWRVYKU9W9tbmVXJBqdHHDmZEKhyNAv1N35kYWaEdgLurFV5oTvAcwA4rM3FkfW 8/1kE/e7sUnsVpINvARNxpFO X-Report-Abuse-To: spam@quarantine11.antispamcloud.com Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 16:47:33 -0000 This is a multi-part message in MIME format. --------------1455D1D7C3FC1353F392291E Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable I certainly do not qualify as a "large operator", but when implementing=20 Cubic for QUIC I found that Hystart was critical for performance.=20 Specifically, the classical slow-start is very prone to overshooting the = capacity of the path, which causes large batches of errors. With=20 web-like traffic, these batches of errors cause increase latency for the = first transactions in a session, and thus a drop in perceived quality.=20 Hystart solves that. With Hystart, I observed that a large fraction of=20 sessions do not experience any packet loss at all. As an aside, we should strive for this suppression of packet losses.=20 That's actually a big reason for moving from Cubic to BBR. Hystart=20 suppresses the losses during the initial phase of the connection, but=20 Cubic still relies on periodically testing the limit of path capacity=20 and causing losses during the subsequent phase. BBR's probing for=20 bottleneck bandwidth is much more conservative, does not cause such=20 losses. It might be possible to adapt Cubic to also not cause losses,=20 for example by ending an epoch early if too many CE marks are received=20 or if the RTT increases. That would be worth trying. -- Christian Huitema On 7/28/2021 3:19 AM, Bob Briscoe wrote: > Yuchung, > > It was during a Mar 2018 ad hoc workshop Jana had organized at the=20 > London IETF entitled 'BBR and the intersection with other work". I=20 > can't remember why I needed to know at the time, but during the break=20 > I approached the major server operators individually, established=20 > whether they used Cubic, and if so asked whether they used Hystart or=20 > disabled it. It's hard to anonymize the results, because IMMSMC all=20 > those that used Cubic said they disabled Hystart. > > If this isn't correct, then maybe people who replied at the time=20 > thought they were being asked something else. Or maybe it was correct=20 > then but isn't now. > > > Bob > > > On 28/07/2021 00:46, Yuchung Cheng wrote: >> Wait -- how is hystart "invariably disabled" in Linux (cubic)!? >> >> What data indicates that >> >> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe > > wrote: >> >> =C2=A0=C2=A0=C2=A0 Any large server operator out there who are using C= ubic, >> =C2=A0=C2=A0=C2=A0 If you're willing to state whether or not your oper= ations disable >> =C2=A0=C2=A0=C2=A0 Hystart, pls do so in reply. >> =C2=A0=C2=A0=C2=A0 Then Praveen can cite this mailing list thread in t= he Hystart++ >> =C2=A0=C2=A0=C2=A0 draft, as requested below. >> >> =C2=A0=C2=A0=C2=A0 If you are willing to reply privately, I would be w= illing to keep >> =C2=A0=C2=A0=C2=A0 your confidences, and provide an anonymized result = to the list. >> >> =C2=A0=C2=A0=C2=A0 Cheers >> >> >> >> =C2=A0=C2=A0=C2=A0 Bob >> >> =C2=A0=C2=A0=C2=A0 On 27/07/2021 01:50, Praveen Balasubramanian wrote:= >>> >>> =C2=A0=C2=A0=C2=A0 Although Hystart is default enabled in Linux, it i= s invariably >>> =C2=A0=C2=A0=C2=A0 disabled. So, it's misleading to just say Hystart = is default >>> =C2=A0=C2=A0=C2=A0 enabled, which implies it's widely used, when peop= le clearly find >>> =C2=A0=C2=A0=C2=A0 it has problems (which motivates Hystart++). I fou= nd this out >>> =C2=A0=C2=A0=C2=A0 through an informal survey I did at the Mar'18 IET= F in London by >>> =C2=A0=C2=A0=C2=A0 asking round the implementers of the most prevalen= t stacks (I >>> =C2=A0=C2=A0=C2=A0 would name names if I could find the note I later = sent to someone >>> =C2=A0=C2=A0=C2=A0 or to some list, but I can't find it - sry). >>>>>>> >>>>>>> =C2=A0=C2=A0=C2=A0 I'd like some citations on this versus anecdat= a if possible >>>>>>> =C2=A0=C2=A0=C2=A0 before I add that caveat to the text. Do large= deployments >>>>>>> =C2=A0=C2=A0=C2=A0 disable this? I haven't come across this sugge= stion in any >>>>>>> =C2=A0=C2=A0=C2=A0 Linux tuning guides to date. >> >> =C2=A0=C2=A0=C2=A0 -- ________________________________________________= ________________ >> =C2=A0=C2=A0=C2=A0 Bob Briscoehttp://bobbriscoe.net/ >> >> =C2=A0=C2=A0=C2=A0 _______________________________________________ >> =C2=A0=C2=A0=C2=A0 tcpm mailing list >> =C2=A0=C2=A0=C2=A0 tcpm@ietf.org >> =C2=A0=C2=A0=C2=A0 https://www.ietf.org/mailman/listinfo/tcpm >> =C2=A0=C2=A0=C2=A0 >> > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm --------------1455D1D7C3FC1353F392291E Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

I certainly do not qualify as a "large operator", but when implementing Cubic for QUIC I found that Hystart was critical for performance. Specifically, the classical slow-start is very prone to overshooting the capacity of the path, which causes large batches of errors. With web-like traffic, these batches of errors cause increase latency for the first transactions in a session, and thus a drop in perceived quality. Hystart solves that. With Hystart, I observed that a large fraction of sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. That's actually a big reason for moving from Cubic to BBR. Hystart suppresses the losses during the initial phase of the connection, but Cubic still relies on periodically testing the limit of path capacity and causing losses during the subsequent phase. BBR's probing for bottleneck bandwidth is much more conservative, does not cause such losses. It might be possible to adapt Cubic to also not cause losses, for example by ending an epoch early if too many CE marks are received or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work". I can't remember why I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now.


Bob


On 28/07/2021 00:46, Yuchung Cheng wrote:
Wait -- how is hystart "invariably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net <mailto:ietf@bobbriscoe.net>> wrote:

    Any large server operator out there who are using Cubic,
    If you're willing to state whether or not your operations disable
    Hystart, pls do so in reply.
    Then Praveen can cite this mailing list thread in the Hystart++
    draft, as requested below.

    If you are willing to reply privately, I would be willing to keep
    your confidences, and provide an anonymized result to the list.

    Cheers



    Bob

    On 27/07/2021 01:50, Praveen Balasubramanian wrote:

    Although Hystart is default enabled in Linux, it is invariably
    disabled. So, it's misleading to just say Hystart is default
    enabled, which implies it's widely used, when people clearly find
    it has problems (which motivates Hystart++). I found this out
    through an informal survey I did at the Mar'18 IETF in London by
    asking round the implementers of the most prevalent stacks (I
    would name names if I could find the note I later sent to someone
    or to some list, but I can't find it - sry).

    I'd like some citations on this versus anecdata if possible
    before I add that caveat to the text. Do large deployments
    disable this? I haven't come across this suggestion in any
    Linux tuning guides to date.

    --     ________________________________________________________________
    Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>

    _______________________________________________
    tcpm mailing list
    tcpm@ietf.org <mailto:tcpm@ietf.org>
    https://www.ietf.org/mailman/listinfo/tcpm
    <https://www.ietf.org/mailman/listinfo/tcpm> 



_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--------------1455D1D7C3FC1353F392291E-- From nobody Wed Jul 28 14:13:08 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 354653A20CC for ; Wed, 28 Jul 2021 14:13:07 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.55 X-Spam-Level: X-Spam-Status: No, score=-2.55 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J4J5cuY1CjiN for ; Wed, 28 Jul 2021 14:13:02 -0700 (PDT) Received: from rn-mailsvcp-ppex-lapp14.apple.com (rn-mailsvcp-ppex-lapp14.rno.apple.com [17.179.253.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 921D93A20CA for ; Wed, 28 Jul 2021 14:13:02 -0700 (PDT) Received: from pps.filterd (rn-mailsvcp-ppex-lapp14.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp14.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 16SJue64026700; Wed, 28 Jul 2021 14:12:57 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=yLkqzVKmsrF7i6fPAprFbLpdhaE5XgdqCDEvCAobIjM=; b=YYQW2/B/UGh/wUxN700Or/15oV/BzQnP354Ct0BYHFapvlyxZtIMSlsvkczAsKCnyTiC RU0ev/hSwKtq+hhfQZR/OJAIA4J30tqJsD7ME35L/qpKYRAT9N71QAzIEW+/xlb673Ye fY54RBRKxz+1153QXcgKudyYIJjPiU06CwTgaZ4mE+AYZO0gW0/oCt5Hf7m1LJESVU9f qlcqH/iErUlPXg9KjAp9Y8YtYK501CAghCxJ8ru9Mg7SCSvsK1hFrMmE/4wyRuiE9uu/ 7yyfBjr5z1Pow9rWyv4CWvqPjxZ+HlVQdy2yNYnlYHMpxyrJYjIGstNDHuU4a9VDy1ug GA== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by rn-mailsvcp-ppex-lapp14.rno.apple.com with ESMTP id 3a235q4xv9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 28 Jul 2021 14:12:57 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QWZ0113C49LC660@rn-mailsvcp-mta-lapp01.rno.apple.com>; Wed, 28 Jul 2021 14:12:57 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QWZ00C0048IJ000@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 14:12:57 -0700 (PDT) X-Va-A: X-Va-T-CD: 34c0898ec7cdaf6a757ba12558229a8b X-Va-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-Va-R-CD: 2eec4d4b333905bf2498bde820845c7b X-Va-CD: 0 X-Va-ID: d4ab3681-5f79-4d6d-9b54-1acbe64fae7e X-V-A: X-V-T-CD: 34c0898ec7cdaf6a757ba12558229a8b X-V-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-V-R-CD: 2eec4d4b333905bf2498bde820845c7b X-V-CD: 0 X-V-ID: 23484147-0e61-4540-98e3-9aa7ca3d2b21 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_10:2021-07-27, 2021-07-28 signatures=0 Received: from smtpclient.apple (unknown [17.234.58.195]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QWZ00C1P49KVG00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Wed, 28 Jul 2021 14:12:57 -0700 (PDT) From: Vidhi Goel Message-id: Content-type: multipart/alternative; boundary="Apple-Mail=_9785424E-2E44-4306-B0EF-54A5614BD58C" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.11\)) Date: Wed, 28 Jul 2021 14:12:56 -0700 In-reply-to: <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> To: Mark Allman , "tcpm@ietf.org Extensions" References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> X-Mailer: Apple Mail (2.3654.100.0.2.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-28_10:2021-07-27, 2021-07-28 signatures=0 Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 21:13:07 -0000 --Apple-Mail=_9785424E-2E44-4306-B0EF-54A5614BD58C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Resurrecting the 3465 thread. In the TCPM meeting at IETF 111, we discussed about this issue of L=3D2 = which is a MUST in RFC 3465. This is a very strict requirement and = stacks like Linux already doesn=E2=80=99t follow it. > The basic principle in the Linux code is that the sender should use = the ACKs to learn about the capacity of the path (both in volumetric and = rate dimensions), and should not ignore that information. This allows = the sender to quickly grow and achieve high throughput, even in the = presence of stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, = etc. >=20 > Considering bursts is important, but that can be tackled as an = orthogonal issue. Bursts are avoided in the Linux TCP ecosystem by the = combination of TSO autosizing, pacing, TSQ, and fair queueing. As Neal described, the congestion controller should use the information = in stretch ACKs to increase its congestion window so that it correctly = adjusts the cwnd based on available link capacity. Burstiness is an orthogonal issue which can be solved by pacing. QUIC loss recovery (RFC 9002) also follows this approach. Mark, As a lot of transport and congestion control drafts reference RFC 3465, = do you think we should update this RFC to reflect the current = deployment? This would also be useful for someone who is just starting = with a new implementation. Thanks, Vidhi > On Nov 27, 2019, at 5:14 AM, Mark Allman wrote: >=20 >=20 >> +Mark Allman >=20 > Just to clear it up, I *was* at BBN long ago when the ABC document > was written. It's a cool place to work. I recommend it. But, I > now hang out at ICSI. >=20 >> I believe that ABC was written to solve the problem with ACK >> counting by counting the number of bytes acknowledged for >> misbehaving receivers. Limiting the increase to 2*MSS was a good >> solution to avoid bursts at the time. >=20 > The main motivation behind ABC was to counteract delayed ACKs. The > common approach at the time was to just bump cwnd by one MSS every > time an ACK rolled in. So, if an ACK covered two segments because > the receiver was delaying the ACKs then the growth rate during slow > start was 1.5x per RTT instead of the 2x that was really > envisioned. During congestion avoidance the growth was 1 MSS every > 2 RTTs instead of the envisioned every RTT. >=20 > A secondary motivation was to counteract these ACK division attacks > that Savage taught us about. I.e., we could ACK an MSS-sized packet > one byte at a time and the sender would then increase the cwnd by > MSS*MSS bytes in the prevalent ACK counting scheme (i.e., cwnd would > get bumped by MSS bytes for every ACK). >=20 > The limit has two roots ... >=20 > (1) The limit is important in slow starts that follow an RTO. As > the RFC discusses, in this case we might retransmit a single > packet and this will cause the receiver's window to slide a > great deal. Therefore, an ACK may indicate that a ton of data > has left the network, but that isn't really the case. So, we > don't want to increase the cwnd based on all the new bytes > ACKed. >=20 > I have since mostly decided that this use of L is crude. > Probably there is a more elegant way to do it by using the > scoreboard and the SACK information to get a better > understanding of what left the network and when. That said, in > this case L is simple and probably about right most of the > time. >=20 > (2) I think there was some general conservativeness to bursts and > using L everywhere quelled some of the worry. Here L=3D2 was used > to exactly offset delayed ACKs. >=20 >> I agree that increasing the congestion window and controlling the >> burst rate are orthogonal issues. >=20 > Yes. In fact, we did subsequent research on mitigating bursts > because we never really thought of ABC as somehow the way to control > bursts (research papers available, but never went into RFCs). >=20 > And, in a world that leverages stretch ACKs as a routine I think > Linux's approach of not using an L may well be correct. Documenting > that and the reasoning behind it in modern networks seems useful to > me. >=20 > allman >=20 --Apple-Mail=_9785424E-2E44-4306-B0EF-54A5614BD58C Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Resurrecting the 3465 thread.

In the TCPM meeting at IETF 111, we discussed about = this issue of L=3D2 which is a MUST in RFC 3465. This is a very strict = requirement and stacks like Linux already doesn=E2=80=99t follow it.
The basic = principle in the Linux code is that the sender should use the ACKs to = learn about the capacity of the path (both in volumetric and rate = dimensions), and should not ignore that information. This allows the = sender to quickly grow and achieve high throughput, even in the = presence of stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, = etc.

Considering= bursts is important, but that can be tackled as an orthogonal issue. = Bursts are avoided in the Linux TCP ecosystem by the combination of TSO = autosizing, pacing, TSQ, and fair = queueing.

As Neal described, the congestion controller should use = the information in stretch ACKs to increase its congestion window so = that it correctly adjusts the cwnd based on available = link capacity.
Burstiness is an orthogonal issue = which can be solved by pacing.

QUIC loss recovery (RFC 9002) also = follows this approach.

Mark,
As a lot of = transport and congestion control drafts reference RFC 3465, do you think = we should update this RFC to reflect the current deployment? This would = also be useful for someone who is just starting with a new = implementation.

Thanks,
Vidhi

On Nov = 27, 2019, at 5:14 AM, Mark Allman <mallman@icir.org> wrote:


+Mark Allman

Just to clear it up, I *was* at = BBN long ago when the ABC document
was written.  It's = a cool place to work.  I recommend it.  But, I
now= hang out at ICSI.

I believe that ABC was written to solve the problem with = ACK
counting by counting the number of bytes acknowledged = for
misbehaving receivers. Limiting the increase to 2*MSS = was a good
solution to avoid bursts at the time.

The main motivation behind ABC = was to counteract delayed ACKs.  The
common approach = at the time was to just bump cwnd by one MSS every
time an = ACK rolled in.  So, if an ACK covered two segments because
the receiver was delaying the ACKs then the growth rate = during slow
start was 1.5x per RTT instead of the 2x that = was really
envisioned.  During congestion avoidance = the growth was 1 MSS every
2 RTTs instead of the = envisioned every RTT.

A secondary = motivation was to counteract these ACK division attacks
that= Savage taught us about.  I.e., we could ACK an MSS-sized packet
one byte at a time and the sender would then increase the = cwnd by
MSS*MSS bytes in the prevalent ACK counting scheme = (i.e., cwnd would
get bumped by MSS bytes for every = ACK).

The limit has two roots ...

(1) The limit is important in slow starts that = follow an RTO.  As
   the RFC = discusses, in this case we might retransmit a single
=    packet and this will cause the receiver's window to = slide a
   great deal.  Therefore, an = ACK may indicate that a ton of data
   has = left the network, but that isn't really the case.  So, we
   don't want to increase the cwnd based on = all the new bytes
   ACKed.

   I have since mostly decided = that this use of L is crude.
   Probably = there is a more elegant way to do it by using the
=    scoreboard and the SACK information to get a better
   understanding of what left the network and = when.  That said, in
   this case L = is simple and probably about right most of the
=    time.

(2) I think there = was some general conservativeness to bursts and
=    using L everywhere quelled some of the worry. =  Here L=3D2 was used
   to exactly = offset delayed ACKs.

I agree that increasing the congestion window = and controlling the
burst rate are orthogonal issues.

Yes.  In fact, we did = subsequent research on mitigating bursts
because we never = really thought of ABC as somehow the way to control
bursts = (research papers available, but never went into RFCs).

And, in a world that leverages stretch ACKs as a routine I = think
Linux's approach of not using an L may well be = correct.  Documenting
that and the reasoning behind = it in modern networks seems useful to
me.

allman


= --Apple-Mail=_9785424E-2E44-4306-B0EF-54A5614BD58C-- From nobody Wed Jul 28 16:21:00 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E4E4F3A0978 for ; Wed, 28 Jul 2021 16:20:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.097 X-Spam-Level: X-Spam-Status: No, score=-18.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uWJtp1Zy505Q for ; Wed, 28 Jul 2021 16:20:54 -0700 (PDT) Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 29B223A0976 for ; Wed, 28 Jul 2021 16:20:54 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id m19so2490402wms.0 for ; Wed, 28 Jul 2021 16:20:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=C25ou07oyl99bXHuP37iyZAnVlRNumHWYU0HF2PbHrA=; b=Yo74OR5JJJGDD/WNnwOyB8tPyogoulJEr1nvfJkSetF3sOTTnyXgIdcxu8yc0sb/EM +CJ56YWyP9eydPuZabCeQTa8b4baY4B7sdD7+FPIyEFF1Mavwq0/vmUtRvMt2f7LOuTb y02xpfGYS2L9/73FbdfK+BqaGecRWVRV7UKAnjieMmZxs4eHdz7z4Snz5jgB2j/CPLBt V3Ix2ZISb7ToOfBz4yNj/AGXAGbarvPJl5UU+BCfK4wTp3bjBx3F/0URwA31SU/ayOys VB+I0TQ9Ck75Eo4Wmva6c5Y6S4gUulXlFQQKxeNq8dYuJMBRDvzuXObeCT47jfWcQgqw +WEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=C25ou07oyl99bXHuP37iyZAnVlRNumHWYU0HF2PbHrA=; b=Mj4IInEEPl9/gjSQHgevR6tQWOt28GcO12xXheBFqVPY51yPJ0xjYVHOSI9TECHO5L P7SVnFyjutyMx4jdOLN54ofVkDkMM+fP2i89gwNyS70RP5hMIe4kYB3vfOLNMv8TYLac ei7shyzXqE2bNh+AR2OESM1cjUMa2VVzbzZnVYsjywYnHExaFzQ5OPH7igBwnKV9HYvo WiVK3+pw1bG4wsOV/3QN5NTgslUItRprxZ6KgUI3KvLonw8tRKtpKTmRz7qpy0YTjEBx YTi4msRHi1T0XqjbwnVmrShnm0RmwwqC63S2731Fq59KodjBIzWRLp9vIoFLu7rsitMF L7Kw== X-Gm-Message-State: AOAM533vLIyI2Ne5e2HjS/w/tVzuKkNfJFZuLMVejMmiPkvBmiG3D4dr YM2cT3JYooLd1AA3aorvIN7mOo1xqG9L7rKqJ3OFHw== X-Google-Smtp-Source: ABdhPJy0F5ZwwkwbPtx0lGD2YqcoX1GKgxB1npNPxu/RCyJmt1bGOQO7PZ/ZLT0U9xfsJMWIP/uSRZDZpNUTAIRv3H8= X-Received: by 2002:a05:600c:198a:: with SMTP id t10mr11496654wmq.32.1627514450979; Wed, 28 Jul 2021 16:20:50 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> In-Reply-To: From: Yuchung Cheng Date: Wed, 28 Jul 2021 16:20:14 -0700 Message-ID: To: Vidhi Goel Cc: Mark Allman , "tcpm@ietf.org Extensions" , Neal Cardwell Content-Type: multipart/alternative; boundary="000000000000b8205605c8373c71" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 23:20:59 -0000 --000000000000b8205605c8373c71 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you Vidhi and Mark for supporting an ABC update. To give this a start, my recommended update is: 1) if the sender uses some form of pacing to send data packets, L is RECOMMENDED to be the effective window worth of packets. Pacing here refers to spread packet transmission following a rate based on the congestion window and round trip. Additionally if SACK is supported, the same L applies in slow start after RTO 2) otherwise L is RECOMMENDED to be 8 Thoughts? On Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel wrote: > Resurrecting the 3465 thread. > > In the TCPM meeting at IETF 111, we discussed about this issue of L=3D2 > which is a MUST in RFC 3465. This is a very strict requirement and stacks > like Linux already doesn=E2=80=99t follow it. > > The basic principle in the Linux code is that the sender should use the > ACKs to learn about the capacity of the path (both in volumetric and rate > dimensions), and should not ignore that information. This allows the send= er > to quickly grow and achieve high throughput, even in the presence of > stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc. > > Considering bursts is important, but that can be tackled as an orthogonal > issue. Bursts are avoided in the Linux TCP ecosystem by the combination o= f > TSO autosizing, pacing, TSQ, and fair queueing. > > > As Neal described, the congestion controller should use the information i= n > stretch ACKs to increase its congestion window so that it correctly adjus= ts > the *cwnd* based on available link capacity. > Burstiness is an orthogonal issue which can be solved by pacing. > > QUIC loss recovery (RFC 9002) also follows this approach. > > *Mark*, > As a lot of transport and congestion control drafts reference RFC 3465, d= o > you think we should update this RFC to reflect the current deployment? Th= is > would also be useful for someone who is just starting with a new > implementation. > > Thanks, > Vidhi > > On Nov 27, 2019, at 5:14 AM, Mark Allman wrote: > > > +Mark Allman > > > Just to clear it up, I *was* at BBN long ago when the ABC document > was written. It's a cool place to work. I recommend it. But, I > now hang out at ICSI. > > I believe that ABC was written to solve the problem with ACK > counting by counting the number of bytes acknowledged for > misbehaving receivers. Limiting the increase to 2*MSS was a good > solution to avoid bursts at the time. > > > The main motivation behind ABC was to counteract delayed ACKs. The > common approach at the time was to just bump cwnd by one MSS every > time an ACK rolled in. So, if an ACK covered two segments because > the receiver was delaying the ACKs then the growth rate during slow > start was 1.5x per RTT instead of the 2x that was really > envisioned. During congestion avoidance the growth was 1 MSS every > 2 RTTs instead of the envisioned every RTT. > > A secondary motivation was to counteract these ACK division attacks > that Savage taught us about. I.e., we could ACK an MSS-sized packet > one byte at a time and the sender would then increase the cwnd by > MSS*MSS bytes in the prevalent ACK counting scheme (i.e., cwnd would > get bumped by MSS bytes for every ACK). > > The limit has two roots ... > > (1) The limit is important in slow starts that follow an RTO. As > the RFC discusses, in this case we might retransmit a single > packet and this will cause the receiver's window to slide a > great deal. Therefore, an ACK may indicate that a ton of data > has left the network, but that isn't really the case. So, we > don't want to increase the cwnd based on all the new bytes > ACKed. > > I have since mostly decided that this use of L is crude. > Probably there is a more elegant way to do it by using the > scoreboard and the SACK information to get a better > understanding of what left the network and when. That said, in > this case L is simple and probably about right most of the > time. > > (2) I think there was some general conservativeness to bursts and > using L everywhere quelled some of the worry. Here L=3D2 was used > to exactly offset delayed ACKs. > > I agree that increasing the congestion window and controlling the > burst rate are orthogonal issues. > > > Yes. In fact, we did subsequent research on mitigating bursts > because we never really thought of ABC as somehow the way to control > bursts (research papers available, but never went into RFCs). > > And, in a world that leverages stretch ACKs as a routine I think > Linux's approach of not using an L may well be correct. Documenting > that and the reasoning behind it in modern networks seems useful to > me. > > allman > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --000000000000b8205605c8373c71 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you Vidhi and Mark for supporting an ABC update.
To give this a start, my recommended update is:
<= br>1) if the sender uses some form of pacing to send data packets, L is REC= OMMENDED to be the effective window worth of packets. Pacing here refers to= spread packet transmission following a rate based on the congestion window= and round trip. Additionally if SACK is supported, the same L applies in s= low start after RTO=C2=A0

2) otherwise L is RECOMM= ENDED to be 8=C2=A0

Thoughts?

=
= On Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org= > wrote:
=
Resurrecting the 3465 thread.

In the TCPM mee= ting at IETF 111, we discussed about this issue of L=3D2 which is a MUST in= RFC 3465. This is a very strict requirement and stacks like Linux already = doesn=E2=80=99t follow it.
<= div class=3D"gmail_quote">
The basic principle in the Linux code is tha= t the sender should use the ACKs to learn about the capacity of the path (b= oth in volumetric and rate dimensions), and should not ignore that informat= ion. This allows the sender to quickly grow and achieve high throughput,=C2= =A0even in the presence of stretch ACKs, which are pervasive, due to TSO/GS= O, GRO/LRO, etc.

Considering bursts is important, = but that can be tackled as an orthogonal issue. Bursts are avoided in the L= inux TCP ecosystem by the combination of TSO autosizing, pacing, TSQ, and f= air queueing.

As Neal describe= d, the congestion controller should use the information in stretch ACKs to = increase its congestion window so that it correctly adjusts the cwnd= based on available link capacity.
Burstiness is an orthogonal is= sue which can be solved by pacing.

QUIC loss recov= ery (RFC 9002) also follows this approach.

Mark= ,
As a lot of transport and congestion control drafts referen= ce RFC 3465, do you think we should update this RFC to reflect the current = deployment? This would also be useful for someone who is just starting with= a new implementation.

Thanks,
Vidhi
=

On Nov 27, 2019, at 5:14 AM, Mark A= llman <mallman@ici= r.org> wrote:


+Mark= Allman

Just to clear it up, I *was* at BBN long ago wh= en the ABC document
was written.=C2=A0 It's a cool place to work.=C2= =A0 I recommend it.=C2=A0 But, I
now hang out at ICSI.

I believe that ABC was written to solve the problem with AC= K
counting by counting the number of bytes acknowledged for
misbehavi= ng receivers. Limiting the increase to 2*MSS was a good
solution to avoi= d bursts at the time.

The main motivation behind ABC wa= s to counteract delayed ACKs.=C2=A0 The
common approach at the time was = to just bump cwnd by one MSS every
time an ACK rolled in.=C2=A0 So, if a= n ACK covered two segments because
the receiver was delaying the ACKs th= en the growth rate during slow
start was 1.5x per RTT instead of the 2x = that was really
envisioned.=C2=A0 During congestion avoidance the growth= was 1 MSS every
2 RTTs instead of the envisioned every RTT.

A se= condary motivation was to counteract these ACK division attacks
that Sav= age taught us about.=C2=A0 I.e., we could ACK an MSS-sized packet
one by= te at a time and the sender would then increase the cwnd by
MSS*MSS byte= s in the prevalent ACK counting scheme (i.e., cwnd would
get bumped by M= SS bytes for every ACK).

The limit has two roots ...

(1) The = limit is important in slow starts that follow an RTO.=C2=A0 As
=C2=A0= =C2=A0=C2=A0the RFC discusses, in this case we might retransmit a single =C2=A0=C2=A0=C2=A0packet and this will cause the receiver's window to= slide a
=C2=A0=C2=A0=C2=A0great deal.=C2=A0 Therefore, an ACK may indi= cate that a ton of data
=C2=A0=C2=A0=C2=A0has left the network, but tha= t isn't really the case.=C2=A0 So, we
=C2=A0=C2=A0=C2=A0don't w= ant to increase the cwnd based on all the new bytes
=C2=A0=C2=A0=C2=A0A= CKed.

=C2=A0=C2=A0=C2=A0I have since mostly decided that this use o= f L is crude.
=C2=A0=C2=A0=C2=A0Probably there is a more elegant way to= do it by using the
=C2=A0=C2=A0=C2=A0scoreboard and the SACK informati= on to get a better
=C2=A0=C2=A0=C2=A0understanding of what left the net= work and when.=C2=A0 That said, in
=C2=A0=C2=A0=C2=A0this case L is sim= ple and probably about right most of the
=C2=A0=C2=A0=C2=A0time.
(2) I think there was some general conservativeness to bursts and
=C2= =A0=C2=A0=C2=A0using L everywhere quelled some of the worry.=C2=A0 Here L= =3D2 was used
=C2=A0=C2=A0=C2=A0to exactly offset delayed ACKs.

=
I agree that increasing the congestion window and= controlling the
burst rate are orthogonal issues.

Y= es.=C2=A0 In fact, we did subsequent research on mitigating bursts
becau= se we never really thought of ABC as somehow the way to control
bursts (= research papers available, but never went into RFCs).

And, in a worl= d that leverages stretch ACKs as a routine I think
Linux's approach = of not using an L may well be correct.=C2=A0 Documenting
that and the re= asoning behind it in modern networks seems useful to
me.

allman

______________________= _________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--000000000000b8205605c8373c71-- From nobody Wed Jul 28 16:36:00 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E92F33A0A9A for ; Wed, 28 Jul 2021 16:35:58 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S5iuJMZ0lvXg for ; Wed, 28 Jul 2021 16:35:53 -0700 (PDT) Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F28A63A0A9C for ; Wed, 28 Jul 2021 16:35:52 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id e25-20020a05600c4b99b0290253418ba0fbso2760207wmp.1 for ; Wed, 28 Jul 2021 16:35:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gApTucgVNPHrWD54YxSivcKID7ZkMCaOSRlIyyfa8P4=; b=i86HC68c8SrNjXv6lck0v34bDZ1qYjrGP2S1xG0lhn8hyE1LqxN6THIGDe2U8UWLQV go4VYi/aW8M73wB9hTgPcr/gy61ckDn3lKA0BGO5xCJ6NL0u/xC9zkRjgpsfpAqgZWKi 81yrh49x8s+9g49GMqKDrBFeEFsLsVOrsnH2FBJzO4/WRC4eNNz4nlC6DV1KgPUJzsPC PMPRgIX81jasnc03AWlsgv4iCj7tmyizSzTPJTi020h10BKVhmASARIqdCHx05jNpK8T trODlabq3UsHwAgfrp8yYPSEaChKDgmsboU+Dz2qCvqG54eQVGhu0MwKEv57/0S4MHig QlcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gApTucgVNPHrWD54YxSivcKID7ZkMCaOSRlIyyfa8P4=; b=cnX96KTpNxBwCK3eeCDuo+rBNw5mVG0WYyxss5Pd9sG8MkwHmh1H7SwaP5Rm7t4jXO 0OO90qgd0FDPik3PZSgjdZIVuscFI0eRm2CQvYy79L/nXbCsGeJUPqYQcreBxeBtp1Hy WrEuDx/QfdHPNWgw8iri6WNIyJWFc20uSE3T94yol6IvLjmzFSSvC/kXtGIOTUmal7ee 3iQ5JBF0IYViEqHtlP+KhPC4M3r4VcuHfV3YTaLSBAXQn4iZAIW1yTmXaqwKIY5S442K PKE1xO/zWvUIy1MlkkMTMxIgttmoGtr/8onV73VZdfSJk8E77ypT8mKrlQxOKIeaKDLi E6ew== X-Gm-Message-State: AOAM5335w4MJYoAZEMR3K8v0+bxmR//GrnoizPK6xaPbdg40BYkzpovN VoPGMf32kyTkuttv5hK/BWgSvck5wln4s0hyCswuOg== X-Google-Smtp-Source: ABdhPJyYrGCookS4yaqG9N775bo5YKTtGjtbBfjbLgLttF9xClNGam2d8KkCa8UAqgPgBDDOd0bmOxZ8bY+1aiZXp6Y= X-Received: by 2002:a7b:ce10:: with SMTP id m16mr11206926wmc.75.1627515346072; Wed, 28 Jul 2021 16:35:46 -0700 (PDT) MIME-Version: 1.0 References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> In-Reply-To: <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> From: Yuchung Cheng Date: Wed, 28 Jul 2021 16:35:08 -0700 Message-ID: To: huitema@huitema.net Cc: Bob Briscoe , "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" , Neal Cardwell Content-Type: multipart/alternative; boundary="00000000000012737505c83772ed" Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Jul 2021 23:35:59 -0000 --00000000000012737505c83772ed Content-Type: text/plain; charset="UTF-8" same experience with Chrisitian For more than half a decade Google traffic used Linux CUBIC Hystart. This includes google.com and YouTube public Internet traffic, as well as internal Google TCP traffic. When we deployed pacing around 2013 we disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as it causes false SS exit. But we continued to use the stock Hystart Delay mechanism. ~2014-15 we switched to BBR which uses a different startup more robust to delay jitters. But IMO the simpler hystart mitigating SS over-shoot is still highly valuable for Cubic and other congestion controls, as Internet bandwidth continues to hike. Did those providers disable hystart b/c of its poor interaction between the ack-train mode and pacing? Maybe it's time to disable the ack-train approach in the upstream Linux when hystart++ is standardized. On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema wrote: > I certainly do not qualify as a "large operator", but when implementing > Cubic for QUIC I found that Hystart was critical for performance. > Specifically, the classical slow-start is very prone to overshooting the > capacity of the path, which causes large batches of errors. With web-like > traffic, these batches of errors cause increase latency for the first > transactions in a session, and thus a drop in perceived quality. Hystart > solves that. With Hystart, I observed that a large fraction of sessions do > not experience any packet loss at all. > > As an aside, we should strive for this suppression of packet losses. > That's actually a big reason for moving from Cubic to BBR. Hystart > suppresses the losses during the initial phase of the connection, but Cubic > still relies on periodically testing the limit of path capacity and causing > losses during the subsequent phase. BBR's probing for bottleneck bandwidth > is much more conservative, does not cause such losses. It might be possible > to adapt Cubic to also not cause losses, for example by ending an epoch > early if too many CE marks are received or if the RTT increases. That would > be worth trying. > > -- Christian Huitema > On 7/28/2021 3:19 AM, Bob Briscoe wrote: > > Yuchung, > > It was during a Mar 2018 ad hoc workshop Jana had organized at the London > IETF entitled 'BBR and the intersection with other work". I can't remember > why I needed to know at the time, but during the break I approached the > major server operators individually, established whether they used Cubic, > and if so asked whether they used Hystart or disabled it. It's hard to > anonymize the results, because IMMSMC all those that used Cubic said they > disabled Hystart. > > If this isn't correct, then maybe people who replied at the time thought > they were being asked something else. Or maybe it was correct then but > isn't now. > > > Bob > > > On 28/07/2021 00:46, Yuchung Cheng wrote: > > Wait -- how is hystart "invariably disabled" in Linux (cubic)!? > > What data indicates that > > On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe > wrote: > > Any large server operator out there who are using Cubic, > If you're willing to state whether or not your operations disable > Hystart, pls do so in reply. > Then Praveen can cite this mailing list thread in the Hystart++ > draft, as requested below. > > If you are willing to reply privately, I would be willing to keep > your confidences, and provide an anonymized result to the list. > > Cheers > > > > Bob > > On 27/07/2021 01:50, Praveen Balasubramanian wrote: > > > Although Hystart is default enabled in Linux, it is invariably > disabled. So, it's misleading to just say Hystart is default > enabled, which implies it's widely used, when people clearly find > it has problems (which motivates Hystart++). I found this out > through an informal survey I did at the Mar'18 IETF in London by > asking round the implementers of the most prevalent stacks (I > would name names if I could find the note I later sent to someone > or to some list, but I can't find it - sry). > > > I'd like some citations on this versus anecdata if possible > before I add that caveat to the text. Do large deployments > disable this? I haven't come across this suggestion in any > Linux tuning guides to date. > > > -- > ________________________________________________________________ > Bob Briscoehttp://bobbriscoe.net/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > > > > > > _______________________________________________ > tcpm mailing listtcpm@ietf.orghttps://www.ietf.org/mailman/listinfo/tcpm > > --00000000000012737505c83772ed Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
same experience with Chrisitian

For mor= e than half a decade Google traffic used=C2=A0Linux CUBIC Hystart. This inc= ludes=C2=A0google.com= =C2=A0and YouTube public Internet traffic, as well as internal Google TCP t= raffic.=C2=A0When we deployed pacing around 2013 we disabled the Hystart AC= K-train mechanism (that Hystart++ prohibits now) as it causes false SS exit= .=C2=A0But we continued to use the stock Hystart Delay mechanism. ~2014-15 = we switched to BBR which uses a different startup more robust to delay jitt= ers. But IMO the simpler hystart mitigating SS over-shoot is still highly v= aluable for Cubic and other congestion controls, as Internet bandwidth cont= inues=C2=A0to hike.

Did those providers disable hy= start b/c of its poor interaction between the ack-train=C2=A0mode and pacin= g? Maybe it's time to disable the ack-train approach in the upstream Li= nux=C2=A0when hystart++=C2=A0is standardized.

On Wed, Jul 28, 2021 at = 9:47 AM Christian Huitema <huitema@huitema.net> wrote:
=20 =20 =20

I certainly do not qualify as a "large operator", but when implementing Cubic for QUIC I found that Hystart was critical for performance. Specifically, the classical slow-start is very prone to overshooting the capacity of the path, which causes large batches of errors. With web-like traffic, these batches of errors cause increase latency for the first transactions in a session, and thus a drop in perceived quality. Hystart solves that. With Hystart, I observed that a large fraction of sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. That's actually a big reason for moving from Cubic to BBR= . Hystart suppresses the losses during the initial phase of the connection, but Cubic still relies on periodically testing the limit of path capacity and causing losses during the subsequent phase. BBR's probing for bottleneck bandwidth is much more conservative, does not cause such losses. It might be possible to adapt Cubic to also not cause losses, for example by ending an epoch early if too many CE marks are received or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work&qu= ot;. I can't remember why I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now.


Bob


On 28/07/2021 00:46, Yuchung Cheng wrote:
Wait -- how is hystart "invariably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@b= obbriscoe.net = <mailto:ietf@bobbriscoe.net>> wrote:

=C2=A0=C2=A0=C2=A0 Any large server operator out there who are usin= g Cubic,
=C2=A0=C2=A0=C2=A0 If you're willing to state whether or not yo= ur operations disable
=C2=A0=C2=A0=C2=A0 Hystart, pls do so in reply.
=C2=A0=C2=A0=C2=A0 Then Praveen can cite this mailing list thread i= n the Hystart++
=C2=A0=C2=A0=C2=A0 draft, as requested below.

=C2=A0=C2=A0=C2=A0 If you are willing to reply privately, I would b= e willing to keep
=C2=A0=C2=A0=C2=A0 your confidences, and provide an anonymized resu= lt to the list.

=C2=A0=C2=A0=C2=A0 Cheers



=C2=A0=C2=A0=C2=A0 Bob

=C2=A0=C2=A0=C2=A0 On 27/07/2021 01:50, Praveen Balasubramanian wro= te:

=C2=A0=C2=A0=C2=A0 Although Hystart is default enabled in Linux, = it is invariably
=C2=A0=C2=A0=C2=A0 disabled. So, it's misleading to just say = Hystart is default
=C2=A0=C2=A0=C2=A0 enabled, which implies it's widely used, w= hen people clearly find
=C2=A0=C2=A0=C2=A0 it has problems (which motivates Hystart++). I= found this out
=C2=A0=C2=A0=C2=A0 through an informal survey I did at the Mar= 9;18 IETF in London by
=C2=A0=C2=A0=C2=A0 asking round the implementers of the most prev= alent stacks (I
=C2=A0=C2=A0=C2=A0 would name names if I could find the note I la= ter sent to someone
=C2=A0=C2=A0=C2=A0 or to some list, but I can't find it - sry= ).

=C2=A0=C2=A0=C2=A0 I'd like some citations on this ve= rsus anecdata if possible
=C2=A0=C2=A0=C2=A0 before I add that caveat to the text. = Do large deployments
=C2=A0=C2=A0=C2=A0 disable this? I haven't come acros= s this suggestion in any
=C2=A0=C2=A0=C2=A0 Linux tuning guides to date.

=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 ________________________________________________________________
=C2=A0=C2=A0=C2=A0 Bob Briscoehttp://bobbriscoe.net/=C2=A0 <http://bob= briscoe.net/>

=C2=A0=C2=A0=C2=A0 _______________________________________________
=C2=A0=C2=A0=C2=A0 tcpm mailing list
=C2=A0=C2=A0=C2=A0 tcpm@ietf.org <= ;mailto:tcpm@ietf.org>
=C2=A0=C2=A0=C2=A0 https://www.ietf.org/mailman/listinfo/tcpm
=C2=A0=C2=A0=C2=A0 <https://www.ietf.org/mailman/listinfo/tcpm>= 



_______________________________________________
tcpm mailing list
tcpm@ietf.org
ht=
tps://www.ietf.org/mailman/listinfo/tcpm
--00000000000012737505c83772ed-- From nobody Wed Jul 28 17:42:01 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1D8933A078E for ; Wed, 28 Jul 2021 17:41:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ICAUsv2MKpye for ; Wed, 28 Jul 2021 17:41:46 -0700 (PDT) Received: from mail-ua1-x933.google.com (mail-ua1-x933.google.com [IPv6:2607:f8b0:4864:20::933]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BAABE3A0765 for ; Wed, 28 Jul 2021 17:41:46 -0700 (PDT) Received: by mail-ua1-x933.google.com with SMTP id t26so1834059uao.12 for ; Wed, 28 Jul 2021 17:41:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+qz7jmPVwLw81N0rAZQfVWW1tWX+aq7F1SQz3Dxt9uI=; b=nWXON9xaqOxPRYpV5hXdDqYl4m4F5xOYHIZORVOm0yKmYwrk6Zg8xXbLUKfUeG4Ur5 NcMY2ftt1Y7Txbe5pXDHDG8zHjlXVTPH1xzutcYsiBhGl0qLbxCzaAHsYzkk2Zig4E+c LuYQEPOC/hmWzXhY06jUGZSK2QqHnVyTCbYHahOSCcyWWP3xYTJUh1tPucjAgD9qeKOA pHuxQSJbn2qbjHkTEByIDcuXLkCvS/bUru1W/89tIJodRuHLD7JxTZMVECrJJJINVgIi 1lAFRBjJw9fE5bDGGRu2qwcErnsBeWk0VTbRfXRq4JVnop2lagEjbBngbK1GVCaitria ecHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+qz7jmPVwLw81N0rAZQfVWW1tWX+aq7F1SQz3Dxt9uI=; b=l0v1FPU8VkKz7Dxc3qBWzGbuKbtrcY50nIKX5OFL4qusrdaGHAa6hejdgPiqrAOGm5 /LRqxJcewzy9D3e30mGiBUwqWjYA0K6/tLYEiGS5kJ2k+1kpZzkIx6hNlQR5CE+cZOF+ tGRv4nW1M0zLptxeVir7+MVvnZ/s+uUE474/SFTlzrz5KOxId3e6KnY/at1BPWJGmyKg 3mIFMXZdNgzY2rjIOe6ZUUv1w/6Q7oukFlRkSWMxKJ7FSfl6P7u3Lc3TE9H1u7itiI+o 7Lz95Xx2bL/eTNPbI4r/4SqKq3a5OIggYL83FMFrH1cGBHMk6jUAqje0IhTgvUqcELwZ XwGw== X-Gm-Message-State: AOAM533RsUQM93nxxWoPh/dUGVK1x6TEpG7BizpQakT7fWJ91e2aqn+g f9tKT7lNISpa7ZEl4Epu7uTsdjAj+TIX3uJzhocgSw== X-Google-Smtp-Source: ABdhPJwxW/QmyWtqbpblkdFQJzs8YHoAPTOms8k3FWnzA1BI7Q1yf/RYV+kj1aVnduISHDiI+q3bBdSNQQLBGimK76A= X-Received: by 2002:ab0:25c5:: with SMTP id y5mr2651090uan.142.1627519304247; Wed, 28 Jul 2021 17:41:44 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> In-Reply-To: From: Neal Cardwell Date: Wed, 28 Jul 2021 20:41:27 -0400 Message-ID: To: Yuchung Cheng Cc: Vidhi Goel , Mark Allman , "tcpm@ietf.org Extensions" Content-Type: multipart/alternative; boundary="000000000000ff1d4605c8385de2" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 00:41:59 -0000 --000000000000ff1d4605c8385de2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I would vote that the guidance should be something like: On an ACK, don't increase the cwnd by more than the amount of data ACKed. This implies: o slow-start is limited to doubling each round trip o the Savage ACK division attacks are squashed I don't think limiting cwnd growth to some constant L less than cwnd is the best road to go down, since: (a) At high speeds (at least >10Gbps) TCP has needed to operate in TSO/GRO/LRO/GRO aggregates of 64KBytes for a decade or more, and is now moving on to larger aggregates to go beyond 100Gbps (see Eric Dumazet's nice talk at the Linux Netdev conference a few weeks ago). Having a constant L is not scalable or future-proof. I would imagine any constant L lower than 64KBytes will be ignored, in practice, with good reason. A constant L bigger than 64KBytes will be out of date soon, if it's not already. (b) If we're worried about bursts (which we should be), then we should recommend pacing. And RFC 7661 already does that: https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2 "A TCP sender in the non-validated phase SHOULD control the maximum burst size, e.g., using a rate-based pacing algorithm in which a sender paces out the cwnd over its estimate of the RTT, or some other method, to prevent many segments being transmitted contiguously at line-rate." So a sender that follows RFC 7661 already has a solution for bursts: pacing. (b) If an implementation ignores RFC 7661 and does not do pacing, then limiting cwnd *growth* to L per ACK does not prevent bursts. A sender that ignores RFC 7661 will be frequently restarting from idle to blast a full cwnd at line rate. Those restart-from-idle cwnd bursts are already super-common in some of the most common types of TCP traffic: web, RPC, streaming video. Having a particular L limit for cwnd growth will not save us from those cwnd-magnitude bursts. Just my two cents on that, neal On Wed, Jul 28, 2021 at 7:20 PM Yuchung Cheng wrote: > Thank you Vidhi and Mark for supporting an ABC update. > > To give this a start, my recommended update is: > > 1) if the sender uses some form of pacing to send data packets, L is > RECOMMENDED to be the effective window worth of packets. Pacing here refe= rs > to spread packet transmission following a rate based on the congestion > window and round trip. Additionally if SACK is supported, the same L > applies in slow start after RTO > > 2) otherwise L is RECOMMENDED to be 8 > > Thoughts? > > > On Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel 40apple.com@dmarc.ietf.org> wrote: > >> Resurrecting the 3465 thread. >> >> In the TCPM meeting at IETF 111, we discussed about this issue of L=3D2 >> which is a MUST in RFC 3465. This is a very strict requirement and stack= s >> like Linux already doesn=E2=80=99t follow it. >> >> The basic principle in the Linux code is that the sender should use the >> ACKs to learn about the capacity of the path (both in volumetric and rat= e >> dimensions), and should not ignore that information. This allows the sen= der >> to quickly grow and achieve high throughput, even in the presence of >> stretch ACKs, which are pervasive, due to TSO/GSO, GRO/LRO, etc. >> >> Considering bursts is important, but that can be tackled as an orthogona= l >> issue. Bursts are avoided in the Linux TCP ecosystem by the combination = of >> TSO autosizing, pacing, TSQ, and fair queueing. >> >> >> As Neal described, the congestion controller should use the information >> in stretch ACKs to increase its congestion window so that it correctly >> adjusts the *cwnd* based on available link capacity. >> Burstiness is an orthogonal issue which can be solved by pacing. >> >> QUIC loss recovery (RFC 9002) also follows this approach. >> >> *Mark*, >> As a lot of transport and congestion control drafts reference RFC 3465, >> do you think we should update this RFC to reflect the current deployment= ? >> This would also be useful for someone who is just starting with a new >> implementation. >> >> Thanks, >> Vidhi >> >> On Nov 27, 2019, at 5:14 AM, Mark Allman wrote: >> >> >> +Mark Allman >> >> >> Just to clear it up, I *was* at BBN long ago when the ABC document >> was written. It's a cool place to work. I recommend it. But, I >> now hang out at ICSI. >> >> I believe that ABC was written to solve the problem with ACK >> counting by counting the number of bytes acknowledged for >> misbehaving receivers. Limiting the increase to 2*MSS was a good >> solution to avoid bursts at the time. >> >> >> The main motivation behind ABC was to counteract delayed ACKs. The >> common approach at the time was to just bump cwnd by one MSS every >> time an ACK rolled in. So, if an ACK covered two segments because >> the receiver was delaying the ACKs then the growth rate during slow >> start was 1.5x per RTT instead of the 2x that was really >> envisioned. During congestion avoidance the growth was 1 MSS every >> 2 RTTs instead of the envisioned every RTT. >> >> A secondary motivation was to counteract these ACK division attacks >> that Savage taught us about. I.e., we could ACK an MSS-sized packet >> one byte at a time and the sender would then increase the cwnd by >> MSS*MSS bytes in the prevalent ACK counting scheme (i.e., cwnd would >> get bumped by MSS bytes for every ACK). >> >> The limit has two roots ... >> >> (1) The limit is important in slow starts that follow an RTO. As >> the RFC discusses, in this case we might retransmit a single >> packet and this will cause the receiver's window to slide a >> great deal. Therefore, an ACK may indicate that a ton of data >> has left the network, but that isn't really the case. So, we >> don't want to increase the cwnd based on all the new bytes >> ACKed. >> >> I have since mostly decided that this use of L is crude. >> Probably there is a more elegant way to do it by using the >> scoreboard and the SACK information to get a better >> understanding of what left the network and when. That said, in >> this case L is simple and probably about right most of the >> time. >> >> (2) I think there was some general conservativeness to bursts and >> using L everywhere quelled some of the worry. Here L=3D2 was used >> to exactly offset delayed ACKs. >> >> I agree that increasing the congestion window and controlling the >> burst rate are orthogonal issues. >> >> >> Yes. In fact, we did subsequent research on mitigating bursts >> because we never really thought of ABC as somehow the way to control >> bursts (research papers available, but never went into RFCs). >> >> And, in a world that leverages stretch ACKs as a routine I think >> Linux's approach of not using an L may well be correct. Documenting >> that and the reasoning behind it in modern networks seems useful to >> me. >> >> allman >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm >> > --000000000000ff1d4605c8385de2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I would vote that the guidance should be = something like:

=C2=A0 On an ACK, don't increase the cwnd by mor= e than the amount of data ACKed.

This implies:

=C2=A0o slow-s= tart is limited to doubling each round trip

=C2=A0o the Savage ACK d= ivision attacks are squashed

I don't think limiting cwnd growth = to some constant L less than cwnd is the best road to go down, since:
(a) At high speeds (at least >10Gbps) TCP has needed to operate in TSO= /GRO/LRO/GRO aggregates of 64KBytes for a decade or more, and is now moving= on to larger aggregates to go beyond 100Gbps (see Eric Dumazet's nice = talk at the Linux Netdev conference a few weeks ago). Having a constant L i= s not scalable or future-proof. I would imagine any constant L lower than 6= 4KBytes will be ignored, in practice, with good reason. A constant=C2=A0L b= igger than 64KBytes will be out of date soon,=C2=A0if it's not already.=

(b) If we're worried about bursts (which we should be), then we= should recommend pacing. And RFC 7661 already does that:
=C2=A0 https://da= tatracker.ietf.org/doc/html/rfc7661#section-4.4.2

=C2=A0 "A= TCP sender in the non-validated phase SHOULD control the maximum
=C2=A0= =C2=A0burst size, e.g., using a rate-based pacing algorithm in which a
= =C2=A0 =C2=A0sender paces out the cwnd over its estimate of the RTT, or som= e
=C2=A0 =C2=A0other method, to prevent many segments being transmitted<= br>=C2=A0 =C2=A0contiguously at line-rate."

=C2=A0 So a sender that follows RFC 7661 already has a solution for bur= sts: pacing.

(b) If an= implementation ignores RFC 7661=C2=A0and does not do pacing, then limiting= cwnd *growth* to L per ACK does not prevent bursts. A sender that ignores = RFC 7661 will be frequently restarting from idle to blast a full cwnd at li= ne rate. Those restart-from-idle cwnd bursts are already super-common in so= me of the most=C2=A0common types of TCP traffic: web, RPC, streaming video.= Having a=C2=A0particular L limit for cwnd growth will not save us from tho= se cwnd-magnitude bursts.

Just my two = cents on that,
neal


On Wed, Jul 28, 2021 at 7:20 PM= Yuchung Cheng <ycheng@google.com> wrote:
Thank you Vidhi and Mark for supporting an ABC update.
<= br>
To give this a start, my recommended update is:
1) if the sender uses some form of pacing to send data packets, L is RECOM= MENDED to be the effective window worth of packets. Pacing here refers to s= pread packet transmission following a rate based on the congestion window a= nd round trip. Additionally if SACK is supported, the same L applies in slo= w start after RTO=C2=A0

2) otherwise L is RECOMMEN= DED to be 8=C2=A0

Thoughts?


On= Wed, Jul 28, 2021 at 2:13 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Resurrecting the 3465 thread.

In the TCPM meeti= ng at IETF 111, we discussed about this issue of L=3D2 which is a MUST in R= FC 3465. This is a very strict requirement and stacks like Linux already do= esn=E2=80=99t follow it.
The basic principle in the Linux code is that = the sender should use the ACKs to learn about the capacity of the path (bot= h in volumetric and rate dimensions), and should not ignore that informatio= n. This allows the sender to quickly grow and achieve high throughput,=C2= =A0even in the presence of stretch ACKs, which are pervasive, due to TSO/GS= O, GRO/LRO, etc.

Considering bursts is important, = but that can be tackled as an orthogonal issue. Bursts are avoided in the L= inux TCP ecosystem by the combination of TSO autosizing, pacing, TSQ, and f= air queueing.

As Neal describe= d, the congestion controller should use the information in stretch ACKs to = increase its congestion window so that it correctly adjusts the cwnd= based on available link capacity.



Vidhi
=



+Mark= Allman

Just to clear it up, I *was* at BBN long ago wh= en the ABC document
was written.=C2=A0 It's a cool place to work.=C2= =A0 I recommend it.=C2=A0 But, I
now hang out at ICSI.

I believe that ABC was written to solve the problem with AC= K
counting by counting the number of bytes acknowledged for
misbehavi= ng receivers. Limiting the increase to 2*MSS was a good
solution to avoi= d bursts at the time.

The main motivation behind ABC wa= s to counteract delayed ACKs.=C2=A0 The
common approach at the time was = to just bump cwnd by one MSS every
time an ACK rolled in.=C2=A0 So, if a= n ACK covered two segments because
the receiver was delaying the ACKs th= en the growth rate during slow
start was 1.5x per RTT instead of the 2x = that was really
envisioned.=C2=A0 During congestion avoidance the growth= was 1 MSS every
2 RTTs instead of the envisioned every RTT.

A se= condary motivation was to counteract these ACK division attacks
that Sav= age taught us about.=C2=A0 I.e., we could ACK an MSS-sized packet
one by= te at a time and the sender would then increase the cwnd by
MSS*MSS byte= s in the prevalent ACK counting scheme (i.e., cwnd would
get bumped by M= SS bytes for every ACK).

The limit has two roots ...

(1) The = limit is important in slow starts that follow an RTO.=C2=A0 As
=C2=A0= =C2=A0=C2=A0the RFC discusses, in this case we might retransmit a single =C2=A0=C2=A0=C2=A0packet and this will cause the receiver's window to= slide a
=C2=A0=C2=A0=C2=A0great deal.=C2=A0 Therefore, an ACK may indi= cate that a ton of data
=C2=A0=C2=A0=C2=A0has left the network, but tha= t isn't really the case.=C2=A0 So, we
=C2=A0=C2=A0=C2=A0don't w= ant to increase the cwnd based on all the new bytes
=C2=A0=C2=A0=C2=A0A= CKed.

=C2=A0=C2=A0=C2=A0I have since mostly decided that this use o= f L is crude.
=C2=A0=C2=A0=C2=A0Probably there is a more elegant way to= do it by using the
=C2=A0=C2=A0=C2=A0scoreboard and the SACK informati= on to get a better
=C2=A0=C2=A0=C2=A0understanding of what left the net= work and when.=C2=A0 That said, in
=C2=A0=C2=A0=C2=A0this case L is sim= ple and probably about right most of the
=C2=A0=C2=A0=C2=A0time.
(2) I think there was some general conservativeness to bursts and
=C2= =A0=C2=A0=C2=A0using L everywhere quelled some of the worry.=C2=A0 Here L= =3D2 was used
=C2=A0=C2=A0=C2=A0to exactly offset delayed ACKs.

=
I agree that increasing the congestion window and= controlling the
burst rate are orthogonal issues.

Y= es.=C2=A0 In fact, we did subsequent research on mitigating bursts
becau= se we never really thought of ABC as somehow the way to control
bursts (= research papers available, but never went into RFCs).

And, in a worl= d that leverages stretch ACKs as a routine I think
Linux's approach = of not using an L may well be correct.=C2=A0 Documenting
that and the re= asoning behind it in modern networks seems useful to
me.

allman

______________________= _________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--000000000000ff1d4605c8385de2-- From nobody Thu Jul 29 01:10:04 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D27C93A1817; Thu, 29 Jul 2021 01:10:02 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iK21wLRlnWDX; Thu, 29 Jul 2021 01:09:57 -0700 (PDT) Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 848D03A1816; Thu, 29 Jul 2021 01:09:57 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 2A8B125A14; Thu, 29 Jul 2021 10:09:55 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1627546195; bh=HTkpz0Sa3yA+MiP3DRwUvHywlNr8fY/YF8seyIrKSeg=; h=From:To:CC:Subject:Date:From; b=QSaY78A90fWIjTtmyvnn2PqvPUpjFzBbrFWO6XJuyUUYvOyM5aAXGC9JsAq6DrHj5 1EDmffMu4UjbJWGjOPNxMltP4Fb8wI/egQkIF5qcFi4UlSRXpfrLtwKjzzdRvoSzLq JtwMdN8n/gi1KUEDXAfwWNaM0fqc6ffbptgoYkdQ= X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4td_r9dsqkoQ; Thu, 29 Jul 2021 10:09:54 +0200 (CEST) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Thu, 29 Jul 2021 10:09:54 +0200 (CEST) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.14; Thu, 29 Jul 2021 10:09:53 +0200 Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.2176.014; Thu, 29 Jul 2021 10:09:53 +0200 From: "Scharf, Michael" To: "tcpm@ietf.org" CC: tcpm-chairs Thread-Topic: IETF 111 draft minutes Thread-Index: AdeEUNq5I5vsPVfDSueWTRzr5ZY+Pw== Date: Thu, 29 Jul 2021 08:09:53 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [134.108.140.248] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Archived-At: Subject: [tcpm] IETF 111 draft minutes X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 08:10:03 -0000 Hi all, Draft minutes for the TCPM meeting can be found at: https://datatracker.iet= f.org/meeting/111/materials/minutes-111-tcpm-01 Please contact the chairs if something needs to be corrected therein. Thanks to Gorry as our very reliable notetaker! Michael From nobody Thu Jul 29 05:41:46 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EEBB63A2180 for ; Thu, 29 Jul 2021 05:41:43 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.897 X-Spam-Level: X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cORPm7zDv8v2 for ; Thu, 29 Jul 2021 05:41:39 -0700 (PDT) Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E4A023A217B for ; Thu, 29 Jul 2021 05:41:38 -0700 (PDT) Received: by mail-oo1-f42.google.com with SMTP id o17-20020a4a64110000b0290263e1ba7ff9so1541243ooc.2 for ; Thu, 29 Jul 2021 05:41:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=/AUx2N6AlQufL1Oh57kHPnhRN6V/oLoJorq43SXm+Ds=; b=tVDQLuaJhhWRRIRBQgmRCIbZ+GW/TeFaIFh9HsW7BFKtg2uRumne5VnwOEDIshdNjX fc6xfyvm8cP/8URg3yZrFvne4xT9+HTB/5jFlisiGlthOacP6L9V7cRiZ/w3FUGxz5u/ d+agicQnBI5cP6/70BGwFBIMUMIZGWUKjOFiIwfanMRBwQ+/PuLlWTs7AWtea6HjWOm/ ZeZ52uIeuqBVZRkZxq42YVNKPzT4QxTdkgEykfP9yec+T8Bjq2td4EdskuVVprvAWQqc nAbQqr/FNuyZIfxSfp0D90KFjtbL2e/vKYZpKaRhTd4YcEw3qLndsS1IC6lDWANQLE3X YyVw== X-Gm-Message-State: AOAM531ygcaITElTNFnzDNn1C2z2Mr6iS8XwUWtOT2HQj6GOtiSeIYwp OztmViQ4xw92PevEZczYLyL4sQ== X-Google-Smtp-Source: ABdhPJyd4Vzd9dOjyrT9GbTwIZFxfHzKfZ4/3rGaaJL0iEMRDgVol/txW+obZc99GchH0j8Q7VdR1Q== X-Received: by 2002:a4a:da0f:: with SMTP id e15mr2836020oou.53.1627562497935; Thu, 29 Jul 2021 05:41:37 -0700 (PDT) Received: from [192.168.1.208] (162-203-32-211.lightspeed.bcvloh.sbcglobal.net. [162.203.32.211]) by smtp.gmail.com with ESMTPSA id i18sm569274oik.3.2021.07.29.05.41.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jul 2021 05:41:37 -0700 (PDT) From: "Mark Allman" To: "Vidhi Goel" Cc: Extensions , "Neal Cardwell" Date: Thu, 29 Jul 2021 08:41:24 -0400 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_B688FF40-D4CD-45C9-8BB7-62DA46BFCCF5_="; micalg=pgp-sha1; protocol="application/pgp-signature" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 12:41:44 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_B688FF40-D4CD-45C9-8BB7-62DA46BFCCF5_= Content-Type: text/plain I just inhaled notes from Vidhi, Yuchung and Neal. And, I am trying to page the recesses of my brain back in ... :-) I have a few thoughts ... - I sort of agree with Yuchung & Neal in spirit, if not particulars. - During the initial slow start phase I think it'd be fine to say either: (a) If you use some sort of burst mitigator/pacing, an implementation can increase cwnd by the number of bytes ACKed on each ACK (i.e., no L). (b) If there is no burst mitigation then we have to figure out if L is still useful for this purpose and whether we want to retain it. Seems like perhaps L=2 is sensible here. L was never meant to be some general burst mitigator. However, ABC clearly *can* aggravate bursting and so perhaps it makes sense to have it also try to limit the impact of the aggravation (in the absence of some general mechanism). - During slow starts that follow RTOs there is a general problem that just because the window slides by X bytes doesn't say anything about the *network*, as that sliding can happen because much of the data was likely queued for the application on the receiver. So, e.g., you can RTO and send one packet and get an ACK back that slides the window 10 packets. That doesn't mean 10 packets left. It means one packet left the network and nine packets are eligible to be sent to the application. So, it is not OK to set the cwnd to 1+10 = 11 packets in response to this ACK. Here L should exist and be 1. - It is possible that someone could work out a SACK-based scheme to try to figure out how much of the ACK is data leaving the network and how much of it was data sitting in the buffer at the receiver. And, if we could do that then we could say "L=1 unless you use a fancy algorithm". My guess, however, is this is likely a fool's errand for two reasons. First, on an RTO the scoreboard is supposed to be cleared and while SACK information will slowly get built back up we lose a bunch of state that we could perhaps use to differentiate between "left network" and "left buffer". Second, receivers are supposed to ACK out of order segments and those that fill in holes immediately. If they do that, it means that every ACK received actually does convey one segment leaving the network and therefore L=1 is correct and even an accurate SACK-based algorithm wouldn't let one increase cwnd more than L=1 would. (One caveat here is I am not sure how this thinking would play with the offload stuff.) allman --=_MailMate_B688FF40-D4CD-45C9-8BB7-62DA46BFCCF5_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iG8EARECAC8WIQR2wAGDYhexl/6SqCRbKutazjIizgUCYQKh9BEcbWFsbG1hbkBp Y2lyLm9yZwAKCRBbKutazjIizi+oAJsFkEKFOpC4Zwj0JvRLzyVAsK9SEwCfbviV V2TFDqugqMuNsmFpj8zUPeo= =w690 -----END PGP SIGNATURE----- --=_MailMate_B688FF40-D4CD-45C9-8BB7-62DA46BFCCF5_=-- From nobody Thu Jul 29 06:39:06 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A2F7A3A2317 for ; Thu, 29 Jul 2021 06:39:04 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I-5SxzKiex90 for ; Thu, 29 Jul 2021 06:38:59 -0700 (PDT) Received: from mail-vk1-xa36.google.com (mail-vk1-xa36.google.com [IPv6:2607:f8b0:4864:20::a36]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9BD2F3A2334 for ; Thu, 29 Jul 2021 06:38:59 -0700 (PDT) Received: by mail-vk1-xa36.google.com with SMTP id i13so1288201vkd.12 for ; Thu, 29 Jul 2021 06:38:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=n5FtIhZupmS9HOoQZdKr7zJaufPs6zXiUtJz8JHSBHU=; b=ZQl/l3f8K/Ec8FzmMLJbASSXVAInH3sHzveywcKaqXojEI89WS0cPZK10ZvnvL3WEw g0s0cd9UEqm1MrN9sJ67dH/YD1Xzo5fK7Ns+E4cjsiRUPOE/olehe0jeN4rsHGiOBwFt JXU4Vk6hk76cz12D0XevZdXP4zvA8CdisJiFSCGq6OtQH5xrlNpC+FvngmtW9p6ezV8b 2rFaMM7crSmKEyT+M+8DJOcL9dnkzAWxnMgopW0Du9xiG4Bs1za7N9tVwPoFqWSh5UDt IdHplruo2cOUEcM/n52+BhrAkgeyVakuofeMWvVCbWeEajHfn+Q0Apjins7wtbqiH6b6 ziVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=n5FtIhZupmS9HOoQZdKr7zJaufPs6zXiUtJz8JHSBHU=; b=qrIcmDod9MEltSOarnKQxRsZI/hTOJkG7juZf732+FcqEjPOIzZIG4zlhzZlomrPZe B+2PzBlj0fIO8sSvsZ6d4zte14YPLd5QaIgC85fP0xlDa7IrkrPIfQxSTWtCZ+tz/UUj xe2SMrK9vBLSo58LX25gJo140CJWFlSvF9fPqfCs3A5LY6qqMjegeZmsKznETegQ3VjW nUu6asgZRrYRxczCXwQgTLknVKNySFQt1q5nJIDev2pW8DEaJvn7WMGIZ+hIK4/x7hgZ Ao5ZVvmPmn3mpxvKuXG5pE+Oa5bnLxQVyNowVR4198lYpokiYtsdAgik81y5f1Aw768F SSVw== X-Gm-Message-State: AOAM532T6/KheYWGyoNjhYrLYGqs4/zacE/Wlo12gtmDo0SdG4XaFBhx 2/0VABWqqrNyGTcPJPpS/wOXraV+fTZmHaw4sUUzQQ== X-Google-Smtp-Source: ABdhPJzsLiZmB12t5aQbLOSJ2ZqVy/0bXGYvytJ4Ep0uqHvPNlukOvYBdt71bpZDiqbF7em33ZdY7TldjXao9CSjx4Q= X-Received: by 2002:a1f:4843:: with SMTP id v64mr3956242vka.11.1627565936328; Thu, 29 Jul 2021 06:38:56 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> In-Reply-To: From: Neal Cardwell Date: Thu, 29 Jul 2021 09:38:39 -0400 Message-ID: To: Mark Allman Cc: Vidhi Goel , Extensions Content-Type: multipart/alternative; boundary="0000000000007c530e05c843393a" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 13:39:05 -0000 --0000000000007c530e05c843393a Content-Type: text/plain; charset="UTF-8" On Thu, Jul 29, 2021 at 8:41 AM Mark Allman wrote: > > I just inhaled notes from Vidhi, Yuchung and Neal. And, I am trying > to page the recesses of my brain back in ... :-) > > I have a few thoughts ... > > - I sort of agree with Yuchung & Neal in spirit, if not > particulars. > > - During the initial slow start phase I think it'd be fine to say > either: > > (a) If you use some sort of burst mitigator/pacing, an > implementation can increase cwnd by the number of bytes > ACKed on each ACK (i.e., no L). > > (b) If there is no burst mitigation then we have to figure out > if L is still useful for this purpose and whether we want to > retain it. Seems like perhaps L=2 is sensible here. L was > never meant to be some general burst mitigator. However, > ABC clearly *can* aggravate bursting and so perhaps it makes > sense to have it also try to limit the impact of the > aggravation (in the absence of some general mechanism). > Even if recommending a static L value, IMHO L=2 is a bit conservative. Given that RFC 6928 and the experience with IW10 seems to show that the vast majority of paths on the public Internet work fine with bursts of 10 packets, this seems to suggest that it is safe to allow an ACK for 5 packets in slow start to increase cwnd by 5 packets and allow the release of 10 packets, which I guess implies an L of at least 5. > - During slow starts that follow RTOs there is a general problem > that just because the window slides by X bytes doesn't say > anything about the *network*, as that sliding can happen because > much of the data was likely queued for the application on the > receiver. So, e.g., you can RTO and send one packet and get an > ACK back that slides the window 10 packets. That doesn't mean > 10 packets left. It means one packet left the network and nine > packets are eligible to be sent to the application. So, it is > not OK to set the cwnd to 1+10 = 11 packets in response to this > ACK. Here L should exist and be 1. > AFAICT this argument only applies to non-SACK connections. For connections with SACK (the vast majority of connections over the public Internet and in datacenters), it is quite feasible to determine how many packets really left the network (and Linux TCP does this; see below). > - It is possible that someone could work out a SACK-based scheme > to try to figure out how much of the ACK is data leaving the > network and how much of it was data sitting in the buffer at the > receiver. And, if we could do that then we could say "L=1 > unless you use a fancy algorithm". The SACK-based scheme to try to figure out how much of the ACK is data leaving the network has been implemented in Linux for at least 20 years or so ( see the 2002 paper "Congestion Control in Linux TCP" by Pasi Sarolahti and Alexey Kuznetsov: https://www.cs.helsinki.fi/research/iwtcp/papers/linuxtcp.pdf ) > My guess, however, is this > is likely a fool's errand for two reasons. First, on an RTO the > scoreboard is supposed to be cleared and while SACK information > will slowly get built back up we lose a bunch of state that we > could perhaps use to differentiate between "left network" and > "left buffer". Clearing the scoreboard on RTO is indeed a "SHOULD" from RFC 2018: After a retransmit timeout the data sender SHOULD turn off all of the SACKed bits, since the timeout might indicate that the data receiver has reneged. The data sender MUST retransmit the segment at the left edge of the window after a retransmit timeout, whether or not the SACKed bit is on for that segment. But our experience with Linux TCP is that clearing the scoreboard on RTO is unnecessary, and causes unnecessary performance problems and redundant retransmissions. While it is true that "the timeout might indicate that the data receiver has reneged", it is vastly more common for timeouts to be due to simple packet loss. And the sender can specifically detect reneging if it finds it times out and hits the specific scenario implied in the RFC (the segment at the left edge of the window is SACKed); and if this specific indication of reneging happens, *then* it makes sense for the sender to clear the SACK scoreboard. This is the approach Linux TCP has used for a decade or more, and it works very well in practice. > Second, receivers are supposed to ACK out of > order segments and those that fill in holes immediately. If they do that, it means that every ACK received actually does > convey one segment leaving the network and therefore L=1 is correct and even an accurate SACK-based algorithm wouldn't let > one increase cwnd more than L=1 would. (One caveat here is I am not sure how this thinking would play with the offload stuff.) > Yes, offload mechanisms are so pervasive in practice, that this kind of reasoning no longer holds in common cases. In common cases, there are hardware (LRO) or software (GRO) mechanisms that are doing aggregation of consecutive byte ranges at a layer that is too low to know whether the byte ranges are out of order or fill a hole, so by the time that out-of-order or hole-filling ranges reach TCP they may be up to 64KBytes in length. So although TCP will try to ACK an out-of-order or hole-filling sequence range "immediately" from the TCP layer's perspective, it may often be up to 64Kbytes or roughly 44 segments in length. best regards, neal --0000000000007c530e05c843393a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 8:41 AM Mark = Allman <mallman@icir.org> wro= te:

I just inhaled notes from Vidhi, Yuchung and Neal.=C2=A0 And, I am trying to page the recesses of my brain back in ... :-)

I have a few thoughts ...

=C2=A0 - I sort of agree with Yuchung & Neal in spirit, if not
=C2=A0 =C2=A0 particulars.

=C2=A0 - During the initial slow start phase I think it'd be fine to sa= y
=C2=A0 =C2=A0 either:

=C2=A0 =C2=A0 (a) If you use some sort of burst mitigator/pacing, an
=C2=A0 =C2=A0 =C2=A0 =C2=A0 implementation can increase cwnd by the number = of bytes
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ACKed on each ACK (i.e., no L).

=C2=A0 =C2=A0 (b) If there is no burst mitigation then we have to figure ou= t
=C2=A0 =C2=A0 =C2=A0 =C2=A0 if L is still useful for this purpose and wheth= er we want to
=C2=A0 =C2=A0 =C2=A0 =C2=A0 retain it.=C2=A0 Seems like perhaps L=3D2 is se= nsible here.=C2=A0 L was
=C2=A0 =C2=A0 =C2=A0 =C2=A0 never meant to be some general burst mitigator.= =C2=A0 However,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ABC clearly *can* aggravate bursting and so per= haps it makes
=C2=A0 =C2=A0 =C2=A0 =C2=A0 sense to have it also try to limit the impact o= f the
=C2=A0 =C2=A0 =C2=A0 =C2=A0 aggravation (in the absence of some general mec= hanism).

Even if recommending a static = L value, IMHO L=3D2 is a bit conservative. Given that RFC 6928 and the expe= rience with IW10 seems to show that the vast majority of paths on the publi= c Internet work fine with bursts of 10 packets, this seems to suggest that = it is safe to allow an ACK for 5 packets in slow start to increase cwnd by = 5 packets and allow the release of 10 packets, which I guess implies an L o= f at least 5.
=C2=A0
=C2=A0 - During slow starts that follow RTOs there is a general problem
=C2=A0 =C2=A0 that just because the window slides by X bytes doesn't sa= y
=C2=A0 =C2=A0 anything about the *network*, as that sliding can happen beca= use
=C2=A0 =C2=A0 much of the data was likely queued for the application on the=
=C2=A0 =C2=A0 receiver.=C2=A0 So, e.g., you can RTO and send one packet and= get an
=C2=A0 =C2=A0 ACK back that slides the window 10 packets.=C2=A0 That doesn&= #39;t mean
=C2=A0 =C2=A0 10 packets left.=C2=A0 It means one packet left the network a= nd nine
=C2=A0 =C2=A0 packets are eligible to be sent to the application.=C2=A0 So,= it is
=C2=A0 =C2=A0 not OK to set the cwnd to 1+10 =3D 11 packets in response to = this
=C2=A0 =C2=A0 ACK.=C2=A0 Here L should exist and be 1.

AFAICT this argument only applies to non-SACK connections. = For connections with SACK (the vast majority of connections over the public= Internet and in datacenters), it is quite feasible to determine how many p= ackets really left the network (and Linux TCP does this; see below).
<= div>=C2=A0
=C2=A0 - It is possible that someone could work out a SACK-based scheme
=C2=A0 =C2=A0 to try to figure out how much of the ACK is data leaving the<= br> =C2=A0 =C2=A0 network and how much of it was data sitting in the buffer at = the
=C2=A0 =C2=A0 receiver.=C2=A0 And, if we could do that then we could say &q= uot;L=3D1
=C2=A0 =C2=A0 unless you use a fancy algorithm".=C2=A0

The SACK-based scheme to try to= figure out how much of the ACK is data leaving the network has been implem= ented in Linux for at least 20 years or so ( see the 2002 paper "Conge= stion Control in Linux TCP" by=C2=A0Pasi Sarolahti and=C2=A0Alexey Kuz= netsov:=C2=A0 https://www.cs.helsinki.fi/research/iwtcp/papers/linuxtcp.pdf<= /a> )
My guess, however, is this =C2=A0 =C2=A0 is likely a fool's errand for two reasons.=C2=A0 First, o= n an RTO the
=C2=A0 =C2=A0 scoreboard is supposed to be cleared and while SACK informati= on
=C2=A0 =C2=A0 will slowly get built back up we lose a bunch of state that w= e
=C2=A0 =C2=A0 could perhaps use to differentiate between "left network= " and
=C2=A0 =C2=A0 "left buffer".

Clea= ring the scoreboard on RTO is indeed a "SHOULD" from RFC 2018:

=
   After a retransmit timeout the data sender SHOULD turn off all of the
   SACKed bits, since the timeout might indicate that the data receiver
   has reneged.  The data sender MUST retransmit the segment at the left
   edge of the window after a retransmit timeout, whether or not the
   SACKed bit is on for that segment.

But our experience w= ith Linux TCP is that clearing the scoreboard on RTO is unnecessary, and ca= uses unnecessary performance problems and redundant retransmissions.=C2=A0 = While it is true that "the timeout might indicate that the data receiver=C2=A0has reneged", it is vast= ly more common for timeouts to be due to simple packet loss. And the= sender can specifically detect reneging if it finds it times out and hits = the specific scenario implied in the RFC (the segment at the left edge of t= he window is SACKed); and if this specific indication of reneging happens, = *then* it makes sense for the sender to clear the SACK scoreboard. This is = the approach Linux TCP has used for a decade or more, and it works very wel= l in practice.
=C2=A0
=C2=A0 Second, receivers are supposed to ACK out of
=C2=A0 =C2=A0 order segments and those that fill in holes immediately.=C2= =A0 If
=C2=A0 =C2=A0 they do that, it means that every ACK received actually does<= br> =C2=A0 =C2=A0 convey one segment leaving the network and therefore L=3D1 is=
=C2=A0 =C2=A0 correct and even an accurate SACK-based algorithm wouldn'= t let
=C2=A0 =C2=A0 one increase cwnd more than L=3D1 would.=C2=A0 (One caveat he= re is I am
=C2=A0 =C2=A0 not sure how this thinking would play with the offload stuff.= )

Yes, offload mechanisms are so p= ervasive in practice, that this kind of reasoning no longer holds in common= cases. In common cases, there are hardware (LRO) or software (GRO) mechani= sms that are doing aggregation of consecutive byte ranges at a layer that i= s too low to know whether the byte ranges are out of order or fill a hole, = so by the time that out-of-order or hole-filling ranges reach TCP they may = be up to 64KBytes in length. So although TCP will try to ACK an out-of-orde= r or hole-filling sequence range "immediately" from the TCP layer= 's perspective, it may=C2=A0often be up to 64Kbytes or roughly 44 segme= nts in length.

best regards,
neal

--0000000000007c530e05c843393a-- From nobody Thu Jul 29 07:06:48 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8ADBD3A241B for ; Thu, 29 Jul 2021 07:06:38 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.896 X-Spam-Level: X-Spam-Status: No, score=-1.896 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NqB7poDqRebc for ; Thu, 29 Jul 2021 07:06:35 -0700 (PDT) Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id ABFC93A2400 for ; Thu, 29 Jul 2021 07:06:35 -0700 (PDT) Received: by mail-ot1-f50.google.com with SMTP id h63-20020a9d14450000b02904ce97efee36so5961020oth.7 for ; Thu, 29 Jul 2021 07:06:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=zIl0FkvPN3/e3SZuLy6t4+c4zFNeVkQcKpvyyWlQJ94=; b=sn5bJN9znOVC8igweZAY0crTZ3AL2rhf200P1T88r7J5sWFWCXrFkuNQT788EfprSD /RA8T3sFB2zAwEcL4IjBZ3UnGfr2TOUBqh+USDpsxt7L51AcBbenYCkXy5936NZ0znbL 4Ryrfh+X6Bqb5oXDs9QzX8BnwF4DbghDq1OyaC5NvcWYlgPH9+J2yaZEXoe1fqueADCT U10GSZ6JGUVSyCucRV4CZspj4XKaFWe2eIN1TAzx/euBAuPGwlb3aRTu6lKuvrR1xuci d2NZ285AWRw6tfWoUKp3FYFXtz7Fi86WGm0X3hQx9GapTShAfEjz70xuWEuqJR/NCMJR FZpw== X-Gm-Message-State: AOAM531nB3PoGY8cobvjmq0MtY3CWqqHUJQeSKVySoGm078GJ5cMX9lb d8XZK5VBOzp/Jh7Tz/o4hWA6/A== X-Google-Smtp-Source: ABdhPJzPuyHUikcKXkDj7f+cPDr2uqG1oJPvsFNFgO6sFIckIoY0Klv0mOcN6SNtYqsoDmomj2xkmA== X-Received: by 2002:a05:6830:140d:: with SMTP id v13mr3557377otp.296.1627567594785; Thu, 29 Jul 2021 07:06:34 -0700 (PDT) Received: from [192.168.1.208] (162-203-32-211.lightspeed.bcvloh.sbcglobal.net. [162.203.32.211]) by smtp.gmail.com with ESMTPSA id f16sm605032oiw.29.2021.07.29.07.06.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 29 Jul 2021 07:06:34 -0700 (PDT) From: "Mark Allman" To: "Neal Cardwell" Cc: "Vidhi Goel" , Extensions Date: Thu, 29 Jul 2021 10:06:31 -0400 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_283CFC4C-4731-4DD2-9C19-E6B5A945CE1B_="; micalg=pgp-sha1; protocol="application/pgp-signature" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 14:06:46 -0000 This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_283CFC4C-4731-4DD2-9C19-E6B5A945CE1B_= Content-Type: text/plain >> (b) If there is no burst mitigation then we have to figure out >> if L is still useful for this purpose and whether we want to >> retain it. Seems like perhaps L=2 is sensible here. L was >> never meant to be some general burst mitigator. However, >> ABC clearly *can* aggravate bursting and so perhaps it makes >> sense to have it also try to limit the impact of the >> aggravation (in the absence of some general mechanism). > > Even if recommending a static L value, IMHO L=2 is a bit > conservative. Well, perhaps. L=2 was designed to exactly counteract delayed ACKs. So, it isn't exactly a new magic number. We could wave our hands and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure we could come up with something that folks felt was fine. However, my feeling is that if we want to worry about bursts then let's worry about bursts in some generic way. And, if you have some way to deal with bursts then L isn't needed. And, if you don't have a way to deal with bursts then a conservative L seems fine. But, perhaps putting the effort into a generic mechanism instead of cooking yet another magic number we need to periodically refresh is probably a better way to spend effort. >> - During slow starts that follow RTOs there is a general >> problem that just because the window slides by X bytes >> doesn't say anything about the *network*, as that sliding can >> happen because much of the data was likely queued for the >> application on the receiver. So, e.g., you can RTO and send >> one packet and get an ACK back that slides the window 10 >> packets. That doesn't mean 10 packets left. It means one >> packet left the network and nine packets are eligible to be >> sent to the application. So, it is not OK to set the cwnd to >> 1+10 = 11 packets in response to this ACK. Here L should >> exist and be 1. > > AFAICT this argument only applies to non-SACK connections. For > connections with SACK (the vast majority of connections over the > public Internet and in datacenters), it is quite feasible to > determine how many packets really left the network (and Linux TCP > does this; see below). If you have an accurate way to figure out how many of the ACKed bytes left the network and how many were just buffered at the receiver then I see no problem with increasing based on byte count as you do in the initial slow start. (I don't remember what the paper you cite says, but my guess is it's often the case that L=1 is a reasonable substitute for something complicated here. But, perhaps I am running the simulation in my head wrong ... it has been a while, admittedly!) > Yes, offload mechanisms are so pervasive in practice, I am trying to build a mental model here. How pervasive would you guess these are? And, where in the network? I have assumed that they are for sure pervasive in data centers and server farms, but not for the vast majority of Internet-connected devices. allman --=_MailMate_283CFC4C-4731-4DD2-9C19-E6B5A945CE1B_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iG8EARECAC8WIQR2wAGDYhexl/6SqCRbKutazjIizgUCYQK15xEcbWFsbG1hbkBp Y2lyLm9yZwAKCRBbKutazjIizjkwAJ44B6ANEkYM5I4q4SzRy62FH2ErOgCdGmdx /lz/QkC/LacsAwm9DPTAitU= =DjSE -----END PGP SIGNATURE----- --=_MailMate_283CFC4C-4731-4DD2-9C19-E6B5A945CE1B_=-- From nobody Thu Jul 29 08:19:41 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A82C23A25C2 for ; Thu, 29 Jul 2021 08:19:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.097 X-Spam-Level: X-Spam-Status: No, score=-18.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J7ilBQmPsuZL for ; Thu, 29 Jul 2021 08:19:35 -0700 (PDT) Received: from mail-ua1-x933.google.com (mail-ua1-x933.google.com [IPv6:2607:f8b0:4864:20::933]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D15F53A25EB for ; Thu, 29 Jul 2021 08:19:34 -0700 (PDT) Received: by mail-ua1-x933.google.com with SMTP id v3so2724683uau.3 for ; Thu, 29 Jul 2021 08:19:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uIXoovpFH5cpbVPTiEaS8+9BS0tKC3WyrV5nZyho+to=; b=C95B1MiPPE00z34L/7VcaUG4hHJ/YM4oRasMcLEjo4XdWli4JKCoHz0tUWnr1f07tC u+TJejetpI7K/6JtGFFCOMnGYA5pEPwTsAzSFB/YNCUWhsthBukkjCyIFXD7bKVWdrPh Z0uatkJGaRsRz30S/pRW5hNjptiJcazyF5GDOqifXAYwbVyql+uW1SBSWAiNvO2ZDMd2 iEhJyLB6vvmEuXQGcppppGW21Hh4CM/4UfoYRjiQ2RnuwBlh1RNVl871y0WALYHng1rS ETVAEB9M/clfVg4EXHxMq9OBeFNCp3h7KHBEOhq9TWdwZHnuNLrfBs1bZIgXd4pTng6c siQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uIXoovpFH5cpbVPTiEaS8+9BS0tKC3WyrV5nZyho+to=; b=lK46ijBkrZx31spTe5ZR0mxY8GncIMHe3VWwam+S5EL1yc3JZYbQdO6V0wDAzstApi 6MZtHmFwxzC2XI2sEa4vjy97ZbwnFv2sg1qM9Yoo1+C1cz/WDHVLSZZlNas2fwQYtqvT NzdEKCXOQeWt7l51GA10G1I8x4P9hK+SaN/8rwEjRsiB9OoDuQoiU/ZxB01IeYV2h9zX +aGlEhyoIz68Ic7hbK6VTJINbgX09zu6eTeH29QXagZUN1YCr16z4hTmplhcdRusvumy FbhpiqY2m/NBLntA7pzeAbBo0oA8B59zVArCw5VB9ffs/0pGP+/s2nxyNRjL3YRkmVOD 3tEw== X-Gm-Message-State: AOAM533D1qjDVSj7um3miYDmEfaOdiI4SCw4TPL8/i5EIxJCC8JscaUL RbteDq+GhPudGLx22UUGRmuS/Y/vujRvftfY8tMHUI2hKLyaNg== X-Google-Smtp-Source: ABdhPJxFzw6yFlPYQ/etGOPglSaNQlVkD9XLUndQ2nwBSUDjWtSGyrKlN3zqxjMnMd+IEFVulh7cruzgS92uU37YFk0= X-Received: by 2002:ab0:42d:: with SMTP id 42mr5027275uav.63.1627571972718; Thu, 29 Jul 2021 08:19:32 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> In-Reply-To: From: Neal Cardwell Date: Thu, 29 Jul 2021 11:19:16 -0400 Message-ID: To: Mark Allman Cc: Vidhi Goel , Extensions Content-Type: multipart/alternative; boundary="000000000000481a3605c844a15c" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 15:19:40 -0000 --000000000000481a3605c844a15c Content-Type: text/plain; charset="UTF-8" On Thu, Jul 29, 2021 at 10:06 AM Mark Allman wrote: > > >> (b) If there is no burst mitigation then we have to figure out > >> if L is still useful for this purpose and whether we want to > >> retain it. Seems like perhaps L=2 is sensible here. L was > >> never meant to be some general burst mitigator. However, > >> ABC clearly *can* aggravate bursting and so perhaps it makes > >> sense to have it also try to limit the impact of the > >> aggravation (in the absence of some general mechanism). > > > > Even if recommending a static L value, IMHO L=2 is a bit > > conservative. > > Well, perhaps. L=2 was designed to exactly counteract delayed ACKs. > So, it isn't exactly a new magic number. We could wave our hands > and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure > we could come up with something that folks felt was fine. However, > my feeling is that if we want to worry about bursts then let's worry > about bursts in some generic way. And, if you have some way to deal > with bursts then L isn't needed. And, if you don't have a way to > deal with bursts then a conservative L seems fine. But, perhaps > putting the effort into a generic mechanism instead of cooking yet > another magic number we need to periodically refresh is probably a > better way to spend effort. > Yes, I very much agree that "putting the effort into a generic mechanism instead of cooking yet another magic number we need to periodically refresh is probably a better way to spend effort." > > >> - During slow starts that follow RTOs there is a general > >> problem that just because the window slides by X bytes > >> doesn't say anything about the *network*, as that sliding can > >> happen because much of the data was likely queued for the > >> application on the receiver. So, e.g., you can RTO and send > >> one packet and get an ACK back that slides the window 10 > >> packets. That doesn't mean 10 packets left. It means one > >> packet left the network and nine packets are eligible to be > >> sent to the application. So, it is not OK to set the cwnd to > >> 1+10 = 11 packets in response to this ACK. Here L should > >> exist and be 1. > > > > AFAICT this argument only applies to non-SACK connections. For > > connections with SACK (the vast majority of connections over the > > public Internet and in datacenters), it is quite feasible to > > determine how many packets really left the network (and Linux TCP > > does this; see below). > > If you have an accurate way to figure out how many of the ACKed > bytes left the network and how many were just buffered at the > receiver then I see no problem with increasing based on byte count > as you do in the initial slow start. > > (I don't remember what the paper you cite says, but my guess is it's > often the case that L=1 is a reasonable substitute for something > complicated here. But, perhaps I am running the simulation in my > head wrong ... it has been a while, admittedly!) > > > Yes, offload mechanisms are so pervasive in practice, > > I am trying to build a mental model here. How pervasive would you > guess these are? And, where in the network? I have assumed that > they are for sure pervasive in data centers and server farms, but > not for the vast majority of Internet-connected devices. > >From my impression looking at public Internet traces, aggregation mechanisms that cause TCP ACKs for more than 2 segments are very common. I suspect that's because the majority of public Internet traffic these days has a bottleneck that is either wifi, cellular, or DOCSIS, and all of these have a shared medium with a large latency overhead for L2 MAC control of gets to speak next. So a lot of batching happens, both in big batches of data that arrive at the client in the same L2 medium time slot, and big batches of ACKs that accumulate while the client waits (often several milliseconds, sometimes even tens of milliseconds) for its chance to send a big stretch ACK or batch of ACKs. This brings up a related point: even if there is some ABC-style per-ACK L limit on cwnd increases, the time structure of most public Internet ACK streams is massively bursty because of these aggregation mechanisms inherent in L2 behavior on most public Internet bottlenecks (wifi, cellular, DOCSIS). So even if there is a limit L that limits the per-ACK behavior to be smooth, if there is no pacing of data segments then the data transmit time structure will still be bursty because the ACK arrivals these days are very bursty. best regards, neal --000000000000481a3605c844a15c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 10:06 AM Mark= Allman <mallman@icir.org> wr= ote:

>>=C2=A0 =C2=A0 =C2=A0(b) If there is no burst mitigation then we hav= e to figure out
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if L is still useful for this pur= pose and whether we want to
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0retain it.=C2=A0 Seems like perha= ps L=3D2 is sensible here.=C2=A0 L was
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0never meant to be some general bu= rst mitigator.=C2=A0 However,
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ABC clearly *can* aggravate burst= ing and so perhaps it makes
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sense to have it also try to limi= t the impact of the
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0aggravation (in the absence of so= me general mechanism).
>
> Even if recommending a static L value, IMHO L=3D2 is a bit
> conservative.

Well, perhaps.=C2=A0 L=3D2 was designed to exactly counteract delayed ACKs.=
So, it isn't exactly a new magic number.=C2=A0 We could wave our hands<= br> and say "5 seems OK" or "10 seems OK" or whatever.=C2= =A0 And, I am sure
we could come up with something that folks felt was fine.=C2=A0 However, my feeling is that if we want to worry about bursts then let's worry about bursts in some generic way.=C2=A0 And, if you have some way to deal with bursts then L isn't needed.=C2=A0 And, if you don't have a way= to
deal with bursts then a conservative L seems fine.=C2=A0 But, perhaps
putting the effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is probably a
better way to spend effort.

Yes, I very= much agree that "putting the effort into a generic mechanism instead = of cooking yet another magic number we need to periodically refresh is prob= ably a better way to spend effort."=C2=A0

>>=C2=A0 =C2=A0- During slow starts that follow RTOs there is a gener= al
>>=C2=A0 =C2=A0 =C2=A0problem that just because the window slides by = X bytes
>>=C2=A0 =C2=A0 =C2=A0doesn't say anything about the *network*, a= s that sliding can
>>=C2=A0 =C2=A0 =C2=A0happen because much of the data was likely queu= ed for the
>>=C2=A0 =C2=A0 =C2=A0application on the receiver.=C2=A0 So, e.g., yo= u can RTO and send
>>=C2=A0 =C2=A0 =C2=A0one packet and get an ACK back that slides the = window 10
>>=C2=A0 =C2=A0 =C2=A0packets.=C2=A0 That doesn't mean 10 packets= left.=C2=A0 It means one
>>=C2=A0 =C2=A0 =C2=A0packet left the network and nine packets are el= igible to be
>>=C2=A0 =C2=A0 =C2=A0sent to the application.=C2=A0 So, it is not OK= to set the cwnd to
>>=C2=A0 =C2=A0 =C2=A01+10 =3D 11 packets in response to this ACK.=C2= =A0 Here L should
>>=C2=A0 =C2=A0 =C2=A0exist and be 1.
>
> AFAICT this argument only applies to non-SACK connections. For
> connections with SACK (the vast majority of connections over the
> public Internet and in datacenters), it is quite feasible to
> determine how many packets really left the network (and Linux TCP
> does this; see below).

If you have an accurate way to figure out how many of the ACKed
bytes left the network and how many were just buffered at the
receiver then I see no problem with increasing based on byte count
as you do in the initial slow start.

(I don't remember what the paper you cite says, but my guess is it'= s
often the case that L=3D1 is a reasonable substitute for something
complicated here.=C2=A0 But, perhaps I am running the simulation in my
head wrong ... it has been a while, admittedly!)

> Yes, offload mechanisms are so pervasive in practice,

I am trying to build a mental model here.=C2=A0 How pervasive would you
guess these are?=C2=A0 And, where in the network?=C2=A0 I have assumed that=
they are for sure pervasive in data centers and server farms, but
not for the vast majority of Internet-connected devices.
From my impression looking at public Internet traces, aggregation mechani= sms that cause TCP ACKs for more than 2 segments are very common. I suspect= that's because the majority of public Internet traffic these days has = a bottleneck that is either wifi, cellular, or DOCSIS, and all of these hav= e a shared medium with a large latency overhead for L2 MAC control of gets = to speak next. So a lot of batching happens, both in big batches of data th= at arrive at the client in the same L2 medium time slot, and big batches of= ACKs that accumulate while the client waits (often several milliseconds, s= ometimes even tens of milliseconds) for its chance to send a big stretch AC= K or batch of ACKs.

This brings up a related point: even if there is some ABC-sty= le per-ACK L limit on cwnd increases, the time structure of most public Int= ernet ACK streams is massively bursty because of these aggregation mechanis= ms inherent in L2 behavior on most public Internet bottlenecks (wifi, cellu= lar, DOCSIS). So even if there is a limit L that limits the per-ACK behavio= r to be smooth, if there is no pacing of data segments then the data transm= it time structure will still be bursty because the ACK arrivals these days = are very bursty.=C2=A0

best regards,
neal

--000000000000481a3605c844a15c-- From nobody Thu Jul 29 09:28:26 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B6273A0B01; Thu, 29 Jul 2021 09:28:25 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tvWIeqmgYI5m; Thu, 29 Jul 2021 09:28:20 -0700 (PDT) Received: from mail-ssdrsserver2.hostinginterface.eu (mail-ssdrsserver2.hostinginterface.eu [185.185.85.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 151B03A0B1A; Thu, 29 Jul 2021 09:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BPrGtN716r4yvWd+B0bESPrc5fXvSINzbyDMGuypfkY=; b=x+MyFkJvxmHRRw4HHu1g7Zcsho HvXzprvamGb3ZMzBWaSwb/bGaX/XW7ZiS9qadO8DmZbq7uPVtOCUiScag28QrhO/05KOLrR7DsZ9P Ua6qUH97YlMRePYPIbbQyz+dzoMwQlwzl5bajBDPcmnnv0ZQvfezKzyQcEo7r41l7vu15Aha2cZR0 ZRQrbbdpZ9kSSnCkWhh9+tGO9zeTshteTU7l3lP6yaYpmPHNQLmPGModiEbjI0yGvDQsL/lZoJOtL v17j9CF5x8CYJC+LNCFBnshf91GRJGilBbhDPZB3ioZdSihQym0vDKyHVUDI88Jc/cjensEW6+/82 VDDt4jrg==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:52882 helo=[192.168.1.11]) by ssdrsserver2.hostinginterface.eu with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1m98tT-0001cI-4a; Thu, 29 Jul 2021 17:28:16 +0100 To: Yuchung Cheng , huitema@huitema.net Cc: "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" , Neal Cardwell References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> From: Bob Briscoe Message-ID: Date: Thu, 29 Jul 2021 17:28:14 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------66B5964856B4360959048994" Content-Language: en-GB X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - ssdrsserver2.hostinginterface.eu X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: ssdrsserver2.hostinginterface.eu: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: ssdrsserver2.hostinginterface.eu: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 16:28:26 -0000 This is a multi-part message in MIME format. --------------66B5964856B4360959048994 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Yuchung, On 29/07/2021 00:35, Yuchung Cheng wrote: > same experience with Chrisitian > > For more than half a decade Google traffic used Linux CUBIC Hystart. > This includes google.com  and YouTube public > Internet traffic, as well as internal Google TCP traffic. When we > deployed pacing around 2013 we disabled the Hystart ACK-train > mechanism (that Hystart++ prohibits now) as it causes false SS > exit. But we continued to use the stock Hystart Delay mechanism. > ~2014-15 we switched to BBR which uses a different startup more robust > to delay jitters. [BB] You or Neal (sry can't remember which) were one of those I surveyed who said they had disabled Hystart. So perhaps this was a misunderstanding over the ACK-train part. > But IMO the simpler hystart mitigating SS over-shoot is still highly > valuable for Cubic and other congestion controls, as Internet > bandwidth continues to hike. > > Did those providers disable hystart b/c of its poor interaction > between the ack-train mode and pacing? [BB] I didn't ask them why. Bob > Maybe it's time to disable the ack-train approach in the upstream > Linux when hystart++ is standardized. > > On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema > wrote: > > I certainly do not qualify as a "large operator", but when > implementing Cubic for QUIC I found that Hystart was critical for > performance. Specifically, the classical slow-start is very prone > to overshooting the capacity of the path, which causes large > batches of errors. With web-like traffic, these batches of errors > cause increase latency for the first transactions in a session, > and thus a drop in perceived quality. Hystart solves that. With > Hystart, I observed that a large fraction of sessions do not > experience any packet loss at all. > > As an aside, we should strive for this suppression of packet > losses. That's actually a big reason for moving from Cubic to BBR. > Hystart suppresses the losses during the initial phase of the > connection, but Cubic still relies on periodically testing the > limit of path capacity and causing losses during the subsequent > phase. BBR's probing for bottleneck bandwidth is much more > conservative, does not cause such losses. It might be possible to > adapt Cubic to also not cause losses, for example by ending an > epoch early if too many CE marks are received or if the RTT > increases. That would be worth trying. > > -- Christian Huitema > > On 7/28/2021 3:19 AM, Bob Briscoe wrote: >> Yuchung, >> >> It was during a Mar 2018 ad hoc workshop Jana had organized at >> the London IETF entitled 'BBR and the intersection with other >> work". I can't remember why I needed to know at the time, but >> during the break I approached the major server operators >> individually, established whether they used Cubic, and if so >> asked whether they used Hystart or disabled it. It's hard to >> anonymize the results, because IMMSMC all those that used Cubic >> said they disabled Hystart. >> >> If this isn't correct, then maybe people who replied at the time >> thought they were being asked something else. Or maybe it was >> correct then but isn't now. >> >> >> Bob >> >> >> On 28/07/2021 00:46, Yuchung Cheng wrote: >>> Wait -- how is hystart "invariably disabled" in Linux (cubic)!? >>> >>> What data indicates that >>> >>> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe >> >>> > wrote: >>> >>>     Any large server operator out there who are using Cubic, >>>     If you're willing to state whether or not your operations >>> disable >>>     Hystart, pls do so in reply. >>>     Then Praveen can cite this mailing list thread in the Hystart++ >>>     draft, as requested below. >>> >>>     If you are willing to reply privately, I would be willing to >>> keep >>>     your confidences, and provide an anonymized result to the list. >>> >>>     Cheers >>> >>> >>> >>>     Bob >>> >>>     On 27/07/2021 01:50, Praveen Balasubramanian wrote: >>>> >>>>     Although Hystart is default enabled in Linux, it is invariably >>>>     disabled. So, it's misleading to just say Hystart is default >>>>     enabled, which implies it's widely used, when people >>>> clearly find >>>>     it has problems (which motivates Hystart++). I found this out >>>>     through an informal survey I did at the Mar'18 IETF in >>>> London by >>>>     asking round the implementers of the most prevalent stacks (I >>>>     would name names if I could find the note I later sent to >>>> someone >>>>     or to some list, but I can't find it - sry). >>>>>>>> >>>>>>>>     I'd like some citations on this versus anecdata if >>>>>>>> possible >>>>>>>>     before I add that caveat to the text. Do large deployments >>>>>>>>     disable this? I haven't come across this suggestion in any >>>>>>>>     Linux tuning guides to date. >>> >>>     -- >>> ________________________________________________________________ >>>     Bob Briscoehttp://bobbriscoe.net/ >>> >>> >>>     _______________________________________________ >>>     tcpm mailing list >>> tcpm@ietf.org >>> >>> https://www.ietf.org/mailman/listinfo/tcpm >>> >>> >>> >>> >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------66B5964856B4360959048994 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Yuchung,

On 29/07/2021 00:35, Yuchung Cheng wrote:
same experience with Chrisitian

For more than half a decade Google traffic used Linux CUBIC Hystart. This includes google.com and YouTube public Internet traffic, as well as internal Google TCP traffic. When we deployed pacing around 2013 we disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as it causes false SS exit. But we continued to use the stock Hystart Delay mechanism. ~2014-15 we switched to BBR which uses a different startup more robust to delay jitters.

[BB] You or Neal (sry can't remember which) were one of those I surveyed who said they had disabled Hystart. So perhaps this was a misunderstanding over the ACK-train part.

But IMO the simpler hystart mitigating SS over-shoot is still highly valuable for Cubic and other congestion controls, as Internet bandwidth continues to hike.

Did those providers disable hystart b/c of its poor interaction between the ack-train mode and pacing?

[BB] I didn't ask them why.



Bob

Maybe it's time to disable the ack-train approach in the upstream Linux when hystart++ is standardized.

On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <huitema@huitema.net> wrote:

I certainly do not qualify as a "large operator", but when implementing Cubic for QUIC I found that Hystart was critical for performance. Specifically, the classical slow-start is very prone to overshooting the capacity of the path, which causes large batches of errors. With web-like traffic, these batches of errors cause increase latency for the first transactions in a session, and thus a drop in perceived quality. Hystart solves that. With Hystart, I observed that a large fraction of sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. That's actually a big reason for moving from Cubic to BBR. Hystart suppresses the losses during the initial phase of the connection, but Cubic still relies on periodically testing the limit of path capacity and causing losses during the subsequent phase. BBR's probing for bottleneck bandwidth is much more conservative, does not cause such losses. It might be possible to adapt Cubic to also not cause losses, for example by ending an epoch early if too many CE marks are received or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work". I can't remember why I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now.


Bob


On 28/07/2021 00:46, Yuchung Cheng wrote:
Wait -- how is hystart "invariably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net <mailto:ietf@bobbriscoe.net>> wrote:

    Any large server operator out there who are using Cubic,
    If you're willing to state whether or not your operations disable
    Hystart, pls do so in reply.
    Then Praveen can cite this mailing list thread in the Hystart++
    draft, as requested below.

    If you are willing to reply privately, I would be willing to keep
    your confidences, and provide an anonymized result to the list.

    Cheers



    Bob

    On 27/07/2021 01:50, Praveen Balasubramanian wrote:

    Although Hystart is default enabled in Linux, it is invariably
    disabled. So, it's misleading to just say Hystart is default
    enabled, which implies it's widely used, when people clearly find
    it has problems (which motivates Hystart++). I found this out
    through an informal survey I did at the Mar'18 IETF in London by
    asking round the implementers of the most prevalent stacks (I
    would name names if I could find the note I later sent to someone
    or to some list, but I can't find it - sry).

    I'd like some citations on this versus anecdata if possible
    before I add that caveat to the text. Do large deployments
    disable this? I haven't come across this suggestion in any
    Linux tuning guides to date.

    --     ________________________________________________________________
    Bob Briscoehttp://bobbriscoe.net/  <http://bobbriscoe.net/>

    _______________________________________________
    tcpm mailing list
    tcpm@ietf.org <mailto:tcpm@ietf.org>
    https://www.ietf.org/mailman/listinfo/tcpm
    <https://www.ietf.org/mailman/listinfo/tcpm> 



_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--------------66B5964856B4360959048994-- From nobody Thu Jul 29 09:40:28 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 988B63A0C15 for ; Thu, 29 Jul 2021 09:40:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.097 X-Spam-Level: X-Spam-Status: No, score=-18.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v-hw39Tf39eB for ; Thu, 29 Jul 2021 09:40:21 -0700 (PDT) Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7F61C3A0C17 for ; Thu, 29 Jul 2021 09:40:21 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id f14-20020a05600c154eb02902519e4abe10so7219909wmg.4 for ; Thu, 29 Jul 2021 09:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZGa4ZwO2eFBnICzGG8n9w0dO7ckJyxxaaPtRgsu3heg=; b=QFY8jnoLcg7tgsjvquv+sJGJBcWBQmeO3ydfMFA45DSW1P8+murAB7pfYiOREQmfdR s2oYo2DGyNhlaF+jIG4JHQW5ia2xvtLQGwcAeiYLzRhsyvjc7j7lC2hML/lrOVGXO658 sQ4ySf1BeXoxbV1+DY+qyqMTHZP/EoOYIMT/wogTpklHeJSpg1lk714G/4KkNJwyUirf GxXGe0avmdhDygSpxl2FMfuKgo0kJkwfzt7RmxI/80CfxDfNviwQam7pJQCqZwCMz+RC QF2EaHA7WZQzukhxxBDl7VT4cyV1BXd+Nn0Ayc7mqDi/l+M0CPkDZgVxWJMEY5j9tBcy Sztw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZGa4ZwO2eFBnICzGG8n9w0dO7ckJyxxaaPtRgsu3heg=; b=XE1D3TzChIzNHqNxT7d7YaZ9iQaA45XL+RujfKASJH6NZnsFdT9DUk10pAAbnDID77 LsqSAA5fEEuOBvoSGomRx9+u4tiTDhQoSZhKJGV0IzcATLWxQNsG4XZPhyQ9Ld0JIP/T GnXAZA0iwB7r/NjX8cLRYrBTgGieS7s8s+WBwxPeiQ2Tsp4Uz36hZr2PsAzmOK4QSD50 u2MuiglN9S0cYZznPYueh+7l9iWjXbNDFIrpM8s2SoPYOO7MLrXMvPaKjoxJ7YeK9FAD vVBTlPhSEV5+8cCfetauVNL25vB7OQGD++XOwFFPICZDntNFE3CQ/xTHJO0lNdDEeUTa 1BQg== X-Gm-Message-State: AOAM5328XFKfFuD0kOMfB7pmcA6V8uSl/CCgleihJMX88cfzcFKh0/wN LRvmzDuklOfg2AgSjmSeyNpdZgPOaXjLBYSEvudJiw== X-Google-Smtp-Source: ABdhPJyQADPVBn/NLKWRN/anEK5ZSx3ca97QyM0YNTs8aMUL07lxJ8ryqYPS+W91LxgKnLdufPGMQEO5Hyk8hIm2oiU= X-Received: by 2002:a05:600c:354a:: with SMTP id i10mr15572350wmq.171.1627576817987; Thu, 29 Jul 2021 09:40:17 -0700 (PDT) MIME-Version: 1.0 References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> In-Reply-To: From: Yuchung Cheng Date: Thu, 29 Jul 2021 09:39:40 -0700 Message-ID: To: Bob Briscoe Cc: huitema@huitema.net, "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" , Neal Cardwell Content-Type: multipart/alternative; boundary="00000000000015945405c845c22d" Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 16:40:27 -0000 --00000000000015945405c845c22d Content-Type: text/plain; charset="UTF-8" On Thu, Jul 29, 2021 at 9:28 AM Bob Briscoe wrote: > Yuchung, > > On 29/07/2021 00:35, Yuchung Cheng wrote: > > same experience with Chrisitian > > For more than half a decade Google traffic used Linux CUBIC Hystart. This > includes google.com and YouTube public Internet traffic, as well as > internal Google TCP traffic. When we deployed pacing around 2013 we > disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as > it causes false SS exit. But we continued to use the stock Hystart Delay > mechanism. ~2014-15 we switched to BBR which uses a different startup more > robust to delay jitters. > > > [BB] You or Neal (sry can't remember which) were one of those I surveyed > who said they had disabled Hystart. So perhaps this was a misunderstanding > over the ACK-train part. > Yes this must be mis-communication. Since that was in 2018, it's either "we don't use hystart b/c BBR doesn't use hystart" or "we don't use hystart ack-train when hystart was used in Cubic before BBR". > But IMO the simpler hystart mitigating SS over-shoot is still highly > valuable for Cubic and other congestion controls, as Internet bandwidth > continues to hike. > > Did those providers disable hystart b/c of its poor interaction between > the ack-train mode and pacing? > > > [BB] I didn't ask them why. > > > > Bob > > Maybe it's time to disable the ack-train approach in the upstream > Linux when hystart++ is standardized. > > On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema > wrote: > >> I certainly do not qualify as a "large operator", but when implementing >> Cubic for QUIC I found that Hystart was critical for performance. >> Specifically, the classical slow-start is very prone to overshooting the >> capacity of the path, which causes large batches of errors. With web-like >> traffic, these batches of errors cause increase latency for the first >> transactions in a session, and thus a drop in perceived quality. Hystart >> solves that. With Hystart, I observed that a large fraction of sessions do >> not experience any packet loss at all. >> >> As an aside, we should strive for this suppression of packet losses. >> That's actually a big reason for moving from Cubic to BBR. Hystart >> suppresses the losses during the initial phase of the connection, but Cubic >> still relies on periodically testing the limit of path capacity and causing >> losses during the subsequent phase. BBR's probing for bottleneck bandwidth >> is much more conservative, does not cause such losses. It might be possible >> to adapt Cubic to also not cause losses, for example by ending an epoch >> early if too many CE marks are received or if the RTT increases. That would >> be worth trying. >> >> -- Christian Huitema >> On 7/28/2021 3:19 AM, Bob Briscoe wrote: >> >> Yuchung, >> >> It was during a Mar 2018 ad hoc workshop Jana had organized at the London >> IETF entitled 'BBR and the intersection with other work". I can't remember >> why I needed to know at the time, but during the break I approached the >> major server operators individually, established whether they used Cubic, >> and if so asked whether they used Hystart or disabled it. It's hard to >> anonymize the results, because IMMSMC all those that used Cubic said they >> disabled Hystart. >> >> If this isn't correct, then maybe people who replied at the time thought >> they were being asked something else. Or maybe it was correct then but >> isn't now. >> >> >> Bob >> >> >> On 28/07/2021 00:46, Yuchung Cheng wrote: >> >> Wait -- how is hystart "invariably disabled" in Linux (cubic)!? >> >> What data indicates that >> >> On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe > > wrote: >> >> Any large server operator out there who are using Cubic, >> If you're willing to state whether or not your operations disable >> Hystart, pls do so in reply. >> Then Praveen can cite this mailing list thread in the Hystart++ >> draft, as requested below. >> >> If you are willing to reply privately, I would be willing to keep >> your confidences, and provide an anonymized result to the list. >> >> Cheers >> >> >> >> Bob >> >> On 27/07/2021 01:50, Praveen Balasubramanian wrote: >> >> >> Although Hystart is default enabled in Linux, it is invariably >> disabled. So, it's misleading to just say Hystart is default >> enabled, which implies it's widely used, when people clearly find >> it has problems (which motivates Hystart++). I found this out >> through an informal survey I did at the Mar'18 IETF in London by >> asking round the implementers of the most prevalent stacks (I >> would name names if I could find the note I later sent to someone >> or to some list, but I can't find it - sry). >> >> >> I'd like some citations on this versus anecdata if possible >> before I add that caveat to the text. Do large deployments >> disable this? I haven't come across this suggestion in any >> Linux tuning guides to date. >> >> >> -- >> ________________________________________________________________ >> Bob Briscoehttp://bobbriscoe.net/ >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm >> >> >> >> >> >> _______________________________________________ >> tcpm mailing listtcpm@ietf.orghttps://www.ietf.org/mailman/listinfo/tcpm >> >> > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > --00000000000015945405c845c22d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 9:28 AM Bob B= riscoe <ietf@bo= bbriscoe.net> wrote:
=20 =20 =20
Yuchung,

On 29/07/2021 00:35, Yuchung Cheng wrote:
=20
same experience with Chrisitian

For more than half a decade Google traffic used=C2=A0Linux CUB= IC Hystart. This includes=C2=A0google.com=C2=A0and YouTube public Internet traffic, as well as internal Google TCP traffic.=C2=A0When we deployed pacing around 2013 we disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as it causes false SS exit.=C2=A0But we continued to use the stoc= k Hystart Delay mechanism. ~2014-15 we switched to BBR which uses a different startup more robust to delay jitters.

[BB] You or Neal (sry can't remember which) were one of those I surveyed who said they had disabled Hystart. So perhaps this was a misunderstanding over the ACK-train part.
Ye= s this must be mis-communication. Since that was in 2018, it's either &= quot;we don't use hystart b/c BBR doesn't use hystart" or &quo= t;we don't use hystart ack-train when hystart was used in Cubic before = BBR".



But IMO the simpler hystart mitigating SS over-shoot is still highly valuable for Cubic and other congestion controls, as Internet bandwidth continues=C2=A0to hike.

Did those providers disable hystart b/c of its poor interaction between the ack-train=C2=A0mode and pacing?

[BB] I didn't ask them why.



Bob

Maybe it's time to disable the ack-train approach in the upstream Linux=C2=A0when hystart++=C2=A0is standardized.

On Wed, Jul 28, 2021 at 9:47 AM Christian Huitema <huitema@huitema.net> wrote:

I certainly do not qualify as a "large operator", = but when implementing Cubic for QUIC I found that Hystart was critical for performance. Specifically, the classical slow-start is very prone to overshooting the capacity of the path, which causes large batches of errors. With web-like traffic, these batches of errors cause increase latency for the first transactions in a session, and thus a drop in perceived quality. Hystart solves that. With Hystart, I observed that a large fraction of sessions do not experience any packet loss at all.

As an aside, we should strive for this suppression of packet losses. That's actually a big reason for moving from Cubic to BBR. Hystart suppresses the losses during the initial phase of the connection, but Cubic still relies on periodically testing the limit of path capacity and causing losses during the subsequent phase. BBR's probing for bottleneck bandwidth is much more conservative, does not cause such losses. It might be possible to adapt Cubic to also not cause losses, for example by ending an epoch early if too many CE marks are received or if the RTT increases. That would be worth trying.

-- Christian Huitema

On 7/28/2021 3:19 AM, Bob Briscoe wrote:
Yuchung,

It was during a Mar 2018 ad hoc workshop Jana had organized at the London IETF entitled 'BBR and the intersection with other work". I can't remember why = I needed to know at the time, but during the break I approached the major server operators individually, established whether they used Cubic, and if so asked whether they used Hystart or disabled it. It's hard to anonymize the results, because IMMSMC all those that used Cubic said they disabled Hystart.

If this isn't correct, then maybe people who replied at the time thought they were being asked something else. Or maybe it was correct then but isn't now.


Bob


On 28/07/2021 00:46, Yuchung Cheng wrote:
Wait -- how is hystart "invari= ably disabled" in Linux (cubic)!?

What data indicates that

On Tue, Jul 27, 2021 at 4:37 PM Bob Briscoe <ietf@bobbriscoe.net <mailto:ietf@bobbris= coe.net>> wrote:

=C2=A0=C2=A0=C2=A0 Any large server operator out there who = are using Cubic,
=C2=A0=C2=A0=C2=A0 If you're willing to state whether o= r not your operations disable
=C2=A0=C2=A0=C2=A0 Hystart, pls do so in reply.
=C2=A0=C2=A0=C2=A0 Then Praveen can cite this mailing list = thread in the Hystart++
=C2=A0=C2=A0=C2=A0 draft, as requested below.

=C2=A0=C2=A0=C2=A0 If you are willing to reply privately, I= would be willing to keep
=C2=A0=C2=A0=C2=A0 your confidences, and provide an anonymi= zed result to the list.

=C2=A0=C2=A0=C2=A0 Cheers



=C2=A0=C2=A0=C2=A0 Bob

=C2=A0=C2=A0=C2=A0 On 27/07/2021 01:50, Praveen Balasubrama= nian wrote:

=C2=A0=C2=A0=C2=A0 Although Hystart is default enabled in= Linux, it is invariably
=C2=A0=C2=A0=C2=A0 disabled. So, it's misleading to j= ust say Hystart is default
=C2=A0=C2=A0=C2=A0 enabled, which implies it's widely= used, when people clearly find
=C2=A0=C2=A0=C2=A0 it has problems (which motivates Hysta= rt++). I found this out
=C2=A0=C2=A0=C2=A0 through an informal survey I did at th= e Mar'18 IETF in London by
=C2=A0=C2=A0=C2=A0 asking round the implementers of the m= ost prevalent stacks (I
=C2=A0=C2=A0=C2=A0 would name names if I could find the n= ote I later sent to someone
=C2=A0=C2=A0=C2=A0 or to some list, but I can't find = it - sry).

=C2=A0=C2=A0=C2=A0 I'd like some citations on= this versus anecdata if possible
=C2=A0=C2=A0=C2=A0 before I add that caveat to th= e text. Do large deployments
=C2=A0=C2=A0=C2=A0 disable this? I haven't co= me across this suggestion in any
=C2=A0=C2=A0=C2=A0 Linux tuning guides to date. <= br>

=C2=A0=C2=A0=C2=A0 -- =C2=A0=C2=A0=C2=A0 ___________________________________________________________= _____
=C2=A0=C2=A0=C2=A0 Bob Briscoehttp://bobbriscoe.net/=C2=A0 <ht= tp://bobbriscoe.net/>

=C2=A0=C2=A0=C2=A0 ________________________________________= _______
=C2=A0=C2=A0=C2=A0 tcpm mailing list
=C2=A0=C2=A0=C2=A0 tcpm@ietf.org <mailto:tcpm@ietf.org>
=C2=A0=C2=A0=C2=A0 https://www.ietf.org/mailman/listinfo/tcpm=
=C2=A0=C2=A0=C2=A0 <https://www.ietf.org/mailman/listinfo/= tcpm> 



_______________________________________________
tcpm mailing list
tcpm@ietf.org
ht=
tps://www.ietf.org/mailman/listinfo/tcpm


--=20
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
--00000000000015945405c845c22d-- From nobody Thu Jul 29 09:59:37 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 49AC83A0DF6 for ; Thu, 29 Jul 2021 09:59:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.097 X-Spam-Level: X-Spam-Status: No, score=-18.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TVjnv6V4ebDC for ; Thu, 29 Jul 2021 09:59:11 -0700 (PDT) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8928D3A0DEB for ; Thu, 29 Jul 2021 09:59:11 -0700 (PDT) Received: by mail-ej1-x62c.google.com with SMTP id yk17so4141182ejb.11 for ; Thu, 29 Jul 2021 09:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XSLKSSwGRpfZcrcGzaPMANDIFJu1pCXMCeDOkR39eig=; b=DyPlrku8YZ48e1C6Njx8rU0p0KBmgwrplnwPZFZdQJMtM1v+e8LcJ/iPsdy3yM4Z5Y XoNZ2mRoP6fV2HKmBHAXHQceGSRcaC7TMRZE1EznghYAwsvOA49+bP+C6Pp/tTOPb+Lc +LCU5zIKA8aM23qofUauAoolBX55WqYQNKnlpEU+eS/a2wDK+bOTDLdOze2b5J6x+3n6 cm3sPR/ofxUI9oWQQL806NTNPb2im7yZ29Sr6su+GUbJ3T2lkvSQN24wLz4XzakKHacI Hs7MwFgNvs/4d+8VfSGAZiIBswTKWrb2D06bgNjy5By/y/1ax+LPVFxekRyRkaH5NQKa enCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XSLKSSwGRpfZcrcGzaPMANDIFJu1pCXMCeDOkR39eig=; b=dgZBnThcABVvzaFSkd2OKWOWBpKX+ongreAURdLH5T3i1HvvSohazLAZxMl/5OndH8 WpeTPVhdiETAk2Jr4X0Blvdl3dYgHJzepVdzCrY0shnwjzUF9nPl52nvhAdBSyBGLJvd 8FTBCA+UagSEwoxRpakp3NeDhmuLpT0EsSv99nZmRaCMw5hMHeLKbkr1XXswNY8AXvJO Snb0ubfwqRKSXP557fKsU03ZzwlZgAC0VX7/aFRjBcrgI+rbL1+6KcZo+spWWCxrkEBs 9hVN+j6eeQQWHf8m0iGe5Sw3Qz+/+pEJBXSwE7Hs0jh1wuuTFOFi+FgqZYuM/cx7bg/Q ElAQ== X-Gm-Message-State: AOAM531xWAGFn7RZSWm/Qu3unbsO0S3iJ59UW/R306aU0gQd3So19WVb QIejUavew8QdgOSS9foBc0Xs1NvJ0RqcfXP38QeX2g== X-Google-Smtp-Source: ABdhPJx0rQb/90je5n1EmRocmSKToSgR4Ky7KFb4jhGY5cjnDbNl2Q54kYRfFmsDgnPtpwO6E+iw8gb5sbJc44bsLNI= X-Received: by 2002:a17:906:e0ce:: with SMTP id gl14mr5574236ejb.168.1627577948957; Thu, 29 Jul 2021 09:59:08 -0700 (PDT) MIME-Version: 1.0 References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> <004bfb54-32d6-0758-cd36-df52542c5a9d@bobbriscoe.net> <4c203f66-6429-d717-8213-8bdf9d3a7b2a@huitema.net> In-Reply-To: From: Neal Cardwell Date: Thu, 29 Jul 2021 12:58:51 -0400 Message-ID: To: Yuchung Cheng Cc: Bob Briscoe , huitema@huitema.net, "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Content-Type: multipart/alternative; boundary="0000000000007e5d1e05c84605c8" Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 16:59:21 -0000 --0000000000007e5d1e05c84605c8 Content-Type: text/plain; charset="UTF-8" On Thu, Jul 29, 2021 at 12:40 PM Yuchung Cheng wrote: > > > On Thu, Jul 29, 2021 at 9:28 AM Bob Briscoe wrote: > >> Yuchung, >> >> On 29/07/2021 00:35, Yuchung Cheng wrote: >> >> same experience with Chrisitian >> >> For more than half a decade Google traffic used Linux CUBIC Hystart. This >> includes google.com and YouTube public Internet traffic, as well as >> internal Google TCP traffic. When we deployed pacing around 2013 we >> disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as >> it causes false SS exit. But we continued to use the stock Hystart Delay >> mechanism. ~2014-15 we switched to BBR which uses a different startup more >> robust to delay jitters. >> >> >> [BB] You or Neal (sry can't remember which) were one of those I surveyed >> who said they had disabled Hystart. So perhaps this was a misunderstanding >> over the ACK-train part. >> > Yes this must be mis-communication. Since that was in 2018, it's either > "we don't use hystart b/c BBR doesn't use hystart" or "we don't use hystart > ack-train when hystart was used in Cubic before BBR". > Yes, my sense of our experience with CUBIC Hystart matches Yuchung's, including the likely sources of mis-communication. thanks, neal --0000000000007e5d1e05c84605c8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 12:40 PM Yuch= ung Cheng <ycheng@google.com>= ; wrote:


On Thu, Jul 29, 2021 at 9:28 AM Bob Briscoe = <ietf@bobbrisco= e.net> wrote:
=20 =20 =20
Yuchung,

On 29/07/2021 00:35, Yuchung Cheng wrote:
=20
same experience with Chrisitian

For more than half a decade Google traffic used=C2=A0Linux CUB= IC Hystart. This includes=C2=A0google.com=C2=A0and YouTube public Internet traffic, as well as internal Google TCP traffic.=C2=A0When we deployed pacing around 2013 we disabled the Hystart ACK-train mechanism (that Hystart++ prohibits now) as it causes false SS exit.=C2=A0But we continued to use the stoc= k Hystart Delay mechanism. ~2014-15 we switched to BBR which uses a different startup more robust to delay jitters.

[BB] You or Neal (sry can't remember which) were one of those I surveyed who said they had disabled Hystart. So perhaps this was a misunderstanding over the ACK-train part.
Ye= s this must be mis-communication. Since that was in 2018, it's either &= quot;we don't use hystart b/c BBR doesn't use hystart" or &quo= t;we don't use hystart ack-train when hystart was used in Cubic before = BBR".

Yes, my sense = of our experience with CUBIC Hystart matches Yuchung's, including the l= ikely sources of mis-communication.

thanks,
<= div>neal

--0000000000007e5d1e05c84605c8-- From nobody Thu Jul 29 13:19:39 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9F91D3A08DC for ; Thu, 29 Jul 2021 13:19:36 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -7.552 X-Spam-Level: X-Spam-Status: No, score=-7.552 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Utpf0XKZfC5h for ; Thu, 29 Jul 2021 13:19:28 -0700 (PDT) Received: from ma1-aaemail-dr-lapp02.apple.com (ma1-aaemail-dr-lapp02.apple.com [17.171.2.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 369B93A08CB for ; Thu, 29 Jul 2021 13:19:27 -0700 (PDT) Received: from pps.filterd (ma1-aaemail-dr-lapp02.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp02.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16TKIdUp011614; Thu, 29 Jul 2021 13:19:25 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=KfnieGMF5eH02mFyLoKxBxd+HQu/kMxqR3J93zgHbp4=; b=JnyvL2qiRSXlwiWAzJVp6qpgdMOsF0D0oSvBe+baTBiXwhEnCVF5a6SJuLaQXyJmJ/qQ PXpYiHRhsx7fjRVP3CC1z7hzdR4BxOsCc6s1+rJxCu2wyucxy97pqN/P2sr0RjJLAIpc qx38EKvf0ek1m5iAb4QLkFuWNk5nzcHl5uHbQ1LE57oiU56rgKlQ99QyNTDCpojCzwXE rLsqV/9X+L3o95g8KCXUQYbdplxLA5oH1td2Qa0qDxgRDpKb+CwWR7Rz29eXNcDAXd70 AtLC+st/TCAndVz0JplrhdZc+EsokLJgdQmo04N+6/tuu125K2+dHPe8rGUNj6q8vus+ Uw== Received: from rn-mailsvcp-mta-lapp04.rno.apple.com (rn-mailsvcp-mta-lapp04.rno.apple.com [10.225.203.152]) by ma1-aaemail-dr-lapp02.apple.com with ESMTP id 3a38p52wg8-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 29 Jul 2021 13:19:25 -0700 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QX0006UGWGDBR50@rn-mailsvcp-mta-lapp04.rno.apple.com>; Thu, 29 Jul 2021 13:19:25 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QX000I00WFDMP00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 29 Jul 2021 13:19:25 -0700 (PDT) X-Va-A: X-Va-T-CD: 4a7e35e76faa9baa589abca3404fc9ee X-Va-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-Va-R-CD: 2eec4d4b333905bf2498bde820845c7b X-Va-CD: 0 X-Va-ID: e589ef69-4f91-48f8-8371-e91c7778ef4c X-V-A: X-V-T-CD: 4a7e35e76faa9baa589abca3404fc9ee X-V-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-V-R-CD: 2eec4d4b333905bf2498bde820845c7b X-V-CD: 0 X-V-ID: f28d182d-9590-4a39-938e-fe4e83e059a5 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-29_16:2021-07-29, 2021-07-29 signatures=0 Received: from smtpclient.apple (unknown [17.11.74.148]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QX0010DIWGC1K00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 29 Jul 2021 13:19:24 -0700 (PDT) From: Vidhi Goel Message-id: <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_4596C771-A40E-41BF-8DB1-BBC0C488B5D1" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.11\)) Date: Thu, 29 Jul 2021 13:19:23 -0700 In-reply-to: Cc: Extensions To: Neal Cardwell , Mark Allman References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> X-Mailer: Apple Mail (2.3654.100.0.2.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-29_16:2021-07-29, 2021-07-29 signatures=0 Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 20:19:37 -0000 --Apple-Mail=_4596C771-A40E-41BF-8DB1-BBC0C488B5D1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. > So, it isn't exactly a new magic number. We could wave our hands > and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure > we could come up with something that folks felt was fine. However, > my feeling is that if we want to worry about bursts then let's worry > about bursts in some generic way. And, if you have some way to deal > with bursts then L isn't needed. And, if you don't have a way to > deal with bursts then a conservative L seems fine. But, perhaps > putting the effort into a generic mechanism instead of cooking yet > another magic number we need to periodically refresh is probably a > better way to spend effort. >=20 > Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort.=E2=80=9D=20= I agree that defining such a number doesn=E2=80=99t fully solve the = problem but it gives some recommendation for implementations that = don=E2=80=99t do pacing. So, defining a somewhat less restrictive value = for L (5 or 10) would be a last resort for implementations that don=E2=80=99= t pace. Thanks, Vidhi > On Jul 29, 2021, at 8:19 AM, Neal Cardwell = wrote: >=20 >=20 >=20 > On Thu, Jul 29, 2021 at 10:06 AM Mark Allman > wrote: >=20 > >> (b) If there is no burst mitigation then we have to figure out > >> if L is still useful for this purpose and whether we want = to > >> retain it. Seems like perhaps L=3D2 is sensible here. L = was > >> never meant to be some general burst mitigator. However, > >> ABC clearly *can* aggravate bursting and so perhaps it = makes > >> sense to have it also try to limit the impact of the > >> aggravation (in the absence of some general mechanism). > > > > Even if recommending a static L value, IMHO L=3D2 is a bit > > conservative. >=20 > Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. > So, it isn't exactly a new magic number. We could wave our hands > and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure > we could come up with something that folks felt was fine. However, > my feeling is that if we want to worry about bursts then let's worry > about bursts in some generic way. And, if you have some way to deal > with bursts then L isn't needed. And, if you don't have a way to > deal with bursts then a conservative L seems fine. But, perhaps > putting the effort into a generic mechanism instead of cooking yet > another magic number we need to periodically refresh is probably a > better way to spend effort. >=20 > Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort."=20 >=20 > >> - During slow starts that follow RTOs there is a general > >> problem that just because the window slides by X bytes > >> doesn't say anything about the *network*, as that sliding can > >> happen because much of the data was likely queued for the > >> application on the receiver. So, e.g., you can RTO and send > >> one packet and get an ACK back that slides the window 10 > >> packets. That doesn't mean 10 packets left. It means one > >> packet left the network and nine packets are eligible to be > >> sent to the application. So, it is not OK to set the cwnd to > >> 1+10 =3D 11 packets in response to this ACK. Here L should > >> exist and be 1. > > > > AFAICT this argument only applies to non-SACK connections. For > > connections with SACK (the vast majority of connections over the > > public Internet and in datacenters), it is quite feasible to > > determine how many packets really left the network (and Linux TCP > > does this; see below). >=20 > If you have an accurate way to figure out how many of the ACKed > bytes left the network and how many were just buffered at the > receiver then I see no problem with increasing based on byte count > as you do in the initial slow start. >=20 > (I don't remember what the paper you cite says, but my guess is it's > often the case that L=3D1 is a reasonable substitute for something > complicated here. But, perhaps I am running the simulation in my > head wrong ... it has been a while, admittedly!) >=20 > > Yes, offload mechanisms are so pervasive in practice, >=20 > I am trying to build a mental model here. How pervasive would you > guess these are? And, where in the network? I have assumed that > they are for sure pervasive in data centers and server farms, but > not for the vast majority of Internet-connected devices. >=20 > =46rom my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs. >=20 > This brings up a related point: even if there is some ABC-style = per-ACK L limit on cwnd increases, the time structure of most public = Internet ACK streams is massively bursty because of these aggregation = mechanisms inherent in L2 behavior on most public Internet bottlenecks = (wifi, cellular, DOCSIS). So even if there is a limit L that limits the = per-ACK behavior to be smooth, if there is no pacing of data segments = then the data transmit time structure will still be bursty because the = ACK arrivals these days are very bursty.=20 >=20 > best regards, > neal --Apple-Mail=_4596C771-A40E-41BF-8DB1-BBC0C488B5D1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort.=E2=80=9D 

I agree that defining such a number doesn=E2=80=99t fully = solve the problem but it gives some recommendation for implementations = that don=E2=80=99t do pacing. So, defining a somewhat less restrictive = value for L (5 or 10) would be a last resort for implementations that = don=E2=80=99t pace.

Thanks,
Vidhi



On Jul 29, 2021, at 8:19 AM, = Neal Cardwell <ncardwell@google.com> wrote:



On Thu, Jul 29, 2021 at = 10:06 AM Mark Allman <mallman@icir.org> wrote:

>>  =    (b) If there is no burst mitigation then we have to figure = out
>>         if L is = still useful for this purpose and whether we want to
>>         retain it.  = Seems like perhaps L=3D2 is sensible here.  L was
>>         never meant to be = some general burst mitigator.  However,
>>  =        ABC clearly *can* aggravate bursting and so = perhaps it makes
>>        =  sense to have it also try to limit the impact of the
>>         aggravation (in the = absence of some general mechanism).
>
> = Even if recommending a static L value, IMHO L=3D2 is a bit
> conservative.

Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort." 

>>   - During slow starts that follow RTOs = there is a general
>>     problem = that just because the window slides by X bytes
>>     doesn't say anything about the = *network*, as that sliding can
>>    =  happen because much of the data was likely queued for the
>>     application on the = receiver.  So, e.g., you can RTO and send
>>     one packet and get an ACK back = that slides the window 10
>>    =  packets.  That doesn't mean 10 packets left.  It means = one
>>     packet left the network = and nine packets are eligible to be
>>    =  sent to the application.  So, it is not OK to set the cwnd = to
>>     1+10 =3D 11 packets in = response to this ACK.  Here L should
>>  =    exist and be 1.
>
> = AFAICT this argument only applies to non-SACK connections. For
> connections with SACK (the vast majority of connections = over the
> public Internet and in datacenters), it is = quite feasible to
> determine how many packets really = left the network (and Linux TCP
> does this; see = below).

If you have an accurate way to = figure out how many of the ACKed
bytes left the network = and how many were just buffered at the
receiver then I see = no problem with increasing based on byte count
as you do = in the initial slow start.

(I don't = remember what the paper you cite says, but my guess is it's
often the case that L=3D1 is a reasonable substitute for = something
complicated here.  But, perhaps I am = running the simulation in my
head wrong ... it has been a = while, admittedly!)

> Yes, offload = mechanisms are so pervasive in practice,

I = am trying to build a mental model here.  How pervasive would you
guess these are?  And, where in the network?  I = have assumed that
they are for sure pervasive in data = centers and server farms, but
not for the vast majority of = Internet-connected devices.

=46r= om my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs.

This brings up a = related point: even if there is some ABC-style per-ACK L limit on cwnd = increases, the time structure of most public Internet ACK streams is = massively bursty because of these aggregation mechanisms inherent in L2 = behavior on most public Internet bottlenecks (wifi, cellular, DOCSIS). = So even if there is a limit L that limits the per-ACK behavior to be = smooth, if there is no pacing of data segments then the data transmit = time structure will still be bursty because the ACK arrivals these days = are very bursty. 

best regards,
neal

= --Apple-Mail=_4596C771-A40E-41BF-8DB1-BBC0C488B5D1-- From nobody Thu Jul 29 13:48:29 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2AD573A0B16 for ; Thu, 29 Jul 2021 13:48:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.097 X-Spam-Level: X-Spam-Status: No, score=-18.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VARmlIo4xM_y for ; Thu, 29 Jul 2021 13:48:23 -0700 (PDT) Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3FDB63A0B12 for ; Thu, 29 Jul 2021 13:48:23 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id o5-20020a1c4d050000b02901fc3a62af78so7668631wmh.3 for ; Thu, 29 Jul 2021 13:48:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8+QZpxAU3xXpM/jNeqsG88s2dnWrwGxZFTiFFO2Zc9s=; b=QJBlmjMqVHMofKZZPGlvHOWRqrXJCxpJa2uZHm7jgkrIQsFPmIJQgr7wcYJB/5R2qO Ofd/Oo9lAXX8WB78Jrf+7nCrD6e9v2SP7+2YErqqpT2kVKFsUgGT0QO5M4AUrJRyvmWF 02Po90Ercnm1PuAX9DBe8avrr7uPNs3U+B3eOy3QOUwU7hcmFz+f0aY7y7yRJ8M44Gr0 9b2I/tstvejEjmIRB67uxwiPuFqBVXhUDnSXHUYQq5QAZ2ZtRr7lIXjILPgxf3ht9xbA Kt+8oGLnSnAPHkAMfbCciBGFCls8TXag7sc3WOdJbkx9gxRmOgceMFKT54IuJF/DzzCa XpRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8+QZpxAU3xXpM/jNeqsG88s2dnWrwGxZFTiFFO2Zc9s=; b=L1Od4on7aMfqF/ES33dN0S4QOYwBUwXazYYvhn0p0FEIW0ucDdEJBQZqtGyQAR20q2 jGVcQDJr+cdDmAe8k0hTbFKVkG4okuNejyLcKYTHyBaM/sK/2BzAh8n3HVLrou4L45YI qzwpiFvI38LT0M+nTfWjvu+EyxRXWyEJDE6dSGcbpNi1kqGWpXAONbJ36lmrBQVU3wUV NdUqo/UHhy5Ku6CRLc/AVJLY2e1dzOQdZEQtLpKLPCivsWARmCtMgjH4f38vwBtXFC89 Yw9vzGQCRGvBu2V3Z8Cn7Ji1acLvCisnvMnIY6TV/wKrDjUKeKPSyyZM3ECBWdM/nTSl YBXQ== X-Gm-Message-State: AOAM531WIGZ632/sLRojFPJ08Vt1Lle/z2wseTzvSzkTX2op7nAx4GKO Z24Aqtx2au1fxBJPPLwUpavKsqgy0IraFuwxo361yQ== X-Google-Smtp-Source: ABdhPJzBEtPNN4A59vh/X75jUZLj7GlO7AKXHwOt3VhN7rXK6kQIfo2kZs58uLcEqXxG/qHUozZiQlsm3wWgT6W5Afw= X-Received: by 2002:a1c:f206:: with SMTP id s6mr6446543wmc.102.1627591700089; Thu, 29 Jul 2021 13:48:20 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> In-Reply-To: <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> From: Yuchung Cheng Date: Thu, 29 Jul 2021 13:47:42 -0700 Message-ID: To: Vidhi Goel Cc: Neal Cardwell , Mark Allman , Extensions Content-Type: multipart/alternative; boundary="00000000000020345805c849392d" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Jul 2021 20:48:28 -0000 --00000000000020345805c849392d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel wrote: > Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >> So, it isn't exactly a new magic number. We could wave our hands >> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >> we could come up with something that folks felt was fine. However, >> my feeling is that if we want to worry about bursts then let's worry >> about bursts in some generic way. And, if you have some way to deal >> with bursts then L isn't needed. And, if you don't have a way to >> deal with bursts then a conservative L seems fine. But, perhaps >> putting the effort into a generic mechanism instead of cooking yet >> another magic number we need to periodically refresh is probably a >> better way to spend effort. >> > > Yes, I very much agree that "putting the effort into a generic mechanism > instead of cooking yet another magic number we need to periodically refre= sh > is probably a better way to spend effort.=E2=80=9D > > > I agree that defining such a number doesn=E2=80=99t fully solve the probl= em but it > gives some recommendation for implementations that don=E2=80=99t do pacin= g. So, > defining a somewhat less restrictive value for L (5 or 10) would be a las= t > resort for implementations that don=E2=80=99t pace. > How about putting a number 10, and also put all the rationales to follow to decide a higher or lower value. It's never one-size for all. Also I believe it's time to move ABC into the standards track, in the era of (bigger and bigger) stretch ACKs. > Thanks, > Vidhi > > > > On Jul 29, 2021, at 8:19 AM, Neal Cardwell wrote: > > > > On Thu, Jul 29, 2021 at 10:06 AM Mark Allman wrote: > >> >> >> (b) If there is no burst mitigation then we have to figure out >> >> if L is still useful for this purpose and whether we want to >> >> retain it. Seems like perhaps L=3D2 is sensible here. L was >> >> never meant to be some general burst mitigator. However, >> >> ABC clearly *can* aggravate bursting and so perhaps it makes >> >> sense to have it also try to limit the impact of the >> >> aggravation (in the absence of some general mechanism). >> > >> > Even if recommending a static L value, IMHO L=3D2 is a bit >> > conservative. >> >> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >> So, it isn't exactly a new magic number. We could wave our hands >> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >> we could come up with something that folks felt was fine. However, >> my feeling is that if we want to worry about bursts then let's worry >> about bursts in some generic way. And, if you have some way to deal >> with bursts then L isn't needed. And, if you don't have a way to >> deal with bursts then a conservative L seems fine. But, perhaps >> putting the effort into a generic mechanism instead of cooking yet >> another magic number we need to periodically refresh is probably a >> better way to spend effort. >> > > Yes, I very much agree that "putting the effort into a generic mechanism > instead of cooking yet another magic number we need to periodically refre= sh > is probably a better way to spend effort." > >> >> >> - During slow starts that follow RTOs there is a general >> >> problem that just because the window slides by X bytes >> >> doesn't say anything about the *network*, as that sliding can >> >> happen because much of the data was likely queued for the >> >> application on the receiver. So, e.g., you can RTO and send >> >> one packet and get an ACK back that slides the window 10 >> >> packets. That doesn't mean 10 packets left. It means one >> >> packet left the network and nine packets are eligible to be >> >> sent to the application. So, it is not OK to set the cwnd to >> >> 1+10 =3D 11 packets in response to this ACK. Here L should >> >> exist and be 1. >> > >> > AFAICT this argument only applies to non-SACK connections. For >> > connections with SACK (the vast majority of connections over the >> > public Internet and in datacenters), it is quite feasible to >> > determine how many packets really left the network (and Linux TCP >> > does this; see below). >> >> If you have an accurate way to figure out how many of the ACKed >> bytes left the network and how many were just buffered at the >> receiver then I see no problem with increasing based on byte count >> as you do in the initial slow start. >> >> (I don't remember what the paper you cite says, but my guess is it's >> often the case that L=3D1 is a reasonable substitute for something >> complicated here. But, perhaps I am running the simulation in my >> head wrong ... it has been a while, admittedly!) >> >> > Yes, offload mechanisms are so pervasive in practice, >> >> I am trying to build a mental model here. How pervasive would you >> guess these are? And, where in the network? I have assumed that >> they are for sure pervasive in data centers and server farms, but >> not for the vast majority of Internet-connected devices. >> > > From my impression looking at public Internet traces, aggregation > mechanisms that cause TCP ACKs for more than 2 segments are very common. = I > suspect that's because the majority of public Internet traffic these days > has a bottleneck that is either wifi, cellular, or DOCSIS, and all of the= se > have a shared medium with a large latency overhead for L2 MAC control of > gets to speak next. So a lot of batching happens, both in big batches of > data that arrive at the client in the same L2 medium time slot, and big > batches of ACKs that accumulate while the client waits (often several > milliseconds, sometimes even tens of milliseconds) for its chance to send= a > big stretch ACK or batch of ACKs. > > This brings up a related point: even if there is some ABC-style per-ACK L > limit on cwnd increases, the time structure of most public Internet ACK > streams is massively bursty because of these aggregation mechanisms > inherent in L2 behavior on most public Internet bottlenecks (wifi, > cellular, DOCSIS). So even if there is a limit L that limits the per-ACK > behavior to be smooth, if there is no pacing of data segments then the da= ta > transmit time structure will still be bursty because the ACK arrivals the= se > days are very bursty. > > best regards, > neal > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --00000000000020345805c849392d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 1:19 PM Vidhi= Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
<= div class=3D"gmail_quote">
Well, perhaps.=C2=A0 L=3D2 was designed to exactly counteract delayed ACKs= .
So, it isn't exactly a new magic number.=C2=A0 We could wave our h= ands
and say "5 seems OK" or "10 seems OK" or whatev= er.=C2=A0 And, I am sure
we could come up with something that folks felt= was fine.=C2=A0 However,
my feeling is that if we want to worry about b= ursts then let's worry
about bursts in some generic way.=C2=A0 And, = if you have some way to deal
with bursts then L isn't needed.=C2=A0 = And, if you don't have a way to
deal with bursts then a conservative= L seems fine.=C2=A0 But, perhaps
putting the effort into a generic mech= anism instead of cooking yet
another magic number we need to periodicall= y refresh is probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a= generic mechanism instead of cooking yet another magic number we need to p= eriodically refresh is probably a better way to spend effort.=E2=80=9D=C2= =A0

I agree that defining such a num= ber doesn=E2=80=99t fully solve the problem but it gives some recommendatio= n for implementations that don=E2=80=99t do pacing. So, defining a somewhat= less restrictive value for L (5 or 10) would be a last resort for implemen= tations that don=E2=80=99t pace.
How about put= ting a number 10, and also put all the rationales to follow to decide a hig= her or lower value. It's never one-size for all.

Also I believe it's time to move ABC into the standards track, in th= e era of (bigger and bigger) stretch ACKs.


Thanks,
Vidhi



On Jul 29, 2021, at 8:19 AM, Neal Cardwell <ncardwell@google.com> wrote:


On Thu, Jul 29, 2021 at 10:06 AM Mark A= llman <mallman@ici= r.org> wrote:

>>=C2=A0 =C2=A0 =C2=A0(b) If there is no burst mitigation th= en we have to figure out
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if L = is still useful for this purpose and whether we want to
>>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0retain it.=C2=A0 Seems like perhaps L=3D2 is sen= sible here.=C2=A0 L was
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0never = meant to be some general burst mitigator.=C2=A0 However,
>>=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0ABC clearly *can* aggravate bursting and so perh= aps it makes
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sense to have it = also try to limit the impact of the
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0aggravation (in the absence of some general mechanism).
>
&g= t; Even if recommending a static L value, IMHO L=3D2 is a bit
> conse= rvative.

Well, perhaps.=C2=A0 L=3D2 was designed to exactly countera= ct delayed ACKs.
So, it isn't exactly a new magic number.=C2=A0 We c= ould wave our hands
and say "5 seems OK" or "10 seems OK&= quot; or whatever.=C2=A0 And, I am sure
we could come up with something = that folks felt was fine.=C2=A0 However,
my feeling is that if we want t= o worry about bursts then let's worry
about bursts in some generic w= ay.=C2=A0 And, if you have some way to deal
with bursts then L isn't= needed.=C2=A0 And, if you don't have a way to
deal with bursts then= a conservative L seems fine.=C2=A0 But, perhaps
putting the effort into= a generic mechanism instead of cooking yet
another magic number we need= to periodically refresh is probably a
better way to spend effort.

Yes, I very much agree that "putting th= e effort into a generic mechanism instead of cooking yet another magic numb= er we need to periodically refresh is probably a better way to spend effort= ."=C2=A0

&g= t;>=C2=A0 =C2=A0- During slow starts that follow RTOs there is a general=
>>=C2=A0 =C2=A0 =C2=A0problem that just because the window slides= by X bytes
>>=C2=A0 =C2=A0 =C2=A0doesn't say anything about t= he *network*, as that sliding can
>>=C2=A0 =C2=A0 =C2=A0happen bec= ause much of the data was likely queued for the
>>=C2=A0 =C2=A0 = =C2=A0application on the receiver.=C2=A0 So, e.g., you can RTO and send
= >>=C2=A0 =C2=A0 =C2=A0one packet and get an ACK back that slides the = window 10
>>=C2=A0 =C2=A0 =C2=A0packets.=C2=A0 That doesn't me= an 10 packets left.=C2=A0 It means one
>>=C2=A0 =C2=A0 =C2=A0packe= t left the network and nine packets are eligible to be
>>=C2=A0 = =C2=A0 =C2=A0sent to the application.=C2=A0 So, it is not OK to set the cwn= d to
>>=C2=A0 =C2=A0 =C2=A01+10 =3D 11 packets in response to this= ACK.=C2=A0 Here L should
>>=C2=A0 =C2=A0 =C2=A0exist and be 1.>
> AFAICT this argument only applies to non-SACK connections. Fo= r
> connections with SACK (the vast majority of connections over the<= br>> public Internet and in datacenters), it is quite feasible to
>= ; determine how many packets really left the network (and Linux TCP
>= does this; see below).

If you have an accurate way to figure out ho= w many of the ACKed
bytes left the network and how many were just buffer= ed at the
receiver then I see no problem with increasing based on byte c= ount
as you do in the initial slow start.

(I don't remember w= hat the paper you cite says, but my guess is it's
often the case tha= t L=3D1 is a reasonable substitute for something
complicated here.=C2=A0= But, perhaps I am running the simulation in my
head wrong ... it has be= en a while, admittedly!)

> Yes, offload mechanisms are so pervasi= ve in practice,

I am trying to build a mental model here.=C2=A0 How = pervasive would you
guess these are?=C2=A0 And, where in the network?=C2= =A0 I have assumed that
they are for sure pervasive in data centers and = server farms, but
not for the vast majority of Internet-connected device= s.

From my impression looking at public Internet traces= , aggregation mechanisms that cause TCP ACKs for more than 2 segments are v= ery common. I suspect that's because the majority of public Internet tr= affic these days has a bottleneck that is either wifi, cellular, or DOCSIS,= and all of these have a shared medium with a large latency overhead for L2= MAC control of gets to speak next. So a lot of batching happens, both in b= ig batches of data that arrive at the client in the same L2 medium time slo= t, and big batches of ACKs that accumulate while the client waits (often se= veral milliseconds, sometimes even tens of milliseconds) for its chance to = send a big stretch ACK or batch of ACKs.

This bri= ngs up a related point: even if there is some ABC-style per-ACK L limit on = cwnd increases, the time structure of most public Internet ACK streams is m= assively bursty because of these aggregation mechanisms inherent in L2 beha= vior on most public Internet bottlenecks (wifi, cellular, DOCSIS). So even = if there is a limit L that limits the per-ACK behavior to be smooth, if the= re is no pacing of data segments then the data transmit time structure will= still be bursty because the ACK arrivals these days are very bursty.=C2=A0=

best regards,
neal

________________= _______________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--00000000000020345805c849392d-- From nobody Thu Jul 29 18:03:28 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AE17C3A134A for ; Thu, 29 Jul 2021 18:03:26 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.551 X-Spam-Level: X-Spam-Status: No, score=-2.551 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J_M-1iRz1Zja for ; Thu, 29 Jul 2021 18:03:22 -0700 (PDT) Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1EA4B3A1349 for ; Thu, 29 Jul 2021 18:03:21 -0700 (PDT) Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 16U12tOi041857; Thu, 29 Jul 2021 18:03:14 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=qdrb3lIZnJLb7niP7zMxybJtyjWyUG6cP53yveihUXo=; b=tQKRqnKve+nPpcEU8cXn+pHe2mZUm/dE2f8xu0kuxSm+f4UdyMi9nbL4JBxtpDWr8edZ BwPP1QTEq1b5Vwg1iCJjz0j41fKdh2w+KdmQx6au9e9eb/QuhqXvrBgTzeMgAbT7j+zV jorWEZQ+N5/lTl2Gdf0gyw6Z6ncJp8x0z4FexNAUc56lNIHszONC9H1X6yCidJwYYG1Y CKLH3tqV5/w3LRnOIai2HSddfMYn8TeCwc7zuw6oJNmTEbY2BeWKQiU45GDVRQZDW+H2 ofkkhF9SP7+XAab4E34anCIqof3YjMkR0ta3aNsi71/cqzLtqJZHtV5Tjg5A1HNQEuQL tA== Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 3a41dvc8j8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 29 Jul 2021 18:03:14 -0700 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QX100YNB9LCK440@rn-mailsvcp-mta-lapp01.rno.apple.com>; Thu, 29 Jul 2021 18:03:12 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QX1008009E4Q800@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 29 Jul 2021 18:03:12 -0700 (PDT) X-Va-A: X-Va-T-CD: 637c60740750869fdac3fa3bd3c9b6e8 X-Va-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-Va-R-CD: 2eec4d4b333905bf2498bde820845c7b X-Va-CD: 0 X-Va-ID: 8e677037-20f6-46b9-ba6b-b3aec8367b9e X-V-A: X-V-T-CD: 637c60740750869fdac3fa3bd3c9b6e8 X-V-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-V-R-CD: 2eec4d4b333905bf2498bde820845c7b X-V-CD: 0 X-V-ID: 0ed97748-1c16-41f3-b41e-eeafa2c52253 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-29_20:2021-07-29, 2021-07-29 signatures=0 Received: from smtpclient.apple (unknown [17.11.74.148]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QX10067Y9LA8600@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 29 Jul 2021 18:03:11 -0700 (PDT) From: Vidhi Goel Message-id: Content-type: multipart/alternative; boundary="Apple-Mail=_32687F68-9803-4041-A39F-EBA71F6C7605" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.11\)) Date: Thu, 29 Jul 2021 18:03:10 -0700 In-reply-to: Cc: Neal Cardwell , Mark Allman , Extensions To: Yuchung Cheng References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> X-Mailer: Apple Mail (2.3654.100.0.2.11) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-29_20:2021-07-29, 2021-07-29 signatures=0 Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 01:03:27 -0000 --Apple-Mail=_32687F68-9803-4041-A39F-EBA71F6C7605 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 >> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >> So, it isn't exactly a new magic number. We could wave our hands >> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >> we could come up with something that folks felt was fine. However, >> my feeling is that if we want to worry about bursts then let's worry >> about bursts in some generic way. And, if you have some way to deal >> with bursts then L isn't needed. And, if you don't have a way to >> deal with bursts then a conservative L seems fine. But, perhaps >> putting the effort into a generic mechanism instead of cooking yet >> another magic number we need to periodically refresh is probably a >> better way to spend effort. >>=20 >> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort.=E2=80=9D=20= >=20 > I agree that defining such a number doesn=E2=80=99t fully solve the = problem but it gives some recommendation for implementations that = don=E2=80=99t do pacing. So, defining a somewhat less restrictive value = for L (5 or 10) would be a last resort for implementations that don=E2=80=99= t pace. > How about putting a number 10, and also put all the rationales to = follow to decide a higher or lower value. It's never one-size for all. That sounds great. Something on the lines of, =E2=80=9CThis document RECOMMENDS using mechanisms like Pacing to = control how many bytes are sent to the network at a point of time. But = if it is not possible to implement pacing, an implementation MAY = implicitly pace their traffic by applying a limit L to the increase in = congestion window per ACK during slow start. In modern stacks, = acknowledgments are aggregated for various reason, CPU optimization, = reducing network load etc. Hence it is common for a sender to receive an = aggregated ACK that acknowledges more than 2 segments. For example, a = stack that implements GRO could aggregate packets up to 64Kbytes or ~44 = segments before passing on to the TCP layer and this would result in a = single ACK to be generated by the TCP stack. Given that an initial = window of 10 packets in current deployments has been working fine, the = draft makes a recommendation to set L=3D10 during slow start. This would = mean that with every ACK, we are probing for a new capacity by sending = 10 packets in addition to the previously discovered capacity. = Implementations MAY choose to set a lower limit if they believe an = increase of 10 is too aggressive." Does this sound like what we would like to say? - Vidhi > On Jul 29, 2021, at 1:47 PM, Yuchung Cheng = wrote: >=20 >=20 >=20 > On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel = > wrote: >> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >> So, it isn't exactly a new magic number. We could wave our hands >> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >> we could come up with something that folks felt was fine. However, >> my feeling is that if we want to worry about bursts then let's worry >> about bursts in some generic way. And, if you have some way to deal >> with bursts then L isn't needed. And, if you don't have a way to >> deal with bursts then a conservative L seems fine. But, perhaps >> putting the effort into a generic mechanism instead of cooking yet >> another magic number we need to periodically refresh is probably a >> better way to spend effort. >>=20 >> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort.=E2=80=9D=20= >=20 > I agree that defining such a number doesn=E2=80=99t fully solve the = problem but it gives some recommendation for implementations that = don=E2=80=99t do pacing. So, defining a somewhat less restrictive value = for L (5 or 10) would be a last resort for implementations that don=E2=80=99= t pace. > How about putting a number 10, and also put all the rationales to = follow to decide a higher or lower value. It's never one-size for all. >=20 > Also I believe it's time to move ABC into the standards track, in the = era of (bigger and bigger) stretch ACKs. >=20 >=20 > Thanks, > Vidhi >=20 >=20 >=20 >> On Jul 29, 2021, at 8:19 AM, Neal Cardwell > wrote: >>=20 >>=20 >>=20 >> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman > wrote: >>=20 >> >> (b) If there is no burst mitigation then we have to figure out >> >> if L is still useful for this purpose and whether we want = to >> >> retain it. Seems like perhaps L=3D2 is sensible here. L = was >> >> never meant to be some general burst mitigator. However, >> >> ABC clearly *can* aggravate bursting and so perhaps it = makes >> >> sense to have it also try to limit the impact of the >> >> aggravation (in the absence of some general mechanism). >> > >> > Even if recommending a static L value, IMHO L=3D2 is a bit >> > conservative. >>=20 >> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >> So, it isn't exactly a new magic number. We could wave our hands >> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >> we could come up with something that folks felt was fine. However, >> my feeling is that if we want to worry about bursts then let's worry >> about bursts in some generic way. And, if you have some way to deal >> with bursts then L isn't needed. And, if you don't have a way to >> deal with bursts then a conservative L seems fine. But, perhaps >> putting the effort into a generic mechanism instead of cooking yet >> another magic number we need to periodically refresh is probably a >> better way to spend effort. >>=20 >> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort."=20 >>=20 >> >> - During slow starts that follow RTOs there is a general >> >> problem that just because the window slides by X bytes >> >> doesn't say anything about the *network*, as that sliding can >> >> happen because much of the data was likely queued for the >> >> application on the receiver. So, e.g., you can RTO and send >> >> one packet and get an ACK back that slides the window 10 >> >> packets. That doesn't mean 10 packets left. It means one >> >> packet left the network and nine packets are eligible to be >> >> sent to the application. So, it is not OK to set the cwnd to >> >> 1+10 =3D 11 packets in response to this ACK. Here L should >> >> exist and be 1. >> > >> > AFAICT this argument only applies to non-SACK connections. For >> > connections with SACK (the vast majority of connections over the >> > public Internet and in datacenters), it is quite feasible to >> > determine how many packets really left the network (and Linux TCP >> > does this; see below). >>=20 >> If you have an accurate way to figure out how many of the ACKed >> bytes left the network and how many were just buffered at the >> receiver then I see no problem with increasing based on byte count >> as you do in the initial slow start. >>=20 >> (I don't remember what the paper you cite says, but my guess is it's >> often the case that L=3D1 is a reasonable substitute for something >> complicated here. But, perhaps I am running the simulation in my >> head wrong ... it has been a while, admittedly!) >>=20 >> > Yes, offload mechanisms are so pervasive in practice, >>=20 >> I am trying to build a mental model here. How pervasive would you >> guess these are? And, where in the network? I have assumed that >> they are for sure pervasive in data centers and server farms, but >> not for the vast majority of Internet-connected devices. >>=20 >> =46rom my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs. >>=20 >> This brings up a related point: even if there is some ABC-style = per-ACK L limit on cwnd increases, the time structure of most public = Internet ACK streams is massively bursty because of these aggregation = mechanisms inherent in L2 behavior on most public Internet bottlenecks = (wifi, cellular, DOCSIS). So even if there is a limit L that limits the = per-ACK behavior to be smooth, if there is no pacing of data segments = then the data transmit time structure will still be bursty because the = ACK arrivals these days are very bursty.=20 >>=20 >> best regards, >> neal >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm = --Apple-Mail=_32687F68-9803-4041-A39F-EBA71F6C7605 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort.=E2=80=9D 

I agree that defining such a number doesn=E2=80=99t fully = solve the problem but it gives some recommendation for implementations = that don=E2=80=99t do pacing. So, defining a somewhat less restrictive = value for L (5 or 10) would be a last resort for implementations that = don=E2=80=99t pace.
How about = putting a number 10, and also put all the rationales to follow to decide = a higher or lower value. It's never one-size for = all.

That sounds great. Something on the lines of,

 =E2=80=9CThis = document RECOMMENDS using mechanisms like Pacing to control how many = bytes are sent to the network at a point of time. But if it is not = possible to implement pacing, an implementation MAY implicitly pace = their traffic by applying a limit L to the increase in congestion window = per ACK during slow start. In modern stacks, acknowledgments are = aggregated for various reason, CPU optimization, reducing network load = etc. Hence it is common for a sender to receive an aggregated ACK that = acknowledges more than 2 segments. For example, a stack that implements = GRO could aggregate packets up to 64Kbytes or ~44 segments before = passing on to the TCP layer and this would result in a single ACK to be = generated by the TCP stack. Given that an initial window of 10 = packets in current deployments has been working = fine, the draft makes a recommendation to set L=3D10 during slow start. = This would mean that with every ACK, we are probing for a new capacity = by sending 10 packets in addition to the previously discovered capacity. = Implementations MAY choose to set a lower limit if they believe an = increase of 10 is too aggressive."

Does this sound like what we would like = to say?

-
Vidhi

On Jul = 29, 2021, at 1:47 PM, Yuchung Cheng <ycheng=3D40google.com@dmarc.ietf.org> wrote:



On Thu, Jul 29, 2021 at = 1:19 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort.=E2=80=9D 

I agree that defining such a number doesn=E2=80=99t fully = solve the problem but it gives some recommendation for implementations = that don=E2=80=99t do pacing. So, defining a somewhat less restrictive = value for L (5 or 10) would be a last resort for implementations that = don=E2=80=99t pace.
How about = putting a number 10, and also put all the rationales to follow to decide = a higher or lower value. It's never one-size for all.

Also I believe it's time = to move ABC into the standards track, in the era of (bigger and bigger) = stretch ACKs.


Thanks,
Vidhi



On Jul 29, 2021, at 8:19 AM, Neal Cardwell = <ncardwell@google.com> wrote:



On Thu, = Jul 29, 2021 at 10:06 AM Mark Allman <mallman@icir.org> wrote:

>>  =    (b) If there is no burst mitigation then we have to figure = out
>>         if L is = still useful for this purpose and whether we want to
>>         retain it.  = Seems like perhaps L=3D2 is sensible here.  L was
>>         never meant to be = some general burst mitigator.  However,
>>  =        ABC clearly *can* aggravate bursting and so = perhaps it makes
>>        =  sense to have it also try to limit the impact of the
>>         aggravation (in the = absence of some general mechanism).
>
> = Even if recommending a static L value, IMHO L=3D2 is a bit
> conservative.

Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort." 

>>   - During slow starts that follow RTOs = there is a general
>>     problem = that just because the window slides by X bytes
>>     doesn't say anything about the = *network*, as that sliding can
>>    =  happen because much of the data was likely queued for the
>>     application on the = receiver.  So, e.g., you can RTO and send
>>     one packet and get an ACK back = that slides the window 10
>>    =  packets.  That doesn't mean 10 packets left.  It means = one
>>     packet left the network = and nine packets are eligible to be
>>    =  sent to the application.  So, it is not OK to set the cwnd = to
>>     1+10 =3D 11 packets in = response to this ACK.  Here L should
>>  =    exist and be 1.
>
> = AFAICT this argument only applies to non-SACK connections. For
> connections with SACK (the vast majority of connections = over the
> public Internet and in datacenters), it is = quite feasible to
> determine how many packets really = left the network (and Linux TCP
> does this; see = below).

If you have an accurate way to = figure out how many of the ACKed
bytes left the network = and how many were just buffered at the
receiver then I see = no problem with increasing based on byte count
as you do = in the initial slow start.

(I don't = remember what the paper you cite says, but my guess is it's
often the case that L=3D1 is a reasonable substitute for = something
complicated here.  But, perhaps I am = running the simulation in my
head wrong ... it has been a = while, admittedly!)

> Yes, offload = mechanisms are so pervasive in practice,

I = am trying to build a mental model here.  How pervasive would you
guess these are?  And, where in the network?  I = have assumed that
they are for sure pervasive in data = centers and server farms, but
not for the vast majority of = Internet-connected devices.

=46r= om my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs.

This brings up a = related point: even if there is some ABC-style per-ACK L limit on cwnd = increases, the time structure of most public Internet ACK streams is = massively bursty because of these aggregation mechanisms inherent in L2 = behavior on most public Internet bottlenecks (wifi, cellular, DOCSIS). = So even if there is a limit L that limits the per-ACK behavior to be = smooth, if there is no pacing of data segments then the data transmit = time structure will still be bursty because the ACK arrivals these days = are very bursty. 

best regards,
neal

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

= --Apple-Mail=_32687F68-9803-4041-A39F-EBA71F6C7605-- From nobody Fri Jul 30 11:05:30 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 72CE63A08C1 for ; Fri, 30 Jul 2021 11:05:20 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HxLaEx5xjxtX for ; Fri, 30 Jul 2021 11:05:15 -0700 (PDT) Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B37FE3A0880 for ; Fri, 30 Jul 2021 11:05:14 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id o5-20020a1c4d050000b02901fc3a62af78so9723586wmh.3 for ; Fri, 30 Jul 2021 11:05:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=O4AvUB35a1Cryzd1gOOPKfPeIGiPXV88JkB+P1a4tM4=; b=tevUfS28Y2FWRg+gLaop42RDZx4SjFIkA5nHG2cPmqsvAgIGuRnjxIpJ+If9AlC0ew 5pJcMMGK27wOsfTdi3lFlh/PvHnGrcXwhJyJGOg0Sgv+nzHMgLtnLIeW+Ox9jiifOPwT dIobZ3E8ItnSmc05mzSGR2jCa16K3ox3IfoC5Y5KVs8/kah0fCpxlMO4zClkuMlnROEo hsZ5pUFzFwQoy+bFVJSKe7nYvqgePlR0D0lWaGg9kcMDxDSsvvw7vLUX1vzUesML281f XN+HqLs1f0z9TpEqBqIHzD+nFSE4oB4l60lHFJX5rdM9g+DqPMpTbaZI5Uicp4zeNh5E QREQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=O4AvUB35a1Cryzd1gOOPKfPeIGiPXV88JkB+P1a4tM4=; b=UdvdlDhUppLOwWyzfeMMXuSJbGBVJI/nJHf33tw9Rr+ZE8uYvFc8BUJ0ED1hY8tZdE iHDDDkU/Xc5FcGreojWbvJSfr+AZK0oElIGmvzgZZvTX3U643xQRDAcbKFM4KDA77sw/ gOaQ+G5ZNJ7eanm4OdqZ6po3lHkDrK0qGREpB7uDEvRXpoZg0h8s79WF37F7IjsWz58b ftcpw2fR64xYBxUImOdQRwZWCeJQBoqHfhdpc2gvWOwqtWPwVii1nlWSKl94BgInDqXO yEdEUiP/SyrUFDl6oYoawapcJSyZOvjbNlEhXEI1+WWElhYBUJcFZC3zzutuDANAiI1O sGRw== X-Gm-Message-State: AOAM533uOvS6L+MCLoYW6lkKCf4PtrdD7r/Bt9rmhMN2tyDU3sLSHwhD 1X+p8shq/YTANbzdXDxOkxavPCgnqpFH9HPZ9oOknA== X-Google-Smtp-Source: ABdhPJyhcfnZ8u7jMfcRGForo0ccMbbviTAwcrtr7dWnozbw8+RYCyW8fimJkmB7W4K1WfgAlWvhVoFDZDgn4IXQ354= X-Received: by 2002:a05:600c:2241:: with SMTP id a1mr4296936wmm.171.1627668311254; Fri, 30 Jul 2021 11:05:11 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> In-Reply-To: From: Yuchung Cheng Date: Fri, 30 Jul 2021 11:04:32 -0700 Message-ID: To: Vidhi Goel Cc: Neal Cardwell , Mark Allman , Extensions Content-Type: multipart/alternative; boundary="00000000000081ff4805c85b0f8f" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 18:05:27 -0000 --00000000000081ff4805c85b0f8f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel wrote: > Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>> >> >> Yes, I very much agree that "putting the effort into a generic mechanism >> instead of cooking yet another magic number we need to periodically refr= esh >> is probably a better way to spend effort.=E2=80=9D >> >> >> I agree that defining such a number doesn=E2=80=99t fully solve the prob= lem but >> it gives some recommendation for implementations that don=E2=80=99t do p= acing. So, >> defining a somewhat less restrictive value for L (5 or 10) would be a la= st >> resort for implementations that don=E2=80=99t pace. >> > How about putting a number 10, and also put all the rationales to follow > to decide a higher or lower value. It's never one-size for all. > > > That sounds great. Something on the lines of, > > =E2=80=9CThis document RECOMMENDS using mechanisms like Pacing to contro= l how > many bytes are sent to the network at a point of time. But if it is not > possible to implement pacing, an implementation MAY implicitly pace their > traffic by applying a limit L to the increase in congestion window per AC= K > during slow start. In modern stacks, acknowledgments are aggregated for > various reason, CPU optimization, reducing network load etc. Hence it is > common for a sender to receive an aggregated ACK that acknowledges more > than 2 segments. For example, a stack that implements GRO could aggregate > packets up to 64Kbytes or ~44 segments before passing on to the TCP layer > and this would result in a single ACK to be generated by the TCP stack. > Given that an initial window of 10 packets in current deployments has > been working fine, the draft makes a recommendation to set L=3D10 during = slow > start. This would mean that with every ACK, we are probing for a new > capacity by sending 10 packets in addition to the previously discovered > capacity. Implementations MAY choose to set a lower limit if they believe > an increase of 10 is too aggressive." > > Does this sound like what we would like to say? > Thanks for taking a shot. I would put more description on Pacing to ensure better implementation. How about: "Pacing here refers to spread packet transmission following a rate based on the congestion window and round trip." with a citation of https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2 I would also refer to IW RFC 6928 in case it gets increased / updated a few years later. Hmm maybe we should also move RFC6928 to the standard track :-) > - > Vidhi > > On Jul 29, 2021, at 1:47 PM, Yuchung Cheng < > ycheng=3D40google.com@dmarc.ietf.org> wrote: > > > > On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel 40apple.com@dmarc.ietf.org> wrote: > >> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>> >> >> Yes, I very much agree that "putting the effort into a generic mechanism >> instead of cooking yet another magic number we need to periodically refr= esh >> is probably a better way to spend effort.=E2=80=9D >> >> >> I agree that defining such a number doesn=E2=80=99t fully solve the prob= lem but >> it gives some recommendation for implementations that don=E2=80=99t do p= acing. So, >> defining a somewhat less restrictive value for L (5 or 10) would be a la= st >> resort for implementations that don=E2=80=99t pace. >> > How about putting a number 10, and also put all the rationales to follow > to decide a higher or lower value. It's never one-size for all. > > Also I believe it's time to move ABC into the standards track, in the era > of (bigger and bigger) stretch ACKs. > > >> Thanks, >> Vidhi >> >> >> >> On Jul 29, 2021, at 8:19 AM, Neal Cardwell wrote: >> >> >> >> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman wrote: >> >>> >>> >> (b) If there is no burst mitigation then we have to figure out >>> >> if L is still useful for this purpose and whether we want to >>> >> retain it. Seems like perhaps L=3D2 is sensible here. L wa= s >>> >> never meant to be some general burst mitigator. However, >>> >> ABC clearly *can* aggravate bursting and so perhaps it makes >>> >> sense to have it also try to limit the impact of the >>> >> aggravation (in the absence of some general mechanism). >>> > >>> > Even if recommending a static L value, IMHO L=3D2 is a bit >>> > conservative. >>> >>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>> >> >> Yes, I very much agree that "putting the effort into a generic mechanism >> instead of cooking yet another magic number we need to periodically refr= esh >> is probably a better way to spend effort." >> >>> >>> >> - During slow starts that follow RTOs there is a general >>> >> problem that just because the window slides by X bytes >>> >> doesn't say anything about the *network*, as that sliding can >>> >> happen because much of the data was likely queued for the >>> >> application on the receiver. So, e.g., you can RTO and send >>> >> one packet and get an ACK back that slides the window 10 >>> >> packets. That doesn't mean 10 packets left. It means one >>> >> packet left the network and nine packets are eligible to be >>> >> sent to the application. So, it is not OK to set the cwnd to >>> >> 1+10 =3D 11 packets in response to this ACK. Here L should >>> >> exist and be 1. >>> > >>> > AFAICT this argument only applies to non-SACK connections. For >>> > connections with SACK (the vast majority of connections over the >>> > public Internet and in datacenters), it is quite feasible to >>> > determine how many packets really left the network (and Linux TCP >>> > does this; see below). >>> >>> If you have an accurate way to figure out how many of the ACKed >>> bytes left the network and how many were just buffered at the >>> receiver then I see no problem with increasing based on byte count >>> as you do in the initial slow start. >>> >>> (I don't remember what the paper you cite says, but my guess is it's >>> often the case that L=3D1 is a reasonable substitute for something >>> complicated here. But, perhaps I am running the simulation in my >>> head wrong ... it has been a while, admittedly!) >>> >>> > Yes, offload mechanisms are so pervasive in practice, >>> >>> I am trying to build a mental model here. How pervasive would you >>> guess these are? And, where in the network? I have assumed that >>> they are for sure pervasive in data centers and server farms, but >>> not for the vast majority of Internet-connected devices. >>> >> >> From my impression looking at public Internet traces, aggregation >> mechanisms that cause TCP ACKs for more than 2 segments are very common.= I >> suspect that's because the majority of public Internet traffic these day= s >> has a bottleneck that is either wifi, cellular, or DOCSIS, and all of th= ese >> have a shared medium with a large latency overhead for L2 MAC control of >> gets to speak next. So a lot of batching happens, both in big batches of >> data that arrive at the client in the same L2 medium time slot, and big >> batches of ACKs that accumulate while the client waits (often several >> milliseconds, sometimes even tens of milliseconds) for its chance to sen= d a >> big stretch ACK or batch of ACKs. >> >> This brings up a related point: even if there is some ABC-style per-ACK = L >> limit on cwnd increases, the time structure of most public Internet ACK >> streams is massively bursty because of these aggregation mechanisms >> inherent in L2 behavior on most public Internet bottlenecks (wifi, >> cellular, DOCSIS). So even if there is a limit L that limits the per-ACK >> behavior to be smooth, if there is no pacing of data segments then the d= ata >> transmit time structure will still be bursty because the ACK arrivals th= ese >> days are very bursty. >> >> best regards, >> neal >> >> >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm > > > --00000000000081ff4805c85b0f8f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Jul 29, 2021 at 6:03 PM Vidhi= Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Well, perhaps.=C2=A0 L=3D2 was designed t= o exactly counteract delayed ACKs.
So, it isn't exactly a new magic = number.=C2=A0 We could wave our hands
and say "5 seems OK" or = "10 seems OK" or whatever.=C2=A0 And, I am sure
we could come = up with something that folks felt was fine.=C2=A0 However,
my feeling is= that if we want to worry about bursts then let's worry
about bursts= in some generic way.=C2=A0 And, if you have some way to deal
with burst= s then L isn't needed.=C2=A0 And, if you don't have a way to
dea= l with bursts then a conservative L seems fine.=C2=A0 But, perhaps
putti= ng the effort into a generic mechanism instead of cooking yet
another ma= gic number we need to periodically refresh is probably a
better way to s= pend effort.

Yes, I very much agree tha= t "putting the effort into a generic mechanism instead of cooking yet = another magic number we need to periodically refresh is probably a better w= ay to spend effort.=E2=80=9D=C2=A0

I= agree that defining such a number doesn=E2=80=99t fully solve the problem = but it gives some recommendation for implementations that don=E2=80=99t do = pacing. So, defining a somewhat less restrictive value for L (5 or 10) woul= d be a last resort for implementations that don=E2=80=99t pace.
=
How about putting a number 10, and also put all the ratio= nales to follow to decide a higher or lower value. It's never one-size = for all.

<= /div>
That sounds great. Something on the lines of,

=C2=A0=E2=80=9CThis document RECOMMENDS using mechanisms like Pacing = to control how many bytes are sent to the network at a point of time. But i= f it is not possible to implement pacing, an implementation MAY implicitly = pace their traffic by applying a limit L to the increase in congestion wind= ow per ACK during slow start. In modern stacks, acknowledgments are aggrega= ted for various reason, CPU optimization, reducing network load etc. Hence = it is common for a sender to receive an aggregated ACK that acknowledges mo= re than 2 segments. For example, a stack that implements GRO could aggregat= e packets up to 64Kbytes or ~44 segments before passing on to the TCP layer= and this would result in a single ACK to be generated by the TCP stack. Gi= ven that an initial window of 10 packets=C2=A0in current deployments=C2=A0has been working fine, the draft mak= es a recommendation to set L=3D10 during slow start. This would mean that w= ith every ACK, we are probing for a new capacity by sending 10 packets in a= ddition to the previously discovered capacity. Implementations MAY choose t= o set a lower limit if they believe an increase of 10 is too aggressive.&qu= ot;

Does this sound like what we would like to say= ?
Thanks for=C2=A0taking a shot. I= would put more description on Pacing to=C2=A0ensure better implementation.= How about:
"Pacing here refers to spread packet transmissio= n following a rate based on the congestion window and round trip." wit= h a citation of=C2=A0https://datatracker.ietf.org/doc/html/rfc7661#section-4.4= .2



I would also = refer to IW RFC 6928=C2=A0in case it gets increased / updated a few years l= ater.
Hmm maybe we should also move RFC6928=C2=A0to the standard = track :-)


-
V= idhi

On Jul 29, 20= 21, at 1:47 PM, Yuchung Cheng <ycheng=3D40google.com@dmarc.ietf.org&g= t; wrote:



On Thu, Jul 29, 2021 = at 1:19 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Well, perhaps.=C2=A0 L=3D2 was designed to exactly counter= act delayed ACKs.
So, it isn't exactly a new magic number.=C2=A0 We = could wave our hands
and say "5 seems OK" or "10 seems OK= " or whatever.=C2=A0 And, I am sure
we could come up with something= that folks felt was fine.=C2=A0 However,
my feeling is that if we want = to worry about bursts then let's worry
about bursts in some generic = way.=C2=A0 And, if you have some way to deal
with bursts then L isn'= t needed.=C2=A0 And, if you don't have a way to
deal with bursts the= n a conservative L seems fine.=C2=A0 But, perhaps
putting the effort int= o a generic mechanism instead of cooking yet
another magic number we nee= d to periodically refresh is probably a
better way to spend effort.
<= /blockquote>

Yes, I very much agree that "putting t= he effort into a generic mechanism instead of cooking yet another magic num= ber we need to periodically refresh is probably a better way to spend effor= t.=E2=80=9D=C2=A0

I agree that defin= ing such a number doesn=E2=80=99t fully solve the problem but it gives some= recommendation for implementations that don=E2=80=99t do pacing. So, defin= ing a somewhat less restrictive value for L (5 or 10) would be a last resor= t for implementations that don=E2=80=99t pace.
How about putting a number 10, and also put all the rationales to follow t= o decide a higher or lower value. It's never one-size for all.

Also I believe it's time to move ABC into the standard= s track, in the era of (bigger and bigger) stretch ACKs.


Thanks,
Vidhi



On Jul 29, 2021, at 8:19 AM, Neal Cardwell <ncardwell@google.com> w= rote:



On Thu, Jul 29, 2021 at 1= 0:06 AM Mark Allman <mallman@icir.org> wrote:

>>=C2=A0 =C2=A0 =C2=A0(b) If there is no burst= mitigation then we have to figure out
>>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0if L is still useful for this purpose and whether we want to
&= gt;>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0retain it.=C2=A0 Seems like perhap= s L=3D2 is sensible here.=C2=A0 L was
>>=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0never meant to be some general burst mitigator.=C2=A0 However,>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ABC clearly *can* aggravate burs= ting and so perhaps it makes
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0s= ense to have it also try to limit the impact of the
>>=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0aggravation (in the absence of some general mechani= sm).
>
> Even if recommending a static L value, IMHO L=3D2 is a= bit
> conservative.

Well, perhaps.=C2=A0 L=3D2 was designed t= o exactly counteract delayed ACKs.
So, it isn't exactly a new magic = number.=C2=A0 We could wave our hands
and say "5 seems OK" or = "10 seems OK" or whatever.=C2=A0 And, I am sure
we could come = up with something that folks felt was fine.=C2=A0 However,
my feeling is= that if we want to worry about bursts then let's worry
about bursts= in some generic way.=C2=A0 And, if you have some way to deal
with burst= s then L isn't needed.=C2=A0 And, if you don't have a way to
dea= l with bursts then a conservative L seems fine.=C2=A0 But, perhaps
putti= ng the effort into a generic mechanism instead of cooking yet
another ma= gic number we need to periodically refresh is probably a
better way to s= pend effort.

Yes, I very much agree tha= t "putting the effort into a generic mechanism instead of cooking yet = another magic number we need to periodically refresh is probably a better w= ay to spend effort."=C2=A0

>>=C2=A0 =C2=A0- During slow starts that follow RTOs = there is a general
>>=C2=A0 =C2=A0 =C2=A0problem that just because= the window slides by X bytes
>>=C2=A0 =C2=A0 =C2=A0doesn't sa= y anything about the *network*, as that sliding can
>>=C2=A0 =C2= =A0 =C2=A0happen because much of the data was likely queued for the
>= >=C2=A0 =C2=A0 =C2=A0application on the receiver.=C2=A0 So, e.g., you ca= n RTO and send
>>=C2=A0 =C2=A0 =C2=A0one packet and get an ACK bac= k that slides the window 10
>>=C2=A0 =C2=A0 =C2=A0packets.=C2=A0 T= hat doesn't mean 10 packets left.=C2=A0 It means one
>>=C2=A0 = =C2=A0 =C2=A0packet left the network and nine packets are eligible to be>>=C2=A0 =C2=A0 =C2=A0sent to the application.=C2=A0 So, it is not O= K to set the cwnd to
>>=C2=A0 =C2=A0 =C2=A01+10 =3D 11 packets in = response to this ACK.=C2=A0 Here L should
>>=C2=A0 =C2=A0 =C2=A0ex= ist and be 1.
>
> AFAICT this argument only applies to non-SACK= connections. For
> connections with SACK (the vast majority of conne= ctions over the
> public Internet and in datacenters), it is quite fe= asible to
> determine how many packets really left the network (and L= inux TCP
> does this; see below).

If you have an accurate way = to figure out how many of the ACKed
bytes left the network and how many = were just buffered at the
receiver then I see no problem with increasing= based on byte count
as you do in the initial slow start.

(I don&= #39;t remember what the paper you cite says, but my guess is it's
of= ten the case that L=3D1 is a reasonable substitute for something
complic= ated here.=C2=A0 But, perhaps I am running the simulation in my
head wro= ng ... it has been a while, admittedly!)

> Yes, offload mechanism= s are so pervasive in practice,

I am trying to build a mental model = here.=C2=A0 How pervasive would you
guess these are?=C2=A0 And, where in= the network?=C2=A0 I have assumed that
they are for sure pervasive in d= ata centers and server farms, but
not for the vast majority of Internet-= connected devices.

From my impression looking at public= Internet traces, aggregation mechanisms that cause TCP ACKs for more than = 2 segments are very common. I suspect that's because the majority of pu= blic Internet traffic these days has a bottleneck that is either wifi, cell= ular, or DOCSIS, and all of these have a shared medium with a large latency= overhead for L2 MAC control of gets to speak next. So a lot of batching ha= ppens, both in big batches of data that arrive at the client in the same L2= medium time slot, and big batches of ACKs that accumulate while the client= waits (often several milliseconds, sometimes even tens of milliseconds) fo= r its chance to send a big stretch ACK or batch of ACKs.

This brings up a related point: even if there is some ABC-style per= -ACK L limit on cwnd increases, the time structure of most public Internet = ACK streams is massively bursty because of these aggregation mechanisms inh= erent in L2 behavior on most public Internet bottlenecks (wifi, cellular, D= OCSIS). So even if there is a limit L that limits the per-ACK behavior to b= e smooth, if there is no pacing of data segments then the data transmit tim= e structure will still be bursty because the ACK arrivals these days are ve= ry bursty.=C2=A0

=
best regards,
neal

= _______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

--00000000000081ff4805c85b0f8f-- From nobody Fri Jul 30 13:26:22 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5373A3A104A for ; Fri, 30 Jul 2021 13:26:12 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.548 X-Spam-Level: X-Spam-Status: No, score=-2.548 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E_jAsmAysqZh for ; Fri, 30 Jul 2021 13:26:08 -0700 (PDT) Received: from rn-mailsvcp-ppex-lapp15.apple.com (rn-mailsvcp-ppex-lapp15.rno.apple.com [17.179.253.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4F9D33A0FEC for ; Fri, 30 Jul 2021 13:25:56 -0700 (PDT) Received: from pps.filterd (rn-mailsvcp-ppex-lapp15.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp15.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 16UKN4mS016056; Fri, 30 Jul 2021 13:25:52 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=6xaULiUaHiLpPTKPDJ/3vsDZzxqR67uNxcGGIw9U8Ow=; b=kCmT/HNDPBdjmSz4w6NU971O63B6L2obvzILj2WhgNYIOvAg1BSxN9M7eug8N7p0l72O 0xLSVmjDWCHfTobgJdcrCX6K7hwR+ch9D4ZQgSni2ZldarMlMhJ9vj10ZWnQAr0sAppM 2VRTEJIJu260kEs/5WQbXQHI2A16mqW8KxEZX4bgCdZFyvlHQ9lbzkdtTsIyRDgI5bMJ hfpvU65babSttNaqlmlV/KP+T1HvUduKUu0W6EJRcY1Sb9L0jUEZhBoDIHbCE0LQP3DN kxItJyt+z1iZFujcZoJn5UBNqzawxyu9kK01NTyffOi5jN+w4pRyI83sRvnStGf5U5s7 dA== Received: from rn-mailsvcp-mta-lapp04.rno.apple.com (rn-mailsvcp-mta-lapp04.rno.apple.com [10.225.203.152]) by rn-mailsvcp-ppex-lapp15.rno.apple.com with ESMTP id 3a235sxq1x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 30 Jul 2021 13:25:52 -0700 Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPS id <0QX200HXCRF4SMB0@rn-mailsvcp-mta-lapp04.rno.apple.com>; Fri, 30 Jul 2021 13:25:52 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) id <0QX200500R6D5Q00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Fri, 30 Jul 2021 13:25:52 -0700 (PDT) X-Va-A: X-Va-T-CD: 2e660d2641f10d05ffec32ae59e5d560 X-Va-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-Va-R-CD: 2eec4d4b333905bf2498bde820845c7b X-Va-CD: 0 X-Va-ID: 0b6d172c-8c58-434a-b3e8-77382d952abf X-V-A: X-V-T-CD: 2e660d2641f10d05ffec32ae59e5d560 X-V-E-CD: 8ad83cf34a8c3732cb73dea81270d36d X-V-R-CD: 2eec4d4b333905bf2498bde820845c7b X-V-CD: 0 X-V-ID: d25c34a7-260c-4cad-88cb-9738bea50e3e X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-30_11:2021-07-30, 2021-07-30 signatures=0 Received: from smtpclient.apple (vimac.scv.apple.com [17.192.171.33]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.9.20210415 64bit (built Apr 15 2021)) with ESMTPSA id <0QX200GKSRF36L00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Fri, 30 Jul 2021 13:25:51 -0700 (PDT) From: Vidhi Goel Message-id: <0CB3BAA7-721D-42F0-B302-B626B26A4D32@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_21329852-9850-4D9D-8F0B-D89603BF7C43" MIME-version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Date: Fri, 30 Jul 2021 13:25:51 -0700 In-reply-to: Cc: Extensions To: Yuchung Cheng , Mark Allman , Neal Cardwell References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-30_11:2021-07-30, 2021-07-30 signatures=0 Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 20:26:20 -0000 --Apple-Mail=_21329852-9850-4D9D-8F0B-D89603BF7C43 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Mark, Yuchung, Neal, Given that we have some ideas / suggestion text on improving 3465, how = do you think we should proceed? Mark, do you want to start a GitHub repo or such with some of the = changes already suggested so far and others can review and / or = contribute to the bis draft? Thanks, Vidhi > On Jul 30, 2021, at 11:04 AM, Yuchung Cheng = wrote: >=20 >=20 >=20 > On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel = > wrote: >>> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>>=20 >>> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort.=E2=80=9D=20= >>=20 >> I agree that defining such a number doesn=E2=80=99t fully solve the = problem but it gives some recommendation for implementations that = don=E2=80=99t do pacing. So, defining a somewhat less restrictive value = for L (5 or 10) would be a last resort for implementations that don=E2=80=99= t pace. >> How about putting a number 10, and also put all the rationales to = follow to decide a higher or lower value. It's never one-size for all. >=20 > That sounds great. Something on the lines of, >=20 > =E2=80=9CThis document RECOMMENDS using mechanisms like Pacing to = control how many bytes are sent to the network at a point of time. But = if it is not possible to implement pacing, an implementation MAY = implicitly pace their traffic by applying a limit L to the increase in = congestion window per ACK during slow start. In modern stacks, = acknowledgments are aggregated for various reason, CPU optimization, = reducing network load etc. Hence it is common for a sender to receive an = aggregated ACK that acknowledges more than 2 segments. For example, a = stack that implements GRO could aggregate packets up to 64Kbytes or ~44 = segments before passing on to the TCP layer and this would result in a = single ACK to be generated by the TCP stack. Given that an initial = window of 10 packets in current deployments has been working fine, the = draft makes a recommendation to set L=3D10 during slow start. This would = mean that with every ACK, we are probing for a new capacity by sending = 10 packets in addition to the previously discovered capacity. = Implementations MAY choose to set a lower limit if they believe an = increase of 10 is too aggressive." >=20 > Does this sound like what we would like to say? > Thanks for taking a shot. I would put more description on Pacing to = ensure better implementation. How about: > "Pacing here refers to spread packet transmission following a rate = based on the congestion window and round trip." with a citation of = https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2 = >=20 >=20 >=20 > I would also refer to IW RFC 6928 in case it gets increased / updated = a few years later. > Hmm maybe we should also move RFC6928 to the standard track :-) >=20 >=20 > - > Vidhi >=20 >> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng = > wrote: >>=20 >>=20 >>=20 >> On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel = > wrote: >>> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>>=20 >>> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort.=E2=80=9D=20= >>=20 >> I agree that defining such a number doesn=E2=80=99t fully solve the = problem but it gives some recommendation for implementations that = don=E2=80=99t do pacing. So, defining a somewhat less restrictive value = for L (5 or 10) would be a last resort for implementations that don=E2=80=99= t pace. >> How about putting a number 10, and also put all the rationales to = follow to decide a higher or lower value. It's never one-size for all. >>=20 >> Also I believe it's time to move ABC into the standards track, in the = era of (bigger and bigger) stretch ACKs. >>=20 >>=20 >> Thanks, >> Vidhi >>=20 >>=20 >>=20 >>> On Jul 29, 2021, at 8:19 AM, Neal Cardwell > wrote: >>>=20 >>>=20 >>>=20 >>> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman > wrote: >>>=20 >>> >> (b) If there is no burst mitigation then we have to figure = out >>> >> if L is still useful for this purpose and whether we want = to >>> >> retain it. Seems like perhaps L=3D2 is sensible here. L = was >>> >> never meant to be some general burst mitigator. However, >>> >> ABC clearly *can* aggravate bursting and so perhaps it = makes >>> >> sense to have it also try to limit the impact of the >>> >> aggravation (in the absence of some general mechanism). >>> > >>> > Even if recommending a static L value, IMHO L=3D2 is a bit >>> > conservative. >>>=20 >>> Well, perhaps. L=3D2 was designed to exactly counteract delayed = ACKs. >>> So, it isn't exactly a new magic number. We could wave our hands >>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>> we could come up with something that folks felt was fine. However, >>> my feeling is that if we want to worry about bursts then let's worry >>> about bursts in some generic way. And, if you have some way to deal >>> with bursts then L isn't needed. And, if you don't have a way to >>> deal with bursts then a conservative L seems fine. But, perhaps >>> putting the effort into a generic mechanism instead of cooking yet >>> another magic number we need to periodically refresh is probably a >>> better way to spend effort. >>>=20 >>> Yes, I very much agree that "putting the effort into a generic = mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend effort."=20 >>>=20 >>> >> - During slow starts that follow RTOs there is a general >>> >> problem that just because the window slides by X bytes >>> >> doesn't say anything about the *network*, as that sliding can >>> >> happen because much of the data was likely queued for the >>> >> application on the receiver. So, e.g., you can RTO and send >>> >> one packet and get an ACK back that slides the window 10 >>> >> packets. That doesn't mean 10 packets left. It means one >>> >> packet left the network and nine packets are eligible to be >>> >> sent to the application. So, it is not OK to set the cwnd to >>> >> 1+10 =3D 11 packets in response to this ACK. Here L should >>> >> exist and be 1. >>> > >>> > AFAICT this argument only applies to non-SACK connections. For >>> > connections with SACK (the vast majority of connections over the >>> > public Internet and in datacenters), it is quite feasible to >>> > determine how many packets really left the network (and Linux TCP >>> > does this; see below). >>>=20 >>> If you have an accurate way to figure out how many of the ACKed >>> bytes left the network and how many were just buffered at the >>> receiver then I see no problem with increasing based on byte count >>> as you do in the initial slow start. >>>=20 >>> (I don't remember what the paper you cite says, but my guess is it's >>> often the case that L=3D1 is a reasonable substitute for something >>> complicated here. But, perhaps I am running the simulation in my >>> head wrong ... it has been a while, admittedly!) >>>=20 >>> > Yes, offload mechanisms are so pervasive in practice, >>>=20 >>> I am trying to build a mental model here. How pervasive would you >>> guess these are? And, where in the network? I have assumed that >>> they are for sure pervasive in data centers and server farms, but >>> not for the vast majority of Internet-connected devices. >>>=20 >>> =46rom my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs. >>>=20 >>> This brings up a related point: even if there is some ABC-style = per-ACK L limit on cwnd increases, the time structure of most public = Internet ACK streams is massively bursty because of these aggregation = mechanisms inherent in L2 behavior on most public Internet bottlenecks = (wifi, cellular, DOCSIS). So even if there is a limit L that limits the = per-ACK behavior to be smooth, if there is no pacing of data segments = then the data transmit time structure will still be bursty because the = ACK arrivals these days are very bursty.=20 >>>=20 >>> best regards, >>> neal >>=20 >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm = --Apple-Mail=_21329852-9850-4D9D-8F0B-D89603BF7C43 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi Mark, Yuchung, Neal,

Given that we have some ideas / suggestion text on = improving 3465, how do you think we should proceed?
Mark, = do you want to start a GitHub repo or such with some of the changes = already suggested so far and others can review and / or contribute to = the bis draft?

Thanks,
Vidhi

On Jul = 30, 2021, at 11:04 AM, Yuchung Cheng <ycheng=3D40google.com@dmarc.ietf.org> wrote:



On Thu, Jul 29, 2021 at = 6:03 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort.=E2=80=9D 

I agree that defining such a number doesn=E2=80=99t fully = solve the problem but it gives some recommendation for implementations = that don=E2=80=99t do pacing. So, defining a somewhat less restrictive = value for L (5 or 10) would be a last resort for implementations that = don=E2=80=99t pace.
How about = putting a number 10, and also put all the rationales to follow to decide = a higher or lower value. It's never one-size for = all.

That sounds great. Something on the lines of,

 =E2=80=9CThis = document RECOMMENDS using mechanisms like Pacing to control how many = bytes are sent to the network at a point of time. But if it is not = possible to implement pacing, an implementation MAY implicitly pace = their traffic by applying a limit L to the increase in congestion window = per ACK during slow start. In modern stacks, acknowledgments are = aggregated for various reason, CPU optimization, reducing network load = etc. Hence it is common for a sender to receive an aggregated ACK that = acknowledges more than 2 segments. For example, a stack that implements = GRO could aggregate packets up to 64Kbytes or ~44 segments before = passing on to the TCP layer and this would result in a single ACK to be = generated by the TCP stack. Given that an initial window of 10 = packets in current = deployments has been working fine, the draft makes a = recommendation to set L=3D10 during slow start. This would mean that = with every ACK, we are probing for a new capacity by sending 10 packets = in addition to the previously discovered capacity. Implementations MAY = choose to set a lower limit if they believe an increase of 10 is too = aggressive."

Does this sound like what we would like to = say?
Thanks = for taking a shot. I would put more description on Pacing = to ensure better implementation. How about:
"Pacing here refers to spread packet transmission following a = rate based on the congestion window and round trip." with a citation = of https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2=



I = would also refer to IW RFC 6928 in case it gets increased / updated = a few years later.
Hmm maybe we should also move = RFC6928 to the standard track :-)


-
Vidhi

On Jul = 29, 2021, at 1:47 PM, Yuchung Cheng <ycheng=3D40google.com@dmarc.ietf.org> wrote:



On Thu, = Jul 29, 2021 at 1:19 PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort.=E2=80=9D 

I agree that defining such a number doesn=E2=80=99t fully = solve the problem but it gives some recommendation for implementations = that don=E2=80=99t do pacing. So, defining a somewhat less restrictive = value for L (5 or 10) would be a last resort for implementations that = don=E2=80=99t pace.
How about = putting a number 10, and also put all the rationales to follow to decide = a higher or lower value. It's never one-size for all.

Also I believe it's time = to move ABC into the standards track, in the era of (bigger and bigger) = stretch ACKs.


Thanks,
Vidhi



On Jul 29, 2021, at 8:19 AM, Neal Cardwell = <ncardwell@google.com> wrote:



On Thu, = Jul 29, 2021 at 10:06 AM Mark Allman <mallman@icir.org> wrote:

>>  =    (b) If there is no burst mitigation then we have to figure = out
>>         if L is = still useful for this purpose and whether we want to
>>         retain it.  = Seems like perhaps L=3D2 is sensible here.  L was
>>         never meant to be = some general burst mitigator.  However,
>>  =        ABC clearly *can* aggravate bursting and so = perhaps it makes
>>        =  sense to have it also try to limit the impact of the
>>         aggravation (in the = absence of some general mechanism).
>
> = Even if recommending a static L value, IMHO L=3D2 is a bit
> conservative.

Well, = perhaps.  L=3D2 was designed to exactly counteract delayed ACKs.
So, it isn't exactly a new magic number.  We could wave = our hands
and say "5 seems OK" or "10 seems OK" or = whatever.  And, I am sure
we could come up with = something that folks felt was fine.  However,
my = feeling is that if we want to worry about bursts then let's worry
about bursts in some generic way.  And, if you have some = way to deal
with bursts then L isn't needed.  And, if = you don't have a way to
deal with bursts then a = conservative L seems fine.  But, perhaps
putting the = effort into a generic mechanism instead of cooking yet
another magic number we need to periodically refresh is = probably a
better way to spend effort.

Yes, I very much agree that "putting the effort into a = generic mechanism instead of cooking yet another magic number we need to = periodically refresh is probably a better way to spend = effort." 

>>   - During slow starts that follow RTOs = there is a general
>>     problem = that just because the window slides by X bytes
>>     doesn't say anything about the = *network*, as that sliding can
>>    =  happen because much of the data was likely queued for the
>>     application on the = receiver.  So, e.g., you can RTO and send
>>     one packet and get an ACK back = that slides the window 10
>>    =  packets.  That doesn't mean 10 packets left.  It means = one
>>     packet left the network = and nine packets are eligible to be
>>    =  sent to the application.  So, it is not OK to set the cwnd = to
>>     1+10 =3D 11 packets in = response to this ACK.  Here L should
>>  =    exist and be 1.
>
> = AFAICT this argument only applies to non-SACK connections. For
> connections with SACK (the vast majority of connections = over the
> public Internet and in datacenters), it is = quite feasible to
> determine how many packets really = left the network (and Linux TCP
> does this; see = below).

If you have an accurate way to = figure out how many of the ACKed
bytes left the network = and how many were just buffered at the
receiver then I see = no problem with increasing based on byte count
as you do = in the initial slow start.

(I don't = remember what the paper you cite says, but my guess is it's
often the case that L=3D1 is a reasonable substitute for = something
complicated here.  But, perhaps I am = running the simulation in my
head wrong ... it has been a = while, admittedly!)

> Yes, offload = mechanisms are so pervasive in practice,

I = am trying to build a mental model here.  How pervasive would you
guess these are?  And, where in the network?  I = have assumed that
they are for sure pervasive in data = centers and server farms, but
not for the vast majority of = Internet-connected devices.

=46r= om my impression looking at public Internet traces, aggregation = mechanisms that cause TCP ACKs for more than 2 segments are very common. = I suspect that's because the majority of public Internet traffic these = days has a bottleneck that is either wifi, cellular, or DOCSIS, and all = of these have a shared medium with a large latency overhead for L2 MAC = control of gets to speak next. So a lot of batching happens, both in big = batches of data that arrive at the client in the same L2 medium time = slot, and big batches of ACKs that accumulate while the client waits = (often several milliseconds, sometimes even tens of milliseconds) for = its chance to send a big stretch ACK or batch of ACKs.

This brings up a = related point: even if there is some ABC-style per-ACK L limit on cwnd = increases, the time structure of most public Internet ACK streams is = massively bursty because of these aggregation mechanisms inherent in L2 = behavior on most public Internet bottlenecks (wifi, cellular, DOCSIS). = So even if there is a limit L that limits the per-ACK behavior to be = smooth, if there is no pacing of data segments then the data transmit = time structure will still be bursty because the ACK arrivals these days = are very bursty. 

best regards,
neal

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

= --Apple-Mail=_21329852-9850-4D9D-8F0B-D89603BF7C43-- From nobody Fri Jul 30 13:36:08 2021 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id A92353A0F02; Fri, 30 Jul 2021 13:35:57 -0700 (PDT) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: Kyle Rose via Datatracker To: Cc: draft-ietf-tcpm-rfc793bis.all@ietf.org, last-call@ietf.org, tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.35.0 Auto-Submitted: auto-generated Precedence: bulk Message-ID: <162767735763.27351.5673596060247016004@ietfa.amsl.com> Reply-To: Kyle Rose Date: Fri, 30 Jul 2021 13:35:57 -0700 Archived-At: Subject: [tcpm] Secdir last call review of draft-ietf-tcpm-rfc793bis-24 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 20:35:58 -0000 Reviewer: Kyle Rose Review result: Ready I have reviewed this document as part of the security directorate's ongoing effort to review all IETF documents being processed by the IESG. These comments were written primarily for the benefit of the security area directors. Document editors and WG chairs should treat these comments just like any other last call comments. This document is Almost Ready for publication. The Security and Privacy Considerations section briefly outlines the implications of TCP's long legacy, stretching back to the halcyon days of a friendlier internet. The paragraphs outlining layered security (e.g., IPsec, TLS) and TCP option-based security (e.g., TCP-AO, tcpcrypt) provide a good summary of the mechanisms since developed to add security to a protocol that has otherwise met the challenges of an internet that is many orders of magnitude larger than it was 40 years ago. The one item I see missing from this section is a mention of lessons learned and subsequently applied to the design of QUIC. I think it is worth mentioning, for instance, that TCP's large surface area of cleartext metadata exposes more information to the path than required to successfully route packets to their destination, including to on-path adversaries that may be able to use this metadata to bolster targeted or pervasive surveillance. There is one more omission, adjacent to (but not explicitly about) security, that I think warrants some text in this document: that is around ossification. Given the lengthy back-and-forth I witnessed as chair of the TCPINC WG regarding the (in)feasibility of protecting segmentation and header values on the public internet, it is probably worth adding to a 793bis document a section that briefly outlines the ossification impact of voluminous cleartext and unprotected/un-GREASEd metadata, maybe with a reference to the wire image as defined by RFC 8546. The reason I think this is worthwhile is that it would be good to have the practical limits of TCP extensibility (i.e., in a world with middleboxes and other deeply TCP-aware network elements) documented where folks might look for it when thinking about new options or other new functionality. I would be happy to help flesh out some text here. From nobody Fri Jul 30 16:05:43 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 984893A153E for ; Fri, 30 Jul 2021 16:05:39 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.096 X-Spam-Level: X-Spam-Status: No, score=-18.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ptJ9BJnNdvzM for ; Fri, 30 Jul 2021 16:05:33 -0700 (PDT) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CB3F93A0B1D for ; Fri, 30 Jul 2021 16:05:32 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id m20-20020a05600c4f54b029024e75a15716so7354157wmq.2 for ; Fri, 30 Jul 2021 16:05:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7ql2chQuXsqycgRQbWCinOP+1x3fBJMukZWdQHmzYdw=; b=D+BV1q3B/6soreWHxhw5SPw1nCBqjDzHMW+rDxajJZxA3f2QL4UhXW+UcOzr98Qco7 sxVGNDi28VSHbDZtM+tDY13NdVV+fG6aA8HAxOyxmkogR+PMwDfcr9A3sHhLz3XGoAcX wRXUlOgEM5mlDvaeHiXdKtL6vinOZ7dWthiIHRbrK7dTyyK5XYCU83YbkCJlmGrhYvIv IQ0rPniAnxZ3mCyISyMwFFFCHwHgY114oT7oRrGBRj8/3W1CMWjan78eJ/LIS8eyjBwE tEcObTV/hX4ONelQtNU3wjQT1mjONxJnCt9xdiCoyiDfCU4wxGS6F5eBmRMYp+oPo71u DFxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7ql2chQuXsqycgRQbWCinOP+1x3fBJMukZWdQHmzYdw=; b=AUT4wHduIQ+2blV2u9QOgE5JyeDvd8QdwNtPBwby59CaJL/7Sp1OdEtvLjK+A7ycB7 38hBF+NFU2HFzDzdHiDUCKOBrM8Y24DwoEGwagSXPatuhg36m2wBNdUDaNoevKb4ispu qNEU93IXOdY1B4cv0rvBcJDvmn5KLhFFBZMqQ+ha/BUqKmd53TDTMYCsdKMnMti6dH7L Yl1ppz6IvhAXWbWziLgmcKDNCb/AqzqSzpLQeSLkABSJtOH2eLk1qUXbqyCWeXxPp29b 6ErpG0KnlCmyMhSw/MtbBP6Vs/c8Ue3VJKrCJJ4+YTrftbDhi2fMFORq9F3FLrbuphsJ WsQQ== X-Gm-Message-State: AOAM530M1qTqbo0YfgsOwtaXRnghW1xqEXT6tQJuhW7kQJ0X3SwMjS6F sdvJaiojrZutKwEUI41GQAD0hkEzhjahuOh/GsLj0Q== X-Google-Smtp-Source: ABdhPJwW7sT6H/7qQtR83cLRjCBN9pY+NWuxOVWkAXEEtg+YiexJXmisW+FBjNPyN3D4juSh+y/gprdiDFnCoF/J+XE= X-Received: by 2002:a7b:ce10:: with SMTP id m16mr5318048wmc.75.1627686329254; Fri, 30 Jul 2021 16:05:29 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> <0CB3BAA7-721D-42F0-B302-B626B26A4D32@apple.com> In-Reply-To: <0CB3BAA7-721D-42F0-B302-B626B26A4D32@apple.com> From: Yuchung Cheng Date: Fri, 30 Jul 2021 16:04:52 -0700 Message-ID: To: Vidhi Goel Cc: Mark Allman , Neal Cardwell , Extensions Content-Type: multipart/alternative; boundary="00000000000076845705c85f418f" Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 23:05:40 -0000 --00000000000076845705c85f418f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable github repo sounds good to me. On Fri, Jul 30, 2021 at 1:26 PM Vidhi Goel wrote: > Hi Mark, Yuchung, Neal, > > Given that we have some ideas / suggestion text on improving 3465, how do > you think we should proceed? > Mark, do you want to start a GitHub repo or such with some of the changes > already suggested so far and others can review and / or contribute to the > bis draft? > > Thanks, > Vidhi > > On Jul 30, 2021, at 11:04 AM, Yuchung Cheng < > ycheng=3D40google.com@dmarc.ietf.org> wrote: > > > > On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel 40apple.com@dmarc.ietf.org> wrote: > >> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>>> So, it isn't exactly a new magic number. We could wave our hands >>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>> we could come up with something that folks felt was fine. However, >>>> my feeling is that if we want to worry about bursts then let's worry >>>> about bursts in some generic way. And, if you have some way to deal >>>> with bursts then L isn't needed. And, if you don't have a way to >>>> deal with bursts then a conservative L seems fine. But, perhaps >>>> putting the effort into a generic mechanism instead of cooking yet >>>> another magic number we need to periodically refresh is probably a >>>> better way to spend effort. >>>> >>> >>> Yes, I very much agree that "putting the effort into a generic mechanis= m >>> instead of cooking yet another magic number we need to periodically ref= resh >>> is probably a better way to spend effort.=E2=80=9D >>> >>> >>> I agree that defining such a number doesn=E2=80=99t fully solve the pro= blem but >>> it gives some recommendation for implementations that don=E2=80=99t do = pacing. So, >>> defining a somewhat less restrictive value for L (5 or 10) would be a l= ast >>> resort for implementations that don=E2=80=99t pace. >>> >> How about putting a number 10, and also put all the rationales to follow >> to decide a higher or lower value. It's never one-size for all. >> >> >> That sounds great. Something on the lines of, >> >> =E2=80=9CThis document RECOMMENDS using mechanisms like Pacing to contr= ol how >> many bytes are sent to the network at a point of time. But if it is not >> possible to implement pacing, an implementation MAY implicitly pace thei= r >> traffic by applying a limit L to the increase in congestion window per A= CK >> during slow start. In modern stacks, acknowledgments are aggregated for >> various reason, CPU optimization, reducing network load etc. Hence it is >> common for a sender to receive an aggregated ACK that acknowledges more >> than 2 segments. For example, a stack that implements GRO could aggregat= e >> packets up to 64Kbytes or ~44 segments before passing on to the TCP laye= r >> and this would result in a single ACK to be generated by the TCP stack. >> Given that an initial window of 10 packets in current deployments has >> been working fine, the draft makes a recommendation to set L=3D10 during= slow >> start. This would mean that with every ACK, we are probing for a new >> capacity by sending 10 packets in addition to the previously discovered >> capacity. Implementations MAY choose to set a lower limit if they believ= e >> an increase of 10 is too aggressive." >> >> Does this sound like what we would like to say? >> > Thanks for taking a shot. I would put more description on Pacing to ensur= e > better implementation. How about: > "Pacing here refers to spread packet transmission following a rate based > on the congestion window and round trip." with a citation of > https://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2 > > > > I would also refer to IW RFC 6928 in case it gets increased / updated a > few years later. > Hmm maybe we should also move RFC6928 to the standard track :-) > > >> - >> Vidhi >> >> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng < >> ycheng=3D40google.com@dmarc.ietf.org> wrote: >> >> >> >> On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel > 40apple.com@dmarc.ietf.org> wrote: >> >>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>>> So, it isn't exactly a new magic number. We could wave our hands >>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>> we could come up with something that folks felt was fine. However, >>>> my feeling is that if we want to worry about bursts then let's worry >>>> about bursts in some generic way. And, if you have some way to deal >>>> with bursts then L isn't needed. And, if you don't have a way to >>>> deal with bursts then a conservative L seems fine. But, perhaps >>>> putting the effort into a generic mechanism instead of cooking yet >>>> another magic number we need to periodically refresh is probably a >>>> better way to spend effort. >>>> >>> >>> Yes, I very much agree that "putting the effort into a generic mechanis= m >>> instead of cooking yet another magic number we need to periodically ref= resh >>> is probably a better way to spend effort.=E2=80=9D >>> >>> >>> I agree that defining such a number doesn=E2=80=99t fully solve the pro= blem but >>> it gives some recommendation for implementations that don=E2=80=99t do = pacing. So, >>> defining a somewhat less restrictive value for L (5 or 10) would be a l= ast >>> resort for implementations that don=E2=80=99t pace. >>> >> How about putting a number 10, and also put all the rationales to follow >> to decide a higher or lower value. It's never one-size for all. >> >> Also I believe it's time to move ABC into the standards track, in the er= a >> of (bigger and bigger) stretch ACKs. >> >> >>> Thanks, >>> Vidhi >>> >>> >>> >>> On Jul 29, 2021, at 8:19 AM, Neal Cardwell wrote= : >>> >>> >>> >>> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman wrote: >>> >>>> >>>> >> (b) If there is no burst mitigation then we have to figure out >>>> >> if L is still useful for this purpose and whether we want t= o >>>> >> retain it. Seems like perhaps L=3D2 is sensible here. L w= as >>>> >> never meant to be some general burst mitigator. However, >>>> >> ABC clearly *can* aggravate bursting and so perhaps it make= s >>>> >> sense to have it also try to limit the impact of the >>>> >> aggravation (in the absence of some general mechanism). >>>> > >>>> > Even if recommending a static L value, IMHO L=3D2 is a bit >>>> > conservative. >>>> >>>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs. >>>> So, it isn't exactly a new magic number. We could wave our hands >>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>> we could come up with something that folks felt was fine. However, >>>> my feeling is that if we want to worry about bursts then let's worry >>>> about bursts in some generic way. And, if you have some way to deal >>>> with bursts then L isn't needed. And, if you don't have a way to >>>> deal with bursts then a conservative L seems fine. But, perhaps >>>> putting the effort into a generic mechanism instead of cooking yet >>>> another magic number we need to periodically refresh is probably a >>>> better way to spend effort. >>>> >>> >>> Yes, I very much agree that "putting the effort into a generic mechanis= m >>> instead of cooking yet another magic number we need to periodically ref= resh >>> is probably a better way to spend effort." >>> >>>> >>>> >> - During slow starts that follow RTOs there is a general >>>> >> problem that just because the window slides by X bytes >>>> >> doesn't say anything about the *network*, as that sliding can >>>> >> happen because much of the data was likely queued for the >>>> >> application on the receiver. So, e.g., you can RTO and send >>>> >> one packet and get an ACK back that slides the window 10 >>>> >> packets. That doesn't mean 10 packets left. It means one >>>> >> packet left the network and nine packets are eligible to be >>>> >> sent to the application. So, it is not OK to set the cwnd to >>>> >> 1+10 =3D 11 packets in response to this ACK. Here L should >>>> >> exist and be 1. >>>> > >>>> > AFAICT this argument only applies to non-SACK connections. For >>>> > connections with SACK (the vast majority of connections over the >>>> > public Internet and in datacenters), it is quite feasible to >>>> > determine how many packets really left the network (and Linux TCP >>>> > does this; see below). >>>> >>>> If you have an accurate way to figure out how many of the ACKed >>>> bytes left the network and how many were just buffered at the >>>> receiver then I see no problem with increasing based on byte count >>>> as you do in the initial slow start. >>>> >>>> (I don't remember what the paper you cite says, but my guess is it's >>>> often the case that L=3D1 is a reasonable substitute for something >>>> complicated here. But, perhaps I am running the simulation in my >>>> head wrong ... it has been a while, admittedly!) >>>> >>>> > Yes, offload mechanisms are so pervasive in practice, >>>> >>>> I am trying to build a mental model here. How pervasive would you >>>> guess these are? And, where in the network? I have assumed that >>>> they are for sure pervasive in data centers and server farms, but >>>> not for the vast majority of Internet-connected devices. >>>> >>> >>> From my impression looking at public Internet traces, aggregation >>> mechanisms that cause TCP ACKs for more than 2 segments are very common= . I >>> suspect that's because the majority of public Internet traffic these da= ys >>> has a bottleneck that is either wifi, cellular, or DOCSIS, and all of t= hese >>> have a shared medium with a large latency overhead for L2 MAC control o= f >>> gets to speak next. So a lot of batching happens, both in big batches o= f >>> data that arrive at the client in the same L2 medium time slot, and big >>> batches of ACKs that accumulate while the client waits (often several >>> milliseconds, sometimes even tens of milliseconds) for its chance to se= nd a >>> big stretch ACK or batch of ACKs. >>> >>> This brings up a related point: even if there is some ABC-style per-ACK >>> L limit on cwnd increases, the time structure of most public Internet A= CK >>> streams is massively bursty because of these aggregation mechanisms >>> inherent in L2 behavior on most public Internet bottlenecks (wifi, >>> cellular, DOCSIS). So even if there is a limit L that limits the per-AC= K >>> behavior to be smooth, if there is no pacing of data segments then the = data >>> transmit time structure will still be bursty because the ACK arrivals t= hese >>> days are very bursty. >>> >>> best regards, >>> neal >>> >>> >>> _______________________________________________ >>> tcpm mailing list >>> tcpm@ietf.org >>> https://www.ietf.org/mailman/listinfo/tcpm >> >> > --00000000000076845705c85f418f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
github repo sounds good to me.

On Fri, Jul 30, 2021 at 1:26= PM Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Hi Mark, Yuchung, Neal,

Given that we have some ideas = / suggestion text on improving 3465, how do you think we should proceed?Mark, do you want to start a GitHub repo or such with some of the changes= already suggested so far and others can review and / or contribute to the = bis draft?

Thanks,
Vidhi

On Jul 30, 2021, at 11:04 AM, Yuchung Cheng <= ;= ycheng=3D40google.com@dmarc.ietf.org> wrote:



On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel <vidhi_g= oel=3D40app= le.com@dmarc.ietf.org> wrote:
Well, perhaps.=C2=A0 L=3D2 was designed to exactly counter= act delayed ACKs.
So, it isn't exactly a new magic number.=C2=A0 We = could wave our hands
and say "5 seems OK" or "10 seems OK= " or whatever.=C2=A0 And, I am sure
we could come up with something= that folks felt was fine.=C2=A0 However,
my feeling is that if we want = to worry about bursts then let's worry
about bursts in some generic = way.=C2=A0 And, if you have some way to deal
with bursts then L isn'= t needed.=C2=A0 And, if you don't have a way to
deal with bursts the= n a conservative L seems fine.=C2=A0 But, perhaps
putting the effort int= o a generic mechanism instead of cooking yet
another magic number we nee= d to periodically refresh is probably a
better way to spend effort.
<= /blockquote>

Yes, I very much agree that "putting t= he effort into a generic mechanism instead of cooking yet another magic num= ber we need to periodically refresh is probably a better way to spend effor= t.=E2=80=9D=C2=A0

I agree that defin= ing such a number doesn=E2=80=99t fully solve the problem but it gives some= recommendation for implementations that don=E2=80=99t do pacing. So, defin= ing a somewhat less restrictive value for L (5 or 10) would be a last resor= t for implementations that don=E2=80=99t pace.
How about putting a number 10, and also put all the rationales to follow t= o decide a higher or lower value. It's never one-size for all.

That so= unds great. Something on the lines of,

=C2=A0=E2= =80=9CThis document RECOMMENDS using mechanisms like Pacing to control how = many bytes are sent to the network at a point of time. But if it is not pos= sible to implement pacing, an implementation MAY implicitly pace their traf= fic by applying a limit L to the increase in congestion window per ACK duri= ng slow start. In modern stacks, acknowledgments are aggregated for various= reason, CPU optimization, reducing network load etc. Hence it is common fo= r a sender to receive an aggregated ACK that acknowledges more than 2 segme= nts. For example, a stack that implements GRO could aggregate packets up to= 64Kbytes or ~44 segments before passing on to the TCP layer and this would= result in a single ACK to be generated by the TCP stack. Given that an ini= tial window of 10 packets=C2=A0in current deployments=C2=A0has= been working fine, the draft makes a recommendation to set L=3D10 during s= low start. This would mean that with every ACK, we are probing for a new ca= pacity by sending 10 packets in addition to the previously discovered capac= ity. Implementations MAY choose to set a lower limit if they believe an inc= rease of 10 is too aggressive."

Does this sou= nd like what we would like to say?
Thanks for=C2=A0taking a shot. I would put more description on Pacing to= =C2=A0ensure better implementation. How about:
"Pacing here = refers to spread packet transmission following a rate based on the congesti= on window and round trip." with a citation of=C2=A0http= s://datatracker.ietf.org/doc/html/rfc7661#section-4.4.2

<= /div>


I would also refer to IW RFC 6928= =C2=A0in case it gets increased / updated a few years later.
Hmm = maybe we should also move RFC6928=C2=A0to the standard track :-)
=

-
Vidhi

On Jul 29, 2021, at 1:47 PM, Yuch= ung Cheng <ycheng=3D40google.com@dmarc.ietf.org> wrote:

<= div>

On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goe= l <vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:
Wel= l, perhaps.=C2=A0 L=3D2 was designed to exactly counteract delayed ACKs.So, it isn't exactly a new magic number.=C2=A0 We could wave our hands=
and say "5 seems OK" or "10 seems OK" or whatever.= =C2=A0 And, I am sure
we could come up with something that folks felt wa= s fine.=C2=A0 However,
my feeling is that if we want to worry about burs= ts then let's worry
about bursts in some generic way.=C2=A0 And, if = you have some way to deal
with bursts then L isn't needed.=C2=A0 And= , if you don't have a way to
deal with bursts then a conservative L = seems fine.=C2=A0 But, perhaps
putting the effort into a generic mechani= sm instead of cooking yet
another magic number we need to periodically r= efresh is probably a
better way to spend effort.
Yes, I very much agree that "putting the effort into a ge= neric mechanism instead of cooking yet another magic number we need to peri= odically refresh is probably a better way to spend effort.=E2=80=9D=C2=A0

I agree that defining such a number d= oesn=E2=80=99t fully solve the problem but it gives some recommendation for= implementations that don=E2=80=99t do pacing. So, defining a somewhat less= restrictive value for L (5 or 10) would be a last resort for implementatio= ns that don=E2=80=99t pace.
How about putting = a number 10, and also put all the rationales to follow to decide a higher o= r lower value. It's never one-size for all.

Al= so I believe it's time to move ABC into the standards track, in the era= of (bigger and bigger) stretch ACKs.


Thanks,
Vidhi



On J= ul 29, 2021, at 8:19 AM, Neal Cardwell <ncardwell@google.com> wrote:



On Thu, Jul 29, 2021 at 10:06 AM Mark Allman= <mallman@icir.org= > wrote:
=
>>=C2=A0 =C2=A0 =C2=A0(b) If there is no burst mitigation then we= have to figure out
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if L is st= ill useful for this purpose and whether we want to
>>=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0retain it.=C2=A0 Seems like perhaps L=3D2 is sensible = here.=C2=A0 L was
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0never meant = to be some general burst mitigator.=C2=A0 However,
>>=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0ABC clearly *can* aggravate bursting and so perhaps it= makes
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0sense to have it also t= ry to limit the impact of the
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= aggravation (in the absence of some general mechanism).
>
> Eve= n if recommending a static L value, IMHO L=3D2 is a bit
> conservativ= e.

Well, perhaps.=C2=A0 L=3D2 was designed to exactly counteract del= ayed ACKs.
So, it isn't exactly a new magic number.=C2=A0 We could w= ave our hands
and say "5 seems OK" or "10 seems OK" = or whatever.=C2=A0 And, I am sure
we could come up with something that f= olks felt was fine.=C2=A0 However,
my feeling is that if we want to worr= y about bursts then let's worry
about bursts in some generic way.=C2= =A0 And, if you have some way to deal
with bursts then L isn't neede= d.=C2=A0 And, if you don't have a way to
deal with bursts then a con= servative L seems fine.=C2=A0 But, perhaps
putting the effort into a gen= eric mechanism instead of cooking yet
another magic number we need to pe= riodically refresh is probably a
better way to spend effort.

Yes, I very much agree that "putting the effo= rt into a generic mechanism instead of cooking yet another magic number we = need to periodically refresh is probably a better way to spend effort."= ;=C2=A0

>>= =C2=A0 =C2=A0- During slow starts that follow RTOs there is a general
&g= t;>=C2=A0 =C2=A0 =C2=A0problem that just because the window slides by X = bytes
>>=C2=A0 =C2=A0 =C2=A0doesn't say anything about the *ne= twork*, as that sliding can
>>=C2=A0 =C2=A0 =C2=A0happen because m= uch of the data was likely queued for the
>>=C2=A0 =C2=A0 =C2=A0ap= plication on the receiver.=C2=A0 So, e.g., you can RTO and send
>>= =C2=A0 =C2=A0 =C2=A0one packet and get an ACK back that slides the window 1= 0
>>=C2=A0 =C2=A0 =C2=A0packets.=C2=A0 That doesn't mean 10 pa= ckets left.=C2=A0 It means one
>>=C2=A0 =C2=A0 =C2=A0packet left t= he network and nine packets are eligible to be
>>=C2=A0 =C2=A0 =C2= =A0sent to the application.=C2=A0 So, it is not OK to set the cwnd to
&g= t;>=C2=A0 =C2=A0 =C2=A01+10 =3D 11 packets in response to this ACK.=C2= =A0 Here L should
>>=C2=A0 =C2=A0 =C2=A0exist and be 1.
>> AFAICT this argument only applies to non-SACK connections. For
>= ; connections with SACK (the vast majority of connections over the
> = public Internet and in datacenters), it is quite feasible to
> determ= ine how many packets really left the network (and Linux TCP
> does th= is; see below).

If you have an accurate way to figure out how many o= f the ACKed
bytes left the network and how many were just buffered at th= e
receiver then I see no problem with increasing based on byte count
= as you do in the initial slow start.

(I don't remember what the = paper you cite says, but my guess is it's
often the case that L=3D1 = is a reasonable substitute for something
complicated here.=C2=A0 But, pe= rhaps I am running the simulation in my
head wrong ... it has been a whi= le, admittedly!)

> Yes, offload mechanisms are so pervasive in pr= actice,

I am trying to build a mental model here.=C2=A0 How pervasiv= e would you
guess these are?=C2=A0 And, where in the network?=C2=A0 I ha= ve assumed that
they are for sure pervasive in data centers and server f= arms, but
not for the vast majority of Internet-connected devices.

From my impression looking at public Internet traces, aggreg= ation mechanisms that cause TCP ACKs for more than 2 segments are very comm= on. I suspect that's because the majority of public Internet traffic th= ese days has a bottleneck that is either wifi, cellular, or DOCSIS, and all= of these have a shared medium with a large latency overhead for L2 MAC con= trol of gets to speak next. So a lot of batching happens, both in big batch= es of data that arrive at the client in the same L2 medium time slot, and b= ig batches of ACKs that accumulate while the client waits (often several mi= lliseconds, sometimes even tens of milliseconds) for its chance to send a b= ig stretch ACK or batch of ACKs.

This brings up a= related point: even if there is some ABC-style per-ACK L limit on cwnd inc= reases, the time structure of most public Internet ACK streams is massively= bursty because of these aggregation mechanisms inherent in L2 behavior on = most public Internet bottlenecks (wifi, cellular, DOCSIS). So even if there= is a limit L that limits the per-ACK behavior to be smooth, if there is no= pacing of data segments then the data transmit time structure will still b= e bursty because the ACK arrivals these days are very bursty.=C2=A0

best regards,
neal

________________________= _______________________
tcpm mailing list
tcpm@ietf.org
https://www.i= etf.org/mailman/listinfo/tcpm
--00000000000076845705c85f418f-- From nobody Fri Jul 30 16:07:46 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BF7703A1543 for ; Fri, 30 Jul 2021 16:07:44 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -18.098 X-Spam-Level: X-Spam-Status: No, score=-18.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.499, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DxoSWpVffjGf for ; Fri, 30 Jul 2021 16:07:39 -0700 (PDT) Received: from mail-ua1-x931.google.com (mail-ua1-x931.google.com [IPv6:2607:f8b0:4864:20::931]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 626923A153A for ; Fri, 30 Jul 2021 16:07:39 -0700 (PDT) Received: by mail-ua1-x931.google.com with SMTP id t26so4583161uao.12 for ; Fri, 30 Jul 2021 16:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=UmW+MHp0SVlEPT72lHxJ8X5OY7rgGAPROdRgspVcQ0k=; b=wSs9D4RKDAKt2f20g7aEh45b5L7wFO/YXp9lzjFpNZBdmFfVfaxxJXEaZqBgBdnYwC hAYVScMcyyr1XdJlJzMMyEgGwqPE3WCoERCVkif3MZO7l1BKa2BIHS81e4BZizybXJ5D GeO5jUvJdDNN851DcUqtu3W5AxYsIo6q7p54rE2l2fJRixLdom9W0zxJZZbN8IK64EBu KaqmKlBrdYGl0lmbsO96721fIJFp+Vcq9VGlLSPblikiyfyafs7d9EqBRYpS7Ycwle1F CHwwZu/yrDqct+1qT7HTo4qT9gq7RNMXcHwUKt+W5jSoeJ66ypB/8T0GhuwgvzHEg5A2 MptQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=UmW+MHp0SVlEPT72lHxJ8X5OY7rgGAPROdRgspVcQ0k=; b=sjuC2agOM/SMVQWB2Ff56vM7d5Tzbzf65bwM9iIF/ifCTiK1J3vWyN70Quad8mj1CQ kGCa/7X3gm3h2KRWw7KKuSdyd8JE49Fwsv//VcTmPQ7nWp8iyKXJL3NPS8ZvTIz8nKR2 9Tnde4hb/D9PwR8YH4xnjSWX2UvrPP4TlgERoQy75JVoDHAUxKotlSl261xcFOmmYcbb sgMgUQ0NelYqLGuK2OVudvDD7yFLh+gctO8GylwhCPNKJdnG/kbQ4P3f0rbAmiQR8uv+ qwfTjXC3Oha4loaBKRLSJGW0lMDbmXJAwY6dEAuIrNeFO4aLaTtyNBPPbzsoXl3cjXJP MElg== X-Gm-Message-State: AOAM530A4BhP/+2nEbcHlnawipvdzMvG2dAEKi2VEIvN7NTlJdErgKRF Br5mdA0ueecDtxj6XPdFn4OQcRlZ/10UkWzKrJngLw== X-Google-Smtp-Source: ABdhPJySKHGG6JJbv9kJy3gwxKVMwdmP0mZWLdWxS45gL5r53p9iBiIDs7hy6jBOCwrvcJ8Om9Y5aj1sIrBR9eHVadw= X-Received: by 2002:ab0:42d:: with SMTP id 42mr5041430uav.63.1627686455969; Fri, 30 Jul 2021 16:07:35 -0700 (PDT) MIME-Version: 1.0 References: <78EF3761-7CAF-459E-A4C0-57CDEAFEA8EE@apple.com> <601D9D4F-A82C-475A-98CC-383C1F876C44@apple.com> <54699CC9-C8F5-4CA3-8815-F7A21AE10429@icsi.berkeley.edu> <11FE4818-87E7-4FD8-8F45-E19CD9A3366A@apple.com> <0CB3BAA7-721D-42F0-B302-B626B26A4D32@apple.com> In-Reply-To: From: Neal Cardwell Date: Fri, 30 Jul 2021 19:07:19 -0400 Message-ID: To: Yuchung Cheng Cc: Vidhi Goel , Mark Allman , Extensions Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Archived-At: Subject: Re: [tcpm] =?utf-8?q?Linux_doesn=E2=80=99t_implement_RFC3465?= X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Jul 2021 23:07:45 -0000 A github repo sounds good to me as well. thanks, neal On Fri, Jul 30, 2021 at 7:05 PM Yuchung Cheng wrote: > > github repo sounds good to me. > > On Fri, Jul 30, 2021 at 1:26 PM Vidhi Goel wrote: >> >> Hi Mark, Yuchung, Neal, >> >> Given that we have some ideas / suggestion text on improving 3465, how d= o you think we should proceed? >> Mark, do you want to start a GitHub repo or such with some of the change= s already suggested so far and others can review and / or contribute to the= bis draft? >> >> Thanks, >> Vidhi >> >> On Jul 30, 2021, at 11:04 AM, Yuchung Cheng wrote: >> >> >> >> On Thu, Jul 29, 2021 at 6:03 PM Vidhi Goel wrote: >>>>> >>>>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs= . >>>>> So, it isn't exactly a new magic number. We could wave our hands >>>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>>> we could come up with something that folks felt was fine. However, >>>>> my feeling is that if we want to worry about bursts then let's worry >>>>> about bursts in some generic way. And, if you have some way to deal >>>>> with bursts then L isn't needed. And, if you don't have a way to >>>>> deal with bursts then a conservative L seems fine. But, perhaps >>>>> putting the effort into a generic mechanism instead of cooking yet >>>>> another magic number we need to periodically refresh is probably a >>>>> better way to spend effort. >>>> >>>> >>>> Yes, I very much agree that "putting the effort into a generic mechani= sm instead of cooking yet another magic number we need to periodically refr= esh is probably a better way to spend effort.=E2=80=9D >>>> >>>> >>>> I agree that defining such a number doesn=E2=80=99t fully solve the pr= oblem but it gives some recommendation for implementations that don=E2=80= =99t do pacing. So, defining a somewhat less restrictive value for L (5 or = 10) would be a last resort for implementations that don=E2=80=99t pace. >>> >>> How about putting a number 10, and also put all the rationales to follo= w to decide a higher or lower value. It's never one-size for all. >>> >>> >>> That sounds great. Something on the lines of, >>> >>> =E2=80=9CThis document RECOMMENDS using mechanisms like Pacing to cont= rol how many bytes are sent to the network at a point of time. But if it is= not possible to implement pacing, an implementation MAY implicitly pace th= eir traffic by applying a limit L to the increase in congestion window per = ACK during slow start. In modern stacks, acknowledgments are aggregated for= various reason, CPU optimization, reducing network load etc. Hence it is c= ommon for a sender to receive an aggregated ACK that acknowledges more than= 2 segments. For example, a stack that implements GRO could aggregate packe= ts up to 64Kbytes or ~44 segments before passing on to the TCP layer and th= is would result in a single ACK to be generated by the TCP stack. Given tha= t an initial window of 10 packets in current deployments has been working f= ine, the draft makes a recommendation to set L=3D10 during slow start. This= would mean that with every ACK, we are probing for a new capacity by sendi= ng 10 packets in addition to the previously discovered capacity. Implementa= tions MAY choose to set a lower limit if they believe an increase of 10 is = too aggressive." >>> >>> Does this sound like what we would like to say? >> >> Thanks for taking a shot. I would put more description on Pacing to ensu= re better implementation. How about: >> "Pacing here refers to spread packet transmission following a rate based= on the congestion window and round trip." with a citation of https://datat= racker.ietf.org/doc/html/rfc7661#section-4.4.2 >> >> >> >> I would also refer to IW RFC 6928 in case it gets increased / updated a = few years later. >> Hmm maybe we should also move RFC6928 to the standard track :-) >> >>> >>> - >>> Vidhi >>> >>> On Jul 29, 2021, at 1:47 PM, Yuchung Cheng wrote: >>> >>> >>> >>> On Thu, Jul 29, 2021 at 1:19 PM Vidhi Goel wrote: >>>>> >>>>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs= . >>>>> So, it isn't exactly a new magic number. We could wave our hands >>>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>>> we could come up with something that folks felt was fine. However, >>>>> my feeling is that if we want to worry about bursts then let's worry >>>>> about bursts in some generic way. And, if you have some way to deal >>>>> with bursts then L isn't needed. And, if you don't have a way to >>>>> deal with bursts then a conservative L seems fine. But, perhaps >>>>> putting the effort into a generic mechanism instead of cooking yet >>>>> another magic number we need to periodically refresh is probably a >>>>> better way to spend effort. >>>> >>>> >>>> Yes, I very much agree that "putting the effort into a generic mechani= sm instead of cooking yet another magic number we need to periodically refr= esh is probably a better way to spend effort.=E2=80=9D >>>> >>>> >>>> I agree that defining such a number doesn=E2=80=99t fully solve the pr= oblem but it gives some recommendation for implementations that don=E2=80= =99t do pacing. So, defining a somewhat less restrictive value for L (5 or = 10) would be a last resort for implementations that don=E2=80=99t pace. >>> >>> How about putting a number 10, and also put all the rationales to follo= w to decide a higher or lower value. It's never one-size for all. >>> >>> Also I believe it's time to move ABC into the standards track, in the e= ra of (bigger and bigger) stretch ACKs. >>> >>>> >>>> Thanks, >>>> Vidhi >>>> >>>> >>>> >>>> On Jul 29, 2021, at 8:19 AM, Neal Cardwell wrot= e: >>>> >>>> >>>> >>>> On Thu, Jul 29, 2021 at 10:06 AM Mark Allman wrote: >>>>> >>>>> >>>>> >> (b) If there is no burst mitigation then we have to figure out >>>>> >> if L is still useful for this purpose and whether we want = to >>>>> >> retain it. Seems like perhaps L=3D2 is sensible here. L = was >>>>> >> never meant to be some general burst mitigator. However, >>>>> >> ABC clearly *can* aggravate bursting and so perhaps it mak= es >>>>> >> sense to have it also try to limit the impact of the >>>>> >> aggravation (in the absence of some general mechanism). >>>>> > >>>>> > Even if recommending a static L value, IMHO L=3D2 is a bit >>>>> > conservative. >>>>> >>>>> Well, perhaps. L=3D2 was designed to exactly counteract delayed ACKs= . >>>>> So, it isn't exactly a new magic number. We could wave our hands >>>>> and say "5 seems OK" or "10 seems OK" or whatever. And, I am sure >>>>> we could come up with something that folks felt was fine. However, >>>>> my feeling is that if we want to worry about bursts then let's worry >>>>> about bursts in some generic way. And, if you have some way to deal >>>>> with bursts then L isn't needed. And, if you don't have a way to >>>>> deal with bursts then a conservative L seems fine. But, perhaps >>>>> putting the effort into a generic mechanism instead of cooking yet >>>>> another magic number we need to periodically refresh is probably a >>>>> better way to spend effort. >>>> >>>> >>>> Yes, I very much agree that "putting the effort into a generic mechani= sm instead of cooking yet another magic number we need to periodically refr= esh is probably a better way to spend effort." >>>>> >>>>> >>>>> >> - During slow starts that follow RTOs there is a general >>>>> >> problem that just because the window slides by X bytes >>>>> >> doesn't say anything about the *network*, as that sliding can >>>>> >> happen because much of the data was likely queued for the >>>>> >> application on the receiver. So, e.g., you can RTO and send >>>>> >> one packet and get an ACK back that slides the window 10 >>>>> >> packets. That doesn't mean 10 packets left. It means one >>>>> >> packet left the network and nine packets are eligible to be >>>>> >> sent to the application. So, it is not OK to set the cwnd to >>>>> >> 1+10 =3D 11 packets in response to this ACK. Here L should >>>>> >> exist and be 1. >>>>> > >>>>> > AFAICT this argument only applies to non-SACK connections. For >>>>> > connections with SACK (the vast majority of connections over the >>>>> > public Internet and in datacenters), it is quite feasible to >>>>> > determine how many packets really left the network (and Linux TCP >>>>> > does this; see below). >>>>> >>>>> If you have an accurate way to figure out how many of the ACKed >>>>> bytes left the network and how many were just buffered at the >>>>> receiver then I see no problem with increasing based on byte count >>>>> as you do in the initial slow start. >>>>> >>>>> (I don't remember what the paper you cite says, but my guess is it's >>>>> often the case that L=3D1 is a reasonable substitute for something >>>>> complicated here. But, perhaps I am running the simulation in my >>>>> head wrong ... it has been a while, admittedly!) >>>>> >>>>> > Yes, offload mechanisms are so pervasive in practice, >>>>> >>>>> I am trying to build a mental model here. How pervasive would you >>>>> guess these are? And, where in the network? I have assumed that >>>>> they are for sure pervasive in data centers and server farms, but >>>>> not for the vast majority of Internet-connected devices. >>>> >>>> >>>> From my impression looking at public Internet traces, aggregation mech= anisms that cause TCP ACKs for more than 2 segments are very common. I susp= ect that's because the majority of public Internet traffic these days has a= bottleneck that is either wifi, cellular, or DOCSIS, and all of these have= a shared medium with a large latency overhead for L2 MAC control of gets t= o speak next. So a lot of batching happens, both in big batches of data tha= t arrive at the client in the same L2 medium time slot, and big batches of = ACKs that accumulate while the client waits (often several milliseconds, so= metimes even tens of milliseconds) for its chance to send a big stretch ACK= or batch of ACKs. >>>> >>>> This brings up a related point: even if there is some ABC-style per-AC= K L limit on cwnd increases, the time structure of most public Internet ACK= streams is massively bursty because of these aggregation mechanisms inhere= nt in L2 behavior on most public Internet bottlenecks (wifi, cellular, DOCS= IS). So even if there is a limit L that limits the per-ACK behavior to be s= mooth, if there is no pacing of data segments then the data transmit time s= tructure will still be bursty because the ACK arrivals these days are very = bursty. >>>> >>>> best regards, >>>> neal >>>> >>>> >>>> _______________________________________________ >>>> tcpm mailing list >>>> tcpm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/tcpm >> >> From nobody Sat Jul 31 00:02:46 2021 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3C2BB3A19C5; Sat, 31 Jul 2021 00:02:44 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zd5BDZIpDdpn; Sat, 31 Jul 2021 00:02:39 -0700 (PDT) Received: from mail-ua1-x932.google.com (mail-ua1-x932.google.com [IPv6:2607:f8b0:4864:20::932]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2F1CB3A19C3; Sat, 31 Jul 2021 00:02:38 -0700 (PDT) Received: by mail-ua1-x932.google.com with SMTP id o10so4939848uaj.0; Sat, 31 Jul 2021 00:02:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iVSWNz8rE4obHrSsWV0bl1O6Ab9h9N8b5xmLWd65BqQ=; b=awXBt0o5dSNx57jarNZGSTDtTegwl7UNgrD7jvomyhiSleNqj65pgRASJA6iPtpJVa D/dLikB6me8SjfaMuq/5yXn8rOPBienKczqSC35VUs4QCt8fQBq8K3H3xYHfxhH3GlRo M97N5GG3FqHbCH3jKeJOgA/jweKXldUz31gVTs/LiPWV+ZRlKhprtBv+hTVM5HWWl2oX D1cx8ClnKRRhiomJCbWj9OICnk0ltObM9dEgYKPo9YXnpTtqYqz/3kXXnPxSn7FI3PJ4 wtnnSHPrCA8eF8GJjWNUhdSEQh3dieXUfKGRivMJY2keed7Metpt9nqmmH4Ot6JRJXp4 8juQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iVSWNz8rE4obHrSsWV0bl1O6Ab9h9N8b5xmLWd65BqQ=; b=ZZnL1EktYVmFHwu3wg5ooHVWMaIcwZOpphjcIHw7bkbFvBOrtlqQ8FO1azV3ftIRnS CE1IBsDgMLqsYr15CLJYffRDP80Xa5c5RjvZm6w5fgM3f2+LlAyfic80jKYF5YqgWWNI DEhQ+Zbdpb/2A8lcd07c2e18BCtSRRxWPyiZY/agwVmEBPgMnULEyfs1pm0Ig1Epfh+y hSd3SE4T6+zMtDDTyavAWyV0zLYEMOLcHoEPNyGGGuA+TsocbUlapPOodsQcavK8pJsV DY3nvSsDY78emCYoVS4Sq+1s5tlrttFr8Swz0qX2xBm/Qfziy5G4SbjE8KEY2owOyEP8 T5PA== X-Gm-Message-State: AOAM531QB9UmbEOqH4b17uocMYf/pgBkkELox/Tbqf5dKKBtcHFHBn07 N480PtBIwkN+3Y4cm6FexT6jSIB9nx5m3/T86bBs2DHzY3d7MA== X-Google-Smtp-Source: ABdhPJzfQ7Ez6lSPYInm0jIzY30JVZ/5IXpiBtwiackK5EfkS+isgiabdU9GzJ3HdZ6a734qK2/i0J90pvZaCZ+Glqw= X-Received: by 2002:ab0:74cc:: with SMTP id f12mr5207936uaq.28.1627714957125; Sat, 31 Jul 2021 00:02:37 -0700 (PDT) MIME-Version: 1.0 References: <162610476442.30543.4667406094304409800@ietfa.amsl.com> <98289918-67d1-2be1-723d-2df66be46fac@bobbriscoe.net> <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> In-Reply-To: <84ac0f00-b828-2503-fd7c-0ef7c6465768@bobbriscoe.net> From: Junho Choi Date: Sat, 31 Jul 2021 00:02:00 -0700 Message-ID: To: Bob Briscoe Cc: Praveen Balasubramanian , "tcpm@ietf.org" , "draft-ietf-tcpm-hystartplusplus@ietf.org" Content-Type: multipart/alternative; boundary="000000000000d0aa6f05c865eba9" Archived-At: Subject: Re: [tcpm] Hystart survey of large server operators X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Jul 2021 07:02:44 -0000 --000000000000d0aa6f05c865eba9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Currently Cloudflare's QUIC library (quiche) and its production QUIC traffic uses CUBIC (rfc8312bis-02) and HyStart++ (draft02) by default. (https://blog.cloudflare.com/cubic-and-hystart-support-in-quiche/). On Tue, Jul 27, 2021 at 4:38 PM Bob Briscoe wrote: > Any large server operator out there who are using Cubic, > If you're willing to state whether or not your operations disable Hystart= , > pls do so in reply. > Then Praveen can cite this mailing list thread in the Hystart++ draft, as > requested below. > > If you are willing to reply privately, I would be willing to keep your > confidences, and provide an anonymized result to the list. > > Cheers > > > > Bob > > On 27/07/2021 01:50, Praveen Balasubramanian wrote: > > Although Hystart is default enabled in Linux, it is invariably disabled. = So, it's misleading to just say Hystart is default enabled, which implies i= t's widely used, when people clearly find it has problems (which motivates = Hystart++). I found this out through an informal survey I did at the Mar'18= IETF in London by asking round the implementers of the most prevalent stac= ks (I would name names if I could find the note I later sent to someone or = to some list, but I can't find it - sry). > > I'd like some citations on this versus anecdata if possible before I add = that caveat to the text. Do large deployments disable this? I haven't come = across this suggestion in any Linux tuning guides to date. > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --=20 Junho Choi | https://saturnsoft.net --000000000000d0aa6f05c865eba9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Currently Cloudflare's QUIC library (quiche) and = its production QUIC traffic uses CUBIC (rfc8312bis-02) and HyStart++ (draft= 02) by default.

On Tue, Jul 27, 2021 at 4:38 PM Bob Brisc= oe <ietf@bobbriscoe.net> w= rote:
=20 =20 =20
Any large server operator out there who are using Cubic,
If you're willing to state whether or not your operations disable Hystart, pls do so in reply.
Then Praveen can cite this mailing list thread in the Hystart++ draft, as requested below.

If you are willing to reply privately, I would be willing to keep your confidences, and provide an anonymized result to the list.

Cheers



Bob

On 27/07/2021 01:50, Praveen Balasubramanian wrote:
Although Hystart is default enabled in Linux, it is invariably d=
isabled. So, it's misleading to just say Hystart is default enabled, wh=
ich implies it's widely used, when people clearly find it has problems =
(which motivates Hystart++). I found this out through an informal survey I =
did at the Mar'18 IETF in London by asking round the implementers of th=
e most prevalent stacks (I would name names if I could find the note I late=
r sent to someone or to some list, but I can't find it - sry).

I'd like some citations on this versus anecdata if p=
ossible before I add that caveat to the text. Do large deployments disable =
this? I haven't come across this suggestion in any Linux tuning guides =
to date.=20


--=20
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


--
Junho Choi <junho= dot choi at gmail.com&g= t; | https://saturnsof= t.net
--000000000000d0aa6f05c865eba9--