From nobody Sun Jan 5 13:23:52 2020 Return-Path: X-Original-To: loops@ietfa.amsl.com Delivered-To: loops@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 486F2120043; Sun, 5 Jan 2020 13:23:41 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Z53JTyJKVLD; Sun, 5 Jan 2020 13:23:36 -0800 (PST) Received: from gabriel-vm-2.zfn.uni-bremen.de (gabriel-vm-2.zfn.uni-bremen.de [134.102.50.17]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 51C60120020; Sun, 5 Jan 2020 13:23:36 -0800 (PST) Received: from [172.16.42.112] (p548DC893.dip0.t-ipconnect.de [84.141.200.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gabriel-vm-2.zfn.uni-bremen.de (Postfix) with ESMTPSA id 47rWpZ35nJz16kF; Sun, 5 Jan 2020 22:23:34 +0100 (CET) Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Content-Type: text/plain; charset=utf-8 From: Carsten Bormann X-Priority: 3 (Normal) In-Reply-To: <24526f0dbdcbdb65951d16cb5770fcbe.squirrel@www.critical.com> Date: Sun, 5 Jan 2020 22:23:33 +0100 Cc: loops@ietf.org, nwcrg@irtf.org, dtn@ietf.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <3D026CEF-AF7D-4CDE-8A5B-8CE5DCF749D1@tzi.org> <163A8040-9B13-43AA-8917-F75D984B0EBF@tzi.org> <26708398-5DEB-478E-8CB8-D406A8B874FF@tzi.org> <7DC0B005-8BA2-4D80-8CDF-4E2AE7EA96F3@tzi.org> <24526f0dbdcbdb65951d16cb5770fcbe.squirrel@www.critical.com> To: Stuart William Card X-Mailer: Apple Mail (2.3608.40.2.2.4) Archived-At: Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room X-BeenThere: loops@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Local Optimizations on Path Segments List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jan 2020 21:23:41 -0000 Hi Stu, I think there are a number of people on the list who have a similar = background with TCP acceleration, where the prominence of the SCPS-TP = protocol varies. (For me, SCPS-TP was mainly an interesting = specification to look at and use as a quarry and source of ideas for our = own, more optimized protocols.) Traditional PEPs were as good as they = were because: (1) they had more intricate (and timely) knowledge of the critical path = segment than the end-to-end protocol could have, (2) techniques such as TCP splitting allowed mitigating some (but not = all) of the latency induced performance issues end-to-end protocols = exhibited over high-latency links. With increasing use of encryption inside the transport layer, (2) is = going away (and will probably only come back when encrypted transports = such as QUIC start talking to halfway-houses in the network, without = giving up the other benefits of encryption). This reduces the benefit = from the introduction of SCPS-TP segments into a sequence of split TCP = segments. (1) will stay relevant for those industries where that knowledge can be = provided (satellite links, vehicular links, maybe more general wireless = links). What LOOPS is trying to do is providing a protocol that can help even in = the absence of both (1) and (2). One useful aspect here may be the = aggregation of traffic, which is quite effective against tail loss = events (and generally can provide a more reasonable basis for = measurements), but also may make it less interesting to interact = specifically with each end-to-end transport flow. Fixing path segments = outside =E2=80=9Cthe critical=E2=80=9D segment (e.g., a WiFi segment = before going wide-area) can also help against mutual exacerbation of = their problems. So LOOPS is not trying to replace PEPs that benefit from (2) (and still = do from (1), today). It could still be run profitably over critical = path segments where a forklift upgrade of PEP infrastructure is not = feasible right away. It is just one more tool in the toolkit that can = be used to assemble a useful end-to-end path for some networking = requirements, in particular where there already is a good reason to run = a tunnel. Probably later than sooner, it might also help with running = such tunnels over multiple paths, but we are not proposing that the = LOOPS activity should be actively looking into that just yet (even = though some of us have been doing this in their day jobs for a decade or = so now). The fact that recovering packets can lead to reordering makes life = harder for a LOOPS-like protocol. As we saw in Singapore, we have quite = different views on how much LOOPS could benefit from a general trend = towards more reordering-robust end-to-end protocols. LOOPS also could = itself be made robust against reordering within its path segment =E2=80=94= this needn=E2=80=99t involve (partial) resequencing at the LOOPS = egress, where we again have different views on how useful that could be. So yes, thanks for reminding us that SCPS-TP is good background material = for optimizing path segments, but you are also right that we have to be = aware of the limitations to its applicability in the LOOPS space. I=E2=80=99m certainly interested in looking at more robust end-to-end = protocols as well (including multipoint =E2=80=94 reopening the = discussion of NORM with a view to 2020=E2=80=99s networks and = applications is very interesting!), and there already is a lot of = attention to how QUIC could evolve to be more synergistic with efforts = spent inside the network. Ultimately, the applications will need to be = part of the picture, evolving the transport abstractions (just as DTN = did in a nicely radical way). So LOOPS is just a start. Much of what = could be done beyond LOOPS is research - maybe it is also worth thinking = about coordinating some of that, by posing the right questions. E.g., = for reliable multicast, we had the idea of using router assist, which = from today=E2=80=99s perspective also would benefit from COINRG-like = computing in the network. Please do keep us informed about interesting developments in the greater = design space here.=20 As a proposed IETF WG, LOOPS itself will need to focus on a more narrow = set of problems and opportunities, and this is the time to discuss = whether we got that narrowing right in the current charter proposal [1]. Gr=C3=BC=C3=9Fe, Carsten [1]: https://github.com/loops-wg/charter/tree/post-105 From nobody Sun Jan 5 17:46:39 2020 Return-Path: X-Original-To: loops@ietfa.amsl.com Delivered-To: loops@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F22E2120058; Sun, 5 Jan 2020 17:46:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.9 X-Spam-Level: X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id biPEcu59lh0E; Sun, 5 Jan 2020 17:46:26 -0800 (PST) Received: from mail.critical.com (mail.critical.com [209.217.211.166]) by ietfa.amsl.com (Postfix) with ESMTP id 91B8C120045; Sun, 5 Jan 2020 17:46:26 -0800 (PST) Received: by mail.critical.com (Postfix, from userid 99) id DC75F3801D; Sun, 5 Jan 2020 20:32:40 -0500 (EST) Received: from www.critical.com (unknown [192.168.6.38]) by mail.critical.com (Postfix) with ESMTP id 9240F1854D; Sun, 5 Jan 2020 20:32:38 -0500 (EST) Received: from 74.78.200.230 (SquirrelMail authenticated user cardsw) by www.critical.com with HTTP; Sun, 5 Jan 2020 20:46:22 -0500 Message-ID: <266cb00cb46ed971f3055a086b421312.squirrel@www.critical.com> In-Reply-To: References: <3D026CEF-AF7D-4CDE-8A5B-8CE5DCF749D1@tzi.org> <163A8040-9B13-43AA-8917-F75D984B0EBF@tzi.org> <26708398-5DEB-478E-8CB8-D406A8B874FF@tzi.org> <7DC0B005-8BA2-4D80-8CDF-4E2AE7EA96F3@tzi.org> <24526f0dbdcbdb65951d16cb5770fcbe.squirrel@www.critical.com> Date: Sun, 5 Jan 2020 20:46:22 -0500 From: "Stuart William Card" To: "Carsten Bormann" Cc: loops@ietf.org, nwcrg@irtf.org, dtn@ietf.org User-Agent: SquirrelMail/1.4.23 [SVN] MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Archived-At: Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room X-BeenThere: loops@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Local Optimizations on Path Segments List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2020 01:46:29 -0000 Carsten -- Thanks for your thoughtful response. I'll try to focus my inputs. :-) > ... > (1) ... more intricate (and timely) knowledge of the critical path segment... > (2) techniques such as TCP splitting allowed mitigating some... > of the latency induced performance issues end-to-end protocols exhibited... > With increasing use of encryption... (2) is going away... > (1) will stay relevant... where that knowledge can be provided... > What LOOPS is trying to do is providing a protocol that can help even in > the absence of both (1) and (2)... > recovering packets can lead to reordering... > applications will need to be part of the picture... > LOOPS itself will need to focus on a more narrow > set of problems and opportunities, and this is the time to discuss whether > we got that narrowing right in the current charter proposal... Agreed. Further, not only transport but also network layer encryption kills simple approaches to (2), leaving AFAIK only multipath or trading increased mean latency to reduce latency variance. Many of us have suffered from suboptimal and sometimes unanticipated interactions between link and transport layer behaviors (retransmission in both, flow control in the data link, congestion control in the transport). Application layer adaptive behaviors (e.g. video streaming that switches dynamically during a session among different source coding rates based upon end to end performance as it varies) further complicate this. LOOPS will be yet another interacting control loop. In the worst case, a non-congestion loss of a link layer frame could lead to: - data link retransmission - LOOPS retransmission - transport retransmission - transport backoff - application rate downshift All at once! Presumably we can't prevent this altogether but want to minimize its occurrence. The charter looks good. I suggest broadening its scope of consideration of cross-layer interactions: not just w/transport congestion control, but also w/retransmission, misordering & resequencing at all layers. -- Stuart W. Card, VP & Chief Scientist Critical Technologies Inc. (CTI) From nobody Mon Jan 6 00:47:32 2020 Return-Path: X-Original-To: loops@ietfa.amsl.com Delivered-To: loops@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AC7021200B3; Mon, 6 Jan 2020 00:47:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.2 X-Spam-Level: X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ohYstDNVA5pC; Mon, 6 Jan 2020 00:47:21 -0800 (PST) Received: from huawei.com (lhrrgout.huawei.com [185.176.76.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8120C12001E; Mon, 6 Jan 2020 00:47:21 -0800 (PST) Received: from lhreml703-cah.china.huawei.com (unknown [172.18.7.107]) by Forcepoint Email with ESMTP id 21A39AEEE36543686AC9; Mon, 6 Jan 2020 08:47:20 +0000 (GMT) Received: from dggeme755-chm.china.huawei.com (10.3.19.101) by lhreml703-cah.china.huawei.com (10.201.108.44) with Microsoft SMTP Server (TLS) id 14.3.408.0; Mon, 6 Jan 2020 08:47:19 +0000 Received: from dggeme758-chm.china.huawei.com (10.3.19.104) by dggeme755-chm.china.huawei.com (10.3.19.101) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1713.5; Mon, 6 Jan 2020 16:47:17 +0800 Received: from dggeme758-chm.china.huawei.com ([10.6.80.69]) by dggeme758-chm.china.huawei.com ([10.6.80.69]) with mapi id 15.01.1713.004; Mon, 6 Jan 2020 16:47:17 +0800 From: "Zhouxingwang (Joe)" To: Stuart William Card , Carsten Bormann CC: "nwcrg@irtf.org" , "loops@ietf.org" , "dtn@ietf.org" Thread-Topic: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room Thread-Index: AQHVxA5wn08ZkzTEl06o0CselVnRRafcV/gAgAD4cVA= Date: Mon, 6 Jan 2020 08:47:17 +0000 Message-ID: References: <3D026CEF-AF7D-4CDE-8A5B-8CE5DCF749D1@tzi.org> <163A8040-9B13-43AA-8917-F75D984B0EBF@tzi.org> <26708398-5DEB-478E-8CB8-D406A8B874FF@tzi.org> <7DC0B005-8BA2-4D80-8CDF-4E2AE7EA96F3@tzi.org> <24526f0dbdcbdb65951d16cb5770fcbe.squirrel@www.critical.com> <266cb00cb46ed971f3055a086b421312.squirrel@www.critical.com> In-Reply-To: <266cb00cb46ed971f3055a086b421312.squirrel@www.critical.com> Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.134.27.230] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-CFilter-Loop: Reflected Archived-At: Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room X-BeenThere: loops@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Local Optimizations on Path Segments List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2020 08:47:25 -0000 Hi, It seems Linux TCP is much more out-of-order tolerant than we think. 1) TCP can dynamically adjust flow reordering level, not stick to 3. (https= ://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) tcp_reordering - INTEGER Initial reordering level of packets in a TCP strea= m. TCP stack can then dynamically adjust flow reordering level between this initial value and tcp_max_reordering Default: 3 tcp_max_reordering - INTEGER Maximal reordering level of packets in a TCP s= tream. 300 is a fairly conservative value, but you might increase it if paths are using per packet load balancing (like bonding rr mode) Defaul= t: 300 2) RACK has been enabled by default in Linux for loss detection since kerne= l4.4. (https://kernelnewbies.org/Linux_4.4)=20 tcp_recovery - INTEGER This value is a bitmap to enable various experimental loss recovery features. RACK: 0x1 enables the RACK loss detection for fast detection of lost retransmissions and tail drops. It also subsumes and disables RFC6675 recovery for SACK connections. RACK: 0x2 makes RACK's reordering window static (min_rtt/4). RACK: 0x4 disables RACK's DUPACK threshold heuristic Default: 0x1 The retransmission conflict between TCP and other layer will not be a big i= ssue. Thanks Joe -----Original Message----- From: nwcrg [mailto:nwcrg-bounces@irtf.org] On Behalf Of Stuart William Car= d Sent: Monday, January 06, 2020 9:46 AM To: Carsten Bormann Cc: nwcrg@irtf.org; loops@ietf.org; dtn@ietf.org Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, = Orchard Room Carsten -- Thanks for your thoughtful response. I'll try to focus my inputs. :-) > ... > (1) ... more intricate (and timely) knowledge of the critical path segment... > (2) techniques such as TCP splitting allowed mitigating some... > of the latency induced performance issues end-to-end protocols exhibited.= .. > With increasing use of encryption... (2) is going away... > (1) will stay relevant... where that knowledge can be provided... > What LOOPS is trying to do is providing a protocol that can help even=20 > in the absence of both (1) and (2)... > recovering packets can lead to reordering... > applications will need to be part of the picture... > LOOPS itself will need to focus on a more narrow set of problems and=20 > opportunities, and this is the time to discuss whether we got that=20 > narrowing right in the current charter proposal... Agreed. Further, not only transport but also network layer encryption kills= simple approaches to (2), leaving AFAIK only multipath or trading increase= d mean latency to reduce latency variance. Many of us have suffered from suboptimal and sometimes unanticipated intera= ctions between link and transport layer behaviors (retransmission in both, = flow control in the data link, congestion control in the transport). Application layer adaptive behaviors (e.g. video streaming that switches dy= namically during a session among different source coding rates based upon e= nd to end performance as it varies) further complicate this. LOOPS will be = yet another interacting control loop. In the worst case, a non-congestion l= oss of a link layer frame could lead to: - data link retransmission - LOOPS retransmission - transport retransmission - transport backoff - application rate downshift All at once! Presumably we can't prevent this altogether but want to minimi= ze its occurrence. The charter looks good. I suggest broadening its scope of consideration of = cross-layer interactions: not just w/transport congestion control, but also= w/retransmission, misordering & resequencing at all layers. -- Stuart W. Card, VP & Chief Scientist Critical Technologies Inc. (CTI) _______________________________________________ nwcrg mailing list nwcrg@irtf.org https://www.irtf.org/mailman/listinfo/nwcrg From nobody Mon Jan 6 00:55:10 2020 Return-Path: X-Original-To: loops@ietfa.amsl.com Delivered-To: loops@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 050B0120098 for ; Mon, 6 Jan 2020 00:55:08 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.899 X-Spam-Level: X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gtwo2boJu7-Z for ; Mon, 6 Jan 2020 00:55:04 -0800 (PST) Received: from mail-out01.uio.no (mail-out01.uio.no [IPv6:2001:700:100:10::50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DE241120116 for ; Mon, 6 Jan 2020 00:55:02 -0800 (PST) Received: from mail-mx12.uio.no ([129.240.10.84]) by mail-out01.uio.no with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92.3) (envelope-from ) id 1ioO9v-00052n-M9; Mon, 06 Jan 2020 09:54:47 +0100 Received: from ti0182q160-4250.bb.online.no ([82.164.31.203] helo=[10.0.0.14]) by mail-mx12.uio.no with esmtpsa (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) user michawe (Exim 4.93) (envelope-from ) id 1ioO9t-0003Gm-JL; Mon, 06 Jan 2020 09:54:47 +0100 From: Michael Welzl Message-Id: <8484FB1B-AAB8-45D8-B838-89AD58F4B3FD@ifi.uio.no> Content-Type: multipart/alternative; boundary="Apple-Mail=_7C9BFF23-E98D-44E9-B975-D72E919C561F" Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Date: Mon, 6 Jan 2020 09:54:41 +0100 In-Reply-To: Cc: Stuart William Card , Carsten Bormann , "loops@ietf.org" To: "Zhouxingwang (Joe)" References: <3D026CEF-AF7D-4CDE-8A5B-8CE5DCF749D1@tzi.org> <163A8040-9B13-43AA-8917-F75D984B0EBF@tzi.org> <26708398-5DEB-478E-8CB8-D406A8B874FF@tzi.org> <7DC0B005-8BA2-4D80-8CDF-4E2AE7EA96F3@tzi.org> <24526f0dbdcbdb65951d16cb5770fcbe.squirrel@www.critical.com> <266cb00cb46ed971f3055a086b421312.squirrel@www.critical.com> X-Mailer: Apple Mail (2.3445.104.11) X-UiO-SPF-Received: Received-SPF: neutral (mail-mx12.uio.no: 82.164.31.203 is neither permitted nor denied by domain of ifi.uio.no) client-ip=82.164.31.203; envelope-from=michawe@ifi.uio.no; helo=[10.0.0.14]; X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, HTML_MESSAGE=0.001, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO) X-UiO-Scanned: DB4F1B750402E1CD1775C04671AA606C46FEFBB6 Archived-At: Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room X-BeenThere: loops@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Local Optimizations on Path Segments List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2020 08:55:08 -0000 --Apple-Mail=_7C9BFF23-E98D-44E9-B975-D72E919C561F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii [dropping NWCRG and DTN to reduce noise a bit] Hi, Yes, Linux TCP has indeed been quite robust to re-ordering for many = years, see here for a quite detailed study: https://www.duo.uio.no/handle/10852/9034 = Cheers, Michael > On Jan 6, 2020, at 9:47 AM, Zhouxingwang (Joe) = wrote: >=20 > Hi, >=20 > It seems Linux TCP is much more out-of-order tolerant than we think. >=20 > 1) TCP can dynamically adjust flow reordering level, not stick to 3. = (https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt) > tcp_reordering - INTEGER Initial reordering level of packets in a TCP = stream. > TCP stack can then dynamically adjust flow reordering level > between this initial value and tcp_max_reordering > Default: 3 > tcp_max_reordering - INTEGER Maximal reordering level of packets in a = TCP stream. 300 is a fairly conservative value, but you might increase = it > if paths are using per packet load balancing (like bonding rr = mode) Default: 300 >=20 > 2) RACK has been enabled by default in Linux for loss detection since = kernel4.4. (https://kernelnewbies.org/Linux_4.4)=20 > tcp_recovery - INTEGER > This value is a bitmap to enable various experimental loss = recovery > features. >=20 > RACK: 0x1 enables the RACK loss detection for fast detection of = lost > retransmissions and tail drops. It also subsumes and = disables > RFC6675 recovery for SACK connections. > RACK: 0x2 makes RACK's reordering window static (min_rtt/4). > RACK: 0x4 disables RACK's DUPACK threshold heuristic >=20 > Default: 0x1 >=20 > The retransmission conflict between TCP and other layer will not be a = big issue. >=20 > Thanks > Joe >=20 > -----Original Message----- > From: nwcrg [mailto:nwcrg-bounces@irtf.org] On Behalf Of Stuart = William Card > Sent: Monday, January 06, 2020 9:46 AM > To: Carsten Bormann > Cc: nwcrg@irtf.org; loops@ietf.org; dtn@ietf.org > Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday = 08:30, Orchard Room >=20 > Carsten -- >=20 > Thanks for your thoughtful response. I'll try to focus my inputs. :-) >=20 >> ... >> (1) ... more intricate (and timely) knowledge of the critical path > segment... >> (2) techniques such as TCP splitting allowed mitigating some... >> of the latency induced performance issues end-to-end protocols = exhibited... >> With increasing use of encryption... (2) is going away... >> (1) will stay relevant... where that knowledge can be provided... >> What LOOPS is trying to do is providing a protocol that can help even=20= >> in the absence of both (1) and (2)... >> recovering packets can lead to reordering... >> applications will need to be part of the picture... >> LOOPS itself will need to focus on a more narrow set of problems and=20= >> opportunities, and this is the time to discuss whether we got that=20 >> narrowing right in the current charter proposal... >=20 > Agreed. Further, not only transport but also network layer encryption = kills simple approaches to (2), leaving AFAIK only multipath or trading = increased mean latency to reduce latency variance. >=20 > Many of us have suffered from suboptimal and sometimes unanticipated = interactions between link and transport layer behaviors (retransmission = in both, flow control in the data link, congestion control in the = transport). > Application layer adaptive behaviors (e.g. video streaming that = switches dynamically during a session among different source coding = rates based upon end to end performance as it varies) further complicate = this. LOOPS will be yet another interacting control loop. In the worst = case, a non-congestion loss of a link layer frame could lead to: > - data link retransmission > - LOOPS retransmission > - transport retransmission > - transport backoff > - application rate downshift > All at once! Presumably we can't prevent this altogether but want to = minimize its occurrence. >=20 > The charter looks good. I suggest broadening its scope of = consideration of cross-layer interactions: not just w/transport = congestion control, but also w/retransmission, misordering & = resequencing at all layers. >=20 > -- > Stuart W. Card, VP & Chief Scientist > Critical Technologies Inc. (CTI) > >=20 >=20 > _______________________________________________ > nwcrg mailing list > nwcrg@irtf.org > https://www.irtf.org/mailman/listinfo/nwcrg >=20 > --=20 > LOOPS mailing list > LOOPS@ietf.org > https://www.ietf.org/mailman/listinfo/loops --Apple-Mail=_7C9BFF23-E98D-44E9-B975-D72E919C561F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii [dropping NWCRG and DTN to reduce noise a bit]


Hi,

Yes, = Linux TCP has indeed been quite robust to re-ordering for many years, = see here for a quite detailed study:
https://www.duo.uio.no/handle/10852/9034

Cheers,
Michael


On Jan 6, 2020, at 9:47 AM, Zhouxingwang (Joe) <zhouxingwang@huawei.com> wrote:

Hi,
It seems Linux TCP is much more out-of-order = tolerant than we think.

1) TCP can = dynamically adjust flow reordering level, not stick to 3. (https://www.kernel.org/doc/Documentation/networking/ip-sysctl.t= xt)
tcp_reordering - INTEGER Initial reordering level = of packets in a TCP stream.
TCP stack can then dynamically = adjust flow reordering level
between this initial value and = tcp_max_reordering
Default: 3
tcp_max_reordering - INTEGER Maximal reordering level of = packets in a TCP stream. 300 is a fairly conservative value, but you = might increase it
if paths are using per packet = load balancing (like bonding rr mode) Default: 300

2) RACK has been enabled by default in Linux for loss = detection since kernel4.4. (https://kernelnewbies.org/Linux_4.4)
tcp_recovery - INTEGER
This = value is a bitmap to enable various experimental loss recovery
= features.

RACK: 0x1 = enables the RACK loss detection for fast detection of lost
=      retransmissions and tail drops. It = also subsumes and disables
=      RFC6675 recovery for SACK connections.
= RACK: 0x2 makes RACK's reordering window static (min_rtt/4).
= RACK: 0x4 disables RACK's DUPACK threshold heuristic

Default: 0x1

The retransmission conflict between TCP and other layer will = not be a big issue.

Thanks
Joe

-----Original = Message-----
From: nwcrg [mailto:nwcrg-bounces@irtf.org] On Behalf Of Stuart = William Card
Sent: Monday, January 06, 2020 9:46 AM
To: Carsten Bormann <cabo@tzi.org>
Cc: nwcrg@irtf.org; loops@ietf.org; dtn@ietf.org
Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on = Tuesday 08:30, Orchard Room

Carsten --

Thanks for your thoughtful response. I'll try = to focus my inputs. :-)

...
(1) ... more intricate (and = timely) knowledge of the critical path
segment...
(2) techniques such as TCP splitting allowed mitigating = some...
of the latency induced performance issues = end-to-end protocols exhibited...
With increasing use of = encryption... (2) is going away...
(1) will stay = relevant... where that knowledge can be provided...
What = LOOPS is trying to do is providing a protocol that can help even
in the absence of both (1) and (2)...
recovering = packets can lead to reordering...
applications will need = to be part of the picture...
LOOPS itself will need to = focus on a more narrow set of problems and
opportunities, = and this is the time to discuss whether we got that
narrowing right in the current charter proposal...

Agreed. Further, not only = transport but also network layer encryption kills simple approaches to = (2), leaving AFAIK only multipath or trading increased mean latency to = reduce latency variance.

Many of us have = suffered from suboptimal and sometimes unanticipated interactions = between link and transport layer behaviors (retransmission in both, flow = control in the data link, congestion control in the transport).
Application layer adaptive behaviors (e.g. video streaming = that switches dynamically during a session among different source coding = rates based upon end to end performance as it varies) further complicate = this. LOOPS will be yet another interacting control loop. In the worst = case, a non-congestion loss of a link layer frame could lead to:
- data link retransmission
- LOOPS = retransmission
- transport retransmission
- = transport backoff
- application rate downshift
All at once! Presumably we can't prevent this altogether but = want to minimize its occurrence.

The = charter looks good. I suggest broadening its scope of consideration of = cross-layer interactions: not just w/transport congestion control, but = also w/retransmission, misordering & resequencing at all layers.

--
Stuart W. Card, VP & = Chief Scientist
Critical Technologies Inc. (CTI)
<stu.card@critical.com>


_______________________________________________
nwcrg mailing list
nwcrg@irtf.org
https://www.irtf.org/mailman/listinfo/nwcrg

--
LOOPS mailing list
LOOPS@ietf.org
https://www.ietf.org/mailman/listinfo/loops

= --Apple-Mail=_7C9BFF23-E98D-44E9-B975-D72E919C561F-- From nobody Thu Jan 16 06:00:14 2020 Return-Path: X-Original-To: loops@ietfa.amsl.com Delivered-To: loops@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0C262120816 for ; Thu, 16 Jan 2020 06:00:12 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.198 X-Spam-Level: X-Spam-Status: No, score=-4.198 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4pHI0IqOCzb6 for ; Thu, 16 Jan 2020 06:00:07 -0800 (PST) Received: from gabriel-vm-2.zfn.uni-bremen.de (gabriel-vm-2.zfn.uni-bremen.de [134.102.50.17]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F3AFE1200CC for ; Thu, 16 Jan 2020 06:00:06 -0800 (PST) Received: from client-0044.vpn.uni-bremen.de (client-0044.vpn.uni-bremen.de [134.102.107.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gabriel-vm-2.zfn.uni-bremen.de (Postfix) with ESMTPSA id 47z5Rm5JMZzyR7; Thu, 16 Jan 2020 15:00:04 +0100 (CET) Content-Type: multipart/alternative; boundary="Apple-Mail=_ECDD0D59-E916-4FB9-A433-1A32794467C5" Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) From: Carsten Bormann Date: Thu, 16 Jan 2020 15:00:03 +0100 Cc: Carsten Bormann X-Mao-Original-Outgoing-Id: 600876003.722854-1d40073c9ecb3cb350bb14747e755527 Message-Id: <1C86A449-11F0-4AD7-AA54-76084F93DCA9@tzi.org> References: <157918220640.26131.868807962153171240.idtracker@ietfa.amsl.com> To: loops@ietf.org X-Mailer: Apple Mail (2.3608.40.2.2.4) Archived-At: Subject: [LOOPS] Fwd: [6lo] Last Call: (6LoWPAN Selective Fragment Recovery) to Proposed Standard X-BeenThere: loops@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Local Optimizations on Path Segments List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2020 14:00:13 -0000 --Apple-Mail=_ECDD0D59-E916-4FB9-A433-1A32794467C5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Fragment recovery is an approach that is somewhat related to LOOPS: On a challenged segment of the path (a 6LoWPAN mesh where larger packets = need to be fragmented and thus are subject to exponentiation of delivery = rates), a local recovery scheme is run. I think it is interesting to see how the various objectives of LOOPS are = addressed in this limited context. (Also, an IETF last call is always a good reason to have a final look at = a document that is proposed to progress.) Gr=C3=BC=C3=9Fe, Carsten > Begin forwarded message: >=20 > From: The IESG > > Subject: [6lo] Last Call: = (6LoWPAN Selective Fragment Recovery) to Proposed Standard > Date: 2020-01-16 at 14:43:26 CET > To: "IETF-Announce" > > Cc: draft-ietf-6lo-fragment-recovery@ietf.org = , 6lo-chairs@ietf.org = , Carles Gomez >, suresh@kaloom.com = , 6lo@ietf.org > Reply-To: last-call@ietf.org > Archived-At: = > >=20 >=20 > The IESG has received a request from the IPv6 over Networks of > Resource-constrained Nodes WG (6lo) to consider the following = document: - > '6LoWPAN Selective Fragment Recovery' > as Proposed Standard >=20 > The IESG plans to make a decision in the next few weeks, and solicits = final > comments on this action. Please send substantive comments to the > last-call@ietf.org mailing lists by = 2020-01-30. Exceptionally, comments may > be sent to iesg@ietf.org instead. In either = case, please retain the beginning > of the Subject line to allow automated sorting. >=20 > Abstract >=20 >=20 > This draft updates RFC 4944 with a simple protocol to recover > individual fragments across a route-over mesh network, with a = minimal > flow control to protect the network against bloat. >=20 >=20 >=20 >=20 > The file can be obtained via > https://datatracker.ietf.org/doc/draft-ietf-6lo-fragment-recovery/ = >=20 > IESG discussion can be tracked via > = https://datatracker.ietf.org/doc/draft-ietf-6lo-fragment-recovery/ballot/ = >=20 >=20 > No IPR declarations have been submitted directly on this I-D. >=20 >=20 >=20 >=20 > _______________________________________________ > 6lo mailing list > 6lo@ietf.org > https://www.ietf.org/mailman/listinfo/6lo = --Apple-Mail=_ECDD0D59-E916-4FB9-A433-1A32794467C5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Fragment recovery is an approach that is somewhat related to = LOOPS:
On a challenged segment of the path (a 6LoWPAN = mesh where larger packets need to be fragmented and thus are subject to = exponentiation of delivery rates), a local recovery scheme is = run.
I think it is interesting to see how the = various objectives of LOOPS are addressed in this limited = context.
(Also, an IETF last call is always a good = reason to have a final look at a document that is proposed to = progress.)

Gr=C3=BC=C3=9Fe, Carsten


Begin forwarded message:

From: = The IESG <iesg-secretary@ietf.org>
Subject: = [6lo] Last Call: = <draft-ietf-6lo-fragment-recovery-08.txt> (6LoWPAN Selective = Fragment Recovery) to Proposed Standard
Date: = 2020-01-16 at 14:43:26 CET
To: = "IETF-Announce" <ietf-announce@ietf.org>
Cc: draft-ietf-6lo-fragment-recovery@ietf.org, 6lo-chairs@ietf.org, = Carles Gomez <carlesgo@entel.upc.edu>, suresh@kaloom.com, 6lo@ietf.org
Reply-To: = last-call@ietf.org
Archived-At: <https://mailarchive.ietf.org/arch/msg/6lo/4dLWt2sqGGu7bPizyh7nf= WsJYCo>


The IESG has received a = request from the IPv6 over Networks of
Resource-constrained = Nodes WG (6lo) to consider the following document: -
'6LoWPAN Selective Fragment Recovery'
=  <draft-ietf-6lo-fragment-recovery-08.txt> as Proposed = Standard

The IESG plans to make a decision = in the next few weeks, and solicits final
comments on this = action. Please send substantive comments to the
last-call@ietf.org = mailing lists by 2020-01-30. Exceptionally, comments may
be = sent to iesg@ietf.org = instead. In either case, please retain the beginning
of = the Subject line to allow automated sorting.

Abstract


=   This draft updates RFC 4944 with a simple protocol to = recover
  individual fragments across a = route-over mesh network, with a minimal
  flow = control to protect the network against bloat.




The file can be = obtained via
https://datatracker.ietf.org/doc/draft-ietf-6lo-fragment-recove= ry/

IESG discussion can be tracked = via
https://datatracker.ietf.org/doc/draft-ietf-6lo-fragment-recove= ry/ballot/


No IPR = declarations have been submitted directly on this I-D.




_______________________________________________
6lo mailing list
6lo@ietf.org
https://www.ietf.org/mailman/listinfo/6lo

= --Apple-Mail=_ECDD0D59-E916-4FB9-A433-1A32794467C5--