From nobody Sun Nov 1 00:58:12 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 384823A0BEE for ; Sun, 1 Nov 2020 00:58:11 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: 0.033 X-Spam-Level: X-Spam-Status: No, score=0.033 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SORBS_HTTP=0.001, RCVD_IN_SORBS_SOCKS=1.927, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vedS08r0Qgmh for ; Sun, 1 Nov 2020 00:58:05 -0700 (PDT) Received: from dash.upc.es (dash.upc.es [147.83.2.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A14B83A0BA1 for ; Sun, 1 Nov 2020 00:58:04 -0700 (PDT) Received: from entelserver.upc.edu (entelserver.upc.es [147.83.40.4]) by dash.upc.es (8.14.4/8.14.4/Debian-4.1ubuntu1) with ESMTP id 0A17w0lT036144; Sun, 1 Nov 2020 08:58:00 +0100 Received: from webmail.entel.upc.edu (webmail.entel.upc.edu [147.83.39.6]) by entelserver.upc.edu (Postfix) with ESMTP id 4AD6D1D53C1; Sun, 1 Nov 2020 08:58:00 +0100 (CET) Received: from 83.38.106.121 by webmail.entel.upc.edu with HTTP; Sun, 1 Nov 2020 08:58:00 +0100 Message-ID: <43bb52606fbf7d70be5043edd1a90649.squirrel@webmail.entel.upc.edu> Date: Sun, 1 Nov 2020 08:58:00 +0100 From: "Carles Gomez Montenegro" To: tcpm@ietf.org Cc: Jon.Crowcroft@cl.cam.ac.uk User-Agent: SquirrelMail/1.4.21-1.fc14 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Virus-Scanned: clamav-milter 0.100.3 at dash X-Virus-Status: Clean X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.3.9 (dash.upc.es [147.83.2.50]); Sun, 01 Nov 2020 08:58:00 +0100 (CET) Archived-At: Subject: [tcpm] [Fwd: New Version Notification for draft-gomez-tcpm-ack-rate-request-01.txt] X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 07:58:11 -0000 Dear TCPM WG, We just submitted an updated version of the "TCP ACK Rate Request option" draft. Comments are very welcome. Cheers, Carles and Jon ---------------------------- Original Message ---------------------------- Subject: New Version Notification for draft-gomez-tcpm-ack-rate-request-01.txt From: internet-drafts@ietf.org Date: Sun, November 1, 2020 8:47 am To: "Carles Gomez" "Jon Crowcroft" -------------------------------------------------------------------------- A new version of I-D, draft-gomez-tcpm-ack-rate-request-01.txt has been successfully submitted by Carles Gomez and posted to the IETF repository. Name: draft-gomez-tcpm-ack-rate-request Revision: 01 Title: TCP ACK Rate Request Option Document date: 2020-10-31 Group: Individual Submission Pages: 7 URL: https://www.ietf.org/archive/id/draft-gomez-tcpm-ack-rate-request-01.txt Status: https://datatracker.ietf.org/doc/draft-gomez-tcpm-ack-rate-request/ Htmlized: https://datatracker.ietf.org/doc/html/draft-gomez-tcpm-ack-rate-request Htmlized: https://tools.ietf.org/html/draft-gomez-tcpm-ack-rate-request-01 Diff: https://www.ietf.org/rfcdiff?url2=draft-gomez-tcpm-ack-rate-request-01 Abstract: TCP Delayed Acknowledgments (ACKs) is a widely deployed mechanism that allows reducing protocol overhead in many scenarios. However, Delayed ACKs may also contribute to suboptimal performance. When a relatively large congestion window (cwnd) can be used, less frequent ACKs may be desirable. On the other hand, in relatively small cwnd scenarios, eliciting an immediate ACK may avoid unnecessary delays that may be incurred by the Delayed ACKs mechanism. This document specifies the TCP ACK Rate Request (TARR) option. This option allows a sender to indicate the ACK rate to be used by a receiver, and it also allows to request immediate ACKs from a receiver. Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. The IETF Secretariat From nobody Sun Nov 1 01:06:53 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0A8BF3A0F41 for ; Sun, 1 Nov 2020 01:06:52 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h5SYxoh1ZF16 for ; Sun, 1 Nov 2020 01:06:50 -0700 (PDT) Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8BD303A0C11 for ; Sun, 1 Nov 2020 01:06:50 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id b3so5054423wrx.11 for ; Sun, 01 Nov 2020 01:06:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=KVs2iQY8pyO8jV1lN7bFjGV8YIkbgHFbGnn4p9bhmyk=; b=ktcMRUKVzHVT2H49svYtYEhxWqwqXd1yFPqGKD2iA/9oJOHAqDWaj1Sp8VzjSYuKxR lSFrtvDtu9nco7w4NLqAULcexsmqIBx6w2CSDvpQnC5YngDnfg0qhzxLBMYb6UBpLiE4 0tQOZKILQDkpDz5MN2IAQRDESoHY5rpo0Yn3HSc6mp6ouCpufZDx8LsF6Io0/LGUx3D1 wVB48CEKt3FsL9hcIz6EgzvxN1ZLKHGPdqcOMY7EvC8n3Jy72YfoqsTAlBuy63G3Ef4e fmrRHapGm3ZYoHNVabzErpPRuyNaRu9GP6h9uCjAciOZzE4e+xWbn+vcGUtrCT+2HYf3 yLHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=KVs2iQY8pyO8jV1lN7bFjGV8YIkbgHFbGnn4p9bhmyk=; b=fKoDX7VkOSED16DhPo3iO8+i7opjMSH/OFdTKiGmaxvfYC6ujThD4Jd8kWPlvlE4m0 npUXmIIrLomI2BFYrRx7l3UTW/Wwh3scCT9snPmQZCcK0knz7yDq2yXdhTDA16stwrTW m5DLhU9lY8zv8fMVWF0BvuBLQF6uFFhH8nbNVoy3nRHZr58VpJqEqgb3H5u/yEyt795E MK+Y/Su4O+YA49nSTKVXfWGd7aj6nc735bMIRG/VaMXRAmJnRYdSsQ5LSf4ZKNnAws+p MiURKBEDjBS5k7/S4aP2PQ4fgdfAw8zKT8hNmNl0o7R/zB1Li429gY+CchEfMg6dqfJ8 utnw== X-Gm-Message-State: AOAM533QixHHKFmY16GqUxtZ6U/4aqoZ6g93HUHdHTlFmFHLe6n/esH+ OBq97WkrivQUwE2QLYVHjorFYkQqEporHX8O1Ox+Iocp/Bw= X-Google-Smtp-Source: ABdhPJyJNLVfJa/1cpFkKS7P3Kg/4JoPD8MIKKqwZrF+QR88BehtoRqmJzlKlaoausb7HVo2N5t3b4LHYNkkYP6hFWk= X-Received: by 2002:adf:f1c1:: with SMTP id z1mr13438015wro.331.1604218008673; Sun, 01 Nov 2020 01:06:48 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Juhamatti Kuusisaari Date: Sun, 1 Nov 2020 10:06:34 +0200 Message-ID: To: tcpm IETF list , "Scharf, Michael" Content-Type: text/plain; charset="UTF-8" Archived-At: Subject: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 08:06:52 -0000 Hello, My comments included below apply also to draft-ietf-tcpm-yang-tcp-00. In brief: * include-tcp-options -> ignore-tcp-options with default false * accept-ao-mismatch -> accept-key-mismatch BR, -- Juhamatti ---------- Forwarded message --------- From: Juhamatti Kuusisaari Date: Fri, 18 Sep 2020 at 11:53 Subject: [tcpm] Comments for draft-scharf-tcpm-yang-tcp-06 To: tcpm IETF list , Scharf, Michael Hello, I read through draft-scharf-tcpm-yang-tcp-06 and overall it looks fine to me. Nevertheless, there are a couple of items that may need clarifications/improvements. (1) I believe "leaf include-tcp-options" should be "leaf ignore-tcp-options" with a false default as the options are included by default in the RFC 5925. In my opinion, this would better emphasize the fact that options really should be included by default and not including them should be a special case. Change suggestion in detail below: leaf include-tcp-options { type boolean; must "../enable-ao = 'true'"; description "Include TCP options in HMAC calculation."; } => leaf ignore-tcp-options { type boolean; default "false"; must "../enable-ao = 'true'"; description "Ignore TCP options in MAC calculation."; } Please also note the "HMAC"->"MAC" change suggestion. And yes, I do realize that a default could be added to the original "include" leaf. After pondering about this, I do think "ignore" leaf would be a better end result for the reasons I mentioned above. (2) There is now a leaf that says: leaf accept-ao-mismatch { type boolean; must "../enable-ao = 'true'"; description "Accept packets with HMAC mismatch."; } It is true that RFC 5925 allows non-existing MKT connections that should be accepted. Then again, the above configuration and its description looks to me that any mismatch would be accepted. So, maybe a configuration setting better reflecting RFC 5925 would be something on the lines of leaf accept-key-mismatch { type boolean; must "../enable-ao = 'true'"; description "Accept TCP segments with a Master Key Tuple (MKT) that is not configured."; } As this configuration option does not have such a strong default as the former one, I do not see a need to change its logic otherwise nor add a default. I would assume that most security aware users would have "false" there as a setting - especially those users that would use a YANG model to do the configuration. Best regards, -- Juhamatti From nobody Sun Nov 1 01:14:51 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7B0EA3A0F44 for ; Sun, 1 Nov 2020 01:14:50 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gu07VIPs2609 for ; Sun, 1 Nov 2020 01:14:48 -0700 (PDT) Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 441673A0BFF for ; Sun, 1 Nov 2020 01:14:47 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 1E2A125A13; Sun, 1 Nov 2020 09:14:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1604218486; bh=XuGw/ewBjnL0h+sw0/bqfiiTKQhiNxv8E64dRL4SRTw=; h=From:To:Subject:Date:References:In-Reply-To:From; b=nmvXoeOZB5yFEaHREIbfH98CmUU7Y+N0srUZKBNFPyrUaSFdGmylw/V2lKysW/arO WF3MT+He1I3wwbrXrcjA+8rbichjWbHKXciAII1cIMeZgEeXVo2cbGp9UEIZCiLEsR j4AHhTqpMzp+kdF93CH4hlr9zcKMPb+pFpiixAQo= X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QuBP4rULnJBi; Sun, 1 Nov 2020 09:14:45 +0100 (CET) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Sun, 1 Nov 2020 09:14:45 +0100 (CET) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Sun, 1 Nov 2020 09:14:45 +0100 Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Sun, 1 Nov 2020 09:14:45 +0100 From: "Scharf, Michael" To: Juhamatti Kuusisaari , tcpm IETF list Thread-Topic: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 Thread-Index: AQHWsCX18jC/qcTIsEiT0l55wWW0SKmy7E5A Date: Sun, 1 Nov 2020 08:14:44 +0000 Message-ID: <8f2a68f8b4e34f2fa568d9182f23c812@hs-esslingen.de> References: In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [134.108.140.248] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 08:14:51 -0000 SGkgSnVoYW1hdHRpLA0KDQpBZGRyZXNzaW5nIHlvdXIgY29tbWVudHMgaXMgb24gbXkgVE9ETyBs aXN0LiBQbGVhc2Ugc3RheSB0dW5lZC4NCg0KVGhlIHZlcnNpb24gZHJhZnQtaWV0Zi10Y3BtLXlh bmctdGNwLTAwIGlzIGp1c3QgYSBjb3B5IG9mIHRoZSBwcmV2aW91cyB2ZXJzaW9uIGluIG9yZGVy IHRvIHNpbXBsaWZ5IGZ1dHVyZSBjaGFuZ2UgdHJhY2tpbmcuIFZlcnNpb24gLTAxIHdpbGwgYWRk cmVzcyB5b3VyIGZlZWRiYWNrIGFzIHdlbGwgYXMgb3RoZXIgY29tbWVudHMuDQoNClRoYW5rcywg YW5kIHNvcnJ5IGZvciB0aGUgZGVsYXllZCByZXNwb25zZQ0KDQpNaWNoYWVsDQoNCj4gLS0tLS1P cmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogSnVoYW1hdHRpIEt1dXNpc2FhcmkgPGp1aGFt YXRrQGdtYWlsLmNvbT4NCj4gU2VudDogU3VuZGF5LCBOb3ZlbWJlciAxLCAyMDIwIDk6MDcgQU0N Cj4gVG86IHRjcG0gSUVURiBsaXN0IDx0Y3BtQGlldGYub3JnPjsgU2NoYXJmLCBNaWNoYWVsIDxN aWNoYWVsLlNjaGFyZkBocy0NCj4gZXNzbGluZ2VuLmRlPg0KPiBTdWJqZWN0OiBbdGNwbV0gQ29t bWVudHMgZm9yIGRyYWZ0LWlldGYtdGNwbS15YW5nLXRjcC0wMA0KPiANCj4gSGVsbG8sDQo+IA0K PiBNeSBjb21tZW50cyBpbmNsdWRlZCBiZWxvdyBhcHBseSBhbHNvIHRvIGRyYWZ0LWlldGYtdGNw bS15YW5nLXRjcC0wMC4NCj4gDQo+IEluIGJyaWVmOg0KPiAgKiBpbmNsdWRlLXRjcC1vcHRpb25z IC0+IGlnbm9yZS10Y3Atb3B0aW9ucyB3aXRoIGRlZmF1bHQgZmFsc2UNCj4gICogYWNjZXB0LWFv LW1pc21hdGNoIC0+IGFjY2VwdC1rZXktbWlzbWF0Y2gNCj4gDQo+IEJSLA0KPiAtLQ0KPiAgSnVo YW1hdHRpDQo+IA0KPiANCj4gLS0tLS0tLS0tLSBGb3J3YXJkZWQgbWVzc2FnZSAtLS0tLS0tLS0N Cj4gRnJvbTogSnVoYW1hdHRpIEt1dXNpc2FhcmkgPGp1aGFtYXRrQGdtYWlsLmNvbT4NCj4gRGF0 ZTogRnJpLCAxOCBTZXAgMjAyMCBhdCAxMTo1Mw0KPiBTdWJqZWN0OiBbdGNwbV0gQ29tbWVudHMg Zm9yIGRyYWZ0LXNjaGFyZi10Y3BtLXlhbmctdGNwLTA2DQo+IFRvOiB0Y3BtIElFVEYgbGlzdCA8 dGNwbUBpZXRmLm9yZz4sIFNjaGFyZiwgTWljaGFlbA0KPiA8TWljaGFlbC5TY2hhcmZAaHMtZXNz bGluZ2VuLmRlPg0KPiANCj4gDQo+IEhlbGxvLA0KPiANCj4gSSByZWFkIHRocm91Z2ggZHJhZnQt c2NoYXJmLXRjcG0teWFuZy10Y3AtMDYgYW5kIG92ZXJhbGwgaXQgbG9va3MgZmluZSB0byBtZS4N Cj4gDQo+IE5ldmVydGhlbGVzcywgdGhlcmUgYXJlIGEgY291cGxlIG9mIGl0ZW1zIHRoYXQgbWF5 IG5lZWQNCj4gY2xhcmlmaWNhdGlvbnMvaW1wcm92ZW1lbnRzLg0KPiANCj4gKDEpIEkgYmVsaWV2 ZSAibGVhZiBpbmNsdWRlLXRjcC1vcHRpb25zIiBzaG91bGQgYmUgImxlYWYNCj4gaWdub3JlLXRj cC1vcHRpb25zIiB3aXRoIGEgZmFsc2UgZGVmYXVsdCBhcyB0aGUgb3B0aW9ucyBhcmUgaW5jbHVk ZWQNCj4gYnkgZGVmYXVsdCBpbiB0aGUgUkZDIDU5MjUuIEluIG15IG9waW5pb24sIHRoaXMgd291 bGQgYmV0dGVyIGVtcGhhc2l6ZQ0KPiB0aGUgZmFjdCB0aGF0IG9wdGlvbnMgcmVhbGx5IHNob3Vs ZCBiZSBpbmNsdWRlZCBieSBkZWZhdWx0IGFuZCBub3QNCj4gaW5jbHVkaW5nIHRoZW0gc2hvdWxk IGJlIGEgc3BlY2lhbCBjYXNlLiBDaGFuZ2Ugc3VnZ2VzdGlvbiBpbiBkZXRhaWwNCj4gYmVsb3c6 DQo+IA0KPiAgICAgICBsZWFmIGluY2x1ZGUtdGNwLW9wdGlvbnMgew0KPiAgICAgICAgIHR5cGUg Ym9vbGVhbjsNCj4gICAgICAgICBtdXN0ICIuLi9lbmFibGUtYW8gPSAndHJ1ZSciOw0KPiAgICAg ICAgIGRlc2NyaXB0aW9uDQo+ICAgICAgICAgICAiSW5jbHVkZSBUQ1Agb3B0aW9ucyBpbiBITUFD IGNhbGN1bGF0aW9uLiI7DQo+ICAgICAgIH0NCj4gPT4NCj4gICAgICAgbGVhZiBpZ25vcmUtdGNw LW9wdGlvbnMgew0KPiAgICAgICAgIHR5cGUgYm9vbGVhbjsNCj4gICAgICAgICBkZWZhdWx0ICJm YWxzZSI7DQo+ICAgICAgICAgbXVzdCAiLi4vZW5hYmxlLWFvID0gJ3RydWUnIjsNCj4gICAgICAg ICBkZXNjcmlwdGlvbg0KPiAgICAgICAgICAgIklnbm9yZSBUQ1Agb3B0aW9ucyBpbiBNQUMgY2Fs Y3VsYXRpb24uIjsNCj4gICAgICAgfQ0KPiANCj4gUGxlYXNlIGFsc28gbm90ZSB0aGUgIkhNQUMi LT4iTUFDIiBjaGFuZ2Ugc3VnZ2VzdGlvbi4gQW5kIHllcywgSSBkbw0KPiByZWFsaXplIHRoYXQg YSBkZWZhdWx0IGNvdWxkIGJlIGFkZGVkIHRvIHRoZSBvcmlnaW5hbCAiaW5jbHVkZSIgbGVhZi4N Cj4gQWZ0ZXIgcG9uZGVyaW5nIGFib3V0IHRoaXMsIEkgZG8gdGhpbmsgImlnbm9yZSIgbGVhZiB3 b3VsZCBiZSBhIGJldHRlcg0KPiBlbmQgcmVzdWx0IGZvciB0aGUgcmVhc29ucyBJIG1lbnRpb25l ZCBhYm92ZS4NCj4gDQo+ICgyKSBUaGVyZSBpcyBub3cgYSBsZWFmIHRoYXQgc2F5czoNCj4gDQo+ ICAgICAgIGxlYWYgYWNjZXB0LWFvLW1pc21hdGNoIHsNCj4gICAgICAgICB0eXBlIGJvb2xlYW47 DQo+ICAgICAgICAgbXVzdCAiLi4vZW5hYmxlLWFvID0gJ3RydWUnIjsNCj4gICAgICAgICBkZXNj cmlwdGlvbg0KPiAgICAgICAgICAgIkFjY2VwdCBwYWNrZXRzIHdpdGggSE1BQyBtaXNtYXRjaC4i Ow0KPiAgICAgICB9DQo+IA0KPiBJdCBpcyB0cnVlIHRoYXQgUkZDIDU5MjUgYWxsb3dzIG5vbi1l eGlzdGluZyBNS1QgY29ubmVjdGlvbnMgdGhhdA0KPiBzaG91bGQgYmUgYWNjZXB0ZWQuIFRoZW4g YWdhaW4sIHRoZSBhYm92ZSBjb25maWd1cmF0aW9uIGFuZCBpdHMNCj4gZGVzY3JpcHRpb24gbG9v a3MgdG8gbWUgdGhhdCBhbnkgbWlzbWF0Y2ggd291bGQgYmUgYWNjZXB0ZWQuIFNvLCBtYXliZQ0K PiBhIGNvbmZpZ3VyYXRpb24gc2V0dGluZyBiZXR0ZXIgcmVmbGVjdGluZyBSRkMgNTkyNSB3b3Vs ZCBiZSBzb21ldGhpbmcNCj4gb24gdGhlIGxpbmVzIG9mDQo+IA0KPiAgICAgICBsZWFmIGFjY2Vw dC1rZXktbWlzbWF0Y2ggew0KPiAgICAgICAgIHR5cGUgYm9vbGVhbjsNCj4gICAgICAgICBtdXN0 ICIuLi9lbmFibGUtYW8gPSAndHJ1ZSciOw0KPiAgICAgICAgIGRlc2NyaXB0aW9uDQo+ICAgICAg ICAgICAiQWNjZXB0IFRDUCBzZWdtZW50cyB3aXRoIGEgTWFzdGVyIEtleSBUdXBsZSAoTUtUKSB0 aGF0IGlzDQo+IG5vdCBjb25maWd1cmVkLiI7DQo+ICAgICAgIH0NCj4gDQo+IEFzIHRoaXMgY29u ZmlndXJhdGlvbiBvcHRpb24gZG9lcyBub3QgaGF2ZSBzdWNoIGEgc3Ryb25nIGRlZmF1bHQgYXMN Cj4gdGhlIGZvcm1lciBvbmUsIEkgZG8gbm90IHNlZSBhIG5lZWQgdG8gY2hhbmdlIGl0cyBsb2dp YyBvdGhlcndpc2Ugbm9yDQo+IGFkZCBhIGRlZmF1bHQuIEkgd291bGQgYXNzdW1lIHRoYXQgbW9z dCBzZWN1cml0eSBhd2FyZSB1c2VycyB3b3VsZA0KPiBoYXZlICJmYWxzZSIgdGhlcmUgYXMgYSBz ZXR0aW5nIC0gZXNwZWNpYWxseSB0aG9zZSB1c2VycyB0aGF0IHdvdWxkDQo+IHVzZSBhIFlBTkcg bW9kZWwgdG8gZG8gdGhlIGNvbmZpZ3VyYXRpb24uDQo+IA0KPiBCZXN0IHJlZ2FyZHMsDQo+IC0t DQo+ICBKdWhhbWF0dGkNCg== From nobody Mon Nov 2 14:00:11 2020 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id CFF773A0822; Mon, 2 Nov 2020 14:00:05 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.21.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <160435440581.5690.5100780803566339971@ietfa.amsl.com> Date: Mon, 02 Nov 2020 14:00:05 -0800 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-rack-12.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2020 22:00:06 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : The RACK-TLP loss detection algorithm for TCP Authors : Yuchung Cheng Neal Cardwell Nandita Dukkipati Priyaranjan Jha Filename : draft-ietf-tcpm-rack-12.txt Pages : 31 Date : 2020-11-02 Abstract: This document presents the RACK-TLP loss detection algorithm for TCP. RACK-TLP uses per-segment transmit timestamps and selective acknowledgements (SACK) and has two parts: RACK ("Recent ACKnowledgment") starts fast recovery quickly using time-based inferences derived from ACK feedback. TLP ("Tail Loss Probe") leverages RACK and sends a probe packet to trigger ACK feedback to avoid retransmission timeout (RTO) events. Compared to the widely used DUPACK threshold approach, RACK-TLP detects losses more efficiently when there are application-limited flights of data, lost retransmissions, or data packet reordering events. It is intended to be an alternative to the DUPACK threshold approach. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/ There are also htmlized versions available at: https://tools.ietf.org/html/draft-ietf-tcpm-rack-12 https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rack-12 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rack-12 Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Nov 2 14:28:57 2020 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 21C423A1586; Mon, 2 Nov 2020 14:28:44 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.21.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <160435612410.7062.6419055780200518274@ietfa.amsl.com> Date: Mon, 02 Nov 2020 14:28:44 -0800 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-rack-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2020 22:28:52 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : The RACK-TLP loss detection algorithm for TCP Authors : Yuchung Cheng Neal Cardwell Nandita Dukkipati Priyaranjan Jha Filename : draft-ietf-tcpm-rack-13.txt Pages : 31 Date : 2020-11-02 Abstract: This document presents the RACK-TLP loss detection algorithm for TCP. RACK-TLP uses per-segment transmit timestamps and selective acknowledgements (SACK) and has two parts: RACK ("Recent ACKnowledgment") starts fast recovery quickly using time-based inferences derived from ACK feedback. TLP ("Tail Loss Probe") leverages RACK and sends a probe packet to trigger ACK feedback to avoid retransmission timeout (RTO) events. Compared to the widely used DUPACK threshold approach, RACK-TLP detects losses more efficiently when there are application-limited flights of data, lost retransmissions, or data packet reordering events. It is intended to be an alternative to the DUPACK threshold approach. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/ There are also htmlized versions available at: https://tools.ietf.org/html/draft-ietf-tcpm-rack-13 https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-rack-13 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rack-13 Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Nov 2 15:39:54 2020 Return-Path: X-Original-To: tcpm@ietf.org Delivered-To: tcpm@ietfa.amsl.com Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id E7C0C3A1301; Mon, 2 Nov 2020 15:39:51 -0800 (PST) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: internet-drafts@ietf.org To: Cc: tcpm@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 7.21.0 Auto-Submitted: auto-generated Precedence: bulk Reply-To: tcpm@ietf.org Message-ID: <160436039192.31069.18212280907615305035@ietfa.amsl.com> Date: Mon, 02 Nov 2020 15:39:51 -0800 Archived-At: Subject: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2020 23:39:52 -0000 A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF. Title : More Accurate ECN Feedback in TCP Authors : Bob Briscoe Mirja Kühlewind Richard Scheffenegger Filename : draft-ietf-tcpm-accurate-ecn-13.txt Pages : 58 Date : 2020-11-02 Abstract: Explicit Congestion Notification (ECN) is a mechanism where network nodes can mark IP packets instead of dropping them to indicate incipient congestion to the end-points. Receivers with an ECN- capable transport protocol feed back this information to the sender. ECN is specified for TCP in such a way that only one feedback signal can be transmitted per Round-Trip Time (RTT). Recent new TCP mechanisms like Congestion Exposure (ConEx), Data Center TCP (DCTCP) or Low Latency Low Loss Scalable Throughput (L4S) need more accurate ECN feedback information whenever more than one marking is received in one RTT. This document specifies a scheme to provide more than one feedback signal per RTT in the TCP header. Given TCP header space is scarce, it allocates a reserved header bit, that was previously used for the ECN-Nonce which has now been declared historic. It also overloads the two existing ECN flags in the TCP header. The resulting extra space is exploited to feed back the IP- ECN field received during the 3-way handshake as well. Supplementary feedback information can optionally be provided in a new TCP option, which is never used on the TCP SYN. The IETF datatracker status page for this draft is: https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ There are also htmlized versions available at: https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13 https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-13 A diff from the previous version is available at: https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-13 Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ From nobody Mon Nov 2 16:08:11 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 47C333A093D for ; Mon, 2 Nov 2020 16:08:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.347 X-Spam-Level: X-Spam-Status: No, score=-2.347 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.247, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hs-WDIbndgUA for ; Mon, 2 Nov 2020 16:08:07 -0800 (PST) Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3B6733A12DE for ; Mon, 2 Nov 2020 16:06:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:References:Cc:To:From:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=BNPOAfTBRUezYaFF6xQmwaTScgvodAwa9t397POu92Q=; b=awB90lB6yi514TI6XFp97LHLzk KscPyuby++02WgM+hzd4O6kSxVyHwJrDekWL8WcA2GtWHtLzn+yO3fthEycPJzNzczZPNM4svi8i7 SNSnawCZLkHtksm33ewAqvA1F/kMICS1En0wYOSZxX9nplNpMbiOwDX5jgDGlCHeFPMsf8ac2SvPg skjKJC6v+8jvbk/XtVWNxxAFbpPlKqsY+EW9XFA3mpme9EDfAXb3RKiv+5/0PIV8oI8v+KqkuZV0t dxHF566S0ai4cayd+E0OZa+ntI9dnBauLulBF9HAGbKwW/Q3oyFMtlUsUs2vKkwr1X38xfinIsDk/ BFldGQgA==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:39504 helo=[192.168.1.3]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kZjpl-00FIBL-Gl; Tue, 03 Nov 2020 00:05:58 +0000 From: Bob Briscoe To: "Scharf, Michael" , "tcpm@ietf.org" Cc: "Scheffenegger, Richard" , Mirja Kuehlewind References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> Message-ID: Date: Tue, 3 Nov 2020 00:05:56 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB X-OutGoing-Spam-Status: No, score=-0.2 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 00:08:10 -0000 Michael, 1/ I just posted a draft-13 that has the two AccECN TCP Option orders. There are also htmlized versions available at: https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13 As agreed, it keys the two option orders on two different Option Kinds, which I've requested in the IANA section (TBD0 and TBD1). This replaces using the initial values of the fields. I've reverted all the complicated text about: * varying the initial field values dependent on option order, * having to keep consistent order throughout a connection, * and choosing which order from each end. All of which is irrelevant now. It is much simpler (I guess we all knew that). Thank you. As mentioned, I found my other ToDo list: 2/ We also introduced considerably more guidance on middlebox handling, esp. ACK filtering and Segmentation Offload. There was enough extra text to be worth breaking that section into subsections. The guidance on ACK filtering for connections with AccECN negotiated updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in the presence of ECN was a research issue). But there is still no guidance for ECN connections when AccECN has not been negotiated. 3/ I also added a SHOULD requirement I had on my ToDo list: Each Data Sender SHOULD disable ECT in one direction if all the feedback for three or four rounds from the start is CE. That's already done in Linux, and in the TCP in iOS and MacOS at least. Bob On 28/10/2020 17:17, Bob Briscoe wrote: > Michael, > > On 28/10/2020 15:55, Scharf, Michael wrote: >> Hi Bob, >> >> What is the future plan regarding the option encoding proposed by the >> authors? >> >> This has been discussed in past meetings and on the list, for >> instance: >> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/ >> >> I have explained several times my pushback against the encoding >> currently described in the document, most notably the requirement to >> keep state in the endpoints to decode the format of a TCP option. I >> have not changed my mind. >> >> I have also mentioned that, for instance, two option codepoints would >> IMHO be a better engineering approach ("KISS principle"). That would >> be a very simple change to the content to the document. Needless to >> say that there are other alternatives for marshalling parameters in >> an unambigous way. > > [BB] Whoops. In my head I had already shifted to your scheme with > two-option kinds. But I see that, in the draft,  I haven't. Whoops - > that was a bug in my ToDo list processing. Sorry about that. I'll > submit another draft before the deadline with that scheme replacing > the current one. > > I've just found some other entries in my ToDo list for this draft > which I missed as well. I'll include them too. > > Sorry again. > > > Bob > >> >> There have been other related comments, e.g. from Ilpo. >> >> Best regards >> >> Michael (with no hat) >> >>> -----Original Message----- >>> From: tcpm On Behalf Of Bob Briscoe >>> Sent: Wednesday, October 28, 2020 2:04 PM >>> To: tcpm@ietf.org >>> Cc: Scheffenegger, Richard ; Mirja >>> Kuehlewind >>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-12.txt >>> >>> TCPM folks, >>> >>> Back in May'20, it was decided to hold back the AccECN (and the ECN++) >>> drafts for a while until the position on L4S in tsvwg was clearer. >>> There >>> is no formal dependency on the L4S drafts in tsvwg, but it made sense >>> not to burn codepoints without more certainty. >>> >>> I've just refreshed the AccECN and the ECN++ drafts, which had expired. >>> >>> I included the following minor changes to AccECN, which you can see via >>> the diff link below, but in summary: >>> * moved wrongly categorized "action if unwilling to provide f/b" to the >>> group of bullets relevant to the receiver >>> * removed refs to long-expired ecn-fallback draft >>> * fixed table 1 after xml2rfc changed formatting >>> >>> Thanks to those who pointed these out. >>> >>> No changes to ECN++ other than update of some refs. >>> >>> Cheers >>> >>> >>> Bob >>> >>> On 28/10/2020 12:47, internet-drafts@ietf.org wrote: >>>> A New Internet-Draft is available from the on-line Internet-Drafts >>>> directories. >>>> This draft is a work item of the TCP Maintenance and Minor >>>> Extensions WG of >>> the IETF. >>>>           Title           : More Accurate ECN Feedback in TCP >>>>           Authors         : Bob Briscoe >>>>                             Mirja Kühlewind >>>>                             Richard Scheffenegger >>>>     Filename        : draft-ietf-tcpm-accurate-ecn-12.txt >>>>     Pages           : 58 >>>>     Date            : 2020-10-28 >>>> >>>> Abstract: >>>>      Explicit Congestion Notification (ECN) is a mechanism where >>>> network >>>>      nodes can mark IP packets instead of dropping them to indicate >>>>      incipient congestion to the end-points.  Receivers with an ECN- >>>>      capable transport protocol feed back this information to the >>>> sender. >>>>      ECN is specified for TCP in such a way that only one feedback >>>> signal >>>>      can be transmitted per Round-Trip Time (RTT).  Recent new TCP >>>>      mechanisms like Congestion Exposure (ConEx), Data Center TCP >>>> (DCTCP) >>>>      or Low Latency Low Loss Scalable Throughput (L4S) need more >>>> accurate >>>>      ECN feedback information whenever more than one marking is >>>> received >>>>      in one RTT.  This document specifies a scheme to provide more >>>> than >>>>      one feedback signal per RTT in the TCP header.  Given TCP header >>>>      space is scarce, it allocates a reserved header bit, that was >>>>      previously used for the ECN-Nonce which has now been declared >>>>      historic.  It also overloads the two existing ECN flags in the >>>> TCP >>>>      header.  The resulting extra space is exploited to feed back >>>> the IP- >>>>      ECN field received during the 3-way handshake as well. >>>> Supplementary >>>>      feedback information can optionally be provided in a new TCP >>>> option, >>>>      which is never used on the TCP SYN. >>>> >>>> >>>> The IETF datatracker status page for this draft is: >>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ >>>> >>>> There are also htmlized versions available at: >>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12 >>>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12 >>>> >>>> A diff from the previous version is available at: >>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-12 >>>> >>>> >>>> Please note that it may take a couple of minutes from the time of >>>> submission >>>> until the htmlized version and diff are available at tools.ietf.org. >>>> >>>> Internet-Drafts are also available by anonymous FTP at: >>>> ftp://ftp.ietf.org/internet-drafts/ >>>> >>>> >>>> _______________________________________________ >>>> tcpm mailing list >>>> tcpm@ietf.org >>>> https://www.ietf.org/mailman/listinfo/tcpm >>> -- >>> ________________________________________________________________ >>> Bob Briscoe http://bobbriscoe.net/ >>>                  PRIVILEGED AND CONFIDENTIAL >>> >>> _______________________________________________ >>> tcpm mailing list >>> tcpm@ietf.org >>> https://www.ietf.org/mailman/listinfo/tcpm >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ From nobody Mon Nov 2 16:26:05 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E8F793A12A9 for ; Mon, 2 Nov 2020 16:26:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -17.599 X-Spam-Level: X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EzscLUkm6CyE for ; Mon, 2 Nov 2020 16:25:59 -0800 (PST) Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 07F0D3A12AB for ; Mon, 2 Nov 2020 16:25:59 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id w14so16562664wrs.9 for ; Mon, 02 Nov 2020 16:25:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hST2jqAwtBkEXrImHujSBwHJr715993K+qvnyWIMlgY=; b=HcNlzNw6gTxxxDnsN+9jhJu/Vklj7tMI8lvtmuY9uyx9piaKEV5vS2aWluxI3fhEA4 8XvqPYrbiBRBUMZ8AHSreT8ToB+99WOwKRu90lgjQfQD1e1O91tNPO4PDCXlqC/0ym/c iBQ97gxsM6gt3afmN0dnutWB35ZJAYbnSY7U4sYhPYERM7c7QYiximvOCi/v0QA8c5NM FpEvPfrwtdzCyS4od5Bl5u5bp76hP4rZE4OymbfiMApQ1kbv4/RJv1stfv2MkZC0lQvz TtanyntGNwzzlcUxfVeThSbrGsI2sdBBTvG5Bk01hvoHZZCMxgWu7DQzFmPWGTk+9WLC ht7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hST2jqAwtBkEXrImHujSBwHJr715993K+qvnyWIMlgY=; b=MO1XBO1+KsQ3qGkY5nVFMYTnuMNqtzB2x3PHTeNodatvILmbWMduP7wyYLZvNPPN+5 ryHQWoKajOJ02/HPOfavp277CrCGMyZ201r0c0uqRid+59ofk8EkkdWHb5EPlzxyN1dN /eUp97f7lE+QaP6dG2aue4/AhGHIzRwRVyx/a1CdzgKfcQCL7hytTYuIoTbxlR1l6Mdc NIeQjftl+N0CkrmxXU6NEJfVRph323qPZcaOJ64IAib6k7KjOUcm4IvrGtYSEI2ew55r KFfBntGtIVcf3z3NokBChVQ6rczr50wXiAT//ExJqBpnqZ/jXoa+iFtjKh1D087vWzyg T4bQ== X-Gm-Message-State: AOAM533bLsCKJxZSOOAdnDC6wIgaiSK9Y3KexqwlqnXMeUwdy3uXvKsc tneV7xzD8FQsDAW7hcbybNe5vPRdNfC07b+OED3rABUiYPI= X-Google-Smtp-Source: ABdhPJzieoFzjdLEpsxKvOql9Eq0OcTTVqRl+envtdoy09Pr3FmuVfsZiZdnvc2M32QwglxuDeoVstB0t5FgoJo2U8M= X-Received: by 2002:a5d:62cf:: with SMTP id o15mr22403569wrv.49.1604363156716; Mon, 02 Nov 2020 16:25:56 -0800 (PST) MIME-Version: 1.0 References: <160435977209.20839.4083263519198538783@ietfa.amsl.com> In-Reply-To: <160435977209.20839.4083263519198538783@ietfa.amsl.com> From: Yuchung Cheng Date: Mon, 2 Nov 2020 16:25:19 -0800 Message-ID: To: "tcpm@ietf.org Extensions" Cc: Eric Dumazet , Kevin Yang , Neal Cardwell , iccrg IRTF list Content-Type: multipart/alternative; boundary="0000000000000c9bc005b328e850" Archived-At: Subject: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 00:26:01 -0000 --0000000000000c9bc005b328e850 Content-Type: text/plain; charset="UTF-8" Hi tcpm & iccrg, We are proposing a new TCP timestamp option to measure the network delay more precisely (e.g. excluding delayed ACK effects) for congestion control, e.g. Swift published SIGCOMM 2020. The precision of the measurements can potentially be further enhanced by NIC hardware timestamps. We are working on a reference implementation for Linux as well. We'll also present it in the upcoming tcpm meeting. Feedbacks are very welcome. ---------- Forwarded message --------- From: Date: Mon, Nov 2, 2020 at 3:29 PM Subject: New Version Notification for draft-yang-tcpm-ets-00.txt To: Eric Dumazet , Yuchung Cheng , Neal Cardwell , Kevin Yang (Yudong) A new version of I-D, draft-yang-tcpm-ets-00.txt has been successfully submitted by Kevin (Yudong) Yang and posted to the IETF repository. Name: draft-yang-tcpm-ets Revision: 00 Title: TCP ETS: Extensible Timestamp Options Document date: 2020-11-02 Group: Individual Submission Pages: 13 URL: https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt Status: https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/ Htmlized: https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets Htmlized: https://tools.ietf.org/html/draft-yang-tcpm-ets-00 Abstract: This document presents ETS: an Extensible TimeStamps option for TCP. It allows hosts to use microseconds as the unit for timestamps to improve the precision of timestamps, and advertise the maximum ACK delay for its own delayed ACK mechanism. Furthermore, it extends the information provided in the [RFC7323] TCP Timestamps Option by including the receiver delay in the TSecr echoing, so that the receiver of the ACK is able to more accurately estimate the portion of the RTT that resulted from time traveling through the network. The ETS option format is extensible, so that future extensions can add further information without the overhead of extra TCP option kind and length fields. Please note that it may take a couple of minutes from the time of submission until the htmlized version and diff are available at tools.ietf.org. The IETF Secretariat --0000000000000c9bc005b328e850 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi tcpm & iccrg,

We are = proposing a new TCP timestamp option to measure the network delay more prec= isely (e.g. excluding delayed ACK effects) for congestion control, e.g. Swi= ft published SIGCOMM 2020. The precision of the measurements can potentiall= y be further enhanced by NIC hardware timestamps. We are working on a refer= ence implementation for Linux as well.

We'll a= lso present it in the upcoming tcpm meeting. Feedbacks are very welcome.=C2= =A0

---------- Forwarded message ---------
From: <<= a href=3D"mailto:internet-drafts@ietf.org" target=3D"_blank">internet-draft= s@ietf.org>
Date: Mon, Nov 2, 2020 at 3:29 PM
Subject: = New Version Notification for draft-yang-tcpm-ets-00.txt
To: Eric Dumazet= <edumazet@goog= le.com>, Yuchung Cheng <ycheng@google.com>, Neal Cardwell <ncardwell@google.com>, Kev= in Yang (Yudong) <yy= d@google.com>



A new version of I-D, draft-yang-tcpm-ets-00.txt
has been successfully submitted by Kevin (Yudong) Yang and posted to the IETF repository.

Name:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0draft-yang-tcpm-ets
Revision:=C2=A0 =C2=A0 =C2=A0 =C2=A000
Title:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TCP ETS: Extensible Timestamp Opti= ons
Document date:=C2=A0 2020-11-02
Group:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Individual Submission
Pages:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 13
URL:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt
Status:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https:= //datatracker.ietf.org/doc/draft-yang-tcpm-ets/
Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https:/= /datatracker.ietf.org/doc/html/draft-yang-tcpm-ets
Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https://tools.= ietf.org/html/draft-yang-tcpm-ets-00


Abstract:
=C2=A0 =C2=A0This document presents ETS: an Extensible TimeStamps option fo= r TCP.
=C2=A0 =C2=A0It allows hosts to use microseconds as the unit for timestamps= to
=C2=A0 =C2=A0improve the precision of timestamps, and advertise the maximum= ACK
=C2=A0 =C2=A0delay for its own delayed ACK mechanism.=C2=A0 Furthermore, it= extends the
=C2=A0 =C2=A0information provided in the [RFC7323] TCP Timestamps Option by=
=C2=A0 =C2=A0including the receiver delay in the TSecr echoing, so that the=
=C2=A0 =C2=A0receiver of the ACK is able to more accurately estimate the po= rtion
=C2=A0 =C2=A0of the RTT that resulted from time traveling through the netwo= rk.
=C2=A0 =C2=A0The ETS option format is extensible, so that future extensions= can
=C2=A0 =C2=A0add further information without the overhead of extra TCP opti= on kind
=C2=A0 =C2=A0and length fields.




Please note that it may take a couple of minutes from the time of submissio= n
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat


--0000000000000c9bc005b328e850-- From nobody Tue Nov 3 02:05:10 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4601E3A15BC for ; Tue, 3 Nov 2020 02:05:09 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Kat9dTGIyZKk for ; Tue, 3 Nov 2020 02:05:06 -0800 (PST) Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1CC7C3A15BE for ; Tue, 3 Nov 2020 02:05:05 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id E73BF25A16; Tue, 3 Nov 2020 11:05:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1604397903; bh=1TlRcbMoRve77XExkY2itRA9dU1CsnfyDEne3feOns8=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=cA+18r0HdryRmVOR/8eeMdR2JhS8iTkClGLrbT6qZXV9hqIA4F+jQSGI3/BURZpWS orp2sfkBu/wH5xSF9KMZ8Z5PHMKgDbP4D5sDGzBD7qnVZfPcL49h+ZjkMqgAfLnuF5 Y617J8OKx2ngwElUe049XIKEWaC5itiuTmg791Aw= X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6mCrx2fejP9n; Tue, 3 Nov 2020 11:05:00 +0100 (CET) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Tue, 3 Nov 2020 11:05:00 +0100 (CET) Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Tue, 3 Nov 2020 11:05:00 +0100 Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Tue, 3 Nov 2020 11:05:00 +0100 From: "Scharf, Michael" To: Bob Briscoe , "tcpm@ietf.org" CC: "Scheffenegger, Richard" , "Mirja Kuehlewind" Thread-Topic: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt Thread-Index: AQHWsXUdzhg7shsO4UuGSaJvCge1xam2LPxA Date: Tue, 3 Nov 2020 10:04:59 +0000 Message-ID: References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [134.108.140.248] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 10:05:09 -0000 PiBNaWNoYWVsLA0KPiANCj4gMS8gSSBqdXN0IHBvc3RlZCBhIGRyYWZ0LTEzIHRoYXQgaGFzIHRo ZSB0d28gQWNjRUNOIFRDUCBPcHRpb24gb3JkZXJzLg0KPiANCj4gVGhlcmUgYXJlIGFsc28gaHRt bGl6ZWQgdmVyc2lvbnMgYXZhaWxhYmxlIGF0Og0KPiBodHRwczovL3Rvb2xzLmlldGYub3JnL2h0 bWwvZHJhZnQtaWV0Zi10Y3BtLWFjY3VyYXRlLWVjbi0xMw0KPiANCj4gQXMgYWdyZWVkLCBpdCBr ZXlzIHRoZSB0d28gb3B0aW9uIG9yZGVycyBvbiB0d28gZGlmZmVyZW50IE9wdGlvbiBLaW5kcywN Cj4gd2hpY2ggSSd2ZSByZXF1ZXN0ZWQgaW4gdGhlIElBTkEgc2VjdGlvbiAoVEJEMCBhbmQgVEJE MSkuIFRoaXMgcmVwbGFjZXMNCj4gdXNpbmcgdGhlIGluaXRpYWwgdmFsdWVzIG9mIHRoZSBmaWVs ZHMuIEkndmUgcmV2ZXJ0ZWQgYWxsIHRoZQ0KPiBjb21wbGljYXRlZCB0ZXh0IGFib3V0Og0KPiAq IHZhcnlpbmcgdGhlIGluaXRpYWwgZmllbGQgdmFsdWVzIGRlcGVuZGVudCBvbiBvcHRpb24gb3Jk ZXIsDQo+ICogaGF2aW5nIHRvIGtlZXAgY29uc2lzdGVudCBvcmRlciB0aHJvdWdob3V0IGEgY29u bmVjdGlvbiwNCj4gKiBhbmQgY2hvb3Npbmcgd2hpY2ggb3JkZXIgZnJvbSBlYWNoIGVuZC4NCj4g QWxsIG9mIHdoaWNoIGlzIGlycmVsZXZhbnQgbm93Lg0KPiANCj4gSXQgaXMgbXVjaCBzaW1wbGVy IChJIGd1ZXNzIHdlIGFsbCBrbmV3IHRoYXQpLiBUaGFuayB5b3UuDQoNClllcCwgYW5kIHNpbXBs aWNpdHkgaXMgSU1ITyBhIGJpZyBwbHVzLiBUaGlzIGFwcHJvYWNoIGlzIElNSE8gbXVjaCBiZXR0 ZXIgdGhhbiB0aGUgcHJldmlvdXMgcHJvcG9zYWwgb2Yga2VlcGluZyBzdGF0ZS4gSSdkIGxpa2Ug dG8gcmVwZWF0IHRoYXQgSSB3b3VsZCBhbHNvIGJlIGZpbmUgd2l0aCBvdGhlciBhbHRlcm5hdGl2 ZSBzb2x1dGlvbnMsIGFzIGxvbmcgYXMgdGhleSBhcmUgbm90IG92ZXJseSBjb21wbGV4Lg0KDQpB bnl3YXksIHRoaXMgbmV3IHRleHQgd29ya3MgZm9yIG1lLiANCg0KTWljaGFlbA0KDQogDQo+IEFz IG1lbnRpb25lZCwgSSBmb3VuZCBteSBvdGhlciBUb0RvIGxpc3Q6DQo+IDIvIFdlIGFsc28gaW50 cm9kdWNlZCBjb25zaWRlcmFibHkgbW9yZSBndWlkYW5jZSBvbiBtaWRkbGVib3ggaGFuZGxpbmcs DQo+IGVzcC4gQUNLIGZpbHRlcmluZyBhbmQgU2VnbWVudGF0aW9uIE9mZmxvYWQuIFRoZXJlIHdh cyBlbm91Z2ggZXh0cmEgdGV4dA0KPiB0byBiZSB3b3J0aCBicmVha2luZyB0aGF0IHNlY3Rpb24g aW50byBzdWJzZWN0aW9ucy4NCj4gDQo+IFRoZSBndWlkYW5jZSBvbiBBQ0sgZmlsdGVyaW5nIGZv ciBjb25uZWN0aW9ucyB3aXRoIEFjY0VDTiBuZWdvdGlhdGVkDQo+IHVwZGF0ZXMgdGhlIChub24t KWd1aWRhbmNlIGluIFJGQzM0NDkgKEJDUDY5IHdoaWNoIHNhaWQgQUNLIGZpbHRlcmluZyBpbg0K PiB0aGUgcHJlc2VuY2Ugb2YgRUNOIHdhcyBhIHJlc2VhcmNoIGlzc3VlKS4gQnV0IHRoZXJlIGlz IHN0aWxsIG5vDQo+IGd1aWRhbmNlIGZvciBFQ04gY29ubmVjdGlvbnMgd2hlbiBBY2NFQ04gaGFz IG5vdCBiZWVuIG5lZ290aWF0ZWQuDQo+IA0KPiAzLyBJIGFsc28gYWRkZWQgYSBTSE9VTEQgcmVx dWlyZW1lbnQgSSBoYWQgb24gbXkgVG9EbyBsaXN0OiBFYWNoIERhdGENCj4gU2VuZGVyIFNIT1VM RCBkaXNhYmxlIEVDVCBpbiBvbmUgZGlyZWN0aW9uIGlmIGFsbCB0aGUgZmVlZGJhY2sgZm9yIHRo cmVlDQo+IG9yIGZvdXIgcm91bmRzIGZyb20gdGhlIHN0YXJ0IGlzIENFLiBUaGF0J3MgYWxyZWFk eSBkb25lIGluIExpbnV4LCBhbmQNCj4gaW4gdGhlIFRDUCBpbiBpT1MgYW5kIE1hY09TIGF0IGxl YXN0Lg0KPiANCj4gDQo+IA0KPiBCb2INCj4gDQo+IA0KPiANCj4gDQo+IA0KPiBPbiAyOC8xMC8y MDIwIDE3OjE3LCBCb2IgQnJpc2NvZSB3cm90ZToNCj4gPiBNaWNoYWVsLA0KPiA+DQo+ID4gT24g MjgvMTAvMjAyMCAxNTo1NSwgU2NoYXJmLCBNaWNoYWVsIHdyb3RlOg0KPiA+PiBIaSBCb2IsDQo+ ID4+DQo+ID4+IFdoYXQgaXMgdGhlIGZ1dHVyZSBwbGFuIHJlZ2FyZGluZyB0aGUgb3B0aW9uIGVu Y29kaW5nIHByb3Bvc2VkIGJ5IHRoZQ0KPiA+PiBhdXRob3JzPw0KPiA+Pg0KPiA+PiBUaGlzIGhh cyBiZWVuIGRpc2N1c3NlZCBpbiBwYXN0IG1lZXRpbmdzIGFuZCBvbiB0aGUgbGlzdCwgZm9yDQo+ ID4+IGluc3RhbmNlOg0KPiA+PiBodHRwczovL21haWxhcmNoaXZlLmlldGYub3JnL2FyY2gvbXNn L3RjcG0vem8tDQo+IDFPUjBuUmZoSG9jWDh5dlR2cEM0Qk5Nby8NCj4gPj4NCj4gPj4gSSBoYXZl IGV4cGxhaW5lZCBzZXZlcmFsIHRpbWVzIG15IHB1c2hiYWNrIGFnYWluc3QgdGhlIGVuY29kaW5n DQo+ID4+IGN1cnJlbnRseSBkZXNjcmliZWQgaW4gdGhlIGRvY3VtZW50LCBtb3N0IG5vdGFibHkg dGhlIHJlcXVpcmVtZW50IHRvDQo+ID4+IGtlZXAgc3RhdGUgaW4gdGhlIGVuZHBvaW50cyB0byBk ZWNvZGUgdGhlIGZvcm1hdCBvZiBhIFRDUCBvcHRpb24uIEkNCj4gPj4gaGF2ZSBub3QgY2hhbmdl ZCBteSBtaW5kLg0KPiA+Pg0KPiA+PiBJIGhhdmUgYWxzbyBtZW50aW9uZWQgdGhhdCwgZm9yIGlu c3RhbmNlLCB0d28gb3B0aW9uIGNvZGVwb2ludHMgd291bGQNCj4gPj4gSU1ITyBiZSBhIGJldHRl ciBlbmdpbmVlcmluZyBhcHByb2FjaCAoIktJU1MgcHJpbmNpcGxlIikuIFRoYXQgd291bGQNCj4g Pj4gYmUgYSB2ZXJ5IHNpbXBsZSBjaGFuZ2UgdG8gdGhlIGNvbnRlbnQgdG8gdGhlIGRvY3VtZW50 LiBOZWVkbGVzcyB0bw0KPiA+PiBzYXkgdGhhdCB0aGVyZSBhcmUgb3RoZXIgYWx0ZXJuYXRpdmVz IGZvciBtYXJzaGFsbGluZyBwYXJhbWV0ZXJzIGluDQo+ID4+IGFuIHVuYW1iaWdvdXMgd2F5Lg0K PiA+DQo+ID4gW0JCXSBXaG9vcHMuIEluIG15IGhlYWQgSSBoYWQgYWxyZWFkeSBzaGlmdGVkIHRv IHlvdXIgc2NoZW1lIHdpdGgNCj4gPiB0d28tb3B0aW9uIGtpbmRzLiBCdXQgSSBzZWUgdGhhdCwg aW4gdGhlIGRyYWZ0LMKgIEkgaGF2ZW4ndC4gV2hvb3BzIC0NCj4gPiB0aGF0IHdhcyBhIGJ1ZyBp biBteSBUb0RvIGxpc3QgcHJvY2Vzc2luZy4gU29ycnkgYWJvdXQgdGhhdC4gSSdsbA0KPiA+IHN1 Ym1pdCBhbm90aGVyIGRyYWZ0IGJlZm9yZSB0aGUgZGVhZGxpbmUgd2l0aCB0aGF0IHNjaGVtZSBy ZXBsYWNpbmcNCj4gPiB0aGUgY3VycmVudCBvbmUuDQo+ID4NCj4gPiBJJ3ZlIGp1c3QgZm91bmQg c29tZSBvdGhlciBlbnRyaWVzIGluIG15IFRvRG8gbGlzdCBmb3IgdGhpcyBkcmFmdA0KPiA+IHdo aWNoIEkgbWlzc2VkIGFzIHdlbGwuIEknbGwgaW5jbHVkZSB0aGVtIHRvby4NCj4gPg0KPiA+IFNv cnJ5IGFnYWluLg0KPiA+DQo+ID4NCj4gPiBCb2INCj4gPg0KPiA+Pg0KPiA+PiBUaGVyZSBoYXZl IGJlZW4gb3RoZXIgcmVsYXRlZCBjb21tZW50cywgZS5nLiBmcm9tIElscG8uDQo+ID4+DQo+ID4+ IEJlc3QgcmVnYXJkcw0KPiA+Pg0KPiA+PiBNaWNoYWVsICh3aXRoIG5vIGhhdCkNCj4gPj4NCj4g Pj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4+PiBGcm9tOiB0Y3BtIDx0Y3BtLWJv dW5jZXNAaWV0Zi5vcmc+IE9uIEJlaGFsZiBPZiBCb2IgQnJpc2NvZQ0KPiA+Pj4gU2VudDogV2Vk bmVzZGF5LCBPY3RvYmVyIDI4LCAyMDIwIDI6MDQgUE0NCj4gPj4+IFRvOiB0Y3BtQGlldGYub3Jn DQo+ID4+PiBDYzogU2NoZWZmZW5lZ2dlciwgUmljaGFyZCA8UmljaGFyZC5TY2hlZmZlbmVnZ2Vy QG5ldGFwcC5jb20+Ow0KPiBNaXJqYQ0KPiA+Pj4gS3VlaGxld2luZCA8aWV0ZkBrdWVobGV3aW5k Lm5ldD4NCj4gPj4+IFN1YmplY3Q6IFJlOiBbdGNwbV0gSS1EIEFjdGlvbjogZHJhZnQtaWV0Zi10 Y3BtLWFjY3VyYXRlLWVjbi0xMi50eHQNCj4gPj4+DQo+ID4+PiBUQ1BNIGZvbGtzLA0KPiA+Pj4N Cj4gPj4+IEJhY2sgaW4gTWF5JzIwLCBpdCB3YXMgZGVjaWRlZCB0byBob2xkIGJhY2sgdGhlIEFj Y0VDTiAoYW5kIHRoZSBFQ04rKykNCj4gPj4+IGRyYWZ0cyBmb3IgYSB3aGlsZSB1bnRpbCB0aGUg cG9zaXRpb24gb24gTDRTIGluIHRzdndnIHdhcyBjbGVhcmVyLg0KPiA+Pj4gVGhlcmUNCj4gPj4+ IGlzIG5vIGZvcm1hbCBkZXBlbmRlbmN5IG9uIHRoZSBMNFMgZHJhZnRzIGluIHRzdndnLCBidXQg aXQgbWFkZSBzZW5zZQ0KPiA+Pj4gbm90IHRvIGJ1cm4gY29kZXBvaW50cyB3aXRob3V0IG1vcmUg Y2VydGFpbnR5Lg0KPiA+Pj4NCj4gPj4+IEkndmUganVzdCByZWZyZXNoZWQgdGhlIEFjY0VDTiBh bmQgdGhlIEVDTisrIGRyYWZ0cywgd2hpY2ggaGFkIGV4cGlyZWQuDQo+ID4+Pg0KPiA+Pj4gSSBp bmNsdWRlZCB0aGUgZm9sbG93aW5nIG1pbm9yIGNoYW5nZXMgdG8gQWNjRUNOLCB3aGljaCB5b3Ug Y2FuIHNlZSB2aWENCj4gPj4+IHRoZSBkaWZmIGxpbmsgYmVsb3csIGJ1dCBpbiBzdW1tYXJ5Og0K PiA+Pj4gKiBtb3ZlZCB3cm9uZ2x5IGNhdGVnb3JpemVkICJhY3Rpb24gaWYgdW53aWxsaW5nIHRv IHByb3ZpZGUgZi9iIiB0byB0aGUNCj4gPj4+IGdyb3VwIG9mIGJ1bGxldHMgcmVsZXZhbnQgdG8g dGhlIHJlY2VpdmVyDQo+ID4+PiAqIHJlbW92ZWQgcmVmcyB0byBsb25nLWV4cGlyZWQgZWNuLWZh bGxiYWNrIGRyYWZ0DQo+ID4+PiAqIGZpeGVkIHRhYmxlIDEgYWZ0ZXIgeG1sMnJmYyBjaGFuZ2Vk IGZvcm1hdHRpbmcNCj4gPj4+DQo+ID4+PiBUaGFua3MgdG8gdGhvc2Ugd2hvIHBvaW50ZWQgdGhl c2Ugb3V0Lg0KPiA+Pj4NCj4gPj4+IE5vIGNoYW5nZXMgdG8gRUNOKysgb3RoZXIgdGhhbiB1cGRh dGUgb2Ygc29tZSByZWZzLg0KPiA+Pj4NCj4gPj4+IENoZWVycw0KPiA+Pj4NCj4gPj4+DQo+ID4+ PiBCb2INCj4gPj4+DQo+ID4+PiBPbiAyOC8xMC8yMDIwIDEyOjQ3LCBpbnRlcm5ldC1kcmFmdHNA aWV0Zi5vcmcgd3JvdGU6DQo+ID4+Pj4gQSBOZXcgSW50ZXJuZXQtRHJhZnQgaXMgYXZhaWxhYmxl IGZyb20gdGhlIG9uLWxpbmUgSW50ZXJuZXQtRHJhZnRzDQo+ID4+Pj4gZGlyZWN0b3JpZXMuDQo+ ID4+Pj4gVGhpcyBkcmFmdCBpcyBhIHdvcmsgaXRlbSBvZiB0aGUgVENQIE1haW50ZW5hbmNlIGFu ZCBNaW5vcg0KPiA+Pj4+IEV4dGVuc2lvbnMgV0cgb2YNCj4gPj4+IHRoZSBJRVRGLg0KPiA+Pj4+ IMKgwqDCoMKgwqDCoMKgwqDCoCBUaXRsZcKgwqDCoMKgwqDCoMKgwqDCoMKgIDogTW9yZSBBY2N1 cmF0ZSBFQ04gRmVlZGJhY2sgaW4gVENQDQo+ID4+Pj4gwqDCoMKgwqDCoMKgwqDCoMKgIEF1dGhv cnPCoMKgwqDCoMKgwqDCoMKgIDogQm9iIEJyaXNjb2UNCj4gPj4+PiDCoMKgwqDCoMKgwqDCoMKg wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgTWlyamEgS8O8aGxld2luZA0K PiA+Pj4+IMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKg wqDCoCBSaWNoYXJkIFNjaGVmZmVuZWdnZXINCj4gPj4+PiDCoMKgwqDCoEZpbGVuYW1lwqDCoMKg wqDCoMKgwqAgOiBkcmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLTEyLnR4dA0KPiA+Pj4+IMKg wqDCoMKgUGFnZXPCoMKgwqDCoMKgwqDCoMKgwqDCoCA6IDU4DQo+ID4+Pj4gwqDCoMKgwqBEYXRl wqDCoMKgwqDCoMKgwqDCoMKgwqDCoCA6IDIwMjAtMTAtMjgNCj4gPj4+Pg0KPiA+Pj4+IEFic3Ry YWN0Og0KPiA+Pj4+IMKgwqDCoMKgIEV4cGxpY2l0IENvbmdlc3Rpb24gTm90aWZpY2F0aW9uIChF Q04pIGlzIGEgbWVjaGFuaXNtIHdoZXJlDQo+ID4+Pj4gbmV0d29yaw0KPiA+Pj4+IMKgwqDCoMKg IG5vZGVzIGNhbiBtYXJrIElQIHBhY2tldHMgaW5zdGVhZCBvZiBkcm9wcGluZyB0aGVtIHRvIGlu ZGljYXRlDQo+ID4+Pj4gwqDCoMKgwqAgaW5jaXBpZW50IGNvbmdlc3Rpb24gdG8gdGhlIGVuZC1w b2ludHMuwqAgUmVjZWl2ZXJzIHdpdGggYW4gRUNOLQ0KPiA+Pj4+IMKgwqDCoMKgIGNhcGFibGUg dHJhbnNwb3J0IHByb3RvY29sIGZlZWQgYmFjayB0aGlzIGluZm9ybWF0aW9uIHRvIHRoZQ0KPiA+ Pj4+IHNlbmRlci4NCj4gPj4+PiDCoMKgwqDCoCBFQ04gaXMgc3BlY2lmaWVkIGZvciBUQ1AgaW4g c3VjaCBhIHdheSB0aGF0IG9ubHkgb25lIGZlZWRiYWNrDQo+ID4+Pj4gc2lnbmFsDQo+ID4+Pj4g wqDCoMKgwqAgY2FuIGJlIHRyYW5zbWl0dGVkIHBlciBSb3VuZC1UcmlwIFRpbWUgKFJUVCkuwqAg UmVjZW50IG5ldyBUQ1ANCj4gPj4+PiDCoMKgwqDCoCBtZWNoYW5pc21zIGxpa2UgQ29uZ2VzdGlv biBFeHBvc3VyZSAoQ29uRXgpLCBEYXRhIENlbnRlciBUQ1ANCj4gPj4+PiAoRENUQ1ApDQo+ID4+ Pj4gwqDCoMKgwqAgb3IgTG93IExhdGVuY3kgTG93IExvc3MgU2NhbGFibGUgVGhyb3VnaHB1dCAo TDRTKSBuZWVkIG1vcmUNCj4gPj4+PiBhY2N1cmF0ZQ0KPiA+Pj4+IMKgwqDCoMKgIEVDTiBmZWVk YmFjayBpbmZvcm1hdGlvbiB3aGVuZXZlciBtb3JlIHRoYW4gb25lIG1hcmtpbmcgaXMNCj4gPj4+ PiByZWNlaXZlZA0KPiA+Pj4+IMKgwqDCoMKgIGluIG9uZSBSVFQuwqAgVGhpcyBkb2N1bWVudCBz cGVjaWZpZXMgYSBzY2hlbWUgdG8gcHJvdmlkZSBtb3JlDQo+ID4+Pj4gdGhhbg0KPiA+Pj4+IMKg wqDCoMKgIG9uZSBmZWVkYmFjayBzaWduYWwgcGVyIFJUVCBpbiB0aGUgVENQIGhlYWRlci7CoCBH aXZlbiBUQ1AgaGVhZGVyDQo+ID4+Pj4gwqDCoMKgwqAgc3BhY2UgaXMgc2NhcmNlLCBpdCBhbGxv Y2F0ZXMgYSByZXNlcnZlZCBoZWFkZXIgYml0LCB0aGF0IHdhcw0KPiA+Pj4+IMKgwqDCoMKgIHBy ZXZpb3VzbHkgdXNlZCBmb3IgdGhlIEVDTi1Ob25jZSB3aGljaCBoYXMgbm93IGJlZW4gZGVjbGFy ZWQNCj4gPj4+PiDCoMKgwqDCoCBoaXN0b3JpYy7CoCBJdCBhbHNvIG92ZXJsb2FkcyB0aGUgdHdv IGV4aXN0aW5nIEVDTiBmbGFncyBpbiB0aGUNCj4gPj4+PiBUQ1ANCj4gPj4+PiDCoMKgwqDCoCBo ZWFkZXIuwqAgVGhlIHJlc3VsdGluZyBleHRyYSBzcGFjZSBpcyBleHBsb2l0ZWQgdG8gZmVlZCBi YWNrDQo+ID4+Pj4gdGhlIElQLQ0KPiA+Pj4+IMKgwqDCoMKgIEVDTiBmaWVsZCByZWNlaXZlZCBk dXJpbmcgdGhlIDMtd2F5IGhhbmRzaGFrZSBhcyB3ZWxsLg0KPiA+Pj4+IFN1cHBsZW1lbnRhcnkN Cj4gPj4+PiDCoMKgwqDCoCBmZWVkYmFjayBpbmZvcm1hdGlvbiBjYW4gb3B0aW9uYWxseSBiZSBw cm92aWRlZCBpbiBhIG5ldyBUQ1ANCj4gPj4+PiBvcHRpb24sDQo+ID4+Pj4gwqDCoMKgwqAgd2hp Y2ggaXMgbmV2ZXIgdXNlZCBvbiB0aGUgVENQIFNZTi4NCj4gPj4+Pg0KPiA+Pj4+DQo+ID4+Pj4g VGhlIElFVEYgZGF0YXRyYWNrZXIgc3RhdHVzIHBhZ2UgZm9yIHRoaXMgZHJhZnQgaXM6DQo+ID4+ Pj4gaHR0cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9kb2MvZHJhZnQtaWV0Zi10Y3BtLWFjY3Vy YXRlLWVjbi8NCj4gPj4+Pg0KPiA+Pj4+IFRoZXJlIGFyZSBhbHNvIGh0bWxpemVkIHZlcnNpb25z IGF2YWlsYWJsZSBhdDoNCj4gPj4+PiBodHRwczovL3Rvb2xzLmlldGYub3JnL2h0bWwvZHJhZnQt aWV0Zi10Y3BtLWFjY3VyYXRlLWVjbi0xMg0KPiA+Pj4+IGh0dHBzOi8vZGF0YXRyYWNrZXIuaWV0 Zi5vcmcvZG9jL2h0bWwvZHJhZnQtaWV0Zi10Y3BtLWFjY3VyYXRlLWVjbi0xMg0KPiA+Pj4+DQo+ ID4+Pj4gQSBkaWZmIGZyb20gdGhlIHByZXZpb3VzIHZlcnNpb24gaXMgYXZhaWxhYmxlIGF0Og0K PiA+Pj4+IGh0dHBzOi8vd3d3LmlldGYub3JnL3JmY2RpZmY/dXJsMj1kcmFmdC1pZXRmLXRjcG0t YWNjdXJhdGUtZWNuLTEyDQo+ID4+Pj4NCj4gPj4+Pg0KPiA+Pj4+IFBsZWFzZSBub3RlIHRoYXQg aXQgbWF5IHRha2UgYSBjb3VwbGUgb2YgbWludXRlcyBmcm9tIHRoZSB0aW1lIG9mDQo+ID4+Pj4g c3VibWlzc2lvbg0KPiA+Pj4+IHVudGlsIHRoZSBodG1saXplZCB2ZXJzaW9uIGFuZCBkaWZmIGFy ZSBhdmFpbGFibGUgYXQgdG9vbHMuaWV0Zi5vcmcuDQo+ID4+Pj4NCj4gPj4+PiBJbnRlcm5ldC1E cmFmdHMgYXJlIGFsc28gYXZhaWxhYmxlIGJ5IGFub255bW91cyBGVFAgYXQ6DQo+ID4+Pj4gZnRw Oi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0cy8NCj4gPj4+Pg0KPiA+Pj4+DQo+ID4+Pj4g X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPj4+PiB0 Y3BtIG1haWxpbmcgbGlzdA0KPiA+Pj4+IHRjcG1AaWV0Zi5vcmcNCj4gPj4+PiBodHRwczovL3d3 dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3RjcG0NCj4gPj4+IC0tDQo+ID4+Pg0KPiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+ IF9fX19fXw0KPiA+Pj4gQm9iIEJyaXNjb2UgaHR0cDovL2JvYmJyaXNjb2UubmV0Lw0KPiA+Pj4g wqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqDCoMKgwqAgUFJJVklMRUdFRCBBTkQgQ09ORklERU5U SUFMDQo+ID4+Pg0KPiA+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX18NCj4gPj4+IHRjcG0gbWFpbGluZyBsaXN0DQo+ID4+PiB0Y3BtQGlldGYub3JnDQo+ ID4+PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3RjcG0NCj4gPj4gX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gPj4gdGNwbSBt YWlsaW5nIGxpc3QNCj4gPj4gdGNwbUBpZXRmLm9yZw0KPiA+PiBodHRwczovL3d3dy5pZXRmLm9y Zy9tYWlsbWFuL2xpc3RpbmZvL3RjcG0NCj4gPg0KPiANCj4gLS0NCj4gX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiBfX19fX18NCj4g Qm9iIEJyaXNjb2UgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHR0cDovL2JvYmJyaXNj b2UubmV0Lw0KDQo= From nobody Tue Nov 3 09:05:16 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D7DA03A0DDA; Tue, 3 Nov 2020 09:05:13 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mu6922a67BcB; Tue, 3 Nov 2020 09:05:12 -0800 (PST) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 608C83A0DD5; Tue, 3 Nov 2020 09:05:12 -0800 (PST) Received: by mail-pf1-x433.google.com with SMTP id e7so14731578pfn.12; Tue, 03 Nov 2020 09:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=+nVOAXW111Ar2VaCYDHzT9FJW5r3TOR0+A4B8Gu/Mfk=; b=u2W/z9IIJ0Jmk4MNOzq5AvMzQ+42xjo78ezNqCVp0+at4+TbSLvlmsmtJNjVimcNzz I9mtQL9Bs/+m3SaDabFSmYkL2N5y62yGW764ZQhUUIr8z5Y9l5HrCl1ScXsneK4UH73L S0q4qb7J0OnRHMJFQCgHqv2PxJPtRL3lVp9SqIekHpKJuJD03FlUMxz6OC7OTMeOlSuL JI8UKZDcLwynRmepbQFnyK8/JZiX8yq3QwOWZgidClKgaHOHkhsxolO/42gKS0Jzimi2 hZudskz8F0JM6KeFs+2+PLiGqymh8xbPW/o2/2TxMwwWZ+sxjIFYqo5+yJBUQtlQ9fkP /E5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=+nVOAXW111Ar2VaCYDHzT9FJW5r3TOR0+A4B8Gu/Mfk=; b=YccEI7J7GPljcoOyWRTghyfURdTSl1+oFOur8G04Pb0++0c484t42p5vpHnVk0tukE 5zFdDSvF/wyWjvkKlfX0Gk5P+QRMHAl020PrkTc4i8yBVO+FIC/VMza4y1h6O/KcQd2G au4G+1Uo1eJdTfuZufKSfimoJG4ElrTpjY733P3f3t06Qu8VewHkwrty2XlAlVIzXvgX ham8jHPj/RqdWxFCRwMzOI6oM0VVjKy9+hmHRYl9FZPhaeZCvldlzv2jwHQIg8gNcz+t Ah00XYSEzBtxYy+iqY9Vex5N5RCvyfKDFmzquDXK8xh7cpk4OpOYlf8K1whkxLZJY/l+ 82Ew== X-Gm-Message-State: AOAM532X1l4mwdHXixmdNnjPFLoTq9B26v4WmhIkkvol8Y9LpWYtK1r8 WaFftw3bRnaYjJ0ov2bYbpz6Iz3leEE= X-Google-Smtp-Source: ABdhPJw6G28kZuU+xJXeQQhA0ZtCkgWemti+l3ZEyg5RxOL+0eCtdt8Zw9APj+oAtfS3/2tR9cspYw== X-Received: by 2002:a63:c750:: with SMTP id v16mr18212355pgg.424.1604423111810; Tue, 03 Nov 2020 09:05:11 -0800 (PST) Received: from ?IPv6:2601:647:5600:5020:a509:db0b:6291:b03b? ([2601:647:5600:5020:a509:db0b:6291:b03b]) by smtp.gmail.com with ESMTPSA id p188sm15675478pgp.65.2020.11.03.09.05.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Nov 2020 09:05:10 -0800 (PST) From: Mahesh Jethanandani Message-Id: <3D34753E-E8E6-4252-A3B7-27C7A1E38BE5@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_46984346-0430-485D-96F6-F484FA3C1543" Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Date: Tue, 3 Nov 2020 09:05:10 -0800 In-Reply-To: Cc: tcpm IETF list , tcpm-chairs To: Michael Tuexen References: X-Mailer: Apple Mail (2.3608.120.23.2.4) Archived-At: Subject: Re: [tcpm] Agenda requests for TCPM@IETF109 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 17:05:14 -0000 --Apple-Mail=_46984346-0430-485D-96F6-F484FA3C1543 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I know this is coming a little late, but would like to request a 5 min = slot to discuss the following draft. > On Oct 19, 2020, at 4:23 AM, Michael Tuexen = wrote: >=20 > Dear all, >=20 > according to the preliminary schedule for IETF 109 (online), TCPM is > scheduled for Tuesday, November 17th, 2020, from 05:00 to 07:00 UTC. >=20 > If you would like to present during this meeting, please provide by = the > end of Friday, October 30th, 2020: >=20 * Title of the presentation - Updates to draft-ietf-tcpm-yang-tcp-00 * Name of the presenter - Mahesh Jethanandani * Name of the Internet Draft - draft-ietf-tcpm-yang-tcp * Time required (including Q/A) - 5 min. Thanks. > Please send us the slides for your presentations by Friday, November = 13th, 2020, > latest. >=20 > Best regards > Michael > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm Mahesh Jethanandani mjethanandani@gmail.com --Apple-Mail=_46984346-0430-485D-96F6-F484FA3C1543 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii I = know this is coming a little late, but would like to request a 5 min = slot to discuss the following draft.

On Oct = 19, 2020, at 4:23 AM, Michael Tuexen <tuexen@fh-muenster.de> wrote:

Dear = all,

according to the preliminary schedule = for IETF 109 (online), TCPM is
scheduled for Tuesday, = November 17th, 2020, from 05:00 to 07:00 UTC.

If you would like to present during this meeting, please = provide by the
end of Friday, October 30th, 2020:


* Title of the presentation - Updates to = draft-ietf-tcpm-yang-tcp-00
* Name of the presenter - = Mahesh Jethanandani
* Name of the Internet Draft - = draft-ietf-tcpm-yang-tcp
* Time required (including Q/A) - = 5 min.

Thanks.

Please send us = the slides for your presentations by Friday, November 13th, 2020,
latest.

Best regards
Michael
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

Mahesh Jethanandani
mjethanandani@gmail.com





= --Apple-Mail=_46984346-0430-485D-96F6-F484FA3C1543-- From nobody Tue Nov 3 09:11:52 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C4DF13A0DFB for ; Tue, 3 Nov 2020 09:11:50 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pkYW5DMiz0eO for ; Tue, 3 Nov 2020 09:11:49 -0800 (PST) Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4158C3A0DEA for ; Tue, 3 Nov 2020 09:11:49 -0800 (PST) Received: by mail-pg1-x52b.google.com with SMTP id w4so3942043pgg.13 for ; Tue, 03 Nov 2020 09:11:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=zpzAaROmCI4YGS3AyurMl2j3jZ3/r7W66vSFCjoBVAo=; b=Z76BKi+LRmLNDo3He6Mp81Hmfu7asW8V6RE3hxwU1yWJIXjPld3mW9Mi5eTLnXJXnR 3jAScIdZkqcODAPLzlsRNLZWUxxgtU0xYJ21Um4dqpogLiKGryGK5/BwQzTF5Ye5RTyf 1wLQ1QqZ3hTLk9f8CsOEa56x4yJvq9wY3vzknVOnrDA1jrLOcJj+YW7XaDEgUjPinl0i KElIS7IVFlh4ER3tFABccxJoFKoI+AnoZGAp6p0ZOtOFYes0Vfsq05/QjdcNWslfDHbv i4+i+nJCF3NvnceUq6LVX3rE2DSNH6yq0g8vyojcL3/IPyMtb4ZCqQWZuNedFk/ooc8d 43Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=zpzAaROmCI4YGS3AyurMl2j3jZ3/r7W66vSFCjoBVAo=; b=nHYT6AMTFtmbsyMVlaMUwyITwFIxpSdwly2RaHUSsf4sCVHty6Y3cTTMiij+FT2HJa fFWKD6/MhtiK/A9Cis/SjEzG9eiCjkM4bKrAqLGOm8yX/l8tq6hj6YzT2Aj7cuAhYU7u wWVqQV/TiG1RmXlSCpBGfVPQ5OubKtDNeNANm22r7ywiryqXvfDrobjS0iFmOrzNbZe5 0U8qaN1FTT1RhypF9ZQQ4nixF0oSSyYLgdlUQlIsOtpOt0WZejBSi+Nwdy6Yr8TpieWS yx0mYf31se/TU8bd2q8Tw1c39WP1mu4DlCoVlbHz3VQCwIoafq5VV34fslKUiw64if+S e8zQ== X-Gm-Message-State: AOAM533BO7q5bn+5wZREaWhQg5Oh+E1pUFvH1caxBymTU7ZLZH48lX8X M9HSo5Aym2eiOOIKsQ/mDeo= X-Google-Smtp-Source: ABdhPJzLjTjYxOesxENLHdyrG66nfBR6l7iRyTIw+52YiTuMvHjaLeC0sGJYuqzKUDPetkqMNHdqrg== X-Received: by 2002:a17:90a:4684:: with SMTP id z4mr131709pjf.97.1604423508774; Tue, 03 Nov 2020 09:11:48 -0800 (PST) Received: from ?IPv6:2601:647:5600:5020:8418:b85d:2a3e:6a8? ([2601:647:5600:5020:8418:b85d:2a3e:6a8]) by smtp.gmail.com with ESMTPSA id l2sm3836754pjy.6.2020.11.03.09.11.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 03 Nov 2020 09:11:47 -0800 (PST) From: Mahesh Jethanandani Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_06980C37-B0AF-43F8-83BC-8CDC6ACA9AEF" Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Date: Tue, 3 Nov 2020 09:11:47 -0800 In-Reply-To: Cc: tcpm IETF list , Michael SCHARF To: Juhamatti Kuusisaari References: X-Mailer: Apple Mail (2.3608.120.23.2.4) Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 17:11:51 -0000 --Apple-Mail=_06980C37-B0AF-43F8-83BC-8CDC6ACA9AEF Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Juhamatti, > On Nov 1, 2020, at 1:06 AM, Juhamatti Kuusisaari = wrote: >=20 > Hello, >=20 > My comments included below apply also to draft-ietf-tcpm-yang-tcp-00. >=20 > In brief: > * include-tcp-options -> ignore-tcp-options with default false > * accept-ao-mismatch -> accept-key-mismatch >=20 > BR, > -- > Juhamatti >=20 >=20 > ---------- Forwarded message --------- > From: Juhamatti Kuusisaari > Date: Fri, 18 Sep 2020 at 11:53 > Subject: [tcpm] Comments for draft-scharf-tcpm-yang-tcp-06 > To: tcpm IETF list , Scharf, Michael > >=20 >=20 > Hello, >=20 > I read through draft-scharf-tcpm-yang-tcp-06 and overall it looks fine = to me. >=20 > Nevertheless, there are a couple of items that may need > clarifications/improvements. >=20 > (1) I believe "leaf include-tcp-options" should be "leaf > ignore-tcp-options" with a false default as the options are included > by default in the RFC 5925. In my opinion, this would better emphasize > the fact that options really should be included by default and not > including them should be a special case. Change suggestion in detail > below: >=20 > leaf include-tcp-options { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Include TCP options in HMAC calculation."; > } > =3D> > leaf ignore-tcp-options { > type boolean; > default "false"; > must "../enable-ao =3D 'true'"; > description > "Ignore TCP options in MAC calculation."; > } >=20 Would it not be simple to add a =E2=80=98default true=E2=80=99 to = =E2=80=98include-tcp-options=E2=80=99 to achieve the same result? = Somehow my head does not comprehend a double negative very well :-). > Please also note the "HMAC"->"MAC" change suggestion. And yes, I do > realize that a default could be added to the original "include" leaf. > After pondering about this, I do think "ignore" leaf would be a better > end result for the reasons I mentioned above. >=20 > (2) There is now a leaf that says: >=20 > leaf accept-ao-mismatch { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Accept packets with HMAC mismatch."; > } >=20 > It is true that RFC 5925 allows non-existing MKT connections that > should be accepted. Then again, the above configuration and its > description looks to me that any mismatch would be accepted. So, maybe > a configuration setting better reflecting RFC 5925 would be something > on the lines of >=20 > leaf accept-key-mismatch { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Accept TCP segments with a Master Key Tuple (MKT) that is > not configured."; > } >=20 > As this configuration option does not have such a strong default as > the former one, I do not see a need to change its logic otherwise nor > add a default. I would assume that most security aware users would > have "false" there as a setting - especially those users that would > use a YANG model to do the configuration. I am fine with making this change. Thanks >=20 > Best regards, > -- > Juhamatti >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm Mahesh Jethanandani mjethanandani@gmail.com --Apple-Mail=_06980C37-B0AF-43F8-83BC-8CDC6ACA9AEF Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Juhamatti,

On Nov 1, 2020, at 1:06 AM, Juhamatti = Kuusisaari <juhamatk@gmail.com> wrote:

Hello,

My comments included = below apply also to draft-ietf-tcpm-yang-tcp-00.

In brief:
* include-tcp-options -> = ignore-tcp-options with default false
* = accept-ao-mismatch -> accept-key-mismatch

BR,
--
Juhamatti

---------- Forwarded message ---------
From: Juhamatti Kuusisaari <juhamatk@gmail.com>
Date: Fri, 18 Sep = 2020 at 11:53
Subject: [tcpm] Comments for = draft-scharf-tcpm-yang-tcp-06
To: tcpm IETF list <tcpm@ietf.org>, Scharf, = Michael
<Michael.Scharf@hs-esslingen.de>


Hello,

I read = through draft-scharf-tcpm-yang-tcp-06 and overall it looks fine to = me.

Nevertheless, there are a couple of = items that may need
clarifications/improvements.

(1) I believe "leaf include-tcp-options" = should be "leaf
ignore-tcp-options" with a false default = as the options are included
by default in the RFC 5925. In = my opinion, this would better emphasize
the fact that = options really should be included by default and not
including them should be a special case. Change suggestion in = detail
below:

=      leaf include-tcp-options {
=        type boolean;
=        must "../enable-ao =3D = 'true'";
=        description
=          "Include TCP = options in HMAC calculation.";
=      }
=3D>
=      leaf ignore-tcp-options {
=        type boolean;
=        default "false";
=        must "../enable-ao =3D = 'true'";
=        description
=          "Ignore TCP = options in MAC calculation.";
=      }


Would it = not be simple to add a =E2=80=98default true=E2=80=99 to = =E2=80=98include-tcp-options=E2=80=99 to achieve the same result? = Somehow my head does not comprehend a double negative very well = :-).

Please also note the "HMAC"->"MAC" change = suggestion. And yes, I do
realize that a default could be = added to the original "include" leaf.
After pondering = about this, I do think "ignore" leaf would be a better
end = result for the reasons I mentioned above.

(2)= There is now a leaf that says:

=      leaf accept-ao-mismatch {
=        type boolean;
=        must "../enable-ao =3D = 'true'";
=        description
=          "Accept packets = with HMAC mismatch.";
     }

It is true that RFC 5925 allows non-existing = MKT connections that
should be accepted. Then again, the = above configuration and its
description looks to me that = any mismatch would be accepted. So, maybe
a configuration = setting better reflecting RFC 5925 would be something
on = the lines of

=      leaf accept-key-mismatch {
=        type boolean;
=        must "../enable-ao =3D = 'true'";
=        description
=          "Accept TCP = segments with a Master Key Tuple (MKT) that is
not = configured.";
     }

As this configuration option does not have = such a strong default as
the former one, I do not see a = need to change its logic otherwise nor
add a default. I = would assume that most security aware users would
have = "false" there as a setting - especially those users that would
use a YANG model to do the configuration.

I am fine = with making this change.

Thanks


Best regards,
--
Juhamatti

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

Mahesh Jethanandani
mjethanandani@gmail.com





= --Apple-Mail=_06980C37-B0AF-43F8-83BC-8CDC6ACA9AEF-- From nobody Tue Nov 3 09:43:28 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 083703A0E58; Tue, 3 Nov 2020 09:43:27 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.613 X-Spam-Level: X-Spam-Status: No, score=-1.613 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.275, SPF_NONE=0.001, T_SPF_HELO_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0EsWKqm4JkUp; Tue, 3 Nov 2020 09:43:24 -0800 (PST) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 577FD3A0E53; Tue, 3 Nov 2020 09:43:23 -0800 (PST) Received: from mb.fritz.box (ip4d16e8c9.dynamic.kabel-deutschland.de [77.22.232.201]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 5085C721BE016; Tue, 3 Nov 2020 18:43:19 +0100 (CET) From: Michael Tuexen Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_1CBD2A3F-586C-494E-A01A-9938143A06FC"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Date: Tue, 3 Nov 2020 18:43:16 +0100 In-Reply-To: <3D34753E-E8E6-4252-A3B7-27C7A1E38BE5@gmail.com> Cc: tcpm IETF list , tcpm-chairs To: Mahesh Jethanandani References: <3D34753E-E8E6-4252-A3B7-27C7A1E38BE5@gmail.com> X-Mailer: Apple Mail (2.3608.120.23.2.4) Archived-At: Subject: Re: [tcpm] Agenda requests for TCPM@IETF109 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 17:43:27 -0000 --Apple-Mail=_1CBD2A3F-586C-494E-A01A-9938143A06FC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On 3. Nov 2020, at 18:05, Mahesh Jethanandani = wrote: >=20 > I know this is coming a little late, but would like to request a 5 min = slot to discuss the following draft. No problem. Best regards Michael >=20 >> On Oct 19, 2020, at 4:23 AM, Michael Tuexen = wrote: >>=20 >> Dear all, >>=20 >> according to the preliminary schedule for IETF 109 (online), TCPM is >> scheduled for Tuesday, November 17th, 2020, from 05:00 to 07:00 UTC. >>=20 >> If you would like to present during this meeting, please provide by = the >> end of Friday, October 30th, 2020: >>=20 >=20 > * Title of the presentation - Updates to draft-ietf-tcpm-yang-tcp-00 > * Name of the presenter - Mahesh Jethanandani > * Name of the Internet Draft - draft-ietf-tcpm-yang-tcp > * Time required (including Q/A) - 5 min. >=20 > Thanks. >=20 >> Please send us the slides for your presentations by Friday, November = 13th, 2020, >> latest. >>=20 >> Best regards >> Michael >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm >=20 > Mahesh Jethanandani > mjethanandani@gmail.com >=20 >=20 >=20 >=20 >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm --Apple-Mail=_1CBD2A3F-586C-494E-A01A-9938143A06FC Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEKow ggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYTAkRFMSsw KQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYDVQQLDBZULVN5 c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFsUm9vdCBDbGFzcyAy MB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNVBAYTAkRFMUUwQwYDVQQK EzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMg ZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlv biBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMtg1/9moUHN0vqH l4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZsFVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8F XRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0peQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+Ba L2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qL NupOkSk9s1FcragMvp0049ENF4N1xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz 9AkH4wKGMUZrAcUQDBHHWekCAwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU k+PYMiba1fFKpZFK4OpL4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYD VR0TAQH/BAgwBgEB/wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGC LB4wCAYGZ4EMAQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUv cmwvVGVsZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYB BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5jZXIw DQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4eTizDnS6 dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/MOaZ/SLick0+ hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3SPXez7vTXTf/D6OWS T1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc22CzeIs2LgtjZeOJVEqM7 h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bPZYoaorVyGTkwggWsMIIElKAD AgECAgcbY7rQHiw9MA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJERTFFMEMGA1UEChM8VmVy ZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYu MRAwDgYDVQQLEwdERk4tUEtJMS0wKwYDVQQDEyRERk4tVmVyZWluIENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IDIwHhcNMTYwNTI0MTEzODQwWhcNMzEwMjIyMjM1OTU5WjCBjTELMAkGA1UEBhMCREUx RTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5lcyBEZXV0c2NoZW4gRm9yc2NodW5n c25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMGA1UEAwwcREZOLVZlcmVpbiBHbG9i YWwgSXNzdWluZyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ07eRxH3h+Gy8Zp 1xCeOdfZojDbchwFfylfS2jxrRnWTOFrG7ELf6Gr4HuLi9gtzm6IOhDuV+UefwRRNuu6cG1joL6W LkDh0YNMZj0cZGnlm6Stcq5oOVGHecwX064vXWNxSzl660Knl5BpBb+Q/6RAcL0D57+eGIgfn5mI TQ5HjUhfZZkQ0tkqSe3BuS0dnxLLFdM/fx5ULzquk1enfnjK1UriGuXtQX1TX8izKvWKMKztFwUk P7agCwf9TRqaA1KgNpzeJIdl5Of6x5ZzJBTN0OgbaJ4YWa52fvfRCng8h0uwN89Tyjo4EPPLR22M ZD08WkVKusqAfLjz56dMTM0CAwEAAaOCAgUwggIBMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0P AQH/BAQDAgEGMCkGA1UdIAQiMCAwDQYLKwYBBAGBrSGCLB4wDwYNKwYBBAGBrSGCLAEBBDAdBgNV HQ4EFgQUazqYi/nyU4na4K2yMh4JH+iqO3QwHwYDVR0jBBgwFoAUk+PYMiba1fFKpZFK4OpL4qIM z+EwgY8GA1UdHwSBhzCBhDBAoD6gPIY6aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9v dC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDBAoD6gPIY6aHR0cDovL2NkcDIucGNhLmRmbi5kZS9n bG9iYWwtcm9vdC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDCB3QYIKwYBBQUHAQEEgdAwgc0wMwYI KwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBKBggrBgEF BQcwAoY+aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1jYS9wdWIvY2FjZXJ0 L2NhY2VydC5jcnQwSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFsLXJv b3QtZzItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCBeEWkTqR/ DlXwCbFqPnjMaDWpHPOVnj/z+N9rOHeJLI21rT7H8pTNoAauusyosa0zCLYkhmI2THhuUPDVbmCN T1IxQ5dGdfBi5G5mUcFCMWdQ5UnnOR7Ln8qGSN4IFP8VSytmm6A4nwDO/afr0X9XLchMX9wQEZc+ lgQCXISoKTlslPwQkgZ7nu7YRrQbtQMMONncsKk/cQYLsgMHM8KNSGMlJTx6e1du94oFOO+4oK4v 9NsH1VuEGMGpuEvObJAaguS5Pfp38dIfMwK/U+d2+dwmJUFvL6Yb+qQTkPp8ftkLYF3sv8pBoGH7 EUkp2KgtdRXYShjqFu9VNCIaE40GMIIF4DCCBMigAwIBAgIMIRX9tDE2QqO3mVLXMA0GCSqGSIb3 DQEBCwUAMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1ciBGb2VyZGVydW5nIGVp bmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYDVQQLDAdERk4tUEtJMSUw IwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBMB4XDTE5MDYwNDE0MjkxMFoXDTIy MDYwMzE0MjkxMFowfDELMAkGA1UEBhMCREUxIDAeBgNVBAoMF0ZhY2hob2Noc2NodWxlIE11ZW5z dGVyMTIwMAYDVQQLDClGYWNoYmVyZWljaCBFbGVrdHJvdGVjaG5payB1bmQgSW5mb3JtYXRpazEX MBUGA1UEAwwOTWljaGFlbCBUdWV4ZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM r8qQcPxLFCxzPtXvRyM9KeQaxyMA8gwUNc7IIiATs6mRQFe5ib/mvwT9nc0bAa+94go6HJDiD3FJ NkTo4u8aBsIcTt5pJtdBQLm88PLakbe3+fp/00//n7xxbTh7mAtFVCf25LxDCKkrdGk/+jglRq/R VdwhZZ3VpYCrx8YfI/hIzdRL3+4I4z/mnQ8K0X8d2MVVPG+9nBEngdnYGez5f/8wIVOgEYYBc21k yvMnVXLu2Ing+LwBd0gDG9Vasv87MNHCOZfJTNBlLhI2UDei/uNg9Zd4ynlMpPWZ7v0oiDWvmv8E OuD4oric8heyD0OYK2FL4qcVC4dn4pnyulfHAgMBAAGjggJOMIICSjA+BgNVHSAENzA1MA8GDSsG AQQBga0hgiwBAQQwEAYOKwYBBAGBrSGCLAEBBAQwEAYOKwYBBAGBrSGCLAIBBAQwCQYDVR0TBAIw ADAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQW BBTxiodBVL/lA4p5iNesIsJRhhgd6zAfBgNVHSMEGDAWgBRrOpiL+fJTidrgrbIyHgkf6Ko7dDAg BgNVHREEGTAXgRV0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0 cDovL2NkcDEucGNhLmRmbi5kZS9kZm4tY2EtZ2xvYmFsLWcyL3B1Yi9jcmwvY2FjcmwuY3JsMD+g PaA7hjlodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2Rmbi1jYS1nbG9iYWwtZzIvcHViL2NybC9jYWNy bC5jcmwwgdsGCCsGAQUFBwEBBIHOMIHLMDMGCCsGAQUFBzABhidodHRwOi8vb2NzcC5wY2EuZGZu LmRlL09DU1AtU2VydmVyL09DU1AwSQYIKwYBBQUHMAKGPWh0dHA6Ly9jZHAxLnBjYS5kZm4uZGUv ZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwSQYIKwYBBQUHMAKGPWh0dHA6 Ly9jZHAyLnBjYS5kZm4uZGUvZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQw DQYJKoZIhvcNAQELBQADggEBABs3VlmIZ1VF3HkaQdjInZYmamRabbdgJ7cz6m6o/agKL7+Vhwx7 1BaaYs2gMa5Nu/GJ3YfdqIsUlYtKdl58Yhp/89E6xBfJkItS+rE8RFdf2XgklGlx7GmsvdA3tId5 b6K6r9a5wpVN0epxY6K8wwpzEib6fJLcHylybQXZ7JSOaLRLp6WU3QPoyTT7FpvAe/0b2MSCbPX4 fc53PCn2aGgXuRFVQeCn1SP1BF3QW1ppkFhcF6G+0JcUxYFAXE6r/71WZBvUHqoG/th2hAwQnS+Y KhUYh4SZQH3/ursXXJYXQ5vyIhkN1FZlmtWA8+ofdCnoqSTbiSX2Aa/kKbTapwgxggOdMIIDmQIB ATCBnjCBjTELMAkGA1UEBhMCREUxRTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5l cyBEZXV0c2NoZW4gRm9yc2NodW5nc25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMG A1UEAwwcREZOLVZlcmVpbiBHbG9iYWwgSXNzdWluZyBDQQIMIRX9tDE2QqO3mVLXMA0GCWCGSAFl AwQCAQUAoIIBzzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMDEx MDMxNzQzMTZaMC8GCSqGSIb3DQEJBDEiBCAJHnpGZDMQ/XrmRlMnPuvlZDctzCP5J03yoEXAnNGp yDCBrwYJKwYBBAGCNxAEMYGhMIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1 ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYD VQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBAgwhFf20 MTZCo7eZUtcwgbEGCyqGSIb3DQEJEAILMYGhoIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8 VmVyZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUu IFYuMRAwDgYDVQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5n IENBAgwhFf20MTZCo7eZUtcwDQYJKoZIhvcNAQEBBQAEggEAC/T/jfia4kDWZfdQfvDcBXmNOOR+ N5EfYIp0pozQWdf5OjKvZyCPs1U4zIi7Xn1x7Ex1gYbb6HDS8aN0nAEnrHIS1UjntSNMp0B9mZhY iZT2VKBaBbYfDHvJUedt4mu/kq/Fd9IohWEnor7QRqzYfpTLiQAsTCVpu4Ii28rsrnBOzvGw/AWH aPXyLp7PdOvsD+VEiNwnQUQSii0HjTiDY2P6VtvohE4s+dcvA0bOPLocQR8ANkuqbUCsMByi1hs8 IicLXK2P4+6kPdYeFBZPK12gkcM7i+ON8IcTurTzbVztTxGq276iC9vSQxW5qXPCq3w1xy5/WAqg DhB4H2RLBgAAAAAAAA== --Apple-Mail=_1CBD2A3F-586C-494E-A01A-9938143A06FC-- From nobody Tue Nov 3 10:58:48 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C4E783A0DC1 for ; Tue, 3 Nov 2020 10:58:47 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -17.599 X-Spam-Level: X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PJd847CebY_P for ; Tue, 3 Nov 2020 10:58:45 -0800 (PST) Received: from mail-il1-x129.google.com (mail-il1-x129.google.com [IPv6:2607:f8b0:4864:20::129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5D5373A0639 for ; Tue, 3 Nov 2020 10:58:45 -0800 (PST) Received: by mail-il1-x129.google.com with SMTP id g7so17085473ilr.12 for ; Tue, 03 Nov 2020 10:58:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=A45NfcF9MivJB6j3/yqzBL5krCmlV8FKzJg8IqFhdnw=; b=RdjTn8x/piCVW0Z+oj1sk4qI32qLVcvctR876SiOBEJlVF26ID9m7wEP5MnSiFvN7o RZ9HoGBnekF2DJvJiuTnnLDhhmr5KalcVtEZC9y0tmZfXCMNtUzSZFewPwT95jkC1DbE 6lMarKGTPYE+lgEI1dvM3y2La+0KSlCp8wjoGSTyZdq84UJjLIykgBeRX3wJdMnjAeH5 /8vF7dknTH/ZlVxUgZ2TXM0tIBHOmIxcjpHNQUBYXmYd1eXGi0bRer0gJ4OHGfQTJO5a WTbw0KG5brIS1lz+8DBwZDmZzMwEtTw4VeMmv8taaZtONUVpndog3AHQQRemZfM//BJ+ xO7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=A45NfcF9MivJB6j3/yqzBL5krCmlV8FKzJg8IqFhdnw=; b=HZYYSyzuOCFaeO4HqYYK6eQd6awAxY3TYLQionhXal48Y5pB6YHXIdPIfKP4/zKp3r cep7igyY68rO7BMX3K+OLPsR8CvCJ/rlmFpATI601WNRG0ypUTkqg5BFStqeJSuxnzh6 0kBZxZwQtbxYpNJ60kqbp0ZUjL7gxxpTghLJPhJ1gfqbyvndUw9mtXl8CUfdXfWWIUBp tik6Qm2YWFINHzpCkTegrHqiN+liW3dfxN6IycZn67B2GfKcUdXWW3NuKSJkiHU6UVL2 Pe15+ioQ6dREiPTVyK8zOlGX238Nji++3dZ6fvGLY8PRTjscu9H2Rv9n3t93O/ZgN+vA UaHQ== X-Gm-Message-State: AOAM533d7Wo6NMNGsa71xb1zcJv713gLRYJaSEIrBAOxtMIi0iegIJvi TQ3JNXNh5zJReedNVUvfCnR0q7IInepzXgCDSHCGpg== X-Google-Smtp-Source: ABdhPJydDkGqSIZQGfDHt1FAjNC10fqUqBXdDDNG8QpB30X5WYvNNioG04LWjb0EKAOyws2sOdFdHhFvwzsboRGPHag= X-Received: by 2002:a92:5b46:: with SMTP id p67mr15463433ilb.150.1604429924329; Tue, 03 Nov 2020 10:58:44 -0800 (PST) MIME-Version: 1.0 References: <3D34753E-E8E6-4252-A3B7-27C7A1E38BE5@gmail.com> In-Reply-To: From: Matt Mathis Date: Tue, 3 Nov 2020 10:58:32 -0800 Message-ID: To: Michael Tuexen Cc: Mahesh Jethanandani , tcpm IETF list , tcpm-chairs Content-Type: multipart/alternative; boundary="000000000000b554ff05b338737c" Archived-At: Subject: Re: [tcpm] Agenda requests for TCPM@IETF109 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2020 18:58:48 -0000 --000000000000b554ff05b338737c Content-Type: text/plain; charset="UTF-8" Oops, I entirely missed this thread. We would like to update PRR to PS by introducing rfc6937bis as a tcpm work item. Zero changes to the algorithms as described in RFC 6937 plus the introduction of an optional heuristic to select which reduction bound. The introduction has been updated, the rest of the document has not. I don't know that there is much to say beyond what is in the introduction: do I need to make a presentation and get a hum? (and how is that done over VC anyhow?) Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay We must not tolerate intolerance; however our response must be carefully measured: too strong would be hypocritical and risks spiraling out of control; too weak risks being mistaken for tacit approval. On Tue, Nov 3, 2020 at 9:43 AM Michael Tuexen wrote: > > On 3. Nov 2020, at 18:05, Mahesh Jethanandani > wrote: > > > > I know this is coming a little late, but would like to request a 5 min > slot to discuss the following draft. > No problem. > > Best regards > Michael > > > >> On Oct 19, 2020, at 4:23 AM, Michael Tuexen > wrote: > >> > >> Dear all, > >> > >> according to the preliminary schedule for IETF 109 (online), TCPM is > >> scheduled for Tuesday, November 17th, 2020, from 05:00 to 07:00 UTC. > >> > >> If you would like to present during this meeting, please provide by the > >> end of Friday, October 30th, 2020: > >> > > > > * Title of the presentation - Updates to draft-ietf-tcpm-yang-tcp-00 > > * Name of the presenter - Mahesh Jethanandani > > * Name of the Internet Draft - draft-ietf-tcpm-yang-tcp > > * Time required (including Q/A) - 5 min. > > > > Thanks. > > > >> Please send us the slides for your presentations by Friday, November > 13th, 2020, > >> latest. > >> > >> Best regards > >> Michael > >> _______________________________________________ > >> tcpm mailing list > >> tcpm@ietf.org > >> https://www.ietf.org/mailman/listinfo/tcpm > > > > Mahesh Jethanandani > > mjethanandani@gmail.com > > > > > > > > > > > > _______________________________________________ > > tcpm mailing list > > tcpm@ietf.org > > https://www.ietf.org/mailman/listinfo/tcpm > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --000000000000b554ff05b338737c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Oops, I entirely missed this thread.

We= would like to update PRR to PS by introducing rfc6937bis as a tcpm work it= em.

Zero changes to the algorithms=C2=A0as describ= ed in RFC 6937 plus the introduction of an optional heuristic to select whi= ch reduction bound.=C2=A0 =C2=A0The introduction has been updated, the rest= of the document has not.

I don't know that th= ere is much to say beyond what is in the introduction: do I need to make a = presentation=C2=A0and get a hum?=C2=A0 (and how is that done over VC anyhow= ?)

=
Thanks,
--MM--
The best way to predict t= he future is to create it. =C2=A0- Alan Kay

We must not tolerate int= olerance;
=C2=A0 =C2=A0 =C2=A0 =C2=A0however our resp= onse must be carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 too strong would be hypocritical and risks spiraling out = of control;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak ri= sks being mistaken for tacit approval.
<= /div>

On Tue, Nov 3, 2020 at 9:43 AM Michael Tuexen <tuexen@fh-muenster.de> wrote:
<= /div>
> On 3. Nov 2020,= at 18:05, Mahesh Jethanandani <mjethanandani@gmail.com> wrote:
>
> I know this is coming a little late, but would like to request a 5 min= slot to discuss the following draft.
No problem.

Best regards
Michael
>
>> On Oct 19, 2020, at 4:23 AM, Michael Tuexen <tuexen@fh-muenster.de> wrot= e:
>>
>> Dear all,
>>
>> according to the preliminary schedule for IETF 109 (online), TCPM = is
>> scheduled for Tuesday, November 17th, 2020, from 05:00 to 07:00 UT= C.
>>
>> If you would like to present during this meeting, please provide b= y the
>> end of Friday, October 30th, 2020:
>>
>
> * Title of the presentation - Updates to draft-ietf-tcpm-yang-tcp-00 > * Name of the presenter - Mahesh Jethanandani
> * Name of the Internet Draft - draft-ietf-tcpm-yang-tcp
> * Time required (including Q/A) - 5 min.
>
> Thanks.
>
>> Please send us the slides for your presentations by Friday, Novemb= er 13th, 2020,
>> latest.
>>
>> Best regards
>> Michael
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm >
> Mahesh Jethanandani
> mjethanan= dani@gmail.com
>
>
>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--000000000000b554ff05b338737c-- From nobody Tue Nov 3 21:11:42 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9494D3A1417 for ; Tue, 3 Nov 2020 21:11:39 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UT9lquzi6Z9w for ; Tue, 3 Nov 2020 21:11:38 -0800 (PST) Received: from mail-io1-xd2c.google.com (mail-io1-xd2c.google.com [IPv6:2607:f8b0:4864:20::d2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 15B803A1418 for ; Tue, 3 Nov 2020 21:11:37 -0800 (PST) Received: by mail-io1-xd2c.google.com with SMTP id s10so6661483ioe.1 for ; Tue, 03 Nov 2020 21:11:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=njg2xofiCSn0475UAwFgE5WQzbOyOsGXXoL90LcIpk4=; b=vLKnOJv5rsDfNPpbtYQyWKSbnu4Yuq4IE+NjLVq2uaUiKEtZt+t8jKoLFH62VUHOgx hBK1kkmm6GN79T0ZF5OTYyA23IZ/0A7Giyl0FNeojqzLTN8hFllQYX6BPjuCwCkmSmWR U75a43I8zZU+NsmVUtlAvPv6BEhBBV7GFoY2as/yl2wAGH9u4l4P6zLD4RVavOngvZ3S HN9Vd2qvqlMBkHojv0v462J8v4CRCQGyaycHUBk9grGaTBN8ZMXhHXPF2/8h4OvqNyfP t8683ACw89mcnNshaAUzSnYbfcx0Y3rU+MFUnoD50+zOuDoDWR7CXvzemC8xzYeWlRsS rK9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=njg2xofiCSn0475UAwFgE5WQzbOyOsGXXoL90LcIpk4=; b=r+RrcClgozGDu7/6I3cXbfpCFUUDcgXJgFQgw7pcEIyBwBQ9oJ79O+tn7wdybl1w16 /rQcpvhxn7Zlezsbjxxknu8iWKSwMg7Y7Q9hdbeCHpMFt5mmdJXb0JRari+kWLXkDe7m KQjMbUze+MCuhy2p7MfSnCyAzWbNY5VF4ZEpuvVeIBj4oHlqRkRM9uWFO2lag6WSgJd9 zT7rSyOF1NE84FzCkTApLJrA5Xq82RtBFFNe8ipbNF7Vncb/IUDzwcYAs26UgUNHkFSS o7/F4QdJKTXW04wtL18LWvfKdzKdqRR5H4uklWsTWBm7O1u+KmzXYOVlkuXESJycN36s eFVg== X-Gm-Message-State: AOAM53121YuZ2UyB0ldU/S+zne14l2imArez1ticeft+zsqD0tNX3MTh SlQ1//aJG+arNS3yDj9ITSe0vQCxJvzJWknbclo= X-Google-Smtp-Source: ABdhPJx5RbOVMa7CEA/KCsjaASVzCKerszVIKDJTkYPn9ZuPTGVQzVq1rDuMMfuN4GMXSQ7/kqk07yAkEXK6FBZvD98= X-Received: by 2002:a6b:1646:: with SMTP id 67mr16558283iow.189.1604466697283; Tue, 03 Nov 2020 21:11:37 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Juhamatti Kuusisaari Date: Wed, 4 Nov 2020 07:11:24 +0200 Message-ID: To: Mahesh Jethanandani Cc: tcpm IETF list , Michael SCHARF Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 05:11:40 -0000 Hello Mahesh, > Would it not be simple to add a =E2=80=98default true=E2=80=99 to =E2=80= =98include-tcp-options=E2=80=99 to achieve the same result? Somehow my head= does not comprehend a double negative very well :-). Yes, =E2=80=98default true=E2=80=99 to =E2=80=98include-tcp-options=E2=80= =99 would achieve the same end result. However, I think it would guide implementers to have a special "include"-mode for options in the TCP AO implementation, especially as TCP MD5 does not have options included. There should be nothing special about including options, the special part is to ignore them. Thus, I think 'ignore-tcp-options' is the way to go. Then again, having 'default true' with =E2=80=98include-tcp-options=E2=80=99 is certain= ly an improvement and another valid choice. -- Juhamatti On Tue, 3 Nov 2020 at 19:11, Mahesh Jethanandani wrote: > > Hi Juhamatti, > > On Nov 1, 2020, at 1:06 AM, Juhamatti Kuusisaari wro= te: > > Hello, > > My comments included below apply also to draft-ietf-tcpm-yang-tcp-00. > > In brief: > * include-tcp-options -> ignore-tcp-options with default false > * accept-ao-mismatch -> accept-key-mismatch > > BR, > -- > Juhamatti > > > ---------- Forwarded message --------- > From: Juhamatti Kuusisaari > Date: Fri, 18 Sep 2020 at 11:53 > Subject: [tcpm] Comments for draft-scharf-tcpm-yang-tcp-06 > To: tcpm IETF list , Scharf, Michael > > > > Hello, > > I read through draft-scharf-tcpm-yang-tcp-06 and overall it looks fine to= me. > > Nevertheless, there are a couple of items that may need > clarifications/improvements. > > (1) I believe "leaf include-tcp-options" should be "leaf > ignore-tcp-options" with a false default as the options are included > by default in the RFC 5925. In my opinion, this would better emphasize > the fact that options really should be included by default and not > including them should be a special case. Change suggestion in detail > below: > > leaf include-tcp-options { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Include TCP options in HMAC calculation."; > } > =3D> > leaf ignore-tcp-options { > type boolean; > default "false"; > must "../enable-ao =3D 'true'"; > description > "Ignore TCP options in MAC calculation."; > } > > > Would it not be simple to add a =E2=80=98default true=E2=80=99 to =E2=80= =98include-tcp-options=E2=80=99 to achieve the same result? Somehow my head= does not comprehend a double negative very well :-). > > Please also note the "HMAC"->"MAC" change suggestion. And yes, I do > realize that a default could be added to the original "include" leaf. > After pondering about this, I do think "ignore" leaf would be a better > end result for the reasons I mentioned above. > > (2) There is now a leaf that says: > > leaf accept-ao-mismatch { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Accept packets with HMAC mismatch."; > } > > It is true that RFC 5925 allows non-existing MKT connections that > should be accepted. Then again, the above configuration and its > description looks to me that any mismatch would be accepted. So, maybe > a configuration setting better reflecting RFC 5925 would be something > on the lines of > > leaf accept-key-mismatch { > type boolean; > must "../enable-ao =3D 'true'"; > description > "Accept TCP segments with a Master Key Tuple (MKT) that is > not configured."; > } > > As this configuration option does not have such a strong default as > the former one, I do not see a need to change its logic otherwise nor > add a default. I would assume that most security aware users would > have "false" there as a setting - especially those users that would > use a YANG model to do the configuration. > > > I am fine with making this change. > > Thanks > > > Best regards, > -- > Juhamatti > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > > > Mahesh Jethanandani > mjethanandani@gmail.com > > > > > From nobody Wed Nov 4 13:36:12 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 266EE3A0B86 for ; Wed, 4 Nov 2020 13:36:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.319 X-Spam-Level: X-Spam-Status: No, score=-1.319 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=strayalpha.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BDlPhix_RdaD for ; Wed, 4 Nov 2020 13:36:09 -0800 (PST) Received: from server217-4.web-hosting.com (server217-4.web-hosting.com [198.54.116.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 73A6D3A05DF for ; Wed, 4 Nov 2020 13:36:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=strayalpha.com; s=default; h=To:References:Message-Id: Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version: Content-Type:Sender:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=IIQURh6Po+QX1K5eX40F+Uybw9ykpelTi2CLzuqYOu4=; b=23ptU7U9GwkM/m6RGK4/7wWmk grjfdJ6hgnMqilKZmWYLxdeet8MP0aOHATBsPLHscaBswR/0ecKNBgRA8TajK+3Ygkr+PmnOnpEYo U8kX7ZbmYBUrYs5/E+HXDw3COXZAb2N/st4VMi7+gmfB6EMdqdhVwv6B8DUS+vYzgNrczzx0IyJGK qcQBVbnTIEMcBZMKNRD84LtJoVvL9+Wc/nNv6f9ML1keY0Yt0PqHo9fXbE3I/FkZLAo0hTp8TYDAy J2YZOhO+8y4K1Xolxe/XcLqUGrKcd5OyKg5PNvd3zznb+bC/CwH6m6d4FPhXSmQQhLKOvbougmEos RuRWUSxSg==; Received: from cpe-172-250-225-198.socal.res.rr.com ([172.250.225.198]:61168 helo=[192.168.1.14]) by server217.web-hosting.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1kaQRo-00319D-Ur; Wed, 04 Nov 2020 16:36:09 -0500 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) From: Joseph Touch In-Reply-To: Date: Wed, 4 Nov 2020 13:36:04 -0800 Cc: Mahesh Jethanandani , tcpm IETF list Content-Transfer-Encoding: quoted-printable Message-Id: <340D363A-3C02-4A64-9DF4-A335B69CC87F@strayalpha.com> References: To: Juhamatti Kuusisaari X-Mailer: Apple Mail (2.3608.120.23.2.1) X-OutGoing-Spam-Status: No, score=0.3 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server217.web-hosting.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - strayalpha.com X-Get-Message-Sender-Via: server217.web-hosting.com: authenticated_id: touch@strayalpha.com X-Authenticated-Sender: server217.web-hosting.com: touch@strayalpha.com X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2020 21:36:11 -0000 Is there a convention in YANG for such situations? I personally would not expect all values to have =E2=80=9Cdefault=3Dtrue=E2= =80=9D or =E2=80=9Con" everywhere. Sometimes you=E2=80=99re actively = turning on something that is typically disabled (default=3Dfalse or = off). I think the =E2=80=9Cinclude-tcp-options=E2=80=9D matches the way it=E2=80= =99s described in the doc (TCP option flag, where true implies includdd) = and the default=3Dtrue would match better. Joe > On Nov 3, 2020, at 9:11 PM, Juhamatti Kuusisaari = wrote: >=20 > Hello Mahesh, >=20 >> Would it not be simple to add a =E2=80=98default true=E2=80=99 to = =E2=80=98include-tcp-options=E2=80=99 to achieve the same result? = Somehow my head does not comprehend a double negative very well :-). >=20 > Yes, =E2=80=98default true=E2=80=99 to =E2=80=98include-tcp-options=E2=80= =99 would achieve the same > end result. However, I think it would guide implementers to have a > special "include"-mode for options in the TCP AO implementation, > especially as TCP MD5 does not have options included. There should be > nothing special about including options, the special part is to ignore > them. Thus, I think 'ignore-tcp-options' is the way to go. Then again, > having 'default true' with =E2=80=98include-tcp-options=E2=80=99 is = certainly an > improvement and another valid choice. >=20 > -- > Juhamatti >=20 > On Tue, 3 Nov 2020 at 19:11, Mahesh Jethanandani > wrote: >>=20 >> Hi Juhamatti, >>=20 >> On Nov 1, 2020, at 1:06 AM, Juhamatti Kuusisaari = wrote: >>=20 >> Hello, >>=20 >> My comments included below apply also to draft-ietf-tcpm-yang-tcp-00. >>=20 >> In brief: >> * include-tcp-options -> ignore-tcp-options with default false >> * accept-ao-mismatch -> accept-key-mismatch >>=20 >> BR, >> -- >> Juhamatti >>=20 >>=20 >> ---------- Forwarded message --------- >> From: Juhamatti Kuusisaari >> Date: Fri, 18 Sep 2020 at 11:53 >> Subject: [tcpm] Comments for draft-scharf-tcpm-yang-tcp-06 >> To: tcpm IETF list , Scharf, Michael >> >>=20 >>=20 >> Hello, >>=20 >> I read through draft-scharf-tcpm-yang-tcp-06 and overall it looks = fine to me. >>=20 >> Nevertheless, there are a couple of items that may need >> clarifications/improvements. >>=20 >> (1) I believe "leaf include-tcp-options" should be "leaf >> ignore-tcp-options" with a false default as the options are included >> by default in the RFC 5925. In my opinion, this would better = emphasize >> the fact that options really should be included by default and not >> including them should be a special case. Change suggestion in detail >> below: >>=20 >> leaf include-tcp-options { >> type boolean; >> must "../enable-ao =3D 'true'"; >> description >> "Include TCP options in HMAC calculation."; >> } >> =3D> >> leaf ignore-tcp-options { >> type boolean; >> default "false"; >> must "../enable-ao =3D 'true'"; >> description >> "Ignore TCP options in MAC calculation."; >> } >>=20 >>=20 >> Would it not be simple to add a =E2=80=98default true=E2=80=99 to = =E2=80=98include-tcp-options=E2=80=99 to achieve the same result? = Somehow my head does not comprehend a double negative very well :-). >>=20 >> Please also note the "HMAC"->"MAC" change suggestion. And yes, I do >> realize that a default could be added to the original "include" leaf. >> After pondering about this, I do think "ignore" leaf would be a = better >> end result for the reasons I mentioned above. >>=20 >> (2) There is now a leaf that says: >>=20 >> leaf accept-ao-mismatch { >> type boolean; >> must "../enable-ao =3D 'true'"; >> description >> "Accept packets with HMAC mismatch."; >> } >>=20 >> It is true that RFC 5925 allows non-existing MKT connections that >> should be accepted. Then again, the above configuration and its >> description looks to me that any mismatch would be accepted. So, = maybe >> a configuration setting better reflecting RFC 5925 would be something >> on the lines of >>=20 >> leaf accept-key-mismatch { >> type boolean; >> must "../enable-ao =3D 'true'"; >> description >> "Accept TCP segments with a Master Key Tuple (MKT) that is >> not configured."; >> } >>=20 >> As this configuration option does not have such a strong default as >> the former one, I do not see a need to change its logic otherwise nor >> add a default. I would assume that most security aware users would >> have "false" there as a setting - especially those users that would >> use a YANG model to do the configuration. >>=20 >>=20 >> I am fine with making this change. >>=20 >> Thanks >>=20 >>=20 >> Best regards, >> -- >> Juhamatti >>=20 >> _______________________________________________ >> tcpm mailing list >> tcpm@ietf.org >> https://www.ietf.org/mailman/listinfo/tcpm >>=20 >>=20 >> Mahesh Jethanandani >> mjethanandani@gmail.com >>=20 >>=20 >>=20 >>=20 >>=20 >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm From nobody Thu Nov 5 03:04:23 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C64A33A0E7E for ; Thu, 5 Nov 2020 03:04:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=btconnect.onmicrosoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mlh3v8JfxHCq for ; Thu, 5 Nov 2020 03:04:12 -0800 (PST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2124.outbound.protection.outlook.com [40.107.20.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2AD2B3A0E7D for ; Thu, 5 Nov 2020 03:04:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=k+6q07V15edVYKSclAI89SE878AHTtcW8S2/SuoVL1uzImEzxgpFM8og2ruRXvMGHLAPlzo3kw5QJWI+ulJqYVM/QIz22KjdyqTXxHo4nKSiqT1V761x955QZzZU1Boo06vHK72i8tOPf2GsKrb40TMVzjYIHzToT/K/byJeh08OG8kEKp8HZIxXjWLSFgkdTtSu7ZLgSWCzlygF2Hg+IO1wlWhc9yZKLNP2WXMk5l94p27eE4Ja+TCkqh16HUbKm9pelYOOp8oYAHgig5xuiXGI6mM5vvNKDBZLACxWJKUHGxVMXNDKu/BSXMqFdA0aCC9xbC1lg6lPCjix8WDMDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wP0cUPfFaOYbLUEkwDs6IyBuXytxuuaV5OsHoQ7ooAI=; b=GRQU9rau8Zcr8CwFC0Y65qT3K6UCNVXwZyz4R+zrmfONgr4qLmAzaza2k8yo4C5P+FscQWf1q53NZbeUM1aQuOH2mcG8zh0YCxGQFh9PuFtZPSLa4zOGQc0t+Zx7OCsGEUbnTC1g7dVInCQ/CNpARQEJjd5OcsGr9sqdFiPg4JbM2eamklQxg7ItslWjp7vXQlWVeCse3ciI8IyvhzMsJAiJnSf3bQNhCQyBtZauPuSsHwJCYoaTjTwxudZ232ggrgHK8ZOXrTZaDigyosYgb/CGqIvdbDpHQf+I2Vo1NTgVn1KiCz+8XRJpraAi+hQc38gKPGQWxmT/lzBqIbWXow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=btconnect.com; dmarc=pass action=none header.from=btconnect.com; dkim=pass header.d=btconnect.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btconnect.onmicrosoft.com; s=selector2-btconnect-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wP0cUPfFaOYbLUEkwDs6IyBuXytxuuaV5OsHoQ7ooAI=; b=E4/E/SkHfSTMwwaFVNYnDsY0K8PXvSp2BtvgOTdiCLUa+RNebeAmbX+aXJxSPpA2R2VApGRcp6fngkGsWlUP7AM1tH4cQL0z2xuyfYWBzSyf/G6DnuRlJchDqreuWyvHZ6vYmsNqlj6hiLp1ffIwXnu3fuEZuSMum9w6FUKLk1k= Received: from AM6PR07MB5784.eurprd07.prod.outlook.com (2603:10a6:20b:95::29) by AS8PR07MB7463.eurprd07.prod.outlook.com (2603:10a6:20b:2a5::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.13; Thu, 5 Nov 2020 11:04:10 +0000 Received: from AM6PR07MB5784.eurprd07.prod.outlook.com ([fe80::8826:83d6:417f:4897]) by AM6PR07MB5784.eurprd07.prod.outlook.com ([fe80::8826:83d6:417f:4897%5]) with mapi id 15.20.3541.015; Thu, 5 Nov 2020 11:04:09 +0000 From: tom petch To: Joseph Touch , Juhamatti Kuusisaari CC: tcpm IETF list Thread-Topic: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 Thread-Index: AQHWsCX8yotO44JUg0uIKWvSPHlD0Km2qIaAgADJDwCAARMdAIAA4AKy Date: Thu, 5 Nov 2020 11:04:09 +0000 Message-ID: References: , <340D363A-3C02-4A64-9DF4-A335B69CC87F@strayalpha.com> In-Reply-To: <340D363A-3C02-4A64-9DF4-A335B69CC87F@strayalpha.com> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: strayalpha.com; dkim=none (message not signed) header.d=none;strayalpha.com; dmarc=none action=none header.from=btconnect.com; x-originating-ip: [86.146.121.140] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9809e221-2c11-47e9-80f7-08d8817a8585 x-ms-traffictypediagnostic: AS8PR07MB7463: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IHR1ivDDkXFe6Q3A8TxvbLaqBAjy1/XTJyWgqHv4fiVs5VuYMlIlEL4x/D1vclReKPlw/ecEm6loolGRdqJdS616+7oq+u8apB3blgP8ModwXAz6AzaEOQJ01bzHz8YguqFTl54Ddg8d7IDnwxVni28ibnPYiEkce/0OatyZDPjEoY5Zq5vEbcwswDwcz6E2rHok36ePrzOiFEY7HHv/xRg4wI+AvRJEx0jtNxXBu26Gh25rHW7Mb/GO6eTwBzdMsu01NuA056KnIO9QXzQT0zPzaHBy7bXhlMKpz35Aq9D9MIOhyupEZJJGUi2WODfytFELp/0aeCFRova8coiIUxS+oxBvLQOVJSAbsHd0Fqhy0BUowv8cj+N4W5YwhVUh4a2QLcuicce+TBt5RbiYFw== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM6PR07MB5784.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(39860400002)(346002)(136003)(396003)(376002)(316002)(4326008)(55016002)(71200400001)(9686003)(5660300002)(186003)(86362001)(83380400001)(110136005)(2906002)(76116006)(91956017)(33656002)(66946007)(8936002)(53546011)(8676002)(6506007)(66446008)(66476007)(478600001)(52536014)(26005)(64756008)(66556008)(966005)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: n0PTTt72DDMGc1casIFcEy+W4WP67YjhggGIQj0PoDYx7GN7eD1sJN6wDnqwSO7D/aPhQeX7FoUk8LbiaP3KbtXG0ubcBvTUoW/MxeQEZptQyj2rwDqLGCHdE+pmNuIfyFuAmKg83AaSRL/q0258PVQJPxALtx1UGk2x2qDclzlwCWty3eQOsOnH5xELzOYAgcBOWQ21XHgB9ETlXsq38Dh0x/n8QmqFPQPlNGQb/UC74J6b5uUOMekE7ChUzbCtr6DWDlPPHSZsV45FIm7GM3kr2OsnxXurtAbi9c5N3bCaGvAwi2Icw64zjB3SSkW2T/edddmnOcSuEFqBweq+WcxsU1BiJTqw2E1Xzg/Sw15NwdWcbTaRRf/8v7Qt4HUNxpaaGjXrfKKKFRf/+qr1ypkZhPPta6WTxYJb12tDlWoSITypsMxbDn9nHwPHlqENl94P0vWQC50bUOy8rr8+TYJ/OXHf1hCi9G5CV6wv/owU+GkNsypy1kVFFF5nG23Y3c5yC/4r+4ol7tU0kBAk18JHJWqvOHreCsH8vKrNU9PvKseTz5Wm7gceIQxtW1ZHFZUvK1a303bRyUa3i5nRa8UAvWObjC47igiLu2HDWRsWZQTcZMQUX4KP60csyQjNdJuGfKov5jloUkuup0bmykKLMnJAl2abpCJ/pPIna5EOHFa3VBnQFBkOg4UEOTd6XFraGdjCVO15RQqhYTSAaPoHEl4CU9NqPQcNSF8egoJXyx8IbKbscisnDOriUbRKdDh3V/d/1PkrvLBFksDfrZU+P4Yeue3MYkk+WhZsbu8nS9qUGxeCTyaDMP7v/Oc/HBDzqVAXIjelxRvzO3RwOJlvVUthWXswh7CJe9xd2FDvmuwPeyU6Dn+OtYck54Zuo2ZafPy3ILnsoOBz/hlgmw== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: btconnect.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM6PR07MB5784.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9809e221-2c11-47e9-80f7-08d8817a8585 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Nov 2020 11:04:09.7808 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: cf8853ed-96e5-465b-9185-806bfe185e30 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M3x914CZnSx5jIYcB03onmS0/ymWayk7yU5lzJ8z3k6AbXKv8zihxa30Nlw2INSneZze26IFKZ4rHaK1IoxF4w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR07MB7463 Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 11:04:15 -0000 From: tcpm on behalf of Joseph Touch =0A= Sent: 04 November 2020 21:36=0A= Is there a convention in YANG for such situations?=0A= =0A= I personally would not expect all values to have =93default=3Dtrue=94 or = =93on" everywhere. Sometimes you=92re actively turning on something that is= typically disabled (default=3Dfalse or off).=0A= =0A= I think the =93include-tcp-options=94 matches the way it=92s described in t= he doc (TCP option flag, where true implies includdd) and the default=3Dtru= e would match better.=0A= =0A= =0A= >From a YANG, rather than TCPM, perspective, I am not clear what you are ask= ing. In this case RFC5925 - which the leaf would benefit from as a referen= ce to - seems clear to me=0A= 'by default including options'=0A= in which case I would expect the YANG to default true.=0A= I think it good practice to be explicit in any YANG description clause of a= boolean as to what true means or false or ideally both).=0A= Also, I think the choice of 'include' in the leaf identifier unfortunate gi= ven that 'include' is a YANG technical term that appears in every YANG modu= le. I know that it used in RFC5925 but still see it as unfortunate.=0A= =0A= Tom Petch=0A= =0A= =0A= Joe=0A= =0A= > On Nov 3, 2020, at 9:11 PM, Juhamatti Kuusisaari wro= te:=0A= >=0A= > Hello Mahesh,=0A= >=0A= >> Would it not be simple to add a =91default true=92 to =91include-tcp-opt= ions=92 to achieve the same result? Somehow my head does not comprehend a d= ouble negative very well :-).=0A= >=0A= > Yes, =91default true=92 to =91include-tcp-options=92 would achieve the sa= me=0A= > end result. However, I think it would guide implementers to have a=0A= > special "include"-mode for options in the TCP AO implementation,=0A= > especially as TCP MD5 does not have options included. There should be=0A= > nothing special about including options, the special part is to ignore=0A= > them. Thus, I think 'ignore-tcp-options' is the way to go. Then again,=0A= > having 'default true' with =91include-tcp-options=92 is certainly an=0A= > improvement and another valid choice.=0A= >=0A= > --=0A= > Juhamatti=0A= >=0A= > On Tue, 3 Nov 2020 at 19:11, Mahesh Jethanandani=0A= > wrote:=0A= >>=0A= >> Hi Juhamatti,=0A= >>=0A= >> On Nov 1, 2020, at 1:06 AM, Juhamatti Kuusisaari wr= ote:=0A= >>=0A= >> Hello,=0A= >>=0A= >> My comments included below apply also to draft-ietf-tcpm-yang-tcp-00.=0A= >>=0A= >> In brief:=0A= >> * include-tcp-options -> ignore-tcp-options with default false=0A= >> * accept-ao-mismatch -> accept-key-mismatch=0A= >>=0A= >> BR,=0A= >> --=0A= >> Juhamatti=0A= >>=0A= >>=0A= >> ---------- Forwarded message ---------=0A= >> From: Juhamatti Kuusisaari =0A= >> Date: Fri, 18 Sep 2020 at 11:53=0A= >> Subject: [tcpm] Comments for draft-scharf-tcpm-yang-tcp-06=0A= >> To: tcpm IETF list , Scharf, Michael=0A= >> =0A= >>=0A= >>=0A= >> Hello,=0A= >>=0A= >> I read through draft-scharf-tcpm-yang-tcp-06 and overall it looks fine t= o me.=0A= >>=0A= >> Nevertheless, there are a couple of items that may need=0A= >> clarifications/improvements.=0A= >>=0A= >> (1) I believe "leaf include-tcp-options" should be "leaf=0A= >> ignore-tcp-options" with a false default as the options are included=0A= >> by default in the RFC 5925. In my opinion, this would better emphasize= =0A= >> the fact that options really should be included by default and not=0A= >> including them should be a special case. Change suggestion in detail=0A= >> below:=0A= >>=0A= >> leaf include-tcp-options {=0A= >> type boolean;=0A= >> must "../enable-ao =3D 'true'";=0A= >> description=0A= >> "Include TCP options in HMAC calculation.";=0A= >> }=0A= >> =3D>=0A= >> leaf ignore-tcp-options {=0A= >> type boolean;=0A= >> default "false";=0A= >> must "../enable-ao =3D 'true'";=0A= >> description=0A= >> "Ignore TCP options in MAC calculation.";=0A= >> }=0A= >>=0A= >>=0A= >> Would it not be simple to add a =91default true=92 to =91include-tcp-opt= ions=92 to achieve the same result? Somehow my head does not comprehend a d= ouble negative very well :-).=0A= >>=0A= >> Please also note the "HMAC"->"MAC" change suggestion. And yes, I do=0A= >> realize that a default could be added to the original "include" leaf.=0A= >> After pondering about this, I do think "ignore" leaf would be a better= =0A= >> end result for the reasons I mentioned above.=0A= >>=0A= >> (2) There is now a leaf that says:=0A= >>=0A= >> leaf accept-ao-mismatch {=0A= >> type boolean;=0A= >> must "../enable-ao =3D 'true'";=0A= >> description=0A= >> "Accept packets with HMAC mismatch.";=0A= >> }=0A= >>=0A= >> It is true that RFC 5925 allows non-existing MKT connections that=0A= >> should be accepted. Then again, the above configuration and its=0A= >> description looks to me that any mismatch would be accepted. So, maybe= =0A= >> a configuration setting better reflecting RFC 5925 would be something=0A= >> on the lines of=0A= >>=0A= >> leaf accept-key-mismatch {=0A= >> type boolean;=0A= >> must "../enable-ao =3D 'true'";=0A= >> description=0A= >> "Accept TCP segments with a Master Key Tuple (MKT) that is=0A= >> not configured.";=0A= >> }=0A= >>=0A= >> As this configuration option does not have such a strong default as=0A= >> the former one, I do not see a need to change its logic otherwise nor=0A= >> add a default. I would assume that most security aware users would=0A= >> have "false" there as a setting - especially those users that would=0A= >> use a YANG model to do the configuration.=0A= >>=0A= >>=0A= >> I am fine with making this change.=0A= >>=0A= >> Thanks=0A= >>=0A= >>=0A= >> Best regards,=0A= >> --=0A= >> Juhamatti=0A= >>=0A= >> _______________________________________________=0A= >> tcpm mailing list=0A= >> tcpm@ietf.org=0A= >> https://www.ietf.org/mailman/listinfo/tcpm=0A= >>=0A= >>=0A= >> Mahesh Jethanandani=0A= >> mjethanandani@gmail.com=0A= >>=0A= >>=0A= >>=0A= >>=0A= >>=0A= >=0A= > _______________________________________________=0A= > tcpm mailing list=0A= > tcpm@ietf.org=0A= > https://www.ietf.org/mailman/listinfo/tcpm=0A= =0A= _______________________________________________=0A= tcpm mailing list=0A= tcpm@ietf.org=0A= https://www.ietf.org/mailman/listinfo/tcpm=0A= From nobody Thu Nov 5 07:30:57 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B045E3A131A for ; Thu, 5 Nov 2020 07:30:56 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.319 X-Spam-Level: X-Spam-Status: No, score=-1.319 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=strayalpha.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Up-b3l3Y3r9K for ; Thu, 5 Nov 2020 07:30:55 -0800 (PST) Received: from server217-4.web-hosting.com (server217-4.web-hosting.com [198.54.116.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4BD7C3A12DF for ; Thu, 5 Nov 2020 07:30:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=strayalpha.com; s=default; h=To:References:Message-Id: Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version: Content-Type:Sender:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=KSM62n5ZIVwRZwepsbDaU93a9/pBnAwhNAQ+zoDdaaw=; b=Ksz1ZH5ee8rpNGRyv2yjU0VQI ASnxgwKZS+1GTFDUuvSPTH9aVJe4Y2eQteO3QP/ridmOAUFborFUZeXITB8ja77+nHP0P9ewF1Y/P nJM73EOBW+N2arJ0+UMzOhi/aFXmK/QI5vebzojgljYOmJC0A7NNNtK9MNnEUBbvw0akAytpsk5MA U54A5r6EVxXpttDtgoRWqp4XIua55NYvASI/PKtjfLtvzAIHwQm/HulWEB7vsfzOgSQsBJZZ/umoq JpsnIu75Bi9WAEXqg/8ICdoNoyU85Loz4GxdUNNg8LW6M0gPakwepSZ/L0G3CHMve5bK1I6F7LbUq x0Q11LzqQ==; Received: from cpe-172-250-225-198.socal.res.rr.com ([172.250.225.198]:61766 helo=[192.168.1.14]) by server217.web-hosting.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1kahDt-002AIU-BM; Thu, 05 Nov 2020 10:30:54 -0500 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) From: Joseph Touch In-Reply-To: Date: Thu, 5 Nov 2020 07:30:48 -0800 Cc: Juhamatti Kuusisaari , tcpm IETF list Content-Transfer-Encoding: quoted-printable Message-Id: References: <340D363A-3C02-4A64-9DF4-A335B69CC87F@strayalpha.com> To: tom petch X-Mailer: Apple Mail (2.3608.120.23.2.1) X-OutGoing-Spam-Status: No, score=-0.5 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server217.web-hosting.com X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - strayalpha.com X-Get-Message-Sender-Via: server217.web-hosting.com: authenticated_id: touch@strayalpha.com X-Authenticated-Sender: server217.web-hosting.com: touch@strayalpha.com X-Source: X-Source-Args: X-Source-Dir: X-From-Rewrite: unmodified, already matched Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 15:30:57 -0000 > On Nov 5, 2020, at 3:04 AM, tom petch wrote: >=20 > From: tcpm on behalf of Joseph Touch = > Sent: 04 November 2020 21:36 > Is there a convention in YANG for such situations? >=20 > I personally would not expect all values to have =E2=80=9Cdefault=3Dtrue= =E2=80=9D or =E2=80=9Con" everywhere. Sometimes you=E2=80=99re actively = turning on something that is typically disabled (default=3Dfalse or = off). >=20 > I think the =E2=80=9Cinclude-tcp-options=E2=80=9D matches the way = it=E2=80=99s described in the doc (TCP option flag, where true implies = includdd) and the default=3Dtrue would match better. >=20 > > =46rom a YANG, rather than TCPM, perspective, I am not clear what you = are asking. The question is whether toggles are typically true by default or false = by default. It doesn=E2=80=99t appear so. > In this case RFC5925 - which the leaf would benefit from as a = reference to - seems clear to me > 'by default including options' > in which case I would expect the YANG to default true. > I think it good practice to be explicit in any YANG description clause = of a boolean as to what true means or false or ideally both). > Also, I think the choice of 'include' in the leaf identifier = unfortunate given that 'include' is a YANG technical term that appears = in every YANG module. It doesn=E2=80=99t need to be; the term could be = =E2=80=9Coptions-covered-by-MAC=E2=80=9D, default=3Dtrue > I know that it used in RFC5925 but still see it as unfortunate. The YANG model and TCP-AO had overlapping development periods. It=E2=80=99= s impossible to expect either to have taken the other completely into = account. Joe From nobody Thu Nov 5 09:13:03 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 242373A186F for ; Thu, 5 Nov 2020 09:13:02 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -1.901 X-Spam-Level: X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=btconnect.onmicrosoft.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uWMne_1m9t9Y for ; Thu, 5 Nov 2020 09:13:00 -0800 (PST) Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2109.outbound.protection.outlook.com [40.107.20.109]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C7A163A1933 for ; Thu, 5 Nov 2020 09:12:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FjugT6LZ1nO1a8fD9JcppGR384Ju/UU4eLh1OLC+z9y7YEpIhf4NqxK7vooh0hau4TPBgAS1tyD7FCBWyfjbCMGXuOJgbzrHxgNyacXfDeGdSw1Pa/TvWso/qg8HuhS/nJPcWL5VV4kvF20yK/ouEjzWbyEbobifj6DgfL/w0p85fDBO+0ASXdN9UzaH5b+yoQoxwgUCATipGhatiMUvI5qiq++M7JXhehulfYalTfGHJ/vRhUyC7v2OX8lWL2kURFrly7qkFDSbQsCOTL0NKuBtRAdi7yyuB7NshxrcUR0IP9jnDPv9fbqreXShpUboxfrkZa/S/9XXGXl+RMRa6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c4W3zz5ijvUBfKYNJKj6KxwJHNROEYVnTGsFepRrOE4=; b=M+mSS3VZidRVfUK1H2k5jh6Hu8XI2jlM0hRBi+ZlKOU/iJyBEXWiTNHtL1Lp8I7Nboktwb5Oh8u8wo9RURGDatEjpSSsve1DaZEJSjfd9tbORGHRnCCInPm0pwlqLfYX2fZURKexEFL7UFLz2Fl0ZnpRsZWS7GS5NqKJuTn2go0TblpS79QlIp9/Rp9zCuK+J39A9XIjMO246VtSQv3gaWmzHCCgVVlYWjz6OpaXo7E4ka5EVPHCe7/tOyNwfFHokV8If3iyfTtnovNRat7exLGlcfSvsADPZexb2tMgDxuzJeD7d4PoNSf9BovqKdEtxggDfPN/YAkLP5iIR909nQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=btconnect.com; dmarc=pass action=none header.from=btconnect.com; dkim=pass header.d=btconnect.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=btconnect.onmicrosoft.com; s=selector2-btconnect-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=c4W3zz5ijvUBfKYNJKj6KxwJHNROEYVnTGsFepRrOE4=; b=Yd84xqir5Kg5vUdjRWYCr1akJScJ6E1U+yGdBGSZhhhxBMZAACYQ4tQnjFE4RPgKVATUSmcrskf9N63hCxpB4vspTRmMPHhxPv1eFNjRBbnCQjy+HqovM3aOQxEWsJghxKcV9erpbVz7/Vo8Qi1Dd05Rtsq8fAty76+XDPrQ4cI= Received: from AM6PR07MB5784.eurprd07.prod.outlook.com (2603:10a6:20b:95::29) by AM6PR0702MB3543.eurprd07.prod.outlook.com (2603:10a6:209:4::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.13; Thu, 5 Nov 2020 17:12:57 +0000 Received: from AM6PR07MB5784.eurprd07.prod.outlook.com ([fe80::8826:83d6:417f:4897]) by AM6PR07MB5784.eurprd07.prod.outlook.com ([fe80::8826:83d6:417f:4897%5]) with mapi id 15.20.3541.015; Thu, 5 Nov 2020 17:12:57 +0000 From: tom petch To: Joseph Touch CC: Juhamatti Kuusisaari , tcpm IETF list Thread-Topic: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 Thread-Index: AQHWsCX8yotO44JUg0uIKWvSPHlD0Km2qIaAgADJDwCAARMdAIAA4AKygABMRQCAABkYFQ== Date: Thu, 5 Nov 2020 17:12:56 +0000 Message-ID: References: <340D363A-3C02-4A64-9DF4-A335B69CC87F@strayalpha.com> , In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: strayalpha.com; dkim=none (message not signed) header.d=none;strayalpha.com; dmarc=none action=none header.from=btconnect.com; x-originating-ip: [86.146.121.140] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 10ac377d-1032-4dc0-9852-08d881ae0a59 x-ms-traffictypediagnostic: AM6PR0702MB3543: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: D8oM2DFK2Owo4ohjo1mE5A+1+BmCpO9FncE/f9ZqUncNNzg2NyxKNNPoMu60fVt8FrFkgZFCXIzFWDnIpSYmkoqBfKr4oTs+q1igqtzD4XogH2EymjZTHLGRND9teNxN/WKwnOYcSf11WOZMOfiV7c8d1jqbHU0yLXTwq7C1GKEdetqCejMPVorkaTZ8KNNTVkU9Z0PEQ3zpEKMXSSwCaT4z8HhjdBgXVUGvADDKBLLZznngdd4CUn5Wstu9goISZNlLBmdM0BlAeAFiVLKARYx5djRhkDuHFsjSDhlMwEA+yMM0DuT+ZKTkoOxEVcHIm6rgj/+vWCGEUzLsPM7eXQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM6PR07MB5784.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(39860400002)(376002)(396003)(346002)(366004)(66556008)(8676002)(26005)(83380400001)(86362001)(6916009)(7696005)(66476007)(66946007)(316002)(53546011)(64756008)(2906002)(76116006)(8936002)(66446008)(186003)(54906003)(71200400001)(91956017)(6506007)(33656002)(52536014)(55016002)(5660300002)(4326008)(9686003)(478600001); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata: yYvk++kkwZZcjf2D85t8Rf39chlWVrqAtvi95wmRf0soNoKB/tGTvyMT2UR7lU2/Iji5LvvWQ72SOHKfWt5zBwddTEg03NsXzHvLBwz7P9NJqWZhcd2dLts94dxumIGZf8Il5KWcmADiQmtlL6ILiZjgL3DBx9HjUVTDcrL8keB+sET3CKmfzqAWpV5DelCHV3V630T8ty4r/FnIcq/IYQx31EZFds3tyu3WQyBl9DOZ2lVFmcHPY6/YkJ8E2SW/rmc1Yg9Vf2Z+aumdXj4Q5yCgBIn9uNRqQBzVqDStWZtKCYyMTDVZnBz9M0hlu2wmGnbpmD76GPpECjTEPV8GyoPhAsMJSvDh2bCV5mkCD3MbfYXAQ5vBWVRSvdGZPqKCwDq2T+BV5Pfh4aQKbKAbqqSwNmxlHQSGKUUTIcAU+CyKSXKb9esbIyFcrZK1RktDnVo1oqdYZWzVdHl5X9UCPO+Cbfkbs/e0UZe9hwfmjCNB2afkoFxp1uy2NRm0nefn5IE0C7UkpnWSRqVqWQLh1l/5oX1Xk74dobCmNE9ktQbQEiLBN/NZ5oBAhCX2JP8iNeszES/aY801MbCmU2qHBZ5g7lavwyOsxva8uPEzmlZks2ZFLiAQnrujOHMzX333ags6dARohxAwwccmhp8xFbp3XHExIvhelrYz5TPzHkg0om1oeOsCzskw4DtSE62COiQGyj4kUnY3FQnfZ89BDAGRmD6alxvAmmfALO54/6VbtoFFP8vi96c0n7mDqnT4Qh5mGX0/Unj9PX9uEy1ifjCYq99JQGtL7cRblmvGz30Lk2gaNtm7JH41bNOx28i+NX6/I2YrIT+SAeZpLj8GQzRbFjwWwjzXRZBLKtocTlxXCi3k7AMF3eiMiBLt91K2LPWn6snIQUuuMVpYJSG4DA== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: btconnect.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM6PR07MB5784.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 10ac377d-1032-4dc0-9852-08d881ae0a59 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Nov 2020 17:12:56.9692 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: cf8853ed-96e5-465b-9185-806bfe185e30 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xnptprvVL9RlsTGS2QRNjbzKmasmi/onmtwXUBFGqqjSHCysvFq1NIAmf+Sd+rhOvgO1VOsSWH/bqHDgu2UnGQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR0702MB3543 Archived-At: Subject: Re: [tcpm] Comments for draft-ietf-tcpm-yang-tcp-00 X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 17:13:02 -0000 From: Joseph Touch =0A= Sent: 05 November 2020 15:30=0A= > On Nov 5, 2020, at 3:04 AM, tom petch wrote:=0A= >=0A= > From: tcpm on behalf of Joseph Touch =0A= > Sent: 04 November 2020 21:36=0A= > Is there a convention in YANG for such situations?=0A= >=0A= > I personally would not expect all values to have =93default=3Dtrue=94 or = =93on" everywhere. Sometimes you=92re actively turning on something that is= typically disabled (default=3Dfalse or off).=0A= >=0A= > I think the =93include-tcp-options=94 matches the way it=92s described in= the doc (TCP option flag, where true implies includdd) and the default=3Dt= rue would match better.=0A= >=0A= > =0A= > From a YANG, rather than TCPM, perspective, I am not clear what you are a= sking.=0A= =0A= The question is whether toggles are typically true by default or false by d= efault. It doesn=92t appear so.=0A= =0A= =0A= Ah yes, clear now. I agree that there is no such convention in YANG. Simp= lifying, if you want a default, then the default must be explicitly specifi= ed in the model. If there is no default and the leaf is not set to a value= e.g. by Netconf, then the leaf does not exist in the YANG data tree, for B= ooleans as for anything else. If you want the leaf always to be present, t= hen YANG constraints allow you to enforce that. As I said, given the wordi= ng in the RFC I would see default true as appropriate here.=0A= As an aside the issue of defaults caused much discussion in Netconf as to w= hether or not they should be returned in a get and this led to three differ= ent processing options for retrieval but the YANG side of things has stayed= the same.=0A= =0A= Tom Petch=0A= =0A= > In this case RFC5925 - which the leaf would benefit from as a reference = to - seems clear to me=0A= > 'by default including options'=0A= > in which case I would expect the YANG to default true.=0A= > I think it good practice to be explicit in any YANG description clause of= a boolean as to what true means or false or ideally both).=0A= > Also, I think the choice of 'include' in the leaf identifier unfortunate = given that 'include' is a YANG technical term that appears in every YANG mo= dule.=0A= =0A= It doesn=92t need to be; the term could be =93options-covered-by-MAC=94, de= fault=3Dtrue=0A= =0A= > I know that it used in RFC5925 but still see it as unfortunate.=0A= =0A= The YANG model and TCP-AO had overlapping development periods. It=92s impos= sible to expect either to have taken the other completely into account.=0A= =0A= Joe=0A= =0A= =0A= From prvs=571d08e49=wesleyr@amazon.com Thu Nov 5 11:32:18 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C96383A19C8 for ; Thu, 5 Nov 2020 11:32:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.599 X-Spam-Level: X-Spam-Status: No, score=-9.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=amazon.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x6DvQP7T4jJD for ; Thu, 5 Nov 2020 11:32:17 -0800 (PST) Received: from smtp-fw-33001.amazon.com (smtp-fw-33001.amazon.com [207.171.190.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A3C913A19C5 for ; Thu, 5 Nov 2020 11:32:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1604604738; x=1636140738; h=from:to:subject:date:message-id:content-id: content-transfer-encoding:mime-version; bh=bA6Y2Yzrj5edLXDuMShq9TgsUscCDgO5Pd9H/kPva9k=; b=DscJwzZLBxVp/N7GP6JOqvHK0OqqGnMSJL7gf1cty+eL7qUUuZHxwQ6C N38JcF7napon3I2bh8sHA4GbUrvZOLhiDE5CUhujqKdLK4LIoA2BfecV/ z0v0KJBx69HxIXl6/NSPH0OH7X7SB2kYs9XpyHFtabf7k6S4iU/ksIKpq Q=; X-IronPort-AV: E=Sophos;i="5.77,454,1596499200"; d="scan'208";a="91073281" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-474bcd9f.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 05 Nov 2020 19:32:10 +0000 Received: from EX13MTAUWC002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1d-474bcd9f.us-east-1.amazon.com (Postfix) with ESMTPS id C731FA2088 for ; Thu, 5 Nov 2020 19:32:09 +0000 (UTC) Received: from EX13D03UWC001.ant.amazon.com (10.43.162.136) by EX13MTAUWC002.ant.amazon.com (10.43.162.240) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 5 Nov 2020 19:32:08 +0000 Received: from EX13D03UWC001.ant.amazon.com (10.43.162.136) by EX13D03UWC001.ant.amazon.com (10.43.162.136) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 5 Nov 2020 19:32:08 +0000 Received: from EX13D03UWC001.ant.amazon.com ([10.43.162.136]) by EX13D03UWC001.ant.amazon.com ([10.43.162.136]) with mapi id 15.00.1497.006; Thu, 5 Nov 2020 19:32:08 +0000 From: "Rosenblum, Wesley" To: "tcpm@ietf.org" Thread-Topic: [tcpm] Cwnd growth after fast convergence in CUBIC Thread-Index: AQHWs6pZgRHriaRhl0KFJQo63h8sSg== Date: Thu, 5 Nov 2020 19:32:08 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/16.42.20101102 x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [10.43.162.144] Content-Type: text/plain; charset="utf-8" Content-ID: <3348514639CC02448DC533B8D21CEF65@amazon.com> Content-Transfer-Encoding: base64 MIME-Version: 1.0 Archived-At: Subject: [tcpm] Cwnd growth after fast convergence in CUBIC X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Nov 2020 19:34:22 -0000 SW4gUkZDODMxMiwgc2VjdGlvbiA0LjYsIHRoZSBmYXN0IGNvbnZlcmdlbmNlIGhldXJpc3RpYyBz cGVjaWZpZXMgdG8gZnVydGhlciByZWR1Y2UgV19tYXggd2hlbiBhIGRlY3JlYXNlIGluIGF2YWls YWJsZSBiYW5kd2lkdGggaXMgZGV0ZWN0ZWQuIEFmdGVyIFdfbWF4IGhhcyBiZWVuIGZ1cnRoZXIg cmVkdWNlZCwgV19jdWJpYygwKSB3aWxsIGJlIGxlc3MgdGhhbiBjd25kLCBzaW5jZSBXX21heCBp cyBub3cgbGVzcyB0aGFuIHRoZSBwcmV2aW91cyBjd25kLiBXaGVuIGVudGVyaW5nIGNvbmdlc3Rp b24gYXZvaWRhbmNlLCB0aGlzIGNhbiBjYXVzZSB0aGUgdGFyZ2V0IGNvbmdlc3Rpb24gd2luZG93 IHRvIGJlIGxlc3MgdGhhbiB0aGUgY3VycmVudCBjb25nZXN0aW9uIHdpbmRvdy4gRm9yIGV4YW1w bGUsIHNheSBjd25kID0gMTAwLCBiZXRhX2N1YmljID0gLjcgYW5kIGEgY29uZ2VzdGlvbiBldmVu dCBvY2N1cnM6DQoNCldfbWF4ID0gY3duZCA9IDEwMDsNCldfbWF4ID0gV19tYXgqKDEuMCtiZXRh X2N1YmljKS8yLjAgPSA4NSAgIC8vIGZ1cnRoZXIgcmVkdWNlIFdfbWF4IGZvciBmYXN0IGNvbnZl cmdlbmNlDQoNCmN3bmQgPSBjd25kICogYmV0YV9jdWJpYyA9IDcwCSAgICAgICAgICAvLyB3aW5k b3cgcmVkdWN0aW9uDQpXX2N1YmljKDApID0gV19tYXggKiBiZXRhX2N1YmljID0gNTkuNQ0KDQpJ ZiB3ZSB3ZXJlIHRvIGVudGVyIGNvbmdlc3Rpb24gYXZvaWRhbmNlIGF0IHRoaXMgcG9pbnQsIHdp dGggYSBzbWFsbCBlbm91Z2ggUlRULCB0aGUgdGFyZ2V0IGNhbmRpZGF0ZSBjb25nZXN0aW9uIHdp bmRvdyBhcyBjYWxjdWxhdGVkIGJ5IFdfY3ViaWModCtSVFQpIG1heSBiZSBsZXNzIHRoYW4gdGhl IGN1cnJlbnQgY29uZ2VzdGlvbiB3aW5kb3cgKH41OS41IDwgNzApLiANCg0KTXkgcXVlc3Rpb24g aXM6IEhvdyBzaG91bGQgdGhlIGN3bmQgYmUgYWRqdXN0ZWQgaW4gY29uZ2VzdGlvbiBhdm9pZGFu Y2Ugd2hlbiBpdCBpcyBiZWxvdyB0aGUgdGFyZ2V0IGNhbmRpZGF0ZSBjb25nZXN0aW9uIHdpbmRv dz8gSWYgd2Ugd2VyZSB0byBmb2xsb3cgc2VjdGlvbiA0LjMgYW5kIGluY3JlbWVudCBjd25kIGJ5 IChXX2N1YmljKHQrUlRUKSAtIGN3bmQpL2N3bmQsIHRoaXMgd291bGQgcmVzdWx0IGluIGEgbmVn YXRpdmUgaW5jcmVtZW50LCB3aGljaCBkb2VzbuKAmXQgbWFrZSBzZW5zZSB0byBtZS4gVGhlIExp bnV4IENVQklDIGltcGxlbWVudGF0aW9uIGFkZHMgYSDigJx2ZXJ5IHNtYWxsIGluY3JlbWVudOKA nSBpbiBzdWNoIGEgY2FzZTogc2VlIGh0dHBzOi8vZ2l0aHViLmNvbS90b3J2YWxkcy9saW51eC9i bG9iL21hc3Rlci9uZXQvaXB2NC90Y3BfY3ViaWMuYyNMMjkzLUwyOTcuIERvZXMgYW55b25lIGtu b3cgd2h5IExpbnV4IGFkZHMgdGhhdCBpbmNyZW1lbnQ/IE15IGluY2xpbmF0aW9uIGlzIHRvIGp1 c3QgbGVhdmUgdGhlIGN3bmQgYXMgaXMgYW5kIG5vdCBtYWtlIGFueSBhZGp1c3RtZW50IHdoZW4g dGFyZ2V0IGN3bmQgPD0gY3duZCwgYnV0IEnigJltIHdvbmRlcmluZyBpZiB0aGVyZSBpcyBhbnkg b3RoZXIgZ3VpZGFuY2UgZm9yIHRoaXMgY2FzZS4NCg0KVGhhbmtzLA0KV2VzbGV5DQoNCg== From nobody Thu Nov 5 20:45:38 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4AAAC3A0CCB; Thu, 5 Nov 2020 20:45:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sy1QCpgz0beL; Thu, 5 Nov 2020 20:45:29 -0800 (PST) Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9A4FE3A0C73; Thu, 5 Nov 2020 20:45:29 -0800 (PST) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 0A64c7Ft001268; Thu, 5 Nov 2020 20:45:20 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=r9BrHqYAJqCPBaIZpYiUBy6M5myL0+4E/vxodOBDoA8=; b=Spt8/MMWSOLEdHIDkWaaSm1Pk6q4lQ9pD6ttXkJP5GB25SYebxZRWsjOUnijQBe+nq2u mMfdlOARTt0joNCC4+R/EJ5CTzWzXyCm5vPKt09hCWl/txLm5kN7zxPiM3m/XWFCJ0di 4C3b+doxvcxgjcB4Hfap0U3DuOgT+fRx/Fa40bMYOznDFMLqoQTd6rxA9H3dFLPl0xxE /5NYh/DnPbzaUnwW86KyEfMUVzyUASUQ8bltyVLz0MLy+86YIak23fr7hwF5cpDHC4E1 LBA11Qjo6sqSEbNGCsjf1C+bzb2lCNGYBhi3K26r5kFCs/lcCKFo2Axm54Y1VzvYZP5+ Qg== Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 34h6k4uv3e-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 05 Nov 2020 20:45:20 -0800 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPS id <0QJC00SJ9YJJ4B50@rn-mailsvcp-mta-lapp02.rno.apple.com>; Thu, 05 Nov 2020 20:45:19 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) id <0QJC00M00YCWFI00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 05 Nov 2020 20:45:19 -0800 (PST) X-Va-A: X-Va-T-CD: 81ca60fce39c2560b6c4a7e5841f9b8f X-Va-E-CD: 64c891999a622faf2fb6ed376907e689 X-Va-R-CD: 2a50cb2984bfcd8ac8065acd6fe50879 X-Va-CD: 0 X-Va-ID: 7944ec92-be68-440c-9829-5030ac8e7eb7 X-V-A: X-V-T-CD: 81ca60fce39c2560b6c4a7e5841f9b8f X-V-E-CD: 64c891999a622faf2fb6ed376907e689 X-V-R-CD: 2a50cb2984bfcd8ac8065acd6fe50879 X-V-CD: 0 X-V-ID: 2756571a-88e6-4f5d-99ba-340a5c2b2657 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_01:2020-11-05, 2020-11-06 signatures=0 Received: from [17.150.216.191] (unknown [17.150.216.191]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPSA id <0QJC00ECJYJIIB00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 05 Nov 2020 20:45:19 -0800 (PST) From: Vidhi Goel Message-id: Content-type: multipart/alternative; boundary="Apple-Mail=_DD06A1D3-FD5F-4D0F-AEA2-FED95F846665" MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Date: Thu, 05 Nov 2020 20:45:18 -0800 In-reply-to: Cc: "tcpm@ietf.org" , Lars Eggert , iccrg@irtf.org To: "Rosenblum, Wesley" References: X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_01:2020-11-05, 2020-11-06 signatures=0 Archived-At: Subject: Re: [tcpm] Cwnd growth after fast convergence in CUBIC X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 04:45:34 -0000 --Apple-Mail=_DD06A1D3-FD5F-4D0F-AEA2-FED95F846665 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 +Lars and ICCRG Hi Wesley, I had noticed the same problem as you described when I implemented = Cubic. This is my take on computing the K factor that avoids the = situation of target cwnd <=3D cwnd. W_max =3D cwnd =3D 100 Cwnd =3D cwnd * beta =3D 70=20 W_max =3D W_max * 0.85 =3D 85 I simplified the formula to calculate K as below: K =3D cbrt(W_max*(1-beta)/C) =3D> K =3D cbrt(W_max - W_max * beta) / C) // As I already computed cwnd =3D Wmax * beta, I replace it by cwnd = (without the convergence factor) =3D> K =3D cbrt(W_max - cwnd / C) This ensures that the target cwnd is never below 70.=20 IMO, we should not be applying beta to W_max after convergence factor = has been applied in the K formula. Perhaps, it was an oversight in K=E2=80= =99s formula. Thanks, Vidhi > On Nov 5, 2020, at 11:32 AM, Rosenblum, Wesley = wrote: >=20 > In RFC8312, section 4.6, the fast convergence heuristic specifies to = further reduce W_max when a decrease in available bandwidth is detected. = After W_max has been further reduced, W_cubic(0) will be less than cwnd, = since W_max is now less than the previous cwnd. When entering congestion = avoidance, this can cause the target congestion window to be less than = the current congestion window. For example, say cwnd =3D 100, beta_cubic = =3D .7 and a congestion event occurs: >=20 > W_max =3D cwnd =3D 100; > W_max =3D W_max*(1.0+beta_cubic)/2.0 =3D 85 // further reduce W_max = for fast convergence >=20 > cwnd =3D cwnd * beta_cubic =3D 70 // window reduction > W_cubic(0) =3D W_max * beta_cubic =3D 59.5 >=20 > If we were to enter congestion avoidance at this point, with a small = enough RTT, the target candidate congestion window as calculated by = W_cubic(t+RTT) may be less than the current congestion window (~59.5 < = 70).=20 >=20 > My question is: How should the cwnd be adjusted in congestion = avoidance when it is below the target candidate congestion window? If we = were to follow section 4.3 and increment cwnd by (W_cubic(t+RTT) - = cwnd)/cwnd, this would result in a negative increment, which doesn=E2=80=99= t make sense to me. The Linux CUBIC implementation adds a =E2=80=9Cvery = small increment=E2=80=9D in such a case: see = https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c#L293-L2= 97. Does anyone know why Linux adds that increment? My inclination is to = just leave the cwnd as is and not make any adjustment when target cwnd = <=3D cwnd, but I=E2=80=99m wondering if there is any other guidance for = this case. >=20 > Thanks, > Wesley >=20 > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm --Apple-Mail=_DD06A1D3-FD5F-4D0F-AEA2-FED95F846665 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
+Lars and ICCRG

Hi = Wesley,

I = had noticed the same problem as you described when I = implemented Cubic. This is my take on computing the K factor = that avoids the situation of target cwnd <=3D cwnd.

W_max =3D cwnd =3D = 100
Cwnd =3D cwnd * beta =3D 70 

W_max =3D W_max * 0.85 =3D= 85

I = simplified the formula to calculate K as below:

K = =3D cbrt(W_max*(1-beta)/C)
=3D> K =3D cbrt(W_max= - W_max * beta) / C)

// As I already = computed cwnd =3D Wmax * beta, I replace it by cwnd (without the = convergence factor)
=3D> K =3D cbrt(W_max - cwnd / C)

This = ensures that the target cwnd is never below = 70. 

IMO, we should not be = applying beta to W_max after convergence factor has been applied in the = K formula. Perhaps, it was an oversight in K=E2=80=99s = formula.


Thanks,
Vidhi


On Nov 5, 2020, at 11:32 AM, Rosenblum, = Wesley <wesleyr=3D40amazon.com@dmarc.ietf.org> wrote:

In = RFC8312, section 4.6, the fast convergence heuristic specifies to = further reduce W_max when a decrease in available bandwidth is detected. = After W_max has been further reduced, W_cubic(0) will be less than cwnd, = since W_max is now less than the previous cwnd. When entering congestion = avoidance, this can cause the target congestion window to be less than = the current congestion window. For example, say cwnd =3D 100, beta_cubic = =3D .7 and a congestion event occurs:

W_max = =3D cwnd =3D 100;
W_max =3D W_max*(1.0+beta_cubic)/2.0 =3D = 85   // further reduce W_max for fast convergence

cwnd =3D cwnd * beta_cubic =3D 70 =          // window = reduction
W_cubic(0) =3D W_max * beta_cubic =3D 59.5

If we were to enter congestion avoidance at = this point, with a small enough RTT, the target candidate congestion = window as calculated by W_cubic(t+RTT) may be less than the current = congestion window (~59.5 < 70).

My = question is: How should the cwnd be adjusted in congestion avoidance = when it is below the target candidate congestion window? If we were to = follow section 4.3 and increment cwnd by (W_cubic(t+RTT) - cwnd)/cwnd, = this would result in a negative increment, which doesn=E2=80=99t make = sense to me. The Linux CUBIC implementation adds a =E2=80=9Cvery small = increment=E2=80=9D in such a case: see https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubi= c.c#L293-L297. Does anyone know why Linux adds that increment? My = inclination is to just leave the cwnd as is and not make any adjustment = when target cwnd <=3D cwnd, but I=E2=80=99m wondering if there is any = other guidance for this case.

Thanks,
Wesley

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

= --Apple-Mail=_DD06A1D3-FD5F-4D0F-AEA2-FED95F846665-- From nobody Fri Nov 6 00:28:33 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 22DF33A0EA8 for ; Fri, 6 Nov 2020 00:28:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.099 X-Spam-Level: X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=eggert.org Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5fTgeqvXmzdB for ; Fri, 6 Nov 2020 00:28:27 -0800 (PST) Received: from mail.eggert.org (mail.eggert.org [91.190.195.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D82AC3A0EB3 for ; Fri, 6 Nov 2020 00:28:26 -0800 (PST) Received: from [IPv6:2a00:ac00:4000:400:1d5b:6b9d:45f1:2c5b] (unknown [IPv6:2a00:ac00:4000:400:1d5b:6b9d:45f1:2c5b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.eggert.org (Postfix) with ESMTPSA id 09AB56103D6; Fri, 6 Nov 2020 10:28:16 +0200 (EET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eggert.org; s=dkim; t=1604651297; bh=n/N7oTfEFfGYC6aiRqekLXBsr3n+OSIJAmCY5SrzE1w=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=aoHfZrd3OBn3ikfA0p7L1ees4ODNS26EHxfu05k4iyuE/XhY5gPVUdObncktzkTDQ sifKQeSPhYm1irXA/gPDz4iDdWtSUGfW7Ob9OvQ3ad0N1tkzIQEb43ap+xxB8jIjjI lnI50N5+BbzWZmKbr3jp9EKDEaA/Folm+Ezx6KIA= From: Lars Eggert Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_2D1B332D-E9BC-4A9A-93CC-2BA541A9C8DD"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Date: Fri, 6 Nov 2020 10:28:16 +0200 In-Reply-To: Cc: "Rosenblum, Wesley" , "tcpm@ietf.org" , iccrg@irtf.org, Martin Thomson To: Vidhi Goel References: X-MailScanner-ID: 09AB56103D6.A3098 X-MailScanner: Found to be clean X-MailScanner-From: lars@eggert.org Archived-At: Subject: Re: [tcpm] [iccrg] Cwnd growth after fast convergence in CUBIC X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 08:28:28 -0000 --Apple-Mail=_2D1B332D-E9BC-4A9A-93CC-2BA541A9C8DD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, On 2020-11-6, at 6:45, Vidhi Goel = wrote: > IMO, we should not be applying beta to W_max after convergence factor = has been applied in the K formula. Perhaps, it was an oversight in K=E2=80= =99s formula. the other authors of the CUBIC RFC are experts on the algorithm, I = mostly just helped get it through the IETF process. That said, the intent was to document what the Linux code was doing at = the time. Is there a difference between the RFC and what Linux has been = doing or is doing now? If yes, it might be time to revise the RFC. (I think Martin Thomson also found some nits recently.) Thanks, Lars --Apple-Mail=_2D1B332D-E9BC-4A9A-93CC-2BA541A9C8DD Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEmpq0ZpSoejRmyhheVLXDCb9wwVcFAl+lCSAACgkQVLXDCb9w wVcqzQ//f/0OkJZH7YViZksJkbrNXAZvDU9sWsB7Zv5MLjM6LhXKcQgjMGMOQJxM //bwIZ1mCbqDsuoJKasx6TIVMy/kjpBfFxp3ap3LmfVammfIa9I59qUM84gjAUDj 8Nz78iY5FMMbAc9H/pAP9rV6nYohSmat8dvodNLkKKxutSB13Ac5olvIvULji2AW FEtkPrZzMub0MPRpmi8jyRBkh6m0kMHf4+1VIzNtKV6/IyiFWQrGYZURk9lR05I+ qQoGKTcAhO76/SimIhTiplJUiSGFy3kn8UaozX+ce5s298nPa/uslD0QvoPecI14 yIJi8bcOK5U85ysJ23j5Xjiu+2l7b3a98+83ktUd6j41cpCx3XMCH3uCYIczV81v yhNVK65tXAERX648LtRY/kRznlv4Qy2ocF1/c0aL55kkPBzviHHiDivBYDN51ya8 9jQgVSoRHObdLn7IXHt79KvGTCIFFoGGQfMfCYmXPrCwtasAuKPwmjNDsH/+PG5e nv0Zcy4NJKkhGm6CzLcwd5JvU0umD2L0hrpCUpWtt8gBSjOwacv28ekj7vxmkMsu /n+O6sb01jbfszHaALAWzLJyszvCVXbZdkGvr9bsWncQeDqhf7hKzCzCHC9EC7NZ iindouSwAmE+AL85JpqIUv3TEodYTSObNmRY/8rKkYlzMLdhzT0= =J6L4 -----END PGP SIGNATURE----- --Apple-Mail=_2D1B332D-E9BC-4A9A-93CC-2BA541A9C8DD-- From nobody Fri Nov 6 00:54:15 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 272193A0EF6 for ; Fri, 6 Nov 2020 00:54:14 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id krbximcJgEGD for ; Fri, 6 Nov 2020 00:54:10 -0800 (PST) Received: from mail-qt1-x835.google.com (mail-qt1-x835.google.com [IPv6:2607:f8b0:4864:20::835]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CEF1D3A0EF5 for ; Fri, 6 Nov 2020 00:54:09 -0800 (PST) Received: by mail-qt1-x835.google.com with SMTP id t5so288531qtp.2 for ; Fri, 06 Nov 2020 00:54:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=E+KBs/aVKYAUOmaBCN1jimxAyQUEOwPabXUMqwSov3E=; b=TMizsD3Bv1hS84Idf18cPFBgh6+WSGIfKRnCkC/pig4MjKBl3TICqUABn5cITB/s8Y w5ogLi8ZRENnzWDZ3/HvnuqETnwGCgx3+Mz8zqqBz1iVx4QZ2bVrQjvil0JUKl5pNIg7 lp+ncb8zctEslBfWGuQJ/fVawzlX4V65/ceDiRXxjqSe2Sghreg8aFRUSdUnO+VcqTG3 lOUu8qs6xm2TvYA+eLdpedU6S65pnSreWR1VoSOjcYkP2l7ncS9CNNnWcdo8CbxH4iu6 ZPLe3iDtpRHezAWDv6dEkQpxWEgorhwhw1uPS6Bg00k1aa/SY16k78y2kkUunkLWvF7F 4UAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=E+KBs/aVKYAUOmaBCN1jimxAyQUEOwPabXUMqwSov3E=; b=mW/0S4O7NyA368CyFazrMudzu1RZBUnJP2UkzqJYp6ABvAql02KXZV6kOBRQbCwG85 Fd4wrvrww/t06O1OyKq5aia0z623TVGcQtIGJyA53XlqCj9KvjT+ol6mD6zK8mzur7zC j/l7uX0w5aVunmgp591gMGChD02E9iLpUqfpMwxGu3EXfLGW9p8FFxq0jVmuKcMom8yp kCagFmHzI/PPPo8tkTyscZCG7C1YeP5cXv2w9ch17tI0xySAPBuyKlLuxeV5U2m4wirE eStLNTEAGgMZkda3pZSkbd+nXc6yr34g6sd5KcrH3F5TkZMhGqu6pqpkfgKRDu6MC0HQ xxQA== X-Gm-Message-State: AOAM531I7zf8m0TyyNygg9lUQ9dUd1JUuf9LjpBOfVqxmocPldmge2Sm EDb3nOtAfu+8M6cRwO3kvqIgd+KBAPInAUQW2oI= X-Google-Smtp-Source: ABdhPJx1zk2FvpMSNqvSmTbftOhmV7JUNn1ybRFqR9dpFNc8o9iwQLPP+SLzWF1R4R/TwgmIDKqj/AM/AUkdpBxJ7rU= X-Received: by 2002:ac8:5a8c:: with SMTP id c12mr548104qtc.310.1604652848823; Fri, 06 Nov 2020 00:54:08 -0800 (PST) MIME-Version: 1.0 References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> In-Reply-To: From: Yoshifumi Nishida Date: Fri, 6 Nov 2020 00:53:57 -0800 Message-ID: To: Bob Briscoe Cc: "Scharf, Michael" , "tcpm@ietf.org" , "Scheffenegger, Richard" , Mirja Kuehlewind Content-Type: multipart/alternative; boundary="0000000000000ac13005b36c5b65" Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 08:54:14 -0000 --0000000000000ac13005b36c5b65 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Bob, Sorry for the late comments, I have a naive question about the encoding schemes in the options. The byte counters will be incremented by every byte the receiver receives. But, do we really need this level of granularity? I am thinking if we can byte-shift the counter values to some extent, we might be able to save option space. Thanks, -- Yoshi On Mon, Nov 2, 2020 at 4:09 PM Bob Briscoe wrote: > Michael, > > 1/ I just posted a draft-13 that has the two AccECN TCP Option orders. > > There are also htmlized versions available at: > https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13 > > As agreed, it keys the two option orders on two different Option Kinds, > which I've requested in the IANA section (TBD0 and TBD1). This replaces > using the initial values of the fields. I've reverted all the > complicated text about: > * varying the initial field values dependent on option order, > * having to keep consistent order throughout a connection, > * and choosing which order from each end. > All of which is irrelevant now. > > It is much simpler (I guess we all knew that). Thank you. > > As mentioned, I found my other ToDo list: > 2/ We also introduced considerably more guidance on middlebox handling, > esp. ACK filtering and Segmentation Offload. There was enough extra text > to be worth breaking that section into subsections. > > The guidance on ACK filtering for connections with AccECN negotiated > updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in > the presence of ECN was a research issue). But there is still no > guidance for ECN connections when AccECN has not been negotiated. > > 3/ I also added a SHOULD requirement I had on my ToDo list: Each Data > Sender SHOULD disable ECT in one direction if all the feedback for three > or four rounds from the start is CE. That's already done in Linux, and > in the TCP in iOS and MacOS at least. > > > > Bob > > > > > > On 28/10/2020 17:17, Bob Briscoe wrote: > > Michael, > > > > On 28/10/2020 15:55, Scharf, Michael wrote: > >> Hi Bob, > >> > >> What is the future plan regarding the option encoding proposed by the > >> authors? > >> > >> This has been discussed in past meetings and on the list, for > >> instance: > >> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo= / > >> > >> I have explained several times my pushback against the encoding > >> currently described in the document, most notably the requirement to > >> keep state in the endpoints to decode the format of a TCP option. I > >> have not changed my mind. > >> > >> I have also mentioned that, for instance, two option codepoints would > >> IMHO be a better engineering approach ("KISS principle"). That would > >> be a very simple change to the content to the document. Needless to > >> say that there are other alternatives for marshalling parameters in > >> an unambigous way. > > > > [BB] Whoops. In my head I had already shifted to your scheme with > > two-option kinds. But I see that, in the draft, I haven't. Whoops - > > that was a bug in my ToDo list processing. Sorry about that. I'll > > submit another draft before the deadline with that scheme replacing > > the current one. > > > > I've just found some other entries in my ToDo list for this draft > > which I missed as well. I'll include them too. > > > > Sorry again. > > > > > > Bob > > > >> > >> There have been other related comments, e.g. from Ilpo. > >> > >> Best regards > >> > >> Michael (with no hat) > >> > >>> -----Original Message----- > >>> From: tcpm On Behalf Of Bob Briscoe > >>> Sent: Wednesday, October 28, 2020 2:04 PM > >>> To: tcpm@ietf.org > >>> Cc: Scheffenegger, Richard ; Mirja > >>> Kuehlewind > >>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-12.txt > >>> > >>> TCPM folks, > >>> > >>> Back in May'20, it was decided to hold back the AccECN (and the ECN++= ) > >>> drafts for a while until the position on L4S in tsvwg was clearer. > >>> There > >>> is no formal dependency on the L4S drafts in tsvwg, but it made sense > >>> not to burn codepoints without more certainty. > >>> > >>> I've just refreshed the AccECN and the ECN++ drafts, which had expire= d. > >>> > >>> I included the following minor changes to AccECN, which you can see v= ia > >>> the diff link below, but in summary: > >>> * moved wrongly categorized "action if unwilling to provide f/b" to t= he > >>> group of bullets relevant to the receiver > >>> * removed refs to long-expired ecn-fallback draft > >>> * fixed table 1 after xml2rfc changed formatting > >>> > >>> Thanks to those who pointed these out. > >>> > >>> No changes to ECN++ other than update of some refs. > >>> > >>> Cheers > >>> > >>> > >>> Bob > >>> > >>> On 28/10/2020 12:47, internet-drafts@ietf.org wrote: > >>>> A New Internet-Draft is available from the on-line Internet-Drafts > >>>> directories. > >>>> This draft is a work item of the TCP Maintenance and Minor > >>>> Extensions WG of > >>> the IETF. > >>>> Title : More Accurate ECN Feedback in TCP > >>>> Authors : Bob Briscoe > >>>> Mirja K=C3=BChlewind > >>>> Richard Scheffenegger > >>>> Filename : draft-ietf-tcpm-accurate-ecn-12.txt > >>>> Pages : 58 > >>>> Date : 2020-10-28 > >>>> > >>>> Abstract: > >>>> Explicit Congestion Notification (ECN) is a mechanism where > >>>> network > >>>> nodes can mark IP packets instead of dropping them to indicate > >>>> incipient congestion to the end-points. Receivers with an ECN- > >>>> capable transport protocol feed back this information to the > >>>> sender. > >>>> ECN is specified for TCP in such a way that only one feedback > >>>> signal > >>>> can be transmitted per Round-Trip Time (RTT). Recent new TCP > >>>> mechanisms like Congestion Exposure (ConEx), Data Center TCP > >>>> (DCTCP) > >>>> or Low Latency Low Loss Scalable Throughput (L4S) need more > >>>> accurate > >>>> ECN feedback information whenever more than one marking is > >>>> received > >>>> in one RTT. This document specifies a scheme to provide more > >>>> than > >>>> one feedback signal per RTT in the TCP header. Given TCP heade= r > >>>> space is scarce, it allocates a reserved header bit, that was > >>>> previously used for the ECN-Nonce which has now been declared > >>>> historic. It also overloads the two existing ECN flags in the > >>>> TCP > >>>> header. The resulting extra space is exploited to feed back > >>>> the IP- > >>>> ECN field received during the 3-way handshake as well. > >>>> Supplementary > >>>> feedback information can optionally be provided in a new TCP > >>>> option, > >>>> which is never used on the TCP SYN. > >>>> > >>>> > >>>> The IETF datatracker status page for this draft is: > >>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ > >>>> > >>>> There are also htmlized versions available at: > >>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12 > >>>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-1= 2 > >>>> > >>>> A diff from the previous version is available at: > >>>> https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-accurate-ecn-12 > >>>> > >>>> > >>>> Please note that it may take a couple of minutes from the time of > >>>> submission > >>>> until the htmlized version and diff are available at tools.ietf.org. > >>>> > >>>> Internet-Drafts are also available by anonymous FTP at: > >>>> ftp://ftp.ietf.org/internet-drafts/ > >>>> > >>>> > >>>> _______________________________________________ > >>>> tcpm mailing list > >>>> tcpm@ietf.org > >>>> https://www.ietf.org/mailman/listinfo/tcpm > >>> -- > >>> ________________________________________________________________ > >>> Bob Briscoe http://bobbriscoe.net/ > >>> PRIVILEGED AND CONFIDENTIAL > >>> > >>> _______________________________________________ > >>> tcpm mailing list > >>> tcpm@ietf.org > >>> https://www.ietf.org/mailman/listinfo/tcpm > >> _______________________________________________ > >> tcpm mailing list > >> tcpm@ietf.org > >> https://www.ietf.org/mailman/listinfo/tcpm > > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > --0000000000000ac13005b36c5b65 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Bob,

Sorry for the late c= omments, I have a naive question=C2=A0about the encoding schemes in the opt= ions.
The byte counters will be incremented by every byte=C2=A0th= e receiver receives. But, do we really need this level of granularity?=C2= =A0
I am thinking if we can byte-shift the counter values to some= extent, we might be able to save option space.

Th= anks,
--
Yoshi


On Mon, Nov 2, 2020 at 4:0= 9 PM Bob Briscoe <ietf@bobbriscoe= .net> wrote:
Michael,

1/ I just posted a draft-13 that has the two AccECN TCP Option orders.

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-tc= pm-accurate-ecn-13

As agreed, it keys the two option orders on two different Option Kinds, which I've requested in the IANA section (TBD0 and TBD1). This replaces=
using the initial values of the fields. I've reverted all the
complicated text about:
* varying the initial field values dependent on option order,
* having to keep consistent order throughout a connection,
* and choosing which order from each end.
All of which is irrelevant now.

It is much simpler (I guess we all knew that). Thank you.

As mentioned, I found my other ToDo list:
2/ We also introduced considerably more guidance on middlebox handling, esp. ACK filtering and Segmentation Offload. There was enough extra text to be worth breaking that section into subsections.

The guidance on ACK filtering for connections with AccECN negotiated
updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in the presence of ECN was a research issue). But there is still no
guidance for ECN connections when AccECN has not been negotiated.

3/ I also added a SHOULD requirement I had on my ToDo list: Each Data
Sender SHOULD disable ECT in one direction if all the feedback for three or four rounds from the start is CE. That's already done in Linux, and =
in the TCP in iOS and MacOS at least.



Bob





On 28/10/2020 17:17, Bob Briscoe wrote:
> Michael,
>
> On 28/10/2020 15:55, Scharf, Michael wrote:
>> Hi Bob,
>>
>> What is the future plan regarding the option encoding proposed by = the
>> authors?
>>
>> This has been discussed in past meetings and on the list, for
>> instance:
>> https://mailarchive.= ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>>
>> I have explained several times my pushback against the encoding >> currently described in the document, most notably the requirement = to
>> keep state in the endpoints to decode the format of a TCP option. = I
>> have not changed my mind.
>>
>> I have also mentioned that, for instance, two option codepoints wo= uld
>> IMHO be a better engineering approach ("KISS principle")= . That would
>> be a very simple change to the content to the document. Needless t= o
>> say that there are other alternatives for marshalling parameters i= n
>> an unambigous way.
>
> [BB] Whoops. In my head I had already shifted to your scheme with
> two-option kinds. But I see that, in the draft,=C2=A0 I haven't. W= hoops -
> that was a bug in my ToDo list processing. Sorry about that. I'll =
> submit another draft before the deadline with that scheme replacing > the current one.
>
> I've just found some other entries in my ToDo list for this draft =
> which I missed as well. I'll include them too.
>
> Sorry again.
>
>
> Bob
>
>>
>> There have been other related comments, e.g. from Ilpo.
>>
>> Best regards
>>
>> Michael (with no hat)
>>
>>> -----Original Message-----
>>> From: tcpm <tcpm-bounces@ietf.org> On Behalf Of Bob Briscoe
>>> Sent: Wednesday, October 28, 2020 2:04 PM
>>> To: tcpm@ie= tf.org
>>> Cc: Scheffenegger, Richard <Richard.Scheffenegger@netapp.com= >; Mirja
>>> Kuehlewind <ietf@kuehlewind.net>
>>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-1= 2.txt
>>>
>>> TCPM folks,
>>>
>>> Back in May'20, it was decided to hold back the AccECN (an= d the ECN++)
>>> drafts for a while until the position on L4S in tsvwg was clea= rer.
>>> There
>>> is no formal dependency on the L4S drafts in tsvwg, but it mad= e sense
>>> not to burn codepoints without more certainty.
>>>
>>> I've just refreshed the AccECN and the ECN++ drafts, which= had expired.
>>>
>>> I included the following minor changes to AccECN, which you ca= n see via
>>> the diff link below, but in summary:
>>> * moved wrongly categorized "action if unwilling to provi= de f/b" to the
>>> group of bullets relevant to the receiver
>>> * removed refs to long-expired ecn-fallback draft
>>> * fixed table 1 after xml2rfc changed formatting
>>>
>>> Thanks to those who pointed these out.
>>>
>>> No changes to ECN++ other than update of some refs.
>>>
>>> Cheers
>>>
>>>
>>> Bob
>>>
>>> On 28/10/2020 12:47, internet-drafts@ietf.org wrote:
>>>> A New Internet-Draft is available from the on-line Interne= t-Drafts
>>>> directories.
>>>> This draft is a work item of the TCP Maintenance and Minor=
>>>> Extensions WG of
>>> the IETF.
>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Tit= le=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : More Accur= ate ECN Feedback in TCP
>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Aut= hors=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : Bob Briscoe
>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Mirja K=C3=BChlewind
>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Richard Scheffenegger
>>>> =C2=A0=C2=A0=C2=A0=C2=A0Filename=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 : draft-ietf-tcpm-accurate-ecn-12.txt
>>>> =C2=A0=C2=A0=C2=A0=C2=A0Pages=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : 58
>>>> =C2=A0=C2=A0=C2=A0=C2=A0Date=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : 2020-10-28
>>>>
>>>> Abstract:
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 Explicit Congestion Notification = (ECN) is a mechanism where
>>>> network
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 nodes can mark IP packets instead= of dropping them to indicate
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 incipient congestion to the end-p= oints.=C2=A0 Receivers with an ECN-
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 capable transport protocol feed b= ack this information to the
>>>> sender.
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN is specified for TCP in such = a way that only one feedback
>>>> signal
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 can be transmitted per Round-Trip= Time (RTT).=C2=A0 Recent new TCP
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 mechanisms like Congestion Exposu= re (ConEx), Data Center TCP
>>>> (DCTCP)
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 or Low Latency Low Loss Scalable = Throughput (L4S) need more
>>>> accurate
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN feedback information whenever= more than one marking is
>>>> received
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 in one RTT.=C2=A0 This document s= pecifies a scheme to provide more
>>>> than
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 one feedback signal per RTT in th= e TCP header.=C2=A0 Given TCP header
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 space is scarce, it allocates a r= eserved header bit, that was
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 previously used for the ECN-Nonce= which has now been declared
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 historic.=C2=A0 It also overloads= the two existing ECN flags in the
>>>> TCP
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 header.=C2=A0 The resulting extra= space is exploited to feed back
>>>> the IP-
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN field received during the 3-w= ay handshake as well.
>>>> Supplementary
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 feedback information can optional= ly be provided in a new TCP
>>>> option,
>>>> =C2=A0=C2=A0=C2=A0=C2=A0 which is never used on the TCP SY= N.
>>>>
>>>>
>>>> The IETF datatracker status page for this draft is:
>>>> https://datatracker.i= etf.org/doc/draft-ietf-tcpm-accurate-ecn/
>>>>
>>>> There are also htmlized versions available at:
>>>> https://tools.ietf.org/h= tml/draft-ietf-tcpm-accurate-ecn-12
>>>> https://datatr= acker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12
>>>>
>>>> A diff from the previous version is available at:
>>>> https://www.ietf= .org/rfcdiff?url2=3Ddraft-ietf-tcpm-accurate-ecn-12
>>>>
>>>>
>>>> Please note that it may take a couple of minutes from the = time of
>>>> submission
>>>> until the htmlized version and diff are available at tools.iet= f.org.
>>>>
>>>> Internet-Drafts are also available by anonymous FTP at: >>>> ftp://ftp.ietf.org/internet-drafts/
>>>>
>>>>
>>>> _______________________________________________
>>>> tcpm mailing list
>>>> tcpm@ie= tf.org
>>>> https://www.ietf.org/mailman/listinfo/tcp= m
>>> --
>>> ______________________________________________________________= __
>>> Bob Briscoe http://bobbriscoe.net/
>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PRIVILEGED AND CONFIDENTIAL
>>>
>>> _______________________________________________
>>> tcpm mailing list
>>> tcpm@ietf.o= rg
>>> https://www.ietf.org/mailman/listinfo/tcpm
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm >

--
________________________________________________________________
Bob Briscoe=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://bobbriscoe.net/

_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm
--0000000000000ac13005b36c5b65-- From nobody Fri Nov 6 01:30:09 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9AAE73A0FC5 for ; Fri, 6 Nov 2020 01:30:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.101 X-Spam-Level: X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3qmfxPj6Vq7l for ; Fri, 6 Nov 2020 01:30:05 -0800 (PST) Received: from EUR05-AM6-obe.outbound.protection.outlook.com (mail-am6eur05on2046.outbound.protection.outlook.com [40.107.22.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D9DD33A0D4D for ; Fri, 6 Nov 2020 01:30:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Rxi50ray8hGJJfRPgLQjsFXagY9gkvbwuRKykdjidLaUWkrLC0ns3+iQ5lTAPHeCbxMpobevdC6q9mKfmXj5KzJzACRD+M1cgrp4lKMMk5jO79GXl0CrBYpZPwdL4YW7sX6afZmDCqGY2dwu7w+SI7kRwumpOk60wdFkrPsVmRGhUuZ/Yf4/p+Ws4zZ7cokwuPd5QhLVcgR484hyDqikjUwQb8cqexGZWyucGQFLnHk9ojEGO72wbKLu27qO52TISmwDUcyXcUH4JUPsDMHrbYQSEU7Vz6ubKQBAQ5+IMGPMux7xcaWKXDu3iAiwoMBpwvpgl7wBfWs0b8mEjISU5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rD8BhHCoD3gky6tieSGcZXaSDYnsFVpyzehBKMflt7Q=; b=jskRaP1uKxpM4KGepcECW+rT/jX4pJN72v+41qaf2oqUy95Rx1l0uYGLbLpyae/wNYx2xl3khozU7ZRhPDQTGYPGfvfFXVYjogkPADIXGk7v1CQkFq+a//heCDoVPn8rVBXZxvNJdHuVJ5hKQeqj0JDZzEDWTd5bT7KTa64ty50vlLOrBoFyej6U7tlDLStlhC/4L6OxlFRaYWCDatCgTIDaqOfO+ybZ33r4rSQl2JMWd7pWUHEpr3AFQYfs88XaXjzwbFhY46t7VxtdfIjERXrTkSSTJ4yO3uhJ71t/92nMkCEpmg4OT0AeKBCjwBgzhNNQt7QdQRETOxiFASDE/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ericsson.com; dmarc=pass action=none header.from=ericsson.com; dkim=pass header.d=ericsson.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rD8BhHCoD3gky6tieSGcZXaSDYnsFVpyzehBKMflt7Q=; b=SI3XUdJJPlDZIg55YnyD5ZEhz3mKi7GceoIKuGUO49NiEewexKgXWxOO4FejwiRwOTkoNwiXjOC3neDuom84d1F9ntN2DrzVFKZs+n3jIowtAifu7YzDsiaVbm7X5YUjhfFs1/HMpGiSkibREy88+jsHzFiqhFFKtaSbscw06jw= Received: from AM0PR07MB3939.eurprd07.prod.outlook.com (2603:10a6:208:40::14) by AM9PR07MB7282.eurprd07.prod.outlook.com (2603:10a6:20b:2c6::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.13; Fri, 6 Nov 2020 09:30:01 +0000 Received: from AM0PR07MB3939.eurprd07.prod.outlook.com ([fe80::50e6:68f7:3047:3cf0]) by AM0PR07MB3939.eurprd07.prod.outlook.com ([fe80::50e6:68f7:3047:3cf0%3]) with mapi id 15.20.3541.018; Fri, 6 Nov 2020 09:30:01 +0000 From: Mirja Kuehlewind To: Yoshifumi Nishida , Bob Briscoe CC: "Scheffenegger, Richard" , "tcpm@ietf.org" , Mirja Kuehlewind Thread-Topic: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt Thread-Index: AQHWsXWYJmIGxpft5kqA2Sqe0rO+j6m60c6AgAAa14A= Date: Fri, 6 Nov 2020 09:30:01 +0000 Message-ID: References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> In-Reply-To: Accept-Language: en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/16.42.20101102 authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=ericsson.com; x-originating-ip: [79.209.205.21] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fa5e799c-79a1-4431-1d5c-08d88236897e x-ms-traffictypediagnostic: AM9PR07MB7282: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: i2VApVPOigJkgBPW3ITgu5Ue8jomYFR8csW1iZpOlkzZ07/aTvJ6DV3d0skwr9c8OFHu4C6XSJwU2eXIyQI/n8Uf6ypR2ObSOZ2ST+GDfSO+aWQhqUkNbju90LqWUUcQ1wibrFbOzIAfSFOi6v0mB/6OOdhrhCwLNVClTU+K51txhIdZdnrjw5q4/bZWp8eDeyidVhxZFRX8OBCIQ3zahQNsnwtwsP/+0S9fLal5QkY4PjZIkLzZwT7eVH+1isxpTVMNPhI1XBeRUQZJ6sbGQpJW+YLXU8/vBIl/6wNDSZ3OA5YmuODqleYhgWl0JFVol3X5Z7oO8NwerKz0CnhbkOFJ9seCniUr5OVu1pK/iedYmP6v1L0sGBbOu2JDqmEu+jfD0HkogjEZyq/e29zYzA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR07MB3939.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(376002)(346002)(396003)(39860400002)(136003)(366004)(6506007)(6512007)(166002)(186003)(2906002)(76116006)(2616005)(86362001)(44832011)(53546011)(71200400001)(478600001)(26005)(316002)(54906003)(6486002)(64756008)(4001150100001)(5660300002)(66574015)(36756003)(8936002)(4326008)(110136005)(33656002)(8676002)(66946007)(66476007)(66446008)(83380400001)(966005)(66556008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: y5s8mccGgXZXhXf20mGxZG036+jz6KoS2a/Cqx/iyTGEI9p4epEM5cbnTrJzqBWai7G+d/12WJXMWJ0nxcmb6OejheO85+AeoqVi5xqOPBd12I5Bq6EC37ByBGz3Mrxmbu9UOlwerPtOpi06faXptQzrF7EZRBSw82p43HskjF9JG6XNywOHFv68zUlt7aotCNIBDlBsk7dHje8sTzsMiBRVZpsluQr/c7RyF1DRMKg2XSfF6LmAwPxfX8ZpQxXHqG8DwV8Oc3kRO4WejdTQR4JTuBYuMEyoDIrdVxvD6gQ99pF1MXv18MELPtleR94YZyRZq7f/y66n17ZyeKwvEV4SzcBiRx9jui2gT2nBgzLcNyHJRBz/hFP79jZV5o2B5p6xGGJLCtyct2MTfJM+TQeW7anOEbFP79PmZpB8B/FngorlX3oVsGm2RsX0T1Us+OoWvmufAECfseIOIPiHdXx+l/TBeu0h8vfk6ucWCaghmhccutBUK8bi32R5GvYHxewouF4Iq/yO85cCO2bl7aBwqsnTU59/1WNvn1MXF/LbuvjqBghf+2HNcGq4RCsaGDT4am9xvesaP0BqelFTROanN2qBWUbqEQuVAYzk9OvQ0TH0DDUp9fzYuJ9rROY8LpOE/FyIsSECuwdmi7Kl1w== x-ms-exchange-transport-forked: True Content-Type: multipart/alternative; boundary="_000_FFA9CFA448E244328DFCBF9EA08FBAB6ericssoncom_" MIME-Version: 1.0 X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM0PR07MB3939.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: fa5e799c-79a1-4431-1d5c-08d88236897e X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Nov 2020 09:30:01.9063 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Q5Hr4TzQnsKAibd5cK0yqAENbiCEHShRUxwl/bEkExXOUyzQtHbaBVze8ALDDRb8b4IYqbiD3w7pqoLC2ZaUqH2LCH6d9wFQx5hUVt6aLl0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM9PR07MB7282 Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 09:30:08 -0000 --_000_FFA9CFA448E244328DFCBF9EA08FBAB6ericssoncom_ Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 SGkgWW9zaGksDQoNCm5vdCBzdXJlIGlmIHRoYXQgcmVhbGx5IHNhdmVzIHRoYXQgbXVjaCBhcyB5 b3UgZG9u4oCZdCBuZWVkIHRvIHNlbmQgdGhlIG9wdGlvbiB3aXRoIGV2ZXJ5IHBhY2tldCwgYXMg c3VjaCB5b3UgY2FuIG1hbmFnZSB0aGUgb3ZlcmhlYWQgZHluYW1pY2FsbHkgYXMgeW91IGNhbiBk ZWNpZGUgaG93IG9mdGVuIHRvIHNlbmQuIEFsc28gaW4gYSBzY2VuYXJpbyB3aGVyZSBtb3N0IG9m IHRoZSB0cmFmZmljIGZsb3dzIG9ubHkgaW4gb24gZGlyZWN0aW9uIHRoZSBBY2NFQ04gaXMgc2Vu dCBpbiB0aGUgb3RoZXIgZGlyZWN0aW9uLg0KDQpNaXJqYQ0KDQoNCg0KRnJvbTogdGNwbSA8dGNw bS1ib3VuY2VzQGlldGYub3JnPiBvbiBiZWhhbGYgb2YgWW9zaGlmdW1pIE5pc2hpZGEgPG5zZC5p ZXRmQGdtYWlsLmNvbT4NCkRhdGU6IEZyaWRheSwgNi4gTm92ZW1iZXIgMjAyMCBhdCAwOTo1NA0K VG86IEJvYiBCcmlzY29lIDxpZXRmQGJvYmJyaXNjb2UubmV0Pg0KQ2M6ICJTY2hlZmZlbmVnZ2Vy LCBSaWNoYXJkIiA8UmljaGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20+LCAidGNwbUBpZXRm Lm9yZyIgPHRjcG1AaWV0Zi5vcmc+LCBNaXJqYSBLdWVobGV3aW5kIDxpZXRmQGt1ZWhsZXdpbmQu bmV0Pg0KU3ViamVjdDogUmU6IFt0Y3BtXSBJLUQgQWN0aW9uOiBkcmFmdC1pZXRmLXRjcG0tYWNj dXJhdGUtZWNuLTEzLnR4dA0KDQpIaSBCb2IsDQoNClNvcnJ5IGZvciB0aGUgbGF0ZSBjb21tZW50 cywgSSBoYXZlIGEgbmFpdmUgcXVlc3Rpb24gYWJvdXQgdGhlIGVuY29kaW5nIHNjaGVtZXMgaW4g dGhlIG9wdGlvbnMuDQpUaGUgYnl0ZSBjb3VudGVycyB3aWxsIGJlIGluY3JlbWVudGVkIGJ5IGV2 ZXJ5IGJ5dGUgdGhlIHJlY2VpdmVyIHJlY2VpdmVzLiBCdXQsIGRvIHdlIHJlYWxseSBuZWVkIHRo aXMgbGV2ZWwgb2YgZ3JhbnVsYXJpdHk/DQpJIGFtIHRoaW5raW5nIGlmIHdlIGNhbiBieXRlLXNo aWZ0IHRoZSBjb3VudGVyIHZhbHVlcyB0byBzb21lIGV4dGVudCwgd2UgbWlnaHQgYmUgYWJsZSB0 byBzYXZlIG9wdGlvbiBzcGFjZS4NCg0KVGhhbmtzLA0KLS0NCllvc2hpDQoNCg0KT24gTW9uLCBO b3YgMiwgMjAyMCBhdCA0OjA5IFBNIEJvYiBCcmlzY29lIDxpZXRmQGJvYmJyaXNjb2UubmV0PG1h aWx0bzppZXRmQGJvYmJyaXNjb2UubmV0Pj4gd3JvdGU6DQpNaWNoYWVsLA0KDQoxLyBJIGp1c3Qg cG9zdGVkIGEgZHJhZnQtMTMgdGhhdCBoYXMgdGhlIHR3byBBY2NFQ04gVENQIE9wdGlvbiBvcmRl cnMuDQoNClRoZXJlIGFyZSBhbHNvIGh0bWxpemVkIHZlcnNpb25zIGF2YWlsYWJsZSBhdDoNCmh0 dHBzOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLTEz DQoNCkFzIGFncmVlZCwgaXQga2V5cyB0aGUgdHdvIG9wdGlvbiBvcmRlcnMgb24gdHdvIGRpZmZl cmVudCBPcHRpb24gS2luZHMsDQp3aGljaCBJJ3ZlIHJlcXVlc3RlZCBpbiB0aGUgSUFOQSBzZWN0 aW9uIChUQkQwIGFuZCBUQkQxKS4gVGhpcyByZXBsYWNlcw0KdXNpbmcgdGhlIGluaXRpYWwgdmFs dWVzIG9mIHRoZSBmaWVsZHMuIEkndmUgcmV2ZXJ0ZWQgYWxsIHRoZQ0KY29tcGxpY2F0ZWQgdGV4 dCBhYm91dDoNCiogdmFyeWluZyB0aGUgaW5pdGlhbCBmaWVsZCB2YWx1ZXMgZGVwZW5kZW50IG9u IG9wdGlvbiBvcmRlciwNCiogaGF2aW5nIHRvIGtlZXAgY29uc2lzdGVudCBvcmRlciB0aHJvdWdo b3V0IGEgY29ubmVjdGlvbiwNCiogYW5kIGNob29zaW5nIHdoaWNoIG9yZGVyIGZyb20gZWFjaCBl bmQuDQpBbGwgb2Ygd2hpY2ggaXMgaXJyZWxldmFudCBub3cuDQoNCkl0IGlzIG11Y2ggc2ltcGxl ciAoSSBndWVzcyB3ZSBhbGwga25ldyB0aGF0KS4gVGhhbmsgeW91Lg0KDQpBcyBtZW50aW9uZWQs IEkgZm91bmQgbXkgb3RoZXIgVG9EbyBsaXN0Og0KMi8gV2UgYWxzbyBpbnRyb2R1Y2VkIGNvbnNp ZGVyYWJseSBtb3JlIGd1aWRhbmNlIG9uIG1pZGRsZWJveCBoYW5kbGluZywNCmVzcC4gQUNLIGZp bHRlcmluZyBhbmQgU2VnbWVudGF0aW9uIE9mZmxvYWQuIFRoZXJlIHdhcyBlbm91Z2ggZXh0cmEg dGV4dA0KdG8gYmUgd29ydGggYnJlYWtpbmcgdGhhdCBzZWN0aW9uIGludG8gc3Vic2VjdGlvbnMu DQoNClRoZSBndWlkYW5jZSBvbiBBQ0sgZmlsdGVyaW5nIGZvciBjb25uZWN0aW9ucyB3aXRoIEFj Y0VDTiBuZWdvdGlhdGVkDQp1cGRhdGVzIHRoZSAobm9uLSlndWlkYW5jZSBpbiBSRkMzNDQ5IChC Q1A2OSB3aGljaCBzYWlkIEFDSyBmaWx0ZXJpbmcgaW4NCnRoZSBwcmVzZW5jZSBvZiBFQ04gd2Fz IGEgcmVzZWFyY2ggaXNzdWUpLiBCdXQgdGhlcmUgaXMgc3RpbGwgbm8NCmd1aWRhbmNlIGZvciBF Q04gY29ubmVjdGlvbnMgd2hlbiBBY2NFQ04gaGFzIG5vdCBiZWVuIG5lZ290aWF0ZWQuDQoNCjMv IEkgYWxzbyBhZGRlZCBhIFNIT1VMRCByZXF1aXJlbWVudCBJIGhhZCBvbiBteSBUb0RvIGxpc3Q6 IEVhY2ggRGF0YQ0KU2VuZGVyIFNIT1VMRCBkaXNhYmxlIEVDVCBpbiBvbmUgZGlyZWN0aW9uIGlm IGFsbCB0aGUgZmVlZGJhY2sgZm9yIHRocmVlDQpvciBmb3VyIHJvdW5kcyBmcm9tIHRoZSBzdGFy dCBpcyBDRS4gVGhhdCdzIGFscmVhZHkgZG9uZSBpbiBMaW51eCwgYW5kDQppbiB0aGUgVENQIGlu IGlPUyBhbmQgTWFjT1MgYXQgbGVhc3QuDQoNCg0KDQpCb2INCg0KDQoNCg0KDQpPbiAyOC8xMC8y MDIwIDE3OjE3LCBCb2IgQnJpc2NvZSB3cm90ZToNCj4gTWljaGFlbCwNCj4NCj4gT24gMjgvMTAv MjAyMCAxNTo1NSwgU2NoYXJmLCBNaWNoYWVsIHdyb3RlOg0KPj4gSGkgQm9iLA0KPj4NCj4+IFdo YXQgaXMgdGhlIGZ1dHVyZSBwbGFuIHJlZ2FyZGluZyB0aGUgb3B0aW9uIGVuY29kaW5nIHByb3Bv c2VkIGJ5IHRoZQ0KPj4gYXV0aG9ycz8NCj4+DQo+PiBUaGlzIGhhcyBiZWVuIGRpc2N1c3NlZCBp biBwYXN0IG1lZXRpbmdzIGFuZCBvbiB0aGUgbGlzdCwgZm9yDQo+PiBpbnN0YW5jZToNCj4+IGh0 dHBzOi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1IwblJmaEhvY1g4 eXZUdnBDNEJOTW8vDQo+Pg0KPj4gSSBoYXZlIGV4cGxhaW5lZCBzZXZlcmFsIHRpbWVzIG15IHB1 c2hiYWNrIGFnYWluc3QgdGhlIGVuY29kaW5nDQo+PiBjdXJyZW50bHkgZGVzY3JpYmVkIGluIHRo ZSBkb2N1bWVudCwgbW9zdCBub3RhYmx5IHRoZSByZXF1aXJlbWVudCB0bw0KPj4ga2VlcCBzdGF0 ZSBpbiB0aGUgZW5kcG9pbnRzIHRvIGRlY29kZSB0aGUgZm9ybWF0IG9mIGEgVENQIG9wdGlvbi4g SQ0KPj4gaGF2ZSBub3QgY2hhbmdlZCBteSBtaW5kLg0KPj4NCj4+IEkgaGF2ZSBhbHNvIG1lbnRp b25lZCB0aGF0LCBmb3IgaW5zdGFuY2UsIHR3byBvcHRpb24gY29kZXBvaW50cyB3b3VsZA0KPj4g SU1ITyBiZSBhIGJldHRlciBlbmdpbmVlcmluZyBhcHByb2FjaCAoIktJU1MgcHJpbmNpcGxlIiku IFRoYXQgd291bGQNCj4+IGJlIGEgdmVyeSBzaW1wbGUgY2hhbmdlIHRvIHRoZSBjb250ZW50IHRv IHRoZSBkb2N1bWVudC4gTmVlZGxlc3MgdG8NCj4+IHNheSB0aGF0IHRoZXJlIGFyZSBvdGhlciBh bHRlcm5hdGl2ZXMgZm9yIG1hcnNoYWxsaW5nIHBhcmFtZXRlcnMgaW4NCj4+IGFuIHVuYW1iaWdv dXMgd2F5Lg0KPg0KPiBbQkJdIFdob29wcy4gSW4gbXkgaGVhZCBJIGhhZCBhbHJlYWR5IHNoaWZ0 ZWQgdG8geW91ciBzY2hlbWUgd2l0aA0KPiB0d28tb3B0aW9uIGtpbmRzLiBCdXQgSSBzZWUgdGhh dCwgaW4gdGhlIGRyYWZ0LCAgSSBoYXZlbid0LiBXaG9vcHMgLQ0KPiB0aGF0IHdhcyBhIGJ1ZyBp biBteSBUb0RvIGxpc3QgcHJvY2Vzc2luZy4gU29ycnkgYWJvdXQgdGhhdC4gSSdsbA0KPiBzdWJt aXQgYW5vdGhlciBkcmFmdCBiZWZvcmUgdGhlIGRlYWRsaW5lIHdpdGggdGhhdCBzY2hlbWUgcmVw bGFjaW5nDQo+IHRoZSBjdXJyZW50IG9uZS4NCj4NCj4gSSd2ZSBqdXN0IGZvdW5kIHNvbWUgb3Ro ZXIgZW50cmllcyBpbiBteSBUb0RvIGxpc3QgZm9yIHRoaXMgZHJhZnQNCj4gd2hpY2ggSSBtaXNz ZWQgYXMgd2VsbC4gSSdsbCBpbmNsdWRlIHRoZW0gdG9vLg0KPg0KPiBTb3JyeSBhZ2Fpbi4NCj4N Cj4NCj4gQm9iDQo+DQo+Pg0KPj4gVGhlcmUgaGF2ZSBiZWVuIG90aGVyIHJlbGF0ZWQgY29tbWVu dHMsIGUuZy4gZnJvbSBJbHBvLg0KPj4NCj4+IEJlc3QgcmVnYXJkcw0KPj4NCj4+IE1pY2hhZWwg KHdpdGggbm8gaGF0KQ0KPj4NCj4+PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPj4+IEZy b206IHRjcG0gPHRjcG0tYm91bmNlc0BpZXRmLm9yZzxtYWlsdG86dGNwbS1ib3VuY2VzQGlldGYu b3JnPj4gT24gQmVoYWxmIE9mIEJvYiBCcmlzY29lDQo+Pj4gU2VudDogV2VkbmVzZGF5LCBPY3Rv YmVyIDI4LCAyMDIwIDI6MDQgUE0NCj4+PiBUbzogdGNwbUBpZXRmLm9yZzxtYWlsdG86dGNwbUBp ZXRmLm9yZz4NCj4+PiBDYzogU2NoZWZmZW5lZ2dlciwgUmljaGFyZCA8UmljaGFyZC5TY2hlZmZl bmVnZ2VyQG5ldGFwcC5jb208bWFpbHRvOlJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29t Pj47IE1pcmphDQo+Pj4gS3VlaGxld2luZCA8aWV0ZkBrdWVobGV3aW5kLm5ldDxtYWlsdG86aWV0 ZkBrdWVobGV3aW5kLm5ldD4+DQo+Pj4gU3ViamVjdDogUmU6IFt0Y3BtXSBJLUQgQWN0aW9uOiBk cmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLTEyLnR4dA0KPj4+DQo+Pj4gVENQTSBmb2xrcywN Cj4+Pg0KPj4+IEJhY2sgaW4gTWF5JzIwLCBpdCB3YXMgZGVjaWRlZCB0byBob2xkIGJhY2sgdGhl IEFjY0VDTiAoYW5kIHRoZSBFQ04rKykNCj4+PiBkcmFmdHMgZm9yIGEgd2hpbGUgdW50aWwgdGhl IHBvc2l0aW9uIG9uIEw0UyBpbiB0c3Z3ZyB3YXMgY2xlYXJlci4NCj4+PiBUaGVyZQ0KPj4+IGlz IG5vIGZvcm1hbCBkZXBlbmRlbmN5IG9uIHRoZSBMNFMgZHJhZnRzIGluIHRzdndnLCBidXQgaXQg bWFkZSBzZW5zZQ0KPj4+IG5vdCB0byBidXJuIGNvZGVwb2ludHMgd2l0aG91dCBtb3JlIGNlcnRh aW50eS4NCj4+Pg0KPj4+IEkndmUganVzdCByZWZyZXNoZWQgdGhlIEFjY0VDTiBhbmQgdGhlIEVD TisrIGRyYWZ0cywgd2hpY2ggaGFkIGV4cGlyZWQuDQo+Pj4NCj4+PiBJIGluY2x1ZGVkIHRoZSBm b2xsb3dpbmcgbWlub3IgY2hhbmdlcyB0byBBY2NFQ04sIHdoaWNoIHlvdSBjYW4gc2VlIHZpYQ0K Pj4+IHRoZSBkaWZmIGxpbmsgYmVsb3csIGJ1dCBpbiBzdW1tYXJ5Og0KPj4+ICogbW92ZWQgd3Jv bmdseSBjYXRlZ29yaXplZCAiYWN0aW9uIGlmIHVud2lsbGluZyB0byBwcm92aWRlIGYvYiIgdG8g dGhlDQo+Pj4gZ3JvdXAgb2YgYnVsbGV0cyByZWxldmFudCB0byB0aGUgcmVjZWl2ZXINCj4+PiAq IHJlbW92ZWQgcmVmcyB0byBsb25nLWV4cGlyZWQgZWNuLWZhbGxiYWNrIGRyYWZ0DQo+Pj4gKiBm aXhlZCB0YWJsZSAxIGFmdGVyIHhtbDJyZmMgY2hhbmdlZCBmb3JtYXR0aW5nDQo+Pj4NCj4+PiBU aGFua3MgdG8gdGhvc2Ugd2hvIHBvaW50ZWQgdGhlc2Ugb3V0Lg0KPj4+DQo+Pj4gTm8gY2hhbmdl cyB0byBFQ04rKyBvdGhlciB0aGFuIHVwZGF0ZSBvZiBzb21lIHJlZnMuDQo+Pj4NCj4+PiBDaGVl cnMNCj4+Pg0KPj4+DQo+Pj4gQm9iDQo+Pj4NCj4+PiBPbiAyOC8xMC8yMDIwIDEyOjQ3LCBpbnRl cm5ldC1kcmFmdHNAaWV0Zi5vcmc8bWFpbHRvOmludGVybmV0LWRyYWZ0c0BpZXRmLm9yZz4gd3Jv dGU6DQo+Pj4+IEEgTmV3IEludGVybmV0LURyYWZ0IGlzIGF2YWlsYWJsZSBmcm9tIHRoZSBvbi1s aW5lIEludGVybmV0LURyYWZ0cw0KPj4+PiBkaXJlY3Rvcmllcy4NCj4+Pj4gVGhpcyBkcmFmdCBp cyBhIHdvcmsgaXRlbSBvZiB0aGUgVENQIE1haW50ZW5hbmNlIGFuZCBNaW5vcg0KPj4+PiBFeHRl bnNpb25zIFdHIG9mDQo+Pj4gdGhlIElFVEYuDQo+Pj4+ICAgICAgICAgICBUaXRsZSAgICAgICAg ICAgOiBNb3JlIEFjY3VyYXRlIEVDTiBGZWVkYmFjayBpbiBUQ1ANCj4+Pj4gICAgICAgICAgIEF1 dGhvcnMgICAgICAgICA6IEJvYiBCcmlzY29lDQo+Pj4+ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBNaXJqYSBLw7xobGV3aW5kDQo+Pj4+ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBS aWNoYXJkIFNjaGVmZmVuZWdnZXINCj4+Pj4gICAgIEZpbGVuYW1lICAgICAgICA6IGRyYWZ0LWll dGYtdGNwbS1hY2N1cmF0ZS1lY24tMTIudHh0DQo+Pj4+ICAgICBQYWdlcyAgICAgICAgICAgOiA1 OA0KPj4+PiAgICAgRGF0ZSAgICAgICAgICAgIDogMjAyMC0xMC0yOA0KPj4+Pg0KPj4+PiBBYnN0 cmFjdDoNCj4+Pj4gICAgICBFeHBsaWNpdCBDb25nZXN0aW9uIE5vdGlmaWNhdGlvbiAoRUNOKSBp cyBhIG1lY2hhbmlzbSB3aGVyZQ0KPj4+PiBuZXR3b3JrDQo+Pj4+ICAgICAgbm9kZXMgY2FuIG1h cmsgSVAgcGFja2V0cyBpbnN0ZWFkIG9mIGRyb3BwaW5nIHRoZW0gdG8gaW5kaWNhdGUNCj4+Pj4g ICAgICBpbmNpcGllbnQgY29uZ2VzdGlvbiB0byB0aGUgZW5kLXBvaW50cy4gIFJlY2VpdmVycyB3 aXRoIGFuIEVDTi0NCj4+Pj4gICAgICBjYXBhYmxlIHRyYW5zcG9ydCBwcm90b2NvbCBmZWVkIGJh Y2sgdGhpcyBpbmZvcm1hdGlvbiB0byB0aGUNCj4+Pj4gc2VuZGVyLg0KPj4+PiAgICAgIEVDTiBp cyBzcGVjaWZpZWQgZm9yIFRDUCBpbiBzdWNoIGEgd2F5IHRoYXQgb25seSBvbmUgZmVlZGJhY2sN Cj4+Pj4gc2lnbmFsDQo+Pj4+ICAgICAgY2FuIGJlIHRyYW5zbWl0dGVkIHBlciBSb3VuZC1Ucmlw IFRpbWUgKFJUVCkuICBSZWNlbnQgbmV3IFRDUA0KPj4+PiAgICAgIG1lY2hhbmlzbXMgbGlrZSBD b25nZXN0aW9uIEV4cG9zdXJlIChDb25FeCksIERhdGEgQ2VudGVyIFRDUA0KPj4+PiAoRENUQ1Ap DQo+Pj4+ICAgICAgb3IgTG93IExhdGVuY3kgTG93IExvc3MgU2NhbGFibGUgVGhyb3VnaHB1dCAo TDRTKSBuZWVkIG1vcmUNCj4+Pj4gYWNjdXJhdGUNCj4+Pj4gICAgICBFQ04gZmVlZGJhY2sgaW5m b3JtYXRpb24gd2hlbmV2ZXIgbW9yZSB0aGFuIG9uZSBtYXJraW5nIGlzDQo+Pj4+IHJlY2VpdmVk DQo+Pj4+ICAgICAgaW4gb25lIFJUVC4gIFRoaXMgZG9jdW1lbnQgc3BlY2lmaWVzIGEgc2NoZW1l IHRvIHByb3ZpZGUgbW9yZQ0KPj4+PiB0aGFuDQo+Pj4+ICAgICAgb25lIGZlZWRiYWNrIHNpZ25h bCBwZXIgUlRUIGluIHRoZSBUQ1AgaGVhZGVyLiAgR2l2ZW4gVENQIGhlYWRlcg0KPj4+PiAgICAg IHNwYWNlIGlzIHNjYXJjZSwgaXQgYWxsb2NhdGVzIGEgcmVzZXJ2ZWQgaGVhZGVyIGJpdCwgdGhh dCB3YXMNCj4+Pj4gICAgICBwcmV2aW91c2x5IHVzZWQgZm9yIHRoZSBFQ04tTm9uY2Ugd2hpY2gg aGFzIG5vdyBiZWVuIGRlY2xhcmVkDQo+Pj4+ICAgICAgaGlzdG9yaWMuICBJdCBhbHNvIG92ZXJs b2FkcyB0aGUgdHdvIGV4aXN0aW5nIEVDTiBmbGFncyBpbiB0aGUNCj4+Pj4gVENQDQo+Pj4+ICAg ICAgaGVhZGVyLiAgVGhlIHJlc3VsdGluZyBleHRyYSBzcGFjZSBpcyBleHBsb2l0ZWQgdG8gZmVl ZCBiYWNrDQo+Pj4+IHRoZSBJUC0NCj4+Pj4gICAgICBFQ04gZmllbGQgcmVjZWl2ZWQgZHVyaW5n IHRoZSAzLXdheSBoYW5kc2hha2UgYXMgd2VsbC4NCj4+Pj4gU3VwcGxlbWVudGFyeQ0KPj4+PiAg ICAgIGZlZWRiYWNrIGluZm9ybWF0aW9uIGNhbiBvcHRpb25hbGx5IGJlIHByb3ZpZGVkIGluIGEg bmV3IFRDUA0KPj4+PiBvcHRpb24sDQo+Pj4+ICAgICAgd2hpY2ggaXMgbmV2ZXIgdXNlZCBvbiB0 aGUgVENQIFNZTi4NCj4+Pj4NCj4+Pj4NCj4+Pj4gVGhlIElFVEYgZGF0YXRyYWNrZXIgc3RhdHVz IHBhZ2UgZm9yIHRoaXMgZHJhZnQgaXM6DQo+Pj4+IGh0dHBzOi8vZGF0YXRyYWNrZXIuaWV0Zi5v cmcvZG9jL2RyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24vDQo+Pj4+DQo+Pj4+IFRoZXJlIGFy ZSBhbHNvIGh0bWxpemVkIHZlcnNpb25zIGF2YWlsYWJsZSBhdDoNCj4+Pj4gaHR0cHM6Ly90b29s cy5pZXRmLm9yZy9odG1sL2RyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTINCj4+Pj4gaHR0 cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9kb2MvaHRtbC9kcmFmdC1pZXRmLXRjcG0tYWNjdXJh dGUtZWNuLTEyDQo+Pj4+DQo+Pj4+IEEgZGlmZiBmcm9tIHRoZSBwcmV2aW91cyB2ZXJzaW9uIGlz IGF2YWlsYWJsZSBhdDoNCj4+Pj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvcmZjZGlmZj91cmwyPWRy YWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTINCj4+Pj4NCj4+Pj4NCj4+Pj4gUGxlYXNlIG5v dGUgdGhhdCBpdCBtYXkgdGFrZSBhIGNvdXBsZSBvZiBtaW51dGVzIGZyb20gdGhlIHRpbWUgb2YN Cj4+Pj4gc3VibWlzc2lvbg0KPj4+PiB1bnRpbCB0aGUgaHRtbGl6ZWQgdmVyc2lvbiBhbmQgZGlm ZiBhcmUgYXZhaWxhYmxlIGF0IHRvb2xzLmlldGYub3JnPGh0dHA6Ly90b29scy5pZXRmLm9yZz4u DQo+Pj4+DQo+Pj4+IEludGVybmV0LURyYWZ0cyBhcmUgYWxzbyBhdmFpbGFibGUgYnkgYW5vbnlt b3VzIEZUUCBhdDoNCj4+Pj4gZnRwOi8vZnRwLmlldGYub3JnL2ludGVybmV0LWRyYWZ0cy8NCj4+ Pj4NCj4+Pj4NCj4+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18NCj4+Pj4gdGNwbSBtYWlsaW5nIGxpc3QNCj4+Pj4gdGNwbUBpZXRmLm9yZzxtYWlsdG86 dGNwbUBpZXRmLm9yZz4NCj4+Pj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5m by90Y3BtDQo+Pj4gLS0NCj4+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+Pj4gQm9iIEJyaXNjb2UgaHR0cDovL2JvYmJy aXNjb2UubmV0LzxodHRwczovL3Byb3RlY3QyLmZpcmVleWUuY29tL3YxL3VybD9rPTk0YTlhYWIy LWNiMzI5MzlmLTk0YTllYTI5LTg2OTJkYzgyODRjYi04MjdiOGU1ODRkMzY2Yjc4JnE9MSZlPWI5 MTQxZjZjLTBmNzAtNDQxNC05MjEyLTgyY2FlNzJiOTNmYyZ1PWh0dHAlM0ElMkYlMkZib2Jicmlz Y29lLm5ldCUyRj4NCj4+PiAgICAgICAgICAgICAgICAgIFBSSVZJTEVHRUQgQU5EIENPTkZJREVO VElBTA0KPj4+DQo+Pj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18NCj4+PiB0Y3BtIG1haWxpbmcgbGlzdA0KPj4+IHRjcG1AaWV0Zi5vcmc8bWFpbHRvOnRj cG1AaWV0Zi5vcmc+DQo+Pj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby90 Y3BtDQo+PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0K Pj4gdGNwbSBtYWlsaW5nIGxpc3QNCj4+IHRjcG1AaWV0Zi5vcmc8bWFpbHRvOnRjcG1AaWV0Zi5v cmc+DQo+PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3RjcG0NCj4NCg0K LS0NCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18NCkJvYiBCcmlzY29lICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGh0 dHA6Ly9ib2JicmlzY29lLm5ldC88aHR0cHM6Ly9wcm90ZWN0Mi5maXJlZXllLmNvbS92MS91cmw/ az0xNjQxZmUyNC00OWRhYzcwOS0xNjQxYmViZi04NjkyZGM4Mjg0Y2ItMDI4MzQ3NWVmNGZhOGUy NyZxPTEmZT1iOTE0MWY2Yy0wZjcwLTQ0MTQtOTIxMi04MmNhZTcyYjkzZmMmdT1odHRwJTNBJTJG JTJGYm9iYnJpc2NvZS5uZXQlMkY+DQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fDQp0Y3BtIG1haWxpbmcgbGlzdA0KdGNwbUBpZXRmLm9yZzxtYWlsdG86 dGNwbUBpZXRmLm9yZz4NCmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vdGNw bQ0K --_000_FFA9CFA448E244328DFCBF9EA08FBAB6ericssoncom_ Content-Type: text/html; charset="utf-8" Content-ID: <5F779350F2440C408334CE729C9A8CDA@eurprd07.prod.outlook.com> Content-Transfer-Encoding: base64 PGh0bWwgeG1sbnM6bz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6b2ZmaWNlIiB4 bWxuczp3PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQtY29tOm9mZmljZTp3b3JkIiB4bWxuczptPSJo dHRwOi8vc2NoZW1hcy5taWNyb3NvZnQuY29tL29mZmljZS8yMDA0LzEyL29tbWwiIHhtbG5zPSJo dHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwIj4NCjxoZWFkPg0KPG1ldGEgaHR0cC1lcXVp dj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9dXRmLTgiPg0KPG1l dGEgbmFtZT0iR2VuZXJhdG9yIiBjb250ZW50PSJNaWNyb3NvZnQgV29yZCAxNSAoZmlsdGVyZWQg bWVkaXVtKSI+DQo8c3R5bGU+PCEtLQ0KLyogRm9udCBEZWZpbml0aW9ucyAqLw0KQGZvbnQtZmFj ZQ0KCXtmb250LWZhbWlseToiQ2FtYnJpYSBNYXRoIjsNCglwYW5vc2UtMToyIDQgNSAzIDUgNCA2 IDMgMiA0O30NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6Q2FsaWJyaTsNCglwYW5vc2UtMToy IDE1IDUgMiAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3Jt YWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0KCWZvbnQtc2l6 ZToxMS4wcHQ7DQoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7fQ0KYTpsaW5rLCBz cGFuLk1zb0h5cGVybGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsN Cgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCnNwYW4uRW1haWxTdHlsZTE4DQoJe21zby1z dHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNl cmlmOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1bHQNCgl7bXNvLXN0eWxlLXR5 cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBwdDt9DQpAcGFnZSBXb3JkU2VjdGlvbjEN Cgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2luOjcyLjBwdCA3Mi4wcHQgNzIuMHB0IDcy LjBwdDt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24xO30NCi0tPjwvc3R5 bGU+DQo8L2hlYWQ+DQo8Ym9keSBsYW5nPSJlbi1ERSIgbGluaz0iYmx1ZSIgdmxpbms9InB1cnBs ZSIgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj4NCjxkaXYgY2xhc3M9IldvcmRTZWN0aW9u MSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJERSIgc3R5bGU9Im1zby1mYXJl YXN0LWxhbmd1YWdlOkVOLVVTIj5IaSBZb3NoaSw8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJERSIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1 YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y bWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT Ij5ub3Qgc3VyZSBpZiB0aGF0IHJlYWxseSBzYXZlcyB0aGF0IG11Y2ggYXMgeW91IGRvbuKAmXQg bmVlZCB0byBzZW5kIHRoZSBvcHRpb24gd2l0aCBldmVyeSBwYWNrZXQsIGFzIHN1Y2ggeW91IGNh biBtYW5hZ2UgdGhlIG92ZXJoZWFkIGR5bmFtaWNhbGx5IGFzIHlvdSBjYW4gZGVjaWRlIGhvdyBv ZnRlbiB0byBzZW5kLiBBbHNvDQogaW4gYSBzY2VuYXJpbyB3aGVyZSBtb3N0IG9mIHRoZSB0cmFm ZmljIGZsb3dzIG9ubHkgaW4gb24gZGlyZWN0aW9uIHRoZSBBY2NFQ04gaXMgc2VudCBpbiB0aGUg b3RoZXIgZGlyZWN0aW9uLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMi PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu IGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPk1pcmphPG86 cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4t VVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48 L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxl PSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9w Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdl OkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6 bm9uZTtib3JkZXItdG9wOnNvbGlkICNCNUM0REYgMS4wcHQ7cGFkZGluZzozLjBwdCAwY20gMGNt IDBjbSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzYuMHB0Ij48 Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEyLjBwdDtjb2xvcjpibGFjayI+RnJvbToNCjwvc3Bh bj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Y29sb3I6YmxhY2siPnRjcG0gJmx0 O3RjcG0tYm91bmNlc0BpZXRmLm9yZyZndDsgb24gYmVoYWxmIG9mIFlvc2hpZnVtaSBOaXNoaWRh ICZsdDtuc2QuaWV0ZkBnbWFpbC5jb20mZ3Q7PGJyPg0KPGI+RGF0ZTogPC9iPkZyaWRheSwgNi4g Tm92ZW1iZXIgMjAyMCBhdCAwOTo1NDxicj4NCjxiPlRvOiA8L2I+Qm9iIEJyaXNjb2UgJmx0O2ll dGZAYm9iYnJpc2NvZS5uZXQmZ3Q7PGJyPg0KPGI+Q2M6IDwvYj4mcXVvdDtTY2hlZmZlbmVnZ2Vy LCBSaWNoYXJkJnF1b3Q7ICZsdDtSaWNoYXJkLlNjaGVmZmVuZWdnZXJAbmV0YXBwLmNvbSZndDss ICZxdW90O3RjcG1AaWV0Zi5vcmcmcXVvdDsgJmx0O3RjcG1AaWV0Zi5vcmcmZ3Q7LCBNaXJqYSBL dWVobGV3aW5kICZsdDtpZXRmQGt1ZWhsZXdpbmQubmV0Jmd0Ozxicj4NCjxiPlN1YmplY3Q6IDwv Yj5SZTogW3RjcG1dIEktRCBBY3Rpb246IGRyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTMu dHh0PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05v cm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0OjM2LjBwdCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8 L2Rpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1s ZWZ0OjM2LjBwdCI+SGkgQm9iLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xh c3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0OjM2LjBwdCI+PG86cD4mbmJzcDs8L286 cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2lu LWxlZnQ6MzYuMHB0Ij5Tb3JyeSBmb3IgdGhlIGxhdGUgY29tbWVudHMsIEkgaGF2ZSBhIG5haXZl IHF1ZXN0aW9uJm5ic3A7YWJvdXQgdGhlIGVuY29kaW5nIHNjaGVtZXMgaW4gdGhlIG9wdGlvbnMu PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHls ZT0ibWFyZ2luLWxlZnQ6MzYuMHB0Ij5UaGUgYnl0ZSBjb3VudGVycyB3aWxsIGJlIGluY3JlbWVu dGVkIGJ5IGV2ZXJ5IGJ5dGUmbmJzcDt0aGUgcmVjZWl2ZXIgcmVjZWl2ZXMuIEJ1dCwgZG8gd2Ug cmVhbGx5IG5lZWQgdGhpcyBsZXZlbCBvZiBncmFudWxhcml0eT8mbmJzcDs8bzpwPjwvbzpwPjwv cD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tbGVm dDozNi4wcHQiPkkgYW0gdGhpbmtpbmcgaWYgd2UgY2FuIGJ5dGUtc2hpZnQgdGhlIGNvdW50ZXIg dmFsdWVzIHRvIHNvbWUgZXh0ZW50LCB3ZSBtaWdodCBiZSBhYmxlIHRvIHNhdmUgb3B0aW9uIHNw YWNlLjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg c3R5bGU9Im1hcmdpbi1sZWZ0OjM2LjBwdCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4N CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzYuMHB0Ij5U aGFua3MsPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFs IiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzYuMHB0Ij4tLTxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8 ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0OjM2LjBwdCI+WW9z aGk8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0 eWxlPSJtYXJnaW4tbGVmdDozNi4wcHQiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8 cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWxlZnQ6MzYuMHB0Ij48bzpwPiZuYnNw OzwvbzpwPjwvcD4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1h cmdpbi1sZWZ0OjM2LjBwdCI+T24gTW9uLCBOb3YgMiwgMjAyMCBhdCA0OjA5IFBNIEJvYiBCcmlz Y29lICZsdDs8YSBocmVmPSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+aWV0ZkBib2Jicmlz Y29lLm5ldDwvYT4mZ3Q7IHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90 ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtwYWRk aW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowY20i Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1sZWZ0OjM2LjBwdCI+TWljaGFl bCw8YnI+DQo8YnI+DQoxLyBJIGp1c3QgcG9zdGVkIGEgZHJhZnQtMTMgdGhhdCBoYXMgdGhlIHR3 byBBY2NFQ04gVENQIE9wdGlvbiBvcmRlcnMuPGJyPg0KPGJyPg0KVGhlcmUgYXJlIGFsc28gaHRt bGl6ZWQgdmVyc2lvbnMgYXZhaWxhYmxlIGF0Ojxicj4NCjxhIGhyZWY9Imh0dHBzOi8vdG9vbHMu aWV0Zi5vcmcvaHRtbC9kcmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLTEzIiB0YXJnZXQ9Il9i bGFuayI+aHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0LWlldGYtdGNwbS1hY2N1cmF0 ZS1lY24tMTM8L2E+PGJyPg0KPGJyPg0KQXMgYWdyZWVkLCBpdCBrZXlzIHRoZSB0d28gb3B0aW9u IG9yZGVycyBvbiB0d28gZGlmZmVyZW50IE9wdGlvbiBLaW5kcywgPGJyPg0Kd2hpY2ggSSd2ZSBy ZXF1ZXN0ZWQgaW4gdGhlIElBTkEgc2VjdGlvbiAoVEJEMCBhbmQgVEJEMSkuIFRoaXMgcmVwbGFj ZXMgPGJyPg0KdXNpbmcgdGhlIGluaXRpYWwgdmFsdWVzIG9mIHRoZSBmaWVsZHMuIEkndmUgcmV2 ZXJ0ZWQgYWxsIHRoZSA8YnI+DQpjb21wbGljYXRlZCB0ZXh0IGFib3V0Ojxicj4NCiogdmFyeWlu ZyB0aGUgaW5pdGlhbCBmaWVsZCB2YWx1ZXMgZGVwZW5kZW50IG9uIG9wdGlvbiBvcmRlciw8YnI+ DQoqIGhhdmluZyB0byBrZWVwIGNvbnNpc3RlbnQgb3JkZXIgdGhyb3VnaG91dCBhIGNvbm5lY3Rp b24sPGJyPg0KKiBhbmQgY2hvb3Npbmcgd2hpY2ggb3JkZXIgZnJvbSBlYWNoIGVuZC48YnI+DQpB bGwgb2Ygd2hpY2ggaXMgaXJyZWxldmFudCBub3cuPGJyPg0KPGJyPg0KSXQgaXMgbXVjaCBzaW1w bGVyIChJIGd1ZXNzIHdlIGFsbCBrbmV3IHRoYXQpLiBUaGFuayB5b3UuPGJyPg0KPGJyPg0KQXMg bWVudGlvbmVkLCBJIGZvdW5kIG15IG90aGVyIFRvRG8gbGlzdDo8YnI+DQoyLyBXZSBhbHNvIGlu dHJvZHVjZWQgY29uc2lkZXJhYmx5IG1vcmUgZ3VpZGFuY2Ugb24gbWlkZGxlYm94IGhhbmRsaW5n LCA8YnI+DQplc3AuIEFDSyBmaWx0ZXJpbmcgYW5kIFNlZ21lbnRhdGlvbiBPZmZsb2FkLiBUaGVy ZSB3YXMgZW5vdWdoIGV4dHJhIHRleHQgPGJyPg0KdG8gYmUgd29ydGggYnJlYWtpbmcgdGhhdCBz ZWN0aW9uIGludG8gc3Vic2VjdGlvbnMuPGJyPg0KPGJyPg0KVGhlIGd1aWRhbmNlIG9uIEFDSyBm aWx0ZXJpbmcgZm9yIGNvbm5lY3Rpb25zIHdpdGggQWNjRUNOIG5lZ290aWF0ZWQgPGJyPg0KdXBk YXRlcyB0aGUgKG5vbi0pZ3VpZGFuY2UgaW4gUkZDMzQ0OSAoQkNQNjkgd2hpY2ggc2FpZCBBQ0sg ZmlsdGVyaW5nIGluIDxicj4NCnRoZSBwcmVzZW5jZSBvZiBFQ04gd2FzIGEgcmVzZWFyY2ggaXNz dWUpLiBCdXQgdGhlcmUgaXMgc3RpbGwgbm8gPGJyPg0KZ3VpZGFuY2UgZm9yIEVDTiBjb25uZWN0 aW9ucyB3aGVuIEFjY0VDTiBoYXMgbm90IGJlZW4gbmVnb3RpYXRlZC48YnI+DQo8YnI+DQozLyBJ IGFsc28gYWRkZWQgYSBTSE9VTEQgcmVxdWlyZW1lbnQgSSBoYWQgb24gbXkgVG9EbyBsaXN0OiBF YWNoIERhdGEgPGJyPg0KU2VuZGVyIFNIT1VMRCBkaXNhYmxlIEVDVCBpbiBvbmUgZGlyZWN0aW9u IGlmIGFsbCB0aGUgZmVlZGJhY2sgZm9yIHRocmVlIDxicj4NCm9yIGZvdXIgcm91bmRzIGZyb20g dGhlIHN0YXJ0IGlzIENFLiBUaGF0J3MgYWxyZWFkeSBkb25lIGluIExpbnV4LCBhbmQgPGJyPg0K aW4gdGhlIFRDUCBpbiBpT1MgYW5kIE1hY09TIGF0IGxlYXN0Ljxicj4NCjxicj4NCjxicj4NCjxi cj4NCkJvYjxicj4NCjxicj4NCjxicj4NCjxicj4NCjxicj4NCjxicj4NCk9uIDI4LzEwLzIwMjAg MTc6MTcsIEJvYiBCcmlzY29lIHdyb3RlOjxicj4NCiZndDsgTWljaGFlbCw8YnI+DQomZ3Q7PGJy Pg0KJmd0OyBPbiAyOC8xMC8yMDIwIDE1OjU1LCBTY2hhcmYsIE1pY2hhZWwgd3JvdGU6PGJyPg0K Jmd0OyZndDsgSGkgQm9iLDxicj4NCiZndDsmZ3Q7PGJyPg0KJmd0OyZndDsgV2hhdCBpcyB0aGUg ZnV0dXJlIHBsYW4gcmVnYXJkaW5nIHRoZSBvcHRpb24gZW5jb2RpbmcgcHJvcG9zZWQgYnkgdGhl IDxicj4NCiZndDsmZ3Q7IGF1dGhvcnM/PGJyPg0KJmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyBUaGlz IGhhcyBiZWVuIGRpc2N1c3NlZCBpbiBwYXN0IG1lZXRpbmdzIGFuZCBvbiB0aGUgbGlzdCwgZm9y IDxicj4NCiZndDsmZ3Q7IGluc3RhbmNlOiA8YnI+DQomZ3Q7Jmd0OyA8YSBocmVmPSJodHRwczov L21haWxhcmNoaXZlLmlldGYub3JnL2FyY2gvbXNnL3RjcG0vem8tMU9SMG5SZmhIb2NYOHl2VHZw QzRCTk1vLyIgdGFyZ2V0PSJfYmxhbmsiPg0KaHR0cHM6Ly9tYWlsYXJjaGl2ZS5pZXRmLm9yZy9h cmNoL21zZy90Y3BtL3pvLTFPUjBuUmZoSG9jWDh5dlR2cEM0Qk5Nby88L2E+PGJyPg0KJmd0OyZn dDs8YnI+DQomZ3Q7Jmd0OyBJIGhhdmUgZXhwbGFpbmVkIHNldmVyYWwgdGltZXMgbXkgcHVzaGJh Y2sgYWdhaW5zdCB0aGUgZW5jb2RpbmcgPGJyPg0KJmd0OyZndDsgY3VycmVudGx5IGRlc2NyaWJl ZCBpbiB0aGUgZG9jdW1lbnQsIG1vc3Qgbm90YWJseSB0aGUgcmVxdWlyZW1lbnQgdG8gPGJyPg0K Jmd0OyZndDsga2VlcCBzdGF0ZSBpbiB0aGUgZW5kcG9pbnRzIHRvIGRlY29kZSB0aGUgZm9ybWF0 IG9mIGEgVENQIG9wdGlvbi4gSSA8YnI+DQomZ3Q7Jmd0OyBoYXZlIG5vdCBjaGFuZ2VkIG15IG1p bmQuPGJyPg0KJmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyBJIGhhdmUgYWxzbyBtZW50aW9uZWQgdGhh dCwgZm9yIGluc3RhbmNlLCB0d28gb3B0aW9uIGNvZGVwb2ludHMgd291bGQgPGJyPg0KJmd0OyZn dDsgSU1ITyBiZSBhIGJldHRlciBlbmdpbmVlcmluZyBhcHByb2FjaCAoJnF1b3Q7S0lTUyBwcmlu Y2lwbGUmcXVvdDspLiBUaGF0IHdvdWxkIDxicj4NCiZndDsmZ3Q7IGJlIGEgdmVyeSBzaW1wbGUg Y2hhbmdlIHRvIHRoZSBjb250ZW50IHRvIHRoZSBkb2N1bWVudC4gTmVlZGxlc3MgdG8gPGJyPg0K Jmd0OyZndDsgc2F5IHRoYXQgdGhlcmUgYXJlIG90aGVyIGFsdGVybmF0aXZlcyBmb3IgbWFyc2hh bGxpbmcgcGFyYW1ldGVycyBpbiA8YnI+DQomZ3Q7Jmd0OyBhbiB1bmFtYmlnb3VzIHdheS48YnI+ DQomZ3Q7PGJyPg0KJmd0OyBbQkJdIFdob29wcy4gSW4gbXkgaGVhZCBJIGhhZCBhbHJlYWR5IHNo aWZ0ZWQgdG8geW91ciBzY2hlbWUgd2l0aCA8YnI+DQomZ3Q7IHR3by1vcHRpb24ga2luZHMuIEJ1 dCBJIHNlZSB0aGF0LCBpbiB0aGUgZHJhZnQsJm5ic3A7IEkgaGF2ZW4ndC4gV2hvb3BzIC0gPGJy Pg0KJmd0OyB0aGF0IHdhcyBhIGJ1ZyBpbiBteSBUb0RvIGxpc3QgcHJvY2Vzc2luZy4gU29ycnkg YWJvdXQgdGhhdC4gSSdsbCA8YnI+DQomZ3Q7IHN1Ym1pdCBhbm90aGVyIGRyYWZ0IGJlZm9yZSB0 aGUgZGVhZGxpbmUgd2l0aCB0aGF0IHNjaGVtZSByZXBsYWNpbmcgPGJyPg0KJmd0OyB0aGUgY3Vy cmVudCBvbmUuPGJyPg0KJmd0Ozxicj4NCiZndDsgSSd2ZSBqdXN0IGZvdW5kIHNvbWUgb3RoZXIg ZW50cmllcyBpbiBteSBUb0RvIGxpc3QgZm9yIHRoaXMgZHJhZnQgPGJyPg0KJmd0OyB3aGljaCBJ IG1pc3NlZCBhcyB3ZWxsLiBJJ2xsIGluY2x1ZGUgdGhlbSB0b28uPGJyPg0KJmd0Ozxicj4NCiZn dDsgU29ycnkgYWdhaW4uPGJyPg0KJmd0Ozxicj4NCiZndDs8YnI+DQomZ3Q7IEJvYjxicj4NCiZn dDs8YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7IFRoZXJlIGhhdmUgYmVlbiBvdGhlciByZWxh dGVkIGNvbW1lbnRzLCBlLmcuIGZyb20gSWxwby48YnI+DQomZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7 IEJlc3QgcmVnYXJkczxicj4NCiZndDsmZ3Q7PGJyPg0KJmd0OyZndDsgTWljaGFlbCAod2l0aCBu byBoYXQpPGJyPg0KJmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyZndDsgLS0tLS1PcmlnaW5hbCBNZXNz YWdlLS0tLS08YnI+DQomZ3Q7Jmd0OyZndDsgRnJvbTogdGNwbSAmbHQ7PGEgaHJlZj0ibWFpbHRv OnRjcG0tYm91bmNlc0BpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPnRjcG0tYm91bmNlc0BpZXRm Lm9yZzwvYT4mZ3Q7IE9uIEJlaGFsZiBPZiBCb2IgQnJpc2NvZTxicj4NCiZndDsmZ3Q7Jmd0OyBT ZW50OiBXZWRuZXNkYXksIE9jdG9iZXIgMjgsIDIwMjAgMjowNCBQTTxicj4NCiZndDsmZ3Q7Jmd0 OyBUbzogPGEgaHJlZj0ibWFpbHRvOnRjcG1AaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj50Y3Bt QGlldGYub3JnPC9hPjxicj4NCiZndDsmZ3Q7Jmd0OyBDYzogU2NoZWZmZW5lZ2dlciwgUmljaGFy ZCAmbHQ7PGEgaHJlZj0ibWFpbHRvOlJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tIiB0 YXJnZXQ9Il9ibGFuayI+UmljaGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb208L2E+Jmd0Ozsg TWlyamE8YnI+DQomZ3Q7Jmd0OyZndDsgS3VlaGxld2luZCAmbHQ7PGEgaHJlZj0ibWFpbHRvOmll dGZAa3VlaGxld2luZC5uZXQiIHRhcmdldD0iX2JsYW5rIj5pZXRmQGt1ZWhsZXdpbmQubmV0PC9h PiZndDs8YnI+DQomZ3Q7Jmd0OyZndDsgU3ViamVjdDogUmU6IFt0Y3BtXSBJLUQgQWN0aW9uOiBk cmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLTEyLnR4dDxicj4NCiZndDsmZ3Q7Jmd0Ozxicj4N CiZndDsmZ3Q7Jmd0OyBUQ1BNIGZvbGtzLDxicj4NCiZndDsmZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7 Jmd0OyBCYWNrIGluIE1heScyMCwgaXQgd2FzIGRlY2lkZWQgdG8gaG9sZCBiYWNrIHRoZSBBY2NF Q04gKGFuZCB0aGUgRUNOKyspPGJyPg0KJmd0OyZndDsmZ3Q7IGRyYWZ0cyBmb3IgYSB3aGlsZSB1 bnRpbCB0aGUgcG9zaXRpb24gb24gTDRTIGluIHRzdndnIHdhcyBjbGVhcmVyLiA8YnI+DQomZ3Q7 Jmd0OyZndDsgVGhlcmU8YnI+DQomZ3Q7Jmd0OyZndDsgaXMgbm8gZm9ybWFsIGRlcGVuZGVuY3kg b24gdGhlIEw0UyBkcmFmdHMgaW4gdHN2d2csIGJ1dCBpdCBtYWRlIHNlbnNlPGJyPg0KJmd0OyZn dDsmZ3Q7IG5vdCB0byBidXJuIGNvZGVwb2ludHMgd2l0aG91dCBtb3JlIGNlcnRhaW50eS48YnI+ DQomZ3Q7Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyZndDsgSSd2ZSBqdXN0IHJlZnJlc2hlZCB0aGUg QWNjRUNOIGFuZCB0aGUgRUNOKysgZHJhZnRzLCB3aGljaCBoYWQgZXhwaXJlZC48YnI+DQomZ3Q7 Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyZndDsgSSBpbmNsdWRlZCB0aGUgZm9sbG93aW5nIG1pbm9y IGNoYW5nZXMgdG8gQWNjRUNOLCB3aGljaCB5b3UgY2FuIHNlZSB2aWE8YnI+DQomZ3Q7Jmd0OyZn dDsgdGhlIGRpZmYgbGluayBiZWxvdywgYnV0IGluIHN1bW1hcnk6PGJyPg0KJmd0OyZndDsmZ3Q7 ICogbW92ZWQgd3JvbmdseSBjYXRlZ29yaXplZCAmcXVvdDthY3Rpb24gaWYgdW53aWxsaW5nIHRv IHByb3ZpZGUgZi9iJnF1b3Q7IHRvIHRoZTxicj4NCiZndDsmZ3Q7Jmd0OyBncm91cCBvZiBidWxs ZXRzIHJlbGV2YW50IHRvIHRoZSByZWNlaXZlcjxicj4NCiZndDsmZ3Q7Jmd0OyAqIHJlbW92ZWQg cmVmcyB0byBsb25nLWV4cGlyZWQgZWNuLWZhbGxiYWNrIGRyYWZ0PGJyPg0KJmd0OyZndDsmZ3Q7 ICogZml4ZWQgdGFibGUgMSBhZnRlciB4bWwycmZjIGNoYW5nZWQgZm9ybWF0dGluZzxicj4NCiZn dDsmZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7Jmd0OyBUaGFua3MgdG8gdGhvc2Ugd2hvIHBvaW50ZWQg dGhlc2Ugb3V0Ljxicj4NCiZndDsmZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7Jmd0OyBObyBjaGFuZ2Vz IHRvIEVDTisrIG90aGVyIHRoYW4gdXBkYXRlIG9mIHNvbWUgcmVmcy48YnI+DQomZ3Q7Jmd0OyZn dDs8YnI+DQomZ3Q7Jmd0OyZndDsgQ2hlZXJzPGJyPg0KJmd0OyZndDsmZ3Q7PGJyPg0KJmd0OyZn dDsmZ3Q7PGJyPg0KJmd0OyZndDsmZ3Q7IEJvYjxicj4NCiZndDsmZ3Q7Jmd0Ozxicj4NCiZndDsm Z3Q7Jmd0OyBPbiAyOC8xMC8yMDIwIDEyOjQ3LCA8YSBocmVmPSJtYWlsdG86aW50ZXJuZXQtZHJh ZnRzQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+DQppbnRlcm5ldC1kcmFmdHNAaWV0Zi5vcmc8 L2E+IHdyb3RlOjxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgQSBOZXcgSW50ZXJuZXQtRHJhZnQgaXMg YXZhaWxhYmxlIGZyb20gdGhlIG9uLWxpbmUgSW50ZXJuZXQtRHJhZnRzIDxicj4NCiZndDsmZ3Q7 Jmd0OyZndDsgZGlyZWN0b3JpZXMuPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBUaGlzIGRyYWZ0IGlz IGEgd29yayBpdGVtIG9mIHRoZSBUQ1AgTWFpbnRlbmFuY2UgYW5kIE1pbm9yIDxicj4NCiZndDsm Z3Q7Jmd0OyZndDsgRXh0ZW5zaW9ucyBXRyBvZjxicj4NCiZndDsmZ3Q7Jmd0OyB0aGUgSUVURi48 YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyBUaXRsZSZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyA6IE1vcmUgQWNjdXJhdGUgRUNOIEZlZWRiYWNr IGluIFRDUDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IEF1dGhvcnMmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgOiBCb2IgQnJpc2NvZTxicj4NCiZndDsmZ3Q7Jmd0 OyZndDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IE1p cmphIEvDvGhsZXdpbmQ8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBSaWNoYXJkIFNjaGVmZmVuZWdnZXI8YnI+DQom Z3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO0ZpbGVuYW1lJm5ic3A7Jm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IDogZHJhZnQtaWV0Zi10Y3BtLWFjY3Vy YXRlLWVjbi0xMi50eHQ8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwO1BhZ2VzJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7 Jm5ic3A7Jm5ic3A7IDogNTg8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNw OyZuYnNwO0RhdGUmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsgOiAyMDIwLTEwLTI4PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0Ozxi cj4NCiZndDsmZ3Q7Jmd0OyZndDsgQWJzdHJhY3Q6PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyAmbmJz cDsmbmJzcDsmbmJzcDsmbmJzcDsgRXhwbGljaXQgQ29uZ2VzdGlvbiBOb3RpZmljYXRpb24gKEVD TikgaXMgYSBtZWNoYW5pc20gd2hlcmUgPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBuZXR3b3JrPGJy Pg0KJmd0OyZndDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgbm9kZXMgY2FuIG1h cmsgSVAgcGFja2V0cyBpbnN0ZWFkIG9mIGRyb3BwaW5nIHRoZW0gdG8gaW5kaWNhdGU8YnI+DQom Z3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBpbmNpcGllbnQgY29uZ2Vz dGlvbiB0byB0aGUgZW5kLXBvaW50cy4mbmJzcDsgUmVjZWl2ZXJzIHdpdGggYW4gRUNOLTxicj4N CiZndDsmZ3Q7Jmd0OyZndDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGNhcGFibGUgdHJhbnNw b3J0IHByb3RvY29sIGZlZWQgYmFjayB0aGlzIGluZm9ybWF0aW9uIHRvIHRoZSA8YnI+DQomZ3Q7 Jmd0OyZndDsmZ3Q7IHNlbmRlci48YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyBFQ04gaXMgc3BlY2lmaWVkIGZvciBUQ1AgaW4gc3VjaCBhIHdheSB0aGF0IG9u bHkgb25lIGZlZWRiYWNrIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgc2lnbmFsPGJyPg0KJmd0OyZn dDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgY2FuIGJlIHRyYW5zbWl0dGVkIHBl ciBSb3VuZC1UcmlwIFRpbWUgKFJUVCkuJm5ic3A7IFJlY2VudCBuZXcgVENQPGJyPg0KJmd0OyZn dDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgbWVjaGFuaXNtcyBsaWtlIENvbmdl c3Rpb24gRXhwb3N1cmUgKENvbkV4KSwgRGF0YSBDZW50ZXIgVENQIDxicj4NCiZndDsmZ3Q7Jmd0 OyZndDsgKERDVENQKTxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i c3A7IG9yIExvdyBMYXRlbmN5IExvdyBMb3NzIFNjYWxhYmxlIFRocm91Z2hwdXQgKEw0UykgbmVl ZCBtb3JlIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgYWNjdXJhdGU8YnI+DQomZ3Q7Jmd0OyZndDsm Z3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBFQ04gZmVlZGJhY2sgaW5mb3JtYXRpb24gd2hl bmV2ZXIgbW9yZSB0aGFuIG9uZSBtYXJraW5nIGlzIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgcmVj ZWl2ZWQ8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBpbiBv bmUgUlRULiZuYnNwOyBUaGlzIGRvY3VtZW50IHNwZWNpZmllcyBhIHNjaGVtZSB0byBwcm92aWRl IG1vcmUgPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyB0aGFuPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyAm bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgb25lIGZlZWRiYWNrIHNpZ25hbCBwZXIgUlRUIGluIHRo ZSBUQ1AgaGVhZGVyLiZuYnNwOyBHaXZlbiBUQ1AgaGVhZGVyPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0 OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgc3BhY2UgaXMgc2NhcmNlLCBpdCBhbGxvY2F0ZXMg YSByZXNlcnZlZCBoZWFkZXIgYml0LCB0aGF0IHdhczxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgJm5i c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IHByZXZpb3VzbHkgdXNlZCBmb3IgdGhlIEVDTi1Ob25jZSB3 aGljaCBoYXMgbm93IGJlZW4gZGVjbGFyZWQ8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyBoaXN0b3JpYy4mbmJzcDsgSXQgYWxzbyBvdmVybG9hZHMgdGhlIHR3 byBleGlzdGluZyBFQ04gZmxhZ3MgaW4gdGhlIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgVENQPGJy Pg0KJmd0OyZndDsmZ3Q7Jmd0OyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgaGVhZGVyLiZuYnNw OyBUaGUgcmVzdWx0aW5nIGV4dHJhIHNwYWNlIGlzIGV4cGxvaXRlZCB0byBmZWVkIGJhY2sgPGJy Pg0KJmd0OyZndDsmZ3Q7Jmd0OyB0aGUgSVAtPGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyAmbmJzcDsm bmJzcDsmbmJzcDsmbmJzcDsgRUNOIGZpZWxkIHJlY2VpdmVkIGR1cmluZyB0aGUgMy13YXkgaGFu ZHNoYWtlIGFzIHdlbGwuIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgU3VwcGxlbWVudGFyeTxicj4N CiZndDsmZ3Q7Jmd0OyZndDsgJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7IGZlZWRiYWNrIGluZm9y bWF0aW9uIGNhbiBvcHRpb25hbGx5IGJlIHByb3ZpZGVkIGluIGEgbmV3IFRDUCA8YnI+DQomZ3Q7 Jmd0OyZndDsmZ3Q7IG9wdGlvbiw8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7ICZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyB3aGljaCBpcyBuZXZlciB1c2VkIG9uIHRoZSBUQ1AgU1lOLjxicj4NCiZndDsm Z3Q7Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBU aGUgSUVURiBkYXRhdHJhY2tlciBzdGF0dXMgcGFnZSBmb3IgdGhpcyBkcmFmdCBpczo8YnI+DQom Z3Q7Jmd0OyZndDsmZ3Q7IDxhIGhyZWY9Imh0dHBzOi8vZGF0YXRyYWNrZXIuaWV0Zi5vcmcvZG9j L2RyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24vIiB0YXJnZXQ9Il9ibGFuayI+DQpodHRwczov L2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9kcmFmdC1pZXRmLXRjcG0tYWNjdXJhdGUtZWNuLzwv YT48YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBUaGVyZSBhcmUg YWxzbyBodG1saXplZCB2ZXJzaW9ucyBhdmFpbGFibGUgYXQ6PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0 OyA8YSBocmVmPSJodHRwczovL3Rvb2xzLmlldGYub3JnL2h0bWwvZHJhZnQtaWV0Zi10Y3BtLWFj Y3VyYXRlLWVjbi0xMiIgdGFyZ2V0PSJfYmxhbmsiPg0KaHR0cHM6Ly90b29scy5pZXRmLm9yZy9o dG1sL2RyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTI8L2E+PGJyPg0KJmd0OyZndDsmZ3Q7 Jmd0OyA8YSBocmVmPSJodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9odG1sL2RyYWZ0 LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTIiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHBzOi8vZGF0 YXRyYWNrZXIuaWV0Zi5vcmcvZG9jL2h0bWwvZHJhZnQtaWV0Zi10Y3BtLWFjY3VyYXRlLWVjbi0x MjwvYT48YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBBIGRpZmYg ZnJvbSB0aGUgcHJldmlvdXMgdmVyc2lvbiBpcyBhdmFpbGFibGUgYXQ6PGJyPg0KJmd0OyZndDsm Z3Q7Jmd0OyA8YSBocmVmPSJodHRwczovL3d3dy5pZXRmLm9yZy9yZmNkaWZmP3VybDI9ZHJhZnQt aWV0Zi10Y3BtLWFjY3VyYXRlLWVjbi0xMiIgdGFyZ2V0PSJfYmxhbmsiPg0KaHR0cHM6Ly93d3cu aWV0Zi5vcmcvcmZjZGlmZj91cmwyPWRyYWZ0LWlldGYtdGNwbS1hY2N1cmF0ZS1lY24tMTI8L2E+ PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0Ozxicj4NCiZndDsmZ3Q7Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0 OyZndDsmZ3Q7IFBsZWFzZSBub3RlIHRoYXQgaXQgbWF5IHRha2UgYSBjb3VwbGUgb2YgbWludXRl cyBmcm9tIHRoZSB0aW1lIG9mIDxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgc3VibWlzc2lvbjxicj4N CiZndDsmZ3Q7Jmd0OyZndDsgdW50aWwgdGhlIGh0bWxpemVkIHZlcnNpb24gYW5kIGRpZmYgYXJl IGF2YWlsYWJsZSBhdCA8YSBocmVmPSJodHRwOi8vdG9vbHMuaWV0Zi5vcmciIHRhcmdldD0iX2Js YW5rIj4NCnRvb2xzLmlldGYub3JnPC9hPi48YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7PGJyPg0KJmd0 OyZndDsmZ3Q7Jmd0OyBJbnRlcm5ldC1EcmFmdHMgYXJlIGFsc28gYXZhaWxhYmxlIGJ5IGFub255 bW91cyBGVFAgYXQ6PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyA8YSBocmVmPSJmdHA6Ly9mdHAuaWV0 Zi5vcmcvaW50ZXJuZXQtZHJhZnRzLyIgdGFyZ2V0PSJfYmxhbmsiPmZ0cDovL2Z0cC5pZXRmLm9y Zy9pbnRlcm5ldC1kcmFmdHMvPC9hPjxicj4NCiZndDsmZ3Q7Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0 OyZndDsmZ3Q7PGJyPg0KJmd0OyZndDsmZ3Q7Jmd0OyBfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXzxicj4NCiZndDsmZ3Q7Jmd0OyZndDsgdGNwbSBtYWlsaW5n IGxpc3Q8YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7IDxhIGhyZWY9Im1haWx0bzp0Y3BtQGlldGYub3Jn IiB0YXJnZXQ9Il9ibGFuayI+dGNwbUBpZXRmLm9yZzwvYT48YnI+DQomZ3Q7Jmd0OyZndDsmZ3Q7 IDxhIGhyZWY9Imh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vdGNwbSIgdGFy Z2V0PSJfYmxhbmsiPmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vdGNwbTwv YT48YnI+DQomZ3Q7Jmd0OyZndDsgLS0gPGJyPg0KJmd0OyZndDsmZ3Q7IF9fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+DQom Z3Q7Jmd0OyZndDsgQm9iIEJyaXNjb2UgPGEgaHJlZj0iaHR0cHM6Ly9wcm90ZWN0Mi5maXJlZXll LmNvbS92MS91cmw/az05NGE5YWFiMi1jYjMyOTM5Zi05NGE5ZWEyOS04NjkyZGM4Mjg0Y2ItODI3 YjhlNTg0ZDM2NmI3OCZhbXA7cT0xJmFtcDtlPWI5MTQxZjZjLTBmNzAtNDQxNC05MjEyLTgyY2Fl NzJiOTNmYyZhbXA7dT1odHRwJTNBJTJGJTJGYm9iYnJpc2NvZS5uZXQlMkYiIHRhcmdldD0iX2Js YW5rIj4NCmh0dHA6Ly9ib2JicmlzY29lLm5ldC88L2E+PGJyPg0KJmd0OyZndDsmZ3Q7ICZuYnNw OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBQUklWSUxFR0VEIEFORCBDT05GSURF TlRJQUw8YnI+DQomZ3Q7Jmd0OyZndDs8YnI+DQomZ3Q7Jmd0OyZndDsgX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+DQomZ3Q7Jmd0OyZndDsgdGNwbSBt YWlsaW5nIGxpc3Q8YnI+DQomZ3Q7Jmd0OyZndDsgPGEgaHJlZj0ibWFpbHRvOnRjcG1AaWV0Zi5v cmciIHRhcmdldD0iX2JsYW5rIj50Y3BtQGlldGYub3JnPC9hPjxicj4NCiZndDsmZ3Q7Jmd0OyA8 YSBocmVmPSJodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3RjcG0iIHRhcmdl dD0iX2JsYW5rIj5odHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL3RjcG08L2E+ PGJyPg0KJmd0OyZndDsgX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX188YnI+DQomZ3Q7Jmd0OyB0Y3BtIG1haWxpbmcgbGlzdDxicj4NCiZndDsmZ3Q7IDxhIGhy ZWY9Im1haWx0bzp0Y3BtQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+dGNwbUBpZXRmLm9yZzwv YT48YnI+DQomZ3Q7Jmd0OyA8YSBocmVmPSJodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xp c3RpbmZvL3RjcG0iIHRhcmdldD0iX2JsYW5rIj5odHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFu L2xpc3RpbmZvL3RjcG08L2E+PGJyPg0KJmd0Ozxicj4NCjxicj4NCi0tIDxicj4NCl9fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188 YnI+DQpCb2IgQnJpc2NvZSZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7 ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg Jm5ic3A7ICZuYnNwOzxhIGhyZWY9Imh0dHBzOi8vcHJvdGVjdDIuZmlyZWV5ZS5jb20vdjEvdXJs P2s9MTY0MWZlMjQtNDlkYWM3MDktMTY0MWJlYmYtODY5MmRjODI4NGNiLTAyODM0NzVlZjRmYThl MjcmYW1wO3E9MSZhbXA7ZT1iOTE0MWY2Yy0wZjcwLTQ0MTQtOTIxMi04MmNhZTcyYjkzZmMmYW1w O3U9aHR0cCUzQSUyRiUyRmJvYmJyaXNjb2UubmV0JTJGIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDov L2JvYmJyaXNjb2UubmV0LzwvYT48YnI+DQo8YnI+DQpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXzxicj4NCnRjcG0gbWFpbGluZyBsaXN0PGJyPg0KPGEgaHJl Zj0ibWFpbHRvOnRjcG1AaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj50Y3BtQGlldGYub3JnPC9h Pjxicj4NCjxhIGhyZWY9Imh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vdGNw bSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8v dGNwbTwvYT48bzpwPjwvbzpwPjwvcD4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPC9kaXY+DQo8 L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg== --_000_FFA9CFA448E244328DFCBF9EA08FBAB6ericssoncom_-- From nobody Fri Nov 6 02:28:24 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DDD4F3A104A for ; Fri, 6 Nov 2020 02:28:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tOQnO_SD6xTF for ; Fri, 6 Nov 2020 02:28:19 -0800 (PST) Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9EA183A1049 for ; Fri, 6 Nov 2020 02:28:19 -0800 (PST) Received: by mail-qv1-xf2d.google.com with SMTP id d1so213370qvl.6 for ; Fri, 06 Nov 2020 02:28:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=o8SJlRcjeJdOIba3giJmbM7rYWge/Fa3+IBChZNVf/w=; b=lZaYwj/JEFo0tR5LI/uHeRustWgxM2whsLNy5GT1z9XJ/QjOYJdVfxVMJhAeY86d7Y uEebY/BGZuDELPenwBMWfVvH7QDnqYMUG4rlxBvqV3shC2a/kKTOSptdxs3RFHRwoItJ ue5NhBZOGGG19/RUo+NISrcfvTiz38LEjjtSMERLDxLTd0Op8hnHCYbMXy1jA5lVjabV D6ah+HIeYu5bKtKdIEvmCvx73LYh9MGUogyuSFkY9yfi4d0U/dMx+IfM+rvZxh6MiNrk nmfGL0XiHs8vzuIbzCTaz+faSvQVAyt6O5u7evixwzb55HEkcjkhZ/ciM9tCTkFAEU6v SqwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=o8SJlRcjeJdOIba3giJmbM7rYWge/Fa3+IBChZNVf/w=; b=KXoUokwr05J2xl5+TnSCJh3SCP8vag9hEbBcKbjPCkyOC3qu+nZAND+9Nv+qTCr4uQ jfMegY+6Ft4/MB3lgm31W3b+9XafQo7Uf17n66jwYArGCoshLJyU9LajrfYzJzzch0+l lXtOCuScLMVwZ9DWd0FZXitAYP9OjJKCsaMOZcCgbNOBTKhjP5gHimtWf3+50JwKoKhs RuIqNsKEssPMlS0m2SB/dVBFEt+xhlJtOtHlTLsaywgQEYGVDzWq02p9zjHdCr170C95 OdKCXn3jiuSEhXNkZ273bxzA+llCzlfiwORbbuKoyKZjwAIlK1covXBHgNDOopYGJpSS aYdg== X-Gm-Message-State: AOAM530hKOuL7sArSuVE0WYZ0uGzOckBdj+AyVGMzjtIG2hyzGg2rMQf LEi0IjAP94M4gW0CmSdxTmdjGa6JfRi0dzxfBVU= X-Google-Smtp-Source: ABdhPJwhshT309X8mFFe4lSxt8enjMUOjR2AZx/QkZ5EKa2MWrcxha0JiPYMca26Oj2CyiQluaDi54wDzjwsUqEp7xw= X-Received: by 2002:a0c:e0c9:: with SMTP id x9mr828662qvk.56.1604658497230; Fri, 06 Nov 2020 02:28:17 -0800 (PST) MIME-Version: 1.0 References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> In-Reply-To: From: Yoshifumi Nishida Date: Fri, 6 Nov 2020 02:28:06 -0800 Message-ID: To: Mirja Kuehlewind Cc: Bob Briscoe , "Scheffenegger, Richard" , "tcpm@ietf.org" , Mirja Kuehlewind Content-Type: multipart/alternative; boundary="000000000000b69a9f05b36dabe6" Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 10:28:23 -0000 --000000000000b69a9f05b36dabe6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mirja, It seems to me that we don't have much freedom for the frequency of the feedback. The draft says: MUST immediately send an ACK once 'n' CE marks have arrived since the previous ACK, where 'n' SHOULD be 2 and MUST be no greater than 6. Or, options are not applied to this? Also, even though traffic flows one direction, I think you might want to send back SACK options, etc. Another thought is bulk flow traffic tends to send full MSS packets. I feel 1 byte granularity might not be very efficient. In any cases, I just thought saving option space won't be a bad thing if it's not overly difficult. Thanks, -- Yoshi On Fri, Nov 6, 2020 at 1:30 AM Mirja Kuehlewind < mirja.kuehlewind@ericsson.com> wrote: > Hi Yoshi, > > > > not sure if that really saves that much as you don=E2=80=99t need to send= the > option with every packet, as such you can manage the overhead dynamically > as you can decide how often to send. Also in a scenario where most of the > traffic flows only in on direction the AccECN is sent in the other > direction. > > > > Mirja > > > > > > > > *From: *tcpm on behalf of Yoshifumi Nishida < > nsd.ietf@gmail.com> > *Date: *Friday, 6. November 2020 at 09:54 > *To: *Bob Briscoe > *Cc: *"Scheffenegger, Richard" , " > tcpm@ietf.org" , Mirja Kuehlewind > *Subject: *Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt > > > > Hi Bob, > > > > Sorry for the late comments, I have a naive question about the encoding > schemes in the options. > > The byte counters will be incremented by every byte the receiver receives= . > But, do we really need this level of granularity? > > I am thinking if we can byte-shift the counter values to some extent, we > might be able to save option space. > > > > Thanks, > > -- > > Yoshi > > > > > > On Mon, Nov 2, 2020 at 4:09 PM Bob Briscoe wrote: > > Michael, > > 1/ I just posted a draft-13 that has the two AccECN TCP Option orders. > > There are also htmlized versions available at: > https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13 > > As agreed, it keys the two option orders on two different Option Kinds, > which I've requested in the IANA section (TBD0 and TBD1). This replaces > using the initial values of the fields. I've reverted all the > complicated text about: > * varying the initial field values dependent on option order, > * having to keep consistent order throughout a connection, > * and choosing which order from each end. > All of which is irrelevant now. > > It is much simpler (I guess we all knew that). Thank you. > > As mentioned, I found my other ToDo list: > 2/ We also introduced considerably more guidance on middlebox handling, > esp. ACK filtering and Segmentation Offload. There was enough extra text > to be worth breaking that section into subsections. > > The guidance on ACK filtering for connections with AccECN negotiated > updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in > the presence of ECN was a research issue). But there is still no > guidance for ECN connections when AccECN has not been negotiated. > > 3/ I also added a SHOULD requirement I had on my ToDo list: Each Data > Sender SHOULD disable ECT in one direction if all the feedback for three > or four rounds from the start is CE. That's already done in Linux, and > in the TCP in iOS and MacOS at least. > > > > Bob > > > > > > On 28/10/2020 17:17, Bob Briscoe wrote: > > Michael, > > > > On 28/10/2020 15:55, Scharf, Michael wrote: > >> Hi Bob, > >> > >> What is the future plan regarding the option encoding proposed by the > >> authors? > >> > >> This has been discussed in past meetings and on the list, for > >> instance: > >> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo= / > >> > >> I have explained several times my pushback against the encoding > >> currently described in the document, most notably the requirement to > >> keep state in the endpoints to decode the format of a TCP option. I > >> have not changed my mind. > >> > >> I have also mentioned that, for instance, two option codepoints would > >> IMHO be a better engineering approach ("KISS principle"). That would > >> be a very simple change to the content to the document. Needless to > >> say that there are other alternatives for marshalling parameters in > >> an unambigous way. > > > > [BB] Whoops. In my head I had already shifted to your scheme with > > two-option kinds. But I see that, in the draft, I haven't. Whoops - > > that was a bug in my ToDo list processing. Sorry about that. I'll > > submit another draft before the deadline with that scheme replacing > > the current one. > > > > I've just found some other entries in my ToDo list for this draft > > which I missed as well. I'll include them too. > > > > Sorry again. > > > > > > Bob > > > >> > >> There have been other related comments, e.g. from Ilpo. > >> > >> Best regards > >> > >> Michael (with no hat) > >> > >>> -----Original Message----- > >>> From: tcpm On Behalf Of Bob Briscoe > >>> Sent: Wednesday, October 28, 2020 2:04 PM > >>> To: tcpm@ietf.org > >>> Cc: Scheffenegger, Richard ; Mirja > >>> Kuehlewind > >>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-12.txt > >>> > >>> TCPM folks, > >>> > >>> Back in May'20, it was decided to hold back the AccECN (and the ECN++= ) > >>> drafts for a while until the position on L4S in tsvwg was clearer. > >>> There > >>> is no formal dependency on the L4S drafts in tsvwg, but it made sense > >>> not to burn codepoints without more certainty. > >>> > >>> I've just refreshed the AccECN and the ECN++ drafts, which had expire= d. > >>> > >>> I included the following minor changes to AccECN, which you can see v= ia > >>> the diff link below, but in summary: > >>> * moved wrongly categorized "action if unwilling to provide f/b" to t= he > >>> group of bullets relevant to the receiver > >>> * removed refs to long-expired ecn-fallback draft > >>> * fixed table 1 after xml2rfc changed formatting > >>> > >>> Thanks to those who pointed these out. > >>> > >>> No changes to ECN++ other than update of some refs. > >>> > >>> Cheers > >>> > >>> > >>> Bob > >>> > >>> On 28/10/2020 12:47, internet-drafts@ietf.org wrote: > >>>> A New Internet-Draft is available from the on-line Internet-Drafts > >>>> directories. > >>>> This draft is a work item of the TCP Maintenance and Minor > >>>> Extensions WG of > >>> the IETF. > >>>> Title : More Accurate ECN Feedback in TCP > >>>> Authors : Bob Briscoe > >>>> Mirja K=C3=BChlewind > >>>> Richard Scheffenegger > >>>> Filename : draft-ietf-tcpm-accurate-ecn-12.txt > >>>> Pages : 58 > >>>> Date : 2020-10-28 > >>>> > >>>> Abstract: > >>>> Explicit Congestion Notification (ECN) is a mechanism where > >>>> network > >>>> nodes can mark IP packets instead of dropping them to indicate > >>>> incipient congestion to the end-points. Receivers with an ECN- > >>>> capable transport protocol feed back this information to the > >>>> sender. > >>>> ECN is specified for TCP in such a way that only one feedback > >>>> signal > >>>> can be transmitted per Round-Trip Time (RTT). Recent new TCP > >>>> mechanisms like Congestion Exposure (ConEx), Data Center TCP > >>>> (DCTCP) > >>>> or Low Latency Low Loss Scalable Throughput (L4S) need more > >>>> accurate > >>>> ECN feedback information whenever more than one marking is > >>>> received > >>>> in one RTT. This document specifies a scheme to provide more > >>>> than > >>>> one feedback signal per RTT in the TCP header. Given TCP heade= r > >>>> space is scarce, it allocates a reserved header bit, that was > >>>> previously used for the ECN-Nonce which has now been declared > >>>> historic. It also overloads the two existing ECN flags in the > >>>> TCP > >>>> header. The resulting extra space is exploited to feed back > >>>> the IP- > >>>> ECN field received during the 3-way handshake as well. > >>>> Supplementary > >>>> feedback information can optionally be provided in a new TCP > >>>> option, > >>>> which is never used on the TCP SYN. > >>>> > >>>> > >>>> The IETF datatracker status page for this draft is: > >>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ > >>>> > >>>> There are also htmlized versions available at: > >>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12 > >>>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-1= 2 > >>>> > >>>> A diff from the previous version is available at: > >>>> https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-accurate-ecn-12 > >>>> > >>>> > >>>> Please note that it may take a couple of minutes from the time of > >>>> submission > >>>> until the htmlized version and diff are available at tools.ietf.org. > >>>> > >>>> Internet-Drafts are also available by anonymous FTP at: > >>>> ftp://ftp.ietf.org/internet-drafts/ > >>>> > >>>> > >>>> _______________________________________________ > >>>> tcpm mailing list > >>>> tcpm@ietf.org > >>>> https://www.ietf.org/mailman/listinfo/tcpm > >>> -- > >>> ________________________________________________________________ > >>> Bob Briscoe http://bobbriscoe.net/ > > >>> PRIVILEGED AND CONFIDENTIAL > >>> > >>> _______________________________________________ > >>> tcpm mailing list > >>> tcpm@ietf.org > >>> https://www.ietf.org/mailman/listinfo/tcpm > >> _______________________________________________ > >> tcpm mailing list > >> tcpm@ietf.org > >> https://www.ietf.org/mailman/listinfo/tcpm > > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > > --000000000000b69a9f05b36dabe6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mirja,

It seems to me that we don't have much freed= om for the frequency of the feedback.=C2=A0
The draft says:
=C2=A0=C2=A0=C2=A0 =C2=A0 MUST immediately send an ACK once 'n' CE marks ha= ve arrived since
      the previous ACK, where=
 'n' SHOULD be 2 and MUST be no greater
      than 6.
Or, options are not applied to this?
Also, even though traffic flows one direction, I think =
you might want to send back SACK options, etc.
<=
font face=3D"arial, sans-serif" style=3D"">Another thought is bulk flow traffic tends to s=
end full MSS packets. I feel 1 byte granularity might not be very efficient=
.
In an=
y cases, I just thought saving option space won't be a bad thing if it&=
#39;s not overly difficult.

Thanks,
--
Yoshi


On Fri, Nov 6, =
2020 at 1:30 AM Mirja Kuehlewind <mirja.kuehlewind@ericsson.com> wrote:











Hi Yoshi,


=C2=A0


not sure if that really saves t=
hat much as you don=E2=80=99t need to send the option with every packet, as=
 such you can manage the overhead dynamically as you can decide how often t=
o send. Also
 in a scenario where most of the traffic flows only in on direction the Acc=
ECN is sent in the other direction.


=C2=A0


Mirja


=C2=A0


=C2=A0


=C2=A0




From:
tcpm <tcpm-bounces@ietf.org>=
; on behalf of Yoshifumi Nishida <nsd.ietf@gmail.com>

Date: Friday, 6. November 2020 at 09:54

To: Bob Briscoe <ietf@bobbriscoe.net>

Cc: "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.c=
om>, "tcpm@i=
etf.org" <tc=
pm@ietf.org>, Mirja Kuehlewind <ietf@kuehlewind.net>

Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt<=
u>






=C2=A0








Hi Bob,






=C2=A0






Sorry for the late commen=
ts, I have a naive question=C2=A0about the encoding schemes in the options.=







The byte counters will be=
 incremented by every byte=C2=A0the receiver receives. But, do we really ne=
ed this level of granularity?=C2=A0






I am thinking if we can b=
yte-shift the counter values to some extent, we might be able to save optio=
n space.






=C2=A0






Thanks,






--






Yoshi






=C2=A0




=C2=A0






On Mon, Nov 2, 2020 at 4:=
09 PM Bob Briscoe <ietf@bobbriscoe.net> wrote:






Michael,



1/ I just posted a draft-13 that has the two AccECN TCP Option orders.



There are also htmlized versions available at:

https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13<=
/a>



As agreed, it keys the two option orders on two different Option Kinds, 
which I've requested in the IANA section (TBD0 and TBD1). This replaces=
 

using the initial values of the fields. I've reverted all the 

complicated text about:

* varying the initial field values dependent on option order,

* having to keep consistent order throughout a connection,

* and choosing which order from each end.

All of which is irrelevant now.



It is much simpler (I guess we all knew that). Thank you.



As mentioned, I found my other ToDo list:

2/ We also introduced considerably more guidance on middlebox handling, 
esp. ACK filtering and Segmentation Offload. There was enough extra text 
to be worth breaking that section into subsections.



The guidance on ACK filtering for connections with AccECN negotiated 

updates the (non-)guidance in RFC3449 (BCP69 which said ACK filtering in 
the presence of ECN was a research issue). But there is still no 

guidance for ECN connections when AccECN has not been negotiated.



3/ I also added a SHOULD requirement I had on my ToDo list: Each Data 

Sender SHOULD disable ECT in one direction if all the feedback for three 
or four rounds from the start is CE. That's already done in Linux, and =


in the TCP in iOS and MacOS at least.







Bob











On 28/10/2020 17:17, Bob Briscoe wrote:

> Michael,

>

> On 28/10/2020 15:55, Scharf, Michael wrote:

>> Hi Bob,

>>

>> What is the future plan regarding the option encoding proposed by =
the 

>> authors?

>>

>> This has been discussed in past meetings and on the list, for 

>> instance: 

>> 
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/=


>>

>> I have explained several times my pushback against the encoding 
>> currently described in the document, most notably the requirement =
to 

>> keep state in the endpoints to decode the format of a TCP option. =
I 

>> have not changed my mind.

>>

>> I have also mentioned that, for instance, two option codepoints wo=
uld 

>> IMHO be a better engineering approach ("KISS principle")=
. That would 

>> be a very simple change to the content to the document. Needless t=
o 

>> say that there are other alternatives for marshalling parameters i=
n 

>> an unambigous way.

>

> [BB] Whoops. In my head I had already shifted to your scheme with 

> two-option kinds. But I see that, in the draft,=C2=A0 I haven't. W=
hoops - 

> that was a bug in my ToDo list processing. Sorry about that. I'll =


> submit another draft before the deadline with that scheme replacing 
> the current one.

>

> I've just found some other entries in my ToDo list for this draft =


> which I missed as well. I'll include them too.

>

> Sorry again.

>

>

> Bob

>

>>

>> There have been other related comments, e.g. from Ilpo.

>>

>> Best regards

>>

>> Michael (with no hat)

>>

>>> -----Original Message-----

>>> From: tcpm <tcpm-bounces@ietf.org> On Behalf Of Bob Briscoe

>>> Sent: Wednesday, October 28, 2020 2:04 PM

>>> To: tcpm@ie=
tf.org

>>> Cc: Scheffenegger, Richard <Richard.Scheffenegger@netapp.com=
>; Mirja

>>> Kuehlewind <ietf@kuehlewind.net>

>>> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-1=
2.txt

>>>

>>> TCPM folks,

>>>

>>> Back in May'20, it was decided to hold back the AccECN (an=
d the ECN++)

>>> drafts for a while until the position on L4S in tsvwg was clea=
rer. 

>>> There

>>> is no formal dependency on the L4S drafts in tsvwg, but it mad=
e sense

>>> not to burn codepoints without more certainty.

>>>

>>> I've just refreshed the AccECN and the ECN++ drafts, which=
 had expired.

>>>

>>> I included the following minor changes to AccECN, which you ca=
n see via

>>> the diff link below, but in summary:

>>> * moved wrongly categorized "action if unwilling to provi=
de f/b" to the

>>> group of bullets relevant to the receiver

>>> * removed refs to long-expired ecn-fallback draft

>>> * fixed table 1 after xml2rfc changed formatting

>>>

>>> Thanks to those who pointed these out.

>>>

>>> No changes to ECN++ other than update of some refs.

>>>

>>> Cheers

>>>

>>>

>>> Bob

>>>

>>> On 28/10/2020 12:47, 
internet-drafts@ietf.org wrote:

>>>> A New Internet-Draft is available from the on-line Interne=
t-Drafts 

>>>> directories.

>>>> This draft is a work item of the TCP Maintenance and Minor=
 

>>>> Extensions WG of

>>> the IETF.

>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Tit=
le=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : More Accur=
ate ECN Feedback in TCP

>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Aut=
hors=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : Bob Briscoe

>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Mirja K=C3=BChlewind

>>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Richard Scheffenegger

>>>> =C2=A0=C2=A0=C2=A0=C2=A0Filename=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 : draft-ietf-tcpm-accurate-ecn-12.txt

>>>> =C2=A0=C2=A0=C2=A0=C2=A0Pages=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : 58

>>>> =C2=A0=C2=A0=C2=A0=C2=A0Date=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 : 2020-10-28

>>>>

>>>> Abstract:

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 Explicit Congestion Notification =
(ECN) is a mechanism where 

>>>> network

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 nodes can mark IP packets instead=
 of dropping them to indicate

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 incipient congestion to the end-p=
oints.=C2=A0 Receivers with an ECN-

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 capable transport protocol feed b=
ack this information to the 

>>>> sender.

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN is specified for TCP in such =
a way that only one feedback 

>>>> signal

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 can be transmitted per Round-Trip=
 Time (RTT).=C2=A0 Recent new TCP

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 mechanisms like Congestion Exposu=
re (ConEx), Data Center TCP 

>>>> (DCTCP)

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 or Low Latency Low Loss Scalable =
Throughput (L4S) need more 

>>>> accurate

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN feedback information whenever=
 more than one marking is 

>>>> received

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 in one RTT.=C2=A0 This document s=
pecifies a scheme to provide more 

>>>> than

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 one feedback signal per RTT in th=
e TCP header.=C2=A0 Given TCP header

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 space is scarce, it allocates a r=
eserved header bit, that was

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 previously used for the ECN-Nonce=
 which has now been declared

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 historic.=C2=A0 It also overloads=
 the two existing ECN flags in the 

>>>> TCP

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 header.=C2=A0 The resulting extra=
 space is exploited to feed back 

>>>> the IP-

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 ECN field received during the 3-w=
ay handshake as well. 

>>>> Supplementary

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 feedback information can optional=
ly be provided in a new TCP 

>>>> option,

>>>> =C2=A0=C2=A0=C2=A0=C2=A0 which is never used on the TCP SY=
N.

>>>>

>>>>

>>>> The IETF datatracker status page for this draft is:

>>>> 
https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/

>>>>

>>>> There are also htmlized versions available at:

>>>> 
https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12

>>>> 
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12
>>>>

>>>> A diff from the previous version is available at:

>>>> 
https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-accurate-ecn-12

>>>>

>>>>

>>>> Please note that it may take a couple of minutes from the =
time of 

>>>> submission

>>>> until the htmlized version and diff are available at 
tools.ietf.org.

>>>>

>>>> Internet-Drafts are also available by anonymous FTP at:
>>>> ftp://ftp.ietf.org/internet-drafts/

>>>>

>>>>

>>>> _______________________________________________

>>>> tcpm mailing list

>>>> tcpm@ie=
tf.org

>>>> https://www.ietf.org/mailman/listinfo/tcpm

>>> -- 

>>> ______________________________________________________________=
__

>>> Bob Briscoe 
http://bobbriscoe.net/

>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 PRIVILEGED AND CONFIDENTIAL

>>>

>>> _______________________________________________

>>> tcpm mailing list

>>> tcpm@ietf.o=
rg

>>> https://www.ietf.org/mailman/listinfo/tcpm

>> _______________________________________________

>> tcpm mailing list

>> tcpm@ietf.org

>> https://www.ietf.org/mailman/listinfo/tcpm

>



-- 

________________________________________________________________

Bob Briscoe=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0http://bobbriscoe.net/



_______________________________________________

tcpm mailing list

tcpm@ietf.org

ht=
tps://www.ietf.org/mailman/listinfo/tcpm
--000000000000b69a9f05b36dabe6-- From nobody Fri Nov 6 05:02:02 2020 Return-Path: X-Original-To: tcpm@ietfa.amsl.com Delivered-To: tcpm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E794F3A115A for ; Fri, 6 Nov 2020 05:02:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.346 X-Spam-Level: X-Spam-Status: No, score=-2.346 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.247, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kw8uT5zFyuPY for ; Fri, 6 Nov 2020 05:01:58 -0800 (PST) Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 31CFF3A1157 for ; Fri, 6 Nov 2020 05:01:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=7lHgC50+yYG6mzTC2EewBFygx15dqFr3hd5l5pr162w=; b=n17n1hGgWDrHHPb8xepFx7GxL 9dSQFr5RqCIB+yDGpuN4kem1shPQSvsNQ7OOTPpgH+0AfDXSh3nW+MVdo+xVP082NYM+pGDvvZr3D YHRGQCw3lyYwb6XAkpEvP3tqEp/xowqwA7zByDah36FV2uRukCHvfqSNtblGHKB/nGYGmOxGuU53r D6IkX/dnkAR1MIptAjoQmRwzJE52xGQMlh8k/AnsB0Rlkw4oQx50b2eDL+J0m2uYEatMvGI+rjWJd QYCHPgIoLqWiYz/luYysnFRQ1d2uPbLK8tPZaKBNS5pGjby+mG7DzaYAwzjG4W4QdGiXWU0E3lf2E Gq7FqW7yA==; Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:37462 helo=[192.168.1.6]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kb1NK-00GNLQ-Bq; Fri, 06 Nov 2020 13:01:56 +0000 To: Yoshifumi Nishida , Mirja Kuehlewind Cc: "Scheffenegger, Richard" , "tcpm@ietf.org" , Mirja Kuehlewind References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net> From: Bob Briscoe Message-ID: <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net> Date: Fri, 6 Nov 2020 13:01:53 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------420C8A00D08317BBE04A003A" Content-Language: en-GB X-OutGoing-Spam-Status: No, score=-1.0 X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net X-AntiAbuse: Original Domain - ietf.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - bobbriscoe.net X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net X-Source: X-Source-Args: X-Source-Dir: Archived-At: Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt X-BeenThere: tcpm@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: TCP Maintenance and Minor Extensions Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Nov 2020 13:02:01 -0000 This is a multi-part message in MIME format. --------------420C8A00D08317BBE04A003A Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Yoshi, On 06/11/2020 10:28, Yoshifumi Nishida wrote: > Hi Mirja, > > It seems to me that we don't have much freedom for the frequency of > the feedback. > The draft says: >     MUST immediately send an ACK once 'n' CE marks have arrived since > the previous ACK, where 'n' SHOULD be 2 and MUST be no greater than 6. > Or, options are not applied to this? [BB] Section 3.2.2 about the ACE field, which is in the main TCP header feeds back the count of CE-marked *packets* . Section 3.2.3 is about the AccECN TCP Option, which is optional and feeds back a count of *bytes* for each marking. If there are any places in the draft where this isn't clear, we can fix them. But, in this case, the  text right under your quote says: These rules for when to send an ACK are designed to be complemented by those inSection 3.2.3.3 , which concern whether the AccECN TCP Option ought to be included on ACKs. > Also, even though traffic flows one direction, I think you might want > to send back SACK options, etc. Last para of the Introduction: It is recommended that the AccECN protocol is implemented alongside SACK [RFC2018 ] and ... Also, interactions with SACK are raised where necessary throughout the draft. In particular this one: If the smallest allowed AccECN Option would leave insufficient space for two SACK blocks on a particular ACK, the Data Receiver MUST give precedence to the SACK option (total 18 octets), because loss feedback is more critical. > Another thought is bulk flow traffic tends to send full MSS packets. I > feel 1 byte granularity might not be very efficient. > In any cases, I just thought saving option space won't be a bad thing > if it's not overly difficult. [BB] If efficiency is needed, field(s) (or the whole option) can be omitted. During the history of this draft we have proposed different ways to encode information more efficiently, but WG discussion decided that simple omission or inclusion of fields gives optional efficiency with minimal complexity. I'm not saying the act of reducing the granularity of each field would be particularly complex. Rather, complexity comes from the resulting ambiguity. Here's an example: * The Linux code currently uses the byte counters to help disambiguate wrap of the 3-bit packet counter. * If for instance the byte counters were truncated by 8 bits for efficiency, it would add many more ambiguous cases for this code to have to take into account. * Such complexity introduces misunderstanding and bugs. The design has gone round and round many times. We have tried to converge on the best tradeoffs. Best Regards Bob > Thanks, > -- > Yoshi > > On Fri, Nov 6, 2020 at 1:30 AM Mirja Kuehlewind > > > wrote: > > Hi Yoshi, > > not sure if that really saves that much as you don’t need to send > the option with every packet, as such you can manage the overhead > dynamically as you can decide how often to send. Also in a > scenario where most of the traffic flows only in on direction the > AccECN is sent in the other direction. > > Mirja > > *From: *tcpm > on behalf of Yoshifumi Nishida > > > *Date: *Friday, 6. November 2020 at 09:54 > *To: *Bob Briscoe > > *Cc: *"Scheffenegger, Richard" >, "tcpm@ietf.org > " >, > Mirja Kuehlewind > > *Subject: *Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt > > Hi Bob, > > Sorry for the late comments, I have a naive question about the > encoding schemes in the options. > > The byte counters will be incremented by every byte the receiver > receives. But, do we really need this level of granularity? > > I am thinking if we can byte-shift the counter values to some > extent, we might be able to save option space. > > Thanks, > > -- > > Yoshi > > On Mon, Nov 2, 2020 at 4:09 PM Bob Briscoe > wrote: > > Michael, > > 1/ I just posted a draft-13 that has the two AccECN TCP Option > orders. > > There are also htmlized versions available at: > https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13 > > As agreed, it keys the two option orders on two different > Option Kinds, > which I've requested in the IANA section (TBD0 and TBD1). This > replaces > using the initial values of the fields. I've reverted all the > complicated text about: > * varying the initial field values dependent on option order, > * having to keep consistent order throughout a connection, > * and choosing which order from each end. > All of which is irrelevant now. > > It is much simpler (I guess we all knew that). Thank you. > > As mentioned, I found my other ToDo list: > 2/ We also introduced considerably more guidance on middlebox > handling, > esp. ACK filtering and Segmentation Offload. There was enough > extra text > to be worth breaking that section into subsections. > > The guidance on ACK filtering for connections with AccECN > negotiated > updates the (non-)guidance in RFC3449 (BCP69 which said ACK > filtering in > the presence of ECN was a research issue). But there is still no > guidance for ECN connections when AccECN has not been negotiated. > > 3/ I also added a SHOULD requirement I had on my ToDo list: > Each Data > Sender SHOULD disable ECT in one direction if all the feedback > for three > or four rounds from the start is CE. That's already done in > Linux, and > in the TCP in iOS and MacOS at least. > > > > Bob > > > > > > On 28/10/2020 17:17, Bob Briscoe wrote: > > Michael, > > > > On 28/10/2020 15:55, Scharf, Michael wrote: > >> Hi Bob, > >> > >> What is the future plan regarding the option encoding > proposed by the > >> authors? > >> > >> This has been discussed in past meetings and on the list, for > >> instance: > >> > https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/ > >> > >> I have explained several times my pushback against the > encoding > >> currently described in the document, most notably the > requirement to > >> keep state in the endpoints to decode the format of a TCP > option. I > >> have not changed my mind. > >> > >> I have also mentioned that, for instance, two option > codepoints would > >> IMHO be a better engineering approach ("KISS principle"). > That would > >> be a very simple change to the content to the document. > Needless to > >> say that there are other alternatives for marshalling > parameters in > >> an unambigous way. > > > > [BB] Whoops. In my head I had already shifted to your scheme > with > > two-option kinds. But I see that, in the draft,  I haven't. > Whoops - > > that was a bug in my ToDo list processing. Sorry about that. > I'll > > submit another draft before the deadline with that scheme > replacing > > the current one. > > > > I've just found some other entries in my ToDo list for this > draft > > which I missed as well. I'll include them too. > > > > Sorry again. > > > > > > Bob > > > >> > >> There have been other related comments, e.g. from Ilpo. > >> > >> Best regards > >> > >> Michael (with no hat) > >> > >>> -----Original Message----- > >>> From: tcpm > On Behalf Of Bob Briscoe > >>> Sent: Wednesday, October 28, 2020 2:04 PM > >>> To: tcpm@ietf.org > >>> Cc: Scheffenegger, Richard > >; Mirja > >>> Kuehlewind > > >>> Subject: Re: [tcpm] I-D Action: > draft-ietf-tcpm-accurate-ecn-12.txt > >>> > >>> TCPM folks, > >>> > >>> Back in May'20, it was decided to hold back the AccECN > (and the ECN++) > >>> drafts for a while until the position on L4S in tsvwg was > clearer. > >>> There > >>> is no formal dependency on the L4S drafts in tsvwg, but it > made sense > >>> not to burn codepoints without more certainty. > >>> > >>> I've just refreshed the AccECN and the ECN++ drafts, which > had expired. > >>> > >>> I included the following minor changes to AccECN, which > you can see via > >>> the diff link below, but in summary: > >>> * moved wrongly categorized "action if unwilling to > provide f/b" to the > >>> group of bullets relevant to the receiver > >>> * removed refs to long-expired ecn-fallback draft > >>> * fixed table 1 after xml2rfc changed formatting > >>> > >>> Thanks to those who pointed these out. > >>> > >>> No changes to ECN++ other than update of some refs. > >>> > >>> Cheers > >>> > >>> > >>> Bob > >>> > >>> On 28/10/2020 12:47, internet-drafts@ietf.org > wrote: > >>>> A New Internet-Draft is available from the on-line > Internet-Drafts > >>>> directories. > >>>> This draft is a work item of the TCP Maintenance and Minor > >>>> Extensions WG of > >>> the IETF. > >>>>           Title           : More Accurate ECN Feedback in TCP > >>>>           Authors         : Bob Briscoe > >>>> Mirja Kühlewind > >>>> Richard Scheffenegger > >>>>     Filename        : draft-ietf-tcpm-accurate-ecn-12.txt > >>>>     Pages           : 58 > >>>>     Date            : 2020-10-28 > >>>> > >>>> Abstract: > >>>>      Explicit Congestion Notification (ECN) is a > mechanism where > >>>> network > >>>>      nodes can mark IP packets instead of dropping them > to indicate > >>>>      incipient congestion to the end-points.  Receivers > with an ECN- > >>>>      capable transport protocol feed back this > information to the > >>>> sender. > >>>>      ECN is specified for TCP in such a way that only one > feedback > >>>> signal > >>>>      can be transmitted per Round-Trip Time (RTT).  > Recent new TCP > >>>>      mechanisms like Congestion Exposure (ConEx), Data > Center TCP > >>>> (DCTCP) > >>>>      or Low Latency Low Loss Scalable Throughput (L4S) > need more > >>>> accurate > >>>>      ECN feedback information whenever more than one > marking is > >>>> received > >>>>      in one RTT.  This document specifies a scheme to > provide more > >>>> than > >>>>      one feedback signal per RTT in the TCP header.  > Given TCP header > >>>>      space is scarce, it allocates a reserved header bit, > that was > >>>>      previously used for the ECN-Nonce which has now been > declared > >>>>      historic.  It also overloads the two existing ECN > flags in the > >>>> TCP > >>>>      header.  The resulting extra space is exploited to > feed back > >>>> the IP- > >>>>      ECN field received during the 3-way handshake as well. > >>>> Supplementary > >>>>      feedback information can optionally be provided in a > new TCP > >>>> option, > >>>>      which is never used on the TCP SYN. > >>>> > >>>> > >>>> The IETF datatracker status page for this draft is: > >>>> > https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/ > >>>> > >>>> There are also htmlized versions available at: > >>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12 > >>>> > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12 > >>>> > >>>> A diff from the previous version is available at: > >>>> > https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-12 > >>>> > >>>> > >>>> Please note that it may take a couple of minutes from the > time of > >>>> submission > >>>> until the htmlized version and diff are available at > tools.ietf.org . > >>>> > >>>> Internet-Drafts are also available by anonymous FTP at: > >>>> ftp://ftp.ietf.org/internet-drafts/ > >>>> > >>>> > >>>> _______________________________________________ > >>>> tcpm mailing list > >>>> tcpm@ietf.org > >>>> https://www.ietf.org/mailman/listinfo/tcpm > >>> -- > >>> > ________________________________________________________________ > >>> Bob Briscoe http://bobbriscoe.net/ > > >>>                  PRIVILEGED AND CONFIDENTIAL > >>> > >>> _______________________________________________ > >>> tcpm mailing list > >>> tcpm@ietf.org > >>> https://www.ietf.org/mailman/listinfo/tcpm > >> _______________________________________________ > >> tcpm mailing list > >> tcpm@ietf.org > >> https://www.ietf.org/mailman/listinfo/tcpm > > > > -- > ________________________________________________________________ > Bob Briscoe http://bobbriscoe.net/ > > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > -- ________________________________________________________________ Bob Briscoe http://bobbriscoe.net/ --------------420C8A00D08317BBE04A003A Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Yoshi,

On 06/11/2020 10:28, Yoshifumi Nishida wrote:
Hi Mirja,

It seems to me that we don't have much freedom for the frequency of the feedback. 
The draft says:
      MUST immediately send an ACK once 'n' CE marks have arrived since
      the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
      than 6.
Or, options are not applied to this?

[BB]
Section 3.2.2 about the ACE field, which is in the main TCP header feeds back the count of CE-marked *packets* .
Section 3.2.3 is about the AccECN TCP Option, which is optional and feeds back a count of *bytes* for each marking.

If there are any places in the draft where this isn't clear, we can fix them. But, in this case, the  text right under your quote says:
   These rules for when to send an ACK are designed to be complemented
   by those in Section 3.2.3.3, which concern whether the AccECN TCP
   Option ought to be included on ACKs.


Also, even though traffic flows one direction, I think you might want to send back SACK options, etc.

Last para of the Introduction:
   It is recommended that the AccECN protocol is implemented alongside
   SACK [RFC2018] and ...

Also, interactions with SACK are raised where necessary throughout the draft. In particular this one:
      If the smallest allowed AccECN Option would leave insufficient
      space for two SACK blocks on a particular ACK, the Data Receiver
      MUST give precedence to the SACK option (total 18 octets), because
      loss feedback is more critical.

Another thought is bulk flow traffic tends to send full MSS packets. I feel 1 byte granularity might not be very efficient.
In any cases, I just thought saving option space won't be a bad thing if it's not overly difficult.

[BB] If efficiency is needed, field(s) (or the whole option) can be omitted.
During the history of this draft we have proposed different ways to encode information more efficiently, but WG discussion decided that simple omission or inclusion of fields gives optional efficiency with minimal complexity.

I'm not saying the act of reducing the granularity of each field would be particularly complex.
Rather, complexity comes from the resulting ambiguity. Here's an example:
* The Linux code currently uses the byte counters to help disambiguate wrap of the 3-bit packet counter.
* If for instance the byte counters were truncated by 8 bits for efficiency, it would add many more ambiguous cases for this code to have to take into account.
* Such complexity introduces misunderstanding and bugs.

The design has gone round and round many times. We have tried to converge on the best tradeoffs.

Best Regards


Bob



Thanks,
--
Yoshi

        

        

          
On Fri, Nov 6, 2020 at 1:30
            AM Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>
            wrote:

          

          

            

              

                
Hi Yoshi,

                
 

                
not sure if that
                    really saves that much as you don’t need to send the
                    option with every packet, as such you can manage the
                    overhead dynamically as you can decide how often to
                    send. Also in a scenario where most of the traffic
                    flows only in on direction the AccECN is sent in the
                    other direction.

                
 

                
Mirja

                
 

                
 

                
 

                

                  
From:
                      tcpm <tcpm-bounces@ietf.org>
                      on behalf of Yoshifumi Nishida <nsd.ietf@gmail.com>

                      Date: Friday, 6. November 2020 at 09:54

                      To: Bob Briscoe <ietf@bobbriscoe.net>

                      Cc: "Scheffenegger, Richard" <Richard.Scheffenegger@netapp.com>,
                      "tcpm@ietf.org" <tcpm@ietf.org>,
                      Mirja Kuehlewind <ietf@kuehlewind.net>

                      Subject: Re: [tcpm] I-D Action:
                      draft-ietf-tcpm-accurate-ecn-13.txt

                

                

                  
 

                

                

                  

                    
Hi
                      Bob,

                  

                  

                    
 

                  

                  

                    
Sorry
                      for the late comments, I have a naive
                      question about the encoding schemes in the
                      options.

                  

                  

                    
The
                      byte counters will be incremented by every
                      byte the receiver receives. But, do we really need
                      this level of granularity? 

                  

                  

                    
I am
                      thinking if we can byte-shift the counter values
                      to some extent, we might be able to save option
                      space.

                  

                  

                    
 

                  

                  

                    
Thanks,

                  

                  

                    
--

                  

                  

                    
Yoshi

                  

                  

                    
 

                  

                  
 

                  

                    

                      
On
                        Mon, Nov 2, 2020 at 4:09 PM Bob Briscoe <ietf@bobbriscoe.net>
                        wrote:

                    

                    

                      
Michael,

                        

                        1/ I just posted a draft-13 that has the two
                        AccECN TCP Option orders.

                        

                        There are also htmlized versions available at:

                        https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-13

                        

                        As agreed, it keys the two option orders on two
                        different Option Kinds, 

                        which I've requested in the IANA section (TBD0
                        and TBD1). This replaces 

                        using the initial values of the fields. I've
                        reverted all the 

                        complicated text about:

                        * varying the initial field values dependent on
                        option order,

                        * having to keep consistent order throughout a
                        connection,

                        * and choosing which order from each end.

                        All of which is irrelevant now.

                        

                        It is much simpler (I guess we all knew that).
                        Thank you.

                        

                        As mentioned, I found my other ToDo list:

                        2/ We also introduced considerably more guidance
                        on middlebox handling, 

                        esp. ACK filtering and Segmentation Offload.
                        There was enough extra text 

                        to be worth breaking that section into
                        subsections.

                        

                        The guidance on ACK filtering for connections
                        with AccECN negotiated 

                        updates the (non-)guidance in RFC3449 (BCP69
                        which said ACK filtering in 

                        the presence of ECN was a research issue). But
                        there is still no 

                        guidance for ECN connections when AccECN has not
                        been negotiated.

                        

                        3/ I also added a SHOULD requirement I had on my
                        ToDo list: Each Data 

                        Sender SHOULD disable ECT in one direction if
                        all the feedback for three 

                        or four rounds from the start is CE. That's
                        already done in Linux, and 

                        in the TCP in iOS and MacOS at least.

                        

                        

                        

                        Bob

                        

                        

                        

                        

                        

                        On 28/10/2020 17:17, Bob Briscoe wrote:

                        > Michael,

                        >

                        > On 28/10/2020 15:55, Scharf, Michael wrote:

                        >> Hi Bob,

                        >>

                        >> What is the future plan regarding the
                        option encoding proposed by the 

                        >> authors?

                        >>

                        >> This has been discussed in past
                        meetings and on the list, for 

                        >> instance: 

                        >> 
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/

                        >>

                        >> I have explained several times my
                        pushback against the encoding 

                        >> currently described in the document,
                        most notably the requirement to 

                        >> keep state in the endpoints to decode
                        the format of a TCP option. I 

                        >> have not changed my mind.

                        >>

                        >> I have also mentioned that, for
                        instance, two option codepoints would 

                        >> IMHO be a better engineering approach
                        ("KISS principle"). That would 

                        >> be a very simple change to the content
                        to the document. Needless to 

                        >> say that there are other alternatives
                        for marshalling parameters in 

                        >> an unambigous way.

                        >

                        > [BB] Whoops. In my head I had already
                        shifted to your scheme with 

                        > two-option kinds. But I see that, in the
                        draft,  I haven't. Whoops - 

                        > that was a bug in my ToDo list processing.
                        Sorry about that. I'll 

                        > submit another draft before the deadline
                        with that scheme replacing 

                        > the current one.

                        >

                        > I've just found some other entries in my
                        ToDo list for this draft 

                        > which I missed as well. I'll include them
                        too.

                        >

                        > Sorry again.

                        >

                        >

                        > Bob

                        >

                        >>

                        >> There have been other related comments,
                        e.g. from Ilpo.

                        >>

                        >> Best regards

                        >>

                        >> Michael (with no hat)

                        >>

                        >>> -----Original Message-----

                        >>> From: tcpm <tcpm-bounces@ietf.org>
                        On Behalf Of Bob Briscoe

                        >>> Sent: Wednesday, October 28, 2020
                        2:04 PM

                        >>> To: tcpm@ietf.org

                        >>> Cc: Scheffenegger, Richard <Richard.Scheffenegger@netapp.com>;
                        Mirja

                        >>> Kuehlewind <ietf@kuehlewind.net>

                        >>> Subject: Re: [tcpm] I-D Action:
                        draft-ietf-tcpm-accurate-ecn-12.txt

                        >>>

                        >>> TCPM folks,

                        >>>

                        >>> Back in May'20, it was decided to
                        hold back the AccECN (and the ECN++)

                        >>> drafts for a while until the
                        position on L4S in tsvwg was clearer. 

                        >>> There

                        >>> is no formal dependency on the L4S
                        drafts in tsvwg, but it made sense

                        >>> not to burn codepoints without more
                        certainty.

                        >>>

                        >>> I've just refreshed the AccECN and
                        the ECN++ drafts, which had expired.

                        >>>

                        >>> I included the following minor
                        changes to AccECN, which you can see via

                        >>> the diff link below, but in
                        summary:

                        >>> * moved wrongly categorized "action
                        if unwilling to provide f/b" to the

                        >>> group of bullets relevant to the
                        receiver

                        >>> * removed refs to long-expired
                        ecn-fallback draft

                        >>> * fixed table 1 after xml2rfc
                        changed formatting

                        >>>

                        >>> Thanks to those who pointed these
                        out.

                        >>>

                        >>> No changes to ECN++ other than
                        update of some refs.

                        >>>

                        >>> Cheers

                        >>>

                        >>>

                        >>> Bob

                        >>>

                        >>> On 28/10/2020 12:47, 
                          internet-drafts@ietf.org wrote:

                        >>>> A New Internet-Draft is
                        available from the on-line Internet-Drafts 

                        >>>> directories.

                        >>>> This draft is a work item of
                        the TCP Maintenance and Minor 

                        >>>> Extensions WG of

                        >>> the IETF.

                        >>>>           Title           :
                        More Accurate ECN Feedback in TCP

                        >>>>           Authors         : Bob
                        Briscoe

                        >>>>                            
                        Mirja Kühlewind

                        >>>>                            
                        Richard Scheffenegger

                        >>>>     Filename        :
                        draft-ietf-tcpm-accurate-ecn-12.txt

                        >>>>     Pages           : 58

                        >>>>     Date            :
                        2020-10-28

                        >>>>

                        >>>> Abstract:

                        >>>>      Explicit Congestion
                        Notification (ECN) is a mechanism where 

                        >>>> network

                        >>>>      nodes can mark IP packets
                        instead of dropping them to indicate

                        >>>>      incipient congestion to
                        the end-points.  Receivers with an ECN-

                        >>>>      capable transport protocol
                        feed back this information to the 

                        >>>> sender.

                        >>>>      ECN is specified for TCP
                        in such a way that only one feedback 

                        >>>> signal

                        >>>>      can be transmitted per
                        Round-Trip Time (RTT).  Recent new TCP

                        >>>>      mechanisms like Congestion
                        Exposure (ConEx), Data Center TCP 

                        >>>> (DCTCP)

                        >>>>      or Low Latency Low Loss
                        Scalable Throughput (L4S) need more 

                        >>>> accurate

                        >>>>      ECN feedback information
                        whenever more than one marking is 

                        >>>> received

                        >>>>      in one RTT.  This document
                        specifies a scheme to provide more 

                        >>>> than

                        >>>>      one feedback signal per
                        RTT in the TCP header.  Given TCP header

                        >>>>      space is scarce, it
                        allocates a reserved header bit, that was

                        >>>>      previously used for the
                        ECN-Nonce which has now been declared

                        >>>>      historic.  It also
                        overloads the two existing ECN flags in the 

                        >>>> TCP

                        >>>>      header.  The resulting
                        extra space is exploited to feed back 

                        >>>> the IP-

                        >>>>      ECN field received during
                        the 3-way handshake as well. 

                        >>>> Supplementary

                        >>>>      feedback information can
                        optionally be provided in a new TCP 

                        >>>> option,

                        >>>>      which is never used on the
                        TCP SYN.

                        >>>>

                        >>>>

                        >>>> The IETF datatracker status
                        page for this draft is:

                        >>>> 
https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/

                        >>>>

                        >>>> There are also htmlized
                        versions available at:

                        >>>> 
https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-12

                        >>>> 
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-12

                        >>>>

                        >>>> A diff from the previous
                        version is available at:

                        >>>> 
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-accurate-ecn-12

                        >>>>

                        >>>>

                        >>>> Please note that it may take a
                        couple of minutes from the time of 

                        >>>> submission

                        >>>> until the htmlized version and
                        diff are available at 
                          tools.ietf.org.

                        >>>>

                        >>>> Internet-Drafts are also
                        available by anonymous FTP at:

                        >>>> ftp://ftp.ietf.org/internet-drafts/

                        >>>>

                        >>>>

                        >>>>
                        _______________________________________________

                        >>>> tcpm mailing list

                        >>>> tcpm@ietf.org

                        >>>> https://www.ietf.org/mailman/listinfo/tcpm

                        >>> -- 

                        >>>
                        ________________________________________________________________

                        >>> Bob Briscoe 
                          http://bobbriscoe.net/

                        >>>                  PRIVILEGED AND
                        CONFIDENTIAL

                        >>>

                        >>>
                        _______________________________________________

                        >>> tcpm mailing list

                        >>> tcpm@ietf.org

                        >>> https://www.ietf.org/mailman/listinfo/tcpm

                        >>
                        _______________________________________________

                        >> tcpm mailing list

                        >> tcpm@ietf.org

                        >> https://www.ietf.org/mailman/listinfo/tcpm

                        >

                        

                        -- 

________________________________________________________________

                        Bob Briscoe                               http://bobbriscoe.net/

                        

                        _______________________________________________

                        tcpm mailing list

                        tcpm@ietf.org

                        https://www.ietf.org/mailman/listinfo/tcpm

                    

                  

                

              

            

          

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------420C8A00D08317BBE04A003A--


From nobody Fri Nov  6 06:01:15 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 67A673A11C0 for ; Fri,  6 Nov 2020 06:01:14 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.145
X-Spam-Level: 
X-Spam-Status: No, score=-2.145 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.247, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CHD3EOI_cDcc for ; Fri,  6 Nov 2020 06:01:12 -0800 (PST)
Received: from pegasus.erg.abdn.ac.uk (pegasus.erg.abdn.ac.uk [137.50.19.135]) by ietfa.amsl.com (Postfix) with ESMTP id 558673A11AF for ; Fri,  6 Nov 2020 06:01:11 -0800 (PST)
Received: from GF-MacBook-Pro.lan (fgrpf.plus.com [212.159.18.54]) by pegasus.erg.abdn.ac.uk (Postfix) with ESMTPSA id 157A11B002AE; Fri,  6 Nov 2020 14:01:08 +0000 (GMT)
To: Bob Briscoe 
References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net>     <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>
Cc: tcpm IETF list 
From: Gorry Fairhurst 
Message-ID: 
Date: Fri, 6 Nov 2020 14:01:07 +0000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:78.0) Gecko/20100101 Thunderbird/78.3.3
MIME-Version: 1.0
In-Reply-To: <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-GB
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 06 Nov 2020 14:01:14 -0000

I see this rev introduced a section on ACK Filtering, which I have 
specific comments on. We need to consider the cost of sending ACKs - 
especially anything that results in them being large - however, this 
needs to be done by choosing a default at the receiver - not as an 
adaptation, the adaption if necessary needs to be introduce more ACKs 
when this is beneficial, as in some cases for DAAS, not in trying to 
figure something from delayed or "missing" ACKs.

1) So let me say first that I think the suggestion for updating the 
receiver to thin, and every 2-6 packets seems reasonable.

     “MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
       than 6.”

I do agree that endpoints can do much more to reduce the return path 
capacity that they consume - and that the endpoints have the opportunity 
to encode the feedback information more efficiently than can be done by 
a device (or queue) in the network. From our experiments, n=10 would 
also often be likely OK.

2. Althouigh in some cases a receiver can predict a good "n", I do not 
believe a receiver can typically find out how to choose “n” by observing 
the ACKs on a connection.

Sure, if the path is modelled as a simple bit-congestive path that has a 
queue, then you could monitoring timing/loss of packets and determine a 
rate that tries not to fill a queue. I would not know how common such a 
bottleneck link is.

I do know that many bottleneck links are much more complex and need to 
consider the scheduling transmission of small ACK packets on the return 
path performance. In many radio technologies, the overhead from 
transmission of ACKs is proportionally much higher than for data 
segments. These layer 2s are often constrained in terms of packets (e.g. 
where lower layer framing and transmission scheduling is an important 
consideration to the efficiency/cost of using the service). This has 
often been “fixed” using ACK Thinning or PEPs. Protocols that send fewer 
ACKs/data packet are not impacted.


2. I do not believe a receiver can find out how top to choose “n” by 
observing the ACKs received on a connection.

Sure, if the path is modelled as a simple bit-congestive path that has a 
queue, then you could monitoring timing/loss of packets and determine a 
rate that tries not to fill a queue. I would not know how common such a 
bottleneck link is.

I do know that many bottleneck links are more complex and needs to 
consider the scheduling transmission of small ACK packets on the return 
path performance. In many radio technologies, the overhead from 
transmission of ACKs is proportionally much higher than for data 
segments -  These layer 2s are constrained in terms of packets (e.g. 
where lower layer framing and transmission scheduling is an important 
consideration to the efficiency/cost of using the service). This has 
often been “fixed” using ACK Thinning or PEPs. Protocols that send fewer 
ACKs/data packet are not impacted.

Scheduling (at the IP layer,can also often result in variations in 
timing - something that is hard to predict and I would suggests make 
automated selection of “n” very hard, and the need to thin is often not 
directly visible to a user of the path. Traffic multiplexed with 
protocols that use different ACK policies (perhaps QUIC?) also make this 
sort of detection less reliable.

3) A queue might be competing for scheduling opportunities against other 
IP flows (e.g., WFQ), or competing for transmission opportunities with a 
shared resource (common in radio links).

ACKs can and do take transmission opportunities away from the resource 
pools that would be used by other services that share the same resource 
pool. This can be reduced by some form of per-flow scheduling with 
deeper per-flow queues - but that in itself can induce unwanted coupling 
between flows - limiting the peak rate of one flow, to achieve a higher 
ACK ratio for another flow.

Consider a QUIC flow sharing a return path with a TCP flow, the higher 
rate of larger QUIC ACKs reduces the transmission opportunities for TCP 
ACKs, which in turn suppresses the TCP flow, limiting the overall system 
performance. If the ACKs use the same transmission resource, they could 
take directly away from transmission opportunities in the forward 
direction, but even if they do not - the rate of ACKs retuned impacts 
the sending rate end to end.

4) The new section adds some cross-layer requirements on sub-IP layers, 
which seem to imply more complexity. I’m not comfortable with such a 
proposal.

It seems that the text suggests not implement thinning and that these 
systems implement logic to detect the presence of the option by 
observing SYNs and then configure the link to do something special. I 
don’t think think this is the correct approach - the result will be 
throttling of the connection speed. Is it necessary that the sub-IP 
layer does not implement thinning for connections that use AccECN feedback?

When the text says: “SHOULD preserve the timing of each ACK”
- does that imply that nodes need to discard rather than delay packets 
when link scheduling or transmission scheduling is used. This seems a 
very odd requirement, it might be better too simply strip-out the AccECN 
option or discard all packets that set this.

I can’t see how queuing the ACKs, or deleting ACKs rather than queuing 
them would improve the performance of AccECN TCP connections.

Gorry


From nobody Fri Nov  6 06:10:36 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D8A253A11CF for ; Fri,  6 Nov 2020 06:10:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.6
X-Spam-Level: 
X-Spam-Status: No, score=-17.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZskFFHtfdCsz for ; Fri,  6 Nov 2020 06:10:33 -0800 (PST)
Received: from mail-ua1-x92f.google.com (mail-ua1-x92f.google.com [IPv6:2607:f8b0:4864:20::92f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id AF76E3A11C9 for ; Fri,  6 Nov 2020 06:10:33 -0800 (PST)
Received: by mail-ua1-x92f.google.com with SMTP id 91so426982uar.5 for ; Fri, 06 Nov 2020 06:10:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=A3Og9aZePRqtwZDsqaPxN4XTl46G86vDz9hUFBWQgTI=; b=dmC8cZ1vyyxCuJoEY36Khc62+OVAi2gfStTVbEhM3mkTEKpOPgv/k7ltEIC9YV/IZh StkwBxPHOrh4+nm4GvwVmzL240fVnpXR/8LJ1MYxV21WhBLXOTgUJ0ZkEg1ly0o2Enuk t19uA1nZHypWsMpHcUrOc6C7jwqtu1Kr3o3lCpBpykWHMQnC++cnzXjJ1zhhb0lxo4K5 4V+eNrwpj6L0G2wYzgq4RuY4WmblDib5WlgosNSBz4XbJLFtOPZYHmA0IExdAB7eX40u iw3dCH+q+etliulw7j/OBvtYapNJqlUBcPMiMZLHqw4fqPzuUbsQ7jSPfBctNaoOKbco tQcA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=A3Og9aZePRqtwZDsqaPxN4XTl46G86vDz9hUFBWQgTI=; b=GZcrVE0qhiqWA3bPnn+7tPQgBWOoT2rjT1hC8zzCm03CAWJrVWD4AcUlBO4nywLXvl dRsiRqrRUJxRK/5wXmJHv47TMOWR2pK0okD8TtgGQ9FpOWJgMFJTwRoC+aBG2jfqSYBi buYqKqu2R7RwW06K+q+UuDKg8tmIpoT9NUziksN40qP/EzRv185HMqqCG3/YNeimnQzX VsmVUeD74l9hgBhL9z5tVm2QFI3+nE2I8gNsZ8b++8FslIFnghUkk3bnQkkYhbsnG9LI 6yVK0XpBw5qSZdIElyqIgTNmu549nQMp4G8Bnk0ZMxW4tSDIe8mK8iFJaEPzPskE4cyM CaqQ==
X-Gm-Message-State: AOAM530f442OxqTvzy5Zx/njGTLPoi1ogmQPA/UKzvsqj/slPrUAUlZY b4uTnorKKf0FHMOpwamY9/NSI0xm0t3fYGMt4onJSg==
X-Google-Smtp-Source: ABdhPJw3VAS+XdM2tqZspoGETTgncVY/dmnljOVDyr3DgRt/J0KOknEuywIOrdaXSKsk0bIfbuQew95keST8GQ+yHw4=
X-Received: by 2002:ab0:424:: with SMTP id 33mr807086uav.33.1604671832394; Fri, 06 Nov 2020 06:10:32 -0800 (PST)
MIME-Version: 1.0
References:   
In-Reply-To: 
From: Neal Cardwell 
Date: Fri, 6 Nov 2020 09:10:15 -0500
Message-ID: 
To: Lars Eggert 
Cc: Vidhi Goel , "tcpm@ietf.org" ,  "Rosenblum, Wesley" , Martin Thomson , iccrg IRTF list 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [iccrg]  Cwnd growth after fast convergence in CUBIC
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 06 Nov 2020 14:10:35 -0000

On Fri, Nov 6, 2020 at 3:28 AM Lars Eggert  wrote:
> That said, the intent was to document what the Linux code was doing
> at the time. Is there a difference between the RFC and what Linux
> has been doing or is doing now? If yes, it might be time to revise
> the RFC.

Yes, it seems that this is a case where the RFC differs from the Linux
TCP CUBIC code.

The RFC says:
      K = cubic_root(W_max*(1-beta_cubic)/C) (Eq. 2)

which is equivalent to:
      K = cubic_root((W_max - W_max*beta_cubic)/C).  [B]

The Linux code does something different: it computes K once per epoch
immediately after the fast recovery or ECN cwnd reduction phase
completes, using:

      K = cubic_root((W_max - cwnd)/C).  [C]

See the computation of bic_K here:
  https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c#L254

For the non-fast-convergence case, the result of [B] and [C] are
equivalent, but for the fast convergence case they are not. So it
looks like in the case of fast convergence the RFC formula produces a
result that is incorrect, and does not match the Linux TCP CUBIC
behavior.

AFAICT,  changing Eq 2 to the following might fix the RFC and make it
match the Linux TCP CUBIC behavior:

      K = cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)

best,
neal


From nobody Fri Nov  6 12:35:20 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0557C3A0CD6 for ; Fri,  6 Nov 2020 12:35:19 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.598
X-Spam-Level: 
X-Spam-Status: No, score=-9.598 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=amazon.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v7hNcPk7GSth for ; Fri,  6 Nov 2020 12:35:17 -0800 (PST)
Received: from smtp-fw-9102.amazon.com (smtp-fw-9102.amazon.com [207.171.184.29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6C9503A0CD4 for ; Fri,  6 Nov 2020 12:35:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1604694918; x=1636230918; h=from:to:cc:subject:date:message-id:content-id: content-transfer-encoding:mime-version; bh=BJXfkvyX/nLPc74TsVwz5PL+h7thrt5WV2yodSClOPU=; b=VE5u696ccInQRvJhdDceOKML0J0FHcIK2kXQeUCAYeaB/qVKU23/qSOZ zx53jF8rbon5bpm/3bHQ1JU+JRieHL8XAYK64axUoUNwODrkwE/eIwFcZ MnoI2JiEBxw7SPtLnrx8TEGrmcpvr/Nl9g47hKNOBt/ktnePGzLQbIZLN I=;
X-IronPort-AV: E=Sophos;i="5.77,457,1596499200"; d="scan'208";a="92746338"
Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 06 Nov 2020 20:34:10 +0000
Received: from EX13MTAUWC001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com (Postfix) with ESMTPS id 7BE6EA3680; Fri,  6 Nov 2020 20:34:08 +0000 (UTC)
Received: from EX13D03UWC001.ant.amazon.com (10.43.162.136) by EX13MTAUWC001.ant.amazon.com (10.43.162.135) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 6 Nov 2020 20:34:07 +0000
Received: from EX13D03UWC001.ant.amazon.com (10.43.162.136) by EX13D03UWC001.ant.amazon.com (10.43.162.136) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 6 Nov 2020 20:34:07 +0000
Received: from EX13D03UWC001.ant.amazon.com ([10.43.162.136]) by EX13D03UWC001.ant.amazon.com ([10.43.162.136]) with mapi id 15.00.1497.006; Fri, 6 Nov 2020 20:34:07 +0000
From: "Rosenblum, Wesley" 
To: Neal Cardwell , Lars Eggert 
CC: Vidhi Goel , "tcpm@ietf.org" , Martin Thomson , iccrg IRTF list 
Thread-Topic: [iccrg] [tcpm] Cwnd growth after fast convergence in CUBIC
Thread-Index: AQHWtHwsGhU3OQMl/0ae2ZWgfkIe8w==
Date: Fri, 6 Nov 2020 20:34:07 +0000
Message-ID: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-MacOutlook/16.42.20101102
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [10.43.161.43]
Content-Type: text/plain; charset="utf-8"
Content-ID: <32515586DBC6CD4B9FF13F0D094D3088@amazon.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] [iccrg]  Cwnd growth after fast convergence in CUBIC
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 06 Nov 2020 20:35:19 -0000

QWdyZWVkLCBhbmQgdGhpcyBpcyBiYXNpY2FsbHkgd2hhdCBWaWRoaSBoYXMgc3VnZ2VzdGVkIGFz
IHdlbGwuIEluIG15IG9wdGlvbiBpdCBpcyBzbGlnaHRseQ0KbW9yZSBjbGVhciB0byBpbnRyb2R1
Y2UgY3duZCBpbnRvIHRoZSBjYWxjdWxhdGlvbiBvZiBLIGFzIFZpZGhpIGhhcyBkb25lLiBJIHN1
cHBvc2UgaXQgY29tZXMgDQpkb3duIHRvIHdoZXJlIGluIGEgcGFydGljdWxhciBpbXBsZW1lbnRh
dGlvbiBvbmUgY2FsY3VsYXRlcyBLLg0KDQpPbmUgbm90ZTogSWYgV19sYXN0X21heCB3ZXJlIHRv
IGJlIGluY29ycG9yYXRlZCBpbnRvIEssIHRoZSBzcGVjaWFsIGNhc2UgbWVudGlvbmVkIGluIHNl
Y3Rpb24gDQo0LjggYWJvdXQgZXhpdGluZyBoeWJyaWQgc2xvdyBzdGFydCB3b3VsZCBhbHNvIG5l
ZWQgdG8gYmUgdXBkYXRlZCB0byBtZW50aW9uIHNldHRpbmcgDQpXX2xhc3RfbWF4ID0gV19tYXgg
PSBjd25kLCBhcyBXX2xhc3RfbWF4ICB3b3VsZCBub3QgaGF2ZSBiZWVuIHNldCBpbiBzdWNoIGEg
Y2FzZS4gDQoNClRoYW5rcywNCldlc2xleQ0KDQrvu79PbiAxMS82LzIwLCA2OjEwIEFNLCAiTmVh
bCBDYXJkd2VsbCIgPG5jYXJkd2VsbD00MGdvb2dsZS5jb21AZG1hcmMuaWV0Zi5vcmc+IHdyb3Rl
Og0KPiAgQUZBSUNULCAgY2hhbmdpbmcgRXEgMiB0byB0aGUgZm9sbG93aW5nIG1pZ2h0IGZpeCB0
aGUgUkZDIGFuZCBtYWtlIGl0DQo+ICBtYXRjaCB0aGUgTGludXggVENQIENVQklDIGJlaGF2aW9y
Og0KPg0KPiAgICAgICAgIEsgPSBjdWJpY19yb290KFdfbWF4IC0gV19sYXN0X21heCpiZXRhX2N1
YmljKS9DKSAoRXEuIDIpDQoNCiAgICBiZXN0LA0KICAgIG5lYWwNCg0K


From nobody Fri Nov  6 13:51:13 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A9E343A0D79 for ; Fri,  6 Nov 2020 13:51:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.346
X-Spam-Level: 
X-Spam-Status: No, score=-2.346 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.247, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lipPw-zNqkha for ; Fri,  6 Nov 2020 13:51:08 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8AF753A0D86 for ; Fri,  6 Nov 2020 13:51:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=WUnFRtVymYWs3FMd58/D5q9QudesrGs7uRh1moIj3YM=; b=tFxCKc8ppNXD+Xm5QAAY/ZI8W 7d2wkDL7KlVng7UzRYK4GlwzmLAb94oB0/mCV2mixgERZhCrqjUcO0ja5wp69OfT38ByyMDE3dCYG C8aL8uivZ6JvqdqpTy4WEo5XPL+a3W+5YiOLmx0h9Pc7eTI+nts8wZVRmtYZS4WJkeqBrT7UFXmJK 5sN0CfvNgNkV5gOSSoOI2K7F60BovQ0iGYIIkqJJF6Hj/luJMk8zLmmZAPVyASntrXBaeqjCXbCVa cziKQZRZBYXno6NRgOuujtChjHXg9f+9iuoICo/gTcgcFK5ZtIDkYC5jqvtbbcrhgriy0Y6B1sH85 nP/l15qog==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:40024 helo=[192.168.1.6]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kb9dR-000miN-HE; Fri, 06 Nov 2020 21:51:06 +0000
To: Gorry Fairhurst 
Cc: tcpm IETF list 
References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net>     <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net> 
From: Bob Briscoe 
Message-ID: <3c7126bd-ebdc-00b1-8b46-79125eb525a1@bobbriscoe.net>
Date: Fri, 6 Nov 2020 21:51:04 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------A930686E6CB10DE8CCAEBDEE"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 06 Nov 2020 21:51:12 -0000

This is a multi-part message in MIME format.
--------------A930686E6CB10DE8CCAEBDEE
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Gorry,

Thanks as always for checking all this... inline tagged [BB]

On 06/11/2020 14:01, Gorry Fairhurst wrote:
> I see this rev introduced a section on ACK Filtering, which I have 
> specific comments on. We need to consider the cost of sending ACKs - 
> especially anything that results in them being large - however, this 
> needs to be done by choosing a default at the receiver - not as an 
> adaptation, the adaption if necessary needs to be introduce more ACKs 
> when this is beneficial, as in some cases for DAAS, not in trying to 
> figure something from delayed or "missing" ACKs.
>
> 1) So let me say first that I think the suggestion for updating the 
> receiver to thin, and every 2-6 packets seems reasonable.
>
>     “MUST immediately send an ACK once 'n' CE marks have arrived since
>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>       than 6.”
>
> I do agree that endpoints can do much more to reduce the return path 
> capacity that they consume - and that the endpoints have the 
> opportunity to encode the feedback information more efficiently than 
> can be done by a device (or queue) in the network. From our 
> experiments, n=10 would also often be likely OK.

NB: n is the no. of *CE marks*, which is only the ACK thinning ratio in 
the worst case if there's 100% marking.
IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
whenever marking probability < 60%.

Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
duration of a 100% marking sequence, whenever some data packets are not 
marked, the receiver aACK ratio can rise. This *could* allow any 
build-up of ACKs queued on the reverse path to relax, which could 
improve the chances of not overflowing into ACK loss.

It's hard to predict when 100% marking in the forward direction will 
coincide with reverse path congestion, and I'm not saying it won't.
Just pointing out that n does not equal the ACK thinning ratio, it's the 
floor of it.

n=6 is a consequence of only having a 3-bit field to count CE marks. We 
could push the envelope and make the max n=7. But not n=8, otherwise 
wrap after 8 marks would be indistinguishable from zero marks.


>
> 2. Althouigh in some cases a receiver can predict a good "n", I do not 
> believe a receiver can typically find out how to choose “n” by 
> observing the ACKs on a connection.
> 2. I do not believe a receiver can find out how top to choose “n” by 
> observing the ACKs received on a connection.

[BB] I think you intended to delete one of these sentences. Whatever, 
both of them seem to contain a typo - otherwise they are circular? 
Surely the receiver is sending the ACKs, so either sentence as it stands 
ends with "...I do not believe a receiver can typically decide how often 
to send an ACK by observing how often it is sending ACKs itself." ???

>
> Sure, if the path is modelled as a simple bit-congestive path that has 
> a queue, then you could monitoring timing/loss of packets and 
> determine a rate that tries not to fill a queue. I would not know how 
> common such a bottleneck link is.
>
> I do know that many bottleneck links are more complex and needs to 
> consider the scheduling transmission of small ACK packets on the 
> return path performance. In many radio technologies, the overhead from 
> transmission of ACKs is proportionally much higher than for data 
> segments -  These layer 2s are constrained in terms of packets (e.g. 
> where lower layer framing and transmission scheduling is an important 
> consideration to the efficiency/cost of using the service). This has 
> often been “fixed” using ACK Thinning or PEPs. Protocols that send 
> fewer ACKs/data packet are not impacted.

[BB] I agree that the receiver alone cannot predict a good 'n'.
AccECN is a not the complete solution, but we've grasped the opportunity 
to take a step in the right direction (within the chartered scope of 
AccECN). AccECN at least allows the data sender to measure the extent of 
congestion marking on the arriving ACK stream (if there's ECN support at 
the bottleneck). Whereas before AccECN, a random ECN mark on one pure 
ACK in a round couldn't be distinguished from 100% of ACKs marked.

>
> Scheduling (at the IP layer,can also often result in variations in 
> timing - something that is hard to predict and I would suggests make 
> automated selection of “n” very hard, and the need to thin is often 
> not directly visible to a user of the path. Traffic multiplexed with 
> protocols that use different ACK policies (perhaps QUIC?) also make 
> this sort of detection less reliable.

[BB] Being hard shouldn't prevent us trying.
The remaining piece would be to advance a TCP ACK congestion control 
protocol on the experimental track (RFC5690 
 is informational).


>
> 3) A queue might be competing for scheduling opportunities against 
> other IP flows (e.g., WFQ), or competing for transmission 
> opportunities with a shared resource (common in radio links).
>
> ACKs can and do take transmission opportunities away from the resource 
> pools that would be used by other services that share the same 
> resource pool. This can be reduced by some form of per-flow scheduling 
> with deeper per-flow queues - but that in itself can induce unwanted 
> coupling between flows - limiting the peak rate of one flow, to 
> achieve a higher ACK ratio for another flow.
>
> Consider a QUIC flow sharing a return path with a TCP flow, the higher 
> rate of larger QUIC ACKs reduces the transmission opportunities for 
> TCP ACKs, which in turn suppresses the TCP flow, limiting the overall 
> system performance. If the ACKs use the same transmission resource, 
> they could take directly away from transmission opportunities in the 
> forward direction, but even if they do not - the rate of ACKs retuned 
> impacts the sending rate end to end.

[BB] This says two things to me:
* if TCP is going to introduce control over the ACK ratio, QUIC and 
other transports ought to as well {Note 1}.
* and all ACK CC algorithms ought to be designed to ensure some degree 
of equitable capacity sharing {Note 2}, including between the ACK stream 
of flow A and the data stream of flow B transferring in the opposite 
direction.

{Note 1}: I don't think this needs any formal "all or nothing" rule 
where the transport ADs co-ordinate multiple WGs. Enough of the people 
involved are the same for the right thing to happen (I think). But some 
AD oversight wouldn't do any harm.
{Note 2}: There are numerous challenges to face, but I think it's 
paramount to first establish which node is considered responsible for 
solving the problem (the bottleneck or one of the transport endpoints). 
This doesn't imply that the IETF will ever ensure that other nodes don't 
'interfere'. For instance, if the sender is deemed responsible for 
determining the level of ACK filtering, it will still have to make 
allowance for bottlenecks attempting to ACK thin as well.



>
> 4) The new section adds some cross-layer requirements on sub-IP 
> layers, which seem to imply more complexity. I’m not comfortable with 
> such a proposal.
>
> It seems that the text suggests not implement thinning and that these 
> systems implement logic to detect the presence of the option by 
> observing SYNs and then configure the link to do something special. 

This was the proposal that I was most interested to hear your views on. 
We can remove it if the WG doesn't want it, but first I want to make 
sure the proposal is fully understood:
* The aim is for the RFC to tell bottleneck implementers (via this text) 
that *only in cases where AccECN has been negotiated* the preferred way 
to improve flow performance is for the link to signal ECN to the 
transport, not to drop the ACKs itself.
* If the link doesn't support ECN, it still does the ACK filtering itself
* Links aren't going to delegate their control unless transports 
actually act on the ECN signals that links emit - and improve 
performance better than links have been. So, it's important that 
transports take on these responsibilities - the data sender takes 
responsibility for determining the ACK rate, asking its correspondent 
receiver does actually thin its own ACKs.

Here's who signals what, and who does what

                link-.
                      \
                       '-[ECN CE]-.
                                   \
                                    '-> receiver
                                     .-
                                    /
             .-[(Acc)ECN feedback]-'
            /
  sender <-'
         -.
           \
            '---[ACK CC ratio]---.
                                  \
                                   '-> receiver(treats as polite 
request, not mandatory)


> I don’t think think this is the correct approach - the result will be 
> throttling of the connection speed. Is it necessary that the sub-IP 
> layer does not implement thinning for connections that use AccECN 
> feedback?

Why do you say the connection speed would be throttled?
Did the text not explain the proposal well?
Or did you understand the proposal but disagree with it?


>
> When the text says: “SHOULD preserve the timing of each ACK”
> - does that imply that nodes need to discard rather than delay packets 
> when link scheduling or transmission scheduling is used. This seems a 
> very odd requirement, it might be better too simply strip-out the 
> AccECN option or discard all packets that set this.

Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
filter ACKs" (if it supports ECN marking).

>
> I can’t see how queuing the ACKs, or deleting ACKs rather than queuing 
> them would improve the performance of AccECN TCP connections.

May be the piece where the link's preferred role is to signal to the 
transport to slow down the ACK rate using ECN wasn't clear.


Bob

>
> Gorry
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------A930686E6CB10DE8CCAEBDEE
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Gorry,

    

    Thanks as always for checking all this... inline tagged [BB]

    

    
On 06/11/2020 14:01, Gorry Fairhurst
      wrote:

    

    
I
      see this rev introduced a section on ACK Filtering, which I have
      specific comments on. We need to consider the cost of sending ACKs
      - especially anything that results in them being large - however,
      this needs to be done by choosing a default at the receiver - not
      as an adaptation, the adaption if necessary needs to be introduce
      more ACKs when this is beneficial, as in some cases for DAAS, not
      in trying to figure something from delayed or "missing" ACKs.
      

      

      1) So let me say first that I think the suggestion for updating
      the receiver to thin, and every 2-6 packets seems reasonable.
      

      

          “MUST immediately send an ACK once 'n' CE marks have arrived
      since
      

            the previous ACK, where 'n' SHOULD be 2 and MUST be no
      greater
      

            than 6.”
      

      

      I do agree that endpoints can do much more to reduce the return
      path capacity that they consume - and that the endpoints have the
      opportunity to encode the feedback information more efficiently
      than can be done by a device (or queue) in the network. From our
      experiments, n=10 would also often be likely OK.
      

    

    

    NB: n is the no. of *CE marks*, which is only the ACK thinning ratio
    in the worst case if there's 100% marking. 

    IOW, with n=6 the receiver could ACK less than 1 in 10 data packets
    whenever marking probability < 60%.

    

    Altho the receiver's ACK ratio will have to fall to 6 to 1 for the
    duration of a 100% marking sequence, whenever some data packets are
    not marked, the receiver aACK ratio can rise. This *could* allow any
    build-up of ACKs queued on the reverse path to relax, which could
    improve the chances of not overflowing into ACK loss.

    

    It's hard to predict when 100% marking in the forward direction will
    coincide with reverse path congestion, and I'm not saying it won't.

    Just pointing out that n does not equal the ACK thinning ratio, it's
    the floor of it.

    

    n=6 is a consequence of only having a 3-bit field to count CE marks.
    We could push the envelope and make the max n=7. But not n=8,
    otherwise wrap after 8 marks would be indistinguishable from zero
    marks.

    

    

    

      

      2. Althouigh in some cases a receiver can predict a good "n", I do
      not believe a receiver can typically find out how to choose “n” by
      observing the ACKs on a connection.
      

    

    

      2. I do not believe a receiver can find out how top to choose “n”
      by observing the ACKs received on a connection.
      

    

    

    [BB] I think you intended to delete one of these sentences.
    Whatever, both of them seem to contain a typo - otherwise they are
    circular? Surely the receiver is sending the ACKs, so either
    sentence as it stands ends with "...I do not believe a receiver can
    typically decide how often to send an ACK by observing how often it
    is sending ACKs itself." ???

    

    

      

      Sure, if the path is modelled as a simple bit-congestive path that
      has a queue, then you could monitoring timing/loss of packets and
      determine a rate that tries not to fill a queue. I would not know
      how common such a bottleneck link is.
      

      

      I do know that many bottleneck links are more complex and needs to
      consider the scheduling transmission of small ACK packets on the
      return path performance. In many radio technologies, the overhead
      from transmission of ACKs is proportionally much higher than for
      data segments -  These layer 2s are constrained in terms of
      packets (e.g. where lower layer framing and transmission
      scheduling is an important consideration to the efficiency/cost of
      using the service). This has often been “fixed” using ACK Thinning
      or PEPs. Protocols that send fewer ACKs/data packet are not
      impacted.
      

    

    

    

    

    [BB] I agree that the receiver alone cannot predict a good 'n'. 

    AccECN is a not the complete solution, but we've grasped the
    opportunity to take a step in the right direction (within the
    chartered scope of AccECN). AccECN at least allows the data sender
    to measure the extent of congestion marking on the arriving ACK
    stream (if there's ECN support at the bottleneck). Whereas before
    AccECN, a random ECN mark on one pure ACK in a round couldn't be
    distinguished from 100% of ACKs marked.

    

    

      

      Scheduling (at the IP layer,can also often result in variations in
      timing - something that is hard to predict and I would suggests
      make automated selection of “n” very hard, and the need to thin is
      often not directly visible to a user of the path. Traffic
      multiplexed with protocols that use different ACK policies
      (perhaps QUIC?) also make this sort of detection less reliable.
      

    

    

    [BB] Being hard shouldn't prevent us trying.

    The remaining piece would be to advance a TCP ACK congestion control
    protocol on the experimental track (RFC5690 is
    informational).

    

    

    

      

      3) A queue might be competing for scheduling opportunities against
      other IP flows (e.g., WFQ), or competing for transmission
      opportunities with a shared resource (common in radio links).
      

      

      ACKs can and do take transmission opportunities away from the
      resource pools that would be used by other services that share the
      same resource pool. This can be reduced by some form of per-flow
      scheduling with deeper per-flow queues - but that in itself can
      induce unwanted coupling between flows - limiting the peak rate of
      one flow, to achieve a higher ACK ratio for another flow.
      

      

      Consider a QUIC flow sharing a return path with a TCP flow, the
      higher rate of larger QUIC ACKs reduces the transmission
      opportunities for TCP ACKs, which in turn suppresses the TCP flow,
      limiting the overall system performance. If the ACKs use the same
      transmission resource, they could take directly away from
      transmission opportunities in the forward direction, but even if
      they do not - the rate of ACKs retuned impacts the sending rate
      end to end.
      

    

    

    [BB] This says two things to me: 

    * if TCP is going to introduce control over the ACK ratio, QUIC and
    other transports ought to as well {Note 1}. 

    * and all ACK CC algorithms ought to be designed to ensure some
    degree of equitable capacity sharing {Note 2}, including between the
    ACK stream of flow A and the data stream of flow B transferring in
    the opposite direction.

    

    {Note 1}: I don't think this needs any formal "all or nothing" rule
    where the transport ADs co-ordinate multiple WGs. Enough of the
    people involved are the same for the right thing to happen (I
    think). But some AD oversight wouldn't do any harm.

    {Note 2}: There are numerous challenges to face, but I think it's
    paramount to first establish which node is considered responsible
    for solving the problem (the bottleneck or one of the transport
    endpoints). This doesn't imply that the IETF will ever ensure that
    other nodes don't 'interfere'. For instance, if the sender is deemed
    responsible for determining the level of ACK filtering, it will
    still have to make allowance for bottlenecks attempting to ACK thin
    as well.

    

    

    

    

      

      4) The new section adds some cross-layer requirements on sub-IP
      layers, which seem to imply more complexity. I’m not comfortable
      with such a proposal.
      

      

      It seems that the text suggests not implement thinning and that
      these systems implement logic to detect the presence of the option
      by observing SYNs and then configure the link to do something
      special. 

    

    This was the proposal that I was most interested to hear your views
    on. We can remove it if the WG doesn't want it, but first I want to
    make sure the proposal is fully understood:

    * The aim is for the RFC to tell bottleneck implementers (via this
    text) that *only in cases where AccECN has been negotiated* the
    preferred way to improve flow performance is for the link to signal
    ECN to the transport, not to drop the ACKs itself. 

    * If the link doesn't support ECN, it still does the ACK filtering
    itself

    * Links aren't going to delegate their control unless transports
    actually act on the ECN signals that links emit - and improve
    performance better than links have been. So, it's important that
    transports take on these responsibilities - the data sender takes
    responsibility for determining the ACK rate, asking its
    correspondent receiver does actually thin its own ACKs.

    

    Here's who signals what, and who does what

    

                   link-.

                           \

                            '-[ECN CE]-.

                                        \

                                         '-> receiver

                                          .-

                                         /

                  .-[(Acc)ECN feedback]-'

                 /

       sender <-'

              -.

                \

                 '---[ACK CC ratio]---.

                                       \

                                        '-> receiver (treats
      as polite request, not mandatory)

    

    

    
I
      don’t think think this is the correct approach - the result will
      be throttling of the connection speed. Is it necessary that the
      sub-IP layer does not implement thinning for connections that use
      AccECN feedback?
      

    

    

    Why do you say the connection speed would be throttled? 

    Did the text not explain the proposal well?

    Or did you understand the proposal but disagree with it?

    

    

    

      

      When the text says: “SHOULD preserve the timing of each ACK”
      

      - does that imply that nodes need to discard rather than delay
      packets when link scheduling or transmission scheduling is used.
      This seems a very odd requirement, it might be better too simply
      strip-out the AccECN option or discard all packets that set this.
      

    

    

    Sorry, that may be clumsy wording. It just means "the link SHOULD
    NOT filter ACKs" (if it supports ECN marking).

    

    

      

      I can’t see how queuing the ACKs, or deleting ACKs rather than
      queuing them would improve the performance of AccECN TCP
      connections.
      

    

    

    May be the piece where the link's preferred role is to signal to the
    transport to slow down the ACK rate using ECN wasn't clear.

    

    

    Bob

    

    

      

      Gorry
      

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------A930686E6CB10DE8CCAEBDEE--


From nobody Fri Nov  6 19:27:47 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E399D3A00E5; Fri,  6 Nov 2020 19:27:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level: 
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1g3Hj8cE2qa1; Fri,  6 Nov 2020 19:27:40 -0800 (PST)
Received: from nwk-aaemail-lapp02.apple.com (nwk-aaemail-lapp02.apple.com [17.151.62.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A55573A011B; Fri,  6 Nov 2020 19:27:40 -0800 (PST)
Received: from pps.filterd (nwk-aaemail-lapp02.apple.com [127.0.0.1]) by nwk-aaemail-lapp02.apple.com (8.16.0.43/8.16.0.42) with SMTP id 0A73AqYK014515; Fri, 6 Nov 2020 19:27:33 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=3UX4fVZU+U6fSWUeS09u4NCMkEfrfxa49nLm8qohjRE=; b=KZ77J5FaBiqtPCcKEx/dJIELo7KBfsVKqkxWYazG8ttKWilB2mri3aIf9th7oHor8xJB BUgFF3h3Q/M1X0JueQcwZT5eZ+UsnpjxCYfWQ4tLNhU9edYgjag2Su1EeTHLoP9H7sTB 2BtM0B6nHsmeUheMGLEnfbfWh/dbQKSFbVjwJhvluDS14nHzLrdS3E3bbGD3eE8ag+B0 9auJuHdc4l1MMmKJCSL7UcS/FN3Xjwe91Ue3bp6emHyg99LpObqQja+IALl1pcwUL/WT qVSI4/AkEKnLH2SvxZSJs1rcmRSv/5qPmSq4H2mlFSpKqzEvz2/EZShKqn0R9RoJts00 rA== 
Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by nwk-aaemail-lapp02.apple.com with ESMTP id 34h4gjxn90-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 06 Nov 2020 19:27:33 -0800
Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPS id <0QJE00MTMPLX6210@rn-mailsvcp-mta-lapp01.rno.apple.com>;  Fri, 06 Nov 2020 19:27:33 -0800 (PST)
Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) id <0QJE00700PIC8L00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Fri, 06 Nov 2020 19:27:33 -0800 (PST)
X-Va-A: 
X-Va-T-CD: d372616deca2a8b45477d746e04bd5f5
X-Va-E-CD: cc88deaa2fe6674c646bde81085e4789
X-Va-R-CD: cf165b39e9253d775e50ccbeb702d059
X-Va-CD: 0
X-Va-ID: f01332ba-9eda-4fc1-a935-e400d50525ca
X-V-A: 
X-V-T-CD: d372616deca2a8b45477d746e04bd5f5
X-V-E-CD: cc88deaa2fe6674c646bde81085e4789
X-V-R-CD: cf165b39e9253d775e50ccbeb702d059
X-V-CD: 0
X-V-ID: cab4c974-66a8-40c9-b070-8db25e25f70f
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_06:2020-11-05, 2020-11-06 signatures=0
Received: from [17.149.232.178] (unknown [17.149.232.178]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPSA id <0QJE00HOWPLWB500@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Fri, 06 Nov 2020 19:27:32 -0800 (PST)
From: Vidhi Goel 
Message-id: 
Content-type: multipart/alternative; boundary="Apple-Mail=_72099EA6-0356-49F9-A78D-EEB817F43989"
MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Date: Fri, 06 Nov 2020 19:27:32 -0800
In-reply-to: 
Cc: Neal Cardwell , "tcpm@ietf.org" , Martin Thomson , iccrg IRTF list , Christoph Paasch 
To: "Rosenblum, Wesley" , Lars Eggert , "Scheffenegger, Richard" 
References: 
X-Mailer: Apple Mail (2.3608.80.23.2.2)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_06:2020-11-05, 2020-11-06 signatures=0
Archived-At: 
Subject: Re: [tcpm] [iccrg]  Cwnd growth after fast convergence in CUBIC
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 07 Nov 2020 03:27:42 -0000

--Apple-Mail=_72099EA6-0356-49F9-A78D-EEB817F43989
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

++

> Agreed, and this is basically what Vidhi has suggested as well. In my =
option it is slightly
> more clear to introduce cwnd into the calculation of K as Vidhi has =
done. I suppose it comes=20
> down to where in a particular implementation one calculates K.
>=20
> One note: If W_last_max were to be incorporated into K, the special =
case mentioned in section=20
> 4.8 about exiting hybrid slow start would also need to be updated to =
mention setting=20
> W_last_max =3D W_max =3D cwnd, as W_last_max  would not have been set =
in such a case.=20

The way we have implemented Cubic, using W_last_max won=E2=80=99t work =
for us either. Previous suggestion of using `cwnd` works for us,
K =3D cbrt(W_max - cwnd / C)


> That said, the intent was to document what the Linux code was doing at =
the time. Is there a difference between the RFC and what Linux has been =
doing or is doing now? If yes, it might be time to revise the RFC.
>=20
> (I think Martin Thomson also found some nits recently.)


Revising the RFC would be good. While we are on the topic, there is one =
more thing that could use a discussion.

Christoph (cc=E2=80=99ed) and I had discussed this earlier regarding =
alpha_aimd used for TCP friendly region.=20

Currently, alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic) // for the =
entire TCP friendly region

But we think that once the cwnd in TF region reaches W_max, we should =
set the alpha_aimd to 1 to have similar behavior as standard TCP =
congestion algorithm (eg. NewReno)

if (W_est < W_max)
	alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic)
else
	alpha_aimd =3D 1


Let me know your thoughts on this. I'd be happy to work on this.


Thanks,
Vidhi

> On Nov 6, 2020, at 12:34 PM, Rosenblum, Wesley  =
wrote:
>=20
> Agreed, and this is basically what Vidhi has suggested as well. In my =
option it is slightly
> more clear to introduce cwnd into the calculation of K as Vidhi has =
done. I suppose it comes=20
> down to where in a particular implementation one calculates K.
>=20
> One note: If W_last_max were to be incorporated into K, the special =
case mentioned in section=20
> 4.8 about exiting hybrid slow start would also need to be updated to =
mention setting=20
> W_last_max =3D W_max =3D cwnd, as W_last_max  would not have been set =
in such a case.=20
>=20
> Thanks,
> Wesley
>=20
> =EF=BB=BFOn 11/6/20, 6:10 AM, "Neal Cardwell" =
 wrote:
>> AFAICT,  changing Eq 2 to the following might fix the RFC and make it
>> match the Linux TCP CUBIC behavior:
>>=20
>>        K =3D cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)
>=20
>    best,
>    neal
>=20


--Apple-Mail=_72099EA6-0356-49F9-A78D-EEB817F43989
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8


++

Agreed, and this is basically what Vidhi has suggested as =
well. In my option it is slightly
more clear to introduce =
cwnd into the calculation of K as Vidhi has done. I suppose it =
comes 
down to where in a particular implementation =
one calculates K.

One note: If W_last_max =
were to be incorporated into K, the special case mentioned in =
section 
4.8 about exiting hybrid slow start would =
also need to be updated to mention setting 
W_last_max =
=3D W_max =3D cwnd, as W_last_max  would not have been set in such =
a case. 

The way =
we have implemented Cubic, using W_last_max won=E2=80=99t work for us =
either. Previous suggestion of using `cwnd` works for us,
K =
=3D cbrt(W_max =
- cwnd / C)


That said, the intent was =
to document what the Linux code was doing at the time. Is there a =
difference between the RFC and what Linux has been doing or is doing =
now? If yes, it might be time to revise the RFC.

(I think Martin Thomson also found some nits =
recently.)

Revising the RFC would be good. While we are on the =
topic, there is one more thing that could use a discussion.

Christoph (cc=E2=80=99ed) and I had discussed =
this earlier regarding alpha_aimd used for TCP friendly =
region. 

Currently, alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic) // for the entire TCP =
friendly region

But we think that once the cwnd in TF =
region reaches W_max, we should set the alpha_aimd to 1 to have similar =
behavior as standard TCP congestion algorithm (eg. =
NewReno)

if (W_est < =
W_max)
	=
alpha_aimd =
=3D 3*(1-beta_cubic)/(1+beta_cubic)
else
	alpha_aimd =3D 1


Let me know your =
thoughts on this. I'd be happy to work on this.


Thanks,
Vidhi

On Nov 6, 2020, at 12:34 PM, =
Rosenblum, Wesley <wesleyr@amazon.com> wrote:

Agreed, and this is basically what Vidhi has suggested as =
well. In my option it is slightly
more clear to introduce =
cwnd into the calculation of K as Vidhi has done. I suppose it comes 
down to where in a particular implementation one calculates =
K.

One note: If W_last_max were to be =
incorporated into K, the special case mentioned in section 
4.8 about exiting hybrid slow start would also need to be =
updated to mention setting 
W_last_max =3D W_max =3D cwnd, =
as W_last_max  would not have been set in such a case. 

Thanks,
Wesley

=EF=BB=BFOn 11/6/20, 6:10 AM, "Neal Cardwell" <ncardwell=3D40google.com@dmarc.ietf.org> wrote:
 AFAICT,  changing =
Eq 2 to the following might fix the RFC and make it
 match =
the Linux TCP CUBIC behavior:

 =
       K =3D cubic_root(W_max - =
W_last_max*beta_cubic)/C) (Eq. 2)

    best,
 =
   neal


=

--Apple-Mail=_72099EA6-0356-49F9-A78D-EEB817F43989--


From nobody Fri Nov  6 20:09:10 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5A2B23A044A; Fri,  6 Nov 2020 20:08:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.101
X-Spam-Level: 
X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s3Ef1C1TOAGB; Fri,  6 Nov 2020 20:08:58 -0800 (PST)
Received: from nwk-aaemail-lapp01.apple.com (nwk-aaemail-lapp01.apple.com [17.151.62.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0D7183A0408; Fri,  6 Nov 2020 20:08:58 -0800 (PST)
Received: from pps.filterd (nwk-aaemail-lapp01.apple.com [127.0.0.1]) by nwk-aaemail-lapp01.apple.com (8.16.0.43/8.16.0.42) with SMTP id 0A73u7Fd033018; Fri, 6 Nov 2020 20:08:51 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=W+yzCzlYMbKisExAGk2t+l+MhoOxE6m/2yCWs0XTygo=; b=m8XSfAbm/A/tErrec7hUiLMod+X1EpTAIcACjIaHGr6dtvLZ37RArYMSPDHfPgXuOWJ7 g/JlsxziW9j5uSctqEkZ1W8oHifbXENeHLFE/rbQIA5Z+hcrC6nHa79bDqL+b7TicT2X X0d3rJTKP6WJzudGLx3/VN6+X/8UIiM59uGg18uvDeXuHAka+2lPE0ip2ditDv1nPKr1 sUq/uge96t6zr20OasWiOD0FmvSweyYsf4yi7Lkf4B1D0WecjF5EZ3pS3rZjXmxWe+Ed 8Ao3obO+FYNl5w8HMbgm4WS9Mu9MNdI8HMLrQ+9CVNuM+eJbfP1TQJj3PeElq3H0GFjP /Q== 
Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by nwk-aaemail-lapp01.apple.com with ESMTP id 34h6v29j7h-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Fri, 06 Nov 2020 20:08:51 -0800
Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPS id <0QJE011NWRIR46E0@rn-mailsvcp-mta-lapp01.rno.apple.com>;  Fri, 06 Nov 2020 20:08:51 -0800 (PST)
Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) id <0QJE00A00R44KP00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Fri, 06 Nov 2020 20:08:51 -0800 (PST)
X-Va-A: 
X-Va-T-CD: c97696a9e84e996401d2d042a4999cf4
X-Va-E-CD: 46ed4047f398203abf674ba2b4f02161
X-Va-R-CD: 3c13deb3c3a974241d79e19958cc67e6
X-Va-CD: 0
X-Va-ID: 06920350-1233-446a-83f6-d4f81f05b665
X-V-A: 
X-V-T-CD: c97696a9e84e996401d2d042a4999cf4
X-V-E-CD: 46ed4047f398203abf674ba2b4f02161
X-V-R-CD: 3c13deb3c3a974241d79e19958cc67e6
X-V-CD: 0
X-V-ID: 1c4f0ea2-83dd-4566-b321-0d5c45c25976
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_06:2020-11-05, 2020-11-06 signatures=0
Received: from [17.149.232.178] (unknown [17.149.232.178]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPSA id <0QJE00N52RIPNC00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Fri, 06 Nov 2020 20:08:50 -0800 (PST)
From: Vidhi Goel 
Message-id: <57A58E6B-0385-47C1-A85F-49017703411D@apple.com>
Content-type: multipart/alternative; boundary="Apple-Mail=_A33D7947-EF98-49DE-8A5B-AFE67E272E4B"
MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Date: Fri, 06 Nov 2020 20:08:49 -0800
In-reply-to: 
Cc: Neal Cardwell , "tcpm@ietf.org" , iccrg IRTF list , Martin Thomson , Christoph Paasch 
To: Lars Eggert , "Scheffenegger, Richard" , "Rosenblum, Wesley" 
References:  
X-Mailer: Apple Mail (2.3608.80.23.2.2)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-06_06:2020-11-05, 2020-11-06 signatures=0
Archived-At: 
Subject: Re: [tcpm] [iccrg]  Cwnd growth after fast convergence in CUBIC
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 07 Nov 2020 04:08:59 -0000

--Apple-Mail=_A33D7947-EF98-49DE-8A5B-AFE67E272E4B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

 +Christoph

> On Nov 6, 2020, at 7:27 PM, Vidhi Goel =
 wrote:
>=20
> ++
>=20
>> Agreed, and this is basically what Vidhi has suggested as well. In my =
option it is slightly
>> more clear to introduce cwnd into the calculation of K as Vidhi has =
done. I suppose it comes=20
>> down to where in a particular implementation one calculates K.
>>=20
>> One note: If W_last_max were to be incorporated into K, the special =
case mentioned in section=20
>> 4.8 about exiting hybrid slow start would also need to be updated to =
mention setting=20
>> W_last_max =3D W_max =3D cwnd, as W_last_max  would not have been set =
in such a case.=20
>=20
> The way we have implemented Cubic, using W_last_max won=E2=80=99t work =
for us either. Previous suggestion of using `cwnd` works for us,
> K =3D cbrt(W_max - cwnd / C)
>=20
>=20
>> That said, the intent was to document what the Linux code was doing =
at the time. Is there a difference between the RFC and what Linux has =
been doing or is doing now? If yes, it might be time to revise the RFC.
>>=20
>> (I think Martin Thomson also found some nits recently.)
>=20
>=20
> Revising the RFC would be good. While we are on the topic, there is =
one more thing that could use a discussion.
>=20
> Christoph (cc=E2=80=99ed) and I had discussed this earlier regarding =
alpha_aimd used for TCP friendly region.=20
>=20
> Currently, alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic) // for the =
entire TCP friendly region
>=20
> But we think that once the cwnd in TF region reaches W_max, we should =
set the alpha_aimd to 1 to have similar behavior as standard TCP =
congestion algorithm (eg. NewReno)
>=20
> if (W_est < W_max)
> 	alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic)
> else
> 	alpha_aimd =3D 1
>=20
>=20
> Let me know your thoughts on this. I'd be happy to work on this.
>=20
>=20
> Thanks,
> Vidhi
>=20
>> On Nov 6, 2020, at 12:34 PM, Rosenblum, Wesley > wrote:
>>=20
>> Agreed, and this is basically what Vidhi has suggested as well. In my =
option it is slightly
>> more clear to introduce cwnd into the calculation of K as Vidhi has =
done. I suppose it comes=20
>> down to where in a particular implementation one calculates K.
>>=20
>> One note: If W_last_max were to be incorporated into K, the special =
case mentioned in section=20
>> 4.8 about exiting hybrid slow start would also need to be updated to =
mention setting=20
>> W_last_max =3D W_max =3D cwnd, as W_last_max  would not have been set =
in such a case.=20
>>=20
>> Thanks,
>> Wesley
>>=20
>> =EF=BB=BFOn 11/6/20, 6:10 AM, "Neal Cardwell" =
> wrote:
>>> AFAICT,  changing Eq 2 to the following might fix the RFC and make =
it
>>> match the Linux TCP CUBIC behavior:
>>>=20
>>>        K =3D cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)
>>=20
>>    best,
>>    neal
>>=20
>=20
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


--Apple-Mail=_A33D7947-EF98-49DE-8A5B-AFE67E272E4B
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

 +Christoph

On Nov 6, 2020, at 7:27 PM, =
Vidhi Goel <vidhi_goel=3D40apple.com@dmarc.ietf.org> =
wrote:

++

Agreed, and this is basically what Vidhi has suggested as =
well. In my option it is slightly
more clear to introduce =
cwnd into the calculation of K as Vidhi has done. I suppose it =
comes 
down to where in a particular implementation =
one calculates K.

One note: If W_last_max =
were to be incorporated into K, the special case mentioned in =
section 
4.8 about exiting hybrid slow start would =
also need to be updated to mention setting 
W_last_max =
=3D W_max =3D cwnd, as W_last_max  would not have been set in such =
a case. 

The way =
we have implemented Cubic, using W_last_max won=E2=80=99t work for us =
either. Previous suggestion of using `cwnd` works for us,
K =
=3D cbrt(W_max =
- cwnd / C)


That said, the intent was =
to document what the Linux code was doing at the time. Is there a =
difference between the RFC and what Linux has been doing or is doing =
now? If yes, it might be time to revise the RFC.

(I think Martin Thomson also found some nits =
recently.)

Revising the RFC would be good. While we are on the =
topic, there is one more thing that could use a discussion.

Christoph =
(cc=E2=80=99ed) and I had discussed this earlier regarding alpha_aimd =
used for TCP friendly region. 

Currently, alpha_aimd =3D 3*(1-beta_cubic)/(1+beta_cubic) // for the entire TCP =
friendly region

But we think that once the cwnd in TF =
region reaches W_max, we should set the alpha_aimd to 1 to have similar =
behavior as standard TCP congestion algorithm (eg. =
NewReno)

if (W_est < W_max)
	alpha_aimd =
=3D 3*(1-beta_cubic)/(1+beta_cubic)
else
	alpha_aimd =3D 1


Let me know your =
thoughts on this. I'd be happy to work on this.


Thanks,
Vidhi

On Nov =
6, 2020, at 12:34 PM, Rosenblum, Wesley <wesleyr@amazon.com> =
wrote:

Agreed, and this is basically what Vidhi has suggested as =
well. In my option it is slightly
more clear to introduce =
cwnd into the calculation of K as Vidhi has done. I suppose it comes 
down to where in a particular implementation one calculates =
K.

One note: If W_last_max were to be =
incorporated into K, the special case mentioned in section 
4.8 about exiting hybrid slow start would also need to be =
updated to mention setting 
W_last_max =3D W_max =3D cwnd, =
as W_last_max  would not have been set in such a case. 

Thanks,
Wesley

=EF=BB=BFOn 11/6/20, 6:10 AM, "Neal Cardwell" <ncardwell=3D40google.com@dmarc.ietf.org> wrote:
 AFAICT,  changing =
Eq 2 to the following might fix the RFC and make it
 match =
the Linux TCP CUBIC behavior:

 =
       K =3D cubic_root(W_max - =
W_last_max*beta_cubic)/C) (Eq. 2)

    best,
 =
   neal


_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm

=

--Apple-Mail=_A33D7947-EF98-49DE-8A5B-AFE67E272E4B--


From nobody Sat Nov  7 08:31:16 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B33943A0A4A for ; Sat,  7 Nov 2020 08:31:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.053
X-Spam-Level: 
X-Spam-Status: No, score=0.053 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VqiXP_Mer1I2 for ; Sat,  7 Nov 2020 08:30:59 -0800 (PST)
Received: from mail-ot1-x330.google.com (mail-ot1-x330.google.com [IPv6:2607:f8b0:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A44933A0A6E for ; Sat,  7 Nov 2020 08:30:59 -0800 (PST)
Received: by mail-ot1-x330.google.com with SMTP id z16so4335167otq.6 for ; Sat, 07 Nov 2020 08:30:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OcTkqBkldkUOg9LicKp+ibMX28fMGlheqrnZE2CvVV8=; b=VIFO9zImnRYQVXSRc9bZlcSf+OwMUYddsfI7gFoVC+U7dZlBZVWeazukUiqAoZHJGx KGr0WoAotsfm/KSAcrptOUP3KeGeqmsG8fosonslj6atBjzy//47ktbMiHR/C2n0vBSY S48kvNHyOEVtEQRPmy3V33TY+tp1lVhFF4x28phsMHOUJHkqxsdW5S2J3Jh7Dp9d/iY9 6k7c3LrRKK7BO5vN0teGNqtujcoWTs3Ym+eJDAITs5eYbVNX88KUNnPxVLZqpN+VO1SZ Uws2Kpu8yofB2oHpqzZHO0u6+uE/BDdOUAXwUudYSnhXiQ0GuJTMMN2DuhMt4x+aWofp JFmA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OcTkqBkldkUOg9LicKp+ibMX28fMGlheqrnZE2CvVV8=; b=O5uTenHovW8yLL7GicnOWZoYNO56b7mNP8YU8hfLTkVqJPr9P4Uo9A/UKxXJsnit8l 90Sgm/YLb4ZT0cMoyVwXuSUo2EuqshRBiwHa4atkaB+vPRu68QXY4sYbFFRYNR4WYCBs S0h58Nhs2d1d6hRgCHqiBcSB8JyQC41cK6pBRcHpQjNfR3fpqo+pWv0QpD30ptAICfPn xJSx9a6cp2Brm3g6Gmc59MbEJAq0Ruh5prusWQuTZEw9fOWSOQOMim8Xpdt8jy+6r7/g wUrFKuZlw8II8eMoj0lfdt1ijZufxbptmgayz0OcochoRC3+T3T7kA+oJQ2bjf/Q+dRn xNVg==
X-Gm-Message-State: AOAM532NoYDDLWIlrKa5pXxnDLYGAzxvhe+WoyZUmkjFb09algJzAisO v6KC7OSHUiZs0H7nxMPzaDFhue7u/jCogFsDxLY=
X-Google-Smtp-Source: ABdhPJzERE2fUPkgTy200FylzQypjrDIcdsPP3YmtD86q3yp4O3F+jmLVmZ/YcxWGEGnomALxPTOz8z+n37W++LHeoc=
X-Received: by 2002:a9d:929:: with SMTP id 38mr5015527otp.170.1604766658341; Sat, 07 Nov 2020 08:30:58 -0800 (PST)
MIME-Version: 1.0
References:    
In-Reply-To: 
From: L X 
Date: Sat, 7 Nov 2020 10:30:46 -0600
Message-ID: 
To: Neal Cardwell 
Cc: Lars Eggert ,  "Rosenblum, Wesley" ,  Vidhi Goel , "tcpm@ietf.org" ,  iccrg IRTF list , Martin Thomson , xu@unl.edu
Content-Type: multipart/alternative; boundary="0000000000009e2f9f05b386da14"
Archived-At: 
Subject: Re: [tcpm] [iccrg]  Cwnd growth after fast convergence in CUBIC
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 07 Nov 2020 16:31:04 -0000

--0000000000009e2f9f05b386da14
Content-Type: text/plain; charset="UTF-8"

Yes, the RFC is different from the Linux Cubic in case of fast convergence.

I would suggest changing Equation (2) of RFC 8312 to
K = cubic_root((W_max* - cwnd)/C) (Eq. 2)

where cwnd is the cwnd at the beginning of a congestion avoidance
stage. That is, W_cubic(0)=the cwnd at the beginning of a congestion
avoidance stage


By the way, sometime the fast recovery stage also changes cwnd based
on the number of packets in flight. As a result the cwnd at the
beginning of a congestion avoidance stage may be different from
W_last_max*beta_cubic.


Therefore, I would prefer to using the following Equation (2) in RFC 8312

K = cubic_root((W_max* - cwnd)/C) (Eq. 2)

instead of

K = cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)


Thank you all

Lisong




On Fri, Nov 6, 2020 at 8:11 AM Neal Cardwell  wrote:

> On Fri, Nov 6, 2020 at 3:28 AM Lars Eggert  wrote:
> > That said, the intent was to document what the Linux code was doing
> > at the time. Is there a difference between the RFC and what Linux
> > has been doing or is doing now? If yes, it might be time to revise
> > the RFC.
>
> Yes, it seems that this is a case where the RFC differs from the Linux
> TCP CUBIC code.
>
> The RFC says:
>       K = cubic_root(W_max*(1-beta_cubic)/C) (Eq. 2)
>
> which is equivalent to:
>       K = cubic_root((W_max - W_max*beta_cubic)/C).  [B]
>
> The Linux code does something different: it computes K once per epoch
> immediately after the fast recovery or ECN cwnd reduction phase
> completes, using:
>
>       K = cubic_root((W_max - cwnd)/C).  [C]
>
> See the computation of bic_K here:
>   https://github.com/torvalds/linux/blob/master/net/ipv4/tcp_cubic.c#L254
>
> For the non-fast-convergence case, the result of [B] and [C] are
> equivalent, but for the fast convergence case they are not. So it
> looks like in the case of fast convergence the RFC formula produces a
> result that is incorrect, and does not match the Linux TCP CUBIC
> behavior.
>
> AFAICT,  changing Eq 2 to the following might fix the RFC and make it
> match the Linux TCP CUBIC behavior:
>
>       K = cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)
>
> best,
> neal
>
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://www.irtf.org/mailman/listinfo/iccrg
>

--0000000000009e2f9f05b386da14
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable



=
I would suggest changing Equation (2) of R=
FC 8312 to=C2=A0
K =3D cubic_root((W_max* - cwnd)/C) (Eq. 2)=

where cwnd is the cwnd at the beginning of a congestion avoidan=
ce stage. That is, W_cubic(0)=3Dthe cwnd at the beginning of a congestion a=
voidance stage

By the way, sometime the fast recovery stage also c=
hanges cwnd based on the number of packets in flight. As a result the cwnd =
at the beginning of a congestion avoidance stage may be different from W_la=
st_max*beta_cubic.

Therefore, I would prefer to using the follo=
wing Equation (2) in RFC 8312
<=
font face=3D"arial, sans-serif">K =3D cubic_root((W_max* - cwnd)/C) (Eq. 2)=

instead of
=
K =3D cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq. 2)

Thank you all
Lisong
=




=

On Fri, No=
v 6, 2020 at 8:11 AM Neal Cardwell <ncardwell=3D40google.com@dmarc.ietf.org> wrote:
=

On Fri, Nov 6, 2020 at 3:=
28 AM Lars Eggert <=
lars@eggert.org> wrote:

> That said, the intent was to document what the Linux code was doing
> at the time. Is there a difference between the RFC and what Linux

> has been doing or is doing now? If yes, it might be time to revise

> the RFC.



Yes, it seems that this is a case where the RFC differs from the Linux

TCP CUBIC code.



The RFC says:

=C2=A0 =C2=A0 =C2=A0 K =3D cubic_root(W_max*(1-beta_cubic)/C) (Eq. 2)



which is equivalent to:

=C2=A0 =C2=A0 =C2=A0 K =3D cubic_root((W_max - W_max*beta_cubic)/C).=C2=A0 =
[B]



The Linux code does something different: it computes K once per epoch

immediately after the fast recovery or ECN cwnd reduction phase

completes, using:



=C2=A0 =C2=A0 =C2=A0 K =3D cubic_root((W_max - cwnd)/C).=C2=A0 [C]



See the computation of bic_K here:

=C2=A0 https://github.com/tor=
valds/linux/blob/master/net/ipv4/tcp_cubic.c#L254



For the non-fast-convergence case, the result of [B] and [C] are

equivalent, but for the fast convergence case they are not. So it

looks like in the case of fast convergence the RFC formula produces a

result that is incorrect, and does not match the Linux TCP CUBIC

behavior.



AFAICT,=C2=A0 changing Eq 2 to the following might fix the RFC and make it<=
br>
match the Linux TCP CUBIC behavior:



=C2=A0 =C2=A0 =C2=A0 K =3D cubic_root(W_max - W_last_max*beta_cubic)/C) (Eq=
. 2)



best,

neal



_______________________________________________

iccrg mailing list

iccrg@irtf.org

https://www.irtf.org/mailman/listinfo/iccrg




--0000000000009e2f9f05b386da14--


From nobody Sun Nov  8 23:57:46 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C652A3A0A4E for ; Sun,  8 Nov 2020 23:57:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h85Ru3_RNDaU for ; Sun,  8 Nov 2020 23:57:42 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3072E3A0A47 for ; Sun,  8 Nov 2020 23:57:41 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id D28BC25A1D for ; Mon,  9 Nov 2020 08:57:39 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1604908659; bh=OZts3zhiUhgBvYWEAwGAgWr0/Mq07z77QrdXXe8JfD8=; h=From:To:Subject:Date:References:In-Reply-To:From; b=t32ArdVIIkZkJoT0mhEeqYKJuG5yORSGjPUFAB2XzaIvNYXNADdpr1aqFhYpBTZNE yjWAq5jSUXHP7Li/AmtOEltlTgJDwruhQoY8MAxSthGIEX/L1wERDNxkGragl9POUD bku2yGKf+fdNow8S0usvJWaLcL8gZEKzHkuErfik=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XSIbKLBbWqI8 for ; Mon,  9 Nov 2020 08:57:38 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS for ; Mon,  9 Nov 2020 08:57:38 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Mon, 9 Nov 2020 08:57:37 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Mon, 9 Nov 2020 08:57:37 +0100
From: "Scharf, Michael" 
To: "tcpm@ietf.org" 
Thread-Topic: [ipr-announce] IPR Disclosure Deutsche Telekom AG's Statement about IPR related to draft-amend-tcpm-mptcp-robe
Thread-Index: AQHWs5w3rLPtGmu0BUG4VdqluVKYaKm/dAKg
Date: Mon, 9 Nov 2020 07:57:37 +0000
Message-ID: <46e73a0d5c7f4113bf28f47b0adc8a23@hs-esslingen.de>
References: <160459864866.28205.11773890483572335654@ietfa.amsl.com>
In-Reply-To: <160459864866.28205.11773890483572335654@ietfa.amsl.com>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: 
Subject: [tcpm] FW: [ipr-announce] IPR Disclosure Deutsche Telekom AG's Statement about IPR related to draft-amend-tcpm-mptcp-robe
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 09 Nov 2020 07:57:44 -0000

FYI

I forward this e-mail just to ensure that the TCPM WG is aware of the IPR d=
isclosure and its content (e.g., the licensing conditions).

Michael

-----Original Message-----
From: ipr-announce  On Behalf Of IETF Secret=
ariat
Sent: Thursday, November 5, 2020 6:51 PM
To: draft-amend-tcpm-mptcp-robe@ietf.org
Cc: ipr-announce@ietf.org
Subject: [ipr-announce] IPR Disclosure Deutsche Telekom AG's Statement abou=
t IPR related to draft-amend-tcpm-mptcp-robe

Dear Markus Amend, Jiao Kang:


An IPR disclosure that pertains to your Internet-Draft entitled "Multipath
TCP Extension for Robust Session Establishment" (draft-amend-tcpm-mptcp-rob=
e)
was submitted to the IETF Secretariat on 2020-11-04 and has been posted on
the "IETF Page of Intellectual Property Rights Disclosures"
(https://datatracker.ietf.org/ipr/4423/). The title of the IPR disclosure i=
s
"Deutsche Telekom AG's Statement about IPR related to
draft-amend-tcpm-mptcp-robe"


Thank you

IETF Secretariat


_______________________________________________
ipr-announce mailing list
ipr-announce@ietf.org
https://www.ietf.org/mailman/listinfo/ipr-announce


From nobody Mon Nov  9 02:50:59 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0A1043A0E6C for ; Mon,  9 Nov 2020 02:50:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4muJ0azGQCF1 for ; Mon,  9 Nov 2020 02:50:55 -0800 (PST)
Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 254B83A0E57 for ; Mon,  9 Nov 2020 02:50:55 -0800 (PST)
Received: by mail-qk1-x734.google.com with SMTP id r7so7444478qkf.3 for ; Mon, 09 Nov 2020 02:50:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8fdKTUOxRUDf7cA/HDXaLq9GLYWI3yIHRM8pS2u5gkU=; b=jWc2U2O9dEA584hTjOYbaz0QIAsDCKPRdZ5GHKCp1C+4guHnxBNJEkymPYFcLA8Xa0 8prcq+w106pOz9O8zoVNDXotTosi3xNOpJjGa9hjcF5GnzhNNkngxX0zTcECS0iaBro5 PFrLZzsxzG7UlEYEr6KzDXKJLNytuafpkCBj4wj68veUGEfG5wF+iG4Oue6ZSRN2xJBx PGqeojOsC+e3/bhP/bcCsYaxSd5oIo36OtwG+U2TYuRyeTSQDVI67odGchVj4it3VmHW ZCuZqhlE89St0hVTUAwpy3R6gNbXUjuw65IuHX2D+38A0aEXWZXbVVgUTfspZaw0dRPJ ILlQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8fdKTUOxRUDf7cA/HDXaLq9GLYWI3yIHRM8pS2u5gkU=; b=nNDjJJDkjnzv3iXSTjWKU0HlydC8OSXG3olaOHhIOZjnlsMYlC2BqNXR1LJLyGf4xN lUUowhJzAtNJn9YLJdJ9G3P/zTKfOkQ7gMdcdPENUdeq8mMKGybRwnkN2FcLNB/QEOgi RYpfbMkRTF/k3XGDuk1hEFzihenGFM6UZD9kA5rnjwYnbz5dG07Iqf9WewZtW4wC5Nyv rDp/y7Z8Lg8ICGQZULTM8X4Yh5ln0vRaxcb1CoLqJy8Nhkf7mpnZwdRsQPn1K9gc3Krz BcWmR/XmJkWJV0Ljpai/+c7/3bHm5ox9xFEdM/LHJJ3913HS6hXPbaG65qKaK/I3HmZP Q/Mw==
X-Gm-Message-State: AOAM532mwn0ipQ/BkGKfils9aZVHXL16VTMzwpZFnuJH6eeA3+/r6/I0 Fbdbfe+VPwJWYrHkyYYZhJ+2dom6/dMo10phKv0=
X-Google-Smtp-Source: ABdhPJxmzW5N+m0kiFnCUYVwuq9FBumXarmWsGaatXy7ASnWTzJCnmTNIBQ58lsuYewcGlQLv2eK2v1jBMFxrSpOEi0=
X-Received: by 2002:a37:87c5:: with SMTP id j188mr12978391qkd.476.1604919054099;  Mon, 09 Nov 2020 02:50:54 -0800 (PST)
MIME-Version: 1.0
References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net>     <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>
In-Reply-To: <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>
From: Yoshifumi Nishida 
Date: Mon, 9 Nov 2020 02:50:42 -0800
Message-ID: 
To: Bob Briscoe 
Cc: Mirja Kuehlewind ,  "Scheffenegger, Richard" , "tcpm@ietf.org" , Mirja Kuehlewind 
Content-Type: multipart/alternative; boundary="0000000000001ce91605b3aa562e"
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 09 Nov 2020 10:50:57 -0000

--0000000000001ce91605b3aa562e
Content-Type: text/plain; charset="UTF-8"

Hi Bob,

On Fri, Nov 6, 2020 at 5:01 AM Bob Briscoe  wrote:

> Yoshi,
>
> On 06/11/2020 10:28, Yoshifumi Nishida wrote:
>
> Hi Mirja,
>
> It seems to me that we don't have much freedom for the frequency of the
> feedback.
> The draft says:
>       MUST immediately send an ACK once 'n' CE marks have arrived since
>
>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>       than 6.
>
> Or, options are not applied to this?
>
>
> [BB]
> Section 3.2.2 about the ACE field, which is in the main TCP header feeds
> back the count of CE-marked *packets* .
> Section 3.2.3 is about the AccECN TCP Option, which is optional and feeds
> back a count of *bytes* for each marking.
>
> If there are any places in the draft where this isn't clear, we can fix
> them. But, in this case, the  text right under your quote says:
>
>    These rules for when to send an ACK are designed to be complemented
>    by those in Section 3.2.3.3 , which concern whether the AccECN TCP
>    Option ought to be included on ACKs.
>
>
> Also, even though traffic flows one direction, I think you might want to send back SACK options, etc.
>
>
> Last para of the Introduction:
>
>    It is recommended that the AccECN protocol is implemented alongside
>    SACK [RFC2018 ] and ...
>
>
> Also, interactions with SACK are raised where necessary throughout the
> draft. In particular this one:
>
>       If the smallest allowed AccECN Option would leave insufficient
>       space for two SACK blocks on a particular ACK, the Data Receiver
>       MUST give precedence to the SACK option (total 18 octets), because
>       loss feedback is more critical.
>
> I see. Thanks for the clarification.

Another thought is bulk flow traffic tends to send full MSS packets. I
feel 1 byte granularity might not be very efficient.
>
> In any cases, I just thought saving option space won't be a bad thing if it's not overly difficult.
>
>
> [BB] If efficiency is needed, field(s) (or the whole option) can be
> omitted.
> During the history of this draft we have proposed different ways to encode
> information more efficiently, but WG discussion decided that simple
> omission or inclusion of fields gives optional efficiency with minimal
> complexity.
>
> I'm not saying the act of reducing the granularity of each field would be
> particularly complex.
> Rather, complexity comes from the resulting ambiguity. Here's an example:
> * The Linux code currently uses the byte counters to help disambiguate
> wrap of the 3-bit packet counter.
> * If for instance the byte counters were truncated by 8 bits for
> efficiency, it would add many more ambiguous cases for this code to have to
> take into account.
> * Such complexity introduces misunderstanding and bugs.
>
> The design has gone round and round many times. We have tried to converge
> on the best tradeoffs.
>
>

OK. I have some concern that mitigating ambiguity by the using option may
require a certain high frequency,
but if the results from implementations look good, I don't have much things
to say here.

Thanks,
--
Yoshi

--0000000000001ce91605b3aa562e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Hi Bob,

On Fri, Nov 6, 2020 at 5:01 AM Bob =
Briscoe <ietf@bobbriscoe.net&=
gt; wrote:

 =20
   =20
 =20
  

    Yoshi,

    

    
On 06/11/2020 10:28, Yoshifumi Nishida
      wrote:

    

    

     =20
      

        
Hi Mirja,

        


        

        
It seems to me
            that we don't have much freedom for the frequency of the
            feedback.=C2=A0

        

        
The draft says:

        
=C2=A0=C2=A0=C2=A0 =C2=A0 MUST
              immediately send an ACK once 'n' CE marks have arrive=
d
              since

        
      =
the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
      than 6.

        
Or, op=
tions are not applied to this?

      

    

    

    [BB]

    Section 3.2.2 about the ACE field, which is in the main TCP header
    feeds back the count of CE-marked *packets* .

    Section 3.2.3 is about the AccECN TCP Option, which is optional and
    feeds back a count of *bytes* for each marking.

    

    If there are any places in the draft where this isn't clear, we can
    fix them. But, in this case, the=C2=A0 text right under your quote says=
:

    
   These rules for when to send an ACK are designed to be compleme=
nted
   by those in Section 3.2.3.3, which co=
ncern whether the AccECN TCP
   Option ought to be included on ACKs.=C2=A0

    

    

      

        
Also, =
even though traffic flows one direction, I think you might want to send bac=
k SACK options, etc.

      

    

    

    Last para of the Introduction:

    
   It is recommended that the AccECN protocol is implemented along=
side
   SACK [RFC2018] and=
 ...

    

    Also, interactions with SACK are raised where necessary throughout
    the draft. In particular this one:

    
      If the smallest allowed AccECN Option would leave insufficie=
nt
      space for two SACK blocks on a particular ACK, the Data Receiver
      MUST give precedence to the SACK option (total 18 octets), because
      loss feedback is more critical.
I see. T=
hanks for the clarification.


    

      

        
<=
font face=3D"arial, sans-serif">Another thought is bulk flow traffic tends to send full MS=
S packets. I feel 1 byte granularity might not be very efficient.

      

    

    

      

        
In any cases, I just thought saving option space =
won't be a bad thing if it's not overly difficult.


      

    

    

    [BB] If efficiency is needed, field(s) (or the whole option) can be
    omitted.

    During the history of this draft we have proposed different ways to
    encode information more efficiently, but WG discussion decided that
    simple omission or inclusion of fields gives optional efficiency
    with minimal complexity.

    

    I'm not saying the act of reducing the granularity of each field
    would be particularly complex. 

    Rather, complexity comes from the resulting ambiguity. Here's an
    example:

    * The Linux code currently uses the byte counters to help
    disambiguate wrap of the 3-bit packet counter. 

    * If for instance the byte counters were truncated by 8 bits for
    efficiency, it would add many more ambiguous cases for this code to
    have to take into account. 

    * Such complexity introduces misunderstanding and bugs. 

    

    The design has gone round and round many times. We have tried to
    converge on the best tradeoffs.
=C2=A0
=
=C2=A0
OK. I have some concern that mitigating ambiguity by the u=
sing=C2=A0option may require a certain high frequency,=C2=A0=C2=A0
but if the results from implementations look good, I don't have much =
things to say here.

Thanks,
--
Yoshi


--0000000000001ce91605b3aa562e--


From nobody Mon Nov  9 16:31:22 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9B32B3A1533 for ; Mon,  9 Nov 2020 16:31:19 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZgiRcJ0GgKcC for ; Mon,  9 Nov 2020 16:31:17 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C79133A1263 for ; Mon,  9 Nov 2020 16:31:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:MIME-Version:Date:Message-ID: Subject:From:To:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=K9lk4Z3tTAvGgaQb2uA8rGNLF3aZjw+lrs6t8yUaSN8=; b=BvqTtfYmyPzy84A89QGY2QYJRx 0HapFn1L/+8fhguPIgj/+IUvUbVtZ6X7dppImxRrmaauRLl3IrvExoe3qRrhE+iTji4FeKR9wuWE4 IfBvb8bM52/3d1lTISqQHCRJRydQyyv1QsLJo1fhTwtV6ipKJvXj/dUmdb8PTCgfJWJxGr3ZCrt2M 5xqEQlaKcwtZtX0yb+zp/WY4CNVeAmT4gWXfvuv7mjQusrBkhes+iR7qe6QgyHObE1YFs92iiEjGp wfIjS0HrDuX+1S4n5Mz4/DrsM81Q3j9qXaX+0K1McQsaGcJsqXkbdeoKKVuawFIqZkdEIP1CPlFjO jeij5d5Q==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:45658 helo=[192.168.1.11]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kcHZ4-004LF9-Gf; Tue, 10 Nov 2020 00:31:16 +0000
To: Richard Scheffenegger , Mirja Kuehlewind , =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= , tcpm IETF list 
From: Bob Briscoe 
Message-ID: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net>
Date: Tue, 10 Nov 2020 00:31:13 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="------------59B0390BB3B319789DFBF248"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 10 Nov 2020 00:31:20 -0000

This is a multi-part message in MIME format.
--------------59B0390BB3B319789DFBF248
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Mirja, Richard, Ilpo, tcpm list,

I've just been reading through the accurate-ecn draft to double-check. I 
think there's a problem with the following text...

3.2.2.5.1.  Data Receiver Safety Procedures

    An AccECN Data Receiver:

    o  SHOULD immediately send an ACK whenever a data packet marked CE
       arrives after the previous [data] packet was not CE.

    o  MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
       than 6.

...
    For the avoidance of doubt, the change-triggered ACK mechanism is
    deliberately worded to solely apply to data packets, and to ignore
    the arrival of a control packet with no payload, because it is
    important that TCP does not acknowledge pure ACKs.


In the first bullet, I think it doesn't matter whether the previous 
packet marked CE was a data packet or a pure ACK (i.e we should remove 
the second occurrence of 'data' that I have put in [square brackets].

The second bullet doesn't consider the possibility that the 'n'th CE 
mark might arrive on a pure ACK. Then, the wording as it stands says the 
Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a 
pure ACK, but I'm not actually sure it does any harm to do so in this 
case (it cannot cause an infinite loop of ACKs). However, given it would 
be unorthodox, we maybe ought to rule it out by rewording anyway?

Thoughts anyone?


Bob

--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------59B0390BB3B319789DFBF248
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  

    
  
  
    Mirja, Richard, Ilpo, tcpm list,

    

    I've just been reading through the accurate-ecn draft to
    double-check. I think there's a problem with the following text...

    

    
3.2.2.5.1.  Data Receiver Safety Procedures

   An AccECN Data Receiver:

   o  SHOULD immediately send an ACK whenever a data packet marked CE
      arrives after the previous [data] packet was not CE.

   o  MUST immediately send an ACK once 'n' CE marks have arrived since
      the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
      than 6.

...
   For the avoidance of doubt, the change-triggered ACK mechanism is
   deliberately worded to solely apply to data packets, and to ignore
   the arrival of a control packet with no payload, because it is
   important that TCP does not acknowledge pure ACKs.  

    

    In the first bullet, I think it doesn't matter whether the previous
    packet marked CE was a data packet or a pure ACK (i.e we should
    remove the second occurrence of 'data' that I have put in [square
    brackets].

    

    The second bullet doesn't consider the possibility that the 'n'th 
    CE mark might arrive on a pure ACK. Then, the wording as it stands
    says the Data Receiver MUST immediately ACK a pure ACK. I know TCP
    never ACKs a pure ACK, but I'm not actually sure it does any harm to
    do so in this case (it cannot cause an infinite loop of ACKs).
    However, given it would be unorthodox, we maybe ought to rule it out
    by rewording anyway?

    

    Thoughts anyone?

    

    

    Bob

    

    
--
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------59B0390BB3B319789DFBF248--


From nobody Tue Nov 10 01:41:47 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1537B3A0E43 for ; Tue, 10 Nov 2020 01:41:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level: 
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4BMkTPaXyWE0 for ; Tue, 10 Nov 2020 01:41:41 -0800 (PST)
Received: from pegasus.erg.abdn.ac.uk (pegasus.erg.abdn.ac.uk [IPv6:2001:630:42:150::2]) by ietfa.amsl.com (Postfix) with ESMTP id 0C4583A0D2B for ; Tue, 10 Nov 2020 01:41:40 -0800 (PST)
Received: from GF-MacBook-Pro.lan (fgrpf.plus.com [212.159.18.54]) by pegasus.erg.abdn.ac.uk (Postfix) with ESMTPSA id A7D7B1B00246; Tue, 10 Nov 2020 09:41:36 +0000 (GMT)
To: Bob Briscoe 
Cc: tcpm IETF list 
References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net>     <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>  <3c7126bd-ebdc-00b1-8b46-79125eb525a1@bobbriscoe.net>
From: Gorry Fairhurst 
Message-ID: <4d56be9b-06e0-6b2a-998c-23263b40e59d@erg.abdn.ac.uk>
Date: Tue, 10 Nov 2020 09:41:35 +0000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:78.0) Gecko/20100101 Thunderbird/78.3.3
MIME-Version: 1.0
In-Reply-To: <3c7126bd-ebdc-00b1-8b46-79125eb525a1@bobbriscoe.net>
Content-Type: multipart/alternative; boundary="------------CA4E695C55348E043307E28C"
Content-Language: en-GB
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 10 Nov 2020 09:41:46 -0000

This is a multi-part message in MIME format.
--------------CA4E695C55348E043307E28C
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit


See in-line, marked [GF], and I will try to unpick why I don't agree 
with some of this text.

On 06/11/2020 21:51, Bob Briscoe wrote:
> Gorry,
>
> Thanks as always for checking all this... inline tagged [BB]
>
> On 06/11/2020 14:01, Gorry Fairhurst wrote:
>> I see this rev introduced a section on ACK Filtering, which I have 
>> specific comments on. We need to consider the cost of sending ACKs - 
>> especially anything that results in them being large - however, this 
>> needs to be done by choosing a default at the receiver - not as an 
>> adaptation, the adaption if necessary needs to be introduce more ACKs 
>> when this is beneficial, as in some cases for DAAS, not in trying to 
>> figure something from delayed or "missing" ACKs.
>>
>> 1) So let me say first that I think the suggestion for updating the 
>> receiver to thin, and every 2-6 packets seems reasonable.
>>
>>     “MUST immediately send an ACK once 'n' CE marks have arrived since
>>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>>       than 6.”
>>
>> I do agree that endpoints can do much more to reduce the return path 
>> capacity that they consume - and that the endpoints have the 
>> opportunity to encode the feedback information more efficiently than 
>> can be done by a device (or queue) in the network. From our 
>> experiments, n=10 would also often be likely OK.
>
> NB: n is the no. of *CE marks*, which is only the ACK thinning ratio 
> in the worst case if there's 100% marking.
> IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
> whenever marking probability < 60%.
>
> Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
> duration of a 100% marking sequence, whenever some data packets are 
> not marked, the receiver aACK ratio can rise. This *could* allow any 
> build-up of ACKs queued on the reverse path to relax, which could 
> improve the chances of not overflowing into ACK loss.
>
> It's hard to predict when 100% marking in the forward direction will 
> coincide with reverse path congestion, and I'm not saying it won't.
> Just pointing out that n does not equal the ACK thinning ratio, it's 
> the floor of it.
>
> n=6 is a consequence of only having a 3-bit field to count CE marks. 
> We could push the envelope and make the max n=7. But not n=8, 
> otherwise wrap after 8 marks would be indistinguishable from zero marks.
>
[GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only when 
there is 100% marking - less with more likely patterns, that seems 
reasonable.

[GF] I also would agree that it is quite OK to increase the ACK-Ratio 
(send more ACKs) in some specific conditions, known to endpoints.

[GF] What concerns me, is that to avoid the need for ACK-Thining and 
related methods within the network, the "normal" condition needs to be 
for a rec eievr to send (far) fewer ACKs.

>>
>> 2. Although in some cases a receiver can predict a good "n", I do not 
>> believe a receiver can typically find out how to choose “n” by 
>> observing the ACKs on a connection.
>> 2. I do not believe a receiver can find out how top to choose “n” by 
>> observing the ACKs received on a connection.
>
> [BB] I think you intended to delete one of these sentences. Whatever, 
> both of them seem to contain a typo - otherwise they are circular? 
> Surely the receiver is sending the ACKs, so either sentence as it 
> stands ends with "...I do not believe a receiver can typically decide 
> how often to send an ACK by observing how often it is sending ACKs 
> itself." ???
>
[GF] So, the topic and solution are a little complex ... let me try to 
write a simpler statement: The receiver alone cannot predict a good ACK 
ratio based on the ACKs the remote endpoint receives. ECN-marking of an 
ACK stream does not provide useful informatiopn, because in this 
specific case, the control packets are small, and we enter the realm of 
packet-congestible links (as in RFC7141), where the overhead of 
transmission time slot scheduling and fitting packets to burst 
transmission plans can be more sigmnificant than the number of bits sent.

>>
>> Sure, if the path is modelled as a simple bit-congestive path that 
>> has a queue, then you could monitoring timing/loss of packets and 
>> determine a rate that tries not to fill a queue. I would not know how 
>> common such a bottleneck link is.
>>
>> I do know that many bottleneck links are more complex and needs to 
>> consider the scheduling transmission of small ACK packets on the 
>> return path performance. In many radio technologies, the overhead 
>> from transmission of ACKs is proportionally much higher than for data 
>> segments -  These layer 2s are constrained in terms of packets (e.g. 
>> where lower layer framing and transmission scheduling is an important 
>> consideration to the efficiency/cost of using the service). This has 
>> often been “fixed” using ACK Thinning or PEPs. Protocols that send 
>> fewer ACKs/data packet are not impacted.
>
> [BB] I agree that the receiver alone cannot predict a good 'n'.
> AccECN is a not the complete solution, but we've grasped the 
> opportunity to take a step in the right direction (within the 
> chartered scope of AccECN). AccECN at least allows the data sender to 
> measure the extent of congestion marking on the arriving ACK stream 
> (if there's ECN support at the bottleneck). Whereas before AccECN, a 
> random ECN mark on one pure ACK in a round couldn't be distinguished 
> from 100% of ACKs marked.
>
[GF] We've relied for decades on ACK-Thinning (or equivalent) in the 
network, where the link KNOWS what the sub-IP layer is doing, and hence 
knows whether an ACK is eating into resources that are scarce; but this 
solution is already problematic for encrypted traffic, and when ACKs 
carry more than ACK information. So, back to my point: To fix this, I 
think we need to reduce the level of feedback at the receiver.

[GF] So ... I see two problems with using ECN marks on the return path 
to vary the ACK Ratio:

(i) I'm not so sure the Layer 2 knows how to mark ACK packets - because 
they are not bit-congestible signals, and  I'd prefer to see ECN marking 
based on queue capacity (bit congestibale). I could see methods that let 
the sub-IP resource allocation ballance the use of forward and return 
transmssion opportunities and use this to CE-mark both directions - and 
see if flows adapt their return path resource consumption to optimise 
their forward data path transmission. It could perform better if it were 
to seperate ACKs (or any short packets? - also sometimes done) from 
"actual" data and then pro-actively CE-mark ACK packets to free-up 
transmission resource for other packets in the uplink or downlink. I'm 
not sure this cross-layer coupling is wise.

(ii) Before a new method is deployed, and the ACK Ratio remains 1:2, 
then the network nodes still do either ACK-Thinning or queue the ACKs.

>>
>> Scheduling (at the IP layer,can also often result in variations in 
>> timing - something that is hard to predict and I would suggests make 
>> automated selection of “n” very hard, and the need to thin is often 
>> not directly visible to a user of the path. Traffic multiplexed with 
>> protocols that use different ACK policies (perhaps QUIC?) also make 
>> this sort of detection less reliable.
>
> [BB] Being hard shouldn't prevent us trying.
> The remaining piece would be to advance a TCP ACK congestion control 
> protocol on the experimental track (RFC5690 
>  is informational).
>

[GF] To me, this is a complicated way to approach the question of how 
much feedback is needed. I actually don't see the justification for 
adding an extra control loop, that will interact with other control 
loops in the network.

>>
>> 3) A queue might be competing for scheduling opportunities against 
>> other IP flows (e.g., WFQ), or competing for transmission 
>> opportunities with a shared resource (common in radio links).
>>
>> ACKs can and do take transmission opportunities away from the 
>> resource pools that would be used by other services that share the 
>> same resource pool. This can be reduced by some form of per-flow 
>> scheduling with deeper per-flow queues - but that in itself can 
>> induce unwanted coupling between flows - limiting the peak rate of 
>> one flow, to achieve a higher ACK ratio for another flow.
>>
>> Consider a QUIC flow sharing a return path with a TCP flow, the 
>> higher rate of larger QUIC ACKs reduces the transmission 
>> opportunities for TCP ACKs, which in turn suppresses the TCP flow, 
>> limiting the overall system performance. If the ACKs use the same 
>> transmission resource, they could take directly away from 
>> transmission opportunities in the forward direction, but even if they 
>> do not - the rate of ACKs retuned impacts the sending rate end to end.
>
> [BB] This says two things to me:
> * if TCP is going to introduce control over the ACK ratio, QUIC and 
> other transports ought to as well {Note 1}.
> * and all ACK CC algorithms ought to be designed to ensure some degree 
> of equitable capacity sharing {Note 2}, including between the ACK 
> stream of flow A and the data stream of flow B transferring in the 
> opposite direction.
>
> {Note 1}: I don't think this needs any formal "all or nothing" rule 
> where the transport ADs co-ordinate multiple WGs. Enough of the people 
> involved are the same for the right thing to happen (I think). But 
> some AD oversight wouldn't do any harm.
> {Note 2}: There are numerous challenges to face, but I think it's 
> paramount to first establish which node is considered responsible for 
> solving the problem (the bottleneck or one of the transport 
> endpoints). This doesn't imply that the IETF will ever ensure that 
> other nodes don't 'interfere'. For instance, if the sender is deemed 
> responsible for determining the level of ACK filtering, it will still 
> have to make allowance for bottlenecks attempting to ACK thin as well.
>
>
>>
>> 4) The new section adds some cross-layer requirements on sub-IP 
>> layers, which seem to imply more complexity. I’m not comfortable with 
>> such a proposal.
>>
>> It seems that the text suggests not implement thinning and that these 
>> systems implement logic to detect the presence of the option by 
>> observing SYNs and then configure the link to do something special. 
>
> This was the proposal that I was most interested to hear your views 
> on. We can remove it if the WG doesn't want it, but first I want to 
> make sure the proposal is fully understood:
> * The aim is for the RFC to tell bottleneck implementers (via this 
> text) that *only in cases where AccECN has been negotiated* the 
> preferred way to improve flow performance is for the link to signal 
> ECN to the transport, not to drop the ACKs itself.

[GF] In principle could be interesting - if this were a separate signal 
for "ACKs" - we don't have this signal. We have a signal for all IP traffic.

> * If the link doesn't support ECN, it still does the ACK filtering itself
> * Links aren't going to delegate their control unless transports 
> actually act on the ECN signals that links emit - and improve 
> performance better than links have been. So, it's important that 
> transports take on these responsibilities - the data sender takes 
> responsibility for determining the ACK rate, asking its correspondent 
> receiver does actually thin its own ACKs.
> Here's who signals what, and who does what
>
>                link-.
>                      \
>                       '-[ECN CE]-.
>                                   \
>                                    '-> receiver
>                                     .-
>                                    /
>             .-[(Acc)ECN feedback]-'
>            /
>  sender <-'
>         -.
>           \
>            '---[ACK CC ratio]---.
>                                  \
>                                   '-> receiver(treats as polite 
> request, not mandatory)
>
>
[GF] That's a useful diagram to help understand!!! The control loops for 
layer 2 channel sharing oftem operate on a similar timescale.

[GF] Let's be sure we start from the same problem:

* Current TCP Specs suggest the ACK Ratio is 1:2. I could say that means 
that 33% of transmission opportunities are used solely for feedback. 
Before we had sophisticated transports, tha was fine, but nowadays this 
is a very high proportion by modern standards, and given the way the 
ACK-Clock no longer has to be for each segment or pair of segments ... 
reality moved away from that when stretch-ACKs became common.

* Many senders (e.g, using offload) might have an ACK Ratio of 1:4, but 
not all.

* Some paths "Thin" ... by how much depends.  In many cases, an ACK 
Ratio 1:2, causes excessive resource usage that predjudices performance, 
an ACK Ratio of 1:4 is still 20% of transmission opportunities. An 
ACK-Ratio of 1:10, would mean 10% of transmission opportunities are used 
for feedback (that's an order of magnitude less control than data 
segments, and is a reasonable operating point). My suggestion is that 
MOST of the time, transports can live with an ACK Ratio 1:10, and in 
many cases this is enough to avoid ACK Thinning measures being triggered.

* A transport can adapt feedback, for sure,  and there will be cases 
where lower rate (e.g. 1:1 or 1:2 initially) or higher rates (1:10,1:20, 
etc as rates increase and rate-based methods are used) will be benefit 
the transport - but changing the default operating point from 1:2 to 
1:10 is likely already significantly reducing the need (and triggering) 
of ACK thinning.

- So, maybe we should do this, rather than encourage a new generation of 
smart ECN-based ACK modification schemes. If we mitigate the feedback 
traffic, we can then focus on deploying ECN for capacity detection.

>> I don’t think think this is the correct approach - the result will be 
>> throttling of the connection speed. Is it necessary that the sub-IP 
>> layer does not implement thinning for connections that use AccECN 
>> feedback?
>
> Why do you say the connection speed would be throttled?
[GF] For example: A sudden increasse in reverse path traffic will need 
to be buffered until the radio transmission schedule changes. That will 
delay feedback to the sender (advertised rwnd/cwnd for TCP), and hence 
the forward rate will then throttle determined by the bottleneck ACK rate.
> Did the text not explain the proposal well?

Well, I think the problem is complex, which is why this email is already 
long.

> Or did you understand the proposal but disagree with it?
>
>
>>
>> When the text says: “SHOULD preserve the timing of each ACK”
>> - does that imply that nodes need to discard rather than delay 
>> packets when link scheduling or transmission scheduling is used. This 
>> seems a very odd requirement, it might be better too simply strip-out 
>> the AccECN option or discard all packets that set this.
>
> Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
> filter ACKs" (if it supports ECN marking).
>>
>> I can’t see how queuing the ACKs, or deleting ACKs rather than 
>> queuing them would improve the performance of AccECN TCP connections.
>
> May be the piece where the link's preferred role is to signal to the 
> transport to slow down the ACK rate using ECN wasn't clear.
>
>
> Bob
>
>>
>> Gorry
>>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

Summary ... because this thread has lots of "discussion" or a 
complicated interaction, and you asked about the AccECN draft text:

+ I agree that AccECN can be used with a sender that uses a more 
appropiate target ACK Ratio - that is good! I agree this is worth saying.

+ ACK Thining, and variants - including uses of compression - is a 
deployed mitigation, and that changing the ACK Ratio can eliminate this 
practice and the need to implement this.

+ I agree that if the path is a simple bottleneck, then ECN-marking 
should be encouraged. However, not appropiate without much more 
consideration for a more complex L2 where the marking is 
packet-congestible. A link predomonantly carrying small ACKs seems to 
lead into the research issues in RFC7141.

+ I would disagree with any encouragement to use cross-layer methods 
with use ECN-marking to seek to adjust ACK feedback traffic and I am 
concerned that using ECN to modify the ACK Ratio might create new layers 
of feedback loop that will aggrevate the problem.

I think the current text goes to far.

Gorry


--------------CA4E695C55348E043307E28C
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    


    

    
See in-line, marked [GF], and I will
      try to unpick why I don't agree with some of this text.

    

    


    

    
On 06/11/2020 21:51, Bob Briscoe wrote:

    

    

      
      Gorry,

      

      Thanks as always for checking all this... inline tagged [BB]

      

      
On 06/11/2020 14:01, Gorry Fairhurst
        wrote:

      

      
I
        see this rev introduced a section on ACK Filtering, which I have
        specific comments on. We need to consider the cost of sending
        ACKs - especially anything that results in them being large -
        however, this needs to be done by choosing a default at the
        receiver - not as an adaptation, the adaption if necessary needs
        to be introduce more ACKs when this is beneficial, as in some
        cases for DAAS, not in trying to figure something from delayed
        or "missing" ACKs. 

        

        1) So let me say first that I think the suggestion for updating
        the receiver to thin, and every 2-6 packets seems reasonable. 

        

            “MUST immediately send an ACK once 'n' CE marks have arrived
        since 

              the previous ACK, where 'n' SHOULD be 2 and MUST be no
        greater 

              than 6.” 

        

        I do agree that endpoints can do much more to reduce the return
        path capacity that they consume - and that the endpoints have
        the opportunity to encode the feedback information more
        efficiently than can be done by a device (or queue) in the
        network. From our experiments, n=10 would also often be likely
        OK. 

      

      

      NB: n is the no. of *CE marks*, which is only the ACK thinning
      ratio in the worst case if there's 100% marking. 

      IOW, with n=6 the receiver could ACK less than 1 in 10 data
      packets whenever marking probability < 60%.

      

      Altho the receiver's ACK ratio will have to fall to 6 to 1 for the
      duration of a 100% marking sequence, whenever some data packets
      are not marked, the receiver aACK ratio can rise. This *could*
      allow any build-up of ACKs queued on the reverse path to relax,
      which could improve the chances of not overflowing into ACK loss.

      

      It's hard to predict when 100% marking in the forward direction
      will coincide with reverse path congestion, and I'm not saying it
      won't.

      Just pointing out that n does not equal the ACK thinning ratio,
      it's the floor of it.

      

      n=6 is a consequence of only having a 3-bit field to count CE
      marks. We could push the envelope and make the max n=7. But not
      n=8, otherwise wrap after 8 marks would be indistinguishable from
      zero marks.

      

    

    
[GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only
      when there is 100% marking - less with more likely patterns, that
      seems reasonable. 

    

    
[GF] I also would agree that it is quite OK to increase the
      ACK-Ratio (send more ACKs) in some specific conditions, known to
      endpoints.

    

    
[GF] What concerns me, is that to avoid the need for ACK-Thining
      and related methods within the network, the "normal" condition
      needs to be for a rec eievr to send (far) fewer ACKs.

    

    

      

        

        2. Although in some cases a receiver can predict a good "n", I
        do not believe a receiver can typically find out how to choose
        “n” by observing the ACKs on a connection. 

      

      

        2. I do not believe a receiver can find out how top to choose
        “n” by observing the ACKs received on a connection. 

      

      

      [BB] I think you intended to delete one of these sentences.
      Whatever, both of them seem to contain a typo - otherwise they are
      circular? Surely the receiver is sending the ACKs, so either
      sentence as it stands ends with "...I do not believe a receiver
      can typically decide how often to send an ACK by observing how
      often it is sending ACKs itself." ???

      

    

    
[GF] So, the topic and solution are a little complex ... let me
      try to write a simpler statement: The receiver alone cannot
      predict a good ACK ratio based on the ACKs the remote endpoint
      receives. ECN-marking of an ACK stream does not provide useful
      informatiopn, because in this specific case, the control packets
      are small, and we enter the realm of packet-congestible links (as
      in RFC7141), where the overhead of transmission time slot
      scheduling and fitting packets to burst transmission plans can be
      more sigmnificant than the number of bits sent.

    

    

      

        

        Sure, if the path is modelled as a simple bit-congestive path
        that has a queue, then you could monitoring timing/loss of
        packets and determine a rate that tries not to fill a queue. I
        would not know how common such a bottleneck link is. 

        

        I do know that many bottleneck links are more complex and needs
        to consider the scheduling transmission of small ACK packets on
        the return path performance. In many radio technologies, the
        overhead from transmission of ACKs is proportionally much higher
        than for data segments -  These layer 2s are constrained in
        terms of packets (e.g. where lower layer framing and
        transmission scheduling is an important consideration to the
        efficiency/cost of using the service). This has often been
        “fixed” using ACK Thinning or PEPs. Protocols that send fewer
        ACKs/data packet are not impacted. 

      

      

      

      

      [BB] I agree that the receiver alone cannot predict a good 'n'. 

      AccECN is a not the complete solution, but we've grasped the
      opportunity to take a step in the right direction (within the
      chartered scope of AccECN). AccECN at least allows the data sender
      to measure the extent of congestion marking on the arriving ACK
      stream (if there's ECN support at the bottleneck). Whereas before
      AccECN, a random ECN mark on one pure ACK in a round couldn't be
      distinguished from 100% of ACKs marked.

      

    

    
[GF] We've relied for decades on ACK-Thinning (or equivalent) in
      the network, where the link KNOWS what the sub-IP layer is doing,
      and hence knows whether an ACK is eating into resources that are
      scarce; but this solution is already problematic for encrypted
      traffic, and when ACKs carry more than ACK information. So, back
      to my point: To fix this, I think we need to reduce the level of
      feedback at the receiver.

    

    
[GF] So ... I see two problems with using ECN marks on the return
      path to vary the ACK Ratio:

    
(i) I'm not so sure the Layer 2 knows how to mark ACK packets -
      because they are not bit-congestible signals, and  I'd prefer to
      see ECN marking based on queue capacity (bit congestibale). I
      could see methods that let the sub-IP resource allocation ballance
      the use of forward and return transmssion opportunities and use
      this to CE-mark both directions - and see if flows adapt their
      return path resource consumption to optimise their forward data
      path transmission. It could perform better if it were to seperate
      ACKs (or any short packets? - also sometimes done) from "actual"
      data and then pro-actively CE-mark ACK packets to free-up
      transmission resource for other packets in the uplink or downlink.
      I'm not sure this cross-layer coupling is wise.  

    

    
(ii) Before a new method is deployed, and the ACK Ratio remains
      1:2, then the network nodes still do either ACK-Thinning or queue
      the ACKs.

    

    

      

        

        Scheduling (at the IP layer,can also often result in variations
        in timing - something that is hard to predict and I would
        suggests make automated selection of “n” very hard, and the need
        to thin is often not directly visible to a user of the path.
        Traffic multiplexed with protocols that use different ACK
        policies (perhaps QUIC?) also make this sort of detection less
        reliable. 

      

      

      [BB] Being hard shouldn't prevent us trying.

      The remaining piece would be to advance a TCP ACK congestion
      control protocol on the experimental track (RFC5690 is informational).

      

    

    

    
 [GF] To me, this is a complicated way to approach the question
      of how much feedback is needed. I actually don't see the
      justification for adding an extra control loop, that will interact
      with other control loops in the network. 

    

    

      

        

        3) A queue might be competing for scheduling opportunities
        against other IP flows (e.g., WFQ), or competing for
        transmission opportunities with a shared resource (common in
        radio links). 

        

        ACKs can and do take transmission opportunities away from the
        resource pools that would be used by other services that share
        the same resource pool. This can be reduced by some form of
        per-flow scheduling with deeper per-flow queues - but that in
        itself can induce unwanted coupling between flows - limiting the
        peak rate of one flow, to achieve a higher ACK ratio for another
        flow. 

        

        Consider a QUIC flow sharing a return path with a TCP flow, the
        higher rate of larger QUIC ACKs reduces the transmission
        opportunities for TCP ACKs, which in turn suppresses the TCP
        flow, limiting the overall system performance. If the ACKs use
        the same transmission resource, they could take directly away
        from transmission opportunities in the forward direction, but
        even if they do not - the rate of ACKs retuned impacts the
        sending rate end to end. 

      

      

      [BB] This says two things to me: 

      * if TCP is going to introduce control over the ACK ratio, QUIC
      and other transports ought to as well {Note 1}. 

      * and all ACK CC algorithms ought to be designed to ensure some
      degree of equitable capacity sharing {Note 2}, including between
      the ACK stream of flow A and the data stream of flow B
      transferring in the opposite direction.

      

      {Note 1}: I don't think this needs any formal "all or nothing"
      rule where the transport ADs co-ordinate multiple WGs. Enough of
      the people involved are the same for the right thing to happen (I
      think). But some AD oversight wouldn't do any harm.

      {Note 2}: There are numerous challenges to face, but I think it's
      paramount to first establish which node is considered responsible
      for solving the problem (the bottleneck or one of the transport
      endpoints). This doesn't imply that the IETF will ever ensure that
      other nodes don't 'interfere'. For instance, if the sender is
      deemed responsible for determining the level of ACK filtering, it
      will still have to make allowance for bottlenecks attempting to
      ACK thin as well.

      

    

    
 

      

        

        4) The new section adds some cross-layer requirements on sub-IP
        layers, which seem to imply more complexity. I’m not comfortable
        with such a proposal. 

        

        It seems that the text suggests not implement thinning and that
        these systems implement logic to detect the presence of the
        option by observing SYNs and then configure the link to do
        something special. 

      

      This was the proposal that I was most interested to hear your
      views on. We can remove it if the WG doesn't want it, but first I
      want to make sure the proposal is fully understood:

      * The aim is for the RFC to tell bottleneck implementers (via this
      text) that *only in cases where AccECN has been negotiated* the
      preferred way to improve flow performance is for the link to
      signal ECN to the transport, not to drop the ACKs itself. 

    

    
[GF] In principle could be interesting - if this were a separate
      signal for "ACKs" - we don't have this signal. We have a signal
      for all IP traffic.

    

    
 *
      If the link doesn't support ECN, it still does the ACK filtering
      itself

      * Links aren't going to delegate their control unless transports
      actually act on the ECN signals that links emit - and improve
      performance better than links have been. So, it's important that
      transports take on these responsibilities - the data sender takes
      responsibility for determining the ACK rate, asking its
      correspondent receiver does actually thin its own ACKs.

    

    

      Here's who signals what, and who does what

      

                     link-.

                             \

                              '-[ECN CE]-.

                                          \

                                           '-> receiver

                                            .-

                                           /

                    .-[(Acc)ECN feedback]-'

                   /

         sender <-'

                -.

                  \

                   '---[ACK CC ratio]---.

                                         \

                                          '-> receiver
        (treats as polite request, not mandatory)

      

      

    

    
[GF] That's a useful diagram to help understand!!! The control
      loops for layer 2 channel sharing oftem operate on a similar
      timescale.

    
[GF] Let's be sure we start from the same problem: 

    

    
* Current TCP Specs suggest the ACK Ratio is 1:2. I could say
      that means that 33% of transmission opportunities are used solely
      for feedback. Before we had sophisticated transports, tha was
      fine, but nowadays this is a very high proportion by modern
      standards, and given the way the ACK-Clock no longer has to be for
      each segment or pair of segments ... reality moved away from that
      when stretch-ACKs became common. 

    

    
* Many senders (e.g, using offload) might have an ACK Ratio of
      1:4, but not all. 

    

    
* Some paths "Thin" ... by how much depends.  In many cases, an
      ACK Ratio 1:2, causes excessive resource usage that predjudices
      performance, an ACK Ratio of 1:4 is still 20% of transmission
      opportunities. An ACK-Ratio of 1:10, would mean 10% of
      transmission opportunities are used for feedback (that's an order
      of magnitude less control than data segments, and is a reasonable
      operating point). My suggestion is that MOST of the time,
      transports can live with an ACK Ratio 1:10, and in many cases this
      is enough to avoid ACK Thinning measures being triggered. 

    

    
* A transport can adapt feedback, for sure,  and there will be
      cases where lower rate (e.g. 1:1 or 1:2 initially) or higher rates
      (1:10,1:20, etc as rates increase and rate-based methods are used)
      will be benefit the transport - but changing the default operating
      point from 1:2 to 1:10 is likely already significantly reducing
      the need (and triggering) of ACK thinning.

    
- So, maybe we should do this, rather than encourage a new
      generation of smart ECN-based ACK modification schemes. If we
      mitigate the feedback traffic, we can then focus on deploying ECN
      for capacity detection.

    

    

      
I
        don’t think think this is the correct approach - the result will
        be throttling of the connection speed. Is it necessary that the
        sub-IP layer does not implement thinning for connections that
        use AccECN feedback? 

      

      

      Why do you say the connection speed would be throttled? 

    

    [GF] For example: A sudden increasse in reverse path traffic will
    need to be buffered until the radio transmission schedule changes.
    That will delay feedback to the sender (advertised rwnd/cwnd for
    TCP), and hence the forward rate will then throttle determined by
    the bottleneck ACK rate.

    

      Did the text not explain the proposal well?

    

    
Well, I think the problem is complex, which is why this email is
      already long.

    

    
 Or
      did you understand the proposal but disagree with it?

      

    

    
 

      

        

        When the text says: “SHOULD preserve the timing of each ACK” 

        - does that imply that nodes need to discard rather than delay
        packets when link scheduling or transmission scheduling is used.
        This seems a very odd requirement, it might be better too simply
        strip-out the AccECN option or discard all packets that set
        this. 

      

      

      Sorry, that may be clumsy wording. It just means "the link SHOULD
      NOT filter ACKs" (if it supports ECN marking).

    

    

      

        

        I can’t see how queuing the ACKs, or deleting ACKs rather than
        queuing them would improve the performance of AccECN TCP
        connections. 

      

      

      May be the piece where the link's preferred role is to signal to
      the transport to slow down the ACK rate using ECN wasn't clear.

      

    

    
 

      Bob

      

      

        

        Gorry 

        

      

      

      
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

    

    
Summary ... because this thread has lots of "discussion" or a
      complicated interaction, and you asked about the AccECN draft
      text: 

    

    
+ I agree that AccECN can be used with a sender that uses a more
      appropiate target ACK Ratio - that is good! I agree this is worth
      saying.

    

    
+ ACK Thining, and variants - including uses of compression - is
      a deployed mitigation, and that changing the ACK Ratio can
      eliminate this practice and the need to implement this.

    

    
+ I agree that if the path is a simple bottleneck, then
      ECN-marking should be encouraged. However, not appropiate without
      much more consideration for a more complex L2 where the marking is
      packet-congestible. A link predomonantly carrying small ACKs seems
      to lead into the research issues in RFC7141.

    

    
+ I would disagree with any encouragement to use cross-layer
      methods with use ECN-marking to seek to adjust ACK feedback
      traffic and I am concerned that using ECN to modify the ACK Ratio
      might create new layers of feedback loop that will aggrevate the
      problem. 

    

    
I think the current text goes to far.

    

    
Gorry

    

  


--------------CA4E695C55348E043307E28C--


From nobody Tue Nov 10 02:08:11 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 432A63A0E85 for ; Tue, 10 Nov 2020 02:08:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f-lymDkH7SqZ for ; Tue, 10 Nov 2020 02:08:07 -0800 (PST)
Received: from wp513.webpack.hosteurope.de (wp513.webpack.hosteurope.de [IPv6:2a01:488:42:1000:50ed:8223::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5FD313A0E83 for ; Tue, 10 Nov 2020 02:08:07 -0800 (PST)
Received: from p200300dee707b6000c59cd27c374f47a.dip0.t-ipconnect.de ([2003:de:e707:b600:c59:cd27:c374:f47a]); authenticated by wp513.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1kcQZH-0004BN-UI; Tue, 10 Nov 2020 11:08:03 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
From: Mirja Kuehlewind 
In-Reply-To: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net>
Date: Tue, 10 Nov 2020 11:08:03 +0100
Cc: Richard Scheffenegger , =?utf-8?Q?Ilpo_J=C3=A4rvinen?= , tcpm IETF list 
Content-Transfer-Encoding: quoted-printable
Message-Id: <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net>
To: Bob Briscoe 
X-Mailer: Apple Mail (2.3608.120.23.2.4)
X-bounce-key: webpack.hosteurope.de;ietf@kuehlewind.net;1605002887;13f8d6f1;
X-HE-SMSGID: 1kcQZH-0004BN-UI
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 10 Nov 2020 10:08:09 -0000

Hi Bob,

Please see below.

> On 10. Nov 2020, at 01:31, Bob Briscoe  =
wrote:
>=20
> Mirja, Richard, Ilpo, tcpm list,
>=20
> I've just been reading through the accurate-ecn draft to double-check. =
I think there's a problem with the following text...
>=20
> 3.2.2.5.1.  Data Receiver Safety Procedures
>=20
>    An AccECN Data Receiver:
>=20
>    o  SHOULD immediately send an ACK whenever a data packet marked CE
>       arrives after the previous [data] packet was not CE.
>=20
>    o  MUST immediately send an ACK once 'n' CE marks have arrived =
since
>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>       than 6.
>=20
> ...
>    For the avoidance of doubt, the change-triggered ACK mechanism is
>    deliberately worded to solely apply to data packets, and to ignore
>    the arrival of a control packet with no payload, because it is
>    important that TCP does not acknowledge pure ACKs. =20
>=20
>=20
> In the first bullet, I think it doesn't matter whether the previous =
packet marked CE was a data packet or a pure ACK (i.e we should remove =
the second occurrence of 'data' that I have put in [square brackets].

I believe it does matter. If the previous packet was a pure ACK and was =
CE marked, you didn=E2=80=99t send an ACK and so you should immediately =
send one now. If the previous packet was a data packet and CE marked, =
you don=E2=80=99t need to send an immediate ACK because you already did =
this with the previous packet. However, it text might be a but ambiguous =
because what is meant is =E2=80=9Cif the previous packet was a data =
packet and CE marked=E2=80=9D. So this does not apply if we e.g. have a =
CE-marked data packets, then a pure ACK that is not CE marked, and then =
another CE-marked data packet. =20

>=20
> The second bullet doesn't consider the possibility that the 'n'th  CE =
mark might arrive on a pure ACK. Then, the wording as it stands says the =
Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a =
pure ACK, but I'm not actually sure it does any harm to do so in this =
case (it cannot cause an infinite loop of ACKs). However, given it would =
be unorthodox, we maybe ought to rule it out by rewording anyway?

I think we also can leave this as it is because if that =E2=80=99n=E2=80=99=
 packet is a pure ACK that still means that you have unacked data =
packets with CE marks and you should trigger an ACK for those packet now =
(rather than waiting for the delayed ACK timer to expire). However this =
case is less important. Also we should probably make sure that this =
doesn=E2=80=99t apply if there are only pure ACKs with CE marks, maybe =
by add =E2=80=9Cif unacknowledged data are outstanding=E2=80=9D or =
something.

Mirja



>=20
> Thoughts anyone?
>=20
>=20
> Bob
>=20
> --
> ________________________________________________________________
> Bob Briscoe                              =20
> http://bobbriscoe.net/


From nobody Tue Nov 10 08:39:08 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 16C293A0AD3 for ; Tue, 10 Nov 2020 08:39:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.101
X-Spam-Level: 
X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zKwEJmhs9J5P for ; Tue, 10 Nov 2020 08:39:02 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 572993A0A2B for ; Tue, 10 Nov 2020 08:39:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=kMcVpvgHPBtoDqyuRSBfESXxBYBbPgTUWdeczv0mksU=; b=J5UNjj4DmcZFcVuDEg+Q1r9zQz rMh6XCPLsH9o9jXde83QxmRJfmxTklDDu+so3qg4KDoNSTMha/DEPJLdvgxsNNICF/y8/EQdVMMDq 647LeojoM7Vhba3EDZnSx7Qwk3nuD8FZza37so2lzzJA7oL0/thP+OIiWRbp4eZl+rrR3HCgfh7/N qYInDoT4EKoS32oX2wlGmwqebchYglr/5XB6jjYAorGjHMT4AT5uFEsLr3/0EtAEtmZIHHUgLFL+9 mPUaZsxjoM8EHkjGmTF3k1IerhcpkMqw4GKdJ36QPY/cvDUSA2Btv4jEKGMV6LkwXamifvjs645X4 AVFCVq7w==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:46816 helo=[192.168.1.11]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kcWfa-009r0h-4c; Tue, 10 Nov 2020 16:39:00 +0000
To: Mirja Kuehlewind 
Cc: tcpm IETF list , Richard Scheffenegger 
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>
From: Bob Briscoe 
Message-ID: <060c8bd8-d64b-3e46-7874-742e35e6d114@bobbriscoe.net>
Date: Tue, 10 Nov 2020 16:38:57 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-0.2
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 10 Nov 2020 16:39:06 -0000

Mirja,

On 10/11/2020 10:08, Mirja Kuehlewind wrote:
> Hi Bob, Please see below.
>> On 10. Nov 2020, at 01:31, Bob Briscoe  
>> wrote: Mirja, Richard, Ilpo, tcpm list, I've just been reading 
>> through the accurate-ecn draft to double-check. I think there's a 
>> problem with the following text... 
>> 3.2.2.5.1.  Data Receiver Safety Procedures
>>
>>     An AccECN Data Receiver:
>>
>>     o  SHOULD immediately send an ACK whenever a data packet marked CE
>>        arrives after the previous [data] packet was not CE.
>>
>>     o  MUST immediately send an ACK once 'n' CE marks have arrived since
>>        the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>>        than 6.
>>
>> ...
>>     For the avoidance of doubt, the change-triggered ACK mechanism is
>>     deliberately worded to solely apply to data packets, and to ignore
>>     the arrival of a control packet with no payload, because it is
>>     important that TCP does not acknowledge pure ACKs.
>> In the first bullet, I think it doesn't matter whether the previous 
>> packet marked CE was a data packet or a pure ACK (i.e we should 
>> remove the second occurrence of 'data' that I have put in [square 
>> brackets]. 
> [MK] I believe it does matter. If the previous packet was a pure ACK 
> and was CE marked, you didn’t send an ACK and so you should 
> immediately send one now. 

[BB] Neither the current text, nor my proposed edit, nor the text you 
propose below, says this. And I don't think it needs to. Surely this 
falls into the case of the second bullet ('n' CE marks in a row before 
needing to feed back anything).

> [MK] If the previous packet was a data packet and CE marked, you don’t 
> need to send an immediate ACK because you already did this with the 
> previous packet. However, it text might be a but ambiguous because 
> what is meant is “if the previous packet was a data packet and CE 
> marked”. So this does not apply if we e.g. have a CE-marked data 
> packets, then a pure ACK that is not CE marked, and then another 
> CE-marked data packet. 

[BB] An AQM will typically mark packets irrespective of their type or 
size (that's a feature not a bug). So there's no reason for the receiver 
to treat some marks differently to others, whatever type or size of 
packets they're on.

Let's revisit your last sentence above with a more general scenario - 
with more than one pure ACK in between the data - actually quite a 
common pattern. I.e. a server receives a small request, sends its larger 
response, then when that response has completed, the client sends 
another small request. So the server's TCP will see one data packet, a 
load of pure ACKs then one data packet again.

Now consider the case where both arriving data packets (the requests) 
are CE-marked. There are three relevant cases for the marking of the 
pure ACKs in between:
1) no CE-marked pure ACKs
2) some non-CE-marked pure ACKs.
3) all CE-marked pure ACKs

In cases 1 & 2, there has been a transition from none to some CE marks, 
but there hasn't been an opportunity to report it yet, so use the first 
opportunity to ACK a data packet.
In case 3, surely the server doesn't need to send feedback unless there 
are 'n' CE marks to report.

So how about this text to replace the first bullet:

    An AccECN Data Receiver:

    o  SHOULD send an ACK at the first opportunity (in response to the
       arrival of a data packet) whenever a mix of non-CE marked
       and CE marked packets have arrived since any previous ACK.

Moving on to the second bullet...
>> The second bullet doesn't consider the possibility that the 'n'th CE 
>> mark might arrive on a pure ACK. Then, the wording as it stands says 
>> the Data Receiver MUST immediately ACK a pure ACK. I know TCP never 
>> ACKs a pure ACK, but I'm not actually sure it does any harm to do so 
>> in this case (it cannot cause an infinite loop of ACKs). However, 
>> given it would be unorthodox, we maybe ought to rule it out by 
>> rewording anyway? 
> [MK] I think we also can leave this as it is because if that ’n’ 
> packet is a pure ACK that still means that you have unacked data 
> packets with CE marks and you should trigger an ACK for those packet 
> now (rather than waiting for the delayed ACK timer to expire). However 
> this case is less important. Also we should probably make sure that 
> this doesn’t apply if there are only pure ACKs with CE marks, maybe by 
> add “if unacknowledged data are outstanding” or something.

[BB] By the end of your last para you start to realize that, if the n'th 
CE mark arrives on pure ACK, it doesn't say anything about whether the 
previous (n-1) CE marks were on data or pure ACKs.

Consider another common scenario; a server delivering chunks of video 
over a high BDP path (e.g. 500 packets per RTT), and going idle between 
times (perhaps in response to an "exceeded high watermark" data packet 
from the server). Now consider the possibility that the reverse path 
bottleneck builds a queue during each chunk, and starts CE marking 
towards the end of each chunk. Then, after the server stops sending a 
chunk, it continues to see a couple of hundred CE-marked pure ACKs 
arriving.

However, by your proposed rule, it doesn't send any feedback, 'cos it 
has no data to acknowledge. So the client cannot regulate its ACK 
stream. That's just tough, 'cos the client will stop sending more pure 
ACKs after it has received the last data, which is before any later 
feedback from the server can reach it.

However, if the client later sends a data packet that the server can 
acknowledge (e.g. "a low watermark" message), the server's ACE counter 
will have wrapped dozens of times, and the client won't be able to work 
out how many. That doesn't matter if a few round trips have elapsed - 
the congestion info will have become stale. But it does matter if it's 
fairly soon.

Instead, if we keep the text of the second bullet... if the server's 'n' 
was 2, it would send a pure ACK for every other pure ACK it received. 
But, what if these pure ACKs from the server were also CE-marked? Then 
the client would feed back pure ACKs using its 'n'. This regression 
would eventually die out, but I don't like it.

This conversation about the second bullet presumes AccECN is being used 
for ACK congestion control. I'm not comfortable with writing anything 
specific about ACK congestion control into a draft on the standards 
track. But I don't want to preclude AccECN being used for ACK CC.


I believe the second bullet needs something changed to limit the ACKing 
of pure ACKs. However, before we put effort into wordsmithing, I'd like 
to hear whether you agree with the general idea of not defining what to 
do for ACK CC, but not precluding it.

Cheers




Bob

>
> Mirja
>
>
>
>> Thoughts anyone?
>>
>>
>> Bob
>>
>> --
>> ________________________________________________________________
>> Bob Briscoe
>> http://bobbriscoe.net/
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


From nobody Wed Nov 11 01:37:42 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 818C43A0FE2 for ; Wed, 11 Nov 2020 01:37:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VAKQSNDQJR_C for ; Wed, 11 Nov 2020 01:37:40 -0800 (PST)
Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0E3A43A0DC4 for ; Wed, 11 Nov 2020 01:37:40 -0800 (PST)
Received: by mail-qk1-x731.google.com with SMTP id v143so1096722qkb.2 for ; Wed, 11 Nov 2020 01:37:39 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZybdvdgbiNSNk5n4j78yCN4HNaQfetsRYsn45xT4w4o=; b=OWIZNgOy4IjCBnVqnj0+vLLqMcpx8XDdz/4O6vK1V403YYcyNMmgVeUl2ZXMKEgxTv CZbxKtk/2+Ky3F2u7jCQavvKYQI8S3Y8mr00zD9FKUraYiLjwADHXm6HcjmcduT/zkCa HQF8O8C64WXkS9t93pdVhft5RO2gmwwOm7lS4P2jFWioi8OimQWH3Kw0vfvFLpsFKwL1 +az2+wFEOyTQQyiAgi0NEyXDN4zEiXj/gGPSBL7Ou2DWAOA00Q0ydTiGNlIvxz0+a6JQ X4/b/fGQ+hMsuUpydLVMEQhORbgnHuH+G0qU4adIPWSBZlDyKRKGdscD6pYLC3OWLT3O czlA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZybdvdgbiNSNk5n4j78yCN4HNaQfetsRYsn45xT4w4o=; b=PsEM4t4JLuLNIPadOQaLdgzumNgEURdNSYf0LTBfxROQXYMZOw1/oO7ssDFSwwtaQA kWGRkvdGR51BWhf0g255kok+jmEaUJt+ALVtdVFdDpjfCq5vvtd3Ljk5RTW5GhQg2aUL 06epJwSZShPTpV0xQ8PzapGaUWUsTJUhLdNVnF7t4IIh3OcnBDBgWL7MgdAfXuPrnQ+L 6CKWY56wACgx9GxB7qdQkZuEsRs8YuIUbzHnQQFbTsJ7HTYAVXzYHARq9PX5cPHulgzf Fv/Sbz7gIB/aLnt/kKXT2eVXg3cuAR+f0/gzjC9+LI3b5AW/ecv7/3SuDXMiop7fU+nu tQ2A==
X-Gm-Message-State: AOAM531/L9bDSLfNMhNM4lob5SartyFhPO+IhKJ/JnMnD51DeSaWYOfF bhb4hBa3TsPoVqnsMhO4r7BZHnYlBAOlPI3L8RM=
X-Google-Smtp-Source: ABdhPJxDdReUvhgSMnqP8dsjEDUU9zDQm8BjAvH7VnfCq2YLLTxfAdIZmhu/af1a1NSNXoS+8i1+wC7MBcToL0WkRrA=
X-Received: by 2002:a05:620a:11a4:: with SMTP id c4mr14569637qkk.8.1605087458986;  Wed, 11 Nov 2020 01:37:38 -0800 (PST)
MIME-Version: 1.0
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>
In-Reply-To: <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>
From: Yoshifumi Nishida 
Date: Wed, 11 Nov 2020 01:37:27 -0800
Message-ID: 
To: Mirja Kuehlewind 
Cc: Bob Briscoe , tcpm IETF list ,  Richard Scheffenegger 
Content-Type: multipart/alternative; boundary="000000000000d38e5a05b3d18bbf"
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 11 Nov 2020 09:37:42 -0000

--000000000000d38e5a05b3d18bbf
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind 
wrote:

> Hi Bob,
>
> Please see below.
>
> > On 10. Nov 2020, at 01:31, Bob Briscoe  wrote:
> >
> > Mirja, Richard, Ilpo, tcpm list,
> >
> > I've just been reading through the accurate-ecn draft to double-check. =
I
> think there's a problem with the following text...
> >
> > 3.2.2.5.1.  Data Receiver Safety Procedures
> >
> >    An AccECN Data Receiver:
> >
> >    o  SHOULD immediately send an ACK whenever a data packet marked CE
> >       arrives after the previous [data] packet was not CE.
> >
> >    o  MUST immediately send an ACK once 'n' CE marks have arrived since
> >       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
> >       than 6.
> >
> > ...
> >    For the avoidance of doubt, the change-triggered ACK mechanism is
> >    deliberately worded to solely apply to data packets, and to ignore
> >    the arrival of a control packet with no payload, because it is
> >    important that TCP does not acknowledge pure ACKs.
> >
> >
> > In the first bullet, I think it doesn't matter whether the previous
> packet marked CE was a data packet or a pure ACK (i.e we should remove th=
e
> second occurrence of 'data' that I have put in [square brackets].
>
> I believe it does matter. If the previous packet was a pure ACK and was C=
E
> marked, you didn=E2=80=99t send an ACK and so you should immediately send=
 one now.
> If the previous packet was a data packet and CE marked, you don=E2=80=99t=
 need to
> send an immediate ACK because you already did this with the previous
> packet. However, it text might be a but ambiguous because what is meant i=
s
> =E2=80=9Cif the previous packet was a data packet and CE marked=E2=80=9D.=
 So this does not
> apply if we e.g. have a CE-marked data packets, then a pure ACK that is n=
ot
> CE marked, and then another CE-marked data packet.
>
> >
> > The second bullet doesn't consider the possibility that the 'n'th  CE
> mark might arrive on a pure ACK. Then, the wording as it stands says the
> Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a pu=
re
> ACK, but I'm not actually sure it does any harm to do so in this case (it
> cannot cause an infinite loop of ACKs). However, given it would be
> unorthodox, we maybe ought to rule it out by rewording anyway?
>

even though it won't cause an infinite loop, can they be dup acks? If so,
it doesn't look good to me especially when early retransmit is activated.


> I think we also can leave this as it is because if that =E2=80=99n=E2=80=
=99 packet is a
> pure ACK that still means that you have unacked data packets with CE mark=
s
> and you should trigger an ACK for those packet now (rather than waiting f=
or
> the delayed ACK timer to expire). However this case is less important. Al=
so
> we should probably make sure that this doesn=E2=80=99t apply if there are=
 only pure
> ACKs with CE marks, maybe by add =E2=80=9Cif unacknowledged data are outs=
tanding=E2=80=9D
> or something.
>
>
"can send a pure ack on a pure ack when the ack value advances"?
--
Yoshi

--000000000000d38e5a05b3d18bbf
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable




=

On Tue, Nov 10, 2020 at 2:08 AM Mirja=
 Kuehlewind <ie=
tf@kuehlewind.net> wrote:
Hi Bob,



Please see below.



> On 10. Nov 2020, at 01:31, Bob Briscoe <research@bobbriscoe.net> wrote:
> 

> Mirja, Richard, Ilpo, tcpm list,

> 

> I've just been reading through the accurate-ecn draft to double-ch=
eck. I think there's a problem with the following text...

> 

> 3.2.2.5.1.=C2=A0 Data Receiver Safety Procedures

> 

>=C2=A0 =C2=A0 An AccECN Data Receiver:

> 

>=C2=A0 =C2=A0 o=C2=A0 SHOULD immediately send an ACK whenever a data pa=
cket marked CE

>=C2=A0 =C2=A0 =C2=A0 =C2=A0arrives after the previous [data] packet was=
 not CE.

> 

>=C2=A0 =C2=A0 o=C2=A0 MUST immediately send an ACK once 'n' CE =
marks have arrived since

>=C2=A0 =C2=A0 =C2=A0 =C2=A0the previous ACK, where 'n' SHOULD b=
e 2 and MUST be no greater

>=C2=A0 =C2=A0 =C2=A0 =C2=A0than 6.

> 

> ...

>=C2=A0 =C2=A0 For the avoidance of doubt, the change-triggered ACK mech=
anism is

>=C2=A0 =C2=A0 deliberately worded to solely apply to data packets, and =
to ignore

>=C2=A0 =C2=A0 the arrival of a control packet with no payload, because =
it is

>=C2=A0 =C2=A0 important that TCP does not acknowledge pure ACKs.=C2=A0 =


> 

> 

> In the first bullet, I think it doesn't matter whether the previou=
s packet marked CE was a data packet or a pure ACK (i.e we should remove th=
e second occurrence of 'data' that I have put in [square brackets].=




I believe it does matter. If the previous packet was a pure ACK and was CE =
marked, you didn=E2=80=99t send an ACK and so you should immediately send o=
ne now. If the previous packet was a data packet and CE marked, you don=E2=
=80=99t need to send an immediate ACK because you already did this with the=
 previous packet. However, it text might be a but ambiguous because what is=
 meant is =E2=80=9Cif the previous packet was a data packet and CE marked=
=E2=80=9D. So this does not apply if we e.g. have a CE-marked data packets,=
 then a pure ACK that is not CE marked, and then another CE-marked data pac=
ket.=C2=A0 



> 

> The second bullet doesn't consider the possibility that the 'n=
'th=C2=A0 CE mark might arrive on a pure ACK. Then, the wording as it s=
tands says the Data Receiver MUST immediately ACK a pure ACK. I know TCP ne=
ver ACKs a pure ACK, but I'm not actually sure it does any harm to do s=
o in this case (it cannot cause an infinite loop of ACKs). However, given i=
t would be unorthodox, we maybe ought to rule it out by rewording anyway?

even though it won't cause an infini=
te=C2=A0loop, can they be dup acks? If so, it doesn't=C2=A0look good to=
 me especially when early retransmit is activated.
=C2=A0

I think we also can leave this as it is because if that =E2=80=99n=E2=80=99=
 packet is a pure ACK that still means that you have unacked data packets w=
ith CE marks and you should trigger an ACK for those packet now (rather tha=
n waiting for the delayed ACK timer to expire). However this case is less i=
mportant. Also we should probably make sure that this doesn=E2=80=99t apply=
 if there are only pure ACKs with CE marks, maybe by add =E2=80=9Cif unackn=
owledged data are outstanding=E2=80=9D or something.
=C2=A0
"can send a pure ack on a pure ack when the ack value advance=
s"?=C2=A0=C2=A0
--
Yoshi=C2=A0
=C2=A0


--000000000000d38e5a05b3d18bbf--


From nobody Wed Nov 11 15:18:14 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 16ED63A11E2 for ; Wed, 11 Nov 2020 15:18:13 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qdCor2gp-tvY for ; Wed, 11 Nov 2020 15:18:10 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 932183A11E1 for ; Wed, 11 Nov 2020 15:18:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=rnVSJ+MAq3lzBuKg0b3hlcElIRAe5tP8UP8wqhUITrg=; b=GH1YKvBYYoZ1dH5BIlKLIgVfq mHf2u+F0F1ufn2KX4sfuM0giJcQCIa7OgLYhOhKi5iuXE+fNOnUiLxLsxZRSOSOAzvG52TAt7g775 /NPV5/iq5JzV8A+HHIffWOO3tSZPpeETYTufOvxZ6l8QBemT38VRCYdSyAQiVfyTyPXg9Km2OcxFK vb8ZkUkxe08+dDlVWxk77+9KJD51Wv5efYuTjN32zbW5ttsoIT5ikj7SjPlBjjHEEvub85wCeAVvC /fVXTD9sx3/zTzzzAljTWnciz+kIsV74wxouyOmBSf2PkDIl5QD6lJwnRYwkayA7pNZCFrQdxBDmP c+7e9X1zw==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:52594 helo=[192.168.1.11]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kczNO-00G3EN-Dy; Wed, 11 Nov 2020 23:18:08 +0000
To: Yoshifumi Nishida , Mirja Kuehlewind 
Cc: tcpm IETF list , Richard Scheffenegger 
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net> 
From: Bob Briscoe 
Message-ID: 
Date: Wed, 11 Nov 2020 23:18:05 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------9E7A6EE86E3F269E27DD4191"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 11 Nov 2020 23:18:13 -0000

This is a multi-part message in MIME format.
--------------9E7A6EE86E3F269E27DD4191
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Yoshi,

On 11/11/2020 09:37, Yoshifumi Nishida wrote:
>
>
> On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind  > wrote:
>
>     Hi Bob,
>
>     Please see below.
>
>     > On 10. Nov 2020, at 01:31, Bob Briscoe      > wrote:
>     >
>     > Mirja, Richard, Ilpo, tcpm list,
>     >
>     > I've just been reading through the accurate-ecn draft to
>     double-check. I think there's a problem with the following text...
>     >
>     > 3.2.2.5.1.  Data Receiver Safety Procedures
>     >
>     >    An AccECN Data Receiver:
>     >
>     >    o  SHOULD immediately send an ACK whenever a data packet
>     marked CE
>     >       arrives after the previous [data] packet was not CE.
>     >
>     >    o  MUST immediately send an ACK once 'n' CE marks have
>     arrived since
>     >       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>     >       than 6.
>     >
>     > ...
>     >    For the avoidance of doubt, the change-triggered ACK mechanism is
>     >    deliberately worded to solely apply to data packets, and to
>     ignore
>     >    the arrival of a control packet with no payload, because it is
>     >    important that TCP does not acknowledge pure ACKs.
>     >
>     >
>     > In the first bullet, I think it doesn't matter whether the
>     previous packet marked CE was a data packet or a pure ACK (i.e we
>     should remove the second occurrence of 'data' that I have put in
>     [square brackets].
>
>     I believe it does matter. If the previous packet was a pure ACK
>     and was CE marked, you didn’t send an ACK and so you should
>     immediately send one now. If the previous packet was a data packet
>     and CE marked, you don’t need to send an immediate ACK because you
>     already did this with the previous packet. However, it text might
>     be a but ambiguous because what is meant is “if the previous
>     packet was a data packet and CE marked”. So this does not apply if
>     we e.g. have a CE-marked data packets, then a pure ACK that is not
>     CE marked, and then another CE-marked data packet.
>
>     >
>     > The second bullet doesn't consider the possibility that the
>     'n'th  CE mark might arrive on a pure ACK. Then, the wording as it
>     stands says the Data Receiver MUST immediately ACK a pure ACK. I
>     know TCP never ACKs a pure ACK, but I'm not actually sure it does
>     any harm to do so in this case (it cannot cause an infinite loop
>     of ACKs). However, given it would be unorthodox, we maybe ought to
>     rule it out by rewording anyway?
>
>
> even though it won't cause an infinite loop, can they be dup acks? If 
> so, it doesn't look good to me especially when early retransmit is 
> activated.

[BB] I'm not expert in TCP loss recovery but, surely if TCP A has 
nothing outstanding, it cannot consider receipt of dup acknowledgements 
from TCP B as a sign of loss? Nonetheless, there would be a chance that 
such a dup acknowledgement might be in flight when TCP A happened to 
have just sent some more data, and they crossed in flight.

To avoid this case, it would be possible for TCP B to send pure ACK(s) 
with SEG.SEQ = SND.NXT-1 (using the same trick as keepalive probes). 
Being out of window, each of these pure ACKs would elicit a further pure 
ACK from TCP A. However, that would acknowledge SND-NXT, so the loop 
would stop there. Nonetheless, if TCP B started sending data again, and 
if TCP B's data crossed the pure ACKs that it had recently elicited from 
TCP A, once these pure ACKs arrived at TCP B, I think it would treat 
them as DupACKs.

It's seems possible for TCP B to recognize that it elicited these 
DupACKs from TCP A, and to therefore work out that it could ignore them.

I've kept on digging into this particular obscure corner of ACK 
congestion control, only 'cos I wanted to see if there was a useful end 
to this hole (so it would be worth the AccECN RFC allowing others to dig 
there later, rather than ruling it out).

Nonethless, the AccECN draft itself doesn't have to go anywhere near 
this subject - it can leave all that for a potential future draft about 
using AccECN for ACK congestion control.

>     I think we also can leave this as it is because if that ’n’ packet
>     is a pure ACK that still means that you have unacked data packets
>     with CE marks and you should trigger an ACK for those packet now
>     (rather than waiting for the delayed ACK timer to expire). However
>     this case is less important. Also we should probably make sure
>     that this doesn’t apply if there are only pure ACKs with CE marks,
>     maybe by add “if unacknowledged data are outstanding” or something.
>
> "can send a pure ack on a pure ack when the ack value advances"?

[BB] Yes. That's a fairly safe thing to require.
I think the best strategy here is to require an immediate ACK in 
response to a data packet if there are at least 'n' CE marks to be 
reported. And to extend that requirement to immediately ACKing a pure 
ACK if the ACK value has advanced since the last ACK. But just to say 
nothing about ACKing pure ACKs in other cases, so we don't preclude a 
case that can be made safe in future (e.g. with something like the 
keepalive approach above).

If you (Yoshi), Mirja and others agree, let's try to work out the 
wording of this second bullet.


Bob


> --
> Yoshi

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------9E7A6EE86E3F269E27DD4191
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Yoshi,

    

    
On 11/11/2020 09:37, Yoshifumi Nishida
      wrote:

    

    

      
      

        


        

        

        

          
On Tue, Nov 10, 2020 at 2:08
            AM Mirja Kuehlewind <ietf@kuehlewind.net>
            wrote:

          

          
Hi Bob,

            

            Please see below.

            

            > On 10. Nov 2020, at 01:31, Bob Briscoe <research@bobbriscoe.net>
            wrote:

            > 

            > Mirja, Richard, Ilpo, tcpm list,

            > 

            > I've just been reading through the accurate-ecn draft
            to double-check. I think there's a problem with the
            following text...

            > 

            > 3.2.2.5.1.  Data Receiver Safety Procedures

            > 

            >    An AccECN Data Receiver:

            > 

            >    o  SHOULD immediately send an ACK whenever a data
            packet marked CE

            >       arrives after the previous [data] packet was not
            CE.

            > 

            >    o  MUST immediately send an ACK once 'n' CE marks
            have arrived since

            >       the previous ACK, where 'n' SHOULD be 2 and MUST
            be no greater

            >       than 6.

            > 

            > ...

            >    For the avoidance of doubt, the change-triggered ACK
            mechanism is

            >    deliberately worded to solely apply to data packets,
            and to ignore

            >    the arrival of a control packet with no payload,
            because it is

            >    important that TCP does not acknowledge pure ACKs. 
            

            > 

            > 

            > In the first bullet, I think it doesn't matter whether
            the previous packet marked CE was a data packet or a pure
            ACK (i.e we should remove the second occurrence of 'data'
            that I have put in [square brackets].

            

            I believe it does matter. If the previous packet was a pure
            ACK and was CE marked, you didn’t send an ACK and so you
            should immediately send one now. If the previous packet was
            a data packet and CE marked, you don’t need to send an
            immediate ACK because you already did this with the previous
            packet. However, it text might be a but ambiguous because
            what is meant is “if the previous packet was a data packet
            and CE marked”. So this does not apply if we e.g. have a
            CE-marked data packets, then a pure ACK that is not CE
            marked, and then another CE-marked data packet.  

            

            > 

            > The second bullet doesn't consider the possibility that
            the 'n'th  CE mark might arrive on a pure ACK. Then, the
            wording as it stands says the Data Receiver MUST immediately
            ACK a pure ACK. I know TCP never ACKs a pure ACK, but I'm
            not actually sure it does any harm to do so in this case (it
            cannot cause an infinite loop of ACKs). However, given it
            would be unorthodox, we maybe ought to rule it out by
            rewording anyway?

          

          


          

          
even though it won't cause an infinite loop, can they be
            dup acks? If so, it doesn't look good to me especially when
            early retransmit is activated.

        

      

    

    

    [BB] I'm not expert in TCP loss recovery but, surely if TCP A has
    nothing outstanding, it cannot consider receipt of dup
    acknowledgements from TCP B as a sign of loss? Nonetheless, there
    would be a chance that such a dup acknowledgement might be in flight
    when TCP A happened to have just sent some more data, and they
    crossed in flight. 

    

    To avoid this case, it would be possible for TCP B to send pure
    ACK(s) with SEG.SEQ = SND.NXT-1 (using the same trick as keepalive
    probes). Being out of window, each of these pure ACKs would elicit a
    further pure ACK from TCP A. However, that would acknowledge
    SND-NXT, so the loop would stop there. Nonetheless, if TCP B started
    sending data again, and if TCP B's data crossed the pure ACKs that
    it had recently elicited from TCP A, once these pure ACKs arrived at
    TCP B, I think it would treat them as DupACKs.

    

    It's seems possible for TCP B to recognize that it elicited these
    DupACKs from TCP A, and to therefore work out that it could ignore
    them. 

    

    I've kept on digging into this particular obscure corner of ACK
    congestion control, only 'cos I wanted to see if there was a useful
    end to this hole (so it would be worth the AccECN RFC allowing
    others to dig there later, rather than ruling it out).

    

    Nonethless, the AccECN draft itself doesn't have to go anywhere near
    this subject - it can leave all that for a potential future draft
    about using AccECN for ACK congestion control. 

    

    

      

        

          
 

          

            I think we also can leave this as it is because if that ’n’
            packet is a pure ACK that still means that you have unacked
            data packets with CE marks and you should trigger an ACK for
            those packet now (rather than waiting for the delayed ACK
            timer to expire). However this case is less important. Also
            we should probably make sure that this doesn’t apply if
            there are only pure ACKs with CE marks, maybe by add “if
            unacknowledged data are outstanding” or something.

             

          

          
"can send a pure ack on a pure ack when the ack value
            advances"?  

          

        

      

    

    

    [BB] Yes. That's a fairly safe thing to require.

    I think the best strategy here is to require an immediate ACK in
    response to a data packet if there are at least 'n' CE marks to be
    reported. And to extend that requirement to immediately ACKing a
    pure ACK if the ACK value has advanced since the last ACK. But just
    to say nothing about ACKing pure ACKs in other cases, so we don't
    preclude a case that can be made safe in future (e.g. with something
    like the keepalive approach above).

    

    If you (Yoshi), Mirja and others agree, let's try to work out the
    wording of this second bullet.

    

    

    Bob

    

    

    

      

        

          
--

          
Yoshi 

          
 

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------9E7A6EE86E3F269E27DD4191--


From nobody Wed Nov 11 18:41:04 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4ACF73A134A for ; Wed, 11 Nov 2020 18:41:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j-EmvbRgFcas for ; Wed, 11 Nov 2020 18:40:58 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 59D323A1348 for ; Wed, 11 Nov 2020 18:40:58 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:References:Cc:To:Subject:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=gWR3JQP9DoWSpgwfUlA0hPqVjxRKlNh+eSYib56yKTQ=; b=hubbgfr3RnepApkHAOghlLkpY hjZA2qYXrazxoFD089nOoBLlE0T9cqsk8AkCnc2n9DAxJjT7VBGNBe+thExs81ZoMT1znzQDPSU2K V+6qaRJcRU4ka/8f7rMslfhqBEfPpz6QdMHndoMuiOk7u7y+3Ue95AIV2ztgdpSLUe2Sy/6fhPcsZ Qn5hIxEuZMAH/5XHawU1+XVZgn2ZM4L2eC66c877+vDzaN7/p1jvxw/Uh67l9KJWhBBhuIYzSzRHb LhMw51EpfyyA86o1ejRw/5VaPdtCMz1TsldAxO9T0QmB4W+25FQwESN2XsVCFNlDpAsG90+IRPQ4j qVz09YPAQ==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:53324 helo=[192.168.1.11]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kd2Xd-00GJ5O-BO; Thu, 12 Nov 2020 02:40:56 +0000
From: Bob Briscoe 
To: Gorry Fairhurst 
Cc: tcpm IETF list 
References: <160388925181.18695.7892567372446756190@ietfa.amsl.com> <4017c549-ac6d-d633-6432-20a6a8a9a342@bobbriscoe.net> <3c8de57b23994824b6c51cf5d7fba7ec@hs-esslingen.de> <5dd8f210-2fe2-bd9b-5e69-4a87016f5416@bobbriscoe.net>     <286e7716-be59-a023-7786-21e8175a3eb3@bobbriscoe.net>  <3c7126bd-ebdc-00b1-8b46-79125eb525a1@bobbriscoe.net> <4d56be9b-06e0-6b2a-998c-23263b40e59d@erg.abdn.ac.uk>
Message-ID: <0bcf246f-3a38-42bf-2252-3fc2086c1131@bobbriscoe.net>
Date: Thu, 12 Nov 2020 02:40:52 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <4d56be9b-06e0-6b2a-998c-23263b40e59d@erg.abdn.ac.uk>
Content-Type: multipart/alternative; boundary="------------C9A351C92F8DE9587981A317"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-accurate-ecn-13.txt - ACK Thining
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 12 Nov 2020 02:41:03 -0000

This is a multi-part message in MIME format.
--------------C9A351C92F8DE9587981A317
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Gorry, [this thread is getting even longer, but I think we might soon 
start to violently agree.]
Inline tagged [BB2]...

On 10/11/2020 09:41, Gorry Fairhurst wrote:
>
> See in-line, marked [GF], and I will try to unpick why I don't agree 
> with some of this text.
>
> On 06/11/2020 21:51, Bob Briscoe wrote:
>> Gorry,
>>
>> Thanks as always for checking all this... inline tagged [BB]
>>
>> On 06/11/2020 14:01, Gorry Fairhurst wrote:
>>> I see this rev introduced a section on ACK Filtering, which I have 
>>> specific comments on. We need to consider the cost of sending ACKs - 
>>> especially anything that results in them being large - however, this 
>>> needs to be done by choosing a default at the receiver - not as an 
>>> adaptation, the adaption if necessary needs to be introduce more 
>>> ACKs when this is beneficial, as in some cases for DAAS, not in 
>>> trying to figure something from delayed or "missing" ACKs.
>>>
>>> 1) So let me say first that I think the suggestion for updating the 
>>> receiver to thin, and every 2-6 packets seems reasonable.
>>>
>>>     “MUST immediately send an ACK once 'n' CE marks have arrived since
>>>       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>>>       than 6.”
>>>
>>> I do agree that endpoints can do much more to reduce the return path 
>>> capacity that they consume - and that the endpoints have the 
>>> opportunity to encode the feedback information more efficiently than 
>>> can be done by a device (or queue) in the network. From our 
>>> experiments, n=10 would also often be likely OK.
>>
>> NB: n is the no. of *CE marks*, which is only the ACK thinning ratio 
>> in the worst case if there's 100% marking.
>> IOW, with n=6 the receiver could ACK less than 1 in 10 data packets 
>> whenever marking probability < 60%.
>>
>> Altho the receiver's ACK ratio will have to fall to 6 to 1 for the 
>> duration of a 100% marking sequence, whenever some data packets are 
>> not marked, the receiver aACK ratio can rise. This *could* allow any 
>> build-up of ACKs queued on the reverse path to relax, which could 
>> improve the chances of not overflowing into ACK loss.
>>
>> It's hard to predict when 100% marking in the forward direction will 
>> coincide with reverse path congestion, and I'm not saying it won't.
>> Just pointing out that n does not equal the ACK thinning ratio, it's 
>> the floor of it.
>>
>> n=6 is a consequence of only having a 3-bit field to count CE marks. 
>> We could push the envelope and make the max n=7. But not n=8, 
>> otherwise wrap after 8 marks would be indistinguishable from zero marks.
>>
> [GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6 only when 
> there is 100% marking - less with more likely patterns, that seems 
> reasonable.
>
> [GF] I also would agree that it is quite OK to increase the ACK-Ratio 
> (send more ACKs) in some specific conditions, known to endpoints.
>
> [GF] What concerns me, is that to avoid the need for ACK-Thining and 
> related methods within the network, the "normal" condition needs to be 
> for a rec eievr to send (far) fewer ACKs.
>

[BB2] "What concerns me is that, ... the normal condition needs to be 
for a receiver to send (far) fewer ACKs".
That's ambiguous to me... it might mean:
a) your concern is that the implication of what I'm saying will need far 
fewer ACKs (which you don't want).

But given what you say later, I'm pretty sure you meant:
b) your concern is that you want the normal condition to be fewer ACKs 
(i.e. you want the opposite of (a))


>>>
>>> 2. Although in some cases a receiver can predict a good "n", I do 
>>> not believe a receiver can typically find out how to choose “n” by 
>>> observing the ACKs on a connection.
>>> 2. I do not believe a receiver can find out how top to choose “n” by 
>>> observing the ACKs received on a connection.
>>
>> [BB] I think you intended to delete one of these sentences. Whatever, 
>> both of them seem to contain a typo - otherwise they are circular? 
>> Surely the receiver is sending the ACKs, so either sentence as it 
>> stands ends with "...I do not believe a receiver can typically decide 
>> how often to send an ACK by observing how often it is sending ACKs 
>> itself." ???
>>
> [GF] So, the topic and solution are a little complex ... let me try to 
> write a simpler statement: The receiver alone cannot predict a good 
> ACK ratio based on the ACKs the remote endpoint receives. ECN-marking 
> of an ACK stream does not provide useful informatiopn, because in this 
> specific case, the control packets are small, and we enter the realm 
> of packet-congestible links (as in RFC7141), where the overhead of 
> transmission time slot scheduling and fitting packets to burst 
> transmission plans can be more sigmnificant than the number of bits sent.
>

[BB2] If a link becomes packet-congested, ECN can (and should) still be 
used to indicate such congestion. The implementation has to be designed 
so that the ECN-marking process isn't compromised by packet processing 
overload. But that's perfectly possible for a good implementer.


>>>
>>> Sure, if the path is modelled as a simple bit-congestive path that 
>>> has a queue, then you could monitoring timing/loss of packets and 
>>> determine a rate that tries not to fill a queue. I would not know 
>>> how common such a bottleneck link is.
>>>
>>> I do know that many bottleneck links are more complex and needs to 
>>> consider the scheduling transmission of small ACK packets on the 
>>> return path performance. In many radio technologies, the overhead 
>>> from transmission of ACKs is proportionally much higher than for 
>>> data segments -  These layer 2s are constrained in terms of packets 
>>> (e.g. where lower layer framing and transmission scheduling is an 
>>> important consideration to the efficiency/cost of using the 
>>> service). This has often been “fixed” using ACK Thinning or PEPs. 
>>> Protocols that send fewer ACKs/data packet are not impacted.
>>
>> [BB] I agree that the receiver alone cannot predict a good 'n'.
>> AccECN is a not the complete solution, but we've grasped the 
>> opportunity to take a step in the right direction (within the 
>> chartered scope of AccECN). AccECN at least allows the data sender to 
>> measure the extent of congestion marking on the arriving ACK stream 
>> (if there's ECN support at the bottleneck). Whereas before AccECN, a 
>> random ECN mark on one pure ACK in a round couldn't be distinguished 
>> from 100% of ACKs marked.
>>
> [GF] We've relied for decades on ACK-Thinning (or equivalent) in the 
> network, where the link KNOWS what the sub-IP layer is doing, and 
> hence knows whether an ACK is eating into resources that are scarce; 
> but this solution is already problematic for encrypted traffic, and 
> when ACKs carry more than ACK information. So, back to my point: To 
> fix this, I think we need to reduce the level of feedback at the receiver.
>

[BB2] Certainly as we get more packets per RTT, we shouldn't need to 
have more ACKs per RTT. Better still, we ought to pack more information 
into each ACK.

However, do you have enough evidence to say reducing the absolute level 
of feedback is /all/ that is necessary? That's quite a strong assertion. 
It presumes that there is not a trade-off. It says, if we achieve some 
lower level of feedback then, in all the cases when the reverse path is 
uncongested, it would never be advantageous to increase the ACK frequency.


> [GF] So ... I see two problems with using ECN marks on the return path 
> to vary the ACK Ratio:
>
> (i) I'm not so sure the Layer 2 knows how to mark ACK packets - 
> because they are not bit-congestible signals, and  I'd prefer to see 
> ECN marking based on queue capacity (bit congestibale).
>

[BB2] I don't see why ECN is hard or inappropriate for 
packet-congestion. Is there any experience that says that? And I don't 
know why you'd prefer to see ECN for bit congestion. Is there some 
reasoning behind that?

> I could see methods that let the sub-IP resource allocation ballance 
> the use of forward and return transmssion opportunities and use this 
> to CE-mark both directions - and see if flows adapt their return path 
> resource consumption to optimise their forward data path transmission. 
> It could perform better if it were to seperate ACKs (or any short 
> packets? - also sometimes done) from "actual" data and then 
> pro-actively CE-mark ACK packets to free-up transmission resource for 
> other packets in the uplink or downlink. I'm not sure this cross-layer 
> coupling is wise.
>

[BB2] Why complicate matters? A link doesn't know which flow in one 
direction relates to which flow in the other (the reverse path of some 
flows might not even go through this link). And a link doesn't know the 
relative importance of ACKs and data, which is specific to different 
congestion controllers. So how can a link balance both directions well? 
What's wrong with the link blindly signalling congestion on packets 
(irrespective of whether data or ACKs), in each direction independently. 
And let the end systems find which degree of self-controlled 
ACK-thinning maximizes their performance.

What is important is to come to consensus on whether the link is or is 
not the best place to control reverse path congestion.  As above, there 
is scope for transports to just reduce the regular ACK load, full stop. 
Beyond that, to find a good trade-off between too much and too few ACKs, 
I certainly think we can agree that neither the transport nor the link 
can do that well on its own. Communication between them seems essential. 
Attempting to communicate from host to network seems like a non-starter, 
given the information the transport would have to convey would be quite 
complex. Congestion signalling from the link to the transport seems 
sufficient (to me). And using ECN seems likely to be significantly 
better than using loss.


> (ii) Before a new method is deployed, and the ACK Ratio remains 1:2, 
> then the network nodes still do either ACK-Thinning or queue the ACKs.
>

[BB2] As above, I'm sure there's scope for reducing ACK frequency 
without (significantly?) affecting performance. And you've been doing 
this research, so you should know. And I guess, maybe, that might make 
most link-based ACK filtering unnecessary. But I don't think we need to 
decide now that this will be enough, and ACK CC will not be necessary 
(see later).

>>>
>>> Scheduling (at the IP layer,can also often result in variations in 
>>> timing - something that is hard to predict and I would suggests make 
>>> automated selection of “n” very hard, and the need to thin is often 
>>> not directly visible to a user of the path. Traffic multiplexed with 
>>> protocols that use different ACK policies (perhaps QUIC?) also make 
>>> this sort of detection less reliable.
>>
>> [BB] Being hard shouldn't prevent us trying.
>> The remaining piece would be to advance a TCP ACK congestion control 
>> protocol on the experimental track (RFC5690 
>>  is informational).
>>
>
> [GF] To me, this is a complicated way to approach the question of how 
> much feedback is needed. I actually don't see the justification for 
> adding an extra control loop, that will interact with other control 
> loops in the network.
>

[BB2] Yes, I understand that you favour the transport picking a higher 
ACK ratio for all paths. But surely the empirical evidence from 
link-based ACK filtering shows that it adapts the ACK ratio to 
conditions, and doesn't just apply a higher ACK ratio all the time. But 
I guess you would argue that's because links are fighting with 
transports that weren't specifically designed to cope with a higher ACK 
ratio, so not filtering at all is better when it's not necessary.

>>>
>>> 3) A queue might be competing for scheduling opportunities against 
>>> other IP flows (e.g., WFQ), or competing for transmission 
>>> opportunities with a shared resource (common in radio links).
>>>
>>> ACKs can and do take transmission opportunities away from the 
>>> resource pools that would be used by other services that share the 
>>> same resource pool. This can be reduced by some form of per-flow 
>>> scheduling with deeper per-flow queues - but that in itself can 
>>> induce unwanted coupling between flows - limiting the peak rate of 
>>> one flow, to achieve a higher ACK ratio for another flow.
>>>
>>> Consider a QUIC flow sharing a return path with a TCP flow, the 
>>> higher rate of larger QUIC ACKs reduces the transmission 
>>> opportunities for TCP ACKs, which in turn suppresses the TCP flow, 
>>> limiting the overall system performance. If the ACKs use the same 
>>> transmission resource, they could take directly away from 
>>> transmission opportunities in the forward direction, but even if 
>>> they do not - the rate of ACKs retuned impacts the sending rate end 
>>> to end.

[BB2] A question:
Has the approach ever been tried where the link just does dumb 
congestion notification (without regard to type or size of packets), and 
the end system congestion controls optimize data transfer performance 
and the tradeoff with ACK ratio? Whether analytically, or empirically?

>>
>> [BB] This says two things to me:
>> * if TCP is going to introduce control over the ACK ratio, QUIC and 
>> other transports ought to as well {Note 1}.
>> * and all ACK CC algorithms ought to be designed to ensure some 
>> degree of equitable capacity sharing {Note 2}, including between the 
>> ACK stream of flow A and the data stream of flow B transferring in 
>> the opposite direction.
>>
>> {Note 1}: I don't think this needs any formal "all or nothing" rule 
>> where the transport ADs co-ordinate multiple WGs. Enough of the 
>> people involved are the same for the right thing to happen (I think). 
>> But some AD oversight wouldn't do any harm.
>> {Note 2}: There are numerous challenges to face, but I think it's 
>> paramount to first establish which node is considered responsible for 
>> solving the problem (the bottleneck or one of the transport 
>> endpoints). This doesn't imply that the IETF will ever ensure that 
>> other nodes don't 'interfere'. For instance, if the sender is deemed 
>> responsible for determining the level of ACK filtering, it will still 
>> have to make allowance for bottlenecks attempting to ACK thin as well.
>>
>>
>>>
>>> 4) The new section adds some cross-layer requirements on sub-IP 
>>> layers, which seem to imply more complexity. I’m not comfortable 
>>> with such a proposal.
>>>
>>> It seems that the text suggests not implement thinning and that 
>>> these systems implement logic to detect the presence of the option 
>>> by observing SYNs and then configure the link to do something special. 
>>
>> This was the proposal that I was most interested to hear your views 
>> on. We can remove it if the WG doesn't want it, but first I want to 
>> make sure the proposal is fully understood:
>> * The aim is for the RFC to tell bottleneck implementers (via this 
>> text) that *only in cases where AccECN has been negotiated* the 
>> preferred way to improve flow performance is for the link to signal 
>> ECN to the transport, not to drop the ACKs itself.
>
> [GF] In principle could be interesting - if this were a separate 
> signal for "ACKs" - we don't have this signal. We have a signal for 
> all IP traffic.
>

[BB2] I don't think the signal should be different for ACKs and data. 
That's unnecessarily complicated for the link, and makes the transport's 
job harder if different links each choose a different balance (it's hard 
enough even if all links use the same balance).

>> * If the link doesn't support ECN, it still does the ACK filtering itself
>> * Links aren't going to delegate their control unless transports 
>> actually act on the ECN signals that links emit - and improve 
>> performance better than links have been. So, it's important that 
>> transports take on these responsibilities - the data sender takes 
>> responsibility for determining the ACK rate, asking [I meant 
>> 'assuming'] its correspondent receiver does actually thin its own ACKs.
>> Here's who signals what, and who does what
>>
>>                link-.
>>                      \
>>                       '-[ECN CE]-.
>>                                   \
>>                                    '-> receiver
>>                                     .-
>>                                    /
>>             .-[(Acc)ECN feedback]-'
>>            /
>>  sender <-'
>>         -.
>>           \
>>            '---[ACK CC ratio]---.
>>                                  \
>>                                   '-> receiver(treats as polite 
>> request, not mandatory)
>>
>>
> [GF] That's a useful diagram to help understand!!! The control loops 
> for layer 2 channel sharing oftem operate on a similar timescale.
>
> [GF] Let's be sure we start from the same problem:
>
> * Current TCP Specs suggest the ACK Ratio is 1:2. I could say that 
> means that 33% of transmission opportunities are used solely for 
> feedback.
>

[BB2] Well, that depends how much aggregation is used to pack packets 
into each xmt opportunity. But your figures here and below are still 
meaningful relative to each other, even if aggregation shifts them all 
downwards in absolute terms.

> Before we had sophisticated transports, tha was fine, but nowadays 
> this is a very high proportion by modern standards, and given the way 
> the ACK-Clock no longer has to be for each segment or pair of segments 
> ... reality moved away from that when stretch-ACKs became common.
>
> * Many senders (e.g, using offload) might have an ACK Ratio of 1:4, 
> but not all.
>
> * Some paths "Thin" ... by how much depends.  In many cases, an ACK 
> Ratio 1:2, causes excessive resource usage that predjudices 
> performance, an ACK Ratio of 1:4 is still 20% of transmission 
> opportunities. An ACK-Ratio of 1:10, would mean 10% of transmission 
> opportunities are used for feedback (that's an order of magnitude less 
> control than data segments, and is a reasonable operating point). My 
> suggestion is that MOST of the time, transports can live with an ACK 
> Ratio 1:10, and in many cases this is enough to avoid ACK Thinning 
> measures being triggered.
>
> * A transport can adapt feedback, for sure,  and there will be cases 
> where lower rate (e.g. 1:1 or 1:2 initially) or higher rates 
> (1:10,1:20, etc as rates increase and rate-based methods are used) 
> will be benefit the transport - but changing the default operating 
> point from 1:2 to 1:10 is likely already significantly reducing the 
> need (and triggering) of ACK thinning.
>

[BB2] The point tho' is that the opportunity for ACK thinning has arisen 
as packet rates have increased, because you only really need a certain 
number of ACKs per RTT to maintain control. E.g if you need 8 ACKs per 
RTT then if the window is 16 you need ACK ratio = 2, but if the window 
is 320, ACK ratio = 40 should work OK. However, just because the average 
flow rate is continually increasing, doesn't mean flows don't still need 
to go slow before they go fast.

If a link enforces an ACK ratio, it doesn't know which flows are 
currently getting started, which flows have small RTTs, etc. etc. Flows 
know that.

> - So, maybe we should do this, rather than encourage a new generation 
> of smart ECN-based ACK modification schemes. If we mitigate the 
> feedback traffic, we can then focus on deploying ECN for capacity 
> detection.
>

[BB2] I'm fine starting out with transports doing more ACK thinning 
themselves.
But it's no use if links still filter the ACKs of these 'good' flows on 
top of the flows filtering themselves.

QUIC has encryption to prevent that (at least until networks find 
heuristics to do QUIC ACK thinning). TCP does not.

So it does not harm to require links not to filter AccECN-capable 
packets (perhaps we can widen that to all ECN-capable packets).
And it also does no harm to recommend that links ECN mark instead. If 
they did, they would of course ECN-mark all ECN-capable packets, of course.

Then:
* if your strategy of cutting down the absolute ACK ratio is sufficient, 
we don't need to introduce ACK CC, but we still have more ECN deployment 
anyway.
* And if your strategy isn't enough, we have the signals for the basis 
of robust ACK CC.

>>> I don’t think think this is the correct approach - the result will 
>>> be throttling of the connection speed. Is it necessary that the 
>>> sub-IP layer does not implement thinning for connections that use 
>>> AccECN feedback?
>>
>> Why do you say the connection speed would be throttled?
> [GF] For example: A sudden increasse in reverse path traffic will need 
> to be buffered until the radio transmission schedule changes. That 
> will delay feedback to the sender (advertised rwnd/cwnd for TCP), and 
> hence the forward rate will then throttle determined by the bottleneck 
> ACK rate.

[BB2] Well, actually the ACKs leaving the head of the buffer should be 
congestion-marked for the backlog behind them when they dequeue, as well 
as for their sojourn time (due to the buffer in front of them when they 
enqueued). I can point you at a paper on this if interested.
But, whatever, this is a better way to get the receiver to thin the ACK 
stream than the network discarding packets from it (better = less 
damaging to performance).

And, bear in mind that the first step is for transports to reduce ACK 
ratio overall, so we need a way for the link to distinguish the old 
transports that probably haven't reduced their ACK ratio, from newer 
ones that probably have.

Encouraging the link to ECN mark (all traffic incl ACKs) is worthwhile 
whether or not it's used for ACK CC.

And we only resort to ACK CC if necessary - but the signals are there if 
we need to develop the algorithms.

>> Did the text not explain the proposal well?
>
> Well, I think the problem is complex, which is why this email is 
> already long.
>
>> Or did you understand the proposal but disagree with it?
>>
>>
>>>
>>> When the text says: “SHOULD preserve the timing of each ACK”
>>> - does that imply that nodes need to discard rather than delay 
>>> packets when link scheduling or transmission scheduling is used. 
>>> This seems a very odd requirement, it might be better too simply 
>>> strip-out the AccECN option or discard all packets that set this.
>>
>> Sorry, that may be clumsy wording. It just means "the link SHOULD NOT 
>> filter ACKs" (if it supports ECN marking).
>>>
>>> I can’t see how queuing the ACKs, or deleting ACKs rather than 
>>> queuing them would improve the performance of AccECN TCP connections.
>>
>> May be the piece where the link's preferred role is to signal to the 
>> transport to slow down the ACK rate using ECN wasn't clear.
>>

Bob


>>
>> Bob
>>
>>>
>>> Gorry
>>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>
> Summary ... because this thread has lots of "discussion" or a 
> complicated interaction, and you asked about the AccECN draft text:
>
> + I agree that AccECN can be used with a sender that uses a more 
> appropiate target ACK Ratio - that is good! I agree this is worth saying.
>
> + ACK Thining, and variants - including uses of compression - is a 
> deployed mitigation, and that changing the ACK Ratio can eliminate 
> this practice and the need to implement this.
>
> + I agree that if the path is a simple bottleneck, then ECN-marking 
> should be encouraged. However, not appropiate without much more 
> consideration for a more complex L2 where the marking is 
> packet-congestible. A link predomonantly carrying small ACKs seems to 
> lead into the research issues in RFC7141.
>
> + I would disagree with any encouragement to use cross-layer methods 
> with use ECN-marking to seek to adjust ACK feedback traffic and I am 
> concerned that using ECN to modify the ACK Ratio might create new 
> layers of feedback loop that will aggrevate the problem.
>
> I think the current text goes to far.
>
> Gorry
>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/


--------------C9A351C92F8DE9587981A317
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Gorry, [this thread is getting even longer, but I think we might
    soon start to violently agree.] 

    Inline tagged [BB2]...

    

    
On 10/11/2020 09:41, Gorry Fairhurst
      wrote:

    

    

      
      


      

      
See in-line, marked [GF], and I will
        try to unpick why I don't agree with some of this text.

      

      


      

      
On 06/11/2020 21:51, Bob Briscoe
        wrote:

      

      

        
        Gorry,

        

        Thanks as always for checking all this... inline tagged [BB]

        

        
On 06/11/2020 14:01, Gorry
          Fairhurst wrote:

        

        
I
          see this rev introduced a section on ACK Filtering, which I
          have specific comments on. We need to consider the cost of
          sending ACKs - especially anything that results in them being
          large - however, this needs to be done by choosing a default
          at the receiver - not as an adaptation, the adaption if
          necessary needs to be introduce more ACKs when this is
          beneficial, as in some cases for DAAS, not in trying to figure
          something from delayed or "missing" ACKs. 

          

          1) So let me say first that I think the suggestion for
          updating the receiver to thin, and every 2-6 packets seems
          reasonable. 

          

              “MUST immediately send an ACK once 'n' CE marks have
          arrived since 

                the previous ACK, where 'n' SHOULD be 2 and MUST be no
          greater 

                than 6.” 

          

          I do agree that endpoints can do much more to reduce the
          return path capacity that they consume - and that the
          endpoints have the opportunity to encode the feedback
          information more efficiently than can be done by a device (or
          queue) in the network. From our experiments, n=10 would also
          often be likely OK. 

        

        

        NB: n is the no. of *CE marks*, which is only the ACK thinning
        ratio in the worst case if there's 100% marking. 

        IOW, with n=6 the receiver could ACK less than 1 in 10 data
        packets whenever marking probability < 60%.

        

        Altho the receiver's ACK ratio will have to fall to 6 to 1 for
        the duration of a 100% marking sequence, whenever some data
        packets are not marked, the receiver aACK ratio can rise. This
        *could* allow any build-up of ACKs queued on the reverse path to
        relax, which could improve the chances of not overflowing into
        ACK loss.

        

        It's hard to predict when 100% marking in the forward direction
        will coincide with reverse path congestion, and I'm not saying
        it won't.

        Just pointing out that n does not equal the ACK thinning ratio,
        it's the floor of it.

        

        n=6 is a consequence of only having a 3-bit field to count CE
        marks. We could push the envelope and make the max n=7. But not
        n=8, otherwise wrap after 8 marks would be indistinguishable
        from zero marks.

        

      

      
[GF] Right, so the 3-bit field, limits the ACK-Ratio tobe 6
        only when there is 100% marking - less with more likely
        patterns, that seems reasonable. 

      

      
[GF] I also would agree that it is quite OK to increase the
        ACK-Ratio (send more ACKs) in some specific conditions, known to
        endpoints.

      

      
[GF] What concerns me, is that to avoid the need for
        ACK-Thining and related methods within the network, the "normal"
        condition needs to be for a rec eievr to send (far) fewer ACKs.

      

    

    

    [BB2] "What concerns me is that, ... the normal condition needs to
    be for a receiver to send (far) fewer ACKs".

    That's ambiguous to me... it might mean:

    a) your concern is that the implication of what I'm saying will need
    far fewer ACKs (which you don't want).

    

    But given what you say later, I'm pretty sure you meant:

    b) your concern is that you want the normal condition to be fewer
    ACKs (i.e. you want the opposite of (a))

    

    

    

      
 

      

        

          

          2. Although in some cases a receiver can predict a good "n", I
          do not believe a receiver can typically find out how to choose
          “n” by observing the ACKs on a connection. 

        

        

          2. I do not believe a receiver can find out how top to choose
          “n” by observing the ACKs received on a connection. 

        

        

        [BB] I think you intended to delete one of these sentences.
        Whatever, both of them seem to contain a typo - otherwise they
        are circular? Surely the receiver is sending the ACKs, so either
        sentence as it stands ends with "...I do not believe a receiver
        can typically decide how often to send an ACK by observing how
        often it is sending ACKs itself." ???

        

      

      
[GF] So, the topic and solution are a little complex ... let me
        try to write a simpler statement: The receiver alone cannot
        predict a good ACK ratio based on the ACKs the remote endpoint
        receives. ECN-marking of an ACK stream does not provide useful
        informatiopn, because in this specific case, the control packets
        are small, and we enter the realm of packet-congestible links
        (as in RFC7141), where the overhead of transmission time slot
        scheduling and fitting packets to burst transmission plans can
        be more sigmnificant than the number of bits sent.

      

    

    

    [BB2] If a link becomes packet-congested, ECN can (and should) still
    be used to indicate such congestion. The implementation has to be
    designed so that the ECN-marking process isn't compromised by packet
    processing overload. But that's perfectly possible for a good
    implementer.

    

    

    

      
 

      

        

          

          Sure, if the path is modelled as a simple bit-congestive path
          that has a queue, then you could monitoring timing/loss of
          packets and determine a rate that tries not to fill a queue. I
          would not know how common such a bottleneck link is. 

          

          I do know that many bottleneck links are more complex and
          needs to consider the scheduling transmission of small ACK
          packets on the return path performance. In many radio
          technologies, the overhead from transmission of ACKs is
          proportionally much higher than for data segments -  These
          layer 2s are constrained in terms of packets (e.g. where lower
          layer framing and transmission scheduling is an important
          consideration to the efficiency/cost of using the service).
          This has often been “fixed” using ACK Thinning or PEPs.
          Protocols that send fewer ACKs/data packet are not impacted. 

        

        

        

        

        [BB] I agree that the receiver alone cannot predict a good 'n'.
        

        AccECN is a not the complete solution, but we've grasped the
        opportunity to take a step in the right direction (within the
        chartered scope of AccECN). AccECN at least allows the data
        sender to measure the extent of congestion marking on the
        arriving ACK stream (if there's ECN support at the bottleneck).
        Whereas before AccECN, a random ECN mark on one pure ACK in a
        round couldn't be distinguished from 100% of ACKs marked.

        

      

      
[GF] We've relied for decades on ACK-Thinning (or equivalent)
        in the network, where the link KNOWS what the sub-IP layer is
        doing, and hence knows whether an ACK is eating into resources
        that are scarce; but this solution is already problematic for
        encrypted traffic, and when ACKs carry more than ACK
        information. So, back to my point: To fix this, I think we need
        to reduce the level of feedback at the receiver.

      

    

    

    [BB2] Certainly as we get more packets per RTT, we shouldn't need to
    have more ACKs per RTT. Better still, we ought to pack more
    information into each ACK.

    

    However, do you have enough evidence to say reducing the absolute
    level of feedback is /all/ that is necessary? That's quite a strong
    assertion. It presumes that there is not a trade-off. It says, if we
    achieve some lower level of feedback then, in all the cases when the
    reverse path is uncongested, it would never be advantageous to
    increase the ACK frequency. 

    

    

    

      
 

      
[GF] So ... I see two problems with using ECN marks on the
        return path to vary the ACK Ratio:

      
(i) I'm not so sure the Layer 2 knows how to mark ACK packets -
        because they are not bit-congestible signals, and  I'd prefer to
        see ECN marking based on queue capacity (bit congestibale). 

    

    

    [BB2] I don't see why ECN is hard or inappropriate for
    packet-congestion. Is there any experience that says that? And I
    don't know why you'd prefer to see ECN for bit congestion. Is there
    some reasoning behind that?

    

    

      
I could see methods that let the sub-IP resource allocation
        ballance the use of forward and return transmssion opportunities
        and use this to CE-mark both directions - and see if flows adapt
        their return path resource consumption to optimise their forward
        data path transmission. It could perform better if it were to
        seperate ACKs (or any short packets? - also sometimes done) from
        "actual" data and then pro-actively CE-mark ACK packets to
        free-up transmission resource for other packets in the uplink or
        downlink. I'm not sure this cross-layer coupling is wise.  

      

    

    

    [BB2] Why complicate matters? A link doesn't know which flow in one
    direction relates to which flow in the other (the reverse path of
    some flows might not even go through this link). And a link doesn't
    know the relative importance of ACKs and data, which is specific to
    different congestion controllers. So how can a link balance both
    directions well? What's wrong with the link blindly signalling
    congestion on packets (irrespective of whether data or ACKs), in
    each direction independently. And let the end systems find which
    degree of self-controlled ACK-thinning maximizes their performance.
    

    

    What is important is to come to consensus on whether the link is or
    is not the best place to control reverse path congestion.  As above,
    there is scope for transports to just reduce the regular ACK load,
    full stop. Beyond that, to find a good trade-off between too much
    and too few ACKs, I certainly think we can agree that neither the
    transport nor the link can do that well on its own. Communication
    between them seems essential. Attempting to communicate from host to
    network seems like a non-starter, given the information the
    transport would have to convey would be quite complex. Congestion
    signalling from the link to the transport seems sufficient (to me).
    And using ECN seems likely to be significantly better than using
    loss.

    

    

    

      
 

      
(ii) Before a new method is deployed, and the ACK Ratio remains
        1:2, then the network nodes still do either ACK-Thinning or
        queue the ACKs.

      

    

    

    [BB2] As above, I'm sure there's scope for reducing ACK frequency
    without (significantly?) affecting performance. And you've been
    doing this research, so you should know. And I guess, maybe, that
    might make most link-based ACK filtering unnecessary. But I don't
    think we need to decide now that this will be enough, and ACK CC
    will not be necessary (see later).

    

    

      
 

      

        

          

          Scheduling (at the IP layer,can also often result in
          variations in timing - something that is hard to predict and I
          would suggests make automated selection of “n” very hard, and
          the need to thin is often not directly visible to a user of
          the path. Traffic multiplexed with protocols that use
          different ACK policies (perhaps QUIC?) also make this sort of
          detection less reliable. 

        

        

        [BB] Being hard shouldn't prevent us trying.

        The remaining piece would be to advance a TCP ACK congestion
        control protocol on the experimental track (RFC5690 is informational).

        

      

      

      
 [GF] To me, this is a complicated way to approach the question
        of how much feedback is needed. I actually don't see the
        justification for adding an extra control loop, that will
        interact with other control loops in the network. 

      

    

    

    [BB2] Yes, I understand that you favour the transport picking a
    higher ACK ratio for all paths. But surely the empirical evidence
    from link-based ACK filtering shows that it adapts the ACK ratio to
    conditions, and doesn't just apply a higher ACK ratio all the time.
    But I guess you would argue that's because links are fighting with
    transports that weren't specifically designed to cope with a higher
    ACK ratio, so not filtering at all is better when it's not
    necessary.

    

    

      
 

      

        

          

          3) A queue might be competing for scheduling opportunities
          against other IP flows (e.g., WFQ), or competing for
          transmission opportunities with a shared resource (common in
          radio links). 

          

          ACKs can and do take transmission opportunities away from the
          resource pools that would be used by other services that share
          the same resource pool. This can be reduced by some form of
          per-flow scheduling with deeper per-flow queues - but that in
          itself can induce unwanted coupling between flows - limiting
          the peak rate of one flow, to achieve a higher ACK ratio for
          another flow. 

          

          Consider a QUIC flow sharing a return path with a TCP flow,
          the higher rate of larger QUIC ACKs reduces the transmission
          opportunities for TCP ACKs, which in turn suppresses the TCP
          flow, limiting the overall system performance. If the ACKs use
          the same transmission resource, they could take directly away
          from transmission opportunities in the forward direction, but
          even if they do not - the rate of ACKs retuned impacts the
          sending rate end to end. 

        

      

    

    

    [BB2] A question:

    Has the approach ever been tried where the link just does dumb
    congestion notification (without regard to type or size of packets),
    and the end system congestion controls optimize data transfer
    performance and the tradeoff with ACK ratio? Whether analytically,
    or empirically?

    

    

      

        

        

        

        [BB] This says two things to me: 

        * if TCP is going to introduce control over the ACK ratio, QUIC
        and other transports ought to as well {Note 1}. 

        * and all ACK CC algorithms ought to be designed to ensure some
        degree of equitable capacity sharing {Note 2}, including between
        the ACK stream of flow A and the data stream of flow B
        transferring in the opposite direction.

        

        {Note 1}: I don't think this needs any formal "all or nothing"
        rule where the transport ADs co-ordinate multiple WGs. Enough of
        the people involved are the same for the right thing to happen
        (I think). But some AD oversight wouldn't do any harm.

        {Note 2}: There are numerous challenges to face, but I think
        it's paramount to first establish which node is considered
        responsible for solving the problem (the bottleneck or one of
        the transport endpoints). This doesn't imply that the IETF will
        ever ensure that other nodes don't 'interfere'. For instance, if
        the sender is deemed responsible for determining the level of
        ACK filtering, it will still have to make allowance for
        bottlenecks attempting to ACK thin as well.

        

      

      

        

        

          

          4) The new section adds some cross-layer requirements on
          sub-IP layers, which seem to imply more complexity. I’m not
          comfortable with such a proposal. 

          

          It seems that the text suggests not implement thinning and
          that these systems implement logic to detect the presence of
          the option by observing SYNs and then configure the link to do
          something special. 

        

        This was the proposal that I was most interested to hear your
        views on. We can remove it if the WG doesn't want it, but first
        I want to make sure the proposal is fully understood:

        * The aim is for the RFC to tell bottleneck implementers (via
        this text) that *only in cases where AccECN has been negotiated*
        the preferred way to improve flow performance is for the link to
        signal ECN to the transport, not to drop the ACKs itself. 

      

      
[GF] In principle could be interesting - if this were a
        separate signal for "ACKs" - we don't have this signal. We have
        a signal for all IP traffic.

      

    

    

    [BB2] I don't think the signal should be different for ACKs and
    data. That's unnecessarily complicated for the link, and makes the
    transport's job harder if different links each choose a different
    balance (it's hard enough even if all links use the same balance).

    

    

      
 

      

        * If the link doesn't support ECN, it still does the ACK
        filtering itself

        * Links aren't going to delegate their control unless transports
        actually act on the ECN signals that links emit - and improve
        performance better than links have been. So, it's important that
        transports take on these responsibilities - the data sender
        takes responsibility for determining the ACK rate, asking [I
        meant 'assuming'] its correspondent receiver does actually thin
        its own ACKs.

      

      

        Here's who signals what, and who does what

        

                       link-.

                               \

                                '-[ECN CE]-.

                                            \

                                             '-> receiver

                                              .-

                                             /

                      .-[(Acc)ECN feedback]-'

                     /

           sender <-'

                  -.

                    \

                     '---[ACK CC ratio]---.

                                           \

                                            '-> receiver
          (treats as polite request, not mandatory)

        

        

      

      
[GF] That's a useful diagram to help understand!!! The control
        loops for layer 2 channel sharing oftem operate on a similar
        timescale.

      
[GF] Let's be sure we start from the same problem: 

      

      
* Current TCP Specs suggest the ACK Ratio is 1:2. I could say
        that means that 33% of transmission opportunities are used
        solely for feedback. 

    

    

    [BB2] Well, that depends how much aggregation is used to pack
    packets into each xmt opportunity. But your figures here and below
    are still meaningful relative to each other, even if aggregation
    shifts them all downwards in absolute terms.

    

    

      
Before we had sophisticated transports, tha was fine, but
        nowadays this is a very high proportion by modern standards, and
        given the way the ACK-Clock no longer has to be for each segment
        or pair of segments ... reality moved away from that when
        stretch-ACKs became common. 

      

      
* Many senders (e.g, using offload) might have an ACK Ratio of
        1:4, but not all. 

      

      
* Some paths "Thin" ... by how much depends.  In many cases, an
        ACK Ratio 1:2, causes excessive resource usage that predjudices
        performance, an ACK Ratio of 1:4 is still 20% of transmission
        opportunities. An ACK-Ratio of 1:10, would mean 10% of
        transmission opportunities are used for feedback (that's an
        order of magnitude less control than data segments, and is a
        reasonable operating point). My suggestion is that MOST of the
        time, transports can live with an ACK Ratio 1:10, and in many
        cases this is enough to avoid ACK Thinning measures being
        triggered. 

      

      
* A transport can adapt feedback, for sure,  and there will be
        cases where lower rate (e.g. 1:1 or 1:2 initially) or higher
        rates (1:10,1:20, etc as rates increase and rate-based methods
        are used) will be benefit the transport - but changing the
        default operating point from 1:2 to 1:10 is likely already
        significantly reducing the need (and triggering) of ACK
        thinning.

    

    

    [BB2] The point tho' is that the opportunity for ACK thinning has
    arisen as packet rates have increased, because you only really need
    a certain number of ACKs per RTT to maintain control. E.g if you
    need 8 ACKs per RTT then if the window is 16 you need ACK ratio = 2,
    but if the window is 320, ACK ratio = 40 should work OK. However,
    just because the average flow rate is continually increasing,
    doesn't mean flows don't still need to go slow before they go fast.
    

    

    If a link enforces an ACK ratio, it doesn't know which flows are
    currently getting started, which flows have small RTTs, etc. etc.
    Flows know that.

    

    

      
- So, maybe we should do this, rather than encourage a new
        generation of smart ECN-based ACK modification schemes. If we
        mitigate the feedback traffic, we can then focus on deploying
        ECN for capacity detection.

      

    

    

    [BB2] I'm fine starting out with transports doing more ACK thinning
    themselves. 

    But it's no use if links still filter the ACKs of these 'good' flows
    on top of the flows filtering themselves. 

    

    QUIC has encryption to prevent that (at least until networks find
    heuristics to do QUIC ACK thinning). TCP does not. 

    

    So it does not harm to require links not to filter AccECN-capable
    packets (perhaps we can widen that to all ECN-capable packets). 

    And it also does no harm to recommend that links ECN mark instead.
    If they did, they would of course ECN-mark all ECN-capable packets,
    of course. 

    

    Then: 

    * if your strategy of cutting down the absolute ACK ratio is
    sufficient, we don't need to introduce ACK CC, but we still have
    more ECN deployment anyway. 

    * And if your strategy isn't enough, we have the signals for the
    basis of robust ACK CC.

    

    

      
 

      

        
I
          don’t think think this is the correct approach - the result
          will be throttling of the connection speed. Is it necessary
          that the sub-IP layer does not implement thinning for
          connections that use AccECN feedback? 

        

        

        Why do you say the connection speed would be throttled? 

      

      [GF] For example: A sudden increasse in reverse path traffic will
      need to be buffered until the radio transmission schedule changes.
      That will delay feedback to the sender (advertised rwnd/cwnd for
      TCP), and hence the forward rate will then throttle determined by
      the bottleneck ACK rate.

    

    

    [BB2] Well, actually the ACKs leaving the head of the buffer should
    be congestion-marked for the backlog behind them when they dequeue,
    as well as for their sojourn time (due to the buffer in front of
    them when they enqueued). I can point you at a paper on this if
    interested. 

    But, whatever, this is a better way to get the receiver to thin the
    ACK stream than the network discarding packets from it (better =
    less damaging to performance).

    

    And, bear in mind that the first step is for transports to reduce
    ACK ratio overall, so we need a way for the link to distinguish the
    old transports that probably haven't reduced their ACK ratio, from
    newer ones that probably have.

    

    Encouraging the link to ECN mark (all traffic incl ACKs) is
    worthwhile whether or not it's used for ACK CC.

    

    And we only resort to ACK CC if necessary - but the signals are
    there if we need to develop the algorithms.

    

    

      

        Did the text not explain the proposal well?

      

      
Well, I think the problem is complex, which is why this email
        is already long.

      

      

        Or did you understand the proposal but disagree with it?

        

      

      

        

        

          

          When the text says: “SHOULD preserve the timing of each ACK” 

          - does that imply that nodes need to discard rather than delay
          packets when link scheduling or transmission scheduling is
          used. This seems a very odd requirement, it might be better
          too simply strip-out the AccECN option or discard all packets
          that set this. 

        

        

        Sorry, that may be clumsy wording. It just means "the link
        SHOULD NOT filter ACKs" (if it supports ECN marking).

      

      

        

          

          I can’t see how queuing the ACKs, or deleting ACKs rather than
          queuing them would improve the performance of AccECN TCP
          connections. 

        

        

        May be the piece where the link's preferred role is to signal to
        the transport to slow down the ACK rate using ECN wasn't clear.

        

      

    

    

    Bob

    

    

    

      

      

      

        

        Bob

        

        

          

          Gorry 

          

        

        

        
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

      

      
Summary ... because this thread has lots of "discussion" or a
        complicated interaction, and you asked about the AccECN draft
        text: 

      

      
+ I agree that AccECN can be used with a sender that uses a
        more appropiate target ACK Ratio - that is good! I agree this is
        worth saying.

      

      
+ ACK Thining, and variants - including uses of compression -
        is a deployed mitigation, and that changing the ACK Ratio can
        eliminate this practice and the need to implement this.

      

      
+ I agree that if the path is a simple bottleneck, then
        ECN-marking should be encouraged. However, not appropiate
        without much more consideration for a more complex L2 where the
        marking is packet-congestible. A link predomonantly carrying
        small ACKs seems to lead into the research issues in RFC7141.

      

      
+ I would disagree with any encouragement to use cross-layer
        methods with use ECN-marking to seek to adjust ACK feedback
        traffic and I am concerned that using ECN to modify the ACK
        Ratio might create new layers of feedback loop that will
        aggrevate the problem. 

      

      
I think the current text goes to far.

      

      
Gorry

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------C9A351C92F8DE9587981A317--


From nobody Thu Nov 12 14:42:04 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 36F553A0E8F for ; Thu, 12 Nov 2020 14:42:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cqQ4m-uAmjLm for ; Thu, 12 Nov 2020 14:42:01 -0800 (PST)
Received: from mail-qk1-x736.google.com (mail-qk1-x736.google.com [IPv6:2607:f8b0:4864:20::736]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1D69B3A0E8E for ; Thu, 12 Nov 2020 14:42:01 -0800 (PST)
Received: by mail-qk1-x736.google.com with SMTP id 11so7152982qkd.5 for ; Thu, 12 Nov 2020 14:42:01 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cm/RDn+iyXC6UlL1dScw+6u5DbZN29R4KaEA5+0EhXI=; b=WPt7X8k0x+BSlBdKGYYXke0wo4B2kxYrBkSTi1sbWwqzXOg9MwgEMPVlG/mfvJOloa CkNBRsfbQ9ULZAMul78/rx2ES+n4YNRjnBle8nsqvBIDqwsk2Y+nABcRa3D9lolx33F+ lH/c2ORsaQmPXiOJXYt1UFCkx03V+s8vN7ByixV1OlfAjOL41b/Hf77cS0+iCZBDB/MA S+23XsXcYUkqHhkNny8GtI0Y2yqoig+Qd8iT48vZcLacOAqF8v1K6llFb1TJ4e+J37SV ybBnWSpnbtyHtmur1cjwzJCKCDwQc9xueKCKqjuTwr5Dr3gIuN7ulRlJw1mXAhnZV7m+ hz/w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cm/RDn+iyXC6UlL1dScw+6u5DbZN29R4KaEA5+0EhXI=; b=LcP/lQoyoInYpaQxc4Xrq/daskxOnBUNU0OKt1Ubvy3dOhJb3knqUWTsdqAIm6PYZj Vu6k2ym9tqH6ZRQrC6j8Z0GQoEmWRYR7U1OdnoMkIVO0EJSZ034CcnMEy6cda7D9mGLE RQuO/Elis6MPcMtkjxwUHH1l9cn0lD9Ns+ox6q3x40IeN6I+glrKiLLMLgmJgk+ED6T6 j8xghO9rda0g0kF3clv9GkvkBPkhiLnWg83oXeGsgMA3PL39joURFMXt+cCx4n4i6LCU fEViJB425PxrkRUcc5CyL5Crw0RIj8sHdeAiVH3HFXQqt3BPqIpxKkxCxzqpvM/fn9vw yWPw==
X-Gm-Message-State: AOAM532p4So6uVLrYbSCrUWRz7UfUyc+w4rD7Y9a+irYO295AYdWe1LO YrTwHfUPuiGq7lf1e/VruTpy0x23HF0T4AYA2Jo=
X-Google-Smtp-Source: ABdhPJzT9qsY6PIZD6+rPC/M+adCKlQTyhokB+tWdPGVDZ2pPDhP7sFxOgVEUrYWYLEv0teDME5nmAfRYmDRo3tLnRI=
X-Received: by 2002:a37:87c5:: with SMTP id j188mr2275647qkd.476.1605220920178;  Thu, 12 Nov 2020 14:42:00 -0800 (PST)
MIME-Version: 1.0
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>  
In-Reply-To: 
From: Yoshifumi Nishida 
Date: Thu, 12 Nov 2020 14:41:49 -0800
Message-ID: 
To: Bob Briscoe 
Cc: Mirja Kuehlewind , tcpm IETF list ,  Richard Scheffenegger 
Content-Type: multipart/alternative; boundary="000000000000bbba6605b3f09ed0"
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 12 Nov 2020 22:42:03 -0000

--000000000000bbba6605b3f09ed0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Bob,

On Wed, Nov 11, 2020 at 3:18 PM Bob Briscoe  wrote=
:

> Yoshi,
>
> On 11/11/2020 09:37, Yoshifumi Nishida wrote:
>
>
>
> On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind 
> wrote:
>
>> Hi Bob,
>>
>> Please see below.
>>
>> > On 10. Nov 2020, at 01:31, Bob Briscoe  wrote=
:
>> >
>> > Mirja, Richard, Ilpo, tcpm list,
>> >
>> > I've just been reading through the accurate-ecn draft to double-check.
>> I think there's a problem with the following text...
>> >
>> > 3.2.2.5.1.  Data Receiver Safety Procedures
>> >
>> >    An AccECN Data Receiver:
>> >
>> >    o  SHOULD immediately send an ACK whenever a data packet marked CE
>> >       arrives after the previous [data] packet was not CE.
>> >
>> >    o  MUST immediately send an ACK once 'n' CE marks have arrived sinc=
e
>> >       the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>> >       than 6.
>> >
>> > ...
>> >    For the avoidance of doubt, the change-triggered ACK mechanism is
>> >    deliberately worded to solely apply to data packets, and to ignore
>> >    the arrival of a control packet with no payload, because it is
>> >    important that TCP does not acknowledge pure ACKs.
>> >
>> >
>> > In the first bullet, I think it doesn't matter whether the previous
>> packet marked CE was a data packet or a pure ACK (i.e we should remove t=
he
>> second occurrence of 'data' that I have put in [square brackets].
>>
>> I believe it does matter. If the previous packet was a pure ACK and was
>> CE marked, you didn=E2=80=99t send an ACK and so you should immediately =
send one
>> now. If the previous packet was a data packet and CE marked, you don=E2=
=80=99t need
>> to send an immediate ACK because you already did this with the previous
>> packet. However, it text might be a but ambiguous because what is meant =
is
>> =E2=80=9Cif the previous packet was a data packet and CE marked=E2=80=9D=
. So this does not
>> apply if we e.g. have a CE-marked data packets, then a pure ACK that is =
not
>> CE marked, and then another CE-marked data packet.
>>
>> >
>> > The second bullet doesn't consider the possibility that the 'n'th  CE
>> mark might arrive on a pure ACK. Then, the wording as it stands says the
>> Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a p=
ure
>> ACK, but I'm not actually sure it does any harm to do so in this case (i=
t
>> cannot cause an infinite loop of ACKs). However, given it would be
>> unorthodox, we maybe ought to rule it out by rewording anyway?
>>
>
> even though it won't cause an infinite loop, can they be dup acks? If so,
> it doesn't look good to me especially when early retransmit is activated.
>
>
> [BB] I'm not expert in TCP loss recovery but, surely if TCP A has nothing
> outstanding, it cannot consider receipt of dup acknowledgements from TCP =
B
> as a sign of loss? Nonetheless, there would be a chance that such a dup
> acknowledgement might be in flight when TCP A happened to have just sent
> some more data, and they crossed in flight.
>
> To avoid this case, it would be possible for TCP B to send pure ACK(s)
> with SEG.SEQ =3D SND.NXT-1 (using the same trick as keepalive probes). Be=
ing
> out of window, each of these pure ACKs would elicit a further pure ACK fr=
om
> TCP A. However, that would acknowledge SND-NXT, so the loop would stop
> there. Nonetheless, if TCP B started sending data again, and if TCP B's
> data crossed the pure ACKs that it had recently elicited from TCP A, once
> these pure ACKs arrived at TCP B, I think it would treat them as DupACKs.
>
> It's seems possible for TCP B to recognize that it elicited these DupACKs
> from TCP A, and to therefore work out that it could ignore them.
>
> I've kept on digging into this particular obscure corner of ACK congestio=
n
> control, only 'cos I wanted to see if there was a useful end to this hole
> (so it would be worth the AccECN RFC allowing others to dig there later,
> rather than ruling it out).
>
> Nonethless, the AccECN draft itself doesn't have to go anywhere near this
> subject - it can leave all that for a potential future draft about using
> AccECN for ACK congestion control.
>

I personally prefer not to specify very detailed logic for ACK CC here
unless it is crucial for the doc.
Since the doc is aiming to be a PS, I think it would be better to avoid
risks for nasty corner cases caused by some tricky techniques.

> I think we also can leave this as it is because if that =E2=80=99n=E2=80=
=99 packet is a
>> pure ACK that still means that you have unacked data packets with CE mar=
ks
>> and you should trigger an ACK for those packet now (rather than waiting =
for
>> the delayed ACK timer to expire). However this case is less important. A=
lso
>> we should probably make sure that this doesn=E2=80=99t apply if there ar=
e only pure
>> ACKs with CE marks, maybe by add =E2=80=9Cif unacknowledged data are out=
standing=E2=80=9D
>> or something.
>>
>>
> "can send a pure ack on a pure ack when the ack value advances"?
>
>
> [BB] Yes. That's a fairly safe thing to require.
> I think the best strategy here is to require an immediate ACK in response
> to a data packet if there are at least 'n' CE marks to be reported. And t=
o
> extend that requirement to immediately ACKing a pure ACK if the ACK value
> has advanced since the last ACK. But just to say nothing about ACKing pur=
e
> ACKs in other cases, so we don't preclude a case that can be made safe in
> future (e.g. with something like the keepalive approach above).
>
> If you (Yoshi), Mirja and others agree, let's try to work out the wording
> of this second bullet.
>

Works for me. If you could have proposed texts, I will take a look.
--
Yoshi

--000000000000bbba6605b3f09ed0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Hi Bob,

On Wed, Nov 11, 2020 at 3:18 PM Bob Briscoe &=
lt;research@bobbriscoe.net&g=
t; wrote:

 =20
   =20
 =20
  

    Yoshi,

    

    
On 11/11/2020 09:37, Yoshifumi Nishida
      wrote:

    

    

     =20
      

        


        

        

        

          
On Tue, Nov 10, 2020 at 2:0=
8
            AM Mirja Kuehlewind <ietf@kuehlewind.net>
            wrote:

          

          
Hi Bob,

            

            Please see below.

            

            > On 10. Nov 2020, at 01:31, Bob Briscoe <research@bobbriscoe.net&g=
t;
            wrote:

            > 

            > Mirja, Richard, Ilpo, tcpm list,

            > 

            > I've just been reading through the accurate-ecn draft
            to double-check. I think there's a problem with the
            following text...

            > 

            > 3.2.2.5.1.=C2=A0 Data Receiver Safety Procedures

            > 

            >=C2=A0 =C2=A0 An AccECN Data Receiver:

            > 

            >=C2=A0 =C2=A0 o=C2=A0 SHOULD immediately send an ACK whenev=
er a data
            packet marked CE

            >=C2=A0 =C2=A0 =C2=A0 =C2=A0arrives after the previous [data=
] packet was not
            CE.

            > 

            >=C2=A0 =C2=A0 o=C2=A0 MUST immediately send an ACK once =
9;n' CE marks
            have arrived since

            >=C2=A0 =C2=A0 =C2=A0 =C2=A0the previous ACK, where 'n&#=
39; SHOULD be 2 and MUST
            be no greater

            >=C2=A0 =C2=A0 =C2=A0 =C2=A0than 6.

            > 

            > ...

            >=C2=A0 =C2=A0 For the avoidance of doubt, the change-trigge=
red ACK
            mechanism is

            >=C2=A0 =C2=A0 deliberately worded to solely apply to data p=
ackets,
            and to ignore

            >=C2=A0 =C2=A0 the arrival of a control packet with no paylo=
ad,
            because it is

            >=C2=A0 =C2=A0 important that TCP does not acknowledge pure =
ACKs.=C2=A0
            

            > 

            > 

            > In the first bullet, I think it doesn't matter whether
            the previous packet marked CE was a data packet or a pure
            ACK (i.e we should remove the second occurrence of 'data=
9;
            that I have put in [square brackets].

            

            I believe it does matter. If the previous packet was a pure
            ACK and was CE marked, you didn=E2=80=99t send an ACK and so yo=
u
            should immediately send one now. If the previous packet was
            a data packet and CE marked, you don=E2=80=99t need to send an
            immediate ACK because you already did this with the previous
            packet. However, it text might be a but ambiguous because
            what is meant is =E2=80=9Cif the previous packet was a data pac=
ket
            and CE marked=E2=80=9D. So this does not apply if we e.g. have =
a
            CE-marked data packets, then a pure ACK that is not CE
            marked, and then another CE-marked data packet.=C2=A0 

            

            > 

            > The second bullet doesn't consider the possibility tha=
t
            the 'n'th=C2=A0 CE mark might arrive on a pure ACK. The=
n, the
            wording as it stands says the Data Receiver MUST immediately
            ACK a pure ACK. I know TCP never ACKs a pure ACK, but I'm
            not actually sure it does any harm to do so in this case (it
            cannot cause an infinite loop of ACKs). However, given it
            would be unorthodox, we maybe ought to rule it out by
            rewording anyway?

          

          


          

          
even though it won't cause an infinite=C2=A0loop, can th=
ey be
            dup acks? If so, it doesn't=C2=A0look good to me especially=
 when
            early retransmit is activated.

        

      

    

    

    [BB] I'm not expert in TCP loss recovery but, surely if TCP A has
    nothing outstanding, it cannot consider receipt of dup
    acknowledgements from TCP B as a sign of loss? Nonetheless, there
    would be a chance that such a dup acknowledgement might be in flight
    when TCP A happened to have just sent some more data, and they
    crossed in flight. 

    

    To avoid this case, it would be possible for TCP B to send pure
    ACK(s) with SEG.SEQ =3D SND.NXT-1 (using the same trick as keepalive
    probes). Being out of window, each of these pure ACKs would elicit a
    further pure ACK from TCP A. However, that would acknowledge
    SND-NXT, so the loop would stop there. Nonetheless, if TCP B started
    sending data again, and if TCP B's data crossed the pure ACKs that
    it had recently elicited from TCP A, once these pure ACKs arrived at
    TCP B, I think it would treat them as DupACKs.

    

    It's seems possible for TCP B to recognize that it elicited these
    DupACKs from TCP A, and to therefore work out that it could ignore
    them. 

    

    I've kept on digging into this particular obscure corner of ACK
    congestion control, only 'cos I wanted to see if there was a useful
    end to this hole (so it would be worth the AccECN RFC allowing
    others to dig there later, rather than ruling it out).

    

    Nonethless, the AccECN draft itself doesn't have to go anywhere nea=
r
    this subject - it can leave all that for a potential future draft
    about using AccECN for ACK congestion control. 
<=
div>
I personally prefer not to specify very detailed logic f=
or ACK CC here unless it is crucial for the doc.=C2=A0
Since the=
=C2=A0doc is aiming to be a PS, I think it would be better to avoid risks=
=C2=A0for nasty corner cases caused by some tricky techniques.=C2=A0
<=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex">

      

        

         =20
          

            I think we also can leave this as it is because if that =E2=80=
=99n=E2=80=99
            packet is a pure ACK that still means that you have unacked
            data packets with CE marks and you should trigger an ACK for
            those packet now (rather than waiting for the delayed ACK
            timer to expire). However this case is less important. Also
            we should probably make sure that this doesn=E2=80=99t apply if
            there are only pure ACKs with CE marks, maybe by add =E2=80=9Ci=
f
            unacknowledged data are outstanding=E2=80=9D or something.

            =C2=A0

          

          
"can send a pure ack on a pure ack when the ack value
            advances"?=C2=A0 

          

        

      

    

    

    [BB] Yes. That's a fairly safe thing to require.

    I think the best strategy here is to require an immediate ACK in
    response to a data packet if there are at least 'n' CE marks to=
 be
    reported. And to extend that requirement to immediately ACKing a
    pure ACK if the ACK value has advanced since the last ACK. But just
    to say nothing about ACKing pure ACKs in other cases, so we don't
    preclude a case that can be made safe in future (e.g. with something
    like the keepalive approach above).

    

    If you (Yoshi), Mirja and others agree, let's try to work out the
    wording of this second bullet.=C2=A0

Works for me. If you could have proposed texts, I will take a look.=
=C2=A0
--
Yoshi


--000000000000bbba6605b3f09ed0--


From nobody Fri Nov 13 04:25:59 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B0F973A0808; Fri, 13 Nov 2020 04:25:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=eggert.org
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t06WBm9nGB9U; Fri, 13 Nov 2020 04:25:56 -0800 (PST)
Received: from mail.eggert.org (mail.eggert.org [91.190.195.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5B3E73A07DF; Fri, 13 Nov 2020 04:25:56 -0800 (PST)
Received: from [IPv6:2a00:ac00:4000:400:8c50:9d35:a73e:dc29] (unknown [IPv6:2a00:ac00:4000:400:8c50:9d35:a73e:dc29]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.eggert.org (Postfix) with ESMTPSA id 31FBE607D82; Fri, 13 Nov 2020 14:25:45 +0200 (EET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eggert.org; s=dkim; t=1605270345; bh=JqKGWo3nugRivgmW3JfUjPf5tAk6m3OuYEF5OATWZZg=; h=From:Subject:Date:Cc:To; b=keWkGkAKEG82MCkubrmuySUDjtTmmGfwa+nzmt5S1vHT+rXMQR7e9RHqsa5g+oYZM kDgJM1Mm9nCccPNSUv4zDyOXX6TIb5dlyFxP/bNOjyAvX9qhCGD7vWbeZ7zjJhQ0nj PQillaTSVphCmXFTvWmCyOceZWAytzvIUTl08yaA=
From: Lars Eggert 
Content-Type: multipart/signed; boundary="Apple-Mail=_15FAE733-600A-4757-92B9-8CC986433E7D"; protocol="application/pgp-signature"; micalg=pgp-sha512
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
Message-Id: 
Date: Fri, 13 Nov 2020 14:25:44 +0200
Cc: sangtae.ha@colorado.edu, L X 
To: tcpm IETF list , QUIC WG 
X-MailScanner-ID: 31FBE607D82.A322E
X-MailScanner: Found to be clean
X-MailScanner-From: lars@eggert.org
Archived-At: 
Subject: [tcpm] Started work on RFC8312bis
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 13 Nov 2020 12:25:58 -0000

--Apple-Mail=_15FAE733-600A-4757-92B9-8CC986433E7D
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hi,

since there are apparently tweaks that need to be made to the CUBIC RFC, =
e.g., based on what's changed in the Linux implementation since =
publication, we started a repo for this work at =
https://github.com/NTAP/rfc8312bis.

We're planning to submit an individual -00 after submissions reopen =
after IETF-109. That initial submission will have only editorial changes =
compared to RFC8312 (due to conversion to Markdown, boilerplate changes, =
etc.)

I'd like to follow the QUIC approach for developing this bis, i.e., if =
you'd like to suggest changes, please open issues in the repo above. For =
those that get traction, we'll develop and discuss concrete text changes =
in pull requests (PRs) in that same repo.

Thanks,
Lars

--Apple-Mail=_15FAE733-600A-4757-92B9-8CC986433E7D
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEmpq0ZpSoejRmyhheVLXDCb9wwVcFAl+ue0gACgkQVLXDCb9w
wVdbGg/8CBb3/qAjEO3ZtRnz2YOUWYex9KcoixlbqEZ8D/G5du0eGLPxFJMrMrsn
nrfbDYJFFC04znAdcfmPY47Lou/vParAadw7nz8KHpbupnZY0NfhjAK+gfgMQ2Qp
pXaSFFCwlLdMqZWiUyiogeggp2RWzajyfyb1szIrPnDjin0yKVajz8JeTlQJI1Yo
LNRwUPddOPIoosv2nAYrajF8IOVHqHMiY0ZH3OLc+h3KURRY38S2gs2Ly/NRiBL4
5AzLuwTuFOY3OEx9PdetVRQn9ezKBeIR010trQcgL7ak2klZ/rtpFWm7in/NS/vr
Yhdg930n5552NPFOXP1ekOkXqQRRZGW2L+p63RdsEujTGic2PFsKWUgDra9+rwPc
p+9Lq0jR2sZ9f095sk/81RdnVRNbhmelVFCMD5oXTI/p6fdGqMWYFaavF1DXiFbi
iTm7hq1GpfaFsEo5kOcoVjdTcBx2CeLY4jF478iEa60oqRYIvhZ3dcBejO98X1gp
A0tMpaj1PwC4m3dcN3zW05t3gA9wFTBTjLLbTU6CHAEIPf9XPPODss1vj/BXIOey
yvIEYgwmxfcbWv+HbSoFUQXR+z95WWuBDBaeALF8KxwJnn4TJR3RyIgFXWJC2oBI
H/ON2e1d91hnQ6kGYXaI1TJHEUqbchm1cQ6pM40kGA7FlIKQnRc=
=sxyF
-----END PGP SIGNATURE-----

--Apple-Mail=_15FAE733-600A-4757-92B9-8CC986433E7D--


From nobody Sat Nov 14 04:13:04 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C41493A156F for ; Sat, 14 Nov 2020 04:13:02 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tT_4FFrFq6Om for ; Sat, 14 Nov 2020 04:13:01 -0800 (PST)
Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E77923A0E61 for ; Sat, 14 Nov 2020 04:13:00 -0800 (PST)
Received: by mail-qt1-x82c.google.com with SMTP id 3so9302735qtx.3 for ; Sat, 14 Nov 2020 04:13:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=s2x4/pSBqKS5eLF/cndhmqCnulHgb8TRoQLsJmZ3rWE=; b=oi+alOTCfHG7FCZq6j6CkMsRE7BG6X6PclpmuKc6B+eaQgn6vn7Y8P2vHVAUX8BpQG 6QTM82vS78K/StUctwHJ+Hv05qSeeO2u8tHX5tkFhpvu/MNj/tqxRoSFWe6XwN+Ti26N Jb6st8WLHjx15G5A+ipWA2GppAluhBc8Wbp/0btGkKk7uF5A4a9YK3fs9yhb6y7X0RIf jKn1wSz71hUcaXgf1joq4y/S267/cqqQry1AQ9WRKAkR+vkHnJTYFZDqAK88+U4mw25E JfgUAMLdOObl74SFx6iwH94MkFIBu3wcJmSxCxlz7p9MDtOOh9EgK5kIR5cX2LKRYEmX ql+g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=s2x4/pSBqKS5eLF/cndhmqCnulHgb8TRoQLsJmZ3rWE=; b=HbItnaBmv4H45vkFT/RHwRfMmGf/1HPz6iQT/X8dyHHA8+xvDKdQC7WAsbpI57ILFZ BclFi5xNYGhwTwawbAnBj3fXSKcNjqgV/5tbZQvvVqHwd9A6D7eg5G9/3LpWBtux9oij IoOpyF0pXqEVFfxZYeRkSCbS2L5KzaLjBR6NB8DDQZwh8DYDSdLrRneKoWUyM+9mieEJ LT9oJY0YojAI6QpJ35JnBi8HzlMQCgWbh6cRvXsJ3sICZS/XE41c7Y4F6fvSw2TouEU8 /hIeRFpIRLa/qKgVfV+6KRKZHwEKaJDSLjlen+hYdA8RQPLgcCsBVPhsukcvWrJmCoPH vmZQ==
X-Gm-Message-State: AOAM532DCatpB77qzUZIgRyHKwGl30AU3UZYAGzoRS7m2z4MEUHneIte bwZIjVcxjYDX2rACzLXZ4JDWlyRjkCEtnbExo+g=
X-Google-Smtp-Source: ABdhPJwIUk+xH2XLdsGOSpTqpzCiqMyIIgGIRT754UFhYOnUmhQCXJiZatectdzX9ji/mLtK0YU9kjDljaFIUSbeRyo=
X-Received: by 2002:ac8:5c15:: with SMTP id i21mr6121759qti.190.1605355979927;  Sat, 14 Nov 2020 04:12:59 -0800 (PST)
MIME-Version: 1.0
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com> 
In-Reply-To: 
From: Yoshifumi Nishida 
Date: Sat, 14 Nov 2020 04:12:48 -0800
Message-ID: 
To: Yuchung Cheng 
Cc: "tcpm@ietf.org Extensions" , Eric Dumazet , iccrg IRTF list ,  Kevin Yang 
Content-Type: multipart/alternative; boundary="000000000000ebf4e805b41010a6"
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 14 Nov 2020 12:13:03 -0000

--000000000000ebf4e805b41010a6
Content-Type: text/plain; charset="UTF-8"

Hi,
Thanks for preparing the draft. I basically like the idea in the docs and
have several comments.

1: I am wondering if we really need to have 32 bits TSval and TSecr in ETS
option in SYN when TSopt is also sent?
    For example, If TSopt has 1 ms granularity, I think ETS option will
need to carry only small fractions of timestamp.

2: "To prevent a false positive PAWS rejection of a valid segment, an ETS
receiver MUST skip the PAWS check"
    -> This is one possible approach, but just skipping PAWS doesn't sound
very good to me.
         How about discarding the segment while updating TS.recent as an
alternative? If the segment is not an old one, it will be retransmitted and
accepted.
         I think this is a bit more conservative.

3: It is not very clear for me how measuring NetworkRTT can contribute to
improving RTO.
    Or, are there some other ways to utilize NetworkRTT?

4: Just out of curiosity, is intended status standard track or
experimental? ExID is usually used for exp or info docs.

Thanks,
--
Yoshi

On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng  wrote:

> Hi tcpm & iccrg,
>
> We are proposing a new TCP timestamp option to measure the network delay
> more precisely (e.g. excluding delayed ACK effects) for congestion control,
> e.g. Swift published SIGCOMM 2020. The precision of the measurements can
> potentially be further enhanced by NIC hardware timestamps. We are working
> on a reference implementation for Linux as well.
>
> We'll also present it in the upcoming tcpm meeting. Feedbacks are very
> welcome.
>
> ---------- Forwarded message ---------
> From: 
> Date: Mon, Nov 2, 2020 at 3:29 PM
> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
> To: Eric Dumazet , Yuchung Cheng ,
> Neal Cardwell , Kevin Yang (Yudong) 
>
>
>
> A new version of I-D, draft-yang-tcpm-ets-00.txt
> has been successfully submitted by Kevin (Yudong) Yang and posted to the
> IETF repository.
>
> Name:           draft-yang-tcpm-ets
> Revision:       00
> Title:          TCP ETS: Extensible Timestamp Options
> Document date:  2020-11-02
> Group:          Individual Submission
> Pages:          13
> URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets
> Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00
>
>
> Abstract:
>    This document presents ETS: an Extensible TimeStamps option for TCP.
>    It allows hosts to use microseconds as the unit for timestamps to
>    improve the precision of timestamps, and advertise the maximum ACK
>    delay for its own delayed ACK mechanism.  Furthermore, it extends the
>    information provided in the [RFC7323] TCP Timestamps Option by
>    including the receiver delay in the TSecr echoing, so that the
>    receiver of the ACK is able to more accurately estimate the portion
>    of the RTT that resulted from time traveling through the network.
>    The ETS option format is extensible, so that future extensions can
>    add further information without the overhead of extra TCP option kind
>    and length fields.
>
>
>
>
> Please note that it may take a couple of minutes from the time of
> submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--000000000000ebf4e805b41010a6
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Hi,
Thanks for preparing the draft. I basic=
ally like the idea in the docs and have several comments.

1: I am wondering if we really need to have 32 bits TSval and TSecr=
 in ETS option in SYN when TSopt is also sent?=C2=A0
=C2=A0 =C2=
=A0 For example, If TSopt has 1 ms granularity, I think ETS option will nee=
d to carry only small=C2=A0fractions of timestamp.=C2=A0

2: "To prevent a false positive PAWS rejection of a valid segme=
nt, an ETS receiver MUST skip the PAWS check"
=C2=A0 =C2=A0 =
-> This is one possible approach, but just skipping PAWS doesn't sou=
nd very good to me.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0How about d=
iscarding the segment while updating TS.recent as an alternative? If the se=
gment is not an old one, it will be retransmitted and accepted.
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I think this is a bit more conservative.<=
/div>

3: It is not very clear for me how measuring Netwo=
rkRTT can contribute to improving RTO.=C2=A0
=C2=A0 =C2=A0 Or, ar=
e there some other ways to utilize NetworkRTT?=C2=A0

4: Just out of curiosity, is intended=C2=A0status standard track or expe=
rimental? ExID is usually used for exp or info docs.

Thanks,
--
Yoshi

On Mon, Nov 2, 2020 at 4:26 PM Yuchu=
ng Cheng <ycheng=3D40goog=
le.com@dmarc.ietf.org> wrote:
Hi tcpm & iccrg,
<=
br>
We are proposing a new TCP timestamp option to measure the ne=
twork delay more precisely (e.g. excluding delayed ACK effects) for congest=
ion control, e.g. Swift published SIGCOMM 2020. The precision of the measur=
ements can potentially be further enhanced by NIC hardware timestamps. We a=
re working on a reference implementation for Linux as well.

<=
/div>
We'll also present it in the upcoming tcpm meeting. Feedbacks=
 are very welcome.=C2=A0

---------- Forwarded message ---------
From: <internet-drafts@ietf.org>
Date: Mon, Nov 2, 2020 at=
 3:29 PM
Subject: New Version Notification for draft-yang-tcpm-ets-00.tx=
t
To: Eric Dumazet <edumazet@google.com>, Yuchung Cheng <ycheng@google.com>, Neal Cardwell=
 <ncardwell@go=
ogle.com>, Kevin Yang (Yudong) <yyd@google.com>




A new version of I-D, draft-yang-tcpm-ets-00.txt

has been successfully submitted by Kevin (Yudong) Yang and posted to the
IETF repository.



Name:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0draft-yang-tcpm-ets

Revision:=C2=A0 =C2=A0 =C2=A0 =C2=A000

Title:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TCP ETS: Extensible Timestamp Opti=
ons

Document date:=C2=A0 2020-11-02

Group:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Individual Submission

Pages:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 13

URL:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt

Status:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https:=
//datatracker.ietf.org/doc/draft-yang-tcpm-ets/

Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https:/=
/datatracker.ietf.org/doc/html/draft-yang-tcpm-ets

Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https://tools.=
ietf.org/html/draft-yang-tcpm-ets-00





Abstract:

=C2=A0 =C2=A0This document presents ETS: an Extensible TimeStamps option fo=
r TCP.

=C2=A0 =C2=A0It allows hosts to use microseconds as the unit for timestamps=
 to

=C2=A0 =C2=A0improve the precision of timestamps, and advertise the maximum=
 ACK

=C2=A0 =C2=A0delay for its own delayed ACK mechanism.=C2=A0 Furthermore, it=
 extends the

=C2=A0 =C2=A0information provided in the [RFC7323] TCP Timestamps Option by=


=C2=A0 =C2=A0including the receiver delay in the TSecr echoing, so that the=


=C2=A0 =C2=A0receiver of the ACK is able to more accurately estimate the po=
rtion

=C2=A0 =C2=A0of the RTT that resulted from time traveling through the netwo=
rk.

=C2=A0 =C2=A0The ETS option format is extensible, so that future extensions=
 can

=C2=A0 =C2=A0add further information without the overhead of extra TCP opti=
on kind

=C2=A0 =C2=A0and length fields.









Please note that it may take a couple of minutes from the time of submissio=
n

until the htmlized version and diff are available at tools.ietf.org.



The IETF Secretariat







_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--000000000000ebf4e805b41010a6--


From nobody Sat Nov 14 08:45:22 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D20A93A0D43 for ; Sat, 14 Nov 2020 08:45:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.599
X-Spam-Level: 
X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d0o5TpIqMIoB for ; Sat, 14 Nov 2020 08:45:15 -0800 (PST)
Received: from mail-yb1-xb35.google.com (mail-yb1-xb35.google.com [IPv6:2607:f8b0:4864:20::b35]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 61DB93A0D44 for ; Sat, 14 Nov 2020 08:45:15 -0800 (PST)
Received: by mail-yb1-xb35.google.com with SMTP id l14so7574360ybq.3 for ; Sat, 14 Nov 2020 08:45:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BCV00OCfiPoleY0fkZBiDHU/qwP4CvyKsCILxZ4QTwg=; b=UGnuYdjYIiV3ZaSPpewmNPr6lBRLzOIiCkolEKVqklfgXmrtBWGuhp0Bh/etY3/NGD x6PQKb6HSCrsBDlMC/SMUwzcvgXob7c1F5+s1+rX+7cR5WbPx/0/dsgJip20gi7lm426 VKsdHD9GJAjkUmgXDHzbLmoHZBv3ezKJqbiqSjS5tj+HV+DwLzhZv7v1JnL4fQCf4HEg +Ih65Z0lUwiMwena3gX27sftEtRcDvqWSrCrv9SB7ibz3WGOSFt/+pxtysWMA4KF5G7J uvZjs84KYw3aoHGjXkrx24I2GsvRovtdWz4kgybm1qkDG587rogxYnOB8MmzmXWj0mRr 4lkA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BCV00OCfiPoleY0fkZBiDHU/qwP4CvyKsCILxZ4QTwg=; b=lH096Qnaf33Eo1F6KLi2CJuwvaGf+M88eXkrV0QVTlrcRVV+lFWOVwygCSheRbjkhk Gc0qhUs5EbcGy5ka2SzN3tN+jt/Qntv6wF5ZrFRAzqWPXC/bHEuwafWSDPV46lyBQ4yr qm+RStZytbsnRZQ6PG5Zwn95Vshb8PjrcdbseKqCIR/nLDGhhMXdz6BZ4Yq0PSpun2MN vkGGfmBshmXeK/HgvaYrWS4QCmR76VkB7qX18TLZY9QqbwYaFRfrr3D43Z/HvvVVoV2e 3x4RFv6bodElUQuobK1iXjNhT+vEr9vGftHXt/8+rMrGV9y8JgopmXpfd53S6D997t2F YD8w==
X-Gm-Message-State: AOAM532CQWAXLYLW73tMmUTxxoc6hTKXGRX/bJG/yQ/c9Q2BlKFlj3Bn ajEz8GBCvCkT06EGXLAYW7JpPUl7aKV5iacgkfwqEQ==
X-Google-Smtp-Source: ABdhPJwtCUkpQblgKKVCP2M7xiFYT3GCIKxkXf0mxAB0hdBnY4HrV7QZX7uw6cwpC3G2QraywSR3rkc9L7/EIHI045M=
X-Received: by 2002:a25:da0f:: with SMTP id n15mr7380943ybf.481.1605372314254;  Sat, 14 Nov 2020 08:45:14 -0800 (PST)
MIME-Version: 1.0
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com>  
In-Reply-To: 
From: Kevin Yang 
Date: Sat, 14 Nov 2020 11:45:03 -0500
Message-ID: 
To: Yoshifumi Nishida 
Cc: Yuchung Cheng ,  "tcpm@ietf.org Extensions" , Eric Dumazet , iccrg IRTF list 
Content-Type: multipart/alternative; boundary="00000000000086889405b413de7d"
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 14 Nov 2020 16:45:21 -0000

--00000000000086889405b413de7d
Content-Type: text/plain; charset="UTF-8"

Thank you Yoshi, very good questions.

1: Our goal is to substitute the TSopt. So we design this new ETSopt not to
depend on TSopt.
    I guess this also answers your question 4, the intent is for the
standard track.

2: I think we should only update TS.recent when we assume the incoming
segment is valid.
    Otherwise, if the incoming is a real old one, we discard it but update
TS.recent;
    then a new segment arrives, we may reject this new seg
because TS.recent > seg.TSval.

3: Your understanding is correct, and in the draft we are not suggesting to
use NetworkRTT
    measurements for RTO calculation.

4: answered by 1.

Kevin

On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida 
wrote:

> Hi,
> Thanks for preparing the draft. I basically like the idea in the docs and
> have several comments.
>
> 1: I am wondering if we really need to have 32 bits TSval and TSecr in ETS
> option in SYN when TSopt is also sent?
>     For example, If TSopt has 1 ms granularity, I think ETS option will
> need to carry only small fractions of timestamp.
>
> 2: "To prevent a false positive PAWS rejection of a valid segment, an ETS
> receiver MUST skip the PAWS check"
>     -> This is one possible approach, but just skipping PAWS doesn't sound
> very good to me.
>          How about discarding the segment while updating TS.recent as an
> alternative? If the segment is not an old one, it will be retransmitted and
> accepted.
>          I think this is a bit more conservative.
>
> 3: It is not very clear for me how measuring NetworkRTT can contribute to
> improving RTO.
>     Or, are there some other ways to utilize NetworkRTT?
>
> 4: Just out of curiosity, is intended status standard track or
> experimental? ExID is usually used for exp or info docs.
>
> Thanks,
> --
> Yoshi
>
> On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng  40google.com@dmarc.ietf.org> wrote:
>
>> Hi tcpm & iccrg,
>>
>> We are proposing a new TCP timestamp option to measure the network delay
>> more precisely (e.g. excluding delayed ACK effects) for congestion control,
>> e.g. Swift published SIGCOMM 2020. The precision of the measurements can
>> potentially be further enhanced by NIC hardware timestamps. We are working
>> on a reference implementation for Linux as well.
>>
>> We'll also present it in the upcoming tcpm meeting. Feedbacks are very
>> welcome.
>>
>> ---------- Forwarded message ---------
>> From: 
>> Date: Mon, Nov 2, 2020 at 3:29 PM
>> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
>> To: Eric Dumazet , Yuchung Cheng ,
>> Neal Cardwell , Kevin Yang (Yudong) > >
>>
>>
>>
>> A new version of I-D, draft-yang-tcpm-ets-00.txt
>> has been successfully submitted by Kevin (Yudong) Yang and posted to the
>> IETF repository.
>>
>> Name:           draft-yang-tcpm-ets
>> Revision:       00
>> Title:          TCP ETS: Extensible Timestamp Options
>> Document date:  2020-11-02
>> Group:          Individual Submission
>> Pages:          13
>> URL:
>> https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt
>> Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets
>> Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00
>>
>>
>> Abstract:
>>    This document presents ETS: an Extensible TimeStamps option for TCP.
>>    It allows hosts to use microseconds as the unit for timestamps to
>>    improve the precision of timestamps, and advertise the maximum ACK
>>    delay for its own delayed ACK mechanism.  Furthermore, it extends the
>>    information provided in the [RFC7323] TCP Timestamps Option by
>>    including the receiver delay in the TSecr echoing, so that the
>>    receiver of the ACK is able to more accurately estimate the portion
>>    of the RTT that resulted from time traveling through the network.
>>    The ETS option format is extensible, so that future extensions can
>>    add further information without the overhead of extra TCP option kind
>>    and length fields.
>>
>>
>>
>>
>> Please note that it may take a couple of minutes from the time of
>> submission
>> until the htmlized version and diff are available at tools.ietf.org.
>>
>> The IETF Secretariat
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>

--00000000000086889405b413de7d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Tha=
nk you Yoshi, very good questions.

1: Our goal is to substitute the TSopt. So we design this new=
 ETSopt not to depend on TSopt.
=C2=A0 =C2=A0 I guess this also answers your question 4, t=
he intent is for the standard track.

2: I think we should only update TS.recent when we assume the i=
ncoming segment is valid.
=C2=A0 =C2=A0 Otherwise, if the incoming is a real old one, we d=
iscard it but update TS.recent;
=C2=A0 =C2=A0 then a new segment arrives, we may reject th=
is new seg because=C2=A0TS.recent > seg.TSval.

3: Your understanding is correct, and in the draft=
 we are not suggesting to use NetworkRTT
=C2=A0 =C2=A0 measurements for RTO calculation.

4: answered by 1.

Kevin

On Sat, Nov 14, 2020=
 at 7:13 AM Yoshifumi Nishida <nsd=
.ietf@gmail.com> wrote:
Hi,
Thanks for preparing the=
 draft. I basically like the idea in the docs and have several comments.

1: I am wondering if we really need to have 32 bits =
TSval and TSecr in ETS option in SYN when TSopt is also sent?=C2=A0
=C2=A0 =C2=A0 For example, If TSopt has 1 ms granularity, I think ETS op=
tion will need to carry only small=C2=A0fractions of timestamp.=C2=A0
=


2: "To prevent a false positive PAWS rejection of =
a valid segment, an ETS receiver MUST skip the PAWS check"
=
=C2=A0 =C2=A0 -> This is one possible approach, but just skipping PAWS d=
oesn't sound very good to me.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0How about discarding the segment while updating TS.recent as an alternat=
ive? If the segment is not an old one, it will be retransmitted and accepte=
d.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I think this is a bit more c=
onservative.

3: It is not very clear for me how me=
asuring NetworkRTT can contribute to improving RTO.=C2=A0
=C2=A0 =
=C2=A0 Or, are there some other ways to utilize NetworkRTT?=C2=A0

4: Just out of curiosity, is intended=C2=A0status standard =
track or experimental? ExID is usually used for exp or info docs.

Thanks,
--
Yoshi

On Mon, Nov 2, 2020 at =
4:26 PM Yuchung Cheng <ycheng=3D40google.com@dmarc.ietf.org> wrote:
H=
i tcpm & iccrg,

We are proposing a new TCP tim=
estamp option to measure the network delay more precisely (e.g. excluding d=
elayed ACK effects) for congestion control, e.g. Swift published SIGCOMM 20=
20. The precision of the measurements can potentially be further enhanced b=
y NIC hardware timestamps. We are working on a reference implementation for=
 Linux as well.

We'll also present it in the u=
pcoming tcpm meeting. Feedbacks are very welcome.=C2=A0

---------- Forwarded=
 message ---------
From: <internet-drafts@ietf.org>
Date: Mon, Nov 2, 2020 at 3:29 PM
Subject: New Version Notificatio=
n for draft-yang-tcpm-ets-00.txt
To: Eric Dumazet <edumazet@google.com>, Yuchung=
 Cheng <ycheng@go=
ogle.com>, Neal Cardwell <ncardwell@google.com>, Kevin Yang (Yudong) <yyd@google.com>




A new version of I-D, draft-yang-tcpm-ets-00.txt

has been successfully submitted by Kevin (Yudong) Yang and posted to the
IETF repository.



Name:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0draft-yang-tcpm-ets

Revision:=C2=A0 =C2=A0 =C2=A0 =C2=A000

Title:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 TCP ETS: Extensible Timestamp Opti=
ons

Document date:=C2=A0 2020-11-02

Group:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Individual Submission

Pages:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 13

URL:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt

Status:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0https:=
//datatracker.ietf.org/doc/draft-yang-tcpm-ets/

Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https:/=
/datatracker.ietf.org/doc/html/draft-yang-tcpm-ets

Htmlized:=C2=A0 =C2=A0 =C2=A0 =C2=A0https://tools.=
ietf.org/html/draft-yang-tcpm-ets-00





Abstract:

=C2=A0 =C2=A0This document presents ETS: an Extensible TimeStamps option fo=
r TCP.

=C2=A0 =C2=A0It allows hosts to use microseconds as the unit for timestamps=
 to

=C2=A0 =C2=A0improve the precision of timestamps, and advertise the maximum=
 ACK

=C2=A0 =C2=A0delay for its own delayed ACK mechanism.=C2=A0 Furthermore, it=
 extends the

=C2=A0 =C2=A0information provided in the [RFC7323] TCP Timestamps Option by=


=C2=A0 =C2=A0including the receiver delay in the TSecr echoing, so that the=


=C2=A0 =C2=A0receiver of the ACK is able to more accurately estimate the po=
rtion

=C2=A0 =C2=A0of the RTT that resulted from time traveling through the netwo=
rk.

=C2=A0 =C2=A0The ETS option format is extensible, so that future extensions=
 can

=C2=A0 =C2=A0add further information without the overhead of extra TCP opti=
on kind

=C2=A0 =C2=A0and length fields.









Please note that it may take a couple of minutes from the time of submissio=
n

until the htmlized version and diff are available at tools.ietf.org.



The IETF Secretariat







_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm






--00000000000086889405b413de7d--


From nobody Sat Nov 14 11:36:12 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3EE163A1297 for ; Sat, 14 Nov 2020 11:36:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.6
X-Spam-Level: 
X-Spam-Status: No, score=-17.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 34IMuKCKCfUi for ; Sat, 14 Nov 2020 11:36:06 -0800 (PST)
Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 27C103A1296 for ; Sat, 14 Nov 2020 11:36:05 -0800 (PST)
Received: by mail-wr1-x433.google.com with SMTP id p8so14071402wrx.5 for ; Sat, 14 Nov 2020 11:36:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=3JrDsbNXwec2/YXbLXxtiz+z8hE5EAkv8s/vTGq70hw=; b=VsNgbyZtZnUKmTVIXkIKl/i9efJRFRt32VG6Lx0jTv3EBPtnzxU57ly00Rit8Sqzsp gibpv/3/8aCg74Al3xNSqRcknA/xR/4OJ8ruGzz6kId/WBzKoRzUxgvRprgEBKnIQTD1 q+BYn60Qk4N8Bon3UV93ONIAEtgnZ66UcHxOLKsbqsUJtigq0tMi/TmMKvVTBhrOKrsw KIEM79hvAMTWiUlEWPGw63MZO8JveFjkJgl4izEw91hY0XZVyGKVDultQZACfFggjcoA vCRvFXuuLXYmQE/BJsGhLTH006PpxP9NFrbky9ykyaQqjjZk4PJycJX2UGfY9IgYo8Ij KPUg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=3JrDsbNXwec2/YXbLXxtiz+z8hE5EAkv8s/vTGq70hw=; b=nckLm6zrAMX4V4ScSlNrrEjGhP07HMZorQ3uZJNyDNeo/pCRdIgtCV0WzIB9r4DEXJ al4IttiU51JTycPfR/Wwr+nrZUN4kKo8ujVhFbq1kAOySC4OYiPyN1UgDz7G98zbWeBn L4IiL/JaFgKGeNCoaQvlFrqIHVDYiQ2WggdGOPLEjU+2Rq92nte1C6uTGv2DkHra2I2s d/NHz9OedkgQhQyRtiUdzSwzzS/UuziAD00AbvNmzi/GgHijsViVXvghqtwuwxG5Szan 8NksNAwJ4ZnsHXNRgLOLyDx0NuJTWiXOX4z4S22Z4zekoqFYPrAOeUIlm3MYbEeQEYMT mHsQ==
X-Gm-Message-State: AOAM532tYTR3coYk6FLqVs7nPFzF2BosyFtSqT2s/47FuoKDMp2duJB+ rMr9QGPEvw2oVu3J/LVzxAF/MTGGQgeVOQ02UmStjQ==
X-Google-Smtp-Source: ABdhPJwvQVIGSfayewhQNViQwjAArKvTYi7URGXs2R6xJNvJ6hJkzEEtHjQGV9K3mw98PkNc0ifSO8h4ZLuxDf71DAI=
X-Received: by 2002:adf:ee12:: with SMTP id y18mr10483351wrn.231.1605382564159;  Sat, 14 Nov 2020 11:36:04 -0800 (PST)
MIME-Version: 1.0
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com>   
In-Reply-To: 
From: Yuchung Cheng 
Date: Sat, 14 Nov 2020 11:35:27 -0800
Message-ID: 
To: Kevin Yang 
Cc: Yoshifumi Nishida , "tcpm@ietf.org Extensions" ,  Eric Dumazet , iccrg IRTF list 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 14 Nov 2020 19:36:09 -0000

On Sat, Nov 14, 2020 at 8:45 AM Kevin Yang
 wrote:
>
> Thank you Yoshi, very good questions.
>
> 1: Our goal is to substitute the TSopt. So we design this new ETSopt not =
to depend on TSopt.
>     I guess this also answers your question 4, the intent is for the stan=
dard track.
>
> 2: I think we should only update TS.recent when we assume the incoming se=
gment is valid.
>     Otherwise, if the incoming is a real old one, we discard it but updat=
e TS.recent;
>     then a new segment arrives, we may reject this new seg because TS.rec=
ent > seg.TSval.
>
> 3: Your understanding is correct, and in the draft we are not suggesting =
to use NetworkRTT
>     measurements for RTO calculation.
>
> 4: answered by 1.
We'll clarify better in the upcoming draft revision: the draft
intends to transition from exp-id to  an official option kind number
from IANA upon becoming
an RFC. But we're following the advice of
https://tools.ietf.org/html/rfc6994 to allow larger scale experiments
for hosts not completely under control of a single entity/network. For
example guest VM TCP communications inside Cloud Data-center networks.


>
> Kevin
>
> On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida  wr=
ote:
>>
>> Hi,
>> Thanks for preparing the draft. I basically like the idea in the docs an=
d have several comments.
>>
>> 1: I am wondering if we really need to have 32 bits TSval and TSecr in E=
TS option in SYN when TSopt is also sent?
>>     For example, If TSopt has 1 ms granularity, I think ETS option will =
need to carry only small fractions of timestamp.
>>
>> 2: "To prevent a false positive PAWS rejection of a valid segment, an ET=
S receiver MUST skip the PAWS check"
>>     -> This is one possible approach, but just skipping PAWS doesn't sou=
nd very good to me.
>>          How about discarding the segment while updating TS.recent as an=
 alternative? If the segment is not an old one, it will be retransmitted an=
d accepted.
>>          I think this is a bit more conservative.
>>
>> 3: It is not very clear for me how measuring NetworkRTT can contribute t=
o improving RTO.
>>     Or, are there some other ways to utilize NetworkRTT?
>>
>> 4: Just out of curiosity, is intended status standard track or experimen=
tal? ExID is usually used for exp or info docs.
>>
>> Thanks,
>> --
>> Yoshi
>>
>> On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng  wrote:
>>>
>>> Hi tcpm & iccrg,
>>>
>>> We are proposing a new TCP timestamp option to measure the network dela=
y more precisely (e.g. excluding delayed ACK effects) for congestion contro=
l, e.g. Swift published SIGCOMM 2020. The precision of the measurements can=
 potentially be further enhanced by NIC hardware timestamps. We are working=
 on a reference implementation for Linux as well.
>>>
>>> We'll also present it in the upcoming tcpm meeting. Feedbacks are very =
welcome.
>>>
>>> ---------- Forwarded message ---------
>>> From: 
>>> Date: Mon, Nov 2, 2020 at 3:29 PM
>>> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
>>> To: Eric Dumazet , Yuchung Cheng , Neal Cardwell , Kevin Yang (Yudong) 
>>>
>>>
>>>
>>> A new version of I-D, draft-yang-tcpm-ets-00.txt
>>> has been successfully submitted by Kevin (Yudong) Yang and posted to th=
e
>>> IETF repository.
>>>
>>> Name:           draft-yang-tcpm-ets
>>> Revision:       00
>>> Title:          TCP ETS: Extensible Timestamp Options
>>> Document date:  2020-11-02
>>> Group:          Individual Submission
>>> Pages:          13
>>> URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.=
txt
>>> Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
>>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm-e=
ts
>>> Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00
>>>
>>>
>>> Abstract:
>>>    This document presents ETS: an Extensible TimeStamps option for TCP.
>>>    It allows hosts to use microseconds as the unit for timestamps to
>>>    improve the precision of timestamps, and advertise the maximum ACK
>>>    delay for its own delayed ACK mechanism.  Furthermore, it extends th=
e
>>>    information provided in the [RFC7323] TCP Timestamps Option by
>>>    including the receiver delay in the TSecr echoing, so that the
>>>    receiver of the ACK is able to more accurately estimate the portion
>>>    of the RTT that resulted from time traveling through the network.
>>>    The ETS option format is extensible, so that future extensions can
>>>    add further information without the overhead of extra TCP option kin=
d
>>>    and length fields.
>>>
>>>
>>>
>>>
>>> Please note that it may take a couple of minutes from the time of submi=
ssion
>>> until the htmlized version and diff are available at tools.ietf.org.
>>>
>>> The IETF Secretariat
>>>
>>>
>>> _______________________________________________
>>> tcpm mailing list
>>> tcpm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Sun Nov 15 23:41:04 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 179003A151E; Sun, 15 Nov 2020 23:40:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level: 
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=eggert.org
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kuftKKAv560Z; Sun, 15 Nov 2020 23:40:57 -0800 (PST)
Received: from mail.eggert.org (mail.eggert.org [IPv6:2a00:ac00:4000:400:211:32ff:fe22:186f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 091B13A150D; Sun, 15 Nov 2020 23:40:57 -0800 (PST)
Received: from [IPv6:2a00:ac00:4000:400:f509:b293:50a1:6ad4] (unknown [IPv6:2a00:ac00:4000:400:f509:b293:50a1:6ad4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.eggert.org (Postfix) with ESMTPSA id E98CC6120A9; Mon, 16 Nov 2020 09:40:42 +0200 (EET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eggert.org; s=dkim; t=1605512443; bh=1K2eoXw1/WgKKTaMV8NafLYpfHuMbqb5cFIw4aBOn5I=; h=From:Subject:References:To:Date; b=jLoEpcPSYQrBUH0qlf7gUkx5sPYtRqAqOi84Z94TuBrtgXDe7wNcMEjBUAcwKWUm1 pyX7N8Tdmk5qoSGHt4/NrFp2cn7b6PqLdcOXgDxdaiBMkSsvisFfR+oUl3f5iffiez dzSDdAOrsoHNzGNjD8Y9JBRumcBxKuyZJvYK4faU=
From: Lars Eggert 
Content-Type: multipart/signed; boundary="Apple-Mail=_4CC1A84C-0157-4617-9096-2710D2C5BC76"; protocol="application/pgp-signature"; micalg=pgp-sha512
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
Message-Id: <4E7BA544-388B-4349-B996-DE561C3C3040@eggert.org>
References: <160551216500.21396.4843161030535839551@ietfa.amsl.com>
To: tcpm IETF list , QUIC WG 
Date: Mon, 16 Nov 2020 09:40:41 +0200
X-MailScanner-ID: E98CC6120A9.A3A5C
X-MailScanner: Found to be clean
X-MailScanner-From: lars@eggert.org
Archived-At: 
Subject: [tcpm] Fwd: New Version Notification for draft-eggert-tcpm-rfc8312bis-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 07:40:59 -0000

--Apple-Mail=_4CC1A84C-0157-4617-9096-2710D2C5BC76
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_14AECCC6-A2ED-4B87-A42D-08C1A1F34C7A"


--Apple-Mail=_14AECCC6-A2ED-4B87-A42D-08C1A1F34C7A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hi,

this is a -00 individual I-D to start the work on revising RFC8312 =
(CUBIC). There are no changes compared to RFC8312 yet (other than =
formatting due to conversion to markdown, etc.) - this serves as a =
baseline to make tracking changes easier as we go forward.

Some change proposals are already under discussion, please see =
https://github.com/NTAP/rfc8312bis/issues and participate in the =
discussion there.

Thanks,
Lars

Begin forwarded message:
>=20
> From: internet-drafts@ietf.org
> Subject: New Version Notification for =
draft-eggert-tcpm-rfc8312bis-00.txt
> Date: November 16, 2020 at 9:36:05 GMT+2
> To: "Lars Eggert" , "Sangtae Ha" =
, "Lisong Xu" 
>=20
>=20
> A new version of I-D, draft-eggert-tcpm-rfc8312bis-00.txt
> has been successfully submitted by Lars Eggert and posted to the
> IETF repository.
>=20
> Name:		draft-eggert-tcpm-rfc8312bis
> Revision:	00
> Title:		CUBIC for Fast Long-Distance Networks
> Document date:	2020-11-15
> Group:		Individual Submission
> Pages:		19
> URL:            =
https://www.ietf.org/archive/id/draft-eggert-tcpm-rfc8312bis-00.txt
> Status:         =
https://datatracker.ietf.org/doc/draft-eggert-tcpm-rfc8312bis/
> Html:           =
https://www.ietf.org/archive/id/draft-eggert-tcpm-rfc8312bis-00.html
> Htmlized:       =
https://tools.ietf.org/html/draft-eggert-tcpm-rfc8312bis-00
>=20
>=20
> Abstract:
>   CUBIC is an extension to the current TCP standards.  It differs from
>   the current TCP standards only in the congestion control algorithm =
on
>   the sender side.  In particular, it uses a cubic function instead of
>   a linear window increase function of the current TCP standards to
>   improve scalability and stability under fast and long-distance
>   networks.  CUBIC and its predecessor algorithm have been adopted as
>   defaults by Linux and have been used for many years.  This document
>   provides a specification of CUBIC to enable third-party
>   implementations and to solicit community feedback through
>   experimentation on the performance of CUBIC.
>=20
>   This documents obsoletes [RFC8312], updating the specification of
>   CUBIC to conform to the current Linux version.
>=20
>=20
>=20
>=20
> Please note that it may take a couple of minutes from the time of =
submission
> until the htmlized version and diff are available at tools.ietf.org.
>=20
> The IETF Secretariat
>=20
>=20


--Apple-Mail=_14AECCC6-A2ED-4B87-A42D-08C1A1F34C7A
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

Hi,

this =
is a -00 individual I-D to start the work on revising RFC8312 (CUBIC). =
There are no changes compared to RFC8312 yet (other than formatting due =
to conversion to markdown, etc.) - this serves as a baseline to make =
tracking changes easier as we go forward.

Some change proposals are already under =
discussion, please see https://github.com/NTAP/rfc8312bis/issues and participate =
in the discussion there.

Thanks,
Lars

Begin forwarded message:

From: =
internet-drafts@ietf.org
Subject: =
New Version =
Notification for draft-eggert-tcpm-rfc8312bis-00.txt
Date: =
November 16, 2020 at 9:36:05 =
GMT+2
To: =
"Lars Eggert" <lars@eggert.org>, =
"Sangtae Ha" <sangtae.ha@colorado.edu>, "Lisong Xu" <xu@unl.edu>

A new version of I-D, draft-eggert-tcpm-rfc8312bis-00.txt
has been successfully submitted by Lars Eggert and posted to =
the
IETF repository.

Name:		=
draft-eggert-tcpm-rfc8312bis
Revision:	00
Title:=
		=
CUBIC for Fast Long-Distance Networks
Document =
date:	=
2020-11-15
Group:		Individual Submission
Pages:=
		=
19
URL: =
           https://www.ietf.org/archive/id/draft-eggert-tcpm-rfc8312bis-00=
.txt
Status: =
        https://datatracker.ietf.org/doc/draft-eggert-tcpm-rfc8312bis/<=
/a>
Html: =
          https://www.ietf.org/archive/id/draft-eggert-tcpm-rfc8312bis-00=
.html
Htmlized:       https://tools.ietf.org/html/draft-eggert-tcpm-rfc8312bis-00=



Abstract:
 =
  CUBIC is an extension to the current TCP standards.  It =
differs from
   the current TCP standards only =
in the congestion control algorithm on
   the =
sender side.  In particular, it uses a cubic function instead of
   a linear window increase function of the current =
TCP standards to
   improve scalability and =
stability under fast and long-distance
 =
  networks.  CUBIC and its predecessor algorithm have =
been adopted as
   defaults by Linux and have =
been used for many years.  This document
 =
  provides a specification of CUBIC to enable third-party
   implementations and to solicit community =
feedback through
   experimentation on the =
performance of CUBIC.

   This =
documents obsoletes [RFC8312], updating the specification of
   CUBIC to conform to the current Linux =
version.




Please note that it may take a couple of minutes from the =
time of submission
until the htmlized version and diff are =
available at tools.ietf.org.

The IETF =
Secretariat



=

--Apple-Mail=_14AECCC6-A2ED-4B87-A42D-08C1A1F34C7A--

--Apple-Mail=_4CC1A84C-0157-4617-9096-2710D2C5BC76
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEmpq0ZpSoejRmyhheVLXDCb9wwVcFAl+yLPkACgkQVLXDCb9w
wVdGfQ//bL7gSLcciFjLLOzmBYVCj3nDH3iJxEWAkx8HPlaVWKKNxBwg/cQWtTmK
yQgornXKFHfHEpzuGboXOelMp9k1Nml2/Rgvmi+xz9zFo3YLF8Pa8/USgeMT7u6x
IGS4iJuWAywYGB1svIfKpyXNXH+qXR96qNYRljAlHrs59j0nBDefo2+VJhyFIN6e
HsUzkA+BNZfdRDnNf4ynVABXL7huREqIdx8nx3yZLO+5V5ZYnf79l0IasUofAjQb
XbT2gwEuSGllWXg/2TcQVpRzldcQwROIZTGuHpAjbnd2c0NQ2zD5/+68sWK5AJiE
rVgi+NMCDkpfkpNVBQGd5nAngt75vxsUU+egcBi7//6Pw7rf71M+4QBucrMuqxXx
za300tfAthKA2qxQmwBr6DDad0XmL2/pV1GcKmK5z1MYbIWOvj/9FmYl/OvKaruf
7MdzWCk7b2VAEq7RfXEE08RPERJwPk4/neIZA0dalma3GGxOvqZKNiHRVwDImR1+
lvJsw1YEsspzg5P1GSUZJgKAGcVZdpvszPAdhD/0FTJbk7UK+wJ5oNC4q1r9F/vE
tIO6HzSWMtw6KVNfYaOXQcg0qS1Np4i5lXlEG0DQ+uHV9x4lEPJ5+RA90RHhLmeg
ezaOA/WXwGMHCBKGnGwqYg0XjGNABuBrq4A0nrt4r9CoREGXxeY=
=mAuU
-----END PGP SIGNATURE-----

--Apple-Mail=_4CC1A84C-0157-4617-9096-2710D2C5BC76--


From nobody Mon Nov 16 05:08:32 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 12C043A0E22 for ; Mon, 16 Nov 2020 05:08:28 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.101
X-Spam-Level: 
X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=uclouvain.be
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ib9dv4cXEIBS for ; Mon, 16 Nov 2020 05:08:25 -0800 (PST)
Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2121.outbound.protection.outlook.com [40.107.20.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1FF713A0E95 for ; Mon, 16 Nov 2020 05:08:20 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lRmyYO7BQofI4hj+nmYwHshK18re/NeswbRbzElXlLM+LFB1vuCBje3Uz0gDfoP10AG+8cPy6LfgKtdTesRRyWMBH3HPHjiXICszGVnQygdsZxhHf6ifIbzexS55LF8mBjsFrN4esINYieI7po1memp8EPUE63R/swm81qR19kmvjHguVTWA+qGSR1dJRF6QV6P6jm+a3i1jtQBGQceSkeIkOHX3OOUQAIyCRoH38fMipCn7OF5Is6nTv6XmnoSVdcWFZ5CVXej/d48mY6W+HQ/RZusmhlVNuSGjmcvBnHZ5j6IPPlovl6SfumLoEnSm5B1HoW9/XKVtzU9f3zZJKw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;  s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XjrlV9UHoIQEo1/VWGtpsnVBGXK2B0Oo6YihzZXXqSs=; b=bHDS1pXWXZe+UHgyxT+MWZbAUgAdVz+yGQG3353MCF5hprudebEhyjSEKCsmmAg1IxkNMev8dFea1+nobrk2xapFGrvxRywyN4Uld33yfj4NzvoaN4YSgKAA8EIG91Z6SZE3Fh/h5nrcmgzlcx6NGhD63P8LlmRQjoU+sSDFx1MkhWuJm7yHhD5m82zIJeN2MtCEpo1FyXgbEKXuVs1499FGqH3GNmo0skEyRVUrUkT6I5XMkZqqtPweNZiqHfrxH3g9KEuPWuZLTV6+vRK4HWz1ilRrEhQjMS9UCGGwZeGnrLiVAJERR+cnFh8qYsyiTfKgBLYEkZ0dUoVuIfVntQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uclouvain.be; dmarc=pass action=none header.from=uclouvain.be; dkim=pass header.d=uclouvain.be; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uclouvain.be; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XjrlV9UHoIQEo1/VWGtpsnVBGXK2B0Oo6YihzZXXqSs=; b=DXjZcwVQKxDjggomr0ZpnavEPFKEhPueoAUwmHUCJOwlwoEs1P7zYQCfBYIpbhF0j1/Fs4wuvq+FWydqdA2pyuoeca8Ezb/okgwoTCUEjAH6Nln7lbg63HRizpTZ/YHsDrF/HYCEjaGI04d9UaLyp1U0MKHtsqErpIuKBSPWq1E=
Authentication-Results: huawei.com; dkim=none (message not signed) header.d=none;huawei.com; dmarc=none action=none header.from=uclouvain.be;
Received: from AM7PR03MB6642.eurprd03.prod.outlook.com (2603:10a6:20b:1bf::6) by AM6PR03MB5527.eurprd03.prod.outlook.com (2603:10a6:20b:fe::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.21; Mon, 16 Nov 2020 13:08:18 +0000
Received: from AM7PR03MB6642.eurprd03.prod.outlook.com ([fe80::b03d:e09f:4450:31cd]) by AM7PR03MB6642.eurprd03.prod.outlook.com ([fe80::b03d:e09f:4450:31cd%5]) with mapi id 15.20.3564.028; Mon, 16 Nov 2020 13:08:18 +0000
Reply-To: Olivier.Bonaventure@uclouvain.be
To: Markus.Amend@telekom.de, tcpm@ietf.org
References: 
From: Olivier Bonaventure 
Message-ID: <00733d01-e95c-3cae-eee2-40fd93f7c0f2@uclouvain.be>
Date: Mon, 16 Nov 2020 14:08:17 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.4.3
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: fr-classic
Content-Transfer-Encoding: 7bit
X-Originating-IP: [2a02:2788:484:140:f93e:7ada:ec60:9c97]
X-ClientProxiedBy: AM0PR04CA0057.eurprd04.prod.outlook.com (2603:10a6:208:1::34) To AM7PR03MB6642.eurprd03.prod.outlook.com (2603:10a6:20b:1bf::6)
MIME-Version: 1.0
X-MS-Exchange-MessageSentRepresentingType: 1
Received: from mbpobo-2.local (2a02:2788:484:140:f93e:7ada:ec60:9c97) by AM0PR04CA0057.eurprd04.prod.outlook.com (2603:10a6:208:1::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3564.28 via Frontend Transport; Mon, 16 Nov 2020 13:08:17 +0000
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 1b93642b-1dfc-4319-1dc7-08d88a30af6c
X-MS-TrafficTypeDiagnostic: AM6PR03MB5527:
X-Microsoft-Antispam-PRVS: 
X-MS-Oob-TLC-OOBClassifiers: OLM:10000;
X-MS-Exchange-SenderADCheck: 1
X-Microsoft-Antispam: BCL:0;
X-Microsoft-Antispam-Message-Info: FhU1ByE3keYiaS0oGo+FBEhwSlZFrgdtXUVWTiRJ2YtlT4x715TFeh6QwCcneH0yK+2rAeM2L3Fhn2TBWJz16XUFlp/PTpSLPVsnOv4dTJsfWMlfPHQI52bM3RfwBsNUNiwGIken7IVdmU2oOOcNSXV0+nciRizrXLW/C87fzt73Kqkbk3fk6ieCirZRkx+G9DfRr0wMCCTSwCB0u2cmD9qH5ypheI8yf/gTtAYlrWulf/uOxhrZDBl2r/7awKdiai3gP9IAiBUbTqbcP3i2d7gO3YMg5hLX2B/O9cxdwDIu2cRwcJpMTQQ39NHd7uuL5knXuT2cXm267u96flcKillZ5Jdfi4rHNwaDssQGXtjXuf4zgTk8QzktDRKtGaApKFUNOB6G/NwrIJJrGcqvwRPA0kIrb0L5ukn3M8cpuy6VQ5JqunLwgIUvSy1UbIvTtvsiKDfTfdQgi8SnA/6Ghg==
X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;  IPV:NLI; SFV:NSPM; H:AM7PR03MB6642.eurprd03.prod.outlook.com; PTR:; CAT:NONE;  SFS:(4636009)(39860400002)(346002)(366004)(136003)(376002)(396003)(478600001)(86362001)(8936002)(2616005)(36756003)(31696002)(4326008)(3480700007)(52116002)(6512007)(3450700001)(186003)(7116003)(16526019)(2906002)(83380400001)(6486002)(8676002)(6506007)(66476007)(966005)(66556008)(31686004)(66946007)(5660300002)(316002)(786003)(43740500002); DIR:OUT; SFP:1102; 
X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?cFZCbFVveTU0bW9zU01Ga2czSm5qSzM4czlFaDRIUEZLRE1ma0k2L1ZXdWRj?= =?utf-8?B?cmpRZHAxOUoweEdDSkxrbEI0WjFtMDZQd29zQXRWcUtBMUp5NU44V0NIZ3VC?= =?utf-8?B?WkplUEVnQ1hXb3hYR0dHSEFiSitTTWxpeDFmWjEzVzd2L0dJTy92ZlZXL25v?= =?utf-8?B?RldLcEYvWnBtVTFGZFNRenZla212d21ROHRlVXV3NmNLakVobkU0Y1ovUjly?= =?utf-8?B?SDJSVXc0SmZBbmdaOVdlODhLTWdPMUZDMHVrL3RNaUMzcHRuZDRCSjBSUm4y?= =?utf-8?B?R3dTS1JtckFKV3RwL25WUW9oM2pVVitCTGpNR0Jwdm9ESlYvTVMyS3Q5djFK?= =?utf-8?B?MlRzNllvOWFIQlZ1T0syVmZtY0lsTURuNVRsR1ZPS09MMzhEdUVjWHp0Y2tT?= =?utf-8?B?d1dydkNXZkYzRzd3R2NpRkZaRHFRb3FRakNBbXZubmVMaW1FMHRlUWlSVVc1?= =?utf-8?B?RWJZREhvVld3b0UyZHY2c1RyU0FWV2dadWlJOFlxS2toRTljc1FtRWs5Vmg0?= =?utf-8?B?aVd5bjJKYXR4ajdPRjNtOWwxeitpMDVXWnBmS2VtMmRKRS9DS3VtMk5lS0x3?= =?utf-8?B?c2hndUdSbnFwdnB6YTRRQWIrYTB1Sy9kREp5SlBWdmVxaDZYTHQrWDI3TEJT?= =?utf-8?B?cFk4eGc0WjdOWnlvclNaMjRYUWtIZFF2dlVWUnlnWjBxMVVDdTlwVWtPYVRN?= =?utf-8?B?Z3kxUnJxclFKTGgrSHNkaTUwRWRQYkZ5R1Bqa1VsSlpkWDZDMncxL1lsek1K?= =?utf-8?B?L0prcHMzdDBlQy9rMHF4dlE5c0dYMFhZVFhaWm9HZ2NXa0tvVmZ1NVphSUs5?= =?utf-8?B?VTRTSzl1SG5mcjhDeWdIcVlKRWxxZHJDaUFGZlBFWTVUM0hXL1EvQ2x2cHFE?= =?utf-8?B?a1lMbEYwNE94Vzd4MEZMNUROWlpnQXV4ZitxUlN5ZG81QmNiaWNWQjBRMlo5?= =?utf-8?B?VmNTRjZBcTFLQTNxcUtGT1VFTlR2enFYWlpLandpYUtpeW9ieU9pOE1NdGh1?= =?utf-8?B?Mmx0VDhnY25ONzNFRzkxb081enJYbHBLOHZ0UFg2TS9QUU1jbHJ0aVFUcG5r?= =?utf-8?B?bHBKRjY0b291QnFGWUxObEtvZklCSDlHRi9Cb3lxcmR6bmVWU0VPTUVIMEJ3?= =?utf-8?B?emlPTWhBdFNRbENieEZQcE9McFk0UVZqWXlWYTdmanF4YzlmT081aGlGaDNW?= =?utf-8?B?c0JmNHE4aVVzdFNnL3dacEJvT2xnUXEzczBNd2RyUGdGenVieHkwNTlWRys5?= =?utf-8?B?c05ER3E1eVRqWWVkVHRDMzlWS0ZPVG9JZEJvdUhEdkZBZm5kdz09?=
X-OriginatorOrg: uclouvain.be
X-MS-Exchange-CrossTenant-Network-Message-Id: 1b93642b-1dfc-4319-1dc7-08d88a30af6c
X-MS-Exchange-CrossTenant-AuthSource: AM7PR03MB6642.eurprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2020 13:08:18.0639 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 7ab090d4-fa2e-4ecf-bc7c-4127b4d582ec
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ipbnJdYSzoqjjh3ZKQBXpmsQP+5layH0VKgtBrprVLFwNC0t1ud2DD6wsa55w40ApN3K/tcGwYgNmXN6GU8sNqHKGFkA+k+rkee69+TCUYUlTUnLsPTRM9CrAoPyL+U0
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR03MB5527
Archived-At: 
Subject: Re: [tcpm] MPTCP RobE
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 13:08:28 -0000

Markus, Jiao,
> 
> as one of the authors of https://datatracker.ietf.org/doc/draft-amend-tcpm-mptcp-robe/, I kindly request feedback from you, in particular however from the RFC6824/8684 authors and MPTCP implementers.
> 
> With MPTCP RobE (Robust Establishment) we address the lack of MPTCP to reliably setup a MPTCP session, when the selected path for the initial session setup is malfunctioned. The MPTCP RFCs does not give any guidelines for this particular problem and it is left to the implementer to take care. For that we propose and compare different strategies in the draft document, separated by their demand on MPTCP protocol changes. Apart from that our claim is, that with these proposals compared to MPTCP, no additional security flaws are raised and any potential computational overhead is minimized.
> 

Sorry for the late reply. I have looked at the document and found that 
handling both RFC6824 and RFC8684 adds a lot of complexity to the 
document. Since RFC8684 is the standards track version of MPTCP, I would 
suggest to only discuss this version of MPTCP in the document.

My main concern about the proposed approach is how this will work with 
load-balancers. When there is a load-balancer in front of a pool of 
servers that use the same IP address, there is no guarantee that SYNs 
from the same client will be delivered to the same server, especially if 
these SYNs are sent over different paths and thus with different source 
addresses. If the two connections that RobE tries to establish terminate 
on different physical servers, they cannot be joined. Given the 
importance of load balancers, I would prefer a solution where the 
application simply tries to open two parallel mptcp connections, prefers 
the first that gets established, adds another subflow to this connection 
and resets the second one. This would slightly increase the delay 
compared to RobE, but would preserve load balancers. In practice, I 
expect that MPTCP stacks will leverage information about the current 
quality of the access networks (e.g. WiFi and xG on smartphones) and 
will only attempt parallel connections when the conditions are 
uncertain. Apple's WiFi assist already includes such functionnality.

Best regards,


Olivier


From nobody Mon Nov 16 08:01:41 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E5D043A1232 for ; Mon, 16 Nov 2020 08:01:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=telekom.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LCQunhzxFGZH for ; Mon, 16 Nov 2020 08:01:38 -0800 (PST)
Received: from mailout41.telekom.de (mailout41.telekom.de [194.25.225.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9ABD83A122E for ; Mon, 16 Nov 2020 08:01:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telekom.de; i=@telekom.de; q=dns/txt; s=dtag1; t=1605542497; x=1637078497; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=kkg/45nyFrzQ43YC8mPO1VIQGXtRqw/CJVLF+ZB0W3k=; b=OGm99duW0mnsyv7Cc6ceRieUC6WVTYUVZIP4eC3nZd0YF1iV0PqUmU0/ r6BlIglB8fHlC03mSG8wYqSst8xCSo064ePtfYywQdU4bwC0g4gsqE3y+ W2yl4PYx1K7vgriI0mqv9LTHrzLGrWd4q5d0IakDFYSOAzFa3PqfmXBkL Y/rSBYz1ltMk/YN/iGLrUVlGzMs3tEIW+KIi2CXM/OG9kp5L1WcAO/hL2 hs9R9SwD7gX1pHYVA8n4OwisCpUi5q42ik3P1oR11TFXU4Yvfu7HT32Ro A3CFcnzhxGdD4gWOnXZsrCZPr09YZ9/zhMEKjwwf7LXsvexkPMxEkY3xy Q==;
IronPort-SDR: s2ZNYjKPVkITuZZgCo8dDoivyWm1JGdOOVv27TrZO7Mg7QHzCt8VIbaW2/a9XM4tviUNM8TNVg D8fCJMn8QWEw==
Received: from qdefcs.de.t-internal.com ([10.171.254.41]) by MAILOUT41.dmznet.de.t-internal.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 16 Nov 2020 17:01:35 +0100
IronPort-SDR: 7vmQ8025018S0lswNdUkaSpIqQNvZT9Z8E8GFz+Ph6qA877+599ONIjKxQ+7Lgp2eQ7tFauZTT wEvKmd/h1PsFJiSbSIckDM5ss75JTSE+M=
X-IronPort-AV: E=Sophos;i="5.77,483,1596492000"; d="scan'208";a="234706716"
X-MGA-submission: =?us-ascii?q?MDGTBR5A00Y+hRfSTCnkpW9KlGQBqmb63SD4uJ?= =?us-ascii?q?siNTUGAaviyjrBl2LGbMoubDgyC5G2mHU6nrWFBiyYbUEFANQeVNe6Rl?= =?us-ascii?q?jX7oPqvJ7XfClIrN2+vwndPPmvSG/ROe5mREhjNjXbZ09dD9zzxQTd+4?= =?us-ascii?q?ws5aM587Uf5aPonejnlACf3w=3D=3D?=
Received: from he199744.emea1.cds.t-internal.com ([10.169.119.52]) by QDEFCV.de.t-internal.com with ESMTP/TLS/ECDHE-RSA-AES128-SHA256; 16 Nov 2020 17:01:36 +0100
Received: from HE199743.EMEA1.cds.t-internal.com (10.169.119.51) by HE199744.emea1.cds.t-internal.com (10.169.119.52) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Nov 2020 17:01:34 +0100
Received: from HE104163.emea1.cds.t-internal.com (10.171.40.38) by HE199743.EMEA1.cds.t-internal.com (10.169.119.51) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Mon, 16 Nov 2020 17:01:34 +0100
Received: from GER01-LEJ-obe.outbound.protection.outlook.de (51.5.80.19) by O365mail05.telekom.de (172.30.0.230) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 16 Nov 2020 17:01:33 +0100
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=alpwERtbOEGegEcxEFwAEKFmcjTR2VHQvpM+ngKNAfnYXnHCtRbLmgAWrqYnyq/XLaYpWzCxe5derCD7IABDZB58zDyFL02BIywdYxqDF7Be3Tol9RN77L+bxIBK51hpx/vKnqlWD+ivCrp+/rVWOMOJQlW+drn8e63grSXZ95anMnz+IvAfZTXTN3TZIrR7HSTL3bb5lW3mrmXR+yl7OJq9QwDfNkFveKa+xCCVcGcZmuUyB3+3lyi2mrA3IqQUbPDxprca+kUMiGpWctA/6OqcntPrL0WBzc5jiJGxMpOhHfrRtIPTwwdLRxSf+/KMoqrXlM/SToaeZARRL4xpvA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;  s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kkg/45nyFrzQ43YC8mPO1VIQGXtRqw/CJVLF+ZB0W3k=; b=XOtWNRs03HRG+UuBzX5UgSV4Ure7aRZH2NoiGWd8yb1TNxJ9dVcNZVIB2F5PRj1tRxpFiRejCJj8G7S+7i3POBXrlzDaExs6gfRM3XrTLBCRgZcVN3pTWlQp/n0iTlJUQm9vvCWYAtN+fXTS5KC2NKZ9wufZ0LQ7B45H/xOyaEiRWMkDGDHXy12tdQDyULwD5G6Qen7W+ag3xHW4INFNc1D7k8UM7yukzQ+9WLTQ57tvOMxoh+jkKLwvMoR6UVgM2qj7dMNeBF9ZYGCuxjeBmBA/PAPY+X64brTtaulf9K33rIH7ekCVh8xSU0rH3BUZVJ7xnPobRw4kQDyDpmqkrg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=telekom.de; dmarc=pass action=none header.from=telekom.de; dkim=pass header.d=telekom.de; arc=none
Received: from LEJPR01MB0635.DEUPRD01.PROD.OUTLOOK.DE (2a01:4180:c012:b::12) by LEJPR01MB0748.DEUPRD01.PROD.OUTLOOK.DE (2a01:4180:c012:e::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3589.15; Mon, 16 Nov 2020 16:01:33 +0000
Received: from LEJPR01MB0635.DEUPRD01.PROD.OUTLOOK.DE ([fe80::cd90:d2c4:eccf:600a]) by LEJPR01MB0635.DEUPRD01.PROD.OUTLOOK.DE ([fe80::cd90:d2c4:eccf:600a%5]) with mapi id 15.20.3589.016; Mon, 16 Nov 2020 16:01:33 +0000
From: 
To: , 
Thread-Topic: MPTCP RobE
Thread-Index: Adat14Qs9LlNljS0SE+s1uwEYxjReQOQgg+AAAWwexA=
Date: Mon, 16 Nov 2020 16:01:33 +0000
Message-ID: 
References:  <00733d01-e95c-3cae-eee2-40fd93f7c0f2@uclouvain.be>
In-Reply-To: <00733d01-e95c-3cae-eee2-40fd93f7c0f2@uclouvain.be>
Accept-Language: de-DE, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: uclouvain.be; dkim=none (message not signed) header.d=none;uclouvain.be; dmarc=none action=none header.from=telekom.de;
x-originating-ip: [212.201.104.11]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 5bfc67fc-7c7e-40aa-0d5e-08d88a48e3cb
x-ms-traffictypediagnostic: LEJPR01MB0748:
x-microsoft-antispam-prvs: 
x-ms-oob-tlc-oobclassifiers: OLM:8882;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: NY9EcYfyEkAPio38/JNnERBeckH08MnK+go012Do3UqWTasNipMlJgBzs2J3Lbvk7/ssZC9+PqWFmsaBi743IWdrRK/8M/Uyk/fmlysAx2XdDheSJvZ9VrS0FzhzwzQDTVGJbaq4wt/YZHdkNFCkX/FBBtfEbfaqbvMJ5b8D5Nk458L8udXL5lH9IgxukIp2oocJjJsfAHp4E9mCUalCzGDe1633a1QZM0rgoAM/cE2wMIhxiw+5OJ3qsYoj8M2WTy3k179NPi/JusWwxzGFkx6mWkuVKxsYPnIGPrN1f5cfkf4tp6NUPlekgoFI4W0M1ofBFk/ixr7f9NJHig/5+5ZRjc1M95YtS0aLn3XQ+z8=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;  IPV:NLI; SFV:NSPM; H:LEJPR01MB0635.DEUPRD01.PROD.OUTLOOK.DE; PTR:; CAT:NONE;  SFS:(39860400002)(376002)(346002)(396003)(366004)(136003)(110136005)(5660300002)(8676002)(478600001)(966005)(86362001)(2906002)(76116006)(64756008)(316002)(66446008)(66556008)(66476007)(66946007)(9686003)(71200400001)(7116003)(55016002)(53546011)(3480700007)(83380400001)(33656002)(4326008)(8936002)(186003)(7696005)(26005); DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata: t5zw+cGACX3khHKG37HmZ1II6Meu8krlzjBqbq+lvwOv47ksKV2M+xkJjSVEzyIB5X5lV+bW39NndrAUurkCO6uDPyboKJ5/Yn7/S0mB6Y0oO8Ot33zGwjphKZI5EDpec2Ryz3DanRLiRupi+sFO3SVIe0VgkhKriceA26J8qWSzXcR3NLxyzQ5IxgVckZWx/H8K9mJeVWpxk0+8e7SLqavh57bYOzqfW7wD8oMLqjMxl0Nqqc3ScPgbW/ZmZTZA717FwkCIcEHPRBsrt4xjCqI6Y901HKq7pL67U8pUjUjru1yXNroC1sROsfLwS6BvD7HqST0g2onDrcAicDOGDD5hhdiekb53osZTKmnxUExTKBc4l4DdAwGDXlrSmkIrMYzPOeHrgWP0BwknV6MfvQqc+P/p0LtIzc05WhxX7lH8/naT0I1U1+kqftdZjsaELQc1BVI/l+ltWlwN7aEVGcmD/8RsEwldHGDHljJ1ALvXPSMTUqCkyl61TyjeYk6xZHgLqT6pIO6dVq2NRx21rvL1eMZl2jrnPqAa/bwyrU9TMO+LTl+ylbGDSNPP9ni7RVLTtKzpHeh/JBTVrWx550lry/d/CGAeeLnNI2mbDb/ggA5ZLQ5JBsqFO/U/8Yk7+iACVIK+Vw7IBJmTmDEUTg==
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: LEJPR01MB0635.DEUPRD01.PROD.OUTLOOK.DE
X-MS-Exchange-CrossTenant-Network-Message-Id: 5bfc67fc-7c7e-40aa-0d5e-08d88a48e3cb
X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Nov 2020 16:01:33.5369 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bde4dffc-4b60-4cf6-8b04-a5eeb25f5c4f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: OsiHbfVBbgg9ytv9eQD7D0M98I7FFUfJpNJr+5NgoSwJHuwxt2Vm1ZZTebCd+YeQ13PsRU7MqW0BW0sZ2CirPA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LEJPR01MB0748
X-TM-SNTS-SMTP: 64D2BE1B28B90255025FFE9198393B1C9EA5393515C86FE71B7EE78B84B4ADA72000:8
X-OriginatorOrg: telekom.de
Archived-At: 
Subject: Re: [tcpm] MPTCP RobE
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 16:01:40 -0000

SG8gT2xpdmllciwNCg0KU2VlIGlubGluZQ0KDQo+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0t
DQo+IEZyb206IE9saXZpZXIgQm9uYXZlbnR1cmUgPE9saXZpZXIuQm9uYXZlbnR1cmVAdWNsb3V2
YWluLmJlPg0KPiBTZW50OiBNb250YWcsIDE2LiBOb3ZlbWJlciAyMDIwIDE0OjA4DQo+IFRvOiBB
bWVuZCwgTWFya3VzIDxNYXJrdXMuQW1lbmRAdGVsZWtvbS5kZT47IHRjcG1AaWV0Zi5vcmcNCj4g
Q2M6IGthbmdqaWFvQGh1YXdlaS5jb20NCj4gU3ViamVjdDogUmU6IE1QVENQIFJvYkUNCj4gDQo+
IE1hcmt1cywgSmlhbywNCj4gPg0KPiA+IGFzIG9uZSBvZiB0aGUgYXV0aG9ycyBvZiBodHRwczov
L2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9kcmFmdC1hbWVuZC10Y3BtLQ0KPiBtcHRjcC1yb2Jl
LywgSSBraW5kbHkgcmVxdWVzdCBmZWVkYmFjayBmcm9tIHlvdSwgaW4gcGFydGljdWxhciBob3dl
dmVyIGZyb20gdGhlDQo+IFJGQzY4MjQvODY4NCBhdXRob3JzIGFuZCBNUFRDUCBpbXBsZW1lbnRl
cnMuDQo+ID4NCj4gPiBXaXRoIE1QVENQIFJvYkUgKFJvYnVzdCBFc3RhYmxpc2htZW50KSB3ZSBh
ZGRyZXNzIHRoZSBsYWNrIG9mIE1QVENQIHRvDQo+IHJlbGlhYmx5IHNldHVwIGEgTVBUQ1Agc2Vz
c2lvbiwgd2hlbiB0aGUgc2VsZWN0ZWQgcGF0aCBmb3IgdGhlIGluaXRpYWwgc2Vzc2lvbg0KPiBz
ZXR1cCBpcyBtYWxmdW5jdGlvbmVkLiBUaGUgTVBUQ1AgUkZDcyBkb2VzIG5vdCBnaXZlIGFueSBn
dWlkZWxpbmVzIGZvciB0aGlzDQo+IHBhcnRpY3VsYXIgcHJvYmxlbSBhbmQgaXQgaXMgbGVmdCB0
byB0aGUgaW1wbGVtZW50ZXIgdG8gdGFrZSBjYXJlLiBGb3IgdGhhdCB3ZQ0KPiBwcm9wb3NlIGFu
ZCBjb21wYXJlIGRpZmZlcmVudCBzdHJhdGVnaWVzIGluIHRoZSBkcmFmdCBkb2N1bWVudCwgc2Vw
YXJhdGVkIGJ5DQo+IHRoZWlyIGRlbWFuZCBvbiBNUFRDUCBwcm90b2NvbCBjaGFuZ2VzLiBBcGFy
dCBmcm9tIHRoYXQgb3VyIGNsYWltIGlzLCB0aGF0IHdpdGgNCj4gdGhlc2UgcHJvcG9zYWxzIGNv
bXBhcmVkIHRvIE1QVENQLCBubyBhZGRpdGlvbmFsIHNlY3VyaXR5IGZsYXdzIGFyZSByYWlzZWQg
YW5kDQo+IGFueSBwb3RlbnRpYWwgY29tcHV0YXRpb25hbCBvdmVyaGVhZCBpcyBtaW5pbWl6ZWQu
DQo+ID4NCj4gDQo+IFNvcnJ5IGZvciB0aGUgbGF0ZSByZXBseS4gSSBoYXZlIGxvb2tlZCBhdCB0
aGUgZG9jdW1lbnQgYW5kIGZvdW5kIHRoYXQNCj4gaGFuZGxpbmcgYm90aCBSRkM2ODI0IGFuZCBS
RkM4Njg0IGFkZHMgYSBsb3Qgb2YgY29tcGxleGl0eSB0byB0aGUNCj4gZG9jdW1lbnQuIFNpbmNl
IFJGQzg2ODQgaXMgdGhlIHN0YW5kYXJkcyB0cmFjayB2ZXJzaW9uIG9mIE1QVENQLCBJIHdvdWxk
DQo+IHN1Z2dlc3QgdG8gb25seSBkaXNjdXNzIHRoaXMgdmVyc2lvbiBvZiBNUFRDUCBpbiB0aGUg
ZG9jdW1lbnQuDQo+IA0KDQpHb29kIGlkZWEsIHdvdWxkIHNpbXBsaWZ5IHNvbWUgdGhpbmdzLiBI
b3dldmVyIG1vc3Qgb2Ygb3VyIHByb3Bvc2FscyBhcmUgTVBUQ1AgdmVyc2lvbiBhZ25vc3RpYywg
ZXhwZWN0aW5nIFJvYkUgZVNJTS4NCg0KPiBNeSBtYWluIGNvbmNlcm4gYWJvdXQgdGhlIHByb3Bv
c2VkIGFwcHJvYWNoIGlzIGhvdyB0aGlzIHdpbGwgd29yayB3aXRoDQo+IGxvYWQtYmFsYW5jZXJz
LiBXaGVuIHRoZXJlIGlzIGEgbG9hZC1iYWxhbmNlciBpbiBmcm9udCBvZiBhIHBvb2wgb2YNCj4g
c2VydmVycyB0aGF0IHVzZSB0aGUgc2FtZSBJUCBhZGRyZXNzLCB0aGVyZSBpcyBubyBndWFyYW50
ZWUgdGhhdCBTWU5zDQo+IGZyb20gdGhlIHNhbWUgY2xpZW50IHdpbGwgYmUgZGVsaXZlcmVkIHRv
IHRoZSBzYW1lIHNlcnZlciwgZXNwZWNpYWxseSBpZg0KPiB0aGVzZSBTWU5zIGFyZSBzZW50IG92
ZXIgZGlmZmVyZW50IHBhdGhzIGFuZCB0aHVzIHdpdGggZGlmZmVyZW50IHNvdXJjZQ0KPiBhZGRy
ZXNzZXMuIElmIHRoZSB0d28gY29ubmVjdGlvbnMgdGhhdCBSb2JFIHRyaWVzIHRvIGVzdGFibGlz
aCB0ZXJtaW5hdGUNCj4gb24gZGlmZmVyZW50IHBoeXNpY2FsIHNlcnZlcnMsIHRoZXkgY2Fubm90
IGJlIGpvaW5lZC4gR2l2ZW4gdGhlDQo+IGltcG9ydGFuY2Ugb2YgbG9hZCBiYWxhbmNlcnMsIEkg
d291bGQgcHJlZmVyIGEgc29sdXRpb24gd2hlcmUgdGhlDQo+IGFwcGxpY2F0aW9uIHNpbXBseSB0
cmllcyB0byBvcGVuIHR3byBwYXJhbGxlbCBtcHRjcCBjb25uZWN0aW9ucywgcHJlZmVycw0KPiB0
aGUgZmlyc3QgdGhhdCBnZXRzIGVzdGFibGlzaGVkLCBhZGRzIGFub3RoZXIgc3ViZmxvdyB0byB0
aGlzIGNvbm5lY3Rpb24NCj4gYW5kIHJlc2V0cyB0aGUgc2Vjb25kIG9uZS4gVGhpcyB3b3VsZCBz
bGlnaHRseSBpbmNyZWFzZSB0aGUgZGVsYXkNCj4gY29tcGFyZWQgdG8gUm9iRSwgYnV0IHdvdWxk
IHByZXNlcnZlIGxvYWQgYmFsYW5jZXJzLiBJbiBwcmFjdGljZSwgSQ0KPiBleHBlY3QgdGhhdCBN
UFRDUCBzdGFja3Mgd2lsbCBsZXZlcmFnZSBpbmZvcm1hdGlvbiBhYm91dCB0aGUgY3VycmVudA0K
PiBxdWFsaXR5IG9mIHRoZSBhY2Nlc3MgbmV0d29ya3MgKGUuZy4gV2lGaSBhbmQgeEcgb24gc21h
cnRwaG9uZXMpIGFuZA0KPiB3aWxsIG9ubHkgYXR0ZW1wdCBwYXJhbGxlbCBjb25uZWN0aW9ucyB3
aGVuIHRoZSBjb25kaXRpb25zIGFyZQ0KPiB1bmNlcnRhaW4uIEFwcGxlJ3MgV2lGaSBhc3Npc3Qg
YWxyZWFkeSBpbmNsdWRlcyBzdWNoIGZ1bmN0aW9ubmFsaXR5Lg0KPiANCg0KWWVzLCB0aGF0IGlz
IGEgdmFsaWQgcG9pbnQsIGJ1dCBzaW1pbGFyIHRvIG15IGFuc3dlciBhYm92ZSwgcHJvYmFibHkg
b25seSBhZmZlY3RzIHRoZSBSb2JFIGVTSU0gb3V0IG9mIGEgYnVuY2ggb2YgcHJvcG9zYWxzLiBT
aW5jZSBSb2JFIGVTSU0gaXMgYW4gb3B0aW9uYWwgZmVhdHVyZSwgd2hpY2ggcmVxdWlyZXMgcmVj
ZWl2ZXIgKGxvYWQgYmFsYW5jZXIpIHN1cHBvcnQsIGxvYWQgYmFsYW5jZXJzIGNhbiBpbXBsaWNp
dGx5IGRlY2xpbmUgc3VwcG9ydCAoc2VlIE1QX0pPSU5fQ0FQIG9yIFJPQkVfZVNJTV9FTikuDQpG
cm9tIG15IHBlcnNwZWN0aXZlIEkgc2VlIG1vc3QgdmFsdWUgb2YgUm9iRSBlU0lNIGluIGUuZy4g
M0dQUCBBVFNTUyAoUHJveHkgc29sdXRpb25zKSBvciByZWFsIGVuZC10by1lbmQgTVBUQ1Agc2Vz
c2lvbnMuDQoNCj4gQmVzdCByZWdhcmRzLA0KPiANCj4gDQo+IE9saXZpZXINCg==


From nobody Mon Nov 16 08:27:02 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AB3A13A127D for ; Mon, 16 Nov 2020 08:27:00 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A1TCFg2sfEK3 for ; Mon, 16 Nov 2020 08:26:48 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2F5A73A127A for ; Mon, 16 Nov 2020 08:26:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:To:Subject:Sender:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=axnmvtFz7lf9+s9zDEu7vqL0ZT14Xe2BHMZKvS5nWZo=; b=TFH535SeTIrrBh41pJa63THq2 AVxveXE+UQv20n0UgdS5dm6wm9c5Fvsj09LKP+kHLWgtSCW4UUeERVugHJP5PpZ3g2p2nrJEnNOEX fcKqaFiZCTzaNlv3+3LUNd86Y61gCSiZi4uAxkvfuOO3WBbCDh2JrZmTL+jWORX7X4k6PFJxJwkkl 6AxbtDn+XTDvv5kDiiZvmFon349Uji5qaPtFtt7ov0+AmvMQwWEUcfnoEam+abDFdDBeIE+aKtKUp sLImJpCdYJfB8hgNTHVDY9s+M96mILfSj/XE11oMY6GDtQK2LQ0aguY4b6N2BdYmZ7X65ht2lTIVU DnJktU5GQ==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:33990 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kehL3-006UmV-MJ; Mon, 16 Nov 2020 16:26:46 +0000
To: Yoshifumi Nishida , "tcpm@ietf.org Extensions" 
References: 
From: Bob Briscoe 
Message-ID: 
Date: Mon, 16 Nov 2020 16:26:45 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------ED3F3044BCC62FCC8822913A"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] bit length for byte counter fields in accurate ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 16:27:01 -0000

This is a multi-part message in MIME format.
--------------ED3F3044BCC62FCC8822913A
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Yoshi,

While searching for a recent email from you just now, I found this one 
unanswered. My apologies.
In case you are still interested in the answer... inline tagged [BB]...

On 30/04/2020 10:06, Yoshifumi Nishida wrote:
> Hi,
> I am still a bit wondering about the length of the field for the byte 
> counter.
> In my understanding, the length of the counter is (at least) 32 bits 
> and encoded into 24 bits field.
> So, I am thinking that reducing the bit size doesn't affect accuracy 
> directly while the potential risks for miscalculation will be 
> increased. Is this incorrect?

Here's the calculations we went through to determine the 24b field size.

Each lowest 24b of the counter are sent every time the receiver sends an 
AccECN Option. Both ends start a connection from the same value, so they 
can maintain 24-bit counters locally just with simple modulo arithmetic 
(example in appendix A). So the field only has to be large enough that, 
if it wraps, the sender can tell, even if some ACKs were lost around the 
time it wrapped.

Each feedback counter increments by the size in bytes of each data 
segment carrying that codepoint.
Theoretical max packet size is 2^16 (IPv4 & IPv6).
You detect wrap most easily by detecting the counter going from the top 
quarter of the range to the bottom quarter (i.e. perhaps highest two 
bits transitioning 0b11->0b00).
So you would have to lose 2^(24-16-2) = 2^6 = 64 ACKs of max size 
packets in a row to miss a wrap.

Losing 64 ACKs in a row is pretty unlikely.

Notice I haven't said that packets of size 2^16 are pretty unlikely. 
'Cos if we get there they won't be.
9000B is fairly common in DCs. That's >2^13.


Bob

>
> Thanks,
> --
> Yoshi
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------ED3F3044BCC62FCC8822913A
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Yoshi,

    

    While searching for a recent email from you just now, I found this
    one unanswered. My apologies.

    In case you are still interested in the answer... inline tagged
    [BB]...

    

    
On 30/04/2020 10:06, Yoshifumi Nishida
      wrote:

    

    

      
      
Hi,
        
I am still a bit wondering about the length of the field
          for the byte counter.

        
In my understanding, the length of the counter is (at
          least) 32 bits and encoded into 24 bits field.

        

        
So, I am thinking that reducing the bit size doesn't affect
          accuracy directly while the potential risks for miscalculation
          will be increased. Is this incorrect?

      

    

    

    Here's the calculations we went through to determine the 24b field
    size.

    

    Each lowest 24b of the counter are sent every time the receiver
    sends an AccECN Option. Both ends start a connection from the same
    value, so they can maintain 24-bit counters locally just with simple
    modulo arithmetic (example in appendix A). So the field only has to
    be large enough that, if it wraps, the sender can tell, even if some
    ACKs were lost around the time it wrapped.

    

    Each feedback counter increments by the size in bytes of each data
    segment carrying that codepoint. 

    Theoretical max packet size is 2^16 (IPv4 & IPv6). 

    You detect wrap most easily by detecting the counter going from the
    top quarter of the range to the bottom quarter (i.e. perhaps highest
    two bits transitioning 0b11->0b00).

    So you would have to lose 2^(24-16-2) = 2^6 = 64 ACKs of max size
    packets in a row to miss a wrap.

    

    Losing 64 ACKs in a row is pretty unlikely.

    

    Notice I haven't said that packets of size 2^16 are pretty unlikely.
    'Cos if we get there they won't be. 

    9000B is fairly common in DCs. That's >2^13.

    

    

    Bob

    

    

      

        


        

        
Thanks,

        
--

        
Yoshi 

      

      

      

      
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------ED3F3044BCC62FCC8822913A--


From nobody Mon Nov 16 08:52:28 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3ABC53A13AA; Mon, 16 Nov 2020 08:52:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1C53RZd0AeE6; Mon, 16 Nov 2020 08:52:20 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A12A03A12B4; Mon, 16 Nov 2020 08:51:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:References:Cc:To:Subject:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=F0qtAn67cI8O90fPtt7aTqSVvKVqLldN5g1GR0Bpl4o=; b=H05hHDeIzBJGFmTX0jKNq/KAn ftkr4Q9dkenKR7rKK3E07AK9/XcYco32fdyyYYFaqxPSf5Y3Mgm2gO94rMRC3SxALq0D7WxGMtxVm sNT2AmLGmIo0qtAWsHofhmiij9ZV6bpxXICLhQGtbN3443PTIX9k0TCxSLGb1UmuXwUQbyh5bhtBw /3gSB2KqfN7FC7kGlDmTHGoK3VSXAa6/XHivx0qkfyNyXeNf/Nh1qnm/0PyfE3n2wPypfiAoZ0V7C pziRpzmCeXo9UVTq0qWM2pDrnwOueXCdpHp3KNfBejTGErD0kHflNagr+GuqkL4eOGm+Pi9rEGdzX pp/yzC/EQ==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:34130 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kehjD-006mH3-9D; Mon, 16 Nov 2020 16:51:44 +0000
From: Bob Briscoe 
To: Yoshifumi Nishida 
Cc: "Scharf, Michael" , Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net> 
Message-ID: <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>
Date: Mon, 16 Nov 2020 16:51:42 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------0705A85F7D664DA337EE6B85"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 16:52:27 -0000

This is a multi-part message in MIME format.
--------------0705A85F7D664DA337EE6B85
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Yoshi, (adding the tcpm list in cc)

On 05/11/2020 06:58, Yoshifumi Nishida wrote:
> Hi Bob,
>
> On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe  > wrote:
>
>     Yoshi,
>
>     On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>>     Hi, folks,
>

>>     In my understanding, I'm not sure if we settled down on using two
>>     option kinds or encoding schemes for 24bits fields in acc ecn draft.
>>     So, I think there're still something to be clarified and hope
>>     things will be settled at the meeting.
>
>     [BB] I know a WG can change it's mind at any time. But I'd rather
>     we just clarified what a previous decision was, to avoid the need
>     to keep re-opening discussion on a question that have been decided
>     then changed three different ways already.
>
>     My memory is not so good these days. I trusted that Michael S
>     remembered the decision correctly, and I seem to remember that
>     decision being made.
>     I've just checked the minutes of the last interim:
>     https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/
>     and they mention Michael's proposal to use two kinds, but don't
>     record any decision.
>     The jabber log gives no clues about any decision.
>
>     I can't find an audio or video recording. Can you point me at one?
>
>
> I thought that it's because there was no clear decision at the meeting.
> But, you can check https://www.youtube.com/watch?v=KsB0_Nb8-kA
> Please let us know if you have any questions or opinions with regard 
> to this.

[BB] I checked the Youtube link you sent below.

First I think we're agreed no-one was fighting for us to keep the 
previous way we did this (using the initial value of the field to set 
the order for the connection).
In my presentation I said there was strong resistance from Michael to do 
it a different way.
(also, offlist, the co-authors including me also didn't like this so 
much. And Ilpo said it made the implementation complex.)

Then came the question of what we do instead. There were three 
alternative proposals:
a) use 2 option kinds
b) add a flags byte
c) somehow use the length field maybe

Michael raised (c) in the meeting as a possibility, but no-one could 
think how to distinguish two options of the same length but a different 
field order using the length field. Michael said he'd post any ideas to 
the list if he could think of any, but that didn't happen.

So we're left choosing between (a) and (b).
I said in the meeting (and on the list when discussing with Ilpo) that 
I'd be happy to go with (b), but only if there was another use for a 
flag. Because it would consume 1B more options space in many packets, 
which is a scarce resource.

Ilpo had a proposed use for another flag (to help synch counters after a 
loss), but I think the discussion about it ended that it wouldn't be 
helpful, 'cos the way it worked depended on itself (circular logic).

In conclusion, I don't think there was an explicit decision to go with 2 
option kinds, but it ended up as the 'last person standing'.
I like it. It's simple. And apparently option kinds are not such a 
scarce resource.

Perhaps we can ratify this in the WG tomorrow.


Bob

>
> Thanks,
> --
> Yoshi

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/


--------------0705A85F7D664DA337EE6B85
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Yoshi, (adding the tcpm list in cc)

    

    
On 05/11/2020 06:58, Yoshifumi Nishida
      wrote:

    

    

      
      

        
Hi Bob,

        


        

        

          
On Wed, Nov 4, 2020 at 3:29
            PM Bob Briscoe <ietf@bobbriscoe.net> wrote:

          

          

            
 Yoshi,

              

              
On 04/11/2020 06:51, Yoshifumi Nishida wrote:

              

              

                

                  
Hi, folks,

                

              

            

          

        

      

    

    

    

      

        

          

            

              

                

                  
In my understanding, I'm not sure if we settled
                    down on using two option kinds or encoding schemes
                    for 24bits fields in acc ecn draft.

                  
So, I think there're still something to be
                    clarified and hope things will be settled at the
                    meeting.

                

              

              

              [BB] I know a WG can change it's mind at any time. But I'd
              rather we just clarified what a previous decision was, to
              avoid the need to keep re-opening discussion on a question
              that have been decided then changed three different ways
              already.

              

              My memory is not so good these days. I trusted that
              Michael S remembered the decision correctly, and I seem to
              remember that decision being made.

              I've just checked the minutes of the last interim:

                  https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/

              and they mention Michael's proposal to use two kinds, but
              don't record any decision.

              The jabber log gives no clues about any decision.

              

              I can't find an audio or video recording. Can you point me
              at one?

            

          

          


          

          
I thought that it's because there was no clear decision
            at the meeting. 

          
          
Please let us know if you have any questions or opinions
            with regard to this.

        

      

    

    

    [BB] I checked the Youtube link you sent below.

    

    First I think we're agreed no-one was fighting for us to keep the
    previous way we did this (using the initial value of the field to
    set the order for the connection). 

    In my presentation I said there was strong resistance from Michael
    to do it a different way. 

    (also, offlist, the co-authors including me also didn't like this so
    much. And Ilpo said it made the implementation complex.)

    

    Then came the question of what we do instead. There were three
    alternative proposals:

    a) use 2 option kinds

    b) add a flags byte

    c) somehow use the length field maybe

    

    Michael raised (c) in the meeting as a possibility, but no-one could
    think how to distinguish two options of the same length but a
    different field order using the length field. Michael said he'd post
    any ideas to the list if he could think of any, but that didn't
    happen.

    

    So we're left choosing between (a) and (b).

    I said in the meeting (and on the list when discussing with Ilpo)
    that I'd be happy to go with (b), but only if there was another use
    for a flag. Because it would consume 1B more options space in many
    packets, which is a scarce resource.

    

    Ilpo had a proposed use for another flag (to help synch counters
    after a loss), but I think the discussion about it ended that it
    wouldn't be helpful, 'cos the way it worked depended on itself
    (circular logic).

    

    In conclusion, I don't think there was an explicit decision to go
    with 2 option kinds, but it ended up as the 'last person standing'.

    I like it. It's simple. And apparently option kinds are not such a
    scarce resource.

    

    Perhaps we can ratify this in the WG tomorrow.

    

    

    Bob

    

    

      

        

          


          

          
Thanks,

          
--

          
Yoshi

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------0705A85F7D664DA337EE6B85--


From nobody Mon Nov 16 09:36:09 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2C1953A09DB; Mon, 16 Nov 2020 09:36:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e08zb2QsuvB6; Mon, 16 Nov 2020 09:36:05 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id F34C33A1337; Mon, 16 Nov 2020 09:36:04 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 1E3BB25A12; Mon, 16 Nov 2020 18:36:03 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1605548163; bh=K9/FSl4wZSWXbeeXJgZtPUPSgZvndsyUurN4gJyV3lo=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=s2hIDOXIMJgOY4LHZzPidOGAf/nhhL3kRmeYAhy+xMUGDQ+bKqdDFMxY6hFt7uF8h waRAx8Wwk8VJe87W2k4Nneugq/5dUISgO3ErjXthMKRxxM1XB/205AlGYFFlFeX4jk ydOcDpXoYfGaJFVi4bemU8f8WSBZr4MxLG3j65P4=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YpXdUxGs5qK2; Mon, 16 Nov 2020 18:36:01 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Mon, 16 Nov 2020 18:36:01 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Mon, 16 Nov 2020 18:36:01 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Mon, 16 Nov 2020 18:36:01 +0100
From: "Scharf, Michael" 
To: Bob Briscoe , Yoshifumi Nishida 
CC: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
Thread-Topic: AccECN field order
Thread-Index: AQHWvDjFKAN34ufmsUi0uesNI2lMmanK+nPQ
Date: Mon, 16 Nov 2020 17:36:00 +0000
Message-ID: 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>
In-Reply-To: <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: multipart/alternative; boundary="_000_b3ae297684b04461be4e5ef5bbe3c83ahsesslingende_"
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 17:36:08 -0000

--_000_b3ae297684b04461be4e5ef5bbe3c83ahsesslingende_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

Qm9iLA0KDQpPbmUgcHJvcG9zYWwgdXNpbmcgdGhlIGxlbmd0aCBmaWVsZCB3aXRoICpvbmUgb3B0
aW9uIGNvZGVwb2ludCBvbmx5KiBpcyBkZXRhaWxlZCBpbjogaHR0cHM6Ly9tYWlsYXJjaGl2ZS5p
ZXRmLm9yZy9hcmNoL21zZy90Y3BtL3pvLTFPUjBuUmZoSG9jWDh5dlR2cEM0Qk5Nby8NCg0KSXQg
aXMgdGhlIHRoaXJkIG9wdGlvbiBtZW50aW9uZWQgaW4gdGhpcyBlLW1haWwuIE9uZSBleGFtcGxl
IHdvdWxkIGJlIHRvIHVzZSBvcHRpb24gbGVuZ3RoIHZhbHVlcyA1LzgvMTEgZm9yIG9uZSBlbmNv
ZGluZyB0eXBlIGFuZCBvcHRpb24gbGVuZ3RoIHZhbHVlcyA2LzkvMTIgZm9yIHRoZSBvdGhlciBl
bmNvZGluZyB0eXBlIChpLmUuLCBvcmRlciBvZiBmaWVsZHMpLiBPciBvbmUgY291bGQgdXNlIHNv
bWUgb3RoZXIgY29tYmluYXRpb24gb2YgbGVuZ3RoIHZhbHVlcyDigJMgdGhlIG9ubHkgcmVxdWly
ZW1lbnQgaXMgdGhhdCBhIGNlcnRhaW4gdmFsdWUgZm9yIHRoZSBvcHRpb24gbGVuZ3RoIGlzIG9u
bHkgdXNlZCBieSBvbmUgb2YgdGhlIG9wdGlvbiBmb3JtYXRzLiBJbiB0aGF0IGFwcHJvYWNoLCB0
aGUgdmFsdWUgb2YgdGhlIGxlbmd0aCBmaWVsZCB3b3VsZCB0aHVzIGRpcmVjdGx5IGRlc2NyaWJl
IHRoZSBlbmNvZGluZyBvZiB0aGUgb3B0aW9uLiBVbmxlc3MgSSBtaXNzIHNvbWV0aGluZywgdGhp
cyB3b3VsZCB3b3JrIGFuZCBpdCB3b3VsZCBqdXN0IHJlcXVpcmUgb25lIG9wdGlvbiBjb2RlcG9p
bnQuDQoNClRodXMsIGFsdGVybmF0aXZlcyB0byB0d28gb3B0aW9uIGNvZGVwb2ludHMgZXhpc3Qg
YW5kIEkgaGF2ZSBleHBsYWluZWQgdGhlbSBvbiB0aGUgbGlzdCBpbiBNYXJjaCAyMDIwLg0KDQpB
bnl3YXksIEkgZG9u4oCZdCByZWFsbHkgY2FyZSBob3cgdGhlIG9wdGlvbnMgYXJlIGVuY29kZWQg
YXMgbG9uZyBhcyB0aGUgcmVjZWl2ZXIgZG9lc27igJl0IG5lZWQgcGVyLWNvbm5lY3Rpb24gc3Rh
dGUgZm9yIGRlY29kaW5nIGEgVENQIG9wdGlvbi4gU28sIHBlcnNvbmFsbHksIEkgd291bGQgYmUg
ZmluZSB3aXRoIHVzaW5nIGUuZy4gdGhlIGxlbmd0aCBmaWVsZCBhcyBkZXNjcmliZWQgaW4gbXkg
b2xkIGUtbWFpbC4gT3IgYW4gYWRkaXRpb25hbCBmbGFnIGJ5dGUuIEFuZCBvbmUgY291bGQgY29t
ZSB1cCB3aXRoIGZ1cnRoZXIgZW5jb2RpbmdzLCBlLmcuLCBieSB1c2luZyBvbmUgb3IgYSBmZXcg
Yml0cyBhcyBhIHNob3J0IOKAnHR5cGXigJ0gZmllbGQgZm9yIGVhY2ggY291bnRlci4gVGhpcyBp
cyBhbGwgYWJvdXQgcHJvdG9jb2wgZW5naW5lZXJpbmcuIEFuZCBhbGwgdGhlc2UgdmFyaWFudHMg
aGF2ZSB0aGVpciBwcm9zIGFuZCBjb25zLg0KDQpJIGFtIGFsc28gZmluZSB3aXRoIHVzaW5nIHR3
byBvcHRpb24gY29kZXBvaW50cyBhcyBzcGVjaWZpZWQgaW4gLTEzOyB0aGlzIGlzIHByb2JhYmx5
IHRoZSBhcHByb2FjaCB0aGF0IGNvbnN1bWVzIHRoZSBsZWFzdCBudW1iZXIgb2YgYml0cy4NCg0K
TWljaGFlbCAody9vIGFueSBoYXQpDQoNCg0KDQpGcm9tOiBCb2IgQnJpc2NvZSA8aWV0ZkBib2Ji
cmlzY29lLm5ldD4NClNlbnQ6IE1vbmRheSwgTm92ZW1iZXIgMTYsIDIwMjAgNTo1MiBQTQ0KVG86
IFlvc2hpZnVtaSBOaXNoaWRhIDxuc2QuaWV0ZkBnbWFpbC5jb20+DQpDYzogU2NoYXJmLCBNaWNo
YWVsIDxNaWNoYWVsLlNjaGFyZkBocy1lc3NsaW5nZW4uZGU+OyBNaWNoYWVsIFR1ZXhlbiA8dHVl
eGVuQGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IE1pcmphIEt1ZWhsZXdp
bmQgPGlldGZAa3VlaGxld2luZC5uZXQ+OyBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxSaWNoYXJk
LlNjaGVmZmVuZWdnZXJAbmV0YXBwLmNvbT47IHRjcG0gSUVURiBsaXN0IDx0Y3BtQGlldGYub3Jn
Pg0KU3ViamVjdDogQWNjRUNOIGZpZWxkIG9yZGVyDQoNCllvc2hpLCAoYWRkaW5nIHRoZSB0Y3Bt
IGxpc3QgaW4gY2MpDQpPbiAwNS8xMS8yMDIwIDA2OjU4LCBZb3NoaWZ1bWkgTmlzaGlkYSB3cm90
ZToNCkhpIEJvYiwNCg0KT24gV2VkLCBOb3YgNCwgMjAyMCBhdCAzOjI5IFBNIEJvYiBCcmlzY29l
IDxpZXRmQGJvYmJyaXNjb2UubmV0PG1haWx0bzppZXRmQGJvYmJyaXNjb2UubmV0Pj4gd3JvdGU6
DQpZb3NoaSwNCk9uIDA0LzExLzIwMjAgMDY6NTEsIFlvc2hpZnVtaSBOaXNoaWRhIHdyb3RlOg0K
SGksIGZvbGtzLA0KDQoNCkluIG15IHVuZGVyc3RhbmRpbmcsIEknbSBub3Qgc3VyZSBpZiB3ZSBz
ZXR0bGVkIGRvd24gb24gdXNpbmcgdHdvIG9wdGlvbiBraW5kcyBvciBlbmNvZGluZyBzY2hlbWVz
IGZvciAyNGJpdHMgZmllbGRzIGluIGFjYyBlY24gZHJhZnQuDQpTbywgSSB0aGluayB0aGVyZSdy
ZSBzdGlsbCBzb21ldGhpbmcgdG8gYmUgY2xhcmlmaWVkIGFuZCBob3BlIHRoaW5ncyB3aWxsIGJl
IHNldHRsZWQgYXQgdGhlIG1lZXRpbmcuDQoNCltCQl0gSSBrbm93IGEgV0cgY2FuIGNoYW5nZSBp
dCdzIG1pbmQgYXQgYW55IHRpbWUuIEJ1dCBJJ2QgcmF0aGVyIHdlIGp1c3QgY2xhcmlmaWVkIHdo
YXQgYSBwcmV2aW91cyBkZWNpc2lvbiB3YXMsIHRvIGF2b2lkIHRoZSBuZWVkIHRvIGtlZXAgcmUt
b3BlbmluZyBkaXNjdXNzaW9uIG9uIGEgcXVlc3Rpb24gdGhhdCBoYXZlIGJlZW4gZGVjaWRlZCB0
aGVuIGNoYW5nZWQgdGhyZWUgZGlmZmVyZW50IHdheXMgYWxyZWFkeS4NCg0KTXkgbWVtb3J5IGlz
IG5vdCBzbyBnb29kIHRoZXNlIGRheXMuIEkgdHJ1c3RlZCB0aGF0IE1pY2hhZWwgUyByZW1lbWJl
cmVkIHRoZSBkZWNpc2lvbiBjb3JyZWN0bHksIGFuZCBJIHNlZW0gdG8gcmVtZW1iZXIgdGhhdCBk
ZWNpc2lvbiBiZWluZyBtYWRlLg0KSSd2ZSBqdXN0IGNoZWNrZWQgdGhlIG1pbnV0ZXMgb2YgdGhl
IGxhc3QgaW50ZXJpbToNCiAgICBodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9taW51
dGVzLWludGVyaW0tMjAyMC10Y3BtLTAxLTIwMjAwNDI5MTYwMC8NCmFuZCB0aGV5IG1lbnRpb24g
TWljaGFlbCdzIHByb3Bvc2FsIHRvIHVzZSB0d28ga2luZHMsIGJ1dCBkb24ndCByZWNvcmQgYW55
IGRlY2lzaW9uLg0KVGhlIGphYmJlciBsb2cgZ2l2ZXMgbm8gY2x1ZXMgYWJvdXQgYW55IGRlY2lz
aW9uLg0KDQpJIGNhbid0IGZpbmQgYW4gYXVkaW8gb3IgdmlkZW8gcmVjb3JkaW5nLiBDYW4geW91
IHBvaW50IG1lIGF0IG9uZT8NCg0KSSB0aG91Z2h0IHRoYXQgaXQncyBiZWNhdXNlIHRoZXJlIHdh
cyBubyBjbGVhciBkZWNpc2lvbiBhdCB0aGUgbWVldGluZy4NCkJ1dCwgeW91IGNhbiBjaGVjayBo
dHRwczovL3d3dy55b3V0dWJlLmNvbS93YXRjaD92PUtzQjBfTmI4LWtBDQpQbGVhc2UgbGV0IHVz
IGtub3cgaWYgeW91IGhhdmUgYW55IHF1ZXN0aW9ucyBvciBvcGluaW9ucyB3aXRoIHJlZ2FyZCB0
byB0aGlzLg0KDQpbQkJdIEkgY2hlY2tlZCB0aGUgWW91dHViZSBsaW5rIHlvdSBzZW50IGJlbG93
Lg0KDQpGaXJzdCBJIHRoaW5rIHdlJ3JlIGFncmVlZCBuby1vbmUgd2FzIGZpZ2h0aW5nIGZvciB1
cyB0byBrZWVwIHRoZSBwcmV2aW91cyB3YXkgd2UgZGlkIHRoaXMgKHVzaW5nIHRoZSBpbml0aWFs
IHZhbHVlIG9mIHRoZSBmaWVsZCB0byBzZXQgdGhlIG9yZGVyIGZvciB0aGUgY29ubmVjdGlvbiku
DQpJbiBteSBwcmVzZW50YXRpb24gSSBzYWlkIHRoZXJlIHdhcyBzdHJvbmcgcmVzaXN0YW5jZSBm
cm9tIE1pY2hhZWwgdG8gZG8gaXQgYSBkaWZmZXJlbnQgd2F5Lg0KKGFsc28sIG9mZmxpc3QsIHRo
ZSBjby1hdXRob3JzIGluY2x1ZGluZyBtZSBhbHNvIGRpZG4ndCBsaWtlIHRoaXMgc28gbXVjaC4g
QW5kIElscG8gc2FpZCBpdCBtYWRlIHRoZSBpbXBsZW1lbnRhdGlvbiBjb21wbGV4LikNCg0KVGhl
biBjYW1lIHRoZSBxdWVzdGlvbiBvZiB3aGF0IHdlIGRvIGluc3RlYWQuIFRoZXJlIHdlcmUgdGhy
ZWUgYWx0ZXJuYXRpdmUgcHJvcG9zYWxzOg0KYSkgdXNlIDIgb3B0aW9uIGtpbmRzDQpiKSBhZGQg
YSBmbGFncyBieXRlDQpjKSBzb21laG93IHVzZSB0aGUgbGVuZ3RoIGZpZWxkIG1heWJlDQoNCk1p
Y2hhZWwgcmFpc2VkIChjKSBpbiB0aGUgbWVldGluZyBhcyBhIHBvc3NpYmlsaXR5LCBidXQgbm8t
b25lIGNvdWxkIHRoaW5rIGhvdyB0byBkaXN0aW5ndWlzaCB0d28gb3B0aW9ucyBvZiB0aGUgc2Ft
ZSBsZW5ndGggYnV0IGEgZGlmZmVyZW50IGZpZWxkIG9yZGVyIHVzaW5nIHRoZSBsZW5ndGggZmll
bGQuIE1pY2hhZWwgc2FpZCBoZSdkIHBvc3QgYW55IGlkZWFzIHRvIHRoZSBsaXN0IGlmIGhlIGNv
dWxkIHRoaW5rIG9mIGFueSwgYnV0IHRoYXQgZGlkbid0IGhhcHBlbi4NCg0KU28gd2UncmUgbGVm
dCBjaG9vc2luZyBiZXR3ZWVuIChhKSBhbmQgKGIpLg0KSSBzYWlkIGluIHRoZSBtZWV0aW5nIChh
bmQgb24gdGhlIGxpc3Qgd2hlbiBkaXNjdXNzaW5nIHdpdGggSWxwbykgdGhhdCBJJ2QgYmUgaGFw
cHkgdG8gZ28gd2l0aCAoYiksIGJ1dCBvbmx5IGlmIHRoZXJlIHdhcyBhbm90aGVyIHVzZSBmb3Ig
YSBmbGFnLiBCZWNhdXNlIGl0IHdvdWxkIGNvbnN1bWUgMUIgbW9yZSBvcHRpb25zIHNwYWNlIGlu
IG1hbnkgcGFja2V0cywgd2hpY2ggaXMgYSBzY2FyY2UgcmVzb3VyY2UuDQoNCklscG8gaGFkIGEg
cHJvcG9zZWQgdXNlIGZvciBhbm90aGVyIGZsYWcgKHRvIGhlbHAgc3luY2ggY291bnRlcnMgYWZ0
ZXIgYSBsb3NzKSwgYnV0IEkgdGhpbmsgdGhlIGRpc2N1c3Npb24gYWJvdXQgaXQgZW5kZWQgdGhh
dCBpdCB3b3VsZG4ndCBiZSBoZWxwZnVsLCAnY29zIHRoZSB3YXkgaXQgd29ya2VkIGRlcGVuZGVk
IG9uIGl0c2VsZiAoY2lyY3VsYXIgbG9naWMpLg0KDQpJbiBjb25jbHVzaW9uLCBJIGRvbid0IHRo
aW5rIHRoZXJlIHdhcyBhbiBleHBsaWNpdCBkZWNpc2lvbiB0byBnbyB3aXRoIDIgb3B0aW9uIGtp
bmRzLCBidXQgaXQgZW5kZWQgdXAgYXMgdGhlICdsYXN0IHBlcnNvbiBzdGFuZGluZycuDQpJIGxp
a2UgaXQuIEl0J3Mgc2ltcGxlLiBBbmQgYXBwYXJlbnRseSBvcHRpb24ga2luZHMgYXJlIG5vdCBz
dWNoIGEgc2NhcmNlIHJlc291cmNlLg0KDQpQZXJoYXBzIHdlIGNhbiByYXRpZnkgdGhpcyBpbiB0
aGUgV0cgdG9tb3Jyb3cuDQoNCg0KQm9iDQoNCg0KDQpUaGFua3MsDQotLQ0KWW9zaGkNCg0KDQoN
Ci0tDQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX18NCg0KQm9iIEJyaXNjb2UgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgaHR0cDovL2JvYmJyaXNjb2UubmV0Lw0K

--_000_b3ae297684b04461be4e5ef5bbe3c83ahsesslingende_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDb25zb2xhczsNCglwYW5vc2UtMToyIDEx
IDYgOSAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWws
IGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0
b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixz
YW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9y
aXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwcmUN
Cgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZvcmZvcm1h
dGllcnQgWmNobiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9u
dC1zaXplOjEwLjBwdDsNCglmb250LWZhbWlseToiQ291cmllciBOZXciO30NCnNwYW4uSFRNTFZv
cmZvcm1hdGllcnRaY2huDQoJe21zby1zdHlsZS1uYW1lOiJIVE1MIFZvcmZvcm1hdGllcnQgWmNo
biI7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZvcmZv
cm1hdGllcnQiOw0KCWZvbnQtZmFtaWx5OkNvbnNvbGFzO30NCnNwYW4uRS1NYWlsRm9ybWF0dm9y
bGFnZTIwDQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OiJD
YWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1bHQN
Cgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBwdDt9DQpAcGFn
ZSBXb3JkU2VjdGlvbjENCgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2luOjcwLjg1cHQg
NzAuODVwdCAyLjBjbSA3MC44NXB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2Vj
dGlvbjE7fQ0KLS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVm
YXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48
IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxv
OmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwh
W2VuZGlmXS0tPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iREUiIGxpbms9ImJsdWUiIHZsaW5rPSJw
dXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+Qm9iLDxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+T25lIHByb3Bvc2FsIHVzaW5nIHRoZSBsZW5ndGggZmllbGQgd2l0aCAqPGI+b25l
IG9wdGlvbiBjb2RlcG9pbnQgb25seTwvYj4qIGlzIGRldGFpbGVkIGluOg0KPGEgaHJlZj0iaHR0
cHM6Ly9tYWlsYXJjaGl2ZS5pZXRmLm9yZy9hcmNoL21zZy90Y3BtL3pvLTFPUjBuUmZoSG9jWDh5
dlR2cEM0Qk5Nby8iPg0KaHR0cHM6Ly9tYWlsYXJjaGl2ZS5pZXRmLm9yZy9hcmNoL21zZy90Y3Bt
L3pvLTFPUjBuUmZoSG9jWDh5dlR2cEM0Qk5Nby88L2E+PG86cD48L286cD48L3NwYW4+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+SXQgaXMgdGhlIHRoaXJkIG9wdGlvbiBtZW50aW9uZWQgaW4gdGhpcyBlLW1haWwu
IE9uZSBleGFtcGxlIHdvdWxkIGJlIHRvIHVzZSBvcHRpb24gbGVuZ3RoIHZhbHVlcyA1LzgvMTEg
Zm9yIG9uZSBlbmNvZGluZyB0eXBlIGFuZCBvcHRpb24gbGVuZ3RoIHZhbHVlcyA2LzkvMTIgZm9y
IHRoZSBvdGhlciBlbmNvZGluZw0KIHR5cGUgKGkuZS4sIG9yZGVyIG9mIGZpZWxkcykuIE9yIG9u
ZSBjb3VsZCB1c2Ugc29tZSBvdGhlciBjb21iaW5hdGlvbiBvZiBsZW5ndGggdmFsdWVzIOKAkyB0
aGUgb25seSByZXF1aXJlbWVudCBpcyB0aGF0IGEgY2VydGFpbiB2YWx1ZSBmb3IgdGhlIG9wdGlv
biBsZW5ndGggaXMgb25seSB1c2VkIGJ5IG9uZSBvZiB0aGUgb3B0aW9uIGZvcm1hdHMuIEluIHRo
YXQgYXBwcm9hY2gsIHRoZSB2YWx1ZSBvZiB0aGUgbGVuZ3RoIGZpZWxkIHdvdWxkIHRodXMNCiBk
aXJlY3RseSBkZXNjcmliZSB0aGUgZW5jb2Rpbmcgb2YgdGhlIG9wdGlvbi4gVW5sZXNzIEkgbWlz
cyBzb21ldGhpbmcsIHRoaXMgd291bGQgd29yayBhbmQgaXQgd291bGQganVzdCByZXF1aXJlIG9u
ZSBvcHRpb24gY29kZXBvaW50LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRodXMs
IGFsdGVybmF0aXZlcyB0byB0d28gb3B0aW9uIGNvZGVwb2ludHMgZXhpc3QgYW5kIEkgaGF2ZSBl
eHBsYWluZWQgdGhlbSBvbiB0aGUgbGlzdCBpbiBNYXJjaCAyMDIwLjxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNv
LWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3Qt
bGFuZ3VhZ2U6RU4tVVMiPkFueXdheSwgSSBkb27igJl0IHJlYWxseSBjYXJlIGhvdyB0aGUgb3B0
aW9ucyBhcmUgZW5jb2RlZCBhcyBsb25nIGFzIHRoZSByZWNlaXZlciBkb2VzbuKAmXQgbmVlZCBw
ZXItY29ubmVjdGlvbiBzdGF0ZSBmb3IgZGVjb2RpbmcgYSBUQ1Agb3B0aW9uLiBTbywgcGVyc29u
YWxseSwgSSB3b3VsZCBiZSBmaW5lIHdpdGggdXNpbmcNCiBlLmcuIHRoZSBsZW5ndGggZmllbGQg
YXMgZGVzY3JpYmVkIGluIG15IG9sZCBlLW1haWwuIE9yIGFuIGFkZGl0aW9uYWwgZmxhZyBieXRl
LiBBbmQgb25lIGNvdWxkIGNvbWUgdXAgd2l0aCBmdXJ0aGVyIGVuY29kaW5ncywgZS5nLiwgYnkg
dXNpbmcgb25lIG9yIGEgZmV3IGJpdHMgYXMgYSBzaG9ydCDigJx0eXBl4oCdIGZpZWxkIGZvciBl
YWNoIGNvdW50ZXIuIFRoaXMgaXMgYWxsIGFib3V0IHByb3RvY29sIGVuZ2luZWVyaW5nLiBBbmQg
YWxsIHRoZXNlDQogdmFyaWFudHMgaGF2ZSB0aGVpciBwcm9zIGFuZCBjb25zLjxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls
ZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwv
cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZh
cmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkkgYW0gYWxzbyBmaW5lIHdpdGggdXNpbmcgdHdvIG9wdGlv
biBjb2RlcG9pbnRzIGFzIHNwZWNpZmllZCBpbiAtMTM7IHRoaXMgaXMgcHJvYmFibHkgdGhlIGFw
cHJvYWNoIHRoYXQgY29uc3VtZXMgdGhlIGxlYXN0IG51bWJlciBvZiBiaXRzLjxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHls
ZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwv
cD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZh
cmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPk1pY2hhZWwgKHcvbyBhbnkgaGF0KTxvOnA+PC9vOnA+PC9z
cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0i
bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVh
c3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3Vh
Z2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxkaXYgc3R5bGU9ImJvcmRl
cjpub25lO2JvcmRlci1sZWZ0OnNvbGlkIGJsdWUgMS41cHQ7cGFkZGluZzowY20gMGNtIDBjbSA0
LjBwdCI+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAj
RTFFMUUxIDEuMHB0O3BhZGRpbmc6My4wcHQgMGNtIDBjbSAwY20iPg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+PGI+RnJvbTo8L2I+IEJvYiBCcmlzY29lICZsdDtpZXRmQGJvYmJyaXNjb2UubmV0Jmd0
OyA8YnI+DQo8Yj5TZW50OjwvYj4gTW9uZGF5LCBOb3ZlbWJlciAxNiwgMjAyMCA1OjUyIFBNPGJy
Pg0KPGI+VG86PC9iPiBZb3NoaWZ1bWkgTmlzaGlkYSAmbHQ7bnNkLmlldGZAZ21haWwuY29tJmd0
Ozxicj4NCjxiPkNjOjwvYj4gU2NoYXJmLCBNaWNoYWVsICZsdDtNaWNoYWVsLlNjaGFyZkBocy1l
c3NsaW5nZW4uZGUmZ3Q7OyBNaWNoYWVsIFR1ZXhlbiAmbHQ7dHVleGVuQGZoLW11ZW5zdGVyLmRl
Jmd0OzsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IE1pcmphIEt1ZWhsZXdpbmQgJmx0O2lldGZAa3Vl
aGxld2luZC5uZXQmZ3Q7OyBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkICZsdDtSaWNoYXJkLlNjaGVm
ZmVuZWdnZXJAbmV0YXBwLmNvbSZndDs7IHRjcG0gSUVURiBsaXN0ICZsdDt0Y3BtQGlldGYub3Jn
Jmd0Ozxicj4NCjxiPlN1YmplY3Q6PC9iPiBBY2NFQ04gZmllbGQgb3JkZXI8bzpwPjwvbzpwPjwv
cD4NCjwvZGl2Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpw
PjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEyLjBwdCI+
WW9zaGksIChhZGRpbmcgdGhlIHRjcG0gbGlzdCBpbiBjYyk8bzpwPjwvbzpwPjwvcD4NCjxkaXY+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5PbiAwNS8xMS8yMDIwIDA2OjU4LCBZb3NoaWZ1bWkgTmlz
aGlkYSB3cm90ZTo8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1h
cmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPkhpIEJvYiw8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K
PGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIFdlZCwgTm92IDQsIDIwMjAgYXQgMzoyOSBQ
TSBCb2IgQnJpc2NvZSAmbHQ7PGEgaHJlZj0ibWFpbHRvOmlldGZAYm9iYnJpc2NvZS5uZXQiPmll
dGZAYm9iYnJpc2NvZS5uZXQ8L2E+Jmd0OyB3cm90ZTo8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0K
PGJsb2NrcXVvdGUgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkICNDQ0NDQ0Mg
MS4wcHQ7cGFkZGluZzowY20gMGNtIDBjbSA2LjBwdDttYXJnaW4tbGVmdDo0LjhwdDttYXJnaW4t
cmlnaHQ6MGNtIj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJv
dHRvbToxMi4wcHQiPllvc2hpLDxvOnA+PC9vOnA+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPk9uIDA0LzExLzIwMjAgMDY6NTEsIFlvc2hpZnVtaSBOaXNoaWRhIHdyb3RlOjxvOnA+
PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luLXRvcDo1LjBwdDtt
YXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
SGksIGZvbGtzLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4N
CjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPGJsb2NrcXVv
dGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0K
PGRpdj4NCjxibG9ja3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAj
Q0NDQ0NDIDEuMHB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7
bWFyZ2luLXJpZ2h0OjBjbSI+DQo8ZGl2Pg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6
NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPkluIG15IHVuZGVyc3RhbmRpbmcsIEknbSBub3Qgc3VyZSBpZiB3ZSBzZXR0bGVkIGRv
d24gb24gdXNpbmcgdHdvIG9wdGlvbiBraW5kcyBvciBlbmNvZGluZyBzY2hlbWVzIGZvciAyNGJp
dHMgZmllbGRzIGluIGFjYyBlY24gZHJhZnQuPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5TbywgSSB0aGluayB0aGVyZSdyZSBzdGlsbCBzb21ldGhp
bmcgdG8gYmUgY2xhcmlmaWVkIGFuZCBob3BlIHRoaW5ncyB3aWxsIGJlIHNldHRsZWQgYXQgdGhl
IG1lZXRpbmcuPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBJIGtub3cgYSBXRyBjYW4gY2hhbmdlIGl0
J3MgbWluZCBhdCBhbnkgdGltZS4gQnV0IEknZCByYXRoZXIgd2UganVzdCBjbGFyaWZpZWQgd2hh
dCBhIHByZXZpb3VzIGRlY2lzaW9uIHdhcywgdG8gYXZvaWQgdGhlIG5lZWQgdG8ga2VlcCByZS1v
cGVuaW5nIGRpc2N1c3Npb24gb24gYSBxdWVzdGlvbiB0aGF0IGhhdmUgYmVlbiBkZWNpZGVkIHRo
ZW4gY2hhbmdlZCB0aHJlZSBkaWZmZXJlbnQgd2F5cyBhbHJlYWR5Ljxicj4NCjxicj4NCk15IG1l
bW9yeSBpcyBub3Qgc28gZ29vZCB0aGVzZSBkYXlzLiBJIHRydXN0ZWQgdGhhdCBNaWNoYWVsIFMg
cmVtZW1iZXJlZCB0aGUgZGVjaXNpb24gY29ycmVjdGx5LCBhbmQgSSBzZWVtIHRvIHJlbWVtYmVy
IHRoYXQgZGVjaXNpb24gYmVpbmcgbWFkZS48YnI+DQpJJ3ZlIGp1c3QgY2hlY2tlZCB0aGUgbWlu
dXRlcyBvZiB0aGUgbGFzdCBpbnRlcmltOjxicj4NCiZuYnNwOyZuYnNwOyZuYnNwOyA8YSBocmVm
PSJodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9taW51dGVzLWludGVyaW0tMjAyMC10
Y3BtLTAxLTIwMjAwNDI5MTYwMC8iIHRhcmdldD0iX2JsYW5rIj4NCmh0dHBzOi8vZGF0YXRyYWNr
ZXIuaWV0Zi5vcmcvZG9jL21pbnV0ZXMtaW50ZXJpbS0yMDIwLXRjcG0tMDEtMjAyMDA0MjkxNjAw
LzwvYT48YnI+DQphbmQgdGhleSBtZW50aW9uIE1pY2hhZWwncyBwcm9wb3NhbCB0byB1c2UgdHdv
IGtpbmRzLCBidXQgZG9uJ3QgcmVjb3JkIGFueSBkZWNpc2lvbi48YnI+DQpUaGUgamFiYmVyIGxv
ZyBnaXZlcyBubyBjbHVlcyBhYm91dCBhbnkgZGVjaXNpb24uPGJyPg0KPGJyPg0KSSBjYW4ndCBm
aW5kIGFuIGF1ZGlvIG9yIHZpZGVvIHJlY29yZGluZy4gQ2FuIHlvdSBwb2ludCBtZSBhdCBvbmU/
PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxkaXY+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPkkgdGhvdWdodCB0aGF0IGl0J3MgYmVjYXVzZSB0aGVyZSB3YXMgbm8gY2xl
YXIgZGVjaXNpb24gYXQgdGhlIG1lZXRpbmcuJm5ic3A7PG86cD48L286cD48L3A+DQo8L2Rpdj4N
CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5CdXQsIHlvdSBjYW4gY2hlY2sgPGEgaHJlZj0i
aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1Lc0IwX05iOC1rQSI+DQpodHRwczovL3d3
dy55b3V0dWJlLmNvbS93YXRjaD92PUtzQjBfTmI4LWtBPC9hPjxvOnA+PC9vOnA+PC9wPg0KPC9k
aXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+UGxlYXNlIGxldCB1cyBrbm93IGlmIHlv
dSBoYXZlIGFueSBxdWVzdGlvbnMgb3Igb3BpbmlvbnMgd2l0aCByZWdhcmQgdG8gdGhpcy48bzpw
PjwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPjxicj4NCltCQl0gSSBjaGVja2VkIHRoZSBZb3V0dWJlIGxpbmsgeW91
IHNlbnQgYmVsb3cuPGJyPg0KPGJyPg0KRmlyc3QgSSB0aGluayB3ZSdyZSBhZ3JlZWQgbm8tb25l
IHdhcyBmaWdodGluZyBmb3IgdXMgdG8ga2VlcCB0aGUgcHJldmlvdXMgd2F5IHdlIGRpZCB0aGlz
ICh1c2luZyB0aGUgaW5pdGlhbCB2YWx1ZSBvZiB0aGUgZmllbGQgdG8gc2V0IHRoZSBvcmRlciBm
b3IgdGhlIGNvbm5lY3Rpb24pLg0KPGJyPg0KSW4gbXkgcHJlc2VudGF0aW9uIEkgc2FpZCB0aGVy
ZSB3YXMgc3Ryb25nIHJlc2lzdGFuY2UgZnJvbSBNaWNoYWVsIHRvIGRvIGl0IGEgZGlmZmVyZW50
IHdheS4NCjxicj4NCihhbHNvLCBvZmZsaXN0LCB0aGUgY28tYXV0aG9ycyBpbmNsdWRpbmcgbWUg
YWxzbyBkaWRuJ3QgbGlrZSB0aGlzIHNvIG11Y2guIEFuZCBJbHBvIHNhaWQgaXQgbWFkZSB0aGUg
aW1wbGVtZW50YXRpb24gY29tcGxleC4pPGJyPg0KPGJyPg0KVGhlbiBjYW1lIHRoZSBxdWVzdGlv
biBvZiB3aGF0IHdlIGRvIGluc3RlYWQuIFRoZXJlIHdlcmUgdGhyZWUgYWx0ZXJuYXRpdmUgcHJv
cG9zYWxzOjxicj4NCmEpIHVzZSAyIG9wdGlvbiBraW5kczxicj4NCmIpIGFkZCBhIGZsYWdzIGJ5
dGU8YnI+DQpjKSBzb21laG93IHVzZSB0aGUgbGVuZ3RoIGZpZWxkIG1heWJlPGJyPg0KPGJyPg0K
TWljaGFlbCByYWlzZWQgKGMpIGluIHRoZSBtZWV0aW5nIGFzIGEgcG9zc2liaWxpdHksIGJ1dCBu
by1vbmUgY291bGQgdGhpbmsgaG93IHRvIGRpc3Rpbmd1aXNoIHR3byBvcHRpb25zIG9mIHRoZSBz
YW1lIGxlbmd0aCBidXQgYSBkaWZmZXJlbnQgZmllbGQgb3JkZXIgdXNpbmcgdGhlIGxlbmd0aCBm
aWVsZC4gTWljaGFlbCBzYWlkIGhlJ2QgcG9zdCBhbnkgaWRlYXMgdG8gdGhlIGxpc3QgaWYgaGUg
Y291bGQgdGhpbmsgb2YgYW55LCBidXQgdGhhdA0KIGRpZG4ndCBoYXBwZW4uPGJyPg0KPGJyPg0K
U28gd2UncmUgbGVmdCBjaG9vc2luZyBiZXR3ZWVuIChhKSBhbmQgKGIpLjxicj4NCkkgc2FpZCBp
biB0aGUgbWVldGluZyAoYW5kIG9uIHRoZSBsaXN0IHdoZW4gZGlzY3Vzc2luZyB3aXRoIElscG8p
IHRoYXQgSSdkIGJlIGhhcHB5IHRvIGdvIHdpdGggKGIpLCBidXQgb25seSBpZiB0aGVyZSB3YXMg
YW5vdGhlciB1c2UgZm9yIGEgZmxhZy4gQmVjYXVzZSBpdCB3b3VsZCBjb25zdW1lIDFCIG1vcmUg
b3B0aW9ucyBzcGFjZSBpbiBtYW55IHBhY2tldHMsIHdoaWNoIGlzIGEgc2NhcmNlIHJlc291cmNl
Ljxicj4NCjxicj4NCklscG8gaGFkIGEgcHJvcG9zZWQgdXNlIGZvciBhbm90aGVyIGZsYWcgKHRv
IGhlbHAgc3luY2ggY291bnRlcnMgYWZ0ZXIgYSBsb3NzKSwgYnV0IEkgdGhpbmsgdGhlIGRpc2N1
c3Npb24gYWJvdXQgaXQgZW5kZWQgdGhhdCBpdCB3b3VsZG4ndCBiZSBoZWxwZnVsLCAnY29zIHRo
ZSB3YXkgaXQgd29ya2VkIGRlcGVuZGVkIG9uIGl0c2VsZiAoY2lyY3VsYXIgbG9naWMpLjxicj4N
Cjxicj4NCkluIGNvbmNsdXNpb24sIEkgZG9uJ3QgdGhpbmsgdGhlcmUgd2FzIGFuIGV4cGxpY2l0
IGRlY2lzaW9uIHRvIGdvIHdpdGggMiBvcHRpb24ga2luZHMsIGJ1dCBpdCBlbmRlZCB1cCBhcyB0
aGUgJ2xhc3QgcGVyc29uIHN0YW5kaW5nJy48YnI+DQpJIGxpa2UgaXQuIEl0J3Mgc2ltcGxlLiBB
bmQgYXBwYXJlbnRseSBvcHRpb24ga2luZHMgYXJlIG5vdCBzdWNoIGEgc2NhcmNlIHJlc291cmNl
Ljxicj4NCjxicj4NClBlcmhhcHMgd2UgY2FuIHJhdGlmeSB0aGlzIGluIHRoZSBXRyB0b21vcnJv
dy48YnI+DQo8YnI+DQo8YnI+DQpCb2I8YnI+DQo8YnI+DQo8YnI+DQo8bzpwPjwvbzpwPjwvcD4N
CjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQi
Pg0KPGRpdj4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8
L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5UaGFua3MsPG86
cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj4tLTxvOnA+
PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+WW9zaGk8bzpw
PjwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPHByZT4tLSA8bzpw
PjwvbzpwPjwvcHJlPg0KPHByZT5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fPG86cD48L286cD48L3ByZT4NCjxwcmU+Qm9iIEJy
aXNjb2UmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsgJm5ic3A7PGEgaHJlZj0iaHR0cDovL2JvYmJyaXNjb2UubmV0LyI+aHR0cDovL2Jv
YmJyaXNjb2UubmV0LzwvYT48bzpwPjwvbzpwPjwvcHJlPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYm9k
eT4NCjwvaHRtbD4NCg==

--_000_b3ae297684b04461be4e5ef5bbe3c83ahsesslingende_--


From nobody Mon Nov 16 09:59:29 2020
Return-Path: 
X-Original-To: tcpm@ietf.org
Delivered-To: tcpm@ietfa.amsl.com
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id C85973A00C4; Mon, 16 Nov 2020 09:59:20 -0800 (PST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
From: internet-drafts@ietf.org
To: 
Cc: tcpm@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 7.22.0
Auto-Submitted: auto-generated
Precedence: bulk
Reply-To: tcpm@ietf.org
Message-ID: <160554956077.12698.6083625541071504486@ietfa.amsl.com>
Date: Mon, 16 Nov 2020 09:59:20 -0800
Archived-At: 
Subject: [tcpm] I-D Action: draft-ietf-tcpm-yang-tcp-01.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 17:59:23 -0000

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : YANG Model for Transmission Control Protocol (TCP) Configuration
        Authors         : Michael Scharf
                          Vishal Murgai
                          Mahesh Jethanandani
	Filename        : draft-ietf-tcpm-yang-tcp-01.txt
	Pages           : 22
	Date            : 2020-11-16

Abstract:
   This document specifies a YANG model for TCP on devices that are
   configured by network management protocols.  The YANG model defines a
   container for all TCP connections and groupings of some of the
   parameters that can be imported and used in TCP implementations or by
   other models that need to configure TCP parameters.  The model
   includes definitions from YANG Groupings for TCP Client and TCP
   Servers (I-D.ietf-netconf-tcp-client-server).  The model is NMDA (RFC
   8342) compliant.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-yang-tcp/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-tcpm-yang-tcp-01
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-yang-tcp-01

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-yang-tcp-01


Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/



From nobody Mon Nov 16 12:54:48 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A4A943A141A for ; Mon, 16 Nov 2020 12:54:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=smcquistin.uk header.b=ABR/ZDQZ; dkim=pass (2048-bit key) header.d=messagingengine.com header.b=OcA7hJwH
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cd2eZg1TKN3K for ; Mon, 16 Nov 2020 12:54:44 -0800 (PST)
Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7D6D13A1416 for ; Mon, 16 Nov 2020 12:54:44 -0800 (PST)
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 693415C0228 for ; Mon, 16 Nov 2020 15:54:43 -0500 (EST)
Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 16 Nov 2020 15:54:43 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=smcquistin.uk; h=from:content-type:mime-version:subject:message-id:references :to:date; s=fm2; bh=Jj+KWWYdbiCqSogH2TXIZIgXbtHNEh2lM6PxbNxlIRA=; b= ABR/ZDQZWKY8YL14sp3g/8+3Krps9odAt8gjMaJFCXdBY6s9/pB5fMwDlN2ik/FJ LTKipQCE0OSgjQRSQ4+1Z8Vfp3hbJvJkkUGmr9Q/tMjfa/5w58rbtK0aih1hysGc g9Sn5q138rNX0CfwkGGuOLUo/19RoCA3miQghKfWV3N+3+O0dhOWQN4GmGjStqk0 KqpCWvPzMtDYzqa9l/4pt41fcpQYzRp4dUMPIqeZPRgbzUaSgl//+D5AYgXIPaTd Id1Ixe6PE1S3ElQiGYTXnRl9Qy775zS2XgmpVvyEfbNA03vCTsCAnYbvSSu6KT6E MfjkODNTFKskHAE30DSQtA==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:references:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=Jj+KWWYdbiCqSogH2 TXIZIgXbtHNEh2lM6PxbNxlIRA=; b=OcA7hJwH0anfqrE9lBJmilgpzGX6dUfm4 9pKZiaq+h9Hrzie/icdjUFY6Y64xcZoDUBFIZng6O869oMc67yzB43LfIJ3p75BL qAtvAbrL5grXRUfNWJWjWqlSjM0MD5cQ8Lyopd7Kxbufgs6onwOKFsOxkdrM9Cr7 xG8u+0f5Rf/gWHZR0MZ8lU1tepwS7KykYBzJW1piNtU1yyzvuJm22/wh+Zu6D453 n2ltI4ANI6U1TxeEyzoM/9x1oHNEyDcdFKQzp8Y541YlLLbcfcOrdmJHAEC62mvq aNSaN+783FhlNi+ltOCs1xm5ozubCy7SeWr3HWKyCX3tSNyREKWOA==
X-ME-Sender:  
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrudefuddgudeggecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhephfgtggfukfhfvfffofesrgdtmh erhhdtjeenucfhrhhomhepufhtvghphhgvnhcuofgtsfhuihhsthhinhcuoehsmhesshhm tghquhhishhtihhnrdhukheqnecuggftrfgrthhtvghrnheptdfhueeljeehheetvddvff fgueejheeihfelteekhfdtkedugfdtueeiueehkeeinecuffhomhgrihhnpehivghtfhdr ohhrghenucfkphepuddtledrudehgedrudelhedrvdehgeenucevlhhushhtvghrufhiii gvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehsmhesshhmtghquhhishhtihhnrdhu kh
X-ME-Proxy:    
Received: from [192.168.1.111] (host109-154-195-254.range109-154.btcentralplus.com [109.154.195.254]) by mail.messagingengine.com (Postfix) with ESMTPA id 84B093064AB2 for ; Mon, 16 Nov 2020 15:54:42 -0500 (EST)
From: Stephen McQuistin 
Content-Type: multipart/alternative; boundary="Apple-Mail=_6D114502-30DC-4128-AE8D-5AACD1EA4575"
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.1\))
Message-Id: 
References: <160435594481.7062.2586551370791864223@ietfa.amsl.com>
To: tcpm@ietf.org
Date: Mon, 16 Nov 2020 20:54:40 +0000
X-Mailer: Apple Mail (2.3654.20.0.2.1)
Archived-At: 
Subject: [tcpm] Fwd: New Version Notification for draft-mcquistin-augmented-tcp-example-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 16 Nov 2020 20:54:46 -0000

--Apple-Mail=_6D114502-30DC-4128-AE8D-5AACD1EA4575
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi all,

I want to highlight this draft (draft-mcquistin-augmented-tcp-example) =
that might be of interest to the group.

The draft tries to mirror draft-ietf-tcpm-rfc793bis, but uses a =
machine-readable packet header diagram language (Augmented Packet Header =
Diagrams: draft-mcquistin-augmented-ascii-diagrams). The draft can be =
parsed by tooling that we=E2=80=99ve created (and described in the =
draft), and then used to generate a parser for the version of TCP it =
describes. This allows for an implementation to be generated directly =
from the document that describes the protocol, ensuring compatibility =
between the code and the specification.

As this is work-in-progress, there are limitations in the =
machine-readable format. This example draft has been written to try to =
explore those limitations (and the benefits of the approach).

It would be great to get feedback on this example specifically (e.g., =
how natural it is to read), and on the broader project. I=E2=80=99m =
happy to discuss on this list, and I=E2=80=99ll also be around on the =
Gather hallway. I=E2=80=99ll be outside Room 8 after both tomorrow=E2=80=99=
s TCPM session, and the TSVWG session on Wednesday.

Thanks!

Stephen=20


> Begin forwarded message:
>=20
> From: internet-drafts@ietf.org
> Subject: New Version Notification for =
draft-mcquistin-augmented-tcp-example-00.txt
> Date: 2 November 2020 at 22:25:44 GMT
> To: "Vivian Band" , "Stephen McQuistin" =
, "Colin Perkins" , "Dejice Jacob" =

>=20
>=20
> A new version of I-D, draft-mcquistin-augmented-tcp-example-00.txt
> has been successfully submitted by Stephen McQuistin and posted to the
> IETF repository.
>=20
> Name:		draft-mcquistin-augmented-tcp-example
> Revision:	00
> Title:		Describing TCP with Augmented Packet Header =
Diagrams
> Document date:	2020-11-02
> Group:		Individual Submission
> Pages:		10
> URL:            =
https://www.ietf.org/archive/id/draft-mcquistin-augmented-tcp-example-00.t=
xt
> Status:         =
https://datatracker.ietf.org/doc/draft-mcquistin-augmented-tcp-example/
> Html:           =
https://www.ietf.org/archive/id/draft-mcquistin-augmented-tcp-example-00.h=
tml
> Htmlized:       =
https://tools.ietf.org/html/draft-mcquistin-augmented-tcp-example-00
>=20
>=20
> Abstract:
>   This document describes TCP, and a number of its extensions, using
>   Augmented Packet Header Diagrams.  This document is an example of =
the
>   Augmented Packet Header Diagram language: it is not intended as a
>   contribution to any ongoing or future work on maintaining or
>   extending TCP.
>=20
>=20
>=20
>=20
> Please note that it may take a couple of minutes from the time of =
submission
> until the htmlized version and diff are available at tools.ietf.org.
>=20
> The IETF Secretariat
>=20
>=20


--Apple-Mail=_6D114502-30DC-4128-AE8D-5AACD1EA4575
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

Hi =
all,

I want to =
highlight this draft (draft-mcquistin-augmented-tcp-example) that might =
be of interest to the group.

The draft tries to mirror =
draft-ietf-tcpm-rfc793bis, but uses a machine-readable packet header =
diagram language (Augmented Packet Header =
Diagrams: draft-mcquistin-augmented-ascii-diagrams). The draft can =
be parsed by tooling that we=E2=80=99ve created (and described in the =
draft), and then used to generate a parser for the version of TCP it =
describes. This allows for an implementation to be generated directly =
from the document that describes the protocol, ensuring compatibility =
between the code and the specification.

As this is work-in-progress, there are =
limitations in the machine-readable format. This example draft has been =
written to try to explore those limitations (and the benefits of the =
approach).

It =
would be great to get feedback on this example specifically (e.g., how =
natural it is to read), and on the broader project. I=E2=80=99m happy to =
discuss on this list, and I=E2=80=99ll also be around on the Gather =
hallway. I=E2=80=99ll be outside Room 8 after both tomorrow=E2=80=99s =
TCPM session, and the TSVWG session on Wednesday.
Thanks!

Stephen 


Begin=
 forwarded message:

From: =
internet-drafts@ietf.org
Subject: =
New Version =
Notification for draft-mcquistin-augmented-tcp-example-00.txt
Date: =
2 November 2020 at 22:25:44 =
GMT
To: =
"Vivian Band" <vivianband0@gmail.com>, "Stephen McQuistin" <sm@smcquistin.uk>, =
"Colin Perkins" <csp@csperkins.org>, "Dejice Jacob" <d.jacob.1@research.gla.ac.uk>

A new version of I-D, =
draft-mcquistin-augmented-tcp-example-00.txt
has been =
successfully submitted by Stephen McQuistin and posted to the
IETF repository.

Name:		=
draft-mcquistin-augmented-tcp-example
Revision:	00
Title:=
		=
Describing TCP with Augmented Packet Header Diagrams
Document date:	2020-11-02
Group:=
		=
Individual Submission
Pages:		10
URL: =
           https://www.ietf.org/archive/id/draft-mcquistin-augmented-tcp-e=
xample-00.txt
Status: =
        https://datatracker.ietf.org/doc/draft-mcquistin-augmented-tcp-=
example/
Html: =
          https://www.ietf.org/archive/id/draft-mcquistin-augmented-tcp-e=
xample-00.html
Htmlized: =
      https://tools.ietf.org/html/draft-mcquistin-augmented-tcp-examp=
le-00


Abstract:
   This document describes TCP, and a number of its =
extensions, using
   Augmented Packet Header =
Diagrams.  This document is an example of the
 =
  Augmented Packet Header Diagram language: it is not intended =
as a
   contribution to any ongoing or future =
work on maintaining or
   extending TCP.




Please note that it may take a couple of minutes from the =
time of submission
until the htmlized version and diff are =
available at tools.ietf.org.

The IETF =
Secretariat



=

--Apple-Mail=_6D114502-30DC-4128-AE8D-5AACD1EA4575--


From nobody Mon Nov 16 16:58:53 2020
Return-Path: 
X-Original-To: tcpm@ietf.org
Delivered-To: tcpm@ietfa.amsl.com
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id CC92A3A17B5; Mon, 16 Nov 2020 16:58:50 -0800 (PST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
From: The IESG 
To: "IETF-Announce" 
X-Test-IDTracker: no
X-IETF-IDTracker: 7.22.0
Auto-Submitted: auto-generated
Precedence: bulk
CC: draft-ietf-tcpm-rack.all@ietf.org, tcpm@ietf.org, tcpm-chairs@ietf.org, draft-ietf-tcpm-rack@ietf.org, martin.h.duke@gmail.com, tuexen@fh-muenster.de
Reply-To: last-call@ietf.org
Sender: 
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Message-ID: <160557473030.20071.3820294165818082636@ietfa.amsl.com>
Date: Mon, 16 Nov 2020 16:58:50 -0800
Archived-At: 
Subject: [tcpm] Last Call:  (The RACK-TLP loss detection algorithm for TCP) to Proposed Standard
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 00:58:51 -0000

The IESG has received a request from the TCP Maintenance and Minor Extensions
WG (tcpm) to consider the following document: - 'The RACK-TLP loss detection
algorithm for TCP'
   as Proposed Standard

The IESG plans to make a decision in the next few weeks, and solicits final
comments on this action. Please send substantive comments to the
last-call@ietf.org mailing lists by 2020-11-30. Exceptionally, comments may
be sent to iesg@ietf.org instead. In either case, please retain the beginning
of the Subject line to allow automated sorting.

Abstract


   This document presents the RACK-TLP loss detection algorithm for TCP.
   RACK-TLP uses per-segment transmit timestamps and selective
   acknowledgements (SACK) and has two parts: RACK ("Recent
   ACKnowledgment") starts fast recovery quickly using time-based
   inferences derived from ACK feedback.  TLP ("Tail Loss Probe")
   leverages RACK and sends a probe packet to trigger ACK feedback to
   avoid retransmission timeout (RTO) events.  Compared to the widely
   used DUPACK threshold approach, RACK-TLP detects losses more
   efficiently when there are application-limited flights of data, lost
   retransmissions, or data packet reordering events.  It is intended to
   be an alternative to the DUPACK threshold approach.




The file can be obtained via
https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/



No IPR declarations have been submitted directly on this I-D.






From nobody Mon Nov 16 17:38:44 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 46DBF3A0D20 for ; Mon, 16 Nov 2020 17:38:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UQuNdj7EpR0j for ; Mon, 16 Nov 2020 17:38:41 -0800 (PST)
Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3D2AB3A0D09 for ; Mon, 16 Nov 2020 17:38:41 -0800 (PST)
Received: from fraeml735-chm.china.huawei.com (unknown [172.18.147.201]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4CZpTF3g3Dz67D5J; Tue, 17 Nov 2020 09:37:05 +0800 (CST)
Received: from fraeml735-chm.china.huawei.com (10.206.15.216) by fraeml735-chm.china.huawei.com (10.206.15.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Tue, 17 Nov 2020 02:38:37 +0100
Received: from DGGEMM405-HUB.china.huawei.com (10.3.20.213) by fraeml735-chm.china.huawei.com (10.206.15.216) with Microsoft SMTP Server (version=TLS1_0, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.1.1913.5 via Frontend Transport; Tue, 17 Nov 2020 02:38:37 +0100
Received: from DGGEMM534-MBS.china.huawei.com ([169.254.7.61]) by DGGEMM405-HUB.china.huawei.com ([10.3.20.213]) with mapi id 14.03.0487.000; Tue, 17 Nov 2020 09:38:34 +0800
From: Kangjiao 
To: "Olivier.Bonaventure@uclouvain.be" , "tcpm@ietf.org" 
Thread-Topic: MPTCP RobE
Thread-Index: Adat14Qs9LlNljS0SE+s1uwEYxjReQN/voaAACqppTA=
Date: Tue, 17 Nov 2020 01:38:33 +0000
Message-ID: <719A2C1D4AC73847B6E1BF21DF1545EAE5DC6385@dggemm534-mbs.china.huawei.com>
References:  <00733d01-e95c-3cae-eee2-40fd93f7c0f2@uclouvain.be>
In-Reply-To: <00733d01-e95c-3cae-eee2-40fd93f7c0f2@uclouvain.be>
Accept-Language: zh-CN, en-US
Content-Language: zh-CN
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.67.102.89]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-CFilter-Loop: Reflected
Archived-At: 
Subject: Re: [tcpm] MPTCP RobE
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 01:38:43 -0000

SGkgT2xpdmllciwNCg0KVGhhbmtzIGZvciB0aGUgcmV2aWV3LiBQbGVhc2Ugc2VlIGEgY2xhcmlm
aWNhdGlvbiBpbmxpbmUgZm9yIHlvdXIgc3VnZ2VzdGlvbi4gVGhhbmtzDQoNCg0KPiAtLS0tLU9y
aWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBPbGl2aWVyIEJvbmF2ZW50dXJlIFttYWlsdG86
T2xpdmllci5Cb25hdmVudHVyZUB1Y2xvdXZhaW4uYmVdDQo+IFNlbnQ6IE1vbmRheSwgTm92ZW1i
ZXIgMTYsIDIwMjAgOTowOCBQTQ0KPiBUbzogTWFya3VzLkFtZW5kQHRlbGVrb20uZGU7IHRjcG1A
aWV0Zi5vcmcNCj4gQ2M6IEthbmdqaWFvIDxrYW5namlhb0BodWF3ZWkuY29tPg0KPiBTdWJqZWN0
OiBSZTogTVBUQ1AgUm9iRQ0KPiANCj4gTWFya3VzLCBKaWFvLA0KPiA+DQo+ID4gYXMgb25lIG9m
IHRoZSBhdXRob3JzIG9mDQo+IGh0dHBzOi8vZGF0YXRyYWNrZXIuaWV0Zi5vcmcvZG9jL2RyYWZ0
LWFtZW5kLXRjcG0tbXB0Y3Atcm9iZS8sIEkga2luZGx5DQo+IHJlcXVlc3QgZmVlZGJhY2sgZnJv
bSB5b3UsIGluIHBhcnRpY3VsYXIgaG93ZXZlciBmcm9tIHRoZSBSRkM2ODI0Lzg2ODQNCj4gYXV0
aG9ycyBhbmQgTVBUQ1AgaW1wbGVtZW50ZXJzLg0KPiA+DQo+ID4gV2l0aCBNUFRDUCBSb2JFIChS
b2J1c3QgRXN0YWJsaXNobWVudCkgd2UgYWRkcmVzcyB0aGUgbGFjayBvZiBNUFRDUCB0bw0KPiBy
ZWxpYWJseSBzZXR1cCBhIE1QVENQIHNlc3Npb24sIHdoZW4gdGhlIHNlbGVjdGVkIHBhdGggZm9y
IHRoZSBpbml0aWFsIHNlc3Npb24NCj4gc2V0dXAgaXMgbWFsZnVuY3Rpb25lZC4gVGhlIE1QVENQ
IFJGQ3MgZG9lcyBub3QgZ2l2ZSBhbnkgZ3VpZGVsaW5lcyBmb3IgdGhpcw0KPiBwYXJ0aWN1bGFy
IHByb2JsZW0gYW5kIGl0IGlzIGxlZnQgdG8gdGhlIGltcGxlbWVudGVyIHRvIHRha2UgY2FyZS4g
Rm9yIHRoYXQgd2UNCj4gcHJvcG9zZSBhbmQgY29tcGFyZSBkaWZmZXJlbnQgc3RyYXRlZ2llcyBp
biB0aGUgZHJhZnQgZG9jdW1lbnQsIHNlcGFyYXRlZCBieQ0KPiB0aGVpciBkZW1hbmQgb24gTVBU
Q1AgcHJvdG9jb2wgY2hhbmdlcy4gQXBhcnQgZnJvbSB0aGF0IG91ciBjbGFpbSBpcywgdGhhdA0K
PiB3aXRoIHRoZXNlIHByb3Bvc2FscyBjb21wYXJlZCB0byBNUFRDUCwgbm8gYWRkaXRpb25hbCBz
ZWN1cml0eSBmbGF3cyBhcmUNCj4gcmFpc2VkIGFuZCBhbnkgcG90ZW50aWFsIGNvbXB1dGF0aW9u
YWwgb3ZlcmhlYWQgaXMgbWluaW1pemVkLg0KPiA+DQo+IA0KPiBTb3JyeSBmb3IgdGhlIGxhdGUg
cmVwbHkuIEkgaGF2ZSBsb29rZWQgYXQgdGhlIGRvY3VtZW50IGFuZCBmb3VuZCB0aGF0IGhhbmRs
aW5nDQo+IGJvdGggUkZDNjgyNCBhbmQgUkZDODY4NCBhZGRzIGEgbG90IG9mIGNvbXBsZXhpdHkg
dG8gdGhlIGRvY3VtZW50LiBTaW5jZQ0KPiBSRkM4Njg0IGlzIHRoZSBzdGFuZGFyZHMgdHJhY2sg
dmVyc2lvbiBvZiBNUFRDUCwgSSB3b3VsZCBzdWdnZXN0IHRvIG9ubHkNCj4gZGlzY3VzcyB0aGlz
IHZlcnNpb24gb2YgTVBUQ1AgaW4gdGhlIGRvY3VtZW50Lg0KPiANCj4gTXkgbWFpbiBjb25jZXJu
IGFib3V0IHRoZSBwcm9wb3NlZCBhcHByb2FjaCBpcyBob3cgdGhpcyB3aWxsIHdvcmsgd2l0aA0K
PiBsb2FkLWJhbGFuY2Vycy4gV2hlbiB0aGVyZSBpcyBhIGxvYWQtYmFsYW5jZXIgaW4gZnJvbnQg
b2YgYSBwb29sIG9mIHNlcnZlcnMgdGhhdA0KPiB1c2UgdGhlIHNhbWUgSVAgYWRkcmVzcywgdGhl
cmUgaXMgbm8gZ3VhcmFudGVlIHRoYXQgU1lOcyBmcm9tIHRoZSBzYW1lIGNsaWVudA0KPiB3aWxs
IGJlIGRlbGl2ZXJlZCB0byB0aGUgc2FtZSBzZXJ2ZXIsIGVzcGVjaWFsbHkgaWYgdGhlc2UgU1lO
cyBhcmUgc2VudCBvdmVyDQo+IGRpZmZlcmVudCBwYXRocyBhbmQgdGh1cyB3aXRoIGRpZmZlcmVu
dCBzb3VyY2UgYWRkcmVzc2VzLiBJZiB0aGUgdHdvDQo+IGNvbm5lY3Rpb25zIHRoYXQgUm9iRSB0
cmllcyB0byBlc3RhYmxpc2ggdGVybWluYXRlIG9uIGRpZmZlcmVudCBwaHlzaWNhbCBzZXJ2ZXJz
LA0KPiB0aGV5IGNhbm5vdCBiZSBqb2luZWQuIEdpdmVuIHRoZSBpbXBvcnRhbmNlIG9mIGxvYWQg
YmFsYW5jZXJzLCBJIHdvdWxkIHByZWZlciBhDQo+IHNvbHV0aW9uIHdoZXJlIHRoZSBhcHBsaWNh
dGlvbiBzaW1wbHkgdHJpZXMgdG8gb3BlbiB0d28gcGFyYWxsZWwgbXB0Y3ANCj4gY29ubmVjdGlv
bnMsIHByZWZlcnMgdGhlIGZpcnN0IHRoYXQgZ2V0cyBlc3RhYmxpc2hlZCwgYWRkcyBhbm90aGVy
IHN1YmZsb3cgdG8NCj4gdGhpcyBjb25uZWN0aW9uIGFuZCByZXNldHMgdGhlIHNlY29uZCBvbmUu
IFRoaXMgd291bGQgc2xpZ2h0bHkgaW5jcmVhc2UgdGhlDQo+DQo+DQo+IGRlbGF5IGNvbXBhcmVk
IHRvIFJvYkUsIGJ1dCB3b3VsZCBwcmVzZXJ2ZSBsb2FkIGJhbGFuY2Vycy4gSW4gcHJhY3RpY2Us
IEkNCj4gZXhwZWN0IHRoYXQgTVBUQ1Agc3RhY2tzIHdpbGwgbGV2ZXJhZ2UgaW5mb3JtYXRpb24g
YWJvdXQgdGhlIGN1cnJlbnQgcXVhbGl0eSBvZg0KPiB0aGUgYWNjZXNzIG5ldHdvcmtzIChlLmcu
IFdpRmkgYW5kIHhHIG9uIHNtYXJ0cGhvbmVzKSBhbmQgd2lsbCBvbmx5IGF0dGVtcHQNCj4gcGFy
YWxsZWwgY29ubmVjdGlvbnMgd2hlbiB0aGUgY29uZGl0aW9ucyBhcmUgdW5jZXJ0YWluLiBBcHBs
ZSdzIFdpRmkgYXNzaXN0DQo+IGFscmVhZHkgaW5jbHVkZXMgc3VjaCBmdW5jdGlvbm5hbGl0eS4N
Cj4NCj4NCg0KSSB0aGluayB0aGlzIHBvaW50IGhhcyBhbHJlYWR5IGJlZW4gY292ZXJlZCBieSB0
aGUgUm9iRV9JUFMgKGluaXRpYWwgcGF0aCBzZWxlY3Rpb24pIGluIHRoZSBkcmFmdCAoU2VjdGlv
biAyLjMpLg0KDQogDQo+IEJlc3QgcmVnYXJkcywNCj4gDQo+IA0KPiBPbGl2aWVyDQo=


From nobody Mon Nov 16 18:08:13 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4D23D3A1882 for ; Mon, 16 Nov 2020 18:08:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bH0jq1SyiUXa for ; Mon, 16 Nov 2020 18:08:10 -0800 (PST)
Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 933593A1881 for ; Mon, 16 Nov 2020 18:08:10 -0800 (PST)
Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.42/8.16.0.42) with SMTP id 0AH23N7I057356 for ; Mon, 16 Nov 2020 18:08:09 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : content-type : mime-version : subject : message-id : date : to; s=20180706; bh=Omn+QKKt49WGAuo4IB9DBGk+YDVt4wJUva1JPJDVOP4=; b=PZ5SqDZ6Rfz5CZ+DAz/xdRR/Yr1l55aoMrjcr/HN/R21+eTHr/Gf0r30mj2YlqQfEbEg qgNqvKD6DUtnoUaFxSQqm3ZSQ4ZUnUFacHAUs+YLBG5hd+jQmeoZNaBZur06WUGGsBWC 8893ZouY47Lbo/9ey6FW1nJd3cLq9gPmq1U0O5A0j+A/1BQ3wVIuqEiNhUgFwSr7UGYB rMOIUTAw/xuBOABLV+d7uXHb7zRI6APRkrGpTeL5kfHahsYPJvAEjTbmzkWwE/2dvFxv nShomPchve1lZH72J+93GZxwRE6nFo/tVsT806TuWKo3GOi1n0QIfxjpFEwnr0XRBdXl Dw== 
Received: from rn-mailsvcp-mta-lapp03.rno.apple.com (rn-mailsvcp-mta-lapp03.rno.apple.com [10.225.203.151]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 34tf4034db-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Mon, 16 Nov 2020 18:08:09 -0800
Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPS id <0QJX00J8N4LI6QC0@rn-mailsvcp-mta-lapp03.rno.apple.com> for tcpm@ietf.org; Mon, 16 Nov 2020 18:08:07 -0800 (PST)
Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) id <0QJX00O00486SV00@rn-mailsvcp-mmp-lapp04.rno.apple.com> for tcpm@ietf.org; Mon, 16 Nov 2020 18:08:07 -0800 (PST)
X-Va-A: 
X-Va-T-CD: a8502bea19e1ef054f10f9e9017ff330
X-Va-E-CD: 96cf5205a610a0be5f6c4a3191595cd9
X-Va-R-CD: 78dc9587f86f7734fbe80b4dfe497d0c
X-Va-CD: 0
X-Va-ID: 034f5ffc-9494-43d4-a38e-66c2ea79fc03
X-V-A: 
X-V-T-CD: a8502bea19e1ef054f10f9e9017ff330
X-V-E-CD: 96cf5205a610a0be5f6c4a3191595cd9
X-V-R-CD: 78dc9587f86f7734fbe80b4dfe497d0c
X-V-CD: 0
X-V-ID: d82b0e78-4dc9-42bf-8ec7-1ebf6c82b708
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-16_13:2020-11-13, 2020-11-16 signatures=0
Received: from [17.149.228.133] (unknown [17.149.228.133]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.6.20200729 64bit (built Jul 29 2020)) with ESMTPSA id <0QJX00QCT4LI4B00@rn-mailsvcp-mmp-lapp04.rno.apple.com> for tcpm@ietf.org; Mon, 16 Nov 2020 18:08:06 -0800 (PST)
From: Vidhi Goel 
Content-type: multipart/alternative; boundary="Apple-Mail=_DF37EB62-D6B1-4CFD-AC68-52405DA89285"
MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\))
Message-id: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>
Date: Mon, 16 Nov 2020 18:08:05 -0800
To: "tcpm@ietf.org Extensions" 
X-Mailer: Apple Mail (2.3608.80.23.2.2)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.312, 18.0.737 definitions=2020-11-16_13:2020-11-13, 2020-11-16 signatures=0
Archived-At: 
Subject: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 02:08:12 -0000

--Apple-Mail=_DF37EB62-D6B1-4CFD-AC68-52405DA89285
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hello,

As Lars mentioned earlier, there are a couple of issues/proposals filed =
for updating the CUBIC RFC 8312 and are currently under discussion. The =
GitHub repo is at https://github.com/NTAP/rfc8312bis =
.
We would like to gather information on which CUBIC implementations =
already or would be ready to adopt these changes.

1. Update K definition (Eq. 2) to account for Fast Convergence
Issue link https://github.com/NTAP/rfc8312bis/issues/1 =

Linux already has this change. Apple's Cubic implementation for both TCP =
and QUIC already use this.

2. Congestion window TCP friendly region after W_max=20
Issue link https://github.com/NTAP/rfc8312bis/issues/2 =

This is a suggestion to improve performance after cwnd > W_max. Apple's =
Cubic implementation for both TCP and QUIC already use this.

=46rom my experience, these changes would improve Cubic=E2=80=99s =
performance.=20

Please let us know if you already or would be ready to implement these =
changes in your CUBIC=E2=80=99s implementation.

Thanks,
Vidhi=

--Apple-Mail=_DF37EB62-D6B1-4CFD-AC68-52405DA89285
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8


Hello,

As Lars mentioned earlier, there are a =
couple of issues/proposals filed for updating the CUBIC RFC 8312 and are =
currently under discussion. The GitHub repo is at https://github.com/NTAP/rfc8312bis.
We=
 would like to gather information on which CUBIC implementations already =
or would be ready to adopt these changes.

1. Update K definition (Eq. 2) to =
account for Fast Convergence
Issue =
link https://github.com/NTAP/rfc8312bis/issues/1
Linux already has this change. Apple's Cubic implementation =
for both TCP and QUIC already use this.

2. Congestion window TCP friendly =
region after W_max 
Issue link https://github.com/NTAP/rfc8312bis/issues/2
This is a suggestion to improve performance after cwnd > =
W_max. Apple's Cubic =
implementation for both TCP and QUIC already use this.

=46rom my experience, =
these changes would improve Cubic=E2=80=99s performance. 

Please let us know if =
you already or would be ready to implement these changes in your =
CUBIC=E2=80=99s implementation.

Thanks,
Vidhi
=

--Apple-Mail=_DF37EB62-D6B1-4CFD-AC68-52405DA89285--


From nobody Mon Nov 16 21:08:26 2020
Return-Path: 
X-Original-To: tcpm@ietf.org
Delivered-To: tcpm@ietfa.amsl.com
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 352E43A0DAE for ; Mon, 16 Nov 2020 21:08:24 -0800 (PST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
To: 
X-Test-IDTracker: no
X-IETF-IDTracker: 7.22.0
Auto-Submitted: auto-generated
Precedence: bulk
Message-ID: <160558970420.27250.13704775328753337761@ietfa.amsl.com>
Date: Mon, 16 Nov 2020 21:08:24 -0800
From: IETF Secretariat 
Archived-At: 
Subject: [tcpm] Milestones changed for tcpm WG
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 05:08:24 -0000

Changed milestone "Submit document on a time-based fast loss detection
algorithm for TCP to the IESG for publication as a Proposed Standard RFC",
resolved as "Done".

URL: https://datatracker.ietf.org/wg/tcpm/about/


From nobody Mon Nov 16 21:10:02 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 617933A0DC0; Mon, 16 Nov 2020 21:10:00 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XXRb9A3csrBz; Mon, 16 Nov 2020 21:09:57 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1D4253A0DBA; Mon, 16 Nov 2020 21:09:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=w07pg8is1Impo9my03Y+7T4BXao2ocNFqI+PQ6yIUEg=; b=cish/pjvxqn7s6RVFPCNiZi+A BPM/9ZLeAPKDPsYQHEXQPjTYGj1a5fSd+AWRJCcUwdc2TGaS1nrJ83n6TaCpXHLsOYx9rll2xROnI wRylHPGfoF6fzAYRuFLLMd1V9rdgKT3qU9YD6aLh/HoMTMfBinKICmNIP516NMIkZ3k8q5P+JMg0B 38CSt4QeWXs4JErRmdTD9iREQWo9iDp1nC0QnYevurPVWI0U1bOR+OgTorHKi6EJMWH7ZyYiTwAma J1JmA5Yt8Un9e6K5RBtG9/0qXGwi+r1JVIfNHBTzTqkuNCXw03d3Vmanzq7zQkZKB1uRcSJWeak9f mlyhUyKSQ==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:37774 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1ketFZ-00FJeQ-G5; Tue, 17 Nov 2020 05:09:55 +0000
To: "Scharf, Michael" , Yoshifumi Nishida 
Cc: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net> 
From: Bob Briscoe 
Message-ID: 
Date: Tue, 17 Nov 2020 05:09:50 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------DD06BAC22A9BCB63AED1641A"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 05:10:00 -0000

This is a multi-part message in MIME format.
--------------DD06BAC22A9BCB63AED1641A
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Michael,

On 16/11/2020 17:36, Scharf, Michael wrote:
>
> Bob,
>
> One proposal using the length field with **one option codepoint only** 
> is detailed in: 
> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>
> It is the third option mentioned in this e-mail. One example would be 
> to use option length values 5/8/11 for one encoding type and option 
> length values 6/9/12 for the other encoding type (i.e., order of 
> fields). Or one could use some other combination of length values – 
> the only requirement is that a certain value for the option length is 
> only used by one of the option formats. In that approach, the value of 
> the length field would thus directly describe the encoding of the 
> option. Unless I miss something, this would work and it would just 
> require one option codepoint.
>
> Thus, alternatives to two option codepoints exist and I have explained 
> them on the list in March 2020.
>

OK, sorry, yes, I remember this now.

As I will explain in the AccECN status update talk today in virtual 
Bangkok, the draft has made provision for different length values than 
5/8/11. It says existing implementations MUST accept length values other 
than those currently defined. But then read in as many whole 3-byte 
fields as they can.

This can be used to add a flags byte on the end in future, for 
extensibility. Or any other form of extensibility the WG might decide in 
the future.

I know a flags byte at the end seems odd compared to at the beginning. 
But (if decided it's needed in future) it's reasonably easy to implement 
by reading the whole option, then processing the last byte, before 
reading the rest of the option.

I believe you will agree that this is a better way to utilize different 
lengths.

And thank you for repeatedly emphasizing that you're happy with the 
2-kind scheme, or other alternatives.


Bob

> Anyway, I don’t really care how the options are encoded as long as the 
> receiver doesn’t need per-connection state for decoding a TCP option. 
> So, personally, I would be fine with using e.g. the length field as 
> described in my old e-mail. Or an additional flag byte. And one could 
> come up with further encodings, e.g., by using one or a few bits as a 
> short “type” field for each counter. This is all about protocol 
> engineering. And all these variants have their pros and cons.
>
> I am also fine with using two option codepoints as specified in -13; 
> this is probably the approach that consumes the least number of bits.
>
> Michael (w/o any hat)
>
> *From:* Bob Briscoe 
> *Sent:* Monday, November 16, 2020 5:52 PM
> *To:* Yoshifumi Nishida 
> *Cc:* Scharf, Michael ; Michael Tuexen 
> ; tcpm-chairs@ietf.org; Mirja Kuehlewind 
> ; Scheffenegger, Richard 
> ; tcpm IETF list 
> *Subject:* AccECN field order
>
> Yoshi, (adding the tcpm list in cc)
>
> On 05/11/2020 06:58, Yoshifumi Nishida wrote:
>
>     Hi Bob,
>
>     On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe      > wrote:
>
>         Yoshi,
>
>         On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>
>             Hi, folks,
>
>
>
>             In my understanding, I'm not sure if we settled down on
>             using two option kinds or encoding schemes for 24bits
>             fields in acc ecn draft.
>
>             So, I think there're still something to be clarified and
>             hope things will be settled at the meeting.
>
>
>         [BB] I know a WG can change it's mind at any time. But I'd
>         rather we just clarified what a previous decision was, to
>         avoid the need to keep re-opening discussion on a question
>         that have been decided then changed three different ways already.
>
>         My memory is not so good these days. I trusted that Michael S
>         remembered the decision correctly, and I seem to remember that
>         decision being made.
>         I've just checked the minutes of the last interim:
>         https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/
>         and they mention Michael's proposal to use two kinds, but
>         don't record any decision.
>         The jabber log gives no clues about any decision.
>
>         I can't find an audio or video recording. Can you point me at one?
>
>     I thought that it's because there was no clear decision at the
>     meeting.
>
>     But, you can check https://www.youtube.com/watch?v=KsB0_Nb8-kA
>     
>
>     Please let us know if you have any questions or opinions with
>     regard to this.
>
>
> [BB] I checked the Youtube link you sent below.
>
> First I think we're agreed no-one was fighting for us to keep the 
> previous way we did this (using the initial value of the field to set 
> the order for the connection).
> In my presentation I said there was strong resistance from Michael to 
> do it a different way.
> (also, offlist, the co-authors including me also didn't like this so 
> much. And Ilpo said it made the implementation complex.)
>
> Then came the question of what we do instead. There were three 
> alternative proposals:
> a) use 2 option kinds
> b) add a flags byte
> c) somehow use the length field maybe
>
> Michael raised (c) in the meeting as a possibility, but no-one could 
> think how to distinguish two options of the same length but a 
> different field order using the length field. Michael said he'd post 
> any ideas to the list if he could think of any, but that didn't happen.
>
> So we're left choosing between (a) and (b).
> I said in the meeting (and on the list when discussing with Ilpo) that 
> I'd be happy to go with (b), but only if there was another use for a 
> flag. Because it would consume 1B more options space in many packets, 
> which is a scarce resource.
>
> Ilpo had a proposed use for another flag (to help synch counters after 
> a loss), but I think the discussion about it ended that it wouldn't be 
> helpful, 'cos the way it worked depended on itself (circular logic).
>
> In conclusion, I don't think there was an explicit decision to go with 
> 2 option kinds, but it ended up as the 'last person standing'.
> I like it. It's simple. And apparently option kinds are not such a 
> scarce resource.
>
> Perhaps we can ratify this in the WG tomorrow.
>
>
> Bob
>
>
>     Thanks,
>
>     --
>
>     Yoshi
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------DD06BAC22A9BCB63AED1641A
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Michael,

    

    
On 16/11/2020 17:36, Scharf, Michael
      wrote:

    

    

      
      
      
      

        
Bob,

        
 

        
One proposal using the length field with *one
              option codepoint only* is detailed in:
            
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/

        
 

        
It is the third option mentioned in this
            e-mail. One example would be to use option length values
            5/8/11 for one encoding type and option length values 6/9/12
            for the other encoding type (i.e., order of fields). Or one
            could use some other combination of length values – the only
            requirement is that a certain value for the option length is
            only used by one of the option formats. In that approach,
            the value of the length field would thus directly describe
            the encoding of the option. Unless I miss something, this
            would work and it would just require one option codepoint.

        
 

        
Thus, alternatives to two option codepoints
            exist and I have explained them on the list in March 2020.

      

    

    

    OK, sorry, yes, I remember this now.

    

    As I will explain in the AccECN status update talk today in virtual
    Bangkok, the draft has made provision for different length values
    than 5/8/11. It says existing implementations MUST accept length
    values other than those currently defined. But then read in as many
    whole 3-byte fields as they can.

    

    This can be used to add a flags byte on the end in future, for
    extensibility. Or any other form of extensibility the WG might
    decide in the future.

    

    I know a flags byte at the end seems odd compared to at the
    beginning. But (if decided it's needed in future) it's reasonably
    easy to implement by reading the whole option, then processing the
    last byte, before reading the rest of the option.

    

    I believe you will agree that this is a better way to utilize
    different lengths.

    

    And thank you for repeatedly emphasizing that you're happy with the
    2-kind scheme, or other alternatives.

    

    

    Bob

    

    

      

        

        
 

        
Anyway, I don’t really care how the options are
            encoded as long as the receiver doesn’t need per-connection
            state for decoding a TCP option. So, personally, I would be
            fine with using e.g. the length field as described in my old
            e-mail. Or an additional flag byte. And one could come up
            with further encodings, e.g., by using one or a few bits as
            a short “type” field for each counter. This is all about
            protocol engineering. And all these variants have their pros
            and cons.

        
 

        
I am also fine with using two option codepoints
            as specified in -13; this is probably the approach that
            consumes the least number of bits.

        
 

        
Michael (w/o any hat)

        
 

        
 

        
 

        

          

            

              
From: Bob Briscoe
                <ietf@bobbriscoe.net> 

                Sent: Monday, November 16, 2020 5:52 PM

                To: Yoshifumi Nishida <nsd.ietf@gmail.com>

                Cc: Scharf, Michael
                <Michael.Scharf@hs-esslingen.de>; Michael Tuexen
                <tuexen@fh-muenster.de>; tcpm-chairs@ietf.org;
                Mirja Kuehlewind <ietf@kuehlewind.net>;
                Scheffenegger, Richard
                <Richard.Scheffenegger@netapp.com>; tcpm IETF list
                <tcpm@ietf.org>

                Subject: AccECN field order

            

          

          
 

          
Yoshi,
            (adding the tcpm list in cc)

          

            
On 05/11/2020 06:58, Yoshifumi Nishida
              wrote:

          

          

            

              

                
Hi Bob,

              

              

                
 

              

              

                

                  
On Wed, Nov 4, 2020 at 3:29 PM
                    Bob Briscoe <ietf@bobbriscoe.net>
                    wrote:

                

                

                  

                    
Yoshi,

                    

                      
On 04/11/2020 06:51,
                        Yoshifumi Nishida wrote:

                    

                    

                      

                        

                          
Hi, folks,

                        

                      

                    

                  

                

              

            

          

          


            

            

          

            

              

                

                  

                    

                      

                        

                          
In my understanding, I'm
                            not sure if we settled down on using two
                            option kinds or encoding schemes for 24bits
                            fields in acc ecn draft.

                        

                        

                          
So, I think there're
                            still something to be clarified and hope
                            things will be settled at the meeting.

                        

                      

                    

                    


                      [BB] I know a WG can change it's mind at any time.
                      But I'd rather we just clarified what a previous
                      decision was, to avoid the need to keep re-opening
                      discussion on a question that have been decided
                      then changed three different ways already.

                      

                      My memory is not so good these days. I trusted
                      that Michael S remembered the decision correctly,
                      and I seem to remember that decision being made.

                      I've just checked the minutes of the last interim:

                          
https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/

                      and they mention Michael's proposal to use two
                      kinds, but don't record any decision.

                      The jabber log gives no clues about any decision.

                      

                      I can't find an audio or video recording. Can you
                      point me at one?

                  

                

                

                  
 

                

                

                  
I thought that it's because there
                    was no clear decision at the meeting. 

                

                
                

                  
Please let us know if you have
                    any questions or opinions with regard to this.

                

              

            

          

          


            [BB] I checked the Youtube link you sent below.

            

            First I think we're agreed no-one was fighting for us to
            keep the previous way we did this (using the initial value
            of the field to set the order for the connection).
            

            In my presentation I said there was strong resistance from
            Michael to do it a different way.
            

            (also, offlist, the co-authors including me also didn't like
            this so much. And Ilpo said it made the implementation
            complex.)

            

            Then came the question of what we do instead. There were
            three alternative proposals:

            a) use 2 option kinds

            b) add a flags byte

            c) somehow use the length field maybe

            

            Michael raised (c) in the meeting as a possibility, but
            no-one could think how to distinguish two options of the
            same length but a different field order using the length
            field. Michael said he'd post any ideas to the list if he
            could think of any, but that didn't happen.

            

            So we're left choosing between (a) and (b).

            I said in the meeting (and on the list when discussing with
            Ilpo) that I'd be happy to go with (b), but only if there
            was another use for a flag. Because it would consume 1B more
            options space in many packets, which is a scarce resource.

            

            Ilpo had a proposed use for another flag (to help synch
            counters after a loss), but I think the discussion about it
            ended that it wouldn't be helpful, 'cos the way it worked
            depended on itself (circular logic).

            

            In conclusion, I don't think there was an explicit decision
            to go with 2 option kinds, but it ended up as the 'last
            person standing'.

            I like it. It's simple. And apparently option kinds are not
            such a scarce resource.

            

            Perhaps we can ratify this in the WG tomorrow.

            

            

            Bob

            

            

            

          

            

              

                

                  
 

                

                

                  
Thanks,

                

                

                  
--

                

                

                  
Yoshi

                

              

            

          

          


            

            

          
-- 

          
________________________________________________________________

          
Bob Briscoe                               http://bobbriscoe.net/

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------DD06BAC22A9BCB63AED1641A--


From nobody Mon Nov 16 22:30:00 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1BD143A1003; Mon, 16 Nov 2020 22:29:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level: 
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qZ4PyZJdu3Gr; Mon, 16 Nov 2020 22:29:57 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7379D3A1007; Mon, 16 Nov 2020 22:29:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605594592; bh=cQMKTo8yzl/5w7sCifwc4f7jbyrZNpsfoXQI1CoW1+A=; h=X-UI-Sender-Class:Subject:To:References:From:Date:In-Reply-To; b=cpCwQSKPClOOar4oTDpHS7O0TPYGMjL34IQGtB0ilKD/RK4L17bN8OZxOjvVBJio9 ajIjd4HMMTIgW/WyWStdgiDv8wJ+HKwzZT0KCaw5VVldmmeLhKI00JeOFHSCw7BCCN qP+H3fkb2utvFXbKx1AGXc8mXdRr02+sgTsl7jDs=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [192.168.233.106] ([185.236.167.136]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MkYbu-1jzkqs1IYo-00m15A; Tue, 17 Nov 2020 07:29:52 +0100
To: Vidhi Goel , "tcpm@ietf.org Extensions" 
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>
From: "Scheffenegger, Richard" 
Message-ID: 
Date: Tue, 17 Nov 2020 07:29:50 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:W9bx9PrM8mByF5SmB0pS8QsSScw+MLZm8G3JV9igoEK/YP1GTaF PUedhFVFcS/biRDG4mwG1wPqiifPSlXv0HKgMfkcr1jn8QZ4EMZgdgtDM7BLSGppDMQmCiX WNQ1+0skXaWNVhr9lwaDDGpRLxZty6vNoudXSkQezGlTZfNhB8fG+c1EYGW3v0Fs0bxWCuc aaXdYcYe0oYHDvk3M/1Ww==
X-UI-Out-Filterresults: notjunk:1;V03:K0:7mCXghvWdbw=:RUjljpffLFA74lXrbNAVW8 dI4cQk/STUoIXwVPySXu57WNz3TGgEjPKoJYjt2mHHVT2yTa6tJZovht4WHpzOvykuvwTgxcf fmKEeKeKF5HHt7i99DAnlEkDBvgi+SY9IbpScGNAvQHb9p8a4OqxPK1al0tCDc2irYROpbnBp JLGt6sMoWI8YjCpLz9oRO+cI2fdIyoIO5F3g+KxcmG4kRx9sk2YziJ/cZaI0mw4YUC0EIFTss 0v506j2R0nnfe2CAog+4GKjVPeUx8wyPvKPgEap0+fKinPG0YpKoqJFOGs8SdvuOpcaa2O0IM n5cJFitZdOqR50qoD7deullXhIUmf4l6KbSOhzd5UXW7ku4gBBT0VSSSPsGh0keJO4HAjqIBU J9LaLPxpPvPIi03/UkPBtTYl94N+LeluUjw8nsUAS39ldULwpoUFR6/2l5ayguyCekVN/AuCT 1QEea9/Qq2AK8T9xxWVIXFcdxmOUhXPUaE+OG39B1JGYYEHI7SKiuiDVeMY6YexoSqRJTfhLX uIsECKx0EcdhRzRAXBf6CepULwvjjueXkJHfI94kU8llzBJ+lVrsd1JtW7e6vrJm1yQ4ddB1o AFdKNnn/Kt+t7Ri+NrC0noB0GQ8E571JcIxLqUbLNOYyLhpqKSvAIYCWID1tqMsTxJOCzbpcO JxA5X5OwjHoNs5sJZ0G0rQwEZVTYVMEhbvfcwJW5+yL+L8ABqDOs72mxn9PM99AMe1ROCuEt0 KUZHIgM0wHeO5TRqouUyIBFRHwzkmESYC2zmJRmPC2svC2pz8nCznAuQMwztEeiwCl80hGVRo k4lw9vao380ZYIxw5mwXDaMoo4nolJTdERxHXOhqd9F3f6ys7omhkM9kvyVPLGAYCI23wg7rS QjVg6xZAN4Jp9ElP+ldA==
Archived-At: 
Subject: Re: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 06:29:59 -0000

We have been working on improving the FreeBSD Cubic implementation, to
bring it up from the originally implemented (early draft) version of
Cubic, to a more production ready state which is inline with the adopted
RFC version.

Having said this, the below changes sound reasonable and I will try to
adopt them for FreeBSD.

Richard


Am 17.11.2020 um 03:08 schrieb Vidhi Goel:
> Hello,
>
> As Lars mentioned earlier, there are a couple of issues/proposals filed
> for updating the CUBIC RFC 8312 and are currently under discussion. The
> GitHub repo is at https://github.com/NTAP/rfc8312bis.
> We would like to gather information on which CUBIC implementations
> already or would be ready to adopt these changes.
>
> 1. Update K definition (Eq. 2) to account for Fast Convergence
> Issue link https://github.com/NTAP/rfc8312bis/issues/1
> Linux already has this change. Apple's Cubic implementation for both TCP
> and QUIC already use this.
>
> 2. Congestion window TCP friendly region after W_max
> Issue link https://github.com/NTAP/rfc8312bis/issues/2
> This is a suggestion to improve performance after cwnd > W_max. Apple's
> Cubic implementation for both TCP and QUIC already use this.
>
>  From my experience, these changes would improve Cubic=E2=80=99s perform=
ance.
>
> Please let us know if you already or would be ready to implement these
> changes in your CUBIC=E2=80=99s implementation.
>
> Thanks,
> Vidhi
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>


From nobody Mon Nov 16 23:15:13 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 03E433A110F for ; Mon, 16 Nov 2020 23:15:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0PgmWXZVoiOV for ; Mon, 16 Nov 2020 23:15:10 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E98603A1111 for ; Mon, 16 Nov 2020 23:15:09 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=G0RmTqWchK+IA0xHuGtb/Ei/LQpYfwEm5iL4Qu9MZCc=; b=BtEaBOAcugFche+8nALZy0N5D +uqC2rHJgzEs7b1FO9JgSwX9xbRu9Xad18r8eQzfhvwcL0xhzNEZwp3+VTXEG5PTAiKlja14uS5xS uiwD6khYr+GvmuabbaBR7raMOhjXV8U67rVfS1hxf7jjBbl9+p4vweJS+xFYzOMC2VcgW75/YAW5W sxbUkpGoczUE8e/8+XrMDeKuWYWXmUj4rF9F4h3s9ZXMJ+D/l4a9xRAi+w5K2cB/mF1nO2JqrAh7n 4wHo52PylrT8nMBXRIHsa/e/we1IMd7H5YvU9asRwhWHdzBERCd2p51WUz1oYSBkVj0i7O6DKZgbv 7CMQTcg5w==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:38186 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1keuhO-00FzXP-Lm; Tue, 17 Nov 2020 06:42:43 +0000
To: Yuchung Cheng , "tcpm@ietf.org Extensions" , Kevin Yang 
Cc: Eric Dumazet , iccrg IRTF list 
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com> 
From: Bob Briscoe 
Message-ID: 
Date: Tue, 17 Nov 2020 06:42:40 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------8A0711D39555F5F55DB53DB2"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 07:15:12 -0000

This is a multi-part message in MIME format.
--------------8A0711D39555F5F55DB53DB2
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi, Kevin, Yuchung,

Here's the draft I referred to in the chat during your talk (and Richard 
mentioned it verbally): 
https://tools.ietf.org/html/draft-scheffenegger-tcpm-timestamp-negotiation-05

Here's the rationale behind the point both Richard and I tried to make 
in the tcpm WG just now:

You could define your new option solely with the last two negotiation 
fields.

Because the existing timestamp fields are already defined as opaque. So 
you can have a separate option just for the negotiation of what the 
existing timestamp option means (ie. its resolution).

Then, you don't force TCP to use a large option on the SYN, where space 
is critical. (But it  can use both options if there is space.)
This also gives you a fall-back if the other peer doesn't support the 
new option, but does support the old timestamp option.

The downside, is 2 extra bytes if TCP does use both options on the SYN. 
There might be other ways to arrange this, but the important thing is 
the requirements:
* fall-back during incremental deployment
* provide alternative ways to minimize space on SYN, so implementers can 
juggle priority of fitting in other options.

Cheers


Bob

On 03/11/2020 00:25, Yuchung Cheng wrote:
> Hi tcpm & iccrg,
>
> We are proposing a new TCP timestamp option to measure the network 
> delay more precisely (e.g. excluding delayed ACK effects) for 
> congestion control, e.g. Swift published SIGCOMM 2020. The precision 
> of the measurements can potentially be further enhanced by NIC 
> hardware timestamps. We are working on a reference implementation for 
> Linux as well.
>
> We'll also present it in the upcoming tcpm meeting. Feedbacks are very 
> welcome.
>
> ---------- Forwarded message ---------
> From: >
> Date: Mon, Nov 2, 2020 at 3:29 PM
> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
> To: Eric Dumazet >, 
> Yuchung Cheng >, Neal 
> Cardwell >, Kevin 
> Yang (Yudong) >
>
>
>
> A new version of I-D, draft-yang-tcpm-ets-00.txt
> has been successfully submitted by Kevin (Yudong) Yang and posted to the
> IETF repository.
>
> Name:           draft-yang-tcpm-ets
> Revision:       00
> Title:          TCP ETS: Extensible Timestamp Options
> Document date:  2020-11-02
> Group:          Individual Submission
> Pages:          13
> URL: https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt
> Status: https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
> Htmlized: https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets
> Htmlized: https://tools.ietf.org/html/draft-yang-tcpm-ets-00
>
>
> Abstract:
>    This document presents ETS: an Extensible TimeStamps option for TCP.
>    It allows hosts to use microseconds as the unit for timestamps to
>    improve the precision of timestamps, and advertise the maximum ACK
>    delay for its own delayed ACK mechanism.  Furthermore, it extends the
>    information provided in the [RFC7323] TCP Timestamps Option by
>    including the receiver delay in the TSecr echoing, so that the
>    receiver of the ACK is able to more accurately estimate the portion
>    of the RTT that resulted from time traveling through the network.
>    The ETS option format is extensible, so that future extensions can
>    add further information without the overhead of extra TCP option kind
>    and length fields.
>
>
>
>
> Please note that it may take a couple of minutes from the time of 
> submission
> until the htmlized version and diff are available at tools.ietf.org 
> .
>
> The IETF Secretariat
>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
                PRIVILEGED AND CONFIDENTIAL


--------------8A0711D39555F5F55DB53DB2
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Hi, Kevin, Yuchung,

    

    
Here's the draft I referred to in
        the chat during your talk (and Richard mentioned it verbally):
https://tools.ietf.org/html/draft-scheffenegger-tcpm-timestamp-negotiation-05

        

      

    
Here's the rationale behind the
        point both Richard and I tried to make in the tcpm WG just now:

        

        You could define your new option solely with the last two
        negotiation fields.

        

        Because the existing timestamp fields are already defined as
        opaque. So you can have a separate option just for the
        negotiation of what the existing timestamp option means (ie. its
        resolution).

        

      

    
Then, you don't force TCP to use a
        large option on the SYN, where space is critical. (But it  can
        use both options if there is space.)

        This also gives you a fall-back if the other peer doesn't
        support the new option, but does support the old timestamp
        option.

      

    

    The downside, is 2 extra bytes if TCP does use both options on the
    SYN. There might be other ways to arrange this, but the important
    thing is the requirements:

    * fall-back during incremental deployment

    * provide alternative ways to minimize space on SYN, so implementers
    can juggle priority of fitting in other options.

    

    Cheers

    

    

    Bob

    

    
On 03/11/2020 00:25, Yuchung Cheng
      wrote:

    

    

      
      

        
Hi tcpm & iccrg,

        


        

        
We are proposing a new TCP timestamp option to measure the
          network delay more precisely (e.g. excluding delayed ACK
          effects) for congestion control, e.g. Swift published SIGCOMM
          2020. The precision of the measurements can potentially be
          further enhanced by NIC hardware timestamps. We are working on
          a reference implementation for Linux as well.

        


        

        
We'll also present it in the upcoming tcpm meeting.
          Feedbacks are very welcome. 

        

        

          
---------- Forwarded message
            ---------

            From: <internet-drafts@ietf.org>

            Date: Mon, Nov 2, 2020 at 3:29 PM

            Subject: New Version Notification for
            draft-yang-tcpm-ets-00.txt

            To: Eric Dumazet <edumazet@google.com>,
            Yuchung Cheng <ycheng@google.com>,
            Neal Cardwell <ncardwell@google.com>,
            Kevin Yang (Yudong) <yyd@google.com>

          

          

          

          

          A new version of I-D, draft-yang-tcpm-ets-00.txt

          has been successfully submitted by Kevin (Yudong) Yang and
          posted to the

          IETF repository.

          

          Name:           draft-yang-tcpm-ets

          Revision:       00

          Title:          TCP ETS: Extensible Timestamp Options

          Document date:  2020-11-02

          Group:          Individual Submission

          Pages:          13

          URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-00.txt

          Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/

          Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm-ets

          Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00

          

          

          Abstract:

             This document presents ETS: an Extensible TimeStamps option
          for TCP.

             It allows hosts to use microseconds as the unit for
          timestamps to

             improve the precision of timestamps, and advertise the
          maximum ACK

             delay for its own delayed ACK mechanism.  Furthermore, it
          extends the

             information provided in the [RFC7323] TCP Timestamps Option
          by

             including the receiver delay in the TSecr echoing, so that
          the

             receiver of the ACK is able to more accurately estimate the
          portion

             of the RTT that resulted from time traveling through the
          network.

             The ETS option format is extensible, so that future
          extensions can

             add further information without the overhead of extra TCP
          option kind

             and length fields.

          

          

          

          

          Please note that it may take a couple of minutes from the time
          of submission

          until the htmlized version and diff are available at tools.ietf.org.

          

          The IETF Secretariat

          

          

        

      

      

      

      
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/
               PRIVILEGED AND CONFIDENTIAL

  


--------------8A0711D39555F5F55DB53DB2--


From nobody Tue Nov 17 00:11:13 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9BB623A03F3; Tue, 17 Nov 2020 00:11:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XPnpVhw-1XqT; Tue, 17 Nov 2020 00:11:06 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A58ED3A0E3C; Tue, 17 Nov 2020 00:10:30 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 0EB7525A15; Tue, 17 Nov 2020 09:10:29 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1605600629; bh=bJbkY24tYCz6JVQ7kw/T5tpPKW/3nw04mj4Cww/SA8c=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=kqriUfBhhgROK+Z5I2XurEwop6YNtw0lL1pKDds73tCwnQjup5r2PjHm9OlFYDwQ0 zruXKFeRpXSVClPk5qPSxEm/30nNzowohU30UetuIGCCGU8o4aSJBlM8AvOIy3G1El jLeTRTHmjCpxXd2oSb6cfjlR2qbhtFeKOeyuk10M=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1fftQpAtbu66; Tue, 17 Nov 2020 09:10:26 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Tue, 17 Nov 2020 09:10:26 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Tue, 17 Nov 2020 09:10:26 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Tue, 17 Nov 2020 09:10:26 +0100
From: "Scharf, Michael" 
To: Bob Briscoe , Yoshifumi Nishida 
CC: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
Thread-Topic: AccECN field order
Thread-Index: AQHWvDjFKAN34ufmsUi0uesNI2lMmanK+nPQgAC8GACAADPpcA==
Date: Tue, 17 Nov 2020 08:10:26 +0000
Message-ID: <30781ea61a794131beafe9997ed9221a@hs-esslingen.de>
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>  
In-Reply-To: 
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: multipart/alternative; boundary="_000_30781ea61a794131beafe9997ed9221ahsesslingende_"
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 08:11:10 -0000

--_000_30781ea61a794131beafe9997ed9221ahsesslingende_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

Qm9iLA0KDQpJIGFtIGZpbmUgd2l0aCB0aGUgdHdvIG9wdGlvbiBraW5kcyBwcm9wb3NlZCBpbiAt
MTMsIGJ1dCBJIGRvbuKAmXQgYnV5IHlvdXIgYXJndW1lbnRzIHdoeSB0aGlzIGVuY29kaW5nIGlz
IGJldHRlciB0aGFuIG90aGVycy4NCg0KTW9zdCBpbXBvcnRhbnRseSwgSSBkb27igJl0IHRoaW5r
IHRoYXQgeW91ciDigJxmb3J3YXJkIGNvbXBhdGliaWxpdHnigJ0gYXJndW1lbnQgaXMgYSB2ZXJ5
IGNvbXBlbGxpbmcgcmVhc29uIGZvciB0d28gY29kZXBvaW50cy4gQWxsIHByb3Bvc2FscyBJIGFt
IGF3YXJlIG9mIGhhdmUgdGhlaXIgcHJvcyBhbmQgY29ucy4gQW5kIEkgYW0gbm90IGF3YXJlIG9m
IGEgcmVhbGx5IGNvbXByZWhlbnNpdmUgZGlzY3Vzc2lvbi4gQXQgbGVhc3QgbmVpdGhlciB0aGUg
ZS1tYWlsIGJlbG93IG5vciB0b2RheeKAmXMgbWVldGluZyBkaXNjdXNzIGFsbCB0cmFkZW9mZnMg
SSB3b3VsZCBpbnZlc3RpZ2F0ZS4NCg0KRm9yIGluc3RhbmNlLCB0aGUgYmVuZWZpdCBvZiB0aGUg
bGVuZ3RoIGVuY29kaW5nIHZhcmlhbnQgd291bGQgYmUgdG8gY29uc3VtZSBvbmx5IG9uZSBUQ1Ag
b3B0aW9uIGNvZGVwb2ludCBub3cuIElmIHdlIGRlY2lkZWQgbGF0ZXIgdG8gdGhhdCBhIGZsYWcg
Ynl0ZSBpcyBuZWVkZWQgKHNheSwgYXQgdGhlIGVuZCksIGEgZm9sbG93LXVwIHByb3Bvc2VkIHN0
YW5kYXJkIGNvdWxkIHNwZWNpZnkgYW5vdGhlciBvcHRpb24gZm9ybWF0IHdpdGggdGhhdCBmbGFn
IGJ5dGUsIHVzaW5nIGEgc2Vjb25kIGNvZGVwb2ludC4NCg0KUmVnYXJkaW5nIHRoZSByZXN1bHRp
bmcgY29kZXBvaW50IGNvbnN1bXB0aW9uICh0d28gb3B0aW9uIGtpbmRzKSBhbmQgdGhlIGFkZGl0
aW9uIG9mIGEgZmxhZyBieXRlLCB0aGlzIGFwcHJvYWNoIHdvdWxkIGJhc2ljYWxseSBlbmQgdXAg
d2l0aCB0aGUgc2FtZSByZXN1bHQgYXMgdGhlIGN1cnJlbnQgcHJvcG9zYWwgaW4gLTEzIHBsdXMg
c29tZSBoeXBvdGhldGljYWwgZnV0dXJlIGFkZGl0aW9uIGxldmVyYWdpbmcgdGhlIGxlbmd0aCBm
aWVsZCBiZXlvbmQgdGhlIGN1cnJlbnRseSBrbm93biB1c2UuDQoNCkFuZCwgeWVzLCB0aGUgbGVu
Z3RoIGVuY29kaW5nIHZhcmlhbnQgbWF5IGJlIGxlc3MgZmxleGlibGUgYW5kL29yIGNvbnN1bWUg
c29tZSBtb3JlIGJpdHMgaW4gc29tZSBjYXNlcywgYnV0IG9uIHRoZSBwbHVzIHNpZGUgaXQgb25s
eSBuZWVkcyBvbmUgY29kZXBvaW50IG5vdyDigJMgYW5kIGdpdmVuIHRoYXQgaXQgaXMgZW50aXJl
bHkgdW5jbGVhciB3aGV0aGVyIGFueSBBY2NFQ04gb3B0aW9uIHdpbGwgaW5kZWVkIGJlIHdpZGVs
eSB1c2VkIGluIGZ1dHVyZSwgdGhpcyBpcyBhIGJpZyBwbHVzLiBUaGUgZmFjdCB0aGF0IGltcGxl
bWVudGVycyBhcmUgcXVpdGUgc2lsZW50IG9uIHRoZSBvcHRpb24gZGVzaWduIGlzIG5vdCBhIGdv
b2Qgc2lnbi4NCg0KVG8gbWUsIG9uZSBwb3RlbnRpYWwgZGlmZmVyZW5jZSBiZXR3ZWVuIHR3byBw
cm9wb3NhbHMgd291bGQgYmUgaW5jcmVtZW50YWwgZGVwbG95bWVudC4gVGhlIHByb3Bvc2FsIGlu
IC0xMyBvbmx5IGhhcyBhbiBhZHZhbnRhZ2UgaWYgbWlkZGxlYm94ZXMgc3VjaCBhcyBmaXJld2Fs
bHMgd2lsbCBpbmRlZWQgcGFzcyBUQ1Agb3B0aW9ucyB3aXRoIGEgZm9ybWF0IHRoYXQgY29udGFp
bnMgY29udGVudCBiZXlvbmQgdGhlIChmaXJzdCkgQWNjdXJhdGUgRUNOIHN0YW5kYXJkIChpLmUu
LCBjdXJyZW50bHkgdW51c2VkIGxlbmd0aCB2YWx1ZXMpLiBJTUhPIGl0IGlzIHRvbyBlYXJseSB0
byBrbm93IHdoZXRoZXIgZmlyZXdhbGxzIHdvdWxkIGluZGVlZCBhbGxvdyB0aGlzIGluIGZ1dHVy
ZS4NCg0KRnJvbSBhIHNlY3VyaXR5IHBlcnNwZWN0aXZlLCBpdCBpcyBub3QgY2xlYXIgdG8gbWUg
d2hldGhlciBhbGxvd2luZyBhcmJpdHJhcnkgdW5zcGVjaWZpZWQgYnl0ZXMgaW4gYSBUQ1Agb3B0
aW9uIGlzIGEgZ29vZCBpZGVhICphdCBhbGwqLiBJdCB3aWxsIGJlIGludGVyZXN0aW5nIHRvIGhl
YXIgdGhlIG9waW5pb24gZnJvbSBTRUMgYXJlYSBvbiB0aGF0LiBQZXJzb25hbGx5LCBJIGFtIG5v
dCBjb252aW5jZWQgdGhhdCB0aGlzIHJlYWxseSBtYWtlcyBzZW5zZSwgYnV0IEkgbXkgY29uY2Vy
bnMgYXJlIG5vdCBzdHJvbmcgZW5vdWdoIHRvIGZvcm1hbGx5IHB1c2ggYmFjay4gSeKAmWxsIGxl
YXZlIGl0IHRvIG90aGVycyB0byB0aGluayBhYm91dCB3aGV0aGVyIHRoaXMgaXMgYSBidWcgb3Ig
YSBmZWF0dXJlLg0KDQpNYXliZSBvbmUgbGVzc29uIGxlYXJudCBpcyB0aGF0IHRoZSBkb2N1bWVu
dCBjb3VsZCBoYXZlIGEgbm9uLW5vcm1hdGl2ZSBhcHBlbmRpeCB0aGF0IGV4cGxhaW5zIHRoZSBy
YXRpb25hbGUgZm9yIHRoZSBmaW5hbGx5IHBpY2tlZCBUQ1Agb3B0aW9uIGVuY29kaW5nLiBUaGF0
IG1heSBhbHNvIGhlbHAgaWYgdGhlcmUgYXJlIGZ1cnRoZXIgcXVlc3Rpb25zIHdoZXRoZXIgdHdv
IGNvZGVwb2ludHMgYXJlIHJlYWxseSByZXF1aXJlZCwgZS5nLiBieSB0aGUgSUVTRyAoaWYgdHdv
IGNvZGVwb2ludHMgYXJlIHN0aWxsIHRoZSBkZXNpZ24gYWZ0ZXIgV0dMQykuIEF0IGxlYXN0IGZv
ciBwYXN0IFRDUCBvcHRpb24gY29kZXBvaW50IGFsbG9jYXRpb25zIEkgcmVjYWxsIHNvbWUgZGlz
Y3Vzc2lvbnMgbGF0ZSBpbiB0aGUgSUVURiBwcm9jZXNzLiBJbiB0aG9zZSBwYXN0IGNhc2VzLCBn
b29kIGFyZ3VtZW50cyBpbiBhbiBhcHBlbmRpeCBhbmQgcnVubmluZyBjb2RlIGhhcyBoZWxwZWQg
YSBsb3QuDQoNCk1pY2hhZWwNCg0KDQoNCkZyb206IEJvYiBCcmlzY29lIDxpZXRmQGJvYmJyaXNj
b2UubmV0Pg0KU2VudDogVHVlc2RheSwgTm92ZW1iZXIgMTcsIDIwMjAgNjoxMCBBTQ0KVG86IFNj
aGFyZiwgTWljaGFlbCA8TWljaGFlbC5TY2hhcmZAaHMtZXNzbGluZ2VuLmRlPjsgWW9zaGlmdW1p
IE5pc2hpZGEgPG5zZC5pZXRmQGdtYWlsLmNvbT4NCkNjOiBNaWNoYWVsIFR1ZXhlbiA8dHVleGVu
QGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IE1pcmphIEt1ZWhsZXdpbmQg
PGlldGZAa3VlaGxld2luZC5uZXQ+OyBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxSaWNoYXJkLlNj
aGVmZmVuZWdnZXJAbmV0YXBwLmNvbT47IHRjcG0gSUVURiBsaXN0IDx0Y3BtQGlldGYub3JnPg0K
U3ViamVjdDogUmU6IEFjY0VDTiBmaWVsZCBvcmRlcg0KDQpNaWNoYWVsLA0KT24gMTYvMTEvMjAy
MCAxNzozNiwgU2NoYXJmLCBNaWNoYWVsIHdyb3RlOg0KQm9iLA0KDQpPbmUgcHJvcG9zYWwgdXNp
bmcgdGhlIGxlbmd0aCBmaWVsZCB3aXRoICpvbmUgb3B0aW9uIGNvZGVwb2ludCBvbmx5KiBpcyBk
ZXRhaWxlZCBpbjogaHR0cHM6Ly9tYWlsYXJjaGl2ZS5pZXRmLm9yZy9hcmNoL21zZy90Y3BtL3pv
LTFPUjBuUmZoSG9jWDh5dlR2cEM0Qk5Nby8NCg0KSXQgaXMgdGhlIHRoaXJkIG9wdGlvbiBtZW50
aW9uZWQgaW4gdGhpcyBlLW1haWwuIE9uZSBleGFtcGxlIHdvdWxkIGJlIHRvIHVzZSBvcHRpb24g
bGVuZ3RoIHZhbHVlcyA1LzgvMTEgZm9yIG9uZSBlbmNvZGluZyB0eXBlIGFuZCBvcHRpb24gbGVu
Z3RoIHZhbHVlcyA2LzkvMTIgZm9yIHRoZSBvdGhlciBlbmNvZGluZyB0eXBlIChpLmUuLCBvcmRl
ciBvZiBmaWVsZHMpLiBPciBvbmUgY291bGQgdXNlIHNvbWUgb3RoZXIgY29tYmluYXRpb24gb2Yg
bGVuZ3RoIHZhbHVlcyDigJMgdGhlIG9ubHkgcmVxdWlyZW1lbnQgaXMgdGhhdCBhIGNlcnRhaW4g
dmFsdWUgZm9yIHRoZSBvcHRpb24gbGVuZ3RoIGlzIG9ubHkgdXNlZCBieSBvbmUgb2YgdGhlIG9w
dGlvbiBmb3JtYXRzLiBJbiB0aGF0IGFwcHJvYWNoLCB0aGUgdmFsdWUgb2YgdGhlIGxlbmd0aCBm
aWVsZCB3b3VsZCB0aHVzIGRpcmVjdGx5IGRlc2NyaWJlIHRoZSBlbmNvZGluZyBvZiB0aGUgb3B0
aW9uLiBVbmxlc3MgSSBtaXNzIHNvbWV0aGluZywgdGhpcyB3b3VsZCB3b3JrIGFuZCBpdCB3b3Vs
ZCBqdXN0IHJlcXVpcmUgb25lIG9wdGlvbiBjb2RlcG9pbnQuDQoNClRodXMsIGFsdGVybmF0aXZl
cyB0byB0d28gb3B0aW9uIGNvZGVwb2ludHMgZXhpc3QgYW5kIEkgaGF2ZSBleHBsYWluZWQgdGhl
bSBvbiB0aGUgbGlzdCBpbiBNYXJjaCAyMDIwLg0KDQpPSywgc29ycnksIHllcywgSSByZW1lbWJl
ciB0aGlzIG5vdy4NCg0KQXMgSSB3aWxsIGV4cGxhaW4gaW4gdGhlIEFjY0VDTiBzdGF0dXMgdXBk
YXRlIHRhbGsgdG9kYXkgaW4gdmlydHVhbCBCYW5na29rLCB0aGUgZHJhZnQgaGFzIG1hZGUgcHJv
dmlzaW9uIGZvciBkaWZmZXJlbnQgbGVuZ3RoIHZhbHVlcyB0aGFuIDUvOC8xMS4gSXQgc2F5cyBl
eGlzdGluZyBpbXBsZW1lbnRhdGlvbnMgTVVTVCBhY2NlcHQgbGVuZ3RoIHZhbHVlcyBvdGhlciB0
aGFuIHRob3NlIGN1cnJlbnRseSBkZWZpbmVkLiBCdXQgdGhlbiByZWFkIGluIGFzIG1hbnkgd2hv
bGUgMy1ieXRlIGZpZWxkcyBhcyB0aGV5IGNhbi4NCg0KVGhpcyBjYW4gYmUgdXNlZCB0byBhZGQg
YSBmbGFncyBieXRlIG9uIHRoZSBlbmQgaW4gZnV0dXJlLCBmb3IgZXh0ZW5zaWJpbGl0eS4gT3Ig
YW55IG90aGVyIGZvcm0gb2YgZXh0ZW5zaWJpbGl0eSB0aGUgV0cgbWlnaHQgZGVjaWRlIGluIHRo
ZSBmdXR1cmUuDQoNCkkga25vdyBhIGZsYWdzIGJ5dGUgYXQgdGhlIGVuZCBzZWVtcyBvZGQgY29t
cGFyZWQgdG8gYXQgdGhlIGJlZ2lubmluZy4gQnV0IChpZiBkZWNpZGVkIGl0J3MgbmVlZGVkIGlu
IGZ1dHVyZSkgaXQncyByZWFzb25hYmx5IGVhc3kgdG8gaW1wbGVtZW50IGJ5IHJlYWRpbmcgdGhl
IHdob2xlIG9wdGlvbiwgdGhlbiBwcm9jZXNzaW5nIHRoZSBsYXN0IGJ5dGUsIGJlZm9yZSByZWFk
aW5nIHRoZSByZXN0IG9mIHRoZSBvcHRpb24uDQoNCkkgYmVsaWV2ZSB5b3Ugd2lsbCBhZ3JlZSB0
aGF0IHRoaXMgaXMgYSBiZXR0ZXIgd2F5IHRvIHV0aWxpemUgZGlmZmVyZW50IGxlbmd0aHMuDQoN
CkFuZCB0aGFuayB5b3UgZm9yIHJlcGVhdGVkbHkgZW1waGFzaXppbmcgdGhhdCB5b3UncmUgaGFw
cHkgd2l0aCB0aGUgMi1raW5kIHNjaGVtZSwgb3Igb3RoZXIgYWx0ZXJuYXRpdmVzLg0KDQoNCkJv
Yg0KDQoNCg0KQW55d2F5LCBJIGRvbuKAmXQgcmVhbGx5IGNhcmUgaG93IHRoZSBvcHRpb25zIGFy
ZSBlbmNvZGVkIGFzIGxvbmcgYXMgdGhlIHJlY2VpdmVyIGRvZXNu4oCZdCBuZWVkIHBlci1jb25u
ZWN0aW9uIHN0YXRlIGZvciBkZWNvZGluZyBhIFRDUCBvcHRpb24uIFNvLCBwZXJzb25hbGx5LCBJ
IHdvdWxkIGJlIGZpbmUgd2l0aCB1c2luZyBlLmcuIHRoZSBsZW5ndGggZmllbGQgYXMgZGVzY3Jp
YmVkIGluIG15IG9sZCBlLW1haWwuIE9yIGFuIGFkZGl0aW9uYWwgZmxhZyBieXRlLiBBbmQgb25l
IGNvdWxkIGNvbWUgdXAgd2l0aCBmdXJ0aGVyIGVuY29kaW5ncywgZS5nLiwgYnkgdXNpbmcgb25l
IG9yIGEgZmV3IGJpdHMgYXMgYSBzaG9ydCDigJx0eXBl4oCdIGZpZWxkIGZvciBlYWNoIGNvdW50
ZXIuIFRoaXMgaXMgYWxsIGFib3V0IHByb3RvY29sIGVuZ2luZWVyaW5nLiBBbmQgYWxsIHRoZXNl
IHZhcmlhbnRzIGhhdmUgdGhlaXIgcHJvcyBhbmQgY29ucy4NCg0KSSBhbSBhbHNvIGZpbmUgd2l0
aCB1c2luZyB0d28gb3B0aW9uIGNvZGVwb2ludHMgYXMgc3BlY2lmaWVkIGluIC0xMzsgdGhpcyBp
cyBwcm9iYWJseSB0aGUgYXBwcm9hY2ggdGhhdCBjb25zdW1lcyB0aGUgbGVhc3QgbnVtYmVyIG9m
IGJpdHMuDQoNCk1pY2hhZWwgKHcvbyBhbnkgaGF0KQ0KDQoNCg0KRnJvbTogQm9iIEJyaXNjb2Ug
PGlldGZAYm9iYnJpc2NvZS5uZXQ+PG1haWx0bzppZXRmQGJvYmJyaXNjb2UubmV0Pg0KU2VudDog
TW9uZGF5LCBOb3ZlbWJlciAxNiwgMjAyMCA1OjUyIFBNDQpUbzogWW9zaGlmdW1pIE5pc2hpZGEg
PG5zZC5pZXRmQGdtYWlsLmNvbT48bWFpbHRvOm5zZC5pZXRmQGdtYWlsLmNvbT4NCkNjOiBTY2hh
cmYsIE1pY2hhZWwgPE1pY2hhZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT48bWFpbHRvOk1pY2hh
ZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT47IE1pY2hhZWwgVHVleGVuIDx0dWV4ZW5AZmgtbXVl
bnN0ZXIuZGU+PG1haWx0bzp0dWV4ZW5AZmgtbXVlbnN0ZXIuZGU+OyB0Y3BtLWNoYWlyc0BpZXRm
Lm9yZzxtYWlsdG86dGNwbS1jaGFpcnNAaWV0Zi5vcmc+OyBNaXJqYSBLdWVobGV3aW5kIDxpZXRm
QGt1ZWhsZXdpbmQubmV0PjxtYWlsdG86aWV0ZkBrdWVobGV3aW5kLm5ldD47IFNjaGVmZmVuZWdn
ZXIsIFJpY2hhcmQgPFJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tPjxtYWlsdG86Umlj
aGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20+OyB0Y3BtIElFVEYgbGlzdCA8dGNwbUBpZXRm
Lm9yZz48bWFpbHRvOnRjcG1AaWV0Zi5vcmc+DQpTdWJqZWN0OiBBY2NFQ04gZmllbGQgb3JkZXIN
Cg0KWW9zaGksIChhZGRpbmcgdGhlIHRjcG0gbGlzdCBpbiBjYykNCk9uIDA1LzExLzIwMjAgMDY6
NTgsIFlvc2hpZnVtaSBOaXNoaWRhIHdyb3RlOg0KSGkgQm9iLA0KDQpPbiBXZWQsIE5vdiA0LCAy
MDIwIGF0IDM6MjkgUE0gQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2NvZS5uZXQ8bWFpbHRvOmll
dGZAYm9iYnJpc2NvZS5uZXQ+PiB3cm90ZToNCllvc2hpLA0KT24gMDQvMTEvMjAyMCAwNjo1MSwg
WW9zaGlmdW1pIE5pc2hpZGEgd3JvdGU6DQpIaSwgZm9sa3MsDQoNCg0KDQpJbiBteSB1bmRlcnN0
YW5kaW5nLCBJJ20gbm90IHN1cmUgaWYgd2Ugc2V0dGxlZCBkb3duIG9uIHVzaW5nIHR3byBvcHRp
b24ga2luZHMgb3IgZW5jb2Rpbmcgc2NoZW1lcyBmb3IgMjRiaXRzIGZpZWxkcyBpbiBhY2MgZWNu
IGRyYWZ0Lg0KU28sIEkgdGhpbmsgdGhlcmUncmUgc3RpbGwgc29tZXRoaW5nIHRvIGJlIGNsYXJp
ZmllZCBhbmQgaG9wZSB0aGluZ3Mgd2lsbCBiZSBzZXR0bGVkIGF0IHRoZSBtZWV0aW5nLg0KDQpb
QkJdIEkga25vdyBhIFdHIGNhbiBjaGFuZ2UgaXQncyBtaW5kIGF0IGFueSB0aW1lLiBCdXQgSSdk
IHJhdGhlciB3ZSBqdXN0IGNsYXJpZmllZCB3aGF0IGEgcHJldmlvdXMgZGVjaXNpb24gd2FzLCB0
byBhdm9pZCB0aGUgbmVlZCB0byBrZWVwIHJlLW9wZW5pbmcgZGlzY3Vzc2lvbiBvbiBhIHF1ZXN0
aW9uIHRoYXQgaGF2ZSBiZWVuIGRlY2lkZWQgdGhlbiBjaGFuZ2VkIHRocmVlIGRpZmZlcmVudCB3
YXlzIGFscmVhZHkuDQoNCk15IG1lbW9yeSBpcyBub3Qgc28gZ29vZCB0aGVzZSBkYXlzLiBJIHRy
dXN0ZWQgdGhhdCBNaWNoYWVsIFMgcmVtZW1iZXJlZCB0aGUgZGVjaXNpb24gY29ycmVjdGx5LCBh
bmQgSSBzZWVtIHRvIHJlbWVtYmVyIHRoYXQgZGVjaXNpb24gYmVpbmcgbWFkZS4NCkkndmUganVz
dCBjaGVja2VkIHRoZSBtaW51dGVzIG9mIHRoZSBsYXN0IGludGVyaW06DQogICAgaHR0cHM6Ly9k
YXRhdHJhY2tlci5pZXRmLm9yZy9kb2MvbWludXRlcy1pbnRlcmltLTIwMjAtdGNwbS0wMS0yMDIw
MDQyOTE2MDAvDQphbmQgdGhleSBtZW50aW9uIE1pY2hhZWwncyBwcm9wb3NhbCB0byB1c2UgdHdv
IGtpbmRzLCBidXQgZG9uJ3QgcmVjb3JkIGFueSBkZWNpc2lvbi4NClRoZSBqYWJiZXIgbG9nIGdp
dmVzIG5vIGNsdWVzIGFib3V0IGFueSBkZWNpc2lvbi4NCg0KSSBjYW4ndCBmaW5kIGFuIGF1ZGlv
IG9yIHZpZGVvIHJlY29yZGluZy4gQ2FuIHlvdSBwb2ludCBtZSBhdCBvbmU/DQoNCkkgdGhvdWdo
dCB0aGF0IGl0J3MgYmVjYXVzZSB0aGVyZSB3YXMgbm8gY2xlYXIgZGVjaXNpb24gYXQgdGhlIG1l
ZXRpbmcuDQpCdXQsIHlvdSBjYW4gY2hlY2sgaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/
dj1Lc0IwX05iOC1rQQ0KUGxlYXNlIGxldCB1cyBrbm93IGlmIHlvdSBoYXZlIGFueSBxdWVzdGlv
bnMgb3Igb3BpbmlvbnMgd2l0aCByZWdhcmQgdG8gdGhpcy4NCg0KW0JCXSBJIGNoZWNrZWQgdGhl
IFlvdXR1YmUgbGluayB5b3Ugc2VudCBiZWxvdy4NCg0KRmlyc3QgSSB0aGluayB3ZSdyZSBhZ3Jl
ZWQgbm8tb25lIHdhcyBmaWdodGluZyBmb3IgdXMgdG8ga2VlcCB0aGUgcHJldmlvdXMgd2F5IHdl
IGRpZCB0aGlzICh1c2luZyB0aGUgaW5pdGlhbCB2YWx1ZSBvZiB0aGUgZmllbGQgdG8gc2V0IHRo
ZSBvcmRlciBmb3IgdGhlIGNvbm5lY3Rpb24pLg0KSW4gbXkgcHJlc2VudGF0aW9uIEkgc2FpZCB0
aGVyZSB3YXMgc3Ryb25nIHJlc2lzdGFuY2UgZnJvbSBNaWNoYWVsIHRvIGRvIGl0IGEgZGlmZmVy
ZW50IHdheS4NCihhbHNvLCBvZmZsaXN0LCB0aGUgY28tYXV0aG9ycyBpbmNsdWRpbmcgbWUgYWxz
byBkaWRuJ3QgbGlrZSB0aGlzIHNvIG11Y2guIEFuZCBJbHBvIHNhaWQgaXQgbWFkZSB0aGUgaW1w
bGVtZW50YXRpb24gY29tcGxleC4pDQoNClRoZW4gY2FtZSB0aGUgcXVlc3Rpb24gb2Ygd2hhdCB3
ZSBkbyBpbnN0ZWFkLiBUaGVyZSB3ZXJlIHRocmVlIGFsdGVybmF0aXZlIHByb3Bvc2FsczoNCmEp
IHVzZSAyIG9wdGlvbiBraW5kcw0KYikgYWRkIGEgZmxhZ3MgYnl0ZQ0KYykgc29tZWhvdyB1c2Ug
dGhlIGxlbmd0aCBmaWVsZCBtYXliZQ0KDQpNaWNoYWVsIHJhaXNlZCAoYykgaW4gdGhlIG1lZXRp
bmcgYXMgYSBwb3NzaWJpbGl0eSwgYnV0IG5vLW9uZSBjb3VsZCB0aGluayBob3cgdG8gZGlzdGlu
Z3Vpc2ggdHdvIG9wdGlvbnMgb2YgdGhlIHNhbWUgbGVuZ3RoIGJ1dCBhIGRpZmZlcmVudCBmaWVs
ZCBvcmRlciB1c2luZyB0aGUgbGVuZ3RoIGZpZWxkLiBNaWNoYWVsIHNhaWQgaGUnZCBwb3N0IGFu
eSBpZGVhcyB0byB0aGUgbGlzdCBpZiBoZSBjb3VsZCB0aGluayBvZiBhbnksIGJ1dCB0aGF0IGRp
ZG4ndCBoYXBwZW4uDQoNClNvIHdlJ3JlIGxlZnQgY2hvb3NpbmcgYmV0d2VlbiAoYSkgYW5kIChi
KS4NCkkgc2FpZCBpbiB0aGUgbWVldGluZyAoYW5kIG9uIHRoZSBsaXN0IHdoZW4gZGlzY3Vzc2lu
ZyB3aXRoIElscG8pIHRoYXQgSSdkIGJlIGhhcHB5IHRvIGdvIHdpdGggKGIpLCBidXQgb25seSBp
ZiB0aGVyZSB3YXMgYW5vdGhlciB1c2UgZm9yIGEgZmxhZy4gQmVjYXVzZSBpdCB3b3VsZCBjb25z
dW1lIDFCIG1vcmUgb3B0aW9ucyBzcGFjZSBpbiBtYW55IHBhY2tldHMsIHdoaWNoIGlzIGEgc2Nh
cmNlIHJlc291cmNlLg0KDQpJbHBvIGhhZCBhIHByb3Bvc2VkIHVzZSBmb3IgYW5vdGhlciBmbGFn
ICh0byBoZWxwIHN5bmNoIGNvdW50ZXJzIGFmdGVyIGEgbG9zcyksIGJ1dCBJIHRoaW5rIHRoZSBk
aXNjdXNzaW9uIGFib3V0IGl0IGVuZGVkIHRoYXQgaXQgd291bGRuJ3QgYmUgaGVscGZ1bCwgJ2Nv
cyB0aGUgd2F5IGl0IHdvcmtlZCBkZXBlbmRlZCBvbiBpdHNlbGYgKGNpcmN1bGFyIGxvZ2ljKS4N
Cg0KSW4gY29uY2x1c2lvbiwgSSBkb24ndCB0aGluayB0aGVyZSB3YXMgYW4gZXhwbGljaXQgZGVj
aXNpb24gdG8gZ28gd2l0aCAyIG9wdGlvbiBraW5kcywgYnV0IGl0IGVuZGVkIHVwIGFzIHRoZSAn
bGFzdCBwZXJzb24gc3RhbmRpbmcnLg0KSSBsaWtlIGl0LiBJdCdzIHNpbXBsZS4gQW5kIGFwcGFy
ZW50bHkgb3B0aW9uIGtpbmRzIGFyZSBub3Qgc3VjaCBhIHNjYXJjZSByZXNvdXJjZS4NCg0KUGVy
aGFwcyB3ZSBjYW4gcmF0aWZ5IHRoaXMgaW4gdGhlIFdHIHRvbW9ycm93Lg0KDQoNCkJvYg0KDQoN
Cg0KDQpUaGFua3MsDQotLQ0KWW9zaGkNCg0KDQoNCg0KLS0NCg0KX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KDQpCb2IgQnJp
c2NvZSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBodHRwOi8vYm9iYnJpc2NvZS5uZXQv
DQoNCg0KDQotLQ0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fDQoNCkJvYiBCcmlzY29lICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIGh0dHA6Ly9ib2JicmlzY29lLm5ldC8NCg==

--_000_30781ea61a794131beafe9997ed9221ahsesslingende_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDb25zb2xhczsNCglwYW5vc2UtMToyIDEx
IDYgOSAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWws
IGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0
b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixz
YW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9y
aXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwcmUN
Cgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZvcmZvcm1h
dGllcnQgWmNobiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9u
dC1zaXplOjEwLjBwdDsNCglmb250LWZhbWlseToiQ291cmllciBOZXciO30NCnNwYW4uSFRNTFZv
cmZvcm1hdGllcnRaY2huDQoJe21zby1zdHlsZS1uYW1lOiJIVE1MIFZvcmZvcm1hdGllcnQgWmNo
biI7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZvcmZv
cm1hdGllcnQiOw0KCWZvbnQtZmFtaWx5OkNvbnNvbGFzO30NCnNwYW4uRS1NYWlsRm9ybWF0dm9y
bGFnZTIxDQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OiJD
YWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1bHQN
Cgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBwdDt9DQpAcGFn
ZSBXb3JkU2VjdGlvbjENCgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2luOjcwLjg1cHQg
NzAuODVwdCAyLjBjbSA3MC44NXB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2Vj
dGlvbjE7fQ0KLS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVm
YXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48
IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxv
OmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwh
W2VuZGlmXS0tPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iREUiIGxpbms9ImJsdWUiIHZsaW5rPSJw
dXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+Qm9iLDxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+SSBhbSBmaW5lIHdpdGggdGhlIHR3byBvcHRpb24ga2luZHMgcHJvcG9zZWQgaW4g
LTEzLCBidXQgSSBkb27igJl0IGJ1eSB5b3VyIGFyZ3VtZW50cyB3aHkgdGhpcyBlbmNvZGluZyBp
cyBiZXR0ZXIgdGhhbiBvdGhlcnMuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpF
Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+TW9z
dCBpbXBvcnRhbnRseSwgSSBkb27igJl0IHRoaW5rIHRoYXQgeW91ciDigJxmb3J3YXJkIGNvbXBh
dGliaWxpdHnigJ0gYXJndW1lbnQgaXMgYSB2ZXJ5IGNvbXBlbGxpbmcgcmVhc29uIGZvciB0d28g
Y29kZXBvaW50cy4gQWxsIHByb3Bvc2FscyBJIGFtIGF3YXJlIG9mIGhhdmUgdGhlaXIgcHJvcyBh
bmQgY29ucy4gQW5kIEkNCiBhbSBub3QgYXdhcmUgb2YgYSByZWFsbHkgY29tcHJlaGVuc2l2ZSBk
aXNjdXNzaW9uLiBBdCBsZWFzdCBuZWl0aGVyIHRoZSBlLW1haWwgYmVsb3cgbm9yIHRvZGF54oCZ
cyBtZWV0aW5nIGRpc2N1c3MgYWxsIHRyYWRlb2ZmcyBJIHdvdWxkIGludmVzdGlnYXRlLjxvOnA+
PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVT
IiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9z
cGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0i
bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkZvciBpbnN0YW5jZSwgdGhlIGJlbmVmaXQgb2Yg
dGhlIGxlbmd0aCBlbmNvZGluZyB2YXJpYW50IHdvdWxkIGJlIHRvIGNvbnN1bWUgb25seSBvbmUg
VENQIG9wdGlvbiBjb2RlcG9pbnQgbm93LiBJZiB3ZSBkZWNpZGVkIGxhdGVyIHRvIHRoYXQgYSBm
bGFnIGJ5dGUgaXMgbmVlZGVkIChzYXksIGF0IHRoZSBlbmQpLCBhDQogZm9sbG93LXVwIHByb3Bv
c2VkIHN0YW5kYXJkIGNvdWxkIHNwZWNpZnkgYW5vdGhlciBvcHRpb24gZm9ybWF0IHdpdGggdGhh
dCBmbGFnIGJ5dGUsIHVzaW5nIGEgc2Vjb25kIGNvZGVwb2ludC48bzpwPjwvbzpwPjwvc3Bhbj48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1m
YXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxh
bmd1YWdlOkVOLVVTIj5SZWdhcmRpbmcgdGhlIHJlc3VsdGluZyBjb2RlcG9pbnQgY29uc3VtcHRp
b24gKHR3byBvcHRpb24ga2luZHMpIGFuZCB0aGUgYWRkaXRpb24gb2YgYSBmbGFnIGJ5dGUsIHRo
aXMgYXBwcm9hY2ggd291bGQgYmFzaWNhbGx5IGVuZCB1cCB3aXRoIHRoZSBzYW1lIHJlc3VsdCBh
cyB0aGUgY3VycmVudCBwcm9wb3NhbCBpbg0KIC0xMyBwbHVzIHNvbWUgaHlwb3RoZXRpY2FsIGZ1
dHVyZSBhZGRpdGlvbiBsZXZlcmFnaW5nIHRoZSBsZW5ndGggZmllbGQgYmV5b25kIHRoZSBjdXJy
ZW50bHkga25vd24gdXNlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt
YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMi
PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkFuZCwgeWVz
LCB0aGUgbGVuZ3RoIGVuY29kaW5nIHZhcmlhbnQgbWF5IGJlIGxlc3MgZmxleGlibGUgYW5kL29y
IGNvbnN1bWUgc29tZSBtb3JlIGJpdHMgaW4gc29tZSBjYXNlcywgYnV0IG9uIHRoZSBwbHVzIHNp
ZGUgaXQgb25seSBuZWVkcyBvbmUgY29kZXBvaW50IG5vdyDigJMgYW5kIGdpdmVuIHRoYXQgaXQg
aXMgZW50aXJlbHkNCiB1bmNsZWFyIHdoZXRoZXIgYW55IEFjY0VDTiBvcHRpb24gd2lsbCBpbmRl
ZWQgYmUgd2lkZWx5IHVzZWQgaW4gZnV0dXJlLCB0aGlzIGlzIGEgYmlnIHBsdXMuIFRoZSBmYWN0
IHRoYXQgaW1wbGVtZW50ZXJzIGFyZSBxdWl0ZSBzaWxlbnQgb24gdGhlIG9wdGlvbiBkZXNpZ24g
aXMgbm90IGEgZ29vZCBzaWduLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRvIG1l
LCBvbmUgcG90ZW50aWFsIGRpZmZlcmVuY2UgYmV0d2VlbiB0d28gcHJvcG9zYWxzIHdvdWxkIGJl
IGluY3JlbWVudGFsIGRlcGxveW1lbnQuIFRoZSBwcm9wb3NhbCBpbiAtMTMgb25seSBoYXMgYW4g
YWR2YW50YWdlIGlmIG1pZGRsZWJveGVzIHN1Y2ggYXMgZmlyZXdhbGxzIHdpbGwgaW5kZWVkIHBh
c3MgVENQDQogb3B0aW9ucyB3aXRoIGEgZm9ybWF0IHRoYXQgY29udGFpbnMgY29udGVudCBiZXlv
bmQgdGhlIChmaXJzdCkgQWNjdXJhdGUgRUNOIHN0YW5kYXJkIChpLmUuLCBjdXJyZW50bHkgdW51
c2VkIGxlbmd0aCB2YWx1ZXMpLiBJTUhPIGl0IGlzIHRvbyBlYXJseSB0byBrbm93IHdoZXRoZXIg
ZmlyZXdhbGxzIHdvdWxkIGluZGVlZCBhbGxvdyB0aGlzIGluIGZ1dHVyZS48bzpwPjwvbzpwPjwv
c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9
Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJl
YXN0LWxhbmd1YWdlOkVOLVVTIj5Gcm9tIGEgc2VjdXJpdHkgcGVyc3BlY3RpdmUsIGl0IGlzIG5v
dCBjbGVhciB0byBtZSB3aGV0aGVyIGFsbG93aW5nIGFyYml0cmFyeSB1bnNwZWNpZmllZCBieXRl
cyBpbiBhIFRDUCBvcHRpb24gaXMgYSBnb29kIGlkZWEgKjxiPmF0IGFsbDwvYj4qLiBJdCB3aWxs
IGJlIGludGVyZXN0aW5nIHRvIGhlYXIgdGhlIG9waW5pb24NCiBmcm9tIFNFQyBhcmVhIG9uIHRo
YXQuIFBlcnNvbmFsbHksIEkgYW0gbm90IGNvbnZpbmNlZCB0aGF0IHRoaXMgcmVhbGx5IG1ha2Vz
IHNlbnNlLCBidXQgSSBteSBjb25jZXJucyBhcmUgbm90IHN0cm9uZyBlbm91Z2ggdG8gZm9ybWFs
bHkgcHVzaCBiYWNrLiBJ4oCZbGwgbGVhdmUgaXQgdG8gb3RoZXJzIHRvIHRoaW5rIGFib3V0IHdo
ZXRoZXIgdGhpcyBpcyBhIGJ1ZyBvciBhIGZlYXR1cmUuPG86cD48L286cD48L3NwYW4+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+TWF5YmUgb25lIGxlc3NvbiBsZWFybnQgaXMgdGhhdCB0aGUgZG9jdW1lbnQgY291
bGQgaGF2ZSBhIG5vbi1ub3JtYXRpdmUgYXBwZW5kaXggdGhhdCBleHBsYWlucyB0aGUgcmF0aW9u
YWxlIGZvciB0aGUgZmluYWxseSBwaWNrZWQgVENQIG9wdGlvbiBlbmNvZGluZy4gVGhhdCBtYXkg
YWxzbyBoZWxwIGlmIHRoZXJlIGFyZQ0KIGZ1cnRoZXIgcXVlc3Rpb25zIHdoZXRoZXIgdHdvIGNv
ZGVwb2ludHMgYXJlIHJlYWxseSByZXF1aXJlZCwgZS5nLiBieSB0aGUgSUVTRyAoaWYgdHdvIGNv
ZGVwb2ludHMgYXJlIHN0aWxsIHRoZSBkZXNpZ24gYWZ0ZXIgV0dMQykuIEF0IGxlYXN0IGZvciBw
YXN0IFRDUCBvcHRpb24gY29kZXBvaW50IGFsbG9jYXRpb25zIEkgcmVjYWxsIHNvbWUgZGlzY3Vz
c2lvbnMgbGF0ZSBpbiB0aGUgSUVURiBwcm9jZXNzLiBJbiB0aG9zZSBwYXN0IGNhc2VzLA0KIGdv
b2QgYXJndW1lbnRzIGluIGFuIGFwcGVuZGl4IGFuZCBydW5uaW5nIGNvZGUgaGFzIGhlbHBlZCBh
IGxvdC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs
YW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNw
OzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5NaWNoYWVsPG86cD48L286cD48
L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxl
PSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9w
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFy
ZWFzdC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5n
dWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9y
ZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgYmx1ZSAxLjVwdDtwYWRkaW5nOjBjbSAwY20gMGNt
IDQuMHB0Ij4NCjxkaXY+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlk
ICNFMUUxRTEgMS4wcHQ7cGFkZGluZzozLjBwdCAwY20gMGNtIDBjbSI+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48Yj48c3BhbiBsYW5nPSJFTi1VUyI+RnJvbTo8L3NwYW4+PC9iPjxzcGFuIGxhbmc9
IkVOLVVTIj4gQm9iIEJyaXNjb2UgJmx0O2lldGZAYm9iYnJpc2NvZS5uZXQmZ3Q7DQo8YnI+DQo8
Yj5TZW50OjwvYj4gVHVlc2RheSwgTm92ZW1iZXIgMTcsIDIwMjAgNjoxMCBBTTxicj4NCjxiPlRv
OjwvYj4gU2NoYXJmLCBNaWNoYWVsICZsdDtNaWNoYWVsLlNjaGFyZkBocy1lc3NsaW5nZW4uZGUm
Z3Q7OyBZb3NoaWZ1bWkgTmlzaGlkYSAmbHQ7bnNkLmlldGZAZ21haWwuY29tJmd0Ozxicj4NCjxi
PkNjOjwvYj4gTWljaGFlbCBUdWV4ZW4gJmx0O3R1ZXhlbkBmaC1tdWVuc3Rlci5kZSZndDs7IHRj
cG0tY2hhaXJzQGlldGYub3JnOyBNaXJqYSBLdWVobGV3aW5kICZsdDtpZXRmQGt1ZWhsZXdpbmQu
bmV0Jmd0OzsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZCAmbHQ7UmljaGFyZC5TY2hlZmZlbmVnZ2Vy
QG5ldGFwcC5jb20mZ3Q7OyB0Y3BtIElFVEYgbGlzdCAmbHQ7dGNwbUBpZXRmLm9yZyZndDs8YnI+
DQo8Yj5TdWJqZWN0OjwvYj4gUmU6IEFjY0VDTiBmaWVsZCBvcmRlcjxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF
Ti1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIg
c3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij5NaWNoYWVsLDxvOnA+PC9vOnA+PC9wPg0KPGRp
dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIDE2LzExLzIwMjAgMTc6MzYsIFNjaGFyZiwgTWlj
aGFlbCB3cm90ZTo8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1h
cmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij48c3BhbiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkJvYiw8L3NwYW4+PG86
cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0ibXNvLWZhcmVh
c3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3Vh
Z2U6RU4tVVMiPk9uZSBwcm9wb3NhbCB1c2luZyB0aGUgbGVuZ3RoIGZpZWxkIHdpdGggKjxiPm9u
ZSBvcHRpb24gY29kZXBvaW50IG9ubHk8L2I+KiBpcyBkZXRhaWxlZCBpbjoNCjxhIGhyZWY9Imh0
dHBzOi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1IwblJmaEhvY1g4
eXZUdnBDNEJOTW8vIj4NCmh0dHBzOi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNw
bS96by0xT1IwblJmaEhvY1g4eXZUdnBDNEJOTW8vPC9hPjwvc3Bhbj48bzpwPjwvbzpwPjwvcD4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVh
c3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3Vh
Z2U6RU4tVVMiPkl0IGlzIHRoZSB0aGlyZCBvcHRpb24gbWVudGlvbmVkIGluIHRoaXMgZS1tYWls
LiBPbmUgZXhhbXBsZSB3b3VsZCBiZSB0byB1c2Ugb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNS84LzEx
IGZvciBvbmUgZW5jb2RpbmcgdHlwZSBhbmQgb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNi85LzEyIGZv
ciB0aGUgb3RoZXIgZW5jb2RpbmcNCiB0eXBlIChpLmUuLCBvcmRlciBvZiBmaWVsZHMpLiBPciBv
bmUgY291bGQgdXNlIHNvbWUgb3RoZXIgY29tYmluYXRpb24gb2YgbGVuZ3RoIHZhbHVlcyDigJMg
dGhlIG9ubHkgcmVxdWlyZW1lbnQgaXMgdGhhdCBhIGNlcnRhaW4gdmFsdWUgZm9yIHRoZSBvcHRp
b24gbGVuZ3RoIGlzIG9ubHkgdXNlZCBieSBvbmUgb2YgdGhlIG9wdGlvbiBmb3JtYXRzLiBJbiB0
aGF0IGFwcHJvYWNoLCB0aGUgdmFsdWUgb2YgdGhlIGxlbmd0aCBmaWVsZCB3b3VsZCB0aHVzDQog
ZGlyZWN0bHkgZGVzY3JpYmUgdGhlIGVuY29kaW5nIG9mIHRoZSBvcHRpb24uIFVubGVzcyBJIG1p
c3Mgc29tZXRoaW5nLCB0aGlzIHdvdWxkIHdvcmsgYW5kIGl0IHdvdWxkIGp1c3QgcmVxdWlyZSBv
bmUgb3B0aW9uIGNvZGVwb2ludC48L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVO
LVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48
c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5UaHVz
LCBhbHRlcm5hdGl2ZXMgdG8gdHdvIG9wdGlvbiBjb2RlcG9pbnRzIGV4aXN0IGFuZCBJIGhhdmUg
ZXhwbGFpbmVkIHRoZW0gb24gdGhlIGxpc3QgaW4gTWFyY2ggMjAyMC48L3NwYW4+PG86cD48L286
cD48L3A+DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQpPSywgc29y
cnksIHllcywgSSByZW1lbWJlciB0aGlzIG5vdy48YnI+DQo8YnI+DQpBcyBJIHdpbGwgZXhwbGFp
biBpbiB0aGUgQWNjRUNOIHN0YXR1cyB1cGRhdGUgdGFsayB0b2RheSBpbiB2aXJ0dWFsIEJhbmdr
b2ssIHRoZSBkcmFmdCBoYXMgbWFkZSBwcm92aXNpb24gZm9yIGRpZmZlcmVudCBsZW5ndGggdmFs
dWVzIHRoYW4gNS84LzExLiBJdCBzYXlzIGV4aXN0aW5nIGltcGxlbWVudGF0aW9ucyBNVVNUIGFj
Y2VwdCBsZW5ndGggdmFsdWVzIG90aGVyIHRoYW4gdGhvc2UgY3VycmVudGx5IGRlZmluZWQuIEJ1
dCB0aGVuIHJlYWQgaW4NCiBhcyBtYW55IHdob2xlIDMtYnl0ZSBmaWVsZHMgYXMgdGhleSBjYW4u
PGJyPg0KPGJyPg0KVGhpcyBjYW4gYmUgdXNlZCB0byBhZGQgYSBmbGFncyBieXRlIG9uIHRoZSBl
bmQgaW4gZnV0dXJlLCBmb3IgZXh0ZW5zaWJpbGl0eS4gT3IgYW55IG90aGVyIGZvcm0gb2YgZXh0
ZW5zaWJpbGl0eSB0aGUgV0cgbWlnaHQgZGVjaWRlIGluIHRoZSBmdXR1cmUuPGJyPg0KPGJyPg0K
SSBrbm93IGEgZmxhZ3MgYnl0ZSBhdCB0aGUgZW5kIHNlZW1zIG9kZCBjb21wYXJlZCB0byBhdCB0
aGUgYmVnaW5uaW5nLiBCdXQgKGlmIGRlY2lkZWQgaXQncyBuZWVkZWQgaW4gZnV0dXJlKSBpdCdz
IHJlYXNvbmFibHkgZWFzeSB0byBpbXBsZW1lbnQgYnkgcmVhZGluZyB0aGUgd2hvbGUgb3B0aW9u
LCB0aGVuIHByb2Nlc3NpbmcgdGhlIGxhc3QgYnl0ZSwgYmVmb3JlIHJlYWRpbmcgdGhlIHJlc3Qg
b2YgdGhlIG9wdGlvbi48YnI+DQo8YnI+DQpJIGJlbGlldmUgeW91IHdpbGwgYWdyZWUgdGhhdCB0
aGlzIGlzIGEgYmV0dGVyIHdheSB0byB1dGlsaXplIGRpZmZlcmVudCBsZW5ndGhzLjxicj4NCjxi
cj4NCkFuZCB0aGFuayB5b3UgZm9yIHJlcGVhdGVkbHkgZW1waGFzaXppbmcgdGhhdCB5b3UncmUg
aGFwcHkgd2l0aCB0aGUgMi1raW5kIHNjaGVtZSwgb3Igb3RoZXIgYWx0ZXJuYXRpdmVzLjxicj4N
Cjxicj4NCjxicj4NCkJvYjxicj4NCjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPGJsb2Nr
cXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxh
bmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVO
LVVTIj5Bbnl3YXksIEkgZG9u4oCZdCByZWFsbHkgY2FyZSBob3cgdGhlIG9wdGlvbnMgYXJlIGVu
Y29kZWQgYXMgbG9uZyBhcyB0aGUgcmVjZWl2ZXIgZG9lc27igJl0IG5lZWQgcGVyLWNvbm5lY3Rp
b24gc3RhdGUgZm9yIGRlY29kaW5nIGEgVENQIG9wdGlvbi4gU28sIHBlcnNvbmFsbHksIEkgd291
bGQgYmUgZmluZSB3aXRoIHVzaW5nDQogZS5nLiB0aGUgbGVuZ3RoIGZpZWxkIGFzIGRlc2NyaWJl
ZCBpbiBteSBvbGQgZS1tYWlsLiBPciBhbiBhZGRpdGlvbmFsIGZsYWcgYnl0ZS4gQW5kIG9uZSBj
b3VsZCBjb21lIHVwIHdpdGggZnVydGhlciBlbmNvZGluZ3MsIGUuZy4sIGJ5IHVzaW5nIG9uZSBv
ciBhIGZldyBiaXRzIGFzIGEgc2hvcnQg4oCcdHlwZeKAnSBmaWVsZCBmb3IgZWFjaCBjb3VudGVy
LiBUaGlzIGlzIGFsbCBhYm91dCBwcm90b2NvbCBlbmdpbmVlcmluZy4gQW5kIGFsbCB0aGVzZQ0K
IHZhcmlhbnRzIGhhdmUgdGhlaXIgcHJvcyBhbmQgY29ucy48L3NwYW4+PG86cD48L286cD48L3A+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJl
YXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj5JIGFtIGFsc28gZmluZSB3aXRoIHVzaW5nIHR3byBvcHRpb24gY29kZXBvaW50
cyBhcyBzcGVjaWZpZWQgaW4gLTEzOyB0aGlzIGlzIHByb2JhYmx5IHRoZSBhcHByb2FjaCB0aGF0
IGNvbnN1bWVzIHRoZSBsZWFzdCBudW1iZXIgb2YgYml0cy48L3NwYW4+PG86cD48L286cD48L3A+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJl
YXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj5NaWNoYWVsICh3L28gYW55IGhhdCk8L3NwYW4+PG86cD48L286cD48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0
LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdl
OkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4m
bmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3Jk
ZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQiPg0KPGRp
dj4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29saWQgI0UxRTFFMSAxLjBw
dDtwYWRkaW5nOjMuMHB0IDBjbSAwY20gMGNtIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPkZy
b206PC9iPiBCb2IgQnJpc2NvZSA8YSBocmVmPSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+
DQombHQ7aWV0ZkBib2JicmlzY29lLm5ldCZndDs8L2E+IDxicj4NCjxiPlNlbnQ6PC9iPiBNb25k
YXksIE5vdmVtYmVyIDE2LCAyMDIwIDU6NTIgUE08YnI+DQo8Yj5Ubzo8L2I+IFlvc2hpZnVtaSBO
aXNoaWRhIDxhIGhyZWY9Im1haWx0bzpuc2QuaWV0ZkBnbWFpbC5jb20iPiZsdDtuc2QuaWV0ZkBn
bWFpbC5jb20mZ3Q7PC9hPjxicj4NCjxiPkNjOjwvYj4gU2NoYXJmLCBNaWNoYWVsIDxhIGhyZWY9
Im1haWx0bzpNaWNoYWVsLlNjaGFyZkBocy1lc3NsaW5nZW4uZGUiPiZsdDtNaWNoYWVsLlNjaGFy
ZkBocy1lc3NsaW5nZW4uZGUmZ3Q7PC9hPjsgTWljaGFlbCBUdWV4ZW4NCjxhIGhyZWY9Im1haWx0
bzp0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUiPiZsdDt0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUmZ3Q7PC9h
PjsgPGEgaHJlZj0ibWFpbHRvOnRjcG0tY2hhaXJzQGlldGYub3JnIj4NCnRjcG0tY2hhaXJzQGll
dGYub3JnPC9hPjsgTWlyamEgS3VlaGxld2luZCA8YSBocmVmPSJtYWlsdG86aWV0ZkBrdWVobGV3
aW5kLm5ldCI+Jmx0O2lldGZAa3VlaGxld2luZC5uZXQmZ3Q7PC9hPjsgU2NoZWZmZW5lZ2dlciwg
UmljaGFyZA0KPGEgaHJlZj0ibWFpbHRvOlJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29t
Ij4mbHQ7UmljaGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20mZ3Q7PC9hPjsgdGNwbSBJRVRG
IGxpc3QNCjxhIGhyZWY9Im1haWx0bzp0Y3BtQGlldGYub3JnIj4mbHQ7dGNwbUBpZXRmLm9yZyZn
dDs8L2E+PGJyPg0KPGI+U3ViamVjdDo8L2I+IEFjY0VDTiBmaWVsZCBvcmRlcjxvOnA+PC9vOnA+
PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzxvOnA+PC9v
OnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0
Ij5Zb3NoaSwgKGFkZGluZyB0aGUgdGNwbSBsaXN0IGluIGNjKTxvOnA+PC9vOnA+PC9wPg0KPGRp
dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIDA1LzExLzIwMjAgMDY6NTgsIFlvc2hpZnVtaSBO
aXNoaWRhIHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0i
bWFyZ2luLXRvcDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8ZGl2Pg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+SGkgQm9iLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+
DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+T24gV2VkLCBOb3YgNCwgMjAyMCBhdCAzOjI5
IFBNIEJvYiBCcmlzY29lICZsdDs8YSBocmVmPSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+
aWV0ZkBib2JicmlzY29lLm5ldDwvYT4mZ3Q7IHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+
DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0ND
QyAxLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdp
bi10b3A6NS4wcHQ7bWFyZ2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPllvc2hp
LDxvOnA+PC9vOnA+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIDA0LzExLzIw
MjAgMDY6NTEsIFlvc2hpZnVtaSBOaXNoaWRhIHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+
DQo8YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luLXRvcDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0
Ij4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SGksIGZvbGtzLDxvOnA+PC9v
OnA+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPC9ibG9ja3F1
b3RlPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
Pjxicj4NCjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1h
cmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0KPGRpdj4NCjxibG9j
a3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAjQ0NDQ0NDIDEuMHB0
O3BhZGRpbmc6MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXRvcDo1
LjBwdDttYXJnaW4tcmlnaHQ6MGNtO21hcmdpbi1ib3R0b206NS4wcHQiPg0KPGRpdj4NCjxibG9j
a3F1b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQiPg0KPGRp
dj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JbiBteSB1bmRlcnN0YW5kaW5nLCBJJ20g
bm90IHN1cmUgaWYgd2Ugc2V0dGxlZCBkb3duIG9uIHVzaW5nIHR3byBvcHRpb24ga2luZHMgb3Ig
ZW5jb2Rpbmcgc2NoZW1lcyBmb3IgMjRiaXRzIGZpZWxkcyBpbiBhY2MgZWNuIGRyYWZ0LjxvOnA+
PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+U28sIEkgdGhp
bmsgdGhlcmUncmUgc3RpbGwgc29tZXRoaW5nIHRvIGJlIGNsYXJpZmllZCBhbmQgaG9wZSB0aGlu
Z3Mgd2lsbCBiZSBzZXR0bGVkIGF0IHRoZSBtZWV0aW5nLjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+
DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCltCQl0g
SSBrbm93IGEgV0cgY2FuIGNoYW5nZSBpdCdzIG1pbmQgYXQgYW55IHRpbWUuIEJ1dCBJJ2QgcmF0
aGVyIHdlIGp1c3QgY2xhcmlmaWVkIHdoYXQgYSBwcmV2aW91cyBkZWNpc2lvbiB3YXMsIHRvIGF2
b2lkIHRoZSBuZWVkIHRvIGtlZXAgcmUtb3BlbmluZyBkaXNjdXNzaW9uIG9uIGEgcXVlc3Rpb24g
dGhhdCBoYXZlIGJlZW4gZGVjaWRlZCB0aGVuIGNoYW5nZWQgdGhyZWUgZGlmZmVyZW50IHdheXMg
YWxyZWFkeS48YnI+DQo8YnI+DQpNeSBtZW1vcnkgaXMgbm90IHNvIGdvb2QgdGhlc2UgZGF5cy4g
SSB0cnVzdGVkIHRoYXQgTWljaGFlbCBTIHJlbWVtYmVyZWQgdGhlIGRlY2lzaW9uIGNvcnJlY3Rs
eSwgYW5kIEkgc2VlbSB0byByZW1lbWJlciB0aGF0IGRlY2lzaW9uIGJlaW5nIG1hZGUuPGJyPg0K
SSd2ZSBqdXN0IGNoZWNrZWQgdGhlIG1pbnV0ZXMgb2YgdGhlIGxhc3QgaW50ZXJpbTo8YnI+DQom
bmJzcDsmbmJzcDsmbmJzcDsgPGEgaHJlZj0iaHR0cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9k
b2MvbWludXRlcy1pbnRlcmltLTIwMjAtdGNwbS0wMS0yMDIwMDQyOTE2MDAvIiB0YXJnZXQ9Il9i
bGFuayI+DQpodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9taW51dGVzLWludGVyaW0t
MjAyMC10Y3BtLTAxLTIwMjAwNDI5MTYwMC88L2E+PGJyPg0KYW5kIHRoZXkgbWVudGlvbiBNaWNo
YWVsJ3MgcHJvcG9zYWwgdG8gdXNlIHR3byBraW5kcywgYnV0IGRvbid0IHJlY29yZCBhbnkgZGVj
aXNpb24uPGJyPg0KVGhlIGphYmJlciBsb2cgZ2l2ZXMgbm8gY2x1ZXMgYWJvdXQgYW55IGRlY2lz
aW9uLjxicj4NCjxicj4NCkkgY2FuJ3QgZmluZCBhbiBhdWRpbyBvciB2aWRlbyByZWNvcmRpbmcu
IENhbiB5b3UgcG9pbnQgbWUgYXQgb25lPzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2Jsb2Nr
cXVvdGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PG86cD48L286cD48L3A+
DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JIHRob3VnaHQgdGhhdCBpdCdz
IGJlY2F1c2UgdGhlcmUgd2FzIG5vIGNsZWFyIGRlY2lzaW9uIGF0IHRoZSBtZWV0aW5nLiZuYnNw
OzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+QnV0
LCB5b3UgY2FuIGNoZWNrIDxhIGhyZWY9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNoP3Y9
S3NCMF9OYjgta0EiPg0KaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1Lc0IwX05iOC1r
QTwvYT48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PlBsZWFzZSBsZXQgdXMga25vdyBpZiB5b3UgaGF2ZSBhbnkgcXVlc3Rpb25zIG9yIG9waW5pb25z
IHdpdGggcmVnYXJkIHRvIHRoaXMuPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9k
aXY+DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQpbQkJdIEkgY2hl
Y2tlZCB0aGUgWW91dHViZSBsaW5rIHlvdSBzZW50IGJlbG93Ljxicj4NCjxicj4NCkZpcnN0IEkg
dGhpbmsgd2UncmUgYWdyZWVkIG5vLW9uZSB3YXMgZmlnaHRpbmcgZm9yIHVzIHRvIGtlZXAgdGhl
IHByZXZpb3VzIHdheSB3ZSBkaWQgdGhpcyAodXNpbmcgdGhlIGluaXRpYWwgdmFsdWUgb2YgdGhl
IGZpZWxkIHRvIHNldCB0aGUgb3JkZXIgZm9yIHRoZSBjb25uZWN0aW9uKS4NCjxicj4NCkluIG15
IHByZXNlbnRhdGlvbiBJIHNhaWQgdGhlcmUgd2FzIHN0cm9uZyByZXNpc3RhbmNlIGZyb20gTWlj
aGFlbCB0byBkbyBpdCBhIGRpZmZlcmVudCB3YXkuDQo8YnI+DQooYWxzbywgb2ZmbGlzdCwgdGhl
IGNvLWF1dGhvcnMgaW5jbHVkaW5nIG1lIGFsc28gZGlkbid0IGxpa2UgdGhpcyBzbyBtdWNoLiBB
bmQgSWxwbyBzYWlkIGl0IG1hZGUgdGhlIGltcGxlbWVudGF0aW9uIGNvbXBsZXguKTxicj4NCjxi
cj4NClRoZW4gY2FtZSB0aGUgcXVlc3Rpb24gb2Ygd2hhdCB3ZSBkbyBpbnN0ZWFkLiBUaGVyZSB3
ZXJlIHRocmVlIGFsdGVybmF0aXZlIHByb3Bvc2Fsczo8YnI+DQphKSB1c2UgMiBvcHRpb24ga2lu
ZHM8YnI+DQpiKSBhZGQgYSBmbGFncyBieXRlPGJyPg0KYykgc29tZWhvdyB1c2UgdGhlIGxlbmd0
aCBmaWVsZCBtYXliZTxicj4NCjxicj4NCk1pY2hhZWwgcmFpc2VkIChjKSBpbiB0aGUgbWVldGlu
ZyBhcyBhIHBvc3NpYmlsaXR5LCBidXQgbm8tb25lIGNvdWxkIHRoaW5rIGhvdyB0byBkaXN0aW5n
dWlzaCB0d28gb3B0aW9ucyBvZiB0aGUgc2FtZSBsZW5ndGggYnV0IGEgZGlmZmVyZW50IGZpZWxk
IG9yZGVyIHVzaW5nIHRoZSBsZW5ndGggZmllbGQuIE1pY2hhZWwgc2FpZCBoZSdkIHBvc3QgYW55
IGlkZWFzIHRvIHRoZSBsaXN0IGlmIGhlIGNvdWxkIHRoaW5rIG9mIGFueSwgYnV0IHRoYXQNCiBk
aWRuJ3QgaGFwcGVuLjxicj4NCjxicj4NClNvIHdlJ3JlIGxlZnQgY2hvb3NpbmcgYmV0d2VlbiAo
YSkgYW5kIChiKS48YnI+DQpJIHNhaWQgaW4gdGhlIG1lZXRpbmcgKGFuZCBvbiB0aGUgbGlzdCB3
aGVuIGRpc2N1c3Npbmcgd2l0aCBJbHBvKSB0aGF0IEknZCBiZSBoYXBweSB0byBnbyB3aXRoIChi
KSwgYnV0IG9ubHkgaWYgdGhlcmUgd2FzIGFub3RoZXIgdXNlIGZvciBhIGZsYWcuIEJlY2F1c2Ug
aXQgd291bGQgY29uc3VtZSAxQiBtb3JlIG9wdGlvbnMgc3BhY2UgaW4gbWFueSBwYWNrZXRzLCB3
aGljaCBpcyBhIHNjYXJjZSByZXNvdXJjZS48YnI+DQo8YnI+DQpJbHBvIGhhZCBhIHByb3Bvc2Vk
IHVzZSBmb3IgYW5vdGhlciBmbGFnICh0byBoZWxwIHN5bmNoIGNvdW50ZXJzIGFmdGVyIGEgbG9z
cyksIGJ1dCBJIHRoaW5rIHRoZSBkaXNjdXNzaW9uIGFib3V0IGl0IGVuZGVkIHRoYXQgaXQgd291
bGRuJ3QgYmUgaGVscGZ1bCwgJ2NvcyB0aGUgd2F5IGl0IHdvcmtlZCBkZXBlbmRlZCBvbiBpdHNl
bGYgKGNpcmN1bGFyIGxvZ2ljKS48YnI+DQo8YnI+DQpJbiBjb25jbHVzaW9uLCBJIGRvbid0IHRo
aW5rIHRoZXJlIHdhcyBhbiBleHBsaWNpdCBkZWNpc2lvbiB0byBnbyB3aXRoIDIgb3B0aW9uIGtp
bmRzLCBidXQgaXQgZW5kZWQgdXAgYXMgdGhlICdsYXN0IHBlcnNvbiBzdGFuZGluZycuPGJyPg0K
SSBsaWtlIGl0LiBJdCdzIHNpbXBsZS4gQW5kIGFwcGFyZW50bHkgb3B0aW9uIGtpbmRzIGFyZSBu
b3Qgc3VjaCBhIHNjYXJjZSByZXNvdXJjZS48YnI+DQo8YnI+DQpQZXJoYXBzIHdlIGNhbiByYXRp
ZnkgdGhpcyBpbiB0aGUgV0cgdG9tb3Jyb3cuPGJyPg0KPGJyPg0KPGJyPg0KQm9iPGJyPg0KPGJy
Pg0KPGJyPg0KPGJyPg0KPG86cD48L286cD48L3A+DQo8YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2lu
LXRvcDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8ZGl2Pg0KPGRpdj4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+VGhhbmtzLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+LS08bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPllvc2hpPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2
Pg0KPC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQo8YnI+
DQo8YnI+DQo8bzpwPjwvbzpwPjwvcD4NCjxwcmU+LS0gPG86cD48L286cD48L3ByZT4NCjxwcmU+
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fXzxvOnA+PC9vOnA+PC9wcmU+DQo8cHJlPkJvYiBCcmlzY29lJm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7ICZuYnNwOzxhIGhy
ZWY9Imh0dHA6Ly9ib2JicmlzY29lLm5ldC8iPmh0dHA6Ly9ib2JicmlzY29lLm5ldC88L2E+PG86
cD48L286cD48L3ByZT4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPHAgY2xhc3M9Ik1zb05vcm1h
bCI+PGJyPg0KPGJyPg0KPG86cD48L286cD48L3A+DQo8cHJlPi0tIDxvOnA+PC9vOnA+PC9wcmU+
DQo8cHJlPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX188bzpwPjwvbzpwPjwvcHJlPg0KPHByZT5Cb2IgQnJpc2NvZSZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDs8YSBocmVmPSJodHRwOi8vYm9iYnJpc2NvZS5uZXQvIj5odHRwOi8vYm9iYnJpc2NvZS5uZXQv
PC9hPjxvOnA+PC9vOnA+PC9wcmU+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5Pg0KPC9odG1sPg0K

--_000_30781ea61a794131beafe9997ed9221ahsesslingende_--



From nobody Tue Nov 17 00:38:24 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B79EE3A0983 for ; Tue, 17 Nov 2020 00:38:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id znLcTPUbGc8F for ; Tue, 17 Nov 2020 00:38:21 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8F01A3A0982 for ; Tue, 17 Nov 2020 00:38:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605602247; bh=SaAA0oBysfZLgN15ZvtNSJN4+1G+zr1l+s4pquBMpk8=; h=X-UI-Sender-Class:Subject:To:References:From:Date:In-Reply-To; b=LyNk/4eDDbImqK1ObDQfa/10gmdcGzKQD24bvtIizLCGUA315JT/UbjS13PzQLnQG EvID3WaiBgMk2X8rdLjNxbFi5TkRiQGa9i1nueT9Ds+SCPxTgEhJdpAJGE+8lnhUEx MnmbnRCUN4yv7aP7S8f81dpqmaZRbKcYC0bYWCH0=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [192.168.233.106] ([185.236.167.136]) by mail.gmx.com (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1M9Fnj-1kaptI1IWN-006Luz; Tue, 17 Nov 2020 09:37:27 +0100
To: Bob Briscoe , Richard Scheffenegger , Mirja Kuehlewind , =?UTF-8?Q?Ilpo_J=c3=a4rvinen?= , tcpm IETF list 
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net>
From: "Scheffenegger, Richard" 
Message-ID: 
Date: Tue, 17 Nov 2020 09:37:26 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:1DcE1dahhspLSIaAr1/7JHnGZ0c9tyCKUPg9I0HZFlWvGtybyBf An3LT56CI+bq3V69RakCWkWydjLWHEihpvefsznOAbdtwQukeU6AtwC68ZVll3ddyLUQuTf o5FKlT5g3tfIBGRQuCtKP7lYeS7gsszYkxXDartn3JgFUGiEXxy8RucBEfxzD3PI+eW3iQq 8MmvZUjvjw65R5IT137/g==
X-UI-Out-Filterresults: notjunk:1;V03:K0:+rQcGNuh5CU=:lAE6Q1gY7hC7prDn4FNQWh RgcQ3abtSRLvVmSIyslH0r2p1yrJBW9Lhom0swroY3fFldqpxfVmeyukAF09Q4uRy92IkFIlt mnXQJYaJdCjvTIf24UFSbw4vooRi84LdWPqeNsEnj/prxSVpBtxFs7od8mJTCDlUe3HBrgLUc cOdyxm8KasBk/F2czIBidiApVXALHV2+mXPoqxcxRzLgQ5nsEl0CbmBPv1LRCCuW3Nz8QcaTj 60XiaTyVF9Zp5OFi20xxQ55FYXBBDL35PhujNr1TaBOY7lvKnIfvd2hWO/m9ypkd+35Rn/wY5 /FURieHvpDVu0aHr3nu+9v+z+ZJg79mjs5Mo2j6AgfTsCmONVTdo+Va4j39uT9euCyhVAUsbG Wh+SoKHsyR6AH03qTnK47Bh0p6s9NXJxoUSt5d39y+BdkJfKQUz/WM2Bo7wwkZUol+PV3lkOp 9cJZQMR0HGK1pbmMYLOgCAEVDTF2idryZSziP+Cbo4Aj5WUGQooPNLo3PyqvSCIZjAksbFHT8 /fuhpCz/GoZ/anZSULRJcfphP6nz8oUYDf4e2v08kgz3xexa0MflcEC1UU3wl3EFKNrx3M1NQ r6Z44iSwWH094DfxHvjpdIFXmDpk8MoYL1sX/X7umtymMwu7RxGcMeeCadMu9I1wATXf5BPIK v6KEzEJkP2DrEDb/Jf6T2IucGow0f6jrmOlCqQqjfQ8CMwVHU5IYOluhSIkoxmSblnqbhAQ1M f8sIND3/v5VRJh+1atVNeCSPBAiwrHjNLGnK7JbzDrfiIfnJwvEfImXSEA9QmkmDV6p52jV1W x/9JTOJ4u1D+Ia4fR1hXD11FVFZtsKEL1HZUNd+xtrDSrxgw3j/MVzeSHLqfna9MYMFg9XcYm U57vyXzDfenah4enjHrA==
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 08:38:23 -0000

Without having read any more recent responses,

If you want to use AccECN to eventually control the ACK flow, I think
the 2nd bullet point is useful.

In order to not continously ACK an ACK (if they all get CE marked), the
sentence may need to be revised to


    o  MUST immediately send an ACK once 'n' CE marks have arrived since
       the previous ACK, where 'n' MUST NOT be lower than 2 and MUST be
       no greater than 6.

That should ensure that any ACK-for-ACK will die down rather rapidely,
if the stream of ACKs on the return path suddenly all get CE marked
(e.g. path change across a device, which erroneously overrides TOS and
happens to paint both IP ECN bits too).

Richard



Am 10.11.2020 um 01:31 schrieb Bob Briscoe:
> Mirja, Richard, Ilpo, tcpm list,
>
> I've just been reading through the accurate-ecn draft to double-check. I
> think there's a problem with the following text...
>
> 3.2.2.5.1.  Data Receiver Safety Procedures
>
>     An AccECN Data Receiver:
>
>     o  SHOULD immediately send an ACK whenever a data packet marked CE
>        arrives after the previous [data] packet was not CE.
>
>     o  MUST immediately send an ACK once 'n' CE marks have arrived since
>        the previous ACK, where 'n' SHOULD be 2 and MUST be no greater
>        than 6.
>
> ...
>     For the avoidance of doubt, the change-triggered ACK mechanism is
>     deliberately worded to solely apply to data packets, and to ignore
>     the arrival of a control packet with no payload, because it is
>     important that TCP does not acknowledge pure ACKs.
>
>
> In the first bullet, I think it doesn't matter whether the previous
> packet marked CE was a data packet or a pure ACK (i.e we should remove
> the second occurrence of 'data' that I have put in [square brackets].
>
> The second bullet doesn't consider the possibility that the 'n'th CE
> mark might arrive on a pure ACK. Then, the wording as it stands says the
> Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs a
> pure ACK, but I'm not actually sure it does any harm to do so in this
> case (it cannot cause an infinite loop of ACKs). However, given it would
> be unorthodox, we maybe ought to rule it out by rewording anyway?
>
> Thoughts anyone?
>
>
> Bob
>
> --
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>


From nobody Tue Nov 17 00:49:28 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id EDAAD3A09D4 for ; Tue, 17 Nov 2020 00:49:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1OFD8j0GAL0z for ; Tue, 17 Nov 2020 00:49:24 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 96F493A09CC for ; Tue, 17 Nov 2020 00:49:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605602917; bh=xwu0iXqJHP+Xd5uskTjFYjNxiH1/nzsh11gMcf1tKvQ=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=YGRhimzuq48y+1X2KCcNBJGKkTV+rKH17a9QpUjc3PHwfOfCsVXsKq2wS6qHAWU5h 4+eudtPxlOzfUV5vzPWj8XJ6VbbSSzxbLnaii+O7Nf31fbBbvRomTScqA2HOb/BjuH DaSoIy+4Q6/YTQKRUSXPRy707dTyDyvc1Tjot6Q8=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [192.168.233.106] ([185.236.167.136]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MsYqp-1kOdw62SMN-00u3wK; Tue, 17 Nov 2020 09:48:37 +0100
To: Bob Briscoe , Mirja Kuehlewind 
Cc: tcpm IETF list , Richard Scheffenegger 
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net> <060c8bd8-d64b-3e46-7874-742e35e6d114@bobbriscoe.net>
From: "Scheffenegger, Richard" 
Message-ID: <221e58f3-ada0-c880-db72-d98af84fedb8@gmx.at>
Date: Tue, 17 Nov 2020 09:48:36 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <060c8bd8-d64b-3e46-7874-742e35e6d114@bobbriscoe.net>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:rcQYiq6JKbF6q8nJpW71oaMXAqvP7ZY9EmR/iBMcVkfTQRef4Al yRlY7Skx9iaudKfFqADDoHgJj9iXZkXADiEkg7QzEOuRuSLWPl7LHx07cgydJl7zZHoE+vU rKanT6e+KZDxr/vQPZnWuo6ICmQOYR2Sv0OjbfYwPk39PpoTGfgp+T8xa4CpzQ+P7MTijOd 4jkY8tDfoPhTsAUQ2r6GQ==
X-UI-Out-Filterresults: notjunk:1;V03:K0:93U8DJeBd3s=:SBtf/0kY9vDB4sbjRG0LmS pfDF6m7FxyLEpNCTdahAQqx7IQsrMSlVANYFGyNJ1aoaCfUzDi++uqJ0ZGOfTyRaL87j0t7EG shRB7Yht3blWFhhMIEfO1GQJxekb1ZxG4PPu6ioV7mS3qVqFHHGTRbluMZRPRXONS/hJi4p1I L24/JHHI1qUz1WqewAL76NT5G3i95u/qi+PzdRjnB4W03KX1JJAPRE8F42C42q3rnt9KQZu6N OmCZENZ8umJmSykbPD7Sno3YS6T5mTbL3DRQN/rJXRDo9zWdzqVaxxqxAOn+RHeXPM1E4SzPt MB4r1EPFiDBgDDfNmss9DbYIvSdz88IH1zPKGOyolWtjQxOkPPwtJW7gJsL4oTdApvoALb0jg DB+sTXzllrOhNUbw1m6dzXd32hQkNd/yjUbz+39i2N3ZMH9PQuA4zgxWs1jaSKcKWJ5hRciHd pOCUVKpWqvqWGuXhIWt991OL5UYdcDLbJBSRSWtNYHSfhnnXWWJSqbJtNzLXjith9P0fvnhJu zBApYFurNRVAXrxaUJhJZSuuGosHs502+mPJwZvskk+jM7rKiSN4L9ovcuvKaxzEMCV7NfkfT YN7Mo+ZQbSUzNELuJUfwC14UEQOwMZBPcB24gdiBPKoBpkuA5sfUMSdBfc4N/yyrOSW+GRBVZ u/RK933ty38Du2w8rm/oSEk1LZPDECfWhajf7sYOP/sBTy4S/ctvXtIMw3gIggAiu9Cf3fHAU iGwXhSZxNxQ0HnQDVwDQgHl4yz7UAOtN5x83SopoYpRjxK7ZIpLvoY0rgO4lFu82mF0/7macG QaKESL6dPrpatueoC8rWKHTVLwQN1F8zrnFCDEzFHXN0HJFhtgvDN/puRrrAKU9rmbsyHF/w1 eaoMIWi+/wYp0aAwTX1A==
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 08:49:26 -0000

See my suggestion; The critical requirement for the 2nd bullet point is,
to have n > 1 for pure, CE-marked ACKs. The exponential decay, after how
many ACKs triggered by CE-marked ACKs these ACK-for-ACK stop, strongly
depends on "n", so the recommendation should be to use a higher value of
n in the case where you only observe a stream of pure ACKs (with or
without marks; only those marked would account against "n" though).

In the case of data packets mixed in, you don't want to delay
excessively, thus "n" may be choosen lower...

Richard



Am 10.11.2020 um 17:38 schrieb Bob Briscoe:
>
> Moving on to the second bullet...
>>> The second bullet doesn't consider the possibility that the 'n'th CE
>>> mark might arrive on a pure ACK. Then, the wording as it stands says
>>> the Data Receiver MUST immediately ACK a pure ACK. I know TCP never
>>> ACKs a pure ACK, but I'm not actually sure it does any harm to do so
>>> in this case (it cannot cause an infinite loop of ACKs). However,
>>> given it would be unorthodox, we maybe ought to rule it out by
>>> rewording anyway?
>> [MK] I think we also can leave this as it is because if that =E2=80=99n=
=E2=80=99
>> packet is a pure ACK that still means that you have unacked data
>> packets with CE marks and you should trigger an ACK for those packet
>> now (rather than waiting for the delayed ACK timer to expire). However
>> this case is less important. Also we should probably make sure that
>> this doesn=E2=80=99t apply if there are only pure ACKs with CE marks, m=
aybe by
>> add =E2=80=9Cif unacknowledged data are outstanding=E2=80=9D or somethi=
ng.
>
> [BB] By the end of your last para you start to realize that, if the n'th
> CE mark arrives on pure ACK, it doesn't say anything about whether the
> previous (n-1) CE marks were on data or pure ACKs.
>
> Consider another common scenario; a server delivering chunks of video
> over a high BDP path (e.g. 500 packets per RTT), and going idle between
> times (perhaps in response to an "exceeded high watermark" data packet
> from the server). Now consider the possibility that the reverse path
> bottleneck builds a queue during each chunk, and starts CE marking
> towards the end of each chunk. Then, after the server stops sending a
> chunk, it continues to see a couple of hundred CE-marked pure ACKs
> arriving.
>
> However, by your proposed rule, it doesn't send any feedback, 'cos it
> has no data to acknowledge. So the client cannot regulate its ACK
> stream. That's just tough, 'cos the client will stop sending more pure
> ACKs after it has received the last data, which is before any later
> feedback from the server can reach it.
>
> However, if the client later sends a data packet that the server can
> acknowledge (e.g. "a low watermark" message), the server's ACE counter
> will have wrapped dozens of times, and the client won't be able to work
> out how many. That doesn't matter if a few round trips have elapsed -
> the congestion info will have become stale. But it does matter if it's
> fairly soon.
>
> Instead, if we keep the text of the second bullet... if the server's 'n'
> was 2, it would send a pure ACK for every other pure ACK it received.
> But, what if these pure ACKs from the server were also CE-marked? Then
> the client would feed back pure ACKs using its 'n'. This regression
> would eventually die out, but I don't like it.
>
> This conversation about the second bullet presumes AccECN is being used
> for ACK congestion control. I'm not comfortable with writing anything
> specific about ACK congestion control into a draft on the standards
> track. But I don't want to preclude AccECN being used for ACK CC.
>
>
> I believe the second bullet needs something changed to limit the ACKing
> of pure ACKs. However, before we put effort into wordsmithing, I'd like
> to hear whether you agree with the general idea of not defining what to
> do for ACK CC, but not precluding it.
>
> Cheers
>
>
>
>
> Bob
>
>>
>> Mirja
>>
>>
>>
>>> Thoughts anyone?
>>>
>>>
>>> Bob
>>>
>>> --
>>> ________________________________________________________________
>>> Bob Briscoe
>>> http://bobbriscoe.net/
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>


From nobody Tue Nov 17 00:54:25 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2708A3A09F4 for ; Tue, 17 Nov 2020 00:54:24 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level: 
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id siLMRQ0n_y7g for ; Tue, 17 Nov 2020 00:54:22 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 516B83A09BB for ; Tue, 17 Nov 2020 00:54:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605603218; bh=iLyy6lK/D6134N1TygD2KyeGpubJbWQR3kvlzqmlviE=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=OPRoQc3qsmjAOvhFXDnkyVr9h9Y5Y/ZjVbJmu2HnshTF4xGTnAWT9zGMBNtw4QSjN za2L9d+kMU3TbQaeb5WjGVX+zFiaAy+4FZUjwE5Rm9/7WEFseEjonvJcUJBZj9ovyl mvfS9Ne1fS7GZxYiAEU9lhJJZf74oaPBZkv6pvss=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [192.168.233.106] ([185.236.167.136]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1N4Qwg-1kF3bg3JB8-011TW6; Tue, 17 Nov 2020 09:53:38 +0100
To: Yoshifumi Nishida , Mirja Kuehlewind 
Cc: tcpm IETF list , Richard Scheffenegger , Bob Briscoe 
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net> 
From: "Scheffenegger, Richard" 
Message-ID: <38c8eb38-3328-5fcd-818f-f92015377716@gmx.at>
Date: Tue, 17 Nov 2020 09:53:38 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:2wcVhteTy1HBtF7GcmgStTXf0WC6YarTN8/Xo+tTf/XIYWBSy3t YvEWV4tZaebTSIPMEIQdBbLflNccysiTKqCmUpXcmMbRSD643rz08x4gIAQburWOsKwt1iN 3/iOXXQv04OcWvFwa3HBcxcjw/RBk/Q15yQz214Ni2xxN3aHyfxHCwGYYgTkw+YAInj9lAy 9DFZ5K1mV1yKrk8Gm/YIg==
X-UI-Out-Filterresults: notjunk:1;V03:K0:3G71AAKv3mE=:z7nRJbnJzvqGsummOcYW7l dYeoe+ghClSDtGzRr59v5rzT63/GvOv8oPrc3V8lcoMuoN0q5k/P8H6HuSFF14qYGTfHoOCMF Bj6+ThMUCQhGbjzrF7a2vgo3FJi4/g3zzcYT/iSwTLL0WOIvtmDQHTgJBIREMWaS6VwbYUFOK /6oEZNh+1CVvqFLCyMqkbpbdzneBia0lXlCoMF8BK0KPx5z/BQerk21WAOZZOQpv3M3p0d58X RvphYbk54KLwpJG62ZEj3yla6K8hTuGc3bhKRkhotiZFIwAsbd42rqsHz9rL06tXbaysaYpV9 2UabMOye1yxrI4fbisNV+lnyM/aZLesm7X2n0zFQZs9GGTvyYBIQrTxaUcQpIpbg/NcX5tz32 ptlz5OEYbOUBrL1ttAdI+V5T1RIirRaBcg1UfhbL6zgfIm/W+1YZB8253dOclfMOKPqBO2sNw +ax4CMgIQqxa6upiYNJq+jW4CrpOh/LCfexcorTyUmYN7rmzqU5Gw0586CIHBivWRIp2ETfAz dwhLnOoAIRjwm3cthL5t9Dbw1lKSjje0gfMBFsH9ui48blWdkCawxY12zk14a2/V5fQ2mIoOz 7KKKDJbrAoEBZvX/daCYvKiSwL/uC5SGQG9Zzf9PyariLmOkTmM/ZPcJLIVZYfpHgnJJjRyh6 ht9A4j8UsUA0PoUGqt/9m0gfaDRvBhR8SPmBXZ5v8JkS08LCIxOdSQgcL8IFKhYRdcKO+15vH S9e+VYEhzna4JBw7RtM0xq1yropX/K+BeJMnWYWAav+4nnlZhe7DT8r/rSt3A6f/lOtlkQwko aNdCN6GUtABGCtXC1Q262aR/tExW8S6bH2bDDlUeDm6Rk/Xif9bBDtcZP9I7lHV3aXBi+rBwS hhi4uJJ1ltEI1NBsh46Q==
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 08:54:24 -0000

This would preclude the use of AccECN for AckCC purposes; effectively
allowing ACKs only to be sent for data packets.. (or a stream of
CE-marked ACKs after an unacked "uneven" data packet was received before).

If the last prior data packet was already ACKs, and then a stream of
CE-marked pure ACKs is observed, we loose synchronizity of the AccECN
counters...

Am 11.11.2020 um 10:37 schrieb Yoshifumi Nishida:
>
> "can send a pure ack on a pure ack when the ack value advances"?
> --
> Yoshi


From nobody Tue Nov 17 03:54:48 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3189E3A122F for ; Tue, 17 Nov 2020 03:54:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.622
X-Spam-Level: 
X-Spam-Status: No, score=-1.622 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.276, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZgjvnyZ-HD73 for ; Tue, 17 Nov 2020 03:54:40 -0800 (PST)
Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 78C813A11DB for ; Tue, 17 Nov 2020 03:54:38 -0800 (PST)
Received: from [IPv6:2a02:8109:1140:c3d:cb2:f97b:5916:11f5] (unknown [IPv6:2a02:8109:1140:c3d:cb2:f97b:5916:11f5]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id B423272275404 for ; Tue, 17 Nov 2020 12:54:35 +0100 (CET)
From: Michael Tuexen 
Content-Type: multipart/signed; boundary="Apple-Mail=_87D849EA-EA0C-441A-9291-41C9AF8792F4"; protocol="application/pkcs7-signature"; micalg=sha-256
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\))
Message-Id: 
Date: Tue, 17 Nov 2020 12:54:35 +0100
To: "tcpm@ietf.org Extensions" 
X-Mailer: Apple Mail (2.3654.20.0.2.21)
Archived-At: 
Subject: [tcpm] Meeting Notes
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 11:54:47 -0000

--Apple-Mail=_87D849EA-EA0C-441A-9291-41C9AF8792F4
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii

Dear all,

a first version of the meeting notes have been uploaded:

https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00

If you have comments, please let the chairs know.

Thanks a lot to Gorry for taking the notes and anyone improving the
notes during the meeting!

Best regards
Michael
--Apple-Mail=_87D849EA-EA0C-441A-9291-41C9AF8792F4
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEKow
ggUSMIID+qADAgECAgkA4wvV+K8l2YEwDQYJKoZIhvcNAQELBQAwgYIxCzAJBgNVBAYTAkRFMSsw
KQYDVQQKDCJULVN5c3RlbXMgRW50ZXJwcmlzZSBTZXJ2aWNlcyBHbWJIMR8wHQYDVQQLDBZULVN5
c3RlbXMgVHJ1c3QgQ2VudGVyMSUwIwYDVQQDDBxULVRlbGVTZWMgR2xvYmFsUm9vdCBDbGFzcyAy
MB4XDTE2MDIyMjEzMzgyMloXDTMxMDIyMjIzNTk1OVowgZUxCzAJBgNVBAYTAkRFMUUwQwYDVQQK
EzxWZXJlaW4genVyIEZvZXJkZXJ1bmcgZWluZXMgRGV1dHNjaGVuIEZvcnNjaHVuZ3NuZXR6ZXMg
ZS4gVi4xEDAOBgNVBAsTB0RGTi1QS0kxLTArBgNVBAMTJERGTi1WZXJlaW4gQ2VydGlmaWNhdGlv
biBBdXRob3JpdHkgMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMtg1/9moUHN0vqH
l4pzq5lN6mc5WqFggEcVToyVsuXPztNXS43O+FZsFVV2B+pG/cgDRWM+cNSrVICxI5y+NyipCf8F
XRgPxJiZN7Mg9mZ4F4fCnQ7MSjLnFp2uDo0peQcAIFTcFV9Kltd4tjTTwXS1nem/wHdN6r1ZB+Ba
L2w8pQDcNb1lDY9/Mm3yWmpLYgHurDg0WUU2SQXaeMpqbVvAgWsRzNI8qIv4cRrKO+KA3Ra0Z3qL
NupOkSk9s1FcragMvp0049ENF4N1xDkesJQLEvHVaY4l9Lg9K7/AjsMeO6W/VRCrKq4Xl14zzsjz
9AkH4wKGMUZrAcUQDBHHWekCAwEAAaOCAXQwggFwMA4GA1UdDwEB/wQEAwIBBjAdBgNVHQ4EFgQU
k+PYMiba1fFKpZFK4OpL4qIMz+EwHwYDVR0jBBgwFoAUv1kgNgB5oKAia4zV8mHSuCzLgkowEgYD
VR0TAQH/BAgwBgEB/wIBAjAzBgNVHSAELDAqMA8GDSsGAQQBga0hgiwBAQQwDQYLKwYBBAGBrSGC
LB4wCAYGZ4EMAQICMEwGA1UdHwRFMEMwQaA/oD2GO2h0dHA6Ly9wa2kwMzM2LnRlbGVzZWMuZGUv
cmwvVGVsZVNlY19HbG9iYWxSb290X0NsYXNzXzIuY3JsMIGGBggrBgEFBQcBAQR6MHgwLAYIKwYB
BQUHMAGGIGh0dHA6Ly9vY3NwMDMzNi50ZWxlc2VjLmRlL29jc3ByMEgGCCsGAQUFBzAChjxodHRw
Oi8vcGtpMDMzNi50ZWxlc2VjLmRlL2NydC9UZWxlU2VjX0dsb2JhbFJvb3RfQ2xhc3NfMi5jZXIw
DQYJKoZIhvcNAQELBQADggEBAIcL/z4Cm2XIVi3WO5qYi3FP2ropqiH5Ri71sqQPrhE4eTizDnS6
dl2e6BiClmLbTDPo3flq3zK9LExHYFV/53RrtCyD2HlrtrdNUAtmB7Xts5et6u5/MOaZ/SLick0+
hFvu+c+Z6n/XUjkurJgARH5pO7917tALOxrN5fcPImxHhPalR6D90Bo0fa3SPXez7vTXTf/D6OWS
T1k+kEcQSrCFWMBvf/iu7QhCnh7U3xQuTY+8npTD5+32GPg8SecmqKc22CzeIs2LgtjZeOJVEqM7
h0S2EQvVDFKvaYwPBt/QolOLV5h7z/0HJPT8vcP9SpIClxvyt7bPZYoaorVyGTkwggWsMIIElKAD
AgECAgcbY7rQHiw9MA0GCSqGSIb3DQEBCwUAMIGVMQswCQYDVQQGEwJERTFFMEMGA1UEChM8VmVy
ZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYu
MRAwDgYDVQQLEwdERk4tUEtJMS0wKwYDVQQDEyRERk4tVmVyZWluIENlcnRpZmljYXRpb24gQXV0
aG9yaXR5IDIwHhcNMTYwNTI0MTEzODQwWhcNMzEwMjIyMjM1OTU5WjCBjTELMAkGA1UEBhMCREUx
RTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5lcyBEZXV0c2NoZW4gRm9yc2NodW5n
c25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMGA1UEAwwcREZOLVZlcmVpbiBHbG9i
YWwgSXNzdWluZyBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ07eRxH3h+Gy8Zp
1xCeOdfZojDbchwFfylfS2jxrRnWTOFrG7ELf6Gr4HuLi9gtzm6IOhDuV+UefwRRNuu6cG1joL6W
LkDh0YNMZj0cZGnlm6Stcq5oOVGHecwX064vXWNxSzl660Knl5BpBb+Q/6RAcL0D57+eGIgfn5mI
TQ5HjUhfZZkQ0tkqSe3BuS0dnxLLFdM/fx5ULzquk1enfnjK1UriGuXtQX1TX8izKvWKMKztFwUk
P7agCwf9TRqaA1KgNpzeJIdl5Of6x5ZzJBTN0OgbaJ4YWa52fvfRCng8h0uwN89Tyjo4EPPLR22M
ZD08WkVKusqAfLjz56dMTM0CAwEAAaOCAgUwggIBMBIGA1UdEwEB/wQIMAYBAf8CAQEwDgYDVR0P
AQH/BAQDAgEGMCkGA1UdIAQiMCAwDQYLKwYBBAGBrSGCLB4wDwYNKwYBBAGBrSGCLAEBBDAdBgNV
HQ4EFgQUazqYi/nyU4na4K2yMh4JH+iqO3QwHwYDVR0jBBgwFoAUk+PYMiba1fFKpZFK4OpL4qIM
z+EwgY8GA1UdHwSBhzCBhDBAoD6gPIY6aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9v
dC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDBAoD6gPIY6aHR0cDovL2NkcDIucGNhLmRmbi5kZS9n
bG9iYWwtcm9vdC1nMi1jYS9wdWIvY3JsL2NhY3JsLmNybDCB3QYIKwYBBQUHAQEEgdAwgc0wMwYI
KwYBBQUHMAGGJ2h0dHA6Ly9vY3NwLnBjYS5kZm4uZGUvT0NTUC1TZXJ2ZXIvT0NTUDBKBggrBgEF
BQcwAoY+aHR0cDovL2NkcDEucGNhLmRmbi5kZS9nbG9iYWwtcm9vdC1nMi1jYS9wdWIvY2FjZXJ0
L2NhY2VydC5jcnQwSgYIKwYBBQUHMAKGPmh0dHA6Ly9jZHAyLnBjYS5kZm4uZGUvZ2xvYmFsLXJv
b3QtZzItY2EvcHViL2NhY2VydC9jYWNlcnQuY3J0MA0GCSqGSIb3DQEBCwUAA4IBAQCBeEWkTqR/
DlXwCbFqPnjMaDWpHPOVnj/z+N9rOHeJLI21rT7H8pTNoAauusyosa0zCLYkhmI2THhuUPDVbmCN
T1IxQ5dGdfBi5G5mUcFCMWdQ5UnnOR7Ln8qGSN4IFP8VSytmm6A4nwDO/afr0X9XLchMX9wQEZc+
lgQCXISoKTlslPwQkgZ7nu7YRrQbtQMMONncsKk/cQYLsgMHM8KNSGMlJTx6e1du94oFOO+4oK4v
9NsH1VuEGMGpuEvObJAaguS5Pfp38dIfMwK/U+d2+dwmJUFvL6Yb+qQTkPp8ftkLYF3sv8pBoGH7
EUkp2KgtdRXYShjqFu9VNCIaE40GMIIF4DCCBMigAwIBAgIMIRX9tDE2QqO3mVLXMA0GCSqGSIb3
DQEBCwUAMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1ciBGb2VyZGVydW5nIGVp
bmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYDVQQLDAdERk4tUEtJMSUw
IwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBMB4XDTE5MDYwNDE0MjkxMFoXDTIy
MDYwMzE0MjkxMFowfDELMAkGA1UEBhMCREUxIDAeBgNVBAoMF0ZhY2hob2Noc2NodWxlIE11ZW5z
dGVyMTIwMAYDVQQLDClGYWNoYmVyZWljaCBFbGVrdHJvdGVjaG5payB1bmQgSW5mb3JtYXRpazEX
MBUGA1UEAwwOTWljaGFlbCBUdWV4ZW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM
r8qQcPxLFCxzPtXvRyM9KeQaxyMA8gwUNc7IIiATs6mRQFe5ib/mvwT9nc0bAa+94go6HJDiD3FJ
NkTo4u8aBsIcTt5pJtdBQLm88PLakbe3+fp/00//n7xxbTh7mAtFVCf25LxDCKkrdGk/+jglRq/R
VdwhZZ3VpYCrx8YfI/hIzdRL3+4I4z/mnQ8K0X8d2MVVPG+9nBEngdnYGez5f/8wIVOgEYYBc21k
yvMnVXLu2Ing+LwBd0gDG9Vasv87MNHCOZfJTNBlLhI2UDei/uNg9Zd4ynlMpPWZ7v0oiDWvmv8E
OuD4oric8heyD0OYK2FL4qcVC4dn4pnyulfHAgMBAAGjggJOMIICSjA+BgNVHSAENzA1MA8GDSsG
AQQBga0hgiwBAQQwEAYOKwYBBAGBrSGCLAEBBAQwEAYOKwYBBAGBrSGCLAIBBAQwCQYDVR0TBAIw
ADAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQW
BBTxiodBVL/lA4p5iNesIsJRhhgd6zAfBgNVHSMEGDAWgBRrOpiL+fJTidrgrbIyHgkf6Ko7dDAg
BgNVHREEGTAXgRV0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUwgY0GA1UdHwSBhTCBgjA/oD2gO4Y5aHR0
cDovL2NkcDEucGNhLmRmbi5kZS9kZm4tY2EtZ2xvYmFsLWcyL3B1Yi9jcmwvY2FjcmwuY3JsMD+g
PaA7hjlodHRwOi8vY2RwMi5wY2EuZGZuLmRlL2Rmbi1jYS1nbG9iYWwtZzIvcHViL2NybC9jYWNy
bC5jcmwwgdsGCCsGAQUFBwEBBIHOMIHLMDMGCCsGAQUFBzABhidodHRwOi8vb2NzcC5wY2EuZGZu
LmRlL09DU1AtU2VydmVyL09DU1AwSQYIKwYBBQUHMAKGPWh0dHA6Ly9jZHAxLnBjYS5kZm4uZGUv
ZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQwSQYIKwYBBQUHMAKGPWh0dHA6
Ly9jZHAyLnBjYS5kZm4uZGUvZGZuLWNhLWdsb2JhbC1nMi9wdWIvY2FjZXJ0L2NhY2VydC5jcnQw
DQYJKoZIhvcNAQELBQADggEBABs3VlmIZ1VF3HkaQdjInZYmamRabbdgJ7cz6m6o/agKL7+Vhwx7
1BaaYs2gMa5Nu/GJ3YfdqIsUlYtKdl58Yhp/89E6xBfJkItS+rE8RFdf2XgklGlx7GmsvdA3tId5
b6K6r9a5wpVN0epxY6K8wwpzEib6fJLcHylybQXZ7JSOaLRLp6WU3QPoyTT7FpvAe/0b2MSCbPX4
fc53PCn2aGgXuRFVQeCn1SP1BF3QW1ppkFhcF6G+0JcUxYFAXE6r/71WZBvUHqoG/th2hAwQnS+Y
KhUYh4SZQH3/ursXXJYXQ5vyIhkN1FZlmtWA8+ofdCnoqSTbiSX2Aa/kKbTapwgxggOdMIIDmQIB
ATCBnjCBjTELMAkGA1UEBhMCREUxRTBDBgNVBAoMPFZlcmVpbiB6dXIgRm9lcmRlcnVuZyBlaW5l
cyBEZXV0c2NoZW4gRm9yc2NodW5nc25ldHplcyBlLiBWLjEQMA4GA1UECwwHREZOLVBLSTElMCMG
A1UEAwwcREZOLVZlcmVpbiBHbG9iYWwgSXNzdWluZyBDQQIMIRX9tDE2QqO3mVLXMA0GCWCGSAFl
AwQCAQUAoIIBzzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMDEx
MTcxMTU0MzVaMC8GCSqGSIb3DQEJBDEiBCDlZUZZ4BCNo/rsHdC1V8jOYJW0J9oblWTyykbTRijg
QjCBrwYJKwYBBAGCNxAEMYGhMIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8VmVyZWluIHp1
ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUuIFYuMRAwDgYD
VQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5nIENBAgwhFf20
MTZCo7eZUtcwgbEGCyqGSIb3DQEJEAILMYGhoIGeMIGNMQswCQYDVQQGEwJERTFFMEMGA1UECgw8
VmVyZWluIHp1ciBGb2VyZGVydW5nIGVpbmVzIERldXRzY2hlbiBGb3JzY2h1bmdzbmV0emVzIGUu
IFYuMRAwDgYDVQQLDAdERk4tUEtJMSUwIwYDVQQDDBxERk4tVmVyZWluIEdsb2JhbCBJc3N1aW5n
IENBAgwhFf20MTZCo7eZUtcwDQYJKoZIhvcNAQELBQAEggEAwMZn3XA3BTX2b9ZVP7FUYzd/hubM
MBop6KycULQ1hprl9wYilbSeVzyumwmDkVrK+ITtc5ZdA53REkNqaQHG6ouZrJRrxD0PV2ikrtdj
uX+StagEkU0Y0TCrJNco9/cDytueBmKFSI5f8iAGZHI905mvkedD6lqXXzMBSDdDpd2V2WavbENA
P7XUmSjrpy48VDZCvMJBq05dMOTr1VXM8BZOqgm/eSHC7Ibf+VembTMDlZ4y2gVc0sRSglUZ6LyR
oK8KUtE4xd8dVOSfq2sEJpcsVsV1xF6h1rLS4dnCO80zPd7ClpdufczK8k51Xd9Cqei6hrHzuCE5
UQzFESNkawAAAAAAAA==
--Apple-Mail=_87D849EA-EA0C-441A-9291-41C9AF8792F4--


From nobody Tue Nov 17 04:20:16 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 29EF03A1201; Tue, 17 Nov 2020 04:20:14 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z9qqQICC3fcq; Tue, 17 Nov 2020 04:20:11 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 76C043A11FE; Tue, 17 Nov 2020 04:20:10 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=upq05HrRwrd88indX46nAn8S9g4BpXQ14j6N5tD4WIA=; b=BiPAArqOLPeCoFxiTYk5vlzX+ pSIv7n6KPoHiaFm725eZmnLR73nZ76CftjDf0qqU4CCGzaFzJopM7s2ictiVYC65WhBaZtmZ4gyff J2lVZ3+YDo95RGMuKwVf1IdpfE2Sz52tJxBWrT/Sok/klZ7X62/vHGt2gpTtGdDmC7bgCVbDznREB 3hxJYbBapRzG4oF2KK2W7lWkh1r5aR4WO3inDnUtYLubPcknKzewo2DsIOk/NJ2slAnG+Pi2iY4FS pOgJ0qVX9D22kuvkA7XJu86RTaJXGH4kiWPaj6ul7swNMHmXQN8SnV4rE0sgErdP+RySdlQxkda9U /Rsi1XN+A==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:39552 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kezxu-0022eD-Uf; Tue, 17 Nov 2020 12:20:08 +0000
To: "Scharf, Michael" , Yoshifumi Nishida 
Cc: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de>
From: Bob Briscoe 
Message-ID: <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net>
Date: Tue, 17 Nov 2020 12:20:05 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <30781ea61a794131beafe9997ed9221a@hs-esslingen.de>
Content-Type: multipart/alternative; boundary="------------D9CED6A3A514B6A2C21BC38E"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 12:20:14 -0000

This is a multi-part message in MIME format.
--------------D9CED6A3A514B6A2C21BC38E
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Michael,

On 17/11/2020 08:10, Scharf, Michael wrote:
>
> Bob,
>
> I am fine with the two option kinds proposed in -13, but I don’t buy 
> your arguments why this encoding is better than others.
>
> Most importantly, I don’t think that your “forward compatibility” 
> argument is a very compelling reason for two codepoints. All proposals 
> I am aware of have their pros and cons. And I am not aware of a really 
> comprehensive discussion. At least neither the e-mail below nor 
> today’s meeting discuss all tradeoffs I would investigate.
>

[BB] You're right - the discussion below is useful and necessary.

> For instance, the benefit of the length encoding variant would be to 
> consume only one TCP option codepoint now. If we decided later to that 
> a flag byte is needed (say, at the end), a follow-up proposed standard 
> could specify another option format with that flag byte, using a 
> second codepoint.
>

[BB]
* The new codepoint would require years to deploy (including in proxies) 
before it works e2e.
* Whereas the 'forward compatibility' approach tries to ensure that 
AccECN implementations that are being built today will be ready to 
interwork with a future extension.
* In contrast, if your hypothetical new AccECN codepoint did need to be 
deployed later, it would need to include a fall-back to the current 
AccECN option when one peer doesn't understand the new codepoint.
* Fall-back would need to avoid a second round of handshake (given this 
is for low latency), which would only work reliably with an option on 
the SYN (because the 3rd ACK is unreliable)
* AccECN has so far avoided using any SYN option space, which is 
important given that is the scarcest resource.

These days, extending TCP is like playing chess - we have to think 
multiple moves ahead! I need to learn to articulate onto the list all my 
reasons for rejecting certain parts of the design space. Sorry about that.

Yes, I admit, the current proposal is at the cost of burning two TCP 
option codepoints. However, you said yourself when you proposed this 
approach, that they are not (currently) a scarce resource. Certainly not 
as scarce as option space. I just checked: roughly 15% of the option 
codepoint allocation space has been used so far. So that's where I would 
suggest we compromise first.


> Regarding the resulting codepoint consumption (two option kinds) and 
> the addition of a flag byte, this approach would basically end up with 
> the same result as the current proposal in -13 plus some hypothetical 
> future addition leveraging the length field beyond the currently known 
> use.
>
> And, yes, the length encoding variant may be less flexible and/or 
> consume some more bits in some cases, but on the plus side it only 
> needs one codepoint now – and given that it is entirely unclear 
> whether any AccECN option will indeed be widely used in future, this 
> is a big plus. The fact that implementers are quite silent on the 
> option design is not a good sign.
>

[BB] Here's another possibility then.

a) [K|L| EE1B | ECEB | EE0B ]
b) [K|L| EE1B | ECEB | EE0B |F]    if F = 0bXXXXXXX0
c) [K|L| EE0B | ECEB | EE1B |F]    if F = 0bXXXXXXX1

We'd have to decide which order alternative (a) takes. I've picked EE1B 
first only 'cos you're right that there is not currently as much 
pressure (on the public Internet) for AccECN with ECT(0).

==Legend==
K:    Kind (1B)
L:    Length (1B)
ExxB: Echo xx byte counter
F:    Flags (1B)

Alternatives a) and b) are equivalent, at least while the only allocated 
flag is for field order.
However, alternativeb) is available in case other flags are allocated in 
future
The flags byte can only be omitted if (L % 3 == 2).
The flags byte is considered present if (L % 3) == 0.

Forward compatibility: Options with (L % 3 == 1) MUST be assumed to 
include a flags byte, and current implementations ignore the last byte.

The flags byte is optional to implement (even if the AccECN option is 
implemented)
If the server includes a flags byte on the SYN/ACK but the client does 
not include one on the 3rd ACK or the first data packet, the server 
assumes the client does not implement the flags byte and uses only 
alternative a) for the remainder of the connection.


> To me, one potential difference between two proposals would be 
> incremental deployment. The proposal in -13 only has an advantage if 
> middleboxes such as firewalls will indeed pass TCP options with a 
> format that contains content beyond the (first) Accurate ECN standard 
> (i.e., currently unused length values). IMHO it is too early to know 
> whether firewalls would indeed allow this in future.
>
> From a security perspective, it is not clear to me whether allowing 
> arbitrary unspecified bytes in a TCP option is a good idea **at all**. 
> It will be interesting to hear the opinion from SEC area on that. 
> Personally, I am not convinced that this really makes sense, but I my 
> concerns are not strong enough to formally push back. I’ll leave it to 
> others to think about whether this is a bug or a feature.
>

[BB] Well, let's first try to deal with security ourselves:
* Octets that are explicitly declared as part of an option cannot be 
used for a buffer overflow attack. I don't really need to, but I could 
add the following text to the forward compatibility wording:
     A receiver considers octets beyond those it uses, but within the 
specified length, as if they are padding.
* And such octets cannot be any different from the current ability of a 
sender to add padding. So there's no new attack possible here.
* And there's no need to specify a max length for any AccECN Option, 
which would just unnecessarily limit flexibility.


> Maybe one lesson learnt is that the document could have a 
> non-normative appendix that explains the rationale for the finally 
> picked TCP option encoding. That may also help if there are further 
> questions whether two codepoints are really required, e.g. by the IESG 
> (if two codepoints are still the design after WGLC). At least for past 
> TCP option codepoint allocations I recall some discussions late in the 
> IETF process. In those past cases, good arguments in an appendix and 
> running code has helped a lot.
>

[BB] I can do that. Appx B is already similar, giving Design Rationale 
for the ACE field.


Bob

> Michael
>
> *From:*Bob Briscoe 
> *Sent:* Tuesday, November 17, 2020 6:10 AM
> *To:* Scharf, Michael ; Yoshifumi 
> Nishida 
> *Cc:* Michael Tuexen ; tcpm-chairs@ietf.org; 
> Mirja Kuehlewind ; Scheffenegger, Richard 
> ; tcpm IETF list 
> *Subject:* Re: AccECN field order
>
> Michael,
>
> On 16/11/2020 17:36, Scharf, Michael wrote:
>
>     Bob,
>
>     One proposal using the length field with **one option codepoint
>     only** is detailed in:
>     https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>
>     It is the third option mentioned in this e-mail. One example would
>     be to use option length values 5/8/11 for one encoding type and
>     option length values 6/9/12 for the other encoding type (i.e.,
>     order of fields). Or one could use some other combination of
>     length values – the only requirement is that a certain value for
>     the option length is only used by one of the option formats. In
>     that approach, the value of the length field would thus directly
>     describe the encoding of the option. Unless I miss something, this
>     would work and it would just require one option codepoint.
>
>     Thus, alternatives to two option codepoints exist and I have
>     explained them on the list in March 2020.
>
>
> OK, sorry, yes, I remember this now.
>
> As I will explain in the AccECN status update talk today in virtual 
> Bangkok, the draft has made provision for different length values than 
> 5/8/11. It says existing implementations MUST accept length values 
> other than those currently defined. But then read in as many whole 
> 3-byte fields as they can.
>
> This can be used to add a flags byte on the end in future, for 
> extensibility. Or any other form of extensibility the WG might decide 
> in the future.
>
> I know a flags byte at the end seems odd compared to at the beginning. 
> But (if decided it's needed in future) it's reasonably easy to 
> implement by reading the whole option, then processing the last byte, 
> before reading the rest of the option.
>
> I believe you will agree that this is a better way to utilize 
> different lengths.
>
> And thank you for repeatedly emphasizing that you're happy with the 
> 2-kind scheme, or other alternatives.
>
>
> Bob
>
>
>     Anyway, I don’t really care how the options are encoded as long as
>     the receiver doesn’t need per-connection state for decoding a TCP
>     option. So, personally, I would be fine with using e.g. the length
>     field as described in my old e-mail. Or an additional flag byte.
>     And one could come up with further encodings, e.g., by using one
>     or a few bits as a short “type” field for each counter. This is
>     all about protocol engineering. And all these variants have their
>     pros and cons.
>
>     I am also fine with using two option codepoints as specified in
>     -13; this is probably the approach that consumes the least number
>     of bits.
>
>     Michael (w/o any hat)
>
>     *From:* Bob Briscoe 
>     
>     *Sent:* Monday, November 16, 2020 5:52 PM
>     *To:* Yoshifumi Nishida 
>     
>     *Cc:* Scharf, Michael 
>     ; Michael Tuexen
>      ;
>     tcpm-chairs@ietf.org ; Mirja
>     Kuehlewind  ;
>     Scheffenegger, Richard 
>     ; tcpm IETF list
>      
>     *Subject:* AccECN field order
>
>     Yoshi, (adding the tcpm list in cc)
>
>     On 05/11/2020 06:58, Yoshifumi Nishida wrote:
>
>         Hi Bob,
>
>         On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe
>         > wrote:
>
>             Yoshi,
>
>             On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>
>                 Hi, folks,
>
>
>
>
>                 In my understanding, I'm not sure if we settled down
>                 on using two option kinds or encoding schemes for
>                 24bits fields in acc ecn draft.
>
>                 So, I think there're still something to be clarified
>                 and hope things will be settled at the meeting.
>
>
>             [BB] I know a WG can change it's mind at any time. But I'd
>             rather we just clarified what a previous decision was, to
>             avoid the need to keep re-opening discussion on a question
>             that have been decided then changed three different ways
>             already.
>
>             My memory is not so good these days. I trusted that
>             Michael S remembered the decision correctly, and I seem to
>             remember that decision being made.
>             I've just checked the minutes of the last interim:
>             https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/
>             and they mention Michael's proposal to use two kinds, but
>             don't record any decision.
>             The jabber log gives no clues about any decision.
>
>             I can't find an audio or video recording. Can you point me
>             at one?
>
>         I thought that it's because there was no clear decision at the
>         meeting.
>
>         But, you can check https://www.youtube.com/watch?v=KsB0_Nb8-kA
>         
>
>         Please let us know if you have any questions or opinions with
>         regard to this.
>
>
>     [BB] I checked the Youtube link you sent below.
>
>     First I think we're agreed no-one was fighting for us to keep the
>     previous way we did this (using the initial value of the field to
>     set the order for the connection).
>     In my presentation I said there was strong resistance from Michael
>     to do it a different way.
>     (also, offlist, the co-authors including me also didn't like this
>     so much. And Ilpo said it made the implementation complex.)
>
>     Then came the question of what we do instead. There were three
>     alternative proposals:
>     a) use 2 option kinds
>     b) add a flags byte
>     c) somehow use the length field maybe
>
>     Michael raised (c) in the meeting as a possibility, but no-one
>     could think how to distinguish two options of the same length but
>     a different field order using the length field. Michael said he'd
>     post any ideas to the list if he could think of any, but that
>     didn't happen.
>
>     So we're left choosing between (a) and (b).
>     I said in the meeting (and on the list when discussing with Ilpo)
>     that I'd be happy to go with (b), but only if there was another
>     use for a flag. Because it would consume 1B more options space in
>     many packets, which is a scarce resource.
>
>     Ilpo had a proposed use for another flag (to help synch counters
>     after a loss), but I think the discussion about it ended that it
>     wouldn't be helpful, 'cos the way it worked depended on itself
>     (circular logic).
>
>     In conclusion, I don't think there was an explicit decision to go
>     with 2 option kinds, but it ended up as the 'last person standing'.
>     I like it. It's simple. And apparently option kinds are not such a
>     scarce resource.
>
>     Perhaps we can ratify this in the WG tomorrow.
>
>
>     Bob
>
>
>
>         Thanks,
>
>         --
>
>         Yoshi
>
>
>
>
>     -- 
>
>     ________________________________________________________________
>
>     Bob Briscoehttp://bobbriscoe.net/
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------D9CED6A3A514B6A2C21BC38E
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Michael,

    

    
On 17/11/2020 08:10, Scharf, Michael
      wrote:

    

    

      
      
      
      

        
Bob,

        
 

        
I am fine with the two option kinds proposed in
            -13, but I don’t buy your arguments why this encoding is
            better than others.

        
 

        
Most importantly, I don’t think that your
            “forward compatibility” argument is a very compelling reason
            for two codepoints. All proposals I am aware of have their
            pros and cons. And I am not aware of a really comprehensive
            discussion. At least neither the e-mail below nor today’s
            meeting discuss all tradeoffs I would investigate.

      

    

    

    [BB] You're right - the discussion below is useful and necessary.

    

    

      

        

        
 

        
For instance, the benefit of the length
            encoding variant would be to consume only one TCP option
            codepoint now. If we decided later to that a flag byte is
            needed (say, at the end), a follow-up proposed standard
            could specify another option format with that flag byte,
            using a second codepoint.

      

    

    

    [BB] 

    * The new codepoint would require years to deploy (including in
    proxies) before it works e2e. 

    * Whereas the 'forward compatibility' approach tries to ensure that
    AccECN implementations that are being built today will be ready to
    interwork with a future extension. 

    * In contrast, if your hypothetical new AccECN codepoint did need to
    be deployed later, it would need to include a fall-back to the
    current AccECN option when one peer doesn't understand the new
    codepoint. 

    * Fall-back would need to avoid a second round of handshake (given
    this is for low latency), which would only work reliably with an
    option on the SYN (because the 3rd ACK is unreliable)

    * AccECN has so far avoided using any SYN option space, which is
    important given that is the scarcest resource.

    

    These days, extending TCP is like playing chess - we have to think
    multiple moves ahead! I need to learn to articulate onto the list
    all my reasons for rejecting certain parts of the design space.
    Sorry about that.

    

    Yes, I admit, the current proposal is at the cost of burning two TCP
    option codepoints. However, you said yourself when you proposed this
    approach, that they are not (currently) a scarce resource. Certainly
    not as scarce as option space. I just checked: roughly 15% of the
    option codepoint allocation space has been used so far. So that's
    where I would suggest we compromise first.

    

    

    

      

        

        
 

        
Regarding the resulting codepoint consumption
            (two option kinds) and the addition of a flag byte, this
            approach would basically end up with the same result as the
            current proposal in -13 plus some hypothetical future
            addition leveraging the length field beyond the currently
            known use.

        
 

        
And, yes, the length encoding variant may be
            less flexible and/or consume some more bits in some cases,
            but on the plus side it only needs one codepoint now – and
            given that it is entirely unclear whether any AccECN option
            will indeed be widely used in future, this is a big plus.
            The fact that implementers are quite silent on the option
            design is not a good sign.

      

    

    

    [BB] Here's another possibility then. 

    

    a) [K|L| EE1B | ECEB | EE0B ]

    b) [K|L| EE1B | ECEB | EE0B |F]    if F = 0bXXXXXXX0

    c) [K|L| EE0B | ECEB | EE1B |F]    if F =
            0bXXXXXXX1

          

    We'd have to decide which order alternative (a) takes. I've picked
    EE1B first only 'cos you're right that there is not currently as
    much
    pressure (on the public Internet) for AccECN with ECT(0).

    

            ==Legend==

            K:    Kind (1B)

            L:    Length (1B)

            ExxB: Echo xx byte counter

            F:    Flags (1B)

            

            Alternatives a) and b) are equivalent, at least while the
            only allocated flag is for field order.

            However, alternative
                            b) is available in case other flags are
                            allocated in future

                          The
                                    flags byte can only be omitted if (L
                                    % 3 == 2).

                                  The
            flags byte is considered present if (L % 3) == 0.

            

            Forward compatibility: Options with (L % 3 == 1) MUST be
            assumed to include a flags byte, and current implementations
            ignore the last byte.

            

            The flags byte is optional to implement (even if the AccECN
            option is implemented)

            If the server includes a flags byte on the SYN/ACK but the
            client does not include one on the 3rd ACK or the first data
            packet, the server assumes the client does not implement the
            flags byte and uses only alternative a) for the remainder of
            the connection.

            

            

          
      
    

      

        

        
 

        
To me, one potential difference between two
            proposals would be incremental deployment. The proposal in
            -13 only has an advantage if middleboxes such as firewalls
            will indeed pass TCP options with a format that contains
            content beyond the (first) Accurate ECN standard (i.e.,
            currently unused length values). IMHO it is too early to
            know whether firewalls would indeed allow this in future.

        
 

        
From a security perspective, it is not clear to
            me whether allowing arbitrary unspecified bytes in a TCP
            option is a good idea *at all*. It will be
            interesting to hear the opinion from SEC area on that.
            Personally, I am not convinced that this really makes sense,
            but I my concerns are not strong enough to formally push
            back. I’ll leave it to others to think about whether this is
            a bug or a feature.

      

    

    

    [BB] Well, let's first try to deal with security ourselves:

    * Octets that are explicitly declared as part of an option cannot be
    used for a buffer overflow attack. I don't really need to, but I
    could add the following text to the forward compatibility wording: 

        A receiver considers octets beyond those it uses, but within the
    specified length, as if they are padding. 

    * And such octets cannot be any different from the current ability
    of a sender to add padding. So there's no new attack possible here.

    * And there's no need to specify a max length for any AccECN Option,
    which would just unnecessarily limit flexibility.

    

    

    

      

        

        
 

        
Maybe one lesson learnt is that the document
            could have a non-normative appendix that explains the
            rationale for the finally picked TCP option encoding. That
            may also help if there are further questions whether two
            codepoints are really required, e.g. by the IESG (if two
            codepoints are still the design after WGLC). At least for
            past TCP option codepoint allocations I recall some
            discussions late in the IETF process. In those past cases,
            good arguments in an appendix and running code has helped a
            lot.

      

    

    

    [BB] I can do that. Appx B is already similar, giving Design
    Rationale for the ACE field.

    

    

    Bob

    

    

      

        

        
 

        
Michael

        
 

        
 

        
 

        

          

            

              
From: Bob Briscoe <ietf@bobbriscoe.net>
                  

                  Sent: Tuesday, November 17, 2020 6:10 AM

                  To: Scharf, Michael
                  <Michael.Scharf@hs-esslingen.de>; Yoshifumi
                  Nishida <nsd.ietf@gmail.com>

                  Cc: Michael Tuexen
                  <tuexen@fh-muenster.de>; tcpm-chairs@ietf.org;
                  Mirja Kuehlewind <ietf@kuehlewind.net>;
                  Scheffenegger, Richard
                  <Richard.Scheffenegger@netapp.com>; tcpm IETF
                  list <tcpm@ietf.org>

                  Subject: Re: AccECN field order

            

          

          
 

          
Michael,

          

            
On 16/11/2020 17:36, Scharf, Michael
              wrote:

          

          

            
Bob,

            
 

            
One
                proposal using the length field with *one option
                  codepoint only* is detailed in:
                
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/

            
 

            
It is
                the third option mentioned in this e-mail. One example
                would be to use option length values 5/8/11 for one
                encoding type and option length values 6/9/12 for the
                other encoding type (i.e., order of fields). Or one
                could use some other combination of length values – the
                only requirement is that a certain value for the option
                length is only used by one of the option formats. In
                that approach, the value of the length field would thus
                directly describe the encoding of the option. Unless I
                miss something, this would work and it would just
                require one option codepoint.

            
 

            
Thus,
                alternatives to two option codepoints exist and I have
                explained them on the list in March 2020.

          

          


            OK, sorry, yes, I remember this now.

            

            As I will explain in the AccECN status update talk today in
            virtual Bangkok, the draft has made provision for different
            length values than 5/8/11. It says existing implementations
            MUST accept length values other than those currently
            defined. But then read in as many whole 3-byte fields as
            they can.

            

            This can be used to add a flags byte on the end in future,
            for extensibility. Or any other form of extensibility the WG
            might decide in the future.

            

            I know a flags byte at the end seems odd compared to at the
            beginning. But (if decided it's needed in future) it's
            reasonably easy to implement by reading the whole option,
            then processing the last byte, before reading the rest of
            the option.

            

            I believe you will agree that this is a better way to
            utilize different lengths.

            

            And thank you for repeatedly emphasizing that you're happy
            with the 2-kind scheme, or other alternatives.

            

            

            Bob

            

            

            

          

            
 

            
Anyway,
                I don’t really care how the options are encoded as long
                as the receiver doesn’t need per-connection state for
                decoding a TCP option. So, personally, I would be fine
                with using e.g. the length field as described in my old
                e-mail. Or an additional flag byte. And one could come
                up with further encodings, e.g., by using one or a few
                bits as a short “type” field for each counter. This is
                all about protocol engineering. And all these variants
                have their pros and cons.

            
 

            
I am
                also fine with using two option codepoints as specified
                in -13; this is probably the approach that consumes the
                least number of bits.

            
 

            
Michael
                (w/o any hat)

            
 

            
 

            
 

            

              

                

                  
From: Bob Briscoe 
                      <ietf@bobbriscoe.net> 

                    Sent: Monday, November 16, 2020 5:52 PM

                    To: Yoshifumi Nishida <nsd.ietf@gmail.com>

                    Cc: Scharf, Michael <Michael.Scharf@hs-esslingen.de>;
                    Michael Tuexen
                    <tuexen@fh-muenster.de>;
                    
                      tcpm-chairs@ietf.org; Mirja Kuehlewind <ietf@kuehlewind.net>;
                    Scheffenegger, Richard
                    <Richard.Scheffenegger@netapp.com>;
                    tcpm IETF list
                    <tcpm@ietf.org>

                    Subject: AccECN field order

                

              

              
 

              
Yoshi,
                (adding the tcpm list in cc)

              

                
On 05/11/2020 06:58, Yoshifumi
                  Nishida wrote:

              

              

                

                  

                    
Hi Bob,

                  

                  

                    
 

                  

                  

                    

                      
On Wed, Nov 4, 2020 at 3:29
                        PM Bob Briscoe <ietf@bobbriscoe.net>
                        wrote:

                    

                    

                      

                        
Yoshi,

                        

                          
On 04/11/2020 06:51,
                            Yoshifumi Nishida wrote:

                        

                        

                          

                            

                              
Hi, folks,

                            

                          

                        

                      

                    

                  

                

              

              


                

                

                

              

                

                  

                    

                      

                        

                          

                            

                              
In my understanding,
                                I'm not sure if we settled down on using
                                two option kinds or encoding schemes for
                                24bits fields in acc ecn draft.

                            

                            

                              
So, I think there're
                                still something to be clarified and hope
                                things will be settled at the meeting.

                            

                          

                        

                        


                          [BB] I know a WG can change it's mind at any
                          time. But I'd rather we just clarified what a
                          previous decision was, to avoid the need to
                          keep re-opening discussion on a question that
                          have been decided then changed three different
                          ways already.

                          

                          My memory is not so good these days. I trusted
                          that Michael S remembered the decision
                          correctly, and I seem to remember that
                          decision being made.

                          I've just checked the minutes of the last
                          interim:

                              
https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/

                          and they mention Michael's proposal to use two
                          kinds, but don't record any decision.

                          The jabber log gives no clues about any
                          decision.

                          

                          I can't find an audio or video recording. Can
                          you point me at one?

                      

                    

                    

                      
 

                    

                    

                      
I thought that it's because
                        there was no clear decision at the meeting. 

                    

                    
                    

                      
Please let us know if you
                        have any questions or opinions with regard to
                        this.

                    

                  

                

              

              


                [BB] I checked the Youtube link you sent below.

                

                First I think we're agreed no-one was fighting for us to
                keep the previous way we did this (using the initial
                value of the field to set the order for the connection).
                

                In my presentation I said there was strong resistance
                from Michael to do it a different way.
                

                (also, offlist, the co-authors including me also didn't
                like this so much. And Ilpo said it made the
                implementation complex.)

                

                Then came the question of what we do instead. There were
                three alternative proposals:

                a) use 2 option kinds

                b) add a flags byte

                c) somehow use the length field maybe

                

                Michael raised (c) in the meeting as a possibility, but
                no-one could think how to distinguish two options of the
                same length but a different field order using the length
                field. Michael said he'd post any ideas to the list if
                he could think of any, but that didn't happen.

                

                So we're left choosing between (a) and (b).

                I said in the meeting (and on the list when discussing
                with Ilpo) that I'd be happy to go with (b), but only if
                there was another use for a flag. Because it would
                consume 1B more options space in many packets, which is
                a scarce resource.

                

                Ilpo had a proposed use for another flag (to help synch
                counters after a loss), but I think the discussion about
                it ended that it wouldn't be helpful, 'cos the way it
                worked depended on itself (circular logic).

                

                In conclusion, I don't think there was an explicit
                decision to go with 2 option kinds, but it ended up as
                the 'last person standing'.

                I like it. It's simple. And apparently option kinds are
                not such a scarce resource.

                

                Perhaps we can ratify this in the WG tomorrow.

                

                

                Bob

                

                

                

                

              

                

                  

                    

                      
 

                    

                    

                      
Thanks,

                    

                    

                      
--

                    

                    

                      
Yoshi

                    

                  

                

              

              


                

                

                

              
-- 

              
________________________________________________________________

              
Bob Briscoe                               http://bobbriscoe.net/

            

          

          


            

            

          
-- 

          
________________________________________________________________

          
Bob Briscoe                               http://bobbriscoe.net/

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------D9CED6A3A514B6A2C21BC38E--


From nobody Tue Nov 17 05:19:25 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6AED13A12CD; Tue, 17 Nov 2020 05:19:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MEu0aEeuY-eV; Tue, 17 Nov 2020 05:19:18 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 764043A0EA9; Tue, 17 Nov 2020 05:18:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605619132; bh=5h6Rx1riUkHM7q9SmGLeIicwWGDgUFKtXpbIBGuB/fg=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=fb/2r7ttZ0TpRiqbemTIcH7TjLperIEFbE29msdcghide930hb87IB11O+o2pjYkA +z7M6M/VJAkVwuaKvabfPJps+ZlWcdqHj7bXFc0nUsTreLmzFOuSzS/76updLbDYiW ap22q5XEIh2TySgQ68qWXM3Y+Q/i8TTcSZ3NTyKg=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [10.249.68.20] ([217.70.211.12]) by mail.gmx.com (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MAfYw-1kYRIS1UBE-00B8Ex; Tue, 17 Nov 2020 14:18:52 +0100
To: Yuchung Cheng , Kevin Yang 
Cc: "tcpm@ietf.org Extensions" , Eric Dumazet , iccrg IRTF list , Neal Cardwell 
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com>    
From: "Scheffenegger, Richard" 
Message-ID: 
Date: Tue, 17 Nov 2020 14:18:49 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:cMLsA6pS/+6A6FwhB6X5We7wYZUp/LltC/HGq58qG2vsevaHI96 r/+au8FrkqY2GBDre1AbQI/nY+y1s4jdcJHqotE0zgDDvAcb+HTaTzCf2cS+8NQVVn4r5/W t3XY3Bj1qcZuqwlK9ldPYSwFVTAndIsuH97/nJ13deSt/MfuDbRvlKS4/R9I6xEjBt7Tyzk e7db3Kmfk3bRHYysJIQKg==
X-UI-Out-Filterresults: notjunk:1;V03:K0:qhWFIo8P1SU=:/519gkFReVuurxchLumxFW 5Rh6Wvs3b+fpVX90X7jfTX2zXbtsLKMZQHAaplG1HkdyTw4ZBiqzrys0KRwCU9ytYu3bwLhbe KI4Y3bgwaCMCwsQeY9IM0fFgqyebVIYxSGmDAAGcXfN7BEFtHTdT9LU8TNUSq+/oeAtHF2jBn +2l+lqMbv5Eqoobgv8IdepL5AIiobaPQKqtMmcvNSXb+yjZlI7VSSTDpO2pAJ8m2t/bZ/vnAO /AyFPJayz8uGqq7w1k05YhI5EP+LUdH7kBQQ3zqTVGiq1HAsHh3E/Ni6zjdOcQ5SsjWOHaUft aD+b7xTpw96b/BKHLb6Ah13atNs22GPPQpmhljNxw3eAKsXJnKWHH6vUHotI4/QcGAbRQvbOd hxkygWFgMgogB6Nibp+SbnwOQx6V42AmH2ii2O2XDhkZSVX838RDDaKKIJudYbPMyKfUghaVt NB3cqQ3oiw2KORCHpn5W7o61Hrl/jm+p+hMM6yHPDsOUaE9dsvaZdgqA2Vni4k+ODcqPvA/Rq MLn8vfAKnavenUev473Ml7jrxj/CdYjLX1WIZugHnPeFC62XWxZBBFVy7i4kAIZEuyTPaNqEJ NLqQgM5t09p1TrU/qLcIIKKLLQRL2NoBvS2E/GbuCqtOEN0Xz+B/ZVV2SJvC6ZXsho6baO1/c Inp2YUUkGYy0rWOGJgpPQyu5a4WdmtkzJHy+o0cJv1RYVHuLka5n+tBdhlLF1B0U0uM0kTvJR UxrbtwYEJPufwmQDYIVThX6OGmBwmC2P4lAB1RmJVKqUv3hy6n83mAqnbtEPNWTtoDsh8wm+/ j0kqGRYtQD0zc4J+IuV48G7LRm/FZtzdgTB04Z8nLQm/7JAmLFUEL4ROnyxsY0TaCIfoyth51 JbGZ+ope/nEBNJJIUG7Q==
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 13:19:20 -0000

Just to capture what Bob and I tried to convey:

There may be a way to extend TSOpt in a backwards-compatible way,
without increasing the size of the option during the SYN handshake, nor
with the need to have both 7323 TSopt and ETSopt present simultaneously
in the SYN and SYN/ACK.

Please have a look at
https://tools.ietf.org/html/draft-scheffenegger-tcpm-timestamp-negotiation=
-05

Which was the signaling / handshake part to

https://tools.ietf.org/html/draft-trammell-tcpm-timestamp-interval-01

As the latter part didn't get much traction (to signal the TS
granularity and even accuracy), these two didn't move forward.


As the SYN TSopt ecr value is currently unused, and the client knows the
initial value for TSval (which should be reflected in the SYN/ACK from
the server), you can use 32 bits there for a feature declaration and
negotiation.

The high level approach there was to use the unused TSecr field (should
be 0 for 7323/1323), and overlay (xor) in the SYN/ACK's TSopt ecr field
the client's TSopt val with the servers 32-bit negotiation field... As
the client can retain knowledge of the initial TSopt value (there should
be a randomized initial value), the client can extract the server side
options by xor'ing it's known initial value when the SYN/ACK is
received. Or fall back to default 7323 operation, if the result of the
XOR is zero...


The major issue would be that TSopt with a length !=3D 10 may be
removed/blocked by certain middleboxes, but a new TCP option has similar
deployment issues.

As Bob mentioned in another thread just now, TCP SYN option space is a
very scarce resource, and making efficient use of it should matter...


Other comments: Graph1 (page 4) has "TSval=3DX" rather than "ETSval=3DX".
Unless you plan to reuse the TSopt in an extensible way (with the same
codepoint, but differentiated by the length of the option), that may
need to be addressed to disambiguate things; You may also want to have
both ETSopt and TSopt in the SYN present, for backwards compatibility if
a separate codepoint is used.

If you want to make use of the timestamp-negotiation design, all of the
extra data shown on page 6 can be stuffed into the TSecr or ETSecr
field, alleviating the need of 4 additional SYN option bytes.

It could be used almost as a drop-in use of timestamp-negotiation, as
long as it can be guaranteed, that at least one bit is set in the 32bit
field (e.g. setting the reserved bit to 1).

EcrDelUnit: I think the codepoint assignment could be improved (but this
is really nit-picking; just that a broken implementation may send out
TSopt with len=3D12, but initialized-to-zero superflous bytes...
0: invalid
1: milliseconds [2^20 nanoseconds?]
2: microseconds [2^10 nanoseconds?]
3: reserved (nanoseconds)

MaxAckDel: 16 bits/microseconds gives a maximum of 65.5 ms; some (older)
stacks are using values as high as 200ms; If such a stack is updated, it
is a good reason to reduce the DelAck value used in that stack, so this
range appears properly selected (FreeBSD recently reduced from 100ms to
40ms).



Best regards,
    Richard


Am 14.11.2020 um 20:35 schrieb Yuchung Cheng:
> On Sat, Nov 14, 2020 at 8:45 AM Kevin Yang
>  wrote:
>>
>> Thank you Yoshi, very good questions.
>>
>> 1: Our goal is to substitute the TSopt. So we design this new ETSopt no=
t to depend on TSopt.
>>      I guess this also answers your question 4, the intent is for the s=
tandard track.
>>
>> 2: I think we should only update TS.recent when we assume the incoming =
segment is valid.
>>      Otherwise, if the incoming is a real old one, we discard it but up=
date TS.recent;
>>      then a new segment arrives, we may reject this new seg because TS.=
recent > seg.TSval.
>>
>> 3: Your understanding is correct, and in the draft we are not suggestin=
g to use NetworkRTT
>>      measurements for RTO calculation.
>>
>> 4: answered by 1.
> We'll clarify better in the upcoming draft revision: the draft
> intends to transition from exp-id to  an official option kind number
> from IANA upon becoming
> an RFC. But we're following the advice of
> https://tools.ietf.org/html/rfc6994 to allow larger scale experiments
> for hosts not completely under control of a single entity/network. For
> example guest VM TCP communications inside Cloud Data-center networks.
>
>
>>
>> Kevin
>>
>> On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida  =
wrote:
>>>
>>> Hi,
>>> Thanks for preparing the draft. I basically like the idea in the docs =
and have several comments.
>>>
>>> 1: I am wondering if we really need to have 32 bits TSval and TSecr in=
 ETS option in SYN when TSopt is also sent?
>>>      For example, If TSopt has 1 ms granularity, I think ETS option wi=
ll need to carry only small fractions of timestamp.
>>>
>>> 2: "To prevent a false positive PAWS rejection of a valid segment, an =
ETS receiver MUST skip the PAWS check"
>>>      -> This is one possible approach, but just skipping PAWS doesn't =
sound very good to me.
>>>           How about discarding the segment while updating TS.recent as=
 an alternative? If the segment is not an old one, it will be retransmitte=
d and accepted.
>>>           I think this is a bit more conservative.
>>>
>>> 3: It is not very clear for me how measuring NetworkRTT can contribute=
 to improving RTO.
>>>      Or, are there some other ways to utilize NetworkRTT?
>>>
>>> 4: Just out of curiosity, is intended status standard track or experim=
ental? ExID is usually used for exp or info docs.
>>>
>>> Thanks,
>>> --
>>> Yoshi
>>>
>>> On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng  wrote:
>>>>
>>>> Hi tcpm & iccrg,
>>>>
>>>> We are proposing a new TCP timestamp option to measure the network de=
lay more precisely (e.g. excluding delayed ACK effects) for congestion con=
trol, e.g. Swift published SIGCOMM 2020. The precision of the measurements=
 can potentially be further enhanced by NIC hardware timestamps. We are wo=
rking on a reference implementation for Linux as well.
>>>>
>>>> We'll also present it in the upcoming tcpm meeting. Feedbacks are ver=
y welcome.
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: 
>>>> Date: Mon, Nov 2, 2020 at 3:29 PM
>>>> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
>>>> To: Eric Dumazet , Yuchung Cheng , Neal Cardwell , Kevin Yang (Yudong) 
>>>>
>>>>
>>>>
>>>> A new version of I-D, draft-yang-tcpm-ets-00.txt
>>>> has been successfully submitted by Kevin (Yudong) Yang and posted to =
the
>>>> IETF repository.
>>>>
>>>> Name:           draft-yang-tcpm-ets
>>>> Revision:       00
>>>> Title:          TCP ETS: Extensible Timestamp Options
>>>> Document date:  2020-11-02
>>>> Group:          Individual Submission
>>>> Pages:          13
>>>> URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-0=
0.txt
>>>> Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/
>>>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcpm=
-ets
>>>> Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00
>>>>
>>>>
>>>> Abstract:
>>>>     This document presents ETS: an Extensible TimeStamps option for T=
CP.
>>>>     It allows hosts to use microseconds as the unit for timestamps to
>>>>     improve the precision of timestamps, and advertise the maximum AC=
K
>>>>     delay for its own delayed ACK mechanism.  Furthermore, it extends=
 the
>>>>     information provided in the [RFC7323] TCP Timestamps Option by
>>>>     including the receiver delay in the TSecr echoing, so that the
>>>>     receiver of the ACK is able to more accurately estimate the porti=
on
>>>>     of the RTT that resulted from time traveling through the network.
>>>>     The ETS option format is extensible, so that future extensions ca=
n
>>>>     add further information without the overhead of extra TCP option =
kind
>>>>     and length fields.
>>>>
>>>>
>>>>
>>>>
>>>> Please note that it may take a couple of minutes from the time of sub=
mission
>>>> until the htmlized version and diff are available at tools.ietf.org.
>>>>
>>>> The IETF Secretariat
>>>>
>>>>
>>>> _______________________________________________
>>>> tcpm mailing list
>>>> tcpm@ietf.org
>>>> https://www.ietf.org/mailman/listinfo/tcpm
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>


From nobody Tue Nov 17 06:25:16 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CC0913A1272; Tue, 17 Nov 2020 06:25:14 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BnmPzpPvtCWK; Tue, 17 Nov 2020 06:25:10 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C951B3A13BD; Tue, 17 Nov 2020 06:25:09 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 7883825A18; Tue, 17 Nov 2020 15:25:07 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1605623107; bh=0xGFMVes45mbbGXOpOGjSczte7givVBBlJCE/LipgRI=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=IgS44Iyo4er1UdNq+MA/v3JbXk+JLD+1tK1hx3ThuJjs40ZnZnRYhEoENAP4H/1F5 5ThrfliNoDBjzG3aNB1EDXSfarhXBKJYWD/BNG2129yeu3W+Q2MvNMKsu2vxwtL9XM BOPvO6LNW6HhzlHorBxz5Iu2dVsLUMF+v1Ac4Dv0=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s4ihcLh0Cx17; Tue, 17 Nov 2020 15:25:04 +0100 (CET)
Received: from rznt8201.rznt.rzdir.fht-esslingen.de (rznt8201.hs-esslingen.de [134.108.48.164]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Tue, 17 Nov 2020 15:25:04 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8201.rznt.rzdir.fht-esslingen.de (134.108.48.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Tue, 17 Nov 2020 15:25:04 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Tue, 17 Nov 2020 15:25:04 +0100
From: "Scharf, Michael" 
To: Bob Briscoe , Yoshifumi Nishida 
CC: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
Thread-Topic: AccECN field order
Thread-Index: AQHWvDjFKAN34ufmsUi0uesNI2lMmanK+nPQgAC8GACAADPpcIAARE2AgAAktJA=
Date: Tue, 17 Nov 2020 14:25:03 +0000
Message-ID: <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de> <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net>
In-Reply-To: <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: multipart/alternative; boundary="_000_5c3c4661887c43529b35bd5f47d10c2bhsesslingende_"
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 14:25:15 -0000

--_000_5c3c4661887c43529b35bd5f47d10c2bhsesslingende_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

SW5saW5lIHNvbWUgY29tbWVudHMgW21zXS4NCg0KR2l2ZW4gdGhhdCBJIGRvbuKAmXQgYXNrIGZv
ciBhbnkgdGV4dCBjaGFuY2VzLCBJ4oCZbGwgbm90IGZvbGxvdy11cCBmb3JldmVyIGluIHRoaXMg
ZGlzY3Vzc2lvbi4gSSBkb27igJl0IG93biBydW5uaW5nIGNvZGUgYW5kIGEgbG90IG9mIGVuY29k
aW5nIGRldGFpbHMgbWFpbmx5IGFmZmVjdCBhbiBpbXBsZW1lbnRhdGlvbi4NCg0KSU1ITyB0aGUg
cmVzdCBvZiB0aGUgV0cgaGFzIHRvIHRoaW5rIGFib3V0IHRoZSBvcHRpb24gZGVzaWduIG1vcmUg
dGhhbiBtZS4gU28sIHRoaXMgaXMgbW9zdGx5IGFib3V0IHRyaWdnZXJpbmcgYSBkaXNjdXNzaW9u
4oCmDQoNCkZyb206IEJvYiBCcmlzY29lIDxpZXRmQGJvYmJyaXNjb2UubmV0Pg0KU2VudDogVHVl
c2RheSwgTm92ZW1iZXIgMTcsIDIwMjAgMToyMCBQTQ0KVG86IFNjaGFyZiwgTWljaGFlbCA8TWlj
aGFlbC5TY2hhcmZAaHMtZXNzbGluZ2VuLmRlPjsgWW9zaGlmdW1pIE5pc2hpZGEgPG5zZC5pZXRm
QGdtYWlsLmNvbT4NCkNjOiBNaWNoYWVsIFR1ZXhlbiA8dHVleGVuQGZoLW11ZW5zdGVyLmRlPjsg
dGNwbS1jaGFpcnNAaWV0Zi5vcmc7IE1pcmphIEt1ZWhsZXdpbmQgPGlldGZAa3VlaGxld2luZC5u
ZXQ+OyBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxSaWNoYXJkLlNjaGVmZmVuZWdnZXJAbmV0YXBw
LmNvbT47IHRjcG0gSUVURiBsaXN0IDx0Y3BtQGlldGYub3JnPg0KU3ViamVjdDogUmU6IEFjY0VD
TiBmaWVsZCBvcmRlcg0KDQpNaWNoYWVsLA0KT24gMTcvMTEvMjAyMCAwODoxMCwgU2NoYXJmLCBN
aWNoYWVsIHdyb3RlOg0KQm9iLA0KDQpJIGFtIGZpbmUgd2l0aCB0aGUgdHdvIG9wdGlvbiBraW5k
cyBwcm9wb3NlZCBpbiAtMTMsIGJ1dCBJIGRvbuKAmXQgYnV5IHlvdXIgYXJndW1lbnRzIHdoeSB0
aGlzIGVuY29kaW5nIGlzIGJldHRlciB0aGFuIG90aGVycy4NCg0KTW9zdCBpbXBvcnRhbnRseSwg
SSBkb27igJl0IHRoaW5rIHRoYXQgeW91ciDigJxmb3J3YXJkIGNvbXBhdGliaWxpdHnigJ0gYXJn
dW1lbnQgaXMgYSB2ZXJ5IGNvbXBlbGxpbmcgcmVhc29uIGZvciB0d28gY29kZXBvaW50cy4gQWxs
IHByb3Bvc2FscyBJIGFtIGF3YXJlIG9mIGhhdmUgdGhlaXIgcHJvcyBhbmQgY29ucy4gQW5kIEkg
YW0gbm90IGF3YXJlIG9mIGEgcmVhbGx5IGNvbXByZWhlbnNpdmUgZGlzY3Vzc2lvbi4gQXQgbGVh
c3QgbmVpdGhlciB0aGUgZS1tYWlsIGJlbG93IG5vciB0b2RheeKAmXMgbWVldGluZyBkaXNjdXNz
IGFsbCB0cmFkZW9mZnMgSSB3b3VsZCBpbnZlc3RpZ2F0ZS4NCg0KW0JCXSBZb3UncmUgcmlnaHQg
LSB0aGUgZGlzY3Vzc2lvbiBiZWxvdyBpcyB1c2VmdWwgYW5kIG5lY2Vzc2FyeS4NCg0KDQoNCkZv
ciBpbnN0YW5jZSwgdGhlIGJlbmVmaXQgb2YgdGhlIGxlbmd0aCBlbmNvZGluZyB2YXJpYW50IHdv
dWxkIGJlIHRvIGNvbnN1bWUgb25seSBvbmUgVENQIG9wdGlvbiBjb2RlcG9pbnQgbm93LiBJZiB3
ZSBkZWNpZGVkIGxhdGVyIHRvIHRoYXQgYSBmbGFnIGJ5dGUgaXMgbmVlZGVkIChzYXksIGF0IHRo
ZSBlbmQpLCBhIGZvbGxvdy11cCBwcm9wb3NlZCBzdGFuZGFyZCBjb3VsZCBzcGVjaWZ5IGFub3Ro
ZXIgb3B0aW9uIGZvcm1hdCB3aXRoIHRoYXQgZmxhZyBieXRlLCB1c2luZyBhIHNlY29uZCBjb2Rl
cG9pbnQuDQoNCltCQl0NCiogVGhlIG5ldyBjb2RlcG9pbnQgd291bGQgcmVxdWlyZSB5ZWFycyB0
byBkZXBsb3kgKGluY2x1ZGluZyBpbiBwcm94aWVzKSBiZWZvcmUgaXQgd29ya3MgZTJlLg0KDQpb
bXNdIFRoYXQgbWF5IGRlcGVuZCAqYSBsb3QqIG9uIHRoZSBlbnZpcm9ubWVudC4gSW4gZW52aXJv
bm1lbnRzIHdoZXJlIEVDTiBjYW4gZ2V0IGRlcGxveWVkIHRvZGF5LCBzYXkgaW4gZGF0YWNlbnRl
cnMsIHJvbGxpbmcgb3V0IGEgbmV3IGtlcm5lbCB2ZXJzaW9uIG1heSBwZXJoYXBzIG5vdCBiZSB0
aGUgYmlnZ2VzdCBvcGVyYXRpb25hbCBjaGFsbGVuZ2UuIEFsc28sIHByb3hpZXMgbWF5IG5vdCBi
ZSBhIHNpZ25pZmljYW50IHByb2JsZW0gaW4gc29tZSBvZiB0aG9zZSBjYXNlcy4gRGVwbG95bWVu
dCByb2FkbWFwcyBhcmUgdmVyeSBoYXJkIHRvIHByZWRpY3QuDQoNCiogV2hlcmVhcyB0aGUgJ2Zv
cndhcmQgY29tcGF0aWJpbGl0eScgYXBwcm9hY2ggdHJpZXMgdG8gZW5zdXJlIHRoYXQgQWNjRUNO
IGltcGxlbWVudGF0aW9ucyB0aGF0IGFyZSBiZWluZyBidWlsdCB0b2RheSB3aWxsIGJlIHJlYWR5
IHRvIGludGVyd29yayB3aXRoIGEgZnV0dXJlIGV4dGVuc2lvbi4NCiogSW4gY29udHJhc3QsIGlm
IHlvdXIgaHlwb3RoZXRpY2FsIG5ldyBBY2NFQ04gY29kZXBvaW50IGRpZCBuZWVkIHRvIGJlIGRl
cGxveWVkIGxhdGVyLCBpdCB3b3VsZCBuZWVkIHRvIGluY2x1ZGUgYSBmYWxsLWJhY2sgdG8gdGhl
IGN1cnJlbnQgQWNjRUNOIG9wdGlvbiB3aGVuIG9uZSBwZWVyIGRvZXNuJ3QgdW5kZXJzdGFuZCB0
aGUgbmV3IGNvZGVwb2ludC4NCg0KW21zXSBUaGlzIGlzIGEgcmVhc29uYWJsZSBwb2ludCwgYWxi
ZWl0IG5vdCBhIHBhcnRpY3VsYXJseSBzdHJvbmcgb25lLCBnaXZlbiB0aGF0IHRoZSBvcHRpb24g
aXMgbm90IHJlYWxseSByZXF1aXJlZCBmb3IgYmFzaWMgaW50ZXJvcGVyYWJpbGl0eS4gQWxzbywg
YSBoeXBvdGhldGljYWwgZm9sbG93LXVwIHN0YW5kYXJkIGNvdWxkIGluY2x1ZGUgYSB3YXkgdG8g
ZXhwbGljaXRseSByZXF1ZXN0IHRoZSBvcHRpb27igKYNCg0KKiBGYWxsLWJhY2sgd291bGQgbmVl
ZCB0byBhdm9pZCBhIHNlY29uZCByb3VuZCBvZiBoYW5kc2hha2UgKGdpdmVuIHRoaXMgaXMgZm9y
IGxvdyBsYXRlbmN5KSwgd2hpY2ggd291bGQgb25seSB3b3JrIHJlbGlhYmx5IHdpdGggYW4gb3B0
aW9uIG9uIHRoZSBTWU4gKGJlY2F1c2UgdGhlIDNyZCBBQ0sgaXMgdW5yZWxpYWJsZSkNCg0KW21z
XSBJIGNhbuKAmXQgZm9sbG93LiBJZiBhbiBlbmRwb2ludCBpcyB1bmhhcHB5IHdpdGggZ2V0dGlu
ZyBBY2NFQ04gb3B0aW9uIOKAnEHigJ0gZnJvbSBhIHBlZXIsIEkgY291bGQgc2VuZCBhIOKAnHBs
ZWFzZSBzZW5kIG1lIHRoZSBiZXR0ZXIgQWNjRUNOIG9wdGlvbiBC4oCdIGFmdGVyIHRoZSAzLVdI
Uy4gSXQgY291bGQgYmUgYW4gdXBncmFkZSwgbm90IGEgZmFsbGJhY2suIEFsc28sIGF0IHRoaXMg
cG9pbnQgd2UgZG9u4oCZdCBrbm93IHdoYXQgb3B0aW9uIEIgd291bGQgYmUgYW5kIGluIHdoaWNo
IGNvbnRleHQgaXQgd291bGQgYmUgbmVlZGVkLiBNYXliZSBvdGhlciBwcm90b2NvbCBmZWF0dXJl
cyB3b3VsZCByZXF1aXJlIG5lZ290aWF0aW9uLCB0b28uIElNSE8gaXQgaXMgYSBzb2x2YWJsZSBw
cm9ibGVtLg0KDQoqIEFjY0VDTiBoYXMgc28gZmFyIGF2b2lkZWQgdXNpbmcgYW55IFNZTiBvcHRp
b24gc3BhY2UsIHdoaWNoIGlzIGltcG9ydGFudCBnaXZlbiB0aGF0IGlzIHRoZSBzY2FyY2VzdCBy
ZXNvdXJjZS4NCg0KW21zXSBZZXAsIGJ1dCBhcyBsb25nIGFzIG5vIHByb3Bvc2FsIG5lZWRzIFNZ
TiBvcHRpb25zLCB0aGlzIGlzIG5vdGhpbmcgdGhhdCB3b3VsZCBtYWtlIG9uZSBzb2x1dGlvbiBi
ZXR0ZXIgdGhhbiBhbm90aGVyLg0KDQpUaGVzZSBkYXlzLCBleHRlbmRpbmcgVENQIGlzIGxpa2Ug
cGxheWluZyBjaGVzcyAtIHdlIGhhdmUgdG8gdGhpbmsgbXVsdGlwbGUgbW92ZXMgYWhlYWQhDQoN
Clttc10gVGhlcmUgaXMgb25lIGZ1bmRhbWVudGFsIGRpZmZlcmVuY2UgYmV0d2VlbiBjaGVzcyBh
bmQgVENQOiBJbiBjaGVzcyB5b3UgcHJlY2lzZWx5IGtub3cgdGhlIGN1cnJlbnQgc2l0dWF0aW9u
IGFuZCBjYW4gcHJlZGljdCBhbGwgcG90ZW50aWFsIG1vdmVzIGluIGFkdmFuY2UgKGFsYmVpdCB0
aGVyZSBhcmUgYSBsb3QpLiBIb3dldmVyLCB3ZSBkb27igJl0IGtub3cgdGhlIGZ1dHVyZSBldm9s
dXRpb24gb2YgVENQLCBFQ04gYW5kIHRoZSBJbnRlcm5ldCBhcyBhIHdob2xlLCBub3IgZnV0dXJl
IHJlcXVpcmVtZW50cy4gSWYgd2UgdHJ5IHRvIGVuZ2luZWVyIGEgVENQIGV4dGVuc2lvbiBmb3Ig
c29tZSBwb3RlbnRpYWwgZnV0dXJlLCB3ZSBtYXkgZmlndXJlIG91dCBsYXRlciB0aGF0IHJlYWxp
dHkgaXMgZGlmZmVyZW50IGZyb20gd2hhdCB3ZSBkZXNpZ25lZCBmb3IuIEFzIG11Y2ggYXMgSSBh
cHByZWNpYXRlIHlvdXIgYXR0ZW1wdHMgdG8gcHJlZGljdCB0aGUgZnV0dXJlLCBpdCB3b3VsZCBi
ZSBhIGJpZyBzdXJwcmlzZSB0byBtZSBpZiB5b3UgY291bGQgY29ycmVjdGx5IHByZWRpY3Qgd2hh
dCBUQ1Agd2lsbCBiZSBpbiAxMCB5ZWFycyBmcm9tIG5vdy4gU28sIGluc3RlYWQgb2YgdHJ5aW5n
IHRvIHdyaXRlIHNwZWNpZmljYXRpb25zIHRoYXQgdGFrZSBpbnRvIGFjY291bnQgZnV0dXJlIHJl
cXVpcmVtZW50cywgd2hpY2ggYXJlIGluIGZhY3QganVzdCB1bmtub3duLCBpdCBpcyBtb3JlIGlt
cG9ydGFudCBmb3IgbWUgdG8gd3JpdGUgZWZmZWN0aXZlLCBlZmZpY2llbnQgYW5kIHNlY3VyZSBz
cGVjaWZpY2F0aW9ucyBmb3Igd2hhdCBpcyBjdXJyZW50bHkga25vd24gZm9yIHN1cmUuDQoNCltt
c10gSSAqZG9u4oCZdCoga25vdyBmb3Igc3VyZSB3aGV0aGVyIHRoZSBjdXJyZW50IEFjY0VDTiBv
cHRpb24gd2lsbCBiZSBuZWVkZWQsIGJ1dCBJIHNlZSBxdWl0ZSBzb21lIGV2aWRlbmNlIHRoYXQg
c3VwcG9ydCBvZiB0aGUgQWNjRUNOIG9wdGlvbihzKSBpcyBub3QgYSB0b3AgcHJpb3JpdHkgdGhl
c2UgZGF5cy4gVGhhdCBpcyBhbHNvIHNvbWV0aGluZyB0byB0aGluayBhYm91dCB3aGVuIGRlc2ln
bmluZyB0aGUgZGV0YWlscy4NCg0KSSBuZWVkIHRvIGxlYXJuIHRvIGFydGljdWxhdGUgb250byB0
aGUgbGlzdCBhbGwgbXkgcmVhc29ucyBmb3IgcmVqZWN0aW5nIGNlcnRhaW4gcGFydHMgb2YgdGhl
IGRlc2lnbiBzcGFjZS4gU29ycnkgYWJvdXQgdGhhdC4NCg0KWWVzLCBJIGFkbWl0LCB0aGUgY3Vy
cmVudCBwcm9wb3NhbCBpcyBhdCB0aGUgY29zdCBvZiBidXJuaW5nIHR3byBUQ1Agb3B0aW9uIGNv
ZGVwb2ludHMuIEhvd2V2ZXIsIHlvdSBzYWlkIHlvdXJzZWxmIHdoZW4geW91IHByb3Bvc2VkIHRo
aXMgYXBwcm9hY2gsIHRoYXQgdGhleSBhcmUgbm90IChjdXJyZW50bHkpIGEgc2NhcmNlIHJlc291
cmNlLiBDZXJ0YWlubHkgbm90IGFzIHNjYXJjZSBhcyBvcHRpb24gc3BhY2UuIEkganVzdCBjaGVj
a2VkOiByb3VnaGx5IDE1JSBvZiB0aGUgb3B0aW9uIGNvZGVwb2ludCBhbGxvY2F0aW9uIHNwYWNl
IGhhcyBiZWVuIHVzZWQgc28gZmFyLiBTbyB0aGF0J3Mgd2hlcmUgSSB3b3VsZCBzdWdnZXN0IHdl
IGNvbXByb21pc2UgZmlyc3QuDQoNClttc10gSSBtYWRlIGEgc2ltaWxhciBjYWxjdWxhdGlvbiBz
b21lIHllYXJzIGFnbyBhbmQgY2FtZSB0byBhIHNpbWlsYXIgY29uY2x1c2lvbi4gSW52ZXN0aW5n
IHR3byBjb2RlcG9pbnRzIGZvciBBY2NFQ04gaXMgc29tZXRoaW5nIHdlIGNhbiBwcm9iYWJseSBh
ZmZvcmQuIEJ1dCBub25ldGhlbGVzcyB3ZSBoYXZlIHRvIHJlYXNvbiBjYXJlZnVsbHkgd2h5IGl0
IGlzIHJlYWxseSB3b3J0aCB0aGUgZWZmb3J0Lg0KDQoNClJlZ2FyZGluZyB0aGUgcmVzdWx0aW5n
IGNvZGVwb2ludCBjb25zdW1wdGlvbiAodHdvIG9wdGlvbiBraW5kcykgYW5kIHRoZSBhZGRpdGlv
biBvZiBhIGZsYWcgYnl0ZSwgdGhpcyBhcHByb2FjaCB3b3VsZCBiYXNpY2FsbHkgZW5kIHVwIHdp
dGggdGhlIHNhbWUgcmVzdWx0IGFzIHRoZSBjdXJyZW50IHByb3Bvc2FsIGluIC0xMyBwbHVzIHNv
bWUgaHlwb3RoZXRpY2FsIGZ1dHVyZSBhZGRpdGlvbiBsZXZlcmFnaW5nIHRoZSBsZW5ndGggZmll
bGQgYmV5b25kIHRoZSBjdXJyZW50bHkga25vd24gdXNlLg0KDQpBbmQsIHllcywgdGhlIGxlbmd0
aCBlbmNvZGluZyB2YXJpYW50IG1heSBiZSBsZXNzIGZsZXhpYmxlIGFuZC9vciBjb25zdW1lIHNv
bWUgbW9yZSBiaXRzIGluIHNvbWUgY2FzZXMsIGJ1dCBvbiB0aGUgcGx1cyBzaWRlIGl0IG9ubHkg
bmVlZHMgb25lIGNvZGVwb2ludCBub3cg4oCTIGFuZCBnaXZlbiB0aGF0IGl0IGlzIGVudGlyZWx5
IHVuY2xlYXIgd2hldGhlciBhbnkgQWNjRUNOIG9wdGlvbiB3aWxsIGluZGVlZCBiZSB3aWRlbHkg
dXNlZCBpbiBmdXR1cmUsIHRoaXMgaXMgYSBiaWcgcGx1cy4gVGhlIGZhY3QgdGhhdCBpbXBsZW1l
bnRlcnMgYXJlIHF1aXRlIHNpbGVudCBvbiB0aGUgb3B0aW9uIGRlc2lnbiBpcyBub3QgYSBnb29k
IHNpZ24uDQoNCltCQl0gSGVyZSdzIGFub3RoZXIgcG9zc2liaWxpdHkgdGhlbi4NCg0KYSkgW0t8
THwgRUUxQiB8IEVDRUIgfCBFRTBCIF0NCmIpIFtLfEx8IEVFMUIgfCBFQ0VCIHwgRUUwQiB8Rl0g
ICAgaWYgRiA9IDBiWFhYWFhYWDANCmMpIFtLfEx8IEVFMEIgfCBFQ0VCIHwgRUUxQiB8Rl0gICAg
aWYgRiA9IDBiWFhYWFhYWDENCg0KV2UnZCBoYXZlIHRvIGRlY2lkZSB3aGljaCBvcmRlciBhbHRl
cm5hdGl2ZSAoYSkgdGFrZXMuIEkndmUgcGlja2VkIEVFMUIgZmlyc3Qgb25seSAnY29zIHlvdSdy
ZSByaWdodCB0aGF0IHRoZXJlIGlzIG5vdCBjdXJyZW50bHkgYXMgbXVjaCBwcmVzc3VyZSAob24g
dGhlIHB1YmxpYyBJbnRlcm5ldCkgZm9yIEFjY0VDTiB3aXRoIEVDVCgwKS4NCg0KPT1MZWdlbmQ9
PQ0KSzogICAgS2luZCAoMUIpDQpMOiAgICBMZW5ndGggKDFCKQ0KRXh4QjogRWNobyB4eCBieXRl
IGNvdW50ZXINCkY6ICAgIEZsYWdzICgxQikNCg0KQWx0ZXJuYXRpdmVzIGEpIGFuZCBiKSBhcmUg
ZXF1aXZhbGVudCwgYXQgbGVhc3Qgd2hpbGUgdGhlIG9ubHkgYWxsb2NhdGVkIGZsYWcgaXMgZm9y
IGZpZWxkIG9yZGVyLg0KSG93ZXZlciwgYWx0ZXJuYXRpdmUgYikgaXMgYXZhaWxhYmxlIGluIGNh
c2Ugb3RoZXIgZmxhZ3MgYXJlIGFsbG9jYXRlZCBpbiBmdXR1cmUNClRoZSBmbGFncyBieXRlIGNh
biBvbmx5IGJlIG9taXR0ZWQgaWYgKEwgJSAzID09IDIpLg0KVGhlIGZsYWdzIGJ5dGUgaXMgY29u
c2lkZXJlZCBwcmVzZW50IGlmIChMICUgMykgPT0gMC4NCg0KRm9yd2FyZCBjb21wYXRpYmlsaXR5
OiBPcHRpb25zIHdpdGggKEwgJSAzID09IDEpIE1VU1QgYmUgYXNzdW1lZCB0byBpbmNsdWRlIGEg
ZmxhZ3MgYnl0ZSwgYW5kIGN1cnJlbnQgaW1wbGVtZW50YXRpb25zIGlnbm9yZSB0aGUgbGFzdCBi
eXRlLg0KDQpUaGUgZmxhZ3MgYnl0ZSBpcyBvcHRpb25hbCB0byBpbXBsZW1lbnQgKGV2ZW4gaWYg
dGhlIEFjY0VDTiBvcHRpb24gaXMgaW1wbGVtZW50ZWQpDQpJZiB0aGUgc2VydmVyIGluY2x1ZGVz
IGEgZmxhZ3MgYnl0ZSBvbiB0aGUgU1lOL0FDSyBidXQgdGhlIGNsaWVudCBkb2VzIG5vdCBpbmNs
dWRlIG9uZSBvbiB0aGUgM3JkIEFDSyBvciB0aGUgZmlyc3QgZGF0YSBwYWNrZXQsIHRoZSBzZXJ2
ZXIgYXNzdW1lcyB0aGUgY2xpZW50IGRvZXMgbm90IGltcGxlbWVudCB0aGUgZmxhZ3MgYnl0ZSBh
bmQgdXNlcyBvbmx5IGFsdGVybmF0aXZlIGEpIGZvciB0aGUgcmVtYWluZGVyIG9mIHRoZSBjb25u
ZWN0aW9uLg0KDQoNClttc10gSSBhbSBub3Qgc3VyZSBpZiB3ZSBuZWVkIHRoYXQsIGJ1dCBJIHRh
a2UgdGhpcyBhcyBleGlzdGVuY2UgcHJvb2YgdGhhdCBvdGhlciBlbmNvZGluZ3MgYXJlIHBvc3Np
YmxlLiBQZXJzb25hbGx5LCBJIHRoaW5rIHRoYXQgb3RoZXIgc29sdXRpb25zIHdvdWxkIGJlIGRv
YWJsZSBpZiB5b3UgdG9vayBvbmUgYml0IGZyb20gb25lIG9yIG11bHRpcGxlIGNvdW50ZXJzIGZv
ciBmb3JtYXQgZW5jb2RpbmcuIEF0IGxlYXN0IGZvciBzb21lIG9mIHRoZSBieXRlIGNvdW50ZXJz
IEkgZG9u4oCZdCB0aGluayB0YWtpbmcgb25lIGJpdCBmb3Igb3RoZXIgcHVycG9zZXMgd291bGQg
cmVzdWx0IGluIGFuIEludGVybmV0IGNvbmdlc3Rpb24gY29sbGFwc2XigKYgQnV0LCBJIGhhdmUg
YWxyZWFkeSBzYWlkIHRoYXQgdGhlIC0xMyBwcm9wb3NhbCB3b3JrcyBmb3IgbWUsIHNvIEnigJls
bCBub3QgdHJ5IHRvIGNvbWUgdXAgd2l0aCBhbm90aGVyIHZhcmlhbnQgbXlzZWxmLg0KDQpUbyBt
ZSwgb25lIHBvdGVudGlhbCBkaWZmZXJlbmNlIGJldHdlZW4gdHdvIHByb3Bvc2FscyB3b3VsZCBi
ZSBpbmNyZW1lbnRhbCBkZXBsb3ltZW50LiBUaGUgcHJvcG9zYWwgaW4gLTEzIG9ubHkgaGFzIGFu
IGFkdmFudGFnZSBpZiBtaWRkbGVib3hlcyBzdWNoIGFzIGZpcmV3YWxscyB3aWxsIGluZGVlZCBw
YXNzIFRDUCBvcHRpb25zIHdpdGggYSBmb3JtYXQgdGhhdCBjb250YWlucyBjb250ZW50IGJleW9u
ZCB0aGUgKGZpcnN0KSBBY2N1cmF0ZSBFQ04gc3RhbmRhcmQgKGkuZS4sIGN1cnJlbnRseSB1bnVz
ZWQgbGVuZ3RoIHZhbHVlcykuIElNSE8gaXQgaXMgdG9vIGVhcmx5IHRvIGtub3cgd2hldGhlciBm
aXJld2FsbHMgd291bGQgaW5kZWVkIGFsbG93IHRoaXMgaW4gZnV0dXJlLg0KDQpGcm9tIGEgc2Vj
dXJpdHkgcGVyc3BlY3RpdmUsIGl0IGlzIG5vdCBjbGVhciB0byBtZSB3aGV0aGVyIGFsbG93aW5n
IGFyYml0cmFyeSB1bnNwZWNpZmllZCBieXRlcyBpbiBhIFRDUCBvcHRpb24gaXMgYSBnb29kIGlk
ZWEgKmF0IGFsbCouIEl0IHdpbGwgYmUgaW50ZXJlc3RpbmcgdG8gaGVhciB0aGUgb3BpbmlvbiBm
cm9tIFNFQyBhcmVhIG9uIHRoYXQuIFBlcnNvbmFsbHksIEkgYW0gbm90IGNvbnZpbmNlZCB0aGF0
IHRoaXMgcmVhbGx5IG1ha2VzIHNlbnNlLCBidXQgSSBteSBjb25jZXJucyBhcmUgbm90IHN0cm9u
ZyBlbm91Z2ggdG8gZm9ybWFsbHkgcHVzaCBiYWNrLiBJ4oCZbGwgbGVhdmUgaXQgdG8gb3RoZXJz
IHRvIHRoaW5rIGFib3V0IHdoZXRoZXIgdGhpcyBpcyBhIGJ1ZyBvciBhIGZlYXR1cmUuDQoNCltC
Ql0gV2VsbCwgbGV0J3MgZmlyc3QgdHJ5IHRvIGRlYWwgd2l0aCBzZWN1cml0eSBvdXJzZWx2ZXM6
DQoqIE9jdGV0cyB0aGF0IGFyZSBleHBsaWNpdGx5IGRlY2xhcmVkIGFzIHBhcnQgb2YgYW4gb3B0
aW9uIGNhbm5vdCBiZSB1c2VkIGZvciBhIGJ1ZmZlciBvdmVyZmxvdyBhdHRhY2suIEkgZG9uJ3Qg
cmVhbGx5IG5lZWQgdG8sIGJ1dCBJIGNvdWxkIGFkZCB0aGUgZm9sbG93aW5nIHRleHQgdG8gdGhl
IGZvcndhcmQgY29tcGF0aWJpbGl0eSB3b3JkaW5nOg0KICAgIEEgcmVjZWl2ZXIgY29uc2lkZXJz
IG9jdGV0cyBiZXlvbmQgdGhvc2UgaXQgdXNlcywgYnV0IHdpdGhpbiB0aGUgc3BlY2lmaWVkIGxl
bmd0aCwgYXMgaWYgdGhleSBhcmUgcGFkZGluZy4NCiogQW5kIHN1Y2ggb2N0ZXRzIGNhbm5vdCBi
ZSBhbnkgZGlmZmVyZW50IGZyb20gdGhlIGN1cnJlbnQgYWJpbGl0eSBvZiBhIHNlbmRlciB0byBh
ZGQgcGFkZGluZy4gU28gdGhlcmUncyBubyBuZXcgYXR0YWNrIHBvc3NpYmxlIGhlcmUuDQoqIEFu
ZCB0aGVyZSdzIG5vIG5lZWQgdG8gc3BlY2lmeSBhIG1heCBsZW5ndGggZm9yIGFueSBBY2NFQ04g
T3B0aW9uLCB3aGljaCB3b3VsZCBqdXN0IHVubmVjZXNzYXJpbHkgbGltaXQgZmxleGliaWxpdHku
DQoNClttc10gSSBoYXZlIG1vcmUgdGhvdWdodCBhYm91dCB0aGUgY292ZXJ0IGNoYW5uZWwgdGhh
dCBzb21lIHVuc3BlY2lmaWVkIGJ5dGVzIGluIGEgc3RhbmRhcmRzIHRyYWNrIFRDUCBvcHRpb24g
Y291bGQgb2ZmZXLigKYgSW4gcGFydGljdWxhciBpZiB0aGUgQWNjRUNOIG9wdGlvbiBnZXRzIHdp
ZGVseSBkZXBsb3llZCBhbmQgdXNlZCwgd291bGQgc29tZSBtYWxpY2lvdXMgdXNlcnMgZmluZCB0
aG9zZSBieXRlcyB1c2VmdWw/IEZvciB3aGF0PyBGb3IgaW5zdGFuY2UsIHdoYXQgaW4gdGhlIEFj
Y0VDTiBwcm90b2NvbCBkZXNpZ24gcHJldmVudHMgdXNlIG9mIHRoZXNlIOKAnHBhZGRpbmfigJ0g
Ynl0ZXMgYXMgc3VwZXItY29va2llPyBUbyBtZSwgaXQgaXMgbXVjaCBtb3JlIGltcG9ydGFudCB0
byB0aGluayBhYm91dCBhYnVzZSBvZiB3aGF0ZXZlciB3ZSBzdGFuZGFyZGl6ZSAqbm93KiwgaW5z
dGVhZCBvZiBlbmdpbmVlcmluZyBmb3Igc29tZSBlbnRpcmVseSB1bmtub3duIGZ1dHVyZS4gSSBh
bSBub3QgYSBzZWN1cml0eSBleHBlcnQsIGJ1dCBJIHN1Z2dlc3QgYSBjYXJlZnVsIGFuYWx5c2lz
IG9mIGFueSBzZWN1cml0eSBpbXBsaWNhdGlvbnMgb2YgdGhlIEFjY0VDTiBvcHRpb24uIFdlIHNl
bGRvbWx5IHNwZWNpZnkgbmV3IFRDUCBvcHRpb25zIGFuZCBtYWxpY2lvdXMgdXNlcnMgd2lsbCBj
ZXJ0YWlubHkgbG9vayBmb3Ig4oCcb3Bwb3J0dW5pdGllc+KAnSBpbiB0aGUgc3BlYy4gU28gZmFy
LCBJIGRvbuKAmXQga25vdyB3aGV0aGVyIHRoYXQgd2hvbGUgaWRlYSBvZiBmb3J3YXJkIGNvbXBh
dGliaWxpdHkgaXMgYSBmZWF0dXJlIG9yIGEgYnVnLg0KDQpNaWNoYWVsDQoNCk1heWJlIG9uZSBs
ZXNzb24gbGVhcm50IGlzIHRoYXQgdGhlIGRvY3VtZW50IGNvdWxkIGhhdmUgYSBub24tbm9ybWF0
aXZlIGFwcGVuZGl4IHRoYXQgZXhwbGFpbnMgdGhlIHJhdGlvbmFsZSBmb3IgdGhlIGZpbmFsbHkg
cGlja2VkIFRDUCBvcHRpb24gZW5jb2RpbmcuIFRoYXQgbWF5IGFsc28gaGVscCBpZiB0aGVyZSBh
cmUgZnVydGhlciBxdWVzdGlvbnMgd2hldGhlciB0d28gY29kZXBvaW50cyBhcmUgcmVhbGx5IHJl
cXVpcmVkLCBlLmcuIGJ5IHRoZSBJRVNHIChpZiB0d28gY29kZXBvaW50cyBhcmUgc3RpbGwgdGhl
IGRlc2lnbiBhZnRlciBXR0xDKS4gQXQgbGVhc3QgZm9yIHBhc3QgVENQIG9wdGlvbiBjb2RlcG9p
bnQgYWxsb2NhdGlvbnMgSSByZWNhbGwgc29tZSBkaXNjdXNzaW9ucyBsYXRlIGluIHRoZSBJRVRG
IHByb2Nlc3MuIEluIHRob3NlIHBhc3QgY2FzZXMsIGdvb2QgYXJndW1lbnRzIGluIGFuIGFwcGVu
ZGl4IGFuZCBydW5uaW5nIGNvZGUgaGFzIGhlbHBlZCBhIGxvdC4NCg0KW0JCXSBJIGNhbiBkbyB0
aGF0LiBBcHB4IEIgaXMgYWxyZWFkeSBzaW1pbGFyLCBnaXZpbmcgRGVzaWduIFJhdGlvbmFsZSBm
b3IgdGhlIEFDRSBmaWVsZC4NCg0KDQpCb2INCg0KDQoNCk1pY2hhZWwNCg0KDQoNCkZyb206IEJv
YiBCcmlzY29lIDxpZXRmQGJvYmJyaXNjb2UubmV0PjxtYWlsdG86aWV0ZkBib2JicmlzY29lLm5l
dD4NClNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDE3LCAyMDIwIDY6MTAgQU0NClRvOiBTY2hhcmYs
IE1pY2hhZWwgPE1pY2hhZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT48bWFpbHRvOk1pY2hhZWwu
U2NoYXJmQGhzLWVzc2xpbmdlbi5kZT47IFlvc2hpZnVtaSBOaXNoaWRhIDxuc2QuaWV0ZkBnbWFp
bC5jb20+PG1haWx0bzpuc2QuaWV0ZkBnbWFpbC5jb20+DQpDYzogTWljaGFlbCBUdWV4ZW4gPHR1
ZXhlbkBmaC1tdWVuc3Rlci5kZT48bWFpbHRvOnR1ZXhlbkBmaC1tdWVuc3Rlci5kZT47IHRjcG0t
Y2hhaXJzQGlldGYub3JnPG1haWx0bzp0Y3BtLWNoYWlyc0BpZXRmLm9yZz47IE1pcmphIEt1ZWhs
ZXdpbmQgPGlldGZAa3VlaGxld2luZC5uZXQ+PG1haWx0bzppZXRmQGt1ZWhsZXdpbmQubmV0Pjsg
U2NoZWZmZW5lZ2dlciwgUmljaGFyZCA8UmljaGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20+
PG1haWx0bzpSaWNoYXJkLlNjaGVmZmVuZWdnZXJAbmV0YXBwLmNvbT47IHRjcG0gSUVURiBsaXN0
IDx0Y3BtQGlldGYub3JnPjxtYWlsdG86dGNwbUBpZXRmLm9yZz4NClN1YmplY3Q6IFJlOiBBY2NF
Q04gZmllbGQgb3JkZXINCg0KTWljaGFlbCwNCk9uIDE2LzExLzIwMjAgMTc6MzYsIFNjaGFyZiwg
TWljaGFlbCB3cm90ZToNCkJvYiwNCg0KT25lIHByb3Bvc2FsIHVzaW5nIHRoZSBsZW5ndGggZmll
bGQgd2l0aCAqb25lIG9wdGlvbiBjb2RlcG9pbnQgb25seSogaXMgZGV0YWlsZWQgaW46IGh0dHBz
Oi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1IwblJmaEhvY1g4eXZU
dnBDNEJOTW8vDQoNCkl0IGlzIHRoZSB0aGlyZCBvcHRpb24gbWVudGlvbmVkIGluIHRoaXMgZS1t
YWlsLiBPbmUgZXhhbXBsZSB3b3VsZCBiZSB0byB1c2Ugb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNS84
LzExIGZvciBvbmUgZW5jb2RpbmcgdHlwZSBhbmQgb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNi85LzEy
IGZvciB0aGUgb3RoZXIgZW5jb2RpbmcgdHlwZSAoaS5lLiwgb3JkZXIgb2YgZmllbGRzKS4gT3Ig
b25lIGNvdWxkIHVzZSBzb21lIG90aGVyIGNvbWJpbmF0aW9uIG9mIGxlbmd0aCB2YWx1ZXMg4oCT
IHRoZSBvbmx5IHJlcXVpcmVtZW50IGlzIHRoYXQgYSBjZXJ0YWluIHZhbHVlIGZvciB0aGUgb3B0
aW9uIGxlbmd0aCBpcyBvbmx5IHVzZWQgYnkgb25lIG9mIHRoZSBvcHRpb24gZm9ybWF0cy4gSW4g
dGhhdCBhcHByb2FjaCwgdGhlIHZhbHVlIG9mIHRoZSBsZW5ndGggZmllbGQgd291bGQgdGh1cyBk
aXJlY3RseSBkZXNjcmliZSB0aGUgZW5jb2Rpbmcgb2YgdGhlIG9wdGlvbi4gVW5sZXNzIEkgbWlz
cyBzb21ldGhpbmcsIHRoaXMgd291bGQgd29yayBhbmQgaXQgd291bGQganVzdCByZXF1aXJlIG9u
ZSBvcHRpb24gY29kZXBvaW50Lg0KDQpUaHVzLCBhbHRlcm5hdGl2ZXMgdG8gdHdvIG9wdGlvbiBj
b2RlcG9pbnRzIGV4aXN0IGFuZCBJIGhhdmUgZXhwbGFpbmVkIHRoZW0gb24gdGhlIGxpc3QgaW4g
TWFyY2ggMjAyMC4NCg0KT0ssIHNvcnJ5LCB5ZXMsIEkgcmVtZW1iZXIgdGhpcyBub3cuDQoNCkFz
IEkgd2lsbCBleHBsYWluIGluIHRoZSBBY2NFQ04gc3RhdHVzIHVwZGF0ZSB0YWxrIHRvZGF5IGlu
IHZpcnR1YWwgQmFuZ2tvaywgdGhlIGRyYWZ0IGhhcyBtYWRlIHByb3Zpc2lvbiBmb3IgZGlmZmVy
ZW50IGxlbmd0aCB2YWx1ZXMgdGhhbiA1LzgvMTEuIEl0IHNheXMgZXhpc3RpbmcgaW1wbGVtZW50
YXRpb25zIE1VU1QgYWNjZXB0IGxlbmd0aCB2YWx1ZXMgb3RoZXIgdGhhbiB0aG9zZSBjdXJyZW50
bHkgZGVmaW5lZC4gQnV0IHRoZW4gcmVhZCBpbiBhcyBtYW55IHdob2xlIDMtYnl0ZSBmaWVsZHMg
YXMgdGhleSBjYW4uDQoNClRoaXMgY2FuIGJlIHVzZWQgdG8gYWRkIGEgZmxhZ3MgYnl0ZSBvbiB0
aGUgZW5kIGluIGZ1dHVyZSwgZm9yIGV4dGVuc2liaWxpdHkuIE9yIGFueSBvdGhlciBmb3JtIG9m
IGV4dGVuc2liaWxpdHkgdGhlIFdHIG1pZ2h0IGRlY2lkZSBpbiB0aGUgZnV0dXJlLg0KDQpJIGtu
b3cgYSBmbGFncyBieXRlIGF0IHRoZSBlbmQgc2VlbXMgb2RkIGNvbXBhcmVkIHRvIGF0IHRoZSBi
ZWdpbm5pbmcuIEJ1dCAoaWYgZGVjaWRlZCBpdCdzIG5lZWRlZCBpbiBmdXR1cmUpIGl0J3MgcmVh
c29uYWJseSBlYXN5IHRvIGltcGxlbWVudCBieSByZWFkaW5nIHRoZSB3aG9sZSBvcHRpb24sIHRo
ZW4gcHJvY2Vzc2luZyB0aGUgbGFzdCBieXRlLCBiZWZvcmUgcmVhZGluZyB0aGUgcmVzdCBvZiB0
aGUgb3B0aW9uLg0KDQpJIGJlbGlldmUgeW91IHdpbGwgYWdyZWUgdGhhdCB0aGlzIGlzIGEgYmV0
dGVyIHdheSB0byB1dGlsaXplIGRpZmZlcmVudCBsZW5ndGhzLg0KDQpBbmQgdGhhbmsgeW91IGZv
ciByZXBlYXRlZGx5IGVtcGhhc2l6aW5nIHRoYXQgeW91J3JlIGhhcHB5IHdpdGggdGhlIDIta2lu
ZCBzY2hlbWUsIG9yIG90aGVyIGFsdGVybmF0aXZlcy4NCg0KDQpCb2INCg0KDQoNCg0KQW55d2F5
LCBJIGRvbuKAmXQgcmVhbGx5IGNhcmUgaG93IHRoZSBvcHRpb25zIGFyZSBlbmNvZGVkIGFzIGxv
bmcgYXMgdGhlIHJlY2VpdmVyIGRvZXNu4oCZdCBuZWVkIHBlci1jb25uZWN0aW9uIHN0YXRlIGZv
ciBkZWNvZGluZyBhIFRDUCBvcHRpb24uIFNvLCBwZXJzb25hbGx5LCBJIHdvdWxkIGJlIGZpbmUg
d2l0aCB1c2luZyBlLmcuIHRoZSBsZW5ndGggZmllbGQgYXMgZGVzY3JpYmVkIGluIG15IG9sZCBl
LW1haWwuIE9yIGFuIGFkZGl0aW9uYWwgZmxhZyBieXRlLiBBbmQgb25lIGNvdWxkIGNvbWUgdXAg
d2l0aCBmdXJ0aGVyIGVuY29kaW5ncywgZS5nLiwgYnkgdXNpbmcgb25lIG9yIGEgZmV3IGJpdHMg
YXMgYSBzaG9ydCDigJx0eXBl4oCdIGZpZWxkIGZvciBlYWNoIGNvdW50ZXIuIFRoaXMgaXMgYWxs
IGFib3V0IHByb3RvY29sIGVuZ2luZWVyaW5nLiBBbmQgYWxsIHRoZXNlIHZhcmlhbnRzIGhhdmUg
dGhlaXIgcHJvcyBhbmQgY29ucy4NCg0KSSBhbSBhbHNvIGZpbmUgd2l0aCB1c2luZyB0d28gb3B0
aW9uIGNvZGVwb2ludHMgYXMgc3BlY2lmaWVkIGluIC0xMzsgdGhpcyBpcyBwcm9iYWJseSB0aGUg
YXBwcm9hY2ggdGhhdCBjb25zdW1lcyB0aGUgbGVhc3QgbnVtYmVyIG9mIGJpdHMuDQoNCk1pY2hh
ZWwgKHcvbyBhbnkgaGF0KQ0KDQoNCg0KRnJvbTogQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2Nv
ZS5uZXQ+PG1haWx0bzppZXRmQGJvYmJyaXNjb2UubmV0Pg0KU2VudDogTW9uZGF5LCBOb3ZlbWJl
ciAxNiwgMjAyMCA1OjUyIFBNDQpUbzogWW9zaGlmdW1pIE5pc2hpZGEgPG5zZC5pZXRmQGdtYWls
LmNvbT48bWFpbHRvOm5zZC5pZXRmQGdtYWlsLmNvbT4NCkNjOiBTY2hhcmYsIE1pY2hhZWwgPE1p
Y2hhZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT48bWFpbHRvOk1pY2hhZWwuU2NoYXJmQGhzLWVz
c2xpbmdlbi5kZT47IE1pY2hhZWwgVHVleGVuIDx0dWV4ZW5AZmgtbXVlbnN0ZXIuZGU+PG1haWx0
bzp0dWV4ZW5AZmgtbXVlbnN0ZXIuZGU+OyB0Y3BtLWNoYWlyc0BpZXRmLm9yZzxtYWlsdG86dGNw
bS1jaGFpcnNAaWV0Zi5vcmc+OyBNaXJqYSBLdWVobGV3aW5kIDxpZXRmQGt1ZWhsZXdpbmQubmV0
PjxtYWlsdG86aWV0ZkBrdWVobGV3aW5kLm5ldD47IFNjaGVmZmVuZWdnZXIsIFJpY2hhcmQgPFJp
Y2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tPjxtYWlsdG86UmljaGFyZC5TY2hlZmZlbmVn
Z2VyQG5ldGFwcC5jb20+OyB0Y3BtIElFVEYgbGlzdCA8dGNwbUBpZXRmLm9yZz48bWFpbHRvOnRj
cG1AaWV0Zi5vcmc+DQpTdWJqZWN0OiBBY2NFQ04gZmllbGQgb3JkZXINCg0KWW9zaGksIChhZGRp
bmcgdGhlIHRjcG0gbGlzdCBpbiBjYykNCk9uIDA1LzExLzIwMjAgMDY6NTgsIFlvc2hpZnVtaSBO
aXNoaWRhIHdyb3RlOg0KSGkgQm9iLA0KDQpPbiBXZWQsIE5vdiA0LCAyMDIwIGF0IDM6MjkgUE0g
Qm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2NvZS5uZXQ8bWFpbHRvOmlldGZAYm9iYnJpc2NvZS5u
ZXQ+PiB3cm90ZToNCllvc2hpLA0KT24gMDQvMTEvMjAyMCAwNjo1MSwgWW9zaGlmdW1pIE5pc2hp
ZGEgd3JvdGU6DQpIaSwgZm9sa3MsDQoNCg0KDQoNCkluIG15IHVuZGVyc3RhbmRpbmcsIEknbSBu
b3Qgc3VyZSBpZiB3ZSBzZXR0bGVkIGRvd24gb24gdXNpbmcgdHdvIG9wdGlvbiBraW5kcyBvciBl
bmNvZGluZyBzY2hlbWVzIGZvciAyNGJpdHMgZmllbGRzIGluIGFjYyBlY24gZHJhZnQuDQpTbywg
SSB0aGluayB0aGVyZSdyZSBzdGlsbCBzb21ldGhpbmcgdG8gYmUgY2xhcmlmaWVkIGFuZCBob3Bl
IHRoaW5ncyB3aWxsIGJlIHNldHRsZWQgYXQgdGhlIG1lZXRpbmcuDQoNCltCQl0gSSBrbm93IGEg
V0cgY2FuIGNoYW5nZSBpdCdzIG1pbmQgYXQgYW55IHRpbWUuIEJ1dCBJJ2QgcmF0aGVyIHdlIGp1
c3QgY2xhcmlmaWVkIHdoYXQgYSBwcmV2aW91cyBkZWNpc2lvbiB3YXMsIHRvIGF2b2lkIHRoZSBu
ZWVkIHRvIGtlZXAgcmUtb3BlbmluZyBkaXNjdXNzaW9uIG9uIGEgcXVlc3Rpb24gdGhhdCBoYXZl
IGJlZW4gZGVjaWRlZCB0aGVuIGNoYW5nZWQgdGhyZWUgZGlmZmVyZW50IHdheXMgYWxyZWFkeS4N
Cg0KTXkgbWVtb3J5IGlzIG5vdCBzbyBnb29kIHRoZXNlIGRheXMuIEkgdHJ1c3RlZCB0aGF0IE1p
Y2hhZWwgUyByZW1lbWJlcmVkIHRoZSBkZWNpc2lvbiBjb3JyZWN0bHksIGFuZCBJIHNlZW0gdG8g
cmVtZW1iZXIgdGhhdCBkZWNpc2lvbiBiZWluZyBtYWRlLg0KSSd2ZSBqdXN0IGNoZWNrZWQgdGhl
IG1pbnV0ZXMgb2YgdGhlIGxhc3QgaW50ZXJpbToNCiAgICBodHRwczovL2RhdGF0cmFja2VyLmll
dGYub3JnL2RvYy9taW51dGVzLWludGVyaW0tMjAyMC10Y3BtLTAxLTIwMjAwNDI5MTYwMC8NCmFu
ZCB0aGV5IG1lbnRpb24gTWljaGFlbCdzIHByb3Bvc2FsIHRvIHVzZSB0d28ga2luZHMsIGJ1dCBk
b24ndCByZWNvcmQgYW55IGRlY2lzaW9uLg0KVGhlIGphYmJlciBsb2cgZ2l2ZXMgbm8gY2x1ZXMg
YWJvdXQgYW55IGRlY2lzaW9uLg0KDQpJIGNhbid0IGZpbmQgYW4gYXVkaW8gb3IgdmlkZW8gcmVj
b3JkaW5nLiBDYW4geW91IHBvaW50IG1lIGF0IG9uZT8NCg0KSSB0aG91Z2h0IHRoYXQgaXQncyBi
ZWNhdXNlIHRoZXJlIHdhcyBubyBjbGVhciBkZWNpc2lvbiBhdCB0aGUgbWVldGluZy4NCkJ1dCwg
eW91IGNhbiBjaGVjayBodHRwczovL3d3dy55b3V0dWJlLmNvbS93YXRjaD92PUtzQjBfTmI4LWtB
DQpQbGVhc2UgbGV0IHVzIGtub3cgaWYgeW91IGhhdmUgYW55IHF1ZXN0aW9ucyBvciBvcGluaW9u
cyB3aXRoIHJlZ2FyZCB0byB0aGlzLg0KDQpbQkJdIEkgY2hlY2tlZCB0aGUgWW91dHViZSBsaW5r
IHlvdSBzZW50IGJlbG93Lg0KDQpGaXJzdCBJIHRoaW5rIHdlJ3JlIGFncmVlZCBuby1vbmUgd2Fz
IGZpZ2h0aW5nIGZvciB1cyB0byBrZWVwIHRoZSBwcmV2aW91cyB3YXkgd2UgZGlkIHRoaXMgKHVz
aW5nIHRoZSBpbml0aWFsIHZhbHVlIG9mIHRoZSBmaWVsZCB0byBzZXQgdGhlIG9yZGVyIGZvciB0
aGUgY29ubmVjdGlvbikuDQpJbiBteSBwcmVzZW50YXRpb24gSSBzYWlkIHRoZXJlIHdhcyBzdHJv
bmcgcmVzaXN0YW5jZSBmcm9tIE1pY2hhZWwgdG8gZG8gaXQgYSBkaWZmZXJlbnQgd2F5Lg0KKGFs
c28sIG9mZmxpc3QsIHRoZSBjby1hdXRob3JzIGluY2x1ZGluZyBtZSBhbHNvIGRpZG4ndCBsaWtl
IHRoaXMgc28gbXVjaC4gQW5kIElscG8gc2FpZCBpdCBtYWRlIHRoZSBpbXBsZW1lbnRhdGlvbiBj
b21wbGV4LikNCg0KVGhlbiBjYW1lIHRoZSBxdWVzdGlvbiBvZiB3aGF0IHdlIGRvIGluc3RlYWQu
IFRoZXJlIHdlcmUgdGhyZWUgYWx0ZXJuYXRpdmUgcHJvcG9zYWxzOg0KYSkgdXNlIDIgb3B0aW9u
IGtpbmRzDQpiKSBhZGQgYSBmbGFncyBieXRlDQpjKSBzb21laG93IHVzZSB0aGUgbGVuZ3RoIGZp
ZWxkIG1heWJlDQoNCk1pY2hhZWwgcmFpc2VkIChjKSBpbiB0aGUgbWVldGluZyBhcyBhIHBvc3Np
YmlsaXR5LCBidXQgbm8tb25lIGNvdWxkIHRoaW5rIGhvdyB0byBkaXN0aW5ndWlzaCB0d28gb3B0
aW9ucyBvZiB0aGUgc2FtZSBsZW5ndGggYnV0IGEgZGlmZmVyZW50IGZpZWxkIG9yZGVyIHVzaW5n
IHRoZSBsZW5ndGggZmllbGQuIE1pY2hhZWwgc2FpZCBoZSdkIHBvc3QgYW55IGlkZWFzIHRvIHRo
ZSBsaXN0IGlmIGhlIGNvdWxkIHRoaW5rIG9mIGFueSwgYnV0IHRoYXQgZGlkbid0IGhhcHBlbi4N
Cg0KU28gd2UncmUgbGVmdCBjaG9vc2luZyBiZXR3ZWVuIChhKSBhbmQgKGIpLg0KSSBzYWlkIGlu
IHRoZSBtZWV0aW5nIChhbmQgb24gdGhlIGxpc3Qgd2hlbiBkaXNjdXNzaW5nIHdpdGggSWxwbykg
dGhhdCBJJ2QgYmUgaGFwcHkgdG8gZ28gd2l0aCAoYiksIGJ1dCBvbmx5IGlmIHRoZXJlIHdhcyBh
bm90aGVyIHVzZSBmb3IgYSBmbGFnLiBCZWNhdXNlIGl0IHdvdWxkIGNvbnN1bWUgMUIgbW9yZSBv
cHRpb25zIHNwYWNlIGluIG1hbnkgcGFja2V0cywgd2hpY2ggaXMgYSBzY2FyY2UgcmVzb3VyY2Uu
DQoNCklscG8gaGFkIGEgcHJvcG9zZWQgdXNlIGZvciBhbm90aGVyIGZsYWcgKHRvIGhlbHAgc3lu
Y2ggY291bnRlcnMgYWZ0ZXIgYSBsb3NzKSwgYnV0IEkgdGhpbmsgdGhlIGRpc2N1c3Npb24gYWJv
dXQgaXQgZW5kZWQgdGhhdCBpdCB3b3VsZG4ndCBiZSBoZWxwZnVsLCAnY29zIHRoZSB3YXkgaXQg
d29ya2VkIGRlcGVuZGVkIG9uIGl0c2VsZiAoY2lyY3VsYXIgbG9naWMpLg0KDQpJbiBjb25jbHVz
aW9uLCBJIGRvbid0IHRoaW5rIHRoZXJlIHdhcyBhbiBleHBsaWNpdCBkZWNpc2lvbiB0byBnbyB3
aXRoIDIgb3B0aW9uIGtpbmRzLCBidXQgaXQgZW5kZWQgdXAgYXMgdGhlICdsYXN0IHBlcnNvbiBz
dGFuZGluZycuDQpJIGxpa2UgaXQuIEl0J3Mgc2ltcGxlLiBBbmQgYXBwYXJlbnRseSBvcHRpb24g
a2luZHMgYXJlIG5vdCBzdWNoIGEgc2NhcmNlIHJlc291cmNlLg0KDQpQZXJoYXBzIHdlIGNhbiBy
YXRpZnkgdGhpcyBpbiB0aGUgV0cgdG9tb3Jyb3cuDQoNCg0KQm9iDQoNCg0KDQoNCg0KVGhhbmtz
LA0KLS0NCllvc2hpDQoNCg0KDQoNCg0KLS0NCg0KX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KDQpCb2IgQnJpc2NvZSAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICBodHRwOi8vYm9iYnJpc2NvZS5uZXQvDQoNCg0KDQoN
Ci0tDQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX18NCg0KQm9iIEJyaXNjb2UgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgaHR0cDovL2JvYmJyaXNjb2UubmV0Lw0KDQoNCg0KLS0NCg0KX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KDQpCb2IgQnJp
c2NvZSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBodHRwOi8vYm9iYnJpc2NvZS5uZXQv
DQo=

--_000_5c3c4661887c43529b35bd5f47d10c2bhsesslingende_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDb25zb2xhczsNCglwYW5vc2UtMToyIDEx
IDYgOSAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWws
IGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0
b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixz
YW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9y
aXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQpwcmUN
Cgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZvcmZvcm1h
dGllcnQgWmNobiI7DQoJbWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9u
dC1zaXplOjEwLjBwdDsNCglmb250LWZhbWlseToiQ291cmllciBOZXciO30NCnR0DQoJe21zby1z
dHlsZS1wcmlvcml0eTo5OTsNCglmb250LWZhbWlseToiQ291cmllciBOZXciO30NCnNwYW4uSFRN
TFZvcmZvcm1hdGllcnRaY2huDQoJe21zby1zdHlsZS1uYW1lOiJIVE1MIFZvcmZvcm1hdGllcnQg
WmNobiI7DQoJbXNvLXN0eWxlLXByaW9yaXR5Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJIVE1MIFZv
cmZvcm1hdGllcnQiOw0KCWZvbnQtZmFtaWx5OkNvbnNvbGFzO30NCnNwYW4uRS1NYWlsRm9ybWF0
dm9ybGFnZTIyDQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5
OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1
bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBwdDt9DQpA
cGFnZSBXb3JkU2VjdGlvbjENCgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2luOjcwLjg1
cHQgNzAuODVwdCAyLjBjbSA3MC44NXB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3Jk
U2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBl
ZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0t
LT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4N
CjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1s
PjwhW2VuZGlmXS0tPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iREUiIGxpbms9ImJsdWUiIHZsaW5r
PSJwdXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3Jt
YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMi
PklubGluZSBzb21lIGNvbW1lbnRzIFttc10uPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5n
dWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1V
UyI+R2l2ZW4gdGhhdCBJIGRvbuKAmXQgYXNrIGZvciBhbnkgdGV4dCBjaGFuY2VzLCBJ4oCZbGwg
bm90IGZvbGxvdy11cCBmb3JldmVyIGluIHRoaXMgZGlzY3Vzc2lvbi4gSSBkb27igJl0IG93biBy
dW5uaW5nIGNvZGUgYW5kIGEgbG90IG9mIGVuY29kaW5nIGRldGFpbHMgbWFpbmx5IGFmZmVjdCBh
biBpbXBsZW1lbnRhdGlvbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT
Ij48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh
biBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5JTUhPIHRo
ZSByZXN0IG9mIHRoZSBXRyBoYXMgdG8gdGhpbmsgYWJvdXQgdGhlIG9wdGlvbiBkZXNpZ24gbW9y
ZSB0aGFuIG1lLiBTbywgdGhpcyBpcyBtb3N0bHkgYWJvdXQgdHJpZ2dlcmluZyBhIGRpc2N1c3Np
b27igKY8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs
YW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNw
OzwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpz
b2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQiPg0KPGRpdj4NCjxkaXYg
c3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29saWQgI0UxRTFFMSAxLjBwdDtwYWRkaW5n
OjMuMHB0IDBjbSAwY20gMGNtIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPkZyb206PC9iPiBC
b2IgQnJpc2NvZSAmbHQ7aWV0ZkBib2JicmlzY29lLm5ldCZndDsgPGJyPg0KPGI+U2VudDo8L2I+
IFR1ZXNkYXksIE5vdmVtYmVyIDE3LCAyMDIwIDE6MjAgUE08YnI+DQo8Yj5Ubzo8L2I+IFNjaGFy
ZiwgTWljaGFlbCAmbHQ7TWljaGFlbC5TY2hhcmZAaHMtZXNzbGluZ2VuLmRlJmd0OzsgWW9zaGlm
dW1pIE5pc2hpZGEgJmx0O25zZC5pZXRmQGdtYWlsLmNvbSZndDs8YnI+DQo8Yj5DYzo8L2I+IE1p
Y2hhZWwgVHVleGVuICZsdDt0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUmZ3Q7OyB0Y3BtLWNoYWlyc0Bp
ZXRmLm9yZzsgTWlyamEgS3VlaGxld2luZCAmbHQ7aWV0ZkBrdWVobGV3aW5kLm5ldCZndDs7IFNj
aGVmZmVuZWdnZXIsIFJpY2hhcmQgJmx0O1JpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29t
Jmd0OzsgdGNwbSBJRVRGIGxpc3QgJmx0O3RjcG1AaWV0Zi5vcmcmZ3Q7PGJyPg0KPGI+U3ViamVj
dDo8L2I+IFJlOiBBY2NFQ04gZmllbGQgb3JkZXI8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9k
aXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiIHN0eWxlPSJtYXJnaW4tYm90dG9tOjEyLjBwdCI+TWljaGFlbCw8bzpwPjwv
bzpwPjwvcD4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5PbiAxNy8xMS8yMDIwIDA4OjEw
LCBTY2hhcmYsIE1pY2hhZWwgd3JvdGU6PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxibG9ja3F1
b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQiPg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5C
b2IsPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5
bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1m
YXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5JIGFtIGZpbmUgd2l0aCB0aGUgdHdvIG9wdGlvbiBraW5k
cyBwcm9wb3NlZCBpbiAtMTMsIGJ1dCBJIGRvbuKAmXQgYnV5IHlvdXIgYXJndW1lbnRzIHdoeSB0
aGlzIGVuY29kaW5nIGlzIGJldHRlciB0aGFuIG90aGVycy48L3NwYW4+PG86cD48L286cD48L3A+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJl
YXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj5Nb3N0IGltcG9ydGFudGx5LCBJIGRvbuKAmXQgdGhpbmsgdGhhdCB5b3VyIOKA
nGZvcndhcmQgY29tcGF0aWJpbGl0eeKAnSBhcmd1bWVudCBpcyBhIHZlcnkgY29tcGVsbGluZyBy
ZWFzb24gZm9yIHR3byBjb2RlcG9pbnRzLiBBbGwgcHJvcG9zYWxzIEkgYW0gYXdhcmUgb2YgaGF2
ZSB0aGVpciBwcm9zIGFuZCBjb25zLiBBbmQgSQ0KIGFtIG5vdCBhd2FyZSBvZiBhIHJlYWxseSBj
b21wcmVoZW5zaXZlIGRpc2N1c3Npb24uIEF0IGxlYXN0IG5laXRoZXIgdGhlIGUtbWFpbCBiZWxv
dyBub3IgdG9kYXnigJlzIG1lZXRpbmcgZGlzY3VzcyBhbGwgdHJhZGVvZmZzIEkgd291bGQgaW52
ZXN0aWdhdGUuPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPC9ibG9ja3F1b3RlPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBZb3UncmUgcmlnaHQgLSB0aGUgZGlzY3Vzc2lvbiBiZWxv
dyBpcyB1c2VmdWwgYW5kIG5lY2Vzc2FyeS48YnI+DQo8YnI+DQo8YnI+DQo8bzpwPjwvbzpwPjwv
cD4NCjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4w
cHQiPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28t
ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+Jm5ic3A7PC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1s
YW5ndWFnZTpFTi1VUyI+Rm9yIGluc3RhbmNlLCB0aGUgYmVuZWZpdCBvZiB0aGUgbGVuZ3RoIGVu
Y29kaW5nIHZhcmlhbnQgd291bGQgYmUgdG8gY29uc3VtZSBvbmx5IG9uZSBUQ1Agb3B0aW9uIGNv
ZGVwb2ludCBub3cuIElmIHdlIGRlY2lkZWQgbGF0ZXIgdG8gdGhhdCBhIGZsYWcgYnl0ZSBpcyBu
ZWVkZWQgKHNheSwgYXQgdGhlIGVuZCksIGENCiBmb2xsb3ctdXAgcHJvcG9zZWQgc3RhbmRhcmQg
Y291bGQgc3BlY2lmeSBhbm90aGVyIG9wdGlvbiBmb3JtYXQgd2l0aCB0aGF0IGZsYWcgYnl0ZSwg
dXNpbmcgYSBzZWNvbmQgY29kZXBvaW50Ljwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjwvYmxvY2tx
dW90ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCltCQl0gPGJyPg0KKiBUaGUgbmV3IGNv
ZGVwb2ludCB3b3VsZCByZXF1aXJlIHllYXJzIHRvIGRlcGxveSAoaW5jbHVkaW5nIGluIHByb3hp
ZXMpIGJlZm9yZSBpdCB3b3JrcyBlMmUuDQo8bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g
bGFuZz0iRU4tVVMiPlttc10gVGhhdCBtYXkgZGVwZW5kICo8Yj5hIGxvdDwvYj4qIG9uIHRoZSBl
bnZpcm9ubWVudC4gSW4gZW52aXJvbm1lbnRzIHdoZXJlIEVDTiBjYW4gZ2V0IGRlcGxveWVkIHRv
ZGF5LCBzYXkgaW4gZGF0YWNlbnRlcnMsIHJvbGxpbmcgb3V0IGEgbmV3IGtlcm5lbCB2ZXJzaW9u
IG1heSBwZXJoYXBzIG5vdCBiZSB0aGUgYmlnZ2VzdCBvcGVyYXRpb25hbCBjaGFsbGVuZ2UuIEFs
c28sDQogcHJveGllcyBtYXkgbm90IGJlIGEgc2lnbmlmaWNhbnQgcHJvYmxlbSBpbiBzb21lIG9m
IHRob3NlIGNhc2VzLiBEZXBsb3ltZW50IHJvYWRtYXBzIGFyZSB2ZXJ5IGhhcmQgdG8gcHJlZGlj
dC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5n
PSJFTi1VUyI+PGJyPg0KKiBXaGVyZWFzIHRoZSAnZm9yd2FyZCBjb21wYXRpYmlsaXR5JyBhcHBy
b2FjaCB0cmllcyB0byBlbnN1cmUgdGhhdCBBY2NFQ04gaW1wbGVtZW50YXRpb25zIHRoYXQgYXJl
IGJlaW5nIGJ1aWx0IHRvZGF5IHdpbGwgYmUgcmVhZHkgdG8gaW50ZXJ3b3JrIHdpdGggYSBmdXR1
cmUgZXh0ZW5zaW9uLg0KPGJyPg0KPC9zcGFuPiogSW4gY29udHJhc3QsIGlmIHlvdXIgaHlwb3Ro
ZXRpY2FsIG5ldyBBY2NFQ04gY29kZXBvaW50IGRpZCBuZWVkIHRvIGJlIGRlcGxveWVkIGxhdGVy
LCBpdCB3b3VsZCBuZWVkIHRvIGluY2x1ZGUgYSBmYWxsLWJhY2sgdG8gdGhlIGN1cnJlbnQgQWNj
RUNOIG9wdGlvbiB3aGVuIG9uZSBwZWVyIGRvZXNuJ3QgdW5kZXJzdGFuZCB0aGUgbmV3IGNvZGVw
b2ludC4NCjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8
L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+W21zXSBU
aGlzIGlzIGEgcmVhc29uYWJsZSBwb2ludCwgYWxiZWl0IG5vdCBhIHBhcnRpY3VsYXJseSBzdHJv
bmcgb25lLCBnaXZlbiB0aGF0IHRoZSBvcHRpb24gaXMgbm90IHJlYWxseSByZXF1aXJlZCBmb3Ig
YmFzaWMgaW50ZXJvcGVyYWJpbGl0eS4gQWxzbywgYSBoeXBvdGhldGljYWwgZm9sbG93LXVwIHN0
YW5kYXJkIGNvdWxkIGluY2x1ZGUgYSB3YXkgdG8gZXhwbGljaXRseQ0KIHJlcXVlc3QgdGhlIG9w
dGlvbuKApjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IGxhbmc9IkVOLVVTIj48YnI+DQoqIEZhbGwtYmFjayB3b3VsZCBuZWVkIHRvIGF2b2lkIGEgc2Vj
b25kIHJvdW5kIG9mIGhhbmRzaGFrZSAoZ2l2ZW4gdGhpcyBpcyBmb3IgbG93IGxhdGVuY3kpLCB3
aGljaCB3b3VsZCBvbmx5IHdvcmsgcmVsaWFibHkgd2l0aCBhbiBvcHRpb24gb24gdGhlIFNZTiAo
YmVjYXVzZSB0aGUgM3JkIEFDSyBpcyB1bnJlbGlhYmxlKTxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpw
Pjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+W21z
XSBJIGNhbuKAmXQgZm9sbG93LiBJZiBhbiBlbmRwb2ludCBpcyB1bmhhcHB5IHdpdGggZ2V0dGlu
ZyBBY2NFQ04gb3B0aW9uIOKAnEHigJ0gZnJvbSBhIHBlZXIsIEkgY291bGQgc2VuZCBhIOKAnHBs
ZWFzZSBzZW5kIG1lIHRoZSBiZXR0ZXIgQWNjRUNOIG9wdGlvbiBC4oCdIGFmdGVyIHRoZSAzLVdI
Uy4gSXQgY291bGQgYmUgYW4gdXBncmFkZSwgbm90IGEgZmFsbGJhY2suIEFsc28sIGF0IHRoaXMN
CiBwb2ludCB3ZSBkb27igJl0IGtub3cgd2hhdCBvcHRpb24gQiB3b3VsZCBiZSBhbmQgaW4gd2hp
Y2ggY29udGV4dCBpdCB3b3VsZCBiZSBuZWVkZWQuIE1heWJlIG90aGVyIHByb3RvY29sIGZlYXR1
cmVzIHdvdWxkIHJlcXVpcmUgbmVnb3RpYXRpb24sIHRvby4gSU1ITyBpdCBpcyBhIHNvbHZhYmxl
IHByb2JsZW0uPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gbGFuZz0iRU4tVVMiPjxicj4NCiogQWNjRUNOIGhhcyBzbyBmYXIgYXZvaWRlZCB1c2luZyBh
bnkgU1lOIG9wdGlvbiBzcGFjZSwgd2hpY2ggaXMgaW1wb3J0YW50IGdpdmVuIHRoYXQgaXMgdGhl
IHNjYXJjZXN0IHJlc291cmNlLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+W21zXSBZZXAsIGJ1dCBhcyBs
b25nIGFzIG5vIHByb3Bvc2FsIG5lZWRzIFNZTiBvcHRpb25zLCB0aGlzIGlzIG5vdGhpbmcgdGhh
dCB3b3VsZCBtYWtlIG9uZSBzb2x1dGlvbiBiZXR0ZXIgdGhhbiBhbm90aGVyLjwvc3Bhbj48c3Bh
biBsYW5nPSJFTi1VUyI+PGJyPg0KPGJyPg0KVGhlc2UgZGF5cywgZXh0ZW5kaW5nIFRDUCBpcyBs
aWtlIHBsYXlpbmcgY2hlc3MgLSB3ZSBoYXZlIHRvIHRoaW5rIG11bHRpcGxlIG1vdmVzIGFoZWFk
ITxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7
PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPlttc10g
VGhlcmUgaXMgb25lIGZ1bmRhbWVudGFsIGRpZmZlcmVuY2UgYmV0d2VlbiBjaGVzcyBhbmQgVENQ
OiBJbiBjaGVzcyB5b3UgcHJlY2lzZWx5IGtub3cgdGhlIGN1cnJlbnQgc2l0dWF0aW9uIGFuZCBj
YW4gcHJlZGljdCBhbGwgcG90ZW50aWFsIG1vdmVzIGluIGFkdmFuY2UgKGFsYmVpdCB0aGVyZSBh
cmUgYSBsb3QpLiBIb3dldmVyLCB3ZSBkb27igJl0IGtub3cgdGhlIGZ1dHVyZQ0KIGV2b2x1dGlv
biBvZiBUQ1AsIEVDTiBhbmQgdGhlIEludGVybmV0IGFzIGEgd2hvbGUsIG5vciBmdXR1cmUgcmVx
dWlyZW1lbnRzLiBJZiB3ZSB0cnkgdG8gZW5naW5lZXIgYSBUQ1AgZXh0ZW5zaW9uIGZvciBzb21l
IHBvdGVudGlhbCBmdXR1cmUsIHdlIG1heSBmaWd1cmUgb3V0IGxhdGVyIHRoYXQgcmVhbGl0eSBp
cyBkaWZmZXJlbnQgZnJvbSB3aGF0IHdlIGRlc2lnbmVkIGZvci4gQXMgbXVjaCBhcyBJIGFwcHJl
Y2lhdGUgeW91ciBhdHRlbXB0cw0KIHRvIHByZWRpY3QgdGhlIGZ1dHVyZSwgaXQgd291bGQgYmUg
YSBiaWcgc3VycHJpc2UgdG8gbWUgaWYgeW91IGNvdWxkIGNvcnJlY3RseSBwcmVkaWN0IHdoYXQg
VENQIHdpbGwgYmUgaW4gMTAgeWVhcnMgZnJvbSBub3cuIFNvLCBpbnN0ZWFkIG9mIHRyeWluZyB0
byB3cml0ZSBzcGVjaWZpY2F0aW9ucyB0aGF0IHRha2UgaW50byBhY2NvdW50IGZ1dHVyZSByZXF1
aXJlbWVudHMsIHdoaWNoIGFyZSBpbiBmYWN0IGp1c3QgdW5rbm93biwgaXQgaXMgbW9yZQ0KIGlt
cG9ydGFudCBmb3IgbWUgdG8gd3JpdGUgZWZmZWN0aXZlLCBlZmZpY2llbnQgYW5kIHNlY3VyZSBz
cGVjaWZpY2F0aW9ucyBmb3Igd2hhdCBpcyBjdXJyZW50bHkga25vd24gZm9yIHN1cmUuPG86cD48
L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMi
PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IGxhbmc9IkVOLVVTIj5bbXNdIEkgKjxiPmRvbuKAmXQ8L2I+KiBrbm93IGZvciBzdXJlIHdoZXRo
ZXIgdGhlIGN1cnJlbnQgQWNjRUNOIG9wdGlvbiB3aWxsIGJlIG5lZWRlZCwgYnV0IEkgc2VlIHF1
aXRlIHNvbWUgZXZpZGVuY2UgdGhhdCBzdXBwb3J0IG9mIHRoZSBBY2NFQ04gb3B0aW9uKHMpIGlz
IG5vdCBhIHRvcCBwcmlvcml0eSB0aGVzZSBkYXlzLiBUaGF0IGlzIGFsc28gc29tZXRoaW5nIHRv
IHRoaW5rDQogYWJvdXQgd2hlbiBkZXNpZ25pbmcgdGhlIGRldGFpbHMuPG86cD48L286cD48L3Nw
YW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+Jm5i
c3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO
LVVTIj5JIG5lZWQgdG8gbGVhcm4gdG8gYXJ0aWN1bGF0ZSBvbnRvIHRoZSBsaXN0IGFsbCBteSBy
ZWFzb25zIGZvciByZWplY3RpbmcgY2VydGFpbiBwYXJ0cyBvZiB0aGUgZGVzaWduIHNwYWNlLg0K
PC9zcGFuPlNvcnJ5IGFib3V0IHRoYXQuPGJyPg0KPGJyPg0KWWVzLCBJIGFkbWl0LCB0aGUgY3Vy
cmVudCBwcm9wb3NhbCBpcyBhdCB0aGUgY29zdCBvZiBidXJuaW5nIHR3byBUQ1Agb3B0aW9uIGNv
ZGVwb2ludHMuIEhvd2V2ZXIsIHlvdSBzYWlkIHlvdXJzZWxmIHdoZW4geW91IHByb3Bvc2VkIHRo
aXMgYXBwcm9hY2gsIHRoYXQgdGhleSBhcmUgbm90IChjdXJyZW50bHkpIGEgc2NhcmNlIHJlc291
cmNlLiBDZXJ0YWlubHkgbm90IGFzIHNjYXJjZSBhcyBvcHRpb24gc3BhY2UuIEkganVzdCBjaGVj
a2VkOiByb3VnaGx5DQogMTUlIG9mIHRoZSBvcHRpb24gY29kZXBvaW50IGFsbG9jYXRpb24gc3Bh
Y2UgaGFzIGJlZW4gdXNlZCBzbyBmYXIuIFNvIHRoYXQncyB3aGVyZSBJIHdvdWxkIHN1Z2dlc3Qg
d2UgY29tcHJvbWlzZSBmaXJzdC48YnI+DQo8YnI+DQo8c3BhbiBsYW5nPSJFTi1VUyI+W21zXSBJ
IG1hZGUgYSBzaW1pbGFyIGNhbGN1bGF0aW9uIHNvbWUgeWVhcnMgYWdvIGFuZCBjYW1lIHRvIGEg
c2ltaWxhciBjb25jbHVzaW9uLiBJbnZlc3RpbmcgdHdvIGNvZGVwb2ludHMgZm9yIEFjY0VDTiBp
cyBzb21ldGhpbmcgd2UgY2FuIHByb2JhYmx5IGFmZm9yZC4gQnV0IG5vbmV0aGVsZXNzIHdlIGhh
dmUgdG8gcmVhc29uIGNhcmVmdWxseSB3aHkgaXQgaXMgcmVhbGx5IHdvcnRoIHRoZSBlZmZvcnQu
PGJyPg0KPGJyPg0KPG86cD48L286cD48L3NwYW4+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1h
cmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4m
bmJzcDs8L3NwYW4+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3Qt
bGFuZ3VhZ2U6RU4tVVMiPlJlZ2FyZGluZyB0aGUgcmVzdWx0aW5nIGNvZGVwb2ludCBjb25zdW1w
dGlvbiAodHdvIG9wdGlvbiBraW5kcykgYW5kIHRoZSBhZGRpdGlvbiBvZiBhIGZsYWcgYnl0ZSwg
dGhpcyBhcHByb2FjaCB3b3VsZCBiYXNpY2FsbHkgZW5kIHVwIHdpdGggdGhlIHNhbWUgcmVzdWx0
IGFzIHRoZSBjdXJyZW50IHByb3Bvc2FsIGluDQogLTEzIHBsdXMgc29tZSBoeXBvdGhldGljYWwg
ZnV0dXJlIGFkZGl0aW9uIGxldmVyYWdpbmcgdGhlIGxlbmd0aCBmaWVsZCBiZXlvbmQgdGhlIGN1
cnJlbnRseSBrbm93biB1c2UuPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1V
UyI+Jm5ic3A7PC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+QW5kLCB5
ZXMsIHRoZSBsZW5ndGggZW5jb2RpbmcgdmFyaWFudCBtYXkgYmUgbGVzcyBmbGV4aWJsZSBhbmQv
b3IgY29uc3VtZSBzb21lIG1vcmUgYml0cyBpbiBzb21lIGNhc2VzLCBidXQgb24gdGhlIHBsdXMg
c2lkZSBpdCBvbmx5IG5lZWRzIG9uZSBjb2RlcG9pbnQgbm93IOKAkyBhbmQgZ2l2ZW4gdGhhdCBp
dCBpcyBlbnRpcmVseQ0KIHVuY2xlYXIgd2hldGhlciBhbnkgQWNjRUNOIG9wdGlvbiB3aWxsIGlu
ZGVlZCBiZSB3aWRlbHkgdXNlZCBpbiBmdXR1cmUsIHRoaXMgaXMgYSBiaWcgcGx1cy4gVGhlIGZh
Y3QgdGhhdCBpbXBsZW1lbnRlcnMgYXJlIHF1aXRlIHNpbGVudCBvbiB0aGUgb3B0aW9uIGRlc2ln
biBpcyBub3QgYSBnb29kIHNpZ24uPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPC9ibG9ja3F1b3Rl
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBIZXJlJ3MgYW5vdGhlciBwb3NzaWJp
bGl0eSB0aGVuLiA8YnI+DQo8YnI+DQo8dHQ+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQi
PmEpIFtLfEx8IEVFMUIgfCBFQ0VCIHwgRUUwQiBdPC9zcGFuPjwvdHQ+PGJyPg0KPHR0PjxzcGFu
IHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij5iKSBbS3xMfCBFRTFCIHwgRUNFQiB8IEVFMEIgfEZd
Jm5ic3A7Jm5ic3A7Jm5ic3A7IGlmIEYgPSAwYlhYWFhYWFgwPC9zcGFuPjwvdHQ+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q291cmllciBOZXcmcXVvdDsi
Pjxicj4NCjx0dD5jKSBbS3xMfCBFRTBCIHwgRUNFQiB8IEVFMUIgfEZdJm5ic3A7Jm5ic3A7Jm5i
c3A7IGlmIEYgPSAwYlhYWFhYWFgxPC90dD48YnI+DQo8L3NwYW4+PGJyPg0KV2UnZCBoYXZlIHRv
IGRlY2lkZSB3aGljaCBvcmRlciBhbHRlcm5hdGl2ZSAoYSkgdGFrZXMuIEkndmUgcGlja2VkIEVF
MUIgZmlyc3Qgb25seSAnY29zIHlvdSdyZSByaWdodCB0aGF0IHRoZXJlIGlzIG5vdCBjdXJyZW50
bHkgYXMgbXVjaCBwcmVzc3VyZSAob24gdGhlIHB1YmxpYyBJbnRlcm5ldCkgZm9yIEFjY0VDTiB3
aXRoIEVDVCgwKS48YnI+DQo8c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWls
eTomcXVvdDtDb3VyaWVyIE5ldyZxdW90OyI+PGJyPg0KPHR0Pj09TGVnZW5kPT08L3R0Pjxicj4N
Cjx0dD5LOiZuYnNwOyZuYnNwOyZuYnNwOyBLaW5kICgxQik8L3R0Pjxicj4NCjx0dD5MOiZuYnNw
OyZuYnNwOyZuYnNwOyBMZW5ndGggKDFCKTwvdHQ+PGJyPg0KPHR0PkV4eEI6IEVjaG8geHggYnl0
ZSBjb3VudGVyPC90dD48YnI+DQo8dHQ+RjombmJzcDsmbmJzcDsmbmJzcDsgRmxhZ3MgKDFCKTwv
dHQ+PGJyPg0KPGJyPg0KPHR0PkFsdGVybmF0aXZlcyBhKSBhbmQgYikgYXJlIGVxdWl2YWxlbnQs
IGF0IGxlYXN0IHdoaWxlIHRoZSBvbmx5IGFsbG9jYXRlZCBmbGFnIGlzIGZvciBmaWVsZCBvcmRl
ci48L3R0Pjxicj4NCjx0dD5Ib3dldmVyLCBhbHRlcm5hdGl2ZSBiKSBpcyBhdmFpbGFibGUgaW4g
Y2FzZSBvdGhlciBmbGFncyBhcmUgYWxsb2NhdGVkIGluIGZ1dHVyZTwvdHQ+PGJyPg0KPHR0PlRo
ZSBmbGFncyBieXRlIGNhbiBvbmx5IGJlIG9taXR0ZWQgaWYgKEwgJSAzID09IDIpLjwvdHQ+PGJy
Pg0KPHR0PlRoZSBmbGFncyBieXRlIGlzIGNvbnNpZGVyZWQgcHJlc2VudCBpZiAoTCAlIDMpID09
IDAuPC90dD48YnI+DQo8YnI+DQo8dHQ+Rm9yd2FyZCBjb21wYXRpYmlsaXR5OiBPcHRpb25zIHdp
dGggKEwgJSAzID09IDEpIE1VU1QgYmUgYXNzdW1lZCB0byBpbmNsdWRlIGEgZmxhZ3MgYnl0ZSwg
YW5kIGN1cnJlbnQgaW1wbGVtZW50YXRpb25zIGlnbm9yZSB0aGUgbGFzdCBieXRlLjwvdHQ+PGJy
Pg0KPGJyPg0KPHR0PlRoZSBmbGFncyBieXRlIGlzIG9wdGlvbmFsIHRvIGltcGxlbWVudCAoZXZl
biBpZiB0aGUgQWNjRUNOIG9wdGlvbiBpcyBpbXBsZW1lbnRlZCk8L3R0Pjxicj4NCjx0dD5JZiB0
aGUgc2VydmVyIGluY2x1ZGVzIGEgZmxhZ3MgYnl0ZSBvbiB0aGUgU1lOL0FDSyBidXQgdGhlIGNs
aWVudCBkb2VzIG5vdCBpbmNsdWRlIG9uZSBvbiB0aGUgM3JkIEFDSyBvciB0aGUgZmlyc3QgZGF0
YSBwYWNrZXQsIHRoZSBzZXJ2ZXIgYXNzdW1lcyB0aGUgY2xpZW50IGRvZXMgbm90IGltcGxlbWVu
dCB0aGUgZmxhZ3MgYnl0ZSBhbmQgdXNlcyBvbmx5IGFsdGVybmF0aXZlIGEpIGZvciB0aGUgcmVt
YWluZGVyIG9mIHRoZSBjb25uZWN0aW9uLjwvdHQ+PGJyPg0KPGJyPg0KPGJyPg0KPC9zcGFuPjxz
cGFuIGxhbmc9IkVOLVVTIj5bbXNdIEkgYW0gbm90IHN1cmUgaWYgd2UgbmVlZCB0aGF0LCBidXQg
SSB0YWtlIHRoaXMgYXMgZXhpc3RlbmNlIHByb29mIHRoYXQgb3RoZXIgZW5jb2RpbmdzIGFyZSBw
b3NzaWJsZS4gUGVyc29uYWxseSwgSSB0aGluayB0aGF0IG90aGVyIHNvbHV0aW9ucyB3b3VsZCBi
ZSBkb2FibGUgaWYgeW91IHRvb2sgb25lIGJpdCBmcm9tIG9uZSBvciBtdWx0aXBsZSBjb3VudGVy
cyBmb3IgZm9ybWF0IGVuY29kaW5nLg0KIEF0IGxlYXN0IGZvciBzb21lIG9mIHRoZSBieXRlIGNv
dW50ZXJzIEkgZG9u4oCZdCB0aGluayB0YWtpbmcgb25lIGJpdCBmb3Igb3RoZXIgcHVycG9zZXMg
d291bGQgcmVzdWx0IGluIGFuIEludGVybmV0IGNvbmdlc3Rpb24gY29sbGFwc2XigKYgQnV0LCBJ
IGhhdmUgYWxyZWFkeSBzYWlkIHRoYXQgdGhlIC0xMyBwcm9wb3NhbCB3b3JrcyBmb3IgbWUsIHNv
IEnigJlsbCBub3QgdHJ5IHRvIGNvbWUgdXAgd2l0aCBhbm90aGVyIHZhcmlhbnQgbXlzZWxmLg0K
PG86cD48L286cD48L3NwYW4+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6NS4w
cHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5n
PSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+
PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPlRvIG1lLCBvbmUgcG90ZW50aWFsIGRpZmZlcmVuY2UgYmV0d2VlbiB0d28gcHJvcG9zYWxz
IHdvdWxkIGJlIGluY3JlbWVudGFsIGRlcGxveW1lbnQuIFRoZSBwcm9wb3NhbCBpbiAtMTMgb25s
eSBoYXMgYW4gYWR2YW50YWdlIGlmIG1pZGRsZWJveGVzIHN1Y2ggYXMgZmlyZXdhbGxzIHdpbGwg
aW5kZWVkIHBhc3MgVENQDQogb3B0aW9ucyB3aXRoIGEgZm9ybWF0IHRoYXQgY29udGFpbnMgY29u
dGVudCBiZXlvbmQgdGhlIChmaXJzdCkgQWNjdXJhdGUgRUNOIHN0YW5kYXJkIChpLmUuLCBjdXJy
ZW50bHkgdW51c2VkIGxlbmd0aCB2YWx1ZXMpLiBJTUhPIGl0IGlzIHRvbyBlYXJseSB0byBrbm93
IHdoZXRoZXIgZmlyZXdhbGxzIHdvdWxkIGluZGVlZCBhbGxvdyB0aGlzIGluIGZ1dHVyZS48L3Nw
YW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48
L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9
Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5Gcm9tIGEgc2VjdXJpdHkgcGVyc3BlY3RpdmUs
IGl0IGlzIG5vdCBjbGVhciB0byBtZSB3aGV0aGVyIGFsbG93aW5nIGFyYml0cmFyeSB1bnNwZWNp
ZmllZCBieXRlcyBpbiBhIFRDUCBvcHRpb24gaXMgYSBnb29kIGlkZWEgKjxiPmF0IGFsbDwvYj4q
LiBJdCB3aWxsIGJlIGludGVyZXN0aW5nIHRvIGhlYXIgdGhlIG9waW5pb24NCiBmcm9tIFNFQyBh
cmVhIG9uIHRoYXQuIFBlcnNvbmFsbHksIEkgYW0gbm90IGNvbnZpbmNlZCB0aGF0IHRoaXMgcmVh
bGx5IG1ha2VzIHNlbnNlLCBidXQgSSBteSBjb25jZXJucyBhcmUgbm90IHN0cm9uZyBlbm91Z2gg
dG8gZm9ybWFsbHkgcHVzaCBiYWNrLiBJ4oCZbGwgbGVhdmUgaXQgdG8gb3RoZXJzIHRvIHRoaW5r
IGFib3V0IHdoZXRoZXIgdGhpcyBpcyBhIGJ1ZyBvciBhIGZlYXR1cmUuPC9zcGFuPjxvOnA+PC9v
OnA+PC9wPg0KPC9ibG9ja3F1b3RlPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBX
ZWxsLCBsZXQncyBmaXJzdCB0cnkgdG8gZGVhbCB3aXRoIHNlY3VyaXR5IG91cnNlbHZlczo8YnI+
DQoqIE9jdGV0cyB0aGF0IGFyZSBleHBsaWNpdGx5IGRlY2xhcmVkIGFzIHBhcnQgb2YgYW4gb3B0
aW9uIGNhbm5vdCBiZSB1c2VkIGZvciBhIGJ1ZmZlciBvdmVyZmxvdyBhdHRhY2suIEkgZG9uJ3Qg
cmVhbGx5IG5lZWQgdG8sIGJ1dCBJIGNvdWxkIGFkZCB0aGUgZm9sbG93aW5nIHRleHQgdG8gdGhl
IGZvcndhcmQgY29tcGF0aWJpbGl0eSB3b3JkaW5nOg0KPGJyPg0KJm5ic3A7Jm5ic3A7Jm5ic3A7
IEEgcmVjZWl2ZXIgY29uc2lkZXJzIG9jdGV0cyBiZXlvbmQgdGhvc2UgaXQgdXNlcywgYnV0IHdp
dGhpbiB0aGUgc3BlY2lmaWVkIGxlbmd0aCwgYXMgaWYgdGhleSBhcmUgcGFkZGluZy4NCjxicj4N
CiogQW5kIHN1Y2ggb2N0ZXRzIGNhbm5vdCBiZSBhbnkgZGlmZmVyZW50IGZyb20gdGhlIGN1cnJl
bnQgYWJpbGl0eSBvZiBhIHNlbmRlciB0byBhZGQgcGFkZGluZy4gU28gdGhlcmUncyBubyBuZXcg
YXR0YWNrIHBvc3NpYmxlIGhlcmUuPGJyPg0KKiBBbmQgdGhlcmUncyBubyBuZWVkIHRvIHNwZWNp
ZnkgYSBtYXggbGVuZ3RoIGZvciBhbnkgQWNjRUNOIE9wdGlvbiwgd2hpY2ggd291bGQganVzdCB1
bm5lY2Vzc2FyaWx5IGxpbWl0IGZsZXhpYmlsaXR5Ljxicj4NCjxicj4NCjxzcGFuIGxhbmc9IkVO
LVVTIj5bbXNdIEkgaGF2ZSBtb3JlIHRob3VnaHQgYWJvdXQgdGhlIGNvdmVydCBjaGFubmVsIHRo
YXQgc29tZSB1bnNwZWNpZmllZCBieXRlcyBpbiBhIHN0YW5kYXJkcyB0cmFjayBUQ1Agb3B0aW9u
IGNvdWxkIG9mZmVy4oCmIEluIHBhcnRpY3VsYXIgaWYgdGhlIEFjY0VDTiBvcHRpb24gZ2V0cyB3
aWRlbHkgZGVwbG95ZWQgYW5kIHVzZWQsIHdvdWxkIHNvbWUgbWFsaWNpb3VzIHVzZXJzIGZpbmQg
dGhvc2UgYnl0ZXMgdXNlZnVsPw0KIEZvciB3aGF0PyBGb3IgaW5zdGFuY2UsIHdoYXQgaW4gdGhl
IEFjY0VDTiBwcm90b2NvbCBkZXNpZ24gcHJldmVudHMgdXNlIG9mIHRoZXNlIOKAnHBhZGRpbmfi
gJ0gYnl0ZXMgYXMgc3VwZXItY29va2llPyBUbyBtZSwgaXQgaXMgbXVjaCBtb3JlIGltcG9ydGFu
dCB0byB0aGluayBhYm91dCBhYnVzZSBvZiB3aGF0ZXZlciB3ZSBzdGFuZGFyZGl6ZSAqPGI+bm93
PC9iPiosIGluc3RlYWQgb2YgZW5naW5lZXJpbmcgZm9yIHNvbWUgZW50aXJlbHkgdW5rbm93bg0K
IGZ1dHVyZS4gSSBhbSBub3QgYSBzZWN1cml0eSBleHBlcnQsIGJ1dCBJIHN1Z2dlc3QgYSBjYXJl
ZnVsIGFuYWx5c2lzIG9mIGFueSBzZWN1cml0eSBpbXBsaWNhdGlvbnMgb2YgdGhlIEFjY0VDTiBv
cHRpb24uIFdlIHNlbGRvbWx5IHNwZWNpZnkgbmV3IFRDUCBvcHRpb25zIGFuZCBtYWxpY2lvdXMg
dXNlcnMgd2lsbCBjZXJ0YWlubHkgbG9vayBmb3Ig4oCcb3Bwb3J0dW5pdGllc+KAnSBpbiB0aGUg
c3BlYy4gU28gZmFyLCBJIGRvbuKAmXQga25vdyB3aGV0aGVyDQogdGhhdCB3aG9sZSBpZGVhIG9m
IGZvcndhcmQgY29tcGF0aWJpbGl0eSBpcyBhIGZlYXR1cmUgb3IgYSBidWcuPG86cD48L286cD48
L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+
Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9
IkVOLVVTIj5NaWNoYWVsPG86cD48L286cD48L3NwYW4+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9
Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT
Ij4mbmJzcDs8L3NwYW4+PHNwYW4gbGFuZz0iRU4tVVMiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVh
c3QtbGFuZ3VhZ2U6RU4tVVMiPk1heWJlIG9uZSBsZXNzb24gbGVhcm50IGlzIHRoYXQgdGhlIGRv
Y3VtZW50IGNvdWxkIGhhdmUgYSBub24tbm9ybWF0aXZlIGFwcGVuZGl4IHRoYXQgZXhwbGFpbnMg
dGhlIHJhdGlvbmFsZSBmb3IgdGhlIGZpbmFsbHkgcGlja2VkIFRDUCBvcHRpb24gZW5jb2Rpbmcu
IFRoYXQgbWF5IGFsc28gaGVscCBpZiB0aGVyZSBhcmUNCiBmdXJ0aGVyIHF1ZXN0aW9ucyB3aGV0
aGVyIHR3byBjb2RlcG9pbnRzIGFyZSByZWFsbHkgcmVxdWlyZWQsIGUuZy4gYnkgdGhlIElFU0cg
KGlmIHR3byBjb2RlcG9pbnRzIGFyZSBzdGlsbCB0aGUgZGVzaWduIGFmdGVyIFdHTEMpLiBBdCBs
ZWFzdCBmb3IgcGFzdCBUQ1Agb3B0aW9uIGNvZGVwb2ludCBhbGxvY2F0aW9ucyBJIHJlY2FsbCBz
b21lIGRpc2N1c3Npb25zIGxhdGUgaW4gdGhlIElFVEYgcHJvY2Vzcy4gSW4gdGhvc2UgcGFzdCBj
YXNlcywNCiBnb29kIGFyZ3VtZW50cyBpbiBhbiBhcHBlbmRpeCBhbmQgcnVubmluZyBjb2RlIGhh
cyBoZWxwZWQgYSBsb3QuPC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPC9ibG9ja3F1b3RlPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBJIGNhbiBkbyB0aGF0LiBBcHB4IEIgaXMgYWxy
ZWFkeSBzaW1pbGFyLCBnaXZpbmcgRGVzaWduIFJhdGlvbmFsZSBmb3IgdGhlIEFDRSBmaWVsZC48
YnI+DQo8YnI+DQo8YnI+DQpCb2I8YnI+DQo8YnI+DQo8YnI+DQo8bzpwPjwvbzpwPjwvcD4NCjxi
bG9ja3F1b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQiPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+Jm5ic3A7PC9zcGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+TWljaGFlbDwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt
YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMi
PiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOzwv
c3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO
LVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPiZuYnNwOzwvc3Bhbj48bzpw
PjwvbzpwPjwvcD4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci1sZWZ0OnNvbGlkIGJs
dWUgMS41cHQ7cGFkZGluZzowY20gMGNtIDBjbSA0LjBwdCI+DQo8ZGl2Pg0KPGRpdiBzdHlsZT0i
Ym9yZGVyOm5vbmU7Ym9yZGVyLXRvcDpzb2xpZCAjRTFFMUUxIDEuMHB0O3BhZGRpbmc6My4wcHQg
MGNtIDBjbSAwY20iPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGI+PHNwYW4gbGFuZz0iRU4tVVMi
PkZyb206PC9zcGFuPjwvYj48c3BhbiBsYW5nPSJFTi1VUyI+IEJvYiBCcmlzY29lDQo8YSBocmVm
PSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+Jmx0O2lldGZAYm9iYnJpc2NvZS5uZXQmZ3Q7
PC9hPiA8YnI+DQo8Yj5TZW50OjwvYj4gVHVlc2RheSwgTm92ZW1iZXIgMTcsIDIwMjAgNjoxMCBB
TTxicj4NCjxiPlRvOjwvYj4gU2NoYXJmLCBNaWNoYWVsIDxhIGhyZWY9Im1haWx0bzpNaWNoYWVs
LlNjaGFyZkBocy1lc3NsaW5nZW4uZGUiPiZsdDtNaWNoYWVsLlNjaGFyZkBocy1lc3NsaW5nZW4u
ZGUmZ3Q7PC9hPjsgWW9zaGlmdW1pIE5pc2hpZGENCjxhIGhyZWY9Im1haWx0bzpuc2QuaWV0ZkBn
bWFpbC5jb20iPiZsdDtuc2QuaWV0ZkBnbWFpbC5jb20mZ3Q7PC9hPjxicj4NCjxiPkNjOjwvYj4g
TWljaGFlbCBUdWV4ZW4gPGEgaHJlZj0ibWFpbHRvOnR1ZXhlbkBmaC1tdWVuc3Rlci5kZSI+Jmx0
O3R1ZXhlbkBmaC1tdWVuc3Rlci5kZSZndDs8L2E+Ow0KPGEgaHJlZj0ibWFpbHRvOnRjcG0tY2hh
aXJzQGlldGYub3JnIj50Y3BtLWNoYWlyc0BpZXRmLm9yZzwvYT47IE1pcmphIEt1ZWhsZXdpbmQg
PGEgaHJlZj0ibWFpbHRvOmlldGZAa3VlaGxld2luZC5uZXQiPg0KJmx0O2lldGZAa3VlaGxld2lu
ZC5uZXQmZ3Q7PC9hPjsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZCA8YSBocmVmPSJtYWlsdG86Umlj
aGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20iPg0KJmx0O1JpY2hhcmQuU2NoZWZmZW5lZ2dl
ckBuZXRhcHAuY29tJmd0OzwvYT47IHRjcG0gSUVURiBsaXN0IDxhIGhyZWY9Im1haWx0bzp0Y3Bt
QGlldGYub3JnIj4NCiZsdDt0Y3BtQGlldGYub3JnJmd0OzwvYT48YnI+DQo8Yj5TdWJqZWN0Ojwv
Yj4gUmU6IEFjY0VDTiBmaWVsZCBvcmRlcjwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0K
PC9kaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyI+Jm5ic3A7PC9z
cGFuPjxvOnA+PC9vOnA+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1i
b3R0b206MTIuMHB0Ij5NaWNoYWVsLDxvOnA+PC9vOnA+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPk9uIDE2LzExLzIwMjAgMTc6MzYsIFNjaGFyZiwgTWljaGFlbCB3cm90ZTo8bzpw
PjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7
bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0i
bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkJvYiw8L3NwYW4+PG86cD48L286cD48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPk9uZSBw
cm9wb3NhbCB1c2luZyB0aGUgbGVuZ3RoIGZpZWxkIHdpdGggKjxiPm9uZSBvcHRpb24gY29kZXBv
aW50IG9ubHk8L2I+KiBpcyBkZXRhaWxlZCBpbjoNCjxhIGhyZWY9Imh0dHBzOi8vbWFpbGFyY2hp
dmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1IwblJmaEhvY1g4eXZUdnBDNEJOTW8vIj4N
Cmh0dHBzOi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1IwblJmaEhv
Y1g4eXZUdnBDNEJOTW8vPC9hPjwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29O
b3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4t
VVMiPiZuYnNwOzwvc3Bhbj48bzpwPjwvbzpwPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxz
cGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPkl0IGlz
IHRoZSB0aGlyZCBvcHRpb24gbWVudGlvbmVkIGluIHRoaXMgZS1tYWlsLiBPbmUgZXhhbXBsZSB3
b3VsZCBiZSB0byB1c2Ugb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNS84LzExIGZvciBvbmUgZW5jb2Rp
bmcgdHlwZSBhbmQgb3B0aW9uIGxlbmd0aCB2YWx1ZXMgNi85LzEyIGZvciB0aGUgb3RoZXIgZW5j
b2RpbmcNCiB0eXBlIChpLmUuLCBvcmRlciBvZiBmaWVsZHMpLiBPciBvbmUgY291bGQgdXNlIHNv
bWUgb3RoZXIgY29tYmluYXRpb24gb2YgbGVuZ3RoIHZhbHVlcyDigJMgdGhlIG9ubHkgcmVxdWly
ZW1lbnQgaXMgdGhhdCBhIGNlcnRhaW4gdmFsdWUgZm9yIHRoZSBvcHRpb24gbGVuZ3RoIGlzIG9u
bHkgdXNlZCBieSBvbmUgb2YgdGhlIG9wdGlvbiBmb3JtYXRzLiBJbiB0aGF0IGFwcHJvYWNoLCB0
aGUgdmFsdWUgb2YgdGhlIGxlbmd0aCBmaWVsZCB3b3VsZCB0aHVzDQogZGlyZWN0bHkgZGVzY3Jp
YmUgdGhlIGVuY29kaW5nIG9mIHRoZSBvcHRpb24uIFVubGVzcyBJIG1pc3Mgc29tZXRoaW5nLCB0
aGlzIHdvdWxkIHdvcmsgYW5kIGl0IHdvdWxkIGp1c3QgcmVxdWlyZSBvbmUgb3B0aW9uIGNvZGVw
b2ludC48L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs
YW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3Nw
YW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5UaHVzLCBhbHRlcm5hdGl2ZXMg
dG8gdHdvIG9wdGlvbiBjb2RlcG9pbnRzIGV4aXN0IGFuZCBJIGhhdmUgZXhwbGFpbmVkIHRoZW0g
b24gdGhlIGxpc3QgaW4gTWFyY2ggMjAyMC48L3NwYW4+PG86cD48L286cD48L3A+DQo8L2Jsb2Nr
cXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQpPSywgc29ycnksIHllcywgSSByZW1l
bWJlciB0aGlzIG5vdy48YnI+DQo8YnI+DQpBcyBJIHdpbGwgZXhwbGFpbiBpbiB0aGUgQWNjRUNO
IHN0YXR1cyB1cGRhdGUgdGFsayB0b2RheSBpbiB2aXJ0dWFsIEJhbmdrb2ssIHRoZSBkcmFmdCBo
YXMgbWFkZSBwcm92aXNpb24gZm9yIGRpZmZlcmVudCBsZW5ndGggdmFsdWVzIHRoYW4gNS84LzEx
LiBJdCBzYXlzIGV4aXN0aW5nIGltcGxlbWVudGF0aW9ucyBNVVNUIGFjY2VwdCBsZW5ndGggdmFs
dWVzIG90aGVyIHRoYW4gdGhvc2UgY3VycmVudGx5IGRlZmluZWQuIEJ1dCB0aGVuIHJlYWQgaW4N
CiBhcyBtYW55IHdob2xlIDMtYnl0ZSBmaWVsZHMgYXMgdGhleSBjYW4uPGJyPg0KPGJyPg0KVGhp
cyBjYW4gYmUgdXNlZCB0byBhZGQgYSBmbGFncyBieXRlIG9uIHRoZSBlbmQgaW4gZnV0dXJlLCBm
b3IgZXh0ZW5zaWJpbGl0eS4gT3IgYW55IG90aGVyIGZvcm0gb2YgZXh0ZW5zaWJpbGl0eSB0aGUg
V0cgbWlnaHQgZGVjaWRlIGluIHRoZSBmdXR1cmUuPGJyPg0KPGJyPg0KSSBrbm93IGEgZmxhZ3Mg
Ynl0ZSBhdCB0aGUgZW5kIHNlZW1zIG9kZCBjb21wYXJlZCB0byBhdCB0aGUgYmVnaW5uaW5nLiBC
dXQgKGlmIGRlY2lkZWQgaXQncyBuZWVkZWQgaW4gZnV0dXJlKSBpdCdzIHJlYXNvbmFibHkgZWFz
eSB0byBpbXBsZW1lbnQgYnkgcmVhZGluZyB0aGUgd2hvbGUgb3B0aW9uLCB0aGVuIHByb2Nlc3Np
bmcgdGhlIGxhc3QgYnl0ZSwgYmVmb3JlIHJlYWRpbmcgdGhlIHJlc3Qgb2YgdGhlIG9wdGlvbi48
YnI+DQo8YnI+DQpJIGJlbGlldmUgeW91IHdpbGwgYWdyZWUgdGhhdCB0aGlzIGlzIGEgYmV0dGVy
IHdheSB0byB1dGlsaXplIGRpZmZlcmVudCBsZW5ndGhzLjxicj4NCjxicj4NCkFuZCB0aGFuayB5
b3UgZm9yIHJlcGVhdGVkbHkgZW1waGFzaXppbmcgdGhhdCB5b3UncmUgaGFwcHkgd2l0aCB0aGUg
Mi1raW5kIHNjaGVtZSwgb3Igb3RoZXIgYWx0ZXJuYXRpdmVzLjxicj4NCjxicj4NCjxicj4NCkJv
Yjxicj4NCjxicj4NCjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPGJsb2NrcXVvdGUgc3R5
bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVO
LVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48
c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5Bbnl3
YXksIEkgZG9u4oCZdCByZWFsbHkgY2FyZSBob3cgdGhlIG9wdGlvbnMgYXJlIGVuY29kZWQgYXMg
bG9uZyBhcyB0aGUgcmVjZWl2ZXIgZG9lc27igJl0IG5lZWQgcGVyLWNvbm5lY3Rpb24gc3RhdGUg
Zm9yIGRlY29kaW5nIGEgVENQIG9wdGlvbi4gU28sIHBlcnNvbmFsbHksIEkgd291bGQgYmUgZmlu
ZSB3aXRoIHVzaW5nDQogZS5nLiB0aGUgbGVuZ3RoIGZpZWxkIGFzIGRlc2NyaWJlZCBpbiBteSBv
bGQgZS1tYWlsLiBPciBhbiBhZGRpdGlvbmFsIGZsYWcgYnl0ZS4gQW5kIG9uZSBjb3VsZCBjb21l
IHVwIHdpdGggZnVydGhlciBlbmNvZGluZ3MsIGUuZy4sIGJ5IHVzaW5nIG9uZSBvciBhIGZldyBi
aXRzIGFzIGEgc2hvcnQg4oCcdHlwZeKAnSBmaWVsZCBmb3IgZWFjaCBjb3VudGVyLiBUaGlzIGlz
IGFsbCBhYm91dCBwcm90b2NvbCBlbmdpbmVlcmluZy4gQW5kIGFsbCB0aGVzZQ0KIHZhcmlhbnRz
IGhhdmUgdGhlaXIgcHJvcyBhbmQgY29ucy48L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT
Ij5JIGFtIGFsc28gZmluZSB3aXRoIHVzaW5nIHR3byBvcHRpb24gY29kZXBvaW50cyBhcyBzcGVj
aWZpZWQgaW4gLTEzOyB0aGlzIGlzIHByb2JhYmx5IHRoZSBhcHByb2FjaCB0aGF0IGNvbnN1bWVz
IHRoZSBsZWFzdCBudW1iZXIgb2YgYml0cy48L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1
YWdlOkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVT
Ij5NaWNoYWVsICh3L28gYW55IGhhdCk8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdl
OkVOLVVTIj4mbmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFs
Ij48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4m
bmJzcDs8L3NwYW4+PG86cD48L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs
YW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4mbmJzcDs8L3Nw
YW4+PG86cD48L286cD48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpz
b2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQiPg0KPGRpdj4NCjxkaXYg
c3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29saWQgI0UxRTFFMSAxLjBwdDtwYWRkaW5n
OjMuMHB0IDBjbSAwY20gMGNtIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPkZyb206PC9iPiBC
b2IgQnJpc2NvZSA8YSBocmVmPSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+DQombHQ7aWV0
ZkBib2JicmlzY29lLm5ldCZndDs8L2E+IDxicj4NCjxiPlNlbnQ6PC9iPiBNb25kYXksIE5vdmVt
YmVyIDE2LCAyMDIwIDU6NTIgUE08YnI+DQo8Yj5Ubzo8L2I+IFlvc2hpZnVtaSBOaXNoaWRhIDxh
IGhyZWY9Im1haWx0bzpuc2QuaWV0ZkBnbWFpbC5jb20iPiZsdDtuc2QuaWV0ZkBnbWFpbC5jb20m
Z3Q7PC9hPjxicj4NCjxiPkNjOjwvYj4gU2NoYXJmLCBNaWNoYWVsIDxhIGhyZWY9Im1haWx0bzpN
aWNoYWVsLlNjaGFyZkBocy1lc3NsaW5nZW4uZGUiPiZsdDtNaWNoYWVsLlNjaGFyZkBocy1lc3Ns
aW5nZW4uZGUmZ3Q7PC9hPjsgTWljaGFlbCBUdWV4ZW4NCjxhIGhyZWY9Im1haWx0bzp0dWV4ZW5A
ZmgtbXVlbnN0ZXIuZGUiPiZsdDt0dWV4ZW5AZmgtbXVlbnN0ZXIuZGUmZ3Q7PC9hPjsgPGEgaHJl
Zj0ibWFpbHRvOnRjcG0tY2hhaXJzQGlldGYub3JnIj4NCnRjcG0tY2hhaXJzQGlldGYub3JnPC9h
PjsgTWlyamEgS3VlaGxld2luZCA8YSBocmVmPSJtYWlsdG86aWV0ZkBrdWVobGV3aW5kLm5ldCI+
Jmx0O2lldGZAa3VlaGxld2luZC5uZXQmZ3Q7PC9hPjsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZA0K
PGEgaHJlZj0ibWFpbHRvOlJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tIj4mbHQ7Umlj
aGFyZC5TY2hlZmZlbmVnZ2VyQG5ldGFwcC5jb20mZ3Q7PC9hPjsgdGNwbSBJRVRGIGxpc3QNCjxh
IGhyZWY9Im1haWx0bzp0Y3BtQGlldGYub3JnIj4mbHQ7dGNwbUBpZXRmLm9yZyZndDs8L2E+PGJy
Pg0KPGI+U3ViamVjdDo8L2I+IEFjY0VDTiBmaWVsZCBvcmRlcjxvOnA+PC9vOnA+PC9wPg0KPC9k
aXY+DQo8L2Rpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzxvOnA+PC9vOnA+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCIgc3R5bGU9Im1hcmdpbi1ib3R0b206MTIuMHB0Ij5Zb3NoaSwg
KGFkZGluZyB0aGUgdGNwbSBsaXN0IGluIGNjKTxvOnA+PC9vOnA+PC9wPg0KPGRpdj4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPk9uIDA1LzExLzIwMjAgMDY6NTgsIFlvc2hpZnVtaSBOaXNoaWRhIHdy
b3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luLXRv
cDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCI+SGkgQm9iLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+Jm5ic3A7PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8ZGl2Pg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+T24gV2VkLCBOb3YgNCwgMjAyMCBhdCAzOjI5IFBNIEJvYiBC
cmlzY29lICZsdDs8YSBocmVmPSJtYWlsdG86aWV0ZkBib2JicmlzY29lLm5ldCI+aWV0ZkBib2Ji
cmlzY29lLm5ldDwvYT4mZ3Q7IHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2tx
dW90ZSBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtw
YWRkaW5nOjBjbSAwY20gMGNtIDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi10b3A6NS4w
cHQ7bWFyZ2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIiBzdHlsZT0ibWFyZ2luLWJvdHRvbToxMi4wcHQiPllvc2hpLDxvOnA+PC9v
OnA+PC9wPg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIDA0LzExLzIwMjAgMDY6NTEs
IFlvc2hpZnVtaSBOaXNoaWRhIHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2tx
dW90ZSBzdHlsZT0ibWFyZ2luLXRvcDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+
DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SGksIGZvbGtzLDxvOnA+PC9vOnA+PC9wPg0K
PC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPC9k
aXY+DQo8L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCjxi
cj4NCjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdp
bi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0KPGRpdj4NCjxibG9ja3F1
b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAjQ0NDQ0NDIDEuMHB0O3Bh
ZGRpbmc6MGNtIDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXRvcDo1LjBw
dDttYXJnaW4tcmlnaHQ6MGNtO21hcmdpbi1ib3R0b206NS4wcHQiPg0KPGRpdj4NCjxibG9ja3F1
b3RlIHN0eWxlPSJtYXJnaW4tdG9wOjUuMHB0O21hcmdpbi1ib3R0b206NS4wcHQiPg0KPGRpdj4N
CjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JbiBteSB1bmRlcnN0YW5kaW5nLCBJJ20gbm90
IHN1cmUgaWYgd2Ugc2V0dGxlZCBkb3duIG9uIHVzaW5nIHR3byBvcHRpb24ga2luZHMgb3IgZW5j
b2Rpbmcgc2NoZW1lcyBmb3IgMjRiaXRzIGZpZWxkcyBpbiBhY2MgZWNuIGRyYWZ0LjxvOnA+PC9v
OnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+U28sIEkgdGhpbmsg
dGhlcmUncmUgc3RpbGwgc29tZXRoaW5nIHRvIGJlIGNsYXJpZmllZCBhbmQgaG9wZSB0aGluZ3Mg
d2lsbCBiZSBzZXR0bGVkIGF0IHRoZSBtZWV0aW5nLjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8
L2Rpdj4NCjwvYmxvY2txdW90ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCltCQl0gSSBr
bm93IGEgV0cgY2FuIGNoYW5nZSBpdCdzIG1pbmQgYXQgYW55IHRpbWUuIEJ1dCBJJ2QgcmF0aGVy
IHdlIGp1c3QgY2xhcmlmaWVkIHdoYXQgYSBwcmV2aW91cyBkZWNpc2lvbiB3YXMsIHRvIGF2b2lk
IHRoZSBuZWVkIHRvIGtlZXAgcmUtb3BlbmluZyBkaXNjdXNzaW9uIG9uIGEgcXVlc3Rpb24gdGhh
dCBoYXZlIGJlZW4gZGVjaWRlZCB0aGVuIGNoYW5nZWQgdGhyZWUgZGlmZmVyZW50IHdheXMgYWxy
ZWFkeS48YnI+DQo8YnI+DQpNeSBtZW1vcnkgaXMgbm90IHNvIGdvb2QgdGhlc2UgZGF5cy4gSSB0
cnVzdGVkIHRoYXQgTWljaGFlbCBTIHJlbWVtYmVyZWQgdGhlIGRlY2lzaW9uIGNvcnJlY3RseSwg
YW5kIEkgc2VlbSB0byByZW1lbWJlciB0aGF0IGRlY2lzaW9uIGJlaW5nIG1hZGUuPGJyPg0KSSd2
ZSBqdXN0IGNoZWNrZWQgdGhlIG1pbnV0ZXMgb2YgdGhlIGxhc3QgaW50ZXJpbTo8YnI+DQombmJz
cDsmbmJzcDsmbmJzcDsgPGEgaHJlZj0iaHR0cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9kb2Mv
bWludXRlcy1pbnRlcmltLTIwMjAtdGNwbS0wMS0yMDIwMDQyOTE2MDAvIiB0YXJnZXQ9Il9ibGFu
ayI+DQpodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9taW51dGVzLWludGVyaW0tMjAy
MC10Y3BtLTAxLTIwMjAwNDI5MTYwMC88L2E+PGJyPg0KYW5kIHRoZXkgbWVudGlvbiBNaWNoYWVs
J3MgcHJvcG9zYWwgdG8gdXNlIHR3byBraW5kcywgYnV0IGRvbid0IHJlY29yZCBhbnkgZGVjaXNp
b24uPGJyPg0KVGhlIGphYmJlciBsb2cgZ2l2ZXMgbm8gY2x1ZXMgYWJvdXQgYW55IGRlY2lzaW9u
Ljxicj4NCjxicj4NCkkgY2FuJ3QgZmluZCBhbiBhdWRpbyBvciB2aWRlbyByZWNvcmRpbmcuIENh
biB5b3UgcG9pbnQgbWUgYXQgb25lPzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2Jsb2NrcXVv
dGU+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+Jm5ic3A7PG86cD48L286cD48L3A+DQo8
L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5JIHRob3VnaHQgdGhhdCBpdCdzIGJl
Y2F1c2UgdGhlcmUgd2FzIG5vIGNsZWFyIGRlY2lzaW9uIGF0IHRoZSBtZWV0aW5nLiZuYnNwOzxv
OnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+QnV0LCB5
b3UgY2FuIGNoZWNrIDxhIGhyZWY9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNoP3Y9S3NC
MF9OYjgta0EiPg0KaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1Lc0IwX05iOC1rQTwv
YT48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPlBs
ZWFzZSBsZXQgdXMga25vdyBpZiB5b3UgaGF2ZSBhbnkgcXVlc3Rpb25zIG9yIG9waW5pb25zIHdp
dGggcmVnYXJkIHRvIHRoaXMuPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+
DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQpbQkJdIEkgY2hlY2tl
ZCB0aGUgWW91dHViZSBsaW5rIHlvdSBzZW50IGJlbG93Ljxicj4NCjxicj4NCkZpcnN0IEkgdGhp
bmsgd2UncmUgYWdyZWVkIG5vLW9uZSB3YXMgZmlnaHRpbmcgZm9yIHVzIHRvIGtlZXAgdGhlIHBy
ZXZpb3VzIHdheSB3ZSBkaWQgdGhpcyAodXNpbmcgdGhlIGluaXRpYWwgdmFsdWUgb2YgdGhlIGZp
ZWxkIHRvIHNldCB0aGUgb3JkZXIgZm9yIHRoZSBjb25uZWN0aW9uKS4NCjxicj4NCkluIG15IHBy
ZXNlbnRhdGlvbiBJIHNhaWQgdGhlcmUgd2FzIHN0cm9uZyByZXNpc3RhbmNlIGZyb20gTWljaGFl
bCB0byBkbyBpdCBhIGRpZmZlcmVudCB3YXkuDQo8YnI+DQooYWxzbywgb2ZmbGlzdCwgdGhlIGNv
LWF1dGhvcnMgaW5jbHVkaW5nIG1lIGFsc28gZGlkbid0IGxpa2UgdGhpcyBzbyBtdWNoLiBBbmQg
SWxwbyBzYWlkIGl0IG1hZGUgdGhlIGltcGxlbWVudGF0aW9uIGNvbXBsZXguKTxicj4NCjxicj4N
ClRoZW4gY2FtZSB0aGUgcXVlc3Rpb24gb2Ygd2hhdCB3ZSBkbyBpbnN0ZWFkLiBUaGVyZSB3ZXJl
IHRocmVlIGFsdGVybmF0aXZlIHByb3Bvc2Fsczo8YnI+DQphKSB1c2UgMiBvcHRpb24ga2luZHM8
YnI+DQpiKSBhZGQgYSBmbGFncyBieXRlPGJyPg0KYykgc29tZWhvdyB1c2UgdGhlIGxlbmd0aCBm
aWVsZCBtYXliZTxicj4NCjxicj4NCk1pY2hhZWwgcmFpc2VkIChjKSBpbiB0aGUgbWVldGluZyBh
cyBhIHBvc3NpYmlsaXR5LCBidXQgbm8tb25lIGNvdWxkIHRoaW5rIGhvdyB0byBkaXN0aW5ndWlz
aCB0d28gb3B0aW9ucyBvZiB0aGUgc2FtZSBsZW5ndGggYnV0IGEgZGlmZmVyZW50IGZpZWxkIG9y
ZGVyIHVzaW5nIHRoZSBsZW5ndGggZmllbGQuIE1pY2hhZWwgc2FpZCBoZSdkIHBvc3QgYW55IGlk
ZWFzIHRvIHRoZSBsaXN0IGlmIGhlIGNvdWxkIHRoaW5rIG9mIGFueSwgYnV0IHRoYXQNCiBkaWRu
J3QgaGFwcGVuLjxicj4NCjxicj4NClNvIHdlJ3JlIGxlZnQgY2hvb3NpbmcgYmV0d2VlbiAoYSkg
YW5kIChiKS48YnI+DQpJIHNhaWQgaW4gdGhlIG1lZXRpbmcgKGFuZCBvbiB0aGUgbGlzdCB3aGVu
IGRpc2N1c3Npbmcgd2l0aCBJbHBvKSB0aGF0IEknZCBiZSBoYXBweSB0byBnbyB3aXRoIChiKSwg
YnV0IG9ubHkgaWYgdGhlcmUgd2FzIGFub3RoZXIgdXNlIGZvciBhIGZsYWcuIEJlY2F1c2UgaXQg
d291bGQgY29uc3VtZSAxQiBtb3JlIG9wdGlvbnMgc3BhY2UgaW4gbWFueSBwYWNrZXRzLCB3aGlj
aCBpcyBhIHNjYXJjZSByZXNvdXJjZS48YnI+DQo8YnI+DQpJbHBvIGhhZCBhIHByb3Bvc2VkIHVz
ZSBmb3IgYW5vdGhlciBmbGFnICh0byBoZWxwIHN5bmNoIGNvdW50ZXJzIGFmdGVyIGEgbG9zcyks
IGJ1dCBJIHRoaW5rIHRoZSBkaXNjdXNzaW9uIGFib3V0IGl0IGVuZGVkIHRoYXQgaXQgd291bGRu
J3QgYmUgaGVscGZ1bCwgJ2NvcyB0aGUgd2F5IGl0IHdvcmtlZCBkZXBlbmRlZCBvbiBpdHNlbGYg
KGNpcmN1bGFyIGxvZ2ljKS48YnI+DQo8YnI+DQpJbiBjb25jbHVzaW9uLCBJIGRvbid0IHRoaW5r
IHRoZXJlIHdhcyBhbiBleHBsaWNpdCBkZWNpc2lvbiB0byBnbyB3aXRoIDIgb3B0aW9uIGtpbmRz
LCBidXQgaXQgZW5kZWQgdXAgYXMgdGhlICdsYXN0IHBlcnNvbiBzdGFuZGluZycuPGJyPg0KSSBs
aWtlIGl0LiBJdCdzIHNpbXBsZS4gQW5kIGFwcGFyZW50bHkgb3B0aW9uIGtpbmRzIGFyZSBub3Qg
c3VjaCBhIHNjYXJjZSByZXNvdXJjZS48YnI+DQo8YnI+DQpQZXJoYXBzIHdlIGNhbiByYXRpZnkg
dGhpcyBpbiB0aGUgV0cgdG9tb3Jyb3cuPGJyPg0KPGJyPg0KPGJyPg0KQm9iPGJyPg0KPGJyPg0K
PGJyPg0KPGJyPg0KPGJyPg0KPG86cD48L286cD48L3A+DQo8YmxvY2txdW90ZSBzdHlsZT0ibWFy
Z2luLXRvcDo1LjBwdDttYXJnaW4tYm90dG9tOjUuMHB0Ij4NCjxkaXY+DQo8ZGl2Pg0KPGRpdj4N
CjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+VGhhbmtzLDxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8
ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+LS08bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRp
dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPllvc2hpPG86cD48L286cD48L3A+DQo8L2Rpdj4NCjwv
ZGl2Pg0KPC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48YnI+DQo8
YnI+DQo8YnI+DQo8YnI+DQo8bzpwPjwvbzpwPjwvcD4NCjxwcmU+LS0gPG86cD48L286cD48L3By
ZT4NCjxwcmU+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fXzxvOnA+PC9vOnA+PC9wcmU+DQo8cHJlPkJvYiBCcmlzY29lJm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7
Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7ICZu
YnNwOzxhIGhyZWY9Imh0dHA6Ly9ib2JicmlzY29lLm5ldC8iPmh0dHA6Ly9ib2JicmlzY29lLm5l
dC88L2E+PG86cD48L286cD48L3ByZT4NCjwvZGl2Pg0KPC9ibG9ja3F1b3RlPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PGJyPg0KPGJyPg0KPGJyPg0KPG86cD48L286cD48L3A+DQo8cHJlPi0tIDxv
OnA+PC9vOnA+PC9wcmU+DQo8cHJlPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX188bzpwPjwvbzpwPjwvcHJlPg0KPHByZT5Cb2Ig
QnJpc2NvZSZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyAmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDs8YSBocmVmPSJodHRwOi8vYm9iYnJpc2NvZS5uZXQvIj5odHRwOi8v
Ym9iYnJpc2NvZS5uZXQvPC9hPjxvOnA+PC9vOnA+PC9wcmU+DQo8L2Rpdj4NCjwvYmxvY2txdW90
ZT4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxicj4NCjxicj4NCjxvOnA+PC9vOnA+PC9wPg0KPHBy
ZT4tLSA8bzpwPjwvbzpwPjwvcHJlPg0KPHByZT5fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPG86cD48L286cD48L3ByZT4NCjxw
cmU+Qm9iIEJyaXNjb2UmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgPGEgaHJlZj0iaHR0cDovL2JvYmJyaXNjb2UubmV0LyI+
aHR0cDovL2JvYmJyaXNjb2UubmV0LzwvYT48bzpwPjwvbzpwPjwvcHJlPg0KPC9kaXY+DQo8L2Rp
dj4NCjwvYm9keT4NCjwvaHRtbD4NCg==

--_000_5c3c4661887c43529b35bd5f47d10c2bhsesslingende_--


From nobody Tue Nov 17 09:50:19 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6E8953A0E8D for ; Tue, 17 Nov 2020 09:50:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.101
X-Spam-Level: 
X-Spam-Status: No, score=-2.101 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NJ7uvfq3HfGs for ; Tue, 17 Nov 2020 09:50:16 -0800 (PST)
Received: from NAM06-BL2-obe.outbound.protection.outlook.com (mail-eopbgr650113.outbound.protection.outlook.com [40.107.65.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 02B5B3A14DF for ; Tue, 17 Nov 2020 09:50:15 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F3Tz0AZEA1Vu5S3mWg7Az8MbmbzsrzmEvGqScCTe1WKrPyR/AIlgW8QADN7QmUzhpMdZOQsTGUvcufanyNGyapwqP3duMzUqG0T3e/nReFn59+pREIWIsHPfHnl1IlJu4kzMRorV77FE7EvlwzICZZtRPjRhtUVdOUSfidmduKYJilcUiD3oZ2aWFkNLyPoF2nikqpYQK/wy2050a2c2sfaTgECBBA2eqUGMutO4NymuZIUvkfoU1f5e3XzPUiD70gexFZ2FhqIf8aiaTJWgBCV3LoE0faKW4PNIpj1eC5zvmjLSwTJGWadBTmY6LcxIsYBC8b6++8Ne0eddg2O7bg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;  s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xsC8oaJ+Ohr5FyREM3MNii3V1PhTcYLp/TIzlZZWnPQ=; b=f/qajaYFsassKQcQZSGlOgFl9D04MiXHO3/dF1DRm+IYtRkH7vLpMuOZbhaKXmMOU555uDdGw2R1NNNYfqOiTdXsrS7uYmzcUR0KnTjisxuhRopO8FPjgxTiI11+O6s/RUvvw1HCMRXZGIdpBZ8MECwkk9mugz2lxBNdDef2tO3xC9G6q4x29GLoOj3X8j6BtGN1tpgQ1LjnEC4UAdNViDPpcwH8k/+yiHLx+OSx6TdSOHDUErvcUMVgW7V/XLofj7iA2IwPiOupbTjGpxCMMVeUiR1UuHGsNIietx7dKsOQJWwtRphknxPNtpLTZvefra44VTIYSPB0tvHbMwCnDA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xsC8oaJ+Ohr5FyREM3MNii3V1PhTcYLp/TIzlZZWnPQ=; b=QXu4QcPD938O6pyqasn0MKhECYb+iYlpIjiySuYkm69hW6qfvsHN50Xe0yGV1IEezGmVD25i9qpK2doyhSkSDNzrpTKZtmVZNyingLngmGsXjNk54FOqUabhma2MlWPvlL1sK7QMp2NOBwRL4pUOJpq/ZDYlGXBWZIqENgJIB60=
Received: from (2603:10b6:408:76::11) by BN3PR00MB0193.namprd00.prod.outlook.com (2a01:111:e400:c5f2::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3627.0; Tue, 17 Nov 2020 17:50:13 +0000
Received: from BN8PR00MB0627.namprd00.prod.outlook.com ([fe80::9110:3be7:364c:5bb4]) by BN8PR00MB0627.namprd00.prod.outlook.com ([fe80::9110:3be7:364c:5bb4%8]) with mapi id 15.20.3627.000; Tue, 17 Nov 2020 17:50:12 +0000
From: Praveen Balasubramanian 
To: "rs.ietf@gmx.at" , "vidhi_goel=40apple.com@dmarc.ietf.org" , "tcpm@ietf.org" 
Thread-Topic: [EXTERNAL] Re: [tcpm] Poll for CUBIC RFC updates
Thread-Index: AQHWvKs9C3JZS2KZGEC+GqZPgCwo6KnMmrcw
Date: Tue, 17 Nov 2020 17:50:12 +0000
Message-ID: 
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com> 
In-Reply-To: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=fbf432f4-7b1d-4f4d-92e9-de3d110b9677; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-11-17T17:49:41Z;  MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: gmx.at; dkim=none (message not signed) header.d=none;gmx.at; dmarc=none action=none header.from=microsoft.com;
x-originating-ip: [2001:4898:80e8:3:95d5:3f73:f682:d302]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: 4554864f-1146-41d9-7c49-08d88b213c00
x-ms-traffictypediagnostic: BN3PR00MB0193:
x-microsoft-antispam-prvs: 
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: bRqwAZBNA/QRIuktqtI/EXz9K6xGUtg5Yx5I0yElp2ayTFMdXiuzvr1tRxrsuOW2xi5OH9fx8C6rwOv/WKGH3w1WGirjAhzrYDddfF9vLQ8w+HFVRtFLd6OXgscY1RAlwhyzQj5NR8cl7NeCIomZMz+hypszvumiCtyoX9xP+em4IxBt+TgCc8sDiYmv0e0GtkOzS2lY08IUx+3UNyx1zrr5GO9SEBkVDgaJXRcNvFfoZ+6imbIPcm41pmAph0IePsJPL9Cm5mqk8hn6nQdH6bmHEmUWlOjqWZ+yPtYdaueSMriN9ZM6z+Iy8cSmwdnrFH5xyOTTjOjL9jiTs3r0E+eRzYSRCPh0LRDwRjoj5Laf2lkZEiSOwy5PnMdaQ5JMVq0v737WesW9Fdc6vzIxHw==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;  IPV:NLI; SFV:NSPM; H:BN8PR00MB0627.namprd00.prod.outlook.com; PTR:; CAT:NONE;  SFS:(4636009)(376002)(366004)(136003)(396003)(346002)(39860400002)(8990500004)(8936002)(316002)(110136005)(83380400001)(8676002)(82960400001)(33656002)(71200400001)(82950400001)(966005)(10290500003)(15650500001)(2906002)(478600001)(53546011)(6506007)(55016002)(86362001)(7696005)(186003)(9686003)(76116006)(64756008)(66556008)(66476007)(66946007)(52536014)(5660300002)(66446008); DIR:OUT; SFP:1102; 
x-ms-exchange-antispam-messagedata: =?utf-8?B?L0w2SDVPQXJ3TGNUYjQ5azNvOFQydjQ0SnQ5QkQ3clUrVDlxa2xuVXlHR3VN?= =?utf-8?B?VzFqMjZ5bnUyY2NvaDlVdzZtemJTeWJYaXpnNlEwMVpsNmZ5U0N0ZnEzcXNl?= =?utf-8?B?N2FEZXMxUFJiVFVWUjV5VWtmZ2xXK1lwQzVmQ0xiQVlaK2NNZGM1ZFVrZitH?= =?utf-8?B?bWRoTUJJMDBmSVBTSEFodFhvVVRNWWNRNC93VFp1Mm5xNG9hTm9QTGpLZnBj?= =?utf-8?B?SEMvdW9sZXVBMlV6ZXZGQjdlT3ZDODJlTFZFMFNPMXRCV1pjQ1hmQXBwcGg1?= =?utf-8?B?UGY4TGJrTlNibEdZLzYwTGlGR2JQRkRLYXBrcWd5KzJBR052SGRuUlhma2Vh?= =?utf-8?B?ZHc5aE9zcTFnbEJiV1drQzBPdUo5cUxHU2JQbVAwVVNWK1lVbWdZWWFCclE0?= =?utf-8?B?cXpsRWNqanRhRXFWWElsd0dnaGxsN3pUUnBDZUcxajc3Z0J6cHMvZHVXMUpW?= =?utf-8?B?MXFmT3BReDFtU3E0OFZ6dXNQWlhNSFBEMm9uTFQzZ3ZIWVpwbm1zZWhHNkhn?= =?utf-8?B?cHVGZW1NbGpVbFU4WTQ0T01nYWdUMmlhUDNuOVhJSyt0TzRiZ2tBSWM0NFRi?= =?utf-8?B?WVNhOG54c3lVY2dmeVNBK3lKLzFsaEorNnhpejFsNVBzeG5VTW9ZR1RsZTRG?= =?utf-8?B?QW5Uc0hXRDlNVVI4bXBYTEVhcnRKMFlFVWJzZTc4c1h1dzc4eC9zZE1yK1Vu?= =?utf-8?B?cmppblJEL2hYbmpENGpNVlgySnViSHFLS2w0NHhsdWlMbFFONHRFRjBzRG4y?= =?utf-8?B?ckZtWU1sT0pYa1ppZnhwaWlZSVNvWEQ0TG8rc0lRV2FEeW5HaWtiTnViZ1dJ?= =?utf-8?B?UXZwblZUUGdNT2FITXBiVlh5NWxWeWF3cWh4MUNDUW13WnN1aGxmRVBTVXli?= =?utf-8?B?TWxqT1hTanpjU2xGbG4zM2FRUmZkcFpEQ3BLaVd3R3YxM1RzNTBnRE9FNmNY?= =?utf-8?B?WlFJNkVlVEhFTFdmT1dxcndiTDUwenpZNzMrKzlLOGUvd3BzYUM0TktKZUNQ?= =?utf-8?B?a3NPa3Uva0o2S2lQRTYrSWY5TVBQaDdGTEZveTdGam5BWnp5RUZVZXJHSEVl?= =?utf-8?B?U2llZDROMVc5MEpBL0VXc0t0dnRxeWRQUzg3amFJTk1hWFIrZGV2eFhmTi9C?= =?utf-8?B?aXNPL2dSdHd1cHpMUDVzOUZYN2R2dXdCdnpydzZmZndESUZTcXlkMzFYc3g3?= =?utf-8?B?SWJtQjBjRUVESFpvc01ld2twdUx5dDlQaDg4aGxWSUlhbUc5TlhpVVFEb21t?= =?utf-8?B?dXpZdzhIZFdRaW1FUGlxYjMyRmNvWmRycEVzM1U4a1psMlVQVHVmMDhLSFRp?= =?utf-8?B?SmxucWFXSVVReGdwTFFIdTExVWtla3lxQVpBcys4WmZvWVRxOVRvaERUdXdB?= =?utf-8?Q?odSO58adI9XT/g7/hZ2e+vmvu7wlpCxg=3D?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN8PR00MB0627.namprd00.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4554864f-1146-41d9-7c49-08d88b213c00
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Nov 2020 17:50:12.8992 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dt10mZGkEY2OBnA6ONKw0uNgGPmR7ZLhXW/xEsFJFHTOnl6Ngutlsz84Ykwls7si4SpHpiapReWlfNo8B5paXNlib9eIg6h/lhJ280dhYgA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN3PR00MB0193
Archived-At: 
Subject: Re: [tcpm] [EXTERNAL] Re:  Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 17:50:18 -0000

QWdyZWVkLCB0aGUgYmVsb3cgY2hhbmdlcyBhcmUgdXNlZnVsIGFuZCB3ZSB3b3VsZCBhZG9wdCB0
aGVtIGZvciB0aGUgV2luZG93cyBUQ1AgaW1wbGVtZW50YXRpb24uIA0KDQotLS0tLU9yaWdpbmFs
IE1lc3NhZ2UtLS0tLQ0KRnJvbTogdGNwbSA8dGNwbS1ib3VuY2VzQGlldGYub3JnPiBPbiBCZWhh
bGYgT2YgU2NoZWZmZW5lZ2dlciwgUmljaGFyZA0KU2VudDogTW9uZGF5LCBOb3ZlbWJlciAxNiwg
MjAyMCAxMDozMCBQTQ0KVG86IFZpZGhpIEdvZWwgPHZpZGhpX2dvZWw9NDBhcHBsZS5jb21AZG1h
cmMuaWV0Zi5vcmc+OyB0Y3BtQGlldGYub3JnIEV4dGVuc2lvbnMgPHRjcG1AaWV0Zi5vcmc+DQpT
dWJqZWN0OiBbRVhURVJOQUxdIFJlOiBbdGNwbV0gUG9sbCBmb3IgQ1VCSUMgUkZDIHVwZGF0ZXMN
Cg0KV2UgaGF2ZSBiZWVuIHdvcmtpbmcgb24gaW1wcm92aW5nIHRoZSBGcmVlQlNEIEN1YmljIGlt
cGxlbWVudGF0aW9uLCB0byBicmluZyBpdCB1cCBmcm9tIHRoZSBvcmlnaW5hbGx5IGltcGxlbWVu
dGVkIChlYXJseSBkcmFmdCkgdmVyc2lvbiBvZiBDdWJpYywgdG8gYSBtb3JlIHByb2R1Y3Rpb24g
cmVhZHkgc3RhdGUgd2hpY2ggaXMgaW5saW5lIHdpdGggdGhlIGFkb3B0ZWQgUkZDIHZlcnNpb24u
DQoNCkhhdmluZyBzYWlkIHRoaXMsIHRoZSBiZWxvdyBjaGFuZ2VzIHNvdW5kIHJlYXNvbmFibGUg
YW5kIEkgd2lsbCB0cnkgdG8gYWRvcHQgdGhlbSBmb3IgRnJlZUJTRC4NCg0KUmljaGFyZA0KDQoN
CkFtIDE3LjExLjIwMjAgdW0gMDM6MDggc2NocmllYiBWaWRoaSBHb2VsOg0KPiBIZWxsbywNCj4N
Cj4gQXMgTGFycyBtZW50aW9uZWQgZWFybGllciwgdGhlcmUgYXJlIGEgY291cGxlIG9mIGlzc3Vl
cy9wcm9wb3NhbHMgDQo+IGZpbGVkIGZvciB1cGRhdGluZyB0aGUgQ1VCSUMgUkZDIDgzMTIgYW5k
IGFyZSBjdXJyZW50bHkgdW5kZXIgDQo+IGRpc2N1c3Npb24uIFRoZSBHaXRIdWIgcmVwbyBpcyBh
dCBodHRwczovL25hbTA2LnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9aHR0
cHMlM0ElMkYlMkZnaXRodWIuY29tJTJGTlRBUCUyRnJmYzgzMTJiaXMmYW1wO2RhdGE9MDQlN0Mw
MSU3Q3ByYXZiJTQwbWljcm9zb2Z0LmNvbSU3QzlkNGU1OTU2ZTFhZDRhYzM0ZmNmMDhkODhhYzIz
YWM3JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMSU3QzYzNzQxMTkx
NDY5NTMzNjU3NCU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaUxD
SlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MzMDAwJmFtcDtz
ZGF0YT1CVmpLaFdFekR6YSUyRnB1Z1l2a0l5QzRtU2VUbk90ZnVBYlMzRUloMk03YTAlM0QmYW1w
O3Jlc2VydmVkPTAuDQo+IFdlIHdvdWxkIGxpa2UgdG8gZ2F0aGVyIGluZm9ybWF0aW9uIG9uIHdo
aWNoIENVQklDIGltcGxlbWVudGF0aW9ucyANCj4gYWxyZWFkeSBvciB3b3VsZCBiZSByZWFkeSB0
byBhZG9wdCB0aGVzZSBjaGFuZ2VzLg0KPg0KPiAxLiBVcGRhdGUgSyBkZWZpbml0aW9uIChFcS4g
MikgdG8gYWNjb3VudCBmb3IgRmFzdCBDb252ZXJnZW5jZSBJc3N1ZSANCj4gbGluayANCj4gaHR0
cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNvbS8/dXJsPWh0dHBzJTNB
JTJGJTJGZ2l0aA0KPiB1Yi5jb20lMkZOVEFQJTJGcmZjODMxMmJpcyUyRmlzc3VlcyUyRjEmYW1w
O2RhdGE9MDQlN0MwMSU3Q3ByYXZiJTQwbWljDQo+IHJvc29mdC5jb20lN0M5ZDRlNTk1NmUxYWQ0
YWMzNGZjZjA4ZDg4YWMyM2FjNyU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQNCj4gN2NkMDExZGI0
NyU3QzElN0MxJTdDNjM3NDExOTE0Njk1MzM2NTc0JTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5
SldJag0KPiBvaU1DNHdMakF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENK
WFZDSTZNbjAlM0QlN0MzMDAwJmFtDQo+IHA7c2RhdGE9VzNTUjN4YmNFTHJKM3Y4MmwlMkJtZ3FY
TTJObUJ4QkJzMDU3RUZjSHlCcWk4JTNEJmFtcDtyZXNlcnZlZD0NCj4gMCBMaW51eCBhbHJlYWR5
IGhhcyB0aGlzIGNoYW5nZS4gQXBwbGUncyBDdWJpYyBpbXBsZW1lbnRhdGlvbiBmb3IgYm90aCAN
Cj4gVENQIGFuZCBRVUlDIGFscmVhZHkgdXNlIHRoaXMuDQo+DQo+IDIuIENvbmdlc3Rpb24gd2lu
ZG93IFRDUCBmcmllbmRseSByZWdpb24gYWZ0ZXIgV19tYXggSXNzdWUgbGluayANCj4gaHR0cHM6
Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNvbS8/dXJsPWh0dHBzJTNBJTJG
JTJGZ2l0aA0KPiB1Yi5jb20lMkZOVEFQJTJGcmZjODMxMmJpcyUyRmlzc3VlcyUyRjImYW1wO2Rh
dGE9MDQlN0MwMSU3Q3ByYXZiJTQwbWljDQo+IHJvc29mdC5jb20lN0M5ZDRlNTk1NmUxYWQ0YWMz
NGZjZjA4ZDg4YWMyM2FjNyU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQNCj4gN2NkMDExZGI0NyU3
QzElN0MxJTdDNjM3NDExOTE0Njk1MzM2NTc0JTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJ
ag0KPiBvaU1DNHdMakF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZD
STZNbjAlM0QlN0MzMDAwJmFtDQo+IHA7c2RhdGE9VjJudzN2S1V2NjlrVHhtJTJCZllGSVJRYldU
REtrNE51cGtxbHkzaEVuS3JNJTNEJmFtcDtyZXNlcnZlZD0NCj4gMCBUaGlzIGlzIGEgc3VnZ2Vz
dGlvbiB0byBpbXByb3ZlIHBlcmZvcm1hbmNlIGFmdGVyIGN3bmQgPiBXX21heC4gDQo+IEFwcGxl
J3MgQ3ViaWMgaW1wbGVtZW50YXRpb24gZm9yIGJvdGggVENQIGFuZCBRVUlDIGFscmVhZHkgdXNl
IHRoaXMuDQo+DQo+ICBGcm9tIG15IGV4cGVyaWVuY2UsIHRoZXNlIGNoYW5nZXMgd291bGQgaW1w
cm92ZSBDdWJpY+KAmXMgcGVyZm9ybWFuY2UuDQo+DQo+IFBsZWFzZSBsZXQgdXMga25vdyBpZiB5
b3UgYWxyZWFkeSBvciB3b3VsZCBiZSByZWFkeSB0byBpbXBsZW1lbnQgdGhlc2UgDQo+IGNoYW5n
ZXMgaW4geW91ciBDVUJJQ+KAmXMgaW1wbGVtZW50YXRpb24uDQo+DQo+IFRoYW5rcywNCj4gVmlk
aGkNCj4NCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18N
Cj4gdGNwbSBtYWlsaW5nIGxpc3QNCj4gdGNwbUBpZXRmLm9yZw0KPiBodHRwczovL25hbTA2LnNh
ZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9aHR0cHMlM0ElMkYlMkZ3d3cuDQo+
IGlldGYub3JnJTJGbWFpbG1hbiUyRmxpc3RpbmZvJTJGdGNwbSZhbXA7ZGF0YT0wNCU3QzAxJTdD
cHJhdmIlNDBtaWNyb3MNCj4gb2Z0LmNvbSU3QzlkNGU1OTU2ZTFhZDRhYzM0ZmNmMDhkODhhYzIz
YWM3JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZA0KPiAwMTFkYjQ3JTdDMSU3QzElN0M2Mzc0
MTE5MTQ2OTUzMzY1NzQlN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlKV0lqb2lNDQo+IEM0d0xq
QXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzMw
MDAmYW1wO3MNCj4gZGF0YT10Q01oMnRGRyUyQkZlSWhCc2NmUk1kdGFYbCUyRnpOY3FjR051UG9i
QTlRcXVNZyUzRCZhbXA7cmVzZXJ2ZWQ9MA0KPg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fXw0KdGNwbSBtYWlsaW5nIGxpc3QNCnRjcG1AaWV0Zi5vcmcN
Cmh0dHBzOi8vbmFtMDYuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRw
cyUzQSUyRiUyRnd3dy5pZXRmLm9yZyUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRnRjcG0mYW1wO2Rh
dGE9MDQlN0MwMSU3Q3ByYXZiJTQwbWljcm9zb2Z0LmNvbSU3QzlkNGU1OTU2ZTFhZDRhYzM0ZmNm
MDhkODhhYzIzYWM3JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMSU3
QzYzNzQxMTkxNDY5NTM0NjU2NyU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdM
akF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0Mz
MDAwJmFtcDtzZGF0YT1XT1B4Y1RIbmlFNGU5bnVOS1BwUGJBRTNPUW9yV0lJMFJzTVJyc29kMHJZ
JTNEJmFtcDtyZXNlcnZlZD0wDQo=


From nobody Tue Nov 17 09:51:27 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D6CF53A2255 for ; Tue, 17 Nov 2020 09:51:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.6
X-Spam-Level: 
X-Spam-Status: No, score=-17.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 80W4-7kyYiTY for ; Tue, 17 Nov 2020 09:51:23 -0800 (PST)
Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B421E3A2256 for ; Tue, 17 Nov 2020 09:51:22 -0800 (PST)
Received: by mail-wr1-x432.google.com with SMTP id o15so24070732wru.6 for ; Tue, 17 Nov 2020 09:51:22 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=kBpWbUTuEJkxR0cOAwZQyfHspuu9UVwmsQR6gE21k9k=; b=Uus0Y//fQ5q2KnBGNm79TVNfQVXxwmoq0ccXWdH+osIFiAQsSDsseeUIBLNykKNMO5 W1AyqJZKXUKgxPKWOnuBracZJbkTbiddZ5ieqS1KixCvp6VFHZJrtLy/BLfKgxp3JYGT bpfugJOOLqvRCUVHCL9470IIlSgXcTFTMXc4wZyRkgX7whHtluOQyihYxvUxGgnXohWq abZT7ZQ9BLM8xANPGaJRxuwvTnVAtaypSx0kso8dP0vtvRz2cYSWwTpNbQh5k9UUWX7W wx3LRHg4fBqNH1+2phA5znfpRFcvx5Hk1OovUxPPcBksnnhUOQfMqpqB4cRj60xT5sdN DMdA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=kBpWbUTuEJkxR0cOAwZQyfHspuu9UVwmsQR6gE21k9k=; b=lfOvD0jl3cToD+rCAZMGAoZpt8aWIlJjxbOh0TGb4MHLnhLxQbdgSVDUY9/56wGpHP y0VzwF6nTtUOShOWQUm5a6Ja0Wvq4F6UmZDSLz1mBwEz0Z1KG8+tONTSj8+CRsn5mWke B+qKKxmm4Vn1kqnQ4cxUEWDmLwWIhaAaANilaWuKi/dVCcdKLpvles6pqHX55VHlPqHW bm0GD3t7jnJwMY3/il6F4mXq/PZeh16d8vlSiwSv7lGefolMxMM5gxsxEKxM0A+59xSm iWd9R1joC49BV6dKPGr/KicqNlx1OhtRuZF+jEvXU/tNOvHaFCDLxSOg1nbNXXyesbTI kiGQ==
X-Gm-Message-State: AOAM533IC6jkCuKSKsZyHF6hyI1uS25KeiAar0ojcO0ClRKmdGkOP70n eHlQkmlgS3eQrN+TLgXd6PRFtPnJxQ+VgTiFnknJ0A==
X-Google-Smtp-Source: ABdhPJwSTMAZTeGItBTZgUelvQrv6JqKQqcrWIHd8aLewJbLdkVnFtpoghVRsHPNefn6TaAaxBgTBQ+X08qnKOYwzPc=
X-Received: by 2002:a5d:4f92:: with SMTP id d18mr777023wru.118.1605635480955;  Tue, 17 Nov 2020 09:51:20 -0800 (PST)
MIME-Version: 1.0
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com> 
In-Reply-To: 
From: Yuchung Cheng 
Date: Tue, 17 Nov 2020 09:50:43 -0800
Message-ID: 
To: "Scheffenegger, Richard" 
Cc: Vidhi Goel ,  "tcpm@ietf.org Extensions" , Neal Cardwell 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 17:51:26 -0000

On Mon, Nov 16, 2020 at 10:30 PM Scheffenegger, Richard  wr=
ote:
>
> We have been working on improving the FreeBSD Cubic implementation, to
> bring it up from the originally implemented (early draft) version of
> Cubic, to a more production ready state which is inline with the adopted
> RFC version.
>
> Having said this, the below changes sound reasonable and I will try to
> adopt them for FreeBSD.
>
> Richard
>
>
> Am 17.11.2020 um 03:08 schrieb Vidhi Goel:
> > Hello,
> >
> > As Lars mentioned earlier, there are a couple of issues/proposals filed
> > for updating the CUBIC RFC 8312 and are currently under discussion. The
> > GitHub repo is at https://github.com/NTAP/rfc8312bis.
> > We would like to gather information on which CUBIC implementations
> > already or would be ready to adopt these changes.
> >
> > 1. Update K definition (Eq. 2) to account for Fast Convergence
> > Issue link https://github.com/NTAP/rfc8312bis/issues/1
> > Linux already has this change. Apple's Cubic implementation for both TC=
P
> > and QUIC already use this.
> >
> > 2. Congestion window TCP friendly region after W_max
> > Issue link https://github.com/NTAP/rfc8312bis/issues/2
> > This is a suggestion to improve performance after cwnd > W_max. Apple's
> > Cubic implementation for both TCP and QUIC already use this.

I'd recommend replacing the modelled TCP Reno window approach in
section 4.2 with an AIMD emulation (Linux's approach).

In our experience, TCP-friendly regions are the predominant mode of
(Linux) Cubic for any regular Internet connection. IOW Cubic is often
"Reno" unless the loss rate is abysmal. The modelled approach is based
on a simple bulk transfer where modern network applications are mostly
structured traffic (burst, idle, repeat). Under such traffic
structures the model has two issues:

The model assumes cwnd overshoot causes losses that are repaired in
one round of fast recovery. In reality, the losses are often due to
bursts to short messages, causing more rounds and even timeouts to
repair. So the overall loss rate "p" tends to be higher than the ideal
model, causing the model to underestimate the window (hence runs in a
more conservative Reno). Instead Linux's approach is to simply emulate
Reno AIMD based on the number of packets per ACK. This also avoids
square-root operation.

>
> >
> >  From my experience, these changes would improve Cubic=E2=80=99s perfor=
mance.
> >
> > Please let us know if you already or would be ready to implement these
> > changes in your CUBIC=E2=80=99s implementation.
> >
> > Thanks,
> > Vidhi
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpm
> >
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Tue Nov 17 11:41:21 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 430363A1588 for ; Tue, 17 Nov 2020 11:41:19 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kmpLfB8iaS14 for ; Tue, 17 Nov 2020 11:41:17 -0800 (PST)
Received: from mx43-out1.antispamcloud.com (mx43-out1.antispamcloud.com [138.201.61.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8EF9E3A1595 for ; Tue, 17 Nov 2020 11:41:17 -0800 (PST)
Received: from xse143.mail2web.com ([66.113.196.143] helo=xse.mail2web.com) by mx13.antispamcloud.com with esmtp (Exim 4.92) (envelope-from ) id 1kf6qf-0005a6-TC for tcpm@ietf.org; Tue, 17 Nov 2020 20:41:11 +0100
Received: from xsmtp21.mail2web.com (unknown [10.100.68.60]) by xse.mail2web.com (Postfix) with ESMTPS id 4CbGWz0FvPz2lTQ for ; Tue, 17 Nov 2020 11:41:03 -0800 (PST)
Received: from [10.5.2.15] (helo=xmail05.myhosting.com) by xsmtp21.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1kf6qc-0004ep-TM for tcpm@ietf.org; Tue, 17 Nov 2020 11:41:02 -0800
Received: (qmail 16105 invoked from network); 17 Nov 2020 19:41:02 -0000
Received: from unknown (HELO [192.168.1.103]) (Authenticated-user:_huitema@huitema.net@[172.58.43.45]) (envelope-sender ) by xmail05.myhosting.com (qmail-ldap-1.03) with ESMTPA for ; 17 Nov 2020 19:41:02 -0000
To: Yuchung Cheng , "Scheffenegger, Richard" 
Cc: Vidhi Goel , "tcpm@ietf.org Extensions" 
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>  
From: Christian Huitema 
Message-ID: 
Date: Tue, 17 Nov 2020 11:41:02 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.3
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-Originating-IP: 66.113.196.143
X-Spampanel-Domain: xsmtpout.mail2web.com
X-Spampanel-Username: 66.113.196.143/32
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.196.143/32@xsmtpout.mail2web.com
X-Spampanel-Outgoing-Class: unsure
X-Spampanel-Outgoing-Evidence: Combined (0.15)
X-Recommended-Action: accept
X-Filter-ID: Mvzo4OR0dZXEDF/gcnlw0cThwnZT+ODcFyeCwCHoUjupSDasLI4SayDByyq9LIhVUZbR67CQ7/vm /hHDJU4RXkTNWdUk1Ol2OGx3IfrIJKywOmJyM1qr8uRnWBrbSAGDPMbZapeW47ess9aQHBKcoLgN zB/4Jkrw1eDLcif59ftMwqPalDEHeRFUVbPHbSjkU7Tmz6iKnkQL9gqsxD347235Nhqq+/HvroPq 8GSPg+5hmwN8D4LrepG7AX8WNwY8PLhBThvdgyPN49yzDQzRHY6jSvfpO+1kZkomjtjB6X5Q5Q9f RUeIpTIC2ySfqvnqLwoxlgatmaBb0rBiK9xbkDrUqzcKIief90MVLZY9LbIZh9+IQ1oS9LBn3VIP 95Jz7ujRlJ9wSMlhvaudJXZ9EIBG/qaR+8r9SKFMmPJLf850OvZYsmoVQuOIhwKLK6IKBNB4LZ0v UHHKTzJX7b1JhLSQQ4vSj0QEim26t/Moy0UPX5E73H1QfrH/5kkrV/Cr0bm2vWdo8usP65i82q1C dZgGrpL44wdx9eXqjQjbvUopOMQJvQ/Ck3iiU+4DQAj3fuQgzT3K9JUHTNiGwfwAmxx/Wk8McinP JEkgAVrOMpacjGLPDGJdOe4i3f4PkPX9cQAgjUq0xsmHmKfJV7oVg2pPsO2a9YK0ZMeLL6baim4M 7OYFXYdC3tRq275m/U3VF32g17ip7E0k1FBlZj4ChLdCp3Zd9clP8wSiJZWbJCj+xRrjVmRxpGtS cvUmgj1Lb4oCRWJNnW/6AGTatscsYl2jZAOanSBpz6Rja2u/0jKpLh8hxKVgQEBahnMCJUbumhJa XJp6Dyip+Pa027T+3/ysta6u1iHEyuS7GD1uvcruwMFbs9NDYQONehnXkUbcJOYIJd4MvQ0Nf4Ec bvHO1diDanHV9KirFAIIecsyj+YNTo81GR+jDXFsz/ZQnbbTizvwlZsrbltGiZoUh+c+5pFVgpT1 b21uZVckGp0ccOa2XhkGbmsUNPNkere1WheNsVXmhO8BzADiszcWR9bz/SDtF09JpSbuuCeiIDK0 C/0=
X-Report-Abuse-To: spam@quarantine11.antispamcloud.com
Archived-At: 
Subject: Re: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 17 Nov 2020 19:41:19 -0000

On 11/17/2020 9:50 AM, Yuchung Cheng wrote:
> On Mon, Nov 16, 2020 at 10:30 PM Scheffenegger, Richard  wrote:
>> We have been working on improving the FreeBSD Cubic implementation, to
>> bring it up from the originally implemented (early draft) version of
>> Cubic, to a more production ready state which is inline with the adopted
>> RFC version.
>>
>> Having said this, the below changes sound reasonable and I will try to
>> adopt them for FreeBSD.
>>
>> Richard
>>
>>
>> Am 17.11.2020 um 03:08 schrieb Vidhi Goel:
>>> Hello,
>>>
>>> As Lars mentioned earlier, there are a couple of issues/proposals filed
>>> for updating the CUBIC RFC 8312 and are currently under discussion. The
>>> GitHub repo is at https://github.com/NTAP/rfc8312bis.
>>> We would like to gather information on which CUBIC implementations
>>> already or would be ready to adopt these changes.
>>>
>>> 1. Update K definition (Eq. 2) to account for Fast Convergence
>>> Issue link https://github.com/NTAP/rfc8312bis/issues/1
>>> Linux already has this change. Apple's Cubic implementation for both TCP
>>> and QUIC already use this.
>>>
>>> 2. Congestion window TCP friendly region after W_max
>>> Issue link https://github.com/NTAP/rfc8312bis/issues/2
>>> This is a suggestion to improve performance after cwnd > W_max. Apple's
>>> Cubic implementation for both TCP and QUIC already use this.
> I'd recommend replacing the modelled TCP Reno window approach in
> section 4.2 with an AIMD emulation (Linux's approach).
I second that. That's exactly what I ended up doing when implementing
Cubic from the RFC, and realizing that the emulation approach was very
hard to make work correctly.
>
> In our experience, TCP-friendly regions are the predominant mode of
> (Linux) Cubic for any regular Internet connection. IOW Cubic is often
> "Reno" unless the loss rate is abysmal. The modelled approach is based
> on a simple bulk transfer where modern network applications are mostly
> structured traffic (burst, idle, repeat). Under such traffic
> structures the model has two issues:
>
> The model assumes cwnd overshoot causes losses that are repaired in
> one round of fast recovery. In reality, the losses are often due to
> bursts to short messages, causing more rounds and even timeouts to
> repair. So the overall loss rate "p" tends to be higher than the ideal
> model, causing the model to underestimate the window (hence runs in a
> more conservative Reno). Instead Linux's approach is to simply emulate
> Reno AIMD based on the number of packets per ACK. This also avoids
> square-root operation.

That, and another issue caused by the use of RTT estimates in the RENO
modelled presented in the RFC. If there is jitter on the network, the
RTT estimates are going to fluctuate quite a bit, and the modelling will
amplify those fluctuations.

-- Christian Huitema


From nobody Tue Nov 17 16:04:27 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 568213A11DC for ; Tue, 17 Nov 2020 16:04:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.599
X-Spam-Level: 
X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m1aleCHYEkfi for ; Tue, 17 Nov 2020 16:04:17 -0800 (PST)
Received: from mail-ua1-x92a.google.com (mail-ua1-x92a.google.com [IPv6:2607:f8b0:4864:20::92a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2D8683A117B for ; Tue, 17 Nov 2020 16:04:11 -0800 (PST)
Received: by mail-ua1-x92a.google.com with SMTP id x13so89374uar.4 for ; Tue, 17 Nov 2020 16:04:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=clshCXq+Jf3lIJ3n+JkRtgp9/ycHeEcFkP1nLfWFk4s=; b=L7WgThhij2eLpHNfja74ySjGjT7A4KMrYu3C4VmTKalhFTxiD3GkSVvFlpU4vSgnar udMopHdt7dHuABpTpakxnhTt1ZIoTEq+leyQnQLeJ3ggxi+WenGhXrtIYhBpfz6e4Uxj pvD7KcsM+pHH1s3+MXCMlxhcJIomuqftir1iNZfpp+DC6jFE48Vuuri8FGIMhtf1WQyN qDhsKAtBllpUzUvymcQD/SFMBQ5GOoUc0d30wvweyk9aKPicvEgZ2zBnBY1s+yGnkCNH T3dTfmrwflaVjWLbCM40rCBuaPEkNttULyZMmjDvb6Hn2AeZeaSeqraYX75AtA+2P/IY nJEA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=clshCXq+Jf3lIJ3n+JkRtgp9/ycHeEcFkP1nLfWFk4s=; b=mkRw1gGz2BaBWgH9QOvXVkDwopQ62KdFJA7nCAbdUQzrH98ZAGCw/SM0iSLI9U84vw WC0Jg6CZpRTxSVs8Ak+E+wLxK5Z/315ZsolFJ1VEot5swVZojSghCyzSu1pUKB7bWAju LNoKM2Iy2peOlyIgqF0JHkrJeRXkZMjm5T8E5ExNzmSyEZsS2ctHiGts2LuhF0td2O/J 2XMPAEWDckvIdc+XZwkrG4PT+E+vRuf/3RdrDjg/OECGdSfBTdtP8jS5TmgWCqnUhI7I B27D4noajpiH5NFcPekaDmgeVwKNpiwOi42/B1Xrpv4tT4QM9TLdT4YHC1Rt1TCpQbMm //+w==
X-Gm-Message-State: AOAM533laIDsfCC3HCMOkKW6Is/COQPjH51gccNFDMSRSDaUUKt6E8Zu is0cXldS86gcjSYwAz+hApK3iazFH6LDL22kLW+Fog==
X-Google-Smtp-Source: ABdhPJysj3jQ7ujKXyIGqpOrBtt+l4peIutP2aVQ23yta4Ahi8kNeXCkBbabwvDzEcoRcRN6WktZLpXbDw8/stgNsMU=
X-Received: by 2002:ab0:254b:: with SMTP id l11mr1733755uan.63.1605657849676;  Tue, 17 Nov 2020 16:04:09 -0800 (PST)
MIME-Version: 1.0
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de> <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net> <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>
In-Reply-To: <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>
From: Neal Cardwell 
Date: Tue, 17 Nov 2020 19:03:53 -0500
Message-ID: 
To: "Scharf, Michael" 
Cc: Bob Briscoe , Yoshifumi Nishida ,  tcpm IETF list , "Scheffenegger, Richard" ,  Michael Tuexen , Mirja Kuehlewind , "tcpm-chairs@ietf.org" 
Content-Type: multipart/alternative; boundary="000000000000c33fae05b45659f0"
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 18 Nov 2020 00:04:25 -0000

--000000000000c33fae05b45659f0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

As someone who contributes to a TCP implementation, I would put in a  vote
for the "Two Option Kinds" proposal outlined in slide 6 here:


https://datatracker.ietf.org/meeting/109/materials/slides-109-tcpm-more-acc=
urate-ecn-feedback-in-tcp-ecn-adding-explicit-congestion-notification-01#pa=
ge=3D6

This approach has the virtue of simplicity, which hopefully may help avoid
middlebox bugs, speed standardization and development, and ease the process
of maintaining fast and stable implementations.

neal


On Tue, Nov 17, 2020 at 9:25 AM Scharf, Michael <
Michael.Scharf@hs-esslingen.de> wrote:

> Inline some comments [ms].
>
>
>
> Given that I don=E2=80=99t ask for any text chances, I=E2=80=99ll not fol=
low-up forever in
> this discussion. I don=E2=80=99t own running code and a lot of encoding d=
etails
> mainly affect an implementation.
>
>
>
> IMHO the rest of the WG has to think about the option design more than me=
.
> So, this is mostly about triggering a discussion=E2=80=A6
>
>
>
> *From:* Bob Briscoe 
> *Sent:* Tuesday, November 17, 2020 1:20 PM
> *To:* Scharf, Michael ; Yoshifumi Nishida
> 
> *Cc:* Michael Tuexen ; tcpm-chairs@ietf.org; Mirja
> Kuehlewind ; Scheffenegger, Richard <
> Richard.Scheffenegger@netapp.com>; tcpm IETF list 
> *Subject:* Re: AccECN field order
>
>
>
> Michael,
>
> On 17/11/2020 08:10, Scharf, Michael wrote:
>
> Bob,
>
>
>
> I am fine with the two option kinds proposed in -13, but I don=E2=80=99t =
buy your
> arguments why this encoding is better than others.
>
>
>
> Most importantly, I don=E2=80=99t think that your =E2=80=9Cforward compat=
ibility=E2=80=9D argument
> is a very compelling reason for two codepoints. All proposals I am aware =
of
> have their pros and cons. And I am not aware of a really comprehensive
> discussion. At least neither the e-mail below nor today=E2=80=99s meeting=
 discuss
> all tradeoffs I would investigate.
>
>
> [BB] You're right - the discussion below is useful and necessary.
>
>
>
>
> For instance, the benefit of the length encoding variant would be to
> consume only one TCP option codepoint now. If we decided later to that a
> flag byte is needed (say, at the end), a follow-up proposed standard coul=
d
> specify another option format with that flag byte, using a second codepoi=
nt.
>
>
> [BB]
> * The new codepoint would require years to deploy (including in proxies)
> before it works e2e.
>
>
>
> [ms] That may depend **a lot** on the environment. In environments where
> ECN can get deployed today, say in datacenters, rolling out a new kernel
> version may perhaps not be the biggest operational challenge. Also, proxi=
es
> may not be a significant problem in some of those cases. Deployment
> roadmaps are very hard to predict.
>
>
> * Whereas the 'forward compatibility' approach tries to ensure that AccEC=
N
> implementations that are being built today will be ready to interwork wit=
h
> a future extension.
> * In contrast, if your hypothetical new AccECN codepoint did need to be
> deployed later, it would need to include a fall-back to the current AccEC=
N
> option when one peer doesn't understand the new codepoint.
>
>
>
> [ms] This is a reasonable point, albeit not a particularly strong one,
> given that the option is not really required for basic interoperability.
> Also, a hypothetical follow-up standard could include a way to explicitly
> request the option=E2=80=A6
>
>
> * Fall-back would need to avoid a second round of handshake (given this i=
s
> for low latency), which would only work reliably with an option on the SY=
N
> (because the 3rd ACK is unreliable)
>
>
>
> [ms] I can=E2=80=99t follow. If an endpoint is unhappy with getting AccEC=
N option
> =E2=80=9CA=E2=80=9D from a peer, I could send a =E2=80=9Cplease send me t=
he better AccECN option B=E2=80=9D
> after the 3-WHS. It could be an upgrade, not a fallback. Also, at this
> point we don=E2=80=99t know what option B would be and in which context i=
t would be
> needed. Maybe other protocol features would require negotiation, too. IMH=
O
> it is a solvable problem.
>
>
> * AccECN has so far avoided using any SYN option space, which is importan=
t
> given that is the scarcest resource.
>
>
>
> [ms] Yep, but as long as no proposal needs SYN options, this is nothing
> that would make one solution better than another.
>
> These days, extending TCP is like playing chess - we have to think
> multiple moves ahead!
>
>
>
> [ms] There is one fundamental difference between chess and TCP: In chess
> you precisely know the current situation and can predict all potential
> moves in advance (albeit there are a lot). However, we don=E2=80=99t know=
 the
> future evolution of TCP, ECN and the Internet as a whole, nor future
> requirements. If we try to engineer a TCP extension for some potential
> future, we may figure out later that reality is different from what we
> designed for. As much as I appreciate your attempts to predict the future=
,
> it would be a big surprise to me if you could correctly predict what TCP
> will be in 10 years from now. So, instead of trying to write specificatio=
ns
> that take into account future requirements, which are in fact just unknow=
n,
> it is more important for me to write effective, efficient and secure
> specifications for what is currently known for sure.
>
>
>
> [ms] I **don=E2=80=99t** know for sure whether the current AccECN option =
will be
> needed, but I see quite some evidence that support of the AccECN option(s=
)
> is not a top priority these days. That is also something to think about
> when designing the details.
>
>
>
> I need to learn to articulate onto the list all my reasons for rejecting
> certain parts of the design space. Sorry about that.
>
> Yes, I admit, the current proposal is at the cost of burning two TCP
> option codepoints. However, you said yourself when you proposed this
> approach, that they are not (currently) a scarce resource. Certainly not =
as
> scarce as option space. I just checked: roughly 15% of the option codepoi=
nt
> allocation space has been used so far. So that's where I would suggest we
> compromise first.
>
> [ms] I made a similar calculation some years ago and came to a similar
> conclusion. Investing two codepoints for AccECN is something we can
> probably afford. But nonetheless we have to reason carefully why it is
> really worth the effort.
>
>
>
> Regarding the resulting codepoint consumption (two option kinds) and the
> addition of a flag byte, this approach would basically end up with the sa=
me
> result as the current proposal in -13 plus some hypothetical future
> addition leveraging the length field beyond the currently known use.
>
>
>
> And, yes, the length encoding variant may be less flexible and/or consume
> some more bits in some cases, but on the plus side it only needs one
> codepoint now =E2=80=93 and given that it is entirely unclear whether any=
 AccECN
> option will indeed be widely used in future, this is a big plus. The fact
> that implementers are quite silent on the option design is not a good sig=
n.
>
>
> [BB] Here's another possibility then.
>
> a) [K|L| EE1B | ECEB | EE0B ]
> b) [K|L| EE1B | ECEB | EE0B |F]    if F =3D 0bXXXXXXX0
> c) [K|L| EE0B | ECEB | EE1B |F]    if F =3D 0bXXXXXXX1
>
> We'd have to decide which order alternative (a) takes. I've picked EE1B
> first only 'cos you're right that there is not currently as much pressure
> (on the public Internet) for AccECN with ECT(0).
>
> =3D=3DLegend=3D=3D
> K:    Kind (1B)
> L:    Length (1B)
> ExxB: Echo xx byte counter
> F:    Flags (1B)
>
> Alternatives a) and b) are equivalent, at least while the only allocated
> flag is for field order.
> However, alternative b) is available in case other flags are allocated in
> future
> The flags byte can only be omitted if (L % 3 =3D=3D 2).
> The flags byte is considered present if (L % 3) =3D=3D 0.
>
> Forward compatibility: Options with (L % 3 =3D=3D 1) MUST be assumed to
> include a flags byte, and current implementations ignore the last byte.
>
> The flags byte is optional to implement (even if the AccECN option is
> implemented)
> If the server includes a flags byte on the SYN/ACK but the client does no=
t
> include one on the 3rd ACK or the first data packet, the server assumes t=
he
> client does not implement the flags byte and uses only alternative a) for
> the remainder of the connection.
>
>
> [ms] I am not sure if we need that, but I take this as existence proof
> that other encodings are possible. Personally, I think that other solutio=
ns
> would be doable if you took one bit from one or multiple counters for
> format encoding. At least for some of the byte counters I don=E2=80=99t t=
hink
> taking one bit for other purposes would result in an Internet congestion
> collapse=E2=80=A6 But, I have already said that the -13 proposal works fo=
r me, so
> I=E2=80=99ll not try to come up with another variant myself.
>
>
>
> To me, one potential difference between two proposals would be incrementa=
l
> deployment. The proposal in -13 only has an advantage if middleboxes such
> as firewalls will indeed pass TCP options with a format that contains
> content beyond the (first) Accurate ECN standard (i.e., currently unused
> length values). IMHO it is too early to know whether firewalls would inde=
ed
> allow this in future.
>
>
>
> From a security perspective, it is not clear to me whether allowing
> arbitrary unspecified bytes in a TCP option is a good idea **at all**. It
> will be interesting to hear the opinion from SEC area on that. Personally=
,
> I am not convinced that this really makes sense, but I my concerns are no=
t
> strong enough to formally push back. I=E2=80=99ll leave it to others to t=
hink about
> whether this is a bug or a feature.
>
>
> [BB] Well, let's first try to deal with security ourselves:
> * Octets that are explicitly declared as part of an option cannot be used
> for a buffer overflow attack. I don't really need to, but I could add the
> following text to the forward compatibility wording:
>     A receiver considers octets beyond those it uses, but within the
> specified length, as if they are padding.
> * And such octets cannot be any different from the current ability of a
> sender to add padding. So there's no new attack possible here.
> * And there's no need to specify a max length for any AccECN Option, whic=
h
> would just unnecessarily limit flexibility.
>
> [ms] I have more thought about the covert channel that some unspecified
> bytes in a standards track TCP option could offer=E2=80=A6 In particular =
if the
> AccECN option gets widely deployed and used, would some malicious users
> find those bytes useful? For what? For instance, what in the AccECN
> protocol design prevents use of these =E2=80=9Cpadding=E2=80=9D bytes as =
super-cookie? To
> me, it is much more important to think about abuse of whatever we
> standardize **now**, instead of engineering for some entirely unknown
> future. I am not a security expert, but I suggest a careful analysis of a=
ny
> security implications of the AccECN option. We seldomly specify new TCP
> options and malicious users will certainly look for =E2=80=9Copportunitie=
s=E2=80=9D in the
> spec. So far, I don=E2=80=99t know whether that whole idea of forward com=
patibility
> is a feature or a bug.
>
>
>
> Michael
>
>
>
> Maybe one lesson learnt is that the document could have a non-normative
> appendix that explains the rationale for the finally picked TCP option
> encoding. That may also help if there are further questions whether two
> codepoints are really required, e.g. by the IESG (if two codepoints are
> still the design after WGLC). At least for past TCP option codepoint
> allocations I recall some discussions late in the IETF process. In those
> past cases, good arguments in an appendix and running code has helped a l=
ot.
>
>
> [BB] I can do that. Appx B is already similar, giving Design Rationale fo=
r
> the ACE field.
>
>
> Bob
>
>
>
>
> Michael
>
>
>
>
>
>
>
> *From:* Bob Briscoe  
> *Sent:* Tuesday, November 17, 2020 6:10 AM
> *To:* Scharf, Michael 
> ; Yoshifumi Nishida 
> 
> *Cc:* Michael Tuexen  ;
> tcpm-chairs@ietf.org; Mirja Kuehlewind 
> ; Scheffenegger, Richard
>  ;
> tcpm IETF list  
> *Subject:* Re: AccECN field order
>
>
>
> Michael,
>
> On 16/11/2020 17:36, Scharf, Michael wrote:
>
> Bob,
>
>
>
> One proposal using the length field with **one option codepoint only** is
> detailed in:
> https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>
>
>
> It is the third option mentioned in this e-mail. One example would be to
> use option length values 5/8/11 for one encoding type and option length
> values 6/9/12 for the other encoding type (i.e., order of fields). Or one
> could use some other combination of length values =E2=80=93 the only requ=
irement is
> that a certain value for the option length is only used by one of the
> option formats. In that approach, the value of the length field would thu=
s
> directly describe the encoding of the option. Unless I miss something, th=
is
> would work and it would just require one option codepoint.
>
>
>
> Thus, alternatives to two option codepoints exist and I have explained
> them on the list in March 2020.
>
>
> OK, sorry, yes, I remember this now.
>
> As I will explain in the AccECN status update talk today in virtual
> Bangkok, the draft has made provision for different length values than
> 5/8/11. It says existing implementations MUST accept length values other
> than those currently defined. But then read in as many whole 3-byte field=
s
> as they can.
>
> This can be used to add a flags byte on the end in future, for
> extensibility. Or any other form of extensibility the WG might decide in
> the future.
>
> I know a flags byte at the end seems odd compared to at the beginning. Bu=
t
> (if decided it's needed in future) it's reasonably easy to implement by
> reading the whole option, then processing the last byte, before reading t=
he
> rest of the option.
>
> I believe you will agree that this is a better way to utilize different
> lengths.
>
> And thank you for repeatedly emphasizing that you're happy with the 2-kin=
d
> scheme, or other alternatives.
>
>
> Bob
>
>
>
>
>
> Anyway, I don=E2=80=99t really care how the options are encoded as long a=
s the
> receiver doesn=E2=80=99t need per-connection state for decoding a TCP opt=
ion. So,
> personally, I would be fine with using e.g. the length field as described
> in my old e-mail. Or an additional flag byte. And one could come up with
> further encodings, e.g., by using one or a few bits as a short =E2=80=9Ct=
ype=E2=80=9D field
> for each counter. This is all about protocol engineering. And all these
> variants have their pros and cons.
>
>
>
> I am also fine with using two option codepoints as specified in -13; this
> is probably the approach that consumes the least number of bits.
>
>
>
> Michael (w/o any hat)
>
>
>
>
>
>
>
> *From:* Bob Briscoe  
> *Sent:* Monday, November 16, 2020 5:52 PM
> *To:* Yoshifumi Nishida  
> *Cc:* Scharf, Michael 
> ; Michael Tuexen 
> ; tcpm-chairs@ietf.org; Mirja Kuehlewind
>  ; Scheffenegger, Richard
>  ;
> tcpm IETF list  
> *Subject:* AccECN field order
>
>
>
> Yoshi, (adding the tcpm list in cc)
>
> On 05/11/2020 06:58, Yoshifumi Nishida wrote:
>
> Hi Bob,
>
>
>
> On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe  wrote:
>
> Yoshi,
>
> On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>
> Hi, folks,
>
>
>
>
>
> In my understanding, I'm not sure if we settled down on using two option
> kinds or encoding schemes for 24bits fields in acc ecn draft.
>
> So, I think there're still something to be clarified and hope things will
> be settled at the meeting.
>
>
> [BB] I know a WG can change it's mind at any time. But I'd rather we just
> clarified what a previous decision was, to avoid the need to keep
> re-opening discussion on a question that have been decided then changed
> three different ways already.
>
> My memory is not so good these days. I trusted that Michael S remembered
> the decision correctly, and I seem to remember that decision being made.
> I've just checked the minutes of the last interim:
>
> https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-20200429160=
0/
> and they mention Michael's proposal to use two kinds, but don't record an=
y
> decision.
> The jabber log gives no clues about any decision.
>
> I can't find an audio or video recording. Can you point me at one?
>
>
>
> I thought that it's because there was no clear decision at the meeting.
>
> But, you can check https://www.youtube.com/watch?v=3DKsB0_Nb8-kA
>
> Please let us know if you have any questions or opinions with regard to
> this.
>
>
> [BB] I checked the Youtube link you sent below.
>
> First I think we're agreed no-one was fighting for us to keep the previou=
s
> way we did this (using the initial value of the field to set the order fo=
r
> the connection).
> In my presentation I said there was strong resistance from Michael to do
> it a different way.
> (also, offlist, the co-authors including me also didn't like this so much=
.
> And Ilpo said it made the implementation complex.)
>
> Then came the question of what we do instead. There were three alternativ=
e
> proposals:
> a) use 2 option kinds
> b) add a flags byte
> c) somehow use the length field maybe
>
> Michael raised (c) in the meeting as a possibility, but no-one could thin=
k
> how to distinguish two options of the same length but a different field
> order using the length field. Michael said he'd post any ideas to the lis=
t
> if he could think of any, but that didn't happen.
>
> So we're left choosing between (a) and (b).
> I said in the meeting (and on the list when discussing with Ilpo) that I'=
d
> be happy to go with (b), but only if there was another use for a flag.
> Because it would consume 1B more options space in many packets, which is =
a
> scarce resource.
>
> Ilpo had a proposed use for another flag (to help synch counters after a
> loss), but I think the discussion about it ended that it wouldn't be
> helpful, 'cos the way it worked depended on itself (circular logic).
>
> In conclusion, I don't think there was an explicit decision to go with 2
> option kinds, but it ended up as the 'last person standing'.
> I like it. It's simple. And apparently option kinds are not such a scarce
> resource.
>
> Perhaps we can ratify this in the WG tomorrow.
>
>
> Bob
>
>
>
>
>
>
> Thanks,
>
> --
>
> Yoshi
>
>
>
>
>
> --
>
> ________________________________________________________________
>
> Bob Briscoe                               http://bobbriscoe.net/
>
>
>
>
> --
>
> ________________________________________________________________
>
> Bob Briscoe                               http://bobbriscoe.net/
>
>
>
> --
>
> ________________________________________________________________
>
> Bob Briscoe                               http://bobbriscoe.net/
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--000000000000c33fae05b45659f0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


As someone who contributes to a TCP implementation, I woul=
d put in a=C2=A0 vote for the "Two Option Kinds" proposal outline=
d in slide 6 here:


This approach has the virtue of simplicity, whi=
ch hopefully may help avoid middlebox bugs, speed standardization and devel=
opment, and ease the process of maintaining fast and stable implementations=
.

neal


On Tue, Nov 17, 2020 at=
 9:25 AM Scharf, Michael <Michael.Scharf@hs-esslingen.de> wrote:











Inline some comments [ms].


=C2=A0


Given that I don=E2=80=99t ask =
for any text chances, I=E2=80=99ll not follow-up forever in this discussion=
. I don=E2=80=99t own running code and a lot of encoding details mainly aff=
ect an implementation.


=C2=A0


IMHO the rest of the WG has to =
think about the option design more than me. So, this is mostly about trigge=
ring a discussion=E2=80=A6


=C2=A0








From: Bob Briscoe <ietf@bobbriscoe.net> 

Sent: Tuesday, November 17, 2020 1:20 PM

To: Scharf, Michael <Michael.Scharf@hs-esslingen.de>; Yoshifumi N=
ishida <nsd.ietf=
@gmail.com>

Cc: Michael Tuexen <tuexen@fh-muenster.de>; tcpm-chairs@ietf.org; Mirja Kuehlewind <=
ietf@kuehlewind.ne=
t>; Scheffenegger, Richard <Richard.Scheffenegger@netapp.com>; =
tcpm IETF list <tcpm@=
ietf.org>

Subject: Re: AccECN field order






=C2=A0


Michael,<=
/p>



On 17/11/2020 08:10, Scharf, Michael wrote:






Bob,


=C2=A0


I am fine with the two option k=
inds proposed in -13, but I don=E2=80=99t buy your arguments why this encod=
ing is better than others.


=C2=A0


Most importantly, I don=E2=80=
=99t think that your =E2=80=9Cforward compatibility=E2=80=9D argument is a =
very compelling reason for two codepoints. All proposals I am aware of have=
 their pros and cons. And I
 am not aware of a really comprehensive discussion. At least neither the e-=
mail below nor today=E2=80=99s meeting discuss all tradeoffs I would invest=
igate.






[BB] You're right - the discussion below is useful and necessary.










=C2=A0


For instance, the benefit of th=
e length encoding variant would be to consume only one TCP option codepoint=
 now. If we decided later to that a flag byte is needed (say, at the end), =
a
 follow-up proposed standard could specify another option format with that =
flag byte, using a second codepoint.






[BB] 

* The new codepoint would require years to deploy (including in proxies) be=
fore it works e2e.



=C2=A0


[ms] That may depend *a lot<=
/b>* on the environment. In environments where ECN can get deployed today, =
say in datacenters, rolling out a new kernel version may perhaps not be the=
 biggest operational challenge. Also,
 proxies may not be a significant problem in some of those cases. Deploymen=
t roadmaps are very hard to predict.




* Whereas the 'forward compatibility' approach tries to ensure that=
 AccECN implementations that are being built today will be ready to interwo=
rk with a future extension.


* In contrast, if your hypothetical new AccECN codepoint did need to=
 be deployed later, it would need to include a fall-back to the current Acc=
ECN option when one peer doesn't understand the new codepoint.



=C2=A0


[ms] This is a reasonable point=
, albeit not a particularly strong one, given that the option is not really=
 required for basic interoperability. Also, a hypothetical follow-up standa=
rd could include a way to explicitly
 request the option=E2=80=A6




* Fall-back would need to avoid a second round of handshake (given this is =
for low latency), which would only work reliably with an option on the SYN =
(because the 3rd ACK is unreliable)


=C2=A0


[ms] I can=E2=80=99t follow. If=
 an endpoint is unhappy with getting AccECN option =E2=80=9CA=E2=80=9D from=
 a peer, I could send a =E2=80=9Cplease send me the better AccECN option B=
=E2=80=9D after the 3-WHS. It could be an upgrade, not a fallback. Also, at=
 this
 point we don=E2=80=99t know what option B would be and in which context it=
 would be needed. Maybe other protocol features would require negotiation, =
too. IMHO it is a solvable problem.




* AccECN has so far avoided using any SYN option space, which is important =
given that is the scarcest resource.


=C2=A0


[ms] Yep, but as long as no pro=
posal needs SYN options, this is nothing that would make one solution bette=
r than another.



These days, extending TCP is like playing chess - we have to think multiple=
 moves ahead!


=C2=A0


[ms] There is one fundamental d=
ifference between chess and TCP: In chess you precisely know the current si=
tuation and can predict all potential moves in advance (albeit there are a =
lot). However, we don=E2=80=99t know the future
 evolution of TCP, ECN and the Internet as a whole, nor future requirements=
. If we try to engineer a TCP extension for some potential future, we may f=
igure out later that reality is different from what we designed for. As muc=
h as I appreciate your attempts
 to predict the future, it would be a big surprise to me if you could corre=
ctly predict what TCP will be in 10 years from now. So, instead of trying t=
o write specifications that take into account future requirements, which ar=
e in fact just unknown, it is more
 important for me to write effective, efficient and secure specifications f=
or what is currently known for sure.


=C2=A0


[ms] I *don=E2=80=99t* k=
now for sure whether the current AccECN option will be needed, but I see qu=
ite some evidence that support of the AccECN option(s) is not a top priorit=
y these days. That is also something to think
 about when designing the details.


=C2=A0


I need to learn to articulate o=
nto the list all my reasons for rejecting certain parts of the design space=
.
Sorry about that.



Yes, I admit, the current proposal is at the cost of burning two TCP option=
 codepoints. However, you said yourself when you proposed this approach, th=
at they are not (currently) a scarce resource. Certainly not as scarce as o=
ption space. I just checked: roughly
 15% of the option codepoint allocation space has been used so far. So that=
's where I would suggest we compromise first.



[ms] I made a similar calculation some years ago and c=
ame to a similar conclusion. Investing two codepoints for AccECN is somethi=
ng we can probably afford. But nonetheless we have to reason carefully why =
it is really worth the effort.








=C2=A0


Regarding the resulting codepoi=
nt consumption (two option kinds) and the addition of a flag byte, this app=
roach would basically end up with the same result as the current proposal i=
n
 -13 plus some hypothetical future addition leveraging the length field bey=
ond the currently known use.


=C2=A0


And, yes, the length encoding v=
ariant may be less flexible and/or consume some more bits in some cases, bu=
t on the plus side it only needs one codepoint now =E2=80=93 and given that=
 it is entirely
 unclear whether any AccECN option will indeed be widely used in future, th=
is is a big plus. The fact that implementers are quite silent on the option=
 design is not a good sign.






[BB] Here's another possibility then. 



a) [K|L| EE1B | ECEB | EE0B ]

b) [K|L| EE1B | ECEB | EE0B |F]=C2=A0=C2=
=A0=C2=A0 if F =3D 0bXXXXXXX0

c) [K|L| EE0B | ECEB | EE1B |F]=C2=A0=C2=A0=C2=A0 if F =3D 0bXXXXXXX1



We'd have to decide which order alternative (a) takes. I've picked =
EE1B first only 'cos you're right that there is not currently as mu=
ch pressure (on the public Internet) for AccECN with ECT(0).



=3D=3DLegend=3D=3D

K:=C2=A0=C2=A0=C2=A0 Kind (1B)

L:=C2=A0=C2=A0=C2=A0 Length (1B)

ExxB: Echo xx byte counter

F:=C2=A0=C2=A0=C2=A0 Flags (1B)



Alternatives a) and b) are equivalent, at least while the only allocate=
d flag is for field order.

However, alternative b) is available in case other flags are allocated =
in future

The flags byte can only be omitted if (L % 3 =3D=3D 2).

The flags byte is considered present if (L % 3) =3D=3D 0.



Forward compatibility: Options with (L % 3 =3D=3D 1) MUST be assumed to=
 include a flags byte, and current implementations ignore the last byte.



The flags byte is optional to implement (even if the AccECN option is i=
mplemented)

If the server includes a flags byte on the SYN/ACK but the client does =
not include one on the 3rd ACK or the first data packet, the server assumes=
 the client does not implement the flags byte and uses only alternative a) =
for the remainder of the connection.





[ms] I am not sure if we need that, but I take =
this as existence proof that other encodings are possible. Personally, I th=
ink that other solutions would be doable if you took one bit from one or mu=
ltiple counters for format encoding.
 At least for some of the byte counters I don=E2=80=99t think taking one bi=
t for other purposes would result in an Internet congestion collapse=E2=80=
=A6 But, I have already said that the -13 proposal works for me, so I=E2=80=
=99ll not try to come up with another variant myself.





=C2=A0


To me, one potential difference=
 between two proposals would be incremental deployment. The proposal in -13=
 only has an advantage if middleboxes such as firewalls will indeed pass TC=
P
 options with a format that contains content beyond the (first) Accurate EC=
N standard (i.e., currently unused length values). IMHO it is too early to =
know whether firewalls would indeed allow this in future.<=
/u>


=C2=A0


From a security perspective, it=
 is not clear to me whether allowing arbitrary unspecified bytes in a TCP o=
ption is a good idea *at all*. It will be interesting to hear the op=
inion
 from SEC area on that. Personally, I am not convinced that this really mak=
es sense, but I my concerns are not strong enough to formally push back. I=
=E2=80=99ll leave it to others to think about whether this is a bug or a fe=
ature.






[BB] Well, let's first try to deal with security ourselves:

* Octets that are explicitly declared as part of an option cannot be used f=
or a buffer overflow attack. I don't really need to, but I could add th=
e following text to the forward compatibility wording:


=C2=A0=C2=A0=C2=A0 A receiver considers octets beyond those it uses, but wi=
thin the specified length, as if they are padding.


* And such octets cannot be any different from the current ability of a sen=
der to add padding. So there's no new attack possible here.

* And there's no need to specify a max length for any AccECN Option, wh=
ich would just unnecessarily limit flexibility.



[ms] I have more thought about the covert channel that=
 some unspecified bytes in a standards track TCP option could offer=E2=80=
=A6 In particular if the AccECN option gets widely deployed and used, would=
 some malicious users find those bytes useful?
 For what? For instance, what in the AccECN protocol design prevents use of=
 these =E2=80=9Cpadding=E2=80=9D bytes as super-cookie? To me, it is much m=
ore important to think about abuse of whatever we standardize *now*,=
 instead of engineering for some entirely unknown
 future. I am not a security expert, but I suggest a careful analysis of an=
y security implications of the AccECN option. We seldomly specify new TCP o=
ptions and malicious users will certainly look for =E2=80=9Copportunities=
=E2=80=9D in the spec. So far, I don=E2=80=99t know whether
 that whole idea of forward compatibility is a feature or a bug.<=
/u>


=C2=A0


Michael



=C2=A0


Maybe one lesson learnt is that=
 the document could have a non-normative appendix that explains the rationa=
le for the finally picked TCP option encoding. That may also help if there =
are
 further questions whether two codepoints are really required, e.g. by the =
IESG (if two codepoints are still the design after WGLC). At least for past=
 TCP option codepoint allocations I recall some discussions late in the IET=
F process. In those past cases,
 good arguments in an appendix and running code has helped a lot.=







[BB] I can do that. Appx B is already similar, giving Design Rationale for =
the ACE field.





Bob










=C2=A0


Michael

=C2=A0


=C2=A0


=C2=A0








From: Bob Briscoe
<ietf@bobbrisco=
e.net> 

Sent: Tuesday, November 17, 2020 6:10 AM

To: Scharf, Michael <Michael.Scharf@hs-esslingen.de>; Yoshifumi N=
ishida
<nsd.ietf@gmail.=
com>

Cc: Michael Tuexen <tuexen@fh-muenster.de>;
tcpm-chairs@ietf.=
org; Mirja Kuehlewind 
<ietf@kuehlewind.net>; Scheffenegger, Richard 
<Richard.Scheffenegger@netapp.com>; tcpm IETF list 
<tcpm@ietf.org>

Subject: Re: AccECN field order






=C2=A0


Michael,<=
/p>



On 16/11/2020 17:36, Scharf, Michael wrote:






Bob,


=C2=A0


One proposal using the length f=
ield with *one option codepoint only* is detailed in:

https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/=



=C2=A0


It is the third option mentione=
d in this e-mail. One example would be to use option length values 5/8/11 f=
or one encoding type and option length values 6/9/12 for the other encoding
 type (i.e., order of fields). Or one could use some other combination of l=
ength values =E2=80=93 the only requirement is that a certain value for the=
 option length is only used by one of the option formats. In that approach,=
 the value of the length field would thus
 directly describe the encoding of the option. Unless I miss something, thi=
s would work and it would just require one option codepoint.<=
u>


=C2=A0


Thus, alternatives to two optio=
n codepoints exist and I have explained them on the list in March 2020.






OK, sorry, yes, I remember this now.



As I will explain in the AccECN status update talk today in virtual Bangkok=
, the draft has made provision for different length values than 5/8/11. It =
says existing implementations MUST accept length values other than those cu=
rrently defined. But then read in
 as many whole 3-byte fields as they can.



This can be used to add a flags byte on the end in future, for extensibilit=
y. Or any other form of extensibility the WG might decide in the future.


I know a flags byte at the end seems odd compared to at the beginning. But =
(if decided it's needed in future) it's reasonably easy to implemen=
t by reading the whole option, then processing the last byte, before readin=
g the rest of the option.



I believe you will agree that this is a better way to utilize different len=
gths.



And thank you for repeatedly emphasizing that you're happy with the 2-k=
ind scheme, or other alternatives.





Bob












=C2=A0


Anyway, I don=E2=80=99t really =
care how the options are encoded as long as the receiver doesn=E2=80=99t ne=
ed per-connection state for decoding a TCP option. So, personally, I would =
be fine with using
 e.g. the length field as described in my old e-mail. Or an additional flag=
 byte. And one could come up with further encodings, e.g., by using one or =
a few bits as a short =E2=80=9Ctype=E2=80=9D field for each counter. This i=
s all about protocol engineering. And all these
 variants have their pros and cons.


=C2=A0


I am also fine with using two o=
ption codepoints as specified in -13; this is probably the approach that co=
nsumes the least number of bits.


=C2=A0


Michael (w/o any hat)=



=C2=A0


=C2=A0


=C2=A0








From: Bob Briscoe 
<ietf@bobbriscoe.net> 

Sent: Monday, November 16, 2020 5:52 PM

To: Yoshifumi Nishida <nsd.ietf@gmail.com>

Cc: Scharf, Michael <Michael.Scharf@hs-esslingen.de>; Michael Tue=
xen
<tuexen@fh-mu=
enster.de>; 
tcpm-chairs@ietf.org; Mirja Kuehlewind <ietf@kuehlewind.net>; Scheffenegger, Ri=
chard
<R=
ichard.Scheffenegger@netapp.com>; tcpm IETF list
<tcpm@ietf.org>

Subject: AccECN field order






=C2=A0


Yoshi, (adding the tcpm=
 list in cc)




On 05/11/2020 06:58, Yoshifumi Nishida wrote:=











Hi Bob,






=C2=A0











Yoshi,



On 04/11/2020 06:51, Yoshifumi Nishida wrote:=











Hi, folks,













































In my understanding, I'm not sure if we settled =
down on using two option kinds or encoding schemes for 24bits fields in acc=
 ecn draft.






So, I think there're still something to be clari=
fied and hope things will be settled at the meeting.










[BB] I know a WG can change it's mind at any time. But I'd rather w=
e just clarified what a previous decision was, to avoid the need to keep re=
-opening discussion on a question that have been decided then changed three=
 different ways already.



My memory is not so good these days. I trusted that Michael S remembered th=
e decision correctly, and I seem to remember that decision being made.

I've just checked the minutes of the last interim:

=C2=A0=C2=A0=C2=A0 
https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/=


and they mention Michael's proposal to use two kinds, but don't rec=
ord any decision.

The jabber log gives no clues about any decision.



I can't find an audio or video recording. Can you point me at one?








=C2=A0






I thought that it's because there was no clear d=
ecision at the meeting.=C2=A0







Please let us know if you have any questions or opin=
ions with regard to this.












[BB] I checked the Youtube link you sent below.



First I think we're agreed no-one was fighting for us to keep the previ=
ous way we did this (using the initial value of the field to set the order =
for the connection).


In my presentation I said there was strong resistance from Michael to do it=
 a different way.


(also, offlist, the co-authors including me also didn't like this so mu=
ch. And Ilpo said it made the implementation complex.)



Then came the question of what we do instead. There were three alternative =
proposals:

a) use 2 option kinds

b) add a flags byte

c) somehow use the length field maybe



Michael raised (c) in the meeting as a possibility, but no-one could think =
how to distinguish two options of the same length but a different field ord=
er using the length field. Michael said he'd post any ideas to the list=
 if he could think of any, but that
 didn't happen.



So we're left choosing between (a) and (b).

I said in the meeting (and on the list when discussing with Ilpo) that I=
9;d be happy to go with (b), but only if there was another use for a flag. =
Because it would consume 1B more options space in many packets, which is a =
scarce resource.



Ilpo had a proposed use for another flag (to help synch counters after a lo=
ss), but I think the discussion about it ended that it wouldn't be help=
ful, 'cos the way it worked depended on itself (circular logic).



In conclusion, I don't think there was an explicit decision to go with =
2 option kinds, but it ended up as the 'last person standing'.

I like it. It's simple. And apparently option kinds are not such a scar=
ce resource.



Perhaps we can ratify this in the WG tomorrow.





Bob




















=C2=A0






Thanks,






--






Yoshi





















-- 


________________________________________________________________


Bob Briscoe=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0http://bobbriscoe.net/















-- 


________________________________________________________________


Bob Briscoe=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0http://bobbriscoe.net/













-- 


________________________________________________________________


Bob Briscoe=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 http://bobbriscoe.net/








_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--000000000000c33fae05b45659f0--


From nobody Tue Nov 17 18:11:28 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 540B53A1287; Tue, 17 Nov 2020 18:11:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uTVartW_dxy3; Tue, 17 Nov 2020 18:11:22 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B1E5D3A128C; Tue, 17 Nov 2020 18:11:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=CcVxdjssv+kheB7yWo5c8AKYaoZLdFXCP6uNJa1yGHg=; b=u+zwCgqKVedlOlF4dZ0C3M75j pk0GTnrRAsko3ZQvoArl7IUW62wTpHfrWaxQLKIzKoKFrdD5FpGvGp4VVedjT/5yAeoN3qzrMal3Q H/c/ywFgLr/OiLBPgeY69ooJQfCzi3aNmfLbogbnQsetG0TvTrLwNYW8uHFBFcSmy3njJebJcJO5J +raKZfiOKF1q4G/R1lgNquFkzY3pCHTGjzuCSL0zGDAdD9r/L7t58bDj68bP4ecMoW4QN/E1U/96k xnW6sRYf1HMD5h9jP77KBvvQ8RDAARk6dQoTKF5bGU2aerJTCAyLdFOBeYJi7bFAORXDXikYs426n CWIojJw0A==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:42966 helo=[192.168.1.4]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kfCwF-00Bejt-HC; Wed, 18 Nov 2020 02:11:17 +0000
To: "Scharf, Michael" , Yoshifumi Nishida 
Cc: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de> <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net> <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>
From: Bob Briscoe 
Message-ID: 
Date: Wed, 18 Nov 2020 02:11:14 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>
Content-Type: multipart/alternative; boundary="------------A7CFC12E4CB829C7B818DDB0"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 18 Nov 2020 02:11:26 -0000

This is a multi-part message in MIME format.
--------------A7CFC12E4CB829C7B818DDB0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Michael,

On 17/11/2020 14:25, Scharf, Michael wrote:
>
> Inline some comments [ms].
>
> Given that I don’t ask for any text chances, I’ll not follow-up 
> forever in this discussion. I don’t own running code and a lot of 
> encoding details mainly affect an implementation.
>
> IMHO the rest of the WG has to think about the option design more than 
> me. So, this is mostly about triggering a discussion…
>
> *From:* Bob Briscoe 
> *Sent:* Tuesday, November 17, 2020 1:20 PM
> *To:* Scharf, Michael ; Yoshifumi 
> Nishida 
> *Cc:* Michael Tuexen ; tcpm-chairs@ietf.org; 
> Mirja Kuehlewind ; Scheffenegger, Richard 
> ; tcpm IETF list 
> *Subject:* Re: AccECN field order
>
> Michael,
>
> On 17/11/2020 08:10, Scharf, Michael wrote:
>
>     Bob,
>
>     I am fine with the two option kinds proposed in -13, but I don’t
>     buy your arguments why this encoding is better than others.
>
>     Most importantly, I don’t think that your “forward compatibility”
>     argument is a very compelling reason for two codepoints. All
>     proposals I am aware of have their pros and cons. And I am not
>     aware of a really comprehensive discussion. At least neither the
>     e-mail below nor today’s meeting discuss all tradeoffs I would
>     investigate.
>
>
> [BB] You're right - the discussion below is useful and necessary.
>
>
>     For instance, the benefit of the length encoding variant would be
>     to consume only one TCP option codepoint now. If we decided later
>     to that a flag byte is needed (say, at the end), a follow-up
>     proposed standard could specify another option format with that
>     flag byte, using a second codepoint.
>
>
> [BB]
> * The new codepoint would require years to deploy (including in 
> proxies) before it works e2e.
>
> [ms] That may depend **a lot** on the environment. In environments 
> where ECN can get deployed today, say in datacenters, rolling out a 
> new kernel version may perhaps not be the biggest operational 
> challenge. Also, proxies may not be a significant problem in some of 
> those cases. Deployment roadmaps are very hard to predict.
>
>
> * Whereas the 'forward compatibility' approach tries to ensure that 
> AccECN implementations that are being built today will be ready to 
> interwork with a future extension.
> * In contrast, if your hypothetical new AccECN codepoint did need to 
> be deployed later, it would need to include a fall-back to the current 
> AccECN option when one peer doesn't understand the new codepoint.
>
> [ms] This is a reasonable point, albeit not a particularly strong one, 
> given that the option is not really required for basic 
> interoperability. Also, a hypothetical follow-up standard could 
> include a way to explicitly request the option…
>
>
> * Fall-back would need to avoid a second round of handshake (given 
> this is for low latency), which would only work reliably with an 
> option on the SYN (because the 3rd ACK is unreliable)
>
> [ms] I can’t follow. If an endpoint is unhappy with getting AccECN 
> option “A” from a peer, I could send a “please send me the better 
> AccECN option B” after the 3-WHS. It could be an upgrade, not a 
> fallback. Also, at this point we don’t know what option B would be and 
> in which context it would be needed. Maybe other protocol features 
> would require negotiation, too. IMHO it is a solvable problem.
>

[BB] Maybe. I prefer to work out full details of a protocol proposal 
before I trust it will be a viable alternative to another proposal.
Through Ilpo's criticism of the handshake reflection stuff we did, I 
learned a lot about how any option cleverness after the SYN/ACK is 
fraught with difficulties, given it cannot be known whether one end or 
the other can be guaranteed to send some data, to be able to piggy-back 
reliable delivery of an option. And you have to cater for all the 
possibilities for reordering the messages once you are past the 3WHS. 
Probably do-able, but complexity could be prohibitive, even for a simple 
upgrade.

>
> * AccECN has so far avoided using any SYN option space, which is 
> important given that is the scarcest resource.
>
> [ms] Yep, but as long as no proposal needs SYN options, this is 
> nothing that would make one solution better than another.
>
> These days, extending TCP is like playing chess - we have to think 
> multiple moves ahead!
>
> [ms] There is one fundamental difference between chess and TCP: In 
> chess you precisely know the current situation and can predict all 
> potential moves in advance (albeit there are a lot). However, we don’t 
> know the future evolution of TCP, ECN and the Internet as a whole, nor 
> future requirements. If we try to engineer a TCP extension for some 
> potential future, we may figure out later that reality is different 
> from what we designed for. As much as I appreciate your attempts to 
> predict the future, it would be a big surprise to me if you could 
> correctly predict what TCP will be in 10 years from now. So, instead 
> of trying to write specifications that take into account future 
> requirements, which are in fact just unknown, it is more important for 
> me to write effective, efficient and secure specifications for what is 
> currently known for sure.
>
> [ms] I **don’t** know for sure whether the current AccECN option will 
> be needed, but I see quite some evidence that support of the AccECN 
> option(s) is not a top priority these days. That is also something to 
> think about when designing the details.
>
> I need to learn to articulate onto the list all my reasons for 
> rejecting certain parts of the design space. Sorry about that.
>
> Yes, I admit, the current proposal is at the cost of burning two TCP 
> option codepoints. However, you said yourself when you proposed this 
> approach, that they are not (currently) a scarce resource. Certainly 
> not as scarce as option space. I just checked: roughly 15% of the 
> option codepoint allocation space has been used so far. So that's 
> where I would suggest we compromise first.
>
> [ms] I made a similar calculation some years ago and came to a similar 
> conclusion. Investing two codepoints for AccECN is something we can 
> probably afford. But nonetheless we have to reason carefully why it is 
> really worth the effort.
>
>     Regarding the resulting codepoint consumption (two option kinds)
>     and the addition of a flag byte, this approach would basically end
>     up with the same result as the current proposal in -13 plus some
>     hypothetical future addition leveraging the length field beyond
>     the currently known use.
>
>     And, yes, the length encoding variant may be less flexible and/or
>     consume some more bits in some cases, but on the plus side it only
>     needs one codepoint now – and given that it is entirely unclear
>     whether any AccECN option will indeed be widely used in future,
>     this is a big plus. The fact that implementers are quite silent on
>     the option design is not a good sign.
>
>
> [BB] Here's another possibility then.
>
> a) [K|L| EE1B | ECEB | EE0B ]
> b) [K|L| EE1B | ECEB | EE0B |F]    if F = 0bXXXXXXX0
> c) [K|L| EE0B | ECEB | EE1B |F]    if F = 0bXXXXXXX1
>
> We'd have to decide which order alternative (a) takes. I've picked 
> EE1B first only 'cos you're right that there is not currently as much 
> pressure (on the public Internet) for AccECN with ECT(0).
>
> ==Legend==
> K:    Kind (1B)
> L:    Length (1B)
> ExxB: Echo xx byte counter
> F:    Flags (1B)
>
> Alternatives a) and b) are equivalent, at least while the only 
> allocated flag is for field order.
> However, alternative b) is available in case other flags are allocated 
> in future
> The flags byte can only be omitted if (L % 3 == 2).
> The flags byte is considered present if (L % 3) == 0.
>
> Forward compatibility: Options with (L % 3 == 1) MUST be assumed to 
> include a flags byte, and current implementations ignore the last byte.
>
> The flags byte is optional to implement (even if the AccECN option is 
> implemented)
> If the server includes a flags byte on the SYN/ACK but the client does 
> not include one on the 3rd ACK or the first data packet, the server 
> assumes the client does not implement the flags byte and uses only 
> alternative a) for the remainder of the connection.
>
>
> [ms] I am not sure if we need that, but I take this as existence proof 
> that other encodings are possible. Personally, I think that other 
> solutions would be doable if you took one bit from one or multiple 
> counters for format encoding. At least for some of the byte counters I 
> don’t think taking one bit for other purposes would result in an 
> Internet congestion collapse… But, I have already said that the -13 
> proposal works for me, so I’ll not try to come up with another variant 
> myself.
>
>     To me, one potential difference between two proposals would be
>     incremental deployment. The proposal in -13 only has an advantage
>     if middleboxes such as firewalls will indeed pass TCP options with
>     a format that contains content beyond the (first) Accurate ECN
>     standard (i.e., currently unused length values). IMHO it is too
>     early to know whether firewalls would indeed allow this in future.
>
>     From a security perspective, it is not clear to me whether
>     allowing arbitrary unspecified bytes in a TCP option is a good
>     idea **at all**. It will be interesting to hear the opinion from
>     SEC area on that. Personally, I am not convinced that this really
>     makes sense, but I my concerns are not strong enough to formally
>     push back. I’ll leave it to others to think about whether this is
>     a bug or a feature.
>
>
> [BB] Well, let's first try to deal with security ourselves:
> * Octets that are explicitly declared as part of an option cannot be 
> used for a buffer overflow attack. I don't really need to, but I could 
> add the following text to the forward compatibility wording:
>     A receiver considers octets beyond those it uses, but within the 
> specified length, as if they are padding.
> * And such octets cannot be any different from the current ability of 
> a sender to add padding. So there's no new attack possible here.
> * And there's no need to specify a max length for any AccECN Option, 
> which would just unnecessarily limit flexibility.
>
> [ms] I have more thought about the covert channel that some 
> unspecified bytes in a standards track TCP option could offer… In 
> particular if the AccECN option gets widely deployed and used, would 
> some malicious users find those bytes useful? For what? For instance, 
> what in the AccECN protocol design prevents use of these “padding” 
> bytes as super-cookie? To me, it is much more important to think about 
> abuse of whatever we standardize **now**, instead of engineering for 
> some entirely unknown future. I am not a security expert, but I 
> suggest a careful analysis of any security implications of the AccECN 
> option. We seldomly specify new TCP options and malicious users will 
> certainly look for “opportunities” in the spec. So far, I don’t know 
> whether that whole idea of forward compatibility is a feature or a bug.
>

[BB] If that were a problem, it could already be done, because (by my 
reading of RFC793) it is valid to set the data offset to stretch out the 
padding to any length within the max option space limits.


Bob

> Michael
>
>     Maybe one lesson learnt is that the document could have a
>     non-normative appendix that explains the rationale for the finally
>     picked TCP option encoding. That may also help if there are
>     further questions whether two codepoints are really required, e.g.
>     by the IESG (if two codepoints are still the design after WGLC).
>     At least for past TCP option codepoint allocations I recall some
>     discussions late in the IETF process. In those past cases, good
>     arguments in an appendix and running code has helped a lot.
>
>
> [BB] I can do that. Appx B is already similar, giving Design Rationale 
> for the ACE field.
>
>
> Bob
>
>
>     Michael
>
>     *From:*Bob Briscoe  
>     *Sent:* Tuesday, November 17, 2020 6:10 AM
>     *To:* Scharf, Michael 
>     ; Yoshifumi Nishida
>      
>     *Cc:* Michael Tuexen 
>     ; tcpm-chairs@ietf.org
>     ; Mirja Kuehlewind
>      ; Scheffenegger,
>     Richard 
>     ; tcpm IETF list
>      
>     *Subject:* Re: AccECN field order
>
>     Michael,
>
>     On 16/11/2020 17:36, Scharf, Michael wrote:
>
>         Bob,
>
>         One proposal using the length field with **one option
>         codepoint only** is detailed in:
>         https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>
>         It is the third option mentioned in this e-mail. One example
>         would be to use option length values 5/8/11 for one encoding
>         type and option length values 6/9/12 for the other encoding
>         type (i.e., order of fields). Or one could use some other
>         combination of length values – the only requirement is that a
>         certain value for the option length is only used by one of the
>         option formats. In that approach, the value of the length
>         field would thus directly describe the encoding of the option.
>         Unless I miss something, this would work and it would just
>         require one option codepoint.
>
>         Thus, alternatives to two option codepoints exist and I have
>         explained them on the list in March 2020.
>
>
>     OK, sorry, yes, I remember this now.
>
>     As I will explain in the AccECN status update talk today in
>     virtual Bangkok, the draft has made provision for different length
>     values than 5/8/11. It says existing implementations MUST accept
>     length values other than those currently defined. But then read in
>     as many whole 3-byte fields as they can.
>
>     This can be used to add a flags byte on the end in future, for
>     extensibility. Or any other form of extensibility the WG might
>     decide in the future.
>
>     I know a flags byte at the end seems odd compared to at the
>     beginning. But (if decided it's needed in future) it's reasonably
>     easy to implement by reading the whole option, then processing the
>     last byte, before reading the rest of the option.
>
>     I believe you will agree that this is a better way to utilize
>     different lengths.
>
>     And thank you for repeatedly emphasizing that you're happy with
>     the 2-kind scheme, or other alternatives.
>
>
>     Bob
>
>
>
>         Anyway, I don’t really care how the options are encoded as
>         long as the receiver doesn’t need per-connection state for
>         decoding a TCP option. So, personally, I would be fine with
>         using e.g. the length field as described in my old e-mail. Or
>         an additional flag byte. And one could come up with further
>         encodings, e.g., by using one or a few bits as a short “type”
>         field for each counter. This is all about protocol
>         engineering. And all these variants have their pros and cons.
>
>         I am also fine with using two option codepoints as specified
>         in -13; this is probably the approach that consumes the least
>         number of bits.
>
>         Michael (w/o any hat)
>
>         *From:* Bob Briscoe 
>         
>         *Sent:* Monday, November 16, 2020 5:52 PM
>         *To:* Yoshifumi Nishida 
>         
>         *Cc:* Scharf, Michael 
>         ; Michael Tuexen
>          ;
>         tcpm-chairs@ietf.org ; Mirja
>         Kuehlewind  ;
>         Scheffenegger, Richard 
>         ; tcpm IETF list
>          
>         *Subject:* AccECN field order
>
>         Yoshi, (adding the tcpm list in cc)
>
>         On 05/11/2020 06:58, Yoshifumi Nishida wrote:
>
>             Hi Bob,
>
>             On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe
>             > wrote:
>
>                 Yoshi,
>
>                 On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>
>                     Hi, folks,
>
>
>
>
>
>                     In my understanding, I'm not sure if we settled
>                     down on using two option kinds or encoding schemes
>                     for 24bits fields in acc ecn draft.
>
>                     So, I think there're still something to be
>                     clarified and hope things will be settled at the
>                     meeting.
>
>
>                 [BB] I know a WG can change it's mind at any time. But
>                 I'd rather we just clarified what a previous decision
>                 was, to avoid the need to keep re-opening discussion
>                 on a question that have been decided then changed
>                 three different ways already.
>
>                 My memory is not so good these days. I trusted that
>                 Michael S remembered the decision correctly, and I
>                 seem to remember that decision being made.
>                 I've just checked the minutes of the last interim:
>                 https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/
>                 and they mention Michael's proposal to use two kinds,
>                 but don't record any decision.
>                 The jabber log gives no clues about any decision.
>
>                 I can't find an audio or video recording. Can you
>                 point me at one?
>
>             I thought that it's because there was no clear decision at
>             the meeting.
>
>             But, you can check https://www.youtube.com/watch?v=KsB0_Nb8-kA
>
>             Please let us know if you have any questions or opinions
>             with regard to this.
>
>
>         [BB] I checked the Youtube link you sent below.
>
>         First I think we're agreed no-one was fighting for us to keep
>         the previous way we did this (using the initial value of the
>         field to set the order for the connection).
>         In my presentation I said there was strong resistance from
>         Michael to do it a different way.
>         (also, offlist, the co-authors including me also didn't like
>         this so much. And Ilpo said it made the implementation complex.)
>
>         Then came the question of what we do instead. There were three
>         alternative proposals:
>         a) use 2 option kinds
>         b) add a flags byte
>         c) somehow use the length field maybe
>
>         Michael raised (c) in the meeting as a possibility, but no-one
>         could think how to distinguish two options of the same length
>         but a different field order using the length field. Michael
>         said he'd post any ideas to the list if he could think of any,
>         but that didn't happen.
>
>         So we're left choosing between (a) and (b).
>         I said in the meeting (and on the list when discussing with
>         Ilpo) that I'd be happy to go with (b), but only if there was
>         another use for a flag. Because it would consume 1B more
>         options space in many packets, which is a scarce resource.
>
>         Ilpo had a proposed use for another flag (to help synch
>         counters after a loss), but I think the discussion about it
>         ended that it wouldn't be helpful, 'cos the way it worked
>         depended on itself (circular logic).
>
>         In conclusion, I don't think there was an explicit decision to
>         go with 2 option kinds, but it ended up as the 'last person
>         standing'.
>         I like it. It's simple. And apparently option kinds are not
>         such a scarce resource.
>
>         Perhaps we can ratify this in the WG tomorrow.
>
>
>         Bob
>
>
>
>
>             Thanks,
>
>             --
>
>             Yoshi
>
>
>
>
>
>         -- 
>
>         ________________________________________________________________
>
>         Bob Briscoehttp://bobbriscoe.net/
>
>
>
>
>     -- 
>
>     ________________________________________________________________
>
>     Bob Briscoehttp://bobbriscoe.net/
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------A7CFC12E4CB829C7B818DDB0
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Michael,

    

    
On 17/11/2020 14:25, Scharf, Michael
      wrote:

    

    

      
      
      
      

        
Inline some comments [ms].

        
 

        
Given that I don’t ask for any text chances,
            I’ll not follow-up forever in this discussion. I don’t own
            running code and a lot of encoding details mainly affect an
            implementation.

        
 

        
IMHO the rest of the WG has to think about the
            option design more than me. So, this is mostly about
            triggering a discussion…

        
 

        

          

            

              
From: Bob Briscoe
                <ietf@bobbriscoe.net> 

                Sent: Tuesday, November 17, 2020 1:20 PM

                To: Scharf, Michael
                <Michael.Scharf@hs-esslingen.de>; Yoshifumi
                Nishida <nsd.ietf@gmail.com>

                Cc: Michael Tuexen <tuexen@fh-muenster.de>;
                tcpm-chairs@ietf.org; Mirja Kuehlewind
                <ietf@kuehlewind.net>; Scheffenegger, Richard
                <Richard.Scheffenegger@netapp.com>; tcpm IETF list
                <tcpm@ietf.org>

                Subject: Re: AccECN field order

            

          

          
 

          
Michael,

          

            
On 17/11/2020 08:10, Scharf, Michael
              wrote:

          

          

            
Bob,

            
 

            
I am
                fine with the two option kinds proposed in -13, but I
                don’t buy your arguments why this encoding is better
                than others.

            
 

            
Most
                importantly, I don’t think that your “forward
                compatibility” argument is a very compelling reason for
                two codepoints. All proposals I am aware of have their
                pros and cons. And I am not aware of a really
                comprehensive discussion. At least neither the e-mail
                below nor today’s meeting discuss all tradeoffs I would
                investigate.

          

          


            [BB] You're right - the discussion below is useful and
            necessary.

            

            

            

          

            
 

            
For
                instance, the benefit of the length encoding variant
                would be to consume only one TCP option codepoint now.
                If we decided later to that a flag byte is needed (say,
                at the end), a follow-up proposed standard could specify
                another option format with that flag byte, using a
                second codepoint.

          

          


            [BB] 

            * The new codepoint would require years to deploy (including
            in proxies) before it works e2e.
            

          
 

          
[ms] That may depend *a
                lot* on the environment. In environments where ECN
              can get deployed today, say in datacenters, rolling out a
              new kernel version may perhaps not be the biggest
              operational challenge. Also, proxies may not be a
              significant problem in some of those cases. Deployment
              roadmaps are very hard to predict.

          


              * Whereas the 'forward compatibility' approach tries to
              ensure that AccECN implementations that are being built
              today will be ready to interwork with a future extension.
              

            * In contrast, if your hypothetical new AccECN
            codepoint did need to be deployed later, it would need to
            include a fall-back to the current AccECN option when one
            peer doesn't understand the new codepoint.
            

          
 

          
[ms] This is a
              reasonable point, albeit not a particularly strong one,
              given that the option is not really required for basic
              interoperability. Also, a hypothetical follow-up standard
              could include a way to explicitly request the option…

          


              * Fall-back would need to avoid a second round of
              handshake (given this is for low latency), which would
              only work reliably with an option on the SYN (because the
              3rd ACK is unreliable)

          
 

          
[ms] I can’t follow.
              If an endpoint is unhappy with getting AccECN option “A”
              from a peer, I could send a “please send me the better
              AccECN option B” after the 3-WHS. It could be an upgrade,
              not a fallback. Also, at this point we don’t know what
              option B would be and in which context it would be needed.
              Maybe other protocol features would require negotiation,
              too. IMHO it is a solvable problem.

        

      

    

    

    [BB] Maybe. I prefer to work out full details of a protocol proposal
    before I trust it will be a viable alternative to another proposal.

    Through Ilpo's criticism of the handshake reflection stuff we did, I
    learned a lot about how any option cleverness after the SYN/ACK is
    fraught with difficulties, given it cannot be known whether one end
    or the other can be guaranteed to send some data, to be able to
    piggy-back reliable delivery of an option. And you have to cater for
    all the possibilities for reordering the messages once you are past
    the 3WHS. Probably do-able, but complexity could be prohibitive,
    even for a simple upgrade.

    

    

      

        

          

          


              * AccECN has so far avoided using any SYN option space,
              which is important given that is the scarcest resource.

          
 

          
[ms] Yep, but as long
              as no proposal needs SYN options, this is nothing that
              would make one solution better than another.

              

              These days, extending TCP is like playing chess - we have
              to think multiple moves ahead!

          
 

          
[ms] There is one
              fundamental difference between chess and TCP: In chess you
              precisely know the current situation and can predict all
              potential moves in advance (albeit there are a lot).
              However, we don’t know the future evolution of TCP, ECN
              and the Internet as a whole, nor future requirements. If
              we try to engineer a TCP extension for some potential
              future, we may figure out later that reality is different
              from what we designed for. As much as I appreciate your
              attempts to predict the future, it would be a big surprise
              to me if you could correctly predict what TCP will be in
              10 years from now. So, instead of trying to write
              specifications that take into account future requirements,
              which are in fact just unknown, it is more important for
              me to write effective, efficient and secure specifications
              for what is currently known for sure.

          
 

          
[ms] I *don’t*
              know for sure whether the current AccECN option will be
              needed, but I see quite some evidence that support of the
              AccECN option(s) is not a top priority these days. That is
              also something to think about when designing the details.

          
 

          
I need to learn to
              articulate onto the list all my reasons for rejecting
              certain parts of the design space.
            Sorry about that.

            

            Yes, I admit, the current proposal is at the cost of burning
            two TCP option codepoints. However, you said yourself when
            you proposed this approach, that they are not (currently) a
            scarce resource. Certainly not as scarce as option space. I
            just checked: roughly 15% of the option codepoint allocation
            space has been used so far. So that's where I would suggest
            we compromise first.

            

            [ms] I made a similar calculation some
              years ago and came to a similar conclusion. Investing two
              codepoints for AccECN is something we can probably afford.
              But nonetheless we have to reason carefully why it is
              really worth the effort.

              

              

          

            
 

            
Regarding
                the resulting codepoint consumption (two option kinds)
                and the addition of a flag byte, this approach would
                basically end up with the same result as the current
                proposal in -13 plus some hypothetical future addition
                leveraging the length field beyond the currently known
                use.

            
 

            
And,
                yes, the length encoding variant may be less flexible
                and/or consume some more bits in some cases, but on the
                plus side it only needs one codepoint now – and given
                that it is entirely unclear whether any AccECN option
                will indeed be widely used in future, this is a big
                plus. The fact that implementers are quite silent on the
                option design is not a good sign.

          

          


            [BB] Here's another possibility then. 

            

            a) [K|L| EE1B | ECEB |
                EE0B ]

            b) [K|L| EE1B | ECEB |
                EE0B |F]    if F = 0bXXXXXXX0

              c) [K|L| EE0B | ECEB | EE1B |F]    if F = 0bXXXXXXX1

            

            We'd have to decide which order alternative (a) takes. I've
            picked EE1B first only 'cos you're right that there is not
            currently as much pressure (on the public Internet) for
            AccECN with ECT(0).

            

              ==Legend==

              K:    Kind (1B)

              L:    Length (1B)

              ExxB: Echo xx byte counter

              F:    Flags (1B)

              

              Alternatives a) and b) are equivalent, at least while
                the only allocated flag is for field order.

              However, alternative b) is available in case other
                flags are allocated in future

              The flags byte can only be omitted if (L % 3 == 2).

              The flags byte is considered present if (L % 3) == 0.

              

              Forward compatibility: Options with (L % 3 == 1) MUST
                be assumed to include a flags byte, and current
                implementations ignore the last byte.

              

              The flags byte is optional to implement (even if the
                AccECN option is implemented)

              If the server includes a flags byte on the SYN/ACK but
                the client does not include one on the 3rd ACK or the
                first data packet, the server assumes the client does
                not implement the flags byte and uses only alternative
                a) for the remainder of the connection.

              

              

            [ms] I am not sure if we need
              that, but I take this as existence proof that other
              encodings are possible. Personally, I think that other
              solutions would be doable if you took one bit from one or
              multiple counters for format encoding. At least for some
              of the byte counters I don’t think taking one bit for
              other purposes would result in an Internet congestion
              collapse… But, I have already said that the -13 proposal
              works for me, so I’ll not try to come up with another
              variant myself.
              

          

            
 

            
To me,
                one potential difference between two proposals would be
                incremental deployment. The proposal in -13 only has an
                advantage if middleboxes such as firewalls will indeed
                pass TCP options with a format that contains content
                beyond the (first) Accurate ECN standard (i.e.,
                currently unused length values). IMHO it is too early to
                know whether firewalls would indeed allow this in
                future.

            
 

            
From a
                security perspective, it is not clear to me whether
                allowing arbitrary unspecified bytes in a TCP option is
                a good idea *at all*. It will be interesting to
                hear the opinion from SEC area on that. Personally, I am
                not convinced that this really makes sense, but I my
                concerns are not strong enough to formally push back.
                I’ll leave it to others to think about whether this is a
                bug or a feature.

          

          


            [BB] Well, let's first try to deal with security ourselves:

            * Octets that are explicitly declared as part of an option
            cannot be used for a buffer overflow attack. I don't really
            need to, but I could add the following text to the forward
            compatibility wording:
            

                A receiver considers octets beyond those it uses, but
            within the specified length, as if they are padding.
            

            * And such octets cannot be any different from the current
            ability of a sender to add padding. So there's no new attack
            possible here.

            * And there's no need to specify a max length for any AccECN
            Option, which would just unnecessarily limit flexibility.

            

            [ms] I have more thought about the covert
              channel that some unspecified bytes in a standards track
              TCP option could offer… In particular if the AccECN option
              gets widely deployed and used, would some malicious users
              find those bytes useful? For what? For instance, what in
              the AccECN protocol design prevents use of these “padding”
              bytes as super-cookie? To me, it is much more important to
              think about abuse of whatever we standardize *now*,
              instead of engineering for some entirely unknown future. I
              am not a security expert, but I suggest a careful analysis
              of any security implications of the AccECN option. We
              seldomly specify new TCP options and malicious users will
              certainly look for “opportunities” in the spec. So far, I
              don’t know whether that whole idea of forward
              compatibility is a feature or a bug.

        

      

    

    

    [BB] If that were a problem, it could already be done, because (by
    my reading of RFC793) it is valid to set the data offset to stretch
    out the padding to any length within the max option space limits.

    

    

    Bob

    

    

      

        

          

          
 

          
Michael

          

            
 

            
Maybe
                one lesson learnt is that the document could have a
                non-normative appendix that explains the rationale for
                the finally picked TCP option encoding. That may also
                help if there are further questions whether two
                codepoints are really required, e.g. by the IESG (if two
                codepoints are still the design after WGLC). At least
                for past TCP option codepoint allocations I recall some
                discussions late in the IETF process. In those past
                cases, good arguments in an appendix and running code
                has helped a lot.

          

          


            [BB] I can do that. Appx B is already similar, giving Design
            Rationale for the ACE field.

            

            

            Bob

            

            

            

          

            
 

            
Michael

            
 

            
 

            
 

            

              

                

                  
From: Bob Briscoe
                      <ietf@bobbriscoe.net>
                      

                      Sent: Tuesday, November 17, 2020 6:10 AM

                      To: Scharf, Michael <Michael.Scharf@hs-esslingen.de>;
                      Yoshifumi Nishida
                      <nsd.ietf@gmail.com>

                      Cc: Michael Tuexen <tuexen@fh-muenster.de>;
                      tcpm-chairs@ietf.org;
                      Mirja Kuehlewind 
                        <ietf@kuehlewind.net>; Scheffenegger,
                      Richard 
                        <Richard.Scheffenegger@netapp.com>;
                      tcpm IETF list 
                        <tcpm@ietf.org>

                      Subject: Re: AccECN field order

                

              

              
 

              
Michael,

              

                
On 16/11/2020 17:36, Scharf,
                  Michael wrote:

              

              

                
Bob,

                
 

                
One
                    proposal using the length field with *one option
                      codepoint only* is detailed in:
                    
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/

                
 

                
It
                    is the third option mentioned in this e-mail. One
                    example would be to use option length values 5/8/11
                    for one encoding type and option length values
                    6/9/12 for the other encoding type (i.e., order of
                    fields). Or one could use some other combination of
                    length values – the only requirement is that a
                    certain value for the option length is only used by
                    one of the option formats. In that approach, the
                    value of the length field would thus directly
                    describe the encoding of the option. Unless I miss
                    something, this would work and it would just require
                    one option codepoint.

                
 

                
Thus,
                    alternatives to two option codepoints exist and I
                    have explained them on the list in March 2020.

              

              


                OK, sorry, yes, I remember this now.

                

                As I will explain in the AccECN status update talk today
                in virtual Bangkok, the draft has made provision for
                different length values than 5/8/11. It says existing
                implementations MUST accept length values other than
                those currently defined. But then read in as many whole
                3-byte fields as they can.

                

                This can be used to add a flags byte on the end in
                future, for extensibility. Or any other form of
                extensibility the WG might decide in the future.

                

                I know a flags byte at the end seems odd compared to at
                the beginning. But (if decided it's needed in future)
                it's reasonably easy to implement by reading the whole
                option, then processing the last byte, before reading
                the rest of the option.

                

                I believe you will agree that this is a better way to
                utilize different lengths.

                

                And thank you for repeatedly emphasizing that you're
                happy with the 2-kind scheme, or other alternatives.

                

                

                Bob

                

                

                

                

              

                
 

                
Anyway,
                    I don’t really care how the options are encoded as
                    long as the receiver doesn’t need per-connection
                    state for decoding a TCP option. So, personally, I
                    would be fine with using e.g. the length field as
                    described in my old e-mail. Or an additional flag
                    byte. And one could come up with further encodings,
                    e.g., by using one or a few bits as a short “type”
                    field for each counter. This is all about protocol
                    engineering. And all these variants have their pros
                    and cons.

                
 

                
I am
                    also fine with using two option codepoints as
                    specified in -13; this is probably the approach that
                    consumes the least number of bits.

                
 

                
Michael
                    (w/o any hat)

                
 

                
 

                
 

                

                  

                    

                      
From: Bob Briscoe 
                          <ietf@bobbriscoe.net> 

                        Sent: Monday, November 16, 2020 5:52 PM

                        To: Yoshifumi Nishida <nsd.ietf@gmail.com>

                        Cc: Scharf, Michael <Michael.Scharf@hs-esslingen.de>;
                        Michael Tuexen
                        <tuexen@fh-muenster.de>;
                        
                          tcpm-chairs@ietf.org; Mirja Kuehlewind <ietf@kuehlewind.net>;
                        Scheffenegger, Richard
                        <Richard.Scheffenegger@netapp.com>;
                        tcpm IETF list
                        <tcpm@ietf.org>

                        Subject: AccECN field order

                    

                  

                  
 

                  
Yoshi,
                    (adding the tcpm list in cc)

                  

                    
On 05/11/2020 06:58, Yoshifumi
                      Nishida wrote:

                  

                  

                    

                      

                        
Hi Bob,

                      

                      

                        
 

                      

                      

                        

                          
On Wed, Nov 4, 2020 at
                            3:29 PM Bob Briscoe <ietf@bobbriscoe.net>
                            wrote:

                        

                        

                          

                            
Yoshi,

                            

                              
On 04/11/2020 06:51,
                                Yoshifumi Nishida wrote:

                            

                            

                              

                                

                                  
Hi, folks,

                                

                              

                            

                          

                        

                      

                    

                  

                  


                    

                    

                    

                    

                  

                    

                      

                        

                          

                            

                              

                                

                                  
In my
                                    understanding, I'm not sure if we
                                    settled down on using two option
                                    kinds or encoding schemes for 24bits
                                    fields in acc ecn draft.

                                

                                

                                  
So, I think
                                    there're still something to be
                                    clarified and hope things will be
                                    settled at the meeting.

                                

                              

                            

                            


                              [BB] I know a WG can change it's mind at
                              any time. But I'd rather we just clarified
                              what a previous decision was, to avoid the
                              need to keep re-opening discussion on a
                              question that have been decided then
                              changed three different ways already.

                              

                              My memory is not so good these days. I
                              trusted that Michael S remembered the
                              decision correctly, and I seem to remember
                              that decision being made.

                              I've just checked the minutes of the last
                              interim:

                                  
https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600/

                              and they mention Michael's proposal to use
                              two kinds, but don't record any decision.

                              The jabber log gives no clues about any
                              decision.

                              

                              I can't find an audio or video recording.
                              Can you point me at one?

                          

                        

                        

                          
 

                        

                        

                          
I thought that it's
                            because there was no clear decision at the
                            meeting. 

                        

                        
                        

                          
Please let us know if you
                            have any questions or opinions with regard
                            to this.

                        

                      

                    

                  

                  


                    [BB] I checked the Youtube link you sent below.

                    

                    First I think we're agreed no-one was fighting for
                    us to keep the previous way we did this (using the
                    initial value of the field to set the order for the
                    connection).
                    

                    In my presentation I said there was strong
                    resistance from Michael to do it a different way.
                    

                    (also, offlist, the co-authors including me also
                    didn't like this so much. And Ilpo said it made the
                    implementation complex.)

                    

                    Then came the question of what we do instead. There
                    were three alternative proposals:

                    a) use 2 option kinds

                    b) add a flags byte

                    c) somehow use the length field maybe

                    

                    Michael raised (c) in the meeting as a possibility,
                    but no-one could think how to distinguish two
                    options of the same length but a different field
                    order using the length field. Michael said he'd post
                    any ideas to the list if he could think of any, but
                    that didn't happen.

                    

                    So we're left choosing between (a) and (b).

                    I said in the meeting (and on the list when
                    discussing with Ilpo) that I'd be happy to go with
                    (b), but only if there was another use for a flag.
                    Because it would consume 1B more options space in
                    many packets, which is a scarce resource.

                    

                    Ilpo had a proposed use for another flag (to help
                    synch counters after a loss), but I think the
                    discussion about it ended that it wouldn't be
                    helpful, 'cos the way it worked depended on itself
                    (circular logic).

                    

                    In conclusion, I don't think there was an explicit
                    decision to go with 2 option kinds, but it ended up
                    as the 'last person standing'.

                    I like it. It's simple. And apparently option kinds
                    are not such a scarce resource.

                    

                    Perhaps we can ratify this in the WG tomorrow.

                    

                    

                    Bob

                    

                    

                    

                    

                    

                  

                    

                      

                        

                          
 

                        

                        

                          
Thanks,

                        

                        

                          
--

                        

                        

                          
Yoshi

                        

                      

                    

                  

                  


                    

                    

                    

                    

                  
-- 

                  
________________________________________________________________

                  
Bob Briscoe                               http://bobbriscoe.net/

                

              

              


                

                

                

              
-- 

              
________________________________________________________________

              
Bob Briscoe                               http://bobbriscoe.net/

            

          

          


            

            

          
-- 

          
________________________________________________________________

          
Bob Briscoe                               http://bobbriscoe.net/

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------A7CFC12E4CB829C7B818DDB0--


From nobody Tue Nov 17 21:36:24 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3ED273A1451; Tue, 17 Nov 2020 21:36:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aUA0epsIqX0W; Tue, 17 Nov 2020 21:36:21 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8DA3B3A144D; Tue, 17 Nov 2020 21:36:20 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 9886425A13; Wed, 18 Nov 2020 06:36:18 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1605677778; bh=dRZ8tpmw9h6b87K9J2gDzbOWbN5WL88gjOPCCYnL04U=; h=From:To:CC:Subject:Date:From; b=nNsyBDHum7CU3v5v23gM78wl8lj7cIYJykC26IQYFn2DKIFrYyG7sLbz0UHDBS+7B 0EJb9JvC3a1gsB8jt/meOf77lJ4P4pJF3rr9KpMNRowhFLmxRShpSoIOwO5YsB2UR8 A8vQuZa8h2Wu2rrFfeW/xnu3JAcQtJzC3LOBJG8c=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EFVQ1SDYzJ21; Wed, 18 Nov 2020 06:36:17 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Wed, 18 Nov 2020 06:36:17 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Wed, 18 Nov 2020 06:36:17 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Wed, 18 Nov 2020 06:36:17 +0100
From: "Scharf, Michael" 
To: Bob Briscoe , Yoshifumi Nishida 
CC: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
Thread-Topic: Security issues of AccECN (was: RE: AccECN field order)
Thread-Index: Ada9a3nGhSM/yBYjQcaGPWFu48rdFA==
Date: Wed, 18 Nov 2020 05:36:16 +0000
Message-ID: <20ac3652d55d4bc19c9c37714c59c95c@hs-esslingen.de>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: multipart/alternative; boundary="_000_20ac3652d55d4bc19c9c37714c59c95chsesslingende_"
MIME-Version: 1.0
Archived-At: 
Subject: [tcpm] Security issues of AccECN (was: RE: AccECN field order)
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 18 Nov 2020 05:36:23 -0000

--_000_20ac3652d55d4bc19c9c37714c59c95chsesslingende_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

T25seSBvbiByZW1hcmsgW21zMl0gcmVnYXJkaW5nIHRoZSBpc3N1ZSBvZiBhIGNvdmVydCBjaGFu
bmVsIGR1ZSB0byDigJ5mb3J3YXJkIGNvbXBhdGliaWxpdHnigJzigKYNCg0KRnJvbSBhIHNlY3Vy
aXR5IHBlcnNwZWN0aXZlLCBpdCBpcyBub3QgY2xlYXIgdG8gbWUgd2hldGhlciBhbGxvd2luZyBh
cmJpdHJhcnkgdW5zcGVjaWZpZWQgYnl0ZXMgaW4gYSBUQ1Agb3B0aW9uIGlzIGEgZ29vZCBpZGVh
ICphdCBhbGwqLiBJdCB3aWxsIGJlIGludGVyZXN0aW5nIHRvIGhlYXIgdGhlIG9waW5pb24gZnJv
bSBTRUMgYXJlYSBvbiB0aGF0LiBQZXJzb25hbGx5LCBJIGFtIG5vdCBjb252aW5jZWQgdGhhdCB0
aGlzIHJlYWxseSBtYWtlcyBzZW5zZSwgYnV0IEkgbXkgY29uY2VybnMgYXJlIG5vdCBzdHJvbmcg
ZW5vdWdoIHRvIGZvcm1hbGx5IHB1c2ggYmFjay4gSeKAmWxsIGxlYXZlIGl0IHRvIG90aGVycyB0
byB0aGluayBhYm91dCB3aGV0aGVyIHRoaXMgaXMgYSBidWcgb3IgYSBmZWF0dXJlLg0KDQpbQkJd
IFdlbGwsIGxldCdzIGZpcnN0IHRyeSB0byBkZWFsIHdpdGggc2VjdXJpdHkgb3Vyc2VsdmVzOg0K
KiBPY3RldHMgdGhhdCBhcmUgZXhwbGljaXRseSBkZWNsYXJlZCBhcyBwYXJ0IG9mIGFuIG9wdGlv
biBjYW5ub3QgYmUgdXNlZCBmb3IgYSBidWZmZXIgb3ZlcmZsb3cgYXR0YWNrLiBJIGRvbid0IHJl
YWxseSBuZWVkIHRvLCBidXQgSSBjb3VsZCBhZGQgdGhlIGZvbGxvd2luZyB0ZXh0IHRvIHRoZSBm
b3J3YXJkIGNvbXBhdGliaWxpdHkgd29yZGluZzoNCiAgICBBIHJlY2VpdmVyIGNvbnNpZGVycyBv
Y3RldHMgYmV5b25kIHRob3NlIGl0IHVzZXMsIGJ1dCB3aXRoaW4gdGhlIHNwZWNpZmllZCBsZW5n
dGgsIGFzIGlmIHRoZXkgYXJlIHBhZGRpbmcuDQoqIEFuZCBzdWNoIG9jdGV0cyBjYW5ub3QgYmUg
YW55IGRpZmZlcmVudCBmcm9tIHRoZSBjdXJyZW50IGFiaWxpdHkgb2YgYSBzZW5kZXIgdG8gYWRk
IHBhZGRpbmcuIFNvIHRoZXJlJ3Mgbm8gbmV3IGF0dGFjayBwb3NzaWJsZSBoZXJlLg0KKiBBbmQg
dGhlcmUncyBubyBuZWVkIHRvIHNwZWNpZnkgYSBtYXggbGVuZ3RoIGZvciBhbnkgQWNjRUNOIE9w
dGlvbiwgd2hpY2ggd291bGQganVzdCB1bm5lY2Vzc2FyaWx5IGxpbWl0IGZsZXhpYmlsaXR5Lg0K
DQpbbXNdIEkgaGF2ZSBtb3JlIHRob3VnaHQgYWJvdXQgdGhlIGNvdmVydCBjaGFubmVsIHRoYXQg
c29tZSB1bnNwZWNpZmllZCBieXRlcyBpbiBhIHN0YW5kYXJkcyB0cmFjayBUQ1Agb3B0aW9uIGNv
dWxkIG9mZmVy4oCmIEluIHBhcnRpY3VsYXIgaWYgdGhlIEFjY0VDTiBvcHRpb24gZ2V0cyB3aWRl
bHkgZGVwbG95ZWQgYW5kIHVzZWQsIHdvdWxkIHNvbWUgbWFsaWNpb3VzIHVzZXJzIGZpbmQgdGhv
c2UgYnl0ZXMgdXNlZnVsPyBGb3Igd2hhdD8gRm9yIGluc3RhbmNlLCB3aGF0IGluIHRoZSBBY2NF
Q04gcHJvdG9jb2wgZGVzaWduIHByZXZlbnRzIHVzZSBvZiB0aGVzZSDigJxwYWRkaW5n4oCdIGJ5
dGVzIGFzIHN1cGVyLWNvb2tpZT8gVG8gbWUsIGl0IGlzIG11Y2ggbW9yZSBpbXBvcnRhbnQgdG8g
dGhpbmsgYWJvdXQgYWJ1c2Ugb2Ygd2hhdGV2ZXIgd2Ugc3RhbmRhcmRpemUgKm5vdyosIGluc3Rl
YWQgb2YgZW5naW5lZXJpbmcgZm9yIHNvbWUgZW50aXJlbHkgdW5rbm93biBmdXR1cmUuIEkgYW0g
bm90IGEgc2VjdXJpdHkgZXhwZXJ0LCBidXQgSSBzdWdnZXN0IGEgY2FyZWZ1bCBhbmFseXNpcyBv
ZiBhbnkgc2VjdXJpdHkgaW1wbGljYXRpb25zIG9mIHRoZSBBY2NFQ04gb3B0aW9uLiBXZSBzZWxk
b21seSBzcGVjaWZ5IG5ldyBUQ1Agb3B0aW9ucyBhbmQgbWFsaWNpb3VzIHVzZXJzIHdpbGwgY2Vy
dGFpbmx5IGxvb2sgZm9yIOKAnG9wcG9ydHVuaXRpZXPigJ0gaW4gdGhlIHNwZWMuIFNvIGZhciwg
SSBkb27igJl0IGtub3cgd2hldGhlciB0aGF0IHdob2xlIGlkZWEgb2YgZm9yd2FyZCBjb21wYXRp
YmlsaXR5IGlzIGEgZmVhdHVyZSBvciBhIGJ1Zy4NCg0KW0JCXSBJZiB0aGF0IHdlcmUgYSBwcm9i
bGVtLCBpdCBjb3VsZCBhbHJlYWR5IGJlIGRvbmUsIGJlY2F1c2UgKGJ5IG15IHJlYWRpbmcgb2Yg
UkZDNzkzKSBpdCBpcyB2YWxpZCB0byBzZXQgdGhlIGRhdGEgb2Zmc2V0IHRvIHN0cmV0Y2ggb3V0
IHRoZSBwYWRkaW5nIHRvIGFueSBsZW5ndGggd2l0aGluIHRoZSBtYXggb3B0aW9uIHNwYWNlIGxp
bWl0cy4NCg0KW21zMl0gSSBkb27igJl0IHVuZGVyc3RhbmQuIFJGQyA3OTMgbWFuZGF0ZXMgdGhh
dCBwYWRkaW5nIGlzIGNvbXBvc2VkIG9mIHplcm9zLCBpLmUuLCBhIGZpcmV3YWxsIG9yIElEUyBj
b3VsZCBkZXRlY3QgYSBub24temVybyBjb3ZlcnQgY2hhbm5lbCBpbiB0aGUgcGFkZGluZyAoYW5k
IGV2ZW4gcmVtb3ZlIHRoZSBwYWRkaW5nKS4gQWxzbywgdGhlcmUgYXJlIElNSE8gbm8gcmVhc29u
YWJsZSBlbmdpbmVlcmluZyByZWFzb25zIGZvciBtYWtpbmcgdGhlIHBhZGRpbmcgbG9uZ2VyIHRo
YW4gbmVlZGVkLCBpLmUuIHN1Y2ggYSBjb3ZlcnQgY2hhbm5lbCBjb3VsZCBiZSBkZXRlY3RlZCBh
cyBhbm9tYWxpdHkuIEFzIGZhciBhcyBJIHVuZGVyc3RhbmQgUkZDIDc5MywgYSBtaWRkbGVib3gg
Y291bGQgYWxzbyBzYWZlbHkgcmVtb3ZlIHRoZSBhZGRpdGlvbmFsIHBhZGRpbmcgdy9vIGltcGFj
dGluZyBhIHN0YW5kYXJkLWNvbXBsaWFudCBUQ1AgY29ubmVjdGlvbi4NCg0KW21zMl0gQWxsb3dp
bmcgYXJiaXRyYXJ5IGJ5dGVzIGluIGEgVENQIG9wdGlvbiBpcyBJTUhPIGEgdmVyeSBkaWZmZXJl
bnQgc3RvcnksIGluIHBhcnRpY3VsYXIgaWYgeW91IGFsbG93IHRoZSB2YWx1ZXMgdG8gaGF2ZSBu
b24temVybyB2YWx1ZXMgYW5kIGZvcmJpZCBtaWRkbGVib3hlcyB0byBjaGVjayBvciByZW1vdmUg
cGFydHMgb2YgdGhlIG9wdGlvbiB0aGF0IGFyZSBub3Qgc3RhbmRhcmRpemVkIGJ5IHRoZSBJRVRG
LiBJZiBJIGNvcnJlY3RseSB1bmRlcnN0YW5kIGRyYWZ0IC0xMywgYW4gYXR0YWNrZXIgY291bGQg
dHJhbnNtaXQgMjkgYnl0ZXMgcGF5bG9hZCBhdCB0aGUgZW5kIG9mIHRoZSBBY2NFQ04gb3B0aW9u
LCBpbiBhbnkgVENQIGNvbm5lY3Rpb24sIGZvciBhbnkgcHVycG9zZS4NCg0KW21zMl0gSSB0aGlu
ayB0aGUg4oCeZm9yd2FyZCBjb21wYXRpYmlsaXR54oCcIChlLmcuLCBTZWN0aW9uIDMuMy4yIGlu
IHRoZSBJLUQpIG5lZWRzIGEgdmVyeSBjYXJlZnVsIHNlY3VyaXR5IGFuYWx5c2lzLiBBcyBmYXIg
YXMgSSB1bmRlcnN0YW5kLCB0aGlzIHdob2xlIGlkZWEgb2Yg4oCeZm9yd2FyZCBjb21wYXRpYmls
aXR54oCcIGlzIG5vdCBuZWVkZWQgZm9yIHVzaW5nIHRoZSAqY3VycmVudGx5KiBzdGFuZGFyZGl6
ZWQgQWNjRUNOIG9wdGlvbnMuIElNSE8geW91IGNvdWxkIGp1c3QgcmVtb3ZlIHRoYXQg4oCeZm9y
d2FyZCBjb21wYXRpYmlsaXR54oCcIGlkZWEgdy9vIGltcGFjdGluZyB0aGUgcmVzdCBvZiB0aGUg
QWNjRUNOIHByb3RvY29sIGFzIGN1cnJlbnRseSBzcGVjaWZpZWQgKGluY2x1ZGluZyB0aGUgY3Vy
cmVudCBmb3JtYXQgb2YgdGhlIHR3byBvcHRpb25zKS4gSWYgdGhlIGlkZWEgb2Yg4oCeZm9yd2Fy
ZCBjb21wYXRpYmlsaXR54oCcIHN0YXlzIGluIHRoZSBkb2N1bWVudCwgSU1ITyB0aGUgc2VjdXJp
dHkgc2VjdGlvbiBuZWVkcyBhIGRlZGljYXRlZCBkaXNjdXNzaW9uIHJlZ2FyZGluZyBwb3RlbnRp
YWwgbWlzdXNlIGFzIGNvdmVydCBjaGFubmVsLiBUaGlzIGNvdWxkIGVpdGhlciBiZSBhIHNlY3Vy
aXR5IG9yIHByaXZhY3kgaXNzdWUsIGRlcGVuZGluZyBvbiB3aG8gcHV0cyB3aGF0IGFkZGl0aW9u
YWwgYnl0ZXMgaW50byB0aGUgb3B0aW9uIGFuZCBjbGFpbXMgdGhhdCB0aGVzZSBieXRlcyB0byBi
ZSBBY2NFQ052Mi4NCg0KTWljaGFlbA0K

--_000_20ac3652d55d4bc19c9c37714c59c95chsesslingende_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDb25zb2xhczsNCglwYW5vc2UtMToyIDEx
IDYgOSAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWws
IGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0
b206LjAwMDFwdDsNCglmb250LXNpemU6MTEuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIixz
YW5zLXNlcmlmO30NCmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsNCgl7bXNvLXN0eWxlLXByaW9y
aXR5Ojk5Ow0KCWNvbG9yOmJsdWU7DQoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9DQphOnZp
c2l0ZWQsIHNwYW4uTXNvSHlwZXJsaW5rRm9sbG93ZWQNCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5
Ow0KCWNvbG9yOnB1cnBsZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCnByZQ0KCXtt
c28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJbXNvLXN0eWxlLWxpbms6IkhUTUwgVm9yZm9ybWF0aWVy
dCBaY2huIjsNCgltYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNp
emU6MTAuMHB0Ow0KCWZvbnQtZmFtaWx5OiJDb3VyaWVyIE5ldyIsc2VyaWY7fQ0KdHQNCgl7bXNv
LXN0eWxlLXByaW9yaXR5Ojk5Ow0KCWZvbnQtZmFtaWx5OiJDb3VyaWVyIE5ldyIsc2VyaWY7fQ0K
c3Bhbi5IVE1MVm9yZm9ybWF0aWVydFpjaG4NCgl7bXNvLXN0eWxlLW5hbWU6IkhUTUwgVm9yZm9y
bWF0aWVydCBaY2huIjsNCgltc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJbXNvLXN0eWxlLWxpbms6
IkhUTUwgVm9yZm9ybWF0aWVydCI7DQoJZm9udC1mYW1pbHk6Q29uc29sYXM7fQ0KcC5tc29ub3Jt
YWwwLCBsaS5tc29ub3JtYWwwLCBkaXYubXNvbm9ybWFsMA0KCXttc28tc3R5bGUtbmFtZTptc29u
b3JtYWw7DQoJbXNvLW1hcmdpbi10b3AtYWx0OmF1dG87DQoJbWFyZ2luLXJpZ2h0OjBjbTsNCglt
c28tbWFyZ2luLWJvdHRvbS1hbHQ6YXV0bzsNCgltYXJnaW4tbGVmdDowY207DQoJZm9udC1zaXpl
OjEyLjBwdDsNCglmb250LWZhbWlseToiVGltZXMgTmV3IFJvbWFuIixzZXJpZjt9DQpzcGFuLkUt
TWFpbEZvcm1hdHZvcmxhZ2UyMQ0KCXttc28tc3R5bGUtdHlwZTpwZXJzb25hbDsNCglmb250LWZh
bWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjsNCgljb2xvcjp3aW5kb3d0ZXh0O30NCnNwYW4uRS1N
YWlsRm9ybWF0dm9ybGFnZTIyDQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZv
bnQtZmFtaWx5OiJDYWxpYnJpIixzYW5zLXNlcmlmOw0KCWNvbG9yOiMxRjQ5N0Q7fQ0KLk1zb0No
cERlZmF1bHQNCgl7bXNvLXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1zaXplOjEwLjBw
dDt9DQpAcGFnZSBXb3JkU2VjdGlvbjENCgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2lu
OjcwLjg1cHQgNzAuODVwdCAyLjBjbSA3MC44NXB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFn
ZTpXb3JkU2VjdGlvbjE7fQ0KLS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxv
OnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtl
bmRpZl0tLT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJl
ZGl0Ij4NCjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0
PjwveG1sPjwhW2VuZGlmXS0tPg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iREUiIGxpbms9ImJsdWUi
IHZsaW5rPSJwdXJwbGUiPg0KPGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEO21zby1mYXJlYXN0LWxhbmd1YWdl
OkVOLVVTIj5Pbmx5IG9uIHJlbWFyayBbbXMyXSByZWdhcmRpbmcgdGhlIGlzc3VlIG9mIGEgY292
ZXJ0IGNoYW5uZWwgZHVlIHRvIOKAnmZvcndhcmQgY29tcGF0aWJpbGl0eeKAnOKApjxvOnA+PC9v
OnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJjb2xvcjoj
MUY0OTdEO21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bh
bj48L3A+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEu
NXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQiPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdp
bi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9u
ZTtib3JkZXItbGVmdDpzb2xpZCBibHVlIDEuNXB0O3BhZGRpbmc6MGNtIDBjbSAwY20gNC4wcHQi
Pg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBw
dCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1m
YXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5Gcm9tIGEgc2VjdXJpdHkgcGVyc3BlY3RpdmUsIGl0IGlz
IG5vdCBjbGVhciB0byBtZSB3aGV0aGVyIGFsbG93aW5nIGFyYml0cmFyeSB1bnNwZWNpZmllZCBi
eXRlcyBpbiBhIFRDUCBvcHRpb24gaXMgYSBnb29kIGlkZWEgKjxiPmF0IGFsbDwvYj4qLiBJdCB3
aWxsIGJlIGludGVyZXN0aW5nIHRvIGhlYXIgdGhlIG9waW5pb24NCiBmcm9tIFNFQyBhcmVhIG9u
IHRoYXQuIFBlcnNvbmFsbHksIEkgYW0gbm90IGNvbnZpbmNlZCB0aGF0IHRoaXMgcmVhbGx5IG1h
a2VzIHNlbnNlLCBidXQgSSBteSBjb25jZXJucyBhcmUgbm90IHN0cm9uZyBlbm91Z2ggdG8gZm9y
bWFsbHkgcHVzaCBiYWNrLiBJ4oCZbGwgbGVhdmUgaXQgdG8gb3RoZXJzIHRvIHRoaW5rIGFib3V0
IHdoZXRoZXIgdGhpcyBpcyBhIGJ1ZyBvciBhIGZlYXR1cmUuPC9zcGFuPjxvOnA+PC9vOnA+PC9w
Pg0KPC9ibG9ja3F1b3RlPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJyPg0KW0JCXSBXZWxsLCBs
ZXQncyBmaXJzdCB0cnkgdG8gZGVhbCB3aXRoIHNlY3VyaXR5IG91cnNlbHZlczo8YnI+DQoqIE9j
dGV0cyB0aGF0IGFyZSBleHBsaWNpdGx5IGRlY2xhcmVkIGFzIHBhcnQgb2YgYW4gb3B0aW9uIGNh
bm5vdCBiZSB1c2VkIGZvciBhIGJ1ZmZlciBvdmVyZmxvdyBhdHRhY2suIEkgZG9uJ3QgcmVhbGx5
IG5lZWQgdG8sIGJ1dCBJIGNvdWxkIGFkZCB0aGUgZm9sbG93aW5nIHRleHQgdG8gdGhlIGZvcndh
cmQgY29tcGF0aWJpbGl0eSB3b3JkaW5nOg0KPGJyPg0KJm5ic3A7Jm5ic3A7Jm5ic3A7IEEgcmVj
ZWl2ZXIgY29uc2lkZXJzIG9jdGV0cyBiZXlvbmQgdGhvc2UgaXQgdXNlcywgYnV0IHdpdGhpbiB0
aGUgc3BlY2lmaWVkIGxlbmd0aCwgYXMgaWYgdGhleSBhcmUgcGFkZGluZy4NCjxicj4NCiogQW5k
IHN1Y2ggb2N0ZXRzIGNhbm5vdCBiZSBhbnkgZGlmZmVyZW50IGZyb20gdGhlIGN1cnJlbnQgYWJp
bGl0eSBvZiBhIHNlbmRlciB0byBhZGQgcGFkZGluZy4gU28gdGhlcmUncyBubyBuZXcgYXR0YWNr
IHBvc3NpYmxlIGhlcmUuPGJyPg0KKiBBbmQgdGhlcmUncyBubyBuZWVkIHRvIHNwZWNpZnkgYSBt
YXggbGVuZ3RoIGZvciBhbnkgQWNjRUNOIE9wdGlvbiwgd2hpY2ggd291bGQganVzdCB1bm5lY2Vz
c2FyaWx5IGxpbWl0IGZsZXhpYmlsaXR5Ljxicj4NCjxicj4NCjxzcGFuIGxhbmc9IkVOLVVTIj5b
bXNdIEkgaGF2ZSBtb3JlIHRob3VnaHQgYWJvdXQgdGhlIGNvdmVydCBjaGFubmVsIHRoYXQgc29t
ZSB1bnNwZWNpZmllZCBieXRlcyBpbiBhIHN0YW5kYXJkcyB0cmFjayBUQ1Agb3B0aW9uIGNvdWxk
IG9mZmVy4oCmIEluIHBhcnRpY3VsYXIgaWYgdGhlIEFjY0VDTiBvcHRpb24gZ2V0cyB3aWRlbHkg
ZGVwbG95ZWQgYW5kIHVzZWQsIHdvdWxkIHNvbWUgbWFsaWNpb3VzIHVzZXJzIGZpbmQgdGhvc2Ug
Ynl0ZXMgdXNlZnVsPw0KIEZvciB3aGF0PyBGb3IgaW5zdGFuY2UsIHdoYXQgaW4gdGhlIEFjY0VD
TiBwcm90b2NvbCBkZXNpZ24gcHJldmVudHMgdXNlIG9mIHRoZXNlIOKAnHBhZGRpbmfigJ0gYnl0
ZXMgYXMgc3VwZXItY29va2llPyBUbyBtZSwgaXQgaXMgbXVjaCBtb3JlIGltcG9ydGFudCB0byB0
aGluayBhYm91dCBhYnVzZSBvZiB3aGF0ZXZlciB3ZSBzdGFuZGFyZGl6ZSAqPGI+bm93PC9iPios
IGluc3RlYWQgb2YgZW5naW5lZXJpbmcgZm9yIHNvbWUgZW50aXJlbHkgdW5rbm93bg0KIGZ1dHVy
ZS4gSSBhbSBub3QgYSBzZWN1cml0eSBleHBlcnQsIGJ1dCBJIHN1Z2dlc3QgYSBjYXJlZnVsIGFu
YWx5c2lzIG9mIGFueSBzZWN1cml0eSBpbXBsaWNhdGlvbnMgb2YgdGhlIEFjY0VDTiBvcHRpb24u
IFdlIHNlbGRvbWx5IHNwZWNpZnkgbmV3IFRDUCBvcHRpb25zIGFuZCBtYWxpY2lvdXMgdXNlcnMg
d2lsbCBjZXJ0YWlubHkgbG9vayBmb3Ig4oCcb3Bwb3J0dW5pdGllc+KAnSBpbiB0aGUgc3BlYy4g
U28gZmFyLCBJIGRvbuKAmXQga25vdyB3aGV0aGVyDQogdGhhdCB3aG9sZSBpZGVhIG9mIGZvcndh
cmQgY29tcGF0aWJpbGl0eSBpcyBhIGZlYXR1cmUgb3IgYSBidWcuPC9zcGFuPjxvOnA+PC9vOnA+
PC9wPg0KPC9kaXY+DQo8L2Jsb2NrcXVvdGU+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBz
dHlsZT0iZm9udC1zaXplOjEyLjBwdDtmb250LWZhbWlseTomcXVvdDtUaW1lcyBOZXcgUm9tYW4m
cXVvdDssc2VyaWYiPjxicj4NCltCQl0gSWYgdGhhdCB3ZXJlIGEgcHJvYmxlbSwgaXQgY291bGQg
YWxyZWFkeSBiZSBkb25lLCBiZWNhdXNlIChieSBteSByZWFkaW5nIG9mIFJGQzc5MykgaXQgaXMg
dmFsaWQgdG8gc2V0IHRoZSBkYXRhIG9mZnNldCB0byBzdHJldGNoIG91dCB0aGUgcGFkZGluZyB0
byBhbnkgbGVuZ3RoIHdpdGhpbiB0aGUgbWF4IG9wdGlvbiBzcGFjZSBsaW1pdHMuPG86cD48L286
cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMx
RjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxzcGFuIHN0eWxlPSJjb2xvcjojMUY0OTdEIj5bbXMyXSBJIGRvbuKAmXQgdW5kZXJzdGFuZC4g
UkZDIDc5MyBtYW5kYXRlcyB0aGF0IHBhZGRpbmcgaXMgY29tcG9zZWQgb2YgemVyb3MsIGkuZS4s
IGEgZmlyZXdhbGwgb3IgSURTIGNvdWxkIGRldGVjdCBhIG5vbi16ZXJvIGNvdmVydCBjaGFubmVs
IGluIHRoZSBwYWRkaW5nIChhbmQgZXZlbiByZW1vdmUgdGhlIHBhZGRpbmcpLiBBbHNvLCB0aGVy
ZSBhcmUgSU1ITw0KIG5vIHJlYXNvbmFibGUgZW5naW5lZXJpbmcgcmVhc29ucyBmb3IgbWFraW5n
IHRoZSBwYWRkaW5nIGxvbmdlciB0aGFuIG5lZWRlZCwgaS5lLiBzdWNoIGEgY292ZXJ0IGNoYW5u
ZWwgY291bGQgYmUgZGV0ZWN0ZWQgYXMgYW5vbWFsaXR5LiBBcyBmYXIgYXMgSSB1bmRlcnN0YW5k
IFJGQyA3OTMsIGEgbWlkZGxlYm94IGNvdWxkIGFsc28gc2FmZWx5IHJlbW92ZSB0aGUgYWRkaXRp
b25hbCBwYWRkaW5nIHcvbyBpbXBhY3RpbmcgYSBzdGFuZGFyZC1jb21wbGlhbnQNCiBUQ1AgY29u
bmVjdGlvbi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh
biBzdHlsZT0iY29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPlttczJdIEFsbG93
aW5nIGFyYml0cmFyeSBieXRlcyBpbiBhIFRDUCBvcHRpb24gaXMgSU1ITyBhIHZlcnkgZGlmZmVy
ZW50IHN0b3J5LCBpbiBwYXJ0aWN1bGFyIGlmIHlvdSBhbGxvdyB0aGUgdmFsdWVzIHRvIGhhdmUg
bm9uLXplcm8gdmFsdWVzIGFuZCBmb3JiaWQgbWlkZGxlYm94ZXMgdG8gY2hlY2sgb3IgcmVtb3Zl
IHBhcnRzIG9mIHRoZSBvcHRpb24gdGhhdA0KIGFyZSBub3Qgc3RhbmRhcmRpemVkIGJ5IHRoZSBJ
RVRGLiBJZiBJIGNvcnJlY3RseSB1bmRlcnN0YW5kIGRyYWZ0IC0xMywgYW4gYXR0YWNrZXIgY291
bGQgdHJhbnNtaXQgMjkgYnl0ZXMgcGF5bG9hZCBhdCB0aGUgZW5kIG9mIHRoZSBBY2NFQ04gb3B0
aW9uLCBpbiBhbnkgVENQIGNvbm5lY3Rpb24sIGZvciBhbnkgcHVycG9zZS48bzpwPjwvbzpwPjwv
c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3
RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gc3R5bGU9ImNvbG9yOiMxRjQ5N0QiPlttczJdIEkgdGhpbmsgdGhlIOKAnmZvcndhcmQgY29t
cGF0aWJpbGl0eeKAnCAoZS5nLiwgU2VjdGlvbiAzLjMuMiBpbiB0aGUgSS1EKSBuZWVkcyBhIHZl
cnkgY2FyZWZ1bCBzZWN1cml0eSBhbmFseXNpcy4gQXMgZmFyIGFzIEkgdW5kZXJzdGFuZCwgdGhp
cyB3aG9sZSBpZGVhIG9mIOKAnmZvcndhcmQgY29tcGF0aWJpbGl0eeKAnCBpcyBub3QgbmVlZGVk
IGZvciB1c2luZyB0aGUNCiAqPGI+Y3VycmVudGx5PC9iPiogc3RhbmRhcmRpemVkIEFjY0VDTiBv
cHRpb25zLiBJTUhPIHlvdSBjb3VsZCBqdXN0IHJlbW92ZSB0aGF0IOKAnmZvcndhcmQgY29tcGF0
aWJpbGl0eeKAnCBpZGVhIHcvbyBpbXBhY3RpbmcgdGhlIHJlc3Qgb2YgdGhlIEFjY0VDTiBwcm90
b2NvbCBhcyBjdXJyZW50bHkgc3BlY2lmaWVkIChpbmNsdWRpbmcgdGhlIGN1cnJlbnQgZm9ybWF0
IG9mIHRoZSB0d28gb3B0aW9ucykuIElmIHRoZSBpZGVhIG9mIOKAnmZvcndhcmQgY29tcGF0aWJp
bGl0eeKAnA0KIHN0YXlzIGluIHRoZSBkb2N1bWVudCwgSU1ITyB0aGUgc2VjdXJpdHkgc2VjdGlv
biBuZWVkcyBhIGRlZGljYXRlZCBkaXNjdXNzaW9uIHJlZ2FyZGluZyBwb3RlbnRpYWwgbWlzdXNl
IGFzIGNvdmVydCBjaGFubmVsLiBUaGlzIGNvdWxkIGVpdGhlciBiZSBhIHNlY3VyaXR5IG9yIHBy
aXZhY3kgaXNzdWUsIGRlcGVuZGluZyBvbiB3aG8gcHV0cyB3aGF0IGFkZGl0aW9uYWwgYnl0ZXMg
aW50byB0aGUgb3B0aW9uIGFuZCBjbGFpbXMgdGhhdCB0aGVzZQ0KIGJ5dGVzIHRvIGJlIEFjY0VD
TnYyLjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0
eWxlPSJjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFz
cz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iY29sb3I6IzFGNDk3RCI+TWljaGFlbDxvOnA+PC9v
OnA+PC9zcGFuPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8L2JvZHk+DQo8L2h0bWw+DQo=

--_000_20ac3652d55d4bc19c9c37714c59c95chsesslingende_--


From nobody Wed Nov 18 00:28:46 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4F4693A1671 for ; Wed, 18 Nov 2020 00:28:44 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=eggert.org
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UeqxNux39Mf9 for ; Wed, 18 Nov 2020 00:28:40 -0800 (PST)
Received: from mail.eggert.org (mail.eggert.org [IPv6:2a00:ac00:4000:400:211:32ff:fe22:186f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2AA663A1670 for ; Wed, 18 Nov 2020 00:28:40 -0800 (PST)
Received: from [IPv6:2a00:ac00:4000:400:ade6:53ee:6b75:dc4d] (unknown [IPv6:2a00:ac00:4000:400:ade6:53ee:6b75:dc4d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.eggert.org (Postfix) with ESMTPSA id 8D4C96103C9; Wed, 18 Nov 2020 10:28:30 +0200 (EET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=eggert.org; s=dkim; t=1605688110; bh=EC9vSn2wClDhH05CvS4N+EYQOoasxcyEgBjCRJnAozQ=; h=From:Subject:Date:In-Reply-To:Cc:To:References; b=TxZBVjPTpt0QvuYeM88nNtqmyzKVHoLOH3K0RIQuH5EihDgIoqaMwdVv6nDQBVwwu mbbtY311Ck976qwQAuG8Ghql0j4KrYbbnS9dXL5Fans0l50gHZ2A5el7XLkOp8uz5y GtGjVG+npqElDNOkUI+TKDBHbLQUjH0sSX+8Dxa8=
From: Lars Eggert 
Message-Id: 
Content-Type: multipart/signed; boundary="Apple-Mail=_F10EAEAE-A65F-40BD-BA18-C860FFBDA028"; protocol="application/pgp-signature"; micalg=pgp-sha512
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
Date: Wed, 18 Nov 2020 10:28:30 +0200
In-Reply-To: 
Cc: "Scheffenegger, Richard" , Vidhi Goel , "tcpm@ietf.org Extensions" 
To: Yuchung Cheng 
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>  
X-MailScanner-ID: 8D4C96103C9.A2FD5
X-MailScanner: Found to be clean
X-MailScanner-From: lars@eggert.org
Archived-At: 
Subject: [tcpm] Replacing with AIMD emulation (was Re: Poll for CUBIC RFC updates)
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 18 Nov 2020 08:28:44 -0000

--Apple-Mail=_F10EAEAE-A65F-40BD-BA18-C860FFBDA028
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On 2020-11-17, at 19:50, Yuchung Cheng =
 wrote:
> I'd recommend replacing the modelled TCP Reno window approach in
> section 4.2 with an AIMD emulation (Linux's approach).



I created https://github.com/NTAP/rfc8312bis/issues/20 to track this =
suggestion.

Thanks,
Lars


--Apple-Mail=_F10EAEAE-A65F-40BD-BA18-C860FFBDA028
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEmpq0ZpSoejRmyhheVLXDCb9wwVcFAl+02y4ACgkQVLXDCb9w
wVfcLxAAvJHPrCHbp++JQgRY6dYasz+0G/4aEunmyaMCS8UvjHOSRhqfuD4agyJt
D136eoQoltchzydEJ/QFbDyNSxvkt2e45XuDe1+YV6TNzQwEA0HYmP6yjtpsgYPk
fjcTUj8sUIlwInRfTdMC4jNH+o832Aqin2hAji1AuRW2ZIP4GL5lEZUGMjonLqWb
BMzn7r+gKIbK7E/VzPwezuTjAxsGM4LzAkwllyUD/2h9gPY9xM8153HqQKx876ap
HDoYAIWrfH0cOrDnDKfIDFQRDFJ8F57GLqtrx9zeKfN9lSfn4f0DixARTv8TsF1Z
+UBuHLcIJ9DADCekyFVCt62o2KNdyFKZB2AWBM68FsKlXmaoNSRVp8RKLgZIQjcO
m5M5fF3gQKbM72E7Q5tYPfuNpQ6Uy+ORWnOPTOc5Qn4A4nZE8eu5hWHo5avjHAGj
MQ302x1b44XB2N6+PcLXdq4tmL4EXJWobjr3mubbTPB76H7RgtXDFONRhY2E0qzg
fg30nmnEOIfrdRd8IF+Ogji9NTE4HySn1Jw93cpNrSnQMbRYrjwpTIoTD+2T20GO
FDm7kqVfOw5Tp95XSZojZC9MQTMtSUs4qmieLDy0c5Hd2QlY0j3t+r/oejrqzrhh
vQEs3vJVBI7xn5YmU5UlAl+gMl/7RQMYhnxlzWKyuNRArMT1tDU=
=SnoU
-----END PGP SIGNATURE-----

--Apple-Mail=_F10EAEAE-A65F-40BD-BA18-C860FFBDA028--


From nobody Wed Nov 18 10:39:45 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9EEA43A067A; Wed, 18 Nov 2020 10:39:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.1
X-Spam-Level: 
X-Spam-Status: No, score=-2.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yN6pv-l4qfxg; Wed, 18 Nov 2020 10:39:41 -0800 (PST)
Received: from cl3.bcs-hosting.net (cl3.bcs-hosting.net [3.11.37.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D289E3A03FA; Wed, 18 Nov 2020 10:39:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=VmCRyIW3YSPjvtjqIb8UoZPVF7Xh3gxuAhnf1R1JNhk=; b=RmrGzajJ4HGRsKrm/zlqwxM/j L5/GNeAdzrGd7j8Asq5otFHlZNdn/M/r+l6CVCU7xadYqDZT+E1QV1QuEy5HLmgzJJjOEI9Id3ThU ggbrKJ8WmHNUGf3zMTW1kTsUw/xIlpggR6Md1/qoFiPS83F2xzWXHXALtrJySiIo8E77pAJo6PE9h VYR6rGng/u6YHryeQ+kxoyyEcPztrITYvLQmYCWb5xQirDqFjSO4Ak71f8IPM9CYIhnFdtXNUo1ht y77cb8oVtmD9Xz8gSj7YSzGsGLiWDugizER+ggLIta4XwxfA5WY0a3TZOeC/7N+G+9zjtWIGEPU/o jUj+yOQgA==;
Received: from 67.153.238.178.in-addr.arpa ([178.238.153.67]:52528 helo=[192.168.1.11]) by cl3.bcs-hosting.net with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1kfSMi-005J6c-P1; Wed, 18 Nov 2020 18:39:38 +0000
To: "Scharf, Michael" , Yoshifumi Nishida 
Cc: Michael Tuexen , "tcpm-chairs@ietf.org" , Mirja Kuehlewind , "Scheffenegger, Richard" , tcpm IETF list 
References: <20ac3652d55d4bc19c9c37714c59c95c@hs-esslingen.de>
From: Bob Briscoe 
Message-ID: 
Date: Wed, 18 Nov 2020 18:38:52 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <20ac3652d55d4bc19c9c37714c59c95c@hs-esslingen.de>
Content-Type: multipart/alternative; boundary="------------781524B6DA23B9AD26587F5D"
Content-Language: en-GB
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - cl3.bcs-hosting.net
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: cl3.bcs-hosting.net: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: cl3.bcs-hosting.net: in@bobbriscoe.net
X-Source: 
X-Source-Args: 
X-Source-Dir: 
Archived-At: 
Subject: Re: [tcpm] Security issues of AccECN (was: RE: AccECN field order)
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 18 Nov 2020 18:39:44 -0000

This is a multi-part message in MIME format.
--------------781524B6DA23B9AD26587F5D
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

Michael,

On 18/11/2020 05:36, Scharf, Michael wrote:
>
> Only on remark [ms2] regarding the issue of a covert channel due to 
> „forward compatibility“…
>
>         From a security perspective, it is not clear to me whether
>         allowing arbitrary unspecified bytes in a TCP option is a good
>         idea **at all**. It will be interesting to hear the opinion
>         from SEC area on that. Personally, I am not convinced that
>         this really makes sense, but I my concerns are not strong
>         enough to formally push back. I’ll leave it to others to think
>         about whether this is a bug or a feature.
>
>
>     [BB] Well, let's first try to deal with security ourselves:
>     * Octets that are explicitly declared as part of an option cannot
>     be used for a buffer overflow attack. I don't really need to, but
>     I could add the following text to the forward compatibility wording:
>         A receiver considers octets beyond those it uses, but within
>     the specified length, as if they are padding.
>     * And such octets cannot be any different from the current ability
>     of a sender to add padding. So there's no new attack possible here.
>     * And there's no need to specify a max length for any AccECN
>     Option, which would just unnecessarily limit flexibility.
>
>     [ms] I have more thought about the covert channel that some
>     unspecified bytes in a standards track TCP option could offer… In
>     particular if the AccECN option gets widely deployed and used,
>     would some malicious users find those bytes useful? For what? For
>     instance, what in the AccECN protocol design prevents use of these
>     “padding” bytes as super-cookie? To me, it is much more important
>     to think about abuse of whatever we standardize **now**, instead
>     of engineering for some entirely unknown future. I am not a
>     security expert, but I suggest a careful analysis of any security
>     implications of the AccECN option. We seldomly specify new TCP
>     options and malicious users will certainly look for
>     “opportunities” in the spec. So far, I don’t know whether that
>     whole idea of forward compatibility is a feature or a bug.
>
>
> [BB] If that were a problem, it could already be done, because (by my 
> reading of RFC793) it is valid to set the data offset to stretch out 
> the padding to any length within the max option space limits.
>
> [ms2] I don’t understand. RFC 793 mandates that padding is composed of 
> zeros, i.e., a firewall or IDS could detect a non-zero covert channel 
> in the padding (and even remove the padding). Also, there are IMHO no 
> reasonable engineering reasons for making the padding longer than 
> needed, i.e. such a covert channel could be detected as anomality. As 
> far as I understand RFC 793, a middlebox could also safely remove the 
> additional padding w/o impacting a standard-compliant TCP connection.
>
> [ms2] Allowing arbitrary bytes in a TCP option is IMHO a very 
> different story, in particular if you allow the values to have 
> non-zero values and forbid middleboxes to check or remove parts of the 
> option that are not standardized by the IETF. If I correctly 
> understand draft -13, an attacker could transmit 29 bytes payload at 
> the end of the AccECN option, in any TCP connection, for any purpose.
>
> [ms2] I think the „forward compatibility“ (e.g., Section 3.3.2 in the 
> I-D) needs a very careful security analysis. As far as I understand, 
> this whole idea of „forward compatibility“ is not needed for using the 
> **currently** standardized AccECN options. IMHO you could just remove 
> that „forward compatibility“ idea w/o impacting the rest of the AccECN 
> protocol as currently specified (including the current format of the 
> two options). If the idea of „forward compatibility“ stays in the 
> document, IMHO the security section needs a dedicated discussion 
> regarding potential misuse as covert channel. This could either be a 
> security or privacy issue, depending on who puts what additional bytes 
> into the option and claims that these bytes to be AccECNv2.
>

[BB] Fair enough.

I will happily mention in Security Considerations that this forward 
compatibility provision creates a covert channel of up to 29B,  but I 
don't think we need to panic about it. I suspect SEC area review will 
think about it for a moment, then let it through. There are all sorts of 
covert channels in TCP/IP headers, not least the packet ID field, or a 
TFO cookie. If they want it changed, we can either limit the additional 
length that we allow for forward compatibility or shut it down 
completely, depending on what we agree.

It is definitely worth noting this, and useful that you've noticed it at 
this stage, but I think even the SEC area will consider that future 
extensibility is more valuable than shutting off a covert channel.


Bob

> Michael
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/


--------------781524B6DA23B9AD26587F5D
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    Michael,

    

    
On 18/11/2020 05:36, Scharf, Michael
      wrote:

    

    

      
      
      
      

        
Only on
            remark [ms2] regarding the issue of a covert channel due to
            „forward compatibility“…

        
 

        

          

            

              

                
From
                    a security perspective, it is not clear to me
                    whether allowing arbitrary unspecified bytes in a
                    TCP option is a good idea *at all*. It will
                    be interesting to hear the opinion from SEC area on
                    that. Personally, I am not convinced that this
                    really makes sense, but I my concerns are not strong
                    enough to formally push back. I’ll leave it to
                    others to think about whether this is a bug or a
                    feature.

              

              


                [BB] Well, let's first try to deal with security
                ourselves:

                * Octets that are explicitly declared as part of an
                option cannot be used for a buffer overflow attack. I
                don't really need to, but I could add the following text
                to the forward compatibility wording:
                

                    A receiver considers octets beyond those it uses,
                but within the specified length, as if they are padding.
                

                * And such octets cannot be any different from the
                current ability of a sender to add padding. So there's
                no new attack possible here.

                * And there's no need to specify a max length for any
                AccECN Option, which would just unnecessarily limit
                flexibility.

                

                [ms] I have more thought about the
                  covert channel that some unspecified bytes in a
                  standards track TCP option could offer… In particular
                  if the AccECN option gets widely deployed and used,
                  would some malicious users find those bytes useful?
                  For what? For instance, what in the AccECN protocol
                  design prevents use of these “padding” bytes as
                  super-cookie? To me, it is much more important to
                  think about abuse of whatever we standardize *now*,
                  instead of engineering for some entirely unknown
                  future. I am not a security expert, but I suggest a
                  careful analysis of any security implications of the
                  AccECN option. We seldomly specify new TCP options and
                  malicious users will certainly look for
                  “opportunities” in the spec. So far, I don’t know
                  whether that whole idea of forward compatibility is a
                  feature or a bug.

            

          

          


              [BB] If that were a problem, it could already be done,
              because (by my reading of RFC793) it is valid to set the
              data offset to stretch out the padding to any length
              within the max option space limits.

          
 

          
[ms2] I don’t
              understand. RFC 793 mandates that padding is composed of
              zeros, i.e., a firewall or IDS could detect a non-zero
              covert channel in the padding (and even remove the
              padding). Also, there are IMHO no reasonable engineering
              reasons for making the padding longer than needed, i.e.
              such a covert channel could be detected as anomality. As
              far as I understand RFC 793, a middlebox could also safely
              remove the additional padding w/o impacting a
              standard-compliant TCP connection.

          
 

          
[ms2]
              Allowing arbitrary bytes in a TCP option is IMHO a very
              different story, in particular if you allow the values to
              have non-zero values and forbid middleboxes to check or
              remove parts of the option that are not standardized by
              the IETF. If I correctly understand draft -13, an attacker
              could transmit 29 bytes payload at the end of the AccECN
              option, in any TCP connection, for any purpose.

          
 

          
[ms2] I think
              the „forward compatibility“ (e.g., Section 3.3.2 in the
              I-D) needs a very careful security analysis. As far as I
              understand, this whole idea of „forward compatibility“ is
              not needed for using the *currently* standardized
              AccECN options. IMHO you could just remove that „forward
              compatibility“ idea w/o impacting the rest of the AccECN
              protocol as currently specified (including the current
              format of the two options). If the idea of „forward
              compatibility“ stays in the document, IMHO the security
              section needs a dedicated discussion regarding potential
              misuse as covert channel. This could either be a security
              or privacy issue, depending on who puts what additional
              bytes into the option and claims that these bytes to be
              AccECNv2.

        

      

    

    

    [BB] Fair enough. 

    

    I will happily mention in Security Considerations that this forward
    compatibility provision creates a covert channel of up to 29B,  but
    I don't think we need to panic about it. I suspect SEC area review
    will think about it for a moment, then let it through. There are all
    sorts of covert channels in TCP/IP headers, not least the packet ID
    field, or a TFO cookie. If they want it changed, we can either limit
    the additional length that we allow for forward compatibility or
    shut it down completely, depending on what we agree.

    

    It is definitely worth noting this, and useful that you've noticed
    it at this stage, but I think even the SEC area will consider that
    future extensibility is more valuable than shutting off a covert
    channel.

    

    

    Bob

    

    

      

        

          

          
 

          
Michael

        

      

    

    

    
-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

  


--------------781524B6DA23B9AD26587F5D--


From nobody Wed Nov 18 19:56:19 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2B3DC3A09BB for ; Wed, 18 Nov 2020 19:56:17 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.589
X-Spam-Level: 
X-Spam-Status: No, score=-17.589 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SEj7n0sKEzeL for ; Wed, 18 Nov 2020 19:56:14 -0800 (PST)
Received: from mail-io1-xd32.google.com (mail-io1-xd32.google.com [IPv6:2607:f8b0:4864:20::d32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7A3103A09B7 for ; Wed, 18 Nov 2020 19:56:14 -0800 (PST)
Received: by mail-io1-xd32.google.com with SMTP id o11so4475059ioo.11 for ; Wed, 18 Nov 2020 19:56:14 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=1GZeKcyPTEZpoPIYBozvaI8+vj0cffGln8Ys2i4QmMo=; b=l56CK/hMOTFHZc1ord+3GL1gmpxzG7u+uD6TWWJXCWPtU4IWHfFjGx6wpIVaJ7Kx8p gm7Kv53Yp0ve+Mc/BRvU00K/xpQ5EWskY8sdYnAQUkcGMGWMpIDNgNRt5Poq2B2o04xF w3iDZcIv+UhIam4J4gwfiD8gPw+QsdzVEykjJqO1d0CfIv60SLSPRxU8W1iyuox7MV1w 5OgwKe7D/NN7DG3T6BDLmRgUQHql8sxm0yPpEy3gdWXdSj+00O5swcA8OSzOEqyE6eSx dlzXpAOwCZpsWqV4bq90o43bavXRpZIxho9oxBwBGkkrvYS+J9Q89IAiXhbuzxxu0RAP +lSA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=1GZeKcyPTEZpoPIYBozvaI8+vj0cffGln8Ys2i4QmMo=; b=fIkDNAfpq5WacwOoSiHAwlIASOvp2AFDpWmxwNgMEYH6r6tNDRSlth5Gg0eS5pzm0S b2wMgn8gAHmBVkNAM+i1WCbDA7IzDoJlNZgTnHB7qD9rB/Afwa+eFcmQyL4b8gsGoVP8 Tgz4Nem3/GmuRgOei2fPjgxzCA7rurt2NmXssOjIjlR3+jdkBIbrnDVYSPNWIs98ckqV LFFOluzum61CeLbEKbLZSBf3rG5YpzgspBMvVNobZYix/deNYtROpPhbNhYpI0mG6r0A GtJMPV3s902ttMCY8Zd1VFEVC8xFAEq9nw9VXShtfhCF93kFb0Ciuax5PsLRvmTOlDd0 fz3g==
X-Gm-Message-State: AOAM530XtSeTvRzTO62AtB8PnPTafVXBGvWZDWU/+k2/zXsankLqDiQy 5rntKz9yIX7HfP6IMct2VmyJNgAiE2eeDnwHlKsqKGTgdt0=
X-Google-Smtp-Source: ABdhPJznwsxEQ+osQlAj07NXJUVZqQzAtl235TSn4z3JWUwjHemdgJGYiFkpGC9ju/87ssUwHl8iqTTl0iyvvWGo8iY=
X-Received: by 2002:a5d:964a:: with SMTP id d10mr14784931ios.5.1605758172486;  Wed, 18 Nov 2020 19:56:12 -0800 (PST)
MIME-Version: 1.0
From: Matt Mathis 
Date: Wed, 18 Nov 2020 19:56:00 -0800
Message-ID: 
To: tcpm IETF list 
Content-Type: multipart/alternative; boundary="00000000000077d42405b46db515"
Archived-At: 
Subject: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 03:56:17 -0000

--00000000000077d42405b46db515
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

The authors of PRR would like to update PRR to Proposed Standard status.
This entails introducing a new document as an tcpm work item.

*Please indicate (non) support and/or comment.*

For more details see the tcpm meeting materials from IETF 109
minutes:
https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00

There were about four "I support this work" remarks at the mic (not
recorded in the minutes), and about as many in the Meetecho chat.

Abridged IETF/tcpm/PRRbis slides:
--
PRR  recap (RFC6937 experimental)
PRR is a special congestion control effective only during fast recovery

   - When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.g.
   loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
   - When inflight < ssthresh, send at the same or twice the delivery_rate
   (more later)
   - Used by all congestion control modules in Linux during fast recovery
      - Can be more dominant than the actual C.C. for lossy flows that=E2=
=80=99re
      in fast recovery constantly (e.g. video streaming through policers)

--
Current Status

   -

   PRR is widely deployed
   -

      At least three major OSs: Linux, Windows, (NetFlix) BSD
      -

      Vast majority of Web traffic for years
      -

   No changes to algorithms published in RFC 6937
   -

      PRR-CRB - Conservative Reduction Bound - strict packet conversion
      during loss recovery
      -

      PRR-SSRB - Slowstart Reduction Bound - one extra segment per ACK
      during loss recovery
      -

   2015 Heuristic to dynamically select which reduction bound
   -

      Only use PRR-SSRB when making good forward progress
      -

         ACKs that advanced snd.una and report no new losses
         -

      Resolves some pathological cases with token bucket policers
      -

         CC estimates ssthresh before it can possibly measure the token rat=
e
         -

         The heuristic makes the best of a bad situation

--
Tentative path forward

   -

   Adopt as a tcpm work item
   -

   Update the text
   -

      Normative RFC 2119 language
      -

      Add MAY use the heuristic...
      -

      Trim redundant and obsolete language
      -

         RFC 6937 repeats itself and is much longer than necessary
         -

         Focus on what an implementer needs to know
         -

         Use non-normative references to RFC 6937 for prior measurement
         work, etc

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.

--00000000000077d42405b46db515
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


The authors of PRR would like=
 to update PRR to Proposed Standard status.=C2=A0 This entails introducing =
a new document as an tcpm work item.

Please indicate (non) s=
upport and/or comment.

For more details see the tcpm meetin=
g materials from IETF 109

There were about four "I support this work" remar=
ks at the mic (not recorded in the minutes), and about as many in the Meete=
cho chat.

Abridged IETF/tc=
pm/PRRbis slides:
--
<=
div>PRR =C2=A0recap (RFC6937 experimental)
PRR is a special congestion control ef=
fective only during fast recovery
  • When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.g. l=
oss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
    • When inflight < ssthresh, send at the same or t=
wice the delivery_rate (more later)
  • =
Used by all congestion control modules in Linux during fast recovery=

    • Can be more dominant than the actual C=
.C. for lossy flows that=E2=80=99re in fast recovery constantly (e.g. video=
 streaming through policers)
-=
-
Current Status
<=
div>
  • PRR is widely deployed
    • At least three major OSs: Linux, Windows, (NetFlix) BSD
      • =

    • Vast majority of Web traffic for years
      <=
/li>
  • No changes to algorithms published in RFC 6=
937
    • PRR-CRB - Conservative Reduction Bound - strict packet conv=
ersion during loss recovery=C2=A0
    • =
P=
RR-SSRB - Slowstart Reduction Bound - one extra segment per ACK during loss=
 recovery
  • 2015 Heuristic to dyn=
amically select which reduction bound
    • Only use PRR-SSRB when ma=
king good forward progress
      • ACKs that advanced snd.una and repor=
t no new losses
    • <=
p style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt">Resolves som=
e pathological cases with token bucket policers
      • CC estimates ss=
thresh before it can possibly measure the token rate
      • The heuristic makes the best of a bad situation
--
Tentative path forward=C2=A0
  • Adopt as a tcpm work item
  • Update the t=
ext
    • Normative RFC 2119 language
    • Add MAY use the =
heuristic...
    • =
RFC 6937 repeats itself and is much longer than necessary
      • Focus on what an implementer=
 needs to know
    • Use non-normative references to RFC 6937 for prior measurement work, e=
tc
<=
div>
Thanks,
--MM--
The best way to predict the future is to creat=
e it. =C2=A0- Alan Kay

We must not tolerate intolerance;
=C2=A0 =C2=A0 =C2=A0 =C2=A0howeve=
r our response must be carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong would b=
e hypocritical and risks spiraling out of control;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks b=
eing mistaken for tacit approval.


--00000000000077d42405b46db515--


From nobody Wed Nov 18 23:42:59 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5BE223A111F for ; Wed, 18 Nov 2020 23:42:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sur3fzOxAghc for ; Wed, 18 Nov 2020 23:42:55 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2FB3E3A0D42 for ; Wed, 18 Nov 2020 23:42:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1605771771; bh=+lyjkry4QfMVaJtHtnCyek/aFITTzYAZw3Fj3dZVJBQ=; h=X-UI-Sender-Class:Subject:To:References:From:Date:In-Reply-To; b=hca4tINIkYRhvqgC0zCE/uqy/cOPR8DvXi78egzesswUl5JPyEKvXwUyZIY/mcwG4 i33wPDlmlgtRmLgBKR+3cPTQPAW5wrUMB0Zo1V6ekDVZNvpg/vywJ9oXUzFF+mjDws UGOj1HFR71xJDERaexH00TrrjyPsTywKp12vaEiY=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [192.168.233.106] ([185.236.167.136]) by mail.gmx.com (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MqaxO-1jtNOd00Rc-00mXHT for ; Thu, 19 Nov 2020 08:42:51 +0100
To: tcpm@ietf.org
References: 
From: "Scheffenegger, Richard" 
Message-ID: 
Date: Thu, 19 Nov 2020 08:42:50 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:L94J8w3bLn9tISf2pc405iOs7ZCxfE3MHJ7FyfO2JL5Xh9Z/nIq Is+Foz80H20sIhLnPUsh96QN5GGt+hPHQsiiJ2nXboKCX1a6rxHIlE7xMHGsCVUS6DezBTH N6lLQqZ+aYL8j5dTOdrgjswNFvui8/7wTTQZQGbfXV4ETVetxxgFHGUdwxX09iueNMXG9X+ jPgRYktQbK/MoFFrqAfIw==
X-UI-Out-Filterresults: notjunk:1;V03:K0:87K4mSLV4EQ=:wXuBjRnXNps53b85/DhV8B 4sR3aEtd5pwEkLMH5JsJ156wjo6kYf0+Pbzvd9esDDtvqZ4eNdoz/3KzeW1O5pRkbx67rDVoF 33ozt8WUPe09x8yvIx13JNOaPp/7/132WUQbj71p256RX21v2O6EDqs3g7hEsIxB9hl645HJR Ci1NopKPIHL8xuLH8LHPDWWTCwvDFegX3oqYYPdxQkFEB+MmwY5t6DgE63hI33m4RDiDTrv33 MoO8Z1CbQ0vifI9Cf8oxwkDwXoIH031M+Mx1F8fA6+MlZXgq0XkRB7nnsdU3xbZFGrh/fUZWN nBnDd2AWRW/OIJNqFwiMFelwgtjzIOnf7U7AyXTjGOjz7zWLgOCoVRIPRRIQUH7tfYfBN+sx7 p9MSaHLxH0sKz/zFQZwPYHsY3VLKeXPkbMp9xGoWk3uvTHJ8IKzPl50ATbA0BMMtH9KDQh9dg hB87/9LBTYWyP4AzSTgMP09OOH3pROoLnG4ssY7PmYMuQrhdO4vX0a9a/ZCmYWHXML1A1ryZ7 sOZUyoZN8vvqBq4Sf3j7CWJxIglnkznYXXSromWKrcPxU/k8ziyFkAoJCb5/pjyM5ODq2soru 4eAe087F/on/3JJF1Hz+G+etU1kdGy9XFgSYGsWnt6uFvddt0b3NuaaZPlw1KPlombybpz1yy H/o6o2kHiGkR169pJKbz2fR3bTRNiogcEsaJOEKwlkoVIuAlxRfSLbnnlTkK0+Zkdqm4g7IZA CZUaaX9UjS6LANwYX76tbpTF2SdCgxl3SuY/69JTVb4tKgmFeCXT28Ve0XwI71L5hh/nXmlEX +1uavSNVOjkKmxq/P8ODRDjreB+a/sgYbZDBUnPvxXCDfIJBOi9Zdu7RXUoDG2Oj6z8tEE8h0 pH216997cShkGIE5TTqw==
Archived-At: 
Subject: Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 07:42:57 -0000

As the heuristic is the one major improvement over RFC6937, that is the
one area I am especially interested in. Currently working to have PRR in
FreeBSD, it would be the best time window to add this heuristig instead
of switching between the two variants in a static fashion.

I support this and will certainly be reviewing the documents and provide
feedback!

Richard


Am 19.11.2020 um 04:56 schrieb Matt Mathis:
> The authors of PRR would like to update PRR to Proposed Standard
> status.=C2=A0 This entails introducing a new document as an tcpm work it=
em.
>
> *Please indicate (non) support and/or comment.*
>
> For more details see the tcpm meeting materials from IETF 109
> minutes:
> https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
> slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
>
> There were about four "I support this work" remarks at the mic (not
> recorded in the minutes), and about as many in the Meetecho chat.
>
> Abridged IETF/tcpm/PRRbis slides:
> --
> PRR =C2=A0recap (RFC6937 experimental)
> PRR is a special congestion control effective only during fast recovery
>
>   * When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.g=
.
>     loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
>   * When inflight < ssthresh, send at the same or twice the
>     delivery_rate (more later)
>   * Used by all congestion control modules in Linux during fast recovery
>       o Can be more dominant than the actual C.C. for lossy flows
>         that=E2=80=99re in fast recovery constantly (e.g. video streamin=
g
>         through policers)
>
> --
> Current Status
>
>   *
>
>     PRR is widely deployed
>
>       o
>
>         At least three major OSs: Linux, Windows, (NetFlix) BSD
>
>       o
>
>         Vast majority of Web traffic for years
>
>   *
>
>     No changes to algorithms published in RFC 6937
>
>       o
>
>         PRR-CRB - Conservative Reduction Bound - strict packet
>         conversion during loss recovery
>
>       o
>
>         PRR-SSRB - Slowstart Reduction Bound - one extra segment per ACK
>         during loss recovery
>
>   *
>
>     2015 Heuristic to dynamically select which reduction bound
>
>       o
>
>         Only use PRR-SSRB when making good forward progress
>
>           +
>
>             ACKs that advanced snd.una and report no new losses
>
>       o
>
>         Resolves some pathological cases with token bucket policers
>
>           +
>
>             CC estimates ssthresh before it can possibly measure the
>             token rate
>
>           +
>
>             The heuristic makes the best of a bad situation
>
> --
> Tentative path forward
>
>   *
>
>     Adopt as a tcpm work item
>
>   *
>
>     Update the text
>
>       o
>
>         Normative RFC 2119 language
>
>       o
>
>         Add MAY use the heuristic...
>
>       o
>
>         Trim redundant and obsolete language
>
>           +
>
>             RFC 6937 repeats itself and is much longer than necessary
>
>           +
>
>             Focus on what an implementer needs to know
>
>           +
>
>             Use non-normative references to RFC 6937 for prior
>             measurement work, etc
>
> Thanks,
> --MM--
> The best way to predict the future is to create it. =C2=A0- Alan Kay
>
> We must not tolerate intolerance;
>  =C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must be carefully measu=
red:
>  =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong would be hypocriti=
cal and risks spiraling out of
> control;
>  =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks being mistaken=
 for tacit approval.
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>


From nobody Thu Nov 19 02:48:22 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 590133A005D; Thu, 19 Nov 2020 02:48:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level: 
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YJMLFOME9dyz; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A14F03A0045; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
Received: by mail-oi1-x233.google.com with SMTP id m143so5801877oig.7; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:from:date:message-id:subject:to; bh=pGNWkFYae1qTiPyvVV66IAP5WPPEXhwnMjgPELWIyxU=; b=jI4F4oWJkorYmqrT8uPwp3PKwQ8ZyTl2TY3B8QPfFYFgBweiFvnB/jHmBZ5V375gbD bP7ALwPsLA7fAmcYmwWjDSTkMs3ajiWxa/ZjPRg4w5q5Q4dD9S+eFzwyWEMDxkJWcpS+ cc16b2E2D3px+zuZMBJlpsAzpZ+IcTaOmqgFdQBE7eo7r3m3y/4A7L1BCBJ1bNYrkf2N bprDjGsrMezBOodTq0J1Buuya+2/V0SKEYt/lgik4Rz/L1iSyQo7YQdfvCPq2qu5dA3R lymely3XuqUsa9fdIlz3WE7bhu4FUNoSX0hYEJgTQfd5lOUoTsWmSVysiexET78543ns XrnQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pGNWkFYae1qTiPyvVV66IAP5WPPEXhwnMjgPELWIyxU=; b=Wa7FMgQQ306A0B9NEpfILPpBYGHmEjosfNjQZ8w5ueh7vpJvG3+myou6671k4Kn5Mj Dm/hMF5vjsP0BFq5k6zhA6Rq1NKNFYm5TWMGrtuNDwnmqLQUqqhBnx9o71oZ0kY/QsTt 0KVYY4b+RKuO5KepX2vTDRVEj2vIcOJxfIjPI4IMyqfOTKCzBROi9Uw/ay5A4wS+u3Ae VgLrhgmyeloggsQxRZdlGgMp3tvIziYO3q+FgUELuef2jldLQPwpxxzfcpHJthA8tVN5 +zIMEzGHek9exCp6fdEwaIw/mwTI2FbEOdW1TIsdiBUs7EguXxMvAYgWqvStEegxuYmK KPOQ==
X-Gm-Message-State: AOAM532edMDJdnaJ+ZNN1N/YoTNf31PdDUpTI7zQfqctb7aU7UftykN1 n2vNiTZfdc0C9Eu6BhHMeOIMP1LDSiG/B6lt1YP5D86nYdw=
X-Google-Smtp-Source: ABdhPJyWdlgBLChqsWqrunfmFa6JmjU6LpUGzV6Nlm9W8mC2QJppj6qr1QHyAwufGHjbdf5p/Ma92g2SRY5OinzKjzQ=
X-Received: by 2002:aca:c4c3:: with SMTP id u186mr2476326oif.4.1605782897451;  Thu, 19 Nov 2020 02:48:17 -0800 (PST)
MIME-Version: 1.0
From: Alexander Azimov 
Date: Thu, 19 Nov 2020 13:48:06 +0300
Message-ID: 
To: tcpm , v6ops@ietf.org
Content-Type: multipart/alternative; boundary="00000000000030651505b473773b"
Archived-At: 
Subject: [tcpm] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 10:48:20 -0000

--00000000000030651505b473773b
Content-Type: text/plain; charset="UTF-8"

Dear colleagues,

I have added in the cc both v6op and tcpm for a reason and let me explain
why.

It's clear that we are moving forward with load balancing that uses flow
label (FL). And the pressure will increase with SRv6 adoption. But at the
moment wide adoption of FL-based load-balancing may create significant
issues for TCP Anycast services.

RFC6437 suggests putting hash from 5-tuple into FL value. And as far as I
know, there is no document that updates this behavior. This description is
perfectly fine, but what is implemented in the Linux kernel is different:
FL is carrying hash from 5-tuple with an additional seed, and this seed is
randomly changed after each RTO/SYN_RTO event. Here are related patches:

   -
   https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.mtv.corp.google.com/
   -
   https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@herbertland.com/
   - https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/


This is a great thing by the way because in the data center environment with
multiple equal paths it gives a way to have pseudo-multipath TCP which
jumps between paths in case of an outage. There might be interest to write
down an informational document for this.

But the problem happens when TCP flows start jumping between anycast
points. If anycast service provides connection tracking (L7/TCP proxy,
stateful L4 balancer) the 'jump' may redirect traffic to another point, and
with the adoption of FL load balancing by the transit providers - to
another location, where no TCP state is available. In other words, an
established TCP session breaks after RTO event. And it's not just a theory
- we already faced this kind of issue in our network, and it is really hard
to debug.

I wonder what you think is a proper solution:

   - Making FL related RTO change as knob instead of default behavior;
   - Adding negotiation behavior in TCP;
   - Something else?

I'm looking forward to your advice. If there is a document that describes
the above problem - please give me a reference.

-- 
Best regards,
Alexander Azimov

--00000000000030651505b473773b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Dear colleagues,

I have added in t=
he cc both v6op and tcpm for a reason and let me explain why.
It's clear that we are moving forward with load balancing t=
hat uses flow label (FL). And the pressure will increase with SRv6 adoption=
. But at the moment wide adoption of FL-based load-balancing may create sig=
nificant issues for TCP Anycast services.

RFC6437 =
suggests putting hash from=C2=A05-tuple into FL value. And as far as I know, there is no document=
 that updates this behavior. This description is perfectly fine, but what i=
s implemented in the Linux kernel is different: FL is carrying=C2=A0hash=C2=
=A0from 5-tuple with an additional seed, and this seed is randomly changed=
=C2=A0after each RTO/SYN_RTO event. Here are related patches:
<=
div>

This is a great thing by the way because in the data ce=
nter=C2=A0environment=C2=A0with multiple equal paths=C2=A0it gives a way to have pseudo-mu=
ltipath=C2=A0TCP which jumps between paths in case of an outage. There migh=
t be interest to write down an informational document for this.

But th=
e problem happens when TCP flows start jumping between anycast points. If a=
nycast service provides connection tracking (L7/TCP proxy, stateful L4 bala=
ncer) the 'jump' may redirect traffic to another point, and with th=
e adoption of FL load balancing by the transit providers - to another locat=
ion, where no TCP state is available. In other words, an established TCP se=
ssion breaks after RTO event. And it's not just a theory - we already f=
aced this kind of issue in our network, and it is really hard to debug.

I wonder what you think is a proper solution:
  • Making FL related RTO change as knob instead of default behavior;
  • A=
dding negotiation=C2=A0behavior in TCP;
  • Something else?
I'm l=
ooking forward to your advice. If there is a document that describes the ab=
ove problem - please give me a reference.

=
-- 
Best regards,
Alexander Azimov


--00000000000030651505b473773b--


From nobody Thu Nov 19 06:21:08 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C02603A107E for ; Thu, 19 Nov 2020 06:21:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.599
X-Spam-Level: 
X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VL6F0yeUgymk for ; Thu, 19 Nov 2020 06:21:02 -0800 (PST)
Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 216EB3A1075 for ; Thu, 19 Nov 2020 06:21:02 -0800 (PST)
Received: by mail-yb1-xb2e.google.com with SMTP id g15so5334599ybq.6 for ; Thu, 19 Nov 2020 06:21:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=5Mtgg0ZeCH0GEQ6sJFLr3gl3PrthcMRPfOJiQQu+760=; b=bPJZbbos6OlR/CaBOH/A1BSvCHJS63gj67m/516PJtBJ+umpDgtFxe/4EfVmI/ewUD +kgzyPboyiCfVrUx4mf48SMhCx5RdasjSATJgWmzPmgjaMLTmLF7v2b/VHqTne/sU/7o ofUcm804fOFBi1AJGsElzRuA9X1JMewvHBolJfz58vcrqkCHnNFgUqhlD1Xz8y1lXop4 LvrTmWY61GsLm8pkM3atXqn/tKyQYBaE7/6J84KfyBV7DMlzygvFWd9poUski9i+242/ +ZXy423eZqyJces2tSghzxIsyRUdSIWz3gslQw0dBUQ1kAP52Muta4sG7RQ1UqtueFZZ LmtA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=5Mtgg0ZeCH0GEQ6sJFLr3gl3PrthcMRPfOJiQQu+760=; b=NrV8ZJ+JyAmPB2Tm2Ls7nBwqIj676Hs3p6eShU9yLJPAIbD6eBceYSCHDVPBEiQrhp 0iFu30Nihe5ldS7mhOgi7RFsuir9AZYpnaU/N9cauGZrqtDQqleROK1StEiI+gyfEl80 oe1K7jqha/2rjHXFFq9N45HneaWVNp/5jDsAF/G+cgYV91e0gtE1IsD4Dy9Q/9FkSK7M iroImBDrZ8ooq0qS11z9DxCJoZZ26Y9gB/TM126oKfHLdWFKsxTDJpLTw9w8FL321Ml5 8+YGo0i/MJ8y1gf5Qr+Z7DP76hHVpLpZ428aHSQGduxl5nEwWBvPtkNu3aGD7z8BNrtL z1jg==
X-Gm-Message-State: AOAM532DSlhgYiESfF0jwQhcTb/dVQFIiYKoKoSKPw//QotkzZMRAMW6 tq1mt66/8bkBX8PGdLiO+KEzksnu8/BtUm0AC6GLig==
X-Google-Smtp-Source: ABdhPJxvhTL8eAn4QX+LAkDPJ61BR1QefUYnES+qhsiyghj3POZjOuktUYrKnPkK8JaKO9BC9NOyiF/RCRP/0+Baqc0=
X-Received: by 2002:a25:3448:: with SMTP id b69mr14095414yba.77.1605795661014;  Thu, 19 Nov 2020 06:21:01 -0800 (PST)
MIME-Version: 1.0
References:  
In-Reply-To: 
From: Ian Swett 
Date: Thu, 19 Nov 2020 09:20:49 -0500
Message-ID: 
To: "Scheffenegger, Richard" 
Cc: "tcpm@ietf.org Extensions" 
Content-Type: multipart/alternative; boundary="000000000000f5961405b4766f77"
Archived-At: 
Subject: Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 14:21:07 -0000

--000000000000f5961405b4766f77
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I support this work.

One question for authors and the group.  This specification could be
applied to QUIC, but my memory is that there are some small changes to
adapt it correctly.  Would noting those changes be in scope for this
document, a separate document in QUIC, or an exercise left to the reader?
I'd be happy to contribute any QUIC specific text if that's helpful.

Thanks, Ian

On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richard 
wrote:

> As the heuristic is the one major improvement over RFC6937, that is the
> one area I am especially interested in. Currently working to have PRR in
> FreeBSD, it would be the best time window to add this heuristig instead
> of switching between the two variants in a static fashion.
>
> I support this and will certainly be reviewing the documents and provide
> feedback!
>
> Richard
>
>
> Am 19.11.2020 um 04:56 schrieb Matt Mathis:
> > The authors of PRR would like to update PRR to Proposed Standard
> > status.  This entails introducing a new document as an tcpm work item.
> >
> > *Please indicate (non) support and/or comment.*
> >
> > For more details see the tcpm meeting materials from IETF 109
> > minutes:
> > https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
> > slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
> >
> > There were about four "I support this work" remarks at the mic (not
> > recorded in the minutes), and about as many in the Meetecho chat.
> >
> > Abridged IETF/tcpm/PRRbis slides:
> > --
> > PRR  recap (RFC6937 experimental)
> > PRR is a special congestion control effective only during fast recovery
> >
> >   * When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.=
g.
> >     loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
> >   * When inflight < ssthresh, send at the same or twice the
> >     delivery_rate (more later)
> >   * Used by all congestion control modules in Linux during fast recover=
y
> >       o Can be more dominant than the actual C.C. for lossy flows
> >         that=E2=80=99re in fast recovery constantly (e.g. video streami=
ng
> >         through policers)
> >
> > --
> > Current Status
> >
> >   *
> >
> >     PRR is widely deployed
> >
> >       o
> >
> >         At least three major OSs: Linux, Windows, (NetFlix) BSD
> >
> >       o
> >
> >         Vast majority of Web traffic for years
> >
> >   *
> >
> >     No changes to algorithms published in RFC 6937
> >
> >       o
> >
> >         PRR-CRB - Conservative Reduction Bound - strict packet
> >         conversion during loss recovery
> >
> >       o
> >
> >         PRR-SSRB - Slowstart Reduction Bound - one extra segment per AC=
K
> >         during loss recovery
> >
> >   *
> >
> >     2015 Heuristic to dynamically select which reduction bound
> >
> >       o
> >
> >         Only use PRR-SSRB when making good forward progress
> >
> >           +
> >
> >             ACKs that advanced snd.una and report no new losses
> >
> >       o
> >
> >         Resolves some pathological cases with token bucket policers
> >
> >           +
> >
> >             CC estimates ssthresh before it can possibly measure the
> >             token rate
> >
> >           +
> >
> >             The heuristic makes the best of a bad situation
> >
> > --
> > Tentative path forward
> >
> >   *
> >
> >     Adopt as a tcpm work item
> >
> >   *
> >
> >     Update the text
> >
> >       o
> >
> >         Normative RFC 2119 language
> >
> >       o
> >
> >         Add MAY use the heuristic...
> >
> >       o
> >
> >         Trim redundant and obsolete language
> >
> >           +
> >
> >             RFC 6937 repeats itself and is much longer than necessary
> >
> >           +
> >
> >             Focus on what an implementer needs to know
> >
> >           +
> >
> >             Use non-normative references to RFC 6937 for prior
> >             measurement work, etc
> >
> > Thanks,
> > --MM--
> > The best way to predict the future is to create it.  - Alan Kay
> >
> > We must not tolerate intolerance;
> >         however our response must be carefully measured:
> >              too strong would be hypocritical and risks spiraling out o=
f
> > control;
> >              too weak risks being mistaken for tacit approval.
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpm
> >
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--000000000000f5961405b4766f77
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


I support this work.

One question for a=
uthors and the group.=C2=A0 This specification=C2=A0could be applied to QUI=
C, but my memory is that there are some small changes to adapt it correctly=
.=C2=A0 Would noting those changes be in scope for this document, a separat=
e document in QUIC, or an exercise left to the reader?=C2=A0 I'd be hap=
py to contribute any QUIC specific text if that's helpful.
Thanks, Ian

On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger=
, Richard <rs.ietf@gmx.at> wrot=
e:
As the heuris=
tic is the one major improvement over RFC6937, that is the

one area I am especially interested in. Currently working to have PRR in
FreeBSD, it would be the best time window to add this heuristig instead

of switching between the two variants in a static fashion.



I support this and will certainly be reviewing the documents and provide
feedback!



Richard





Am 19.11.2020 um 04:56 schrieb Matt Mathis:

> The authors of PRR would like to update PRR to Proposed Standard

> status.=C2=A0 This entails introducing a new document as an tcpm work =
item.

>

> *Please indicate (non) support and/or comment.*

>

> For more details see the tcpm meeting materials from IETF 109

> minutes:

> https://datatracker.ietf.=
org/meeting/109/materials/minutes-109-tcpm-00

> slides: https://tools.ietf.org/html/=
draft-mathis-tcpm-rfc6937bis-00

>

> There were about four "I support this work" remarks at the m=
ic (not

> recorded in the minutes), and about as many in the Meetecho chat.

>

> Abridged IETF/tcpm/PRRbis slides:

> --

> PRR =C2=A0recap (RFC6937 experimental)

> PRR is a special congestion control effective only during fast recover=
y

>

>=C2=A0 =C2=A0* When inflight >=3D ssthresh, send at loss_beta*rate_b=
efore_loss (e.g.

>=C2=A0 =C2=A0 =C2=A0loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 =
for Cubic)

>=C2=A0 =C2=A0* When inflight < ssthresh, send at the same or twice t=
he

>=C2=A0 =C2=A0 =C2=A0delivery_rate (more later)

>=C2=A0 =C2=A0* Used by all congestion control modules in Linux during f=
ast recovery

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o Can be more dominant than the actual C.C. =
for lossy flows

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0that=E2=80=99re in fast recovery cons=
tantly (e.g. video streaming

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0through policers)

>

> --

> Current Status

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0PRR is widely deployed

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0At least three major OSs: Linux, Wind=
ows, (NetFlix) BSD

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Vast majority of Web traffic for year=
s

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0No changes to algorithms published in RFC 6937

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-CRB - Conservative Reduction Boun=
d - strict packet

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0conversion during loss recovery

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-SSRB - Slowstart Reduction Bound =
- one extra segment per ACK

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0during loss recovery

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A02015 Heuristic to dynamically select which reductio=
n bound

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Only use PRR-SSRB when making good fo=
rward progress

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ACKs that advanced snd.=
una and report no new losses

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Resolves some pathological cases with=
 token bucket policers

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CC estimates ssthresh b=
efore it can possibly measure the

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0token rate

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The heuristic makes the=
 best of a bad situation

>

> --

> Tentative path forward

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Adopt as a tcpm work item

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Update the text

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Normative RFC 2119 language

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Add MAY use the heuristic...

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Trim redundant and obsolete language<=
br>
>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RFC 6937 repeats itself=
 and is much longer than necessary

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Focus on what an implem=
enter needs to know

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Use non-normative refer=
ences to RFC 6937 for prior

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0measurement work, etc
>

> Thanks,

> --MM--

> The best way to predict the future is to create it. =C2=A0- Alan Kay
>

> We must not tolerate intolerance;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must be carefull=
y measured:

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong would be hy=
pocritical and risks spiraling out of

> control;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks being m=
istaken for tacit approval.

>

> _______________________________________________

> tcpm mailing list

> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

>



_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--000000000000f5961405b4766f77--


From nobody Thu Nov 19 06:35:32 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 75A243A0317 for ; Thu, 19 Nov 2020 06:35:30 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.096
X-Spam-Level: 
X-Spam-Status: No, score=-2.096 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hjh_NZAbTBHv for ; Thu, 19 Nov 2020 06:35:28 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A318C3A03F1 for ; Thu, 19 Nov 2020 06:35:27 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 975C325A23; Thu, 19 Nov 2020 15:35:25 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1605796525; bh=mQJDjsXqsNZqQ1M1BXL87yWjM09SM0ajKPzQNZxUMzw=; h=From:To:CC:Subject:Date:References:In-Reply-To:From; b=quTvUtflJmieA1M3oN+C1Wy35v5iDY71aphlvLucSK5HqcTq9B8u4jZFyk1GFerR4 YrVwk2iNcIqEG+PHqToDq7LunfwYibg0HQrBAhcktibRsMF6iyyDOfQ+moBqw53iJa 4HWLiO2b4ra41DO6Gnj37udHbiOQsAdB9ZVTyWps=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id owwFwl312lHh; Thu, 19 Nov 2020 15:35:23 +0100 (CET)
Received: from rznt8201.rznt.rzdir.fht-esslingen.de (rznt8201.hs-esslingen.de [134.108.48.164]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS; Thu, 19 Nov 2020 15:35:23 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8201.rznt.rzdir.fht-esslingen.de (134.108.48.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 19 Nov 2020 15:35:06 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Thu, 19 Nov 2020 15:35:06 +0100
From: "Scharf, Michael" 
To: Ian Swett , "Scheffenegger, Richard" 
CC: "tcpm@ietf.org Extensions" 
Thread-Topic: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
Thread-Index: AQHWvif4PUvJugweikmfHJ92YBnS6anPAhUAgABvMoCAABIcAA==
Date: Thu, 19 Nov 2020 14:35:06 +0000
Message-ID: 
References:   
In-Reply-To: 
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: multipart/alternative; boundary="_000_d91010d73f23429db4e873b4dc210107hsesslingende_"
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 14:35:31 -0000

--_000_d91010d73f23429db4e873b4dc210107hsesslingende_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

VGhlIFRDUE0gY2hhcnRlciB3b3VsZCBhbGxvdyBUQ1BNIHRvIHNwZWNpZnkgYSBUQ1AgbWVjaGFu
aXNtIGluY2x1ZGluZyBtaW5vciBhZGFwdGF0aW9ucyBuZWVkZWQgZm9yIFFVSUMuIFRoZSBUQ1BN
IGNoYXJ0ZXIgZXhwbGljaXRseSBzdGF0ZXM6DQoNCuKAnFRDUCdzIGNvbmdlc3Rpb24gY29udHJv
bCBhbGdvcml0aG1zIGFyZSB0aGUgbW9kZWwgZm9sbG93ZWQgYnkNCm90aGVyIElFVEYgdHJhbnNw
b3J0cyAoZS5nLiwgU0NUUCBvciBEQ0NQKSwgd2hpY2ggYXJlIHN0YW5kYXJkaXplZCBpbg0Kb3Ro
ZXIgd29ya2luZyBncm91cHMsIHN1Y2ggYXMgdGhlIFRyYW5zcG9ydCBBcmVhIFdHICh0c3Z3Zyku
IEluIHRoZQ0KcGFzdCwgdGhlIElFVEYgaGFzIHdvcmtlZCBvbiBzZXZlcmFsIGRvY3VtZW50cyBh
Ym91dCBhbGdvcml0aG1zIHRoYXQNCmFyZSBzcGVjaWZpZWQgZm9yIG11bHRpcGxlIHByb3RvY29s
cyAoZS5nLiwgVENQIGFuZCBTQ1RQKSBpbiB0aGUNCnNhbWUgZG9jdW1lbnQuIFdoaWNoIFdHIHNo
ZXBoZXJkcyBzdWNoIGRvY3VtZW50cyB3aWxsIGJlIGRldGVybWluZWQNCm9uIGEgY2FzZS1ieS1j
YXNlIGJhc2lzLiBJbiBhbnkgY2FzZSwgdGhlIFRDUE0gV0cgd2lsbCByZW1haW4gaW4NCmNsb3Nl
IGNvbnRhY3Qgd2l0aCBvdGhlciByZWxldmFudCBXR3Mgd29ya2luZyBvbiB0aGVzZSBwcm90b2Nv
bHMgdG8NCmVuc3VyZSBvcGVubmVzcyBhbmQgc3RyaW5nZW50IHJldmlldyBmcm9tIGFsbCBhbmds
ZXMu4oCdDQoNCihUaGlzIHdhcyB3cml0dGVuIGJlZm9yZSBRVUlDIHN0YW5kYXJkaXphdGlvbiBz
dGFydGVkLikNCg0KSGF2aW5nIG9uZSBkb2N1bWVudCBmb3IgVENQIGFuZCBRVUlDIHdvdWxkIHRo
dXMgYmUgYWxsb3dlZCBieSB0aGUgVENQTSBjaGFydGVyLiBJbiB0aGF0IGNhc2UsIHRoZSBkb2N1
bWVudCB3b3VsZCBwcm9iYWJseSBiZSBsYXN0IGNhbGxlZCBhbHNvIGluIHRoZSBRVUlDIHdvcmtp
bmcgZ3JvdXAg4oCTIGF0IGxlYXN0IHRoaXMgaXMgd2hhdCB3ZSBoYXZlIGRvbmUgaW4gc2ltaWxh
ciBjYXNlcyB3aXRoIFRTVldHIGluIHRoZSBwYXN0Lg0KDQpJIHRoaW5rIHRoZSBmb3JtYWwgZGV0
YWlscyBjb3VsZCBiZSBmaWd1cmVkIG91dCBpZiB0aGVyZSBpcyBjb25zZW5zdXMgdG8gaGF2ZSBv
bmUgZG9jdW1lbnQgb25seS4gSXQgaXMgdXAgdG8gdGhlIFRDUE0gYW5kIFFVSUMgY29tbXVuaXR5
IHRvIGRlY2lkZeKApg0KDQpNaWNoYWVsDQoNCg0KRnJvbTogdGNwbSA8dGNwbS1ib3VuY2VzQGll
dGYub3JnPiBPbiBCZWhhbGYgT2YgSWFuIFN3ZXR0DQpTZW50OiBUaHVyc2RheSwgTm92ZW1iZXIg
MTksIDIwMjAgMzoyMSBQTQ0KVG86IFNjaGVmZmVuZWdnZXIsIFJpY2hhcmQgPHJzLmlldGZAZ214
LmF0Pg0KQ2M6IHRjcG1AaWV0Zi5vcmcgRXh0ZW5zaW9ucyA8dGNwbUBpZXRmLm9yZz4NClN1Ympl
Y3Q6IFJlOiBbdGNwbV0gSW50cm9kdWNlIFJGQyA2OTM3IGJpcyAoUHJvcG9ydGlvbmFsIFJhdGUg
UmVkdWN0aW9uKSBhcyBhIHRjcG0gd29yayBpdGVtPyBJTlBVVCBORUVERUQNCg0KSSBzdXBwb3J0
IHRoaXMgd29yay4NCg0KT25lIHF1ZXN0aW9uIGZvciBhdXRob3JzIGFuZCB0aGUgZ3JvdXAuICBU
aGlzIHNwZWNpZmljYXRpb24gY291bGQgYmUgYXBwbGllZCB0byBRVUlDLCBidXQgbXkgbWVtb3J5
IGlzIHRoYXQgdGhlcmUgYXJlIHNvbWUgc21hbGwgY2hhbmdlcyB0byBhZGFwdCBpdCBjb3JyZWN0
bHkuICBXb3VsZCBub3RpbmcgdGhvc2UgY2hhbmdlcyBiZSBpbiBzY29wZSBmb3IgdGhpcyBkb2N1
bWVudCwgYSBzZXBhcmF0ZSBkb2N1bWVudCBpbiBRVUlDLCBvciBhbiBleGVyY2lzZSBsZWZ0IHRv
IHRoZSByZWFkZXI/ICBJJ2QgYmUgaGFwcHkgdG8gY29udHJpYnV0ZSBhbnkgUVVJQyBzcGVjaWZp
YyB0ZXh0IGlmIHRoYXQncyBoZWxwZnVsLg0KDQpUaGFua3MsIElhbg0KDQpPbiBUaHUsIE5vdiAx
OSwgMjAyMCBhdCAyOjQzIEFNIFNjaGVmZmVuZWdnZXIsIFJpY2hhcmQgPHJzLmlldGZAZ214LmF0
PG1haWx0bzpycy5pZXRmQGdteC5hdD4+IHdyb3RlOg0KQXMgdGhlIGhldXJpc3RpYyBpcyB0aGUg
b25lIG1ham9yIGltcHJvdmVtZW50IG92ZXIgUkZDNjkzNywgdGhhdCBpcyB0aGUNCm9uZSBhcmVh
IEkgYW0gZXNwZWNpYWxseSBpbnRlcmVzdGVkIGluLiBDdXJyZW50bHkgd29ya2luZyB0byBoYXZl
IFBSUiBpbg0KRnJlZUJTRCwgaXQgd291bGQgYmUgdGhlIGJlc3QgdGltZSB3aW5kb3cgdG8gYWRk
IHRoaXMgaGV1cmlzdGlnIGluc3RlYWQNCm9mIHN3aXRjaGluZyBiZXR3ZWVuIHRoZSB0d28gdmFy
aWFudHMgaW4gYSBzdGF0aWMgZmFzaGlvbi4NCg0KSSBzdXBwb3J0IHRoaXMgYW5kIHdpbGwgY2Vy
dGFpbmx5IGJlIHJldmlld2luZyB0aGUgZG9jdW1lbnRzIGFuZCBwcm92aWRlDQpmZWVkYmFjayEN
Cg0KUmljaGFyZA0KDQoNCkFtIDE5LjExLjIwMjAgdW0gMDQ6NTYgc2NocmllYiBNYXR0IE1hdGhp
czoNCj4gVGhlIGF1dGhvcnMgb2YgUFJSIHdvdWxkIGxpa2UgdG8gdXBkYXRlIFBSUiB0byBQcm9w
b3NlZCBTdGFuZGFyZA0KPiBzdGF0dXMuICBUaGlzIGVudGFpbHMgaW50cm9kdWNpbmcgYSBuZXcg
ZG9jdW1lbnQgYXMgYW4gdGNwbSB3b3JrIGl0ZW0uDQo+DQo+ICpQbGVhc2UgaW5kaWNhdGUgKG5v
bikgc3VwcG9ydCBhbmQvb3IgY29tbWVudC4qDQo+DQo+IEZvciBtb3JlIGRldGFpbHMgc2VlIHRo
ZSB0Y3BtIG1lZXRpbmcgbWF0ZXJpYWxzIGZyb20gSUVURiAxMDkNCj4gbWludXRlczoNCj4gaHR0
cHM6Ly9kYXRhdHJhY2tlci5pZXRmLm9yZy9tZWV0aW5nLzEwOS9tYXRlcmlhbHMvbWludXRlcy0x
MDktdGNwbS0wMA0KPiBzbGlkZXM6IGh0dHBzOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFmdC1t
YXRoaXMtdGNwbS1yZmM2OTM3YmlzLTAwDQo+DQo+IFRoZXJlIHdlcmUgYWJvdXQgZm91ciAiSSBz
dXBwb3J0IHRoaXMgd29yayIgcmVtYXJrcyBhdCB0aGUgbWljIChub3QNCj4gcmVjb3JkZWQgaW4g
dGhlIG1pbnV0ZXMpLCBhbmQgYWJvdXQgYXMgbWFueSBpbiB0aGUgTWVldGVjaG8gY2hhdC4NCj4N
Cj4gQWJyaWRnZWQgSUVURi90Y3BtL1BSUmJpcyBzbGlkZXM6DQo+IC0tDQo+IFBSUiAgcmVjYXAg
KFJGQzY5MzcgZXhwZXJpbWVudGFsKQ0KPiBQUlIgaXMgYSBzcGVjaWFsIGNvbmdlc3Rpb24gY29u
dHJvbCBlZmZlY3RpdmUgb25seSBkdXJpbmcgZmFzdCByZWNvdmVyeQ0KPg0KPiAgICogV2hlbiBp
bmZsaWdodCA+PSBzc3RocmVzaCwgc2VuZCBhdCBsb3NzX2JldGEqcmF0ZV9iZWZvcmVfbG9zcyAo
ZS5nLg0KPiAgICAgbG9zc19iZXRhID0gMC41IGZvciBSZW5vIChha2EgcmF0ZS1oYWx2aW5nKSwg
MC43IGZvciBDdWJpYykNCj4gICAqIFdoZW4gaW5mbGlnaHQgPCBzc3RocmVzaCwgc2VuZCBhdCB0
aGUgc2FtZSBvciB0d2ljZSB0aGUNCj4gICAgIGRlbGl2ZXJ5X3JhdGUgKG1vcmUgbGF0ZXIpDQo+
ICAgKiBVc2VkIGJ5IGFsbCBjb25nZXN0aW9uIGNvbnRyb2wgbW9kdWxlcyBpbiBMaW51eCBkdXJp
bmcgZmFzdCByZWNvdmVyeQ0KPiAgICAgICBvIENhbiBiZSBtb3JlIGRvbWluYW50IHRoYW4gdGhl
IGFjdHVhbCBDLkMuIGZvciBsb3NzeSBmbG93cw0KPiAgICAgICAgIHRoYXTigJlyZSBpbiBmYXN0
IHJlY292ZXJ5IGNvbnN0YW50bHkgKGUuZy4gdmlkZW8gc3RyZWFtaW5nDQo+ICAgICAgICAgdGhy
b3VnaCBwb2xpY2VycykNCj4NCj4gLS0NCj4gQ3VycmVudCBTdGF0dXMNCj4NCj4gICAqDQo+DQo+
ICAgICBQUlIgaXMgd2lkZWx5IGRlcGxveWVkDQo+DQo+ICAgICAgIG8NCj4NCj4gICAgICAgICBB
dCBsZWFzdCB0aHJlZSBtYWpvciBPU3M6IExpbnV4LCBXaW5kb3dzLCAoTmV0RmxpeCkgQlNEDQo+
DQo+ICAgICAgIG8NCj4NCj4gICAgICAgICBWYXN0IG1ham9yaXR5IG9mIFdlYiB0cmFmZmljIGZv
ciB5ZWFycw0KPg0KPiAgICoNCj4NCj4gICAgIE5vIGNoYW5nZXMgdG8gYWxnb3JpdGhtcyBwdWJs
aXNoZWQgaW4gUkZDIDY5MzcNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAgIFBSUi1DUkIgLSBD
b25zZXJ2YXRpdmUgUmVkdWN0aW9uIEJvdW5kIC0gc3RyaWN0IHBhY2tldA0KPiAgICAgICAgIGNv
bnZlcnNpb24gZHVyaW5nIGxvc3MgcmVjb3ZlcnkNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAg
IFBSUi1TU1JCIC0gU2xvd3N0YXJ0IFJlZHVjdGlvbiBCb3VuZCAtIG9uZSBleHRyYSBzZWdtZW50
IHBlciBBQ0sNCj4gICAgICAgICBkdXJpbmcgbG9zcyByZWNvdmVyeQ0KPg0KPiAgICoNCj4NCj4g
ICAgIDIwMTUgSGV1cmlzdGljIHRvIGR5bmFtaWNhbGx5IHNlbGVjdCB3aGljaCByZWR1Y3Rpb24g
Ym91bmQNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAgIE9ubHkgdXNlIFBSUi1TU1JCIHdoZW4g
bWFraW5nIGdvb2QgZm9yd2FyZCBwcm9ncmVzcw0KPg0KPiAgICAgICAgICAgKw0KPg0KPiAgICAg
ICAgICAgICBBQ0tzIHRoYXQgYWR2YW5jZWQgc25kLnVuYSBhbmQgcmVwb3J0IG5vIG5ldyBsb3Nz
ZXMNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAgIFJlc29sdmVzIHNvbWUgcGF0aG9sb2dpY2Fs
IGNhc2VzIHdpdGggdG9rZW4gYnVja2V0IHBvbGljZXJzDQo+DQo+ICAgICAgICAgICArDQo+DQo+
ICAgICAgICAgICAgIENDIGVzdGltYXRlcyBzc3RocmVzaCBiZWZvcmUgaXQgY2FuIHBvc3NpYmx5
IG1lYXN1cmUgdGhlDQo+ICAgICAgICAgICAgIHRva2VuIHJhdGUNCj4NCj4gICAgICAgICAgICsN
Cj4NCj4gICAgICAgICAgICAgVGhlIGhldXJpc3RpYyBtYWtlcyB0aGUgYmVzdCBvZiBhIGJhZCBz
aXR1YXRpb24NCj4NCj4gLS0NCj4gVGVudGF0aXZlIHBhdGggZm9yd2FyZA0KPg0KPiAgICoNCj4N
Cj4gICAgIEFkb3B0IGFzIGEgdGNwbSB3b3JrIGl0ZW0NCj4NCj4gICAqDQo+DQo+ICAgICBVcGRh
dGUgdGhlIHRleHQNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAgIE5vcm1hdGl2ZSBSRkMgMjEx
OSBsYW5ndWFnZQ0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgQWRkIE1BWSB1c2UgdGhlIGhl
dXJpc3RpYy4uLg0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgVHJpbSByZWR1bmRhbnQgYW5k
IG9ic29sZXRlIGxhbmd1YWdlDQo+DQo+ICAgICAgICAgICArDQo+DQo+ICAgICAgICAgICAgIFJG
QyA2OTM3IHJlcGVhdHMgaXRzZWxmIGFuZCBpcyBtdWNoIGxvbmdlciB0aGFuIG5lY2Vzc2FyeQ0K
Pg0KPiAgICAgICAgICAgKw0KPg0KPiAgICAgICAgICAgICBGb2N1cyBvbiB3aGF0IGFuIGltcGxl
bWVudGVyIG5lZWRzIHRvIGtub3cNCj4NCj4gICAgICAgICAgICsNCj4NCj4gICAgICAgICAgICAg
VXNlIG5vbi1ub3JtYXRpdmUgcmVmZXJlbmNlcyB0byBSRkMgNjkzNyBmb3IgcHJpb3INCj4gICAg
ICAgICAgICAgbWVhc3VyZW1lbnQgd29yaywgZXRjDQo+DQo+IFRoYW5rcywNCj4gLS1NTS0tDQo+
IFRoZSBiZXN0IHdheSB0byBwcmVkaWN0IHRoZSBmdXR1cmUgaXMgdG8gY3JlYXRlIGl0LiAgLSBB
bGFuIEtheQ0KPg0KPiBXZSBtdXN0IG5vdCB0b2xlcmF0ZSBpbnRvbGVyYW5jZTsNCj4gICAgICAg
ICBob3dldmVyIG91ciByZXNwb25zZSBtdXN0IGJlIGNhcmVmdWxseSBtZWFzdXJlZDoNCj4gICAg
ICAgICAgICAgIHRvbyBzdHJvbmcgd291bGQgYmUgaHlwb2NyaXRpY2FsIGFuZCByaXNrcyBzcGly
YWxpbmcgb3V0IG9mDQo+IGNvbnRyb2w7DQo+ICAgICAgICAgICAgICB0b28gd2VhayByaXNrcyBi
ZWluZyBtaXN0YWtlbiBmb3IgdGFjaXQgYXBwcm92YWwuDQo+DQo+IF9fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IHRjcG0gbWFpbGluZyBsaXN0DQo+IHRj
cG1AaWV0Zi5vcmc8bWFpbHRvOnRjcG1AaWV0Zi5vcmc+DQo+IGh0dHBzOi8vd3d3LmlldGYub3Jn
L21haWxtYW4vbGlzdGluZm8vdGNwbQ0KPg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fXw0KdGNwbSBtYWlsaW5nIGxpc3QNCnRjcG1AaWV0Zi5vcmc8bWFp
bHRvOnRjcG1AaWV0Zi5vcmc+DQpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZv
L3RjcG0NCg==

--_000_d91010d73f23429db4e873b4dc210107hsesslingende_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KLyogU3R5bGUgRGVmaW5pdGlvbnMgKi8NCnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWws
IGRpdi5Nc29Ob3JtYWwNCgl7bWFyZ2luOjBjbTsNCgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJ
Zm9udC1zaXplOjExLjBwdDsNCglmb250LWZhbWlseToiQ2FsaWJyaSIsc2Fucy1zZXJpZjt9DQph
OmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rDQoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsNCgljb2xv
cjpibHVlOw0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0Kc3Bhbi5FLU1haWxGb3JtYXR2
b3JsYWdlMTgNCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6
IkNhbGlicmkiLHNhbnMtc2VyaWY7DQoJY29sb3I6d2luZG93dGV4dDt9DQouTXNvQ2hwRGVmYXVs
dA0KCXttc28tc3R5bGUtdHlwZTpleHBvcnQtb25seTsNCglmb250LWZhbWlseToiQ2FsaWJyaSIs
c2Fucy1zZXJpZjsNCgltc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUzt9DQpAcGFnZSBXb3JkU2Vj
dGlvbjENCgl7c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7DQoJbWFyZ2luOjcwLjg1cHQgNzAuODVwdCAy
LjBjbSA3MC44NXB0O30NCmRpdi5Xb3JkU2VjdGlvbjENCgl7cGFnZTpXb3JkU2VjdGlvbjE7fQ0K
LS0+PC9zdHlsZT48IS0tW2lmIGd0ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlZGVmYXVsdHMgdjpl
eHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+DQo8L3htbD48IVtlbmRpZl0tLT48IS0tW2lmIGd0
ZSBtc28gOV0+PHhtbD4NCjxvOnNoYXBlbGF5b3V0IHY6ZXh0PSJlZGl0Ij4NCjxvOmlkbWFwIHY6
ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPg0KPC9vOnNoYXBlbGF5b3V0PjwveG1sPjwhW2VuZGlmXS0t
Pg0KPC9oZWFkPg0KPGJvZHkgbGFuZz0iREUiIGxpbms9ImJsdWUiIHZsaW5rPSJwdXJwbGUiPg0K
PGRpdiBjbGFzcz0iV29yZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxh
bmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPlRoZSBUQ1BNIGNo
YXJ0ZXIgd291bGQgYWxsb3cgVENQTSB0byBzcGVjaWZ5IGEgVENQIG1lY2hhbmlzbSBpbmNsdWRp
bmcgbWlub3IgYWRhcHRhdGlvbnMgbmVlZGVkIGZvciBRVUlDLiBUaGUgVENQTSBjaGFydGVyIGV4
cGxpY2l0bHkgc3RhdGVzOjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3Jt
YWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMi
PjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFu
IGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVMiPuKAnFRDUCdz
IGNvbmdlc3Rpb24gY29udHJvbCBhbGdvcml0aG1zIGFyZSB0aGUgbW9kZWwgZm9sbG93ZWQgYnk8
bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJF
Ti1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5vdGhlciBJRVRGIHRyYW5z
cG9ydHMgKGUuZy4sIFNDVFAgb3IgRENDUCksIHdoaWNoIGFyZSBzdGFuZGFyZGl6ZWQgaW48bzpw
PjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5vdGhlciB3b3JraW5nIGdyb3Vw
cywgc3VjaCBhcyB0aGUgVHJhbnNwb3J0IEFyZWEgV0cgKHRzdndnKS4gSW4gdGhlPG86cD48L286
cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0
eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+cGFzdCwgdGhlIElFVEYgaGFzIHdvcmtl
ZCBvbiBzZXZlcmFsIGRvY3VtZW50cyBhYm91dCBhbGdvcml0aG1zIHRoYXQ8bzpwPjwvbzpwPjwv
c3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9
Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj5hcmUgc3BlY2lmaWVkIGZvciBtdWx0aXBsZSBw
cm90b2NvbHMgKGUuZy4sIFRDUCBhbmQgU0NUUCkgaW4gdGhlPG86cD48L286cD48L3NwYW4+PC9w
Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFy
ZWFzdC1sYW5ndWFnZTpFTi1VUyI+c2FtZSBkb2N1bWVudC4gV2hpY2ggV0cgc2hlcGhlcmRzIHN1
Y2ggZG9jdW1lbnRzIHdpbGwgYmUgZGV0ZXJtaW5lZDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0ibXNvLWZhcmVhc3Qt
bGFuZ3VhZ2U6RU4tVVMiPm9uIGEgY2FzZS1ieS1jYXNlIGJhc2lzLiBJbiBhbnkgY2FzZSwgdGhl
IFRDUE0gV0cgd2lsbCByZW1haW4gaW48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdl
OkVOLVVTIj5jbG9zZSBjb250YWN0IHdpdGggb3RoZXIgcmVsZXZhbnQgV0dzIHdvcmtpbmcgb24g
dGhlc2UgcHJvdG9jb2xzIHRvPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1V
UyI+ZW5zdXJlIG9wZW5uZXNzIGFuZCBzdHJpbmdlbnQgcmV2aWV3IGZyb20gYWxsIGFuZ2xlcy7i
gJ08bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5n
PSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwv
bzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIg
c3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj4oVGhpcyB3YXMgd3JpdHRlbiBiZWZv
cmUgUVVJQyBzdGFuZGFyZGl6YXRpb24gc3RhcnRlZC4pPG86cD48L286cD48L3NwYW4+PC9wPg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFz
dC1sYW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+SGF2aW5nIG9uZSBkb2N1bWVudCBmb3IgVENQIGFuZCBRVUlDIHdvdWxkIHRodXMg
YmUgYWxsb3dlZCBieSB0aGUgVENQTSBjaGFydGVyLiBJbiB0aGF0IGNhc2UsIHRoZSBkb2N1bWVu
dCB3b3VsZCBwcm9iYWJseSBiZSBsYXN0IGNhbGxlZCBhbHNvIGluIHRoZSBRVUlDIHdvcmtpbmcg
Z3JvdXAg4oCTIGF0IGxlYXN0IHRoaXMNCiBpcyB3aGF0IHdlIGhhdmUgZG9uZSBpbiBzaW1pbGFy
IGNhc2VzIHdpdGggVFNWV0cgaW4gdGhlIHBhc3QuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAg
Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1s
YW5ndWFnZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpF
Ti1VUyI+SSB0aGluayB0aGUgZm9ybWFsIGRldGFpbHMgY291bGQgYmUgZmlndXJlZCBvdXQgaWYg
dGhlcmUgaXMgY29uc2Vuc3VzIHRvIGhhdmUgb25lIGRvY3VtZW50IG9ubHkuIEl0IGlzIHVwIHRv
IHRoZSBUQ1BNIGFuZCBRVUlDIGNvbW11bml0eSB0byBkZWNpZGXigKY8bzpwPjwvbzpwPjwvc3Bh
bj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1z
by1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9Im1zby1mYXJlYXN0
LWxhbmd1YWdlOkVOLVVTIj5NaWNoYWVsPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn
ZTpFTi1VUyI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1h
bCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1VUyI+
PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9y
ZGVyLWxlZnQ6c29saWQgYmx1ZSAxLjVwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDQuMHB0Ij4NCjxk
aXY+DQo8ZGl2IHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNFMUUxRTEgMS4w
cHQ7cGFkZGluZzozLjBwdCAwY20gMGNtIDBjbSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48Yj48
c3BhbiBsYW5nPSJFTi1VUyI+RnJvbTo8L3NwYW4+PC9iPjxzcGFuIGxhbmc9IkVOLVVTIj4gdGNw
bSAmbHQ7dGNwbS1ib3VuY2VzQGlldGYub3JnJmd0Ow0KPGI+T24gQmVoYWxmIE9mIDwvYj5JYW4g
U3dldHQ8YnI+DQo8Yj5TZW50OjwvYj4gVGh1cnNkYXksIE5vdmVtYmVyIDE5LCAyMDIwIDM6MjEg
UE08YnI+DQo8Yj5Ubzo8L2I+IFNjaGVmZmVuZWdnZXIsIFJpY2hhcmQgJmx0O3JzLmlldGZAZ214
LmF0Jmd0Ozxicj4NCjxiPkNjOjwvYj4gdGNwbUBpZXRmLm9yZyBFeHRlbnNpb25zICZsdDt0Y3Bt
QGlldGYub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6PC9iPiBSZTogW3RjcG1dIEludHJvZHVjZSBS
RkMgNjkzNyBiaXMgKFByb3BvcnRpb25hbCBSYXRlIFJlZHVjdGlvbikgYXMgYSB0Y3BtIHdvcmsg
aXRlbT8NCjwvc3Bhbj5JTlBVVCBORUVERUQ8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+
DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxkaXY+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIj5JIHN1cHBvcnQgdGhpcyB3b3JrLjxvOnA+PC9vOnA+PC9wPg0KPGRp
dj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8
ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+T25lIHF1ZXN0aW9uIGZvciBhdXRob3JzIGFuZCB0
aGUgZ3JvdXAuJm5ic3A7IFRoaXMgc3BlY2lmaWNhdGlvbiZuYnNwO2NvdWxkIGJlIGFwcGxpZWQg
dG8gUVVJQywgYnV0IG15IG1lbW9yeSBpcyB0aGF0IHRoZXJlIGFyZSBzb21lIHNtYWxsIGNoYW5n
ZXMgdG8gYWRhcHQgaXQgY29ycmVjdGx5LiZuYnNwOyBXb3VsZCBub3RpbmcgdGhvc2UgY2hhbmdl
cyBiZSBpbiBzY29wZSBmb3IgdGhpcyBkb2N1bWVudCwgYSBzZXBhcmF0ZSBkb2N1bWVudA0KIGlu
IFFVSUMsIG9yIGFuIGV4ZXJjaXNlIGxlZnQgdG8gdGhlIHJlYWRlcj8mbmJzcDsgSSdkIGJlIGhh
cHB5IHRvIGNvbnRyaWJ1dGUgYW55IFFVSUMgc3BlY2lmaWMgdGV4dCBpZiB0aGF0J3MgaGVscGZ1
bC48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxv
OnA+Jm5ic3A7PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
VGhhbmtzLCBJYW48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPC9kaXY+DQo8cCBjbGFzcz0iTXNv
Tm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCI+T24gVGh1LCBOb3YgMTksIDIwMjAgYXQgMjo0MyBBTSBTY2hlZmZlbmVnZ2VyLCBS
aWNoYXJkICZsdDs8YSBocmVmPSJtYWlsdG86cnMuaWV0ZkBnbXguYXQiPnJzLmlldGZAZ214LmF0
PC9hPiZndDsgd3JvdGU6PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxibG9ja3F1b3RlIHN0eWxl
PSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAjQ0NDQ0NDIDEuMHB0O3BhZGRpbmc6MGNt
IDBjbSAwY20gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXJpZ2h0OjBjbSI+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIj5BcyB0aGUgaGV1cmlzdGljIGlzIHRoZSBvbmUgbWFqb3IgaW1wcm92
ZW1lbnQgb3ZlciBSRkM2OTM3LCB0aGF0IGlzIHRoZTxicj4NCm9uZSBhcmVhIEkgYW0gZXNwZWNp
YWxseSBpbnRlcmVzdGVkIGluLiBDdXJyZW50bHkgd29ya2luZyB0byBoYXZlIFBSUiBpbjxicj4N
CkZyZWVCU0QsIGl0IHdvdWxkIGJlIHRoZSBiZXN0IHRpbWUgd2luZG93IHRvIGFkZCB0aGlzIGhl
dXJpc3RpZyBpbnN0ZWFkPGJyPg0Kb2Ygc3dpdGNoaW5nIGJldHdlZW4gdGhlIHR3byB2YXJpYW50
cyBpbiBhIHN0YXRpYyBmYXNoaW9uLjxicj4NCjxicj4NCkkgc3VwcG9ydCB0aGlzIGFuZCB3aWxs
IGNlcnRhaW5seSBiZSByZXZpZXdpbmcgdGhlIGRvY3VtZW50cyBhbmQgcHJvdmlkZTxicj4NCmZl
ZWRiYWNrITxicj4NCjxicj4NClJpY2hhcmQ8YnI+DQo8YnI+DQo8YnI+DQpBbSAxOS4xMS4yMDIw
IHVtIDA0OjU2IHNjaHJpZWIgTWF0dCBNYXRoaXM6PGJyPg0KJmd0OyBUaGUgYXV0aG9ycyBvZiBQ
UlIgd291bGQgbGlrZSB0byB1cGRhdGUgUFJSIHRvIFByb3Bvc2VkIFN0YW5kYXJkPGJyPg0KJmd0
OyBzdGF0dXMuJm5ic3A7IFRoaXMgZW50YWlscyBpbnRyb2R1Y2luZyBhIG5ldyBkb2N1bWVudCBh
cyBhbiB0Y3BtIHdvcmsgaXRlbS48YnI+DQomZ3Q7PGJyPg0KJmd0OyAqUGxlYXNlIGluZGljYXRl
IChub24pIHN1cHBvcnQgYW5kL29yIGNvbW1lbnQuKjxicj4NCiZndDs8YnI+DQomZ3Q7IEZvciBt
b3JlIGRldGFpbHMgc2VlIHRoZSB0Y3BtIG1lZXRpbmcgbWF0ZXJpYWxzIGZyb20gSUVURiAxMDk8
YnI+DQomZ3Q7IG1pbnV0ZXM6PGJyPg0KJmd0OyA8YSBocmVmPSJodHRwczovL2RhdGF0cmFja2Vy
LmlldGYub3JnL21lZXRpbmcvMTA5L21hdGVyaWFscy9taW51dGVzLTEwOS10Y3BtLTAwIiB0YXJn
ZXQ9Il9ibGFuayI+DQpodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL21lZXRpbmcvMTA5L21h
dGVyaWFscy9taW51dGVzLTEwOS10Y3BtLTAwPC9hPjxicj4NCiZndDsgc2xpZGVzOiA8YSBocmVm
PSJodHRwczovL3Rvb2xzLmlldGYub3JnL2h0bWwvZHJhZnQtbWF0aGlzLXRjcG0tcmZjNjkzN2Jp
cy0wMCIgdGFyZ2V0PSJfYmxhbmsiPg0KaHR0cHM6Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0
LW1hdGhpcy10Y3BtLXJmYzY5MzdiaXMtMDA8L2E+PGJyPg0KJmd0Ozxicj4NCiZndDsgVGhlcmUg
d2VyZSBhYm91dCBmb3VyICZxdW90O0kgc3VwcG9ydCB0aGlzIHdvcmsmcXVvdDsgcmVtYXJrcyBh
dCB0aGUgbWljIChub3Q8YnI+DQomZ3Q7IHJlY29yZGVkIGluIHRoZSBtaW51dGVzKSwgYW5kIGFi
b3V0IGFzIG1hbnkgaW4gdGhlIE1lZXRlY2hvIGNoYXQuPGJyPg0KJmd0Ozxicj4NCiZndDsgQWJy
aWRnZWQgSUVURi90Y3BtL1BSUmJpcyBzbGlkZXM6PGJyPg0KJmd0OyAtLTxicj4NCiZndDsgUFJS
ICZuYnNwO3JlY2FwIChSRkM2OTM3IGV4cGVyaW1lbnRhbCk8YnI+DQomZ3Q7IFBSUiBpcyBhIHNw
ZWNpYWwgY29uZ2VzdGlvbiBjb250cm9sIGVmZmVjdGl2ZSBvbmx5IGR1cmluZyBmYXN0IHJlY292
ZXJ5PGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7KiBXaGVuIGluZmxpZ2h0ICZndDs9
IHNzdGhyZXNoLCBzZW5kIGF0IGxvc3NfYmV0YSpyYXRlX2JlZm9yZV9sb3NzIChlLmcuPGJyPg0K
Jmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7bG9zc19iZXRhID0gMC41IGZvciBSZW5vIChha2EgcmF0
ZS1oYWx2aW5nKSwgMC43IGZvciBDdWJpYyk8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyogV2hlbiBp
bmZsaWdodCAmbHQ7IHNzdGhyZXNoLCBzZW5kIGF0IHRoZSBzYW1lIG9yIHR3aWNlIHRoZTxicj4N
CiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwO2RlbGl2ZXJ5X3JhdGUgKG1vcmUgbGF0ZXIpPGJyPg0K
Jmd0OyZuYnNwOyAmbmJzcDsqIFVzZWQgYnkgYWxsIGNvbmdlc3Rpb24gY29udHJvbCBtb2R1bGVz
IGluIExpbnV4IGR1cmluZyBmYXN0IHJlY292ZXJ5PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwO28gQ2FuIGJlIG1vcmUgZG9taW5hbnQgdGhhbiB0aGUgYWN0dWFsIEMuQy4gZm9y
IGxvc3N5IGZsb3dzPGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDt0
aGF04oCZcmUgaW4gZmFzdCByZWNvdmVyeSBjb25zdGFudGx5IChlLmcuIHZpZGVvIHN0cmVhbWlu
Zzxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7dGhyb3VnaCBwb2xp
Y2Vycyk8YnI+DQomZ3Q7PGJyPg0KJmd0OyAtLTxicj4NCiZndDsgQ3VycmVudCBTdGF0dXM8YnI+
DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsqPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsg
Jm5ic3A7ICZuYnNwO1BSUiBpcyB3aWRlbHkgZGVwbG95ZWQ8YnI+DQomZ3Q7PGJyPg0KJmd0OyZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtBdCBsZWFzdCB0aHJlZSBtYWpvciBPU3M6IExpbnV4LCBX
aW5kb3dzLCAoTmV0RmxpeCkgQlNEPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDtvPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7VmFzdCBtYWpvcml0eSBvZiBXZWIgdHJhZmZpYyBmb3IgeWVhcnM8YnI+DQomZ3Q7
PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsqPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7
ICZuYnNwO05vIGNoYW5nZXMgdG8gYWxnb3JpdGhtcyBwdWJsaXNoZWQgaW4gUkZDIDY5Mzc8YnI+
DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288YnI+DQomZ3Q7PGJy
Pg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtQUlItQ1JCIC0gQ29uc2Vy
dmF0aXZlIFJlZHVjdGlvbiBCb3VuZCAtIHN0cmljdCBwYWNrZXQ8YnI+DQomZ3Q7Jm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO2NvbnZlcnNpb24gZHVyaW5nIGxvc3MgcmVjb3Zlcnk8
YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288YnI+DQomZ3Q7
PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtQUlItU1NSQiAtIFNs
b3dzdGFydCBSZWR1Y3Rpb24gQm91bmQgLSBvbmUgZXh0cmEgc2VnbWVudCBwZXIgQUNLPGJyPg0K
Jmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtkdXJpbmcgbG9zcyByZWNvdmVy
eTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyo8YnI+DQomZ3Q7PGJyPg0KJmd0OyZu
YnNwOyAmbmJzcDsgJm5ic3A7MjAxNSBIZXVyaXN0aWMgdG8gZHluYW1pY2FsbHkgc2VsZWN0IHdo
aWNoIHJlZHVjdGlvbiBib3VuZDxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7bzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwO09ubHkgdXNlIFBSUi1TU1JCIHdoZW4gbWFraW5nIGdvb2QgZm9yd2FyZCBwcm9ncmVz
czxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsmIzQzOzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7QUNLcyB0aGF0IGFkdmFuY2VkIHNuZC51bmEgYW5kIHJlcG9y
dCBubyBuZXcgbG9zc2VzPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDtvPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7UmVzb2x2ZXMgc29tZSBwYXRob2xvZ2ljYWwgY2FzZXMgd2l0aCB0b2tlbiBidWNrZXQgcG9s
aWNlcnM8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7JiM0Mzs8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO0NDIGVzdGltYXRlcyBzc3RocmVzaCBiZWZvcmUgaXQg
Y2FuIHBvc3NpYmx5IG1lYXN1cmUgdGhlPGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO3Rva2VuIHJhdGU8YnI+DQomZ3Q7PGJyPg0KJmd0OyZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7JiM0Mzs8YnI+DQomZ3Q7PGJy
Pg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO1Ro
ZSBoZXVyaXN0aWMgbWFrZXMgdGhlIGJlc3Qgb2YgYSBiYWQgc2l0dWF0aW9uPGJyPg0KJmd0Ozxi
cj4NCiZndDsgLS08YnI+DQomZ3Q7IFRlbnRhdGl2ZSBwYXRoIGZvcndhcmQ8YnI+DQomZ3Q7PGJy
Pg0KJmd0OyZuYnNwOyAmbmJzcDsqPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZu
YnNwO0Fkb3B0IGFzIGEgdGNwbSB3b3JrIGl0ZW08YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAm
bmJzcDsqPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwO1VwZGF0ZSB0aGUg
dGV4dDxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7bzxicj4N
CiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO05vcm1hdGl2
ZSBSRkMgMjExOSBsYW5ndWFnZTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7bzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwO0FkZCBNQVkgdXNlIHRoZSBoZXVyaXN0aWMuLi48YnI+DQomZ3Q7PGJyPg0KJmd0OyZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtUcmltIHJlZHVuZGFudCBhbmQgb2Jzb2xldGUgbGFuZ3Vh
Z2U8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7JiM0Mzs8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDsgJm5ic3A7ICZuYnNwO1JGQyA2OTM3IHJlcGVhdHMgaXRzZWxmIGFuZCBpcyBtdWNo
IGxvbmdlciB0aGFuIG5lY2Vzc2FyeTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsmIzQzOzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7
ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7Rm9jdXMgb24gd2hhdCBh
biBpbXBsZW1lbnRlciBuZWVkcyB0byBrbm93PGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyYjNDM7PGJyPg0KJmd0Ozxicj4NCiZndDsm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtVc2Ugbm9uLW5v
cm1hdGl2ZSByZWZlcmVuY2VzIHRvIFJGQyA2OTM3IGZvciBwcmlvcjxicj4NCiZndDsmbmJzcDsg
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDttZWFzdXJlbWVudCB3b3Jr
LCBldGM8YnI+DQomZ3Q7PGJyPg0KJmd0OyBUaGFua3MsPGJyPg0KJmd0OyAtLU1NLS08YnI+DQom
Z3Q7IFRoZSBiZXN0IHdheSB0byBwcmVkaWN0IHRoZSBmdXR1cmUgaXMgdG8gY3JlYXRlIGl0LiAm
bmJzcDstIEFsYW4gS2F5PGJyPg0KJmd0Ozxicj4NCiZndDsgV2UgbXVzdCBub3QgdG9sZXJhdGUg
aW50b2xlcmFuY2U7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDto
b3dldmVyIG91ciByZXNwb25zZSBtdXN0IGJlIGNhcmVmdWxseSBtZWFzdXJlZDo8YnI+DQomZ3Q7
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IHRvbyBzdHJv
bmcgd291bGQgYmUgaHlwb2NyaXRpY2FsIGFuZCByaXNrcyBzcGlyYWxpbmcgb3V0IG9mPGJyPg0K
Jmd0OyBjb250cm9sOzxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDsgdG9vIHdlYWsgcmlza3MgYmVpbmcgbWlzdGFrZW4gZm9yIHRhY2l0IGFw
cHJvdmFsLjxicj4NCiZndDs8YnI+DQomZ3Q7IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fPGJyPg0KJmd0OyB0Y3BtIG1haWxpbmcgbGlzdDxicj4NCiZndDsg
PGEgaHJlZj0ibWFpbHRvOnRjcG1AaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj50Y3BtQGlldGYu
b3JnPC9hPjxicj4NCiZndDsgPGEgaHJlZj0iaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9s
aXN0aW5mby90Y3BtIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1h
bi9saXN0aW5mby90Y3BtPC9hPjxicj4NCiZndDs8YnI+DQo8YnI+DQpfX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4NCnRjcG0gbWFpbGluZyBsaXN0PGJy
Pg0KPGEgaHJlZj0ibWFpbHRvOnRjcG1AaWV0Zi5vcmciIHRhcmdldD0iX2JsYW5rIj50Y3BtQGll
dGYub3JnPC9hPjxicj4NCjxhIGhyZWY9Imh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlz
dGluZm8vdGNwbSIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4v
bGlzdGluZm8vdGNwbTwvYT48bzpwPjwvbzpwPjwvcD4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0K
PC9kaXY+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==

--_000_d91010d73f23429db4e873b4dc210107hsesslingende_--


From nobody Thu Nov 19 06:37:36 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0E8D03A03F3 for ; Thu, 19 Nov 2020 06:37:35 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level: 
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vk0s9yOOuJ11 for ; Thu, 19 Nov 2020 06:37:32 -0800 (PST)
Received: from pegasus.erg.abdn.ac.uk (pegasus.erg.abdn.ac.uk [IPv6:2001:630:42:150::2]) by ietfa.amsl.com (Postfix) with ESMTP id 053973A03EF for ; Thu, 19 Nov 2020 06:37:31 -0800 (PST)
Received: from GF-MacBook-Pro.lan (fgrpf.plus.com [212.159.18.54]) by pegasus.erg.abdn.ac.uk (Postfix) with ESMTPSA id 74DBE1B0022B; Thu, 19 Nov 2020 14:37:27 +0000 (GMT)
To: Ian Swett , "Scheffenegger, Richard" 
Cc: "tcpm@ietf.org Extensions" 
References:   
From: Gorry Fairhurst 
Message-ID: <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk>
Date: Thu, 19 Nov 2020 14:37:27 +0000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:78.0) Gecko/20100101 Thunderbird/78.4.3
MIME-Version: 1.0
In-Reply-To: 
Content-Type: multipart/alternative; boundary="------------89914FAB99531C7529433F90"
Content-Language: en-GB
Archived-At: 
Subject: Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 14:37:35 -0000

This is a multi-part message in MIME format.
--------------89914FAB99531C7529433F90
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 19/11/2020 14:20, Ian Swett wrote:
> I support this work.
>
> One question for authors and the group.  This specification could be 
> applied to QUIC, but my memory is that there are some small changes to 
> adapt it correctly.  Would noting those changes be in scope for this 
> document, a separate document in QUIC, or an exercise left to the 
> reader?  I'd be happy to contribute any QUIC specific text if that's 
> helpful.
>
> Thanks, Ian

I'd hope the differences for QUIC ARE captured in the RFC series. If 
they are small, it might be possible to add to this draft perhaps?

Gorry

>
> On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richard  > wrote:
>
>     As the heuristic is the one major improvement over RFC6937, that
>     is the
>     one area I am especially interested in. Currently working to have
>     PRR in
>     FreeBSD, it would be the best time window to add this heuristig
>     instead
>     of switching between the two variants in a static fashion.
>
>     I support this and will certainly be reviewing the documents and
>     provide
>     feedback!
>
>     Richard
>
>
>     Am 19.11.2020 um 04:56 schrieb Matt Mathis:
>     > The authors of PRR would like to update PRR to Proposed Standard
>     > status.  This entails introducing a new document as an tcpm work
>     item.
>     >
>     > *Please indicate (non) support and/or comment.*
>     >
>     > For more details see the tcpm meeting materials from IETF 109
>     > minutes:
>     >
>     https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
>     
>     > slides:
>     https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
>     
>     >
>     > There were about four "I support this work" remarks at the mic (not
>     > recorded in the minutes), and about as many in the Meetecho chat.
>     >
>     > Abridged IETF/tcpm/PRRbis slides:
>     > --
>     > PRR  recap (RFC6937 experimental)
>     > PRR is a special congestion control effective only during fast
>     recovery
>     >
>     >   * When inflight >= ssthresh, send at
>     loss_beta*rate_before_loss (e.g.
>     >     loss_beta = 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
>     >   * When inflight < ssthresh, send at the same or twice the
>     >     delivery_rate (more later)
>     >   * Used by all congestion control modules in Linux during fast
>     recovery
>     >       o Can be more dominant than the actual C.C. for lossy flows
>     >         that’re in fast recovery constantly (e.g. video streaming
>     >         through policers)
>     >
>     > --
>     > Current Status
>     >
>     >   *
>     >
>     >     PRR is widely deployed
>     >
>     >       o
>     >
>     >         At least three major OSs: Linux, Windows, (NetFlix) BSD
>     >
>     >       o
>     >
>     >         Vast majority of Web traffic for years
>     >
>     >   *
>     >
>     >     No changes to algorithms published in RFC 6937
>     >
>     >       o
>     >
>     >         PRR-CRB - Conservative Reduction Bound - strict packet
>     >         conversion during loss recovery
>     >
>     >       o
>     >
>     >         PRR-SSRB - Slowstart Reduction Bound - one extra segment
>     per ACK
>     >         during loss recovery
>     >
>     >   *
>     >
>     >     2015 Heuristic to dynamically select which reduction bound
>     >
>     >       o
>     >
>     >         Only use PRR-SSRB when making good forward progress
>     >
>     >           +
>     >
>     >             ACKs that advanced snd.una and report no new losses
>     >
>     >       o
>     >
>     >         Resolves some pathological cases with token bucket policers
>     >
>     >           +
>     >
>     >             CC estimates ssthresh before it can possibly measure the
>     >             token rate
>     >
>     >           +
>     >
>     >             The heuristic makes the best of a bad situation
>     >
>     > --
>     > Tentative path forward
>     >
>     >   *
>     >
>     >     Adopt as a tcpm work item
>     >
>     >   *
>     >
>     >     Update the text
>     >
>     >       o
>     >
>     >         Normative RFC 2119 language
>     >
>     >       o
>     >
>     >         Add MAY use the heuristic...
>     >
>     >       o
>     >
>     >         Trim redundant and obsolete language
>     >
>     >           +
>     >
>     >             RFC 6937 repeats itself and is much longer than
>     necessary
>     >
>     >           +
>     >
>     >             Focus on what an implementer needs to know
>     >
>     >           +
>     >
>     >             Use non-normative references to RFC 6937 for prior
>     >             measurement work, etc
>     >
>     > Thanks,
>     > --MM--
>     > The best way to predict the future is to create it.  - Alan Kay
>     >
>     > We must not tolerate intolerance;
>     >         however our response must be carefully measured:
>     >              too strong would be hypocritical and risks
>     spiraling out of
>     > control;
>     >              too weak risks being mistaken for tacit approval.
>     >
>     > _______________________________________________
>     > tcpm mailing list
>     > tcpm@ietf.org 
>     > https://www.ietf.org/mailman/listinfo/tcpm
>     
>     >
>
>     _______________________________________________
>     tcpm mailing list
>     tcpm@ietf.org 
>     https://www.ietf.org/mailman/listinfo/tcpm
>     
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm



--------------89914FAB99531C7529433F90
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit


  
    
  
  
    
On 19/11/2020 14:20, Ian Swett wrote:

    

    

      
      
I support this work.
        


        

        
One question for authors and the group.  This
          specification could be applied to QUIC, but my memory is that
          there are some small changes to adapt it correctly.  Would
          noting those changes be in scope for this document, a separate
          document in QUIC, or an exercise left to the reader?  I'd be
          happy to contribute any QUIC specific text if that's helpful.

        


        

        
Thanks, Ian

      

    

    
I'd hope the differences for QUIC ARE captured in the RFC series.
      If they are small, it might be possible to add to this draft
      perhaps? 

    

    
Gorry

    

    


      

        
On Thu, Nov 19, 2020 at 2:43
          AM Scheffenegger, Richard <rs.ietf@gmx.at> wrote:

        

        
As
          the heuristic is the one major improvement over RFC6937, that
          is the

          one area I am especially interested in. Currently working to
          have PRR in

          FreeBSD, it would be the best time window to add this
          heuristig instead

          of switching between the two variants in a static fashion.

          

          I support this and will certainly be reviewing the documents
          and provide

          feedback!

          

          Richard

          

          

          Am 19.11.2020 um 04:56 schrieb Matt Mathis:

          > The authors of PRR would like to update PRR to Proposed
          Standard

          > status.  This entails introducing a new document as an
          tcpm work item.

          >

          > *Please indicate (non) support and/or comment.*

          >

          > For more details see the tcpm meeting materials from IETF
          109

          > minutes:

          > https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00

          > slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00

          >

          > There were about four "I support this work" remarks at
          the mic (not

          > recorded in the minutes), and about as many in the
          Meetecho chat.

          >

          > Abridged IETF/tcpm/PRRbis slides:

          > --

          > PRR  recap (RFC6937 experimental)

          > PRR is a special congestion control effective only during
          fast recovery

          >

          >   * When inflight >= ssthresh, send at
          loss_beta*rate_before_loss (e.g.

          >     loss_beta = 0.5 for Reno (aka rate-halving), 0.7 for
          Cubic)

          >   * When inflight < ssthresh, send at the same or
          twice the

          >     delivery_rate (more later)

          >   * Used by all congestion control modules in Linux
          during fast recovery

          >       o Can be more dominant than the actual C.C. for
          lossy flows

          >         that’re in fast recovery constantly (e.g. video
          streaming

          >         through policers)

          >

          > --

          > Current Status

          >

          >   *

          >

          >     PRR is widely deployed

          >

          >       o

          >

          >         At least three major OSs: Linux, Windows,
          (NetFlix) BSD

          >

          >       o

          >

          >         Vast majority of Web traffic for years

          >

          >   *

          >

          >     No changes to algorithms published in RFC 6937

          >

          >       o

          >

          >         PRR-CRB - Conservative Reduction Bound - strict
          packet

          >         conversion during loss recovery

          >

          >       o

          >

          >         PRR-SSRB - Slowstart Reduction Bound - one extra
          segment per ACK

          >         during loss recovery

          >

          >   *

          >

          >     2015 Heuristic to dynamically select which reduction
          bound

          >

          >       o

          >

          >         Only use PRR-SSRB when making good forward
          progress

          >

          >           +

          >

          >             ACKs that advanced snd.una and report no new
          losses

          >

          >       o

          >

          >         Resolves some pathological cases with token
          bucket policers

          >

          >           +

          >

          >             CC estimates ssthresh before it can possibly
          measure the

          >             token rate

          >

          >           +

          >

          >             The heuristic makes the best of a bad
          situation

          >

          > --

          > Tentative path forward

          >

          >   *

          >

          >     Adopt as a tcpm work item

          >

          >   *

          >

          >     Update the text

          >

          >       o

          >

          >         Normative RFC 2119 language

          >

          >       o

          >

          >         Add MAY use the heuristic...

          >

          >       o

          >

          >         Trim redundant and obsolete language

          >

          >           +

          >

          >             RFC 6937 repeats itself and is much longer
          than necessary

          >

          >           +

          >

          >             Focus on what an implementer needs to know

          >

          >           +

          >

          >             Use non-normative references to RFC 6937 for
          prior

          >             measurement work, etc

          >

          > Thanks,

          > --MM--

          > The best way to predict the future is to create it.  -
          Alan Kay

          >

          > We must not tolerate intolerance;

          >         however our response must be carefully measured:

          >              too strong would be hypocritical and risks
          spiraling out of

          > control;

          >              too weak risks being mistaken for tacit
          approval.

          >

          > _______________________________________________

          > tcpm mailing list

          > tcpm@ietf.org

          > https://www.ietf.org/mailman/listinfo/tcpm

          >

          

          _______________________________________________

          tcpm mailing list

          tcpm@ietf.org

          https://www.ietf.org/mailman/listinfo/tcpm

        

      

      

      

      
_______________________________________________
tcpm mailing list
tcpm@ietf.org
https://www.ietf.org/mailman/listinfo/tcpm


    

    


    

  


--------------89914FAB99531C7529433F90--


From nobody Thu Nov 19 07:36:49 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 068BB3A0B49 for ; Thu, 19 Nov 2020 07:36:45 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LTne5N03sC3x for ; Thu, 19 Nov 2020 07:36:42 -0800 (PST)
Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 62A5C3A0B4E for ; Thu, 19 Nov 2020 07:36:42 -0800 (PST)
Received: by mail-ej1-x62e.google.com with SMTP id a16so8535463ejj.5 for ; Thu, 19 Nov 2020 07:36:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=hJdKs49Yd3P35X76Yj6DwlhGajrRbKeAN5aH77qCz5o=; b=YGRcf8fbjwVmFVrc7uB27kO0NfhCMIzOwERqI7bHFT7+PSEFSdHGamw/Wk4ZA2P3qw yL0QLgSN32DLod8V9IJu5XooktLdqWKYN00o78UklXc5RhidIh0XUotbkww4ZfGfHiPN Pmp6pr7DireofDiH/7xvZr/cHT+ufDPdesON548i4cqi6ihLHVOC+zcUQ0JS2lfPhy59 bxxHNS/XF1WLA35kKTYHRoMp8KkT0HD5AK6A7R+xndpbtZdiNKetjcC23SwbC7s02uoo H6lQXeFyK23EO80vyc6RNVvi7SFQKTxkNNVu3ohzeN98zy/erSjYg7R2QSTl8nDeSYYl 17Gw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=hJdKs49Yd3P35X76Yj6DwlhGajrRbKeAN5aH77qCz5o=; b=Ts5sVdgfjHMGsOa+L2GrAkcq1hwdD6rN/nt3LGQ0mHQ0to7iXPrtiUCr36davy4LrA ZwWjPl/6kR2bz636bwnA2i3AkoXFvYWmua7ontGMcRObpODlrZgqFPefJWiTE74kDRuO thIGRPJeLVEe4wJT7SX3qvgQcL/K6ebXTRgQNTyFO+wwfcQrlUPZun5j1+kuQhVJlxO8 NFTYCnq8AC0GRuLO0Wydt1KvOCwnmZFtfOPLjkgd1nLc3HpOFJgcqXfK/iUJczNLDyHz /627WQMjOaV+x4bj9XXx0tIehl/otOMlPj8KoBYExwRfYEg0e7UiF/RQPcI8kOZfNN0b PhCg==
X-Gm-Message-State: AOAM531gNY5Cak25+E404UsJn0pzGevuqrDRhxg8UaZ9GHezIYAMLxuY l7dE46l0o8Zi4x6qRbSFVy7aruL5JrnfoodvmXXryg==
X-Google-Smtp-Source: ABdhPJx9T5g80W7SOLsW5YUhmbrZ3PgbRI/T5I/UTpvq7C295GaULVcLX+AO3ZYo9Nwp5yZpS0fzMkX9uRhTj7NXLYA=
X-Received: by 2002:a17:906:8058:: with SMTP id x24mr29683123ejw.272.1605800200587;  Thu, 19 Nov 2020 07:36:40 -0800 (PST)
MIME-Version: 1.0
References: 
In-Reply-To: 
From: Tom Herbert 
Date: Thu, 19 Nov 2020 08:36:29 -0700
Message-ID: 
To: Alexander Azimov 
Cc: tcpm , IPv6 Operations 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 15:36:45 -0000

On Thu, Nov 19, 2020 at 3:49 AM Alexander Azimov  wro=
te:
>
> Dear colleagues,
>
> I have added in the cc both v6op and tcpm for a reason and let me explain=
 why.
>
> It's clear that we are moving forward with load balancing that uses flow =
label (FL). And the pressure will increase with SRv6 adoption. But at the m=
oment wide adoption of FL-based load-balancing may create significant issue=
s for TCP Anycast services.
>
> RFC6437 suggests putting hash from 5-tuple into FL value. And as far as I=
 know, there is no document that updates this behavior. This description is=
 perfectly fine, but what is implemented in the Linux kernel is different: =
FL is carrying hash from 5-tuple with an additional seed, and this seed is =
randomly changed after each RTO/SYN_RTO event. Here are related patches:
>
> https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.m=
tv.corp.google.com/
> https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@he=
rbertland.com/
> https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
>
Alexander,

The initial intent of the Linux flow label code was that the hash was
persistent for the lifetime of the connection; in fact we fixed a
nasty bug where the hash would change when a connection moved to TW
state causing some firewalls not to see the final ACK so they can
remove state. The feature where the flow hash is recalculated for a
failing connection should be optional and not the default behavior. If
this isn't what the implementation is doing I can take it up on Linux
netdev to fix it.

Have you been able to show this behavior is actually happening on a
running system?

Tom

>
> This is a great thing by the way because in the data center environment w=
ith multiple equal paths it gives a way to have pseudo-multipath TCP which =
jumps between paths in case of an outage. There might be interest to write =
down an informational document for this.
>
> But the problem happens when TCP flows start jumping between anycast poin=
ts. If anycast service provides connection tracking (L7/TCP proxy, stateful=
 L4 balancer) the 'jump' may redirect traffic to another point, and with th=
e adoption of FL load balancing by the transit providers - to another locat=
ion, where no TCP state is available. In other words, an established TCP se=
ssion breaks after RTO event. And it's not just a theory - we already faced=
 this kind of issue in our network, and it is really hard to debug.
>
> I wonder what you think is a proper solution:
>
> Making FL related RTO change as knob instead of default behavior;
> Adding negotiation behavior in TCP;
> Something else?
>
> I'm looking forward to your advice. If there is a document that describes=
 the above problem - please give me a reference.
>
> --
> Best regards,
> Alexander Azimov
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops


From nobody Thu Nov 19 08:40:59 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B4D763A09F1 for ; Thu, 19 Nov 2020 08:40:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.599
X-Spam-Level: 
X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7qntK6DdE1md for ; Thu, 19 Nov 2020 08:40:55 -0800 (PST)
Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 984FF3A09E7 for ; Thu, 19 Nov 2020 08:40:55 -0800 (PST)
Received: by mail-il1-x135.google.com with SMTP id w10so5930079ilq.5 for ; Thu, 19 Nov 2020 08:40:55 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=scKkLfm13nZWHBBntXR5uk/QImHB9n9PkObd15R5FrE=; b=pFJmLp2m0wW8mcKRG5es7YqRbKzEQRBzOzi7969TwQxRT9DNKv1eHnP5XMJb8BObjU lUioEGaHm33lDb+LQ/28410/wFgwB+okvp0H1Rs8eMKp/hi41GaayX8I6LCVe/gqpl8u RkPn6U5MGBDRzG3345ms8Lef+QPch2EbzW2SVexw3lUUHp21eIIvK67XdvfOse5rVRkO YzcziZxd7AoDAO0Zl5ug7k2Agil7aq/dXtxZ5Zk3hVvx9+/n11s6wRC+M/mVO/pDUdZJ DRc+m28OjupfcesXQepD/YtzNo5WRpn1rmVdR4Z1IMrL4wFSw19RWIfHNz1BPE3n5Cq/ 5ErA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=scKkLfm13nZWHBBntXR5uk/QImHB9n9PkObd15R5FrE=; b=dXMNdNHOnVm/QNHolYyz9o0JIWsHg2FyO1JVDnOh55b+4eSi5ZUSYkYsWRKzKHWwzz hPNsJgLSEfAc0AQ/swCJ3RM5tn1L/jPFDRkoUiHcQyqzVzsW2vdDbfDMT1GGvAxz2wzn 5fpnHa4rllAJym9uctfsSqxp5ck0wc/e/08jn9Bergfb1JwDbrwCQZq/+79In4WwXVhh n2mqofl4AOz6YpyMkRd8uJpm0cxKPys3Jhbn4HEF1UIu1MBlyvtNMNa4PD23OqReIptQ kW6VCUYf1KskLRK5v/3v6HsOk5hmFKy4Lhlj+SCZYPSHsVGxzlXQY+/7YVDTGb0b6Q7e JQHw==
X-Gm-Message-State: AOAM530Aq0LYtEv3UunLLWuhFWzZW5LEk4DMAGgozT7/m4bbHCj/6Vy5 XsGaU1xyg5uTL/+bt1WaHbXJss013ZyG5rnJOAqYNw==
X-Google-Smtp-Source: ABdhPJwJahrqBSbSNm2d0BWH+ZpTAkLd8JDQ9hPFslSWBbm4pTbimPxXIlZ61UnB7pfN0mrLhhIyN4DR30989y+Vd8k=
X-Received: by 2002:a92:d18a:: with SMTP id z10mr21165942ilz.61.1605804054663;  Thu, 19 Nov 2020 08:40:54 -0800 (PST)
MIME-Version: 1.0
References:    <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk>
In-Reply-To: <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk>
From: Matt Mathis 
Date: Thu, 19 Nov 2020 08:40:42 -0800
Message-ID: 
To: Gorry Fairhurst , Ian Swett 
Cc: "Scheffenegger, Richard" , "tcpm@ietf.org Extensions" 
Content-Type: multipart/alternative; boundary="000000000000425d5f05b478642e"
Archived-At: 
Subject: Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 16:40:58 -0000

--000000000000425d5f05b478642e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

If we can get away with a non-normative description of the heuristic, QUIC
is no problem at all.    If we need normative descriptions, then TCP and
QUIC have to be different, and we need to include normative references to
both recovery machines and recovery state variables, which are not in scope
for the current doc.

I would prefer to stay with a non-normative description of the heuristic,
and scrub the rest of the document for language that conflicts with QUIC.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
       however our response must be carefully measured:
            too strong would be hypocritical and risks spiraling out of
control;
            too weak risks being mistaken for tacit approval.


On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst 
wrote:

> On 19/11/2020 14:20, Ian Swett wrote:
>
> I support this work.
>
> One question for authors and the group.  This specification could be
> applied to QUIC, but my memory is that there are some small changes to
> adapt it correctly.  Would noting those changes be in scope for this
> document, a separate document in QUIC, or an exercise left to the reader?
> I'd be happy to contribute any QUIC specific text if that's helpful.
>
> Thanks, Ian
>
> I'd hope the differences for QUIC ARE captured in the RFC series. If they
> are small, it might be possible to add to this draft perhaps?
>
> Gorry
>
>
> On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richard 
> wrote:
>
>> As the heuristic is the one major improvement over RFC6937, that is the
>> one area I am especially interested in. Currently working to have PRR in
>> FreeBSD, it would be the best time window to add this heuristig instead
>> of switching between the two variants in a static fashion.
>>
>> I support this and will certainly be reviewing the documents and provide
>> feedback!
>>
>> Richard
>>
>>
>> Am 19.11.2020 um 04:56 schrieb Matt Mathis:
>> > The authors of PRR would like to update PRR to Proposed Standard
>> > status.  This entails introducing a new document as an tcpm work item.
>> >
>> > *Please indicate (non) support and/or comment.*
>> >
>> > For more details see the tcpm meeting materials from IETF 109
>> > minutes:
>> > https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
>> > slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
>> >
>> > There were about four "I support this work" remarks at the mic (not
>> > recorded in the minutes), and about as many in the Meetecho chat.
>> >
>> > Abridged IETF/tcpm/PRRbis slides:
>> > --
>> > PRR  recap (RFC6937 experimental)
>> > PRR is a special congestion control effective only during fast recover=
y
>> >
>> >   * When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e=
.g.
>> >     loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
>> >   * When inflight < ssthresh, send at the same or twice the
>> >     delivery_rate (more later)
>> >   * Used by all congestion control modules in Linux during fast recove=
ry
>> >       o Can be more dominant than the actual C.C. for lossy flows
>> >         that=E2=80=99re in fast recovery constantly (e.g. video stream=
ing
>> >         through policers)
>> >
>> > --
>> > Current Status
>> >
>> >   *
>> >
>> >     PRR is widely deployed
>> >
>> >       o
>> >
>> >         At least three major OSs: Linux, Windows, (NetFlix) BSD
>> >
>> >       o
>> >
>> >         Vast majority of Web traffic for years
>> >
>> >   *
>> >
>> >     No changes to algorithms published in RFC 6937
>> >
>> >       o
>> >
>> >         PRR-CRB - Conservative Reduction Bound - strict packet
>> >         conversion during loss recovery
>> >
>> >       o
>> >
>> >         PRR-SSRB - Slowstart Reduction Bound - one extra segment per A=
CK
>> >         during loss recovery
>> >
>> >   *
>> >
>> >     2015 Heuristic to dynamically select which reduction bound
>> >
>> >       o
>> >
>> >         Only use PRR-SSRB when making good forward progress
>> >
>> >           +
>> >
>> >             ACKs that advanced snd.una and report no new losses
>> >
>> >       o
>> >
>> >         Resolves some pathological cases with token bucket policers
>> >
>> >           +
>> >
>> >             CC estimates ssthresh before it can possibly measure the
>> >             token rate
>> >
>> >           +
>> >
>> >             The heuristic makes the best of a bad situation
>> >
>> > --
>> > Tentative path forward
>> >
>> >   *
>> >
>> >     Adopt as a tcpm work item
>> >
>> >   *
>> >
>> >     Update the text
>> >
>> >       o
>> >
>> >         Normative RFC 2119 language
>> >
>> >       o
>> >
>> >         Add MAY use the heuristic...
>> >
>> >       o
>> >
>> >         Trim redundant and obsolete language
>> >
>> >           +
>> >
>> >             RFC 6937 repeats itself and is much longer than necessary
>> >
>> >           +
>> >
>> >             Focus on what an implementer needs to know
>> >
>> >           +
>> >
>> >             Use non-normative references to RFC 6937 for prior
>> >             measurement work, etc
>> >
>> > Thanks,
>> > --MM--
>> > The best way to predict the future is to create it.  - Alan Kay
>> >
>> > We must not tolerate intolerance;
>> >         however our response must be carefully measured:
>> >              too strong would be hypocritical and risks spiraling out =
of
>> > control;
>> >              too weak risks being mistaken for tacit approval.
>> >
>> > _______________________________________________
>> > tcpm mailing list
>> > tcpm@ietf.org
>> > https://www.ietf.org/mailman/listinfo/tcpm
>> >
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>
> _______________________________________________
> tcpm mailing listtcpm@ietf.orghttps://www.ietf.org/mailman/listinfo/tcpm
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--000000000000425d5f05b478642e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


If we can get away with a non-normative description of the=
 heuristic, QUIC is no problem at all.=C2=A0 =C2=A0 If we need normative de=
scriptions, then TCP and QUIC have to be different, and we need to include =
normative references to both recovery machines and recovery state=C2=A0vari=
ables, which are not in scope for the current doc.

I wou=
ld prefer to stay with a non-normative description of the heuristic, and sc=
rub the rest of the document for language that conflicts with QUIC.

Thanks,
--MM--
The best way to predict the futu=
re is to create it. =C2=A0- Alan Kay

We must not tolerate intoleranc=
e;
=C2=A0 =C2=A0 =C2=A0 =C2=A0however our response mu=
st be carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 too strong would be hypocritical and risks spiraling out of cont=
rol;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks bei=
ng mistaken for tacit approval.

On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst <gorry@erg.abdn.ac.uk> wrote:

 =20
   =20
 =20
  

    
On 19/11/2020 14:20, Ian Swett wrote:

    

    

     =20
      
I support this work.
        


        

        
One question for authors and the group.=C2=A0 This
          specification=C2=A0could be applied to QUIC, but my memory is tha=
t
          there are some small changes to adapt it correctly.=C2=A0 Would
          noting those changes be in scope for this document, a separate
          document in QUIC, or an exercise left to the reader?=C2=A0 I'=
d be
          happy to contribute any QUIC specific text if that's helpful.=


        


        

        
Thanks, Ian

      

    

    
I'd hope the differences for QUIC ARE captured in the RFC series=
.
      If they are small, it might be possible to add to this draft
      perhaps? 

    

    
Gorry

    

    


      

        
On Thu, Nov 19, 2020 at 2:43
          AM Scheffenegger, Richard <rs.ietf@gmx.at> wrote:

        

        
As
          the heuristic is the one major improvement over RFC6937, that
          is the

          one area I am especially interested in. Currently working to
          have PRR in

          FreeBSD, it would be the best time window to add this
          heuristig instead

          of switching between the two variants in a static fashion.

          

          I support this and will certainly be reviewing the documents
          and provide

          feedback!

          

          Richard

          

          

          Am 19.11.2020 um 04:56 schrieb Matt Mathis:

          > The authors of PRR would like to update PRR to Proposed
          Standard

          > status.=C2=A0 This entails introducing a new document as an
          tcpm work item.

          >

          > *Please indicate (non) support and/or comment.*

          >

          > For more details see the tcpm meeting materials from IETF
          109

          > minutes:

          > https://datatra=
cker.ietf.org/meeting/109/materials/minutes-109-tcpm-00

          > slides: https://tools.ietf=
.org/html/draft-mathis-tcpm-rfc6937bis-00

          >

          > There were about four "I support this work" remark=
s at
          the mic (not

          > recorded in the minutes), and about as many in the
          Meetecho chat.

          >

          > Abridged IETF/tcpm/PRRbis slides:

          > --

          > PRR =C2=A0recap (RFC6937 experimental)

          > PRR is a special congestion control effective only during
          fast recovery

          >

          >=C2=A0 =C2=A0* When inflight >=3D ssthresh, send at
          loss_beta*rate_before_loss (e.g.

          >=C2=A0 =C2=A0 =C2=A0loss_beta =3D 0.5 for Reno (aka rate-halv=
ing), 0.7 for
          Cubic)

          >=C2=A0 =C2=A0* When inflight < ssthresh, send at the same =
or
          twice the

          >=C2=A0 =C2=A0 =C2=A0delivery_rate (more later)

          >=C2=A0 =C2=A0* Used by all congestion control modules in Linu=
x
          during fast recovery

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o Can be more dominant than the ac=
tual C.C. for
          lossy flows

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0that=E2=80=99re in fast rec=
overy constantly (e.g. video
          streaming

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0through policers)

          >

          > --

          > Current Status

          >

          >=C2=A0 =C2=A0*

          >

          >=C2=A0 =C2=A0 =C2=A0PRR is widely deployed

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0At least three major OSs: L=
inux, Windows,
          (NetFlix) BSD

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Vast majority of Web traffi=
c for years

          >

          >=C2=A0 =C2=A0*

          >

          >=C2=A0 =C2=A0 =C2=A0No changes to algorithms published in RFC=
 6937

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-CRB - Conservative Redu=
ction Bound - strict
          packet

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0conversion during loss reco=
very

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-SSRB - Slowstart Reduct=
ion Bound - one extra
          segment per ACK

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0during loss recovery

          >

          >=C2=A0 =C2=A0*

          >

          >=C2=A0 =C2=A0 =C2=A02015 Heuristic to dynamically select whic=
h reduction
          bound

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Only use PRR-SSRB when maki=
ng good forward
          progress

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ACKs that adv=
anced snd.una and report no new
          losses

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Resolves some pathological =
cases with token
          bucket policers

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CC estimates =
ssthresh before it can possibly
          measure the

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0token rate
          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The heuristic=
 makes the best of a bad
          situation

          >

          > --

          > Tentative path forward

          >

          >=C2=A0 =C2=A0*

          >

          >=C2=A0 =C2=A0 =C2=A0Adopt as a tcpm work item

          >

          >=C2=A0 =C2=A0*

          >

          >=C2=A0 =C2=A0 =C2=A0Update the text

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Normative RFC 2119 language=


          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Add MAY use the heuristic..=
.

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0o

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Trim redundant and obsolete=
 language

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RFC 6937 repe=
ats itself and is much longer
          than necessary

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Focus on what=
 an implementer needs to know

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

          >

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Use non-norma=
tive references to RFC 6937 for
          prior

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0measurement w=
ork, etc

          >

          > Thanks,

          > --MM--

          > The best way to predict the future is to create it. =C2=A0-
          Alan Kay

          >

          > We must not tolerate intolerance;

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must b=
e carefully measured:

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong w=
ould be hypocritical and risks
          spiraling out of

          > control;

          >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak ris=
ks being mistaken for tacit
          approval.

          >

          > _______________________________________________

          > tcpm mailing list

          > tcpm@ietf=
.org

          > https://www.ietf.org/mailman/listinfo/tcp=
m

          >

          

          _______________________________________________

          tcpm mailing list

          tcpm@ietf.org<=
/a>

          https://www.ietf.org/mailman/listinfo/tcpm
        

      

      

      

      
_______________________________________________
tcpm mailing list
tcpm@ietf.org
ht=
tps://www.ietf.org/mailman/listinfo/tcpm


    

    


    

  


_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--000000000000425d5f05b478642e--


From nobody Thu Nov 19 09:37:07 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 81A1B3A0F5E for ; Thu, 19 Nov 2020 09:37:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2
X-Spam-Level: 
X-Spam-Status: No, score=-2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9,  DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cca39e6WZM6c for ; Thu, 19 Nov 2020 09:36:58 -0800 (PST)
Received: from NAM06-BL2-obe.outbound.protection.outlook.com (mail-eopbgr650097.outbound.protection.outlook.com [40.107.65.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B0A603A0ECA for ; Thu, 19 Nov 2020 09:36:58 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AYyNw47bJ5MH4wpoXsGgI539ZeJ28UKmcifA97jZHS6WkDjBI+8O5IQt0Jnl8em32d7C8fXp8QahmVlkBjisNjNX1l7O4bZlbLrJeQRaz1D9vfWftBJaf3UUYpNv9otZOClsjT0KeVxiXbqRtPYx+/wKCLuvyBI0k+sDBQa3DKLwnE8xQms0A9WChS3+KPMOZQbNzZ/ZpPpGmFTpKOxpwntdv++wrnBRvDzwdoTFaoFQgWsCMYpbP+lQx3eCAqJD053tFAYL2wr8XLIQeAbpf3Scm0X+OF3w2sNPg5trWhz40YVbvz/g7aXBW7nS0ZRe9cbVAKlxns3uQ8uuAYL0Aw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;  s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UrW5ewHFhP+8QrALu8v4IakBsHsLlnp2MHDch3W6tZk=; b=c4Pw3jTbeDTu28fiiiNAuH02h0wn7zZICFW0GqwMMIet84Lf3Rjh/144qWACRF/wWjJCiz6fZmosMIEHJ7nfpNm44aTT5G4E2H8OGTX80e0d4hd2UX0dIGqmdhtUhnQVJMIJzNkrHS39MoMVme6FffZxWL3p0MFQYfeFyO/s2ADVBmARDTacShYWGJDAC0I48Ai/TVX3D4xUSu3uYM8oy8E3oBlsUQcItYrQfgThmmWG01vKPjRGUvF2PgWgMfWZyQMGc27IgXGFHs8/TMZE/iNFojd8QFSHqP/WbUjjKcfL2CIiarWI7UO55xlcuFl6y0MmpbleXKFwSX8Y4iRXKA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UrW5ewHFhP+8QrALu8v4IakBsHsLlnp2MHDch3W6tZk=; b=KkKpxWH8rS0PjjtKaeFveAeWBCFErGirrIUd+Z46gwWogoVv3Bbo/3jFrh8yx88hQsAXpARnR+4ykLqL+K26wZX78i2IpEr/BQOCNEwnzegJyhQKIGYx5vHcbUHYLlf+AM3nvnhcLNpqtwzHnXifSa3iI2g7yTmjmF+kpLFypPg=
Received: from (2603:10b6:5:166::20) by DM6PR00MB0588.namprd00.prod.outlook.com (2603:10b6:5:16f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3629.0; Thu, 19 Nov 2020 17:36:49 +0000
Received: from DM6PR00MB0635.namprd00.prod.outlook.com ([fe80::b8ed:6d05:5b73:e9d]) by DM6PR00MB0635.namprd00.prod.outlook.com ([fe80::b8ed:6d05:5b73:e9d%5]) with mapi id 15.20.3633.000; Thu, 19 Nov 2020 17:36:49 +0000
From: Praveen Balasubramanian 
To: "mattmathis=40google.com@dmarc.ietf.org" , "gorry@erg.abdn.ac.uk" , "ianswett=40google.com@dmarc.ietf.org" 
CC: "tcpm@ietf.org" 
Thread-Topic: [EXTERNAL] Re: [tcpm] Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
Thread-Index: AQHWvkekOsRQcbriw0WyAJwy2+VO3qnPgcyAgAAEpYCAACJwAIAADcRw
Date: Thu, 19 Nov 2020 17:36:48 +0000
Message-ID: 
References:    <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk> 
In-Reply-To: 
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=e7e1453b-49de-4ade-a47b-261499faf5a8; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-11-19T17:29:57Z;  MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: dmarc.ietf.org; dkim=none (message not signed) header.d=none;dmarc.ietf.org; dmarc=none action=none header.from=microsoft.com;
x-originating-ip: [2001:4898:80e8:0:95d8:3f73:f682:d302]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: 706b8cc1-e512-470d-b9b0-08d88cb1b1b1
x-ms-traffictypediagnostic: DM6PR00MB0588:
x-microsoft-antispam-prvs: 
x-ms-oob-tlc-oobclassifiers: OLM:8273;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: Ieblx4DISyz+XmwjIDkW78IY9pV69hBpRzBhtDv2ryTO8VjLyn6A2YPW8qEZ0lXSOV1f8TE2p6TdxzLa8nP4U3tbmwGh7nvpPxXoQXhv42ev5ib/jFDfOoeyspV33tEkQWHPvzff96cXQcM6a6tcyZlMZd+Db2TZ01BIsvdlVIU6kd+DWzYkqXhnJjwWAbmKiuYX9A/ZB/YJ0YmRAXWrXgB9nNp+YAlcdt3DnaIaPf/cVMbe2iNa+mkkZsFV7SuzaYnDcoZpBQTGDKf6C6MvJBAVDdxsnrNvskgR1ilwUAIR7o/8z4A55pvT1nFPAfgKd4Ljf/XC0MNoXXA67UC2Pay14zIt6FnzxgBhIXkZary5xN8+CW1J2aesDbKBA6KpoMHoAGFJHrTArkehtQ36Aw==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;  IPV:NLI; SFV:NSPM; H:DM6PR00MB0635.namprd00.prod.outlook.com; PTR:; CAT:NONE;  SFS:(4636009)(346002)(136003)(376002)(39860400002)(366004)(396003)(966005)(66946007)(64756008)(2906002)(316002)(8676002)(6506007)(66446008)(66556008)(66476007)(33656002)(10290500003)(4326008)(71200400001)(8936002)(9686003)(52536014)(478600001)(7696005)(53546011)(86362001)(83380400001)(5660300002)(76116006)(166002)(186003)(55016002)(82960400001)(110136005)(8990500004)(82950400001); DIR:OUT; SFP:1102; 
x-ms-exchange-antispam-messagedata: =?utf-8?B?ZlFuSzFQeGJlNnU3bUErSGMyRThaY3VPU0tMdk9nL2JRRzZkMGwrMVhLWG0y?= =?utf-8?B?VW1JQ3A5OVFaN2t0Wkw4a1pRQkRrNDFIOXpOTHJmSlN1N3NyR1Q1OWRwbkZL?= =?utf-8?B?Ri9VdFdwTEtCQy8ydGJRNVBrRUo2WlhTNytpNGhUMldXWEU4NVg3bE1sWG5x?= =?utf-8?B?TjNHQVErZ3o2Y1lQbnZULzRnbUs3ZyswSW9tOTBXUVFqaVY2WlkzMzAwQUxY?= =?utf-8?B?c2tGU0NXV2Mvazk1Qi9SdnlPNXZ5UHp5RG1wMCs5ZFBtdTBTb0R0cDNrZGc1?= =?utf-8?B?aU50aWVwV3VBNk84dVQ5WWlCU3d5T2xXckJwUXJtSzFFcmhZZkxTNW9haUpB?= =?utf-8?B?b0RkakFieVFhMU04VUk5b08wcTFNUGVBTnh3NHJKaUhkS3lNdThQK2tyUk9n?= =?utf-8?B?dlJjWHlocjFvOGl2NkgydWZ1WHQ4TnF1dzNscElYdm5Tc1hhOE03L2o3eGZj?= =?utf-8?B?MFBtK0JmNmhmWkhWejlkNlNtSk0zanJkK0l6V3RqQUdMY3g2elNUWklpVFRz?= =?utf-8?B?UHBtMzEvL1VXSGZ0Q1Q1U2NOM2p5RHduRGJrUDROdFBIVUxRQVM0RzZxSnlr?= =?utf-8?B?V0dtMXdOcXNhQlBxamdHdVJnc2NneUIrZkhwWmN4UDROeTFoSjU2SGhPYjFE?= =?utf-8?B?c1g3Wkt6SU9UNFAvbmE3K29rNituZXRkd1JpU0xtM1p2Y1RRYzAvTUkvT1J5?= =?utf-8?B?TTVrendSSDZDV2FrZUw1R2tsTjljR1l6Um9jbkhaU0xVUVo0RTFpbTkwTmlD?= =?utf-8?B?SitJTUpRRHAxeGF0T1AveXN3Yk1lV1VrQjZVSWduM0FqM20wc0R0Vk5JMXpS?= =?utf-8?B?OUpGTTc5NHFYV2NIUndGOFUwaWhRL2JlWHBDSHg4NTE3d1ROeUZTTk1XaUxK?= =?utf-8?B?OGppa20yMnY0RWcxcXkwMlRxdFYrVTZXMWFHZ2tkMk4zc1NIcEpjbU9JWnFk?= =?utf-8?B?b2cxNzVweFpmM1ZQbXQ5dDhxRlh0bmI2aCtkNFlqV0lFWUdEcVRQODdVNEtB?= =?utf-8?B?LzlTamdRdUY0cFpGdjRJRTZUU0xJT1hFa1dQZFllVEU3dGdmRjBJRFk0NlNB?= =?utf-8?B?b3N2UXFDZzJ2Qnk3ci95ZjE2U2dCb3QvV2hNQkdtVXh0TFNPWGpCKzBWU1Rm?= =?utf-8?B?YVowd2IwbHI0RVA1S2s2d3loVWNMVjROWFo4VEsxanVJZjFHQmdzQXdaMzlY?= =?utf-8?B?b0luZDh0VEwrd3dnOThkaDV5ZTVreFhDMDRyc2dwL2l5SXdHQ21YTUdlNU8z?= =?utf-8?B?ZlpWRkMwS3R4L1hpajkrWlArRkdBMUl2b0tuVVc0aGJ5Unk2VHh1a0I1U2Ra?= =?utf-8?B?R2xObUxlcFlPY3YvTURhWnF4RXJYZ3pyQmY3SjV6QVpxNnF0c1g1WEZMT0tL?= =?utf-8?Q?7g28wuyaWZ4bqDkskjA6rP5Zc0qz9RU0=3D?=
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_DM6PR00MB06350BB2986E364D2D8DBBDDB6E09DM6PR00MB0635namp_"
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR00MB0635.namprd00.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 706b8cc1-e512-470d-b9b0-08d88cb1b1b1
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Nov 2020 17:36:48.9990 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RoCKt0zG/cHR3EuysTXJAQfmk5Dl/Z7ZW50bX5hXCWWYJjCgGze67xOLqlSsvutGZ4melOK9aQ3dAP/0ztxdW62sseaYtWzylC3XXLn5l7g=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR00MB0588
Archived-At: 
Subject: Re: [tcpm] [EXTERNAL] Re: Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 17:37:05 -0000

--_000_DM6PR00MB06350BB2986E364D2D8DBBDDB6E09DM6PR00MB0635namp_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

SSBhbSBzdXBwb3J0aXZlIG9mIHVwZGF0aW5nIFBSUiB0byBQUyBzdGF0dXMgYW5kIHdpbGwgcmV2
aWV3IHRoZSBkb2N1bWVudC4NCg0KUmUuIFFVSUMuIFRoZSBjdXJyZW50IGxvc3MgcmVjb3Zlcnkg
c3BlYyBmb3IgUVVJQyBhbHJlYWR5IHJlZmVycyB0byBQUlIgYXMgYSBNQVkgYW5kIGhhcyBzaW1p
bGFyIG90aGVyIHJlZmVyZW5jZXMgdG8gVENQIFJGQ3Mgd2l0aG91dCBhbnkgYWRkaXRpb25hbCB0
ZXh0IGZvciBRVUlDIHNwZWNpZmljIGFkYXB0YXRpb25zLiBUaGUgY3VycmVudCB0aXRsZSBvZiBS
RkMgNjkzNyBzY29wZXMgaXQgdG8gVENQOiBQcm9wb3J0aW9uYWwgUmF0ZSBSZWR1Y3Rpb24gZm9y
IFRDUC4gSSBhbHNvIHNlZSBhYm91dCAzOCBoaXRzIGZvciB0aGUgd29yZCBUQ1AgaW4gdGhlIHRl
eHQuDQoNCkZyb206IHRjcG0gPHRjcG0tYm91bmNlc0BpZXRmLm9yZz4gT24gQmVoYWxmIE9mIE1h
dHQgTWF0aGlzDQpTZW50OiBUaHVyc2RheSwgTm92ZW1iZXIgMTksIDIwMjAgODo0MSBBTQ0KVG86
IEdvcnJ5IEZhaXJodXJzdCA8Z29ycnlAZXJnLmFiZG4uYWMudWs+OyBJYW4gU3dldHQgPGlhbnN3
ZXR0PTQwZ29vZ2xlLmNvbUBkbWFyYy5pZXRmLm9yZz4NCkNjOiB0Y3BtQGlldGYub3JnIEV4dGVu
c2lvbnMgPHRjcG1AaWV0Zi5vcmc+DQpTdWJqZWN0OiBbRVhURVJOQUxdIFJlOiBbdGNwbV0gSW50
cm9kdWNlIFJGQyA2OTM3IGJpcyAoUHJvcG9ydGlvbmFsIFJhdGUgUmVkdWN0aW9uKSBhcyBhIHRj
cG0gd29yayBpdGVtPyBJTlBVVCBORUVERUQNCg0KSWYgd2UgY2FuIGdldCBhd2F5IHdpdGggYSBu
b24tbm9ybWF0aXZlIGRlc2NyaXB0aW9uIG9mIHRoZSBoZXVyaXN0aWMsIFFVSUMgaXMgbm8gcHJv
YmxlbSBhdCBhbGwuICAgIElmIHdlIG5lZWQgbm9ybWF0aXZlIGRlc2NyaXB0aW9ucywgdGhlbiBU
Q1AgYW5kIFFVSUMgaGF2ZSB0byBiZSBkaWZmZXJlbnQsIGFuZCB3ZSBuZWVkIHRvIGluY2x1ZGUg
bm9ybWF0aXZlIHJlZmVyZW5jZXMgdG8gYm90aCByZWNvdmVyeSBtYWNoaW5lcyBhbmQgcmVjb3Zl
cnkgc3RhdGUgdmFyaWFibGVzLCB3aGljaCBhcmUgbm90IGluIHNjb3BlIGZvciB0aGUgY3VycmVu
dCBkb2MuDQoNCkkgd291bGQgcHJlZmVyIHRvIHN0YXkgd2l0aCBhIG5vbi1ub3JtYXRpdmUgZGVz
Y3JpcHRpb24gb2YgdGhlIGhldXJpc3RpYywgYW5kIHNjcnViIHRoZSByZXN0IG9mIHRoZSBkb2N1
bWVudCBmb3IgbGFuZ3VhZ2UgdGhhdCBjb25mbGljdHMgd2l0aCBRVUlDLg0KDQpUaGFua3MsDQot
LU1NLS0NClRoZSBiZXN0IHdheSB0byBwcmVkaWN0IHRoZSBmdXR1cmUgaXMgdG8gY3JlYXRlIGl0
LiAgLSBBbGFuIEtheQ0KDQpXZSBtdXN0IG5vdCB0b2xlcmF0ZSBpbnRvbGVyYW5jZTsNCiAgICAg
ICBob3dldmVyIG91ciByZXNwb25zZSBtdXN0IGJlIGNhcmVmdWxseSBtZWFzdXJlZDoNCiAgICAg
ICAgICAgIHRvbyBzdHJvbmcgd291bGQgYmUgaHlwb2NyaXRpY2FsIGFuZCByaXNrcyBzcGlyYWxp
bmcgb3V0IG9mIGNvbnRyb2w7DQogICAgICAgICAgICB0b28gd2VhayByaXNrcyBiZWluZyBtaXN0
YWtlbiBmb3IgdGFjaXQgYXBwcm92YWwuDQoNCg0KT24gVGh1LCBOb3YgMTksIDIwMjAgYXQgNjoz
NyBBTSBHb3JyeSBGYWlyaHVyc3QgPGdvcnJ5QGVyZy5hYmRuLmFjLnVrPG1haWx0bzpnb3JyeUBl
cmcuYWJkbi5hYy51az4+IHdyb3RlOg0KT24gMTkvMTEvMjAyMCAxNDoyMCwgSWFuIFN3ZXR0IHdy
b3RlOg0KSSBzdXBwb3J0IHRoaXMgd29yay4NCg0KT25lIHF1ZXN0aW9uIGZvciBhdXRob3JzIGFu
ZCB0aGUgZ3JvdXAuICBUaGlzIHNwZWNpZmljYXRpb24gY291bGQgYmUgYXBwbGllZCB0byBRVUlD
LCBidXQgbXkgbWVtb3J5IGlzIHRoYXQgdGhlcmUgYXJlIHNvbWUgc21hbGwgY2hhbmdlcyB0byBh
ZGFwdCBpdCBjb3JyZWN0bHkuICBXb3VsZCBub3RpbmcgdGhvc2UgY2hhbmdlcyBiZSBpbiBzY29w
ZSBmb3IgdGhpcyBkb2N1bWVudCwgYSBzZXBhcmF0ZSBkb2N1bWVudCBpbiBRVUlDLCBvciBhbiBl
eGVyY2lzZSBsZWZ0IHRvIHRoZSByZWFkZXI/ICBJJ2QgYmUgaGFwcHkgdG8gY29udHJpYnV0ZSBh
bnkgUVVJQyBzcGVjaWZpYyB0ZXh0IGlmIHRoYXQncyBoZWxwZnVsLg0KDQpUaGFua3MsIElhbg0K
DQpJJ2QgaG9wZSB0aGUgZGlmZmVyZW5jZXMgZm9yIFFVSUMgQVJFIGNhcHR1cmVkIGluIHRoZSBS
RkMgc2VyaWVzLiBJZiB0aGV5IGFyZSBzbWFsbCwgaXQgbWlnaHQgYmUgcG9zc2libGUgdG8gYWRk
IHRvIHRoaXMgZHJhZnQgcGVyaGFwcz8NCg0KR29ycnkNCg0KT24gVGh1LCBOb3YgMTksIDIwMjAg
YXQgMjo0MyBBTSBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkIDxycy5pZXRmQGdteC5hdDxtYWlsdG86
cnMuaWV0ZkBnbXguYXQ+PiB3cm90ZToNCkFzIHRoZSBoZXVyaXN0aWMgaXMgdGhlIG9uZSBtYWpv
ciBpbXByb3ZlbWVudCBvdmVyIFJGQzY5MzcsIHRoYXQgaXMgdGhlDQpvbmUgYXJlYSBJIGFtIGVz
cGVjaWFsbHkgaW50ZXJlc3RlZCBpbi4gQ3VycmVudGx5IHdvcmtpbmcgdG8gaGF2ZSBQUlIgaW4N
CkZyZWVCU0QsIGl0IHdvdWxkIGJlIHRoZSBiZXN0IHRpbWUgd2luZG93IHRvIGFkZCB0aGlzIGhl
dXJpc3RpZyBpbnN0ZWFkDQpvZiBzd2l0Y2hpbmcgYmV0d2VlbiB0aGUgdHdvIHZhcmlhbnRzIGlu
IGEgc3RhdGljIGZhc2hpb24uDQoNCkkgc3VwcG9ydCB0aGlzIGFuZCB3aWxsIGNlcnRhaW5seSBi
ZSByZXZpZXdpbmcgdGhlIGRvY3VtZW50cyBhbmQgcHJvdmlkZQ0KZmVlZGJhY2shDQoNClJpY2hh
cmQNCg0KDQpBbSAxOS4xMS4yMDIwIHVtIDA0OjU2IHNjaHJpZWIgTWF0dCBNYXRoaXM6DQo+IFRo
ZSBhdXRob3JzIG9mIFBSUiB3b3VsZCBsaWtlIHRvIHVwZGF0ZSBQUlIgdG8gUHJvcG9zZWQgU3Rh
bmRhcmQNCj4gc3RhdHVzLiAgVGhpcyBlbnRhaWxzIGludHJvZHVjaW5nIGEgbmV3IGRvY3VtZW50
IGFzIGFuIHRjcG0gd29yayBpdGVtLg0KPg0KPiAqUGxlYXNlIGluZGljYXRlIChub24pIHN1cHBv
cnQgYW5kL29yIGNvbW1lbnQuKg0KPg0KPiBGb3IgbW9yZSBkZXRhaWxzIHNlZSB0aGUgdGNwbSBt
ZWV0aW5nIG1hdGVyaWFscyBmcm9tIElFVEYgMTA5DQo+IG1pbnV0ZXM6DQo+IGh0dHBzOi8vZGF0
YXRyYWNrZXIuaWV0Zi5vcmcvbWVldGluZy8xMDkvbWF0ZXJpYWxzL21pbnV0ZXMtMTA5LXRjcG0t
MDA8aHR0cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNvbS8/dXJsPWh0
dHBzJTNBJTJGJTJGZGF0YXRyYWNrZXIuaWV0Zi5vcmclMkZtZWV0aW5nJTJGMTA5JTJGbWF0ZXJp
YWxzJTJGbWludXRlcy0xMDktdGNwbS0wMCZkYXRhPTA0JTdDMDElN0NwcmF2YiU0MG1pY3Jvc29m
dC5jb20lN0NjOTUyNTRiY2JhMGY0MWMxNGYzNjA4ZDg4Y2E5ZWFjNSU3QzcyZjk4OGJmODZmMTQx
YWY5MWFiMmQ3Y2QwMTFkYjQ3JTdDMSU3QzAlN0M2Mzc0MTQwMDg3MDUzMDM0ODYlN0NVbmtub3du
JTdDVFdGcGJHWnNiM2Q4ZXlKV0lqb2lNQzR3TGpBd01EQWlMQ0pRSWpvaVYybHVNeklpTENKQlRp
STZJazFoYVd3aUxDSlhWQ0k2TW4wJTNEJTdDMTAwMCZzZGF0YT1Hb3RNNEFoakRqcWJSczZva3VN
RWVBUXhRQlVtbXVOYUZoSTdnckJ0UXU0JTNEJnJlc2VydmVkPTA+DQo+IHNsaWRlczogaHR0cHM6
Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0LW1hdGhpcy10Y3BtLXJmYzY5MzdiaXMtMDA8aHR0
cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNvbS8/dXJsPWh0dHBzJTNB
JTJGJTJGdG9vbHMuaWV0Zi5vcmclMkZodG1sJTJGZHJhZnQtbWF0aGlzLXRjcG0tcmZjNjkzN2Jp
cy0wMCZkYXRhPTA0JTdDMDElN0NwcmF2YiU0MG1pY3Jvc29mdC5jb20lN0NjOTUyNTRiY2JhMGY0
MWMxNGYzNjA4ZDg4Y2E5ZWFjNSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JTdD
MSU3QzAlN0M2Mzc0MTQwMDg3MDUzMDM0ODYlN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlKV0lq
b2lNQzR3TGpBd01EQWlMQ0pRSWpvaVYybHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2TW4w
JTNEJTdDMTAwMCZzZGF0YT0xSTh3aXhFbUtybmclMkZxREF6WWFqUXB2ODVkYWdJbGVSZHZxQ1N1
cG9hVVElM0QmcmVzZXJ2ZWQ9MD4NCj4NCj4gVGhlcmUgd2VyZSBhYm91dCBmb3VyICJJIHN1cHBv
cnQgdGhpcyB3b3JrIiByZW1hcmtzIGF0IHRoZSBtaWMgKG5vdA0KPiByZWNvcmRlZCBpbiB0aGUg
bWludXRlcyksIGFuZCBhYm91dCBhcyBtYW55IGluIHRoZSBNZWV0ZWNobyBjaGF0Lg0KPg0KPiBB
YnJpZGdlZCBJRVRGL3RjcG0vUFJSYmlzIHNsaWRlczoNCj4gLS0NCj4gUFJSICByZWNhcCAoUkZD
NjkzNyBleHBlcmltZW50YWwpDQo+IFBSUiBpcyBhIHNwZWNpYWwgY29uZ2VzdGlvbiBjb250cm9s
IGVmZmVjdGl2ZSBvbmx5IGR1cmluZyBmYXN0IHJlY292ZXJ5DQo+DQo+ICAgKiBXaGVuIGluZmxp
Z2h0ID49IHNzdGhyZXNoLCBzZW5kIGF0IGxvc3NfYmV0YSpyYXRlX2JlZm9yZV9sb3NzIChlLmcu
DQo+ICAgICBsb3NzX2JldGEgPSAwLjUgZm9yIFJlbm8gKGFrYSByYXRlLWhhbHZpbmcpLCAwLjcg
Zm9yIEN1YmljKQ0KPiAgICogV2hlbiBpbmZsaWdodCA8IHNzdGhyZXNoLCBzZW5kIGF0IHRoZSBz
YW1lIG9yIHR3aWNlIHRoZQ0KPiAgICAgZGVsaXZlcnlfcmF0ZSAobW9yZSBsYXRlcikNCj4gICAq
IFVzZWQgYnkgYWxsIGNvbmdlc3Rpb24gY29udHJvbCBtb2R1bGVzIGluIExpbnV4IGR1cmluZyBm
YXN0IHJlY292ZXJ5DQo+ICAgICAgIG8gQ2FuIGJlIG1vcmUgZG9taW5hbnQgdGhhbiB0aGUgYWN0
dWFsIEMuQy4gZm9yIGxvc3N5IGZsb3dzDQo+ICAgICAgICAgdGhhdOKAmXJlIGluIGZhc3QgcmVj
b3ZlcnkgY29uc3RhbnRseSAoZS5nLiB2aWRlbyBzdHJlYW1pbmcNCj4gICAgICAgICB0aHJvdWdo
IHBvbGljZXJzKQ0KPg0KPiAtLQ0KPiBDdXJyZW50IFN0YXR1cw0KPg0KPiAgICoNCj4NCj4gICAg
IFBSUiBpcyB3aWRlbHkgZGVwbG95ZWQNCj4NCj4gICAgICAgbw0KPg0KPiAgICAgICAgIEF0IGxl
YXN0IHRocmVlIG1ham9yIE9TczogTGludXgsIFdpbmRvd3MsIChOZXRGbGl4KSBCU0QNCj4NCj4g
ICAgICAgbw0KPg0KPiAgICAgICAgIFZhc3QgbWFqb3JpdHkgb2YgV2ViIHRyYWZmaWMgZm9yIHll
YXJzDQo+DQo+ICAgKg0KPg0KPiAgICAgTm8gY2hhbmdlcyB0byBhbGdvcml0aG1zIHB1Ymxpc2hl
ZCBpbiBSRkMgNjkzNw0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgUFJSLUNSQiAtIENvbnNl
cnZhdGl2ZSBSZWR1Y3Rpb24gQm91bmQgLSBzdHJpY3QgcGFja2V0DQo+ICAgICAgICAgY29udmVy
c2lvbiBkdXJpbmcgbG9zcyByZWNvdmVyeQ0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgUFJS
LVNTUkIgLSBTbG93c3RhcnQgUmVkdWN0aW9uIEJvdW5kIC0gb25lIGV4dHJhIHNlZ21lbnQgcGVy
IEFDSw0KPiAgICAgICAgIGR1cmluZyBsb3NzIHJlY292ZXJ5DQo+DQo+ICAgKg0KPg0KPiAgICAg
MjAxNSBIZXVyaXN0aWMgdG8gZHluYW1pY2FsbHkgc2VsZWN0IHdoaWNoIHJlZHVjdGlvbiBib3Vu
ZA0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgT25seSB1c2UgUFJSLVNTUkIgd2hlbiBtYWtp
bmcgZ29vZCBmb3J3YXJkIHByb2dyZXNzDQo+DQo+ICAgICAgICAgICArDQo+DQo+ICAgICAgICAg
ICAgIEFDS3MgdGhhdCBhZHZhbmNlZCBzbmQudW5hIGFuZCByZXBvcnQgbm8gbmV3IGxvc3Nlcw0K
Pg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgUmVzb2x2ZXMgc29tZSBwYXRob2xvZ2ljYWwgY2Fz
ZXMgd2l0aCB0b2tlbiBidWNrZXQgcG9saWNlcnMNCj4NCj4gICAgICAgICAgICsNCj4NCj4gICAg
ICAgICAgICAgQ0MgZXN0aW1hdGVzIHNzdGhyZXNoIGJlZm9yZSBpdCBjYW4gcG9zc2libHkgbWVh
c3VyZSB0aGUNCj4gICAgICAgICAgICAgdG9rZW4gcmF0ZQ0KPg0KPiAgICAgICAgICAgKw0KPg0K
PiAgICAgICAgICAgICBUaGUgaGV1cmlzdGljIG1ha2VzIHRoZSBiZXN0IG9mIGEgYmFkIHNpdHVh
dGlvbg0KPg0KPiAtLQ0KPiBUZW50YXRpdmUgcGF0aCBmb3J3YXJkDQo+DQo+ICAgKg0KPg0KPiAg
ICAgQWRvcHQgYXMgYSB0Y3BtIHdvcmsgaXRlbQ0KPg0KPiAgICoNCj4NCj4gICAgIFVwZGF0ZSB0
aGUgdGV4dA0KPg0KPiAgICAgICBvDQo+DQo+ICAgICAgICAgTm9ybWF0aXZlIFJGQyAyMTE5IGxh
bmd1YWdlDQo+DQo+ICAgICAgIG8NCj4NCj4gICAgICAgICBBZGQgTUFZIHVzZSB0aGUgaGV1cmlz
dGljLi4uDQo+DQo+ICAgICAgIG8NCj4NCj4gICAgICAgICBUcmltIHJlZHVuZGFudCBhbmQgb2Jz
b2xldGUgbGFuZ3VhZ2UNCj4NCj4gICAgICAgICAgICsNCj4NCj4gICAgICAgICAgICAgUkZDIDY5
MzcgcmVwZWF0cyBpdHNlbGYgYW5kIGlzIG11Y2ggbG9uZ2VyIHRoYW4gbmVjZXNzYXJ5DQo+DQo+
ICAgICAgICAgICArDQo+DQo+ICAgICAgICAgICAgIEZvY3VzIG9uIHdoYXQgYW4gaW1wbGVtZW50
ZXIgbmVlZHMgdG8ga25vdw0KPg0KPiAgICAgICAgICAgKw0KPg0KPiAgICAgICAgICAgICBVc2Ug
bm9uLW5vcm1hdGl2ZSByZWZlcmVuY2VzIHRvIFJGQyA2OTM3IGZvciBwcmlvcg0KPiAgICAgICAg
ICAgICBtZWFzdXJlbWVudCB3b3JrLCBldGMNCj4NCj4gVGhhbmtzLA0KPiAtLU1NLS0NCj4gVGhl
IGJlc3Qgd2F5IHRvIHByZWRpY3QgdGhlIGZ1dHVyZSBpcyB0byBjcmVhdGUgaXQuICAtIEFsYW4g
S2F5DQo+DQo+IFdlIG11c3Qgbm90IHRvbGVyYXRlIGludG9sZXJhbmNlOw0KPiAgICAgICAgIGhv
d2V2ZXIgb3VyIHJlc3BvbnNlIG11c3QgYmUgY2FyZWZ1bGx5IG1lYXN1cmVkOg0KPiAgICAgICAg
ICAgICAgdG9vIHN0cm9uZyB3b3VsZCBiZSBoeXBvY3JpdGljYWwgYW5kIHJpc2tzIHNwaXJhbGlu
ZyBvdXQgb2YNCj4gY29udHJvbDsNCj4gICAgICAgICAgICAgIHRvbyB3ZWFrIHJpc2tzIGJlaW5n
IG1pc3Rha2VuIGZvciB0YWNpdCBhcHByb3ZhbC4NCj4NCj4gX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gdGNwbSBtYWlsaW5nIGxpc3QNCj4gdGNwbUBp
ZXRmLm9yZzxtYWlsdG86dGNwbUBpZXRmLm9yZz4NCj4gaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFp
bG1hbi9saXN0aW5mby90Y3BtPGh0dHBzOi8vbmFtMDYuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0
bG9vay5jb20vP3VybD1odHRwcyUzQSUyRiUyRnd3dy5pZXRmLm9yZyUyRm1haWxtYW4lMkZsaXN0
aW5mbyUyRnRjcG0mZGF0YT0wNCU3QzAxJTdDcHJhdmIlNDBtaWNyb3NvZnQuY29tJTdDYzk1MjU0
YmNiYTBmNDFjMTRmMzYwOGQ4OGNhOWVhYzUlN0M3MmY5ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDEx
ZGI0NyU3QzElN0MwJTdDNjM3NDE0MDA4NzA1MzEzNDQwJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNk
OGV5SldJam9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pY
VkNJNk1uMCUzRCU3QzEwMDAmc2RhdGE9SlpzTUJGM1ZlaFV2SFNKZmdnUFliUGNvMzRoOVVpVEcl
MkJLYnlYVTFCeiUyQkElM0QmcmVzZXJ2ZWQ9MD4NCj4NCg0KX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX18NCnRjcG0gbWFpbGluZyBsaXN0DQp0Y3BtQGlldGYu
b3JnPG1haWx0bzp0Y3BtQGlldGYub3JnPg0KaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9s
aXN0aW5mby90Y3BtPGh0dHBzOi8vbmFtMDYuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5j
b20vP3VybD1odHRwcyUzQSUyRiUyRnd3dy5pZXRmLm9yZyUyRm1haWxtYW4lMkZsaXN0aW5mbyUy
RnRjcG0mZGF0YT0wNCU3QzAxJTdDcHJhdmIlNDBtaWNyb3NvZnQuY29tJTdDYzk1MjU0YmNiYTBm
NDFjMTRmMzYwOGQ4OGNhOWVhYzUlN0M3MmY5ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyU3
QzElN0MwJTdDNjM3NDE0MDA4NzA1MzEzNDQwJTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJ
am9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1u
MCUzRCU3QzEwMDAmc2RhdGE9SlpzTUJGM1ZlaFV2SFNKZmdnUFliUGNvMzRoOVVpVEclMkJLYnlY
VTFCeiUyQkElM0QmcmVzZXJ2ZWQ9MD4NCg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fXw0KDQp0Y3BtIG1haWxpbmcgbGlzdA0KDQp0Y3BtQGlldGYub3Jn
PG1haWx0bzp0Y3BtQGlldGYub3JnPg0KDQpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xp
c3RpbmZvL3RjcG08aHR0cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlvbi5vdXRsb29rLmNv
bS8/dXJsPWh0dHBzJTNBJTJGJTJGd3d3LmlldGYub3JnJTJGbWFpbG1hbiUyRmxpc3RpbmZvJTJG
dGNwbSZkYXRhPTA0JTdDMDElN0NwcmF2YiU0MG1pY3Jvc29mdC5jb20lN0NjOTUyNTRiY2JhMGY0
MWMxNGYzNjA4ZDg4Y2E5ZWFjNSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3JTdD
MSU3QzAlN0M2Mzc0MTQwMDg3MDUzMTM0NDAlN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlKV0lq
b2lNQzR3TGpBd01EQWlMQ0pRSWpvaVYybHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2TW4w
JTNEJTdDMTAwMCZzZGF0YT1KWnNNQkYzVmVoVXZIU0pmZ2dQWWJQY28zNGg5VWlURyUyQktieVhV
MUJ6JTJCQSUzRCZyZXNlcnZlZD0wPg0KDQoNCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fDQp0Y3BtIG1haWxpbmcgbGlzdA0KdGNwbUBpZXRmLm9yZzxtYWls
dG86dGNwbUBpZXRmLm9yZz4NCmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8v
dGNwbTxodHRwczovL25hbTA2LnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29tLz91cmw9
aHR0cHMlM0ElMkYlMkZ3d3cuaWV0Zi5vcmclMkZtYWlsbWFuJTJGbGlzdGluZm8lMkZ0Y3BtJmRh
dGE9MDQlN0MwMSU3Q3ByYXZiJTQwbWljcm9zb2Z0LmNvbSU3Q2M5NTI1NGJjYmEwZjQxYzE0ZjM2
MDhkODhjYTllYWM1JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCU3
QzYzNzQxNDAwODcwNTMyMzM5NyU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdM
akF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0Mx
MDAwJnNkYXRhPUpYbEJseXhETUIycVFVbHglMkJZNDRiN2ZaZmdjajhIYURjRUFrZkF2N0FBSSUz
RCZyZXNlcnZlZD0wPg0K

--_000_DM6PR00MB06350BB2986E364D2D8DBBDDB6E09DM6PR00MB0635namp_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
IkNhbWJyaWEgTWF0aCI7DQoJcGFub3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9DQpAZm9udC1m
YWNlDQoJe2ZvbnQtZmFtaWx5OkNhbGlicmk7DQoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAy
IDQ7fQ0KQGZvbnQtZmFjZQ0KCXtmb250LWZhbWlseTpDb25zb2xhczsNCglwYW5vc2UtMToyIDEx
IDYgOSAyIDIgNCAzIDIgNDt9DQovKiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWws
IGxpLk1zb05vcm1hbCwgZGl2Lk1zb05vcm1hbA0KCXttYXJnaW46MGluOw0KCWZvbnQtc2l6ZTox
MS4wcHQ7DQoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2VyaWY7fQ0KYTpsaW5rLCBzcGFu
Lk1zb0h5cGVybGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6Ymx1ZTsNCgl0
ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCnByZQ0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7
DQoJbXNvLXN0eWxlLWxpbms6IkhUTUwgUHJlZm9ybWF0dGVkIENoYXIiOw0KCW1hcmdpbjowaW47
DQoJZm9udC1zaXplOjEwLjBwdDsNCglmb250LWZhbWlseToiQ291cmllciBOZXciO30NCnNwYW4u
SFRNTFByZWZvcm1hdHRlZENoYXINCgl7bXNvLXN0eWxlLW5hbWU6IkhUTUwgUHJlZm9ybWF0dGVk
IENoYXIiOw0KCW1zby1zdHlsZS1wcmlvcml0eTo5OTsNCgltc28tc3R5bGUtbGluazoiSFRNTCBQ
cmVmb3JtYXR0ZWQiOw0KCWZvbnQtZmFtaWx5OkNvbnNvbGFzO30NCnNwYW4uRW1haWxTdHlsZTIx
DQoJe21zby1zdHlsZS10eXBlOnBlcnNvbmFsLXJlcGx5Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJp
IixzYW5zLXNlcmlmOw0KCWNvbG9yOndpbmRvd3RleHQ7fQ0KLk1zb0NocERlZmF1bHQNCgl7bXNv
LXN0eWxlLXR5cGU6ZXhwb3J0LW9ubHk7DQoJZm9udC1mYW1pbHk6IkNhbGlicmkiLHNhbnMtc2Vy
aWY7fQ0KQHBhZ2UgV29yZFNlY3Rpb24xDQoJe3NpemU6OC41aW4gMTEuMGluOw0KCW1hcmdpbjox
LjBpbiAxLjBpbiAxLjBpbiAxLjBpbjt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNl
Y3Rpb24xO30NCi0tPjwvc3R5bGU+PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWRl
ZmF1bHRzIHY6ZXh0PSJlZGl0IiBzcGlkbWF4PSIxMDI2IiAvPg0KPC94bWw+PCFbZW5kaWZdLS0+
PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWxheW91dCB2OmV4dD0iZWRpdCI+DQo8
bzppZG1hcCB2OmV4dD0iZWRpdCIgZGF0YT0iMSIgLz4NCjwvbzpzaGFwZWxheW91dD48L3htbD48
IVtlbmRpZl0tLT4NCjwvaGVhZD4NCjxib2R5IGxhbmc9IkVOLVVTIiBsaW5rPSJibHVlIiB2bGlu
az0icHVycGxlIiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPg0KPGRpdiBjbGFzcz0iV29y
ZFNlY3Rpb24xIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPkkgYW0gc3VwcG9ydGl2ZSBvZiB1cGRh
dGluZyBQUlIgdG8gUFMgc3RhdHVzIGFuZCB3aWxsIHJldmlldyB0aGUgZG9jdW1lbnQuPG86cD48
L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjxw
IGNsYXNzPSJNc29Ob3JtYWwiPlJlLiBRVUlDLiBUaGUgY3VycmVudCBsb3NzIHJlY292ZXJ5IHNw
ZWMgZm9yIFFVSUMgYWxyZWFkeSByZWZlcnMgdG8gUFJSIGFzIGEgTUFZIGFuZCBoYXMgc2ltaWxh
ciBvdGhlciByZWZlcmVuY2VzIHRvIFRDUCBSRkNzIHdpdGhvdXQgYW55IGFkZGl0aW9uYWwgdGV4
dCBmb3IgUVVJQyBzcGVjaWZpYyBhZGFwdGF0aW9ucy4gVGhlIGN1cnJlbnQgdGl0bGUgb2YgUkZD
IDY5Mzcgc2NvcGVzIGl0IHRvIFRDUDogUHJvcG9ydGlvbmFsDQogUmF0ZSBSZWR1Y3Rpb24gZm9y
IFRDUC4gSSBhbHNvIHNlZSBhYm91dCAzOCBoaXRzIGZvciB0aGUgd29yZCBUQ1AgaW4gdGhlIHRl
eHQuIDxvOnA+DQo8L286cD48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwv
bzpwPjwvcD4NCjxkaXYgc3R5bGU9ImJvcmRlcjpub25lO2JvcmRlci10b3A6c29saWQgI0UxRTFF
MSAxLjBwdDtwYWRkaW5nOjMuMHB0IDBpbiAwaW4gMGluIj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwi
PjxiPkZyb206PC9iPiB0Y3BtICZsdDt0Y3BtLWJvdW5jZXNAaWV0Zi5vcmcmZ3Q7IDxiPk9uIEJl
aGFsZiBPZiA8L2I+DQpNYXR0IE1hdGhpczxicj4NCjxiPlNlbnQ6PC9iPiBUaHVyc2RheSwgTm92
ZW1iZXIgMTksIDIwMjAgODo0MSBBTTxicj4NCjxiPlRvOjwvYj4gR29ycnkgRmFpcmh1cnN0ICZs
dDtnb3JyeUBlcmcuYWJkbi5hYy51ayZndDs7IElhbiBTd2V0dCAmbHQ7aWFuc3dldHQ9NDBnb29n
bGUuY29tQGRtYXJjLmlldGYub3JnJmd0Ozxicj4NCjxiPkNjOjwvYj4gdGNwbUBpZXRmLm9yZyBF
eHRlbnNpb25zICZsdDt0Y3BtQGlldGYub3JnJmd0Ozxicj4NCjxiPlN1YmplY3Q6PC9iPiBbRVhU
RVJOQUxdIFJlOiBbdGNwbV0gSW50cm9kdWNlIFJGQyA2OTM3IGJpcyAoUHJvcG9ydGlvbmFsIFJh
dGUgUmVkdWN0aW9uKSBhcyBhIHRjcG0gd29yayBpdGVtPyBJTlBVVCBORUVERUQ8bzpwPjwvbzpw
PjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+
DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+SWYgd2UgY2FuIGdldCBhd2F5IHdpdGggYSBu
b24tbm9ybWF0aXZlIGRlc2NyaXB0aW9uIG9mIHRoZSBoZXVyaXN0aWMsIFFVSUMgaXMgbm8gcHJv
YmxlbSBhdCBhbGwuJm5ic3A7ICZuYnNwOyBJZiB3ZSBuZWVkIG5vcm1hdGl2ZSBkZXNjcmlwdGlv
bnMsIHRoZW4gVENQIGFuZCBRVUlDIGhhdmUgdG8gYmUgZGlmZmVyZW50LCBhbmQgd2UgbmVlZCB0
byBpbmNsdWRlIG5vcm1hdGl2ZSByZWZlcmVuY2VzIHRvIGJvdGggcmVjb3ZlcnkNCiBtYWNoaW5l
cyBhbmQgcmVjb3Zlcnkgc3RhdGUmbmJzcDt2YXJpYWJsZXMsIHdoaWNoIGFyZSBub3QgaW4gc2Nv
cGUgZm9yIHRoZSBjdXJyZW50IGRvYy48bzpwPjwvbzpwPjwvcD4NCjxkaXY+DQo8cCBjbGFzcz0i
TXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNz
PSJNc29Ob3JtYWwiPkkgd291bGQgcHJlZmVyIHRvIHN0YXkgd2l0aCBhIG5vbi1ub3JtYXRpdmUg
ZGVzY3JpcHRpb24gb2YgdGhlIGhldXJpc3RpYywgYW5kIHNjcnViIHRoZSByZXN0IG9mIHRoZSBk
b2N1bWVudCBmb3IgbGFuZ3VhZ2UgdGhhdCBjb25mbGljdHMgd2l0aCBRVUlDLjxvOnA+PC9vOnA+
PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGJyIGNsZWFyPSJhbGwi
Pg0KPG86cD48L286cD48L3A+DQo8ZGl2Pg0KPGRpdj4NCjxkaXY+DQo8ZGl2Pg0KPGRpdj4NCjxk
aXY+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPlRoYW5rcyw8bzpwPjwvbzpw
PjwvcD4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+LS1NTS0tPGJyPg0KVGhlIGJlc3Qg
d2F5IHRvIHByZWRpY3QgdGhlIGZ1dHVyZSBpcyB0byBjcmVhdGUgaXQuICZuYnNwOy0gQWxhbiBL
YXk8YnI+DQo8YnI+DQpXZSBtdXN0IG5vdCB0b2xlcmF0ZSBpbnRvbGVyYW5jZTs8bzpwPjwvbzpw
PjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZuYnNwOyAmbmJzcDsg
Jm5ic3A7ICZuYnNwO2hvd2V2ZXIgb3VyIHJlc3BvbnNlIG11c3QgYmUgY2FyZWZ1bGx5IG1lYXN1
cmVkOiZuYnNwOzxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgdG9vIHN0cm9u
ZyB3b3VsZCBiZSBoeXBvY3JpdGljYWwgYW5kIHJpc2tzIHNwaXJhbGluZyBvdXQgb2YgY29udHJv
bDs8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPiZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IHRvbyB3ZWFrIHJpc2tzIGJl
aW5nIG1pc3Rha2VuIGZvciB0YWNpdCBhcHByb3ZhbC48bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0K
PC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xhc3M9
Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNs
YXNzPSJNc29Ob3JtYWwiPk9uIFRodSwgTm92IDE5LCAyMDIwIGF0IDY6MzcgQU0gR29ycnkgRmFp
cmh1cnN0ICZsdDs8YSBocmVmPSJtYWlsdG86Z29ycnlAZXJnLmFiZG4uYWMudWsiPmdvcnJ5QGVy
Zy5hYmRuLmFjLnVrPC9hPiZndDsgd3JvdGU6PG86cD48L286cD48L3A+DQo8L2Rpdj4NCjxibG9j
a3F1b3RlIHN0eWxlPSJib3JkZXI6bm9uZTtib3JkZXItbGVmdDpzb2xpZCAjQ0NDQ0NDIDEuMHB0
O3BhZGRpbmc6MGluIDBpbiAwaW4gNi4wcHQ7bWFyZ2luLWxlZnQ6NC44cHQ7bWFyZ2luLXJpZ2h0
OjBpbiI+DQo8ZGl2Pg0KPGRpdj4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPk9uIDE5LzExLzIwMjAg
MTQ6MjAsIElhbiBTd2V0dCB3cm90ZTo8bzpwPjwvbzpwPjwvcD4NCjwvZGl2Pg0KPGJsb2NrcXVv
dGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFyZ2luLWJvdHRvbTo1LjBwdCI+DQo8ZGl2Pg0K
PHAgY2xhc3M9Ik1zb05vcm1hbCI+SSBzdXBwb3J0IHRoaXMgd29yay4gPG86cD48L286cD48L3A+
DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rp
dj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj5PbmUgcXVlc3Rpb24gZm9yIGF1dGhvcnMg
YW5kIHRoZSBncm91cC4mbmJzcDsgVGhpcyBzcGVjaWZpY2F0aW9uJm5ic3A7Y291bGQgYmUgYXBw
bGllZCB0byBRVUlDLCBidXQgbXkgbWVtb3J5IGlzIHRoYXQgdGhlcmUgYXJlIHNvbWUgc21hbGwg
Y2hhbmdlcyB0byBhZGFwdCBpdCBjb3JyZWN0bHkuJm5ic3A7IFdvdWxkIG5vdGluZyB0aG9zZSBj
aGFuZ2VzIGJlIGluIHNjb3BlIGZvciB0aGlzIGRvY3VtZW50LCBhIHNlcGFyYXRlIGRvY3VtZW50
DQogaW4gUVVJQywgb3IgYW4gZXhlcmNpc2UgbGVmdCB0byB0aGUgcmVhZGVyPyZuYnNwOyBJJ2Qg
YmUgaGFwcHkgdG8gY29udHJpYnV0ZSBhbnkgUVVJQyBzcGVjaWZpYyB0ZXh0IGlmIHRoYXQncyBo
ZWxwZnVsLjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1h
bCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8L2Rpdj4NCjxkaXY+DQo8cCBjbGFzcz0iTXNvTm9y
bWFsIj5UaGFua3MsIElhbjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYmxvY2tx
dW90ZT4NCjxwPkknZCBob3BlIHRoZSBkaWZmZXJlbmNlcyBmb3IgUVVJQyBBUkUgY2FwdHVyZWQg
aW4gdGhlIFJGQyBzZXJpZXMuIElmIHRoZXkgYXJlIHNtYWxsLCBpdCBtaWdodCBiZSBwb3NzaWJs
ZSB0byBhZGQgdG8gdGhpcyBkcmFmdCBwZXJoYXBzPw0KPG86cD48L286cD48L3A+DQo8cD5Hb3Jy
eTxvOnA+PC9vOnA+PC9wPg0KPGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbi10b3A6NS4wcHQ7bWFy
Z2luLWJvdHRvbTo1LjBwdCI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpw
PjwvcD4NCjxkaXY+DQo8ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+T24gVGh1LCBOb3YgMTks
IDIwMjAgYXQgMjo0MyBBTSBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJkICZsdDs8YSBocmVmPSJtYWls
dG86cnMuaWV0ZkBnbXguYXQiIHRhcmdldD0iX2JsYW5rIj5ycy5pZXRmQGdteC5hdDwvYT4mZ3Q7
IHdyb3RlOjxvOnA+PC9vOnA+PC9wPg0KPC9kaXY+DQo8YmxvY2txdW90ZSBzdHlsZT0iYm9yZGVy
Om5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgI0NDQ0NDQyAxLjBwdDtwYWRkaW5nOjBpbiAwaW4gMGlu
IDYuMHB0O21hcmdpbi1sZWZ0OjQuOHB0O21hcmdpbi1yaWdodDowaW4iPg0KPHAgY2xhc3M9Ik1z
b05vcm1hbCI+QXMgdGhlIGhldXJpc3RpYyBpcyB0aGUgb25lIG1ham9yIGltcHJvdmVtZW50IG92
ZXIgUkZDNjkzNywgdGhhdCBpcyB0aGU8YnI+DQpvbmUgYXJlYSBJIGFtIGVzcGVjaWFsbHkgaW50
ZXJlc3RlZCBpbi4gQ3VycmVudGx5IHdvcmtpbmcgdG8gaGF2ZSBQUlIgaW48YnI+DQpGcmVlQlNE
LCBpdCB3b3VsZCBiZSB0aGUgYmVzdCB0aW1lIHdpbmRvdyB0byBhZGQgdGhpcyBoZXVyaXN0aWcg
aW5zdGVhZDxicj4NCm9mIHN3aXRjaGluZyBiZXR3ZWVuIHRoZSB0d28gdmFyaWFudHMgaW4gYSBz
dGF0aWMgZmFzaGlvbi48YnI+DQo8YnI+DQpJIHN1cHBvcnQgdGhpcyBhbmQgd2lsbCBjZXJ0YWlu
bHkgYmUgcmV2aWV3aW5nIHRoZSBkb2N1bWVudHMgYW5kIHByb3ZpZGU8YnI+DQpmZWVkYmFjayE8
YnI+DQo8YnI+DQpSaWNoYXJkPGJyPg0KPGJyPg0KPGJyPg0KQW0gMTkuMTEuMjAyMCB1bSAwNDo1
NiBzY2hyaWViIE1hdHQgTWF0aGlzOjxicj4NCiZndDsgVGhlIGF1dGhvcnMgb2YgUFJSIHdvdWxk
IGxpa2UgdG8gdXBkYXRlIFBSUiB0byBQcm9wb3NlZCBTdGFuZGFyZDxicj4NCiZndDsgc3RhdHVz
LiZuYnNwOyBUaGlzIGVudGFpbHMgaW50cm9kdWNpbmcgYSBuZXcgZG9jdW1lbnQgYXMgYW4gdGNw
bSB3b3JrIGl0ZW0uPGJyPg0KJmd0Ozxicj4NCiZndDsgKlBsZWFzZSBpbmRpY2F0ZSAobm9uKSBz
dXBwb3J0IGFuZC9vciBjb21tZW50Lio8YnI+DQomZ3Q7PGJyPg0KJmd0OyBGb3IgbW9yZSBkZXRh
aWxzIHNlZSB0aGUgdGNwbSBtZWV0aW5nIG1hdGVyaWFscyBmcm9tIElFVEYgMTA5PGJyPg0KJmd0
OyBtaW51dGVzOjxicj4NCiZndDsgPGEgaHJlZj0iaHR0cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJv
dGVjdGlvbi5vdXRsb29rLmNvbS8/dXJsPWh0dHBzJTNBJTJGJTJGZGF0YXRyYWNrZXIuaWV0Zi5v
cmclMkZtZWV0aW5nJTJGMTA5JTJGbWF0ZXJpYWxzJTJGbWludXRlcy0xMDktdGNwbS0wMCZhbXA7
ZGF0YT0wNCU3QzAxJTdDcHJhdmIlNDBtaWNyb3NvZnQuY29tJTdDYzk1MjU0YmNiYTBmNDFjMTRm
MzYwOGQ4OGNhOWVhYzUlN0M3MmY5ODhiZjg2ZjE0MWFmOTFhYjJkN2NkMDExZGI0NyU3QzElN0Mw
JTdDNjM3NDE0MDA4NzA1MzAzNDg2JTdDVW5rbm93biU3Q1RXRnBiR1pzYjNkOGV5SldJam9pTUM0
d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsxaGFXd2lMQ0pYVkNJNk1uMCUzRCU3
QzEwMDAmYW1wO3NkYXRhPUdvdE00QWhqRGpxYlJzNm9rdU1FZUFReFFCVW1tdU5hRmhJN2dyQnRR
dTQlM0QmYW1wO3Jlc2VydmVkPTAiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHBzOi8vZGF0YXRyYWNr
ZXIuaWV0Zi5vcmcvbWVldGluZy8xMDkvbWF0ZXJpYWxzL21pbnV0ZXMtMTA5LXRjcG0tMDA8L2E+
PGJyPg0KJmd0OyBzbGlkZXM6IDxhIGhyZWY9Imh0dHBzOi8vbmFtMDYuc2FmZWxpbmtzLnByb3Rl
Y3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwcyUzQSUyRiUyRnRvb2xzLmlldGYub3JnJTJGaHRt
bCUyRmRyYWZ0LW1hdGhpcy10Y3BtLXJmYzY5MzdiaXMtMDAmYW1wO2RhdGE9MDQlN0MwMSU3Q3By
YXZiJTQwbWljcm9zb2Z0LmNvbSU3Q2M5NTI1NGJjYmEwZjQxYzE0ZjM2MDhkODhjYTllYWM1JTdD
NzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCU3QzYzNzQxNDAwODcwNTMw
MzQ4NiU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaUxDSlFJam9p
VjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MxMDAwJmFtcDtzZGF0YT0x
STh3aXhFbUtybmclMkZxREF6WWFqUXB2ODVkYWdJbGVSZHZxQ1N1cG9hVVElM0QmYW1wO3Jlc2Vy
dmVkPTAiIHRhcmdldD0iX2JsYW5rIj4NCmh0dHBzOi8vdG9vbHMuaWV0Zi5vcmcvaHRtbC9kcmFm
dC1tYXRoaXMtdGNwbS1yZmM2OTM3YmlzLTAwPC9hPjxicj4NCiZndDs8YnI+DQomZ3Q7IFRoZXJl
IHdlcmUgYWJvdXQgZm91ciAmcXVvdDtJIHN1cHBvcnQgdGhpcyB3b3JrJnF1b3Q7IHJlbWFya3Mg
YXQgdGhlIG1pYyAobm90PGJyPg0KJmd0OyByZWNvcmRlZCBpbiB0aGUgbWludXRlcyksIGFuZCBh
Ym91dCBhcyBtYW55IGluIHRoZSBNZWV0ZWNobyBjaGF0Ljxicj4NCiZndDs8YnI+DQomZ3Q7IEFi
cmlkZ2VkIElFVEYvdGNwbS9QUlJiaXMgc2xpZGVzOjxicj4NCiZndDsgLS08YnI+DQomZ3Q7IFBS
UiAmbmJzcDtyZWNhcCAoUkZDNjkzNyBleHBlcmltZW50YWwpPGJyPg0KJmd0OyBQUlIgaXMgYSBz
cGVjaWFsIGNvbmdlc3Rpb24gY29udHJvbCBlZmZlY3RpdmUgb25seSBkdXJpbmcgZmFzdCByZWNv
dmVyeTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyogV2hlbiBpbmZsaWdodCAmZ3Q7
PSBzc3RocmVzaCwgc2VuZCBhdCBsb3NzX2JldGEqcmF0ZV9iZWZvcmVfbG9zcyAoZS5nLjxicj4N
CiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwO2xvc3NfYmV0YSA9IDAuNSBmb3IgUmVubyAoYWthIHJh
dGUtaGFsdmluZyksIDAuNyBmb3IgQ3ViaWMpPGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsqIFdoZW4g
aW5mbGlnaHQgJmx0OyBzc3RocmVzaCwgc2VuZCBhdCB0aGUgc2FtZSBvciB0d2ljZSB0aGU8YnI+
DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDtkZWxpdmVyeV9yYXRlIChtb3JlIGxhdGVyKTxicj4N
CiZndDsmbmJzcDsgJm5ic3A7KiBVc2VkIGJ5IGFsbCBjb25nZXN0aW9uIGNvbnRyb2wgbW9kdWxl
cyBpbiBMaW51eCBkdXJpbmcgZmFzdCByZWNvdmVyeTxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDtvIENhbiBiZSBtb3JlIGRvbWluYW50IHRoYW4gdGhlIGFjdHVhbCBDLkMuIGZv
ciBsb3NzeSBmbG93czxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
dGhhdOKAmXJlIGluIGZhc3QgcmVjb3ZlcnkgY29uc3RhbnRseSAoZS5nLiB2aWRlbyBzdHJlYW1p
bmc8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO3Rocm91Z2ggcG9s
aWNlcnMpPGJyPg0KJmd0Ozxicj4NCiZndDsgLS08YnI+DQomZ3Q7IEN1cnJlbnQgU3RhdHVzPGJy
Pg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7Kjxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7
ICZuYnNwOyAmbmJzcDtQUlIgaXMgd2lkZWx5IGRlcGxveWVkPGJyPg0KJmd0Ozxicj4NCiZndDsm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtvPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7QXQgbGVhc3QgdGhyZWUgbWFqb3IgT1NzOiBMaW51eCwg
V2luZG93cywgKE5ldEZsaXgpIEJTRDxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7bzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwO1Zhc3QgbWFqb3JpdHkgb2YgV2ViIHRyYWZmaWMgZm9yIHllYXJzPGJyPg0KJmd0
Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7Kjxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNw
OyAmbmJzcDtObyBjaGFuZ2VzIHRvIGFsZ29yaXRobXMgcHVibGlzaGVkIGluIFJGQyA2OTM3PGJy
Pg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtvPGJyPg0KJmd0Ozxi
cj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7UFJSLUNSQiAtIENvbnNl
cnZhdGl2ZSBSZWR1Y3Rpb24gQm91bmQgLSBzdHJpY3QgcGFja2V0PGJyPg0KJmd0OyZuYnNwOyAm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtjb252ZXJzaW9uIGR1cmluZyBsb3NzIHJlY292ZXJ5
PGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtvPGJyPg0KJmd0
Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7UFJSLVNTUkIgLSBT
bG93c3RhcnQgUmVkdWN0aW9uIEJvdW5kIC0gb25lIGV4dHJhIHNlZ21lbnQgcGVyIEFDSzxicj4N
CiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ZHVyaW5nIGxvc3MgcmVjb3Zl
cnk8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsqPGJyPg0KJmd0Ozxicj4NCiZndDsm
bmJzcDsgJm5ic3A7ICZuYnNwOzIwMTUgSGV1cmlzdGljIHRvIGR5bmFtaWNhbGx5IHNlbGVjdCB3
aGljaCByZWR1Y3Rpb24gYm91bmQ8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwO288YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNw
OyAmbmJzcDtPbmx5IHVzZSBQUlItU1NSQiB3aGVuIG1ha2luZyBnb29kIGZvcndhcmQgcHJvZ3Jl
c3M8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7Kzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7QUNLcyB0aGF0IGFkdmFuY2VkIHNuZC51bmEgYW5kIHJlcG9ydCBu
byBuZXcgbG9zc2VzPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDtvPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
UmVzb2x2ZXMgc29tZSBwYXRob2xvZ2ljYWwgY2FzZXMgd2l0aCB0b2tlbiBidWNrZXQgcG9saWNl
cnM8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg
Jm5ic3A7Kzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7Q0MgZXN0aW1hdGVzIHNzdGhyZXNoIGJlZm9yZSBpdCBjYW4gcG9z
c2libHkgbWVhc3VyZSB0aGU8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7dG9rZW4gcmF0ZTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsrPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtUaGUgaGV1cmlzdGlj
IG1ha2VzIHRoZSBiZXN0IG9mIGEgYmFkIHNpdHVhdGlvbjxicj4NCiZndDs8YnI+DQomZ3Q7IC0t
PGJyPg0KJmd0OyBUZW50YXRpdmUgcGF0aCBmb3J3YXJkPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJz
cDsgJm5ic3A7Kjxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDtBZG9wdCBh
cyBhIHRjcG0gd29yayBpdGVtPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7Kjxicj4N
CiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDtVcGRhdGUgdGhlIHRleHQ8YnI+DQom
Z3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288YnI+DQomZ3Q7PGJyPg0K
Jmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtOb3JtYXRpdmUgUkZDIDIxMTkg
bGFuZ3VhZ2U8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwO288
YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtBZGQg
TUFZIHVzZSB0aGUgaGV1cmlzdGljLi4uPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7
ICZuYnNwOyAmbmJzcDtvPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7VHJpbSByZWR1bmRhbnQgYW5kIG9ic29sZXRlIGxhbmd1YWdlPGJyPg0KJmd0
Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOys8YnI+
DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7
ICZuYnNwO1JGQyA2OTM3IHJlcGVhdHMgaXRzZWxmIGFuZCBpcyBtdWNoIGxvbmdlciB0aGFuIG5l
Y2Vzc2FyeTxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsrPGJyPg0KJmd0Ozxicj4NCiZndDsmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz
cDsgJm5ic3A7ICZuYnNwOyAmbmJzcDtGb2N1cyBvbiB3aGF0IGFuIGltcGxlbWVudGVyIG5lZWRz
IHRvIGtub3c8YnI+DQomZ3Q7PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm
bmJzcDsgJm5ic3A7Kzxicj4NCiZndDs8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i
c3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7VXNlIG5vbi1ub3JtYXRpdmUgcmVmZXJlbmNlcyB0byBS
RkMgNjkzNyBmb3IgcHJpb3I8YnI+DQomZ3Q7Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7bWVhc3VyZW1lbnQgd29yaywgZXRjPGJyPg0KJmd0Ozxicj4NCiZn
dDsgVGhhbmtzLDxicj4NCiZndDsgLS1NTS0tPGJyPg0KJmd0OyBUaGUgYmVzdCB3YXkgdG8gcHJl
ZGljdCB0aGUgZnV0dXJlIGlzIHRvIGNyZWF0ZSBpdC4gJm5ic3A7LSBBbGFuIEtheTxicj4NCiZn
dDs8YnI+DQomZ3Q7IFdlIG11c3Qgbm90IHRvbGVyYXRlIGludG9sZXJhbmNlOzxicj4NCiZndDsm
bmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7aG93ZXZlciBvdXIgcmVzcG9uc2UgbXVz
dCBiZSBjYXJlZnVsbHkgbWVhc3VyZWQ6PGJyPg0KJmd0OyZuYnNwOyAmbmJzcDsgJm5ic3A7ICZu
YnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyB0b28gc3Ryb25nIHdvdWxkIGJlIGh5cG9jcml0aWNh
bCBhbmQgcmlza3Mgc3BpcmFsaW5nIG91dCBvZjxicj4NCiZndDsgY29udHJvbDs8YnI+DQomZ3Q7
Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IHRvbyB3ZWFr
IHJpc2tzIGJlaW5nIG1pc3Rha2VuIGZvciB0YWNpdCBhcHByb3ZhbC48YnI+DQomZ3Q7PGJyPg0K
Jmd0OyBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4N
CiZndDsgdGNwbSBtYWlsaW5nIGxpc3Q8YnI+DQomZ3Q7IDxhIGhyZWY9Im1haWx0bzp0Y3BtQGll
dGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+dGNwbUBpZXRmLm9yZzwvYT48YnI+DQomZ3Q7IDxhIGhy
ZWY9Imh0dHBzOi8vbmFtMDYuc2FmZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1o
dHRwcyUzQSUyRiUyRnd3dy5pZXRmLm9yZyUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRnRjcG0mYW1w
O2RhdGE9MDQlN0MwMSU3Q3ByYXZiJTQwbWljcm9zb2Z0LmNvbSU3Q2M5NTI1NGJjYmEwZjQxYzE0
ZjM2MDhkODhjYTllYWM1JTdDNzJmOTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdD
MCU3QzYzNzQxNDAwODcwNTMxMzQ0MCU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1D
NHdMakF3TURBaUxDSlFJam9pVjJsdU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0Ql
N0MxMDAwJmFtcDtzZGF0YT1KWnNNQkYzVmVoVXZIU0pmZ2dQWWJQY28zNGg5VWlURyUyQktieVhV
MUJ6JTJCQSUzRCZhbXA7cmVzZXJ2ZWQ9MCIgdGFyZ2V0PSJfYmxhbmsiPg0KaHR0cHM6Ly93d3cu
aWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby90Y3BtPC9hPjxicj4NCiZndDs8YnI+DQo8YnI+DQpf
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4NCnRjcG0g
bWFpbGluZyBsaXN0PGJyPg0KPGEgaHJlZj0ibWFpbHRvOnRjcG1AaWV0Zi5vcmciIHRhcmdldD0i
X2JsYW5rIj50Y3BtQGlldGYub3JnPC9hPjxicj4NCjxhIGhyZWY9Imh0dHBzOi8vbmFtMDYuc2Fm
ZWxpbmtzLnByb3RlY3Rpb24ub3V0bG9vay5jb20vP3VybD1odHRwcyUzQSUyRiUyRnd3dy5pZXRm
Lm9yZyUyRm1haWxtYW4lMkZsaXN0aW5mbyUyRnRjcG0mYW1wO2RhdGE9MDQlN0MwMSU3Q3ByYXZi
JTQwbWljcm9zb2Z0LmNvbSU3Q2M5NTI1NGJjYmEwZjQxYzE0ZjM2MDhkODhjYTllYWM1JTdDNzJm
OTg4YmY4NmYxNDFhZjkxYWIyZDdjZDAxMWRiNDclN0MxJTdDMCU3QzYzNzQxNDAwODcwNTMxMzQ0
MCU3Q1Vua25vd24lN0NUV0ZwYkdac2IzZDhleUpXSWpvaU1DNHdMakF3TURBaUxDSlFJam9pVjJs
dU16SWlMQ0pCVGlJNklrMWhhV3dpTENKWFZDSTZNbjAlM0QlN0MxMDAwJmFtcDtzZGF0YT1KWnNN
QkYzVmVoVXZIU0pmZ2dQWWJQY28zNGg5VWlURyUyQktieVhVMUJ6JTJCQSUzRCZhbXA7cmVzZXJ2
ZWQ9MCIgdGFyZ2V0PSJfYmxhbmsiPmh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGlu
Zm8vdGNwbTwvYT48bzpwPjwvbzpwPjwvcD4NCjwvYmxvY2txdW90ZT4NCjwvZGl2Pg0KPHAgY2xh
c3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+DQo8cHJlPl9fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPG86cD48L286cD48L3ByZT4NCjxwcmU+
dGNwbSBtYWlsaW5nIGxpc3Q8bzpwPjwvbzpwPjwvcHJlPg0KPHByZT48YSBocmVmPSJtYWlsdG86
dGNwbUBpZXRmLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPnRjcG1AaWV0Zi5vcmc8L2E+PG86cD48L286
cD48L3ByZT4NCjxwcmU+PGEgaHJlZj0iaHR0cHM6Ly9uYW0wNi5zYWZlbGlua3MucHJvdGVjdGlv
bi5vdXRsb29rLmNvbS8/dXJsPWh0dHBzJTNBJTJGJTJGd3d3LmlldGYub3JnJTJGbWFpbG1hbiUy
Rmxpc3RpbmZvJTJGdGNwbSZhbXA7ZGF0YT0wNCU3QzAxJTdDcHJhdmIlNDBtaWNyb3NvZnQuY29t
JTdDYzk1MjU0YmNiYTBmNDFjMTRmMzYwOGQ4OGNhOWVhYzUlN0M3MmY5ODhiZjg2ZjE0MWFmOTFh
YjJkN2NkMDExZGI0NyU3QzElN0MwJTdDNjM3NDE0MDA4NzA1MzEzNDQwJTdDVW5rbm93biU3Q1RX
RnBiR1pzYjNkOGV5SldJam9pTUM0d0xqQXdNREFpTENKUUlqb2lWMmx1TXpJaUxDSkJUaUk2SWsx
aGFXd2lMQ0pYVkNJNk1uMCUzRCU3QzEwMDAmYW1wO3NkYXRhPUpac01CRjNWZWhVdkhTSmZnZ1BZ
YlBjbzM0aDlVaVRHJTJCS2J5WFUxQnolMkJBJTNEJmFtcDtyZXNlcnZlZD0wIiB0YXJnZXQ9Il9i
bGFuayI+aHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby90Y3BtPC9hPjxvOnA+
PC9vOnA+PC9wcmU+DQo8L2Jsb2NrcXVvdGU+DQo8cD48bzpwPiZuYnNwOzwvbzpwPjwvcD4NCjwv
ZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX188YnI+DQp0Y3BtIG1haWxpbmcgbGlzdDxicj4NCjxhIGhyZWY9Im1h
aWx0bzp0Y3BtQGlldGYub3JnIiB0YXJnZXQ9Il9ibGFuayI+dGNwbUBpZXRmLm9yZzwvYT48YnI+
DQo8YSBocmVmPSJodHRwczovL25hbTA2LnNhZmVsaW5rcy5wcm90ZWN0aW9uLm91dGxvb2suY29t
Lz91cmw9aHR0cHMlM0ElMkYlMkZ3d3cuaWV0Zi5vcmclMkZtYWlsbWFuJTJGbGlzdGluZm8lMkZ0
Y3BtJmFtcDtkYXRhPTA0JTdDMDElN0NwcmF2YiU0MG1pY3Jvc29mdC5jb20lN0NjOTUyNTRiY2Jh
MGY0MWMxNGYzNjA4ZDg4Y2E5ZWFjNSU3QzcyZjk4OGJmODZmMTQxYWY5MWFiMmQ3Y2QwMTFkYjQ3
JTdDMSU3QzAlN0M2Mzc0MTQwMDg3MDUzMjMzOTclN0NVbmtub3duJTdDVFdGcGJHWnNiM2Q4ZXlK
V0lqb2lNQzR3TGpBd01EQWlMQ0pRSWpvaVYybHVNeklpTENKQlRpSTZJazFoYVd3aUxDSlhWQ0k2
TW4wJTNEJTdDMTAwMCZhbXA7c2RhdGE9SlhsQmx5eERNQjJxUVVseCUyQlk0NGI3ZlpmZ2NqOEhh
RGNFQWtmQXY3QUFJJTNEJmFtcDtyZXNlcnZlZD0wIiB0YXJnZXQ9Il9ibGFuayI+aHR0cHM6Ly93
d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby90Y3BtPC9hPjxvOnA+PC9vOnA+PC9wPg0KPC9i
bG9ja3F1b3RlPg0KPC9kaXY+DQo8L2Rpdj4NCjwvYm9keT4NCjwvaHRtbD4NCg==

--_000_DM6PR00MB06350BB2986E364D2D8DBBDDB6E09DM6PR00MB0635namp_--


From nobody Thu Nov 19 12:56:30 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7683F3A1229 for ; Thu, 19 Nov 2020 12:56:29 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.499
X-Spam-Level: 
X-Spam-Status: No, score=-17.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwjGM0Q01uJ5 for ; Thu, 19 Nov 2020 12:56:26 -0800 (PST)
Received: from mail-vs1-xe34.google.com (mail-vs1-xe34.google.com [IPv6:2607:f8b0:4864:20::e34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0F6F03A1219 for ; Thu, 19 Nov 2020 12:56:25 -0800 (PST)
Received: by mail-vs1-xe34.google.com with SMTP id x11so3828522vsx.12 for ; Thu, 19 Nov 2020 12:56:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=u3p6mGz5Yq06DQtaV3ij8nPLJ8NYgt7N92DjnC5RV4k=; b=wSsZBf+jmD1oNs69TEQbWxH23MP5cvDzYKcwAzJCpUmVJ2ZVR8o5FkNxD67xfm/AeI YVOsXk1wXyDkRgvxU/1rC02XTrXaSpRJSzxVQGG8H3gwq7rfRi9jTD2sSJHEy9xFAhu6 UeH2EsxQjh5o68WA4UWBmxShXt9qvwrrtFoNjYE9rj28v2rDS2/vHxjH+1fI091CeCKZ 8oTO8RN2DAGlqUD3WNJxm0pFl987xG3jZ3GQ4ZZOh7WO1TlB6/DGr6rSIjsY49JtpMcb cgyw88Cjb0xB1930zGSb57Rnm8/2+JF/YnkdhWOeidX3W4mxgBc8qbwEOAeLahaxBGIZ TOaw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=u3p6mGz5Yq06DQtaV3ij8nPLJ8NYgt7N92DjnC5RV4k=; b=MA32JFOCKT7AHMaE0GAB/NwySWKL6FZxX+DJdvP4CA2BXJMqceX7oLxDB9OxC2vfCZ j3ER+R6G+p/cgWTFICFc7ZRQDiJ7TY0I4f+vX454jcm8akqLuGg0jzrhJKYMJnLQXBMy C6muIzkndmBFWDqmcn1MZp0q3hB6wOrBth98q8gJMduAnfzMsuzO48l/AOB83Mu9YBRd ZQJP5rJfXRifNtRteL+fghTgkVWfOydEhgFJWEAq+vJ7SkaLsJKiwvFYZfL9RT5eVYP6 OAE2QktmwW/4GHEOThfGrqc9CyEJFo7AUnUOih/8Ci4XedwFRe9qcU596DHN10rdI5wq MjFQ==
X-Gm-Message-State: AOAM5314kCRbsTqS/07aJ+bVMqJyQKVFK7LACxC3oAnt/mWupi9UuH8R udRCt5SiVQTyyktl39O8hqDODqrBRooFZX2T4hm/0/hHH3Lg5w==
X-Google-Smtp-Source: ABdhPJwhlDvFCLepBAFrr6zYa2RmTD32GlUbur4Tjn3t98/aePMqoFuvicA0BR7PLh8cedyInyKvu5SrjvZgNS04Ls4=
X-Received: by 2002:a67:ee16:: with SMTP id f22mr9371376vsp.8.1605819384540; Thu, 19 Nov 2020 12:56:24 -0800 (PST)
MIME-Version: 1.0
References:    <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk>  
In-Reply-To: 
From: Neal Cardwell 
Date: Thu, 19 Nov 2020 15:56:08 -0500
Message-ID: 
To: Praveen Balasubramanian 
Cc: "mattmathis=40google.com@dmarc.ietf.org" ,  "gorry@erg.abdn.ac.uk" ,  "ianswett=40google.com@dmarc.ietf.org" ,  "tcpm@ietf.org" 
Content-Type: multipart/alternative; boundary="000000000000fdb90f05b47bf5ca"
Archived-At: 
Subject: Re: [tcpm] [EXTERNAL] Re: Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 19 Nov 2020 20:56:29 -0000

--000000000000fdb90f05b47bf5ca
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I am very supportive of updating PRR to PS status and adopting this as a
TCPM work item. I will be happy to review the document as well.

The "Tentative path forward" outlined by Matt sounds good to me.

best,
neal




On Thu, Nov 19, 2020 at 12:37 PM Praveen Balasubramanian  wrote:

> I am supportive of updating PRR to PS status and will review the document=
.
>
>
>
> Re. QUIC. The current loss recovery spec for QUIC already refers to PRR a=
s
> a MAY and has similar other references to TCP RFCs without any additional
> text for QUIC specific adaptations. The current title of RFC 6937 scopes =
it
> to TCP: Proportional Rate Reduction for TCP. I also see about 38 hits for
> the word TCP in the text.
>
>
>
> *From:* tcpm  *On Behalf Of * Matt Mathis
> *Sent:* Thursday, November 19, 2020 8:41 AM
> *To:* Gorry Fairhurst ; Ian Swett  40google.com@dmarc.ietf.org>
> *Cc:* tcpm@ietf.org Extensions 
> *Subject:* [EXTERNAL] Re: [tcpm] Introduce RFC 6937 bis (Proportional
> Rate Reduction) as a tcpm work item? INPUT NEEDED
>
>
>
> If we can get away with a non-normative description of the heuristic, QUI=
C
> is no problem at all.    If we need normative descriptions, then TCP and
> QUIC have to be different, and we need to include normative references to
> both recovery machines and recovery state variables, which are not in sco=
pe
> for the current doc.
>
>
>
> I would prefer to stay with a non-normative description of the heuristic,
> and scrub the rest of the document for language that conflicts with QUIC.
>
>
> Thanks,
>
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
>
> We must not tolerate intolerance;
>
>        however our response must be carefully measured:
>
>             too strong would be hypocritical and risks spiraling out of
> control;
>
>             too weak risks being mistaken for tacit approval.
>
>
>
>
>
> On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst 
> wrote:
>
> On 19/11/2020 14:20, Ian Swett wrote:
>
> I support this work.
>
>
>
> One question for authors and the group.  This specification could be
> applied to QUIC, but my memory is that there are some small changes to
> adapt it correctly.  Would noting those changes be in scope for this
> document, a separate document in QUIC, or an exercise left to the reader?
> I'd be happy to contribute any QUIC specific text if that's helpful.
>
>
>
> Thanks, Ian
>
> I'd hope the differences for QUIC ARE captured in the RFC series. If they
> are small, it might be possible to add to this draft perhaps?
>
> Gorry
>
>
>
> On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richard 
> wrote:
>
> As the heuristic is the one major improvement over RFC6937, that is the
> one area I am especially interested in. Currently working to have PRR in
> FreeBSD, it would be the best time window to add this heuristig instead
> of switching between the two variants in a static fashion.
>
> I support this and will certainly be reviewing the documents and provide
> feedback!
>
> Richard
>
>
> Am 19.11.2020 um 04:56 schrieb Matt Mathis:
> > The authors of PRR would like to update PRR to Proposed Standard
> > status.  This entails introducing a new document as an tcpm work item.
> >
> > *Please indicate (non) support and/or comment.*
> >
> > For more details see the tcpm meeting materials from IETF 109
> > minutes:
> > https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
> 
> > slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
> 
> >
> > There were about four "I support this work" remarks at the mic (not
> > recorded in the minutes), and about as many in the Meetecho chat.
> >
> > Abridged IETF/tcpm/PRRbis slides:
> > --
> > PRR  recap (RFC6937 experimental)
> > PRR is a special congestion control effective only during fast recovery
> >
> >   * When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.=
g.
> >     loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
> >   * When inflight < ssthresh, send at the same or twice the
> >     delivery_rate (more later)
> >   * Used by all congestion control modules in Linux during fast recover=
y
> >       o Can be more dominant than the actual C.C. for lossy flows
> >         that=E2=80=99re in fast recovery constantly (e.g. video streami=
ng
> >         through policers)
> >
> > --
> > Current Status
> >
> >   *
> >
> >     PRR is widely deployed
> >
> >       o
> >
> >         At least three major OSs: Linux, Windows, (NetFlix) BSD
> >
> >       o
> >
> >         Vast majority of Web traffic for years
> >
> >   *
> >
> >     No changes to algorithms published in RFC 6937
> >
> >       o
> >
> >         PRR-CRB - Conservative Reduction Bound - strict packet
> >         conversion during loss recovery
> >
> >       o
> >
> >         PRR-SSRB - Slowstart Reduction Bound - one extra segment per AC=
K
> >         during loss recovery
> >
> >   *
> >
> >     2015 Heuristic to dynamically select which reduction bound
> >
> >       o
> >
> >         Only use PRR-SSRB when making good forward progress
> >
> >           +
> >
> >             ACKs that advanced snd.una and report no new losses
> >
> >       o
> >
> >         Resolves some pathological cases with token bucket policers
> >
> >           +
> >
> >             CC estimates ssthresh before it can possibly measure the
> >             token rate
> >
> >           +
> >
> >             The heuristic makes the best of a bad situation
> >
> > --
> > Tentative path forward
> >
> >   *
> >
> >     Adopt as a tcpm work item
> >
> >   *
> >
> >     Update the text
> >
> >       o
> >
> >         Normative RFC 2119 language
> >
> >       o
> >
> >         Add MAY use the heuristic...
> >
> >       o
> >
> >         Trim redundant and obsolete language
> >
> >           +
> >
> >             RFC 6937 repeats itself and is much longer than necessary
> >
> >           +
> >
> >             Focus on what an implementer needs to know
> >
> >           +
> >
> >             Use non-normative references to RFC 6937 for prior
> >             measurement work, etc
> >
> > Thanks,
> > --MM--
> > The best way to predict the future is to create it.  - Alan Kay
> >
> > We must not tolerate intolerance;
> >         however our response must be carefully measured:
> >              too strong would be hypocritical and risks spiraling out o=
f
> > control;
> >              too weak risks being mistaken for tacit approval.
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpm
> 
> >
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
> 
>
>
>
> _______________________________________________
>
> tcpm mailing list
>
> tcpm@ietf.org
>
> https://www.ietf.org/mailman/listinfo/tcpm 
>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
> 
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--000000000000fdb90f05b47bf5ca
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


I am very supportive of updating PRR to PS status and adop=
ting this as a TCPM work item. I will be happy to review the document as we=
ll.

The "Tentative path forward" outlined by Matt sounds g=
ood to me.

best,
neal





--000000000000fdb90f05b47bf5ca--


From nobody Fri Nov 20 01:06:11 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D9D6F3A1AEA; Fri, 20 Nov 2020 01:06:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AmiyMDZw35W1; Fri, 20 Nov 2020 01:06:00 -0800 (PST)
Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9B00D3A1B1F; Fri, 20 Nov 2020 01:05:59 -0800 (PST)
Received: by mail-ot1-x32f.google.com with SMTP id 92so5033909otd.5; Fri, 20 Nov 2020 01:05:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=agEg0jGErrQU2uilzY71nL9vrFGMYJJmt9ZMWE4illA=; b=eWvGGkdwyHMOt7yaqDvOt7D1r5r6784HDjt0JMbI24RIzakYNV1303KlBF+k/kgb+9 uMssSGKp25lMcCjQiltLnXU3K0OB4kjbtAfglZXTtIdT7qkdn2R691fJ6WpvHBbespeV mO8xLchVxO++UviJHTw1py3JZWIvmAqkxZZuqT9tx6ERB2Ta9cOm8NVwsNIJmRV/MOTi +ZqIAN97YgBe+5zRKtydu/xaJy+lAKqIDDaFVyoZIkBFS6+ZT3YgUiIVok530hiApUtY gx+Oq6C7lgELu8FdV3ZXPnXNS2HIxqmhn/UMAkOUDIbzBpZm/PTMi9fGIi3MLJLkUWp5 eaqg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=agEg0jGErrQU2uilzY71nL9vrFGMYJJmt9ZMWE4illA=; b=NtRPn+q8enWDQ9pmc1NeVpLuTXzQWZIDZtqmC6PfMsVhOUayzgr4S6nPV3XYKCdOGF syW5bsUT6anR7XrM1jyIOJ+s5mn2WVBubSCGBsscAxXtiAvep8gBFhNRyV6n8icwIZsz b00BECP5DNHiuNDLJY8gV9s0Bs7BbpAwvYSJ9me+2qwDy/MmvXTK366QflhxaX3PAPg4 7bT2dnsr+tX/o6ti4njnS8+AjstEwmR/sYTTE0c3mW/VbPfn7yz1pLfoQOq6XnU+tTvF aqOCLojnMB7v6Rjrm6M8U6camcKctx0vMfdVNIq+HbFN7SFTd3tGxm8C1RnJ89z+OwNs Rb9g==
X-Gm-Message-State: AOAM532+f2HfKgfb4/6O/a5dQBfV4gTG1mx7FOtubVcTK0unPC28pNXR n4/StRUDMaSQc0xWtDmEwOlmvEOsXjTU3cE3+HBI60lkjPg=
X-Google-Smtp-Source: ABdhPJzaQgRtXz4bVzyB3AOgxLbJjdAx98U5VdH4vO+hPDjg/7jkOljQKU223cG2vFug/+MXdEXlcNNWZwSGeTsT23c=
X-Received: by 2002:a9d:4c92:: with SMTP id m18mr13553418otf.248.1605863158810;  Fri, 20 Nov 2020 01:05:58 -0800 (PST)
MIME-Version: 1.0
References:  
In-Reply-To: 
From: Alexander Azimov 
Date: Fri, 20 Nov 2020 12:05:46 +0300
Message-ID: 
To: Tom Herbert 
Cc: tcpm , IPv6 Operations 
Content-Type: multipart/alternative; boundary="0000000000002389c305b4862791"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 20 Nov 2020 09:06:06 -0000

--0000000000002389c305b4862791
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Tom,

As far as I know, the auto_flowlabels default is 1, which stands for:

> 1: automatic flow labels are enabled by default, they can be disabled on =
a
> per socket basis using the IPV6_AUTOFLOWLABEL socket option


And you get it all-in-one: FL changes after both SYN_RTO and RTO events.
The change after SYN_RTO looks safe and can bring benefit for the end-user
experience, while the FL change after the RTO outside of a controlled
environment is causing a problem.

And yes, we were able to detect it on a working system. One of the
developers contacted NOC team that he has constant connection failures
while pushing a significant amount of data, and it began when he started
working from home. It proved out that there was RTO happening during the
lifetime of the TCP session, the session jumped to another stateful L4
balancer and, as a result, the session got a timeout. At the moment we had
to disable FL load balancing at our border routers and DCI - this solved
the issue at the moment.

If it is possible to remove from the default behavior change of flow label
after RTO event, while keeping it for SYN_RTO would be great.

=D1=87=D1=82, 19 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 18:36, Tom H=
erbert :

> On Thu, Nov 19, 2020 at 3:49 AM Alexander Azimov 
> wrote:
> >
> > Dear colleagues,
> >
> > I have added in the cc both v6op and tcpm for a reason and let me
> explain why.
> >
> > It's clear that we are moving forward with load balancing that uses flo=
w
> label (FL). And the pressure will increase with SRv6 adoption. But at the
> moment wide adoption of FL-based load-balancing may create significant
> issues for TCP Anycast services.
> >
> > RFC6437 suggests putting hash from 5-tuple into FL value. And as far as
> I know, there is no document that updates this behavior. This description
> is perfectly fine, but what is implemented in the Linux kernel is
> different: FL is carrying hash from 5-tuple with an additional seed, and
> this seed is randomly changed after each RTO/SYN_RTO event. Here are
> related patches:
> >
> >
> https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.m=
tv.corp.google.com/
> >
> https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@he=
rbertland.com/
> > https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
> >
> Alexander,
>
> The initial intent of the Linux flow label code was that the hash was
> persistent for the lifetime of the connection; in fact we fixed a
> nasty bug where the hash would change when a connection moved to TW
> state causing some firewalls not to see the final ACK so they can
> remove state. The feature where the flow hash is recalculated for a
> failing connection should be optional and not the default behavior. If
> this isn't what the implementation is doing I can take it up on Linux
> netdev to fix it.
>
> Have you been able to show this behavior is actually happening on a
> running system?
>
> Tom
>
> >
> > This is a great thing by the way because in the data center environment
> with multiple equal paths it gives a way to have pseudo-multipath TCP whi=
ch
> jumps between paths in case of an outage. There might be interest to writ=
e
> down an informational document for this.
> >
> > But the problem happens when TCP flows start jumping between anycast
> points. If anycast service provides connection tracking (L7/TCP proxy,
> stateful L4 balancer) the 'jump' may redirect traffic to another point, a=
nd
> with the adoption of FL load balancing by the transit providers - to
> another location, where no TCP state is available. In other words, an
> established TCP session breaks after RTO event. And it's not just a theor=
y
> - we already faced this kind of issue in our network, and it is really ha=
rd
> to debug.
> >
> > I wonder what you think is a proper solution:
> >
> > Making FL related RTO change as knob instead of default behavior;
> > Adding negotiation behavior in TCP;
> > Something else?
> >
> > I'm looking forward to your advice. If there is a document that
> describes the above problem - please give me a reference.
> >
> > --
> > Best regards,
> > Alexander Azimov
> > _______________________________________________
> > v6ops mailing list
> > v6ops@ietf.org
> > https://www.ietf.org/mailman/listinfo/v6ops
>


--=20
Best regards,
Alexander Azimov

--0000000000002389c305b4862791
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Tom,

As far as I know, the auto_flowlab=
els default is 1, which stands for:
1: automatic flow labels are enabled by default, they can b=
e disabled on a per socket basis using the IPV6_AUTOFLOWLABEL socket option=


And you get it all-in-one: FL changes afte=
r both SYN_RTO and RTO events. The change after SYN_RTO looks safe and can =
bring benefit for the end-user experience, while the FL change after the RT=
O=C2=A0outside of a controlled environment is=C2=A0causing a problem.=C2=A0=


And yes, we were able to detect it on a working s=
ystem. One of the developers contacted NOC team that he has constant connec=
tion failures while pushing a=C2=A0significant=C2=A0amount of data, and it =
began when he started working from home. It proved out that there was RTO h=
appening during the lifetime of the TCP session, the session jumped to anot=
her stateful L4 balancer and, as a result,=C2=A0the session got a timeout. =
At the moment we had to disable=C2=A0FL load balancing at our=C2=A0border r=
outers and DCI - this solved the issue at the moment.=C2=A0

If it is=
 possible to remove from the default behavior change of flow label after RT=
O event, while keeping it for SYN_RTO would be great.

=D1=87=D1=82=
, 19 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 18:36, Tom Herbert <<=
a href=3D"mailto:tom@herbertland.com">tom@herbertland.com>:
On Thu, Nov 19, 2020 at =
3:49 AM Alexander Azimov <a.e.azimov@gmail.com> wrote:

>

> Dear colleagues,

>

> I have added in the cc both v6op and tcpm for a reason and let me expl=
ain why.

>

> It's clear that we are moving forward with load balancing that use=
s flow label (FL). And the pressure will increase with SRv6 adoption. But a=
t the moment wide adoption of FL-based load-balancing may create significan=
t issues for TCP Anycast services.

>

> RFC6437 suggests putting hash from 5-tuple into FL value. And as far a=
s I know, there is no document that updates this behavior. This description=
 is perfectly fine, but what is implemented in the Linux kernel is differen=
t: FL is carrying hash from 5-tuple with an additional seed, and this seed =
is randomly changed after each RTO/SYN_RTO event. Here are related patches:=


>

> htt=
ps://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.mtv.co=
rp.google.com/

> https://=
lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@herbertland.=
com/

> https://lore.kernel.org/n=
etdev/20160928020337.3057238-1-brakmo@fb.com/

>

Alexander,



The initial intent of the Linux flow label code was that the hash was

persistent for the lifetime of the connection; in fact we fixed a

nasty bug where the hash would change when a connection moved to TW

state causing some firewalls not to see the final ACK so they can

remove state. The feature where the flow hash is recalculated for a

failing connection should be optional and not the default behavior. If

this isn't what the implementation is doing I can take it up on Linux
netdev to fix it.



Have you been able to show this behavior is actually happening on a

running system?



Tom



>

> This is a great thing by the way because in the data center environmen=
t with multiple equal paths it gives a way to have pseudo-multipath TCP whi=
ch jumps between paths in case of an outage. There might be interest to wri=
te down an informational document for this.

>

> But the problem happens when TCP flows start jumping between anycast p=
oints. If anycast service provides connection tracking (L7/TCP proxy, state=
ful L4 balancer) the 'jump' may redirect traffic to another point, =
and with the adoption of FL load balancing by the transit providers - to an=
other location, where no TCP state is available. In other words, an establi=
shed TCP session breaks after RTO event. And it's not just a theory - w=
e already faced this kind of issue in our network, and it is really hard to=
 debug.

>

> I wonder what you think is a proper solution:

>

> Making FL related RTO change as knob instead of default behavior;

> Adding negotiation behavior in TCP;

> Something else?

>

> I'm looking forward to your advice. If there is a document that de=
scribes the above problem - please give me a reference.

>

> --

> Best regards,

> Alexander Azimov

> _______________________________________________

> v6ops mailing list

> v6ops@ietf.org=


> https://www.ietf.org/mailman/listinfo/v6ops




-- 
Best regards,
Alexander Azi=
mov


--0000000000002389c305b4862791--


From nobody Fri Nov 20 08:26:46 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A5FD33A0E1B for ; Fri, 20 Nov 2020 08:26:44 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.6
X-Spam-Level: 
X-Spam-Status: No, score=-17.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Sfrbc_bHYSpy for ; Fri, 20 Nov 2020 08:26:43 -0800 (PST)
Received: from mail-vs1-xe2b.google.com (mail-vs1-xe2b.google.com [IPv6:2607:f8b0:4864:20::e2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 419623A0C04 for ; Fri, 20 Nov 2020 08:26:43 -0800 (PST)
Received: by mail-vs1-xe2b.google.com with SMTP id r5so5292408vsp.7 for ; Fri, 20 Nov 2020 08:26:43 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=jPdytSQLvDwtQROw4No7Faby1eesvhFWKvKMJk3DjU4=; b=cORYg0Tm+XWlckqMqygZH+VVTRJwYcHqpxzR0gH4uE3LoR5yd3KZk/SDDX0Odtrunl IjoySTkQqdDTFG/23p2f2j+gBet+gzSe/2x8cGpXWi8um7L3ullNBbRm/AWv1ukFtI4Y AiqsnJxM+0D3r/69yh1ZIm8hsnmUAed76DVEeOmVtm0eOSBF+uSUNvE5dSHy1cfzulcT yCvMk7Xhy7X5YoyxF7gEFwPasJe4nI9/4GaO5jhBEuLk3wzx0g/mLtNW8B6BY1ykGbBa Gs/GUksYbyo9MMwBeuMJh2hkvCcAg9t/vAfKRmaf0DQtGd9+JClCwraFsnYTuKaTGhDE d9zg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=jPdytSQLvDwtQROw4No7Faby1eesvhFWKvKMJk3DjU4=; b=nuY3KNvPiCfamvg3JJaowfLIMSTdMhh92wL+CFj9uaVqAngTMCq+PeaDP66DXsbYb1 m1KTj7ifDTEPkiIj9kOoB8DpR4w+86m6M9twSVbqL2A+Qu0G9WNPv3RdlPHBaSmJFl0q jUbFbZzG/1m6s8iltBTSTW9PIviOiGgyKHKSNVbw8ODXcw69PYueR8Vg/0TtwHqfZwMq r8iuQB3zVC0t8Y3yRtajGGzSP5ztY8jMyjXU6yy2o2cvWnWMNehlSeWpZhlx1OOZDZHf RvAbDDQpJdhs+XtPnML1xqX5R3i6hruMzc5FlX2nJeIuZBgrAb6W3PXjh+cwUXa1NWz/ K3sw==
X-Gm-Message-State: AOAM531ImBTd2ozE1eoiYidWxOTQDMolCrJ/qY4GZ8L9tXa67TiR0wcw ExTKniBKuUSNAtom2hS2pYUz91iA9oGSRCWJHD2HI6fWcqb7qQ==
X-Google-Smtp-Source: ABdhPJwIN3pd7rhau2WqrvWf/im0DZk1fcTNsNW3WeOBF9/GPZzanHyudnntoonQpSjWVX92WBxjI1CvfHLcl6mFF3g=
X-Received: by 2002:a67:ff01:: with SMTP id v1mr13777055vsp.16.1605889601850;  Fri, 20 Nov 2020 08:26:41 -0800 (PST)
MIME-Version: 1.0
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>
In-Reply-To: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com>
From: Neal Cardwell 
Date: Fri, 20 Nov 2020 11:26:25 -0500
Message-ID: 
To: Vidhi Goel 
Cc: "tcpm@ietf.org Extensions" 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 20 Nov 2020 16:26:45 -0000

On Mon, Nov 16, 2020 at 9:08 PM Vidhi Goel
 wrote:
>
> Hello,
>
> As Lars mentioned earlier, there are a couple of issues/proposals filed f=
or updating the CUBIC RFC 8312 and are currently under discussion. The GitH=
ub repo is at https://github.com/NTAP/rfc8312bis.
> We would like to gather information on which CUBIC implementations alread=
y or would be ready to adopt these changes.
>
> 1. Update K definition (Eq. 2) to account for Fast Convergence
> Issue link https://github.com/NTAP/rfc8312bis/issues/1
> Linux already has this change. Apple's Cubic implementation for both TCP =
and QUIC already use this.
>
> 2. Congestion window TCP friendly region after W_max
> Issue link https://github.com/NTAP/rfc8312bis/issues/2
> This is a suggestion to improve performance after cwnd > W_max. Apple's C=
ubic implementation for both TCP and QUIC already use this.
>
> From my experience, these changes would improve Cubic=E2=80=99s performan=
ce.
>
> Please let us know if you already or would be ready to implement these ch=
anges in your CUBIC=E2=80=99s implementation.

Yuchung and I chatted about (2.) (
https://github.com/NTAP/rfc8312bis/issues/2 ) and we agreed this looks
like a nice and worthwhile fix, which we should be able to integrate
into the Linux TCP CUBIC implementation.

Thanks for this effort!

best,
neal


From nobody Fri Nov 20 12:39:17 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 574223A113F for ; Fri, 20 Nov 2020 12:39:15 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.599
X-Spam-Level: 
X-Spam-Status: No, score=-17.599 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SxDsRIVZfDCm for ; Fri, 20 Nov 2020 12:39:13 -0800 (PST)
Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9634C3A113B for ; Fri, 20 Nov 2020 12:39:13 -0800 (PST)
Received: by mail-yb1-xb34.google.com with SMTP id g15so9748765ybq.6 for ; Fri, 20 Nov 2020 12:39:13 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=S8V+rdiTYKYxzFWlDU3T7wpsF90MTJOCxqr3y7yl6/Y=; b=a97J1ZYDaKUXe0hh1BE9hybEdbhlBYCTVc94K1NnPDFlor4VPkzPsLTrYlF9St2/B/ qMwbxrE11KQGjNVzE/MmkLpdiSxaSXoc0Zw9UEzoU6iv3ti1PFHtsvbWxtKl38nWZgAt 2iSWXXwJClcDVygKKS1IOHe+XoAGzGnCBapBJ8NlCpj7RQqpv1ahhBl1z07N0/dSo1vY /6reQLVLlV67qXiFUvlQCJ0ZzL6WvexVLQsVoq0bBIsK3To9ThNk7P3CvEXY3oUpukUR IUEqjkKdA8ck63kT5fF2rP2tR3Lp79Z2fH/xVgR7SQxTqk932fvsu3D2CRS01/9xGvLU pzAw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=S8V+rdiTYKYxzFWlDU3T7wpsF90MTJOCxqr3y7yl6/Y=; b=Olql1I4jP2kZ9RyxCH4XizZAM/RwzMGCJsPWo8GFhdEYvI/CKActb+JW54WAUUavup yGGKQUD7r0UjO2on5yJ1Qrr4rLhI7lXYKUQn0+14O9SLuhksg13K6Bwe4GrfSx1JcnJ5 aSKribgeF2W0r7GaiuHX5uDS3/Jicwmr5eT+WS3bF5EYHbSUoIcULaLV+oX93a80feHr VAjoDj3c4Pu3h6Nq9lmJu68vJuXTLtlsNYxWNhzhp/Du3neH3FDMqn2hNBSQ6X26wXwB PdTxC266t2IWPKQgmSJQvY29V1RMvOaN3sWgGZiSwo/aOSe/TtYJZjbr1OVu64ML91st 66aQ==
X-Gm-Message-State: AOAM531eyNqs+ZnyHOXInd6FhOjMA4K5gPdAMcIFJBtBA9yx84RLhJq0 dHjc65rBt6qw+YV5S3AR+BJ0ryx/z3B8JZ/sDj3KLQ==
X-Google-Smtp-Source: ABdhPJx64G+7AhIdcQ+XqS+4h8/ndsgGR6U2wyeOePLvZI3NuD7CStSuWad9Pyj0wZ0to1fN1yBkavMdHtx+V2VD3vY=
X-Received: by 2002:a25:a381:: with SMTP id e1mr7672914ybi.119.1605904752319;  Fri, 20 Nov 2020 12:39:12 -0800 (PST)
MIME-Version: 1.0
References: <02721B41-690E-4CC2-BC26-223EDA4BFA45@apple.com> 
In-Reply-To: 
From: Ian Swett 
Date: Fri, 20 Nov 2020 15:39:01 -0500
Message-ID: 
To: Neal Cardwell 
Cc: Vidhi Goel ,  "tcpm@ietf.org Extensions" 
Content-Type: multipart/alternative; boundary="0000000000004e992205b48fd64e"
Archived-At: 
Subject: Re: [tcpm] Poll for CUBIC RFC updates
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 20 Nov 2020 20:39:15 -0000

--0000000000004e992205b48fd64e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

As with the PRR update, I'd like to see some text(ideally non-normative)
describing any changes necessary for implementing Cubic in QUIC.

On Fri, Nov 20, 2020 at 11:26 AM Neal Cardwell  wrote:

> On Mon, Nov 16, 2020 at 9:08 PM Vidhi Goel
>  wrote:
> >
> > Hello,
> >
> > As Lars mentioned earlier, there are a couple of issues/proposals filed
> for updating the CUBIC RFC 8312 and are currently under discussion. The
> GitHub repo is at https://github.com/NTAP/rfc8312bis.
> > We would like to gather information on which CUBIC implementations
> already or would be ready to adopt these changes.
> >
> > 1. Update K definition (Eq. 2) to account for Fast Convergence
> > Issue link https://github.com/NTAP/rfc8312bis/issues/1
> > Linux already has this change. Apple's Cubic implementation for both TC=
P
> and QUIC already use this.
> >
> > 2. Congestion window TCP friendly region after W_max
> > Issue link https://github.com/NTAP/rfc8312bis/issues/2
> > This is a suggestion to improve performance after cwnd > W_max. Apple's
> Cubic implementation for both TCP and QUIC already use this.
> >
> > From my experience, these changes would improve Cubic=E2=80=99s perform=
ance.
> >
> > Please let us know if you already or would be ready to implement these
> changes in your CUBIC=E2=80=99s implementation.
>
> Yuchung and I chatted about (2.) (
> https://github.com/NTAP/rfc8312bis/issues/2 ) and we agreed this looks
> like a nice and worthwhile fix, which we should be able to integrate
> into the Linux TCP CUBIC implementation.
>
> Thanks for this effort!
>
> best,
> neal
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--0000000000004e992205b48fd64e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


As with the PRR update, I'd like to see some text(idea=
lly non-normative) describing any changes necessary for implementing Cubic =
in QUIC.

On Fri, Nov 20, 2020 at 11:26 AM Neal Cardwell <ncardwell=3D40google.com@dmarc.ietf.org&=
gt; wrote:
On Mo=
n, Nov 16, 2020 at 9:08 PM Vidhi Goel

<vidhi_goel=3D40apple.com@dmarc.ietf.org> wrote:

>

> Hello,

>

> As Lars mentioned earlier, there are a couple of issues/proposals file=
d for updating the CUBIC RFC 8312 and are currently under discussion. The G=
itHub repo is at https://github.com/NTAP/rfc8312bis.

> We would like to gather information on which CUBIC implementations alr=
eady or would be ready to adopt these changes.

>

> 1. Update K definition (Eq. 2) to account for Fast Convergence

> Issue link https://github.com/NTAP/rfc8312bis/issues=
/1

> Linux already has this change. Apple's Cubic implementation for bo=
th TCP and QUIC already use this.

>

> 2. Congestion window TCP friendly region after W_max

> Issue link https://github.com/NTAP/rfc8312bis/issues=
/2

> This is a suggestion to improve performance after cwnd > W_max. App=
le's Cubic implementation for both TCP and QUIC already use this.

>

> From my experience, these changes would improve Cubic=E2=80=99s perfor=
mance.

>

> Please let us know if you already or would be ready to implement these=
 changes in your CUBIC=E2=80=99s implementation.



Yuchung and I chatted about (2.) (

https://github.com/NTAP/rfc8312bis/issues/2 ) and we =
agreed this looks

like a nice and worthwhile fix, which we should be able to integrate

into the Linux TCP CUBIC implementation.



Thanks for this effort!



best,

neal



_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--0000000000004e992205b48fd64e--


From nobody Fri Nov 20 12:52:41 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A68B83A11AB for ; Fri, 20 Nov 2020 12:52:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.499
X-Spam-Level: 
X-Spam-Status: No, score=-17.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V3e5GjLiyD8I for ; Fri, 20 Nov 2020 12:52:36 -0800 (PST)
Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B950B3A1196 for ; Fri, 20 Nov 2020 12:52:35 -0800 (PST)
Received: by mail-wr1-x433.google.com with SMTP id r17so11602994wrw.1 for ; Fri, 20 Nov 2020 12:52:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=8HbzU08XINrzkx9PP9FjPLXao6DwNJLVnt18gx/y/aI=; b=RdOxIiBZeUsw8oiW1E5TJ33V0aKw0dQto+mkg8uvM7kJ21Rd7uhXHFLTiLptVQfnbr llW0xEz8n99jVoO+0uu0nVAJ9zsXMG8xxkoVVXT1oi98kmyhQV5ijjpjuODAJIFLGkjA k2e1IMISdtmc8EgkgZl0+Vj+oenMSkp7jffC3L8SJkNsbXTikBvMgWTmTo+FGLAfkbJQ 4osd0TXsMWrgWIgcvlUrMLVC/yfWhcwJonX7fvZwVfoZNDSdTTcK5EbspPMZn8eTSHpB 7s5OMBGcWkmX6/Ks3u082HiuBCVOdG6rBye0yrDZsOGchoj/m/O5RSlvdadfzvYkNQYP SxAw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=8HbzU08XINrzkx9PP9FjPLXao6DwNJLVnt18gx/y/aI=; b=NarZzyKA4uroWAuyMyDpmSIXhr62Dl5Ke4c3ng8Ig7pClzQibMIFW/NXSZBhja0Aj7 Z1mHOoR6sGXpyTz34V3prcbz6w3mP71DUUVPWcfpu9hYYq0D5UuAkA1Gy2IYt8K08UAi XASw/DQyP3KohJ+DveTumq3D/Mp5TRXpxRuuzwab1i+humVtiM0Ja6mjioueIPx3kXmh TnJJPTogCeeQVcNqSD6EgZip+QnmJlE1LQMp28M3MtMT+pzW9l1ZMP0K1Mp7TJ87pIn9 odEXAso2vEXyEl/L/vguwRoYFQy9xO3Cz2t9yKXN1HB9GycbGDV2sjahw11hunjw+clx fvDA==
X-Gm-Message-State: AOAM53197ngqCh9xveWWFGFb8xuMTJUXEHwpukpBkPLHk84jolRkstvq snT6zSBId867o0LZPNmugDLJFCUiJSLCl62lsfcR1VtdSVU=
X-Google-Smtp-Source: ABdhPJzKwoXU5IiqNsP4fj7YPOd+1nYopZTbP2VQ/LEqpt+1f5TFwBbrTVgFzJTs373qA9dUiRA6vJdbW4pvWgil+jE=
X-Received: by 2002:a5d:6892:: with SMTP id h18mr17922147wru.49.1605905553834;  Fri, 20 Nov 2020 12:52:33 -0800 (PST)
MIME-Version: 1.0
References:    <885d9aa0-50b2-3101-64b8-7a40f25ca612@erg.abdn.ac.uk>   
In-Reply-To: 
From: Yuchung Cheng 
Date: Fri, 20 Nov 2020 12:51:55 -0800
Message-ID: 
To: Neal Cardwell 
Cc: Praveen Balasubramanian ,  "mattmathis=40google.com@dmarc.ietf.org" , "tcpm@ietf.org" ,  "ianswett=40google.com@dmarc.ietf.org" ,  Nandita Dukkipati 
Content-Type: multipart/alternative; boundary="00000000000014dab505b4900681"
Archived-At: 
Subject: Re: [tcpm] [EXTERNAL] Re: Introduce RFC 6937 bis (Proportional Rate Reduction) as a tcpm work item? INPUT NEEDED
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 20 Nov 2020 20:52:40 -0000

--00000000000014dab505b4900681
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for everybody's reply and support

One additional thing we authors would like more feedback on is ECN. The
current RFC and proposed -bis do not cover ECN handling. However in Linux,
ECN handling uses the same PRR algorithm (aka "CWR" aka "cwnd reduction")
just like fast recovery. In other words, if RFC3168 or DCTCP triggers a
congestion window reduction, PRR takes over to control the transmission
until SND_UNA crosses SND.NXT upon reduction.

This arrangement was inherented from the original Linux rate-halving design
before the PRR was introduced. This paper has details
https://www.cs.helsinki.fi/research/iwtcp/papers/linuxtcp.pdf

Compared to fast recovery, this part of the implementation is far less
tested due to the lack of wide classic-ECN usage on the Internet. It's also
unclear how other PRR implementations have applied to ECN handling like
Linux does. Adding ECN handling in -bis PS makes me uneasy.

On Thu, Nov 19, 2020 at 12:57 PM Neal Cardwell  wrote:

> I am very supportive of updating PRR to PS status and adopting this as a
> TCPM work item. I will be happy to review the document as well.
>
> The "Tentative path forward" outlined by Matt sounds good to me.
>
> best,
> neal
>
>
>
>
> On Thu, Nov 19, 2020 at 12:37 PM Praveen Balasubramanian  40microsoft.com@dmarc.ietf.org> wrote:
>
>> I am supportive of updating PRR to PS status and will review the documen=
t.
>>
>>
>>
>> Re. QUIC. The current loss recovery spec for QUIC already refers to PRR
>> as a MAY and has similar other references to TCP RFCs without any
>> additional text for QUIC specific adaptations. The current title of RFC
>> 6937 scopes it to TCP: Proportional Rate Reduction for TCP. I also see
>> about 38 hits for the word TCP in the text.
>>
>>
>>
>> *From:* tcpm  *On Behalf Of * Matt Mathis
>> *Sent:* Thursday, November 19, 2020 8:41 AM
>> *To:* Gorry Fairhurst ; Ian Swett > 40google.com@dmarc.ietf.org>
>> *Cc:* tcpm@ietf.org Extensions 
>> *Subject:* [EXTERNAL] Re: [tcpm] Introduce RFC 6937 bis (Proportional
>> Rate Reduction) as a tcpm work item? INPUT NEEDED
>>
>>
>>
>> If we can get away with a non-normative description of the heuristic,
>> QUIC is no problem at all.    If we need normative descriptions, then TC=
P
>> and QUIC have to be different, and we need to include normative referenc=
es
>> to both recovery machines and recovery state variables, which are not in
>> scope for the current doc.
>>
>>
>>
>> I would prefer to stay with a non-normative description of the heuristic=
,
>> and scrub the rest of the document for language that conflicts with QUIC=
.
>>
>>
>> Thanks,
>>
>> --MM--
>> The best way to predict the future is to create it.  - Alan Kay
>>
>> We must not tolerate intolerance;
>>
>>        however our response must be carefully measured:
>>
>>             too strong would be hypocritical and risks spiraling out of
>> control;
>>
>>             too weak risks being mistaken for tacit approval.
>>
>>
>>
>>
>>
>> On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst 
>> wrote:
>>
>> On 19/11/2020 14:20, Ian Swett wrote:
>>
>> I support this work.
>>
>>
>>
>> One question for authors and the group.  This specification could be
>> applied to QUIC, but my memory is that there are some small changes to
>> adapt it correctly.  Would noting those changes be in scope for this
>> document, a separate document in QUIC, or an exercise left to the reader=
?
>> I'd be happy to contribute any QUIC specific text if that's helpful.
>>
>>
>>
>> Thanks, Ian
>>
>> I'd hope the differences for QUIC ARE captured in the RFC series. If the=
y
>> are small, it might be possible to add to this draft perhaps?
>>
>> Gorry
>>
>>
>>
>> On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richard 
>> wrote:
>>
>> As the heuristic is the one major improvement over RFC6937, that is the
>> one area I am especially interested in. Currently working to have PRR in
>> FreeBSD, it would be the best time window to add this heuristig instead
>> of switching between the two variants in a static fashion.
>>
>> I support this and will certainly be reviewing the documents and provide
>> feedback!
>>
>> Richard
>>
>>
>> Am 19.11.2020 um 04:56 schrieb Matt Mathis:
>> > The authors of PRR would like to update PRR to Proposed Standard
>> > status.  This entails introducing a new document as an tcpm work item.
>> >
>> > *Please indicate (non) support and/or comment.*
>> >
>> > For more details see the tcpm meeting materials from IETF 109
>> > minutes:
>> > https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
>> 
>> > slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
>> 
>> >
>> > There were about four "I support this work" remarks at the mic (not
>> > recorded in the minutes), and about as many in the Meetecho chat.
>> >
>> > Abridged IETF/tcpm/PRRbis slides:
>> > --
>> > PRR  recap (RFC6937 experimental)
>> > PRR is a special congestion control effective only during fast recover=
y
>> >
>> >   * When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e=
.g.
>> >     loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
>> >   * When inflight < ssthresh, send at the same or twice the
>> >     delivery_rate (more later)
>> >   * Used by all congestion control modules in Linux during fast recove=
ry
>> >       o Can be more dominant than the actual C.C. for lossy flows
>> >         that=E2=80=99re in fast recovery constantly (e.g. video stream=
ing
>> >         through policers)
>> >
>> > --
>> > Current Status
>> >
>> >   *
>> >
>> >     PRR is widely deployed
>> >
>> >       o
>> >
>> >         At least three major OSs: Linux, Windows, (NetFlix) BSD
>> >
>> >       o
>> >
>> >         Vast majority of Web traffic for years
>> >
>> >   *
>> >
>> >     No changes to algorithms published in RFC 6937
>> >
>> >       o
>> >
>> >         PRR-CRB - Conservative Reduction Bound - strict packet
>> >         conversion during loss recovery
>> >
>> >       o
>> >
>> >         PRR-SSRB - Slowstart Reduction Bound - one extra segment per A=
CK
>> >         during loss recovery
>> >
>> >   *
>> >
>> >     2015 Heuristic to dynamically select which reduction bound
>> >
>> >       o
>> >
>> >         Only use PRR-SSRB when making good forward progress
>> >
>> >           +
>> >
>> >             ACKs that advanced snd.una and report no new losses
>> >
>> >       o
>> >
>> >         Resolves some pathological cases with token bucket policers
>> >
>> >           +
>> >
>> >             CC estimates ssthresh before it can possibly measure the
>> >             token rate
>> >
>> >           +
>> >
>> >             The heuristic makes the best of a bad situation
>> >
>> > --
>> > Tentative path forward
>> >
>> >   *
>> >
>> >     Adopt as a tcpm work item
>> >
>> >   *
>> >
>> >     Update the text
>> >
>> >       o
>> >
>> >         Normative RFC 2119 language
>> >
>> >       o
>> >
>> >         Add MAY use the heuristic...
>> >
>> >       o
>> >
>> >         Trim redundant and obsolete language
>> >
>> >           +
>> >
>> >             RFC 6937 repeats itself and is much longer than necessary
>> >
>> >           +
>> >
>> >             Focus on what an implementer needs to know
>> >
>> >           +
>> >
>> >             Use non-normative references to RFC 6937 for prior
>> >             measurement work, etc
>> >
>> > Thanks,
>> > --MM--
>> > The best way to predict the future is to create it.  - Alan Kay
>> >
>> > We must not tolerate intolerance;
>> >         however our response must be carefully measured:
>> >              too strong would be hypocritical and risks spiraling out =
of
>> > control;
>> >              too weak risks being mistaken for tacit approval.
>> >
>> > _______________________________________________
>> > tcpm mailing list
>> > tcpm@ietf.org
>> > https://www.ietf.org/mailman/listinfo/tcpm
>> 
>> >
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>> 
>>
>>
>>
>> _______________________________________________
>>
>> tcpm mailing list
>>
>> tcpm@ietf.org
>>
>> https://www.ietf.org/mailman/listinfo/tcpm 
>>
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>> 
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--00000000000014dab505b4900681
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Thanks for everybody's reply and support

One additional thing we authors would like more feedback on is =
ECN. The current RFC and proposed -bis do not cover ECN handling. However i=
n Linux, ECN handling uses the same PRR algorithm (aka "CWR" aka =
"cwnd reduction") just like fast recovery. In other words, if RFC=
3168 or DCTCP triggers a congestion window reduction, PRR takes over to con=
trol the transmission until SND_UNA crosses SND.NXT=C2=A0upon reduction.=C2=
=A0

This arrangement was inherented from the original Li=
nux rate-halving design before the PRR was introduced. This paper has detai=
ls=C2=A0https://www.cs.helsinki.fi/research/iwtcp/papers/l=
inuxtcp.pdf

Compared to fast recovery, this part of =
the implementation is far less tested due to the lack of wide classic-ECN u=
sage on the Internet. It's also unclear how other PRR implementations h=
ave applied to ECN handling like Linux does. Adding ECN handling in -bis PS=
 makes me uneasy.

On Thu, Nov 19, 2020 at 12:57 PM Neal Cardwel=
l <ncardwell=3D40google.com@dmarc.ietf.org> wrote:
I am very supportive of=
 updating PRR to PS status and adopting this as a TCPM work item. I will be=
 happy to review the document as well.

The "Tentative path forw=
ard" outlined by Matt sounds good to me.

best,
neal



On Thu, Nov 19, 2020 at 12:37 PM Praveen Balasub=
ramanian <pravb=3D40microsoft.com@dmarc.ietf.org> wrote:











I am supportive of updating PRR to PS status and wil=
l review the document.


=C2=A0


Re. QUIC. The current loss recovery spec for QUIC al=
ready refers to PRR as a MAY and has similar other references to TCP RFCs w=
ithout any additional text for QUIC specific adaptations. The current title=
 of RFC 6937 scopes it to TCP: Proportional
 Rate Reduction for TCP. I also see about 38 hits for the word TCP in the t=
ext. 



=C2=A0




From: tcpm <tcpm-bounces@ietf.org> On Behalf Of =

Matt Mathis

Sent: Thursday, November 19, 2020 8:41 AM

To: Gorry Fairhurst <gorry@erg.abdn.ac.uk>; Ian Swett <ianswett=3D40google.com@dma=
rc.ietf.org>

Cc: tcpm@ietf.org=
 Extensions <tcpm=
@ietf.org>

Subject: [EXTERNAL] Re: [tcpm] Introduce RFC 6937 bis (Proportional =
Rate Reduction) as a tcpm work item? INPUT NEEDED




=C2=A0




If we can get away with a non-normative description =
of the heuristic, QUIC is no problem at all.=C2=A0 =C2=A0 If we need normat=
ive descriptions, then TCP and QUIC have to be different, and we need to in=
clude normative references to both recovery
 machines and recovery state=C2=A0variables, which are not in scope for the=
 current doc.




=C2=A0






I would prefer to stay with a non-normative descript=
ion of the heuristic, and scrub the rest of the document for language that =
conflicts with QUIC.



























Thanks,




--MM--

The best way to predict the future is to create it. =C2=A0- Alan Kay



We must not tolerate intolerance;






=C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must=
 be carefully measured:=C2=A0






=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong=
 would be hypocritical and risks spiraling out of control;





=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak r=
isks being mistaken for tacit approval.
















=C2=A0






=C2=A0






On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst <=
gorry@erg.abdn.ac=
.uk> wrote:










On 19/11/2020 14:20, Ian Swett wrote:<=
/p>







I support this work. 




=C2=A0






One question for authors and the group.=C2=A0 This s=
pecification=C2=A0could be applied to QUIC, but my memory is that there are=
 some small changes to adapt it correctly.=C2=A0 Would noting those changes=
 be in scope for this document, a separate document
 in QUIC, or an exercise left to the reader?=C2=A0 I'd be happy to cont=
ribute any QUIC specific text if that's helpful.






=C2=A0






Thanks, Ian








I'd hope the differences for QUIC ARE captured in the RFC series. If=
 they are small, it might be possible to add to this draft perhaps?



Gorry




=C2=A0






On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richa=
rd <rs.ietf@gmx.at> wrote:






As the heuristic is the one major improvement over R=
FC6937, that is the

one area I am especially interested in. Currently working to have PRR in
FreeBSD, it would be the best time window to add this heuristig instead

of switching between the two variants in a static fashion.



I support this and will certainly be reviewing the documents and provide
feedback!



Richard





Am 19.11.2020 um 04:56 schrieb Matt Mathis:

> The authors of PRR would like to update PRR to Proposed Standard

> status.=C2=A0 This entails introducing a new document as an tcpm work =
item.

>

> *Please indicate (non) support and/or comment.*

>

> For more details see the tcpm meeting materials from IETF 109

> minutes:

> 
https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00<=
br>
> slides: 
https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00

>

> There were about four "I support this work" remarks at the m=
ic (not

> recorded in the minutes), and about as many in the Meetecho chat.

>

> Abridged IETF/tcpm/PRRbis slides:

> --

> PRR =C2=A0recap (RFC6937 experimental)

> PRR is a special congestion control effective only during fast recover=
y

>

>=C2=A0 =C2=A0* When inflight >=3D ssthresh, send at loss_beta*rate_b=
efore_loss (e.g.

>=C2=A0 =C2=A0 =C2=A0loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 =
for Cubic)

>=C2=A0 =C2=A0* When inflight < ssthresh, send at the same or twice t=
he

>=C2=A0 =C2=A0 =C2=A0delivery_rate (more later)

>=C2=A0 =C2=A0* Used by all congestion control modules in Linux during f=
ast recovery

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o Can be more dominant than the actual C.C. =
for lossy flows

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0that=E2=80=99re in fast recovery cons=
tantly (e.g. video streaming

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0through policers)

>

> --

> Current Status

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0PRR is widely deployed

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0At least three major OSs: Linux, Wind=
ows, (NetFlix) BSD

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Vast majority of Web traffic for year=
s

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0No changes to algorithms published in RFC 6937

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-CRB - Conservative Reduction Boun=
d - strict packet

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0conversion during loss recovery

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-SSRB - Slowstart Reduction Bound =
- one extra segment per ACK

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0during loss recovery

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A02015 Heuristic to dynamically select which reductio=
n bound

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Only use PRR-SSRB when making good fo=
rward progress

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ACKs that advanced snd.=
una and report no new losses

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Resolves some pathological cases with=
 token bucket policers

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CC estimates ssthresh b=
efore it can possibly measure the

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0token rate

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The heuristic makes the=
 best of a bad situation

>

> --

> Tentative path forward

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Adopt as a tcpm work item

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Update the text

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Normative RFC 2119 language

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Add MAY use the heuristic...

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Trim redundant and obsolete language<=
br>
>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RFC 6937 repeats itself=
 and is much longer than necessary

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Focus on what an implem=
enter needs to know

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Use non-normative refer=
ences to RFC 6937 for prior

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0measurement work, etc
>

> Thanks,

> --MM--

> The best way to predict the future is to create it. =C2=A0- Alan Kay
>

> We must not tolerate intolerance;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must be carefull=
y measured:

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong would be hy=
pocritical and risks spiraling out of

> control;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks being m=
istaken for tacit approval.

>

> _______________________________________________

> tcpm mailing list

> tcpm@ietf.org
> 
https://www.ietf.org/mailman/listinfo/tcpm

>



_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm






=C2=A0


_______________________________________________


tcpm mailing list


tcpm@ietf.org


https://www.ietf.org/mailman/listinfo/tcpm=





=C2=A0




_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm










_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm



_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm



--00000000000014dab505b4900681--


From nobody Fri Nov 20 14:01:49 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4F9F13A09E7 for ; Fri, 20 Nov 2020 14:01:47 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, T_SPF_PERMERROR=0.01] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TCpLHrCd_M2b for ; Fri, 20 Nov 2020 14:01:45 -0800 (PST)
Received: from mx36-out10.antispamcloud.com (mx36-out10.antispamcloud.com [209.126.121.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1A7803A09DB for ; Fri, 20 Nov 2020 14:01:44 -0800 (PST)
Received: from xse500.mail2web.com ([66.113.197.246] helo=xse.mail2web.com) by mx168.antispamcloud.com with esmtp (Exim 4.92) (envelope-from ) id 1kgETG-000Bpl-9P for tcpm@ietf.org; Fri, 20 Nov 2020 23:01:40 +0100
Received: from xsmtp21.mail2web.com (unknown [10.100.68.60]) by xse.mail2web.com (Postfix) with ESMTPS id 4Cd9TY5DhyzLpb for ; Fri, 20 Nov 2020 14:00:33 -0800 (PST)
Received: from [10.5.2.49] (helo=xmail11.myhosting.com) by xsmtp21.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from ) id 1kgESH-0003ww-Jz for tcpm@ietf.org; Fri, 20 Nov 2020 14:00:33 -0800
Received: (qmail 2130 invoked from network); 20 Nov 2020 22:00:31 -0000
Received: from unknown (HELO [192.168.1.103]) (Authenticated-user:_huitema@huitema.net@[172.58.43.57]) (envelope-sender ) by xmail11.myhosting.com (qmail-ldap-1.03) with ESMTPA for ; 20 Nov 2020 22:00:31 -0000
To: "tcpm@ietf.org Extensions" 
From: Christian Huitema 
Message-ID: <76d7fb6e-aa5e-b8cd-8593-dd9054134315@huitema.net>
Date: Fri, 20 Nov 2020 14:00:32 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-Originating-IP: 66.113.197.246
X-Spampanel-Domain: xsmtpout.mail2web.com
X-Spampanel-Username: 66.113.197.0/24
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.197.0/24@xsmtpout.mail2web.com
X-Spampanel-Outgoing-Class: unsure
X-Spampanel-Outgoing-Evidence: Combined (0.15)
X-Recommended-Action: accept
X-Filter-ID: Mvzo4OR0dZXEDF/gcnlw0cThwnZT+ODcFyeCwCHoUjupSDasLI4SayDByyq9LIhVUZbR67CQ7/vm /hHDJU4RXkTNWdUk1Ol2OGx3IfrIJKywOmJyM1qr8uRnWBrbSAGDPMbZapeW47ess9aQHBKcoLgN zB/4Jkrw1eDLcif59fv0A4NcvQna04qXDndrwNVbU7Tmz6iKnkQL9gqsxD347235Nhqq+/HvroPq 8GSPg+60/QPNqXybIny9WGhadIo/YxFBjiwvt8YGG8KnqhbQXbyme9ldZJ7uNXfg/GfS8fUvP/L5 rCqHDsKZM+xa1iwJX+gRCHfMVnsAk591zk0uilUI+ZL4xWiN8NS6C+dmX6OEdA4u1aThyWrQ/ou2 +v/lmX4Em37yFgrCB6NHRn1g+f3uncIqYSL3lhh5c81YyJqFoLZMmkWsaurVZfvqROaDnDtHb8z5 dpPkEuJ8Snwqla7jUnW3hy14Yji8fo+4xCnSRo4Rcu5Z37rMuDjCny5fE9ykbJ7I9co1MAEE3ruN Xsm8UJsAPvDcVSKtDCYkioPY5Qx4fJOk03R5fJtf/Dv/dkIzS7m4GUpXCY1Y3j3ilZpnfSx1WjM8 Y221ho/pD3/YTgm/NnC/c8YEQ5zoBxO0nqpf6M4tqOKD1AG+cAoIj3DNpq25zsgdGo9JRJ5tAzrv 5pWb82qAoDl3ILGSF0vmDvI0DEihOd7XzCAIcFZdY11677oPXF7r5zsW33ZNlirOC6HI6/h+rh4v 4bH4tTksPbw/1QCLpp89wymCao3Axhqf27VeHOkXe82afzWlDIurGLlRdjzZKLablPido/hYHHAS JNUmoOHSoqgqxfHmWRWcrRhLeB34s3hUb32GO+2IGBo7+fbvq3aSAF+5z35/08QV3No+S2msRDep v5w/kkG0v17AmegcpQ0tml/sN9lmMy/o83jVXTcfb9k0nLWblJy7uxV6dw8jzlsaNZe6hynMJcjx DydxsJEju76A7X1QIVydqXpZ6MHhiKws9Iiut28r9wo4SqUIg8Yh9hAM0n3LLzx/F2gT3wl8JQJv Bho=
X-Report-Abuse-To: spam@quarantine11.antispamcloud.com
Archived-At: 
Subject: [tcpm] Add handling of spurious losses in Cubic RFC update
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 20 Nov 2020 22:01:47 -0000

I just opened an issue in the Github depot: 
https://github.com/NTAP/rfc8312bis/issues/23

Adding handling of spurious losses in Cubic is not very hard: basically, 
reset the epoch parameters to what they would be if there was no loss. 
It would be good if that was documented in the revised RFC.

-- Christian Huitema


From nobody Sun Nov 22 17:47:28 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6069F3A0A90 for ; Sun, 22 Nov 2020 17:47:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -15.7
X-Spam-Level: 
X-Spam-Status: No, score=-15.7 tagged_above=-999 required=5 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id s2SBP4TBoyrj for ; Sun, 22 Nov 2020 17:47:20 -0800 (PST)
Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 26FA13A0A87 for ; Sun, 22 Nov 2020 17:47:20 -0800 (PST)
Received: by mail-ej1-x62e.google.com with SMTP id oq3so21070224ejb.7 for ; Sun, 22 Nov 2020 17:47:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=DldHftOtIdTMZeIp55IOgdofwcGLa5bq6Hs7biB8e1Q=; b=ZBNTrOHehtCuF/zRKsWP6LAiA7bnUBazjXt7EW6OXaJca/uGucFMNEwpo68HoKBAqu wCyuWWGjaMSGCHAUeHuJYbqWQd2FQTQwKP4hN9VVPXEjh+gLCtG8iCFcs6pc1kHwZqJ1 06t1o78rpORwwWaSUXMcwV9bEYyIkdprx+xOacoxfiJfe9odVR3+yV4RSiGC8FoxPeXH VtoQGV9X++sJaySUWObTJY+S83sbjfkfTTBBMQu8n6TqIluEPuvBclDmbrolDQlaTlXb PXpnQKF6Ntr0jSWa9oQlRfWQ9OrtNneHU/fzY2RsK8lghuRQtSzMOeHeNvkQIxRCkIpM 95dA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=DldHftOtIdTMZeIp55IOgdofwcGLa5bq6Hs7biB8e1Q=; b=E2Ntx+Pd17Jf7J/fz0xjZ3WOVDxO4V7hARuTT2Bjo7s1r3IfJvKBfWq461DP4V+a6N Ws+ky5q4sLIPQO9shgpcastFQhufp7aU0Nl0aeW8JiU73/Km+KF4U4xx2XOqAqndwdjm bIp4o9ev48gHtkzeQRFQ+XGNqGmb/bCQwCeTY1tW3dhCZjYUOHlC/KL1eNTLmEi/HptY NX7XMOS/onUlYW9/JHc0rToLaSlsBoAqyxeHvXj2cgPI5ScCKIgvB2e0rlpuFNQ8AROd 8yta+YbcTW9pMzi6KqqfNwiSxcOMGaRgYJiKCsuagGV3nEiOCwQEfvmGq73RgaWru5uE 9AOQ==
X-Gm-Message-State: AOAM532GMs7VQ2pF7BggwQYvfPYjABoQckqlgY8ye6Y6YeElAGQQgvwC sq/SQSneVyYnAFtTwZpJX70zdOm/Q/HKVcAjp4TjNQ==
X-Google-Smtp-Source: ABdhPJwBxS0Myi3pcNLE6b72HwU2eGhk+BrxfLdyAwK0JNwBPLIdS9/ZwsI52K33EimlTSAF79LK54SbuWl9RYifz4Y=
X-Received: by 2002:a17:906:2b4e:: with SMTP id b14mr45100804ejg.354.1606096038005;  Sun, 22 Nov 2020 17:47:18 -0800 (PST)
MIME-Version: 1.0
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com>     
In-Reply-To: 
From: Kevin Yang 
Date: Sun, 22 Nov 2020 20:47:06 -0500
Message-ID: 
To: "Scheffenegger, Richard" 
Cc: Yuchung Cheng , "tcpm@ietf.org Extensions" , Eric Dumazet , iccrg IRTF list , Neal Cardwell 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 23 Nov 2020 01:47:23 -0000

Thank you Bob and Richard, for the nice feedback!

To Bob's feedback:
>You could define your new option solely with the last two negotiation fiel=
ds
>Because the existing timestamp fields are already defined as opaque...
That's a good point. We actually did consider this in our early
iterations among the authors.
Some drawbacks from relying on TSopt:

1. It has 2 more bytes for kind and len. As you pointed out, it might
be a way to save
on established segments, but not on SYNs.

2. In addition to TSval (in old TSopt), the new ETSopt needs to
include another timestamp
to be the "ETSval" in usec. We can not fill a usec value in TSval(of
old TSopt). Otherwise,
because the passive side may not support this new protocol, it treats
TSval in msec and may trigger
false positive drops by PAWS on the passive side.

I reply Richard's feedback inline below:

On Tue, Nov 17, 2020 at 8:19 AM Scheffenegger, Richard  wro=
te:
>
> Just to capture what Bob and I tried to convey:
>
> There may be a way to extend TSOpt in a backwards-compatible way,
> without increasing the size of the option during the SYN handshake, nor
> with the need to have both 7323 TSopt and ETSopt present simultaneously
> in the SYN and SYN/ACK.
>
> Please have a look at
> https://tools.ietf.org/html/draft-scheffenegger-tcpm-timestamp-negotiatio=
n-05
>
> Which was the signaling / handshake part to
>
> https://tools.ietf.org/html/draft-trammell-tcpm-timestamp-interval-01
>
> As the latter part didn't get much traction (to signal the TS
> granularity and even accuracy), these two didn't move forward.
>
>
> As the SYN TSopt ecr value is currently unused, and the client knows the
> initial value for TSval (which should be reflected in the SYN/ACK from
> the server), you can use 32 bits there for a feature declaration and
> negotiation.
>
> The high level approach there was to use the unused TSecr field (should
> be 0 for 7323/1323), and overlay (xor) in the SYN/ACK's TSopt ecr field
> the client's TSopt val with the servers 32-bit negotiation field... As
> the client can retain knowledge of the initial TSopt value (there should
> be a randomized initial value), the client can extract the server side
> options by xor'ing it's known initial value when the SYN/ACK is
> received. Or fall back to default 7323 operation, if the result of the
> XOR is zero...
>

I agree that the TSecr in  might be used for other purposes. But I'm
not sure if we can alter the TSecr in  by XOR other info.
One specific example I came up with is that a sender may send out many
SYNs because of timeout,
then when it receives a , it does not know which TSval should
be used for xor op.

>
> The major issue would be that TSopt with a length !=3D 10 may be
> removed/blocked by certain middleboxes, but a new TCP option has similar
> deployment issues.
>
> As Bob mentioned in another thread just now, TCP SYN option space is a
> very scarce resource, and making efficient use of it should matter...
>
>
> Other comments: Graph1 (page 4) has "TSval=3DX" rather than "ETSval=3DX".
> Unless you plan to reuse the TSopt in an extensible way (with the same
> codepoint, but differentiated by the length of the option), that may
> need to be addressed to disambiguate things; You may also want to have
> both ETSopt and TSopt in the SYN present, for backwards compatibility if
> a separate codepoint is used.

Good point, we should revise it to ETSval for disambiguation.

> If you want to make use of the timestamp-negotiation design, all of the
> extra data shown on page 6 can be stuffed into the TSecr or ETSecr
> field, alleviating the need of 4 additional SYN option bytes.
>
> It could be used almost as a drop-in use of timestamp-negotiation, as
> long as it can be guaranteed, that at least one bit is set in the 32bit
> field (e.g. setting the reserved bit to 1).

In this way we need to add another "ETSval'' in usec if we use the
RFC7323/1323 TSopt,
the reason is explained above.

> EcrDelUnit: I think the codepoint assignment could be improved (but this
> is really nit-picking; just that a broken implementation may send out
> TSopt with len=3D12, but initialized-to-zero superflous bytes...
> 0: invalid
> 1: milliseconds [2^20 nanoseconds?]
> 2: microseconds [2^10 nanoseconds?]
> 3: reserved (nanoseconds)

Thanks for the nice suggestion. I agree we probably should have
all-zero codepoint to be
"no information". For 2^k nanosec unit, I guess that is for bit-op
convenience and performance?

>
> MaxAckDel: 16 bits/microseconds gives a maximum of 65.5 ms; some (older)
> stacks are using values as high as 200ms; If such a stack is updated, it
> is a good reason to reduce the DelAck value used in that stack, so this
> range appears properly selected (FreeBSD recently reduced from 100ms to
> 40ms).
>
>
>
> Best regards,
>     Richard
>
>
> Am 14.11.2020 um 20:35 schrieb Yuchung Cheng:
> > On Sat, Nov 14, 2020 at 8:45 AM Kevin Yang
> >  wrote:
> >>
> >> Thank you Yoshi, very good questions.
> >>
> >> 1: Our goal is to substitute the TSopt. So we design this new ETSopt n=
ot to depend on TSopt.
> >>      I guess this also answers your question 4, the intent is for the =
standard track.
> >>
> >> 2: I think we should only update TS.recent when we assume the incoming=
 segment is valid.
> >>      Otherwise, if the incoming is a real old one, we discard it but u=
pdate TS.recent;
> >>      then a new segment arrives, we may reject this new seg because TS=
.recent > seg.TSval.
> >>
> >> 3: Your understanding is correct, and in the draft we are not suggesti=
ng to use NetworkRTT
> >>      measurements for RTO calculation.
> >>
> >> 4: answered by 1.
> > We'll clarify better in the upcoming draft revision: the draft
> > intends to transition from exp-id to  an official option kind number
> > from IANA upon becoming
> > an RFC. But we're following the advice of
> > https://tools.ietf.org/html/rfc6994 to allow larger scale experiments
> > for hosts not completely under control of a single entity/network. For
> > example guest VM TCP communications inside Cloud Data-center networks.
> >
> >
> >>
> >> Kevin
> >>
> >> On Sat, Nov 14, 2020 at 7:13 AM Yoshifumi Nishida =
 wrote:
> >>>
> >>> Hi,
> >>> Thanks for preparing the draft. I basically like the idea in the docs=
 and have several comments.
> >>>
> >>> 1: I am wondering if we really need to have 32 bits TSval and TSecr i=
n ETS option in SYN when TSopt is also sent?
> >>>      For example, If TSopt has 1 ms granularity, I think ETS option w=
ill need to carry only small fractions of timestamp.
> >>>
> >>> 2: "To prevent a false positive PAWS rejection of a valid segment, an=
 ETS receiver MUST skip the PAWS check"
> >>>      -> This is one possible approach, but just skipping PAWS doesn't=
 sound very good to me.
> >>>           How about discarding the segment while updating TS.recent a=
s an alternative? If the segment is not an old one, it will be retransmitte=
d and accepted.
> >>>           I think this is a bit more conservative.
> >>>
> >>> 3: It is not very clear for me how measuring NetworkRTT can contribut=
e to improving RTO.
> >>>      Or, are there some other ways to utilize NetworkRTT?
> >>>
> >>> 4: Just out of curiosity, is intended status standard track or experi=
mental? ExID is usually used for exp or info docs.
> >>>
> >>> Thanks,
> >>> --
> >>> Yoshi
> >>>
> >>> On Mon, Nov 2, 2020 at 4:26 PM Yuchung Cheng  wrote:
> >>>>
> >>>> Hi tcpm & iccrg,
> >>>>
> >>>> We are proposing a new TCP timestamp option to measure the network d=
elay more precisely (e.g. excluding delayed ACK effects) for congestion con=
trol, e.g. Swift published SIGCOMM 2020. The precision of the measurements =
can potentially be further enhanced by NIC hardware timestamps. We are work=
ing on a reference implementation for Linux as well.
> >>>>
> >>>> We'll also present it in the upcoming tcpm meeting. Feedbacks are ve=
ry welcome.
> >>>>
> >>>> ---------- Forwarded message ---------
> >>>> From: 
> >>>> Date: Mon, Nov 2, 2020 at 3:29 PM
> >>>> Subject: New Version Notification for draft-yang-tcpm-ets-00.txt
> >>>> To: Eric Dumazet , Yuchung Cheng , Neal Cardwell , Kevin Yang (Yudong) 
> >>>>
> >>>>
> >>>>
> >>>> A new version of I-D, draft-yang-tcpm-ets-00.txt
> >>>> has been successfully submitted by Kevin (Yudong) Yang and posted to=
 the
> >>>> IETF repository.
> >>>>
> >>>> Name:           draft-yang-tcpm-ets
> >>>> Revision:       00
> >>>> Title:          TCP ETS: Extensible Timestamp Options
> >>>> Document date:  2020-11-02
> >>>> Group:          Individual Submission
> >>>> Pages:          13
> >>>> URL:            https://www.ietf.org/archive/id/draft-yang-tcpm-ets-=
00.txt
> >>>> Status:         https://datatracker.ietf.org/doc/draft-yang-tcpm-ets=
/
> >>>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-yang-tcp=
m-ets
> >>>> Htmlized:       https://tools.ietf.org/html/draft-yang-tcpm-ets-00
> >>>>
> >>>>
> >>>> Abstract:
> >>>>     This document presents ETS: an Extensible TimeStamps option for =
TCP.
> >>>>     It allows hosts to use microseconds as the unit for timestamps t=
o
> >>>>     improve the precision of timestamps, and advertise the maximum A=
CK
> >>>>     delay for its own delayed ACK mechanism.  Furthermore, it extend=
s the
> >>>>     information provided in the [RFC7323] TCP Timestamps Option by
> >>>>     including the receiver delay in the TSecr echoing, so that the
> >>>>     receiver of the ACK is able to more accurately estimate the port=
ion
> >>>>     of the RTT that resulted from time traveling through the network=
.
> >>>>     The ETS option format is extensible, so that future extensions c=
an
> >>>>     add further information without the overhead of extra TCP option=
 kind
> >>>>     and length fields.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Please note that it may take a couple of minutes from the time of su=
bmission
> >>>> until the htmlized version and diff are available at tools.ietf.org.
> >>>>
> >>>> The IETF Secretariat
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> tcpm mailing list
> >>>> tcpm@ietf.org
> >>>> https://www.ietf.org/mailman/listinfo/tcpm
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpm
> >


From nobody Mon Nov 23 03:15:15 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E510D3A0803 for ; Mon, 23 Nov 2020 03:15:12 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.003
X-Spam-Level: 
X-Spam-Status: No, score=0.003 tagged_above=-999 required=5 tests=[SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5de5r1BOLrtf for ; Mon, 23 Nov 2020 03:15:11 -0800 (PST)
Received: from wp513.webpack.hosteurope.de (wp513.webpack.hosteurope.de [IPv6:2a01:488:42:1000:50ed:8223::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 095643A07F9 for ; Mon, 23 Nov 2020 03:15:10 -0800 (PST)
Received: from p200300dee707b600cc0e7695c5e06a36.dip0.t-ipconnect.de ([2003:de:e707:b600:cc0e:7695:c5e0:6a36]); authenticated by wp513.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1kh9oI-0003fj-M6; Mon, 23 Nov 2020 12:15:06 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
From: Mirja Kuehlewind 
In-Reply-To: 
Date: Mon, 23 Nov 2020 12:15:06 +0100
Cc: Bob Briscoe , tcpm IETF list , Richard Scheffenegger 
Content-Transfer-Encoding: quoted-printable
Message-Id: <89EBF75E-B224-43CE-A7D5-B6F91C154E02@kuehlewind.net>
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net>   
To: Yoshifumi Nishida 
X-Mailer: Apple Mail (2.3608.120.23.2.4)
X-bounce-key: webpack.hosteurope.de;ietf@kuehlewind.net;1606130111;912e2d19;
X-HE-SMSGID: 1kh9oI-0003fj-M6
Archived-At: 
Subject: Re: [tcpm] Possible error in accurate-ecn
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 23 Nov 2020 11:15:13 -0000

Hi Bob, hi all,

Sorry for my late reply. See inline.

> On 12. Nov 2020, at 23:41, Yoshifumi Nishida  =
wrote:
>=20
> Hi Bob,
>=20
> On Wed, Nov 11, 2020 at 3:18 PM Bob Briscoe  =
wrote:
> Yoshi,
>=20
> On 11/11/2020 09:37, Yoshifumi Nishida wrote:
>>=20
>>=20
>> On Tue, Nov 10, 2020 at 2:08 AM Mirja Kuehlewind =
 wrote:
>> Hi Bob,
>>=20
>> Please see below.
>>=20
>> > On 10. Nov 2020, at 01:31, Bob Briscoe  =
wrote:
>> >=20
>> > Mirja, Richard, Ilpo, tcpm list,
>> >=20
>> > I've just been reading through the accurate-ecn draft to =
double-check. I think there's a problem with the following text...
>> >=20
>> > 3.2.2.5.1.  Data Receiver Safety Procedures
>> >=20
>> >    An AccECN Data Receiver:
>> >=20
>> >    o  SHOULD immediately send an ACK whenever a data packet marked =
CE
>> >       arrives after the previous [data] packet was not CE.
>> >=20
>> >    o  MUST immediately send an ACK once 'n' CE marks have arrived =
since
>> >       the previous ACK, where 'n' SHOULD be 2 and MUST be no =
greater
>> >       than 6.
>> >=20
>> > ...
>> >    For the avoidance of doubt, the change-triggered ACK mechanism =
is
>> >    deliberately worded to solely apply to data packets, and to =
ignore
>> >    the arrival of a control packet with no payload, because it is
>> >    important that TCP does not acknowledge pure ACKs. =20
>> >=20
>> >=20
>> > In the first bullet, I think it doesn't matter whether the previous =
packet marked CE was a data packet or a pure ACK (i.e we should remove =
the second occurrence of 'data' that I have put in [square brackets].
>>=20
>> I believe it does matter. If the previous packet was a pure ACK and =
was CE marked, you didn=E2=80=99t send an ACK and so you should =
immediately send one now. If the previous packet was a data packet and =
CE marked, you don=E2=80=99t need to send an immediate ACK because you =
already did this with the previous packet. However, it text might be a =
but ambiguous because what is meant is =E2=80=9Cif the previous packet =
was a data packet and CE marked=E2=80=9D. So this does not apply if we =
e.g. have a CE-marked data packets, then a pure ACK that is not CE =
marked, and then another CE-marked data packet. =20
>>=20
>> >=20
>> > The second bullet doesn't consider the possibility that the 'n'th  =
CE mark might arrive on a pure ACK. Then, the wording as it stands says =
the Data Receiver MUST immediately ACK a pure ACK. I know TCP never ACKs =
a pure ACK, but I'm not actually sure it does any harm to do so in this =
case (it cannot cause an infinite loop of ACKs). However, given it would =
be unorthodox, we maybe ought to rule it out by rewording anyway?
>>=20
>> even though it won't cause an infinite loop, can they be dup acks? If =
so, it doesn't look good to me especially when early retransmit is =
activated.
>=20
> [BB] I'm not expert in TCP loss recovery but, surely if TCP A has =
nothing outstanding, it cannot consider receipt of dup acknowledgements =
from TCP B as a sign of loss? Nonetheless, there would be a chance that =
such a dup acknowledgement might be in flight when TCP A happened to =
have just sent some more data, and they crossed in flight.=20
>=20
> To avoid this case, it would be possible for TCP B to send pure ACK(s) =
with SEG.SEQ =3D SND.NXT-1 (using the same trick as keepalive probes). =
Being out of window, each of these pure ACKs would elicit a further pure =
ACK from TCP A. However, that would acknowledge SND-NXT, so the loop =
would stop there. Nonetheless, if TCP B started sending data again, and =
if TCP B's data crossed the pure ACKs that it had recently elicited from =
TCP A, once these pure ACKs arrived at TCP B, I think it would treat =
them as DupACKs.
>=20
> It's seems possible for TCP B to recognize that it elicited these =
DupACKs from TCP A, and to therefore work out that it could ignore them.=20=

>=20
> I've kept on digging into this particular obscure corner of ACK =
congestion control, only 'cos I wanted to see if there was a useful end =
to this hole (so it would be worth the AccECN RFC allowing others to dig =
there later, rather than ruling it out).
>=20
> Nonethless, the AccECN draft itself doesn't have to go anywhere near =
this subject - it can leave all that for a potential future draft about =
using AccECN for ACK congestion control.=20
>=20
> I personally prefer not to specify very detailed logic for ACK CC here =
unless it is crucial for the doc.=20
> Since the doc is aiming to be a PS, I think it would be better to =
avoid risks for nasty corner cases caused by some tricky techniques.=20

I would leave anything on ACK CC also for a separate draft.

>> I think we also can leave this as it is because if that =E2=80=99n=E2=80=
=99 packet is a pure ACK that still means that you have unacked data =
packets with CE marks and you should trigger an ACK for those packet now =
(rather than waiting for the delayed ACK timer to expire). However this =
case is less important. Also             we should probably make sure =
that this doesn=E2=80=99t apply if there are only pure ACKs with CE =
marks, maybe by add =E2=80=9Cif unacknowledged data are outstanding=E2=80=9D=
 or something.
>> =20
>> "can send a pure ack on a pure ack when the ack value advances"? =20
>=20
> [BB] Yes. That's a fairly safe thing to require.
> I think the best strategy here is to require an immediate ACK in =
response to a data packet if there are at least 'n' CE marks to be =
reported. And to extend that requirement to immediately ACKing a pure =
ACK if the ACK value has advanced since the last ACK. But just to say =
nothing about ACKing pure ACKs in other cases, so we don't preclude a =
case that can be made safe in future (e.g. with something like the =
keepalive approach above).
>=20
> If you (Yoshi), Mirja and others agree, let's try to work out the =
wording of this second bullet.=20
>=20
> Works for me. If you could have proposed texts, I will take a look.=20
> --
> Yoshi

Works for me too.

On Richard=E2=80=99s point that this could lead to synchronicity lost of =
the AccECN
counters: Only if you don=E2=80=99t use the option. And as long as ACK =
CC is not implemented, it=E2=80=99s probably also we can live with.

Mirja



From nobody Mon Nov 23 03:39:01 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 715083A08E7; Mon, 23 Nov 2020 03:38:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.003
X-Spam-Level: 
X-Spam-Status: No, score=0.003 tagged_above=-999 required=5 tests=[SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id S_tFqCrThHGf; Mon, 23 Nov 2020 03:38:56 -0800 (PST)
Received: from wp513.webpack.hosteurope.de (wp513.webpack.hosteurope.de [IPv6:2a01:488:42:1000:50ed:8223::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C36F03A08D6; Mon, 23 Nov 2020 03:38:55 -0800 (PST)
Received: from p200300dee707b600cc0e7695c5e06a36.dip0.t-ipconnect.de ([2003:de:e707:b600:cc0e:7695:c5e0:6a36]); authenticated by wp513.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1khABG-0001dP-PN; Mon, 23 Nov 2020 12:38:50 +0100
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\))
From: Mirja Kuehlewind 
In-Reply-To: 
Date: Mon, 23 Nov 2020 12:38:50 +0100
Cc: Yoshifumi Nishida , tcpm IETF list , "Scheffenegger, Richard" , Michael Tuexen , "tcpm-chairs@ietf.org" 
Content-Transfer-Encoding: quoted-printable
Message-Id: 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de> <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net> <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de> 
To: Neal Cardwell , "Scharf, Michael" , Bob Briscoe 
X-Mailer: Apple Mail (2.3608.120.23.2.4)
X-bounce-key: webpack.hosteurope.de;ietf@kuehlewind.net;1606131536;35050fbb;
X-HE-SMSGID: 1khABG-0001dP-PN
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 23 Nov 2020 11:38:59 -0000

Hi all,

I think I also support two options as this is the most simple solution.=20=


We probably need to look at section 3.2.3.2. (Path Traversal of the =
AccECN Option) again, as I think this section assumes mostly one option. =
Or I guess we could say that only one option should be used for a =
connection.

Regarding use of the option length for extensibility. I know new options =
are hard to deploy but that=E2=80=99s the extensibility mechanism we =
have with TCP. Not sure we want to invent a new now. If we think we need =
more extensibility for AccECN for any reason, we should specify valid =
option length values that include a flags field now.

Mirja



> On 18. Nov 2020, at 01:03, Neal Cardwell  wrote:
>=20
> As someone who contributes to a TCP implementation, I would put in a  =
vote for the "Two Option Kinds" proposal outlined in slide 6 here:
>=20
>  =
https://datatracker.ietf.org/meeting/109/materials/slides-109-tcpm-more-ac=
curate-ecn-feedback-in-tcp-ecn-adding-explicit-congestion-notification-01#=
page=3D6
>=20
> This approach has the virtue of simplicity, which hopefully may help =
avoid middlebox bugs, speed standardization and development, and ease =
the process of maintaining fast and stable implementations.
>=20
> neal
>=20
>=20
> On Tue, Nov 17, 2020 at 9:25 AM Scharf, Michael =
 wrote:
> Inline some comments [ms].
>=20
> =20
>=20
> Given that I don=E2=80=99t ask for any text chances, I=E2=80=99ll not =
follow-up forever in this discussion. I don=E2=80=99t own running code =
and a lot of encoding details mainly affect an implementation.
>=20
> =20
>=20
> IMHO the rest of the WG has to think about the option design more than =
me. So, this is mostly about triggering a discussion=E2=80=A6
>=20
> =20
>=20
> From: Bob Briscoe =20
> Sent: Tuesday, November 17, 2020 1:20 PM
> To: Scharf, Michael ; Yoshifumi =
Nishida 
> Cc: Michael Tuexen ; tcpm-chairs@ietf.org; =
Mirja Kuehlewind ; Scheffenegger, Richard =
; tcpm IETF list 
> Subject: Re: AccECN field order
>=20
> =20
>=20
> Michael,
>=20
> On 17/11/2020 08:10, Scharf, Michael wrote:
>=20
> Bob,
>=20
> =20
>=20
> I am fine with the two option kinds proposed in -13, but I don=E2=80=99t=
 buy your arguments why this encoding is better than others.
>=20
> =20
>=20
> Most importantly, I don=E2=80=99t think that your =E2=80=9Cforward =
compatibility=E2=80=9D argument is a very compelling reason for two =
codepoints. All proposals I am aware of have their pros and cons. And I =
am not aware of a really comprehensive discussion. At least neither the =
e-mail below nor today=E2=80=99s meeting discuss all tradeoffs I would =
investigate.
>=20
>=20
> [BB] You're right - the discussion below is useful and necessary.
>=20
>=20
>=20
> =20
>=20
> For instance, the benefit of the length encoding variant would be to =
consume only one TCP option codepoint now. If we decided later to that a =
flag byte is needed (say, at the end), a follow-up proposed standard =
could specify another option format with that flag byte, using a second =
codepoint.
>=20
>=20
> [BB]=20
> * The new codepoint would require years to deploy (including in =
proxies) before it works e2e.
>=20
> =20
>=20
> [ms] That may depend *a lot* on the environment. In environments where =
ECN can get deployed today, say in datacenters, rolling out a new kernel =
version may perhaps not be the biggest operational challenge. Also, =
proxies may not be a significant problem in some of those cases. =
Deployment roadmaps are very hard to predict.
>=20
>=20
> * Whereas the 'forward compatibility' approach tries to ensure that =
AccECN implementations that are being built today will be ready to =
interwork with a future extension.=20
> * In contrast, if your hypothetical new AccECN codepoint did need to =
be deployed later, it would need to include a fall-back to the current =
AccECN option when one peer doesn't understand the new codepoint.
>=20
> =20
>=20
> [ms] This is a reasonable point, albeit not a particularly strong one, =
given that the option is not really required for basic interoperability. =
Also, a hypothetical follow-up standard could include a way to =
explicitly request the option=E2=80=A6
>=20
>=20
> * Fall-back would need to avoid a second round of handshake (given =
this is for low latency), which would only work reliably with an option =
on the SYN (because the 3rd ACK is unreliable)
>=20
> =20
>=20
> [ms] I can=E2=80=99t follow. If an endpoint is unhappy with getting =
AccECN option =E2=80=9CA=E2=80=9D from a peer, I could send a =E2=80=9Cple=
ase send me the better AccECN option B=E2=80=9D after the 3-WHS. It =
could be an upgrade, not a fallback. Also, at this point we don=E2=80=99t =
know what option B would be and in which context it would be needed. =
Maybe other protocol features would require negotiation, too. IMHO it is =
a solvable problem.
>=20
>=20
> * AccECN has so far avoided using any SYN option space, which is =
important given that is the scarcest resource.
>=20
> =20
>=20
> [ms] Yep, but as long as no proposal needs SYN options, this is =
nothing that would make one solution better than another.
>=20
> These days, extending TCP is like playing chess - we have to think =
multiple moves ahead!
>=20
> =20
>=20
> [ms] There is one fundamental difference between chess and TCP: In =
chess you precisely know the current situation and can predict all =
potential moves in advance (albeit there are a lot). However, we don=E2=80=
=99t know the future evolution of TCP, ECN and the Internet as a whole, =
nor future requirements. If we try to engineer a TCP extension for some =
potential future, we may figure out later that reality is different from =
what we designed for. As much as I appreciate your attempts to predict =
the future, it would be a big surprise to me if you could correctly =
predict what TCP will be in 10 years from now. So, instead of trying to =
write specifications that take into account future requirements, which =
are in fact just unknown, it is more important for me to write =
effective, efficient and secure specifications for what is currently =
known for sure.
>=20
> =20
>=20
> [ms] I *don=E2=80=99t* know for sure whether the current AccECN option =
will be needed, but I see quite some evidence that support of the AccECN =
option(s) is not a top priority these days. That is also something to =
think about when designing the details.
>=20
> =20
>=20
> I need to learn to articulate onto the list all my reasons for =
rejecting certain parts of the design space. Sorry about that.
>=20
> Yes, I admit, the current proposal is at the cost of burning two TCP =
option codepoints. However, you said yourself when you proposed this =
approach, that they are not (currently) a scarce resource. Certainly not =
as scarce as option space. I just checked: roughly 15% of the option =
codepoint allocation space has been used so far. So that's where I would =
suggest we compromise first.
>=20
> [ms] I made a similar calculation some years ago and came to a similar =
conclusion. Investing two codepoints for AccECN is something we can =
probably afford. But nonetheless we have to reason carefully why it is =
really worth the effort.
>=20
>=20
> =20
>=20
> Regarding the resulting codepoint consumption (two option kinds) and =
the addition of a flag byte, this approach would basically end up with =
the same result as the current proposal in -13 plus some hypothetical =
future addition leveraging the length field beyond the currently known =
use.
>=20
> =20
>=20
> And, yes, the length encoding variant may be less flexible and/or =
consume some more bits in some cases, but on the plus side it only needs =
one codepoint now =E2=80=93 and given that it is entirely unclear =
whether any AccECN option will indeed be widely used in future, this is =
a big plus. The fact that implementers are quite silent on the option =
design is not a good sign.
>=20
>=20
> [BB] Here's another possibility then.=20
>=20
> a) [K|L| EE1B | ECEB | EE0B ]
> b) [K|L| EE1B | ECEB | EE0B |F]    if F =3D 0bXXXXXXX0
> c) [K|L| EE0B | ECEB | EE1B |F]    if F =3D 0bXXXXXXX1
>=20
> We'd have to decide which order alternative (a) takes. I've picked =
EE1B first only 'cos you're right that there is not currently as much =
pressure (on the public Internet) for AccECN with ECT(0).
>=20
> =3D=3DLegend=3D=3D
> K:    Kind (1B)
> L:    Length (1B)
> ExxB: Echo xx byte counter
> F:    Flags (1B)
>=20
> Alternatives a) and b) are equivalent, at least while the only =
allocated flag is for field order.
> However, alternative b) is available in case other flags are allocated =
in future
> The flags byte can only be omitted if (L % 3 =3D=3D 2).
> The flags byte is considered present if (L % 3) =3D=3D 0.
>=20
> Forward compatibility: Options with (L % 3 =3D=3D 1) MUST be assumed =
to include a flags byte, and current implementations ignore the last =
byte.
>=20
> The flags byte is optional to implement (even if the AccECN option is =
implemented)
> If the server includes a flags byte on the SYN/ACK but the client does =
not include one on the 3rd ACK or the first data packet, the server =
assumes the client does not implement the flags byte and uses only =
alternative a) for the remainder of the connection.
>=20
>=20
> [ms] I am not sure if we need that, but I take this as existence proof =
that other encodings are possible. Personally, I think that other =
solutions would be doable if you took one bit from one or multiple =
counters for format encoding. At least for some of the byte counters I =
don=E2=80=99t think taking one bit for other purposes would result in an =
Internet congestion collapse=E2=80=A6 But, I have already said that the =
-13 proposal works for me, so I=E2=80=99ll not try to come up with =
another variant myself.
>=20
> =20
>=20
> To me, one potential difference between two proposals would be =
incremental deployment. The proposal in -13 only has an advantage if =
middleboxes such as firewalls will indeed pass TCP options with a format =
that contains content beyond the (first) Accurate ECN standard (i.e., =
currently unused length values). IMHO it is too early to know whether =
firewalls would indeed allow this in future.
>=20
> =20
>=20
> =46rom a security perspective, it is not clear to me whether allowing =
arbitrary unspecified bytes in a TCP option is a good idea *at all*. It =
will be interesting to hear the opinion from SEC area on that. =
Personally, I am not convinced that this really makes sense, but I my =
concerns are not strong enough to formally push back. I=E2=80=99ll leave =
it to others to think about whether this is a bug or a feature.
>=20
>=20
> [BB] Well, let's first try to deal with security ourselves:
> * Octets that are explicitly declared as part of an option cannot be =
used for a buffer overflow attack. I don't really need to, but I could =
add the following text to the forward compatibility wording:=20
>     A receiver considers octets beyond those it uses, but within the =
specified length, as if they are padding.=20
> * And such octets cannot be any different from the current ability of =
a sender to add padding. So there's no new attack possible here.
> * And there's no need to specify a max length for any AccECN Option, =
which would just unnecessarily limit flexibility.
>=20
> [ms] I have more thought about the covert channel that some =
unspecified bytes in a standards track TCP option could offer=E2=80=A6 =
In particular if the AccECN option gets widely deployed and used, would =
some malicious users find those bytes useful? For what? For instance, =
what in the AccECN protocol design prevents use of these =E2=80=9Cpadding=E2=
=80=9D bytes as super-cookie? To me, it is much more important to think =
about abuse of whatever we standardize *now*, instead of engineering for =
some entirely unknown future. I am not a security expert, but I suggest =
a careful analysis of any security implications of the AccECN option. We =
seldomly specify new TCP options and malicious users will certainly look =
for =E2=80=9Copportunities=E2=80=9D in the spec. So far, I don=E2=80=99t =
know whether that whole idea of forward compatibility is a feature or a =
bug.
>=20
> =20
>=20
> Michael
>=20
> =20
>=20
> Maybe one lesson learnt is that the document could have a =
non-normative appendix that explains the rationale for the finally =
picked TCP option encoding. That may also help if there are further =
questions whether two codepoints are really required, e.g. by the IESG =
(if two codepoints are still the design after WGLC). At least for past =
TCP option codepoint allocations I recall some discussions late in the =
IETF process. In those past cases, good arguments in an appendix and =
running code has helped a lot.
>=20
>=20
> [BB] I can do that. Appx B is already similar, giving Design Rationale =
for the ACE field.
>=20
>=20
> Bob
>=20
>=20
>=20
> =20
>=20
> Michael
>=20
> =20
>=20
> =20
>=20
> =20
>=20
> From: Bob Briscoe =20
> Sent: Tuesday, November 17, 2020 6:10 AM
> To: Scharf, Michael ; Yoshifumi =
Nishida 
> Cc: Michael Tuexen ; tcpm-chairs@ietf.org; =
Mirja Kuehlewind ; Scheffenegger, Richard =
; tcpm IETF list 
> Subject: Re: AccECN field order
>=20
> =20
>=20
> Michael,
>=20
> On 16/11/2020 17:36, Scharf, Michael wrote:
>=20
> Bob,
>=20
> =20
>=20
> One proposal using the length field with *one option codepoint only* =
is detailed in: =
https://mailarchive.ietf.org/arch/msg/tcpm/zo-1OR0nRfhHocX8yvTvpC4BNMo/
>=20
> =20
>=20
> It is the third option mentioned in this e-mail. One example would be =
to use option length values 5/8/11 for one encoding type and option =
length values 6/9/12 for the other encoding type (i.e., order of =
fields). Or one could use some other combination of length values =E2=80=93=
 the only requirement is that a certain value for the option length is =
only used by one of the option formats. In that approach, the value of =
the length field would thus directly describe the encoding of the =
option. Unless I miss something, this would work and it would just =
require one option codepoint.
>=20
> =20
>=20
> Thus, alternatives to two option codepoints exist and I have explained =
them on the list in March 2020.
>=20
>=20
> OK, sorry, yes, I remember this now.
>=20
> As I will explain in the AccECN status update talk today in virtual =
Bangkok, the draft has made provision for different length values than =
5/8/11. It says existing implementations MUST accept length values other =
than those currently defined. But then read in as many whole 3-byte =
fields as they can.
>=20
> This can be used to add a flags byte on the end in future, for =
extensibility. Or any other form of extensibility the WG might decide in =
the future.
>=20
> I know a flags byte at the end seems odd compared to at the beginning. =
But (if decided it's needed in future) it's reasonably easy to implement =
by reading the whole option, then processing the last byte, before =
reading the rest of the option.
>=20
> I believe you will agree that this is a better way to utilize =
different lengths.
>=20
> And thank you for repeatedly emphasizing that you're happy with the =
2-kind scheme, or other alternatives.
>=20
>=20
> Bob
>=20
>=20
>=20
>=20
> =20
>=20
> Anyway, I don=E2=80=99t really care how the options are encoded as =
long as the receiver doesn=E2=80=99t need per-connection state for =
decoding a TCP option. So, personally, I would be fine with using e.g. =
the length field as described in my old e-mail. Or an additional flag =
byte. And one could come up with further encodings, e.g., by using one =
or a few bits as a short =E2=80=9Ctype=E2=80=9D field for each counter. =
This is all about protocol engineering. And all these variants have =
their pros and cons.
>=20
> =20
>=20
> I am also fine with using two option codepoints as specified in -13; =
this is probably the approach that consumes the least number of bits.
>=20
> =20
>=20
> Michael (w/o any hat)
>=20
> =20
>=20
> =20
>=20
> =20
>=20
> From: Bob Briscoe =20
> Sent: Monday, November 16, 2020 5:52 PM
> To: Yoshifumi Nishida 
> Cc: Scharf, Michael ; Michael Tuexen =
; tcpm-chairs@ietf.org; Mirja Kuehlewind =
; Scheffenegger, Richard =
; tcpm IETF list 
> Subject: AccECN field order
>=20
> =20
>=20
> Yoshi, (adding the tcpm list in cc)
>=20
> On 05/11/2020 06:58, Yoshifumi Nishida wrote:
>=20
> Hi Bob,
>=20
> =20
>=20
> On Wed, Nov 4, 2020 at 3:29 PM Bob Briscoe  =
wrote:
>=20
> Yoshi,
>=20
> On 04/11/2020 06:51, Yoshifumi Nishida wrote:
>=20
> Hi, folks,
>=20
>=20
>=20
>=20
>=20
>=20
> In my understanding, I'm not sure if we settled down on using two =
option kinds or encoding schemes for 24bits fields in acc ecn draft.
>=20
> So, I think there're still something to be clarified and hope things =
will be settled at the meeting.
>=20
>=20
> [BB] I know a WG can change it's mind at any time. But I'd rather we =
just clarified what a previous decision was, to avoid the need to keep =
re-opening discussion on a question that have been decided then changed =
three different ways already.
>=20
> My memory is not so good these days. I trusted that Michael S =
remembered the decision correctly, and I seem to remember that decision =
being made.
> I've just checked the minutes of the last interim:
>     =
https://datatracker.ietf.org/doc/minutes-interim-2020-tcpm-01-202004291600=
/
> and they mention Michael's proposal to use two kinds, but don't record =
any decision.
> The jabber log gives no clues about any decision.
>=20
> I can't find an audio or video recording. Can you point me at one?
>=20
> =20
>=20
> I thought that it's because there was no clear decision at the =
meeting.=20
>=20
> But, you can check https://www.youtube.com/watch?v=3DKsB0_Nb8-kA
>=20
> Please let us know if you have any questions or opinions with regard =
to this.
>=20
>=20
> [BB] I checked the Youtube link you sent below.
>=20
> First I think we're agreed no-one was fighting for us to keep the =
previous way we did this (using the initial value of the field to set =
the order for the connection).=20
> In my presentation I said there was strong resistance from Michael to =
do it a different way.=20
> (also, offlist, the co-authors including me also didn't like this so =
much. And Ilpo said it made the implementation complex.)
>=20
> Then came the question of what we do instead. There were three =
alternative proposals:
> a) use 2 option kinds
> b) add a flags byte
> c) somehow use the length field maybe
>=20
> Michael raised (c) in the meeting as a possibility, but no-one could =
think how to distinguish two options of the same length but a different =
field order using the length field. Michael said he'd post any ideas to =
the list if he could think of any, but that didn't happen.
>=20
> So we're left choosing between (a) and (b).
> I said in the meeting (and on the list when discussing with Ilpo) that =
I'd be happy to go with (b), but only if there was another use for a =
flag. Because it would consume 1B more options space in many packets, =
which is a scarce resource.
>=20
> Ilpo had a proposed use for another flag (to help synch counters after =
a loss), but I think the discussion about it ended that it wouldn't be =
helpful, 'cos the way it worked depended on itself (circular logic).
>=20
> In conclusion, I don't think there was an explicit decision to go with =
2 option kinds, but it ended up as the 'last person standing'.
> I like it. It's simple. And apparently option kinds are not such a =
scarce resource.
>=20
> Perhaps we can ratify this in the WG tomorrow.
>=20
>=20
> Bob
>=20
>=20
>=20
>=20
>=20
> =20
>=20
> Thanks,
>=20
> --
>=20
> Yoshi
>=20
>=20
>=20
>=20
>=20
>=20
> --=20
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>=20
>=20
>=20
>=20
> --=20
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>=20
>=20
>=20
> --=20
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Mon Nov 23 04:42:54 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E990B3A0A22; Mon, 23 Nov 2020 04:42:52 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.001
X-Spam-Level: 
X-Spam-Status: No, score=-0.001 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=netapp.onmicrosoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oZW6f7MaLbwK; Mon, 23 Nov 2020 04:42:50 -0800 (PST)
Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2085.outbound.protection.outlook.com [40.107.244.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 15D173A0A1D; Mon, 23 Nov 2020 04:42:49 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jBQceuMTJsoCf5xOVUut+Nu2FDfZzZfj4T952phTgNIoVGX4yuduqhdufV8xFNoZoQS+sO76pmeUA2vflA819dieywr0lJXbbtGYx7mOj1UQMsyF9TOtjbBlqL4jYPaWdbRuaYWFQoX7Lc59743T4+LQOWc0ME2EjQI8gG1BaP/IxpThYEc8txddb4ejt5NuqeKiVB0ewsn5jtAOvQCRsjFCiRWsjWJvfVB6PLe4mrr+rT8cwAh5O8qjTvA0afqdImT3IgE7lBDzdMaUmOa9Qb7lLGbjLAR79fbpK99Vx1rtSXCT33NsmJHDoDbafr/eIgQgpOKXgIH7GF1SrjaDdA==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;  s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=minE/xoqH22FxPd1v9ZUM3UUR9e/Ln1jfLHkmzjiLbs=; b=hTSa1yWu2q8d8BHsThROAG6Dc1/tFKTBajo/VdEMjIZGJJ+acExfbPXGs2Pk5k8OY9B/nTblgXp3mDXGLveYkb6s7zEC09p//vc4F13KHTdldmL7ael1GauDXyiXAf2pDeqYMXFMTlWgVkdUcxVXHwHpYBXblCc4VqSELqdnmmmPv9IdFSnzAThzE0b5v2AvY/OY80LVrt+CZ+XT5j/xYp7QBbhMXRvVvv5lPeMrtspflCgJzIrapIYVCrYbCC92C1LLeWgParJ1U/lbmGhZTmwKbkxHD3cHe5QkPrX+s+09wkBDiTw3JWmMTC8YOidVniofMgj1MNsKPFC8m+i2Ig==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=netapp.com; dmarc=pass action=none header.from=netapp.com; dkim=pass header.d=netapp.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netapp.onmicrosoft.com; s=selector1-netapp-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=minE/xoqH22FxPd1v9ZUM3UUR9e/Ln1jfLHkmzjiLbs=; b=ZqbrbrAnkxlVxCy7c0Xne+FWsP8l3WNxiIX+5z62DiLkbiz8jkrq697HoEv72lMUbDCH0o+PTJlMIdF5wDrPtPpXX1wWYL71eMnFxiiAbvbaj2Ix4CWpYyOYoOn4rNng5lXUJh2d2mqpngO1jMXpmxkIutLsS9NcLGPMcYGb5p4=
Received: from SN4PR0601MB3728.namprd06.prod.outlook.com (2603:10b6:803:51::24) by SN4PR0601MB3725.namprd06.prod.outlook.com (2603:10b6:803:44::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3541.23; Mon, 23 Nov 2020 12:42:48 +0000
Received: from SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::84a9:fc74:7eed:ca3a]) by SN4PR0601MB3728.namprd06.prod.outlook.com ([fe80::84a9:fc74:7eed:ca3a%7]) with mapi id 15.20.3541.030; Mon, 23 Nov 2020 12:42:47 +0000
From: "Scheffenegger, Richard" 
To: Mirja Kuehlewind , Neal Cardwell , "Scharf, Michael" , Bob Briscoe 
CC: Yoshifumi Nishida , tcpm IETF list , Michael Tuexen , "tcpm-chairs@ietf.org" 
Thread-Topic: [tcpm] AccECN field order
Thread-Index: AQHWvDjzOJzuvj4he0uzhveqyjTda6nLBXMAgADB2wCAADJ1AIAARcGAgAAi6oCAAKG5gIAIndMAgAAQZaA=
Date: Mon, 23 Nov 2020 12:42:47 +0000
Message-ID: 
References: <42eee5b7-fc0d-9576-c2ab-128706611a96@bobbriscoe.net>    <279fb3d5-0000-f704-d88f-08ab0fa9e83a@bobbriscoe.net>  <9bdf71e9-4af0-f5ee-f2f7-e63349956500@bobbriscoe.net>   <30781ea61a794131beafe9997ed9221a@hs-esslingen.de> <1c927b36-7228-91f3-4d58-6f3545c88a57@bobbriscoe.net> <5c3c4661887c43529b35bd5f47d10c2b@hs-esslingen.de>  
In-Reply-To: 
Accept-Language: de-AT, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
authentication-results: kuehlewind.net; dkim=none (message not signed) header.d=none;kuehlewind.net; dmarc=none action=none header.from=netapp.com;
x-originating-ip: [185.236.167.136]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 07d31466-4868-4fba-eba3-08d88fad483b
x-ms-traffictypediagnostic: SN4PR0601MB3725:
x-microsoft-antispam-prvs: 
x-ms-oob-tlc-oobclassifiers: OLM:8882;
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: TjHt7e7ugogmSPLjG9/d//2R88eMqFEnkbqibngROlbHYxCn9TFLItjzfTS/RHuU4/yy2uwYsFHACH0OfU32afWPrjrS75jRbixQRStp7KuqsEx0Usf633OoZqkNf2/OlSQajOW3ubDTcPxHmr5r6XAN/U1C6xPSeH2oYZmjWhVq8Et4bm53/okEUiRp4lEYZFcUswyTmiNjXQLyDELN2igPmOJAprj96Wr5YcMjZe8Z25SmABMKpA3b4HDNOnZod9PVo5Z7hQpWv2tN4+y0wW45DCyrqiXIY4KC9DG/4loVJS96UBVovmZ+sYESdZsU3x2HqM11kexFUwrBOCVvSBwl8faef9JtGHSLDq8bvxdoCejxsF+5seRtRFGigRdJO2KiwAOaTzIXpKKicNPa9g==
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;  IPV:NLI; SFV:NSPM; H:SN4PR0601MB3728.namprd06.prod.outlook.com; PTR:; CAT:NONE;  SFS:(4636009)(136003)(376002)(346002)(366004)(39860400002)(396003)(6506007)(8936002)(2906002)(53546011)(110136005)(33656002)(52536014)(316002)(30864003)(5660300002)(54906003)(66476007)(64756008)(66446008)(86362001)(66556008)(76116006)(66946007)(966005)(7696005)(478600001)(66574015)(26005)(186003)(71200400001)(9686003)(83380400001)(4326008)(55016002)(8676002)(579004); DIR:OUT; SFP:1101; 
x-ms-exchange-antispam-messagedata: cGJP9nnKV0i1LEExHkjaBrG7AU4p0DQL9S60dHCgM58jSIgHyjN7a2sv9bC1Q/XSlcOJCw26xohj5rpfqFELqxLltPfgF4z/W3AAe5ZnmfoRtP4zKDCZMoqyu4wMvVY+iELO9S0ePgDh/RHnT/HTIMSGWyLwrHJ+K63RNb/1vx+GNs+YRaF0NW6rogGfbNjytRo2MJtdzYmUyiE9LH7xND6c03CskQNm5g7vpO/pdAZ2l9FoPCcFhcZfu7kLQAdMJ9CQTtCv4bKOfkCZfrAPSJinZR/1+XridXMlNPa3i3FElJeqlSyL+1NPSg0iljrZJkHc+UIow95IA482U4sm016VVMoQ/TrOx42yszuxbcdAZWAo0Z8M9Ux8IQwHpn1oG0/W1RdGFT7ISGT7lCNhZo3jl+ODGxmBQ3+DfBCgs4i3CGiBiV3GkWKIBU7rvyMADw0+3YE8v837Q258GWeKsz/LT9ZZ4sEb2B/ujJ7tiaWI3O466eiYGVjtFzbPs0aMfpVVaw9GgpPFyvqe/Ep1/1zm5NI6Pn/0QITxU887+f9GS9oFFEz541gKmvBXkgN71Anls80wkOCLq4MPOy+08en87o8bxIaIi9uIKl0AZ+jj/58QvTt1Wp3g9sGfTnGhS2tXa2BS78MRMX6/gfHgEg==
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-OriginatorOrg: netapp.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN4PR0601MB3728.namprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 07d31466-4868-4fba-eba3-08d88fad483b
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Nov 2020 12:42:47.5686 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4b0911a0-929b-4715-944b-c03745165b3a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: HhH2snMeuhRWZ88G0MJMAM5/QgMKlU+vxWCshnm5jOY1aTwwg5gCq8VK+E8WNh+IaiaL0cQIHktRJMne8UbeVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0601MB3725
Archived-At: 
Subject: Re: [tcpm] AccECN field order
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 23 Nov 2020 12:42:53 -0000

SSBhbSBhbHNvIGluIGZhdm9yIG9mIHVzaW5nIHR3byBkaXN0aW5jdCBvcHRpb24gY29kZXMuDQoN
Ckhvd2V2ZXIsDQoNCj4gT3IgSSBndWVzcyB3ZSBjb3VsZCBzYXkgdGhhdCBvbmx5IG9uZSBvcHRp
b24gc2hvdWxkIGJlIHVzZWQgZm9yIGEgY29ubmVjdGlvbi4NCg0KSSBhbSBvcHBvc2VkIHRvIHRo
aXMgLSB3aXRoIHR3byBvcHRpb24gY29kZXMsIGEgc2VuZGVyIHNob3VsZCBiZSBmcmVlIHRvIGNo
b29zZSB3aGljaGV2ZXIgaGUgcHJldmVycyBhdCBhbnkgb25lIHBvaW50OyBGb3IgZXhhbXBsZSwg
aWYgdGhlIHNlbmRlciBhIHNlZ21lbnQgKGUuZy4gbWF5IGJlIHRoZSByZWNlaXZlciBzaWRlKSAg
aGFzIHRvIHVwZGF0ZSB0aGUgRUUxIGNvdXRlciwgYnV0IGFsc28gY29udmV5IFNBQ0sgaW5mb3Jt
YXRpb24sIGl0IHdvdWxkIGJlIGdvb2QgdG8gdXNlIHRoZSBvbmUsIHdoZXJlIHRoZSBFRTEgZmll
bGQgaXMgZmlyc3QsIGFuZCBvbWl0IGFsbCByZW1haW5pbmcgZmllbGRzOw0KDQpMYXRlciBpbiB0
aGUgc2FtZSBjb25uZWN0aW9uLCB0aGF0IHZlcnkgc2FtZSBob3N0IG1heSBjaG9vc2UgdG8gdXBk
YXRlIHRoZSBDRSBmaWVsZCB0b2dldGhlciB3aXRoIHNvbWUgU0FDSyBpbmZvcm1hdGlvbiwgYW5k
IHVzZSB0aGUgb3RoZXIgY29kZXBvaW50IGluIG9yZGVyIHRvIHByZXNlcnZlIGFzIG11Y2ggVENQ
IG9wdGlvbiBzcGFjZSBmb3IgdGhlIFNBQ0sgb3B0aW9uLi4uDQoNCg0KQXMgZm9yIGV4dGVuc2li
aWxpdHk6IEkgd291bGQgd2FudCB0byBhdm9pZCB0aGF0IHRvcGljLCBhbmQganVzdCBzcGVjaWZp
eSB0aGUgdGhyZWUgY29ycmVjdCBzdXBwb3J0ZWQgbGVuZ3RocyBoZXJlLCBhbmQgbm90IG1lbnRp
b24gKG90aGVyIHRoYW4gaWdub3JpbmcgIm9kZCIgZnJhY3Rpb25hbHMgb2YgdGhlIDMtYnl0ZSB3
aWRlIGNvdW50ZXJzKSB3aGF0IGhhcHBlbnMgd2l0aCBvdGhlciBsZW5ndGhzLi4uIA0KDQoNClJp
Y2hhcmQgU2NoZWZmZW5lZ2dlcg0KDQotLS0tLVVyc3Byw7xuZ2xpY2hlIE5hY2hyaWNodC0tLS0t
DQpWb246IE1pcmphIEt1ZWhsZXdpbmQgPGlldGZAa3VlaGxld2luZC5uZXQ+IA0KR2VzZW5kZXQ6
IE1vbnRhZywgMjMuIE5vdmVtYmVyIDIwMjAgMTI6MzkNCkFuOiBOZWFsIENhcmR3ZWxsIDxuY2Fy
ZHdlbGxAZ29vZ2xlLmNvbT47IFNjaGFyZiwgTWljaGFlbCA8TWljaGFlbC5TY2hhcmZAaHMtZXNz
bGluZ2VuLmRlPjsgQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2NvZS5uZXQ+DQpDYzogWW9zaGlm
dW1pIE5pc2hpZGEgPG5zZC5pZXRmQGdtYWlsLmNvbT47IHRjcG0gSUVURiBsaXN0IDx0Y3BtQGll
dGYub3JnPjsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZCA8UmljaGFyZC5TY2hlZmZlbmVnZ2VyQG5l
dGFwcC5jb20+OyBNaWNoYWVsIFR1ZXhlbiA8dHVleGVuQGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1j
aGFpcnNAaWV0Zi5vcmcNCkJldHJlZmY6IFJlOiBbdGNwbV0gQWNjRUNOIGZpZWxkIG9yZGVyDQoN
Ck5ldEFwcCBTZWN1cml0eSBXQVJOSU5HOiBUaGlzIGlzIGFuIGV4dGVybmFsIGVtYWlsLiBEbyBu
b3QgY2xpY2sgbGlua3Mgb3Igb3BlbiBhdHRhY2htZW50cyB1bmxlc3MgeW91IHJlY29nbml6ZSB0
aGUgc2VuZGVyIGFuZCBrbm93IHRoZSBjb250ZW50IGlzIHNhZmUuDQoNCg0KDQoNCkhpIGFsbCwN
Cg0KSSB0aGluayBJIGFsc28gc3VwcG9ydCB0d28gb3B0aW9ucyBhcyB0aGlzIGlzIHRoZSBtb3N0
IHNpbXBsZSBzb2x1dGlvbi4NCg0KV2UgcHJvYmFibHkgbmVlZCB0byBsb29rIGF0IHNlY3Rpb24g
My4yLjMuMi4gKFBhdGggVHJhdmVyc2FsIG9mIHRoZSBBY2NFQ04gT3B0aW9uKSBhZ2FpbiwgYXMg
SSB0aGluayB0aGlzIHNlY3Rpb24gYXNzdW1lcyBtb3N0bHkgb25lIG9wdGlvbi4gT3IgSSBndWVz
cyB3ZSBjb3VsZCBzYXkgdGhhdCBvbmx5IG9uZSBvcHRpb24gc2hvdWxkIGJlIHVzZWQgZm9yIGEg
Y29ubmVjdGlvbi4NCg0KUmVnYXJkaW5nIHVzZSBvZiB0aGUgb3B0aW9uIGxlbmd0aCBmb3IgZXh0
ZW5zaWJpbGl0eS4gSSBrbm93IG5ldyBvcHRpb25zIGFyZSBoYXJkIHRvIGRlcGxveSBidXQgdGhh
dOKAmXMgdGhlIGV4dGVuc2liaWxpdHkgbWVjaGFuaXNtIHdlIGhhdmUgd2l0aCBUQ1AuIE5vdCBz
dXJlIHdlIHdhbnQgdG8gaW52ZW50IGEgbmV3IG5vdy4gSWYgd2UgdGhpbmsgd2UgbmVlZCBtb3Jl
IGV4dGVuc2liaWxpdHkgZm9yIEFjY0VDTiBmb3IgYW55IHJlYXNvbiwgd2Ugc2hvdWxkIHNwZWNp
ZnkgdmFsaWQgb3B0aW9uIGxlbmd0aCB2YWx1ZXMgdGhhdCBpbmNsdWRlIGEgZmxhZ3MgZmllbGQg
bm93Lg0KDQpNaXJqYQ0KDQoNCg0KPiBPbiAxOC4gTm92IDIwMjAsIGF0IDAxOjAzLCBOZWFsIENh
cmR3ZWxsIDxuY2FyZHdlbGxAZ29vZ2xlLmNvbT4gd3JvdGU6DQo+DQo+IEFzIHNvbWVvbmUgd2hv
IGNvbnRyaWJ1dGVzIHRvIGEgVENQIGltcGxlbWVudGF0aW9uLCBJIHdvdWxkIHB1dCBpbiBhICB2
b3RlIGZvciB0aGUgIlR3byBPcHRpb24gS2luZHMiIHByb3Bvc2FsIG91dGxpbmVkIGluIHNsaWRl
IDYgaGVyZToNCj4NCj4gIA0KPiBodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL21lZXRpbmcv
MTA5L21hdGVyaWFscy9zbGlkZXMtMTA5LXRjcG0tbW9yDQo+IGUtYWNjdXJhdGUtZWNuLWZlZWRi
YWNrLWluLXRjcC1lY24tYWRkaW5nLWV4cGxpY2l0LWNvbmdlc3Rpb24tbm90aWZpY2ENCj4gdGlv
bi0wMSNwYWdlPTYNCj4NCj4gVGhpcyBhcHByb2FjaCBoYXMgdGhlIHZpcnR1ZSBvZiBzaW1wbGlj
aXR5LCB3aGljaCBob3BlZnVsbHkgbWF5IGhlbHAgYXZvaWQgbWlkZGxlYm94IGJ1Z3MsIHNwZWVk
IHN0YW5kYXJkaXphdGlvbiBhbmQgZGV2ZWxvcG1lbnQsIGFuZCBlYXNlIHRoZSBwcm9jZXNzIG9m
IG1haW50YWluaW5nIGZhc3QgYW5kIHN0YWJsZSBpbXBsZW1lbnRhdGlvbnMuDQo+DQo+IG5lYWwN
Cj4NCj4NCj4gT24gVHVlLCBOb3YgMTcsIDIwMjAgYXQgOToyNSBBTSBTY2hhcmYsIE1pY2hhZWwg
PE1pY2hhZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT4gd3JvdGU6DQo+IElubGluZSBzb21lIGNv
bW1lbnRzIFttc10uDQo+DQo+DQo+DQo+IEdpdmVuIHRoYXQgSSBkb27igJl0IGFzayBmb3IgYW55
IHRleHQgY2hhbmNlcywgSeKAmWxsIG5vdCBmb2xsb3ctdXAgZm9yZXZlciBpbiB0aGlzIGRpc2N1
c3Npb24uIEkgZG9u4oCZdCBvd24gcnVubmluZyBjb2RlIGFuZCBhIGxvdCBvZiBlbmNvZGluZyBk
ZXRhaWxzIG1haW5seSBhZmZlY3QgYW4gaW1wbGVtZW50YXRpb24uDQo+DQo+DQo+DQo+IElNSE8g
dGhlIHJlc3Qgb2YgdGhlIFdHIGhhcyB0byB0aGluayBhYm91dCB0aGUgb3B0aW9uIGRlc2lnbiBt
b3JlIHRoYW4gDQo+IG1lLiBTbywgdGhpcyBpcyBtb3N0bHkgYWJvdXQgdHJpZ2dlcmluZyBhIGRp
c2N1c3Npb27igKYNCj4NCj4NCj4NCj4gRnJvbTogQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2Nv
ZS5uZXQ+DQo+IFNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDE3LCAyMDIwIDE6MjAgUE0NCj4gVG86
IFNjaGFyZiwgTWljaGFlbCA8TWljaGFlbC5TY2hhcmZAaHMtZXNzbGluZ2VuLmRlPjsgWW9zaGlm
dW1pIA0KPiBOaXNoaWRhIDxuc2QuaWV0ZkBnbWFpbC5jb20+DQo+IENjOiBNaWNoYWVsIFR1ZXhl
biA8dHVleGVuQGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IA0KPiBNaXJq
YSBLdWVobGV3aW5kIDxpZXRmQGt1ZWhsZXdpbmQubmV0PjsgU2NoZWZmZW5lZ2dlciwgUmljaGFy
ZCANCj4gPFJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tPjsgdGNwbSBJRVRGIGxpc3Qg
PHRjcG1AaWV0Zi5vcmc+DQo+IFN1YmplY3Q6IFJlOiBBY2NFQ04gZmllbGQgb3JkZXINCj4NCj4N
Cj4NCj4gTWljaGFlbCwNCj4NCj4gT24gMTcvMTEvMjAyMCAwODoxMCwgU2NoYXJmLCBNaWNoYWVs
IHdyb3RlOg0KPg0KPiBCb2IsDQo+DQo+DQo+DQo+IEkgYW0gZmluZSB3aXRoIHRoZSB0d28gb3B0
aW9uIGtpbmRzIHByb3Bvc2VkIGluIC0xMywgYnV0IEkgZG9u4oCZdCBidXkgeW91ciBhcmd1bWVu
dHMgd2h5IHRoaXMgZW5jb2RpbmcgaXMgYmV0dGVyIHRoYW4gb3RoZXJzLg0KPg0KPg0KPg0KPiBN
b3N0IGltcG9ydGFudGx5LCBJIGRvbuKAmXQgdGhpbmsgdGhhdCB5b3VyIOKAnGZvcndhcmQgY29t
cGF0aWJpbGl0eeKAnSBhcmd1bWVudCBpcyBhIHZlcnkgY29tcGVsbGluZyByZWFzb24gZm9yIHR3
byBjb2RlcG9pbnRzLiBBbGwgcHJvcG9zYWxzIEkgYW0gYXdhcmUgb2YgaGF2ZSB0aGVpciBwcm9z
IGFuZCBjb25zLiBBbmQgSSBhbSBub3QgYXdhcmUgb2YgYSByZWFsbHkgY29tcHJlaGVuc2l2ZSBk
aXNjdXNzaW9uLiBBdCBsZWFzdCBuZWl0aGVyIHRoZSBlLW1haWwgYmVsb3cgbm9yIHRvZGF54oCZ
cyBtZWV0aW5nIGRpc2N1c3MgYWxsIHRyYWRlb2ZmcyBJIHdvdWxkIGludmVzdGlnYXRlLg0KPg0K
Pg0KPiBbQkJdIFlvdSdyZSByaWdodCAtIHRoZSBkaXNjdXNzaW9uIGJlbG93IGlzIHVzZWZ1bCBh
bmQgbmVjZXNzYXJ5Lg0KPg0KPg0KPg0KPg0KPg0KPiBGb3IgaW5zdGFuY2UsIHRoZSBiZW5lZml0
IG9mIHRoZSBsZW5ndGggZW5jb2RpbmcgdmFyaWFudCB3b3VsZCBiZSB0byBjb25zdW1lIG9ubHkg
b25lIFRDUCBvcHRpb24gY29kZXBvaW50IG5vdy4gSWYgd2UgZGVjaWRlZCBsYXRlciB0byB0aGF0
IGEgZmxhZyBieXRlIGlzIG5lZWRlZCAoc2F5LCBhdCB0aGUgZW5kKSwgYSBmb2xsb3ctdXAgcHJv
cG9zZWQgc3RhbmRhcmQgY291bGQgc3BlY2lmeSBhbm90aGVyIG9wdGlvbiBmb3JtYXQgd2l0aCB0
aGF0IGZsYWcgYnl0ZSwgdXNpbmcgYSBzZWNvbmQgY29kZXBvaW50Lg0KPg0KPg0KPiBbQkJdDQo+
ICogVGhlIG5ldyBjb2RlcG9pbnQgd291bGQgcmVxdWlyZSB5ZWFycyB0byBkZXBsb3kgKGluY2x1
ZGluZyBpbiBwcm94aWVzKSBiZWZvcmUgaXQgd29ya3MgZTJlLg0KPg0KPg0KPg0KPiBbbXNdIFRo
YXQgbWF5IGRlcGVuZCAqYSBsb3QqIG9uIHRoZSBlbnZpcm9ubWVudC4gSW4gZW52aXJvbm1lbnRz
IHdoZXJlIEVDTiBjYW4gZ2V0IGRlcGxveWVkIHRvZGF5LCBzYXkgaW4gZGF0YWNlbnRlcnMsIHJv
bGxpbmcgb3V0IGEgbmV3IGtlcm5lbCB2ZXJzaW9uIG1heSBwZXJoYXBzIG5vdCBiZSB0aGUgYmln
Z2VzdCBvcGVyYXRpb25hbCBjaGFsbGVuZ2UuIEFsc28sIHByb3hpZXMgbWF5IG5vdCBiZSBhIHNp
Z25pZmljYW50IHByb2JsZW0gaW4gc29tZSBvZiB0aG9zZSBjYXNlcy4gRGVwbG95bWVudCByb2Fk
bWFwcyBhcmUgdmVyeSBoYXJkIHRvIHByZWRpY3QuDQo+DQo+DQo+ICogV2hlcmVhcyB0aGUgJ2Zv
cndhcmQgY29tcGF0aWJpbGl0eScgYXBwcm9hY2ggdHJpZXMgdG8gZW5zdXJlIHRoYXQgQWNjRUNO
IGltcGxlbWVudGF0aW9ucyB0aGF0IGFyZSBiZWluZyBidWlsdCB0b2RheSB3aWxsIGJlIHJlYWR5
IHRvIGludGVyd29yayB3aXRoIGEgZnV0dXJlIGV4dGVuc2lvbi4NCj4gKiBJbiBjb250cmFzdCwg
aWYgeW91ciBoeXBvdGhldGljYWwgbmV3IEFjY0VDTiBjb2RlcG9pbnQgZGlkIG5lZWQgdG8gYmUg
ZGVwbG95ZWQgbGF0ZXIsIGl0IHdvdWxkIG5lZWQgdG8gaW5jbHVkZSBhIGZhbGwtYmFjayB0byB0
aGUgY3VycmVudCBBY2NFQ04gb3B0aW9uIHdoZW4gb25lIHBlZXIgZG9lc24ndCB1bmRlcnN0YW5k
IHRoZSBuZXcgY29kZXBvaW50Lg0KPg0KPg0KPg0KPiBbbXNdIFRoaXMgaXMgYSByZWFzb25hYmxl
IHBvaW50LCBhbGJlaXQgbm90IGEgcGFydGljdWxhcmx5IHN0cm9uZyBvbmUsIA0KPiBnaXZlbiB0
aGF0IHRoZSBvcHRpb24gaXMgbm90IHJlYWxseSByZXF1aXJlZCBmb3IgYmFzaWMgDQo+IGludGVy
b3BlcmFiaWxpdHkuIEFsc28sIGEgaHlwb3RoZXRpY2FsIGZvbGxvdy11cCBzdGFuZGFyZCBjb3Vs
ZCANCj4gaW5jbHVkZSBhIHdheSB0byBleHBsaWNpdGx5IHJlcXVlc3QgdGhlIG9wdGlvbuKApg0K
Pg0KPg0KPiAqIEZhbGwtYmFjayB3b3VsZCBuZWVkIHRvIGF2b2lkIGEgc2Vjb25kIHJvdW5kIG9m
IGhhbmRzaGFrZSAoZ2l2ZW4gDQo+IHRoaXMgaXMgZm9yIGxvdyBsYXRlbmN5KSwgd2hpY2ggd291
bGQgb25seSB3b3JrIHJlbGlhYmx5IHdpdGggYW4gDQo+IG9wdGlvbiBvbiB0aGUgU1lOIChiZWNh
dXNlIHRoZSAzcmQgQUNLIGlzIHVucmVsaWFibGUpDQo+DQo+DQo+DQo+IFttc10gSSBjYW7igJl0
IGZvbGxvdy4gSWYgYW4gZW5kcG9pbnQgaXMgdW5oYXBweSB3aXRoIGdldHRpbmcgQWNjRUNOIG9w
dGlvbiDigJxB4oCdIGZyb20gYSBwZWVyLCBJIGNvdWxkIHNlbmQgYSDigJxwbGVhc2Ugc2VuZCBt
ZSB0aGUgYmV0dGVyIEFjY0VDTiBvcHRpb24gQuKAnSBhZnRlciB0aGUgMy1XSFMuIEl0IGNvdWxk
IGJlIGFuIHVwZ3JhZGUsIG5vdCBhIGZhbGxiYWNrLiBBbHNvLCBhdCB0aGlzIHBvaW50IHdlIGRv
buKAmXQga25vdyB3aGF0IG9wdGlvbiBCIHdvdWxkIGJlIGFuZCBpbiB3aGljaCBjb250ZXh0IGl0
IHdvdWxkIGJlIG5lZWRlZC4gTWF5YmUgb3RoZXIgcHJvdG9jb2wgZmVhdHVyZXMgd291bGQgcmVx
dWlyZSBuZWdvdGlhdGlvbiwgdG9vLiBJTUhPIGl0IGlzIGEgc29sdmFibGUgcHJvYmxlbS4NCj4N
Cj4NCj4gKiBBY2NFQ04gaGFzIHNvIGZhciBhdm9pZGVkIHVzaW5nIGFueSBTWU4gb3B0aW9uIHNw
YWNlLCB3aGljaCBpcyBpbXBvcnRhbnQgZ2l2ZW4gdGhhdCBpcyB0aGUgc2NhcmNlc3QgcmVzb3Vy
Y2UuDQo+DQo+DQo+DQo+IFttc10gWWVwLCBidXQgYXMgbG9uZyBhcyBubyBwcm9wb3NhbCBuZWVk
cyBTWU4gb3B0aW9ucywgdGhpcyBpcyBub3RoaW5nIHRoYXQgd291bGQgbWFrZSBvbmUgc29sdXRp
b24gYmV0dGVyIHRoYW4gYW5vdGhlci4NCj4NCj4gVGhlc2UgZGF5cywgZXh0ZW5kaW5nIFRDUCBp
cyBsaWtlIHBsYXlpbmcgY2hlc3MgLSB3ZSBoYXZlIHRvIHRoaW5rIG11bHRpcGxlIG1vdmVzIGFo
ZWFkIQ0KPg0KPg0KPg0KPiBbbXNdIFRoZXJlIGlzIG9uZSBmdW5kYW1lbnRhbCBkaWZmZXJlbmNl
IGJldHdlZW4gY2hlc3MgYW5kIFRDUDogSW4gY2hlc3MgeW91IHByZWNpc2VseSBrbm93IHRoZSBj
dXJyZW50IHNpdHVhdGlvbiBhbmQgY2FuIHByZWRpY3QgYWxsIHBvdGVudGlhbCBtb3ZlcyBpbiBh
ZHZhbmNlIChhbGJlaXQgdGhlcmUgYXJlIGEgbG90KS4gSG93ZXZlciwgd2UgZG9u4oCZdCBrbm93
IHRoZSBmdXR1cmUgZXZvbHV0aW9uIG9mIFRDUCwgRUNOIGFuZCB0aGUgSW50ZXJuZXQgYXMgYSB3
aG9sZSwgbm9yIGZ1dHVyZSByZXF1aXJlbWVudHMuIElmIHdlIHRyeSB0byBlbmdpbmVlciBhIFRD
UCBleHRlbnNpb24gZm9yIHNvbWUgcG90ZW50aWFsIGZ1dHVyZSwgd2UgbWF5IGZpZ3VyZSBvdXQg
bGF0ZXIgdGhhdCByZWFsaXR5IGlzIGRpZmZlcmVudCBmcm9tIHdoYXQgd2UgZGVzaWduZWQgZm9y
LiBBcyBtdWNoIGFzIEkgYXBwcmVjaWF0ZSB5b3VyIGF0dGVtcHRzIHRvIHByZWRpY3QgdGhlIGZ1
dHVyZSwgaXQgd291bGQgYmUgYSBiaWcgc3VycHJpc2UgdG8gbWUgaWYgeW91IGNvdWxkIGNvcnJl
Y3RseSBwcmVkaWN0IHdoYXQgVENQIHdpbGwgYmUgaW4gMTAgeWVhcnMgZnJvbSBub3cuIFNvLCBp
bnN0ZWFkIG9mIHRyeWluZyB0byB3cml0ZSBzcGVjaWZpY2F0aW9ucyB0aGF0IHRha2UgaW50byBh
Y2NvdW50IGZ1dHVyZSByZXF1aXJlbWVudHMsIHdoaWNoIGFyZSBpbiBmYWN0IGp1c3QgdW5rbm93
biwgaXQgaXMgbW9yZSBpbXBvcnRhbnQgZm9yIG1lIHRvIHdyaXRlIGVmZmVjdGl2ZSwgZWZmaWNp
ZW50IGFuZCBzZWN1cmUgc3BlY2lmaWNhdGlvbnMgZm9yIHdoYXQgaXMgY3VycmVudGx5IGtub3du
IGZvciBzdXJlLg0KPg0KPg0KPg0KPiBbbXNdIEkgKmRvbuKAmXQqIGtub3cgZm9yIHN1cmUgd2hl
dGhlciB0aGUgY3VycmVudCBBY2NFQ04gb3B0aW9uIHdpbGwgYmUgbmVlZGVkLCBidXQgSSBzZWUg
cXVpdGUgc29tZSBldmlkZW5jZSB0aGF0IHN1cHBvcnQgb2YgdGhlIEFjY0VDTiBvcHRpb24ocykg
aXMgbm90IGEgdG9wIHByaW9yaXR5IHRoZXNlIGRheXMuIFRoYXQgaXMgYWxzbyBzb21ldGhpbmcg
dG8gdGhpbmsgYWJvdXQgd2hlbiBkZXNpZ25pbmcgdGhlIGRldGFpbHMuDQo+DQo+DQo+DQo+IEkg
bmVlZCB0byBsZWFybiB0byBhcnRpY3VsYXRlIG9udG8gdGhlIGxpc3QgYWxsIG15IHJlYXNvbnMg
Zm9yIHJlamVjdGluZyBjZXJ0YWluIHBhcnRzIG9mIHRoZSBkZXNpZ24gc3BhY2UuIFNvcnJ5IGFi
b3V0IHRoYXQuDQo+DQo+IFllcywgSSBhZG1pdCwgdGhlIGN1cnJlbnQgcHJvcG9zYWwgaXMgYXQg
dGhlIGNvc3Qgb2YgYnVybmluZyB0d28gVENQIG9wdGlvbiBjb2RlcG9pbnRzLiBIb3dldmVyLCB5
b3Ugc2FpZCB5b3Vyc2VsZiB3aGVuIHlvdSBwcm9wb3NlZCB0aGlzIGFwcHJvYWNoLCB0aGF0IHRo
ZXkgYXJlIG5vdCAoY3VycmVudGx5KSBhIHNjYXJjZSByZXNvdXJjZS4gQ2VydGFpbmx5IG5vdCBh
cyBzY2FyY2UgYXMgb3B0aW9uIHNwYWNlLiBJIGp1c3QgY2hlY2tlZDogcm91Z2hseSAxNSUgb2Yg
dGhlIG9wdGlvbiBjb2RlcG9pbnQgYWxsb2NhdGlvbiBzcGFjZSBoYXMgYmVlbiB1c2VkIHNvIGZh
ci4gU28gdGhhdCdzIHdoZXJlIEkgd291bGQgc3VnZ2VzdCB3ZSBjb21wcm9taXNlIGZpcnN0Lg0K
Pg0KPiBbbXNdIEkgbWFkZSBhIHNpbWlsYXIgY2FsY3VsYXRpb24gc29tZSB5ZWFycyBhZ28gYW5k
IGNhbWUgdG8gYSBzaW1pbGFyIGNvbmNsdXNpb24uIEludmVzdGluZyB0d28gY29kZXBvaW50cyBm
b3IgQWNjRUNOIGlzIHNvbWV0aGluZyB3ZSBjYW4gcHJvYmFibHkgYWZmb3JkLiBCdXQgbm9uZXRo
ZWxlc3Mgd2UgaGF2ZSB0byByZWFzb24gY2FyZWZ1bGx5IHdoeSBpdCBpcyByZWFsbHkgd29ydGgg
dGhlIGVmZm9ydC4NCj4NCj4NCj4NCj4NCj4gUmVnYXJkaW5nIHRoZSByZXN1bHRpbmcgY29kZXBv
aW50IGNvbnN1bXB0aW9uICh0d28gb3B0aW9uIGtpbmRzKSBhbmQgdGhlIGFkZGl0aW9uIG9mIGEg
ZmxhZyBieXRlLCB0aGlzIGFwcHJvYWNoIHdvdWxkIGJhc2ljYWxseSBlbmQgdXAgd2l0aCB0aGUg
c2FtZSByZXN1bHQgYXMgdGhlIGN1cnJlbnQgcHJvcG9zYWwgaW4gLTEzIHBsdXMgc29tZSBoeXBv
dGhldGljYWwgZnV0dXJlIGFkZGl0aW9uIGxldmVyYWdpbmcgdGhlIGxlbmd0aCBmaWVsZCBiZXlv
bmQgdGhlIGN1cnJlbnRseSBrbm93biB1c2UuDQo+DQo+DQo+DQo+IEFuZCwgeWVzLCB0aGUgbGVu
Z3RoIGVuY29kaW5nIHZhcmlhbnQgbWF5IGJlIGxlc3MgZmxleGlibGUgYW5kL29yIGNvbnN1bWUg
c29tZSBtb3JlIGJpdHMgaW4gc29tZSBjYXNlcywgYnV0IG9uIHRoZSBwbHVzIHNpZGUgaXQgb25s
eSBuZWVkcyBvbmUgY29kZXBvaW50IG5vdyDigJMgYW5kIGdpdmVuIHRoYXQgaXQgaXMgZW50aXJl
bHkgdW5jbGVhciB3aGV0aGVyIGFueSBBY2NFQ04gb3B0aW9uIHdpbGwgaW5kZWVkIGJlIHdpZGVs
eSB1c2VkIGluIGZ1dHVyZSwgdGhpcyBpcyBhIGJpZyBwbHVzLiBUaGUgZmFjdCB0aGF0IGltcGxl
bWVudGVycyBhcmUgcXVpdGUgc2lsZW50IG9uIHRoZSBvcHRpb24gZGVzaWduIGlzIG5vdCBhIGdv
b2Qgc2lnbi4NCj4NCj4NCj4gW0JCXSBIZXJlJ3MgYW5vdGhlciBwb3NzaWJpbGl0eSB0aGVuLg0K
Pg0KPiBhKSBbS3xMfCBFRTFCIHwgRUNFQiB8IEVFMEIgXQ0KPiBiKSBbS3xMfCBFRTFCIHwgRUNF
QiB8IEVFMEIgfEZdICAgIGlmIEYgPSAwYlhYWFhYWFgwDQo+IGMpIFtLfEx8IEVFMEIgfCBFQ0VC
IHwgRUUxQiB8Rl0gICAgaWYgRiA9IDBiWFhYWFhYWDENCj4NCj4gV2UnZCBoYXZlIHRvIGRlY2lk
ZSB3aGljaCBvcmRlciBhbHRlcm5hdGl2ZSAoYSkgdGFrZXMuIEkndmUgcGlja2VkIEVFMUIgZmly
c3Qgb25seSAnY29zIHlvdSdyZSByaWdodCB0aGF0IHRoZXJlIGlzIG5vdCBjdXJyZW50bHkgYXMg
bXVjaCBwcmVzc3VyZSAob24gdGhlIHB1YmxpYyBJbnRlcm5ldCkgZm9yIEFjY0VDTiB3aXRoIEVD
VCgwKS4NCj4NCj4gPT1MZWdlbmQ9PQ0KPiBLOiAgICBLaW5kICgxQikNCj4gTDogICAgTGVuZ3Ro
ICgxQikNCj4gRXh4QjogRWNobyB4eCBieXRlIGNvdW50ZXINCj4gRjogICAgRmxhZ3MgKDFCKQ0K
Pg0KPiBBbHRlcm5hdGl2ZXMgYSkgYW5kIGIpIGFyZSBlcXVpdmFsZW50LCBhdCBsZWFzdCB3aGls
ZSB0aGUgb25seSBhbGxvY2F0ZWQgZmxhZyBpcyBmb3IgZmllbGQgb3JkZXIuDQo+IEhvd2V2ZXIs
IGFsdGVybmF0aXZlIGIpIGlzIGF2YWlsYWJsZSBpbiBjYXNlIG90aGVyIGZsYWdzIGFyZSBhbGxv
Y2F0ZWQgDQo+IGluIGZ1dHVyZSBUaGUgZmxhZ3MgYnl0ZSBjYW4gb25seSBiZSBvbWl0dGVkIGlm
IChMICUgMyA9PSAyKS4NCj4gVGhlIGZsYWdzIGJ5dGUgaXMgY29uc2lkZXJlZCBwcmVzZW50IGlm
IChMICUgMykgPT0gMC4NCj4NCj4gRm9yd2FyZCBjb21wYXRpYmlsaXR5OiBPcHRpb25zIHdpdGgg
KEwgJSAzID09IDEpIE1VU1QgYmUgYXNzdW1lZCB0byBpbmNsdWRlIGEgZmxhZ3MgYnl0ZSwgYW5k
IGN1cnJlbnQgaW1wbGVtZW50YXRpb25zIGlnbm9yZSB0aGUgbGFzdCBieXRlLg0KPg0KPiBUaGUg
ZmxhZ3MgYnl0ZSBpcyBvcHRpb25hbCB0byBpbXBsZW1lbnQgKGV2ZW4gaWYgdGhlIEFjY0VDTiBv
cHRpb24gaXMgDQo+IGltcGxlbWVudGVkKSBJZiB0aGUgc2VydmVyIGluY2x1ZGVzIGEgZmxhZ3Mg
Ynl0ZSBvbiB0aGUgU1lOL0FDSyBidXQgdGhlIGNsaWVudCBkb2VzIG5vdCBpbmNsdWRlIG9uZSBv
biB0aGUgM3JkIEFDSyBvciB0aGUgZmlyc3QgZGF0YSBwYWNrZXQsIHRoZSBzZXJ2ZXIgYXNzdW1l
cyB0aGUgY2xpZW50IGRvZXMgbm90IGltcGxlbWVudCB0aGUgZmxhZ3MgYnl0ZSBhbmQgdXNlcyBv
bmx5IGFsdGVybmF0aXZlIGEpIGZvciB0aGUgcmVtYWluZGVyIG9mIHRoZSBjb25uZWN0aW9uLg0K
Pg0KPg0KPiBbbXNdIEkgYW0gbm90IHN1cmUgaWYgd2UgbmVlZCB0aGF0LCBidXQgSSB0YWtlIHRo
aXMgYXMgZXhpc3RlbmNlIHByb29mIHRoYXQgb3RoZXIgZW5jb2RpbmdzIGFyZSBwb3NzaWJsZS4g
UGVyc29uYWxseSwgSSB0aGluayB0aGF0IG90aGVyIHNvbHV0aW9ucyB3b3VsZCBiZSBkb2FibGUg
aWYgeW91IHRvb2sgb25lIGJpdCBmcm9tIG9uZSBvciBtdWx0aXBsZSBjb3VudGVycyBmb3IgZm9y
bWF0IGVuY29kaW5nLiBBdCBsZWFzdCBmb3Igc29tZSBvZiB0aGUgYnl0ZSBjb3VudGVycyBJIGRv
buKAmXQgdGhpbmsgdGFraW5nIG9uZSBiaXQgZm9yIG90aGVyIHB1cnBvc2VzIHdvdWxkIHJlc3Vs
dCBpbiBhbiBJbnRlcm5ldCBjb25nZXN0aW9uIGNvbGxhcHNl4oCmIEJ1dCwgSSBoYXZlIGFscmVh
ZHkgc2FpZCB0aGF0IHRoZSAtMTMgcHJvcG9zYWwgd29ya3MgZm9yIG1lLCBzbyBJ4oCZbGwgbm90
IHRyeSB0byBjb21lIHVwIHdpdGggYW5vdGhlciB2YXJpYW50IG15c2VsZi4NCj4NCj4NCj4NCj4g
VG8gbWUsIG9uZSBwb3RlbnRpYWwgZGlmZmVyZW5jZSBiZXR3ZWVuIHR3byBwcm9wb3NhbHMgd291
bGQgYmUgaW5jcmVtZW50YWwgZGVwbG95bWVudC4gVGhlIHByb3Bvc2FsIGluIC0xMyBvbmx5IGhh
cyBhbiBhZHZhbnRhZ2UgaWYgbWlkZGxlYm94ZXMgc3VjaCBhcyBmaXJld2FsbHMgd2lsbCBpbmRl
ZWQgcGFzcyBUQ1Agb3B0aW9ucyB3aXRoIGEgZm9ybWF0IHRoYXQgY29udGFpbnMgY29udGVudCBi
ZXlvbmQgdGhlIChmaXJzdCkgQWNjdXJhdGUgRUNOIHN0YW5kYXJkIChpLmUuLCBjdXJyZW50bHkg
dW51c2VkIGxlbmd0aCB2YWx1ZXMpLiBJTUhPIGl0IGlzIHRvbyBlYXJseSB0byBrbm93IHdoZXRo
ZXIgZmlyZXdhbGxzIHdvdWxkIGluZGVlZCBhbGxvdyB0aGlzIGluIGZ1dHVyZS4NCj4NCj4NCj4N
Cj4gRnJvbSBhIHNlY3VyaXR5IHBlcnNwZWN0aXZlLCBpdCBpcyBub3QgY2xlYXIgdG8gbWUgd2hl
dGhlciBhbGxvd2luZyBhcmJpdHJhcnkgdW5zcGVjaWZpZWQgYnl0ZXMgaW4gYSBUQ1Agb3B0aW9u
IGlzIGEgZ29vZCBpZGVhICphdCBhbGwqLiBJdCB3aWxsIGJlIGludGVyZXN0aW5nIHRvIGhlYXIg
dGhlIG9waW5pb24gZnJvbSBTRUMgYXJlYSBvbiB0aGF0LiBQZXJzb25hbGx5LCBJIGFtIG5vdCBj
b252aW5jZWQgdGhhdCB0aGlzIHJlYWxseSBtYWtlcyBzZW5zZSwgYnV0IEkgbXkgY29uY2VybnMg
YXJlIG5vdCBzdHJvbmcgZW5vdWdoIHRvIGZvcm1hbGx5IHB1c2ggYmFjay4gSeKAmWxsIGxlYXZl
IGl0IHRvIG90aGVycyB0byB0aGluayBhYm91dCB3aGV0aGVyIHRoaXMgaXMgYSBidWcgb3IgYSBm
ZWF0dXJlLg0KPg0KPg0KPiBbQkJdIFdlbGwsIGxldCdzIGZpcnN0IHRyeSB0byBkZWFsIHdpdGgg
c2VjdXJpdHkgb3Vyc2VsdmVzOg0KPiAqIE9jdGV0cyB0aGF0IGFyZSBleHBsaWNpdGx5IGRlY2xh
cmVkIGFzIHBhcnQgb2YgYW4gb3B0aW9uIGNhbm5vdCBiZSB1c2VkIGZvciBhIGJ1ZmZlciBvdmVy
ZmxvdyBhdHRhY2suIEkgZG9uJ3QgcmVhbGx5IG5lZWQgdG8sIGJ1dCBJIGNvdWxkIGFkZCB0aGUg
Zm9sbG93aW5nIHRleHQgdG8gdGhlIGZvcndhcmQgY29tcGF0aWJpbGl0eSB3b3JkaW5nOg0KPiAg
ICAgQSByZWNlaXZlciBjb25zaWRlcnMgb2N0ZXRzIGJleW9uZCB0aG9zZSBpdCB1c2VzLCBidXQg
d2l0aGluIHRoZSBzcGVjaWZpZWQgbGVuZ3RoLCBhcyBpZiB0aGV5IGFyZSBwYWRkaW5nLg0KPiAq
IEFuZCBzdWNoIG9jdGV0cyBjYW5ub3QgYmUgYW55IGRpZmZlcmVudCBmcm9tIHRoZSBjdXJyZW50
IGFiaWxpdHkgb2YgYSBzZW5kZXIgdG8gYWRkIHBhZGRpbmcuIFNvIHRoZXJlJ3Mgbm8gbmV3IGF0
dGFjayBwb3NzaWJsZSBoZXJlLg0KPiAqIEFuZCB0aGVyZSdzIG5vIG5lZWQgdG8gc3BlY2lmeSBh
IG1heCBsZW5ndGggZm9yIGFueSBBY2NFQ04gT3B0aW9uLCB3aGljaCB3b3VsZCBqdXN0IHVubmVj
ZXNzYXJpbHkgbGltaXQgZmxleGliaWxpdHkuDQo+DQo+IFttc10gSSBoYXZlIG1vcmUgdGhvdWdo
dCBhYm91dCB0aGUgY292ZXJ0IGNoYW5uZWwgdGhhdCBzb21lIHVuc3BlY2lmaWVkIGJ5dGVzIGlu
IGEgc3RhbmRhcmRzIHRyYWNrIFRDUCBvcHRpb24gY291bGQgb2ZmZXLigKYgSW4gcGFydGljdWxh
ciBpZiB0aGUgQWNjRUNOIG9wdGlvbiBnZXRzIHdpZGVseSBkZXBsb3llZCBhbmQgdXNlZCwgd291
bGQgc29tZSBtYWxpY2lvdXMgdXNlcnMgZmluZCB0aG9zZSBieXRlcyB1c2VmdWw/IEZvciB3aGF0
PyBGb3IgaW5zdGFuY2UsIHdoYXQgaW4gdGhlIEFjY0VDTiBwcm90b2NvbCBkZXNpZ24gcHJldmVu
dHMgdXNlIG9mIHRoZXNlIOKAnHBhZGRpbmfigJ0gYnl0ZXMgYXMgc3VwZXItY29va2llPyBUbyBt
ZSwgaXQgaXMgbXVjaCBtb3JlIGltcG9ydGFudCB0byB0aGluayBhYm91dCBhYnVzZSBvZiB3aGF0
ZXZlciB3ZSBzdGFuZGFyZGl6ZSAqbm93KiwgaW5zdGVhZCBvZiBlbmdpbmVlcmluZyBmb3Igc29t
ZSBlbnRpcmVseSB1bmtub3duIGZ1dHVyZS4gSSBhbSBub3QgYSBzZWN1cml0eSBleHBlcnQsIGJ1
dCBJIHN1Z2dlc3QgYSBjYXJlZnVsIGFuYWx5c2lzIG9mIGFueSBzZWN1cml0eSBpbXBsaWNhdGlv
bnMgb2YgdGhlIEFjY0VDTiBvcHRpb24uIFdlIHNlbGRvbWx5IHNwZWNpZnkgbmV3IFRDUCBvcHRp
b25zIGFuZCBtYWxpY2lvdXMgdXNlcnMgd2lsbCBjZXJ0YWlubHkgbG9vayBmb3Ig4oCcb3Bwb3J0
dW5pdGllc+KAnSBpbiB0aGUgc3BlYy4gU28gZmFyLCBJIGRvbuKAmXQga25vdyB3aGV0aGVyIHRo
YXQgd2hvbGUgaWRlYSBvZiBmb3J3YXJkIGNvbXBhdGliaWxpdHkgaXMgYSBmZWF0dXJlIG9yIGEg
YnVnLg0KPg0KPg0KPg0KPiBNaWNoYWVsDQo+DQo+DQo+DQo+IE1heWJlIG9uZSBsZXNzb24gbGVh
cm50IGlzIHRoYXQgdGhlIGRvY3VtZW50IGNvdWxkIGhhdmUgYSBub24tbm9ybWF0aXZlIGFwcGVu
ZGl4IHRoYXQgZXhwbGFpbnMgdGhlIHJhdGlvbmFsZSBmb3IgdGhlIGZpbmFsbHkgcGlja2VkIFRD
UCBvcHRpb24gZW5jb2RpbmcuIFRoYXQgbWF5IGFsc28gaGVscCBpZiB0aGVyZSBhcmUgZnVydGhl
ciBxdWVzdGlvbnMgd2hldGhlciB0d28gY29kZXBvaW50cyBhcmUgcmVhbGx5IHJlcXVpcmVkLCBl
LmcuIGJ5IHRoZSBJRVNHIChpZiB0d28gY29kZXBvaW50cyBhcmUgc3RpbGwgdGhlIGRlc2lnbiBh
ZnRlciBXR0xDKS4gQXQgbGVhc3QgZm9yIHBhc3QgVENQIG9wdGlvbiBjb2RlcG9pbnQgYWxsb2Nh
dGlvbnMgSSByZWNhbGwgc29tZSBkaXNjdXNzaW9ucyBsYXRlIGluIHRoZSBJRVRGIHByb2Nlc3Mu
IEluIHRob3NlIHBhc3QgY2FzZXMsIGdvb2QgYXJndW1lbnRzIGluIGFuIGFwcGVuZGl4IGFuZCBy
dW5uaW5nIGNvZGUgaGFzIGhlbHBlZCBhIGxvdC4NCj4NCj4NCj4gW0JCXSBJIGNhbiBkbyB0aGF0
LiBBcHB4IEIgaXMgYWxyZWFkeSBzaW1pbGFyLCBnaXZpbmcgRGVzaWduIFJhdGlvbmFsZSBmb3Ig
dGhlIEFDRSBmaWVsZC4NCj4NCj4NCj4gQm9iDQo+DQo+DQo+DQo+DQo+DQo+IE1pY2hhZWwNCj4N
Cj4NCj4NCj4NCj4NCj4NCj4NCj4gRnJvbTogQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2NvZS5u
ZXQ+DQo+IFNlbnQ6IFR1ZXNkYXksIE5vdmVtYmVyIDE3LCAyMDIwIDY6MTAgQU0NCj4gVG86IFNj
aGFyZiwgTWljaGFlbCA8TWljaGFlbC5TY2hhcmZAaHMtZXNzbGluZ2VuLmRlPjsgWW9zaGlmdW1p
IA0KPiBOaXNoaWRhIDxuc2QuaWV0ZkBnbWFpbC5jb20+DQo+IENjOiBNaWNoYWVsIFR1ZXhlbiA8
dHVleGVuQGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IA0KPiBNaXJqYSBL
dWVobGV3aW5kIDxpZXRmQGt1ZWhsZXdpbmQubmV0PjsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZCAN
Cj4gPFJpY2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tPjsgdGNwbSBJRVRGIGxpc3QgPHRj
cG1AaWV0Zi5vcmc+DQo+IFN1YmplY3Q6IFJlOiBBY2NFQ04gZmllbGQgb3JkZXINCj4NCj4NCj4N
Cj4gTWljaGFlbCwNCj4NCj4gT24gMTYvMTEvMjAyMCAxNzozNiwgU2NoYXJmLCBNaWNoYWVsIHdy
b3RlOg0KPg0KPiBCb2IsDQo+DQo+DQo+DQo+IE9uZSBwcm9wb3NhbCB1c2luZyB0aGUgbGVuZ3Ro
IGZpZWxkIHdpdGggKm9uZSBvcHRpb24gY29kZXBvaW50IG9ubHkqIA0KPiBpcyBkZXRhaWxlZCBp
bjogDQo+IGh0dHBzOi8vbWFpbGFyY2hpdmUuaWV0Zi5vcmcvYXJjaC9tc2cvdGNwbS96by0xT1Iw
blJmaEhvY1g4eXZUdnBDNEJOTW8NCj4gLw0KPg0KPg0KPg0KPiBJdCBpcyB0aGUgdGhpcmQgb3B0
aW9uIG1lbnRpb25lZCBpbiB0aGlzIGUtbWFpbC4gT25lIGV4YW1wbGUgd291bGQgYmUgdG8gdXNl
IG9wdGlvbiBsZW5ndGggdmFsdWVzIDUvOC8xMSBmb3Igb25lIGVuY29kaW5nIHR5cGUgYW5kIG9w
dGlvbiBsZW5ndGggdmFsdWVzIDYvOS8xMiBmb3IgdGhlIG90aGVyIGVuY29kaW5nIHR5cGUgKGku
ZS4sIG9yZGVyIG9mIGZpZWxkcykuIE9yIG9uZSBjb3VsZCB1c2Ugc29tZSBvdGhlciBjb21iaW5h
dGlvbiBvZiBsZW5ndGggdmFsdWVzIOKAkyB0aGUgb25seSByZXF1aXJlbWVudCBpcyB0aGF0IGEg
Y2VydGFpbiB2YWx1ZSBmb3IgdGhlIG9wdGlvbiBsZW5ndGggaXMgb25seSB1c2VkIGJ5IG9uZSBv
ZiB0aGUgb3B0aW9uIGZvcm1hdHMuIEluIHRoYXQgYXBwcm9hY2gsIHRoZSB2YWx1ZSBvZiB0aGUg
bGVuZ3RoIGZpZWxkIHdvdWxkIHRodXMgZGlyZWN0bHkgZGVzY3JpYmUgdGhlIGVuY29kaW5nIG9m
IHRoZSBvcHRpb24uIFVubGVzcyBJIG1pc3Mgc29tZXRoaW5nLCB0aGlzIHdvdWxkIHdvcmsgYW5k
IGl0IHdvdWxkIGp1c3QgcmVxdWlyZSBvbmUgb3B0aW9uIGNvZGVwb2ludC4NCj4NCj4NCj4NCj4g
VGh1cywgYWx0ZXJuYXRpdmVzIHRvIHR3byBvcHRpb24gY29kZXBvaW50cyBleGlzdCBhbmQgSSBo
YXZlIGV4cGxhaW5lZCB0aGVtIG9uIHRoZSBsaXN0IGluIE1hcmNoIDIwMjAuDQo+DQo+DQo+IE9L
LCBzb3JyeSwgeWVzLCBJIHJlbWVtYmVyIHRoaXMgbm93Lg0KPg0KPiBBcyBJIHdpbGwgZXhwbGFp
biBpbiB0aGUgQWNjRUNOIHN0YXR1cyB1cGRhdGUgdGFsayB0b2RheSBpbiB2aXJ0dWFsIEJhbmdr
b2ssIHRoZSBkcmFmdCBoYXMgbWFkZSBwcm92aXNpb24gZm9yIGRpZmZlcmVudCBsZW5ndGggdmFs
dWVzIHRoYW4gNS84LzExLiBJdCBzYXlzIGV4aXN0aW5nIGltcGxlbWVudGF0aW9ucyBNVVNUIGFj
Y2VwdCBsZW5ndGggdmFsdWVzIG90aGVyIHRoYW4gdGhvc2UgY3VycmVudGx5IGRlZmluZWQuIEJ1
dCB0aGVuIHJlYWQgaW4gYXMgbWFueSB3aG9sZSAzLWJ5dGUgZmllbGRzIGFzIHRoZXkgY2FuLg0K
Pg0KPiBUaGlzIGNhbiBiZSB1c2VkIHRvIGFkZCBhIGZsYWdzIGJ5dGUgb24gdGhlIGVuZCBpbiBm
dXR1cmUsIGZvciBleHRlbnNpYmlsaXR5LiBPciBhbnkgb3RoZXIgZm9ybSBvZiBleHRlbnNpYmls
aXR5IHRoZSBXRyBtaWdodCBkZWNpZGUgaW4gdGhlIGZ1dHVyZS4NCj4NCj4gSSBrbm93IGEgZmxh
Z3MgYnl0ZSBhdCB0aGUgZW5kIHNlZW1zIG9kZCBjb21wYXJlZCB0byBhdCB0aGUgYmVnaW5uaW5n
LiBCdXQgKGlmIGRlY2lkZWQgaXQncyBuZWVkZWQgaW4gZnV0dXJlKSBpdCdzIHJlYXNvbmFibHkg
ZWFzeSB0byBpbXBsZW1lbnQgYnkgcmVhZGluZyB0aGUgd2hvbGUgb3B0aW9uLCB0aGVuIHByb2Nl
c3NpbmcgdGhlIGxhc3QgYnl0ZSwgYmVmb3JlIHJlYWRpbmcgdGhlIHJlc3Qgb2YgdGhlIG9wdGlv
bi4NCj4NCj4gSSBiZWxpZXZlIHlvdSB3aWxsIGFncmVlIHRoYXQgdGhpcyBpcyBhIGJldHRlciB3
YXkgdG8gdXRpbGl6ZSBkaWZmZXJlbnQgbGVuZ3Rocy4NCj4NCj4gQW5kIHRoYW5rIHlvdSBmb3Ig
cmVwZWF0ZWRseSBlbXBoYXNpemluZyB0aGF0IHlvdSdyZSBoYXBweSB3aXRoIHRoZSAyLWtpbmQg
c2NoZW1lLCBvciBvdGhlciBhbHRlcm5hdGl2ZXMuDQo+DQo+DQo+IEJvYg0KPg0KPg0KPg0KPg0K
Pg0KPg0KPiBBbnl3YXksIEkgZG9u4oCZdCByZWFsbHkgY2FyZSBob3cgdGhlIG9wdGlvbnMgYXJl
IGVuY29kZWQgYXMgbG9uZyBhcyB0aGUgcmVjZWl2ZXIgZG9lc27igJl0IG5lZWQgcGVyLWNvbm5l
Y3Rpb24gc3RhdGUgZm9yIGRlY29kaW5nIGEgVENQIG9wdGlvbi4gU28sIHBlcnNvbmFsbHksIEkg
d291bGQgYmUgZmluZSB3aXRoIHVzaW5nIGUuZy4gdGhlIGxlbmd0aCBmaWVsZCBhcyBkZXNjcmli
ZWQgaW4gbXkgb2xkIGUtbWFpbC4gT3IgYW4gYWRkaXRpb25hbCBmbGFnIGJ5dGUuIEFuZCBvbmUg
Y291bGQgY29tZSB1cCB3aXRoIGZ1cnRoZXIgZW5jb2RpbmdzLCBlLmcuLCBieSB1c2luZyBvbmUg
b3IgYSBmZXcgYml0cyBhcyBhIHNob3J0IOKAnHR5cGXigJ0gZmllbGQgZm9yIGVhY2ggY291bnRl
ci4gVGhpcyBpcyBhbGwgYWJvdXQgcHJvdG9jb2wgZW5naW5lZXJpbmcuIEFuZCBhbGwgdGhlc2Ug
dmFyaWFudHMgaGF2ZSB0aGVpciBwcm9zIGFuZCBjb25zLg0KPg0KPg0KPg0KPiBJIGFtIGFsc28g
ZmluZSB3aXRoIHVzaW5nIHR3byBvcHRpb24gY29kZXBvaW50cyBhcyBzcGVjaWZpZWQgaW4gLTEz
OyB0aGlzIGlzIHByb2JhYmx5IHRoZSBhcHByb2FjaCB0aGF0IGNvbnN1bWVzIHRoZSBsZWFzdCBu
dW1iZXIgb2YgYml0cy4NCj4NCj4NCj4NCj4gTWljaGFlbCAody9vIGFueSBoYXQpDQo+DQo+DQo+
DQo+DQo+DQo+DQo+DQo+IEZyb206IEJvYiBCcmlzY29lIDxpZXRmQGJvYmJyaXNjb2UubmV0Pg0K
PiBTZW50OiBNb25kYXksIE5vdmVtYmVyIDE2LCAyMDIwIDU6NTIgUE0NCj4gVG86IFlvc2hpZnVt
aSBOaXNoaWRhIDxuc2QuaWV0ZkBnbWFpbC5jb20+DQo+IENjOiBTY2hhcmYsIE1pY2hhZWwgPE1p
Y2hhZWwuU2NoYXJmQGhzLWVzc2xpbmdlbi5kZT47IE1pY2hhZWwgVHVleGVuIA0KPiA8dHVleGVu
QGZoLW11ZW5zdGVyLmRlPjsgdGNwbS1jaGFpcnNAaWV0Zi5vcmc7IE1pcmphIEt1ZWhsZXdpbmQg
DQo+IDxpZXRmQGt1ZWhsZXdpbmQubmV0PjsgU2NoZWZmZW5lZ2dlciwgUmljaGFyZCANCj4gPFJp
Y2hhcmQuU2NoZWZmZW5lZ2dlckBuZXRhcHAuY29tPjsgdGNwbSBJRVRGIGxpc3QgPHRjcG1AaWV0
Zi5vcmc+DQo+IFN1YmplY3Q6IEFjY0VDTiBmaWVsZCBvcmRlcg0KPg0KPg0KPg0KPiBZb3NoaSwg
KGFkZGluZyB0aGUgdGNwbSBsaXN0IGluIGNjKQ0KPg0KPiBPbiAwNS8xMS8yMDIwIDA2OjU4LCBZ
b3NoaWZ1bWkgTmlzaGlkYSB3cm90ZToNCj4NCj4gSGkgQm9iLA0KPg0KPg0KPg0KPiBPbiBXZWQs
IE5vdiA0LCAyMDIwIGF0IDM6MjkgUE0gQm9iIEJyaXNjb2UgPGlldGZAYm9iYnJpc2NvZS5uZXQ+
IHdyb3RlOg0KPg0KPiBZb3NoaSwNCj4NCj4gT24gMDQvMTEvMjAyMCAwNjo1MSwgWW9zaGlmdW1p
IE5pc2hpZGEgd3JvdGU6DQo+DQo+IEhpLCBmb2xrcywNCj4NCj4NCj4NCj4NCj4NCj4NCj4gSW4g
bXkgdW5kZXJzdGFuZGluZywgSSdtIG5vdCBzdXJlIGlmIHdlIHNldHRsZWQgZG93biBvbiB1c2lu
ZyB0d28gb3B0aW9uIGtpbmRzIG9yIGVuY29kaW5nIHNjaGVtZXMgZm9yIDI0Yml0cyBmaWVsZHMg
aW4gYWNjIGVjbiBkcmFmdC4NCj4NCj4gU28sIEkgdGhpbmsgdGhlcmUncmUgc3RpbGwgc29tZXRo
aW5nIHRvIGJlIGNsYXJpZmllZCBhbmQgaG9wZSB0aGluZ3Mgd2lsbCBiZSBzZXR0bGVkIGF0IHRo
ZSBtZWV0aW5nLg0KPg0KPg0KPiBbQkJdIEkga25vdyBhIFdHIGNhbiBjaGFuZ2UgaXQncyBtaW5k
IGF0IGFueSB0aW1lLiBCdXQgSSdkIHJhdGhlciB3ZSBqdXN0IGNsYXJpZmllZCB3aGF0IGEgcHJl
dmlvdXMgZGVjaXNpb24gd2FzLCB0byBhdm9pZCB0aGUgbmVlZCB0byBrZWVwIHJlLW9wZW5pbmcg
ZGlzY3Vzc2lvbiBvbiBhIHF1ZXN0aW9uIHRoYXQgaGF2ZSBiZWVuIGRlY2lkZWQgdGhlbiBjaGFu
Z2VkIHRocmVlIGRpZmZlcmVudCB3YXlzIGFscmVhZHkuDQo+DQo+IE15IG1lbW9yeSBpcyBub3Qg
c28gZ29vZCB0aGVzZSBkYXlzLiBJIHRydXN0ZWQgdGhhdCBNaWNoYWVsIFMgcmVtZW1iZXJlZCB0
aGUgZGVjaXNpb24gY29ycmVjdGx5LCBhbmQgSSBzZWVtIHRvIHJlbWVtYmVyIHRoYXQgZGVjaXNp
b24gYmVpbmcgbWFkZS4NCj4gSSd2ZSBqdXN0IGNoZWNrZWQgdGhlIG1pbnV0ZXMgb2YgdGhlIGxh
c3QgaW50ZXJpbToNCj4gICAgIA0KPiBodHRwczovL2RhdGF0cmFja2VyLmlldGYub3JnL2RvYy9t
aW51dGVzLWludGVyaW0tMjAyMC10Y3BtLTAxLTIwMjAwNDI5DQo+IDE2MDAvIGFuZCB0aGV5IG1l
bnRpb24gTWljaGFlbCdzIHByb3Bvc2FsIHRvIHVzZSB0d28ga2luZHMsIGJ1dCBkb24ndCANCj4g
cmVjb3JkIGFueSBkZWNpc2lvbi4NCj4gVGhlIGphYmJlciBsb2cgZ2l2ZXMgbm8gY2x1ZXMgYWJv
dXQgYW55IGRlY2lzaW9uLg0KPg0KPiBJIGNhbid0IGZpbmQgYW4gYXVkaW8gb3IgdmlkZW8gcmVj
b3JkaW5nLiBDYW4geW91IHBvaW50IG1lIGF0IG9uZT8NCj4NCj4NCj4NCj4gSSB0aG91Z2h0IHRo
YXQgaXQncyBiZWNhdXNlIHRoZXJlIHdhcyBubyBjbGVhciBkZWNpc2lvbiBhdCB0aGUgbWVldGlu
Zy4NCj4NCj4gQnV0LCB5b3UgY2FuIGNoZWNrIGh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL3dhdGNo
P3Y9S3NCMF9OYjgta0ENCj4NCj4gUGxlYXNlIGxldCB1cyBrbm93IGlmIHlvdSBoYXZlIGFueSBx
dWVzdGlvbnMgb3Igb3BpbmlvbnMgd2l0aCByZWdhcmQgdG8gdGhpcy4NCj4NCj4NCj4gW0JCXSBJ
IGNoZWNrZWQgdGhlIFlvdXR1YmUgbGluayB5b3Ugc2VudCBiZWxvdy4NCj4NCj4gRmlyc3QgSSB0
aGluayB3ZSdyZSBhZ3JlZWQgbm8tb25lIHdhcyBmaWdodGluZyBmb3IgdXMgdG8ga2VlcCB0aGUg
cHJldmlvdXMgd2F5IHdlIGRpZCB0aGlzICh1c2luZyB0aGUgaW5pdGlhbCB2YWx1ZSBvZiB0aGUg
ZmllbGQgdG8gc2V0IHRoZSBvcmRlciBmb3IgdGhlIGNvbm5lY3Rpb24pLg0KPiBJbiBteSBwcmVz
ZW50YXRpb24gSSBzYWlkIHRoZXJlIHdhcyBzdHJvbmcgcmVzaXN0YW5jZSBmcm9tIE1pY2hhZWwg
dG8gZG8gaXQgYSBkaWZmZXJlbnQgd2F5Lg0KPiAoYWxzbywgb2ZmbGlzdCwgdGhlIGNvLWF1dGhv
cnMgaW5jbHVkaW5nIG1lIGFsc28gZGlkbid0IGxpa2UgdGhpcyBzbyANCj4gbXVjaC4gQW5kIEls
cG8gc2FpZCBpdCBtYWRlIHRoZSBpbXBsZW1lbnRhdGlvbiBjb21wbGV4LikNCj4NCj4gVGhlbiBj
YW1lIHRoZSBxdWVzdGlvbiBvZiB3aGF0IHdlIGRvIGluc3RlYWQuIFRoZXJlIHdlcmUgdGhyZWUg
YWx0ZXJuYXRpdmUgcHJvcG9zYWxzOg0KPiBhKSB1c2UgMiBvcHRpb24ga2luZHMNCj4gYikgYWRk
IGEgZmxhZ3MgYnl0ZQ0KPiBjKSBzb21laG93IHVzZSB0aGUgbGVuZ3RoIGZpZWxkIG1heWJlDQo+
DQo+IE1pY2hhZWwgcmFpc2VkIChjKSBpbiB0aGUgbWVldGluZyBhcyBhIHBvc3NpYmlsaXR5LCBi
dXQgbm8tb25lIGNvdWxkIHRoaW5rIGhvdyB0byBkaXN0aW5ndWlzaCB0d28gb3B0aW9ucyBvZiB0
aGUgc2FtZSBsZW5ndGggYnV0IGEgZGlmZmVyZW50IGZpZWxkIG9yZGVyIHVzaW5nIHRoZSBsZW5n
dGggZmllbGQuIE1pY2hhZWwgc2FpZCBoZSdkIHBvc3QgYW55IGlkZWFzIHRvIHRoZSBsaXN0IGlm
IGhlIGNvdWxkIHRoaW5rIG9mIGFueSwgYnV0IHRoYXQgZGlkbid0IGhhcHBlbi4NCj4NCj4gU28g
d2UncmUgbGVmdCBjaG9vc2luZyBiZXR3ZWVuIChhKSBhbmQgKGIpLg0KPiBJIHNhaWQgaW4gdGhl
IG1lZXRpbmcgKGFuZCBvbiB0aGUgbGlzdCB3aGVuIGRpc2N1c3Npbmcgd2l0aCBJbHBvKSB0aGF0
IEknZCBiZSBoYXBweSB0byBnbyB3aXRoIChiKSwgYnV0IG9ubHkgaWYgdGhlcmUgd2FzIGFub3Ro
ZXIgdXNlIGZvciBhIGZsYWcuIEJlY2F1c2UgaXQgd291bGQgY29uc3VtZSAxQiBtb3JlIG9wdGlv
bnMgc3BhY2UgaW4gbWFueSBwYWNrZXRzLCB3aGljaCBpcyBhIHNjYXJjZSByZXNvdXJjZS4NCj4N
Cj4gSWxwbyBoYWQgYSBwcm9wb3NlZCB1c2UgZm9yIGFub3RoZXIgZmxhZyAodG8gaGVscCBzeW5j
aCBjb3VudGVycyBhZnRlciBhIGxvc3MpLCBidXQgSSB0aGluayB0aGUgZGlzY3Vzc2lvbiBhYm91
dCBpdCBlbmRlZCB0aGF0IGl0IHdvdWxkbid0IGJlIGhlbHBmdWwsICdjb3MgdGhlIHdheSBpdCB3
b3JrZWQgZGVwZW5kZWQgb24gaXRzZWxmIChjaXJjdWxhciBsb2dpYykuDQo+DQo+IEluIGNvbmNs
dXNpb24sIEkgZG9uJ3QgdGhpbmsgdGhlcmUgd2FzIGFuIGV4cGxpY2l0IGRlY2lzaW9uIHRvIGdv
IHdpdGggMiBvcHRpb24ga2luZHMsIGJ1dCBpdCBlbmRlZCB1cCBhcyB0aGUgJ2xhc3QgcGVyc29u
IHN0YW5kaW5nJy4NCj4gSSBsaWtlIGl0LiBJdCdzIHNpbXBsZS4gQW5kIGFwcGFyZW50bHkgb3B0
aW9uIGtpbmRzIGFyZSBub3Qgc3VjaCBhIHNjYXJjZSByZXNvdXJjZS4NCj4NCj4gUGVyaGFwcyB3
ZSBjYW4gcmF0aWZ5IHRoaXMgaW4gdGhlIFdHIHRvbW9ycm93Lg0KPg0KPg0KPiBCb2INCj4NCj4N
Cj4NCj4NCj4NCj4NCj4NCj4gVGhhbmtzLA0KPg0KPiAtLQ0KPg0KPiBZb3NoaQ0KPg0KPg0KPg0K
Pg0KPg0KPg0KPiAtLQ0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fDQo+IEJvYiBCcmlzY29lICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIGh0dHA6Ly9ib2JicmlzY29lLm5ldC8NCj4NCj4NCj4NCj4NCj4gLS0NCj4g
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fXw0KPiBCb2IgQnJpc2NvZSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBodHRw
Oi8vYm9iYnJpc2NvZS5uZXQvDQo+DQo+DQo+DQo+IC0tDQo+IF9fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4gQm9iIEJyaXNj
b2UgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgaHR0cDovL2JvYmJyaXNjb2UubmV0Lw0K
PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiB0Y3Bt
IG1haWxpbmcgbGlzdA0KPiB0Y3BtQGlldGYub3JnDQo+IGh0dHBzOi8vd3d3LmlldGYub3JnL21h
aWxtYW4vbGlzdGluZm8vdGNwbQ0KDQo=


From nobody Mon Nov 23 17:24:55 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5C7A03A09D8; Mon, 23 Nov 2020 17:24:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0
X-Spam-Level: 
X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id a4R-UF7n4lLw; Mon, 23 Nov 2020 17:24:50 -0800 (PST)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7B3033A09D4; Mon, 23 Nov 2020 17:24:50 -0800 (PST)
Received: by rfc-editor.org (Postfix, from userid 30) id DC4AEF40780; Mon, 23 Nov 2020 17:24:47 -0800 (PST)
To: ietf-announce@ietf.org, rfc-dist@rfc-editor.org
X-PHP-Originating-Script: 1005:ams_util_lib.php
From: rfc-editor@rfc-editor.org
Cc: rfc-editor@rfc-editor.org, drafts-update-ref@iana.org, tcpm@ietf.org
Content-type: text/plain; charset=UTF-8
Message-Id: <20201124012447.DC4AEF40780@rfc-editor.org>
Date: Mon, 23 Nov 2020 17:24:47 -0800 (PST)
Archived-At: 
Subject: [tcpm] =?utf-8?q?BCP_233=2C_RFC_8961_on_Requirements_for_Time-Ba?= =?utf-8?q?sed_Loss_Detection?=
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 01:24:54 -0000

A new Request for Comments is now available in online RFC libraries.

        BCP 233        
        RFC 8961

        Title:      Requirements for Time-Based Loss Detection 
        Author:     M. Allman
        Status:     Best Current Practice
        Stream:     IETF
        Date:       November 2020
        Mailbox:    mallman@icir.org
        Pages:      12
        See Also:   BCP 233

        I-D Tag:    draft-ietf-tcpm-rto-consider-17.txt

        URL:        https://www.rfc-editor.org/info/rfc8961

        DOI:        10.17487/RFC8961

Many protocols must detect packet loss for various reasons (e.g., to
ensure reliability using retransmissions or to understand the level
of congestion along a network path).  While many mechanisms have been
designed to detect loss, ultimately, protocols can only count on the
passage of time without delivery confirmation to declare a packet
"lost". Each implementation of a time-based loss detection mechanism
represents a balance between correctness and timeliness; therefore,
no implementation suits all situations.  This document provides
high-level requirements for time-based loss detectors appropriate for
general use in unicast communication across the Internet.  Within the
requirements, implementations have latitude to define particulars
that best address each situation.

This document is a product of the TCP Maintenance and Minor Extensions Working Group of the IETF.


BCP: This document specifies an Internet Best Current Practices for the
Internet Community, and requests discussion and suggestions for 
improvements. Distribution of this memo is unlimited.

This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
  https://www.ietf.org/mailman/listinfo/ietf-announce
  https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist

For searching the RFC series, see https://www.rfc-editor.org/search
For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk

Requests for special distribution should be addressed to either the
author of the RFC in question, or to rfc-editor@rfc-editor.org.  Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.


The RFC Editor Team
Association Management Solutions, LLC


From nobody Tue Nov 24 05:24:03 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1CFCC3A0CD0; Tue, 24 Nov 2020 05:23:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.001
X-Spam-Level: 
X-Spam-Status: No, score=-0.001 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=gmx.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hsziFYaSnuol; Tue, 24 Nov 2020 05:23:01 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 098593A0CD8; Tue, 24 Nov 2020 05:23:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1606224174; bh=gwBLMDOH1k4ygv4CcVVdLFQg4My6RbPXb3qTBPCxscs=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=Lz3qbjrN1iqglggOa7hcM2qKO3KtOetR8wMab8MmiEMmJCt61DGw5pfDx1KKU/XHR ZVt9xUNLpwDlAfhe4dEcXBBddZGpHW223cZOak2FC3gcnTRrDzDdrKBcDQCajugtbv k1f2vPDgUHHCKGE2WWaoBMSXvVwWRkAFymeR8cSE=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [10.249.68.57] ([217.70.211.16]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1Mxm3K-1kJ1eK0lfq-00zFpa; Tue, 24 Nov 2020 14:22:54 +0100
To: Kevin Yang 
Cc: Yuchung Cheng , "tcpm@ietf.org Extensions" , Eric Dumazet , iccrg IRTF list , Neal Cardwell 
References: <160435977209.20839.4083263519198538783@ietfa.amsl.com>      
From: "Scheffenegger, Richard" 
Message-ID: <8b5b8012-0914-c833-6390-0e98a97074ad@gmx.at>
Date: Tue, 24 Nov 2020 14:22:51 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Provags-ID: V03:K1:WYgCs+QDkJAIx993BpbNafeFXUNw1tZ/SwiKlD0v1spwYbWc44F mbk9sn7lePa9SB2vMA1P8qYNqwIQhZ341Ck0LntN+0MOIFW0K7IhSpRTHyw9NMgUUVCHeQj Ykx0uqiAfg2VGfKbEK5ilRtvVFb+vzKiEe4usXXAxRnmA+hoGHNLJdFHunr9AZlghPp1mxX V+AahwUL++xYcJL6MRiSg==
X-UI-Out-Filterresults: notjunk:1;V03:K0:fu2x13jebEY=:yDYBfVvkvBklPBMfjKA4vc 2uY4z87VaTLw0BQZpa2qOs0hKd41phafoa9iqr587PnV4TvgNh6dS7MAVfmr37az0Dxh11oSU tMjlX9O7nMS4hfM5zS9xLH1X8fY1mtkhMxoUP78nKi2CRcTpw+ZZunJEqkGY52k4LsVPChaEo ukJ7NTwSNng1NAoVluogkAIrIcFJPLk+eoQvQ2D0MWfiGlNisl9CYLGqIsbyp6HaWVEwWWk2c zDLv3gGf6GjjluYfT3uX5TpULXvmoJ2m82LuhY+AramOnjoVtXkjj90/K3Rcu8Ee6e/DpLM/k eRJCWZCWBWOBdBq6ExrBPlo9izLtCHGH4mhPMJyR+2Elu7y4hPGqjkbvPKx0XaFvIRwzic9St WvmMpeGi6UbXcvuW1QiczRj47XMhPk1gwIWI0hleu1pwh462AEtvm1QlP8bFCLfH837pfxP5T A6RMfD/ftY8e5GLPcvmj+nwxm29ZIJrsIeBOfiufoZbdibyA/rCI2P31bDUNpMxamAWBgRbJY u95e2rP08SWWeorbvdFbGUUtTK0KoInRk/Th1A6DKWhrh8bFSQbaa9sI/EytnhtykZbgiyi4X o58pFbgVNqpPhv0QimOeDid1jAhjIEWOd+PyDIH8jvQnw/1PglphWYQ3UWZ66DitViSIrEHkQ SZJGA8t5Wz9CpFOEY3IDEm9jNh19CIOlUljHln8NeeUO6fAYmhOIRWy6u+7lGdGbNKLq1whtb inT2gCstBGBHKFCJ2gRbWAqXW7RIGM5FHywHArClLr7KmB6jxGUuS5Y1ld+M5UHdfimQhZIRf qN2clBvUIvNS/akSNDqnGkIfYOjvJc4jgPRj9uoJkSoZVuB3VXE4QT4MMHkRs4yBE3adej0SD mzcepYtfWk1RB2BRCjfQ==
Archived-At: 
Subject: Re: [tcpm] Fwd: New Version Notification for draft-yang-tcpm-ets-00.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 13:23:03 -0000

Am 23.11.2020 um 02:47 schrieb Kevin Yang:
>> The high level approach there was to use the unused TSecr field (should
>> be 0 for 7323/1323), and overlay (xor) in the SYN/ACK's TSopt ecr field
>> the client's TSopt val with the servers 32-bit negotiation field... As
>> the client can retain knowledge of the initial TSopt value (there shoul=
d
>> be a randomized initial value), the client can extract the server side
>> options by xor'ing it's known initial value when the SYN/ACK is
>> received. Or fall back to default 7323 operation, if the result of the
>> XOR is zero...
>>
>
> I agree that the TSecr in  might be used for other purposes. But I'=
m
> not sure if we can alter the TSecr in  by XOR other info.
> One specific example I came up with is that a sender may send out many
> SYNs because of timeout,
> then when it receives a , it does not know which TSval should
> be used for xor op.

(E)TS opt is not necessarily the old differentiation between subsequent
SYN packets; extending some reserved bits (during the SYN handshake) may
be another easy way to have sufficient differentiation available.

E.g. The EcrDel value may be restricted to no less than 4 or 8 usec (any
lower value rounded up), yielding sufficient bits which would expected
to be zero in the SYN,ACK; The TSval offset can then be selected by the
sender in such a way, to change at these locations for each
retransmission of the SYN, allowing the sender to disambiguate which SYN
a particular SYN,ACK pairs up with...



>> It could be used almost as a drop-in use of timestamp-negotiation, as
>> long as it can be guaranteed, that at least one bit is set in the 32bit
>> field (e.g. setting the reserved bit to 1).
>
> In this way we need to add another "ETSval'' in usec if we use the
> RFC7323/1323 TSopt,
> the reason is explained above.

As long as some more coarse time resolution is acceptable during the
(special case handling) 3WHS, all you need is a few bits to disambiguate
retransmitted SYN packets. We did spent some thought on the
retransmitted SYN problem during the TS-nego design phase. A single
reserved bit is not sufficient, though;

>
>> EcrDelUnit: I think the codepoint assignment could be improved (but thi=
s
>> is really nit-picking; just that a broken implementation may send out
>> TSopt with len=3D12, but initialized-to-zero superflous bytes...
>> 0: invalid
>> 1: milliseconds [2^20 nanoseconds?]
>> 2: microseconds [2^10 nanoseconds?]
>> 3: reserved (nanoseconds)
>
> Thanks for the nice suggestion. I agree we probably should have
> all-zero codepoint to be
> "no information". For 2^k nanosec unit, I guess that is for bit-op
> convenience and performance?

Yes, simple binary shift operations instead of multiply/divide, and
internally representing timestamp values in nanoseconds
(timespec/tv_nsec). The error when using timeval/tv_usec may even be
small enough to not matter in many environments (eg. simply use a
"microsecond 2^10" value directly, without adjusting for the 2.4% error,
as you are interested in deltas on short timescales, not across long
timespans?


From nobody Tue Nov 24 11:06:42 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E503E3A164D; Tue, 24 Nov 2020 11:06:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hckVkfhsEO0U; Tue, 24 Nov 2020 11:06:34 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2216F3A164C; Tue, 24 Nov 2020 11:06:28 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:3143:1df2:74fd:43d0] (unknown [IPv6:2800:810:464:8164:3143:1df2:74fd:43d0]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 73FFC283AE6; Tue, 24 Nov 2020 19:06:24 +0000 (UTC)
To: Alexander Azimov , tcpm , v6ops@ietf.org
References: 
From: Fernando Gont 
Message-ID: 
Date: Tue, 24 Nov 2020 16:05:50 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 19:06:37 -0000

On 19/11/20 07:48, Alexander Azimov wrote:
> Dear colleagues,
> 
> I have added in the cc both v6op and tcpm for a reason and let me 
> explain why.
> 
> It's clear that we are moving forward with load balancing that uses flow 
> label (FL). And the pressure will increase with SRv6 adoption. But at 
> the moment wide adoption of FL-based load-balancing may create 
> significant issues for TCP Anycast services.
> 
> RFC6437 suggests putting hash from 5-tuple into FL value. And as far as 
> I know, there is no document that updates this behavior. This 
> description is perfectly fine, but what is implemented in the Linux 
> kernel is different: FL is carrying hash from 5-tuple with an additional 
> seed, and this seed is randomly changed after each RTO/SYN_RTO event. 

Changing the FL upon RTO is a bug.

I guess/assume that when you say SYN-RTO, you really mean "user 
timeout", rather than RTO. If you don't, then that's also a bug.
If you do, I fail to understand what's the reason for wanting the FL to 
change in that case, because as a result of port randomization, it 0s 
unlikely that the same four-tuple is employed for the next connection retry.


> Here are related patches:
> 
>   * https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.mtv.corp.google.com/
>   * https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@herbertland.com/
>   * https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
> 
> 
> This is a great thing by the way because in the data center 
> environment with multiple equal paths it gives a way to have 
> pseudo-multipath TCP which jumps between paths in case of an outage. 
> There might be interest to writedown an informational document for this.

That's a bad idea, since specs-wise the Flow-Label is not guaranteed to 
remain unchanged from source to destination. If you want to ahve 
multiple paths, then you should implement that in routing.



> I wonder what you think is a proper solution:
> 
>   * Making FL related RTO change as knob instead of default behavior;
>   * Adding negotiation behavior in TCP;
>   * Something else?

Just make the FL a function of the connection "identifier". And keeo it 
constant for the lifetime of that conenction.

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Tue Nov 24 11:28:47 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2EC9B3A1836; Tue, 24 Nov 2020 11:28:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EfVz3TBW_-Bq; Tue, 24 Nov 2020 11:28:39 -0800 (PST)
Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 503FD3A0E2A; Tue, 24 Nov 2020 11:28:33 -0800 (PST)
Received: by mail-ot1-x329.google.com with SMTP id n11so20453734ota.2; Tue, 24 Nov 2020 11:28:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=d2Xk5BepEbljT/Z1DDKDV6TQTWWdAO2+ntkk96OgV6Q=; b=Ny9BF7iUIP3MYWQ1UBgx6UE4q4FXPMsuuVnlVV9p3l1+Mz1f0ySXLW4aeRLzUdjEpU WK/0/E6p61UvOIN8OrHq1My1YXyBVkV0hlztYZO8YsGishtE8XQnrfxY6pm83ZP98owO r9Sm5Cl1fnxR/cvuEktOvCS29S+ih/R/NLQJnVBe5Ae/SDYGMuJ+dHJ1GLf3IO2NX8kT Ly+ga4UhJEiAkiXW77nQrvC86UQvP8KJpA1lsL0kU9H3OsCgzgmXo3ydBHMV76bbBOVf mqgn3yH3KpLfp1yrsNs6MZSh5XIihkZYz9A/j4Rw3PaQyOF8aXmSV48U/dRwNtC5uu8E M0fQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=d2Xk5BepEbljT/Z1DDKDV6TQTWWdAO2+ntkk96OgV6Q=; b=hhzcYOFA8+wRpzuXEapTCOB5VKZB1oy8XQVcr8Pgt7gE89OJ0UJJG0cPa0H1YhWy4n 7OA5IWnbh9cv0I4CIl6cf6kbdUdiYCE7XU9wAZ7d0WV0uj+OJkQX6oOttQ7SIqgjfZa7 sWxDiMTeAZVKWkR9Jt8qhX5RJFZpuh0ys0z5Fh6wEuVHHYvJ9GNod25fx3gzpBwrZLEQ eEcZ1VTQU1bYxrRqlKe6Y/YurmX5p/2+XBRfJGCb1M6QFEzzPfeggX1rYVOtzVFbMdWM J7IvxPS+ahRBwzd/J5dBmgJMQSXW/NdFvhqNrntuKNz5bkf084U1z7aEibXi/39lrbTS eeVQ==
X-Gm-Message-State: AOAM531BzvcM+EREv9CF30SNTX05bT+V3NgjpIk5zNx8G1MrAsWHCOXG prDg3MFqQiMXnoHe8kJuwLcLM4Vwj/69XnuLlBioeLYlN+M=
X-Google-Smtp-Source: ABdhPJwpCkE7Wo9kQRbR/wp1Wr+ihnnRJ5+6pRxlD+/U08o7Gwg896MomdUurN34kHYD2hWwb2a6bX/kH9pay7nM5Ug=
X-Received: by 2002:a9d:4c92:: with SMTP id m18mr108833otf.248.1606246112526;  Tue, 24 Nov 2020 11:28:32 -0800 (PST)
MIME-Version: 1.0
References:  
In-Reply-To: 
From: Alexander Azimov 
Date: Tue, 24 Nov 2020 22:28:21 +0300
Message-ID: 
To: Fernando Gont 
Cc: tcpm , IPv6 Operations 
Content-Type: multipart/alternative; boundary="000000000000f5860905b4df50ad"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 19:28:41 -0000

--000000000000f5860905b4df50ad
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Fernando,

Stating that FL change during TCP session lifetime is a bug - is a bit
harsh.

It is a fantastic idea to change the FL value if RTO or SYN_RTO happens
in a controlled environment.
These are very specific TCP timeouts, that provide enough guarantee that
there will be no out-of-order packets, though your packets will reach
the destination even in case of an outage inside your network. Zero
influence on your services in case of the network outage - doesn't sound
like a bug for me.

The problem is with the current Linux (though I haven't checked other OSes)
defaults. Specifically, with the default behavior after RTO event.

=D0=B2=D1=82, 24 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:06, Ferna=
ndo Gont :

> On 19/11/20 07:48, Alexander Azimov wrote:
> > Dear colleagues,
> >
> > I have added in the cc both v6op and tcpm for a reason and let me
> > explain why.
> >
> > It's clear that we are moving forward with load balancing that uses flo=
w
> > label (FL). And the pressure will increase with SRv6 adoption. But at
> > the moment wide adoption of FL-based load-balancing may create
> > significant issues for TCP Anycast services.
> >
> > RFC6437 suggests putting hash from 5-tuple into FL value. And as far as
> > I know, there is no document that updates this behavior. This
> > description is perfectly fine, but what is implemented in the Linux
> > kernel is different: FL is carrying hash from 5-tuple with an additiona=
l
> > seed, and this seed is randomly changed after each RTO/SYN_RTO event.
>
> Changing the FL upon RTO is a bug.
>
> I guess/assume that when you say SYN-RTO, you really mean "user
> timeout", rather than RTO. If you don't, then that's also a bug.
> If you do, I fail to understand what's the reason for wanting the FL to
> change in that case, because as a result of port randomization, it 0s
> unlikely that the same four-tuple is employed for the next connection
> retry.
>
>
> > Here are related patches:
> >
> >   *
> https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.m=
tv.corp.google.com/
> >   *
> https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@he=
rbertland.com/
> >   *
> https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
> >
> >
> > This is a great thing by the way because in the data center
> > environment with multiple equal paths it gives a way to have
> > pseudo-multipath TCP which jumps between paths in case of an outage.
> > There might be interest to writedown an informational document for this=
.
>
> That's a bad idea, since specs-wise the Flow-Label is not guaranteed to
> remain unchanged from source to destination. If you want to ahve
> multiple paths, then you should implement that in routing.
>
>
>
> > I wonder what you think is a proper solution:
> >
> >   * Making FL related RTO change as knob instead of default behavior;
> >   * Adding negotiation behavior in TCP;
> >   * Something else?
>
> Just make the FL a function of the connection "identifier". And keeo it
> constant for the lifetime of that conenction.
>
> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
>

--=20
Best regards,
Alexander Azimov

--000000000000f5860905b4df50ad
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Hi Fernando,

Stating that FL=
 change during TCP session lifetime is a bug - is a bit harsh.

It is=
 a fantastic idea to change the FL value if RTO or SYN_RTO happens in=C2=A0=
a controlled=C2=A0environment.=C2=A0
These are very specific TCP timeout=
s, that provide enough guarantee that there will be no out-of-order packets=
, though your packets will reach the=C2=A0destination=C2=A0even in case of =
an outage inside your network. Zero influence on your services in case of t=
he network outage - doesn't sound like a bug for me.

The problem=
 is with the current Linux (though I haven't checked other OSes) defaul=
ts. Specifically, with the default behavior after RTO event.

=D0=
=B2=D1=82, 24 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:06, Fernando=
 Gont <fernando@gont.com.ar&=
gt;:
On 19/11/20=
 07:48, Alexander Azimov wrote:

> Dear colleagues,

> 

> I have added in the cc both v6op and tcpm for a reason and let me 

> explain why.

> 

> It's clear that we are moving forward with load balancing that use=
s flow 

> label (FL). And the pressure will increase with SRv6 adoption. But at =


> the moment wide adoption of FL-based load-balancing may create 

> significant issues for TCP Anycast services.

> 

> RFC6437 suggests putting hash from 5-tuple into FL value. And as far a=
s 

> I know, there is no document that updates this behavior. This 

> description is perfectly fine, but what is implemented in the Linux 
> kernel is different: FL is carrying=C2=A0hash=C2=A0from 5-tuple with a=
n additional 

> seed, and this seed is randomly changed=C2=A0after each RTO/SYN_RTO ev=
ent. 



Changing the FL upon RTO is a bug.



I guess/assume that when you say SYN-RTO, you really mean "user 

timeout", rather than RTO. If you don't, then that's also a bu=
g.

If you do, I fail to understand what's the reason for wanting the FL to=
 

change in that case, because as a result of port randomization, it 0s 

unlikely that the same four-tuple is employed for the next connection retry=
.





> Here are related patches:

> 

>=C2=A0 =C2=A0* https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20=
628@tomh.mtv.corp.google.com/

>=C2=A0 =C2=A0* https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-to=
m@herbertland.com/

>=C2=A0 =C2=A0* https://lor=
e.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/

> 

> 

> This is a great thing by the way because in the data center 

> environment=C2=A0with multiple equal paths=C2=A0it gives a way to have=
 

> pseudo-multipath=C2=A0TCP which jumps between paths in case of an outa=
ge. 

> There might be interest to writedown an informational document for thi=
s.



That's a bad idea, since specs-wise the Flow-Label is not guaranteed to=
 

remain unchanged from source to destination. If you want to ahve 

multiple paths, then you should implement that in routing.







> I wonder what you think is a proper solution:

> 

>=C2=A0 =C2=A0* Making FL related RTO change as knob instead of default =
behavior;

>=C2=A0 =C2=A0* Adding negotiation=C2=A0behavior in TCP;

>=C2=A0 =C2=A0* Something else?



Just make the FL a function of the connection "identifier". And k=
eeo it 

constant for the lifetime of that conenction.



Thanks,

-- 

Fernando Gont

e-mail: fernando@=
gont.com.ar || fgont@si6networks.com

PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1










-- 
Best regards,
Alexander Azi=
mov


--000000000000f5860905b4df50ad--


From nobody Tue Nov 24 11:44:13 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 52D3B3A18BB for ; Tue, 24 Nov 2020 11:44:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MB7GbrnmGb4K for ; Tue, 24 Nov 2020 11:44:06 -0800 (PST)
Received: from mail-ej1-x635.google.com (mail-ej1-x635.google.com [IPv6:2a00:1450:4864:20::635]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 306113A18BA for ; Tue, 24 Nov 2020 11:44:06 -0800 (PST)
Received: by mail-ej1-x635.google.com with SMTP id f23so30256046ejk.2 for ; Tue, 24 Nov 2020 11:44:06 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FU2O2RNZOW//Q1QmbUgngJsTZOfiFc8CERjeQLE9RyM=; b=MuM2a2ACT0B2bBsONHR4Y3o1+oAtOwPb+z4PmSsGwxi6u3joaI1HmUOMSmDzAaC8ff CTTKWzxMxGFxR2AAFCPQYOr5iEllhF+MDy396vDZxevgBjsuMUyE2iAZsUBOoJfrvi0p U4iuqFBkvnoG+w0KpTfvL2BBZ+Njln4ss8DEJBIa8gSZ8myS8Q2m7HqMRO69upbk64em 0EqB/PV6bVRm5sgc59EYDld1xedD++td0r1sjydx/tE2JGtNCI2RHniak3muw8Rm6OSM oLE3chsp8+YWhVQZ26mZbW6mwb99JHzgNhtRtceXu4LL4nskp6dD8I4lIbKSWUsd8Uv9 Uyxg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FU2O2RNZOW//Q1QmbUgngJsTZOfiFc8CERjeQLE9RyM=; b=n2GJjIJtROZhaSJ5+KG7vNjQKDa3XRnvY/AsoP4bZu+0gXEbfgDFljWBRzmlfQ29RK fEGNAe+xqgjoIPHwOcYMNyXWzHbmwJAvML9RqwTh3UNSvncoEZ3J7SZzMeZLDAfYkjZr 0Kpxq8stP3mUmjPLHJ55ZfvRhfbzR3JxivTytkVOUuL572dOGPmgM0sTpeWZ3iCiuqDD fu6uurULGWCEtlnTMQ565qcRhftABefni+YwmXK0XuNeM2mBiS0HVLJkqsc3ggo11Tr7 0F4ombTZ2GtIEWbWGTLNXIibMnpY83cf/ZiJhp3tKnD6Fd+o0WZ2Gjvx88pe3zQOzU6G kvxQ==
X-Gm-Message-State: AOAM532/UaN1kFRJo2aSxzwoRM+8RbtvNeWynGLhbkn6zNUGqN+cUmh6 MLwG5/XOMsQ5biRUP/216qZRIYS+PEXrWgcEe70EZg==
X-Google-Smtp-Source: ABdhPJx20aiuCkrIyqkT15ZaWAeIEao5pLNrlTzgaz7F+bCqqZHC3tHUWhpJRQq1DRNlnXGgT4aFYBb8WVuoop/BAxU=
X-Received: by 2002:a17:906:e2da:: with SMTP id gr26mr5337739ejb.265.1606247044480;  Tue, 24 Nov 2020 11:44:04 -0800 (PST)
MIME-Version: 1.0
References:  
In-Reply-To: 
From: Tom Herbert 
Date: Tue, 24 Nov 2020 12:43:52 -0700
Message-ID: 
To: Fernando Gont 
Cc: Alexander Azimov , tcpm ,  IPv6 Operations 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 19:44:08 -0000

On Tue, Nov 24, 2020 at 12:07 PM Fernando Gont  wrote:
>
> On 19/11/20 07:48, Alexander Azimov wrote:
> > Dear colleagues,
> >
> > I have added in the cc both v6op and tcpm for a reason and let me
> > explain why.
> >
> > It's clear that we are moving forward with load balancing that uses flow
> > label (FL). And the pressure will increase with SRv6 adoption. But at
> > the moment wide adoption of FL-based load-balancing may create
> > significant issues for TCP Anycast services.
> >
> > RFC6437 suggests putting hash from 5-tuple into FL value. And as far as
> > I know, there is no document that updates this behavior. This
> > description is perfectly fine, but what is implemented in the Linux
> > kernel is different: FL is carrying hash from 5-tuple with an additional
> > seed, and this seed is randomly changed after each RTO/SYN_RTO event.
>
> Changing the FL upon RTO is a bug.
>
> I guess/assume that when you say SYN-RTO, you really mean "user
> timeout", rather than RTO. If you don't, then that's also a bug.
> If you do, I fail to understand what's the reason for wanting the FL to
> change in that case, because as a result of port randomization, it 0s
> unlikely that the same four-tuple is employed for the next connection retry.
>
>
> > Here are related patches:
> >
> >   * https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.mtv.corp.google.com/
> >   * https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@herbertland.com/
> >   * https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
> >
> >
> > This is a great thing by the way because in the data center
> > environment with multiple equal paths it gives a way to have
> > pseudo-multipath TCP which jumps between paths in case of an outage.
> > There might be interest to writedown an informational document for this.
>
> That's a bad idea, since specs-wise the Flow-Label is not guaranteed to
> remain unchanged from source to destination. If you want to ahve
> multiple paths, then you should implement that in routing.
>
>
>
> > I wonder what you think is a proper solution:
> >
> >   * Making FL related RTO change as knob instead of default behavior;
> >   * Adding negotiation behavior in TCP;
> >   * Something else?
>
> Just make the FL a function of the connection "identifier". And keeo it
> constant for the lifetime of that conenction.

Fernando,

Modulating the flow label is a means to affect the routing of packets
through the network that uses flow labels as input to the ECMP hash.
The basic idea is that the flow label associated with a connection is
randomly changed when the stack observes that the connection is
failing (e.g. and an RTO). There is nothing in the specs that prevents
this since the source is at liberty to set the flow label as it sees
fit. The feature is useful in large datacenter networks, like
pparently Facebook where the patches originate, since information
discerned by TCP can opportunistically be applied to route selection.
The practical issue is that there are stateful devices like firewalls
that require consistent routing in the network in which case changing
the flow label can confuse them. As I mentioned, the original intent
was that the flow label randomization feature should be opt-in instead
of on by default.

Tom
>
> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops


From nobody Tue Nov 24 11:45:43 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C0AB93A18BF; Tue, 24 Nov 2020 11:45:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1DIlQD3lTVTI; Tue, 24 Nov 2020 11:45:40 -0800 (PST)
Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 19B733A18BD; Tue, 24 Nov 2020 11:45:40 -0800 (PST)
Received: by mail-pg1-x533.google.com with SMTP id 81so138414pgf.0; Tue, 24 Nov 2020 11:45:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=l3QdHBNYMCLYS5OFoHm7WqZ7++WzPT2C5Rx7LvQUUW8=; b=fUeU8mppvgPYZNhUoXlm+/13drnVp2Uz0wzmjMDva3PcBGuNfBCbLNvxFqT6/YxU53 taU44jHVA3ZyZpC6D3LOcFPlXRJcPyztJoWbfP3FwfDqIEpb5cMTUumc3qLxTGJHKkZU dhE38FdGD30JlADytwQDHi9VbEhnahvJdPO/eaaK8H/1tRya8BXWAU8COhN4MJtODoZU vewS+IlLCk18vNI2D2LQyR96V6ImDg5PY3uGVfPFbo2fHkBEOri1gja55q6TqtbDSL0X /X9PpCYNyvMsYMy6x2pgKEYgfZBa/2yw6wtSMTnCVc2EGPfNP1JOJDDansFPkps04jhp trug==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=l3QdHBNYMCLYS5OFoHm7WqZ7++WzPT2C5Rx7LvQUUW8=; b=fclLeMBErhrEyMMi7tIvnGUg4mERU0tsvmmWZvumsollFUIkvGX61X6OuDDTxKbdrp 1HtdAesJEPiPbasB+lLUo3qkEIyZX1CCeNG7ad7zCxmPhvVMhWmd64ow0kSoA6ZuuFBi 9XtvQMKMAkPgJVgd5vBbqBjXant5zmyd4dKjXniMEyMqun2JYDf3Pwdhd8FMRwRamRsB LHI8yqRoObfwo86GCCtCZwP5L8TdnriGdmAuLd0ejQF4ChIMgWvVhQ5XXhj0AjUyDErq zyjlMCiPVz44+30OoPuW8gF7oCrRQRXrw+qs+LGwzdDjx3egK4H4LxlGBslA7X64Otmh pjHQ==
X-Gm-Message-State: AOAM532mBOHkDmnravYhpn/TPBCuU0S/x0TQiLvLQ5gOvqwFhPcIgnSN gt6PddEB+XvdeJTqu8YRll0=
X-Google-Smtp-Source: ABdhPJx0ZmKNteCUBaHAg7f9GJobBYwVWBd1yWDfJknzzAaae67YUowh0xal3/HjhpB3r1vHP4Bm5A==
X-Received: by 2002:a17:90b:b15:: with SMTP id bf21mr186253pjb.21.1606247139592;  Tue, 24 Nov 2020 11:45:39 -0800 (PST)
Received: from [192.168.178.20] ([151.210.131.28]) by smtp.gmail.com with ESMTPSA id z12sm15943744pfg.123.2020.11.24.11.45.36 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Nov 2020 11:45:38 -0800 (PST)
To: Alexander Azimov , Fernando Gont 
Cc: IPv6 Operations , tcpm 
References:   
From: Brian E Carpenter 
Message-ID: 
Date: Wed, 25 Nov 2020 08:45:34 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 19:45:42 -0000

On 25-Nov-20 08:28, Alexander Azimov wrote:
> Hi Fernando,
>=20
> Stating that FL change during TCP session lifetime is a bug - is a bit =
harsh.

It's a bug. It definitely breaks server load balancing, which in practice=
 is more important than en route load balancing, because it will probably=
 switch your session to a different server. (OK, I know that there are me=
chanisms sometimes for rescuing a session that has unexpectedly moved fro=
m one server to another, but in general it's a fail.)

I assume you have read https://tools.ietf.org/html/rfc7098

Regards
    Brian

>=20
> It is a fantastic idea to change the FL value if RTO or SYN_RTO happens=
 in=C2=A0a controlled=C2=A0environment.=C2=A0
> These are very specific TCP timeouts, that provide enough guarantee tha=
t there will be no out-of-order packets, though your packets will reach t=
he=C2=A0destination=C2=A0even in case of an outage inside your network. Z=
ero influence on your services in case of the network outage - doesn't so=
und like a bug for me.
>=20
> The problem is with the current Linux (though I haven't checked other O=
Ses) defaults. Specifically, with the default behavior after RTO event.
>=20
> =D0=B2=D1=82, 24 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:06, F=
ernando Gont >:
>=20
>     On 19/11/20 07:48, Alexander Azimov wrote:
>     > Dear colleagues,
>     >
>     > I have added in the cc both v6op and tcpm for a reason and let me=

>     > explain why.
>     >
>     > It's clear that we are moving forward with load balancing that us=
es flow
>     > label (FL). And the pressure will increase with SRv6 adoption. Bu=
t at
>     > the moment wide adoption of FL-based load-balancing may create
>     > significant issues for TCP Anycast services.
>     >
>     > RFC6437 suggests putting hash from 5-tuple into FL value. And as =
far as
>     > I know, there is no document that updates this behavior. This
>     > description is perfectly fine, but what is implemented in the Lin=
ux
>     > kernel is different: FL is carrying=C2=A0hash=C2=A0from 5-tuple w=
ith an additional
>     > seed, and this seed is randomly changed=C2=A0after each RTO/SYN_R=
TO event.
>=20
>     Changing the FL upon RTO is a bug.
>=20
>     I guess/assume that when you say SYN-RTO, you really mean "user
>     timeout", rather than RTO. If you don't, then that's also a bug.
>     If you do, I fail to understand what's the reason for wanting the F=
L to
>     change in that case, because as a result of port randomization, it =
0s
>     unlikely that the same four-tuple is employed for the next connecti=
on retry.
>=20
>=20
>     > Here are related patches:
>     >
>     >=C2=A0 =C2=A0* https://lore.kernel.org/netdev/alpine.DEB.2.02.1407=
012100290.20628@tomh.mtv.corp.google.com/
>     >=C2=A0 =C2=A0* https://lore.kernel.org/netdev/1438124526-2129341-1=
-git-send-email-tom@herbertland.com/
>     >=C2=A0 =C2=A0* https://lore.kernel.org/netdev/20160928020337.30572=
38-1-brakmo@fb.com/
>     >
>     >
>     > This is a great thing by the way because in the data center
>     > environment=C2=A0with multiple equal paths=C2=A0it gives a way to=
 have
>     > pseudo-multipath=C2=A0TCP which jumps between paths in case of an=
 outage.
>     > There might be interest to writedown an informational document fo=
r this.
>=20
>     That's a bad idea, since specs-wise the Flow-Label is not guarantee=
d to
>     remain unchanged from source to destination. If you want to ahve
>     multiple paths, then you should implement that in routing.
>=20
>=20
>=20
>     > I wonder what you think is a proper solution:
>     >
>     >=C2=A0 =C2=A0* Making FL related RTO change as knob instead of def=
ault behavior;
>     >=C2=A0 =C2=A0* Adding negotiation=C2=A0behavior in TCP;
>     >=C2=A0 =C2=A0* Something else?
>=20
>     Just make the FL a function of the connection "identifier". And kee=
o it
>     constant for the lifetime of that conenction.
>=20
>     Thanks,
>     --=20
>     Fernando Gont
>     e-mail: fernando@gont.com.ar  || fgont=
@si6networks.com 
>     PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>=20
>=20
>=20
>=20
>=20
> --=20
> Best regards,
> Alexander Azimov
>=20
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
>=20


From nobody Tue Nov 24 12:00:07 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1768E3A097A for ; Tue, 24 Nov 2020 12:00:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QbuXgN_rhw_7 for ; Tue, 24 Nov 2020 11:59:59 -0800 (PST)
Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2C2153A08EB for ; Tue, 24 Nov 2020 11:59:59 -0800 (PST)
Received: by mail-ed1-x52c.google.com with SMTP id m16so100144edr.3 for ; Tue, 24 Nov 2020 11:59:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=v9qkRwTfoCBHPKCZzc5O5Y/ldG9QNYn+YcU6f8uxqr4=; b=kQhmeAmeFd+WKt4xHlJsXCB3LXhm6A+7N2IH3n89b95q6V/RPHxdfyR6L614ayw6vx RnKnt8JpPnY0uKJbM5klgdc4z7etJKzuV1juS7lZWDGKJYfVShxN3/Za0K2MpGa5vOCm XK/W23CtyglYxdkvK4TtcDOL+YyjK0DvJyGAyqApeQ3VgtuyY93Lnf4hANRrrA/taupM +1dad4RyQ8Y8YwaWvNDkAYVb3fzsxpv0L9c1V++bpACq1HNKUc2QGKdyR8vpUfrQuPeF 0WNiR+qxKVyGl8k/hLrW2gFjy+LCaDxCQyN2/7bb9eXrl25DHeJRitDODI4qNFrHKCpR QOLw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=v9qkRwTfoCBHPKCZzc5O5Y/ldG9QNYn+YcU6f8uxqr4=; b=ZSNnG0BmdAYCRXlab7kHsM0jzUVTAkPHF/5UWkTrCCnW8Dd0X16g8qcoqbmtWfAifn ZSVNi5+jit1y6MUD3/PCfV8v4cjgNULMH7Hm/JuwsyipGY06WhVjo59rhdny3qxXYolK AJAQh74ObH0Fm5orifqMNHIfpL6ecR0tmuHC9JmwiV/8C9c7WJcjTjye9mTUcwZC3MF0 6W9j5AI1YQUKnvWq8dmPsyJ6vS9N8wR3z4dMW6IiiIV64x+GqboMS5PIIf6ca6zB6W49 xGq6WkzCKgh/BJbAuFsMzFGUViofgWefFptFIHtUKRb8cXVcbZDfBMP/ROue3aPD8ET+ 8FcA==
X-Gm-Message-State: AOAM531vTUFG3IO330P2+YLfJpWlX9z6OSEZ0mn8nqCqGh9HzJErkRTn pihREMOnYHmCjDSDnXYb6Oe8n3Fuv5bz9fbyQlPpbzLs3Fw=
X-Google-Smtp-Source: ABdhPJwbpZfa9c+WP7F5mmDQAcP/BL3v0oiq1XtN3FWFonl3mtwYUWf6fqoQiwNbswdL2ZV3FUsIaRmC9hg1i7B47Xw=
X-Received: by 2002:aa7:dc49:: with SMTP id g9mr105933edu.383.1606247997550; Tue, 24 Nov 2020 11:59:57 -0800 (PST)
MIME-Version: 1.0
References:    
In-Reply-To: 
From: Tom Herbert 
Date: Tue, 24 Nov 2020 12:59:46 -0700
Message-ID: 
To: Brian E Carpenter 
Cc: Alexander Azimov , Fernando Gont ,  IPv6 Operations , tcpm 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Tue, 24 Nov 2020 20:00:03 -0000

On Tue, Nov 24, 2020 at 12:46 PM Brian E Carpenter
 wrote:
>
> On 25-Nov-20 08:28, Alexander Azimov wrote:
> > Hi Fernando,
> >
> > Stating that FL change during TCP session lifetime is a bug - is a bit =
harsh.
>
> It's a bug. It definitely breaks server load balancing, which in practice=
 is more important than en route load balancing, because it will probably s=
witch your session to a different server. (OK, I know that there are mechan=
isms sometimes for rescuing a session that has unexpectedly moved from one =
server to another, but in general it's a fail.)
>
Brian,

IMO it's not that this breaks server load balancing, it's really that
server load balancing and other stateful mechanisms break multipath,
multihoming, and adaptive routing! In any case, I would agree that in
the current Internet it is pragmatic in the current Internet to aim
for consistent routing as the default.

Tom

> I assume you have read https://tools.ietf.org/html/rfc7098
>
> Regards
>     Brian
>
> >
> > It is a fantastic idea to change the FL value if RTO or SYN_RTO happens=
 in a controlled environment.
> > These are very specific TCP timeouts, that provide enough guarantee tha=
t there will be no out-of-order packets, though your packets will reach the=
 destination even in case of an outage inside your network. Zero influence =
on your services in case of the network outage - doesn't sound like a bug f=
or me.
> >
> > The problem is with the current Linux (though I haven't checked other O=
Ses) defaults. Specifically, with the default behavior after RTO event.
> >
> > =D0=B2=D1=82, 24 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:06, F=
ernando Gont >:
> >
> >     On 19/11/20 07:48, Alexander Azimov wrote:
> >     > Dear colleagues,
> >     >
> >     > I have added in the cc both v6op and tcpm for a reason and let me
> >     > explain why.
> >     >
> >     > It's clear that we are moving forward with load balancing that us=
es flow
> >     > label (FL). And the pressure will increase with SRv6 adoption. Bu=
t at
> >     > the moment wide adoption of FL-based load-balancing may create
> >     > significant issues for TCP Anycast services.
> >     >
> >     > RFC6437 suggests putting hash from 5-tuple into FL value. And as =
far as
> >     > I know, there is no document that updates this behavior. This
> >     > description is perfectly fine, but what is implemented in the Lin=
ux
> >     > kernel is different: FL is carrying hash from 5-tuple with an add=
itional
> >     > seed, and this seed is randomly changed after each RTO/SYN_RTO ev=
ent.
> >
> >     Changing the FL upon RTO is a bug.
> >
> >     I guess/assume that when you say SYN-RTO, you really mean "user
> >     timeout", rather than RTO. If you don't, then that's also a bug.
> >     If you do, I fail to understand what's the reason for wanting the F=
L to
> >     change in that case, because as a result of port randomization, it =
0s
> >     unlikely that the same four-tuple is employed for the next connecti=
on retry.
> >
> >
> >     > Here are related patches:
> >     >
> >     >   * https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.=
20628@tomh.mtv.corp.google.com/
> >     >   * https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-=
email-tom@herbertland.com/
> >     >   * https://lore.kernel.org/netdev/20160928020337.3057238-1-brakm=
o@fb.com/
> >     >
> >     >
> >     > This is a great thing by the way because in the data center
> >     > environment with multiple equal paths it gives a way to have
> >     > pseudo-multipath TCP which jumps between paths in case of an outa=
ge.
> >     > There might be interest to writedown an informational document fo=
r this.
> >
> >     That's a bad idea, since specs-wise the Flow-Label is not guarantee=
d to
> >     remain unchanged from source to destination. If you want to ahve
> >     multiple paths, then you should implement that in routing.
> >
> >
> >
> >     > I wonder what you think is a proper solution:
> >     >
> >     >   * Making FL related RTO change as knob instead of default behav=
ior;
> >     >   * Adding negotiation behavior in TCP;
> >     >   * Something else?
> >
> >     Just make the FL a function of the connection "identifier". And kee=
o it
> >     constant for the lifetime of that conenction.
> >
> >     Thanks,
> >     --
> >     Fernando Gont
> >     e-mail: fernando@gont.com.ar  || fgont=
@si6networks.com 
> >     PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> >
> >
> >
> >
> >
> > --
> > Best regards,
> > Alexander Azimov
> >
> > _______________________________________________
> > v6ops mailing list
> > v6ops@ietf.org
> > https://www.ietf.org/mailman/listinfo/v6ops
> >
>
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops


From nobody Tue Nov 24 23:13:39 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 23A6C3A112D; Tue, 24 Nov 2020 23:13:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G6ejUeGjDEYm; Tue, 24 Nov 2020 23:13:35 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [IPv6:2001:67c:27e4::14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BA0E03A1129; Tue, 24 Nov 2020 23:13:33 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:dc8c:7589:8676:bd67] (unknown [IPv6:2800:810:464:8164:dc8c:7589:8676:bd67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 085AF28050F; Wed, 25 Nov 2020 07:13:28 +0000 (UTC)
To: Alexander Azimov 
Cc: tcpm , IPv6 Operations 
References:   
From: Fernando Gont 
Message-ID: 
Date: Wed, 25 Nov 2020 04:04:10 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 8bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 07:13:38 -0000

On 24/11/20 16:28, Alexander Azimov wrote:
> Hi Fernando,
> 
> Stating that FL change during TCP session lifetime is a bug - is a bit 
> harsh.

Apologies if it came across like that -- the intent was to be blunt, 
rather than harsh.




> It is a fantastic idea to change the FL value if RTO or SYN_RTO happens 
> in a controlled environment.

As noted in my previous email, I'm not sure what you mean by RTO -- but 
if you really mean "retransmission timeout", doing that fails the 
definition of FL by RFC6437 and others. -- a retransmitted packet 
belongs to the ame flow, not to a different one.

That aside, please define "controlled environment". :-)



> These are very specific TCP timeouts, that provide enough guarantee that 
> there will be no out-of-order packets,

Strictly speaking, that's incorrect:: the RTO is an estimation, and may 
be fired incorrectly (although the whole point of measuring the RTT anc 
computing the RTO is for that case to be rather rare).


>  though your packets will reach 
> the destination even in case of an outage inside your network. Zero 
> influence on your services in case of the network outage - doesn't sound 
> like a bug for me.

It seems to me that this sort of behavior is the reason for which folks 
actually disable the use of the FL. (i.e., think what happens for load 
balancers).

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Tue Nov 24 23:13:49 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BB6A13A1139; Tue, 24 Nov 2020 23:13:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YESaIJGHZQ3F; Tue, 24 Nov 2020 23:13:37 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2A2403A112A; Tue, 24 Nov 2020 23:13:36 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:dc8c:7589:8676:bd67] (unknown [IPv6:2800:810:464:8164:dc8c:7589:8676:bd67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 21EE3280546; Wed, 25 Nov 2020 07:13:32 +0000 (UTC)
To: Tom Herbert 
Cc: Alexander Azimov , tcpm , IPv6 Operations 
References:   
From: Fernando Gont 
Message-ID: <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>
Date: Wed, 25 Nov 2020 04:13:20 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 07:13:43 -0000

Hi, Tom,

On 24/11/20 16:43, Tom Herbert wrote:
[....]
> Modulating the flow label is a means to affect the routing of packets
> through the network that uses flow labels as input to the ECMP hash.

What's the point?

1) You cannot tell *if* the FL is being used.

2) Changing the FL does not necessarily mean that packets will employ a 
different link.

3) If the network is failing, shouldn't you handle this via routing?



> The basic idea is that the flow label associated with a connection is
> randomly changed when the stack observes that the connection is
> failing (e.g. and an RTO). There is nothing in the specs that prevents
> this since the source is at liberty to set the flow label as it sees
> fit.

The FL is expected to remain constant for the life of a flow. A 
retransmitted packet is part of the same flow as the 
originally-transmitted packet. So this seems to be contradicting the 
very specification of the FL.

For instance, If a RTO for a flow causes the FL to change, then one may 
possibly argue that the FL is not naming/labeling what is said/expected 
to be anming/labeling.



> The feature is useful in large datacenter networks, like
> pparently Facebook where the patches originate, since information
> discerned by TCP can opportunistically be applied to route selection.
> The practical issue is that there are stateful devices like firewalls
> that require consistent routing in the network in which case changing
> the flow label can confuse them. As I mentioned, the original intent
> was that the flow label randomization feature should be opt-in instead
> of on by default.

So... where is the "source" of the packet that would be "modulating" the FL?

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Wed Nov 25 11:35:28 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 663E43A1BDF for ; Wed, 25 Nov 2020 11:35:22 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FH3W9f9aklsB for ; Wed, 25 Nov 2020 11:35:21 -0800 (PST)
Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BE6353A1BE0 for ; Wed, 25 Nov 2020 11:35:20 -0800 (PST)
Received: by mail-ed1-x52e.google.com with SMTP id r22so3788050edw.6 for ; Wed, 25 Nov 2020 11:35:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iKRu8L6k2Wzc+EUc36nWFqtRrs07O1b+4cGtQZ2Jces=; b=1CvkNK/u5S1DLQfhdC2b/IBPyBXy52CQhbS5AFdaB3PGbNzGHFDnhEj1V95zZXxeFO ClJo5TU6yt5UiQSGBJ0xT4eEp4bMOClopYjjdLfXMdk+eOflH9Wj36QuSGb9L0l4swby rGf84txa/rjncjzUrbE9pdBJdEfWcBLndyjMKtQs9h02kQwwkbjwnU7PmB47Zq6FSzDx hB+B3cKSG+d50iHAPGMwzWfXZSOvlzjgdwcdr225s0xWZs8kPnBQ4sh+HVUTlr3fnmHW ee7hTPzwGCOb7LdvtLv81ltmKtCpmWgETwc/3xMc91NAZLpq8A5KEweGTNLoh9LJW4Tj reIg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iKRu8L6k2Wzc+EUc36nWFqtRrs07O1b+4cGtQZ2Jces=; b=GKKO9Gd33Kh2A/jpVtdoA3/27UWDLuY0qn8pnXANVa9oI+Kn6+ntjxvlDp+BMuOZMn gr1rek9U+ODGt4gImhM7DCwaZz3LF6zvleZf7BAjPabmNEuEabC3NTRA4Gg60bJErynm Oma1c98DnKKJ3feUi4/pSi83fyLEFM1ae534coWMZlVMddzzWWvSD0NtPbr/AXdriXIC Bx27qX4iBUPGejkFDoRh2j7cNmG1F4Mz4OTRAq46imfv+y2ZNXdBqIjz3cUGngt0NBNr +1HEt4qj3O34PYsIANT665w7HFoFIWyaVUdcbGFDxZGvvPaMdCa0V5SuQr/BvrTjC2Gm WY4g==
X-Gm-Message-State: AOAM533KKtHjYRMI27jMHY8ewqnMUlcZLK7cMsidfQj4BnF/vxph9gvu mPqYhM5MNmkbz/pQg4D/ZSQHA8qEsK2edsjOejLNYA==
X-Google-Smtp-Source: ABdhPJyZWH9g/uec0zdhtlFbmrH5jXXcQo/IL4Ps6aWcgsrIkDSWeqa0u+giKJ4nEDMo7jW1qe0ATkp6XJTJouXGM5Y=
X-Received: by 2002:a05:6402:114b:: with SMTP id g11mr5111616edw.228.1606332918541;  Wed, 25 Nov 2020 11:35:18 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>
In-Reply-To: <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>
From: Tom Herbert 
Date: Wed, 25 Nov 2020 12:35:07 -0700
Message-ID: 
To: Fernando Gont 
Cc: Alexander Azimov , tcpm ,  IPv6 Operations 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 19:35:23 -0000

Hi Fernando, comments in line...

On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
>
> Hi, Tom,
>
> On 24/11/20 16:43, Tom Herbert wrote:
> [....]
> > Modulating the flow label is a means to affect the routing of packets
> > through the network that uses flow labels as input to the ECMP hash.
>
> What's the point?
>
> 1) You cannot tell *if* the FL is being used.
>
Generally true, but in a limited domain this information could be
discerned. I'd note that it's also generally true that we don't know
if there is a load balancer or stateful firewall in the path that
requires consistent routing, but in a limited domain we could know
that also.

> 2) Changing the FL does not necessarily mean that packets will employ a
> different link.

It's an opportunistic mechanism. If a connection is failing and we get
a better path that fixes it by simply changing the flow label then
what's the harm?

>
> 3) If the network is failing, shouldn't you handle this via routing?
>
Sure, but then that requires an out of band feedback loop from a TCP
implementation to the network infrastructure to indicate there is a
problem and then the network needs to respond. That's significant
infrastructure and higher reaction time than doing something in TCP
and IP. Think of modulating the flow label is an inexpensive form of
source routing within a limited domain that doesn't need any
infrastructure or heavyweight protocols or something like segment
routing.

>
>
> > The basic idea is that the flow label associated with a connection is
> > randomly changed when the stack observes that the connection is
> > failing (e.g. and an RTO). There is nothing in the specs that prevents
> > this since the source is at liberty to set the flow label as it sees
> > fit.
>
> The FL is expected to remain constant for the life of a flow. A
> retransmitted packet is part of the same flow as the
> originally-transmitted packet. So this seems to be contradicting the
> very specification of the FL.
>
> For instance, If a RTO for a flow causes the FL to change, then one may
> possibly argue that the FL is not naming/labeling what is said/expected
> to be anming/labeling.

Specifically, RFC6437 states:

"It is therefore RECOMMENDED that source hosts support the flow label
by setting the flow label field for all packets of a given flow to the
same value chosen from an approximation to a discrete uniform
distribution."

So that is clearly a just recommendation, and not a requirement (and
definitely not a MUST). Furthermore, RFC6437 states:

"A forwarding node MUST either leave a non-zero flow label value
unchanged or change it only for compelling operational security
reasons as described in Section 6.1."

So there's no guarantee in the protocol specs that flow labels are
consistent for the life of the connection, which means that the
network cannot assume that and thus it would be incorrect if the
network tried to enforce flow label consistency as a protocol
requirement. As I said, it is prudent to try to be consistent with
flow labels and the default behavior in Linux should be changed,
however I do not believe there's a valid claim of non-conformance that
motivates removal of the feature that is already deployed.

Tom




>
>
>
> > The feature is useful in large datacenter networks, like
> > pparently Facebook where the patches originate, since information
> > discerned by TCP can opportunistically be applied to route selection.
> > The practical issue is that there are stateful devices like firewalls
> > that require consistent routing in the network in which case changing
> > the flow label can confuse them. As I mentioned, the original intent
> > was that the flow label randomization feature should be opt-in instead
> > of on by default.
>
> So... where is the "source" of the packet that would be "modulating" the FL?
>
> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>


From nobody Wed Nov 25 11:40:07 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B619D3A1BE7; Wed, 25 Nov 2020 11:40:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.121
X-Spam-Level: 
X-Spam-Status: No, score=-2.121 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=joelhalpern.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KC6yqgv_rFZp; Wed, 25 Nov 2020 11:39:57 -0800 (PST)
Received: from mailb2.tigertech.net (mailb2.tigertech.net [208.80.4.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7FA513A1BE5; Wed, 25 Nov 2020 11:39:57 -0800 (PST)
Received: from localhost (localhost [127.0.0.1]) by mailb2.tigertech.net (Postfix) with ESMTP id 4ChB711msQz1nsT2; Wed, 25 Nov 2020 11:39:57 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelhalpern.com; s=2.tigertech; t=1606333197; bh=a0jtMrN6cPnLUIYqiIhsA6vO3/yEn3HPNFwYzRC7LOE=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=VeMclSfdHXMmXzVzafNvQ1GUZujsxm+RWZMvXNyXHLl9XfMOkIDsB3OyZBVY80VnE 7YyREH5qS4hSagNwa1vvv2r+LQOmbHKgwisiszb8qhaLm7XyoEvoYYYs+CrtRt7WZ8 AAhQCUaAAqeOnQCz1hS8pwZuJMngUDJ0lWp9WH0U=
X-Quarantine-ID: 
X-Virus-Scanned: Debian amavisd-new at b2.tigertech.net
Received: from [192.168.128.43] (unknown [50.225.209.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mailb2.tigertech.net (Postfix) with ESMTPSA id 4ChB704hsQz1nsWT; Wed, 25 Nov 2020 11:39:56 -0800 (PST)
To: Tom Herbert 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar> 
From: "Joel M. Halpern" 
Message-ID: <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>
Date: Wed, 25 Nov 2020 14:39:55 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 19:40:02 -0000

This kind of thing is why, as I understand it, MPTCP has discovery 
mechanisms ot know if both sides use it, and can select alternative 
addresses for communication.

Trying to guess flow labels that might avoid a problem because it might 
be an ECMP problem, is just flailing about.  Not a good design for 
operational protocols.

And in general, designing protocols around "I know exactly what is going 
on"  (the requirement for what you describe that goes well beyond just 
"limited domains") is also a recipe for failure.

The Flow Label RFCs are actually very explicit that a flow label is 
supposed to be stable for the life of the flow.  Otherwise, it isn't a 
flow label.

Yours,
Joel

On 11/25/2020 2:35 PM, Tom Herbert wrote:
> Hi Fernando, comments in line...
> 
> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
>>
>> Hi, Tom,
>>
>> On 24/11/20 16:43, Tom Herbert wrote:
>> [....]
>>> Modulating the flow label is a means to affect the routing of packets
>>> through the network that uses flow labels as input to the ECMP hash.
>>
>> What's the point?
>>
>> 1) You cannot tell *if* the FL is being used.
>>
> Generally true, but in a limited domain this information could be
> discerned. I'd note that it's also generally true that we don't know
> if there is a load balancer or stateful firewall in the path that
> requires consistent routing, but in a limited domain we could know
> that also.
> 
>> 2) Changing the FL does not necessarily mean that packets will employ a
>> different link.
> 
> It's an opportunistic mechanism. If a connection is failing and we get
> a better path that fixes it by simply changing the flow label then
> what's the harm?
> 
>>
>> 3) If the network is failing, shouldn't you handle this via routing?
>>
> Sure, but then that requires an out of band feedback loop from a TCP
> implementation to the network infrastructure to indicate there is a
> problem and then the network needs to respond. That's significant
> infrastructure and higher reaction time than doing something in TCP
> and IP. Think of modulating the flow label is an inexpensive form of
> source routing within a limited domain that doesn't need any
> infrastructure or heavyweight protocols or something like segment
> routing.
> 
>>
>>
>>> The basic idea is that the flow label associated with a connection is
>>> randomly changed when the stack observes that the connection is
>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>>> this since the source is at liberty to set the flow label as it sees
>>> fit.
>>
>> The FL is expected to remain constant for the life of a flow. A
>> retransmitted packet is part of the same flow as the
>> originally-transmitted packet. So this seems to be contradicting the
>> very specification of the FL.
>>
>> For instance, If a RTO for a flow causes the FL to change, then one may
>> possibly argue that the FL is not naming/labeling what is said/expected
>> to be anming/labeling.
> 
> Specifically, RFC6437 states:
> 
> "It is therefore RECOMMENDED that source hosts support the flow label
> by setting the flow label field for all packets of a given flow to the
> same value chosen from an approximation to a discrete uniform
> distribution."
> 
> So that is clearly a just recommendation, and not a requirement (and
> definitely not a MUST). Furthermore, RFC6437 states:
> 
> "A forwarding node MUST either leave a non-zero flow label value
> unchanged or change it only for compelling operational security
> reasons as described in Section 6.1."
> 
> So there's no guarantee in the protocol specs that flow labels are
> consistent for the life of the connection, which means that the
> network cannot assume that and thus it would be incorrect if the
> network tried to enforce flow label consistency as a protocol
> requirement. As I said, it is prudent to try to be consistent with
> flow labels and the default behavior in Linux should be changed,
> however I do not believe there's a valid claim of non-conformance that
> motivates removal of the feature that is already deployed.
> 
> Tom
> 
> 
> 
> 
>>
>>
>>
>>> The feature is useful in large datacenter networks, like
>>> pparently Facebook where the patches originate, since information
>>> discerned by TCP can opportunistically be applied to route selection.
>>> The practical issue is that there are stateful devices like firewalls
>>> that require consistent routing in the network in which case changing
>>> the flow label can confuse them. As I mentioned, the original intent
>>> was that the flow label randomization feature should be opt-in instead
>>> of on by default.
>>
>> So... where is the "source" of the packet that would be "modulating" the FL?
>>
>> Thanks,
>> --
>> Fernando Gont
>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>
>>
>>
> 
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
> 


From nobody Wed Nov 25 12:04:52 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7456B3A1C64; Wed, 25 Nov 2020 12:04:50 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FGikAEzXeGsZ; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A31523A1C63; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
Received: by mail-pg1-x534.google.com with SMTP id t37so3360256pga.7; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=42KFVBc8+ZWoF+JqgQiO4syNYF9y+7EKA+BfDce5prs=; b=p0nwqbnzoS0Kzdv0SJYKxq8qfCOJIx2wTDYVQ6sNBIYtMQ7DOk7KF+GAm2c3VcSzcW J5PX4LeXKtll/YPNybnblPgzEaBacea+w0eVSBbDiNtcuQYtWLeJmI3WRfsdfdJ1DZRq mSOshJh5ZiPp4s8uay9uD+dbqhOaotX9fgTc4V7fiXONnieN+l1ow57Dm1gTy1o3RxOS W8VEJNBgrrJdicJSu1SXlkiHzcqW3GiD+ZBDF2UoFbSwA3CSG+F6gQJnCFbl5zfDk7Cg sXbDJVWoHZp36m7yBx2OFTzC914DSIslXWdxRMjmc6GI1clsBpPyT40v1Xn9tFmxhSiN 7QpA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=42KFVBc8+ZWoF+JqgQiO4syNYF9y+7EKA+BfDce5prs=; b=pG2N5BUkXGxqYJgcWj65TvIbnx9E4s6RiI4pdtQUBUsF4SkbrC1ds1Vd1QFbl6ku2W mXOfMjrbYuggcrKVolPgF/KVAjdIJgZKGg997kecW8hxAZ25OR5AAcrZwnSHBB6nakpA NFoMCblx4zlQ3Xf3JPbSY5uc84WVteJQ0mFrYSbtipuvSYMerI8pllCC8pYViZalHl1x Du1r84pR8BfUmGzp+vYA7TU6BySzYp9SxrKIUIKfSkiOmgy+f6B7NoaDuTG5YXK3NCgK 0jlIdQ/fRch6FVGexFTFCa+W5z2fSuLBHqbmAzAzgSRhcCZ+ncfGIMYIz724iKPq+nBG pUfA==
X-Gm-Message-State: AOAM530S38nfO9+rmWQyjje1dUeoT1kHQhQc6lJ67xZ5pY+nFeol73Rw TZkuuqkjLYMCR1pmAxaI0qL3EoKkiWwlNQ==
X-Google-Smtp-Source: ABdhPJyMBMxaHkYZ66l0Y2iytk3PaXXjZ9PdYsd+8ReFqvo7Lq/Qz2hVMJgQ+dWTNMF8bIufpr2T7g==
X-Received: by 2002:a62:5e81:0:b029:197:baa5:1792 with SMTP id s123-20020a625e810000b0290197baa51792mr4501345pfb.80.1606334686622;  Wed, 25 Nov 2020 12:04:46 -0800 (PST)
Received: from [192.168.178.20] ([151.210.131.28]) by smtp.gmail.com with ESMTPSA id y5sm3834165pjt.42.2020.11.25.12.04.43 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Nov 2020 12:04:45 -0800 (PST)
To: "Joel M. Halpern" , Tom Herbert 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>
From: Brian E Carpenter 
Message-ID: <2645ef05-c1e6-e12e-df49-20d318ba39ee@gmail.com>
Date: Thu, 26 Nov 2020 09:04:42 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 20:04:50 -0000

Hi Joel,

On 26-Nov-20 08:39, Joel M. Halpern wrote:
> This kind of thing is why, as I understand it, MPTCP has discovery 
> mechanisms ot know if both sides use it, and can select alternative 
> addresses for communication.
> 
> Trying to guess flow labels that might avoid a problem because it might 
> be an ECMP problem, is just flailing about.  Not a good design for 
> operational protocols.
> 
> And in general, designing protocols around "I know exactly what is going 
> on"  (the requirement for what you describe that goes well beyond just 
> "limited domains") is also a recipe for failure.
> 
> The Flow Label RFCs are actually very explicit that a flow label is 
> supposed to be stable for the life of the flow.  Otherwise, it isn't a 
> flow label.

That's correct. However, the first paragraph of RFC6437 tells you that
it's the prerogative of the source of a flow to decide what is a flow.
It even says "a flow is not necessarily 1:1 mapped to a transport
connection." Therefore, Tom is correct that a source is conformant even
if it changes its mind halfway through a TCP connection. However, it's
only safe to do that if the source also knows that no harm will result,
which it cannot know in the general case. In a limited domain, the source
could know that there is no server load balancing in place, in which case
changing the label is not automatically damaging.

Thus, this is not something that should ever become a default behaviour,
and I think this thread began with a complaint that in some o/s release
it was the default.

Regards
   Brian

> 
> Yours,
> Joel
> 
> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>> Hi Fernando, comments in line...
>>
>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
>>>
>>> Hi, Tom,
>>>
>>> On 24/11/20 16:43, Tom Herbert wrote:
>>> [....]
>>>> Modulating the flow label is a means to affect the routing of packets
>>>> through the network that uses flow labels as input to the ECMP hash.
>>>
>>> What's the point?
>>>
>>> 1) You cannot tell *if* the FL is being used.
>>>
>> Generally true, but in a limited domain this information could be
>> discerned. I'd note that it's also generally true that we don't know
>> if there is a load balancer or stateful firewall in the path that
>> requires consistent routing, but in a limited domain we could know
>> that also.
>>
>>> 2) Changing the FL does not necessarily mean that packets will employ a
>>> different link.
>>
>> It's an opportunistic mechanism. If a connection is failing and we get
>> a better path that fixes it by simply changing the flow label then
>> what's the harm?
>>
>>>
>>> 3) If the network is failing, shouldn't you handle this via routing?
>>>
>> Sure, but then that requires an out of band feedback loop from a TCP
>> implementation to the network infrastructure to indicate there is a
>> problem and then the network needs to respond. That's significant
>> infrastructure and higher reaction time than doing something in TCP
>> and IP. Think of modulating the flow label is an inexpensive form of
>> source routing within a limited domain that doesn't need any
>> infrastructure or heavyweight protocols or something like segment
>> routing.
>>
>>>
>>>
>>>> The basic idea is that the flow label associated with a connection is
>>>> randomly changed when the stack observes that the connection is
>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>>>> this since the source is at liberty to set the flow label as it sees
>>>> fit.
>>>
>>> The FL is expected to remain constant for the life of a flow. A
>>> retransmitted packet is part of the same flow as the
>>> originally-transmitted packet. So this seems to be contradicting the
>>> very specification of the FL.
>>>
>>> For instance, If a RTO for a flow causes the FL to change, then one may
>>> possibly argue that the FL is not naming/labeling what is said/expected
>>> to be anming/labeling.
>>
>> Specifically, RFC6437 states:
>>
>> "It is therefore RECOMMENDED that source hosts support the flow label
>> by setting the flow label field for all packets of a given flow to the
>> same value chosen from an approximation to a discrete uniform
>> distribution."
>>
>> So that is clearly a just recommendation, and not a requirement (and
>> definitely not a MUST). Furthermore, RFC6437 states:
>>
>> "A forwarding node MUST either leave a non-zero flow label value
>> unchanged or change it only for compelling operational security
>> reasons as described in Section 6.1."
>>
>> So there's no guarantee in the protocol specs that flow labels are
>> consistent for the life of the connection, which means that the
>> network cannot assume that and thus it would be incorrect if the
>> network tried to enforce flow label consistency as a protocol
>> requirement. As I said, it is prudent to try to be consistent with
>> flow labels and the default behavior in Linux should be changed,
>> however I do not believe there's a valid claim of non-conformance that
>> motivates removal of the feature that is already deployed.
>>
>> Tom
>>
>>
>>
>>
>>>
>>>
>>>
>>>> The feature is useful in large datacenter networks, like
>>>> pparently Facebook where the patches originate, since information
>>>> discerned by TCP can opportunistically be applied to route selection.
>>>> The practical issue is that there are stateful devices like firewalls
>>>> that require consistent routing in the network in which case changing
>>>> the flow label can confuse them. As I mentioned, the original intent
>>>> was that the flow label randomization feature should be opt-in instead
>>>> of on by default.
>>>
>>> So... where is the "source" of the packet that would be "modulating" the FL?
>>>
>>> Thanks,
>>> --
>>> Fernando Gont
>>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>>
>>>
>>>
>>
>> _______________________________________________
>> v6ops mailing list
>> v6ops@ietf.org
>> https://www.ietf.org/mailman/listinfo/v6ops
>>
> 
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
> 


From nobody Wed Nov 25 12:05:24 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 518A53A1C9C for ; Wed, 25 Nov 2020 12:05:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ksRQbPCNbOES for ; Wed, 25 Nov 2020 12:05:16 -0800 (PST)
Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 47E523A1CA5 for ; Wed, 25 Nov 2020 12:05:13 -0800 (PST)
Received: by mail-ej1-x633.google.com with SMTP id lt17so4213586ejb.3 for ; Wed, 25 Nov 2020 12:05:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=fcooyIZCSMch3fTbFOLokrdujEjrdUAGttzOwcEB5tU=; b=IqGI+/ARxrRXUbowuvoNX4o+J1DVc/2mFsiy30gS02MSOOWJxSDrHcPzIA46kw6+Z4 yAVGqY7AZ2Bd9AjlncwVe/sbiYLOOP55H+3VHa90yXDQi7ErrSMNUvCs8xtd0osm192G KvmxVo5Ynf4Lx7EyOGljQ5hkk9O1AZifcChgS7LA+DqbGP5mWD/Kv8qxzSzcAdETwYh+ G9e0T+5w8c2elNOkFC+5AIuvd5EIY2NXpKVrHwYL0rt3STPlexoZBfyDMwXUWSMwArIo Ws7maAAyxKLVExWTWpV+6WOBqCJ7pUTmQTYrzyEWMYakPRwsSwCMeHF428CkEbaprZp1 ti8A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=fcooyIZCSMch3fTbFOLokrdujEjrdUAGttzOwcEB5tU=; b=X63mQN5GH7aDislB9DluFD3YZcnhM/HaRaVnkwZoRJfi/dkDkybJzkBj5cjmyrqyFx LCxAX0bY/0IGOD6iCxLqOYhiwzJ/FOgnU/RMAfpZVKbM5CxtckAhvyIHjM3J8+RYh7gH f5rMrXZSGAT7wjcYgwvghAygKe+47mhdikmCefmSXFMLFUtWMiNpDar7ZSP2gJJbvpLw 9GNAnU+Al4frmNl1yl9IzEEOmKsVY6qrK2s5sw3S8rQ8ZLICJrwIbSRAzq035r4F960j +KTSBovghx+C9Dj0CM89OVPM5pSWu+AzDhEHNNw4cFVOj2RYdEkYxGpqbzmj1X36JJeU IqAA==
X-Gm-Message-State: AOAM530DQ+eD3tfrfiVBMoILwXostGeR+46CiyHtyorI9zvjDUWeE6LL lr4HRgNjU+ViZV0aiDqAZh1YChOeNvnWvH9MJOOs4A==
X-Google-Smtp-Source: ABdhPJwx/znQ66iAKYmdAJEV4MnLbSaRZ1ZT/TpKPYbZpXPm7wOw19C9hZH7ijmbZM/ujPfIVZxHUjuyl2jFhYvWPCY=
X-Received: by 2002:a17:906:3294:: with SMTP id 20mr4584338ejw.239.1606334711385;  Wed, 25 Nov 2020 12:05:11 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>
In-Reply-To: <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>
From: Tom Herbert 
Date: Wed, 25 Nov 2020 13:05:00 -0700
Message-ID: 
To: "Joel M. Halpern" 
Cc: IPv6 Operations , tcpm 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 20:05:23 -0000

Joel,

Is there an RFC that clearly and unambiguously states that a host MUST
use the same flow label for the lifetime _and_ clearly defines exactly
what a flow is with respect to such a requirement (for instance, how
would you define a flow and enforce such a requirement in UDP? IPsec?
other encapsulations?). If there is such a requirement then we'll
change the code to be conformant.

Tom

On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern  wrote:
>
> This kind of thing is why, as I understand it, MPTCP has discovery
> mechanisms ot know if both sides use it, and can select alternative
> addresses for communication.
>
> Trying to guess flow labels that might avoid a problem because it might
> be an ECMP problem, is just flailing about.  Not a good design for
> operational protocols.
>
> And in general, designing protocols around "I know exactly what is going
> on"  (the requirement for what you describe that goes well beyond just
> "limited domains") is also a recipe for failure.
>
> The Flow Label RFCs are actually very explicit that a flow label is
> supposed to be stable for the life of the flow.  Otherwise, it isn't a
> flow label.
>
> Yours,
> Joel
>
> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> > Hi Fernando, comments in line...
> >
> > On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
> >>
> >> Hi, Tom,
> >>
> >> On 24/11/20 16:43, Tom Herbert wrote:
> >> [....]
> >>> Modulating the flow label is a means to affect the routing of packets
> >>> through the network that uses flow labels as input to the ECMP hash.
> >>
> >> What's the point?
> >>
> >> 1) You cannot tell *if* the FL is being used.
> >>
> > Generally true, but in a limited domain this information could be
> > discerned. I'd note that it's also generally true that we don't know
> > if there is a load balancer or stateful firewall in the path that
> > requires consistent routing, but in a limited domain we could know
> > that also.
> >
> >> 2) Changing the FL does not necessarily mean that packets will employ a
> >> different link.
> >
> > It's an opportunistic mechanism. If a connection is failing and we get
> > a better path that fixes it by simply changing the flow label then
> > what's the harm?
> >
> >>
> >> 3) If the network is failing, shouldn't you handle this via routing?
> >>
> > Sure, but then that requires an out of band feedback loop from a TCP
> > implementation to the network infrastructure to indicate there is a
> > problem and then the network needs to respond. That's significant
> > infrastructure and higher reaction time than doing something in TCP
> > and IP. Think of modulating the flow label is an inexpensive form of
> > source routing within a limited domain that doesn't need any
> > infrastructure or heavyweight protocols or something like segment
> > routing.
> >
> >>
> >>
> >>> The basic idea is that the flow label associated with a connection is
> >>> randomly changed when the stack observes that the connection is
> >>> failing (e.g. and an RTO). There is nothing in the specs that prevents
> >>> this since the source is at liberty to set the flow label as it sees
> >>> fit.
> >>
> >> The FL is expected to remain constant for the life of a flow. A
> >> retransmitted packet is part of the same flow as the
> >> originally-transmitted packet. So this seems to be contradicting the
> >> very specification of the FL.
> >>
> >> For instance, If a RTO for a flow causes the FL to change, then one may
> >> possibly argue that the FL is not naming/labeling what is said/expected
> >> to be anming/labeling.
> >
> > Specifically, RFC6437 states:
> >
> > "It is therefore RECOMMENDED that source hosts support the flow label
> > by setting the flow label field for all packets of a given flow to the
> > same value chosen from an approximation to a discrete uniform
> > distribution."
> >
> > So that is clearly a just recommendation, and not a requirement (and
> > definitely not a MUST). Furthermore, RFC6437 states:
> >
> > "A forwarding node MUST either leave a non-zero flow label value
> > unchanged or change it only for compelling operational security
> > reasons as described in Section 6.1."
> >
> > So there's no guarantee in the protocol specs that flow labels are
> > consistent for the life of the connection, which means that the
> > network cannot assume that and thus it would be incorrect if the
> > network tried to enforce flow label consistency as a protocol
> > requirement. As I said, it is prudent to try to be consistent with
> > flow labels and the default behavior in Linux should be changed,
> > however I do not believe there's a valid claim of non-conformance that
> > motivates removal of the feature that is already deployed.
> >
> > Tom
> >
> >
> >
> >
> >>
> >>
> >>
> >>> The feature is useful in large datacenter networks, like
> >>> pparently Facebook where the patches originate, since information
> >>> discerned by TCP can opportunistically be applied to route selection.
> >>> The practical issue is that there are stateful devices like firewalls
> >>> that require consistent routing in the network in which case changing
> >>> the flow label can confuse them. As I mentioned, the original intent
> >>> was that the flow label randomization feature should be opt-in instead
> >>> of on by default.
> >>
> >> So... where is the "source" of the packet that would be "modulating" the FL?
> >>
> >> Thanks,
> >> --
> >> Fernando Gont
> >> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> >> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> >>
> >>
> >>
> >
> > _______________________________________________
> > v6ops mailing list
> > v6ops@ietf.org
> > https://www.ietf.org/mailman/listinfo/v6ops
> >


From nobody Wed Nov 25 13:28:07 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 50E293A1ECE; Wed, 25 Nov 2020 13:27:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.121
X-Spam-Level: 
X-Spam-Status: No, score=-2.121 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=joelhalpern.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DpDmqAOQC1Tg; Wed, 25 Nov 2020 13:27:52 -0800 (PST)
Received: from mailb2.tigertech.net (mailb2.tigertech.net [208.80.4.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 543973A1E94; Wed, 25 Nov 2020 13:27:52 -0800 (PST)
Received: from localhost (localhost [127.0.0.1]) by mailb2.tigertech.net (Postfix) with ESMTP id 4ChDWX1HK2z1nwrK; Wed, 25 Nov 2020 13:27:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=joelhalpern.com; s=2.tigertech; t=1606339672; bh=feWobQX63XBosVSXdSvhuC/jwbLg/k8cOu5WhaTI6f8=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=QE7FapcL2gi4lyg3s2h1hKLqCHHw2bAIGZfBwYGIYf55OkSUzmToKJA7uGUITTbya OsKgnpZXXvPYNM7ob5uRdlwlerkXBd884kYq5DgNdrXjIGmQ8jj5Nqz17BbCXQcAyv Oi1zplX6ro8yuHc+OmhK5r8dfGIWtTdpmNN8+2ws=
X-Quarantine-ID: <9Ztez0OfM0yx>
X-Virus-Scanned: Debian amavisd-new at b2.tigertech.net
Received: from [192.168.128.43] (unknown [50.225.209.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mailb2.tigertech.net (Postfix) with ESMTPSA id 4ChDWW0PnXz1nwr0; Wed, 25 Nov 2020 13:27:50 -0800 (PST)
To: Tom Herbert 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com> 
From: "Joel M. Halpern" 
Message-ID: 
Date: Wed, 25 Nov 2020 16:27:48 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 21:28:03 -0000

No, as Brian says, there are escape clauses in the flow definitions.

But changing the flow label due to traffic problems does not correspond 
to packets being in actually different flows.
If one were using UDP, and mixing loss sensitive packets with loss 
insensitive packets for a special application, sure, one could use two 
flow labels.  But that is not what you are describing.

Yours,
Joel

On 11/25/2020 3:05 PM, Tom Herbert wrote:
> Joel,
> 
> Is there an RFC that clearly and unambiguously states that a host MUST
> use the same flow label for the lifetime _and_ clearly defines exactly
> what a flow is with respect to such a requirement (for instance, how
> would you define a flow and enforce such a requirement in UDP? IPsec?
> other encapsulations?). If there is such a requirement then we'll
> change the code to be conformant.
> 
> Tom
> 
> On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern  wrote:
>>
>> This kind of thing is why, as I understand it, MPTCP has discovery
>> mechanisms ot know if both sides use it, and can select alternative
>> addresses for communication.
>>
>> Trying to guess flow labels that might avoid a problem because it might
>> be an ECMP problem, is just flailing about.  Not a good design for
>> operational protocols.
>>
>> And in general, designing protocols around "I know exactly what is going
>> on"  (the requirement for what you describe that goes well beyond just
>> "limited domains") is also a recipe for failure.
>>
>> The Flow Label RFCs are actually very explicit that a flow label is
>> supposed to be stable for the life of the flow.  Otherwise, it isn't a
>> flow label.
>>
>> Yours,
>> Joel
>>
>> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>>> Hi Fernando, comments in line...
>>>
>>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
>>>>
>>>> Hi, Tom,
>>>>
>>>> On 24/11/20 16:43, Tom Herbert wrote:
>>>> [....]
>>>>> Modulating the flow label is a means to affect the routing of packets
>>>>> through the network that uses flow labels as input to the ECMP hash.
>>>>
>>>> What's the point?
>>>>
>>>> 1) You cannot tell *if* the FL is being used.
>>>>
>>> Generally true, but in a limited domain this information could be
>>> discerned. I'd note that it's also generally true that we don't know
>>> if there is a load balancer or stateful firewall in the path that
>>> requires consistent routing, but in a limited domain we could know
>>> that also.
>>>
>>>> 2) Changing the FL does not necessarily mean that packets will employ a
>>>> different link.
>>>
>>> It's an opportunistic mechanism. If a connection is failing and we get
>>> a better path that fixes it by simply changing the flow label then
>>> what's the harm?
>>>
>>>>
>>>> 3) If the network is failing, shouldn't you handle this via routing?
>>>>
>>> Sure, but then that requires an out of band feedback loop from a TCP
>>> implementation to the network infrastructure to indicate there is a
>>> problem and then the network needs to respond. That's significant
>>> infrastructure and higher reaction time than doing something in TCP
>>> and IP. Think of modulating the flow label is an inexpensive form of
>>> source routing within a limited domain that doesn't need any
>>> infrastructure or heavyweight protocols or something like segment
>>> routing.
>>>
>>>>
>>>>
>>>>> The basic idea is that the flow label associated with a connection is
>>>>> randomly changed when the stack observes that the connection is
>>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>>>>> this since the source is at liberty to set the flow label as it sees
>>>>> fit.
>>>>
>>>> The FL is expected to remain constant for the life of a flow. A
>>>> retransmitted packet is part of the same flow as the
>>>> originally-transmitted packet. So this seems to be contradicting the
>>>> very specification of the FL.
>>>>
>>>> For instance, If a RTO for a flow causes the FL to change, then one may
>>>> possibly argue that the FL is not naming/labeling what is said/expected
>>>> to be anming/labeling.
>>>
>>> Specifically, RFC6437 states:
>>>
>>> "It is therefore RECOMMENDED that source hosts support the flow label
>>> by setting the flow label field for all packets of a given flow to the
>>> same value chosen from an approximation to a discrete uniform
>>> distribution."
>>>
>>> So that is clearly a just recommendation, and not a requirement (and
>>> definitely not a MUST). Furthermore, RFC6437 states:
>>>
>>> "A forwarding node MUST either leave a non-zero flow label value
>>> unchanged or change it only for compelling operational security
>>> reasons as described in Section 6.1."
>>>
>>> So there's no guarantee in the protocol specs that flow labels are
>>> consistent for the life of the connection, which means that the
>>> network cannot assume that and thus it would be incorrect if the
>>> network tried to enforce flow label consistency as a protocol
>>> requirement. As I said, it is prudent to try to be consistent with
>>> flow labels and the default behavior in Linux should be changed,
>>> however I do not believe there's a valid claim of non-conformance that
>>> motivates removal of the feature that is already deployed.
>>>
>>> Tom
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>>> The feature is useful in large datacenter networks, like
>>>>> pparently Facebook where the patches originate, since information
>>>>> discerned by TCP can opportunistically be applied to route selection.
>>>>> The practical issue is that there are stateful devices like firewalls
>>>>> that require consistent routing in the network in which case changing
>>>>> the flow label can confuse them. As I mentioned, the original intent
>>>>> was that the flow label randomization feature should be opt-in instead
>>>>> of on by default.
>>>>
>>>> So... where is the "source" of the packet that would be "modulating" the FL?
>>>>
>>>> Thanks,
>>>> --
>>>> Fernando Gont
>>>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
>>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>>>
>>>>
>>>>
>>>
>>> _______________________________________________
>>> v6ops mailing list
>>> v6ops@ietf.org
>>> https://www.ietf.org/mailman/listinfo/v6ops
>>>


From nobody Wed Nov 25 14:13:31 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 71CA43A1EE9 for ; Wed, 25 Nov 2020 14:13:29 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.896
X-Spam-Level: 
X-Spam-Status: No, score=-1.896 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fc4SMlwK4ftJ for ; Wed, 25 Nov 2020 14:13:27 -0800 (PST)
Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D488F3A1EE1 for ; Wed, 25 Nov 2020 14:13:26 -0800 (PST)
Received: by mail-ej1-x62d.google.com with SMTP id oq3so5163695ejb.7 for ; Wed, 25 Nov 2020 14:13:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=WxdQiywIPlAyDaIo00bMFE94JVqujKhxjM2KopTZ6Mg=; b=E8CBiwMuIw3ttPfv0pyd7ev5NZBQjwN3T1CSOIreCT5esXuu14bAdfqYyWodpbV444 feo9IKDzxEKJMk9w59Quz9dvy+C+LqqAkpUxqTLjD/F7uwxcmMDBB+GTSHSqpICirOAM SJzgEBX3LX1EBPpDMPle4tvLCLQ7GHnC84//CVoO8p72FdLo9qywcj8Q5npJBhxxX+Sq J+YK45z+CJU2hIrBzIcuK6+CRzfMo7mQRd49Wz0HJRHjQ6X/Hm3WS5/bY6mX0WyTXjAJ s7nrf5G9N06+GrvQZI6zNLs7sgUfFCqTkurJ1fKV9yD5lL4RgUvGB2IR1rWzJj4Ouxts 1P9w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=WxdQiywIPlAyDaIo00bMFE94JVqujKhxjM2KopTZ6Mg=; b=NcT/DyoQRgi2oh0za84Z+yzB28Zk9uNwpVdDBGWC9h19uhWVyG2jpRfeaeAbKfDLkY wqVdpup3QKNQbkDyk7YjAWO2YrptxPz1qrIly6hHqYB3WOusg23HFFZ2GbTdtYq1lBlf gwOmdReSgC9UDWebWnJSOObCBv1P2I5OTb334fM/lk+kfCIzjLpwekuTwNm/2cblwWts jSFIE2VetOh5x1bqGUTKM3I11cLP8milMYB6PBBA4pLlb2lzVMZo6vMDalER3CSfjIFH dh4qCxcXMSGgaWGjSrgabo9o1rteqzbadyrm895lCU8gqacaBpRFlYmr7Ft3cJ/SCeck wFlA==
X-Gm-Message-State: AOAM531n10x74ORR8tIxyT0GW12nILBi4Yg2E8Jzgzgdif5KAOW7K+A5 qLX3GY+wRZyKXsYLZdR3digsmwIWAp/y+NHR2cPzWQ==
X-Google-Smtp-Source: ABdhPJwuken19hAvQqUqHZreGho/udbFEVyecKPAEhXbyr+FG36vuGUBMU0z+Zw4yvdKvEBXtZ+4peX1pZWhGpj7a34=
X-Received: by 2002:a17:906:180b:: with SMTP id v11mr79086eje.41.1606342403708;  Wed, 25 Nov 2020 14:13:23 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>  
In-Reply-To: 
From: Tom Herbert 
Date: Wed, 25 Nov 2020 14:13:11 -0800
Message-ID: 
To: "Joel M. Halpern" 
Cc: IPv6 Operations , tcpm 
Content-Type: multipart/alternative; boundary="0000000000005c70cc05b4f5bcdb"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 22:13:29 -0000

--0000000000005c70cc05b4f5bcdb
Content-Type: text/plain; charset="UTF-8"

Joel, is there a normative definition of a flow?

On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern  wrote:

> No, as Brian says, there are escape clauses in the flow definitions.
>
> But changing the flow label due to traffic problems does not correspond
> to packets being in actually different flows.
> If one were using UDP, and mixing loss sensitive packets with loss
> insensitive packets for a special application, sure, one could use two
> flow labels.  But that is not what you are describing.
>
> Yours,
> Joel
>
> On 11/25/2020 3:05 PM, Tom Herbert wrote:
> > Joel,
> >
> > Is there an RFC that clearly and unambiguously states that a host MUST
> > use the same flow label for the lifetime _and_ clearly defines exactly
> > what a flow is with respect to such a requirement (for instance, how
> > would you define a flow and enforce such a requirement in UDP? IPsec?
> > other encapsulations?). If there is such a requirement then we'll
> > change the code to be conformant.
> >
> > Tom
> >
> > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern 
> wrote:
> >>
> >> This kind of thing is why, as I understand it, MPTCP has discovery
> >> mechanisms ot know if both sides use it, and can select alternative
> >> addresses for communication.
> >>
> >> Trying to guess flow labels that might avoid a problem because it might
> >> be an ECMP problem, is just flailing about.  Not a good design for
> >> operational protocols.
> >>
> >> And in general, designing protocols around "I know exactly what is going
> >> on"  (the requirement for what you describe that goes well beyond just
> >> "limited domains") is also a recipe for failure.
> >>
> >> The Flow Label RFCs are actually very explicit that a flow label is
> >> supposed to be stable for the life of the flow.  Otherwise, it isn't a
> >> flow label.
> >>
> >> Yours,
> >> Joel
> >>
> >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> >>> Hi Fernando, comments in line...
> >>>
> >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont 
> wrote:
> >>>>
> >>>> Hi, Tom,
> >>>>
> >>>> On 24/11/20 16:43, Tom Herbert wrote:
> >>>> [....]
> >>>>> Modulating the flow label is a means to affect the routing of packets
> >>>>> through the network that uses flow labels as input to the ECMP hash.
> >>>>
> >>>> What's the point?
> >>>>
> >>>> 1) You cannot tell *if* the FL is being used.
> >>>>
> >>> Generally true, but in a limited domain this information could be
> >>> discerned. I'd note that it's also generally true that we don't know
> >>> if there is a load balancer or stateful firewall in the path that
> >>> requires consistent routing, but in a limited domain we could know
> >>> that also.
> >>>
> >>>> 2) Changing the FL does not necessarily mean that packets will employ
> a
> >>>> different link.
> >>>
> >>> It's an opportunistic mechanism. If a connection is failing and we get
> >>> a better path that fixes it by simply changing the flow label then
> >>> what's the harm?
> >>>
> >>>>
> >>>> 3) If the network is failing, shouldn't you handle this via routing?
> >>>>
> >>> Sure, but then that requires an out of band feedback loop from a TCP
> >>> implementation to the network infrastructure to indicate there is a
> >>> problem and then the network needs to respond. That's significant
> >>> infrastructure and higher reaction time than doing something in TCP
> >>> and IP. Think of modulating the flow label is an inexpensive form of
> >>> source routing within a limited domain that doesn't need any
> >>> infrastructure or heavyweight protocols or something like segment
> >>> routing.
> >>>
> >>>>
> >>>>
> >>>>> The basic idea is that the flow label associated with a connection is
> >>>>> randomly changed when the stack observes that the connection is
> >>>>> failing (e.g. and an RTO). There is nothing in the specs that
> prevents
> >>>>> this since the source is at liberty to set the flow label as it sees
> >>>>> fit.
> >>>>
> >>>> The FL is expected to remain constant for the life of a flow. A
> >>>> retransmitted packet is part of the same flow as the
> >>>> originally-transmitted packet. So this seems to be contradicting the
> >>>> very specification of the FL.
> >>>>
> >>>> For instance, If a RTO for a flow causes the FL to change, then one
> may
> >>>> possibly argue that the FL is not naming/labeling what is
> said/expected
> >>>> to be anming/labeling.
> >>>
> >>> Specifically, RFC6437 states:
> >>>
> >>> "It is therefore RECOMMENDED that source hosts support the flow label
> >>> by setting the flow label field for all packets of a given flow to the
> >>> same value chosen from an approximation to a discrete uniform
> >>> distribution."
> >>>
> >>> So that is clearly a just recommendation, and not a requirement (and
> >>> definitely not a MUST). Furthermore, RFC6437 states:
> >>>
> >>> "A forwarding node MUST either leave a non-zero flow label value
> >>> unchanged or change it only for compelling operational security
> >>> reasons as described in Section 6.1."
> >>>
> >>> So there's no guarantee in the protocol specs that flow labels are
> >>> consistent for the life of the connection, which means that the
> >>> network cannot assume that and thus it would be incorrect if the
> >>> network tried to enforce flow label consistency as a protocol
> >>> requirement. As I said, it is prudent to try to be consistent with
> >>> flow labels and the default behavior in Linux should be changed,
> >>> however I do not believe there's a valid claim of non-conformance that
> >>> motivates removal of the feature that is already deployed.
> >>>
> >>> Tom
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>>
> >>>>
> >>>>> The feature is useful in large datacenter networks, like
> >>>>> pparently Facebook where the patches originate, since information
> >>>>> discerned by TCP can opportunistically be applied to route selection.
> >>>>> The practical issue is that there are stateful devices like firewalls
> >>>>> that require consistent routing in the network in which case changing
> >>>>> the flow label can confuse them. As I mentioned, the original intent
> >>>>> was that the flow label randomization feature should be opt-in
> instead
> >>>>> of on by default.
> >>>>
> >>>> So... where is the "source" of the packet that would be "modulating"
> the FL?
> >>>>
> >>>> Thanks,
> >>>> --
> >>>> Fernando Gont
> >>>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> >>>>
> >>>>
> >>>>
> >>>
> >>> _______________________________________________
> >>> v6ops mailing list
> >>> v6ops@ietf.org
> >>> https://www.ietf.org/mailman/listinfo/v6ops
> >>>
>

--0000000000005c70cc05b4f5bcdb
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Joel, is there a normative definition of a flow?
=

On Wed=
, Nov 25, 2020, 1:27 PM Joel M. Halpern <jmh@joelhalpern.com> wrote:
No, as Brian says, there are escape clauses in the flow definitions=
.



But changing the flow label due to traffic problems does not correspond 
to packets being in actually different flows.

If one were using UDP, and mixing loss sensitive packets with loss 

insensitive packets for a special application, sure, one could use two 

flow labels.=C2=A0 But that is not what you are describing.



Yours,

Joel



On 11/25/2020 3:05 PM, Tom Herbert wrote:

> Joel,

> 

> Is there an RFC that clearly and unambiguously states that a host MUST=


> use the same flow label for the lifetime _and_ clearly defines exactly=


> what a flow is with respect to such a requirement (for instance, how
> would you define a flow and enforce such a requirement in UDP? IPsec?<=
br>
> other encapsulations?). If there is such a requirement then we'll<=
br>
> change the code to be conformant.

> 

> Tom

> 

> On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <jmh@joelhalpern.=
com> wrote:

>>

>> This kind of thing is why, as I understand it, MPTCP has discovery=


>> mechanisms ot know if both sides use it, and can select alternativ=
e

>> addresses for communication.

>>

>> Trying to guess flow labels that might avoid a problem because it =
might

>> be an ECMP problem, is just flailing about.=C2=A0 Not a good desig=
n for

>> operational protocols.

>>

>> And in general, designing protocols around "I know exactly wh=
at is going

>> on"=C2=A0 (the requirement for what you describe that goes we=
ll beyond just

>> "limited domains") is also a recipe for failure.

>>

>> The Flow Label RFCs are actually very explicit that a flow label i=
s

>> supposed to be stable for the life of the flow.=C2=A0 Otherwise, i=
t isn't a

>> flow label.

>>

>> Yours,

>> Joel

>>

>> On 11/25/2020 2:35 PM, Tom Herbert wrote:

>>> Hi Fernando, comments in line...

>>>

>>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <fernando@=
gont.com.ar> wrote:

>>>>

>>>> Hi, Tom,

>>>>

>>>> On 24/11/20 16:43, Tom Herbert wrote:

>>>> [....]

>>>>> Modulating the flow label is a means to affect the rou=
ting of packets

>>>>> through the network that uses flow labels as input to =
the ECMP hash.

>>>>

>>>> What's the point?

>>>>

>>>> 1) You cannot tell *if* the FL is being used.

>>>>

>>> Generally true, but in a limited domain this information could=
 be

>>> discerned. I'd note that it's also generally true that=
 we don't know

>>> if there is a load balancer or stateful firewall in the path t=
hat

>>> requires consistent routing, but in a limited domain we could =
know

>>> that also.

>>>

>>>> 2) Changing the FL does not necessarily mean that packets =
will employ a

>>>> different link.

>>>

>>> It's an opportunistic mechanism. If a connection is failin=
g and we get

>>> a better path that fixes it by simply changing the flow label =
then

>>> what's the harm?

>>>

>>>>

>>>> 3) If the network is failing, shouldn't you handle thi=
s via routing?

>>>>

>>> Sure, but then that requires an out of band feedback loop from=
 a TCP

>>> implementation to the network infrastructure to indicate there=
 is a

>>> problem and then the network needs to respond. That's sign=
ificant

>>> infrastructure and higher reaction time than doing something i=
n TCP

>>> and IP. Think of modulating the flow label is an inexpensive f=
orm of

>>> source routing within a limited domain that doesn't need a=
ny

>>> infrastructure or heavyweight protocols or something like segm=
ent

>>> routing.

>>>

>>>>

>>>>

>>>>> The basic idea is that the flow label associated with =
a connection is

>>>>> randomly changed when the stack observes that the conn=
ection is

>>>>> failing (e.g. and an RTO). There is nothing in the spe=
cs that prevents

>>>>> this since the source is at liberty to set the flow la=
bel as it sees

>>>>> fit.

>>>>

>>>> The FL is expected to remain constant for the life of a fl=
ow. A

>>>> retransmitted packet is part of the same flow as the

>>>> originally-transmitted packet. So this seems to be contrad=
icting the

>>>> very specification of the FL.

>>>>

>>>> For instance, If a RTO for a flow causes the FL to change,=
 then one may

>>>> possibly argue that the FL is not naming/labeling what is =
said/expected

>>>> to be anming/labeling.

>>>

>>> Specifically, RFC6437 states:

>>>

>>> "It is therefore RECOMMENDED that source hosts support th=
e flow label

>>> by setting the flow label field for all packets of a given flo=
w to the

>>> same value chosen from an approximation to a discrete uniform<=
br>
>>> distribution."

>>>

>>> So that is clearly a just recommendation, and not a requiremen=
t (and

>>> definitely not a MUST). Furthermore, RFC6437 states:

>>>

>>> "A forwarding node MUST either leave a non-zero flow labe=
l value

>>> unchanged or change it only for compelling operational securit=
y

>>> reasons as described in Section 6.1."

>>>

>>> So there's no guarantee in the protocol specs that flow la=
bels are

>>> consistent for the life of the connection, which means that th=
e

>>> network cannot assume that and thus it would be incorrect if t=
he

>>> network tried to enforce flow label consistency as a protocol<=
br>
>>> requirement. As I said, it is prudent to try to be consistent =
with

>>> flow labels and the default behavior in Linux should be change=
d,

>>> however I do not believe there's a valid claim of non-conf=
ormance that

>>> motivates removal of the feature that is already deployed.

>>>

>>> Tom

>>>

>>>

>>>

>>>

>>>>

>>>>

>>>>

>>>>> The feature is useful in large datacenter networks, li=
ke

>>>>> pparently Facebook where the patches originate, since =
information

>>>>> discerned by TCP can opportunistically be applied to r=
oute selection.

>>>>> The practical issue is that there are stateful devices=
 like firewalls

>>>>> that require consistent routing in the network in whic=
h case changing

>>>>> the flow label can confuse them. As I mentioned, the o=
riginal intent

>>>>> was that the flow label randomization feature should b=
e opt-in instead

>>>>> of on by default.

>>>>

>>>> So... where is the "source" of the packet that w=
ould be "modulating" the FL?

>>>>

>>>> Thanks,

>>>> --

>>>> Fernando Gont

>>>> e-mail: fernando@gont.com.ar || fgont@si6networks=
.com

>>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D=
076 FFF1

>>>>

>>>>

>>>>

>>>

>>> _______________________________________________

>>> v6ops mailing list

>>> v6ops@ietf.org

>>> https://www.ietf.org/mailman/list=
info/v6ops

>>>




--0000000000005c70cc05b4f5bcdb--


From nobody Wed Nov 25 14:33:55 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BCAB43A1F16; Wed, 25 Nov 2020 14:33:52 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level: 
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sRoGAESFMVrn; Wed, 25 Nov 2020 14:33:48 -0800 (PST)
Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3F4723A0CDD; Wed, 25 Nov 2020 14:33:48 -0800 (PST)
Received: by mail-pf1-x432.google.com with SMTP id y7so3673493pfq.11; Wed, 25 Nov 2020 14:33:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=QMExF4ke5oEJQZHMD9QX5vBOxSZXJ0QGEhOgBuvsF68=; b=XXKOl+9VHpwdkDmWkAZ68KAN77ncRDkqSg8VHhDPtuP1Ymb/1PTHHrlRNH+gDnnYg/ UxeECbWjBxW8Dq9wUy8xSA41WgPfD58CyAdZJRzVA/3drhKCc/U/2bFHu/PtGTJairx5 YUZX8ajIX6wa5NJlNe1KzcJLVQypQKvPXPZGXvxiKSBf8BDcDPnOhMSnnvlsbv1fEeXN hwFo4be6/ct0yCL29SIaPqhCiKUDSVvxmGJ6ULauDga+33EBd/Ea0KLZ5L+jqN72w6xR /hBSqnBmjPdyrJo/dfPDoh7cDz60SBbqyhvfyjQ3Eq2yw2b+takcExXGSHBegvv1G+fE laig==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=QMExF4ke5oEJQZHMD9QX5vBOxSZXJ0QGEhOgBuvsF68=; b=miNbsJ0ipSXXxXzCToY+3ssk33A9/0/+l9gOamEM/v6nmqIOyd1LChVfK8N9s0FEqn gstrO/1zPj/GA3h9yWX2aHu/jxfQeH9lcdqEaMMrYN1crZ7UWIfCNm/cMCjyc+PqbBr9 T33e7/QE0vgeoLRBjjz/XdyfcjJZ9/umKvjKEqgQngcgzyUoPjY/R/FI4XF04kcKNIp8 XZUYx1Wkfr8eUNxl68PzW2fF7iiP7QukPvBuvSLtROuFxsDhDKqNkON8943+7ZBxhGrW Znrbn2W0V1ZQNx9YjB6lYEhN0V+YcrQtizxi9aIO2AQyC/BgllRK/B8w4h6CS4MKkgz9 8HMQ==
X-Gm-Message-State: AOAM532TiTJg6ozxIk4VTl8YyLg30uLhWgD2KvUFIqKLBZGbRzEfz9mM /wPCfao6hUj1XHrSsIGbPsigj+f2O815IA==
X-Google-Smtp-Source: ABdhPJwmnNgppsFseDMBGGxRCGj6ModoBe7YEFHh4HtVI/7MZfghzOBteOU0yHNHAbO2HQxCXVj80w==
X-Received: by 2002:a17:90b:941:: with SMTP id dw1mr6736698pjb.147.1606343627166;  Wed, 25 Nov 2020 14:33:47 -0800 (PST)
Received: from [192.168.178.20] ([151.210.131.28]) by smtp.gmail.com with ESMTPSA id s18sm2893351pfc.5.2020.11.25.14.33.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Nov 2020 14:33:46 -0800 (PST)
To: Tom Herbert , "Joel M. Halpern" 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>   
From: Brian E Carpenter 
Message-ID: 
Date: Thu, 26 Nov 2020 11:33:42 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 22:33:53 -0000

I'm not Joel, but I did once spend some time grepping RFCs to find out wh=
ether
"flow" or "microflow" was the preferred term. In RFC2474, which is normat=
ive,
we have:

   Microflow: a single instance of an application-to-application flow of
   packets which is identified by source address, destination address,
   protocol id, and source port, destination port (where applicable).

But in the flow label work we explicitly avoided being that precise, and
did not use the term "microflow". There might be some load balancing
scenarios where you want a broader definition, even including bidirection=
al
flows. There are expired drafts on that topic:
draft-tarreau-extend-flow-label-balancing
draft-wang-6man-flow-label-reflection

Regards
   Brian

On 26-Nov-20 11:13, Tom Herbert wrote:
> Joel, is there a normative definition of a flow?
>=20
> On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern > wrote:
>=20
>     No, as Brian says, there are escape clauses in the flow definitions=
=2E
>=20
>     But changing the flow label due to traffic problems does not corres=
pond
>     to packets being in actually different flows.
>     If one were using UDP, and mixing loss sensitive packets with loss
>     insensitive packets for a special application, sure, one could use =
two
>     flow labels.=C2=A0 But that is not what you are describing.
>=20
>     Yours,
>     Joel
>=20
>     On 11/25/2020 3:05 PM, Tom Herbert wrote:
>     > Joel,
>     >
>     > Is there an RFC that clearly and unambiguously states that a host=
 MUST
>     > use the same flow label for the lifetime _and_ clearly defines ex=
actly
>     > what a flow is with respect to such a requirement (for instance, =
how
>     > would you define a flow and enforce such a requirement in UDP? IP=
sec?
>     > other encapsulations?). If there is such a requirement then we'll=

>     > change the code to be conformant.
>     >
>     > Tom
>     >
>     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern > wrote:
>     >>
>     >> This kind of thing is why, as I understand it, MPTCP has discove=
ry
>     >> mechanisms ot know if both sides use it, and can select alternat=
ive
>     >> addresses for communication.
>     >>
>     >> Trying to guess flow labels that might avoid a problem because i=
t might
>     >> be an ECMP problem, is just flailing about.=C2=A0 Not a good des=
ign for
>     >> operational protocols.
>     >>
>     >> And in general, designing protocols around "I know exactly what =
is going
>     >> on"=C2=A0 (the requirement for what you describe that goes well =
beyond just
>     >> "limited domains") is also a recipe for failure.
>     >>
>     >> The Flow Label RFCs are actually very explicit that a flow label=
 is
>     >> supposed to be stable for the life of the flow.=C2=A0 Otherwise,=
 it isn't a
>     >> flow label.
>     >>
>     >> Yours,
>     >> Joel
>     >>
>     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>     >>> Hi Fernando, comments in line...
>     >>>
>     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont > wrote:
>     >>>>
>     >>>> Hi, Tom,
>     >>>>
>     >>>> On 24/11/20 16:43, Tom Herbert wrote:
>     >>>> [....]
>     >>>>> Modulating the flow label is a means to affect the routing of=
 packets
>     >>>>> through the network that uses flow labels as input to the ECM=
P hash.
>     >>>>
>     >>>> What's the point?
>     >>>>
>     >>>> 1) You cannot tell *if* the FL is being used.
>     >>>>
>     >>> Generally true, but in a limited domain this information could =
be
>     >>> discerned. I'd note that it's also generally true that we don't=
 know
>     >>> if there is a load balancer or stateful firewall in the path th=
at
>     >>> requires consistent routing, but in a limited domain we could k=
now
>     >>> that also.
>     >>>
>     >>>> 2) Changing the FL does not necessarily mean that packets will=
 employ a
>     >>>> different link.
>     >>>
>     >>> It's an opportunistic mechanism. If a connection is failing and=
 we get
>     >>> a better path that fixes it by simply changing the flow label t=
hen
>     >>> what's the harm?
>     >>>
>     >>>>
>     >>>> 3) If the network is failing, shouldn't you handle this via ro=
uting?
>     >>>>
>     >>> Sure, but then that requires an out of band feedback loop from =
a TCP
>     >>> implementation to the network infrastructure to indicate there =
is a
>     >>> problem and then the network needs to respond. That's significa=
nt
>     >>> infrastructure and higher reaction time than doing something in=
 TCP
>     >>> and IP. Think of modulating the flow label is an inexpensive fo=
rm of
>     >>> source routing within a limited domain that doesn't need any
>     >>> infrastructure or heavyweight protocols or something like segme=
nt
>     >>> routing.
>     >>>
>     >>>>
>     >>>>
>     >>>>> The basic idea is that the flow label associated with a conne=
ction is
>     >>>>> randomly changed when the stack observes that the connection =
is
>     >>>>> failing (e.g. and an RTO). There is nothing in the specs that=
 prevents
>     >>>>> this since the source is at liberty to set the flow label as =
it sees
>     >>>>> fit.
>     >>>>
>     >>>> The FL is expected to remain constant for the life of a flow. =
A
>     >>>> retransmitted packet is part of the same flow as the
>     >>>> originally-transmitted packet. So this seems to be contradicti=
ng the
>     >>>> very specification of the FL.
>     >>>>
>     >>>> For instance, If a RTO for a flow causes the FL to change, the=
n one may
>     >>>> possibly argue that the FL is not naming/labeling what is said=
/expected
>     >>>> to be anming/labeling.
>     >>>
>     >>> Specifically, RFC6437 states:
>     >>>
>     >>> "It is therefore RECOMMENDED that source hosts support the flow=
 label
>     >>> by setting the flow label field for all packets of a given flow=
 to the
>     >>> same value chosen from an approximation to a discrete uniform
>     >>> distribution."
>     >>>
>     >>> So that is clearly a just recommendation, and not a requirement=
 (and
>     >>> definitely not a MUST). Furthermore, RFC6437 states:
>     >>>
>     >>> "A forwarding node MUST either leave a non-zero flow label valu=
e
>     >>> unchanged or change it only for compelling operational security=

>     >>> reasons as described in Section 6.1."
>     >>>
>     >>> So there's no guarantee in the protocol specs that flow labels =
are
>     >>> consistent for the life of the connection, which means that the=

>     >>> network cannot assume that and thus it would be incorrect if th=
e
>     >>> network tried to enforce flow label consistency as a protocol
>     >>> requirement. As I said, it is prudent to try to be consistent w=
ith
>     >>> flow labels and the default behavior in Linux should be changed=
,
>     >>> however I do not believe there's a valid claim of non-conforman=
ce that
>     >>> motivates removal of the feature that is already deployed.
>     >>>
>     >>> Tom
>     >>>
>     >>>
>     >>>
>     >>>
>     >>>>
>     >>>>
>     >>>>
>     >>>>> The feature is useful in large datacenter networks, like
>     >>>>> pparently Facebook where the patches originate, since informa=
tion
>     >>>>> discerned by TCP can opportunistically be applied to route se=
lection.
>     >>>>> The practical issue is that there are stateful devices like f=
irewalls
>     >>>>> that require consistent routing in the network in which case =
changing
>     >>>>> the flow label can confuse them. As I mentioned, the original=
 intent
>     >>>>> was that the flow label randomization feature should be opt-i=
n instead
>     >>>>> of on by default.
>     >>>>
>     >>>> So... where is the "source" of the packet that would be "modul=
ating" the FL?
>     >>>>
>     >>>> Thanks,
>     >>>> --
>     >>>> Fernando Gont
>     >>>> e-mail: fernando@gont.com.ar  || =
fgont@si6networks.com 
>     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 =
FFF1
>     >>>>
>     >>>>
>     >>>>
>     >>>
>     >>> _______________________________________________
>     >>> v6ops mailing list
>     >>> v6ops@ietf.org 
>     >>> https://www.ietf.org/mailman/listinfo/v6ops
>     >>>
>=20
>=20
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
>=20


From nobody Wed Nov 25 15:04:15 2020
Return-Path: 
X-Original-To: tcpm@ietf.org
Delivered-To: tcpm@ietfa.amsl.com
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id F39BE3A0FAF; Wed, 25 Nov 2020 15:04:08 -0800 (PST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
From: internet-drafts@ietf.org
To: 
Cc: tcpm@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 7.23.0
Auto-Submitted: auto-generated
Precedence: bulk
Reply-To: tcpm@ietf.org
Message-ID: <160634544894.28725.8891473900084027588@ietfa.amsl.com>
Date: Wed, 25 Nov 2020 15:04:08 -0800
Archived-At: 
Subject: [tcpm] I-D Action: draft-ietf-tcpm-2140bis-06.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Wed, 25 Nov 2020 23:04:09 -0000

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the TCP Maintenance and Minor Extensions WG of the IETF.

        Title           : TCP Control Block Interdependence
        Authors         : Joe Touch
                          Michael Welzl
                          Safiqul Islam
	Filename        : draft-ietf-tcpm-2140bis-06.txt
	Pages           : 34
	Date            : 2020-11-25

Abstract:
   This memo provides guidance to TCP implementers that are intended to
   help improve convergence to steady-state operation without affecting
   interoperability. It updates and replaces RFC 2140's description of
   interdependent TCP control blocks and the ways that part of TCP
   state can be shared among similar concurrent or consecutive
   connections. TCP state includes a combination of parameters, such as
   connection state, current round-trip time estimates, congestion
   control information, and process information. Most of this state is
   maintained on a per-connection basis in the TCP Control Block (TCB),
   but implementations can (and do) share certain TCB information
   across connections to the same host. Such sharing is intended to
   improve overall transient transport performance, while maintaining
   backward-compatibility with existing implementations. The sharing
   described herein is limited to only the TCB initialization and so
   has no effect on the long-term behavior of TCP after a connection
   has been established.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-tcpm-2140bis/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-ietf-tcpm-2140bis-06
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-2140bis-06

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-2140bis-06


Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/



From nobody Wed Nov 25 16:16:15 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 797053A074E for ; Wed, 25 Nov 2020 16:16:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.102
X-Spam-Level: 
X-Spam-Status: No, score=0.102 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EQPA6EeQV3Yh for ; Wed, 25 Nov 2020 16:16:02 -0800 (PST)
Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4C4AB3A0794 for ; Wed, 25 Nov 2020 16:16:02 -0800 (PST)
Received: by mail-ed1-x535.google.com with SMTP id r22so331172edw.6 for ; Wed, 25 Nov 2020 16:16:02 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=o6M9JzL0xm0OD2nnsrd2Ip3bYZXIG6TMOIrJr7YwqcY=; b=ORz+VeyKgpzNz+a7eEu+bo+4ZXt3BbGwivunw4/42rXkxolzdD84YOHFMDBdJsn0zh LZy5Q0UGPESGvDTqO0DH754N7xd6l64XJB26ttgfnzmNHEbSD13+NK1aKvhuui4Jc6l1 U7KGPJ4XDEhH0Pu/D3iDq3ZL3A0j6zmSuXM17j/nSPRvXefGOecQdAB0X/wLm5m8bnsv ne4NDRxWBf6B8PnZV0O5tEGS871YMspRa2tatUh8FH2FWNuHQgp0cARTzwKBCn9uN64V S1V3mBNnRIUmz8dD/+Odbv/X/vSreRokwXDoy7r9P351YSf+riHo8lcAq3YFIRvvDNrU ykWQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=o6M9JzL0xm0OD2nnsrd2Ip3bYZXIG6TMOIrJr7YwqcY=; b=A/LJkgf7rflpaHlG+4RgWo4WxcKaM20IDspaOk/xjH7lFttW28H5wzy9cLcLLpgW8x Mlp+PfPcGFKDITdIvPrRTmtjSRKhZ0VaOQmQO70S4Z12JDGuZiPLr0MHAbMHF+jQUo0L E2yB4OUNc6VRanTHhn2Kz4zWbb/nQ6MvWUuOaU8EgRqyWXrxd5I96ao9Nye0jl3CU95/ xclB1eSdoQUKrAML32R0sR7jIpiqW0cVNgE/2jAIuEIu8TxOh4/MvMio++TC2jETzR7O B/3PUQRLjLtA7U5KzrZ+5zbQBJjt8vtz87CnPGktw8QKm9KaAEs+dc79N5XZbXaHCmHz 4Cug==
X-Gm-Message-State: AOAM530K7yBXmVku4Pt/7JCh9Y4iI38b8uK6PwO9C5q5KlxJl9lGbcWc URpuoFzsMggKAojMh+vqtydTxmNLlUoyJ4ibJ3XS/Q==
X-Google-Smtp-Source: ABdhPJxZ0t1PhJb3eiLV8a/TuJs+dxV77o+hZkO1ZtQ0Kx0x71j8K1LV6/sgOkrrw4ZSaZog9rtzHEQWLkRCWZmgmd4=
X-Received: by 2002:aa7:c313:: with SMTP id l19mr151566edq.293.1606349760354;  Wed, 25 Nov 2020 16:16:00 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>    
In-Reply-To: 
From: Tom Herbert 
Date: Wed, 25 Nov 2020 17:15:49 -0700
Message-ID: 
To: Brian E Carpenter 
Cc: "Joel M. Halpern" , IPv6 Operations ,  tcpm 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 00:16:13 -0000

On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
 wrote:
>
> I'm not Joel, but I did once spend some time grepping RFCs to find out whether
> "flow" or "microflow" was the preferred term. In RFC2474, which is normative,
> we have:
>
>    Microflow: a single instance of an application-to-application flow of
>    packets which is identified by source address, destination address,
>    protocol id, and source port, destination port (where applicable).
>
> But in the flow label work we explicitly avoided being that precise, and
> did not use the term "microflow". There might be some load balancing
> scenarios where you want a broader definition, even including bidirectional
> flows. There are expired drafts on that topic:
> draft-tarreau-extend-flow-label-balancing
> draft-wang-6man-flow-label-reflection
>
Brian,

Random Packet Spraying
(https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf)
is an interesting idea where packets for a single connection are
purposely distributed across multiple paths for load distribution. Per
packet randomized flow labels with flow label aware ECMP makes this
quite easy to do without requiring any special support in switches
like you'd need with IPv4. I'm not necessarily advocating this, but it
does highlight one potential use case of having a flow label that
doesn;t have rigidly defined requirements on the host.

Tom

> Regards
>    Brian
>
> On 26-Nov-20 11:13, Tom Herbert wrote:
> > Joel, is there a normative definition of a flow?
> >
> > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern > wrote:
> >
> >     No, as Brian says, there are escape clauses in the flow definitions.
> >
> >     But changing the flow label due to traffic problems does not correspond
> >     to packets being in actually different flows.
> >     If one were using UDP, and mixing loss sensitive packets with loss
> >     insensitive packets for a special application, sure, one could use two
> >     flow labels.  But that is not what you are describing.
> >
> >     Yours,
> >     Joel
> >
> >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
> >     > Joel,
> >     >
> >     > Is there an RFC that clearly and unambiguously states that a host MUST
> >     > use the same flow label for the lifetime _and_ clearly defines exactly
> >     > what a flow is with respect to such a requirement (for instance, how
> >     > would you define a flow and enforce such a requirement in UDP? IPsec?
> >     > other encapsulations?). If there is such a requirement then we'll
> >     > change the code to be conformant.
> >     >
> >     > Tom
> >     >
> >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern > wrote:
> >     >>
> >     >> This kind of thing is why, as I understand it, MPTCP has discovery
> >     >> mechanisms ot know if both sides use it, and can select alternative
> >     >> addresses for communication.
> >     >>
> >     >> Trying to guess flow labels that might avoid a problem because it might
> >     >> be an ECMP problem, is just flailing about.  Not a good design for
> >     >> operational protocols.
> >     >>
> >     >> And in general, designing protocols around "I know exactly what is going
> >     >> on"  (the requirement for what you describe that goes well beyond just
> >     >> "limited domains") is also a recipe for failure.
> >     >>
> >     >> The Flow Label RFCs are actually very explicit that a flow label is
> >     >> supposed to be stable for the life of the flow.  Otherwise, it isn't a
> >     >> flow label.
> >     >>
> >     >> Yours,
> >     >> Joel
> >     >>
> >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> >     >>> Hi Fernando, comments in line...
> >     >>>
> >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont > wrote:
> >     >>>>
> >     >>>> Hi, Tom,
> >     >>>>
> >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
> >     >>>> [....]
> >     >>>>> Modulating the flow label is a means to affect the routing of packets
> >     >>>>> through the network that uses flow labels as input to the ECMP hash.
> >     >>>>
> >     >>>> What's the point?
> >     >>>>
> >     >>>> 1) You cannot tell *if* the FL is being used.
> >     >>>>
> >     >>> Generally true, but in a limited domain this information could be
> >     >>> discerned. I'd note that it's also generally true that we don't know
> >     >>> if there is a load balancer or stateful firewall in the path that
> >     >>> requires consistent routing, but in a limited domain we could know
> >     >>> that also.
> >     >>>
> >     >>>> 2) Changing the FL does not necessarily mean that packets will employ a
> >     >>>> different link.
> >     >>>
> >     >>> It's an opportunistic mechanism. If a connection is failing and we get
> >     >>> a better path that fixes it by simply changing the flow label then
> >     >>> what's the harm?
> >     >>>
> >     >>>>
> >     >>>> 3) If the network is failing, shouldn't you handle this via routing?
> >     >>>>
> >     >>> Sure, but then that requires an out of band feedback loop from a TCP
> >     >>> implementation to the network infrastructure to indicate there is a
> >     >>> problem and then the network needs to respond. That's significant
> >     >>> infrastructure and higher reaction time than doing something in TCP
> >     >>> and IP. Think of modulating the flow label is an inexpensive form of
> >     >>> source routing within a limited domain that doesn't need any
> >     >>> infrastructure or heavyweight protocols or something like segment
> >     >>> routing.
> >     >>>
> >     >>>>
> >     >>>>
> >     >>>>> The basic idea is that the flow label associated with a connection is
> >     >>>>> randomly changed when the stack observes that the connection is
> >     >>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
> >     >>>>> this since the source is at liberty to set the flow label as it sees
> >     >>>>> fit.
> >     >>>>
> >     >>>> The FL is expected to remain constant for the life of a flow. A
> >     >>>> retransmitted packet is part of the same flow as the
> >     >>>> originally-transmitted packet. So this seems to be contradicting the
> >     >>>> very specification of the FL.
> >     >>>>
> >     >>>> For instance, If a RTO for a flow causes the FL to change, then one may
> >     >>>> possibly argue that the FL is not naming/labeling what is said/expected
> >     >>>> to be anming/labeling.
> >     >>>
> >     >>> Specifically, RFC6437 states:
> >     >>>
> >     >>> "It is therefore RECOMMENDED that source hosts support the flow label
> >     >>> by setting the flow label field for all packets of a given flow to the
> >     >>> same value chosen from an approximation to a discrete uniform
> >     >>> distribution."
> >     >>>
> >     >>> So that is clearly a just recommendation, and not a requirement (and
> >     >>> definitely not a MUST). Furthermore, RFC6437 states:
> >     >>>
> >     >>> "A forwarding node MUST either leave a non-zero flow label value
> >     >>> unchanged or change it only for compelling operational security
> >     >>> reasons as described in Section 6.1."
> >     >>>
> >     >>> So there's no guarantee in the protocol specs that flow labels are
> >     >>> consistent for the life of the connection, which means that the
> >     >>> network cannot assume that and thus it would be incorrect if the
> >     >>> network tried to enforce flow label consistency as a protocol
> >     >>> requirement. As I said, it is prudent to try to be consistent with
> >     >>> flow labels and the default behavior in Linux should be changed,
> >     >>> however I do not believe there's a valid claim of non-conformance that
> >     >>> motivates removal of the feature that is already deployed.
> >     >>>
> >     >>> Tom
> >     >>>
> >     >>>
> >     >>>
> >     >>>
> >     >>>>
> >     >>>>
> >     >>>>
> >     >>>>> The feature is useful in large datacenter networks, like
> >     >>>>> pparently Facebook where the patches originate, since information
> >     >>>>> discerned by TCP can opportunistically be applied to route selection.
> >     >>>>> The practical issue is that there are stateful devices like firewalls
> >     >>>>> that require consistent routing in the network in which case changing
> >     >>>>> the flow label can confuse them. As I mentioned, the original intent
> >     >>>>> was that the flow label randomization feature should be opt-in instead
> >     >>>>> of on by default.
> >     >>>>
> >     >>>> So... where is the "source" of the packet that would be "modulating" the FL?
> >     >>>>
> >     >>>> Thanks,
> >     >>>> --
> >     >>>> Fernando Gont
> >     >>>> e-mail: fernando@gont.com.ar  || fgont@si6networks.com 
> >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> >     >>>>
> >     >>>>
> >     >>>>
> >     >>>
> >     >>> _______________________________________________
> >     >>> v6ops mailing list
> >     >>> v6ops@ietf.org 
> >     >>> https://www.ietf.org/mailman/listinfo/v6ops
> >     >>>
> >
> >
> > _______________________________________________
> > v6ops mailing list
> > v6ops@ietf.org
> > https://www.ietf.org/mailman/listinfo/v6ops
> >
>


From nobody Thu Nov 26 01:34:20 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A3CAF3A0B6E; Thu, 26 Nov 2020 01:34:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.098
X-Spam-Level: 
X-Spam-Status: No, score=-0.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2iXUn606boMO; Thu, 26 Nov 2020 01:34:16 -0800 (PST)
Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3ACB83A0B74; Thu, 26 Nov 2020 01:34:16 -0800 (PST)
Received: by mail-ot1-x332.google.com with SMTP id k3so1322159otp.12; Thu, 26 Nov 2020 01:34:16 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6i2UIRIcCmDOKlC10XLkrbtGGakS9jR6ru7+roBSLN4=; b=hK13hU8dJhykLpH5ROHQXEEpRkDgF63YawuhUBPgwRHCys1x9skeXGlbx7o/ur7vyv lZY5YJJNWfQ7PfHIMYSDSEKhlmWka8SLXAZj8hhzt6F06cnc7uOwqeOk1G9QZgoZ5MRB cOwNng3XoVeKX+Hould2J+86t8lwAN+94JbfV85GMz+ApfOruRxEBMbkS7KOzbOh7fCz t1W7cQiqLSEVKchUF9vR8T7ABD9PmCwfMBY5jStEBIFFnDxCkrj5RtC3/vlYa/HHAvE1 XN3b30+PGjcxrGN1TYS2SgT0lMGtxReUVWsp8UpdDWY79DdLwRiHxZucIEL3SvgFaOWV Ba6Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6i2UIRIcCmDOKlC10XLkrbtGGakS9jR6ru7+roBSLN4=; b=TL/UiqjYtof0xjU93NFQSqcZW7/RAvh3MgHn/ddAMa6ScpCDriJurbKVrOjRPeNQid qouiwzzLz8cd150wOehvrdNBg82Mgn8IuKiuHfs5rEGRRPNyEAyZyjl2Zbs9ROrKrr5T qQeLMkS3sSC7uZAlsihBisUtTHRFJJ75Isksu28qcQUL8pNi2BFlX4iuRFCOAw5yIBv0 +2PiwyXowB81eZQWRloImUhxod+G0iaUpiQixK8cQslLIRNiMr06L4UOY/irIEFy55e5 UD5Rt+sBciuipxKdaEw7JV8KJSJd2C3QHCiyAwR//FU53i1Y9Dy9Czai6NlwTs/tdhzP oWsw==
X-Gm-Message-State: AOAM533wZYaB5j9fyko/JZ7RfO6lVdR+UibVR+f4cYgpwbYxr60JCXTF yoXlW40BDkwcn9Qi8qZ2oBnt2QDOZKEwr1eCrLyYSanRlMs=
X-Google-Smtp-Source: ABdhPJy27wvl/Nc0VfP/i7FiZJzgBFRmkpFp22fUaewtPjtM73MoAi6fEbfN1c38zLmyiihRwohwYVFpNjQb4JnP4T8=
X-Received: by 2002:a9d:4c92:: with SMTP id m18mr1814874otf.248.1606383255507;  Thu, 26 Nov 2020 01:34:15 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>     
In-Reply-To: 
From: Alexander Azimov 
Date: Thu, 26 Nov 2020 12:34:04 +0300
Message-ID: 
To: Tom Herbert 
Cc: Brian E Carpenter , tcpm ,  IPv6 Operations 
Content-Type: multipart/alternative; boundary="0000000000005157af05b4ff3f8f"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 09:34:19 -0000

--0000000000005157af05b4ff3f8f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Dear colleagues,

We started discussing an incorrect default behavior, and Tom has already
confirmed that it will be fixed.

Later the thread turned into an argument if the flow label can be changed
during connection/flow lifetime according to current RFC documents, though
these documents can be updated. This looks a bit weird for me because I
always thought that it is the IETF community's responsibility to document
proper solutions. If something undocumented but worthy happens in the
industry - IETF should catch up. So, I would like to get back to the
discussion of reasons and their safety.

The general idea of changing the routing path upon network
outage/degradation looks obvious. Getting kind of source-based routing can
significantly reduce the reaction time and improve end-user experience. The
flow label has a perfect match here: transparent for the application, set
by the source, not part of 5-tuple, while it can be used in the load
balancing. IMO poorly documented. I would like to learn your feedback for
the next wording:

Let say that the flow label is a hash of values from the IP packet's
5-tuple and random number. Then

1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
recalculated.
2. In the case of RTO timeout expiration in the established TCP session the
flow label MAY be recalculated. This setting MUST be switched off by
default.
3. Otherwise flow label SHOULD be preserved unchanged.

The first point stands for redirecting connection from the degraded path
before the connection is established, and it looks safe. The second one can
also improve performance if it is set on the server-side. Please comment if
you see security flaws in such a design.

=D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, Tom H=
erbert :

> On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
>  wrote:
> >
> > I'm not Joel, but I did once spend some time grepping RFCs to find out
> whether
> > "flow" or "microflow" was the preferred term. In RFC2474, which is
> normative,
> > we have:
> >
> >    Microflow: a single instance of an application-to-application flow o=
f
> >    packets which is identified by source address, destination address,
> >    protocol id, and source port, destination port (where applicable).
> >
> > But in the flow label work we explicitly avoided being that precise, an=
d
> > did not use the term "microflow". There might be some load balancing
> > scenarios where you want a broader definition, even including
> bidirectional
> > flows. There are expired drafts on that topic:
> > draft-tarreau-extend-flow-label-balancing
> > draft-wang-6man-flow-label-reflection
> >
> Brian,
>
> Random Packet Spraying
> (
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.529&rep=
=3Drep1&type=3Dpdf
> )
> is an interesting idea where packets for a single connection are
> purposely distributed across multiple paths for load distribution. Per
> packet randomized flow labels with flow label aware ECMP makes this
> quite easy to do without requiring any special support in switches
> like you'd need with IPv4. I'm not necessarily advocating this, but it
> does highlight one potential use case of having a flow label that
> doesn;t have rigidly defined requirements on the host.
>
> Tom
>
> > Regards
> >    Brian
> >
> > On 26-Nov-20 11:13, Tom Herbert wrote:
> > > Joel, is there a normative definition of a flow?
> > >
> > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern  > wrote:
> > >
> > >     No, as Brian says, there are escape clauses in the flow
> definitions.
> > >
> > >     But changing the flow label due to traffic problems does not
> correspond
> > >     to packets being in actually different flows.
> > >     If one were using UDP, and mixing loss sensitive packets with los=
s
> > >     insensitive packets for a special application, sure, one could us=
e
> two
> > >     flow labels.  But that is not what you are describing.
> > >
> > >     Yours,
> > >     Joel
> > >
> > >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
> > >     > Joel,
> > >     >
> > >     > Is there an RFC that clearly and unambiguously states that a
> host MUST
> > >     > use the same flow label for the lifetime _and_ clearly defines
> exactly
> > >     > what a flow is with respect to such a requirement (for instance=
,
> how
> > >     > would you define a flow and enforce such a requirement in UDP?
> IPsec?
> > >     > other encapsulations?). If there is such a requirement then we'=
ll
> > >     > change the code to be conformant.
> > >     >
> > >     > Tom
> > >     >
> > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <
> jmh@joelhalpern.com > wrote:
> > >     >>
> > >     >> This kind of thing is why, as I understand it, MPTCP has
> discovery
> > >     >> mechanisms ot know if both sides use it, and can select
> alternative
> > >     >> addresses for communication.
> > >     >>
> > >     >> Trying to guess flow labels that might avoid a problem because
> it might
> > >     >> be an ECMP problem, is just flailing about.  Not a good design
> for
> > >     >> operational protocols.
> > >     >>
> > >     >> And in general, designing protocols around "I know exactly wha=
t
> is going
> > >     >> on"  (the requirement for what you describe that goes well
> beyond just
> > >     >> "limited domains") is also a recipe for failure.
> > >     >>
> > >     >> The Flow Label RFCs are actually very explicit that a flow
> label is
> > >     >> supposed to be stable for the life of the flow.  Otherwise, it
> isn't a
> > >     >> flow label.
> > >     >>
> > >     >> Yours,
> > >     >> Joel
> > >     >>
> > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> > >     >>> Hi Fernando, comments in line...
> > >     >>>
> > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <
> fernando@gont.com.ar > wrote:
> > >     >>>>
> > >     >>>> Hi, Tom,
> > >     >>>>
> > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
> > >     >>>> [....]
> > >     >>>>> Modulating the flow label is a means to affect the routing
> of packets
> > >     >>>>> through the network that uses flow labels as input to the
> ECMP hash.
> > >     >>>>
> > >     >>>> What's the point?
> > >     >>>>
> > >     >>>> 1) You cannot tell *if* the FL is being used.
> > >     >>>>
> > >     >>> Generally true, but in a limited domain this information coul=
d
> be
> > >     >>> discerned. I'd note that it's also generally true that we
> don't know
> > >     >>> if there is a load balancer or stateful firewall in the path
> that
> > >     >>> requires consistent routing, but in a limited domain we could
> know
> > >     >>> that also.
> > >     >>>
> > >     >>>> 2) Changing the FL does not necessarily mean that packets
> will employ a
> > >     >>>> different link.
> > >     >>>
> > >     >>> It's an opportunistic mechanism. If a connection is failing
> and we get
> > >     >>> a better path that fixes it by simply changing the flow label
> then
> > >     >>> what's the harm?
> > >     >>>
> > >     >>>>
> > >     >>>> 3) If the network is failing, shouldn't you handle this via
> routing?
> > >     >>>>
> > >     >>> Sure, but then that requires an out of band feedback loop fro=
m
> a TCP
> > >     >>> implementation to the network infrastructure to indicate ther=
e
> is a
> > >     >>> problem and then the network needs to respond. That's
> significant
> > >     >>> infrastructure and higher reaction time than doing something
> in TCP
> > >     >>> and IP. Think of modulating the flow label is an inexpensive
> form of
> > >     >>> source routing within a limited domain that doesn't need any
> > >     >>> infrastructure or heavyweight protocols or something like
> segment
> > >     >>> routing.
> > >     >>>
> > >     >>>>
> > >     >>>>
> > >     >>>>> The basic idea is that the flow label associated with a
> connection is
> > >     >>>>> randomly changed when the stack observes that the connectio=
n
> is
> > >     >>>>> failing (e.g. and an RTO). There is nothing in the specs
> that prevents
> > >     >>>>> this since the source is at liberty to set the flow label a=
s
> it sees
> > >     >>>>> fit.
> > >     >>>>
> > >     >>>> The FL is expected to remain constant for the life of a flow=
.
> A
> > >     >>>> retransmitted packet is part of the same flow as the
> > >     >>>> originally-transmitted packet. So this seems to be
> contradicting the
> > >     >>>> very specification of the FL.
> > >     >>>>
> > >     >>>> For instance, If a RTO for a flow causes the FL to change,
> then one may
> > >     >>>> possibly argue that the FL is not naming/labeling what is
> said/expected
> > >     >>>> to be anming/labeling.
> > >     >>>
> > >     >>> Specifically, RFC6437 states:
> > >     >>>
> > >     >>> "It is therefore RECOMMENDED that source hosts support the
> flow label
> > >     >>> by setting the flow label field for all packets of a given
> flow to the
> > >     >>> same value chosen from an approximation to a discrete uniform
> > >     >>> distribution."
> > >     >>>
> > >     >>> So that is clearly a just recommendation, and not a
> requirement (and
> > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
> > >     >>>
> > >     >>> "A forwarding node MUST either leave a non-zero flow label
> value
> > >     >>> unchanged or change it only for compelling operational securi=
ty
> > >     >>> reasons as described in Section 6.1."
> > >     >>>
> > >     >>> So there's no guarantee in the protocol specs that flow label=
s
> are
> > >     >>> consistent for the life of the connection, which means that t=
he
> > >     >>> network cannot assume that and thus it would be incorrect if
> the
> > >     >>> network tried to enforce flow label consistency as a protocol
> > >     >>> requirement. As I said, it is prudent to try to be consistent
> with
> > >     >>> flow labels and the default behavior in Linux should be
> changed,
> > >     >>> however I do not believe there's a valid claim of
> non-conformance that
> > >     >>> motivates removal of the feature that is already deployed.
> > >     >>>
> > >     >>> Tom
> > >     >>>
> > >     >>>
> > >     >>>
> > >     >>>
> > >     >>>>
> > >     >>>>
> > >     >>>>
> > >     >>>>> The feature is useful in large datacenter networks, like
> > >     >>>>> pparently Facebook where the patches originate, since
> information
> > >     >>>>> discerned by TCP can opportunistically be applied to route
> selection.
> > >     >>>>> The practical issue is that there are stateful devices like
> firewalls
> > >     >>>>> that require consistent routing in the network in which cas=
e
> changing
> > >     >>>>> the flow label can confuse them. As I mentioned, the
> original intent
> > >     >>>>> was that the flow label randomization feature should be
> opt-in instead
> > >     >>>>> of on by default.
> > >     >>>>
> > >     >>>> So... where is the "source" of the packet that would be
> "modulating" the FL?
> > >     >>>>
> > >     >>>> Thanks,
> > >     >>>> --
> > >     >>>> Fernando Gont
> > >     >>>> e-mail: fernando@gont.com.ar 
> || fgont@si6networks.com 
> > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D07=
6
> FFF1
> > >     >>>>
> > >     >>>>
> > >     >>>>
> > >     >>>
> > >     >>> _______________________________________________
> > >     >>> v6ops mailing list
> > >     >>> v6ops@ietf.org 
> > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
> > >     >>>
> > >
> > >
> > > _______________________________________________
> > > v6ops mailing list
> > > v6ops@ietf.org
> > > https://www.ietf.org/mailman/listinfo/v6ops
> > >
> >
>
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
>


--=20
Best regards,
Alexander Azimov

--0000000000005157af05b4ff3f8f
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Dear colleagues,

We started discussing an incorrect=
 default behavior, and Tom has already confirmed that it will be fixed.
=

Later the thread turned into an argument if the flow label can be chang=
ed during connection/flow lifetime according to current RFC documents, thou=
gh these documents can be updated. This looks a bit weird for me because I =
always thought that it is the IETF community's responsibility to docume=
nt proper solutions. If something undocumented but worthy happens in the in=
dustry - IETF should catch up. So, I would like to get back to the discussi=
on of reasons and their safety. 

The general idea of changing the ro=
uting path upon network outage/degradation looks obvious. Getting kind of s=
ource-based routing can significantly reduce the reaction=20

 time and improve end-user experience. The flow label has a perfect match h=
ere: transparent for the application, set by the source, not part of 5-tupl=
e, while it can be used in the load balancing. IMO poorly documented. I wou=
ld like to learn your feedback for the next wording:

Let say that th=
e flow label is a hash of values from the IP packet's 5-tuple and rando=
m number. Then

1. In the case of SYN or SYN-ACK retransmission flow =
label SHOULD be recalculated.
2. In the case of RTO timeout expiration i=
n the established TCP session the flow label MAY be=20

recalculated. This setting MUST be switched off by default.
3. Otherwise=
 flow label SHOULD be preserved unchanged.

The first point stands fo=
r redirecting connection from the degraded path before the connection is es=
tablished, and it looks safe. The second one can also improve performance i=
f it is set on the server-side. Please comment if you see security flaws in=
 such a design.

=D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =
=D0=B2 03:16, Tom Herbert <tom@he=
rbertland.com>:
On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter

<brian.=
e.carpenter@gmail.com> wrote:

>

> I'm not Joel, but I did once spend some time grepping RFCs to find=
 out whether

> "flow" or "microflow" was the preferred term. In R=
FC2474, which is normative,

> we have:

>

>=C2=A0 =C2=A0 Microflow: a single instance of an application-to-applica=
tion flow of

>=C2=A0 =C2=A0 packets which is identified by source address, destinatio=
n address,

>=C2=A0 =C2=A0 protocol id, and source port, destination port (where app=
licable).

>

> But in the flow label work we explicitly avoided being that precise, a=
nd

> did not use the term "microflow". There might be some load b=
alancing

> scenarios where you want a broader definition, even including bidirect=
ional

> flows. There are expired drafts on that topic:

> draft-tarreau-extend-flow-label-balancing

> draft-wang-6man-flow-label-reflection

>

Brian,



Random Packet Spraying

(ht=
tps://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.529&rep=
=3Drep1&type=3Dpdf)

is an interesting idea where packets for a single connection are

purposely distributed across multiple paths for load distribution. Per

packet randomized flow labels with flow label aware ECMP makes this

quite easy to do without requiring any special support in switches

like you'd need with IPv4. I'm not necessarily advocating this, but=
 it

does highlight one potential use case of having a flow label that

doesn;t have rigidly defined requirements on the host.



Tom



> Regards

>=C2=A0 =C2=A0 Brian

>

> On 26-Nov-20 11:13, Tom Herbert wrote:

> > Joel, is there a normative definition of a flow?

> >

> > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <jmh@joelhalpern.com <mailt=
o:jmh@joelhalpern.=
com>> wrote:

> >

> >=C2=A0 =C2=A0 =C2=A0No, as Brian says, there are escape clauses in=
 the flow definitions.

> >

> >=C2=A0 =C2=A0 =C2=A0But changing the flow label due to traffic pro=
blems does not correspond

> >=C2=A0 =C2=A0 =C2=A0to packets being in actually different flows.<=
br>
> >=C2=A0 =C2=A0 =C2=A0If one were using UDP, and mixing loss sensiti=
ve packets with loss

> >=C2=A0 =C2=A0 =C2=A0insensitive packets for a special application,=
 sure, one could use two

> >=C2=A0 =C2=A0 =C2=A0flow labels.=C2=A0 But that is not what you ar=
e describing.

> >

> >=C2=A0 =C2=A0 =C2=A0Yours,

> >=C2=A0 =C2=A0 =C2=A0Joel

> >

> >=C2=A0 =C2=A0 =C2=A0On 11/25/2020 3:05 PM, Tom Herbert wrote:

> >=C2=A0 =C2=A0 =C2=A0> Joel,

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> Is there an RFC that clearly and unambigu=
ously states that a host MUST

> >=C2=A0 =C2=A0 =C2=A0> use the same flow label for the lifetime =
_and_ clearly defines exactly

> >=C2=A0 =C2=A0 =C2=A0> what a flow is with respect to such a req=
uirement (for instance, how

> >=C2=A0 =C2=A0 =C2=A0> would you define a flow and enforce such =
a requirement in UDP? IPsec?

> >=C2=A0 =C2=A0 =C2=A0> other encapsulations?). If there is such =
a requirement then we'll

> >=C2=A0 =C2=A0 =C2=A0> change the code to be conformant.

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> Tom

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> On Wed, Nov 25, 2020 at 12:39 PM Joel M. =
Halpern <jmh@jo=
elhalpern.com <mailto:jmh@joelhalpern.com>> wrote:

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> This kind of thing is why, as I under=
stand it, MPTCP has discovery

> >=C2=A0 =C2=A0 =C2=A0>> mechanisms ot know if both sides use =
it, and can select alternative

> >=C2=A0 =C2=A0 =C2=A0>> addresses for communication.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> Trying to guess flow labels that migh=
t avoid a problem because it might

> >=C2=A0 =C2=A0 =C2=A0>> be an ECMP problem, is just flailing =
about.=C2=A0 Not a good design for

> >=C2=A0 =C2=A0 =C2=A0>> operational protocols.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> And in general, designing protocols a=
round "I know exactly what is going

> >=C2=A0 =C2=A0 =C2=A0>> on"=C2=A0 (the requirement for w=
hat you describe that goes well beyond just

> >=C2=A0 =C2=A0 =C2=A0>> "limited domains") is also =
a recipe for failure.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> The Flow Label RFCs are actually very=
 explicit that a flow label is

> >=C2=A0 =C2=A0 =C2=A0>> supposed to be stable for the life of=
 the flow.=C2=A0 Otherwise, it isn't a

> >=C2=A0 =C2=A0 =C2=A0>> flow label.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> Yours,

> >=C2=A0 =C2=A0 =C2=A0>> Joel

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> On 11/25/2020 2:35 PM, Tom Herbert wr=
ote:

> >=C2=A0 =C2=A0 =C2=A0>>> Hi Fernando, comments in line...<=
br>
> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> On Wed, Nov 25, 2020 at 12:13 AM =
Fernando Gont <fernando@gont.com.ar <mailto:fernando@gont.com.ar>> wrote:

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> Hi, Tom,

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> On 24/11/20 16:43, Tom Herber=
t wrote:

> >=C2=A0 =C2=A0 =C2=A0>>>> [....]

> >=C2=A0 =C2=A0 =C2=A0>>>>> Modulating the flow label=
 is a means to affect the routing of packets

> >=C2=A0 =C2=A0 =C2=A0>>>>> through the network that =
uses flow labels as input to the ECMP hash.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> What's the point?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 1) You cannot tell *if* the F=
L is being used.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Generally true, but in a limited =
domain this information could be

> >=C2=A0 =C2=A0 =C2=A0>>> discerned. I'd note that it&#=
39;s also generally true that we don't know

> >=C2=A0 =C2=A0 =C2=A0>>> if there is a load balancer or st=
ateful firewall in the path that

> >=C2=A0 =C2=A0 =C2=A0>>> requires consistent routing, but =
in a limited domain we could know

> >=C2=A0 =C2=A0 =C2=A0>>> that also.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 2) Changing the FL does not n=
ecessarily mean that packets will employ a

> >=C2=A0 =C2=A0 =C2=A0>>>> different link.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> It's an opportunistic mechani=
sm. If a connection is failing and we get

> >=C2=A0 =C2=A0 =C2=A0>>> a better path that fixes it by si=
mply changing the flow label then

> >=C2=A0 =C2=A0 =C2=A0>>> what's the harm?

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 3) If the network is failing,=
 shouldn't you handle this via routing?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Sure, but then that requires an o=
ut of band feedback loop from a TCP

> >=C2=A0 =C2=A0 =C2=A0>>> implementation to the network inf=
rastructure to indicate there is a

> >=C2=A0 =C2=A0 =C2=A0>>> problem and then the network need=
s to respond. That's significant

> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure and higher reactio=
n time than doing something in TCP

> >=C2=A0 =C2=A0 =C2=A0>>> and IP. Think of modulating the f=
low label is an inexpensive form of

> >=C2=A0 =C2=A0 =C2=A0>>> source routing within a limited d=
omain that doesn't need any

> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure or heavyweight pro=
tocols or something like segment

> >=C2=A0 =C2=A0 =C2=A0>>> routing.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>> The basic idea is that th=
e flow label associated with a connection is

> >=C2=A0 =C2=A0 =C2=A0>>>>> randomly changed when the=
 stack observes that the connection is

> >=C2=A0 =C2=A0 =C2=A0>>>>> failing (e.g. and an RTO)=
. There is nothing in the specs that prevents

> >=C2=A0 =C2=A0 =C2=A0>>>>> this since the source is =
at liberty to set the flow label as it sees

> >=C2=A0 =C2=A0 =C2=A0>>>>> fit.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> The FL is expected to remain =
constant for the life of a flow. A

> >=C2=A0 =C2=A0 =C2=A0>>>> retransmitted packet is part =
of the same flow as the

> >=C2=A0 =C2=A0 =C2=A0>>>> originally-transmitted packet=
. So this seems to be contradicting the

> >=C2=A0 =C2=A0 =C2=A0>>>> very specification of the FL.=


> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> For instance, If a RTO for a =
flow causes the FL to change, then one may

> >=C2=A0 =C2=A0 =C2=A0>>>> possibly argue that the FL is=
 not naming/labeling what is said/expected

> >=C2=A0 =C2=A0 =C2=A0>>>> to be anming/labeling.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Specifically, RFC6437 states:

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> "It is therefore RECOMMENDED=
 that source hosts support the flow label

> >=C2=A0 =C2=A0 =C2=A0>>> by setting the flow label field f=
or all packets of a given flow to the

> >=C2=A0 =C2=A0 =C2=A0>>> same value chosen from an approxi=
mation to a discrete uniform

> >=C2=A0 =C2=A0 =C2=A0>>> distribution."

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> So that is clearly a just recomme=
ndation, and not a requirement (and

> >=C2=A0 =C2=A0 =C2=A0>>> definitely not a MUST). Furthermo=
re, RFC6437 states:

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> "A forwarding node MUST eith=
er leave a non-zero flow label value

> >=C2=A0 =C2=A0 =C2=A0>>> unchanged or change it only for c=
ompelling operational security

> >=C2=A0 =C2=A0 =C2=A0>>> reasons as described in Section 6=
.1."

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> So there's no guarantee in th=
e protocol specs that flow labels are

> >=C2=A0 =C2=A0 =C2=A0>>> consistent for the life of the co=
nnection, which means that the

> >=C2=A0 =C2=A0 =C2=A0>>> network cannot assume that and th=
us it would be incorrect if the

> >=C2=A0 =C2=A0 =C2=A0>>> network tried to enforce flow lab=
el consistency as a protocol

> >=C2=A0 =C2=A0 =C2=A0>>> requirement. As I said, it is pru=
dent to try to be consistent with

> >=C2=A0 =C2=A0 =C2=A0>>> flow labels and the default behav=
ior in Linux should be changed,

> >=C2=A0 =C2=A0 =C2=A0>>> however I do not believe there=
9;s a valid claim of non-conformance that

> >=C2=A0 =C2=A0 =C2=A0>>> motivates removal of the feature =
that is already deployed.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Tom

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>> The feature is useful in =
large datacenter networks, like

> >=C2=A0 =C2=A0 =C2=A0>>>>> pparently Facebook where =
the patches originate, since information

> >=C2=A0 =C2=A0 =C2=A0>>>>> discerned by TCP can oppo=
rtunistically be applied to route selection.

> >=C2=A0 =C2=A0 =C2=A0>>>>> The practical issue is th=
at there are stateful devices like firewalls

> >=C2=A0 =C2=A0 =C2=A0>>>>> that require consistent r=
outing in the network in which case changing

> >=C2=A0 =C2=A0 =C2=A0>>>>> the flow label can confus=
e them. As I mentioned, the original intent

> >=C2=A0 =C2=A0 =C2=A0>>>>> was that the flow label r=
andomization feature should be opt-in instead

> >=C2=A0 =C2=A0 =C2=A0>>>>> of on by default.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> So... where is the "sour=
ce" of the packet that would be "modulating" the FL?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> Thanks,

> >=C2=A0 =C2=A0 =C2=A0>>>> --

> >=C2=A0 =C2=A0 =C2=A0>>>> Fernando Gont

> >=C2=A0 =C2=A0 =C2=A0>>>> e-mail: fernando@gont.com.ar <mailto:fernando@gont.com.a=
r> || fgo=
nt@si6networks.com <mailto:fgont@si6networks.com>

> >=C2=A0 =C2=A0 =C2=A0>>>> PGP Fingerprint: 7809 84F5 32=
2E 45C7 F1C9 3945 96EE A9EF D076 FFF1

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> _________________________________=
______________

> >=C2=A0 =C2=A0 =C2=A0>>> v6ops mailing list

> >=C2=A0 =C2=A0 =C2=A0>>> v6ops@ietf.org <mailto:v6ops@ietf.org>

> >=C2=A0 =C2=A0 =C2=A0>>> https://www.iet=
f.org/mailman/listinfo/v6ops

> >=C2=A0 =C2=A0 =C2=A0>>>

> >

> >

> > _______________________________________________

> > v6ops mailing list

> > v6ops@ietf.or=
g

> > https://www.ietf.org/mailman/listinfo/v6ops=


> >

>



_______________________________________________

v6ops mailing list

v6ops@ietf.org

https://www.ietf.org/mailman/listinfo/v6ops




-- 
Best regards,
Alexander Azi=
mov


--0000000000005157af05b4ff3f8f--


From nobody Thu Nov 26 04:47:38 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2369D3A12ED for ; Thu, 26 Nov 2020 04:47:37 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2zi5X7OgWI9Z for ; Thu, 26 Nov 2020 04:47:35 -0800 (PST)
Received: from mail-out01.uio.no (mail-out01.uio.no [IPv6:2001:700:100:10::50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id DDAEB3A12EC for ; Thu, 26 Nov 2020 04:47:34 -0800 (PST)
Received: from mail-mx12.uio.no ([129.240.10.84]) by mail-out01.uio.no with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93.0.4) (envelope-from ) id 1kiGgO-0001cp-I2 for tcpm@ietf.org; Thu, 26 Nov 2020 13:47:32 +0100
Received: from boomerang.ifi.uio.no ([129.240.68.135]) by mail-mx12.uio.no with esmtpsa (TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256) user michawe (Exim 4.93.0.4) (envelope-from ) id 1kiGgN-000DKK-VZ for tcpm@ietf.org; Thu, 26 Nov 2020 13:47:32 +0100
From: Michael Welzl 
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Date: Thu, 26 Nov 2020 13:47:31 +0100
References: <160634544894.28725.8891473900084027588@ietfa.amsl.com>
To: tcpm IETF list 
In-Reply-To: <160634544894.28725.8891473900084027588@ietfa.amsl.com>
Message-Id: <7F478CFF-A60E-4513-AFCB-1DEE985F9880@ifi.uio.no>
X-Mailer: Apple Mail (2.3445.9.1)
X-UiO-SPF-Received: Received-SPF: neutral (mail-mx12.uio.no: 129.240.68.135 is neither permitted nor denied by domain of ifi.uio.no) client-ip=129.240.68.135; envelope-from=michawe@ifi.uio.no; helo=boomerang.ifi.uio.no; 
X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, AWL=0.019, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO)
X-UiO-Scanned: 943B8685A07B418AB99FB8B2845EAF0D710D029F
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-2140bis-06.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 12:47:37 -0000

Dear all,

This is our update to address the WGLC comments, where concrete =
suggestions were made regarding the appendix.

Cheers,
Michael


> On 26 Nov 2020, at 00:04, internet-drafts@ietf.org wrote:
>=20
>=20
> A New Internet-Draft is available from the on-line Internet-Drafts =
directories.
> This draft is a work item of the TCP Maintenance and Minor Extensions =
WG of the IETF.
>=20
>        Title           : TCP Control Block Interdependence
>        Authors         : Joe Touch
>                          Michael Welzl
>                          Safiqul Islam
> 	Filename        : draft-ietf-tcpm-2140bis-06.txt
> 	Pages           : 34
> 	Date            : 2020-11-25
>=20
> Abstract:
>   This memo provides guidance to TCP implementers that are intended to
>   help improve convergence to steady-state operation without affecting
>   interoperability. It updates and replaces RFC 2140's description of
>   interdependent TCP control blocks and the ways that part of TCP
>   state can be shared among similar concurrent or consecutive
>   connections. TCP state includes a combination of parameters, such as
>   connection state, current round-trip time estimates, congestion
>   control information, and process information. Most of this state is
>   maintained on a per-connection basis in the TCP Control Block (TCB),
>   but implementations can (and do) share certain TCB information
>   across connections to the same host. Such sharing is intended to
>   improve overall transient transport performance, while maintaining
>   backward-compatibility with existing implementations. The sharing
>   described herein is limited to only the TCB initialization and so
>   has no effect on the long-term behavior of TCP after a connection
>   has been established.
>=20
>=20
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-tcpm-2140bis/
>=20
> There are also htmlized versions available at:
> https://tools.ietf.org/html/draft-ietf-tcpm-2140bis-06
> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-2140bis-06
>=20
> A diff from the previous version is available at:
> https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-2140bis-06
>=20
>=20
> Please note that it may take a couple of minutes from the time of =
submission
> until the htmlized version and diff are available at tools.ietf.org.
>=20
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>=20
>=20
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Thu Nov 26 06:12:49 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AA8833A1192 for ; Thu, 26 Nov 2020 06:12:47 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=hs-esslingen.de
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id I0PV7qaeJHfi for ; Thu, 26 Nov 2020 06:12:45 -0800 (PST)
Received: from mail.hs-esslingen.de (mail.hs-esslingen.de [134.108.32.78]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7F3633A1190 for ; Thu, 26 Nov 2020 06:12:45 -0800 (PST)
Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.hs-esslingen.de (Postfix) with ESMTP id 58CAD25A23 for ; Thu, 26 Nov 2020 15:12:43 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hs-esslingen.de; s=mail; t=1606399963; bh=Zz3V1jP9YhYW6PmHA3PQ/QpIfoMdZC+il1ECIs0SX/w=; h=From:To:Subject:Date:References:In-Reply-To:From; b=sR2e0FYK/VkDgAXz0RAhU2a9U7g46hQceRWzCq3kw/OChdKcjJku650movN6oj4Zz 2A4Pd9FJkHudwg+UVj6YIto/zu0RQY1wW1g3Z2IPiI/TGKn2BAwtSBSo0m5C/kq0/p tyggRzuaEfn/lSG9TbGi6sW2waKnoeZq4v4UsBTk=
X-Virus-Scanned: by amavisd-new-2.7.1 (20120429) (Debian) at hs-esslingen.de
Received: from mail.hs-esslingen.de ([127.0.0.1]) by localhost (hs-esslingen.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BTV54y8DmGZ5 for ; Thu, 26 Nov 2020 15:12:39 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (rznt8202.hs-esslingen.de [134.108.48.165]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.hs-esslingen.de (Postfix) with ESMTPS for ; Thu, 26 Nov 2020 15:12:39 +0100 (CET)
Received: from rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) by rznt8202.rznt.rzdir.fht-esslingen.de (134.108.48.165) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1979.3; Thu, 26 Nov 2020 15:12:39 +0100
Received: from rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0]) by rznt8202.rznt.rzdir.fht-esslingen.de ([fe80::aca4:171a:3ee1:57e0%3]) with mapi id 15.01.1979.006; Thu, 26 Nov 2020 15:12:39 +0100
From: "Scharf, Michael" 
To: tcpm IETF list 
Thread-Topic: [tcpm] I-D Action: draft-ietf-tcpm-2140bis-06.txt
Thread-Index: AQHWw39gQ+AvYqKOHkSoKPNVa7NXuqnaTNmAgAAi5ZA=
Date: Thu, 26 Nov 2020 14:12:39 +0000
Message-ID: <94c8342fcda84ede98e0591867cd8933@hs-esslingen.de>
References: <160634544894.28725.8891473900084027588@ietfa.amsl.com> <7F478CFF-A60E-4513-AFCB-1DEE985F9880@ifi.uio.no>
In-Reply-To: <7F478CFF-A60E-4513-AFCB-1DEE985F9880@ifi.uio.no>
Accept-Language: de-DE, en-US
Content-Language: de-DE
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [134.108.140.248]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: 
Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-2140bis-06.txt
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 14:12:48 -0000

The appendix was the only remaining open issue I am aware of. If I there ar=
e no objections to -06, I will close the WGLC and prepare my shepherd write=
-up.

Thanks!

Michael


> -----Original Message-----
> From: tcpm  On Behalf Of Michael Welzl
> Sent: Thursday, November 26, 2020 1:48 PM
> To: tcpm IETF list 
> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-2140bis-06.txt
>=20
> Dear all,
>=20
> This is our update to address the WGLC comments, where concrete
> suggestions were made regarding the appendix.
>=20
> Cheers,
> Michael
>=20
>=20
> > On 26 Nov 2020, at 00:04, internet-drafts@ietf.org wrote:
> >
> >
> > A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> > This draft is a work item of the TCP Maintenance and Minor Extensions W=
G
> of the IETF.
> >
> >        Title           : TCP Control Block Interdependence
> >        Authors         : Joe Touch
> >                          Michael Welzl
> >                          Safiqul Islam
> > 	Filename        : draft-ietf-tcpm-2140bis-06.txt
> > 	Pages           : 34
> > 	Date            : 2020-11-25
> >
> > Abstract:
> >   This memo provides guidance to TCP implementers that are intended to
> >   help improve convergence to steady-state operation without affecting
> >   interoperability. It updates and replaces RFC 2140's description of
> >   interdependent TCP control blocks and the ways that part of TCP
> >   state can be shared among similar concurrent or consecutive
> >   connections. TCP state includes a combination of parameters, such as
> >   connection state, current round-trip time estimates, congestion
> >   control information, and process information. Most of this state is
> >   maintained on a per-connection basis in the TCP Control Block (TCB),
> >   but implementations can (and do) share certain TCB information
> >   across connections to the same host. Such sharing is intended to
> >   improve overall transient transport performance, while maintaining
> >   backward-compatibility with existing implementations. The sharing
> >   described herein is limited to only the TCB initialization and so
> >   has no effect on the long-term behavior of TCP after a connection
> >   has been established.
> >
> >
> > The IETF datatracker status page for this draft is:
> > https://datatracker.ietf.org/doc/draft-ietf-tcpm-2140bis/
> >
> > There are also htmlized versions available at:
> > https://tools.ietf.org/html/draft-ietf-tcpm-2140bis-06
> > https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-2140bis-06
> >
> > A diff from the previous version is available at:
> > https://www.ietf.org/rfcdiff?url2=3Ddraft-ietf-tcpm-2140bis-06
> >
> >
> > Please note that it may take a couple of minutes from the time of
> submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > Internet-Drafts are also available by anonymous FTP at:
> > ftp://ftp.ietf.org/internet-drafts/
> >
> >
> > _______________________________________________
> > tcpm mailing list
> > tcpm@ietf.org
> > https://www.ietf.org/mailman/listinfo/tcpm
>=20
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Thu Nov 26 08:39:41 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CBF8E3A14AA for ; Thu, 26 Nov 2020 08:39:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -15.6
X-Spam-Level: 
X-Spam-Status: No, score=-15.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W2xYufzYUr1S for ; Thu, 26 Nov 2020 08:39:37 -0800 (PST)
Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7EE563A14A6 for ; Thu, 26 Nov 2020 08:39:36 -0800 (PST)
Received: by mail-wm1-x332.google.com with SMTP id h21so3004625wmb.2 for ; Thu, 26 Nov 2020 08:39:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=edWfoAmVLZbYlmpbCJONOdlNJLI3bMwOAVJSH9ZaLCQ=; b=iUoi9IwLGLbG5Rpoemq2UQDon2zWNuLsqbnW3WmSzRecaosWeumRIntXHtuIjTBcsR duEZLqgZXuBCkGjGSD8BYjUjPYO87CU8nfbxKqzya4QP04zGCwq8y24OlhwQzOQh7dIL EYpnVfzIe7Jx2jh3f0HxJtU2WUKyDxJLJJxS0thyA94UfP9+FXCqvcZl4wJ/cYqI+2mO Rx02+uRRwYDbHzUMhU1ypuVKlagMuwJHIkGY/l37kRz3iM4XdtxS3NohJZnavzy1z3ub MYeDooXhpqTSmrYa/+FdOnruWCWJY95yiG9TLGyDOKrUbAXXLCk71ejAcjZ4VqSytsjn I+2w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=edWfoAmVLZbYlmpbCJONOdlNJLI3bMwOAVJSH9ZaLCQ=; b=PFh0vQf36RkpV6S7+aphFhBYYSS3fUGYFDNMHchggxtOX+b+fHpvsWLfAxuI5i2R3F rAdlqaxTXSxPOkBIi/hFHkXWTJ9Gc2gmJsNfW7GyR27YqxWK6pERX19cp3DuV+yVY7v3 YSKFo45zFhFofyLkfeaeSjiPgc37tTiQ+0oS9eidZqZTFh9cuNsZtcLucGaxLc+qSAmE mRUzC3x5ZvODQg+2yshsHIiQMZiT1wnqRb1uKk/2S365ELIz2lpLV1VH0SkDQRnwP+AT H52S/LGEGGgXOukzXVMFhDL6IOjPcy20JYDZ90YOvSaBFg58bnLv1WI9n8wCDchW06c4 lZCA==
X-Gm-Message-State: AOAM531RfekOBwNBZZBxQ9LydhRs+JWaJAoSk0yI9abxlLQx7RMOTNLd eecW59IajYhX73/VQJbKLU3j0SFcXUS+ap4hkm8JUQ==
X-Google-Smtp-Source: ABdhPJzqXxadN+VVfaKqKqKd6ElReYkMJOQ8O5ba/SnFB+r0DbboFc/wdwFUdnF2KFMXogyMhIABcSyHr2oNVDNz2sY=
X-Received: by 2002:a7b:cd11:: with SMTP id f17mr4215376wmj.127.1606408774663;  Thu, 26 Nov 2020 08:39:34 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>      
In-Reply-To: 
From: Yuchung Cheng 
Date: Thu, 26 Nov 2020 08:38:57 -0800
Message-ID: 
To: Alexander Azimov 
Cc: Tom Herbert , IPv6 Operations , tcpm , Brian E Carpenter 
Content-Type: multipart/alternative; boundary="00000000000061231205b5053001"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 16:39:40 -0000

--00000000000061231205b5053001
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Nov 26, 2020 at 1:34 AM Alexander Azimov 
wrote:

> Dear colleagues,
>
> We started discussing an incorrect default behavior, and Tom has already
> confirmed that it will be fixed.
>
> Later the thread turned into an argument if the flow label can be changed
> during connection/flow lifetime according to current RFC documents, thoug=
h
> these documents can be updated. This looks a bit weird for me because I
> always thought that it is the IETF community's responsibility to document
> proper solutions. If something undocumented but worthy happens in the
> industry - IETF should catch up. So, I would like to get back to the
> discussion of reasons and their safety.
>
> The general idea of changing the routing path upon network
> outage/degradation looks obvious. Getting kind of source-based routing ca=
n
> significantly reduce the reaction time and improve end-user experience. T=
he
> flow label has a perfect match here: transparent for the application, set
> by the source, not part of 5-tuple, while it can be used in the load
> balancing. IMO poorly documented. I would like to learn your feedback for
> the next wording:
>
> Let say that the flow label is a hash of values from the IP packet's
> 5-tuple and random number. Then
>
> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
> recalculated.
> 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.
> 3. Otherwise flow label SHOULD be preserved unchanged.
>

For completion, we might want to discuss reverse-path failure that
RTO-triggered rehash in (1)(2) alone can not repair the connection in
established or handshake phases.

https://patchwork.ozlabs.org/project/netdev/patch/20180829215356.235336-1-y=
cheng@google.com/

I also agree with Tom that the general feature is very useful in keeping
TCP connected with partial route failures, but the default should be an
opt-in.


>
> The first point stands for redirecting connection from the degraded path
> before the connection is established, and it looks safe. The second one c=
an
> also improve performance if it is set on the server-side. Please comment =
if
> you see security flaws in such a design.
>
> =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, Tom=
 Herbert :
>
>> On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
>>  wrote:
>> >
>> > I'm not Joel, but I did once spend some time grepping RFCs to find out
>> whether
>> > "flow" or "microflow" was the preferred term. In RFC2474, which is
>> normative,
>> > we have:
>> >
>> >    Microflow: a single instance of an application-to-application flow =
of
>> >    packets which is identified by source address, destination address,
>> >    protocol id, and source port, destination port (where applicable).
>> >
>> > But in the flow label work we explicitly avoided being that precise, a=
nd
>> > did not use the term "microflow". There might be some load balancing
>> > scenarios where you want a broader definition, even including
>> bidirectional
>> > flows. There are expired drafts on that topic:
>> > draft-tarreau-extend-flow-label-balancing
>> > draft-wang-6man-flow-label-reflection
>> >
>> Brian,
>>
>> Random Packet Spraying
>> (
>> https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.529&rep=
=3Drep1&type=3Dpdf
>> )
>> is an interesting idea where packets for a single connection are
>> purposely distributed across multiple paths for load distribution. Per
>> packet randomized flow labels with flow label aware ECMP makes this
>> quite easy to do without requiring any special support in switches
>> like you'd need with IPv4. I'm not necessarily advocating this, but it
>> does highlight one potential use case of having a flow label that
>> doesn;t have rigidly defined requirements on the host.
>>
>> Tom
>>
>> > Regards
>> >    Brian
>> >
>> > On 26-Nov-20 11:13, Tom Herbert wrote:
>> > > Joel, is there a normative definition of a flow?
>> > >
>> > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern > > wrote:
>> > >
>> > >     No, as Brian says, there are escape clauses in the flow
>> definitions.
>> > >
>> > >     But changing the flow label due to traffic problems does not
>> correspond
>> > >     to packets being in actually different flows.
>> > >     If one were using UDP, and mixing loss sensitive packets with lo=
ss
>> > >     insensitive packets for a special application, sure, one could
>> use two
>> > >     flow labels.  But that is not what you are describing.
>> > >
>> > >     Yours,
>> > >     Joel
>> > >
>> > >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
>> > >     > Joel,
>> > >     >
>> > >     > Is there an RFC that clearly and unambiguously states that a
>> host MUST
>> > >     > use the same flow label for the lifetime _and_ clearly defines
>> exactly
>> > >     > what a flow is with respect to such a requirement (for
>> instance, how
>> > >     > would you define a flow and enforce such a requirement in UDP?
>> IPsec?
>> > >     > other encapsulations?). If there is such a requirement then
>> we'll
>> > >     > change the code to be conformant.
>> > >     >
>> > >     > Tom
>> > >     >
>> > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <
>> jmh@joelhalpern.com > wrote:
>> > >     >>
>> > >     >> This kind of thing is why, as I understand it, MPTCP has
>> discovery
>> > >     >> mechanisms ot know if both sides use it, and can select
>> alternative
>> > >     >> addresses for communication.
>> > >     >>
>> > >     >> Trying to guess flow labels that might avoid a problem becaus=
e
>> it might
>> > >     >> be an ECMP problem, is just flailing about.  Not a good desig=
n
>> for
>> > >     >> operational protocols.
>> > >     >>
>> > >     >> And in general, designing protocols around "I know exactly
>> what is going
>> > >     >> on"  (the requirement for what you describe that goes well
>> beyond just
>> > >     >> "limited domains") is also a recipe for failure.
>> > >     >>
>> > >     >> The Flow Label RFCs are actually very explicit that a flow
>> label is
>> > >     >> supposed to be stable for the life of the flow.  Otherwise, i=
t
>> isn't a
>> > >     >> flow label.
>> > >     >>
>> > >     >> Yours,
>> > >     >> Joel
>> > >     >>
>> > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>> > >     >>> Hi Fernando, comments in line...
>> > >     >>>
>> > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <
>> fernando@gont.com.ar > wrote:
>> > >     >>>>
>> > >     >>>> Hi, Tom,
>> > >     >>>>
>> > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
>> > >     >>>> [....]
>> > >     >>>>> Modulating the flow label is a means to affect the routing
>> of packets
>> > >     >>>>> through the network that uses flow labels as input to the
>> ECMP hash.
>> > >     >>>>
>> > >     >>>> What's the point?
>> > >     >>>>
>> > >     >>>> 1) You cannot tell *if* the FL is being used.
>> > >     >>>>
>> > >     >>> Generally true, but in a limited domain this information
>> could be
>> > >     >>> discerned. I'd note that it's also generally true that we
>> don't know
>> > >     >>> if there is a load balancer or stateful firewall in the path
>> that
>> > >     >>> requires consistent routing, but in a limited domain we coul=
d
>> know
>> > >     >>> that also.
>> > >     >>>
>> > >     >>>> 2) Changing the FL does not necessarily mean that packets
>> will employ a
>> > >     >>>> different link.
>> > >     >>>
>> > >     >>> It's an opportunistic mechanism. If a connection is failing
>> and we get
>> > >     >>> a better path that fixes it by simply changing the flow labe=
l
>> then
>> > >     >>> what's the harm?
>> > >     >>>
>> > >     >>>>
>> > >     >>>> 3) If the network is failing, shouldn't you handle this via
>> routing?
>> > >     >>>>
>> > >     >>> Sure, but then that requires an out of band feedback loop
>> from a TCP
>> > >     >>> implementation to the network infrastructure to indicate
>> there is a
>> > >     >>> problem and then the network needs to respond. That's
>> significant
>> > >     >>> infrastructure and higher reaction time than doing something
>> in TCP
>> > >     >>> and IP. Think of modulating the flow label is an inexpensive
>> form of
>> > >     >>> source routing within a limited domain that doesn't need any
>> > >     >>> infrastructure or heavyweight protocols or something like
>> segment
>> > >     >>> routing.
>> > >     >>>
>> > >     >>>>
>> > >     >>>>
>> > >     >>>>> The basic idea is that the flow label associated with a
>> connection is
>> > >     >>>>> randomly changed when the stack observes that the
>> connection is
>> > >     >>>>> failing (e.g. and an RTO). There is nothing in the specs
>> that prevents
>> > >     >>>>> this since the source is at liberty to set the flow label
>> as it sees
>> > >     >>>>> fit.
>> > >     >>>>
>> > >     >>>> The FL is expected to remain constant for the life of a
>> flow. A
>> > >     >>>> retransmitted packet is part of the same flow as the
>> > >     >>>> originally-transmitted packet. So this seems to be
>> contradicting the
>> > >     >>>> very specification of the FL.
>> > >     >>>>
>> > >     >>>> For instance, If a RTO for a flow causes the FL to change,
>> then one may
>> > >     >>>> possibly argue that the FL is not naming/labeling what is
>> said/expected
>> > >     >>>> to be anming/labeling.
>> > >     >>>
>> > >     >>> Specifically, RFC6437 states:
>> > >     >>>
>> > >     >>> "It is therefore RECOMMENDED that source hosts support the
>> flow label
>> > >     >>> by setting the flow label field for all packets of a given
>> flow to the
>> > >     >>> same value chosen from an approximation to a discrete unifor=
m
>> > >     >>> distribution."
>> > >     >>>
>> > >     >>> So that is clearly a just recommendation, and not a
>> requirement (and
>> > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
>> > >     >>>
>> > >     >>> "A forwarding node MUST either leave a non-zero flow label
>> value
>> > >     >>> unchanged or change it only for compelling operational
>> security
>> > >     >>> reasons as described in Section 6.1."
>> > >     >>>
>> > >     >>> So there's no guarantee in the protocol specs that flow
>> labels are
>> > >     >>> consistent for the life of the connection, which means that
>> the
>> > >     >>> network cannot assume that and thus it would be incorrect if
>> the
>> > >     >>> network tried to enforce flow label consistency as a protoco=
l
>> > >     >>> requirement. As I said, it is prudent to try to be consisten=
t
>> with
>> > >     >>> flow labels and the default behavior in Linux should be
>> changed,
>> > >     >>> however I do not believe there's a valid claim of
>> non-conformance that
>> > >     >>> motivates removal of the feature that is already deployed.
>> > >     >>>
>> > >     >>> Tom
>> > >     >>>
>> > >     >>>
>> > >     >>>
>> > >     >>>
>> > >     >>>>
>> > >     >>>>
>> > >     >>>>
>> > >     >>>>> The feature is useful in large datacenter networks, like
>> > >     >>>>> pparently Facebook where the patches originate, since
>> information
>> > >     >>>>> discerned by TCP can opportunistically be applied to route
>> selection.
>> > >     >>>>> The practical issue is that there are stateful devices lik=
e
>> firewalls
>> > >     >>>>> that require consistent routing in the network in which
>> case changing
>> > >     >>>>> the flow label can confuse them. As I mentioned, the
>> original intent
>> > >     >>>>> was that the flow label randomization feature should be
>> opt-in instead
>> > >     >>>>> of on by default.
>> > >     >>>>
>> > >     >>>> So... where is the "source" of the packet that would be
>> "modulating" the FL?
>> > >     >>>>
>> > >     >>>> Thanks,
>> > >     >>>> --
>> > >     >>>> Fernando Gont
>> > >     >>>> e-mail: fernando@gont.com.ar 
>> || fgont@si6networks.com 
>> > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF
>> D076 FFF1
>> > >     >>>>
>> > >     >>>>
>> > >     >>>>
>> > >     >>>
>> > >     >>> _______________________________________________
>> > >     >>> v6ops mailing list
>> > >     >>> v6ops@ietf.org 
>> > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
>> > >     >>>
>> > >
>> > >
>> > > _______________________________________________
>> > > v6ops mailing list
>> > > v6ops@ietf.org
>> > > https://www.ietf.org/mailman/listinfo/v6ops
>> > >
>> >
>>
>> _______________________________________________
>> v6ops mailing list
>> v6ops@ietf.org
>> https://www.ietf.org/mailman/listinfo/v6ops
>>
>
>
> --
> Best regards,
> Alexander Azimov
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--00000000000061231205b5053001
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable





On Thu, Nov 26, 2020 at 1:34 AM Alexander Azimov <a.e.azimov@gmail.com> wro=
te:
Dear colleagues,

We started discussing an incorrect default beh=
avior, and Tom has already confirmed that it will be fixed.

Later th=
e thread turned into an argument if the flow label can be changed during co=
nnection/flow lifetime according to current RFC documents, though these doc=
uments can be updated. This looks a bit weird for me because I always thoug=
ht that it is the IETF community's responsibility to document proper so=
lutions. If something undocumented but worthy happens in the industry - IET=
F should catch up. So, I would like to get back to the discussion of reason=
s and their safety. 

The general idea of changing the routing path u=
pon network outage/degradation looks obvious. Getting kind of source-based =
routing can significantly reduce the reaction=20

 time and improve end-user experience. The flow label has a perfect match h=
ere: transparent for the application, set by the source, not part of 5-tupl=
e, while it can be used in the load balancing. IMO poorly documented. I wou=
ld like to learn your feedback for the next wording:

Let say that th=
e flow label is a hash of values from the IP packet's 5-tuple and rando=
m number. Then

1. In the case of SYN or SYN-ACK retransmission flow =
label SHOULD be recalculated.
2. In the case of RTO timeout expiration i=
n the established TCP session the flow label MAY be=20

recalculated. This setting MUST be switched off by default.
3. Otherwise=
 flow label SHOULD be preserved unchanged.
=C2=
=A0
For completion, we might want to discuss reverse-path failure=
 that RTO-triggered rehash in (1)(2) alone can not repair the connection in=
 established or handshake phases.

I also agree with Tom t=
hat the general feature is very useful in keeping TCP connected with partia=
l route failures, but the default should be an opt-in.
=C2=A0

Th=
e first point stands for redirecting connection from the degraded path befo=
re the connection is established, and it looks safe. The second one can als=
o improve performance if it is set on the server-side. Please comment if yo=
u see security flaws in such a design.

=D1=87=D1=82, 26 =D0=BD=D0=BE=D1=
=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, Tom Herbert <tom@herbertland.com>:
<=
blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l=
eft:1px solid rgb(204,204,204);padding-left:1ex">On Wed, Nov 25, 2020 at 3:=
33 PM Brian E Carpenter

<brian.=
e.carpenter@gmail.com> wrote:

>

> I'm not Joel, but I did once spend some time grepping RFCs to find=
 out whether

> "flow" or "microflow" was the preferred term. In R=
FC2474, which is normative,

> we have:

>

>=C2=A0 =C2=A0 Microflow: a single instance of an application-to-applica=
tion flow of

>=C2=A0 =C2=A0 packets which is identified by source address, destinatio=
n address,

>=C2=A0 =C2=A0 protocol id, and source port, destination port (where app=
licable).

>

> But in the flow label work we explicitly avoided being that precise, a=
nd

> did not use the term "microflow". There might be some load b=
alancing

> scenarios where you want a broader definition, even including bidirect=
ional

> flows. There are expired drafts on that topic:

> draft-tarreau-extend-flow-label-balancing

> draft-wang-6man-flow-label-reflection

>

Brian,



Random Packet Spraying

(ht=
tps://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.529&rep=
=3Drep1&type=3Dpdf)

is an interesting idea where packets for a single connection are

purposely distributed across multiple paths for load distribution. Per

packet randomized flow labels with flow label aware ECMP makes this

quite easy to do without requiring any special support in switches

like you'd need with IPv4. I'm not necessarily advocating this, but=
 it

does highlight one potential use case of having a flow label that

doesn;t have rigidly defined requirements on the host.



Tom



> Regards

>=C2=A0 =C2=A0 Brian

>

> On 26-Nov-20 11:13, Tom Herbert wrote:

> > Joel, is there a normative definition of a flow?

> >

> > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <jmh@joelhalpern.com <mailt=
o:jmh@joelhalpern.=
com>> wrote:

> >

> >=C2=A0 =C2=A0 =C2=A0No, as Brian says, there are escape clauses in=
 the flow definitions.

> >

> >=C2=A0 =C2=A0 =C2=A0But changing the flow label due to traffic pro=
blems does not correspond

> >=C2=A0 =C2=A0 =C2=A0to packets being in actually different flows.<=
br>
> >=C2=A0 =C2=A0 =C2=A0If one were using UDP, and mixing loss sensiti=
ve packets with loss

> >=C2=A0 =C2=A0 =C2=A0insensitive packets for a special application,=
 sure, one could use two

> >=C2=A0 =C2=A0 =C2=A0flow labels.=C2=A0 But that is not what you ar=
e describing.

> >

> >=C2=A0 =C2=A0 =C2=A0Yours,

> >=C2=A0 =C2=A0 =C2=A0Joel

> >

> >=C2=A0 =C2=A0 =C2=A0On 11/25/2020 3:05 PM, Tom Herbert wrote:

> >=C2=A0 =C2=A0 =C2=A0> Joel,

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> Is there an RFC that clearly and unambigu=
ously states that a host MUST

> >=C2=A0 =C2=A0 =C2=A0> use the same flow label for the lifetime =
_and_ clearly defines exactly

> >=C2=A0 =C2=A0 =C2=A0> what a flow is with respect to such a req=
uirement (for instance, how

> >=C2=A0 =C2=A0 =C2=A0> would you define a flow and enforce such =
a requirement in UDP? IPsec?

> >=C2=A0 =C2=A0 =C2=A0> other encapsulations?). If there is such =
a requirement then we'll

> >=C2=A0 =C2=A0 =C2=A0> change the code to be conformant.

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> Tom

> >=C2=A0 =C2=A0 =C2=A0>

> >=C2=A0 =C2=A0 =C2=A0> On Wed, Nov 25, 2020 at 12:39 PM Joel M. =
Halpern <jmh@jo=
elhalpern.com <mailto:jmh@joelhalpern.com>> wrote:

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> This kind of thing is why, as I under=
stand it, MPTCP has discovery

> >=C2=A0 =C2=A0 =C2=A0>> mechanisms ot know if both sides use =
it, and can select alternative

> >=C2=A0 =C2=A0 =C2=A0>> addresses for communication.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> Trying to guess flow labels that migh=
t avoid a problem because it might

> >=C2=A0 =C2=A0 =C2=A0>> be an ECMP problem, is just flailing =
about.=C2=A0 Not a good design for

> >=C2=A0 =C2=A0 =C2=A0>> operational protocols.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> And in general, designing protocols a=
round "I know exactly what is going

> >=C2=A0 =C2=A0 =C2=A0>> on"=C2=A0 (the requirement for w=
hat you describe that goes well beyond just

> >=C2=A0 =C2=A0 =C2=A0>> "limited domains") is also =
a recipe for failure.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> The Flow Label RFCs are actually very=
 explicit that a flow label is

> >=C2=A0 =C2=A0 =C2=A0>> supposed to be stable for the life of=
 the flow.=C2=A0 Otherwise, it isn't a

> >=C2=A0 =C2=A0 =C2=A0>> flow label.

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> Yours,

> >=C2=A0 =C2=A0 =C2=A0>> Joel

> >=C2=A0 =C2=A0 =C2=A0>>

> >=C2=A0 =C2=A0 =C2=A0>> On 11/25/2020 2:35 PM, Tom Herbert wr=
ote:

> >=C2=A0 =C2=A0 =C2=A0>>> Hi Fernando, comments in line...<=
br>
> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> On Wed, Nov 25, 2020 at 12:13 AM =
Fernando Gont <fernando@gont.com.ar <mailto:fernando@gont.com.ar>> wrote:

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> Hi, Tom,

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> On 24/11/20 16:43, Tom Herber=
t wrote:

> >=C2=A0 =C2=A0 =C2=A0>>>> [....]

> >=C2=A0 =C2=A0 =C2=A0>>>>> Modulating the flow label=
 is a means to affect the routing of packets

> >=C2=A0 =C2=A0 =C2=A0>>>>> through the network that =
uses flow labels as input to the ECMP hash.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> What's the point?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 1) You cannot tell *if* the F=
L is being used.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Generally true, but in a limited =
domain this information could be

> >=C2=A0 =C2=A0 =C2=A0>>> discerned. I'd note that it&#=
39;s also generally true that we don't know

> >=C2=A0 =C2=A0 =C2=A0>>> if there is a load balancer or st=
ateful firewall in the path that

> >=C2=A0 =C2=A0 =C2=A0>>> requires consistent routing, but =
in a limited domain we could know

> >=C2=A0 =C2=A0 =C2=A0>>> that also.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 2) Changing the FL does not n=
ecessarily mean that packets will employ a

> >=C2=A0 =C2=A0 =C2=A0>>>> different link.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> It's an opportunistic mechani=
sm. If a connection is failing and we get

> >=C2=A0 =C2=A0 =C2=A0>>> a better path that fixes it by si=
mply changing the flow label then

> >=C2=A0 =C2=A0 =C2=A0>>> what's the harm?

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> 3) If the network is failing,=
 shouldn't you handle this via routing?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Sure, but then that requires an o=
ut of band feedback loop from a TCP

> >=C2=A0 =C2=A0 =C2=A0>>> implementation to the network inf=
rastructure to indicate there is a

> >=C2=A0 =C2=A0 =C2=A0>>> problem and then the network need=
s to respond. That's significant

> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure and higher reactio=
n time than doing something in TCP

> >=C2=A0 =C2=A0 =C2=A0>>> and IP. Think of modulating the f=
low label is an inexpensive form of

> >=C2=A0 =C2=A0 =C2=A0>>> source routing within a limited d=
omain that doesn't need any

> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure or heavyweight pro=
tocols or something like segment

> >=C2=A0 =C2=A0 =C2=A0>>> routing.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>> The basic idea is that th=
e flow label associated with a connection is

> >=C2=A0 =C2=A0 =C2=A0>>>>> randomly changed when the=
 stack observes that the connection is

> >=C2=A0 =C2=A0 =C2=A0>>>>> failing (e.g. and an RTO)=
. There is nothing in the specs that prevents

> >=C2=A0 =C2=A0 =C2=A0>>>>> this since the source is =
at liberty to set the flow label as it sees

> >=C2=A0 =C2=A0 =C2=A0>>>>> fit.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> The FL is expected to remain =
constant for the life of a flow. A

> >=C2=A0 =C2=A0 =C2=A0>>>> retransmitted packet is part =
of the same flow as the

> >=C2=A0 =C2=A0 =C2=A0>>>> originally-transmitted packet=
. So this seems to be contradicting the

> >=C2=A0 =C2=A0 =C2=A0>>>> very specification of the FL.=


> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> For instance, If a RTO for a =
flow causes the FL to change, then one may

> >=C2=A0 =C2=A0 =C2=A0>>>> possibly argue that the FL is=
 not naming/labeling what is said/expected

> >=C2=A0 =C2=A0 =C2=A0>>>> to be anming/labeling.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Specifically, RFC6437 states:

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> "It is therefore RECOMMENDED=
 that source hosts support the flow label

> >=C2=A0 =C2=A0 =C2=A0>>> by setting the flow label field f=
or all packets of a given flow to the

> >=C2=A0 =C2=A0 =C2=A0>>> same value chosen from an approxi=
mation to a discrete uniform

> >=C2=A0 =C2=A0 =C2=A0>>> distribution."

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> So that is clearly a just recomme=
ndation, and not a requirement (and

> >=C2=A0 =C2=A0 =C2=A0>>> definitely not a MUST). Furthermo=
re, RFC6437 states:

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> "A forwarding node MUST eith=
er leave a non-zero flow label value

> >=C2=A0 =C2=A0 =C2=A0>>> unchanged or change it only for c=
ompelling operational security

> >=C2=A0 =C2=A0 =C2=A0>>> reasons as described in Section 6=
.1."

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> So there's no guarantee in th=
e protocol specs that flow labels are

> >=C2=A0 =C2=A0 =C2=A0>>> consistent for the life of the co=
nnection, which means that the

> >=C2=A0 =C2=A0 =C2=A0>>> network cannot assume that and th=
us it would be incorrect if the

> >=C2=A0 =C2=A0 =C2=A0>>> network tried to enforce flow lab=
el consistency as a protocol

> >=C2=A0 =C2=A0 =C2=A0>>> requirement. As I said, it is pru=
dent to try to be consistent with

> >=C2=A0 =C2=A0 =C2=A0>>> flow labels and the default behav=
ior in Linux should be changed,

> >=C2=A0 =C2=A0 =C2=A0>>> however I do not believe there=
9;s a valid claim of non-conformance that

> >=C2=A0 =C2=A0 =C2=A0>>> motivates removal of the feature =
that is already deployed.

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> Tom

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>> The feature is useful in =
large datacenter networks, like

> >=C2=A0 =C2=A0 =C2=A0>>>>> pparently Facebook where =
the patches originate, since information

> >=C2=A0 =C2=A0 =C2=A0>>>>> discerned by TCP can oppo=
rtunistically be applied to route selection.

> >=C2=A0 =C2=A0 =C2=A0>>>>> The practical issue is th=
at there are stateful devices like firewalls

> >=C2=A0 =C2=A0 =C2=A0>>>>> that require consistent r=
outing in the network in which case changing

> >=C2=A0 =C2=A0 =C2=A0>>>>> the flow label can confus=
e them. As I mentioned, the original intent

> >=C2=A0 =C2=A0 =C2=A0>>>>> was that the flow label r=
andomization feature should be opt-in instead

> >=C2=A0 =C2=A0 =C2=A0>>>>> of on by default.

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> So... where is the "sour=
ce" of the packet that would be "modulating" the FL?

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>> Thanks,

> >=C2=A0 =C2=A0 =C2=A0>>>> --

> >=C2=A0 =C2=A0 =C2=A0>>>> Fernando Gont

> >=C2=A0 =C2=A0 =C2=A0>>>> e-mail: fernando@gont.com.ar <mailto:fernando@gont.com.a=
r> || fgo=
nt@si6networks.com <mailto:fgont@si6networks.com>

> >=C2=A0 =C2=A0 =C2=A0>>>> PGP Fingerprint: 7809 84F5 32=
2E 45C7 F1C9 3945 96EE A9EF D076 FFF1

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>>

> >=C2=A0 =C2=A0 =C2=A0>>>

> >=C2=A0 =C2=A0 =C2=A0>>> _________________________________=
______________

> >=C2=A0 =C2=A0 =C2=A0>>> v6ops mailing list

> >=C2=A0 =C2=A0 =C2=A0>>> v6ops@ietf.org <mailto:v6ops@ietf.org>

> >=C2=A0 =C2=A0 =C2=A0>>> https://www.iet=
f.org/mailman/listinfo/v6ops

> >=C2=A0 =C2=A0 =C2=A0>>>

> >

> >

> > _______________________________________________

> > v6ops mailing list

> > v6ops@ietf.or=
g

> > https://www.ietf.org/mailman/listinfo/v6ops=


> >

>



_______________________________________________

v6ops mailing list

v6ops@ietf.org

https://www.ietf.org/mailman/listinfo/v6ops




-- 
Best regards,
Alexander Azimov

_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm






--00000000000061231205b5053001--


From nobody Thu Nov 26 11:53:07 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7C5EB3A0937; Thu, 26 Nov 2020 11:53:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.1
X-Spam-Level: 
X-Spam-Status: No, score=-0.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pmK6Qy6hjEjq; Thu, 26 Nov 2020 11:53:04 -0800 (PST)
Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 49A1D3A0934; Thu, 26 Nov 2020 11:53:04 -0800 (PST)
Received: by mail-pf1-x434.google.com with SMTP id b6so2470864pfp.7; Thu, 26 Nov 2020 11:53:04 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=7KsfzhLZojDWOSaUcpbCg7Z+oYHfH1TZPipK+ib5TZM=; b=RaOv69KgSn1labbz2LL1KLRiZfhWImU+CRWRCPqrUeOFF3cv0YAAOHKTcXvxsZklB0 kO1OBlVQbmng/WfPs4shnTlF2Ss9ljQ6W7fBJMc0yWN3+FD9ff8V7fGcAiqnvfYCVVsH agn33WxBkruzdMJ2QRAoUvCxWiXxJQfwOvZ7XTvKbwC3/dxb6osqTfqOdupnG8Posnux GzJhHhMvpr72KSgq+r4KL012Gkk7SLMpRWl8sDfklTgXjKzmpHFBubLf2j52rL7Go2w2 RqSAznNV6KS0Bb8tesKkGtQw9mFLHLvUx4mPrEhigpYSaUvNzO74hZSDkdcULUaDPRu3 8rmA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=7KsfzhLZojDWOSaUcpbCg7Z+oYHfH1TZPipK+ib5TZM=; b=lDe3ZZ3QQCk2nCQbBQueFaZksZocp1w+oplnL4eKzSDiOJt/rTdeIT5+DTomwmpGfG rSSSUz5yo+rL9zCQevEG47nwFdmb1SY+5yIhYAmqOAxkTKjDcHGXcreCXAQPAETG7xs6 53UEOpWOJuk0zCVp+3SMrrSUWPsSYgUM5IEPUDc0V98jyJY5UdxSQK6RuIyN8udCNk+t PH6VMNTR8ZFZBwop+czaretmULtUqvYhr6fVBhsSmun/faYif86SrNkOkfQ50StBiLYk Ry39cep7wcWqCV+s/iIZvbQPNs7BEmDFHRHFWhnywTlMXudluUv68i9Wod9xhhgoFegw 150Q==
X-Gm-Message-State: AOAM531QnKffRni0PQEiSQkszeUOAmxdqOhRt+HeTDXW/41Umu7qDZNw 30d0iL3EpmSsMnlhLY2fpHF5WYJ0RSbO4g==
X-Google-Smtp-Source: ABdhPJxwQkKaIB2EbIoW1srp6ULXjykpHUb+Gs9uEpFKxhU/589BqhoF+njIlywiiMSpPCHfEOVojw==
X-Received: by 2002:a63:5941:: with SMTP id j1mr3728838pgm.59.1606420383173; Thu, 26 Nov 2020 11:53:03 -0800 (PST)
Received: from [192.168.178.20] ([151.210.131.28]) by smtp.gmail.com with ESMTPSA id m5sm7300617pjl.19.2020.11.26.11.53.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Nov 2020 11:53:02 -0800 (PST)
To: Alexander Azimov , Tom Herbert 
Cc: tcpm , IPv6 Operations 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>      
From: Brian E Carpenter 
Message-ID: 
Date: Fri, 27 Nov 2020 08:52:57 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 19:53:07 -0000

Alexander,

> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be re=
calculated.
=2E..
> The first point stands for redirecting connection from the degraded pat=
h before the connection is established, and it looks safe.

I am convinced that it is often unsafe, because that depends on exactly h=
ow a server farm load balancer handles such cases. If it maintains the TC=
P session on the same actual server if there is SYN(-ACK) retransmission,=
 the flow label must not be changed.
The benefit for possible broken routing is quite hypothetical, whereas th=
e risk to
session persistence at the server is real. (Remember that the source cann=
ot know by magic that there is server load balancing at the far end, but =
it's the common case today.)

> 2. In the case of RTO timeout expiration in the established TCP session=
 the flow label MAY be recalculated. This setting MUST be switched off by=
 default.

As noted before, this is only safe in a limited domain where it is known =
with certainty that server load balancing is not in use. I think it's the=
refore a corner case and specifically, the appropriate normative statemen=
t would be SHOULD NOT, which is defined very carefully in RFC2119 exactly=
 for this sort of case.

But in fact I think we should not attempt to legislate on this point. The=
 value is marginal at best. As others have said, this is not the way to t=
ackle operational routing problems.

Regards
   Brian Carpenter

On 26-Nov-20 22:34, Alexander Azimov wrote:
> Dear colleagues,
>=20
> We started discussing an incorrect default behavior, and Tom has alread=
y confirmed that it will be fixed.
>=20
> Later the thread turned into an argument if the flow label can be chang=
ed during connection/flow lifetime according to current RFC documents, th=
ough these documents can be updated. This looks a bit weird for me becaus=
e I always thought that it is the IETF community's responsibility to docu=
ment proper solutions. If something undocumented but worthy happens in th=
e industry - IETF should catch up. So, I would like to get back to the di=
scussion of reasons and their safety.
>=20
> The general idea of changing the routing path upon network outage/degra=
dation looks obvious. Getting kind of source-based routing can significan=
tly reduce the reaction time and improve end-user experience. The flow la=
bel has a perfect match here: transparent for the application, set by the=
 source, not part of 5-tuple, while it can be used in the load balancing.=
 IMO poorly documented. I would like to learn your feedback for the next =
wording:
>=20
> Let say that the flow label is a hash of values from the IP packet's 5-=
tuple and random number. Then
>=20
> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be re=
calculated.
> 2. In the case of RTO timeout expiration in the established TCP session=
 the flow label MAY be recalculated. This setting MUST be switched off by=
 default.
> 3. Otherwise flow label SHOULD be preserved unchanged.
>=20
> The first point stands for redirecting connection from the degraded pat=
h before the connection is established, and it looks safe. The second one=
 can also improve performance if it is set on the server-side. Please com=
ment if you see security flaws in such a design.
>=20
> =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, T=
om Herbert >:
>=20
>     On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
>     > =
wrote:
>     >
>     > I'm not Joel, but I did once spend some time grepping RFCs to fin=
d out whether
>     > "flow" or "microflow" was the preferred term. In RFC2474, which i=
s normative,
>     > we have:
>     >
>     >=C2=A0 =C2=A0 Microflow: a single instance of an application-to-ap=
plication flow of
>     >=C2=A0 =C2=A0 packets which is identified by source address, desti=
nation address,
>     >=C2=A0 =C2=A0 protocol id, and source port, destination port (wher=
e applicable).
>     >
>     > But in the flow label work we explicitly avoided being that preci=
se, and
>     > did not use the term "microflow". There might be some load balanc=
ing
>     > scenarios where you want a broader definition, even including bid=
irectional
>     > flows. There are expired drafts on that topic:
>     > draft-tarreau-extend-flow-label-balancing
>     > draft-wang-6man-flow-label-reflection
>     >
>     Brian,
>=20
>     Random Packet Spraying
>     (https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.52=
9&rep=3Drep1&type=3Dpdf)
>     is an interesting idea where packets for a single connection are
>     purposely distributed across multiple paths for load distribution. =
Per
>     packet randomized flow labels with flow label aware ECMP makes this=

>     quite easy to do without requiring any special support in switches
>     like you'd need with IPv4. I'm not necessarily advocating this, but=
 it
>     does highlight one potential use case of having a flow label that
>     doesn;t have rigidly defined requirements on the host.
>=20
>     Tom
>=20
>     > Regards
>     >=C2=A0 =C2=A0 Brian
>     >
>     > On 26-Nov-20 11:13, Tom Herbert wrote:
>     > > Joel, is there a normative definition of a flow?
>     > >
>     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern  >> wrote:
>     > >
>     > >=C2=A0 =C2=A0 =C2=A0No, as Brian says, there are escape clauses =
in the flow definitions.
>     > >
>     > >=C2=A0 =C2=A0 =C2=A0But changing the flow label due to traffic p=
roblems does not correspond
>     > >=C2=A0 =C2=A0 =C2=A0to packets being in actually different flows=
=2E
>     > >=C2=A0 =C2=A0 =C2=A0If one were using UDP, and mixing loss sensi=
tive packets with loss
>     > >=C2=A0 =C2=A0 =C2=A0insensitive packets for a special applicatio=
n, sure, one could use two
>     > >=C2=A0 =C2=A0 =C2=A0flow labels.=C2=A0 But that is not what you =
are describing.
>     > >
>     > >=C2=A0 =C2=A0 =C2=A0Yours,
>     > >=C2=A0 =C2=A0 =C2=A0Joel
>     > >
>     > >=C2=A0 =C2=A0 =C2=A0On 11/25/2020 3:05 PM, Tom Herbert wrote:
>     > >=C2=A0 =C2=A0 =C2=A0> Joel,
>     > >=C2=A0 =C2=A0 =C2=A0>
>     > >=C2=A0 =C2=A0 =C2=A0> Is there an RFC that clearly and unambiguo=
usly states that a host MUST
>     > >=C2=A0 =C2=A0 =C2=A0> use the same flow label for the lifetime _=
and_ clearly defines exactly
>     > >=C2=A0 =C2=A0 =C2=A0> what a flow is with respect to such a requ=
irement (for instance, how
>     > >=C2=A0 =C2=A0 =C2=A0> would you define a flow and enforce such a=
 requirement in UDP? IPsec?
>     > >=C2=A0 =C2=A0 =C2=A0> other encapsulations?). If there is such a=
 requirement then we'll
>     > >=C2=A0 =C2=A0 =C2=A0> change the code to be conformant.
>     > >=C2=A0 =C2=A0 =C2=A0>
>     > >=C2=A0 =C2=A0 =C2=A0> Tom
>     > >=C2=A0 =C2=A0 =C2=A0>
>     > >=C2=A0 =C2=A0 =C2=A0> On Wed, Nov 25, 2020 at 12:39 PM Joel M. H=
alpern  >> wrote:
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> This kind of thing is why, as I understan=
d it, MPTCP has discovery
>     > >=C2=A0 =C2=A0 =C2=A0>> mechanisms ot know if both sides use it, =
and can select alternative
>     > >=C2=A0 =C2=A0 =C2=A0>> addresses for communication.
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> Trying to guess flow labels that might av=
oid a problem because it might
>     > >=C2=A0 =C2=A0 =C2=A0>> be an ECMP problem, is just flailing abou=
t.=C2=A0 Not a good design for
>     > >=C2=A0 =C2=A0 =C2=A0>> operational protocols.
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> And in general, designing protocols aroun=
d "I know exactly what is going
>     > >=C2=A0 =C2=A0 =C2=A0>> on"=C2=A0 (the requirement for what you d=
escribe that goes well beyond just
>     > >=C2=A0 =C2=A0 =C2=A0>> "limited domains") is also a recipe for f=
ailure.
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> The Flow Label RFCs are actually very exp=
licit that a flow label is
>     > >=C2=A0 =C2=A0 =C2=A0>> supposed to be stable for the life of the=
 flow.=C2=A0 Otherwise, it isn't a
>     > >=C2=A0 =C2=A0 =C2=A0>> flow label.
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> Yours,
>     > >=C2=A0 =C2=A0 =C2=A0>> Joel
>     > >=C2=A0 =C2=A0 =C2=A0>>
>     > >=C2=A0 =C2=A0 =C2=A0>> On 11/25/2020 2:35 PM, Tom Herbert wrote:=

>     > >=C2=A0 =C2=A0 =C2=A0>>> Hi Fernando, comments in line...
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> On Wed, Nov 25, 2020 at 12:13 AM Fernand=
o Gont  >> wrote:
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> Hi, Tom,
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> On 24/11/20 16:43, Tom Herbert wrote:
>     > >=C2=A0 =C2=A0 =C2=A0>>>> [....]
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> Modulating the flow label is a means t=
o affect the routing of packets
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> through the network that uses flow lab=
els as input to the ECMP hash.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> What's the point?
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> 1) You cannot tell *if* the FL is being=
 used.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> Generally true, but in a limited domain =
this information could be
>     > >=C2=A0 =C2=A0 =C2=A0>>> discerned. I'd note that it's also gener=
ally true that we don't know
>     > >=C2=A0 =C2=A0 =C2=A0>>> if there is a load balancer or stateful =
firewall in the path that
>     > >=C2=A0 =C2=A0 =C2=A0>>> requires consistent routing, but in a li=
mited domain we could know
>     > >=C2=A0 =C2=A0 =C2=A0>>> that also.
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> 2) Changing the FL does not necessarily=
 mean that packets will employ a
>     > >=C2=A0 =C2=A0 =C2=A0>>>> different link.
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> It's an opportunistic mechanism. If a co=
nnection is failing and we get
>     > >=C2=A0 =C2=A0 =C2=A0>>> a better path that fixes it by simply ch=
anging the flow label then
>     > >=C2=A0 =C2=A0 =C2=A0>>> what's the harm?
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> 3) If the network is failing, shouldn't=
 you handle this via routing?
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> Sure, but then that requires an out of b=
and feedback loop from a TCP
>     > >=C2=A0 =C2=A0 =C2=A0>>> implementation to the network infrastruc=
ture to indicate there is a
>     > >=C2=A0 =C2=A0 =C2=A0>>> problem and then the network needs to re=
spond. That's significant
>     > >=C2=A0 =C2=A0 =C2=A0>>> infrastructure and higher reaction time =
than doing something in TCP
>     > >=C2=A0 =C2=A0 =C2=A0>>> and IP. Think of modulating the flow lab=
el is an inexpensive form of
>     > >=C2=A0 =C2=A0 =C2=A0>>> source routing within a limited domain t=
hat doesn't need any
>     > >=C2=A0 =C2=A0 =C2=A0>>> infrastructure or heavyweight protocols =
or something like segment
>     > >=C2=A0 =C2=A0 =C2=A0>>> routing.
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> The basic idea is that the flow label =
associated with a connection is
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> randomly changed when the stack observ=
es that the connection is
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> failing (e.g. and an RTO). There is no=
thing in the specs that prevents
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> this since the source is at liberty to=
 set the flow label as it sees
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> fit.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> The FL is expected to remain constant f=
or the life of a flow. A
>     > >=C2=A0 =C2=A0 =C2=A0>>>> retransmitted packet is part of the sam=
e flow as the
>     > >=C2=A0 =C2=A0 =C2=A0>>>> originally-transmitted packet. So this =
seems to be contradicting the
>     > >=C2=A0 =C2=A0 =C2=A0>>>> very specification of the FL.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> For instance, If a RTO for a flow cause=
s the FL to change, then one may
>     > >=C2=A0 =C2=A0 =C2=A0>>>> possibly argue that the FL is not namin=
g/labeling what is said/expected
>     > >=C2=A0 =C2=A0 =C2=A0>>>> to be anming/labeling.
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> Specifically, RFC6437 states:
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> "It is therefore RECOMMENDED that source=
 hosts support the flow label
>     > >=C2=A0 =C2=A0 =C2=A0>>> by setting the flow label field for all =
packets of a given flow to the
>     > >=C2=A0 =C2=A0 =C2=A0>>> same value chosen from an approximation =
to a discrete uniform
>     > >=C2=A0 =C2=A0 =C2=A0>>> distribution."
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> So that is clearly a just recommendation=
, and not a requirement (and
>     > >=C2=A0 =C2=A0 =C2=A0>>> definitely not a MUST). Furthermore, RFC=
6437 states:
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> "A forwarding node MUST either leave a n=
on-zero flow label value
>     > >=C2=A0 =C2=A0 =C2=A0>>> unchanged or change it only for compelli=
ng operational security
>     > >=C2=A0 =C2=A0 =C2=A0>>> reasons as described in Section 6.1."
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> So there's no guarantee in the protocol =
specs that flow labels are
>     > >=C2=A0 =C2=A0 =C2=A0>>> consistent for the life of the connectio=
n, which means that the
>     > >=C2=A0 =C2=A0 =C2=A0>>> network cannot assume that and thus it w=
ould be incorrect if the
>     > >=C2=A0 =C2=A0 =C2=A0>>> network tried to enforce flow label cons=
istency as a protocol
>     > >=C2=A0 =C2=A0 =C2=A0>>> requirement. As I said, it is prudent to=
 try to be consistent with
>     > >=C2=A0 =C2=A0 =C2=A0>>> flow labels and the default behavior in =
Linux should be changed,
>     > >=C2=A0 =C2=A0 =C2=A0>>> however I do not believe there's a valid=
 claim of non-conformance that
>     > >=C2=A0 =C2=A0 =C2=A0>>> motivates removal of the feature that is=
 already deployed.
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> Tom
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> The feature is useful in large datacen=
ter networks, like
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> pparently Facebook where the patches o=
riginate, since information
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> discerned by TCP can opportunistically=
 be applied to route selection.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> The practical issue is that there are =
stateful devices like firewalls
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> that require consistent routing in the=
 network in which case changing
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> the flow label can confuse them. As I =
mentioned, the original intent
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> was that the flow label randomization =
feature should be opt-in instead
>     > >=C2=A0 =C2=A0 =C2=A0>>>>> of on by default.
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> So... where is the "source" of the pack=
et that would be "modulating" the FL?
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>> Thanks,
>     > >=C2=A0 =C2=A0 =C2=A0>>>> --
>     > >=C2=A0 =C2=A0 =C2=A0>>>> Fernando Gont
>     > >=C2=A0 =C2=A0 =C2=A0>>>> e-mail: fernando@gont.com.ar  > || fgont@si6networks.com  >
>     > >=C2=A0 =C2=A0 =C2=A0>>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1=
C9 3945 96EE A9EF D076 FFF1
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >=C2=A0 =C2=A0 =C2=A0>>> ________________________________________=
_______
>     > >=C2=A0 =C2=A0 =C2=A0>>> v6ops mailing list
>     > >=C2=A0 =C2=A0 =C2=A0>>> v6ops@ietf.org  <=
mailto:v6ops@ietf.org >
>     > >=C2=A0 =C2=A0 =C2=A0>>> https://www.ietf.org/mailman/listinfo/v6=
ops
>     > >=C2=A0 =C2=A0 =C2=A0>>>
>     > >
>     > >
>     > > _______________________________________________
>     > > v6ops mailing list
>     > > v6ops@ietf.org 
>     > > https://www.ietf.org/mailman/listinfo/v6ops
>     > >
>     >
>=20
>     _______________________________________________
>     v6ops mailing list
>     v6ops@ietf.org 
>     https://www.ietf.org/mailman/listinfo/v6ops
>=20
>=20
>=20
> --=20
> Best regards,
> Alexander Azimov


From nobody Thu Nov 26 12:36:51 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 47C533A0D64; Thu, 26 Nov 2020 12:36:49 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.098
X-Spam-Level: 
X-Spam-Status: No, score=-0.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 68bCO_P2Ao9g; Thu, 26 Nov 2020 12:36:45 -0800 (PST)
Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 643133A0B2D; Thu, 26 Nov 2020 12:36:45 -0800 (PST)
Received: by mail-ot1-x32e.google.com with SMTP id h19so2879095otr.1; Thu, 26 Nov 2020 12:36:45 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sWGdLHcAR8etsExYzs670N+p1KTnMTLkY8Z+fzFhauE=; b=sQA9zaqveiAc4OxYR98OuVpk1txhAnk3+zlVDj2mSxjmQYnCipnPGm0AqYI851Omt1 EfDjXNkSh4QqDkjPBqrRu1/cQUwbqfn/SaDJgWPMt4CtgzJ+4R0kff5Stu+UUpTKcsBA fvmC2uh7pEV2GAODULkbkXP65uZ8FaaTmnir86+V4oV+LBsvD6KMgF+BFPFkeMITjNkF 1/zEkHw32/XNs99dt0DAzKTMp9cmiGM4jGay9ErlToLQYN0VUmTEs+p94xJ8PLTZXYM+ zSgidtPAbEzEyZEwZO5XZFohQZCYkphJ5MPjkSffZvl0+q78pfToGoqIzsvEPwwOTjrA OxpA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sWGdLHcAR8etsExYzs670N+p1KTnMTLkY8Z+fzFhauE=; b=YgvqhuxZSmqLpTpvGeYjqNwk3MS0Jf0cctFuWw7KHwoAg52Gb6Vkqih6misnufUSgN je4iN0avqaAa6sGqKBZD6Ebzq5sq81EqO0Ry95OSbxAHXw2JgD6Dj7caU1aPn3SHDyVd OayxKcz05nmHnoRhw0B55FOOKdIhWhwBtZ1H8KXqkdFhyGwyE9X9JpgAM//HzuTqemf4 425z5y4wDFLRPflqss40JrV/gnSbxbRgSrw6Ga8FwPCZCMLmmRjS5VdgSw+qV7VssfBv Q+gyOrwzCS/NzWqkwSlTzQjK3hQ0z/ZrD8l7qmY+ASj1d8jE87L0GiYEtggaypnfi/CO IqeA==
X-Gm-Message-State: AOAM532wRoVW7X8+/UpVCvQt/y9aTLTLrRwgVrQW7thZRRXWUJuf7iwj W+dO4vd779h/dcplmJL5KqDP658r90NlfBUr/m0=
X-Google-Smtp-Source: ABdhPJworRl9dvv3KZByEiXhc2KOE102EhSQIm9Ij/0foxU77PRcrf1eX4x8RY3kN9BN+ucdqBmL6YLOf1ondo9xUI8=
X-Received: by 2002:a9d:4c92:: with SMTP id m18mr3590603otf.248.1606423004656;  Thu, 26 Nov 2020 12:36:44 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>       
In-Reply-To: 
From: Alexander Azimov 
Date: Thu, 26 Nov 2020 23:36:33 +0300
Message-ID: 
To: Brian E Carpenter 
Cc: Tom Herbert , tcpm , IPv6 Operations 
Content-Type: multipart/alternative; boundary="0000000000008d395f05b5088044"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Thu, 26 Nov 2020 20:36:49 -0000

--0000000000008d395f05b5088044
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Brian,

May I ask you to give more details on this issue:

I am convinced that it is often unsafe, because that depends on exactly how
> a server farm load balancer handles such cases. If it maintains the TCP
> session on the same actual server if there is SYN(-ACK) retransmission, t=
he
> flow label must not be changed.

If a retransmitted SYN goes to another server/anycast point this won't
bring much harm. At least we haven't experienced such problems in our
datacenters.

=D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:53, Brian=
 E Carpenter :

> Alexander,
>
> > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
> recalculated.
> ...
> > The first point stands for redirecting connection from the degraded pat=
h
> before the connection is established, and it looks safe.
>
> I am convinced that it is often unsafe, because that depends on exactly
> how a server farm load balancer handles such cases. If it maintains the T=
CP
> session on the same actual server if there is SYN(-ACK) retransmission, t=
he
> flow label must not be changed.
> The benefit for possible broken routing is quite hypothetical, whereas th=
e
> risk to
> session persistence at the server is real. (Remember that the source
> cannot know by magic that there is server load balancing at the far end,
> but it's the common case today.)
>
> > 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.
>
> As noted before, this is only safe in a limited domain where it is known
> with certainty that server load balancing is not in use. I think it's
> therefore a corner case and specifically, the appropriate normative
> statement would be SHOULD NOT, which is defined very carefully in RFC2119
> exactly for this sort of case.
>
> But in fact I think we should not attempt to legislate on this point. The
> value is marginal at best. As others have said, this is not the way to
> tackle operational routing problems.
>
> Regards
>    Brian Carpenter
>
> On 26-Nov-20 22:34, Alexander Azimov wrote:
> > Dear colleagues,
> >
> > We started discussing an incorrect default behavior, and Tom has alread=
y
> confirmed that it will be fixed.
> >
> > Later the thread turned into an argument if the flow label can be
> changed during connection/flow lifetime according to current RFC document=
s,
> though these documents can be updated. This looks a bit weird for me
> because I always thought that it is the IETF community's responsibility t=
o
> document proper solutions. If something undocumented but worthy happens i=
n
> the industry - IETF should catch up. So, I would like to get back to the
> discussion of reasons and their safety.
> >
> > The general idea of changing the routing path upon network
> outage/degradation looks obvious. Getting kind of source-based routing ca=
n
> significantly reduce the reaction time and improve end-user experience. T=
he
> flow label has a perfect match here: transparent for the application, set
> by the source, not part of 5-tuple, while it can be used in the load
> balancing. IMO poorly documented. I would like to learn your feedback for
> the next wording:
> >
> > Let say that the flow label is a hash of values from the IP packet's
> 5-tuple and random number. Then
> >
> > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
> recalculated.
> > 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.
> > 3. Otherwise flow label SHOULD be preserved unchanged.
> >
> > The first point stands for redirecting connection from the degraded pat=
h
> before the connection is established, and it looks safe. The second one c=
an
> also improve performance if it is set on the server-side. Please comment =
if
> you see security flaws in such a design.
> >
> > =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, T=
om Herbert  tom@herbertland.com>>:
> >
> >     On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
> >     >
> wrote:
> >     >
> >     > I'm not Joel, but I did once spend some time grepping RFCs to fin=
d
> out whether
> >     > "flow" or "microflow" was the preferred term. In RFC2474, which i=
s
> normative,
> >     > we have:
> >     >
> >     >    Microflow: a single instance of an application-to-application
> flow of
> >     >    packets which is identified by source address, destination
> address,
> >     >    protocol id, and source port, destination port (where
> applicable).
> >     >
> >     > But in the flow label work we explicitly avoided being that
> precise, and
> >     > did not use the term "microflow". There might be some load
> balancing
> >     > scenarios where you want a broader definition, even including
> bidirectional
> >     > flows. There are expired drafts on that topic:
> >     > draft-tarreau-extend-flow-label-balancing
> >     > draft-wang-6man-flow-label-reflection
> >     >
> >     Brian,
> >
> >     Random Packet Spraying
> >     (
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D10.1.1.297.529&rep=
=3Drep1&type=3Dpdf
> )
> >     is an interesting idea where packets for a single connection are
> >     purposely distributed across multiple paths for load distribution.
> Per
> >     packet randomized flow labels with flow label aware ECMP makes this
> >     quite easy to do without requiring any special support in switches
> >     like you'd need with IPv4. I'm not necessarily advocating this, but
> it
> >     does highlight one potential use case of having a flow label that
> >     doesn;t have rigidly defined requirements on the host.
> >
> >     Tom
> >
> >     > Regards
> >     >    Brian
> >     >
> >     > On 26-Nov-20 11:13, Tom Herbert wrote:
> >     > > Joel, is there a normative definition of a flow?
> >     > >
> >     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <
> jmh@joelhalpern.com   jmh@joelhalpern.com >> wrote:
> >     > >
> >     > >     No, as Brian says, there are escape clauses in the flow
> definitions.
> >     > >
> >     > >     But changing the flow label due to traffic problems does no=
t
> correspond
> >     > >     to packets being in actually different flows.
> >     > >     If one were using UDP, and mixing loss sensitive packets
> with loss
> >     > >     insensitive packets for a special application, sure, one
> could use two
> >     > >     flow labels.  But that is not what you are describing.
> >     > >
> >     > >     Yours,
> >     > >     Joel
> >     > >
> >     > >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
> >     > >     > Joel,
> >     > >     >
> >     > >     > Is there an RFC that clearly and unambiguously states tha=
t
> a host MUST
> >     > >     > use the same flow label for the lifetime _and_ clearly
> defines exactly
> >     > >     > what a flow is with respect to such a requirement (for
> instance, how
> >     > >     > would you define a flow and enforce such a requirement in
> UDP? IPsec?
> >     > >     > other encapsulations?). If there is such a requirement
> then we'll
> >     > >     > change the code to be conformant.
> >     > >     >
> >     > >     > Tom
> >     > >     >
> >     > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <
> jmh@joelhalpern.com   jmh@joelhalpern.com >> wrote:
> >     > >     >>
> >     > >     >> This kind of thing is why, as I understand it, MPTCP has
> discovery
> >     > >     >> mechanisms ot know if both sides use it, and can select
> alternative
> >     > >     >> addresses for communication.
> >     > >     >>
> >     > >     >> Trying to guess flow labels that might avoid a problem
> because it might
> >     > >     >> be an ECMP problem, is just flailing about.  Not a good
> design for
> >     > >     >> operational protocols.
> >     > >     >>
> >     > >     >> And in general, designing protocols around "I know
> exactly what is going
> >     > >     >> on"  (the requirement for what you describe that goes
> well beyond just
> >     > >     >> "limited domains") is also a recipe for failure.
> >     > >     >>
> >     > >     >> The Flow Label RFCs are actually very explicit that a
> flow label is
> >     > >     >> supposed to be stable for the life of the flow.
> Otherwise, it isn't a
> >     > >     >> flow label.
> >     > >     >>
> >     > >     >> Yours,
> >     > >     >> Joel
> >     > >     >>
> >     > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> >     > >     >>> Hi Fernando, comments in line...
> >     > >     >>>
> >     > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <
> fernando@gont.com.ar   fernando@gont.com.ar >> wrote:
> >     > >     >>>>
> >     > >     >>>> Hi, Tom,
> >     > >     >>>>
> >     > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
> >     > >     >>>> [....]
> >     > >     >>>>> Modulating the flow label is a means to affect the
> routing of packets
> >     > >     >>>>> through the network that uses flow labels as input to
> the ECMP hash.
> >     > >     >>>>
> >     > >     >>>> What's the point?
> >     > >     >>>>
> >     > >     >>>> 1) You cannot tell *if* the FL is being used.
> >     > >     >>>>
> >     > >     >>> Generally true, but in a limited domain this informatio=
n
> could be
> >     > >     >>> discerned. I'd note that it's also generally true that
> we don't know
> >     > >     >>> if there is a load balancer or stateful firewall in the
> path that
> >     > >     >>> requires consistent routing, but in a limited domain we
> could know
> >     > >     >>> that also.
> >     > >     >>>
> >     > >     >>>> 2) Changing the FL does not necessarily mean that
> packets will employ a
> >     > >     >>>> different link.
> >     > >     >>>
> >     > >     >>> It's an opportunistic mechanism. If a connection is
> failing and we get
> >     > >     >>> a better path that fixes it by simply changing the flow
> label then
> >     > >     >>> what's the harm?
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>> 3) If the network is failing, shouldn't you handle thi=
s
> via routing?
> >     > >     >>>>
> >     > >     >>> Sure, but then that requires an out of band feedback
> loop from a TCP
> >     > >     >>> implementation to the network infrastructure to indicat=
e
> there is a
> >     > >     >>> problem and then the network needs to respond. That's
> significant
> >     > >     >>> infrastructure and higher reaction time than doing
> something in TCP
> >     > >     >>> and IP. Think of modulating the flow label is an
> inexpensive form of
> >     > >     >>> source routing within a limited domain that doesn't nee=
d
> any
> >     > >     >>> infrastructure or heavyweight protocols or something
> like segment
> >     > >     >>> routing.
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>> The basic idea is that the flow label associated with
> a connection is
> >     > >     >>>>> randomly changed when the stack observes that the
> connection is
> >     > >     >>>>> failing (e.g. and an RTO). There is nothing in the
> specs that prevents
> >     > >     >>>>> this since the source is at liberty to set the flow
> label as it sees
> >     > >     >>>>> fit.
> >     > >     >>>>
> >     > >     >>>> The FL is expected to remain constant for the life of =
a
> flow. A
> >     > >     >>>> retransmitted packet is part of the same flow as the
> >     > >     >>>> originally-transmitted packet. So this seems to be
> contradicting the
> >     > >     >>>> very specification of the FL.
> >     > >     >>>>
> >     > >     >>>> For instance, If a RTO for a flow causes the FL to
> change, then one may
> >     > >     >>>> possibly argue that the FL is not naming/labeling what
> is said/expected
> >     > >     >>>> to be anming/labeling.
> >     > >     >>>
> >     > >     >>> Specifically, RFC6437 states:
> >     > >     >>>
> >     > >     >>> "It is therefore RECOMMENDED that source hosts support
> the flow label
> >     > >     >>> by setting the flow label field for all packets of a
> given flow to the
> >     > >     >>> same value chosen from an approximation to a discrete
> uniform
> >     > >     >>> distribution."
> >     > >     >>>
> >     > >     >>> So that is clearly a just recommendation, and not a
> requirement (and
> >     > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
> >     > >     >>>
> >     > >     >>> "A forwarding node MUST either leave a non-zero flow
> label value
> >     > >     >>> unchanged or change it only for compelling operational
> security
> >     > >     >>> reasons as described in Section 6.1."
> >     > >     >>>
> >     > >     >>> So there's no guarantee in the protocol specs that flow
> labels are
> >     > >     >>> consistent for the life of the connection, which means
> that the
> >     > >     >>> network cannot assume that and thus it would be
> incorrect if the
> >     > >     >>> network tried to enforce flow label consistency as a
> protocol
> >     > >     >>> requirement. As I said, it is prudent to try to be
> consistent with
> >     > >     >>> flow labels and the default behavior in Linux should be
> changed,
> >     > >     >>> however I do not believe there's a valid claim of
> non-conformance that
> >     > >     >>> motivates removal of the feature that is already
> deployed.
> >     > >     >>>
> >     > >     >>> Tom
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>> The feature is useful in large datacenter networks,
> like
> >     > >     >>>>> pparently Facebook where the patches originate, since
> information
> >     > >     >>>>> discerned by TCP can opportunistically be applied to
> route selection.
> >     > >     >>>>> The practical issue is that there are stateful device=
s
> like firewalls
> >     > >     >>>>> that require consistent routing in the network in
> which case changing
> >     > >     >>>>> the flow label can confuse them. As I mentioned, the
> original intent
> >     > >     >>>>> was that the flow label randomization feature should
> be opt-in instead
> >     > >     >>>>> of on by default.
> >     > >     >>>>
> >     > >     >>>> So... where is the "source" of the packet that would b=
e
> "modulating" the FL?
> >     > >     >>>>
> >     > >     >>>> Thanks,
> >     > >     >>>> --
> >     > >     >>>> Fernando Gont
> >     > >     >>>> e-mail: fernando@gont.com.ar  fernando@gont.com.ar>  fernando@gont.com.ar>> || fgont@si6networks.com  fgont@si6networks.com>  fgont@si6networks.com>>
> >     > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE
> A9EF D076 FFF1
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>
> >     > >     >>> _______________________________________________
> >     > >     >>> v6ops mailing list
> >     > >     >>> v6ops@ietf.org   v6ops@ietf.org >
> >     > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
> >     > >     >>>
> >     > >
> >     > >
> >     > > _______________________________________________
> >     > > v6ops mailing list
> >     > > v6ops@ietf.org 
> >     > > https://www.ietf.org/mailman/listinfo/v6ops
> >     > >
> >     >
> >
> >     _______________________________________________
> >     v6ops mailing list
> >     v6ops@ietf.org 
> >     https://www.ietf.org/mailman/listinfo/v6ops
> >
> >
> >
> > --
> > Best regards,
> > Alexander Azimov
>
>

--=20
Best regards,
Alexander Azimov

--0000000000008d395f05b5088044
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Brian,

May I ask you to give more detai=
ls on this issue:

I am convinced that it is often unsafe, because that depends o=
n exactly how a server farm load balancer handles such cases. If it maintai=
ns the TCP session on the same actual server if there is SYN(-ACK) retransm=
ission, the flow label must not be changed.=C2=A0=C2=A0
If=
 a retransmitted SYN goes to another server/anycast point this won't br=
ing much harm. At least we haven't experienced such problems in our dat=
acenters.

=D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =
=D0=B2 22:53, Brian E Carpenter <brian.e.carpenter@gmail.com>:
Alexander,



> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be r=
ecalculated.

...

> The first point stands for redirecting connection from the degraded pa=
th before the connection is established, and it looks safe.



I am convinced that it is often unsafe, because that depends on exactly how=
 a server farm load balancer handles such cases. If it maintains the TCP se=
ssion on the same actual server if there is SYN(-ACK) retransmission, the f=
low label must not be changed.

The benefit for possible broken routing is quite hypothetical, whereas the =
risk to

session persistence at the server is real. (Remember that the source cannot=
 know by magic that there is server load balancing at the far end, but it&#=
39;s the common case today.)



> 2. In the case of RTO timeout expiration in the established TCP sessio=
n the flow label MAY be recalculated. This setting MUST be switched off by =
default.



As noted before, this is only safe in a limited domain where it is known wi=
th certainty that server load balancing is not in use. I think it's the=
refore a corner case and specifically, the appropriate normative statement =
would be SHOULD NOT, which is defined very carefully in RFC2119 exactly for=
 this sort of case.



But in fact I think we should not attempt to legislate on this point. The v=
alue is marginal at best. As others have said, this is not the way to tackl=
e operational routing problems.



Regards

=C2=A0 =C2=A0Brian Carpenter



On 26-Nov-20 22:34, Alexander Azimov wrote:

> Dear colleagues,

> 

> We started discussing an incorrect default behavior, and Tom has alrea=
dy confirmed that it will be fixed.

> 

> Later the thread turned into an argument if the flow label can be chan=
ged during connection/flow lifetime according to current RFC documents, tho=
ugh these documents can be updated. This looks a bit weird for me because I=
 always thought that it is the IETF community's responsibility to docum=
ent proper solutions. If something undocumented but worthy happens in the i=
ndustry - IETF should catch up. So, I would like to get back to the discuss=
ion of reasons and their safety.

> 

> The general idea of changing the routing path upon network outage/degr=
adation looks obvious. Getting kind of source-based routing can significant=
ly reduce the reaction time and improve end-user experience. The flow label=
 has a perfect match here: transparent for the application, set by the sour=
ce, not part of 5-tuple, while it can be used in the load balancing. IMO po=
orly documented. I would like to learn your feedback for the next wording:<=
br>
> 

> Let say that the flow label is a hash of values from the IP packet'=
;s 5-tuple and random number. Then

> 

> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be r=
ecalculated.

> 2. In the case of RTO timeout expiration in the established TCP sessio=
n the flow label MAY be recalculated. This setting MUST be switched off by =
default.

> 3. Otherwise flow label SHOULD be preserved unchanged.

> 

> The first point stands for redirecting connection from the degraded pa=
th before the connection is established, and it looks safe. The second one =
can also improve performance if it is set on the server-side. Please commen=
t if you see security flaws in such a design.

> 

> =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03:16, =
Tom Herbert <to=
m@herbertland.com <mailto:tom@herbertland.com>>:

> 

>=C2=A0 =C2=A0 =C2=A0On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
>=C2=A0 =C2=A0 =C2=A0<brian.e.carpenter@gmail.com <mailto:brian.e.carpenter@gmail.=
com>> wrote:

>=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> I'm not Joel, but I did once spend some ti=
me grepping RFCs to find out whether

>=C2=A0 =C2=A0 =C2=A0> "flow" or "microflow" was =
the preferred term. In RFC2474, which is normative,

>=C2=A0 =C2=A0 =C2=A0> we have:

>=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 Microflow: a single instance of a=
n application-to-application flow of

>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 packets which is identified by so=
urce address, destination address,

>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 protocol id, and source port, des=
tination port (where applicable).

>=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> But in the flow label work we explicitly avoid=
ed being that precise, and

>=C2=A0 =C2=A0 =C2=A0> did not use the term "microflow". Th=
ere might be some load balancing

>=C2=A0 =C2=A0 =C2=A0> scenarios where you want a broader definition,=
 even including bidirectional

>=C2=A0 =C2=A0 =C2=A0> flows. There are expired drafts on that topic:=


>=C2=A0 =C2=A0 =C2=A0> draft-tarreau-extend-flow-label-balancing

>=C2=A0 =C2=A0 =C2=A0> draft-wang-6man-flow-label-reflection

>=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0Brian,

> 

>=C2=A0 =C2=A0 =C2=A0Random Packet Spraying

>=C2=A0 =C2=A0 =C2=A0(https://citeseerx.ist.psu.edu/viewdoc/download?doi=3D=
10.1.1.297.529&rep=3Drep1&type=3Dpdf)

>=C2=A0 =C2=A0 =C2=A0is an interesting idea where packets for a single c=
onnection are

>=C2=A0 =C2=A0 =C2=A0purposely distributed across multiple paths for loa=
d distribution. Per

>=C2=A0 =C2=A0 =C2=A0packet randomized flow labels with flow label aware=
 ECMP makes this

>=C2=A0 =C2=A0 =C2=A0quite easy to do without requiring any special supp=
ort in switches

>=C2=A0 =C2=A0 =C2=A0like you'd need with IPv4. I'm not necessar=
ily advocating this, but it

>=C2=A0 =C2=A0 =C2=A0does highlight one potential use case of having a f=
low label that

>=C2=A0 =C2=A0 =C2=A0doesn;t have rigidly defined requirements on the ho=
st.

> 

>=C2=A0 =C2=A0 =C2=A0Tom

> 

>=C2=A0 =C2=A0 =C2=A0> Regards

>=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 Brian

>=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> On 26-Nov-20 11:13, Tom Herbert wrote:

>=C2=A0 =C2=A0 =C2=A0> > Joel, is there a normative definition of =
a flow?

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> > On Wed, Nov 25, 2020, 1:27 PM Joel M. Hal=
pern <jmh@joelh=
alpern.com <mailto:jmh@joelhalpern.com> <mailto:jmh@joelhalpern.com <mailto:jmh@joelhalpern.com>&g=
t;> wrote:

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0No, as Brian says, the=
re are escape clauses in the flow definitions.

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0But changing the flow =
label due to traffic problems does not correspond

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0to packets being in ac=
tually different flows.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0If one were using UDP,=
 and mixing loss sensitive packets with loss

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0insensitive packets fo=
r a special application, sure, one could use two

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0flow labels.=C2=A0 But=
 that is not what you are describing.

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0Yours,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0Joel

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0On 11/25/2020 3:05 PM,=
 Tom Herbert wrote:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Joel,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Is there an RFC t=
hat clearly and unambiguously states that a host MUST

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> use the same flow=
 label for the lifetime _and_ clearly defines exactly

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> what a flow is wi=
th respect to such a requirement (for instance, how

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> would you define =
a flow and enforce such a requirement in UDP? IPsec?

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> other encapsulati=
ons?). If there is such a requirement then we'll

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> change the code t=
o be conformant.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Tom

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> On Wed, Nov 25, 2=
020 at 12:39 PM Joel M. Halpern <jmh@joelhalpern.com <mailto:jmh@joelhalpern.com> <mailto:<=
a href=3D"mailto:jmh@joelhalpern.com" target=3D"_blank">jmh@joelhalpern.com=
 <mailto:jm=
h@joelhalpern.com>>> wrote:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> This kind of =
thing is why, as I understand it, MPTCP has discovery

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> mechanisms ot=
 know if both sides use it, and can select alternative

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> addresses for=
 communication.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Trying to gue=
ss flow labels that might avoid a problem because it might

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> be an ECMP pr=
oblem, is just flailing about.=C2=A0 Not a good design for

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> operational p=
rotocols.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> And in genera=
l, designing protocols around "I know exactly what is going

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> on"=C2=
=A0 (the requirement for what you describe that goes well beyond just

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> "limited=
 domains") is also a recipe for failure.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> The Flow Labe=
l RFCs are actually very explicit that a flow label is

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> supposed to b=
e stable for the life of the flow.=C2=A0 Otherwise, it isn't a

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> flow label.
>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Yours,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Joel

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> On 11/25/2020=
 2:35 PM, Tom Herbert wrote:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Hi Fernan=
do, comments in line...

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> On Wed, N=
ov 25, 2020 at 12:13 AM Fernando Gont <fernando@gont.com.ar <mailto:fernando@gont.com.ar> &l=
t;mailto:fernando=
@gont.com.ar <mailto:fernando@gont.com.ar>>> wrote:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Hi, T=
om,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> On 24=
/11/20 16:43, Tom Herbert wrote:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> [....=
]

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> M=
odulating the flow label is a means to affect the routing of packets

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> t=
hrough the network that uses flow labels as input to the ECMP hash.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> What&=
#39;s the point?

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 1) Yo=
u cannot tell *if* the FL is being used.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Generally=
 true, but in a limited domain this information could be

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> discerned=
. I'd note that it's also generally true that we don't know

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> if there =
is a load balancer or stateful firewall in the path that

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> requires =
consistent routing, but in a limited domain we could know

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> that also=
.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 2) Ch=
anging the FL does not necessarily mean that packets will employ a

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> diffe=
rent link.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> It's =
an opportunistic mechanism. If a connection is failing and we get

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> a better =
path that fixes it by simply changing the flow label then

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> what'=
s the harm?

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 3) If=
 the network is failing, shouldn't you handle this via routing?

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Sure, but=
 then that requires an out of band feedback loop from a TCP

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> implement=
ation to the network infrastructure to indicate there is a

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> problem a=
nd then the network needs to respond. That's significant

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> infrastru=
cture and higher reaction time than doing something in TCP

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> and IP. T=
hink of modulating the flow label is an inexpensive form of

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> source ro=
uting within a limited domain that doesn't need any

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> infrastru=
cture or heavyweight protocols or something like segment

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> routing.<=
br>
>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> T=
he basic idea is that the flow label associated with a connection is

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> r=
andomly changed when the stack observes that the connection is

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> f=
ailing (e.g. and an RTO). There is nothing in the specs that prevents

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> t=
his since the source is at liberty to set the flow label as it sees

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> f=
it.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> The F=
L is expected to remain constant for the life of a flow. A

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> retra=
nsmitted packet is part of the same flow as the

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> origi=
nally-transmitted packet. So this seems to be contradicting the

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> very =
specification of the FL.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> For i=
nstance, If a RTO for a flow causes the FL to change, then one may

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> possi=
bly argue that the FL is not naming/labeling what is said/expected

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> to be=
 anming/labeling.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Specifica=
lly, RFC6437 states:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> "It =
is therefore RECOMMENDED that source hosts support the flow label

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> by settin=
g the flow label field for all packets of a given flow to the

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> same valu=
e chosen from an approximation to a discrete uniform

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> distribut=
ion."

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> So that i=
s clearly a just recommendation, and not a requirement (and

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> definitel=
y not a MUST). Furthermore, RFC6437 states:

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> "A f=
orwarding node MUST either leave a non-zero flow label value

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> unchanged=
 or change it only for compelling operational security

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> reasons a=
s described in Section 6.1."

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> So there&=
#39;s no guarantee in the protocol specs that flow labels are

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> consisten=
t for the life of the connection, which means that the

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> network c=
annot assume that and thus it would be incorrect if the

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> network t=
ried to enforce flow label consistency as a protocol

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> requireme=
nt. As I said, it is prudent to try to be consistent with

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> flow labe=
ls and the default behavior in Linux should be changed,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> however I=
 do not believe there's a valid claim of non-conformance that

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> motivates=
 removal of the feature that is already deployed.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Tom

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> T=
he feature is useful in large datacenter networks, like

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> p=
parently Facebook where the patches originate, since information

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> d=
iscerned by TCP can opportunistically be applied to route selection.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> T=
he practical issue is that there are stateful devices like firewalls

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> t=
hat require consistent routing in the network in which case changing

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> t=
he flow label can confuse them. As I mentioned, the original intent

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> w=
as that the flow label randomization feature should be opt-in instead

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> o=
f on by default.

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> So...=
 where is the "source" of the packet that would be "modulati=
ng" the FL?

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Thank=
s,

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> --
>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Ferna=
ndo Gont

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> e-mai=
l: fernando@gont.=
com.ar <mailto:fernando@gont.com.ar> <mailto:fernando@gont.com.ar <mailto:fernando@gont.com.ar>=
> || fgont@si=
6networks.com <mailto:fgont@si6networks.com> <mailto:fgont@si6networks.com <mailto:=
fgont@si6network=
s.com>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> PGP F=
ingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> _________=
______________________________________

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> v6ops mai=
ling list

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> v6ops@ietf.org <mailto:=
v6ops@ietf.org> =
<mailto:v6ops@ietf.o=
rg <mailto:v6ops=
@ietf.org>>

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> https://www.ietf.org/mailman/listinfo/v6ops

>=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0> > _________________________________________=
______

>=C2=A0 =C2=A0 =C2=A0> > v6ops mailing list

>=C2=A0 =C2=A0 =C2=A0> > v6ops@ietf.org <mailto:v6ops@ietf.org>

>=C2=A0 =C2=A0 =C2=A0> > https://www.ietf.org/ma=
ilman/listinfo/v6ops

>=C2=A0 =C2=A0 =C2=A0> >

>=C2=A0 =C2=A0 =C2=A0>

> 

>=C2=A0 =C2=A0 =C2=A0_______________________________________________

>=C2=A0 =C2=A0 =C2=A0v6ops mailing list

>=C2=A0 =C2=A0 =C2=A0v6ops@ietf.org <mailto:v6ops@ietf.org>

>=C2=A0 =C2=A0 =C2=A0https://www.ietf.org/mailman/list=
info/v6ops

> 

> 

> 

> -- 

> Best regards,

> Alexander Azimov






-- 
Best regards,
Alexander Azi=
mov


--0000000000008d395f05b5088044--


From nobody Thu Nov 26 17:50:53 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1A25C3A0F0C; Thu, 26 Nov 2020 17:50:48 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.1
X-Spam-Level: 
X-Spam-Status: No, score=-0.1 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FvzQ5EVJphLQ; Thu, 26 Nov 2020 17:50:43 -0800 (PST)
Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E40ED3A0F09; Thu, 26 Nov 2020 17:50:42 -0800 (PST)
Received: by mail-pf1-x435.google.com with SMTP id b6so3087853pfp.7; Thu, 26 Nov 2020 17:50:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=VRIc/uAp7ODSAwelFB8KJGjxlq6At3N43SFSJFknG7s=; b=fBTtyvxlK/Yd1K/UX9PIgGLxqEdD3RWH1FR2/nZWQb5FKw1JZxqSLJ7DiaVkzcpPUA hTpEdQnxJKjQsoG4Zd8CXiTFApnljmtOJrIJaGBPn5GTQyxREjIEwdHgVqSXNOSEirDD NfwEYNYDkILbB+E/Mydw2G4QnojQ4Zoo0pZzBfBaO4ZUVRQf+MsbD+Hqa29i9jmxxg6o Jhqy4iUxc3t/GEPiYbQcmRmgXl5NqSotjwOHF9bzYSRj4JLcE7o1YfdWe3PrJyD8wnZt 3e9pOvxA5zOyM1x9tW/1idxWrXuXIwpFgy3pv5vWtOM9kyO/awLul0dGJSSB1sMFJrsD jhhQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=VRIc/uAp7ODSAwelFB8KJGjxlq6At3N43SFSJFknG7s=; b=qpUA08okS+QMLl3De9cTob3t7SdgVUJxH9SYc+n/TM9qu//0gdyJ7xizNoM1E1ofsg m4YOvtdgDHQPzfVUxarE99r7Zv9rkauvaJ4RG4UdMaZwVENtbDtBx2rhll+c70py05xr +9OF/wXSb3yeKhSmF9Jc842fLvTRX+JegR38j1WTGe7RQhza2HNjWrCA85SYgyOESlHl lvGcDgDFiIB8ZaQ622YKWu4Lu0ov9TBrKYnJnObgFcf2EQwme6xBkhoRKFMA4GZUr7+o MwmyTzplAhiVGhdbmCGOmz2uXAnoBaTfcpcsPuZQMsR0+ylqQyHmZb8mzpY/xPx5/uo2 95VQ==
X-Gm-Message-State: AOAM533CRUe7fxAojsd/+gVxFZjeCd7kHRn+dPlzq9Y9RYV/2u4a3+RX OmWOt8dWVOtyBec3y9QjqgVzga/0Q/A2ng==
X-Google-Smtp-Source: ABdhPJxTTM27aqAL2+fOI9Tt3SW/Gl4n1SeqmljlXH+oG2jetqMGVE/sM16CjqIsyWuDrZS3qZqVyQ==
X-Received: by 2002:a63:211:: with SMTP id 17mr4688968pgc.56.1606441841817; Thu, 26 Nov 2020 17:50:41 -0800 (PST)
Received: from [192.168.178.20] ([151.210.131.28]) by smtp.gmail.com with ESMTPSA id q14sm6020665pfl.163.2020.11.26.17.50.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Nov 2020 17:50:40 -0800 (PST)
To: Alexander Azimov 
Cc: Tom Herbert , tcpm , IPv6 Operations 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>        
From: Brian E Carpenter 
Message-ID: 
Date: Fri, 27 Nov 2020 14:50:36 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 01:50:48 -0000

On 27-Nov-20 09:36, Alexander Azimov wrote:
> Brian,
>=20
> May I ask you to give more details on this issue:
>=20
>     I am convinced that it is often unsafe, because that depends on exa=
ctly how a server farm load balancer handles such cases. If it maintains =
the TCP session on the same actual server if there is SYN(-ACK) retransmi=
ssion, the flow label must not be changed.=C2=A0=C2=A0
>=20
> If a retransmitted SYN goes to another server/anycast point this won't =
bring much harm. At least we haven't experienced such problems in our dat=
acenters.

I think depends on details that are definitively out of the IETF's scope.=
 When exactly does the load balancing system assign a session to a given =
server, how much state does that create, what resources does it reserve, =
and when are the state and the reserved resources released?

Also, in some cases, I could imagine that the data centre will not see a =
problem, but the client might see one that they would not otherwise have =
seen. Again, it depends on details that are out of our scope. So I think =
it's a topic that we should intentionally avoid. (Possibly it's a topic t=
hat QUIC should consider, when they consider server load balancing in gen=
eral.)

Regards
   Brian


>=20
> =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 22:53, B=
rian E Carpenter >:
>=20
>     Alexander,
>=20
>     > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD=
 be recalculated.
>     ...
>     > The first point stands for redirecting connection from the degrad=
ed path before the connection is established, and it looks safe.
>=20
>     I am convinced that it is often unsafe, because that depends on exa=
ctly how a server farm load balancer handles such cases. If it maintains =
the TCP session on the same actual server if there is SYN(-ACK) retransmi=
ssion, the flow label must not be changed.
>     The benefit for possible broken routing is quite hypothetical, wher=
eas the risk to
>     session persistence at the server is real. (Remember that the sourc=
e cannot know by magic that there is server load balancing at the far end=
, but it's the common case today.)
>=20
>     > 2. In the case of RTO timeout expiration in the established TCP s=
ession the flow label MAY be recalculated. This setting MUST be switched =
off by default.
>=20
>     As noted before, this is only safe in a limited domain where it is =
known with certainty that server load balancing is not in use. I think it=
's therefore a corner case and specifically, the appropriate normative st=
atement would be SHOULD NOT, which is defined very carefully in RFC2119 e=
xactly for this sort of case.
>=20
>     But in fact I think we should not attempt to legislate on this poin=
t. The value is marginal at best. As others have said, this is not the wa=
y to tackle operational routing problems.
>=20
>     Regards
>     =C2=A0 =C2=A0Brian Carpenter
>=20
>     On 26-Nov-20 22:34, Alexander Azimov wrote:
>     > Dear colleagues,
>     >
>     > We started discussing an incorrect default behavior, and Tom has =
already confirmed that it will be fixed.
>     >
>     > Later the thread turned into an argument if the flow label can be=
 changed during connection/flow lifetime according to current RFC documen=
ts, though these documents can be updated. This looks a bit weird for me =
because I always thought that it is the IETF community's responsibility t=
o document proper solutions. If something undocumented but worthy happens=
 in the industry - IETF should catch up. So, I would like to get back to =
the discussion of reasons and their safety.
>     >
>     > The general idea of changing the routing path upon network outage=
/degradation looks obvious. Getting kind of source-based routing can sign=
ificantly reduce the reaction time and improve end-user experience. The f=
low label has a perfect match here: transparent for the application, set =
by the source, not part of 5-tuple, while it can be used in the load bala=
ncing. IMO poorly documented. I would like to learn your feedback for the=
 next wording:
>     >
>     > Let say that the flow label is a hash of values from the IP packe=
t's 5-tuple and random number. Then
>     >
>     > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD=
 be recalculated.
>     > 2. In the case of RTO timeout expiration in the established TCP s=
ession the flow label MAY be recalculated. This setting MUST be switched =
off by default.
>     > 3. Otherwise flow label SHOULD be preserved unchanged.
>     >
>     > The first point stands for redirecting connection from the degrad=
ed path before the connection is established, and it looks safe. The seco=
nd one can also improve performance if it is set on the server-side. Plea=
se comment if you see security flaws in such a design.
>     >
>     > =D1=87=D1=82, 26 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 03=
:16, Tom Herbert  >>:
>     >
>     >=C2=A0 =C2=A0 =C2=A0On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpen=
ter
>     >=C2=A0 =C2=A0 =C2=A0 >> wrote:
>     >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> I'm not Joel, but I did once spend some time=
 grepping RFCs to find out whether
>     >=C2=A0 =C2=A0 =C2=A0> "flow" or "microflow" was the preferred term=
=2E In RFC2474, which is normative,
>     >=C2=A0 =C2=A0 =C2=A0> we have:
>     >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 Microflow: a single instance of=
 an application-to-application flow of
>     >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 packets which is identified by =
source address, destination address,
>     >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 protocol id, and source port, d=
estination port (where applicable).
>     >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> But in the flow label work we explicitly avo=
ided being that precise, and
>     >=C2=A0 =C2=A0 =C2=A0> did not use the term "microflow". There migh=
t be some load balancing
>     >=C2=A0 =C2=A0 =C2=A0> scenarios where you want a broader definitio=
n, even including bidirectional
>     >=C2=A0 =C2=A0 =C2=A0> flows. There are expired drafts on that topi=
c:
>     >=C2=A0 =C2=A0 =C2=A0> draft-tarreau-extend-flow-label-balancing
>     >=C2=A0 =C2=A0 =C2=A0> draft-wang-6man-flow-label-reflection
>     >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0Brian,
>     >
>     >=C2=A0 =C2=A0 =C2=A0Random Packet Spraying
>     >=C2=A0 =C2=A0 =C2=A0(https://citeseerx.ist.psu.edu/viewdoc/downloa=
d?doi=3D10.1.1.297.529&rep=3Drep1&type=3Dpdf)
>     >=C2=A0 =C2=A0 =C2=A0is an interesting idea where packets for a sin=
gle connection are
>     >=C2=A0 =C2=A0 =C2=A0purposely distributed across multiple paths fo=
r load distribution. Per
>     >=C2=A0 =C2=A0 =C2=A0packet randomized flow labels with flow label =
aware ECMP makes this
>     >=C2=A0 =C2=A0 =C2=A0quite easy to do without requiring any special=
 support in switches
>     >=C2=A0 =C2=A0 =C2=A0like you'd need with IPv4. I'm not necessarily=
 advocating this, but it
>     >=C2=A0 =C2=A0 =C2=A0does highlight one potential use case of havin=
g a flow label that
>     >=C2=A0 =C2=A0 =C2=A0doesn;t have rigidly defined requirements on t=
he host.
>     >
>     >=C2=A0 =C2=A0 =C2=A0Tom
>     >
>     >=C2=A0 =C2=A0 =C2=A0> Regards
>     >=C2=A0 =C2=A0 =C2=A0>=C2=A0 =C2=A0 Brian
>     >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> On 26-Nov-20 11:13, Tom Herbert wrote:
>     >=C2=A0 =C2=A0 =C2=A0> > Joel, is there a normative definition of a=
 flow?
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halp=
ern  >  >>> wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0No, as Brian says, ther=
e are escape clauses in the flow definitions.
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0But changing the flow l=
abel due to traffic problems does not correspond
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0to packets being in act=
ually different flows.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0If one were using UDP, =
and mixing loss sensitive packets with loss
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0insensitive packets for=
 a special application, sure, one could use two
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0flow labels.=C2=A0 But =
that is not what you are describing.
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0Yours,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0Joel
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0On 11/25/2020 3:05 PM, =
Tom Herbert wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Joel,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Is there an RFC that =
clearly and unambiguously states that a host MUST
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> use the same flow lab=
el for the lifetime _and_ clearly defines exactly
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> what a flow is with r=
espect to such a requirement (for instance, how
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> would you define a fl=
ow and enforce such a requirement in UDP? IPsec?
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> other encapsulations?=
). If there is such a requirement then we'll
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> change the code to be=
 conformant.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> Tom
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0> On Wed, Nov 25, 2020 =
at 12:39 PM Joel M. Halpern  >  >>> wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> This kind of thing i=
s why, as I understand it, MPTCP has discovery
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> mechanisms ot know i=
f both sides use it, and can select alternative
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> addresses for commun=
ication.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Trying to guess flow=
 labels that might avoid a problem because it might
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> be an ECMP problem, =
is just flailing about.=C2=A0 Not a good design for
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> operational protocol=
s.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> And in general, desi=
gning protocols around "I know exactly what is going
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> on"=C2=A0 (the requi=
rement for what you describe that goes well beyond just
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> "limited domains") i=
s also a recipe for failure.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> The Flow Label RFCs =
are actually very explicit that a flow label is
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> supposed to be stabl=
e for the life of the flow.=C2=A0 Otherwise, it isn't a
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> flow label.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Yours,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> Joel
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>> On 11/25/2020 2:35 P=
M, Tom Herbert wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Hi Fernando, commen=
ts in line...
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> On Wed, Nov 25, 202=
0 at 12:13 AM Fernando Gont  >  >>> wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Hi, Tom,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> On 24/11/20 16:43,=
 Tom Herbert wrote:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> [....]
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> Modulating the fl=
ow label is a means to affect the routing of packets
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> through the netwo=
rk that uses flow labels as input to the ECMP hash.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> What's the point?
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 1) You cannot tell=
 *if* the FL is being used.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Generally true, but=
 in a limited domain this information could be
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> discerned. I'd note=
 that it's also generally true that we don't know
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> if there is a load =
balancer or stateful firewall in the path that
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> requires consistent=
 routing, but in a limited domain we could know
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> that also.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 2) Changing the FL=
 does not necessarily mean that packets will employ a
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> different link.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> It's an opportunist=
ic mechanism. If a connection is failing and we get
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> a better path that =
fixes it by simply changing the flow label then
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> what's the harm?
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> 3) If the network =
is failing, shouldn't you handle this via routing?
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Sure, but then that=
 requires an out of band feedback loop from a TCP
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> implementation to t=
he network infrastructure to indicate there is a
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> problem and then th=
e network needs to respond. That's significant
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure and =
higher reaction time than doing something in TCP
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> and IP. Think of mo=
dulating the flow label is an inexpensive form of
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> source routing with=
in a limited domain that doesn't need any
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> infrastructure or h=
eavyweight protocols or something like segment
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> routing.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> The basic idea is=
 that the flow label associated with a connection is
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> randomly changed =
when the stack observes that the connection is
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> failing (e.g. and=
 an RTO). There is nothing in the specs that prevents
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> this since the so=
urce is at liberty to set the flow label as it sees
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> fit.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> The FL is expected=
 to remain constant for the life of a flow. A
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> retransmitted pack=
et is part of the same flow as the
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> originally-transmi=
tted packet. So this seems to be contradicting the
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> very specification=
 of the FL.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> For instance, If a=
 RTO for a flow causes the FL to change, then one may
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> possibly argue tha=
t the FL is not naming/labeling what is said/expected
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> to be anming/label=
ing.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Specifically, RFC64=
37 states:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> "It is therefore RE=
COMMENDED that source hosts support the flow label
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> by setting the flow=
 label field for all packets of a given flow to the
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> same value chosen f=
rom an approximation to a discrete uniform
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> distribution."
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> So that is clearly =
a just recommendation, and not a requirement (and
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> definitely not a MU=
ST). Furthermore, RFC6437 states:
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> "A forwarding node =
MUST either leave a non-zero flow label value
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> unchanged or change=
 it only for compelling operational security
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> reasons as describe=
d in Section 6.1."
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> So there's no guara=
ntee in the protocol specs that flow labels are
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> consistent for the =
life of the connection, which means that the
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> network cannot assu=
me that and thus it would be incorrect if the
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> network tried to en=
force flow label consistency as a protocol
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> requirement. As I s=
aid, it is prudent to try to be consistent with
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> flow labels and the=
 default behavior in Linux should be changed,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> however I do not be=
lieve there's a valid claim of non-conformance that
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> motivates removal o=
f the feature that is already deployed.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> Tom
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> The feature is us=
eful in large datacenter networks, like
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> pparently Faceboo=
k where the patches originate, since information
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> discerned by TCP =
can opportunistically be applied to route selection.
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> The practical iss=
ue is that there are stateful devices like firewalls
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> that require cons=
istent routing in the network in which case changing
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> the flow label ca=
n confuse them. As I mentioned, the original intent
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> was that the flow=
 label randomization feature should be opt-in instead
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>> of on by default.=

>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> So... where is the=
 "source" of the packet that would be "modulating" the FL?
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Thanks,
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> --
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> Fernando Gont
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> e-mail: fernando@g=
ont.com.ar  >  >=
> || fgont@si6networks.com  >  >>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>> PGP Fingerprint: 7=
809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> ___________________=
____________________________
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> v6ops mailing list
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> v6ops@ietf.org  >  >>
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>> https://www.ietf.or=
g/mailman/listinfo/v6ops
>     >=C2=A0 =C2=A0 =C2=A0> >=C2=A0 =C2=A0 =C2=A0>>>
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0> > __________________________________________=
_____
>     >=C2=A0 =C2=A0 =C2=A0> > v6ops mailing list
>     >=C2=A0 =C2=A0 =C2=A0> > v6ops@ietf.org  >
>     >=C2=A0 =C2=A0 =C2=A0> > https://www.ietf.org/mailman/listinfo/v6op=
s
>     >=C2=A0 =C2=A0 =C2=A0> >
>     >=C2=A0 =C2=A0 =C2=A0>
>     >
>     >=C2=A0 =C2=A0 =C2=A0______________________________________________=
_
>     >=C2=A0 =C2=A0 =C2=A0v6ops mailing list
>     >=C2=A0 =C2=A0 =C2=A0v6ops@ietf.org  >
>     >=C2=A0 =C2=A0 =C2=A0https://www.ietf.org/mailman/listinfo/v6ops
>     >
>     >
>     >
>     > --
>     > Best regards,
>     > Alexander Azimov
>=20
>=20
>=20
> --=20
> Best regards,
> Alexander Azimov


From nobody Fri Nov 27 09:33:39 2020
Return-Path: 
X-Original-To: tcpm@ietf.org
Delivered-To: tcpm@ietfa.amsl.com
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 2D8183A0AB4; Fri, 27 Nov 2020 09:33:29 -0800 (PST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
From: Paul Wouters via Datatracker 
To: 
Cc: tcpm@ietf.org, last-call@ietf.org, draft-ietf-tcpm-rack.all@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 7.23.0
Auto-Submitted: auto-generated
Precedence: bulk
Message-ID: <160649840913.22803.17626724526067078564@ietfa.amsl.com>
Reply-To: Paul Wouters 
Date: Fri, 27 Nov 2020 09:33:29 -0800
Archived-At: 
Subject: [tcpm] Secdir last call review of draft-ietf-tcpm-rack-13
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 17:33:29 -0000

Reviewer: Paul Wouters
Review result: Ready

I have reviewed this document as part of the security directorate's 
ongoing effort to review all IETF documents being processed by the 
IESG.  These comments were written primarily for the benefit of the 
security area directors.  Document editors and WG chairs should treat 
these comments just like any other last call comments.

The summary of the review is Ready

I am not a TCP expert, but the text in the Security Considerations is clear
and shows an actual increase in security compared to other methods of
packet loss detection and retransmission.





From nobody Fri Nov 27 12:42:04 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3E5D83A0E87; Fri, 27 Nov 2020 12:42:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BC9DmJ34equ1; Fri, 27 Nov 2020 12:42:00 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [IPv6:2001:67c:27e4::14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 955643A0E84; Fri, 27 Nov 2020 12:41:57 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 3C33C283BC9; Fri, 27 Nov 2020 20:41:52 +0000 (UTC)
To: Tom Herbert 
Cc: Alexander Azimov , tcpm , IPv6 Operations 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar> 
From: Fernando Gont 
Message-ID: <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>
Date: Fri, 27 Nov 2020 17:41:40 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 20:42:03 -0000

Hello, Tom,

On 25/11/20 16:35, Tom Herbert wrote:
> Hi Fernando, comments in line...
> 
> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
>>
>> Hi, Tom,
>>
>> On 24/11/20 16:43, Tom Herbert wrote:
>> [....]
>>> Modulating the flow label is a means to affect the routing of packets
>>> through the network that uses flow labels as input to the ECMP hash.
>>
>> What's the point?
>>
>> 1) You cannot tell *if* the FL is being used.
>>
> Generally true, but in a limited domain this information could be
> discerned. I'd note that it's also generally true that we don't know
> if there is a load balancer or stateful firewall in the path that
> requires consistent routing, but in a limited domain we could know
> that also.

Exactly. So my take is that the drawbacks for the general case outweigh 
the benefits of the specific case.



>> 2) Changing the FL does not necessarily mean that packets will employ a
>> different link.
> 
> It's an opportunistic mechanism. If a connection is failing and we get
> a better path that fixes it by simply changing the flow label then
> what's the harm?

Complexity, and the possible negative implications for the general case.

That said, why don't you fix the problem where you should (i.e., routing)?



>> 3) If the network is failing, shouldn't you handle this via routing?
>>
> Sure, but then that requires an out of band feedback loop from a TCP
> implementation to the network infrastructure to indicate there is a
> problem and then the network needs to respond.

Not really. TCP does not deal with paths (not should it). If the network 
detects a link is failing, it should route around the problem.



> That's significant
> infrastructure and higher reaction time than doing something in TCP
> and IP. Think of modulating the flow label is an inexpensive form of
> source routing within a limited domain that doesn't need any
> infrastructure or heavyweight protocols or something like segment
> routing.

Downsides:

* It breaks load balancing
* It depends on the source doing something -- but you don't necessarily 
control the source -- and you cannot tell if the source implements it
* It doesn't solve the problem for nodes that do not behave as expected

My take is that if you really care about this, you're already (or 
should!) be solving the problem in a different (and more general) way.



>>> The basic idea is that the flow label associated with a connection is
>>> randomly changed when the stack observes that the connection is
>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>>> this since the source is at liberty to set the flow label as it sees
>>> fit.
>>
>> The FL is expected to remain constant for the life of a flow. A
>> retransmitted packet is part of the same flow as the
>> originally-transmitted packet. So this seems to be contradicting the
>> very specification of the FL.
>>
>> For instance, If a RTO for a flow causes the FL to change, then one may
>> possibly argue that the FL is not naming/labeling what is said/expected
>> to be anming/labeling.
> 
> Specifically, RFC6437 states:
> 
> "It is therefore RECOMMENDED that source hosts support the flow label
> by setting the flow label field for all packets of a given flow to the
> same value chosen from an approximation to a discrete uniform
> distribution."
> 
> So that is clearly a just recommendation, and not a requirement (and
> definitely not a MUST). Furthermore, RFC6437 states:
> 
> "A forwarding node MUST either leave a non-zero flow label value
> unchanged or change it only for compelling operational security
> reasons as described in Section 6.1."

My concern is that we may leave the FL unusable.




> So there's no guarantee in the protocol specs that flow labels are
> consistent for the life of the connection, which means that the
> network cannot assume that and thus it would be incorrect if the
> network tried to enforce flow label consistency as a protocol
> requirement.

You're probably right in that respect. And, in retrospective, I think it 
would have been better if we had specified the FL to be unmutable.



> As I said, it is prudent to try to be consistent with
> flow labels and the default behavior in Linux should be changed,

Sorry: is the current Linux behavior to modulate the FL?


> however I do not believe there's a valid claim of non-conformance that
> motivates removal of the feature that is already deployed.

I personally consider the changing the FL for a flow to be a bug. That 
said, if the default setting is the correct/compliant behavior, at the 
end of the day we're all adults, and whoever overrides the default 
behavior is supposed to know what he/she does -- and if they don't, they 
get what they deserve  ;-)

So, in that light, I wouldn't push to remove the optional behavior, 
*provided* the default behavior is correct.

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Fri Nov 27 12:47:39 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5924C3A0E8C; Fri, 27 Nov 2020 12:47:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.099
X-Spam-Level: 
X-Spam-Status: No, score=0.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O7Oq7HXekToy; Fri, 27 Nov 2020 12:47:36 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EBC083A0E87; Fri, 27 Nov 2020 12:47:35 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id A8F92283BC9; Fri, 27 Nov 2020 20:47:32 +0000 (UTC)
To: Tom Herbert , Brian E Carpenter 
Cc: tcpm , IPv6 Operations 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>     
From: Fernando Gont 
Message-ID: 
Date: Fri, 27 Nov 2020 17:47:21 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 20:47:38 -0000

Tom,

On 25/11/20 21:15, Tom Herbert wrote:
> On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
>  wrote:
>>
>> I'm not Joel, but I did once spend some time grepping RFCs to find out whether
>> "flow" or "microflow" was the preferred term. In RFC2474, which is normative,
>> we have:
>>
>>     Microflow: a single instance of an application-to-application flow of
>>     packets which is identified by source address, destination address,
>>     protocol id, and source port, destination port (where applicable).
>>
>> But in the flow label work we explicitly avoided being that precise, and
>> did not use the term "microflow". There might be some load balancing
>> scenarios where you want a broader definition, even including bidirectional
>> flows. There are expired drafts on that topic:
>> draft-tarreau-extend-flow-label-balancing
>> draft-wang-6man-flow-label-reflection
>>
> Brian,
> 
> Random Packet Spraying
> (https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf)
> is an interesting idea where packets for a single connection are
> purposely distributed across multiple paths for load distribution. Per
> packet randomized flow labels with flow label aware ECMP makes this
> quite easy to do without requiring any special support in switches
> like you'd need with IPv4. I'm not necessarily advocating this, but it
> does highlight one potential use case of having a flow label that
> doesn;t have rigidly defined requirements on the host.

You don't need to randomize the FL of packets corresponding to a flow to 
get this --- simply do round-robin at the router that wants load-sharing 
over different links.

OTOH, if you do randomize the FL of each packet, then routers that want 
to avoid packet reordering simply can't.

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Fri Nov 27 12:52:32 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A6B083A0E8B; Fri, 27 Nov 2020 12:52:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w4JRSMgJeH_P; Fri, 27 Nov 2020 12:52:24 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6AB773A0DB2; Fri, 27 Nov 2020 12:52:24 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 5DB63283BC9; Fri, 27 Nov 2020 20:52:21 +0000 (UTC)
To: Alexander Azimov , Tom Herbert 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com>      
From: Fernando Gont 
Message-ID: <5cb3b43e-913f-d6cd-f7e2-192f14daf4c2@gont.com.ar>
Date: Fri, 27 Nov 2020 17:52:14 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 20:52:27 -0000

On 26/11/20 06:34, Alexander Azimov wrote:
> Dear colleagues,
> 
> We started discussing an incorrect default behavior, and Tom has already 
> confirmed that it will be fixed.
> 
> Later the thread turned into an argument if the flow label can be 
> changed during connection/flow lifetime according to current RFC 
> documents, though these documents can be updated. This looks a bit weird 
> for me because I always thought that it is the IETF community's 
> responsibility to document proper solutions. If something undocumented 
> but worthy happens in the industry - IETF should catch up. So, I would 
> like to get back to the discussion of reasons and their safety.
> 
> The general idea of changing the routing path upon network 
> outage/degradation looks obvious. Getting kind of source-based routing 
> can significantly reduce the reaction time and improve end-user 
> experience. The flow label has a perfect match here: transparent for the 
> application, set by the source, not part of 5-tuple, while it can be 
> used in the load balancing. 

If you have two links, then, upon change of the FL, you have 50% change 
that the path will be different -- not good.  What would you do if the 
failure continues? -- keep changing the FL?



> Let say that the flow label is a hash of values from the IP packet's 
> 5-tuple and random number. Then
> 
> 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be 
> recalculated.

This one MUST be switched of by default, too. (along with #2).


> 2. In the case of RTO timeout expiration in the established TCP session 
> the flow label MAY be recalculated. This setting MUST be switched off by 
> default.
> 3. Otherwise flow label SHOULD be preserved unchanged.
> 
> The first point stands for redirecting connection from the degraded path 
> before the connection is established, and it looks safe. The second one 
> can also improve performance if it is set on the server-side. Please 
> comment if you see security flaws in such a design.

At the very least, one should analyze the case where in #1, a packet 
gets delayed, you change the FL and retransmit, and finally you get a 
response to the original packet (and/or to both packets).

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Fri Nov 27 15:31:59 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BC5F73A0365 for ; Fri, 27 Nov 2020 15:31:53 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GtTQeQt6Fb1J for ; Fri, 27 Nov 2020 15:31:51 -0800 (PST)
Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5FE553A02BC for ; Fri, 27 Nov 2020 15:31:51 -0800 (PST)
Received: by mail-ed1-x531.google.com with SMTP id r22so7249543edw.6 for ; Fri, 27 Nov 2020 15:31:51 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=KHVcGF66BJ6131S4PRwX4pt0p5yKDCntZmdun1QTP5M=; b=mKwTKQTn6zil88N7l1F7J3cF+4dM+MN17z4N/yEcbwYwoaanjnzBDtrl0xIo/4BVCt a0CnB3ACIDyLbdewD6UQ+mg5g9l2mizdJ5Ouj4UlgCzCrZTOuF1D7ipQKeDCSxC7TKnf JcMTO1/2oCTPWOZljZjSbREwstus6i6fwryq7d5Rfr3PMv++XHm5zUKNb+xuxcpw3Fwv DcIDd8+aUVosbHz78rf6GAR91ctIgZRavh0NEEvtXO4VxJbmm/CNEpOaKxoUb1wP+tDU fivBSK3JRTz1QkGcqj8OIZqFBNHLl5qGswSkS+ZX3rhecU0jZCX+jayiN17We29eXOOH FD5g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=KHVcGF66BJ6131S4PRwX4pt0p5yKDCntZmdun1QTP5M=; b=rn96Hn2FmySd+Wgm65tfQm6aoHH0Q/3/+vhZwoG+Z96NLasJh91ov+wPRwl90WnQBc WqdYjS7pYq6SNfprJ1+X6IvxJtQ6DudtWQVi+kv8gJvVZWl6Qp6a/uulxmhRKregPKal G0yXLX6qqLFBCrT0/8g30wug+vjUQVXTDihwyesme4OqF7C/5dS1DtrWSWC308w+vg5K zubbtQdK/2Q+Ah6V78GgFvvNQqS9zK1Ayz90P4dlKWyofK0y0JGk5J9hWeECMKQ8bVWh iWCWGmp9EYBJZrohaif6qC5xakyP8sB4GqKiQs6/T7KaLU7Ssf7tY0Y6YmQham4TRZyv OMcg==
X-Gm-Message-State: AOAM5322Vr8Vg0+QmV6qJq1hkQB2aQvpqFv8cZrzxVMJncxnRkOS3MOX PDaRVfC3ZHLB7hm+oz5YzFZivKWwutR+tdxtLaKo9w==
X-Google-Smtp-Source: ABdhPJx8PFZFGZ5tLxeT6YigWlaxceNYqyBlTdoN2FKAzW/VOcBIJEY8PTEJJAs3PVCO46/h4A8r3grp9WeRTWnUxAM=
X-Received: by 2002:a50:c050:: with SMTP id u16mr10447493edd.177.1606519909574;  Fri, 27 Nov 2020 15:31:49 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>
In-Reply-To: <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>
From: Tom Herbert 
Date: Fri, 27 Nov 2020 16:31:38 -0700
Message-ID: 
To: Fernando Gont 
Cc: Alexander Azimov , tcpm ,  IPv6 Operations 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Fri, 27 Nov 2020 23:31:54 -0000

On Fri, Nov 27, 2020 at 1:41 PM Fernando Gont  wrote:
>
> Hello, Tom,
>
> On 25/11/20 16:35, Tom Herbert wrote:
> > Hi Fernando, comments in line...
> >
> > On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont  wrote:
> >>
> >> Hi, Tom,
> >>
> >> On 24/11/20 16:43, Tom Herbert wrote:
> >> [....]
> >>> Modulating the flow label is a means to affect the routing of packets
> >>> through the network that uses flow labels as input to the ECMP hash.
> >>
> >> What's the point?
> >>
> >> 1) You cannot tell *if* the FL is being used.
> >>
> > Generally true, but in a limited domain this information could be
> > discerned. I'd note that it's also generally true that we don't know
> > if there is a load balancer or stateful firewall in the path that
> > requires consistent routing, but in a limited domain we could know
> > that also.
>
> Exactly. So my take is that the drawbacks for the general case outweigh
> the benefits of the specific case.
>
>
>
> >> 2) Changing the FL does not necessarily mean that packets will employ a
> >> different link.
> >
> > It's an opportunistic mechanism. If a connection is failing and we get
> > a better path that fixes it by simply changing the flow label then
> > what's the harm?
>
> Complexity, and the possible negative implications for the general case.
>
> That said, why don't you fix the problem where you should (i.e., routing)?
>
>
>
> >> 3) If the network is failing, shouldn't you handle this via routing?
> >>
> > Sure, but then that requires an out of band feedback loop from a TCP
> > implementation to the network infrastructure to indicate there is a
> > problem and then the network needs to respond.
>
> Not really. TCP does not deal with paths (not should it). If the network
> detects a link is failing, it should route around the problem.
>

Fernando,

All we know in this case is that *TCP* has detected a connection is
failing and there may be a potential problem with a path, we don't
know that the network has detected any problems like some link is
failing or will ever detect any problem. So in such a scenario, what
recourse does the host have to try to salvage its connections? If the
answer is that the host should always patiently wait for the network
to figure out what is happening, then that's going to be a hard sell
to application and host stack developers (remember the network is a
black box from their perspective that they really don't trust). If the
answer is that the host is supposed to inform the network of issues
with it's connections and then host and network work together to solve
then that's great; but then what is the general protocol that has been
established to facilitate that and solved the problem of how hosts and
network work together in concert to solve user's problems?

Tom

>
>
> > That's significant
> > infrastructure and higher reaction time than doing something in TCP
> > and IP. Think of modulating the flow label is an inexpensive form of
> > source routing within a limited domain that doesn't need any
> > infrastructure or heavyweight protocols or something like segment
> > routing.
>
> Downsides:
>
> * It breaks load balancing
> * It depends on the source doing something -- but you don't necessarily
> control the source -- and you cannot tell if the source implements it
> * It doesn't solve the problem for nodes that do not behave as expected
>
> My take is that if you really care about this, you're already (or
> should!) be solving the problem in a different (and more general) way.
>
>
>
> >>> The basic idea is that the flow label associated with a connection is
> >>> randomly changed when the stack observes that the connection is
> >>> failing (e.g. and an RTO). There is nothing in the specs that prevents
> >>> this since the source is at liberty to set the flow label as it sees
> >>> fit.
> >>
> >> The FL is expected to remain constant for the life of a flow. A
> >> retransmitted packet is part of the same flow as the
> >> originally-transmitted packet. So this seems to be contradicting the
> >> very specification of the FL.
> >>
> >> For instance, If a RTO for a flow causes the FL to change, then one may
> >> possibly argue that the FL is not naming/labeling what is said/expected
> >> to be anming/labeling.
> >
> > Specifically, RFC6437 states:
> >
> > "It is therefore RECOMMENDED that source hosts support the flow label
> > by setting the flow label field for all packets of a given flow to the
> > same value chosen from an approximation to a discrete uniform
> > distribution."
> >
> > So that is clearly a just recommendation, and not a requirement (and
> > definitely not a MUST). Furthermore, RFC6437 states:
> >
> > "A forwarding node MUST either leave a non-zero flow label value
> > unchanged or change it only for compelling operational security
> > reasons as described in Section 6.1."
>
> My concern is that we may leave the FL unusable.
>
>
>
>
> > So there's no guarantee in the protocol specs that flow labels are
> > consistent for the life of the connection, which means that the
> > network cannot assume that and thus it would be incorrect if the
> > network tried to enforce flow label consistency as a protocol
> > requirement.
>
> You're probably right in that respect. And, in retrospective, I think it
> would have been better if we had specified the FL to be unmutable.
>
>
>
> > As I said, it is prudent to try to be consistent with
> > flow labels and the default behavior in Linux should be changed,
>
> Sorry: is the current Linux behavior to modulate the FL?
>
>
> > however I do not believe there's a valid claim of non-conformance that
> > motivates removal of the feature that is already deployed.
>
> I personally consider the changing the FL for a flow to be a bug. That
> said, if the default setting is the correct/compliant behavior, at the
> end of the day we're all adults, and whoever overrides the default
> behavior is supposed to know what he/she does -- and if they don't, they
> get what they deserve  ;-)
>
> So, in that light, I wouldn't push to remove the optional behavior,
> *provided* the default behavior is correct.
>
> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>


From nobody Fri Nov 27 16:32:33 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 068923A091C; Fri, 27 Nov 2020 16:32:31 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.595
X-Spam-Level: 
X-Spam-Status: No, score=-1.595 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FROM_LOCAL_NOVOWEL=0.5, HK_RANDOM_ENVFROM=0.001, HK_RANDOM_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ACLvHYowSiE4; Fri, 27 Nov 2020 16:32:28 -0800 (PST)
Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6AE713A080A; Fri, 27 Nov 2020 16:32:28 -0800 (PST)
Received: by mail-ot1-x32a.google.com with SMTP id n11so6114479ota.2; Fri, 27 Nov 2020 16:32:28 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lJdZkoENN4nHZDUXLR5mmCBIff5HJLIMuuKUOEUIh+I=; b=WbNVCEw1fSUZdoKYBMhCwKTSJ+iqlluGp0e506Kzgz61XEg1lrUaUIMp2NAQNw2/hA EGtg7eDH3j6N0IXPHPd5FjKMfiWMVdTmxhDkEBA7OliBM5bsjJqs4sRqJ/biPrHnqLg5 Jyhy0HORHn/UHy2yQrvCEk5UcBH65kga4esUPJ5f2HMVKF1v7u7uHHRN/689AG/3Nkr9 GjDtQJjboJ9iu/o0WvllGmzMW7qWTx+tiC67nF945A09Q6SDeWw5bJWDzow4oc9nBbnY 4SF4WO803RbsLKh3nKOzYhuYZnmJoUXsmrudNfRhsoJvbSn4e074Zu4hAFjw+xwfq1UF IBXA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lJdZkoENN4nHZDUXLR5mmCBIff5HJLIMuuKUOEUIh+I=; b=dtuvatoQIWmHViF5+riOuaPUthzMFEEjQhwUE2u8+Fp/Q5r2+FFhAnx1PLd/IKKDD1 XUY6v4fpBPg2OQ9Zc6DFPVXct099nWda6q0BkUL7PNTrp8nRp+o7hDuDTRIarbviHdin szl+WuQwZOT9RsqAf98AHH5AbJBeiDCsNACfewzXPwM73xOpoqaxblXs7C6bqEghmWa3 qvNmJFgD5q79uH7o583VsgYGJ6pIGnwn1WwviN6cq/vJtm5r7DXiV8s4DceJjGxTHfzq r9Gzpc95vf97oCr/FqEUUxhat+V17gHiUfa0pYpPZZBXpMHCrrQ1PAha1gPLT48tktG8 YNbw==
X-Gm-Message-State: AOAM532C7/sQcjpViDADVpqsbctg605Q1j47Rb3zXBTK929gcDipa25D dGhC2QjJCK1VNqdl4WSZqGxdKElBnicUKeoUAU3MS+3v
X-Google-Smtp-Source: ABdhPJxsW03VUVHB6PEpJ9nomQfw1NCSK+HtRHkkAqswTeprA5JEUWIG9lS/kR8YLk8qkhDTp1svTR5liCovaSrreUQ=
X-Received: by 2002:a05:6830:160d:: with SMTP id g13mr8680090otr.74.1606523547462;  Fri, 27 Nov 2020 16:32:27 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar> 
In-Reply-To: 
From: Mark Smith 
Date: Sat, 28 Nov 2020 11:32:14 +1100
Message-ID: 
To: Tom Herbert 
Cc: Fernando Gont , IPv6 Operations , tcpm 
Content-Type: multipart/alternative; boundary="0000000000005eae5b05b51fe993"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 00:32:31 -0000

--0000000000005eae5b05b51fe993
Content-Type: text/plain; charset="UTF-8"

On Sat, 28 Nov 2020, 10:32 Tom Herbert,  wrote:

> On Fri, Nov 27, 2020 at 1:41 PM Fernando Gont 
> wrote:
> >
> > Hello, Tom,
> >
> > On 25/11/20 16:35, Tom Herbert wrote:
> > > Hi Fernando, comments in line...
> > >
> > > On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont 
> wrote:
> > >>
> > >> Hi, Tom,
> > >>
> > >> On 24/11/20 16:43, Tom Herbert wrote:
> > >> [....]
> > >>> Modulating the flow label is a means to affect the routing of packets
> > >>> through the network that uses flow labels as input to the ECMP hash.
> > >>
> > >> What's the point?
> > >>
> > >> 1) You cannot tell *if* the FL is being used.
> > >>
> > > Generally true, but in a limited domain this information could be
> > > discerned. I'd note that it's also generally true that we don't know
> > > if there is a load balancer or stateful firewall in the path that
> > > requires consistent routing, but in a limited domain we could know
> > > that also.
> >
> > Exactly. So my take is that the drawbacks for the general case outweigh
> > the benefits of the specific case.
> >
> >
> >
> > >> 2) Changing the FL does not necessarily mean that packets will employ
> a
> > >> different link.
> > >
> > > It's an opportunistic mechanism. If a connection is failing and we get
> > > a better path that fixes it by simply changing the flow label then
> > > what's the harm?
> >
> > Complexity, and the possible negative implications for the general case.
> >
> > That said, why don't you fix the problem where you should (i.e.,
> routing)?
> >
> >
> >
> > >> 3) If the network is failing, shouldn't you handle this via routing?
> > >>
> > > Sure, but then that requires an out of band feedback loop from a TCP
> > > implementation to the network infrastructure to indicate there is a
> > > problem and then the network needs to respond.
> >
> > Not really. TCP does not deal with paths (not should it). If the network
> > detects a link is failing, it should route around the problem.
> >
>
> Fernando,
>
> All we know in this case is that *TCP* has detected a connection is
> failing and there may be a potential problem with a path, we don't
> know that the network has detected any problems like some link is
> failing or will ever detect any problem. So in such a scenario, what
> recourse does the host have to try to salvage its connections?


Retransmit their packets, let the network take care of trying to deliver
them again.


If the
> answer is that the host should always patiently wait for the network
> to figure out what is happening,


Yes.

Steering packets is the network's job. People like me who run the network
expect to be in charge of that. We have fancy mechanisms that mean
alternate paths are switched to very quickly.

The trouble with having two parties trying to solve the same problem at the
same time without coordination is they're just as likely to work against
each other as they are inadvertently to work with each other.

If hosts want the network to respect the flow label, then hosts need to
ensure that the a flow label value for a flow of packets with the same
source and destination address consistently represents a flow.

If hosts start trying to take control of steering traffic by varying the
flow label value within a flow, making the network's job harder, and the
network being blamed for the possibility bad network performance that may
result, the response will be both quick and easy - network operators will
switch off looking the Flow Label.

The Flow Label is the IPv6 header analogue for transport layer protocol
ports, so that load balancing mechanisms don't have to dig further into the
packet. Can or are transport layer ports varied in TCP SYNs, SYN/ACKs and
ACKs within the same TCP connection?


then that's going to be a hard sell
> to application and host stack developers (remember the network is a
> black box from their perspective that they really don't trust). If the
> answer is that the host is supposed to inform the network of issues
> with it's connections and then host and network work together to solve
> then that's great; but then what is the general protocol that has been
> established to facilitate that and solved the problem of how hosts and
> network work together in concert to solve user's problems?
>
> Tom
>
> >
> >
> > > That's significant
> > > infrastructure and higher reaction time than doing something in TCP
> > > and IP. Think of modulating the flow label is an inexpensive form of
> > > source routing within a limited domain that doesn't need any
> > > infrastructure or heavyweight protocols or something like segment
> > > routing.
> >
> > Downsides:
> >
> > * It breaks load balancing
> > * It depends on the source doing something -- but you don't necessarily
> > control the source -- and you cannot tell if the source implements it
> > * It doesn't solve the problem for nodes that do not behave as expected
> >
> > My take is that if you really care about this, you're already (or
> > should!) be solving the problem in a different (and more general) way.
> >
> >
> >
> > >>> The basic idea is that the flow label associated with a connection is
> > >>> randomly changed when the stack observes that the connection is
> > >>> failing (e.g. and an RTO). There is nothing in the specs that
> prevents
> > >>> this since the source is at liberty to set the flow label as it sees
> > >>> fit.
> > >>
> > >> The FL is expected to remain constant for the life of a flow. A
> > >> retransmitted packet is part of the same flow as the
> > >> originally-transmitted packet. So this seems to be contradicting the
> > >> very specification of the FL.
> > >>
> > >> For instance, If a RTO for a flow causes the FL to change, then one
> may
> > >> possibly argue that the FL is not naming/labeling what is
> said/expected
> > >> to be anming/labeling.
> > >
> > > Specifically, RFC6437 states:
> > >
> > > "It is therefore RECOMMENDED that source hosts support the flow label
> > > by setting the flow label field for all packets of a given flow to the
> > > same value chosen from an approximation to a discrete uniform
> > > distribution."
> > >
> > > So that is clearly a just recommendation, and not a requirement (and
> > > definitely not a MUST). Furthermore, RFC6437 states:
> > >
> > > "A forwarding node MUST either leave a non-zero flow label value
> > > unchanged or change it only for compelling operational security
> > > reasons as described in Section 6.1."
> >
> > My concern is that we may leave the FL unusable.
> >
> >
> >
> >
> > > So there's no guarantee in the protocol specs that flow labels are
> > > consistent for the life of the connection, which means that the
> > > network cannot assume that and thus it would be incorrect if the
> > > network tried to enforce flow label consistency as a protocol
> > > requirement.
> >
> > You're probably right in that respect. And, in retrospective, I think it
> > would have been better if we had specified the FL to be unmutable.
> >
> >
> >
> > > As I said, it is prudent to try to be consistent with
> > > flow labels and the default behavior in Linux should be changed,
> >
> > Sorry: is the current Linux behavior to modulate the FL?
> >
> >
> > > however I do not believe there's a valid claim of non-conformance that
> > > motivates removal of the feature that is already deployed.
> >
> > I personally consider the changing the FL for a flow to be a bug. That
> > said, if the default setting is the correct/compliant behavior, at the
> > end of the day we're all adults, and whoever overrides the default
> > behavior is supposed to know what he/she does -- and if they don't, they
> > get what they deserve  ;-)
> >
> > So, in that light, I wouldn't push to remove the optional behavior,
> > *provided* the default behavior is correct.
> >
> > Thanks,
> > --
> > Fernando Gont
> > e-mail: fernando@gont.com.ar || fgont@si6networks.com
> > PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> >
> >
> >
>
> _______________________________________________
> v6ops mailing list
> v6ops@ietf.org
> https://www.ietf.org/mailman/listinfo/v6ops
>

--0000000000005eae5b05b51fe993
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable




On Sat, 28 Nov 2020, 10:32 Tom Herbert, <tom@herbe=
rtland.com> wrote:
On Fri, =
Nov 27, 2020 at 1:41 PM Fernando Gont <fernando@gont.com.ar=
> wrote:

>

> Hello, Tom,

>

> On 25/11/20 16:35, Tom Herbert wrote:

> > Hi Fernando, comments in line...

> >

> > On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <f=
ernando@gont.com.ar> wrote:

> >>

> >> Hi, Tom,

> >>

> >> On 24/11/20 16:43, Tom Herbert wrote:

> >> [....]

> >>> Modulating the flow label is a means to affect the routin=
g of packets

> >>> through the network that uses flow labels as input to the=
 ECMP hash.

> >>

> >> What's the point?

> >>

> >> 1) You cannot tell *if* the FL is being used.

> >>

> > Generally true, but in a limited domain this information could be=


> > discerned. I'd note that it's also generally true that we=
 don't know

> > if there is a load balancer or stateful firewall in the path that=


> > requires consistent routing, but in a limited domain we could kno=
w

> > that also.

>

> Exactly. So my take is that the drawbacks for the general case outweig=
h

> the benefits of the specific case.

>

>

>

> >> 2) Changing the FL does not necessarily mean that packets wil=
l employ a

> >> different link.

> >

> > It's an opportunistic mechanism. If a connection is failing a=
nd we get

> > a better path that fixes it by simply changing the flow label the=
n

> > what's the harm?

>

> Complexity, and the possible negative implications for the general cas=
e.

>

> That said, why don't you fix the problem where you should (i.e., r=
outing)?

>

>

>

> >> 3) If the network is failing, shouldn't you handle this v=
ia routing?

> >>

> > Sure, but then that requires an out of band feedback loop from a =
TCP

> > implementation to the network infrastructure to indicate there is=
 a

> > problem and then the network needs to respond.

>

> Not really. TCP does not deal with paths (not should it). If the netwo=
rk

> detects a link is failing, it should route around the problem.

>



Fernando,



All we know in this case is that *TCP* has detected a connection is

failing and there may be a potential problem with a path, we don't

know that the network has detected any problems like some link is

failing or will ever detect any problem. So in such a scenario, what

recourse does the host have to try to salvage its connections?
=


Retransmit their =
packets, let the network take care of trying to deliver them again.


 If the

answer is that the host should always patiently wait for the network

to figure out what is happening,
=

Yes.

Steering packets is the network's job. People like me who run=
 the network expect to be in charge of that. We have fancy mechanisms that =
mean alternate paths are switched to very quickly.
<=
br>
The trouble with having two parties trying to so=
lve the same problem at the same time without coordination is they're j=
ust as likely to work against each other as they are inadvertently to work =
with each other.

If host=
s want the network to respect the flow label, then hosts need to ensure tha=
t the a flow label value for a flow of packets with the same source and des=
tination address consistently represents a flow.
If hosts start trying to take control of steering =
traffic by varying the flow label value within a flow, making the network&#=
39;s job harder, and the network being blamed for the possibility bad netwo=
rk performance that may result, the response will be both quick and easy - =
network operators will switch off looking the Flow Label.

The Flow Label is the IPv6 header analogu=
e for transport layer protocol ports, so that load balancing mechanisms don=
't have to dig further into the packet. Can or are transport layer port=
s varied in TCP SYNs, SYN/ACKs and ACKs within the same TCP connection?=C2=
=A0


 then t=
hat's going to be a hard sell

to application and host stack developers (remember the network is a

black box from their perspective that they really don't trust). If the<=
br>
answer is that the host is supposed to inform the network of issues

with it's connections and then host and network work together to solve<=
br>
then that's great; but then what is the general protocol that has been<=
br>
established to facilitate that and solved the problem of how hosts and

network work together in concert to solve user's problems?



Tom



>

>

> > That's significant

> > infrastructure and higher reaction time than doing something in T=
CP

> > and IP. Think of modulating the flow label is an inexpensive form=
 of

> > source routing within a limited domain that doesn't need any<=
br>
> > infrastructure or heavyweight protocols or something like segment=


> > routing.

>

> Downsides:

>

> * It breaks load balancing

> * It depends on the source doing something -- but you don't necess=
arily

> control the source -- and you cannot tell if the source implements it<=
br>
> * It doesn't solve the problem for nodes that do not behave as exp=
ected

>

> My take is that if you really care about this, you're already (or<=
br>
> should!) be solving the problem in a different (and more general) way.=


>

>

>

> >>> The basic idea is that the flow label associated with a c=
onnection is

> >>> randomly changed when the stack observes that the connect=
ion is

> >>> failing (e.g. and an RTO). There is nothing in the specs =
that prevents

> >>> this since the source is at liberty to set the flow label=
 as it sees

> >>> fit.

> >>

> >> The FL is expected to remain constant for the life of a flow.=
 A

> >> retransmitted packet is part of the same flow as the

> >> originally-transmitted packet. So this seems to be contradict=
ing the

> >> very specification of the FL.

> >>

> >> For instance, If a RTO for a flow causes the FL to change, th=
en one may

> >> possibly argue that the FL is not naming/labeling what is sai=
d/expected

> >> to be anming/labeling.

> >

> > Specifically, RFC6437 states:

> >

> > "It is therefore RECOMMENDED that source hosts support the f=
low label

> > by setting the flow label field for all packets of a given flow t=
o the

> > same value chosen from an approximation to a discrete uniform

> > distribution."

> >

> > So that is clearly a just recommendation, and not a requirement (=
and

> > definitely not a MUST). Furthermore, RFC6437 states:

> >

> > "A forwarding node MUST either leave a non-zero flow label v=
alue

> > unchanged or change it only for compelling operational security
> > reasons as described in Section 6.1."

>

> My concern is that we may leave the FL unusable.

>

>

>

>

> > So there's no guarantee in the protocol specs that flow label=
s are

> > consistent for the life of the connection, which means that the
> > network cannot assume that and thus it would be incorrect if the<=
br>
> > network tried to enforce flow label consistency as a protocol

> > requirement.

>

> You're probably right in that respect. And, in retrospective, I th=
ink it

> would have been better if we had specified the FL to be unmutable.

>

>

>

> > As I said, it is prudent to try to be consistent with

> > flow labels and the default behavior in Linux should be changed,<=
br>
>

> Sorry: is the current Linux behavior to modulate the FL?

>

>

> > however I do not believe there's a valid claim of non-conform=
ance that

> > motivates removal of the feature that is already deployed.

>

> I personally consider the changing the FL for a flow to be a bug. That=


> said, if the default setting is the correct/compliant behavior, at the=


> end of the day we're all adults, and whoever overrides the default=


> behavior is supposed to know what he/she does -- and if they don't=
, they

> get what they deserve=C2=A0 ;-)

>

> So, in that light, I wouldn't push to remove the optional behavior=
,

> *provided* the default behavior is correct.

>

> Thanks,

> --

> Fernando Gont

> e-mail: fernando@gont.com.ar || fgont@s=
i6networks.com

> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

>

>

>



_______________________________________________

v6ops mailing list

v6ops@ietf.org

https://www.ietf.org/mailman/listin=
fo/v6ops




--0000000000005eae5b05b51fe993--


From nobody Fri Nov 27 16:38:41 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 520523A097E; Fri, 27 Nov 2020 16:38:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.318
X-Spam-Level: 
X-Spam-Status: No, score=-1.318 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=strayalpha.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xxLz0qNCyTee; Fri, 27 Nov 2020 16:38:39 -0800 (PST)
Received: from server217-4.web-hosting.com (server217-4.web-hosting.com [198.54.116.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EEA9C3A097B; Fri, 27 Nov 2020 16:38:38 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=strayalpha.com; s=default; h=To:References:Message-Id:Cc:Date:In-Reply-To: From:Subject:Mime-Version:Content-Type:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=qYDeYqGvc4mP5lAu9gL9lz0FLtNfydMAsQBWgZsnP90=; b=Qs56f/cQD8vMQ/zb0LSoSvhmb xiBishrKNOiR3i713iX6defy52LjXRc9Z1ebzrGtOhpqEAqj0p7vNgv03yuehadcK4Qz3DtEhxAU2 AyncXeD448+zynmiTh24Oy1C9Rbf2cwb2123Zl2LMn1wOdNhDlRw5IQO6Tcz5uDGX4hinczqfUZqT 20hAkfZYpVKG5Pqc+bLyMcklnVNAEHsaWbUaNsmSR+AmMXLMXpnI5a8u6UPluyJ9kuXnq3cKZNGHz 5q6FihjKOZUBRNzgBDAFF6IJVEwqZ873ELiHcO1toQ9u4x9ev71e1nd4fBFjCEGH6W5io5P8I6cQ2 cpCVUThRw==;
Received: from cpe-172-250-225-198.socal.res.rr.com ([172.250.225.198]:59617 helo=[192.168.1.14]) by server217.web-hosting.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1kioFn-001EKs-23; Fri, 27 Nov 2020 19:38:23 -0500
Content-Type: multipart/alternative; boundary="Apple-Mail=_44B5C294-B782-46B8-80C6-5F938156FF6E"
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\))
From: Joseph Touch 
In-Reply-To: 
Date: Fri, 27 Nov 2020 16:38:18 -0800
Cc: Tom Herbert , IPv6 Operations , tcpm , Fernando Gont 
Message-Id: <2DA153B5-1F7B-40CF-85F9-1BB3CAEF2E1E@strayalpha.com>
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>  
To: Mark Smith 
X-Mailer: Apple Mail (2.3654.20.0.2.21)
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server217.web-hosting.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - strayalpha.com
X-Get-Message-Sender-Via: server217.web-hosting.com: authenticated_id: touch@strayalpha.com
X-Authenticated-Sender: server217.web-hosting.com: touch@strayalpha.com
X-Source: 
X-Source-Args: 
X-Source-Dir: 
X-From-Rewrite: unmodified, already matched
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 00:38:40 -0000

--Apple-Mail=_44B5C294-B782-46B8-80C6-5F938156FF6E
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8



> On Nov 27, 2020, at 4:32 PM, Mark Smith  =
wrote:
>=20
> Steering packets is the network's job.

+1

> The Flow Label is the IPv6 header analogue for transport layer =
protocol ports, so that load balancing mechanisms don't have to dig =
further into the packet.

+1

> Can or are transport layer ports varied in TCP SYNs, SYN/ACKs and ACKs =
within the same TCP connection?

In general, no.

I proposed a method to decouple the SYN destination port (which =
currently is both part of the connection ID *and* identifies the =
service) from the port used for the rest of the connection, but largely =
to provide a separate service ID (originally draft-touch-tcp-portnames, =
later recast as a service name option in draft-touch-tcpm-sno). But that =
didn=E2=80=99t as much use different ports as decouple the service from =
connection ID number.

Joe


--Apple-Mail=_44B5C294-B782-46B8-80C6-5F938156FF6E
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8



On Nov 27, 2020, at 4:32 PM, Mark Smith <markzzzsmith@gmail.com> wrote:

Steering packets is the network's job. =


+1

The Flow Label is the IPv6 header analogue for =
transport layer protocol ports, so that load balancing mechanisms don't =
have to dig further into the packet. 

+1

Can or are transport layer ports =
varied in TCP SYNs, SYN/ACKs and ACKs within the same TCP =
connection?

In =
general, no.

I proposed a method to =
decouple the SYN destination port (which currently is both part of the =
connection ID *and* identifies the service) from the port used for the =
rest of the connection, but largely to provide a separate service ID =
(originally draft-touch-tcp-portnames, later recast as a service name =
option in draft-touch-tcpm-sno). But that didn=E2=80=99t as much =
use different ports as decouple the service from connection ID =
number.

Joe

=

--Apple-Mail=_44B5C294-B782-46B8-80C6-5F938156FF6E--


From nobody Fri Nov 27 19:50:25 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C93B83A0DEA; Fri, 27 Nov 2020 19:50:24 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Level: 
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Eb-n3T9NpCMH; Fri, 27 Nov 2020 19:50:21 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [IPv6:2001:67c:27e4::14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8F6843A0DE8; Fri, 27 Nov 2020 19:50:12 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id B7F0D2845F1; Sat, 28 Nov 2020 03:50:08 +0000 (UTC)
To: Tom Herbert 
Cc: Alexander Azimov , tcpm , IPv6 Operations 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar> 
From: Fernando Gont 
Message-ID: 
Date: Sat, 28 Nov 2020 00:49:52 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 03:50:25 -0000

On 27/11/20 20:31, Tom Herbert wrote:
[....]
> 
> All we know in this case is that *TCP* has detected a connection is
> failing and there may be a potential problem with a path, we don't
> know that the network has detected any problems like some link is
> failing or will ever detect any problem. So in such a scenario, what
> recourse does the host have to try to salvage its connections?

None.

TCP times out and restransmits. And *the network* routes around failing 
paths.



> If the
> answer is that the host should always patiently wait for the network
> to figure out what is happening, then that's going to be a hard sell
> to application and host stack developers (remember the network is a
> black box from their perspective that they really don't trust).

Either that, of you switch to a different destination address if the 
transport protocol allows.



> If the
> answer is that the host is supposed to inform the network of issues
> with it's connections and then host and network work together to solve
> then that's great; 

The only thing the host can do is to switch to a different transport 
protocol if it allows.


> but then what is the general protocol that has been
> established to facilitate that and solved the problem of how hosts and
> network work together in concert to solve user's problems?

They don't really work together (modulo congestion control, one might 
argue), but rther complement each other (TCP times out and retransmits, 
and the network routes around failing paths).

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Fri Nov 27 19:53:25 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 27F063A0DEC; Fri, 27 Nov 2020 19:52:48 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level: 
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uVi5Btv9E5tD; Fri, 27 Nov 2020 19:52:46 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [IPv6:2001:67c:27e4::14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BA86A3A0DEA; Fri, 27 Nov 2020 19:52:45 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 512EA2845F1; Sat, 28 Nov 2020 03:52:42 +0000 (UTC)
To: Mark Smith , Tom Herbert 
Cc: IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>  
From: Fernando Gont 
Message-ID: 
Date: Sat, 28 Nov 2020 00:52:35 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 03:52:48 -0000

On 27/11/20 21:32, Mark Smith wrote:
> 
> On Sat, 28 Nov 2020, 10:32 Tom Herbert,  > wrote:
[....]
> 
> If hosts start trying to take control of steering traffic by varying the 
> flow label value within a flow, making the network's job harder, and the 
> network being blamed for the possibility bad network performance that 
> may result, the response will be both quick and easy - network operators 
> will switch off looking the Flow Label.

THat is *exactly* why I noted that I'm concerned that we seem to be on 
the path of making the FLow Label unusable.

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Fri Nov 27 22:42:40 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0F3EF3A0C4C; Fri, 27 Nov 2020 22:42:35 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.595
X-Spam-Level: 
X-Spam-Status: No, score=-1.595 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, FROM_LOCAL_NOVOWEL=0.5, HK_RANDOM_ENVFROM=0.001, HK_RANDOM_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JpLQ6e7oJDdU; Fri, 27 Nov 2020 22:42:33 -0800 (PST)
Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 8DAD53A0C3A; Fri, 27 Nov 2020 22:42:33 -0800 (PST)
Received: by mail-oi1-x233.google.com with SMTP id v202so8232261oia.9; Fri, 27 Nov 2020 22:42:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mls8hKOkaDFJeKzYEIPlR9kLFnDA6/K7gXNXn/1CPK0=; b=TUWncmNeKVWQHjTRQHG6+jeU1+rE99koDeADd3RGHo4XHKI+3gbvo/I1N5bBzj4OnA r59xxcAiIAcTMOwPYpCmx2/JIyT1Wl1dVNNf52myZnl05REzRyCgiD1WwJqiW/dCxncx ygS8i6vLghUaMOZLcugejtgzZFn3h8NgTyWBhU6N1cT2AMS5GjFXW/RPweIcbtEJHCBA 2HhZx/niZh/G9iu/ePduxVtPrwxLGhkN4nWofwBPn4Ye/3FRJBYB5y7+Tuw2mzmDKi+j /us8VnB++3SnZVEgA5vqOSIZ6oGu0BRWVj0VePCjXqX1Vkyhbf/CPbVBcxPmHVrsB+CH pTZg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mls8hKOkaDFJeKzYEIPlR9kLFnDA6/K7gXNXn/1CPK0=; b=lDH6eOescL4e6z8I1d0lqNptgZYqeWmcPowXkpUjhnzNIkNLmkp3InyTsi6iGesap+ l7qoEM+3+ghSWRJjvB45SffUhCVN34d3uAYv6uKmMyZaZue8+AjlkYnBsCg3GQt4W1PW CImNCtdIKs9mdN6OADeky0ejPqPcUmRGH5CmQTSJ70gLrHP2Ac7Hvw6eMJAQt7+S7sZi e4CgmuwlLCxZ8QaQX2MjTY663/0OnhUylXZkv3hpboBRqOPPwqKiW6Mz/3hsX/yERyrb nHdtsHRwmflek7bx3rpU5/h5WFcKmLvXxMX1vmZ4yBB13EjgY1ur6QvoH0RjTgriJb5F V0zA==
X-Gm-Message-State: AOAM531R9LdsKE1HHsSesPYfGmDczcZQFM/lg+D76SRaEwZYMNyNSNoF sAm5YxAbu0YFGkM9B4Un7v26fC/Mfxpg6zh+LZFE3+iQ
X-Google-Smtp-Source: ABdhPJzeP1w0ZuWCXyA8QXpxAqEodGa7Ikotu2dRtTd037Kz4wVOm9Q00n37Bb5wGBeyofKPrlnswZFRyy0q4nqtrnk=
X-Received: by 2002:aca:534f:: with SMTP id h76mr8042040oib.60.1606545752887;  Fri, 27 Nov 2020 22:42:32 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>   
In-Reply-To: 
From: Mark Smith 
Date: Sat, 28 Nov 2020 17:42:20 +1100
Message-ID: 
To: Fernando Gont 
Cc: Tom Herbert , IPv6 Operations , tcpm 
Content-Type: multipart/alternative; boundary="000000000000ea965605b5251478"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 06:42:35 -0000

--000000000000ea965605b5251478
Content-Type: text/plain; charset="UTF-8"

On Sat, 28 Nov 2020, 14:52 Fernando Gont,  wrote:

> On 27/11/20 21:32, Mark Smith wrote:
> >
> > On Sat, 28 Nov 2020, 10:32 Tom Herbert,  > > wrote:
> [....]
> >
> > If hosts start trying to take control of steering traffic by varying the
> > flow label value within a flow, making the network's job harder, and the
> > network being blamed for the possibility bad network performance that
> > may result, the response will be both quick and easy - network operators
> > will switch off looking the Flow Label.
>
> THat is *exactly* why I noted that I'm concerned that we seem to be on
> the path of making the FLow Label unusable.
>

I see the Flow Label as a hint, but only a hint, of what the hosts wants
from the network. It's useful as an expression of what the host would like
to have.

However, if that hint causes me any trouble, I'll ignore it and then
devolve to trying to deliver the packet based just on its destination
address.



> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
>

--000000000000ea965605b5251478
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable




On Sat, 28 Nov 2020, 14:52 Fernando Gont, <ferna=
ndo@gont.com.ar> wrote:
On 2=
7/11/20 21:32, Mark Smith wrote:

> 

> On Sat, 28 Nov 2020, 10:32 Tom Herbert, <tom@herbertlan=
d.com 

> <mailto:tom@herbertland.com>> wrote:

[....]

> 

> If hosts start trying to take control of steering traffic by varying t=
he 

> flow label value within a flow, making the network's job harder, a=
nd the 

> network being blamed for the possibility bad network performance that =


> may result, the response will be both quick and easy - network operato=
rs 

> will switch off looking the Flow Label.



THat is *exactly* why I noted that I'm concerned that we seem to be on =


the path of making the FLow Label unusable.

I see the Flow Label as a hint, =
but only a hint, of what the hosts wants from the network. It's useful =
as an expression of what the host would like to have.

However, if that hint causes me any trouble, =
I'll ignore it and then devolve to trying to deliver the packet based j=
ust on its destination address.





Thanks,

-- 

Fernando Gont

e-mail: fernando@gont.com.ar || fgont@si6net=
works.com

PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1










--000000000000ea965605b5251478--


From nobody Sat Nov 28 00:42:37 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6A6203A1001; Sat, 28 Nov 2020 00:42:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.888
X-Spam-Level: 
X-Spam-Status: No, score=-1.888 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, T_SPF_TEMPERROR=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CQYkQX6_Rhn8; Sat, 28 Nov 2020 00:42:32 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [IPv6:2001:67c:27e4::14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id C58E93A0FFA; Sat, 28 Nov 2020 00:42:27 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id D94F7283B75; Sat, 28 Nov 2020 08:42:23 +0000 (UTC)
To: Mark Smith 
Cc: Tom Herbert , IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>    
From: Fernando Gont 
Message-ID: <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar>
Date: Sat, 28 Nov 2020 05:01:50 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 08:42:37 -0000

On 28/11/20 03:42, Mark Smith wrote:
> 
> 
> On Sat, 28 Nov 2020, 14:52 Fernando Gont,  > wrote:
> 
>     On 27/11/20 21:32, Mark Smith wrote:
>      >
>      > On Sat, 28 Nov 2020, 10:32 Tom Herbert,      
>      > >> wrote:
>     [....]
>      >
>      > If hosts start trying to take control of steering traffic by
>     varying the
>      > flow label value within a flow, making the network's job harder,
>     and the
>      > network being blamed for the possibility bad network performance
>     that
>      > may result, the response will be both quick and easy - network
>     operators
>      > will switch off looking the Flow Label.
> 
>     THat is *exactly* why I noted that I'm concerned that we seem to be on
>     the path of making the FLow Label unusable.
> 
> 
> I see the Flow Label as a hint, but only a hint, of what the hosts wants 
> from the network. It's useful as an expression of what the host would 
> like to have.

As you correctly noted, it's supposed to be an identifier of the flow, 
because it's hard to get for the port numbers.

I don't think it can bee seen as ad expression of what the host wants to 
ahve, because the Flow ID has no semantics, and the host doesn't even 
know what's in the network.



> However, if that hint causes me any trouble, I'll ignore it and then 
> devolve to trying to deliver the packet based just on its destination 
> address.

If you *needed* (or could make use of) the FL, then, if it causes 
trouble, you either resort to finding the five-tuple, or quite likely 
end of dropping packets.

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Sat Nov 28 02:15:09 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 121263A09CF; Sat, 28 Nov 2020 02:15:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level: 
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FCLBI1v714-W; Sat, 28 Nov 2020 02:15:06 -0800 (PST)
Received: from mail-oi1-x243.google.com (mail-oi1-x243.google.com [IPv6:2607:f8b0:4864:20::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E8ABD3A09D1; Sat, 28 Nov 2020 02:15:05 -0800 (PST)
Received: by mail-oi1-x243.google.com with SMTP id w15so8581353oie.13; Sat, 28 Nov 2020 02:15:05 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=zbbSXTw6J2WlwBxKU5jp8a9Qh4yv9XE2pdrbPYNjLYw=; b=JYYpTQDlk7uZCVbajlVoKu/lqtTbwV2h9rZpefklzIhNiFu9RZ65W/Hhi9lHGl6VV6 ecdQlOpHLgD5AdAtiAwB1N4otAI4H8fZgMLIxKT+JaS03wLI/SkwyliOaliJK25lEHSc yKN5Vd836wcRqdRlkzgE1LTTxa2l2/U6UwZEaUVegZLhY+7YxScnjRMEpAHshYHzLgzh tNYWvzLmxRrNZ4lWyXt6J5+DC+sf8zM+ulS5bMzHjKeo8JdIiLXxJXOEs6sHw4ZQh5gn Zmt6GO4Ha0Mn+6ywECr9dwrEjQcRGrxC7gpavJJkn4tbaaLbVXlAGuiHiZYZgnHq/u9S //zQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=zbbSXTw6J2WlwBxKU5jp8a9Qh4yv9XE2pdrbPYNjLYw=; b=UOAt08Cl2Uef41CkURhOBB6pU6XmBvBvvf9dOyqdetBehAbUwyiIVeelve0GSXwgmC e7buq0vyAvFLLnGdTF5dCSmn0YHNUN7e7DYaV+PfftDRZxS/PgiWBmOv9rpa7QeKftz9 kfX9oK2zzubEYN2ty8Mf8W2gpFiMpAoIgGViEMoTQ3W0ePGW3eK4rxY8+SZhhKm1bGlt 7FuQYzk0TSy0u+/TX5l9eJa6zNh1B7VEsNxqV3STKFDf+iVxMjWe+/Wwnjw21OxYDIy7 5YWNfVQ5pVF2B7s5ZVsTPLgcnuSPlu8JxDQLSY/6+tbNRlwQ6WZN/VvCAl9LajVS/Hle Dw3Q==
X-Gm-Message-State: AOAM530nOpHkH/U8XrmBoNL0TukiD/S01dbAXXEsAHBxXfFmSB3rsKA1 NtdZIpU19s7jIf9/f5gk85o284VtxtO8YeS6Yls=
X-Google-Smtp-Source: ABdhPJzln2y//E7MWV1jfqXR2XXNBZmg9TR1Zwc5yDCgAbkseGhcHON7c8UiC0mYNpoVilNfMwplm5AXNoupTJyYg9c=
X-Received: by 2002:aca:55cb:: with SMTP id j194mr8397781oib.4.1606558505002;  Sat, 28 Nov 2020 02:15:05 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>     <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar>
In-Reply-To: <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar>
From: Alexander Azimov 
Date: Sat, 28 Nov 2020 13:14:51 +0300
Message-ID: 
To: Fernando Gont 
Cc: Mark Smith , Tom Herbert ,  IPv6 Operations , tcpm 
Content-Type: multipart/alternative; boundary="000000000000006b8e05b5280dd0"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 10:15:08 -0000

--000000000000006b8e05b5280dd0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Gentlemen,

At the very least, one should analyze the case where in #1, a packet
> gets delayed, you change the FL and retransmit, and finally you get a
> response to the original packet (and/or to both packets).

As the suggestion strictly speaks about changing FL upon RTO timeouts I
don't see how it differs from getting two responses in case of RTO without
FL change. Nothing special here.

If you have two links, then, upon change of the FL, you have 50% change
> that the path will be different -- not good.  What would you do if the
> failure continues? -- keep changing the FL?

Yes, and if only one path is affected - you have a chance of hitting a good
path with 1 - (1/2)^n. And it's much better than nothing.

> If a retransmitted SYN goes to another server/anycast point this won't
> bring much harm. At least we haven't experienced such problems in our
> datacenters.
> I think depends on details that are definitively out of the IETF's scope.
> When exactly does the load balancing system assign a session to a given
> server, how much state does that create, what resources does it reserve,
> and when are the state and the reserved resources released?
> Also, in some cases, I could imagine that the data centre will not see a
> problem, but the client might see one that they would not otherwise have
> seen.

I can't say that these details look convincing to me. I also can't agree
that operational considerations are out of IETF's scope. The protocols are
designed with real operations in mind, aren't they?

Steering packets is the network's job. People like me who run the network
> expect to be in charge of that. We have fancy mechanisms that mean
> alternate paths are switched to very quickly.

While not questioning innovative technologies that take care of traffic, a
'good' repair time counts in seconds. 'Bad' repair time, when the control
plane becomes broken or a famous silent drop happens counts in minutes and
requires a network engineer to be summoned. If TCP can fix itself in
milliseconds - it's great. And it is needed by real operations. You also
missing part when something happens outside of your domain, where you can't
instantly summon an engineer. If you tell me that you've been never
affected by the network outage of your peering partner I will be greatly
suprised.



=D1=81=D0=B1, 28 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 11:42, Ferna=
ndo Gont :

> On 28/11/20 03:42, Mark Smith wrote:
> >
> >
> > On Sat, 28 Nov 2020, 14:52 Fernando Gont,  > > wrote:
> >
> >     On 27/11/20 21:32, Mark Smith wrote:
> >      >
> >      > On Sat, 28 Nov 2020, 10:32 Tom Herbert,  >     
> >      > >> wrote=
:
> >     [....]
> >      >
> >      > If hosts start trying to take control of steering traffic by
> >     varying the
> >      > flow label value within a flow, making the network's job harder,
> >     and the
> >      > network being blamed for the possibility bad network performance
> >     that
> >      > may result, the response will be both quick and easy - network
> >     operators
> >      > will switch off looking the Flow Label.
> >
> >     THat is *exactly* why I noted that I'm concerned that we seem to be
> on
> >     the path of making the FLow Label unusable.
> >
> >
> > I see the Flow Label as a hint, but only a hint, of what the hosts want=
s
> > from the network. It's useful as an expression of what the host would
> > like to have.
>
> As you correctly noted, it's supposed to be an identifier of the flow,
> because it's hard to get for the port numbers.
>
> I don't think it can bee seen as ad expression of what the host wants to
> ahve, because the Flow ID has no semantics, and the host doesn't even
> know what's in the network.
>
>
>
> > However, if that hint causes me any trouble, I'll ignore it and then
> > devolve to trying to deliver the packet based just on its destination
> > address.
>
> If you *needed* (or could make use of) the FL, then, if it causes
> trouble, you either resort to finding the five-tuple, or quite likely
> end of dropping packets.
>
> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>


--=20
Best regards,
Alexander Azimov

--000000000000006b8e05b5280dd0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Gentlemen,

At the very least, one should analyze the case where in #1=
, a packet
gets delayed, you change the FL and retransmit, and finally y=
ou get a
response to the original packet (and/or to both packets).=C2=A0=
=C2=A0
As the suggestion strictly speaks about changing FL=
 upon RTO timeouts I don't see how it differs from getting two response=
s in case of RTO without FL change. Nothing special here.

If you have two links,=
 then, upon change of the FL, you have 50% change
that the path will be =
different -- not good.=C2=A0 What would you do if the
failure continues?=
 -- keep changing the FL?
Yes, and if only one path is aff=
ected - you have a chance of hitting a good path with 1 - (1/2)^n. And it&#=
39;s much better than nothing.

> If a retransmitted SYN goes to another server/anycast point this =
won't bring much harm. At least we haven't experienced such problem=
s in our datacenters.
I think depends on details that are definit=
ively out of the IETF's scope. When exactly does the load balancing sys=
tem assign a session to a given server, how much state does that create, wh=
at resources does it reserve, and when are the state and the reserved resou=
rces released?
Also, in some cases, I could imagine that the data centre=
 will not see a problem, but the client might see one that they would not o=
therwise have seen.=C2=A0=C2=A0
I can't say that these=
 details look convincing to me. I also can't agree that operational con=
siderations are out of IETF's scope. The protocols are designed with re=
al operations in mind, aren't they?

Steering packets is the network'=
s job. People like me who run the network expect to be in charge of that. W=
e have fancy mechanisms that mean alternate paths are switched to very quic=
kly.
While not questioning innovative technologies that ta=
ke care of traffic, a 'good' repair time counts in seconds. 'Ba=
d'=C2=A0repair time, when the control plane becomes broken or a famous =
silent drop happens counts in=C2=A0minutes and requires a network=C2=A0engi=
neer to be summoned. If TCP can fix itself in milliseconds - it's great=
. And it is needed by real operations. You also missing part when something=
 happens outside of your domain, where you can't instantly summon an en=
gineer. If you tell me that you've been never affected by the network o=
utage of your peering partner I will be greatly suprised.=C2=A0
<=
br>


=D1=81=D0=B1, 28 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=
=B3. =D0=B2 11:42, Fernando Gont <fernando@gont.com.ar>:
On 28/11/20 03:42, Mark Smith wrote:

> 

> 

> On Sat, 28 Nov 2020, 14:52 Fernando Gont, <fernando@gont.com.ar 

> <mailto:f=
ernando@gont.com.ar>> wrote:

> 

>=C2=A0 =C2=A0 =C2=A0On 27/11/20 21:32, Mark Smith wrote:

>=C2=A0 =C2=A0 =C2=A0 >

>=C2=A0 =C2=A0 =C2=A0 > On Sat, 28 Nov 2020, 10:32 Tom Herbert, <<=
a href=3D"mailto:tom@herbertland.com" target=3D"_blank">tom@herbertland.com=


>=C2=A0 =C2=A0 =C2=A0<mailto:tom@herbertland.com>

>=C2=A0 =C2=A0 =C2=A0 > <mailto:tom@herbertland.com <mailto:tom@herbertland.com>>>=
 wrote:

>=C2=A0 =C2=A0 =C2=A0[....]

>=C2=A0 =C2=A0 =C2=A0 >

>=C2=A0 =C2=A0 =C2=A0 > If hosts start trying to take control of stee=
ring traffic by

>=C2=A0 =C2=A0 =C2=A0varying the

>=C2=A0 =C2=A0 =C2=A0 > flow label value within a flow, making the ne=
twork's job harder,

>=C2=A0 =C2=A0 =C2=A0and the

>=C2=A0 =C2=A0 =C2=A0 > network being blamed for the possibility bad =
network performance

>=C2=A0 =C2=A0 =C2=A0that

>=C2=A0 =C2=A0 =C2=A0 > may result, the response will be both quick a=
nd easy - network

>=C2=A0 =C2=A0 =C2=A0operators

>=C2=A0 =C2=A0 =C2=A0 > will switch off looking the Flow Label.

> 

>=C2=A0 =C2=A0 =C2=A0THat is *exactly* why I noted that I'm concerne=
d that we seem to be on

>=C2=A0 =C2=A0 =C2=A0the path of making the FLow Label unusable.

> 

> 

> I see the Flow Label as a hint, but only a hint, of what the hosts wan=
ts 

> from the network. It's useful as an expression of what the host wo=
uld 

> like to have.



As you correctly noted, it's supposed to be an identifier of the flow, =


because it's hard to get for the port numbers.



I don't think it can bee seen as ad expression of what the host wants t=
o 

ahve, because the Flow ID has no semantics, and the host doesn't even <=
br>
know what's in the network.







> However, if that hint causes me any trouble, I'll ignore it and th=
en 

> devolve to trying to deliver the packet based just on its destination =


> address.



If you *needed* (or could make use of) the FL, then, if it causes 

trouble, you either resort to finding the five-tuple, or quite likely 

end of dropping packets.



Thanks,

-- 

Fernando Gont

e-mail: fernando@=
gont.com.ar || fgont@si6networks.com

PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1







_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




-- 
Best regards,
Alexander Azi=
mov


--000000000000006b8e05b5280dd0--


From nobody Sat Nov 28 10:09:21 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1667F3A0E08; Sat, 28 Nov 2020 10:09:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.499
X-Spam-Level: 
X-Spam-Status: No, score=-1.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.399, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KcEylFFdyPxD; Sat, 28 Nov 2020 10:09:17 -0800 (PST)
Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9880F3A0DFC; Sat, 28 Nov 2020 10:09:15 -0800 (PST)
Received: from [IPv6:2a02:8109:1140:c3d:813c:9bb5:ab11:9982] (unknown [IPv6:2a02:8109:1140:c3d:813c:9bb5:ab11:9982]) (Authenticated sender: lurchi) by drew.franken.de (Postfix) with ESMTPSA id 6B18372246344; Sat, 28 Nov 2020 19:09:11 +0100 (CET)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.20.0.2.21\))
From: Michael Tuexen 
In-Reply-To: 
Date: Sat, 28 Nov 2020 19:09:08 +0100
Cc: Fernando Gont , IPv6 Operations , tcpm 
Content-Transfer-Encoding: quoted-printable
Message-Id: <80E049A2-534C-4D89-B67D-4384A6CFA8B6@lurchi.franken.de>
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar> 
To: Tom Herbert 
X-Mailer: Apple Mail (2.3654.20.0.2.21)
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 18:09:20 -0000

> On 28. Nov 2020, at 00:31, Tom Herbert  wrote:
>=20
> On Fri, Nov 27, 2020 at 1:41 PM Fernando Gont  =
wrote:
>>=20
>> Hello, Tom,
>>=20
>> On 25/11/20 16:35, Tom Herbert wrote:
>>> Hi Fernando, comments in line...
>>>=20
>>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont =
 wrote:
>>>>=20
>>>> Hi, Tom,
>>>>=20
>>>> On 24/11/20 16:43, Tom Herbert wrote:
>>>> [....]
>>>>> Modulating the flow label is a means to affect the routing of =
packets
>>>>> through the network that uses flow labels as input to the ECMP =
hash.
>>>>=20
>>>> What's the point?
>>>>=20
>>>> 1) You cannot tell *if* the FL is being used.
>>>>=20
>>> Generally true, but in a limited domain this information could be
>>> discerned. I'd note that it's also generally true that we don't know
>>> if there is a load balancer or stateful firewall in the path that
>>> requires consistent routing, but in a limited domain we could know
>>> that also.
>>=20
>> Exactly. So my take is that the drawbacks for the general case =
outweigh
>> the benefits of the specific case.
>>=20
>>=20
>>=20
>>>> 2) Changing the FL does not necessarily mean that packets will =
employ a
>>>> different link.
>>>=20
>>> It's an opportunistic mechanism. If a connection is failing and we =
get
>>> a better path that fixes it by simply changing the flow label then
>>> what's the harm?
>>=20
>> Complexity, and the possible negative implications for the general =
case.
>>=20
>> That said, why don't you fix the problem where you should (i.e., =
routing)?
>>=20
>>=20
>>=20
>>>> 3) If the network is failing, shouldn't you handle this via =
routing?
>>>>=20
>>> Sure, but then that requires an out of band feedback loop from a TCP
>>> implementation to the network infrastructure to indicate there is a
>>> problem and then the network needs to respond.
>>=20
>> Not really. TCP does not deal with paths (not should it). If the =
network
>> detects a link is failing, it should route around the problem.
>>=20
>=20
> Fernando,
>=20
> All we know in this case is that *TCP* has detected a connection is
> failing and there may be a potential problem with a path, we don't
Hi Tom,

it is perfectly fine that if a TCP connection fails and the application
triggers a setup of a new TCP connection, the new TCP connections
uses a flowlabel different from the one used for the old TCP connection.
However, a single RTO does not indicate a connection failure.
Normally you would need several consecutive RTOs to happen before TCP
declares the connection to be failed.

Best regards
Michael
> know that the network has detected any problems like some link is
> failing or will ever detect any problem. So in such a scenario, what
> recourse does the host have to try to salvage its connections? If the
> answer is that the host should always patiently wait for the network
> to figure out what is happening, then that's going to be a hard sell
> to application and host stack developers (remember the network is a
> black box from their perspective that they really don't trust). If the
> answer is that the host is supposed to inform the network of issues
> with it's connections and then host and network work together to solve
> then that's great; but then what is the general protocol that has been
> established to facilitate that and solved the problem of how hosts and
> network work together in concert to solve user's problems?
>=20
> Tom
>=20
>>=20
>>=20
>>> That's significant
>>> infrastructure and higher reaction time than doing something in TCP
>>> and IP. Think of modulating the flow label is an inexpensive form of
>>> source routing within a limited domain that doesn't need any
>>> infrastructure or heavyweight protocols or something like segment
>>> routing.
>>=20
>> Downsides:
>>=20
>> * It breaks load balancing
>> * It depends on the source doing something -- but you don't =
necessarily
>> control the source -- and you cannot tell if the source implements it
>> * It doesn't solve the problem for nodes that do not behave as =
expected
>>=20
>> My take is that if you really care about this, you're already (or
>> should!) be solving the problem in a different (and more general) =
way.
>>=20
>>=20
>>=20
>>>>> The basic idea is that the flow label associated with a connection =
is
>>>>> randomly changed when the stack observes that the connection is
>>>>> failing (e.g. and an RTO). There is nothing in the specs that =
prevents
>>>>> this since the source is at liberty to set the flow label as it =
sees
>>>>> fit.
>>>>=20
>>>> The FL is expected to remain constant for the life of a flow. A
>>>> retransmitted packet is part of the same flow as the
>>>> originally-transmitted packet. So this seems to be contradicting =
the
>>>> very specification of the FL.
>>>>=20
>>>> For instance, If a RTO for a flow causes the FL to change, then one =
may
>>>> possibly argue that the FL is not naming/labeling what is =
said/expected
>>>> to be anming/labeling.
>>>=20
>>> Specifically, RFC6437 states:
>>>=20
>>> "It is therefore RECOMMENDED that source hosts support the flow =
label
>>> by setting the flow label field for all packets of a given flow to =
the
>>> same value chosen from an approximation to a discrete uniform
>>> distribution."
>>>=20
>>> So that is clearly a just recommendation, and not a requirement (and
>>> definitely not a MUST). Furthermore, RFC6437 states:
>>>=20
>>> "A forwarding node MUST either leave a non-zero flow label value
>>> unchanged or change it only for compelling operational security
>>> reasons as described in Section 6.1."
>>=20
>> My concern is that we may leave the FL unusable.
>>=20
>>=20
>>=20
>>=20
>>> So there's no guarantee in the protocol specs that flow labels are
>>> consistent for the life of the connection, which means that the
>>> network cannot assume that and thus it would be incorrect if the
>>> network tried to enforce flow label consistency as a protocol
>>> requirement.
>>=20
>> You're probably right in that respect. And, in retrospective, I think =
it
>> would have been better if we had specified the FL to be unmutable.
>>=20
>>=20
>>=20
>>> As I said, it is prudent to try to be consistent with
>>> flow labels and the default behavior in Linux should be changed,
>>=20
>> Sorry: is the current Linux behavior to modulate the FL?
>>=20
>>=20
>>> however I do not believe there's a valid claim of non-conformance =
that
>>> motivates removal of the feature that is already deployed.
>>=20
>> I personally consider the changing the FL for a flow to be a bug. =
That
>> said, if the default setting is the correct/compliant behavior, at =
the
>> end of the day we're all adults, and whoever overrides the default
>> behavior is supposed to know what he/she does -- and if they don't, =
they
>> get what they deserve  ;-)
>>=20
>> So, in that light, I wouldn't push to remove the optional behavior,
>> *provided* the default behavior is correct.
>>=20
>> Thanks,
>> --
>> Fernando Gont
>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>=20
>>=20
>>=20
>=20
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm


From nobody Sat Nov 28 10:27:52 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5173A3A0E8C for ; Sat, 28 Nov 2020 10:27:50 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7UhlkCuz03Px for ; Sat, 28 Nov 2020 10:27:47 -0800 (PST)
Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A6D2A3A0E85 for ; Sat, 28 Nov 2020 10:27:46 -0800 (PST)
Received: by mail-ej1-x642.google.com with SMTP id a16so12426679ejj.5 for ; Sat, 28 Nov 2020 10:27:46 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=wWl/NMyNYxy387O+iIu6O4b8CdadQrZa/jeMx3Om5tg=; b=dNgTTawMw7VpBXjsMXb8333PuiTIN6tthkD5Ce1+L1hE+f5KpmZwout31RmBskI+1o b0ZTql8dcOy1NJ/B1mZwKFgGlxCxSvikm0sOY+/vH6j8EhZUoHXNI2Ebc7e0hp2BG6z/ y91bbKqEkVXzRxR4JLSuLFnYJUBZeHZ2Y7925uVMA+ZAttlrbCXr9AlTXcIeSdDvaQVc aWmEo3l4sgMgugqf/VB89gX24ZpKHjo2u8R0St0XsGwz9cuhAGlfFXbhcHXFblYxZFBd XeMmsFsEQNn1cP2JrZG7i8SRFEzqb0ul9AUXmqlgLewRl9NHhAoLS3xiXOb9NODVC8Wm ICEA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=wWl/NMyNYxy387O+iIu6O4b8CdadQrZa/jeMx3Om5tg=; b=EoExjlKsCeYxWr9PJucMpAWgg1+EMCo4mCJ63yS0IYdZJa9YnyLBwRlKHdbDMC+Rzf /D7xG8wwbEvLdtF7TeeSxxA3Aky7OU6rHesV6aakAq/n3VbZ6h0jpCRb5kYmTjwrJtxH V4dnBF+B8wNYinrr7N+a7FXZcNZVmK5LhSe1NbBwuckVqPCYB8mrOVZL0NXPySkkfl4n LucMnSbV2O+ZOI3ouXde6jiAka4ooTm/CARIAOz4Tb66dW9KHsaLVfvaUlE9KdUfrRh1 4Pb1dxJkrV9Tzu/CHMs+6u7Y92zSQLbDqB9dZ7955JYCFTSbAnQ1W/hHUQZGuRToqcuH GdzA==
X-Gm-Message-State: AOAM5322pRCCd6HdxEEbT3tO3sXIbXL9sQiq63YHyN18jYvha1Mw53Sl f9/+U/3RlQMHiGrKSslQMgRioxC5P0w5F/3YckYLOw==
X-Google-Smtp-Source: ABdhPJzZZSeTYj48bvJ9nSIQ7zcFM6+A+2Wb+/H6bNI7joGqmXwJYywQ0bf2CQxfAQ178s1cPq0FN2S0iiRn/6dTPlg=
X-Received: by 2002:a17:906:f153:: with SMTP id gw19mr1182793ejb.272.1606588064811;  Sat, 28 Nov 2020 10:27:44 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>     <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar> 
In-Reply-To: 
From: Tom Herbert 
Date: Sat, 28 Nov 2020 11:27:33 -0700
Message-ID: 
To: Alexander Azimov 
Cc: Fernando Gont , Mark Smith ,  IPv6 Operations , tcpm 
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sat, 28 Nov 2020 18:27:50 -0000

On Sat, Nov 28, 2020 at 3:15 AM Alexander Azimov  wro=
te:
>
> Gentlemen,
>
>> At the very least, one should analyze the case where in #1, a packet
>> gets delayed, you change the FL and retransmit, and finally you get a
>> response to the original packet (and/or to both packets).
>
> As the suggestion strictly speaks about changing FL upon RTO timeouts I d=
on't see how it differs from getting two responses in case of RTO without F=
L change. Nothing special here.
>
>> If you have two links, then, upon change of the FL, you have 50% change
>> that the path will be different -- not good.  What would you do if the
>> failure continues? -- keep changing the FL?
>
> Yes, and if only one path is affected - you have a chance of hitting a go=
od path with 1 - (1/2)^n. And it's much better than nothing.
>
>> > If a retransmitted SYN goes to another server/anycast point this won't=
 bring much harm. At least we haven't experienced such problems in our data=
centers.
>> I think depends on details that are definitively out of the IETF's scope=
. When exactly does the load balancing system assign a session to a given s=
erver, how much state does that create, what resources does it reserve, and=
 when are the state and the reserved resources released?
>> Also, in some cases, I could imagine that the data centre will not see a=
 problem, but the client might see one that they would not otherwise have s=
een.
>
> I can't say that these details look convincing to me. I also can't agree =
that operational considerations are out of IETF's scope. The protocols are =
designed with real operations in mind, aren't they?
>
>> Steering packets is the network's job. People like me who run the networ=
k expect to be in charge of that. We have fancy mechanisms that mean altern=
ate paths are switched to very quickly.
>
> While not questioning innovative technologies that take care of traffic, =
a 'good' repair time counts in seconds. 'Bad' repair time, when the control=
 plane becomes broken or a famous silent drop happens counts in minutes and=
 requires a network engineer to be summoned. If TCP can fix itself in milli=
seconds - it's great. And it is needed by real operations. You also missing=
 part when something happens outside of your domain, where you can't instan=
tly summon an engineer. If you tell me that you've been never affected by t=
he network outage of your peering partner I will be greatly suprised.
>
+1

I would also point out this feature has been in deployment for at
least five years in at least two hyperscalers-- there has been no
major meltdown. We'll make the feature opt-in because we may be
exposed to non-conformant devices in the network that make incorrect
assumptions, but I have yet to see any convincing argument that we
should remove the feature becauses it's fundamentally harmful or not
providing benefits.

As for the mentality of "the network will fix the problem", that's
obviously an easy statement to make from a router vendor point of
view, but that luxury simply doesn't match the realities of how host
stacks are implemented deployed at global scale. This starts with the
fact that there is no such thing as "the network"; Linux stack for
instance needs to run on every network on Earth and I have no idea
which of these networks provides adequate security, fault detection
and repair, are well managed, or even have devices that conform to the
standards (non-conformant network devices are common). The only
recourse we, i.e. host side developers, have is to stick with the
least common denominator in the general case, and allow optional, but
still protocol conformant, features in limited use cases such as
limited domains (hence why the flow label modulation feature should be
optional as we've already established in this thread).

Tom

>
>
> =D1=81=D0=B1, 28 =D0=BD=D0=BE=D1=8F=D0=B1. 2020 =D0=B3. =D0=B2 11:42, Fer=
nando Gont :
>>
>> On 28/11/20 03:42, Mark Smith wrote:
>> >
>> >
>> > On Sat, 28 Nov 2020, 14:52 Fernando Gont, > > > wrote:
>> >
>> >     On 27/11/20 21:32, Mark Smith wrote:
>> >      >
>> >      > On Sat, 28 Nov 2020, 10:32 Tom Herbert, > >     
>> >      > >> wrot=
e:
>> >     [....]
>> >      >
>> >      > If hosts start trying to take control of steering traffic by
>> >     varying the
>> >      > flow label value within a flow, making the network's job harder=
,
>> >     and the
>> >      > network being blamed for the possibility bad network performanc=
e
>> >     that
>> >      > may result, the response will be both quick and easy - network
>> >     operators
>> >      > will switch off looking the Flow Label.
>> >
>> >     THat is *exactly* why I noted that I'm concerned that we seem to b=
e on
>> >     the path of making the FLow Label unusable.
>> >
>> >
>> > I see the Flow Label as a hint, but only a hint, of what the hosts wan=
ts
>> > from the network. It's useful as an expression of what the host would
>> > like to have.
>>
>> As you correctly noted, it's supposed to be an identifier of the flow,
>> because it's hard to get for the port numbers.
>>
>> I don't think it can bee seen as ad expression of what the host wants to
>> ahve, because the Flow ID has no semantics, and the host doesn't even
>> know what's in the network.
>>
>>
>>
>> > However, if that hint causes me any trouble, I'll ignore it and then
>> > devolve to trying to deliver the packet based just on its destination
>> > address.
>>
>> If you *needed* (or could make use of) the FL, then, if it causes
>> trouble, you either resort to finding the five-tuple, or quite likely
>> end of dropping packets.
>>
>> Thanks,
>> --
>> Fernando Gont
>> e-mail: fernando@gont.com.ar || fgont@si6networks.com
>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>>
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> tcpm@ietf.org
>> https://www.ietf.org/mailman/listinfo/tcpm
>
>
>
> --
> Best regards,
> Alexander Azimov


From nobody Sat Nov 28 21:53:35 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9140F3A11AA; Sat, 28 Nov 2020 21:53:32 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level: 
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 04Z64h8ti81m; Sat, 28 Nov 2020 21:53:28 -0800 (PST)
Received: from fgont.go6lab.si (fgont.go6lab.si [91.239.96.14]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9C9DC3A11A9; Sat, 28 Nov 2020 21:53:13 -0800 (PST)
Received: from [IPv6:2800:810:464:8164:d564:510f:6fb3:4524] (unknown [IPv6:2800:810:464:8164:d564:510f:6fb3:4524]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by fgont.go6lab.si (Postfix) with ESMTPSA id 69F7B280878; Sun, 29 Nov 2020 05:53:09 +0000 (UTC)
To: Tom Herbert , Alexander Azimov 
Cc: Mark Smith , IPv6 Operations , tcpm 
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>     <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar>  
From: Fernando Gont 
Message-ID: 
Date: Sun, 29 Nov 2020 02:53:01 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: 
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sun, 29 Nov 2020 05:53:33 -0000

Tom,

On 28/11/20 15:27, Tom Herbert wrote:
[....]
>> While not questioning innovative technologies that take care of traffic, a 'good' repair time counts in seconds. 'Bad' repair time, when the control plane becomes broken or a famous silent drop happens counts in minutes and requires a network engineer to be summoned. If TCP can fix itself in milliseconds - it's great. And it is needed by real operations. You also missing part when something happens outside of your domain, where you can't instantly summon an engineer. If you tell me that you've been never affected by the network outage of your peering partner I will be greatly suprised.
>>
> +1
> 
> I would also point out this feature has been in deployment for at
> least five years in at least two hyperscalers-- there has been no
> major meltdown. We'll make the feature opt-in because we may be
> exposed to non-conformant devices in the network that make incorrect
> assumptions, but I have yet to see any convincing argument that we
> should remove the feature becauses it's fundamentally harmful or not
> providing benefits.

It breaks FL-based load-sharing.



> As for the mentality of "the network will fix the problem", that's
> obviously an easy statement to make from a router vendor point of
> view, but that luxury simply doesn't match the realities of how host
> stacks are implemented deployed at global scale.

Not sure what you mean. We rely on that for the current Internet (hosts 
do not do source-routing).



> This starts with the
> fact that there is no such thing as "the network"; Linux stack for
> instance needs to run on every network on Earth and I have no idea
> which of these networks provides adequate security, fault detection
> and repair, are well managed, or even have devices that conform to the
> standards (non-conformant network devices are common). The only
> recourse we, i.e. host side developers, have is to stick with the
> least common denominator in the general case, and allow optional, but
> still protocol conformant, features in limited use cases such as
> limited domains (hence why the flow label modulation feature should be
> optional as we've already established in this thread).

Modulo the comments I've already made, I'm afraid we might be in the 
path of making the FL unusable...

Thanks,
-- 
Fernando Gont
e-mail: fernando@gont.com.ar || fgont@si6networks.com
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1




From nobody Sun Nov 29 08:26:18 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B69043A0990 for ; Sun, 29 Nov 2020 08:26:16 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.897
X-Spam-Level: 
X-Spam-Status: No, score=-1.897 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=herbertland-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5_Q6Tl8SKekl for ; Sun, 29 Nov 2020 08:26:13 -0800 (PST)
Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 007243A098E for ; Sun, 29 Nov 2020 08:26:12 -0800 (PST)
Received: by mail-ed1-x535.google.com with SMTP id v22so11870968edt.9 for ; Sun, 29 Nov 2020 08:26:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=herbertland-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4nl8RWUyjAga5aJnJqFHpeZR5YaJoGcPySw6TFLDQYM=; b=RhHJn99s6A/08Hcn9dK+jJifriDjIwn5CxKmNsgO3AWBlhdi5+55F21nWyvHrhfzVo lK8LbKJ3y4Af8zMf1/53SM4/Zov+FzDQrdGyY1d/LyhYrjSHLIpCt1I5O6kcGST+kyxF YtxWdhDV+uWRDymRt+d2fwggj3n4ZnlefLPo7nn7r1I58aZAhUv6du7c2jWj+EdJEdsM BMmPKT+veYl0YgAQcpG1svYesFGz53vGzy8Yk0ZDAOt1RIMWuLMcv1WS5yNmWiY4kwZ7 m9M4F94RymBZyz3SF1Y9tU7ZryFtzfeQ7pRYTK0tkxAI/fGC2G6w5vyTGpyqcJ3U0qkL 0E1g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4nl8RWUyjAga5aJnJqFHpeZR5YaJoGcPySw6TFLDQYM=; b=Rs7ZeBGt5XNYzP9y30I+RwGzWEV9pGUDd6ZJ761WikCJdEHvGOteoezCbkTaEMy2Be Pk7dwyfRbKORNDWHe4pgUErJybcG6chKIRbj/8r1E40FP77YWTOyM/qY1qz/Kaci8LQG kP9MJHsuWsP9llm4+Smwsb2f4lgbso5F9Jw+1267XrQ2rH+mOd0oGA0p9KCgur9qcAsx 5vq8yRpG5dT9pLG9P3v9YkOd2Ik+rIas3sOyXlcRtV9qd5LVnaT2qFfdE/gwmHOUs6O+ H75l5CdPXlLwUrWoAbczJG+9hnyZ0ABhmng4wzWq5Hvx+vHj4hQWF0qod0q7cDnEu7z3 GtkA==
X-Gm-Message-State: AOAM532HExKzvKuzQNHUpDNbyViRVzVl8uYnvXK63kzWfPXUyeSZDqta UfHdAdNY3QGjEyZlw4VCNkC7gITI1bmkm2a7a8TdxQ==
X-Google-Smtp-Source: ABdhPJx9ImKGbB+SwsnYzYtsUAtryedC9qBssWQQMBqqzsQ92sbx0l9c//dzyrGoO1RK3B4tTNnHfEirb+P78EsJFVU=
X-Received: by 2002:aa7:dd52:: with SMTP id o18mr2084264edw.177.1606667171298;  Sun, 29 Nov 2020 08:26:11 -0800 (PST)
MIME-Version: 1.0
References:    <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar>  <96b6d04b-e5bb-ba79-0281-e9599109be95@gont.com.ar>     <0d38980a-f1c5-fac5-a9b1-0711d61353d1@gont.com.ar>   
In-Reply-To: 
From: Tom Herbert 
Date: Sun, 29 Nov 2020 09:25:59 -0700
Message-ID: 
To: Fernando Gont 
Cc: Alexander Azimov , Mark Smith ,  IPv6 Operations , tcpm 
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
Subject: Re: [tcpm] [v6ops] Flow Label Load Balancing
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Sun, 29 Nov 2020 16:26:17 -0000

> Modulo the comments I've already made, I'm afraid we might be in the
> path of making the FL unusable...
>

Fernanda,

I really don't understand how this makes the flow label "unusable", to
the contrary this is a useful case with good effect and justifies the
work in the host to set the field. In any case, it would be great if
you could provide the draft with the normative requirements that would
make the flow label "usable".

Please include requirements for both hosts and network nodes. If there
is going to the be a requirement that hosts MUST make a 1:1 mapping to
a transport connection and the flow label MUST be consistent for the
life of connection, then I would expect there to also be a requirement
that the flow label MUST be immutable by all devices in the path-- in
particular the requirement of RFC6437 would need to be updated: "A
forwarding node MUST either leave a non-zero flow label value
unchanged or change it only for compelling operational security
reasons..."-- if hosts are not allowed to change a field they are
primarily responsible for setting, then the network should not be
allowed to modify it either.

Additionally, if the underlying design principle is that flow
identification in a packet is immutable and persistent, then I'd like
to see the discussion on how that rectifies with NAT where
intermediate nodes routinely change IP addresses and transport port
numbers without the host endpoints' knowledge-- unlike the flow label,
port numbers are formally part of the transport connection
identification, so changing them in flight is far more invasive and
potentially harmful than changing the flow label (we've seen far more
problems caused by NAT than flow label due to NAT state evictions,
packets of a flow being re-routed to not hit the same NAT device,
etc.).

Tom


> Thanks,
> --
> Fernando Gont
> e-mail: fernando@gont.com.ar || fgont@si6networks.com
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>


From nobody Mon Nov 30 10:12:44 2020
Return-Path: 
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8801E3A0FCA for ; Mon, 30 Nov 2020 10:12:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.087
X-Spam-Level: 
X-Spam-Status: No, score=-2.087 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EvsOkL8llVBS for ; Mon, 30 Nov 2020 10:12:40 -0800 (PST)
Received: from mail-qk1-x72c.google.com (mail-qk1-x72c.google.com [IPv6:2607:f8b0:4864:20::72c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5BB6A3A0FBB for ; Mon, 30 Nov 2020 10:12:40 -0800 (PST)
Received: by mail-qk1-x72c.google.com with SMTP id q5so11701395qkc.12 for ; Mon, 30 Nov 2020 10:12:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;  h=mime-version:references:in-reply-to:from:date:message-id:subject:to;  bh=z8yQ6nVGonb6lZybt/hO/u7botVkOhF4rA2S7eG582o=; b=sSMgBVTYHxPa3SCFiV/6ZIYbwNZwFz/NLOu69mfAju5kAr8m/2AnRnAaoddQouphnZ zRy25bj0qbbSIjBvKK50EpWAoqMyuniDG0gYEluh+FI0SdN7rDyMFqSczvuJLFfWGKgX 9cbM/Uy8tMkKmlWPZjB8InDpPaak4Ps0DUqo+Wuev/4JMcy45ey1RUjaB2Cv7Gjqas/q VpQU2xZn3c085WM3qagCMgV2gdpGx9M3tdbOuzVTkQPoP/h1qhwfGSB5kX82tUBTCH8C sl9DiAkLquGRToweHjsF1w5/HEHlsddj6edHW2zd1Zi/g42J5PpMZKnpfSzZ27aP8IYk kfgg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=z8yQ6nVGonb6lZybt/hO/u7botVkOhF4rA2S7eG582o=; b=Ia+RkzwgqdbBdE+x8sNbagXhiBpEFRYyvvTmd30zq6i2KkHW4tqAce/9ZCUL/7CID3 KAksjaTK6JHq8vIeJeZ2ApxMk2RE+7FRXfeiWkDl/uElcM4cmSRMnkQVxD0VmAzUEOQl Sy8ORonBnMPUlGC2obn1DpcGOgRAJ7gOubZT6820RucMpXUarizctAWkY6YIcY/GqLta j9Ae3id7k6GLFALeBv+IfPl1qfAp1xNhKemd4lN6OWLIJie7/xe+kVEpnxFwRqJilvOt lxVcrLMyYBuNrbkk7gx+tQsd2A8cJ+ymTEE9FiN1oZsX9o8sKKcoGgXLp/ERp5HhMBWY ubew==
X-Gm-Message-State: AOAM531LK4WmrG4rd8gX8h9ZtjPlKYlIOC+3Djna8HP9Enh8qw1UxnIr brYyc8Xf6ALellQN0cee1voaMElJ13+jxkxYUGlgkGXn
X-Google-Smtp-Source: ABdhPJyDSlIQdX2hyGzsAABIfWd35J6GYneSY8kF5h1PSv60f0TM082Xl1UrFBNAqQoQSEFRE/+GvRgiaD3FXgvPPRk=
X-Received: by 2002:a05:620a:11a4:: with SMTP id c4mr24402672qkk.8.1606759959033;  Mon, 30 Nov 2020 10:12:39 -0800 (PST)
MIME-Version: 1.0
References: 
In-Reply-To: 
From: Yoshifumi Nishida 
Date: Mon, 30 Nov 2020 10:12:27 -0800
Message-ID: 
To: tcpm IETF list 
Content-Type: multipart/alternative; boundary="00000000000098fa1605b556f45e"
Archived-At: 
Subject: [tcpm] Request for feedback on WG Adoption of RFC6937bis
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group 
List-Unsubscribe: , 
List-Archive: 
List-Post: 
List-Help: 
List-Subscribe: , 
X-List-Received-Date: Mon, 30 Nov 2020 18:12:43 -0000

--00000000000098fa1605b556f45e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi folks,

This mail starts a WG adoption call for RFC6937bis draft.
As we've already seen positive supports for this work, we are not asking
people who already expressed their opinions this time. (but, if you want to
do it for some reason, please go ahead)
However, if you haven't done it yet, especially if you're against it, we
would like to encourage you to provide feedback.

We are thinking that the intended status of the doc will be PS based on the
feedback we've seen so far. But, if you have opinions on it, please let us
know.

Thank you so much,
--
Yoshi on behalf of tcpm co-chairs


On Wed, Nov 18, 2020 at 7:56 PM Matt Mathis  wrote:

> The authors of PRR would like to update PRR to Proposed Standard status.
> This entails introducing a new document as an tcpm work item.
>
> *Please indicate (non) support and/or comment.*
>
> For more details see the tcpm meeting materials from IETF 109
> minutes:
> https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00
> slides: https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00
>
> There were about four "I support this work" remarks at the mic (not
> recorded in the minutes), and about as many in the Meetecho chat.
>
> Abridged IETF/tcpm/PRRbis slides:
> --
> PRR  recap (RFC6937 experimental)
> PRR is a special congestion control effective only during fast recovery
>
>    - When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.g=
.
>    loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
>    - When inflight < ssthresh, send at the same or twice the
>    delivery_rate (more later)
>    - Used by all congestion control modules in Linux during fast recovery
>       - Can be more dominant than the actual C.C. for lossy flows that=E2=
=80=99re
>       in fast recovery constantly (e.g. video streaming through policers)
>
> --
> Current Status
>
>    -
>
>    PRR is widely deployed
>    -
>
>       At least three major OSs: Linux, Windows, (NetFlix) BSD
>       -
>
>       Vast majority of Web traffic for years
>       -
>
>    No changes to algorithms published in RFC 6937
>    -
>
>       PRR-CRB - Conservative Reduction Bound - strict packet conversion
>       during loss recovery
>       -
>
>       PRR-SSRB - Slowstart Reduction Bound - one extra segment per ACK
>       during loss recovery
>       -
>
>    2015 Heuristic to dynamically select which reduction bound
>    -
>
>       Only use PRR-SSRB when making good forward progress
>       -
>
>          ACKs that advanced snd.una and report no new losses
>          -
>
>       Resolves some pathological cases with token bucket policers
>       -
>
>          CC estimates ssthresh before it can possibly measure the token
>          rate
>          -
>
>          The heuristic makes the best of a bad situation
>
> --
> Tentative path forward
>
>    -
>
>    Adopt as a tcpm work item
>    -
>
>    Update the text
>    -
>
>       Normative RFC 2119 language
>       -
>
>       Add MAY use the heuristic...
>       -
>
>       Trim redundant and obsolete language
>       -
>
>          RFC 6937 repeats itself and is much longer than necessary
>          -
>
>          Focus on what an implementer needs to know
>          -
>
>          Use non-normative references to RFC 6937 for prior measurement
>          work, etc
>
> Thanks,
> --MM--
> The best way to predict the future is to create it.  - Alan Kay
>
> We must not tolerate intolerance;
>        however our response must be carefully measured:
>             too strong would be hypocritical and risks spiraling out of
> control;
>             too weak risks being mistaken for tacit approval.
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>

--00000000000098fa1605b556f45e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable


Hi folks,

This mail starts a=
 WG adoption call for RFC6937bis draft.
As we've already seen=
 positive supports for this work, we are not asking people who already expr=
essed their opinions this time. (but, if you want to do it for some reason,=
 please go ahead)=C2=A0
However, if you haven't done it yet, =
especially if you're against it, we would like to encourage you to prov=
ide feedback.

We are thinking that the intended st=
atus of the doc will be PS based on the feedback we've seen so far. But=
, if you have opinions on it, please let us know.

=
Thank you so much,
--
Yoshi on behalf of tcpm co-chairs=



On Wed, Nov 18, 2020 at 7:56 PM Matt Mathis <mattmathis=
=3D40google.com@dmarc.ietf.o=
rg> wrote:
The authors of PRR would li=
ke to update PRR to Proposed Standard status.=C2=A0 This entails introducin=
g a new document as an tcpm work item.

Please indicate (non)=
 support and/or comment.

<=
/font>
For more details see the tcpm meet=
ing materials from IETF 109

There were about four "I support this work" rem=
arks at the mic (not recorded in the minutes), and about as many in the Mee=
techo chat.

Abridged IETF/=
tcpm/PRRbis slides:
--
PRR =C2=A0recap (RFC6937 experimental)
PRR is a special congestion control =
effective only during fast recovery
  • When inflight >=3D ssthresh, send at loss_beta*rate_before_loss (e.g.=
 loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 for Cubic)
    • <=
li>When inflight < ssthresh, send at the same or=
 twice the delivery_rate (more later)
  • Used by all congestion control modules in Linux during fast recovery
    • Can be more dominant than the actual=
 C.C. for lossy flows that=E2=80=99re in fast recovery constantly (e.g. vid=
eo streaming through policers)
--
Current Status
  • <=
span style=3D"background-color:transparent;font-variant-numeric:normal;font=
-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">PR=
R is widely deployed
    • At least three major OSs: Linux, Windows, =
(NetFlix) BSD
    • Vast majority of Web =
traffic for years
  • <=
p style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt">No changes t=
o algorithms published in RFC 6937
    • PRR-CRB - Conservative Reduc=
tion Bound - strict packet conversion during loss recovery=C2=A0
      =

    • PRR-SSRB - Slowstart Reduction Bound - one ex=
tra segment per ACK during loss recovery
  • 2015 Heuristic to dynamically select which reduction bound<=
/p>
    • Only use PRR-SSRB when making good forward progress
    • ACKs t=
hat advanced snd.una and report no new losses
  • Resolves some pathological cases with token bucket policers=

    • CC estimates ssthresh before it can possibly measure the token=
 rate
    • The heuristic makes the best=
 of a bad situation
--
Tentative path forward=C2=A0
  • Adopt as a tcpm work item
  • Update the text
  • Normative RFC 2119 language
  • Add MAY use the heuristic...
    <=
/li>
  • Trim redundant and ob=
solete language
    • RFC 6937 repeats itse=
lf and is much longer than necessary
    • Focus on what an implementer needs to know
    • Use non-norm=
ative references to RFC 6937 for prior measurement work, etc
=

Th=
anks,
--MM--
The best way to predict=
 the future is to create it. =C2=A0- Alan Kay

We must not tolerate i=
ntolerance;
=C2=A0 =C2=
=A0 =C2=A0 =C2=A0however our response must be carefully measured:=C2=A0
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 too strong would be hypocritical and risks spiraling out of control;=

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 too weak risks being mistaken for tacit approval.

_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm




--00000000000098fa1605b556f45e--


On Thu, Nov 19, 2020 at 12:37 PM Praveen B=
alasubramanian <pravb=3D40microsoft.com@dmarc.ietf.org> wrote:











I am supportive of updating PRR to PS status and wil=
l review the document.


=C2=A0


Re. QUIC. The current loss recovery spec for QUIC al=
ready refers to PRR as a MAY and has similar other references to TCP RFCs w=
ithout any additional text for QUIC specific adaptations. The current title=
 of RFC 6937 scopes it to TCP: Proportional
 Rate Reduction for TCP. I also see about 38 hits for the word TCP in the t=
ext. 



=C2=A0




From: tcpm <tcpm-bounces@ietf.org> On Behalf Of =

Matt Mathis

Sent: Thursday, November 19, 2020 8:41 AM

To: Gorry Fairhurst <gorry@erg.abdn.ac.uk>; Ian Swett <ianswett=3D40google.com@dma=
rc.ietf.org>

Cc: tcpm@ietf.org=
 Extensions <tcpm=
@ietf.org>

Subject: [EXTERNAL] Re: [tcpm] Introduce RFC 6937 bis (Proportional =
Rate Reduction) as a tcpm work item? INPUT NEEDED




=C2=A0




If we can get away with a non-normative description =
of the heuristic, QUIC is no problem at all.=C2=A0 =C2=A0 If we need normat=
ive descriptions, then TCP and QUIC have to be different, and we need to in=
clude normative references to both recovery
 machines and recovery state=C2=A0variables, which are not in scope for the=
 current doc.




=C2=A0






I would prefer to stay with a non-normative descript=
ion of the heuristic, and scrub the rest of the document for language that =
conflicts with QUIC.



























Thanks,




--MM--

The best way to predict the future is to create it. =C2=A0- Alan Kay



We must not tolerate intolerance;






=C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must=
 be carefully measured:=C2=A0






=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong=
 would be hypocritical and risks spiraling out of control;





=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak r=
isks being mistaken for tacit approval.
















=C2=A0






=C2=A0






On Thu, Nov 19, 2020 at 6:37 AM Gorry Fairhurst <=
gorry@erg.abdn.ac=
.uk> wrote:










On 19/11/2020 14:20, Ian Swett wrote:<=
/p>







I support this work. 




=C2=A0






One question for authors and the group.=C2=A0 This s=
pecification=C2=A0could be applied to QUIC, but my memory is that there are=
 some small changes to adapt it correctly.=C2=A0 Would noting those changes=
 be in scope for this document, a separate document
 in QUIC, or an exercise left to the reader?=C2=A0 I'd be happy to cont=
ribute any QUIC specific text if that's helpful.






=C2=A0






Thanks, Ian








I'd hope the differences for QUIC ARE captured in the RFC series. If=
 they are small, it might be possible to add to this draft perhaps?



Gorry




=C2=A0






On Thu, Nov 19, 2020 at 2:43 AM Scheffenegger, Richa=
rd <rs.ietf@gmx.at> wrote:






As the heuristic is the one major improvement over R=
FC6937, that is the

one area I am especially interested in. Currently working to have PRR in
FreeBSD, it would be the best time window to add this heuristig instead

of switching between the two variants in a static fashion.



I support this and will certainly be reviewing the documents and provide
feedback!



Richard





Am 19.11.2020 um 04:56 schrieb Matt Mathis:

> The authors of PRR would like to update PRR to Proposed Standard

> status.=C2=A0 This entails introducing a new document as an tcpm work =
item.

>

> *Please indicate (non) support and/or comment.*

>

> For more details see the tcpm meeting materials from IETF 109

> minutes:

> 
https://datatracker.ietf.org/meeting/109/materials/minutes-109-tcpm-00<=
br>
> slides: 
https://tools.ietf.org/html/draft-mathis-tcpm-rfc6937bis-00

>

> There were about four "I support this work" remarks at the m=
ic (not

> recorded in the minutes), and about as many in the Meetecho chat.

>

> Abridged IETF/tcpm/PRRbis slides:

> --

> PRR =C2=A0recap (RFC6937 experimental)

> PRR is a special congestion control effective only during fast recover=
y

>

>=C2=A0 =C2=A0* When inflight >=3D ssthresh, send at loss_beta*rate_b=
efore_loss (e.g.

>=C2=A0 =C2=A0 =C2=A0loss_beta =3D 0.5 for Reno (aka rate-halving), 0.7 =
for Cubic)

>=C2=A0 =C2=A0* When inflight < ssthresh, send at the same or twice t=
he

>=C2=A0 =C2=A0 =C2=A0delivery_rate (more later)

>=C2=A0 =C2=A0* Used by all congestion control modules in Linux during f=
ast recovery

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o Can be more dominant than the actual C.C. =
for lossy flows

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0that=E2=80=99re in fast recovery cons=
tantly (e.g. video streaming

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0through policers)

>

> --

> Current Status

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0PRR is widely deployed

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0At least three major OSs: Linux, Wind=
ows, (NetFlix) BSD

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Vast majority of Web traffic for year=
s

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0No changes to algorithms published in RFC 6937

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-CRB - Conservative Reduction Boun=
d - strict packet

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0conversion during loss recovery

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PRR-SSRB - Slowstart Reduction Bound =
- one extra segment per ACK

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0during loss recovery

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A02015 Heuristic to dynamically select which reductio=
n bound

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Only use PRR-SSRB when making good fo=
rward progress

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ACKs that advanced snd.=
una and report no new losses

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Resolves some pathological cases with=
 token bucket policers

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0CC estimates ssthresh b=
efore it can possibly measure the

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0token rate

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The heuristic makes the=
 best of a bad situation

>

> --

> Tentative path forward

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Adopt as a tcpm work item

>

>=C2=A0 =C2=A0*

>

>=C2=A0 =C2=A0 =C2=A0Update the text

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Normative RFC 2119 language

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Add MAY use the heuristic...

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0o

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Trim redundant and obsolete language<=
br>
>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0RFC 6937 repeats itself=
 and is much longer than necessary

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Focus on what an implem=
enter needs to know

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0+

>

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Use non-normative refer=
ences to RFC 6937 for prior

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0measurement work, etc
>

> Thanks,

> --MM--

> The best way to predict the future is to create it. =C2=A0- Alan Kay
>

> We must not tolerate intolerance;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0however our response must be carefull=
y measured:

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too strong would be hy=
pocritical and risks spiraling out of

> control;

>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 too weak risks being m=
istaken for tacit approval.

>

> _______________________________________________

> tcpm mailing list

> tcpm@ietf.org
> 
https://www.ietf.org/mailman/listinfo/tcpm

>



_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm






=C2=A0


_______________________________________________


tcpm mailing list


tcpm@ietf.org


https://www.ietf.org/mailman/listinfo/tcpm=





=C2=A0




_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm










_______________________________________________

tcpm mailing list

tcpm@ietf.org

https://www.ietf.org/mailman/listinfo/tcpm