From c.raiciu@cs.ucl.ac.uk Tue Feb 1 06:57:53 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 229DD3A6C09 for ; Tue, 1 Feb 2011 06:57:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.299 X-Spam-Level: X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8w8CBE-s8KYU for ; Tue, 1 Feb 2011 06:57:51 -0800 (PST) Received: from bells2.cs.ucl.ac.uk (bells2.cs.ucl.ac.uk [128.16.5.33]) by core3.amsl.com (Postfix) with ESMTP id BF6893A6D23 for ; Tue, 1 Feb 2011 06:57:51 -0800 (PST) Received: from [92.81.118.146] (helo=[192.168.1.2]) by bells2.cs.ucl.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (C.Raiciu authenticated) (Exim 4.54) id 1PkHMp-000Cdx-7c; Tue, 01 Feb 2011 14:38:03 +0000 Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: text/plain; charset=windows-1252 From: Costin Raiciu In-Reply-To: <7A7B916F-128B-42C4-B535-99ED507882D1@lurchi.franken.de> Date: Tue, 1 Feb 2011 17:01:00 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <396A98EB-D6A3-4492-AB8B-F539D4BAD0C9@cs.ucl.ac.uk> References: <9510D26531EF184D9017DF24659BB87F3275F300AE@EMV65-UKRD.domain1.systemhost.net> <613383.57734.qm@web39323.mail.mud.yahoo.com> <7A7B916F-128B-42C4-B535-99ED507882D1@lurchi.franken.de> To: =?iso-8859-1?Q?Michael_T=FCxen?= X-Mailer: Apple Mail (2.1081) Cc: multipathtcp@ietf.org, Dirceu Cavendish , iccrg Subject: Re: [multipathtcp] [Iccrg] Re: Please review "Coupled Congestion Control for Multipath Transport Protocols" X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 14:57:53 -0000 Hi Michael,=20 Sorry for the late reply - we haven't slept much lately because of the = Sigcomm deadline. When aiming for the same throughput as tcp on the best path, we equalize = rates in bytes per second. I have analyzed the formula in the draft again, and it is wrong for that = context. The draft keeps the cwnd in bytes, and the correct formula in = that case completely excludes MSS from=20 the calculation of alpha. This is also the formula derived in the = document you mention. The formula for stacks that keep cwnd in packets needs to include MSS, = hence the term cwnd_i *mss_i.=20 We will update the draft to include both versions of the formula; also, = I will recheck the formula for packet based stacks. Thanks for the comments, and well spotted! Costin On 1 Feb 2011, at 09:27, Michael T=FCxen wrote: >>=20 >> Hi Philip, >>=20 >> I have some questions regarding section 3: >>=20 >> (1) It is stated: >> "Since we require that the total throughput is no worse >> than the throughput a single TCP would get on the best path..." >> and later: >> "The formula is derived by equalizing the rate of the multipath = flow >> with the rate of a TCP running on the best path, and solving for >> alpha." >> So what is considered here: >> * throughput in bytes/sec >> * rate in packets/sec >> * rate in bytes/sec >>=20 >> (2) I don't understand the formula for alpha completely. I understand = how >> the corresponding formula in "Practical Congestion Control for = Multipath >> Transport Protocol" is derived. However, there the congestion = windows >> are measured in MTUs (or MSS or packets). So it is more packet = rate >> based. I could understand the formula in the ID if cwnd_i / mss_i = would >> be used instead of cwnd_i * mss_i. So is it a typo or could = someone provide >> a hint why cwnd_i * mss_i is used. >>=20 >> Best regards >> Michael >>=20 >> On Jan 19, 2011, at 3:25 PM, = wrote: >>=20 >>> The Multipath TCP WG has developed a congestion control algorithm = that enables a TCP connection to use multiple paths. =20 >>> http://tools.ietf.org/html/draft-ietf-mptcp-congestion >>>=20 >>> We=92re hoping to WG Last Call this document in February, so it = would be great if anyone from ICCRG could review it >>>=20 >>> Thanks, >>> Phil & Yoshifumi >>> MPTCP WG Chairs >>>=20 >>> Abstract >>>=20 >>> Often endpoints are connected by multiple paths, but = communications >>> are usually restricted to a single path per connection. Resource >>> usage within the network would be more efficient were it possible = for >>> these multiple paths to be used concurrently. Multipath TCP is a >>> proposal to achieve multipath transport in TCP. >>>=20 >>> New congestion control algorithms are needed for multipath = transport >>> protocols such as Multipath TCP, as single path algorithms have a >>> series of issues in the multipath context. One of the prominent >>> problems is that running existing algorithms such as TCP New Reno >>> independently on each path would give the multipath flow more than >>> its fair share at a bottleneck link traversed by more than one of = its >>> subflows. Further, it is desirable that a source with multiple = paths >>> available will transfer more traffic using the least congested of = the >>> paths, hence achieving resource pooling. This would increase the >>> overall utilization of the network and also its robustness to >>> failure. >>>=20 >>> This document presents a congestion control algorithm which = couples >>> the congestion control algorithms running on different subflows by >>> linking their increase functions, and dynamically controls the >>> overall aggresiveness of the multipath flow. The result is a >>> practical algorithm that is fair to TCP at bottlenecks while = moving >>> traffic away from congested links. >>>=20 >>> _______________________________________________ >>> multipathtcp mailing list >>> multipathtcp@ietf.org >>> https://www.ietf.org/mailman/listinfo/multipathtcp >>=20 >>=20 >> _______________________________________________ >> Iccrg mailing list >> Iccrg@cs.ucl.ac.uk >> http://oakham.cs.ucl.ac.uk/mailman/listinfo/iccrg >>=20 >>=20 >=20 > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp From prvs=0013439074=alan.ford@roke.co.uk Tue Feb 1 09:39:22 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 770693A6D27 for ; Tue, 1 Feb 2011 09:39:22 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.421 X-Spam-Level: X-Spam-Status: No, score=-3.421 tagged_above=-999 required=5 tests=[AWL=0.178, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8wWfcvH+zQ7x for ; Tue, 1 Feb 2011 09:39:21 -0800 (PST) Received: from ixe-mta-29.emailfiltering.com (ixe-mta-29-tx.emailfiltering.com [194.116.199.160]) by core3.amsl.com (Postfix) with ESMTP id 2EA123A6BFF for ; Tue, 1 Feb 2011 09:39:21 -0800 (PST) Received: from salt-ext.roke.co.uk ([109.207.29.2]) by ixe-mta-29.emailfiltering.com with emfmta (version 4.8.0.417) vanilla id 273546158 for multipathtcp@ietf.org; 13e06ab57c769b99; Tue, 01 Feb 2011 17:42:38 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 x-cr-hashedpuzzle: ALiR Ax0j CrKw DUMv Dev4 HJuO IFYz IZWw JY72 Kt2j MQDo NHwN NuVs Q+HE RLyU S31q; 2; YgBvAGIALgBiAHIAaQBzAGMAbwBlAEAAYgB0AC4AYwBvAG0AOwBtAHUAbAB0AGkAcABhAHQAaAB0AGMAcABAAGkAZQB0AGYALgBvAHIAZwA=; Sosha1_v1; 7; {E79E4CEF-7131-4DE3-849D-1A011769E6F1}; YQBsAGEAbgAuAGYAbwByAGQAQAByAG8AawBlAC4AYwBvAC4AdQBrAA==; Tue, 01 Feb 2011 17:42:30 GMT; UgBFADoAIABtAHAAdABjAHAALQBtAHUAbAB0AGkAYQBkAGQAcgBlAHMAcwBlAGQAOgAgAEgAbwB3ACAAbABvAG4AZwAgAGIAZQBmAG8AcgBlACAAbQB1AGwAdABpAHAAYQB0AGgAIABzAHQAYQByAHQAcwA/AA== MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable x-cr-puzzleid: {E79E4CEF-7131-4DE3-849D-1A011769E6F1} Content-class: urn:content-classes:message Date: Tue, 1 Feb 2011 17:42:30 -0000 Message-ID: <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> In-Reply-To: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: mptcp-multiaddressed: How long before multipath starts? Thread-Index: AcvBanSXeQ9IQlNxQqycE8YF6qII+gAy3zHw References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> From: "Ford, Alan" To: "Bob Briscoe" Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 17:39:22 -0000 Hi Bob, all, Thanks for your comments. You are of course right that MPTCP is not appropriate for short flows, the problem naturally being the difficulty in deciding when a flow is long-lived or not. The choice of these heuristics would be a very implementation-specific thing, and I'm not sure how much there is we can recommend for the protocol draft. However, we'll add a little commentary on this in the next revision (probably recommending waiting for large scale experimental results, however - as you say, there may well be a numeric heuristic we can use). In the API draft, we already make a comment (Section 4.4): The enabling of MPTCP, either by application or by system defaults, does not necessarily mean that MPTCP will always be used. Both endpoints must support MPTCP, and there must be multiple addresses at at least one endpoint, for MPTCP to be used. Even if those requirements are met, however, MPTCP may not be immediately used on a connection. It may make sense for multiple paths to be brought into operation only after a given period of time, or if the connection is saturated. However we can probably say a bit more about this. An enhanced API should indeed allow an application to signal this (and change its mind during operation); we will make more mention of that in the Extended API appendix in the next revision. This would be an enhancement to the basic "MPTCP on/off" decision - the enabling of MPTCP can be on by default but there needs to be a way of stating that there is no need for additional subflows at this time. Come to think of it, this could have protocol implications and may require an additional flag on the initial MPTCP SYN exchange. Regards, Alan > -----Original Message----- > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > Sent: 31 January 2011 17:15 > To: draft-ietf-mptcp-multiaddressed@tools.ietf.org > Cc: multipathtcp@ietf.org > Subject: mptcp-multiaddressed: How long before multipath starts? >=20 > Alan Costin, Mark, or ANOther, >=20 > The large majority of TCP flows are very short. If there will only be > one or two packets to send, there will be no point sending off loads > of requests to open sub-flows straight after the first 3-way handshake. >=20 > So I imagined there would be some advice in one of the MPTCP I-Ds on > how long a MPTCP should wait until it attempts to open sub-flows. >=20 > I can't find this from a quick scan. I would have expected it to be > in the heuristics section of draft-ietf-mptcp-multiaddressed: > >=20 > In general, a TCP stack doesn't know how much data the application > might eventually want to send. So I guess the basic advice might be > to wait n_1 RTTs before opening a second subflow, then n_2 RTTs > before opening a third and so on. >=20 > Has anyone done any research on this to find the best tradeoff (ie > best values of n_1, n_2, etc) between improved performance/resilience > and wasting signalling packets? I imagine this heuristic would be > clocked by RTTs, not real-time or bytes, because that would likely be > the relevant parameter for the signalling tradeoff. >=20 > Then, in the API draft, there might be some discussion on whether the > application can tell the transport how much data it intends to send, > so the heuristics can be less blind. >=20 > Thoughts? >=20 >=20 > Bob >=20 >=20 > ________________________________________________________________ > Bob Briscoe, BT Innovate & Design From Michael.Tuexen@lurchi.franken.de Tue Feb 1 12:27:21 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 216873A6C62 for ; Tue, 1 Feb 2011 12:27:21 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.196 X-Spam-Level: X-Spam-Status: No, score=-2.196 tagged_above=-999 required=5 tests=[AWL=0.103, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9kaIYmaNikeN for ; Tue, 1 Feb 2011 12:27:20 -0800 (PST) Received: from mail-n.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) by core3.amsl.com (Postfix) with ESMTP id 118793A6C5C for ; Tue, 1 Feb 2011 12:27:19 -0800 (PST) Received: from [192.168.1.113] (p508FB1D6.dip.t-dialin.net [80.143.177.214]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTP id AE4181C0B4600; Tue, 1 Feb 2011 21:30:35 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=windows-1252 From: =?iso-8859-1?Q?Michael_T=FCxen?= In-Reply-To: <396A98EB-D6A3-4492-AB8B-F539D4BAD0C9@cs.ucl.ac.uk> Date: Tue, 1 Feb 2011 21:30:34 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <103AD4FB-9CBE-4343-B858-CE491369350D@lurchi.franken.de> References: <9510D26531EF184D9017DF24659BB87F3275F300AE@EMV65-UKRD.domain1.systemhost.net> <613383.57734.qm@web39323.mail.mud.yahoo.com> <7A7B916F-128B-42C4-B535-99ED507882D1@lurchi.franken.de> <396A98EB-D6A3-4492-AB8B-F539D4BAD0C9@cs.ucl.ac.uk> To: Costin Raiciu X-Mailer: Apple Mail (2.1082) Cc: multipathtcp@ietf.org, Dirceu Cavendish , iccrg Subject: Re: [multipathtcp] [Iccrg] Re: Please review "Coupled Congestion Control for Multipath Transport Protocols" X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 20:27:21 -0000 On Feb 1, 2011, at 4:01 PM, Costin Raiciu wrote: > Hi Michael,=20 >=20 > Sorry for the late reply - we haven't slept much lately because of the = Sigcomm deadline. Hi Costin, no problem... >=20 > When aiming for the same throughput as tcp on the best path, we = equalize rates in bytes per second. So you want the same throughput in bytes/sec as TCP for the best path, = not the same packets/sec as TCP for the best path, right? >=20 > I have analyzed the formula in the draft again, and it is wrong for = that context. The draft keeps the cwnd in bytes, and the correct formula = in that case completely excludes MSS from=20 > the calculation of alpha. This is also the formula derived in the = document you mention. Doesn't that formula considers the cwnd on packets? My understanding is = that the formula in the paper wants to get the same packet/sec since the cwnd = is measured in packets, not bytes. Or am I missing something? >=20 > The formula for stacks that keep cwnd in packets needs to include MSS, = hence the term cwnd_i *mss_i.=20 >=20 > We will update the draft to include both versions of the formula; = also, I will recheck the formula for packet based stacks. Great, thanks. I'll take a look at the formulas as soon as you resubmit = the ID... Thanks for the answer. Best regards Michael >=20 > Thanks for the comments, and well spotted! > Costin >=20 > On 1 Feb 2011, at 09:27, Michael T=FCxen wrote: >=20 >>>=20 >>> Hi Philip, >>>=20 >>> I have some questions regarding section 3: >>>=20 >>> (1) It is stated: >>> "Since we require that the total throughput is no worse >>> than the throughput a single TCP would get on the best path..." >>> and later: >>> "The formula is derived by equalizing the rate of the multipath = flow >>> with the rate of a TCP running on the best path, and solving for >>> alpha." >>> So what is considered here: >>> * throughput in bytes/sec >>> * rate in packets/sec >>> * rate in bytes/sec >>>=20 >>> (2) I don't understand the formula for alpha completely. I = understand how >>> the corresponding formula in "Practical Congestion Control for = Multipath >>> Transport Protocol" is derived. However, there the congestion = windows >>> are measured in MTUs (or MSS or packets). So it is more packet = rate >>> based. I could understand the formula in the ID if cwnd_i / mss_i = would >>> be used instead of cwnd_i * mss_i. So is it a typo or could = someone provide >>> a hint why cwnd_i * mss_i is used. >>>=20 >>> Best regards >>> Michael >>>=20 >>> On Jan 19, 2011, at 3:25 PM, = wrote: >>>=20 >>>> The Multipath TCP WG has developed a congestion control algorithm = that enables a TCP connection to use multiple paths. =20 >>>> http://tools.ietf.org/html/draft-ietf-mptcp-congestion >>>>=20 >>>> We=92re hoping to WG Last Call this document in February, so it = would be great if anyone from ICCRG could review it >>>>=20 >>>> Thanks, >>>> Phil & Yoshifumi >>>> MPTCP WG Chairs >>>>=20 >>>> Abstract >>>>=20 >>>> Often endpoints are connected by multiple paths, but = communications >>>> are usually restricted to a single path per connection. Resource >>>> usage within the network would be more efficient were it possible = for >>>> these multiple paths to be used concurrently. Multipath TCP is a >>>> proposal to achieve multipath transport in TCP. >>>>=20 >>>> New congestion control algorithms are needed for multipath = transport >>>> protocols such as Multipath TCP, as single path algorithms have a >>>> series of issues in the multipath context. One of the prominent >>>> problems is that running existing algorithms such as TCP New Reno >>>> independently on each path would give the multipath flow more than >>>> its fair share at a bottleneck link traversed by more than one of = its >>>> subflows. Further, it is desirable that a source with multiple = paths >>>> available will transfer more traffic using the least congested of = the >>>> paths, hence achieving resource pooling. This would increase the >>>> overall utilization of the network and also its robustness to >>>> failure. >>>>=20 >>>> This document presents a congestion control algorithm which = couples >>>> the congestion control algorithms running on different subflows by >>>> linking their increase functions, and dynamically controls the >>>> overall aggresiveness of the multipath flow. The result is a >>>> practical algorithm that is fair to TCP at bottlenecks while = moving >>>> traffic away from congested links. >>>>=20 >>>> _______________________________________________ >>>> multipathtcp mailing list >>>> multipathtcp@ietf.org >>>> https://www.ietf.org/mailman/listinfo/multipathtcp >>>=20 >>>=20 >>> _______________________________________________ >>> Iccrg mailing list >>> Iccrg@cs.ucl.ac.uk >>> http://oakham.cs.ucl.ac.uk/mailman/listinfo/iccrg >>>=20 >>>=20 >>=20 >> _______________________________________________ >> multipathtcp mailing list >> multipathtcp@ietf.org >> https://www.ietf.org/mailman/listinfo/multipathtcp >=20 >=20 From dokaspar.ietf@gmail.com Wed Feb 2 01:44:52 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3E6FF3A7129 for ; Wed, 2 Feb 2011 01:44:52 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.949 X-Spam-Level: X-Spam-Status: No, score=-2.949 tagged_above=-999 required=5 tests=[AWL=0.650, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OdjjmkiQQtWh for ; Wed, 2 Feb 2011 01:44:51 -0800 (PST) Received: from mail-iy0-f172.google.com (mail-iy0-f172.google.com [209.85.210.172]) by core3.amsl.com (Postfix) with ESMTP id 84A213A6BB3 for ; Wed, 2 Feb 2011 01:44:51 -0800 (PST) Received: by iyi42 with SMTP id 42so7560157iyi.31 for ; Wed, 02 Feb 2011 01:48:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=xl1A7xVc2jFB0SmjpTGRtt7+6JVZ8h3BXKvjXAE7foI=; b=OahAKa3A4PISA8/tk59Oe/OzF8aLGmeb4ScWOHUh0EkRidGnH0APK8nlW632Or2rSN odJRs3IUL5CS4Dm0pylHcyTna9k2e/TYuQAxB4takqK0+1dWEfR/Iv45ux/whkB8QJR0 ek6EOQ2Lo24qSs6HG5wUjPaYC0vsnYFdI8WX4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=KUA58Jh3qZb9tiDf3usiEr6gEcHwQ/wpvMsSY5XNyUPG6W+tssR6IfW90qOjPpe2fF MATFbni1W8v/ZP6pNpNiv++24J4OcJrvxf+TR7NRQss5QaerhoKlJs0Nlj1iH7U5p4xk WeeRYmjcqZlLhxV8KZcu13my/iBZEto2JiwMU= MIME-Version: 1.0 Received: by 10.231.19.201 with SMTP id c9mr9622424ibb.183.1296640090664; Wed, 02 Feb 2011 01:48:10 -0800 (PST) Received: by 10.231.184.68 with HTTP; Wed, 2 Feb 2011 01:48:10 -0800 (PST) In-Reply-To: <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> Date: Wed, 2 Feb 2011 10:48:10 +0100 Message-ID: From: Dominik Kaspar To: "Ford, Alan" Content-Type: text/plain; charset=ISO-8859-1 Cc: multipathtcp@ietf.org, Bob Briscoe Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Feb 2011 09:44:52 -0000 Hi, My question might be slightly off topic, but this thread on short flows motivated me to ask it anyway. Are there any existing models to predict the TCP throughput after a given elapsed time (e.g., the throughput after 2 seconds, after 5 seconds, etc...)? The models I am aware of seem not to be applicable because they all estimate a long-term average. I'd be grateful for any hints and pointers. Best regards, Dominik From c.raiciu@cs.ucl.ac.uk Wed Feb 2 02:07:21 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 994393A6EE9 for ; Wed, 2 Feb 2011 02:07:21 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.432 X-Spam-Level: X-Spam-Status: No, score=-2.432 tagged_above=-999 required=5 tests=[AWL=0.167, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M8PbOYqxNg56 for ; Wed, 2 Feb 2011 02:07:19 -0800 (PST) Received: from bells2.cs.ucl.ac.uk (bells2.cs.ucl.ac.uk [128.16.5.33]) by core3.amsl.com (Postfix) with ESMTP id 760843A6CB6 for ; Wed, 2 Feb 2011 02:07:19 -0800 (PST) Received: from [141.85.37.210] (helo=[192.168.1.66]) by bells2.cs.ucl.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (C.Raiciu authenticated) (Exim 4.54) id 1PkZIt-000CtZ-Vp; Wed, 02 Feb 2011 09:47:12 +0000 Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: multipart/mixed; boundary=Apple-Mail-19-506656188 From: Costin Raiciu In-Reply-To: Date: Wed, 2 Feb 2011 12:10:35 +0200 Message-Id: <79CA19E3-7F9C-4F05-8C2B-72C19C84C9A9@cs.ucl.ac.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> To: Dominik Kaspar X-Mailer: Apple Mail (2.1081) Cc: multipathtcp@ietf.org, Bob Briscoe Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Feb 2011 10:07:21 -0000 --Apple-Mail-19-506656188 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, The short flows question is a valid one; we don't have any heuristics = yet, but we did some measurements for a recent submission. We measured file transfer completion times for different file sizes. The = setup was very simple: two computer connected with two gigabit links. We = compared TCP with UCL-BE's Linux kernel implementation of multipath TCP = to understand when it is worth using multipath TCP. These results (and = probably a lot more) could be the starting point to define the = heuristics Bob is asking about. Attached is the figure showing the results.=20 --Apple-Mail-19-506656188 Content-Disposition: inline; filename=shortfiles.eps Content-Type: image/eps; x-unix-mode=0644; name="shortfiles.eps" Content-Transfer-Encoding: 7bit %!PS-Adobe-2.0 EPSF-2.0 %%Title: shortfiles.eps %%Creator: gnuplot 4.4 patchlevel 2 %%CreationDate: Mon Jan 31 15:48:39 2011 %%DocumentFonts: (atend) %%BoundingBox: 50 50 248 226 %%EndComments %%BeginProlog /gnudict 256 dict def gnudict begin % % The following true/false flags may be edited by hand if desired. % The unit line width and grayscale image gamma correction may also be changed. % /Color false def /Blacktext false def /Solid false def /Dashlength 1 def /Landscape false def /Level1 false def /Rounded false def /ClipToBoundingBox false def /TransparentPatterns false def /gnulinewidth 5.000 def /userlinewidth gnulinewidth def /Gamma 1.0 def % /vshift -60 def /dl1 { 10.0 Dashlength mul mul Rounded { currentlinewidth 0.75 mul sub dup 0 le { pop 0.01 } if } if } def /dl2 { 10.0 Dashlength mul mul Rounded { currentlinewidth 0.75 mul add } if } def /hpt_ 31.5 def /vpt_ 31.5 def /hpt hpt_ def /vpt vpt_ def Level1 {} { /SDict 10 dict def systemdict /pdfmark known not { userdict /pdfmark systemdict /cleartomark get put } if SDict begin [ /Title (shortfiles.eps) /Subject (gnuplot plot) /Creator (gnuplot 4.4 patchlevel 2) /Author (mjh) % /Producer (gnuplot) % /Keywords () /CreationDate (Mon Jan 31 15:48:39 2011) /DOCINFO pdfmark end } ifelse /doclip { ClipToBoundingBox { newpath 50 50 moveto 248 50 lineto 248 226 lineto 50 226 lineto closepath clip } if } def % % Gnuplot Prolog Version 4.4 (August 2010) % %/SuppressPDFMark true def % /M {moveto} bind def /L {lineto} bind def /R {rmoveto} bind def /V {rlineto} bind def /N {newpath moveto} bind def /Z {closepath} bind def /C {setrgbcolor} bind def /f {rlineto fill} bind def /g {setgray} bind def /Gshow {show} def % May be redefined later in the file to support UTF-8 /vpt2 vpt 2 mul def /hpt2 hpt 2 mul def /Lshow {currentpoint stroke M 0 vshift R Blacktext {gsave 0 setgray show grestore} {show} ifelse} def /Rshow {currentpoint stroke M dup stringwidth pop neg vshift R Blacktext {gsave 0 setgray show grestore} {show} ifelse} def /Cshow {currentpoint stroke M dup stringwidth pop -2 div vshift R Blacktext {gsave 0 setgray show grestore} {show} ifelse} def /UP {dup vpt_ mul /vpt exch def hpt_ mul /hpt exch def /hpt2 hpt 2 mul def /vpt2 vpt 2 mul def} def /DL {Color {setrgbcolor Solid {pop []} if 0 setdash} {pop pop pop 0 setgray Solid {pop []} if 0 setdash} ifelse} def /BL {stroke userlinewidth 2 mul setlinewidth Rounded {1 setlinejoin 1 setlinecap} if} def /AL {stroke userlinewidth 2 div setlinewidth Rounded {1 setlinejoin 1 setlinecap} if} def /UL {dup gnulinewidth mul /userlinewidth exch def dup 1 lt {pop 1} if 10 mul /udl exch def} def /PL {stroke userlinewidth setlinewidth Rounded {1 setlinejoin 1 setlinecap} if} def 3.8 setmiterlimit % Default Line colors /LCw {1 1 1} def /LCb {0 0 0} def /LCa {0 0 0} def /LC0 {1 0 0} def /LC1 {0 1 0} def /LC2 {0 0 1} def /LC3 {1 0 1} def /LC4 {0 1 1} def /LC5 {1 1 0} def /LC6 {0 0 0} def /LC7 {1 0.3 0} def /LC8 {0.5 0.5 0.5} def % Default Line Types /LTw {PL [] 1 setgray} def /LTb {BL [] LCb DL} def /LTa {AL [1 udl mul 2 udl mul] 0 setdash LCa setrgbcolor} def /LT0 {PL [] LC0 DL} def /LT1 {PL [4 dl1 2 dl2] LC1 DL} def /LT2 {PL [2 dl1 3 dl2] LC2 DL} def /LT3 {PL [1 dl1 1.5 dl2] LC3 DL} def /LT4 {PL [6 dl1 2 dl2 1 dl1 2 dl2] LC4 DL} def /LT5 {PL [3 dl1 3 dl2 1 dl1 3 dl2] LC5 DL} def /LT6 {PL [2 dl1 2 dl2 2 dl1 6 dl2] LC6 DL} def /LT7 {PL [1 dl1 2 dl2 6 dl1 2 dl2 1 dl1 2 dl2] LC7 DL} def /LT8 {PL [2 dl1 2 dl2 2 dl1 2 dl2 2 dl1 2 dl2 2 dl1 4 dl2] LC8 DL} def /Pnt {stroke [] 0 setdash gsave 1 setlinecap M 0 0 V stroke grestore} def /Dia {stroke [] 0 setdash 2 copy vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath stroke Pnt} def /Pls {stroke [] 0 setdash vpt sub M 0 vpt2 V currentpoint stroke M hpt neg vpt neg R hpt2 0 V stroke } def /Box {stroke [] 0 setdash 2 copy exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath stroke Pnt} def /Crs {stroke [] 0 setdash exch hpt sub exch vpt add M hpt2 vpt2 neg V currentpoint stroke M hpt2 neg 0 R hpt2 vpt2 V stroke} def /TriU {stroke [] 0 setdash 2 copy vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath stroke Pnt} def /Star {2 copy Pls Crs} def /BoxF {stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath fill} def /TriUF {stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath fill} def /TriD {stroke [] 0 setdash 2 copy vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath stroke Pnt} def /TriDF {stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath fill} def /DiaF {stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath fill} def /Pent {stroke [] 0 setdash 2 copy gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath stroke grestore Pnt} def /PentF {stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath fill grestore} def /Circle {stroke [] 0 setdash 2 copy hpt 0 360 arc stroke Pnt} def /CircleF {stroke [] 0 setdash hpt 0 360 arc fill} def /C0 {BL [] 0 setdash 2 copy moveto vpt 90 450 arc} bind def /C1 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc closepath fill vpt 0 360 arc closepath} bind def /C2 {BL [] 0 setdash 2 copy moveto 2 copy vpt 90 180 arc closepath fill vpt 0 360 arc closepath} bind def /C3 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 180 arc closepath fill vpt 0 360 arc closepath} bind def /C4 {BL [] 0 setdash 2 copy moveto 2 copy vpt 180 270 arc closepath fill vpt 0 360 arc closepath} bind def /C5 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc 2 copy moveto 2 copy vpt 180 270 arc closepath fill vpt 0 360 arc} bind def /C6 {BL [] 0 setdash 2 copy moveto 2 copy vpt 90 270 arc closepath fill vpt 0 360 arc closepath} bind def /C7 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 270 arc closepath fill vpt 0 360 arc closepath} bind def /C8 {BL [] 0 setdash 2 copy moveto 2 copy vpt 270 360 arc closepath fill vpt 0 360 arc closepath} bind def /C9 {BL [] 0 setdash 2 copy moveto 2 copy vpt 270 450 arc closepath fill vpt 0 360 arc closepath} bind def /C10 {BL [] 0 setdash 2 copy 2 copy moveto vpt 270 360 arc closepath fill 2 copy moveto 2 copy vpt 90 180 arc closepath fill vpt 0 360 arc closepath} bind def /C11 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 180 arc closepath fill 2 copy moveto 2 copy vpt 270 360 arc closepath fill vpt 0 360 arc closepath} bind def /C12 {BL [] 0 setdash 2 copy moveto 2 copy vpt 180 360 arc closepath fill vpt 0 360 arc closepath} bind def /C13 {BL [] 0 setdash 2 copy moveto 2 copy vpt 0 90 arc closepath fill 2 copy moveto 2 copy vpt 180 360 arc closepath fill vpt 0 360 arc closepath} bind def /C14 {BL [] 0 setdash 2 copy moveto 2 copy vpt 90 360 arc closepath fill vpt 0 360 arc} bind def /C15 {BL [] 0 setdash 2 copy vpt 0 360 arc closepath fill vpt 0 360 arc closepath} bind def /Rec {newpath 4 2 roll moveto 1 index 0 rlineto 0 exch rlineto neg 0 rlineto closepath} bind def /Square {dup Rec} bind def /Bsquare {vpt sub exch vpt sub exch vpt2 Square} bind def /S0 {BL [] 0 setdash 2 copy moveto 0 vpt rlineto BL Bsquare} bind def /S1 {BL [] 0 setdash 2 copy vpt Square fill Bsquare} bind def /S2 {BL [] 0 setdash 2 copy exch vpt sub exch vpt Square fill Bsquare} bind def /S3 {BL [] 0 setdash 2 copy exch vpt sub exch vpt2 vpt Rec fill Bsquare} bind def /S4 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt Square fill Bsquare} bind def /S5 {BL [] 0 setdash 2 copy 2 copy vpt Square fill exch vpt sub exch vpt sub vpt Square fill Bsquare} bind def /S6 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt vpt2 Rec fill Bsquare} bind def /S7 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt vpt2 Rec fill 2 copy vpt Square fill Bsquare} bind def /S8 {BL [] 0 setdash 2 copy vpt sub vpt Square fill Bsquare} bind def /S9 {BL [] 0 setdash 2 copy vpt sub vpt vpt2 Rec fill Bsquare} bind def /S10 {BL [] 0 setdash 2 copy vpt sub vpt Square fill 2 copy exch vpt sub exch vpt Square fill Bsquare} bind def /S11 {BL [] 0 setdash 2 copy vpt sub vpt Square fill 2 copy exch vpt sub exch vpt2 vpt Rec fill Bsquare} bind def /S12 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill Bsquare} bind def /S13 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill 2 copy vpt Square fill Bsquare} bind def /S14 {BL [] 0 setdash 2 copy exch vpt sub exch vpt sub vpt2 vpt Rec fill 2 copy exch vpt sub exch vpt Square fill Bsquare} bind def /S15 {BL [] 0 setdash 2 copy Bsquare fill Bsquare} bind def /D0 {gsave translate 45 rotate 0 0 S0 stroke grestore} bind def /D1 {gsave translate 45 rotate 0 0 S1 stroke grestore} bind def /D2 {gsave translate 45 rotate 0 0 S2 stroke grestore} bind def /D3 {gsave translate 45 rotate 0 0 S3 stroke grestore} bind def /D4 {gsave translate 45 rotate 0 0 S4 stroke grestore} bind def /D5 {gsave translate 45 rotate 0 0 S5 stroke grestore} bind def /D6 {gsave translate 45 rotate 0 0 S6 stroke grestore} bind def /D7 {gsave translate 45 rotate 0 0 S7 stroke grestore} bind def /D8 {gsave translate 45 rotate 0 0 S8 stroke grestore} bind def /D9 {gsave translate 45 rotate 0 0 S9 stroke grestore} bind def /D10 {gsave translate 45 rotate 0 0 S10 stroke grestore} bind def /D11 {gsave translate 45 rotate 0 0 S11 stroke grestore} bind def /D12 {gsave translate 45 rotate 0 0 S12 stroke grestore} bind def /D13 {gsave translate 45 rotate 0 0 S13 stroke grestore} bind def /D14 {gsave translate 45 rotate 0 0 S14 stroke grestore} bind def /D15 {gsave translate 45 rotate 0 0 S15 stroke grestore} bind def /DiaE {stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V closepath stroke} def /BoxE {stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V closepath stroke} def /TriUE {stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V closepath stroke} def /TriDE {stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V closepath stroke} def /PentE {stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat closepath stroke grestore} def /CircE {stroke [] 0 setdash hpt 0 360 arc stroke} def /Opaque {gsave closepath 1 setgray fill grestore 0 setgray closepath} def /DiaW {stroke [] 0 setdash vpt add M hpt neg vpt neg V hpt vpt neg V hpt vpt V hpt neg vpt V Opaque stroke} def /BoxW {stroke [] 0 setdash exch hpt sub exch vpt add M 0 vpt2 neg V hpt2 0 V 0 vpt2 V hpt2 neg 0 V Opaque stroke} def /TriUW {stroke [] 0 setdash vpt 1.12 mul add M hpt neg vpt -1.62 mul V hpt 2 mul 0 V hpt neg vpt 1.62 mul V Opaque stroke} def /TriDW {stroke [] 0 setdash vpt 1.12 mul sub M hpt neg vpt 1.62 mul V hpt 2 mul 0 V hpt neg vpt -1.62 mul V Opaque stroke} def /PentW {stroke [] 0 setdash gsave translate 0 hpt M 4 {72 rotate 0 hpt L} repeat Opaque stroke grestore} def /CircW {stroke [] 0 setdash hpt 0 360 arc Opaque stroke} def /BoxFill {gsave Rec 1 setgray fill grestore} def /Density { /Fillden exch def currentrgbcolor /ColB exch def /ColG exch def /ColR exch def /ColR ColR Fillden mul Fillden sub 1 add def /ColG ColG Fillden mul Fillden sub 1 add def /ColB ColB Fillden mul Fillden sub 1 add def ColR ColG ColB setrgbcolor} def /BoxColFill {gsave Rec PolyFill} def /PolyFill {gsave Density fill grestore grestore} def /h {rlineto rlineto rlineto gsave closepath fill grestore} bind def % % PostScript Level 1 Pattern Fill routine for rectangles % Usage: x y w h s a XX PatternFill % x,y = lower left corner of box to be filled % w,h = width and height of box % a = angle in degrees between lines and x-axis % XX = 0/1 for no/yes cross-hatch % /PatternFill {gsave /PFa [ 9 2 roll ] def PFa 0 get PFa 2 get 2 div add PFa 1 get PFa 3 get 2 div add translate PFa 2 get -2 div PFa 3 get -2 div PFa 2 get PFa 3 get Rec gsave 1 setgray fill grestore clip currentlinewidth 0.5 mul setlinewidth /PFs PFa 2 get dup mul PFa 3 get dup mul add sqrt def 0 0 M PFa 5 get rotate PFs -2 div dup translate 0 1 PFs PFa 4 get div 1 add floor cvi {PFa 4 get mul 0 M 0 PFs V} for 0 PFa 6 get ne { 0 1 PFs PFa 4 get div 1 add floor cvi {PFa 4 get mul 0 2 1 roll M PFs 0 V} for } if stroke grestore} def % /languagelevel where {pop languagelevel} {1} ifelse 2 lt {/InterpretLevel1 true def} {/InterpretLevel1 Level1 def} ifelse % % PostScript level 2 pattern fill definitions % /Level2PatternFill { /Tile8x8 {/PaintType 2 /PatternType 1 /TilingType 1 /BBox [0 0 8 8] /XStep 8 /YStep 8} bind def /KeepColor {currentrgbcolor [/Pattern /DeviceRGB] setcolorspace} bind def << Tile8x8 /PaintProc {0.5 setlinewidth pop 0 0 M 8 8 L 0 8 M 8 0 L stroke} >> matrix makepattern /Pat1 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop 0 0 M 8 8 L 0 8 M 8 0 L stroke 0 4 M 4 8 L 8 4 L 4 0 L 0 4 L stroke} >> matrix makepattern /Pat2 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop 0 0 M 0 8 L 8 8 L 8 0 L 0 0 L fill} >> matrix makepattern /Pat3 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop -4 8 M 8 -4 L 0 12 M 12 0 L stroke} >> matrix makepattern /Pat4 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop -4 0 M 8 12 L 0 -4 M 12 8 L stroke} >> matrix makepattern /Pat5 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop -2 8 M 4 -4 L 0 12 M 8 -4 L 4 12 M 10 0 L stroke} >> matrix makepattern /Pat6 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop -2 0 M 4 12 L 0 -4 M 8 12 L 4 -4 M 10 8 L stroke} >> matrix makepattern /Pat7 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop 8 -2 M -4 4 L 12 0 M -4 8 L 12 4 M 0 10 L stroke} >> matrix makepattern /Pat8 exch def << Tile8x8 /PaintProc {0.5 setlinewidth pop 0 -2 M 12 4 L -4 0 M 12 8 L -4 4 M 8 10 L stroke} >> matrix makepattern /Pat9 exch def /Pattern1 {PatternBgnd KeepColor Pat1 setpattern} bind def /Pattern2 {PatternBgnd KeepColor Pat2 setpattern} bind def /Pattern3 {PatternBgnd KeepColor Pat3 setpattern} bind def /Pattern4 {PatternBgnd KeepColor Landscape {Pat5} {Pat4} ifelse setpattern} bind def /Pattern5 {PatternBgnd KeepColor Landscape {Pat4} {Pat5} ifelse setpattern} bind def /Pattern6 {PatternBgnd KeepColor Landscape {Pat9} {Pat6} ifelse setpattern} bind def /Pattern7 {PatternBgnd KeepColor Landscape {Pat8} {Pat7} ifelse setpattern} bind def } def % % %End of PostScript Level 2 code % /PatternBgnd { TransparentPatterns {} {gsave 1 setgray fill grestore} ifelse } def % % Substitute for Level 2 pattern fill codes with % grayscale if Level 2 support is not selected. % /Level1PatternFill { /Pattern1 {0.250 Density} bind def /Pattern2 {0.500 Density} bind def /Pattern3 {0.750 Density} bind def /Pattern4 {0.125 Density} bind def /Pattern5 {0.375 Density} bind def /Pattern6 {0.625 Density} bind def /Pattern7 {0.875 Density} bind def } def % % Now test for support of Level 2 code % Level1 {Level1PatternFill} {Level2PatternFill} ifelse % /Symbol-Oblique /Symbol findfont [1 0 .167 1 0 0] makefont dup length dict begin {1 index /FID eq {pop pop} {def} ifelse} forall currentdict end definefont pop end %%EndProlog gnudict begin gsave doclip 50 50 translate 0.050 0.050 scale 0 setgray newpath (Helvetica) findfont 180 scalefont setfont 1.000 UL LTb 594 576 M 63 0 V 2978 0 R -63 0 V 486 576 M ( 0) Rshow 1.000 UL LTb 594 967 M 63 0 V 2978 0 R -63 0 V 486 967 M ( 5) Rshow 1.000 UL LTb 594 1357 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 10) Rshow 1.000 UL LTb 594 1748 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 15) Rshow 1.000 UL LTb 594 2139 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 20) Rshow 1.000 UL LTb 594 2530 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 25) Rshow 1.000 UL LTb 594 2920 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 30) Rshow 1.000 UL LTb 594 3311 M 63 0 V 2978 0 R -63 0 V -3086 0 R ( 35) Rshow 1.000 UL LTb 967 576 M 0 63 V 0 2672 R 0 -63 V 967 396 M ( 50) Cshow 1.000 UL LTb 1349 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 100) Cshow 1.000 UL LTb 1730 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 150) Cshow 1.000 UL LTb 2111 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 200) Cshow 1.000 UL LTb 2492 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 250) Cshow 1.000 UL LTb 2873 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 300) Cshow 1.000 UL LTb 3254 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 350) Cshow 1.000 UL LTb 3635 576 M 0 63 V 0 2672 R 0 -63 V 0 -2852 R ( 400) Cshow 1.000 UL LTb 1.000 UL LTb 594 3311 N 594 576 L 3041 0 V 0 2735 V -3041 0 V Z stroke LCb setrgbcolor 180 1943 M currentpoint gsave translate -270 rotate 0 0 M (Download Time \(ms\)) Cshow grestore LTb LCb setrgbcolor 2114 126 M (File size \(pkts\)) Cshow LTb 1.000 UP 1.000 UL LTb % Begin plot #1 1.000 UP 1.000 UL LT0 LCb setrgbcolor 2816 909 M (MPTCP) Rshow LT0 2924 909 M 495 0 V 594 689 M 8 106 V 15 65 V 30 117 V 61 155 V 122 172 V 244 144 V 488 309 V 975 388 V 1098 456 V 594 689 Pls 602 795 Pls 617 860 Pls 647 977 Pls 708 1132 Pls 830 1304 Pls 1074 1448 Pls 1562 1757 Pls 2537 2145 Pls 3171 909 Pls % End plot #1 % Begin plot #2 1.000 UP 1.000 UL LT1 LCb setrgbcolor 2816 729 M (TCP) Rshow LT1 2924 729 M 495 0 V 594 679 M 8 16 V 15 98 V 30 61 V 61 120 V 122 197 V 244 177 V 488 365 V 975 765 V 1095 833 V 594 679 Crs 602 695 Crs 617 793 Crs 647 854 Crs 708 974 Crs 830 1171 Crs 1074 1348 Crs 1562 1713 Crs 2537 2478 Crs 3171 729 Crs % End plot #2 1.000 UL LTb 594 3311 N 594 576 L 3041 0 V 0 2735 V -3041 0 V Z stroke 1.000 UP 1.000 UL LTb stroke grestore end showpage %%Trailer %%DocumentFonts: Helvetica --Apple-Mail-19-506656188 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii This is the description with put in the paper. Should MPTCP be enabled for all TCP connections in a data center? We = connected two dual-interface machines to a gigabit switch, and measured = the time to setup a connection and transfer a short file. TCP can only = use one inter- face; MPTCP can also use the second, but only after the = first subflow has negotiated the use of MPTCP and the second subflow has = been established. Figure 17 shows that TCP is quicker for files of less = than about 150 packets, but much slower thereafter. Closer analysis = shows that our MPTCP implementation slowstarts slightly slower than TCP. = The largest difference is about two RTTs just as TCP fills the link, and = decreases thereafter. This different does not seem fundamen- tal, and is = likely a quirk of slowstart in our implementation. As it is, it would be = better to delay 5ms until the first subflow leaves slowstart before = starting the second. Cheers, Costin On 2 Feb 2011, at 11:48, Dominik Kaspar wrote: > Hi, >=20 > My question might be slightly off topic, but this thread on short > flows motivated me to ask it anyway. >=20 > Are there any existing models to predict the TCP throughput after a > given elapsed time (e.g., the throughput after 2 seconds, after 5 > seconds, etc...)? The models I am aware of seem not to be applicable > because they all estimate a long-term average. I'd be grateful for any > hints and pointers. >=20 > Best regards, > Dominik > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp --Apple-Mail-19-506656188-- From bob.briscoe@bt.com Thu Feb 3 15:42:12 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DC6D63A6B42 for ; Thu, 3 Feb 2011 15:42:12 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.299 X-Spam-Level: X-Spam-Status: No, score=-3.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sMfVKyZAfxT3 for ; Thu, 3 Feb 2011 15:42:11 -0800 (PST) Received: from smtp3.smtp.bt.com (smtp3.smtp.bt.com [217.32.164.138]) by core3.amsl.com (Postfix) with ESMTP id 0C3FC3A6B3D for ; Thu, 3 Feb 2011 15:42:10 -0800 (PST) Received: from i2kc08-ukbr.domain1.systemhost.net ([193.113.197.71]) by smtp3.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 3 Feb 2011 23:45:33 +0000 Received: from cbibipnt08.iuser.iroot.adidom.com ([147.149.100.81]) by i2kc08-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Thu, 3 Feb 2011 23:45:32 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt08.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1296776731982; Thu, 3 Feb 2011 23:45:31 +0000 Received: from MUT.jungle.bt.co.uk ([10.73.210.189]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p13NjUHQ017151; Thu, 3 Feb 2011 23:45:30 GMT Message-Id: <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Thu, 03 Feb 2011 23:45:29 +0000 To: "Ford, Alan" From: Bob Briscoe In-Reply-To: <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad. roke.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 03 Feb 2011 23:45:32.0928 (UTC) FILETIME=[726F9000:01CBC3FC] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Feb 2011 23:42:13 -0000 Alan, Surely this shouldn't be thought of as primarily an API issue? - the basic MPTCP implementation should be able to decide when to start opening sub-flows, whether or not the app wants to give it more information thru an API. That's why I suggested at least something needs to be said on this in the heuristics section of (section-3). Let's put some ball-park numbers on this. My intention is NOT to start a discussion about whether these heuristics are correct, just to show this question will need an answer in a basic implementation (without any API). I think everyone would agree that it wouldn't be worth creating any sub-flows if the transfer needed no more than 3 packets. With an initial window of 3, a TCP server can get 3 packets to a client in 2RTTs (counting from when the client sends the initial SYN). Any more sub-flows would be redundant (other than for sending copies for resilience). Accurate data on what proportion of flows on the Internet consist of no more than 3 packets is hard to come by. Perhaps 50% perhaps 75%. But even if it's only 25%, that's enough to have to address the question of whether a basic MPTCP implementation will need some logic to decide whether and when it is going to start opening sub-flows. Indeed, if one can work out a good default behaviour based on a tradeoff between the cost of signalling for extra sub-flows and their benefit, I cannot see much reason for an app to want to override this - not much need for an API. To kick-off a discussion on a baseline heuristic, if IW=N, I would propose we should ignore the cost of extra SYNs and SYN-ACKs along the same arguments as Wischik's "Short Messages". Then, as soon as the server sees some data buffered for the client (implying the app has got a request for data from the client), it would be worth the server opening one sub-flow for each N segments it will have to send to empty the buffer (limited by the no. of distinct sub-flows that are possible). Incidentally, this would be a neat way of increasing the effective IW while spreading out the packets over different RTTs, so they don't all arrive in the queue at the client back-to-back. The earliest a server could request more sub-flows is in parallel to its first SYN-ACK. But at that point it has no data from the client yet, so it doesn't know what the client wants, nor how big it will be. It could optimistically start a sub-flow though. At this point, we start to need some research on whether this would be worth it... Basically, I'm saying that we shouldn't expect implementers to have to be researchers. At minimum the results of even the most basic research should be documented in the heuristics section, so an implementer who isn't a researcher can just do what it says. Bob At 17:42 01/02/2011, Ford, Alan wrote: >Hi Bob, all, > >Thanks for your comments. You are of course right that MPTCP is not >appropriate for short flows, the problem naturally being the difficulty >in deciding when a flow is long-lived or not. > >The choice of these heuristics would be a very implementation-specific >thing, and I'm not sure how much there is we can recommend for the >protocol draft. However, we'll add a little commentary on this in the >next revision (probably recommending waiting for large scale >experimental results, however - as you say, there may well be a numeric >heuristic we can use). > >In the API draft, we already make a comment (Section 4.4): > > The enabling of MPTCP, either by application or by system defaults, > does not necessarily mean that MPTCP will always be used. Both > endpoints must support MPTCP, and there must be multiple addresses at > at least one endpoint, for MPTCP to be used. Even if those > requirements are met, however, MPTCP may not be immediately used on a > connection. It may make sense for multiple paths to be brought into > operation only after a given period of time, or if the connection is > saturated. > >However we can probably say a bit more about this. An enhanced API >should indeed allow an application to signal this (and change its mind >during operation); we will make more mention of that in the Extended API >appendix in the next revision. This would be an enhancement to the basic >"MPTCP on/off" decision - the enabling of MPTCP can be on by default but >there needs to be a way of stating that there is no need for additional >subflows at this time. Come to think of it, this could have protocol >implications and may require an additional flag on the initial MPTCP SYN >exchange. > >Regards, >Alan > > > -----Original Message----- > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > Sent: 31 January 2011 17:15 > > To: draft-ietf-mptcp-multiaddressed@tools.ietf.org > > Cc: multipathtcp@ietf.org > > Subject: mptcp-multiaddressed: How long before multipath starts? > > > > Alan Costin, Mark, or ANOther, > > > > The large majority of TCP flows are very short. If there will only be > > one or two packets to send, there will be no point sending off loads > > of requests to open sub-flows straight after the first 3-way >handshake. > > > > So I imagined there would be some advice in one of the MPTCP I-Ds on > > how long a MPTCP should wait until it attempts to open sub-flows. > > > > I can't find this from a quick scan. I would have expected it to be > > in the heuristics section of draft-ietf-mptcp-multiaddressed: > > >.8> > > > > In general, a TCP stack doesn't know how much data the application > > might eventually want to send. So I guess the basic advice might be > > to wait n_1 RTTs before opening a second subflow, then n_2 RTTs > > before opening a third and so on. > > > > Has anyone done any research on this to find the best tradeoff (ie > > best values of n_1, n_2, etc) between improved performance/resilience > > and wasting signalling packets? I imagine this heuristic would be > > clocked by RTTs, not real-time or bytes, because that would likely be > > the relevant parameter for the signalling tradeoff. > > > > Then, in the API draft, there might be some discussion on whether the > > application can tell the transport how much data it intends to send, > > so the heuristics can be less blind. > > > > Thoughts? > > > > > > Bob > > > > > > ________________________________________________________________ > > Bob Briscoe, BT Innovate & Design ________________________________________________________________ Bob Briscoe, BT Innovate & Design From prvs=6016b890a8=alan.ford@roke.co.uk Fri Feb 4 05:16:09 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 662AF3A6BA1 for ; Fri, 4 Feb 2011 05:16:09 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.127 X-Spam-Level: X-Spam-Status: No, score=-3.127 tagged_above=-999 required=5 tests=[AWL=-0.128, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A2lWf2RnjGVN for ; Fri, 4 Feb 2011 05:16:08 -0800 (PST) Received: from ixe-mta-29.emailfiltering.com (ixe-mta-29-tx.emailfiltering.com [194.116.199.160]) by core3.amsl.com (Postfix) with ESMTP id 9BD7B3A691A for ; Fri, 4 Feb 2011 05:16:07 -0800 (PST) Received: from salt-ext.roke.co.uk ([109.207.29.2]) by ixe-mta-29.emailfiltering.com with emfmta (version 4.8.0.417) vanilla id 276968842 for multipathtcp@ietf.org; faf13b90e1d46ec1; Fri, 04 Feb 2011 13:19:31 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 x-cr-hashedpuzzle: A8i0 BAic BFP4 Bguo BrGA B9mY DJVr Jl1C KlIL Kmbl KtSx QPJC TX9+ UWPc VN98 V1Tt; 2; YgBvAGIALgBiAHIAaQBzAGMAbwBlAEAAYgB0AC4AYwBvAG0AOwBtAHUAbAB0AGkAcABhAHQAaAB0AGMAcABAAGkAZQB0AGYALgBvAHIAZwA=; Sosha1_v1; 7; {70CDC4C7-A40F-400E-954F-1E116764EDA9}; YQBsAGEAbgAuAGYAbwByAGQAQAByAG8AawBlAC4AYwBvAC4AdQBrAA==; Fri, 04 Feb 2011 13:19:23 GMT; UgBFADoAIABtAHAAdABjAHAALQBtAHUAbAB0AGkAYQBkAGQAcgBlAHMAcwBlAGQAOgAgAEgAbwB3ACAAbABvAG4AZwAgAGIAZQBmAG8AcgBlACAAbQB1AGwAdABpAHAAYQB0AGgAIABzAHQAYQByAHQAcwA/AA== MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable x-cr-puzzleid: {70CDC4C7-A40F-400E-954F-1E116764EDA9} Content-class: urn:content-classes:message Date: Fri, 4 Feb 2011 13:19:23 -0000 Message-ID: <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad.roke.co.uk> In-Reply-To: <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: mptcp-multiaddressed: How long before multipath starts? Thread-Index: AcvD/JmP5vU762A3QoOUxXCiWgLguwAbp5dQ References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> From: "Ford, Alan" To: "Bob Briscoe" Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2011 13:16:09 -0000 Bob, On the surface this sounds like a sensible heuristic to use - essentially, you are saying, do not use MPTCP until there has been congestion and thus you want to initiate resource pooling. However, there are a couple of (smallish) issues with this approach. In the client/server model, it is the server who is in a position to know its buffer is filling up and thus how many new subflows it wants. But in many cases (e.g. client addresses behind NATs) only the client is in a position to create these subflows - and the client won't know about the amount of data in the server's buffer. (Some intelligence could be applied, e.g. opening subflows at the client side when the window size has stayed static for N packets - this is probably good enough for most cases, although adding more than one extra subflow would have to wait until both subflow windows have stayed static). The other issue with this approach is that it does not provide resource pooling for resilience, only throughput. Therefore, a second heuristic about time/number of packets elapsed should be applied, to provide resilience to longer-lived flows. Most of this behaviour can be controlled by the endpoints, and does not require any protocol changes. Or are there any cases where we may want a flag to say "I speak MPTCP, but don't want to use it yet"? Regards, Alan > -----Original Message----- > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > Sent: 03 February 2011 23:45 > To: Ford, Alan > Cc: multipathtcp@ietf.org > Subject: RE: mptcp-multiaddressed: How long before multipath starts? >=20 > Alan, >=20 > Surely this shouldn't be thought of as primarily an API issue? - the > basic MPTCP implementation should be able to decide when to start > opening sub-flows, whether or not the app wants to give it more > information thru an API. >=20 > That's why I suggested at least something needs to be said on this in > the heuristics section of (section-3). >=20 >=20 > Let's put some ball-park numbers on this. My intention is NOT to > start a discussion about whether these heuristics are correct, just > to show this question will need an answer in a basic implementation > (without any API). >=20 > I think everyone would agree that it wouldn't be worth creating any > sub-flows if the transfer needed no more than 3 packets. With an > initial window of 3, a TCP server can get 3 packets to a client in > 2RTTs (counting from when the client sends the initial SYN). Any more > sub-flows would be redundant (other than for sending copies for resilience). >=20 > Accurate data on what proportion of flows on the Internet consist of > no more than 3 packets is hard to come by. Perhaps 50% perhaps 75%. > But even if it's only 25%, that's enough to have to address the > question of whether a basic MPTCP implementation will need some logic > to decide whether and when it is going to start opening sub-flows. >=20 >=20 > Indeed, if one can work out a good default behaviour based on a > tradeoff between the cost of signalling for extra sub-flows and their > benefit, I cannot see much reason for an app to want to override this > - not much need for an API. >=20 >=20 >=20 > To kick-off a discussion on a baseline heuristic, if IW=3DN, I would > propose we should ignore the cost of extra SYNs and SYN-ACKs along > the same arguments as Wischik's "Short Messages". Then, as soon as > the server sees some data buffered for the client (implying the app > has got a request for data from the client), it would be worth the > server opening one sub-flow for each N segments it will have to send > to empty the buffer (limited by the no. of distinct sub-flows that > are possible). >=20 > Incidentally, this would be a neat way of increasing the effective IW > while spreading out the packets over different RTTs, so they don't > all arrive in the queue at the client back-to-back. >=20 > The earliest a server could request more sub-flows is in parallel to > its first SYN-ACK. But at that point it has no data from the client > yet, so it doesn't know what the client wants, nor how big it will > be. It could optimistically start a sub-flow though. At this point, > we start to need some research on whether this would be worth it... >=20 >=20 > Basically, I'm saying that we shouldn't expect implementers to have > to be researchers. At minimum the results of even the most basic > research should be documented in the heuristics section, so an > implementer who isn't a researcher can just do what it says. >=20 >=20 > Bob >=20 > At 17:42 01/02/2011, Ford, Alan wrote: > >Hi Bob, all, > > > >Thanks for your comments. You are of course right that MPTCP is not > >appropriate for short flows, the problem naturally being the difficulty > >in deciding when a flow is long-lived or not. > > > >The choice of these heuristics would be a very implementation-specific > >thing, and I'm not sure how much there is we can recommend for the > >protocol draft. However, we'll add a little commentary on this in the > >next revision (probably recommending waiting for large scale > >experimental results, however - as you say, there may well be a numeric > >heuristic we can use). > > > >In the API draft, we already make a comment (Section 4.4): > > > > The enabling of MPTCP, either by application or by system defaults, > > does not necessarily mean that MPTCP will always be used. Both > > endpoints must support MPTCP, and there must be multiple addresses at > > at least one endpoint, for MPTCP to be used. Even if those > > requirements are met, however, MPTCP may not be immediately used on a > > connection. It may make sense for multiple paths to be brought into > > operation only after a given period of time, or if the connection is > > saturated. > > > >However we can probably say a bit more about this. An enhanced API > >should indeed allow an application to signal this (and change its mind > >during operation); we will make more mention of that in the Extended API > >appendix in the next revision. This would be an enhancement to the basic > >"MPTCP on/off" decision - the enabling of MPTCP can be on by default but > >there needs to be a way of stating that there is no need for additional > >subflows at this time. Come to think of it, this could have protocol > >implications and may require an additional flag on the initial MPTCP SYN > >exchange. > > > >Regards, > >Alan > > > > > -----Original Message----- > > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > > Sent: 31 January 2011 17:15 > > > To: draft-ietf-mptcp-multiaddressed@tools.ietf.org > > > Cc: multipathtcp@ietf.org > > > Subject: mptcp-multiaddressed: How long before multipath starts? > > > > > > Alan Costin, Mark, or ANOther, > > > > > > The large majority of TCP flows are very short. If there will only be > > > one or two packets to send, there will be no point sending off loads > > > of requests to open sub-flows straight after the first 3-way > >handshake. > > > > > > So I imagined there would be some advice in one of the MPTCP I-Ds on > > > how long a MPTCP should wait until it attempts to open sub-flows. > > > > > > I can't find this from a quick scan. I would have expected it to be > > > in the heuristics section of draft-ietf-mptcp-multiaddressed: > > > > > >.8> > > > > > > In general, a TCP stack doesn't know how much data the application > > > might eventually want to send. So I guess the basic advice might be > > > to wait n_1 RTTs before opening a second subflow, then n_2 RTTs > > > before opening a third and so on. > > > > > > Has anyone done any research on this to find the best tradeoff (ie > > > best values of n_1, n_2, etc) between improved performance/resilience > > > and wasting signalling packets? I imagine this heuristic would be > > > clocked by RTTs, not real-time or bytes, because that would likely be > > > the relevant parameter for the signalling tradeoff. > > > > > > Then, in the API draft, there might be some discussion on whether the > > > application can tell the transport how much data it intends to send, > > > so the heuristics can be less blind. > > > > > > Thoughts? > > > > > > > > > Bob > > > > > > > > > ________________________________________________________________ > > > Bob Briscoe, BT Innovate & Design >=20 > ________________________________________________________________ > Bob Briscoe, BT Innovate & Design From bob.briscoe@bt.com Fri Feb 4 09:50:18 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 224D53A6A0E for ; Fri, 4 Feb 2011 09:50:18 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.239 X-Spam-Level: X-Spam-Status: No, score=-3.239 tagged_above=-999 required=5 tests=[AWL=-0.240, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id H5-qcWOquttj for ; Fri, 4 Feb 2011 09:50:15 -0800 (PST) Received: from smtp4.smtp.bt.com (smtp4.smtp.bt.com [217.32.164.151]) by core3.amsl.com (Postfix) with ESMTP id D3A373A69DC for ; Fri, 4 Feb 2011 09:50:14 -0800 (PST) Received: from i2kc06-ukbr.domain1.systemhost.net ([193.113.197.70]) by smtp4.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 4 Feb 2011 17:53:39 +0000 Received: from cbibipnt05.iuser.iroot.adidom.com ([147.149.196.177]) by i2kc06-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Fri, 4 Feb 2011 17:53:38 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt05.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1296842017713; Fri, 4 Feb 2011 17:53:37 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p14HrZij020929; Fri, 4 Feb 2011 17:53:35 GMT Message-Id: <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Fri, 04 Feb 2011 17:53:34 +0000 To: "Ford, Alan" From: Bob Briscoe In-Reply-To: <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad. roke.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad.roke.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 04 Feb 2011 17:53:38.0587 (UTC) FILETIME=[73B88EB0:01CBC494] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2011 17:50:18 -0000 Alan, At 13:19 04/02/2011, Ford, Alan wrote: >Bob, > >On the surface this sounds like a sensible heuristic to use - >essentially, you are saying, do not use MPTCP until there has been >congestion and thus you want to initiate resource pooling. I didn't intend to say that, and I don't think I did. I thought I said the decision should depend on how many packet's worth of data the end-point has to send. >However, there are a couple of (smallish) issues with this approach. In >the client/server model, it is the server who is in a position to know >its buffer is filling up and thus how many new subflows it wants. But in >many cases (e.g. client addresses behind NATs) only the client is in a >position to create these subflows - and the client won't know about the >amount of data in the server's buffer. Good point. Then an API at the client *would* be useful... In my first mail I said that, in the common case of an HTTP GET, the first data packet from the server usually contains an HTTP header that says how many octets the object will be. So, as long as the client has a handy casio calculator (or even a processor) it can work out how many packets that implies, and decide whether to start more sub-flows. >(Some intelligence could be >applied, e.g. opening subflows at the client side when the window size >has stayed static for N packets - this is probably good enough for most >cases, although adding more than one extra subflow would have to wait >until both subflow windows have stayed static). Ooh. I would assume it wants to be opening sub-flows long before it would have time to measure such heuristics. The idea of MPTCP is surely to have all the sub-flows there, and let congestion control decide how much to use them. >The other issue with this approach is that it does not provide resource >pooling for resilience, only throughput. Therefore, a second heuristic >about time/number of packets elapsed should be applied, to provide >resilience to longer-lived flows. I assume "this approach" is what you *thought* I said (something to do with congestion). I believe, what I actually said satisfies your concern here about resilience. >Most of this behaviour can be controlled by the endpoints, and does not >require any protocol changes. Or are there any cases where we may want a >flag to say "I speak MPTCP, but don't want to use it yet"? I am not saying there is any need for a richer wire protocol here. If I speak MPTCP but don't want to open subflows, surely I just don't open a sub-flow, or I don't respond when asked to open one. But a protocol is more than the wire protocol. It also includes the algorithms. I'm saying some baseline algo for the decision when to open sub-flows needs to be in the multi-address spec. But made clear that it's not mandatory and better heuristics would be encouraged. Otherwise, if most implementers take the easy option, they will open all possible sub-flows as soon as possible, and we will have storms of redundant control packets due to all the flows on the Internet (25%?) that are no more than 3 packets. For instance, take Wischik's diagram of the files you access when you log in to gmail: . At the time each was a separate HTTP request. Even if some of these were within one TCP connection there's may-be 50 TCP connections just to login to gmail (I've just accessed CNN.com and opened 20 new TCPs in the process). Each currently requires a 3-way hand-shake before data flows, and most are single packet objects - only one is bigger than 3x 1500B packets. A naive MPTCP that opened 3 additional sub-flows for each object (say) would send 3 redundant SYNs, SYN-ACKs, ACKs, FINs, FIN-ACKs then 3 more FINs & FIN-ACKs in the other direction.* That's 21x50 = 1050 redundant control packets for a fairly normal case. Even if I'm over-estimating by a factor of 10, just think of all the unnecessary interrupt handling, if nothing else. Bob * PS. I haven't actually checked whether MPTCP closes sub-flows like this - apologies if I am over-egging the pudding. >Regards, >Alan > > > -----Original Message----- > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > Sent: 03 February 2011 23:45 > > To: Ford, Alan > > Cc: multipathtcp@ietf.org > > Subject: RE: mptcp-multiaddressed: How long before multipath starts? > > > > Alan, > > > > Surely this shouldn't be thought of as primarily an API issue? - the > > basic MPTCP implementation should be able to decide when to start > > opening sub-flows, whether or not the app wants to give it more > > information thru an API. > > > > That's why I suggested at least something needs to be said on this in > > the heuristics section of >(section-3). > > > > > > Let's put some ball-park numbers on this. My intention is NOT to > > start a discussion about whether these heuristics are correct, just > > to show this question will need an answer in a basic implementation > > (without any API). > > > > I think everyone would agree that it wouldn't be worth creating any > > sub-flows if the transfer needed no more than 3 packets. With an > > initial window of 3, a TCP server can get 3 packets to a client in > > 2RTTs (counting from when the client sends the initial SYN). Any more > > sub-flows would be redundant (other than for sending copies for >resilience). > > > > Accurate data on what proportion of flows on the Internet consist of > > no more than 3 packets is hard to come by. Perhaps 50% perhaps 75%. > > But even if it's only 25%, that's enough to have to address the > > question of whether a basic MPTCP implementation will need some logic > > to decide whether and when it is going to start opening sub-flows. > > > > > > Indeed, if one can work out a good default behaviour based on a > > tradeoff between the cost of signalling for extra sub-flows and their > > benefit, I cannot see much reason for an app to want to override this > > - not much need for an API. > > > > > > > > To kick-off a discussion on a baseline heuristic, if IW=N, I would > > propose we should ignore the cost of extra SYNs and SYN-ACKs along > > the same arguments as Wischik's "Short Messages". Then, as soon as > > the server sees some data buffered for the client (implying the app > > has got a request for data from the client), it would be worth the > > server opening one sub-flow for each N segments it will have to send > > to empty the buffer (limited by the no. of distinct sub-flows that > > are possible). > > > > Incidentally, this would be a neat way of increasing the effective IW > > while spreading out the packets over different RTTs, so they don't > > all arrive in the queue at the client back-to-back. > > > > The earliest a server could request more sub-flows is in parallel to > > its first SYN-ACK. But at that point it has no data from the client > > yet, so it doesn't know what the client wants, nor how big it will > > be. It could optimistically start a sub-flow though. At this point, > > we start to need some research on whether this would be worth it... > > > > > > Basically, I'm saying that we shouldn't expect implementers to have > > to be researchers. At minimum the results of even the most basic > > research should be documented in the heuristics section, so an > > implementer who isn't a researcher can just do what it says. > > > > > > Bob > > > > At 17:42 01/02/2011, Ford, Alan wrote: > > >Hi Bob, all, > > > > > >Thanks for your comments. You are of course right that MPTCP is not > > >appropriate for short flows, the problem naturally being the >difficulty > > >in deciding when a flow is long-lived or not. > > > > > >The choice of these heuristics would be a very >implementation-specific > > >thing, and I'm not sure how much there is we can recommend for the > > >protocol draft. However, we'll add a little commentary on this in the > > >next revision (probably recommending waiting for large scale > > >experimental results, however - as you say, there may well be a >numeric > > >heuristic we can use). > > > > > >In the API draft, we already make a comment (Section 4.4): > > > > > > The enabling of MPTCP, either by application or by system >defaults, > > > does not necessarily mean that MPTCP will always be used. Both > > > endpoints must support MPTCP, and there must be multiple >addresses at > > > at least one endpoint, for MPTCP to be used. Even if those > > > requirements are met, however, MPTCP may not be immediately used >on a > > > connection. It may make sense for multiple paths to be brought >into > > > operation only after a given period of time, or if the connection >is > > > saturated. > > > > > >However we can probably say a bit more about this. An enhanced API > > >should indeed allow an application to signal this (and change its >mind > > >during operation); we will make more mention of that in the Extended >API > > >appendix in the next revision. This would be an enhancement to the >basic > > >"MPTCP on/off" decision - the enabling of MPTCP can be on by default >but > > >there needs to be a way of stating that there is no need for >additional > > >subflows at this time. Come to think of it, this could have protocol > > >implications and may require an additional flag on the initial MPTCP >SYN > > >exchange. > > > > > >Regards, > > >Alan > > > > > > > -----Original Message----- > > > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > > > Sent: 31 January 2011 17:15 > > > > To: draft-ietf-mptcp-multiaddressed@tools.ietf.org > > > > Cc: multipathtcp@ietf.org > > > > Subject: mptcp-multiaddressed: How long before multipath starts? > > > > > > > > Alan Costin, Mark, or ANOther, > > > > > > > > The large majority of TCP flows are very short. If there will only >be > > > > one or two packets to send, there will be no point sending off >loads > > > > of requests to open sub-flows straight after the first 3-way > > >handshake. > > > > > > > > So I imagined there would be some advice in one of the MPTCP I-Ds >on > > > > how long a MPTCP should wait until it attempts to open sub-flows. > > > > > > > > I can't find this from a quick scan. I would have expected it to >be > > > > in the heuristics section of draft-ietf-mptcp-multiaddressed: > > > > > > > >3> >.8> > > > > > > > > In general, a TCP stack doesn't know how much data the application > > > > might eventually want to send. So I guess the basic advice might >be > > > > to wait n_1 RTTs before opening a second subflow, then n_2 RTTs > > > > before opening a third and so on. > > > > > > > > Has anyone done any research on this to find the best tradeoff (ie > > > > best values of n_1, n_2, etc) between improved >performance/resilience > > > > and wasting signalling packets? I imagine this heuristic would be > > > > clocked by RTTs, not real-time or bytes, because that would likely >be > > > > the relevant parameter for the signalling tradeoff. > > > > > > > > Then, in the API draft, there might be some discussion on whether >the > > > > application can tell the transport how much data it intends to >send, > > > > so the heuristics can be less blind. > > > > > > > > Thoughts? > > > > > > > > > > > > Bob > > > > > > > > > > > > ________________________________________________________________ > > > > Bob Briscoe, BT Innovate & Design > > > > ________________________________________________________________ > > Bob Briscoe, BT Innovate & Design ________________________________________________________________ Bob Briscoe, BT Innovate & Design From christoph.paasch@uclouvain.be Fri Feb 4 13:51:01 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BDF653A69D4 for ; Fri, 4 Feb 2011 13:51:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.299 X-Spam-Level: X-Spam-Status: No, score=-6.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_21=0.6, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id d6ARtBWJw7VW for ; Fri, 4 Feb 2011 13:50:58 -0800 (PST) Received: from smtp1.sgsi.ucl.ac.be (smtp.sgsi.ucl.ac.be [130.104.5.67]) by core3.amsl.com (Postfix) with ESMTP id E6C293A69D3 for ; Fri, 4 Feb 2011 13:50:57 -0800 (PST) Received: from cpaasch-mac.localnet (208.117-201-80.adsl-dyn.isp.belgacom.be [80.201.117.208]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: cpaasch@smtp1.sgsi.ucl.ac.be) by smtp1.sgsi.ucl.ac.be (Postfix) with ESMTPSA id B96EDE998F; Fri, 4 Feb 2011 22:54:18 +0100 (CET) X-DKIM: Sendmail DKIM Filter v2.8.2 smtp1.sgsi.ucl.ac.be B96EDE998F DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=uclouvain.be; s=selucl; t=1296856458; bh=/H2jw6kklA7ciFn1mphTxxoE4Vk+Bd4AEQiSguFSYTE=; h=From:Reply-To:To:Subject:Date:Cc:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id; b=zurTcrqgw5+urZ5VCeAC9zej+jGaJFzFE9Op6MXBu/iuAQur3fgUgh7R+MgDa+MfI 6lV6gkNR03JrwGAqZCN7krfkNa75yO9Xlo1lrY6QL4YlxQOBTh9DVXypN8erlVw+LU jhLY1RS+GFS+Bhy4od198oU4vhyVdTjISqHkWcIU= From: Christoph Paasch Organization: =?iso-8859-1?q?Universit=E9_Catholique_de?= Louvain To: multipathtcp@ietf.org, Bob Briscoe Date: Fri, 4 Feb 2011 22:54:18 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.35-26-generic; KDE/4.6.0; x86_64; ; ) References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad.roke.co.uk> <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> In-Reply-To: <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201102042254.18276.christoph.paasch@uclouvain.be> X-Virus-Scanned: clamav-milter 0.96.5-exp at smtp-1.sipr-dc.ucl.ac.be X-Virus-Status: Clean X-Sgsi-Spamcheck: SASL authenticated, X-SGSI-MailScanner-ID: B96EDE998F.00000 X-SGSI-MailScanner: Found to be clean X-SGSI-From: christoph.paasch@uclouvain.be X-SGSI-Spam-Status: No Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list Reply-To: christoph.paasch@uclouvain.be List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2011 21:51:01 -0000 Hi all, additionally to the control-overhead that Bob is talking about (all the=20 additional SYN's,...) for short-lived flows, there is also the overhead for= =20 the implementation in generating the keys at the connection initiation (eve= n=20 if the flow will be only very short), and trying to send SYN's even if the= =20 peer may ignore these SYN's (due to the application setting some kind of=20 socket-option). I think that the initial handshake should be as close as possible to regula= r=20 TCP (just a token should be exchanged), and later over this initial subflow= ,=20 the control-information for establishing a new subflow should be exchanged. Have a look at two past mails, I sended to the list: http://www.ietf.org/mail-archive/web/multipathtcp/current/msg01300.html http://www.ietf.org/mail-archive/web/multipathtcp/current/msg01301.html That way, we are sure that both sides are willing to open a new subflow,=20 *before* sending any SYN that tries to establish the new subflow. If we just ignore the SYN (if we do not want to open a new subflow), this w= ill=20 be costly from an implementation point-of-view. Because the sender of the S= YN=20 is uselessly allocating the memory for the sub-socket, retransmitting it's= =20 SYN,... What do you think? Cheers, Christoph On Friday, February 04, 2011 wrote Bob Briscoe: > Alan, >=20 > At 13:19 04/02/2011, Ford, Alan wrote: > >Bob, > > > >On the surface this sounds like a sensible heuristic to use - > >essentially, you are saying, do not use MPTCP until there has been > >congestion and thus you want to initiate resource pooling. >=20 > I didn't intend to say that, and I don't think I did. I thought I > said the decision should depend on how many packet's worth of data > the end-point has to send. >=20 > >However, there are a couple of (smallish) issues with this approach. In > >the client/server model, it is the server who is in a position to know > >its buffer is filling up and thus how many new subflows it wants. But in > >many cases (e.g. client addresses behind NATs) only the client is in a > >position to create these subflows - and the client won't know about the > >amount of data in the server's buffer. >=20 > Good point. >=20 > Then an API at the client *would* be useful... In my first mail I > said that, in the common case of an HTTP GET, the first data packet > from the server usually contains an HTTP header that says how many > octets the object will be. So, as long as the client has a handy > casio calculator (or even a processor) it can work out how many > packets that implies, and decide whether to start more sub-flows. >=20 > >(Some intelligence could be > >applied, e.g. opening subflows at the client side when the window size > >has stayed static for N packets - this is probably good enough for most > >cases, although adding more than one extra subflow would have to wait > >until both subflow windows have stayed static). >=20 > Ooh. I would assume it wants to be opening sub-flows long before it > would have time to measure such heuristics. The idea of MPTCP is > surely to have all the sub-flows there, and let congestion control > decide how much to use them. >=20 > >The other issue with this approach is that it does not provide resource > >pooling for resilience, only throughput. Therefore, a second heuristic > >about time/number of packets elapsed should be applied, to provide > >resilience to longer-lived flows. >=20 > I assume "this approach" is what you *thought* I said (something to > do with congestion). I believe, what I actually said satisfies your > concern here about resilience. >=20 > >Most of this behaviour can be controlled by the endpoints, and does not > >require any protocol changes. Or are there any cases where we may want a > >flag to say "I speak MPTCP, but don't want to use it yet"? >=20 > I am not saying there is any need for a richer wire protocol here. If > I speak MPTCP but don't want to open subflows, surely I just don't > open a sub-flow, or I don't respond when asked to open one. >=20 > But a protocol is more than the wire protocol. It also includes the > algorithms. I'm saying some baseline algo for the decision when to > open sub-flows needs to be in the multi-address spec. But made clear > that it's not mandatory and better heuristics would be encouraged. >=20 > Otherwise, if most implementers take the easy option, they will open > all possible sub-flows as soon as possible, and we will have storms > of redundant control packets due to all the flows on the Internet > (25%?) that are no more than 3 packets. >=20 > For instance, take Wischik's diagram of the files you access when you > log in to gmail: > . At > the time each was a separate HTTP request. Even if some of these were > within one TCP connection there's may-be 50 TCP connections just to > login to gmail (I've just accessed CNN.com and opened 20 new TCPs in > the process). >=20 > Each currently requires a 3-way hand-shake before data flows, and > most are single packet objects - only one is bigger than 3x 1500B > packets. A naive MPTCP that opened 3 additional sub-flows for each > object (say) would send 3 redundant SYNs, SYN-ACKs, ACKs, FINs, > FIN-ACKs then 3 more FINs & FIN-ACKs in the other direction.* That's > 21x50 =3D 1050 redundant control packets for a fairly normal case. >=20 > Even if I'm over-estimating by a factor of 10, just think of all the > unnecessary interrupt handling, if nothing else. >=20 >=20 > Bob >=20 > * PS. I haven't actually checked whether MPTCP closes sub-flows like > this - apologies if I am over-egging the pudding. >=20 > >Regards, > >Alan > > > > > -----Original Message----- > > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > > Sent: 03 February 2011 23:45 > > > To: Ford, Alan > > > Cc: multipathtcp@ietf.org > > > Subject: RE: mptcp-multiaddressed: How long before multipath starts? > > >=20 > > > Alan, > > >=20 > > > Surely this shouldn't be thought of as primarily an API issue? - the > > > basic MPTCP implementation should be able to decide when to start > > > opening sub-flows, whether or not the app wants to give it more > > > information thru an API. > > >=20 > > > That's why I suggested at least something needs to be said on this in > > > the heuristics section of > > > >(section-3). > > > > > Let's put some ball-park numbers on this. My intention is NOT to > > > start a discussion about whether these heuristics are correct, just > > > to show this question will need an answer in a basic implementation > > > (without any API). > > >=20 > > > I think everyone would agree that it wouldn't be worth creating any > > > sub-flows if the transfer needed no more than 3 packets. With an > > > initial window of 3, a TCP server can get 3 packets to a client in > > > 2RTTs (counting from when the client sends the initial SYN). Any more > > > sub-flows would be redundant (other than for sending copies for > > > >resilience). > > > > > Accurate data on what proportion of flows on the Internet consist of > > > no more than 3 packets is hard to come by. Perhaps 50% perhaps 75%. > > > But even if it's only 25%, that's enough to have to address the > > > question of whether a basic MPTCP implementation will need some logic > > > to decide whether and when it is going to start opening sub-flows. > > >=20 > > >=20 > > > Indeed, if one can work out a good default behaviour based on a > > > tradeoff between the cost of signalling for extra sub-flows and their > > > benefit, I cannot see much reason for an app to want to override this > > > - not much need for an API. > > >=20 > > >=20 > > >=20 > > > To kick-off a discussion on a baseline heuristic, if IW=3DN, I would > > > propose we should ignore the cost of extra SYNs and SYN-ACKs along > > > the same arguments as Wischik's "Short Messages". Then, as soon as > > > the server sees some data buffered for the client (implying the app > > > has got a request for data from the client), it would be worth the > > > server opening one sub-flow for each N segments it will have to send > > > to empty the buffer (limited by the no. of distinct sub-flows that > > > are possible). > > >=20 > > > Incidentally, this would be a neat way of increasing the effective IW > > > while spreading out the packets over different RTTs, so they don't > > > all arrive in the queue at the client back-to-back. > > >=20 > > > The earliest a server could request more sub-flows is in parallel to > > > its first SYN-ACK. But at that point it has no data from the client > > > yet, so it doesn't know what the client wants, nor how big it will > > > be. It could optimistically start a sub-flow though. At this point, > > > we start to need some research on whether this would be worth it... > > >=20 > > >=20 > > > Basically, I'm saying that we shouldn't expect implementers to have > > > to be researchers. At minimum the results of even the most basic > > > research should be documented in the heuristics section, so an > > > implementer who isn't a researcher can just do what it says. > > >=20 > > >=20 > > > Bob > > >=20 > > > At 17:42 01/02/2011, Ford, Alan wrote: > > > >Hi Bob, all, > > > > > > > >Thanks for your comments. You are of course right that MPTCP is not > > > >appropriate for short flows, the problem naturally being the > > > >difficulty > > > > > >in deciding when a flow is long-lived or not. > > > > > > > >The choice of these heuristics would be a very > > > >implementation-specific > > > > > >thing, and I'm not sure how much there is we can recommend for the > > > >protocol draft. However, we'll add a little commentary on this in the > > > >next revision (probably recommending waiting for large scale > > > >experimental results, however - as you say, there may well be a > > > >numeric > > > > > >heuristic we can use). > > > > > > > >In the API draft, we already make a comment (Section 4.4): > > > > The enabling of MPTCP, either by application or by system > > > >defaults, > > > > > > does not necessarily mean that MPTCP will always be used. Both > > > > endpoints must support MPTCP, and there must be multiple > > > >addresses at > > > > > > at least one endpoint, for MPTCP to be used. Even if those > > > > requirements are met, however, MPTCP may not be immediately used > > > >on a > > > > > > connection. It may make sense for multiple paths to be brought > > > >into > > > > > > operation only after a given period of time, or if the connection > > > >is > > > > > > saturated. > > > > > > > >However we can probably say a bit more about this. An enhanced API > > > >should indeed allow an application to signal this (and change its > > > >mind > > > > > >during operation); we will make more mention of that in the Extended > > > >API > > > > > >appendix in the next revision. This would be an enhancement to the > > > >basic > > > > > >"MPTCP on/off" decision - the enabling of MPTCP can be on by default > > > >but > > > > > >there needs to be a way of stating that there is no need for > > > >additional > > > > > >subflows at this time. Come to think of it, this could have protocol > > > >implications and may require an additional flag on the initial MPTCP > > > >SYN > > > > > >exchange. > > > > > > > >Regards, > > > >Alan > > > > > > > > > -----Original Message----- > > > > > From: Bob Briscoe [mailto:bob.briscoe@bt.com] > > > > > Sent: 31 January 2011 17:15 > > > > > To: draft-ietf-mptcp-multiaddressed@tools.ietf.org > > > > > Cc: multipathtcp@ietf.org > > > > > Subject: mptcp-multiaddressed: How long before multipath starts? > > > > >=20 > > > > > Alan Costin, Mark, or ANOther, > > > > >=20 > > > > > The large majority of TCP flows are very short. If there will only > > > >be > > > > > > > one or two packets to send, there will be no point sending off > > > >loads > > > > > > > of requests to open sub-flows straight after the first 3-way > > > > > > > >handshake. > > > > > > > > > So I imagined there would be some advice in one of the MPTCP I-Ds > > > >on > > > > > > > how long a MPTCP should wait until it attempts to open sub-flows. > > > > >=20 > > > > > I can't find this from a quick scan. I would have expected it to > > > >be > > > > > > > in the heuristics section of draft-ietf-mptcp-multiaddressed: > > > > > >3> >.8> > > > > > > > In general, a TCP stack doesn't know how much data the application > > > > > might eventually want to send. So I guess the basic advice might > > > >be > > > > > > > to wait n_1 RTTs before opening a second subflow, then n_2 RTTs > > > > > before opening a third and so on. > > > > >=20 > > > > > Has anyone done any research on this to find the best tradeoff (ie > > > > > best values of n_1, n_2, etc) between improved > > > >performance/resilience > > > > > > > and wasting signalling packets? I imagine this heuristic would be > > > > > clocked by RTTs, not real-time or bytes, because that would likely > > > >be > > > > > > > the relevant parameter for the signalling tradeoff. > > > > >=20 > > > > > Then, in the API draft, there might be some discussion on whether > > > >the > > > > > > > application can tell the transport how much data it intends to > > > >send, > > > > > > > so the heuristics can be less blind. > > > > >=20 > > > > > Thoughts? > > > > >=20 > > > > >=20 > > > > > Bob > > > > >=20 > > > > >=20 > > > > > ________________________________________________________________ > > > > > Bob Briscoe, BT Innovate & Design > > >=20 > > > ________________________________________________________________ > > > Bob Briscoe, BT Innovate & Design >=20 > ________________________________________________________________ > Bob Briscoe, BT Innovate & Design >=20 > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp =2D- Christoph Paasch PhD Student IP Networking Lab --- http://inl.info.ucl.ac.be MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp Universit=E9 Catholique de Louvain www.rollerbulls.be =2D- From yoshifumi.nishida@gmail.com Mon Feb 7 00:33:26 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4344A3A6CBB for ; Mon, 7 Feb 2011 00:33:26 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.977 X-Spam-Level: X-Spam-Status: No, score=-102.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kOg35fmc29EX for ; Mon, 7 Feb 2011 00:33:25 -0800 (PST) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by core3.amsl.com (Postfix) with ESMTP id CC64E3A6CC4 for ; Mon, 7 Feb 2011 00:33:24 -0800 (PST) Received: by wwa36 with SMTP id 36so4810561wwa.13 for ; Mon, 07 Feb 2011 00:33:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=ff7NDhR5Git/oXz8GQp0kBqUX6RW9fcjb44n+uZpcTY=; b=FgTQbFKl++pLvlkzKAUHvoTh0hsdX79H/759EPQh2SjO6QMWAjIiRCagWIQvSi4mvI Dl2R1txD+TJFHsAAvQnXbmQc/fUD8Kygrwv2D70yFgAt5S1oarEoSrLxzuo4EeXd3FXG Nw5UFZR3532EwueN2mQtKGJ0GjrCO/v8PetBg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=n0fsMnzss2NWnfB9tRHcop6LsEXsatt0cuhiI+5P51DAxq1+dG1jXYkR2O7eKdP97E T3vAucLkvzXLsWV7/uBI2VW//Q2nWcTkBV7eWtKikCsa54bBTM1k1TeHWP1aweMbiRmL ZYNh15M0Qp/vEjAcaOoGXKjhccdCTbELM4Du4= MIME-Version: 1.0 Received: by 10.217.2.73 with SMTP id o51mr2159003wes.66.1297067607654; Mon, 07 Feb 2011 00:33:27 -0800 (PST) Sender: yoshifumi.nishida@gmail.com Received: by 10.216.89.11 with HTTP; Mon, 7 Feb 2011 00:33:27 -0800 (PST) In-Reply-To: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> Date: Mon, 7 Feb 2011 00:33:27 -0800 X-Google-Sender-Auth: TD8CqG4Ox7ygyjfF2UIP11n29P0 Message-ID: From: Yoshifumi Nishida To: Bob Briscoe Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 08:33:26 -0000 Hi, I personally prefer to initiate another subflow after the preceding flow enters congestion avoidance. It looks a simple rule and we can avoid multiple subflows to perform slow-start at the same time. Or, does it hurt performance? Thanks, -- Yoshifumi Nishida nishida@sfc.wide.ad.jp 2011/1/31 Bob Briscoe : > Alan Costin, Mark, or ANOther, > > The large majority of TCP flows are very short. If there will only be one= or > two packets to send, there will be no point sending off loads of requests= to > open sub-flows straight after the first 3-way handshake. > > So I imagined there would be some advice in one of the MPTCP I-Ds on how > long a MPTCP should wait until it attempts to open sub-flows. > > I can't find this from a quick scan. I would have expected it to be in th= e > heuristics section of draft-ietf-mptcp-multiaddressed: > > > In general, a TCP stack doesn't know how much data the application might > eventually want to send. So I guess the basic advice might be to wait n_1 > RTTs before opening a second subflow, then n_2 RTTs before opening a thir= d > and so on. > > Has anyone done any research on this to find the best tradeoff (ie best > values of n_1, n_2, etc) between improved performance/resilience and wast= ing > signalling packets? I imagine this heuristic would be clocked by RTTs, no= t > real-time or bytes, because that would likely be the relevant parameter f= or > the signalling tradeoff. > > Then, in the API draft, there might be some discussion on whether the > application can tell the transport how much data it intends to send, so t= he > heuristics can be less blind. > > Thoughts? > > > Bob > > > ________________________________________________________________ > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0BT Innovate & Design > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp > From christoph.paasch@uclouvain.be Mon Feb 7 01:10:07 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6B5C93A6CF9 for ; Mon, 7 Feb 2011 01:10:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.599 X-Spam-Level: X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0Rfa4k0xm+pU for ; Mon, 7 Feb 2011 01:10:06 -0800 (PST) Received: from smtp4.sgsi.ucl.ac.be (smtp.sgsi.ucl.ac.be [130.104.5.67]) by core3.amsl.com (Postfix) with ESMTP id 0A2AD3A6CC8 for ; Mon, 7 Feb 2011 01:10:05 -0800 (PST) Received: from cpaasch-mac.localnet (130.92-67-87.adsl-dyn.isp.belgacom.be [87.67.92.130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: cpaasch@smtp4.sgsi.ucl.ac.be) by smtp4.sgsi.ucl.ac.be (Postfix) with ESMTPSA id 4C971F28FF; Mon, 7 Feb 2011 10:09:04 +0100 (CET) X-DKIM: Sendmail DKIM Filter v2.8.2 smtp4.sgsi.ucl.ac.be 4C971F28FF DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=uclouvain.be; s=selucl; t=1297069744; bh=F33OC1BEkTSl5fREZM2Rfg/i3E/d3RdpptIhHvTNvrs=; h=From:Reply-To:To:Subject:Date:Cc:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id; b=d5k7xmp6c6yvCMCkHoeQp5BPYwzr96AWUmVZ/o8QYW0QTdpdP/hpRSLmWdzHJr93l GBcSPFhx4UPmImsoq0yLtVcDNwuLRP45jpCCQJSbPxZjvwDO1JTJo/yhH9RYYX4GF/ +ciFt3iFsGCOGDoFia09yd1D1VbxdqGjgSuDRVeQ= From: Christoph Paasch Organization: =?iso-8859-1?q?Universit=E9_Catholique_de?= Louvain To: multipathtcp@ietf.org Date: Mon, 7 Feb 2011 10:09:03 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.35-26-generic; KDE/4.6.0; x86_64; ; ) References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <201102071009.03576.christoph.paasch@uclouvain.be> X-Virus-Scanned: clamav-milter 0.96.5-exp at smtp-4.sipr-dc.ucl.ac.be X-Virus-Status: Clean X-Sgsi-Spamcheck: SASL authenticated, X-SGSI-MailScanner-ID: 4C971F28FF.00000 X-SGSI-MailScanner: Found to be clean X-SGSI-From: christoph.paasch@uclouvain.be X-SGSI-Spam-Status: No Cc: Bob Briscoe Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list Reply-To: christoph.paasch@uclouvain.be List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 09:10:07 -0000 Hi, this sounds like a reasonable approach, to avoid creating congestion due to= =20 several subflows being in slow-start. However, I see one problem: After the initial subflow enters congestion avoidance, we try the next one.= =20 However the SYN's of this new subflow get blocked somewhere on their path.= =20 Then we will be stuck with only the initial subflow for a long time (until = the=20 TCP-stack gives up retransmitting the SYN's of the second subflow). And onl= y=20 then we will try the third subflow. Cheers, Christoph On Monday, February 07, 2011 wrote Yoshifumi Nishida: > Hi, > I personally prefer to initiate another subflow after the preceding > flow enters congestion avoidance. It looks a simple rule and we can > avoid multiple subflows to perform slow-start at the same time. > Or, does it hurt performance? >=20 > Thanks, > -- > Yoshifumi Nishida > nishida@sfc.wide.ad.jp >=20 > 2011/1/31 Bob Briscoe : > > Alan Costin, Mark, or ANOther, > >=20 > > The large majority of TCP flows are very short. If there will only be o= ne > > or two packets to send, there will be no point sending off loads of > > requests to open sub-flows straight after the first 3-way handshake. > >=20 > > So I imagined there would be some advice in one of the MPTCP I-Ds on how > > long a MPTCP should wait until it attempts to open sub-flows. > >=20 > > I can't find this from a quick scan. I would have expected it to be in > > the heuristics section of draft-ietf-mptcp-multiaddressed: > > > 8> > >=20 > > In general, a TCP stack doesn't know how much data the application might > > eventually want to send. So I guess the basic advice might be to wait n= _1 > > RTTs before opening a second subflow, then n_2 RTTs before opening a > > third and so on. > >=20 > > Has anyone done any research on this to find the best tradeoff (ie best > > values of n_1, n_2, etc) between improved performance/resilience and > > wasting signalling packets? I imagine this heuristic would be clocked by > > RTTs, not real-time or bytes, because that would likely be the relevant > > parameter for the signalling tradeoff. > >=20 > > Then, in the API draft, there might be some discussion on whether the > > application can tell the transport how much data it intends to send, so > > the heuristics can be less blind. > >=20 > > Thoughts? > >=20 > >=20 > > Bob > >=20 > >=20 > > ________________________________________________________________ > > Bob Briscoe, BT Innovate & Design > > _______________________________________________ > > multipathtcp mailing list > > multipathtcp@ietf.org > > https://www.ietf.org/mailman/listinfo/multipathtcp >=20 > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp =2D- Christoph Paasch PhD Student IP Networking Lab --- http://inl.info.ucl.ac.be MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp Universit=E9 Catholique de Louvain www.rollerbulls.be =2D- From mark.j.handley@gmail.com Mon Feb 7 01:46:01 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B50A03A6D7C for ; Mon, 7 Feb 2011 01:46:00 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.977 X-Spam-Level: X-Spam-Status: No, score=-2.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ec88BasxAuCP for ; Mon, 7 Feb 2011 01:45:57 -0800 (PST) Received: from mail-wy0-f172.google.com (mail-wy0-f172.google.com [74.125.82.172]) by core3.amsl.com (Postfix) with ESMTP id 9FCC03A6D70 for ; Mon, 7 Feb 2011 01:45:56 -0800 (PST) Received: by wyf23 with SMTP id 23so4587284wyf.31 for ; Mon, 07 Feb 2011 01:45:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:from :date:x-google-sender-auth:message-id:subject:to:content-type; bh=mg9Pwz+yhGAtkZOlXbpUUXunYQCZV+w1aOULDYpb9cY=; b=xf3tE8L/UQAJrtetMoWrglWTcn2ykie4RrgiZ38CXY4DA9sW3iE58gFY+nxr60DDvw j9vhhJOVQk01gTHQYQQ4Gxk6DSjFoZ7MS2iG66FghmMaEoJiVAWmQjeNL+TFfudZGlNC xhzv9z+HqmmRIMMMip4ZfaAORWyaHIIRl7Lf4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; b=DA15csMhc6ohVsaEuIXt4U+Zgi1lx/hRAStT63miyUz0mJqooEEUtV0eJXz3hGgngp ldxqaXcQq8a2I7KXOZn2+yz8Ifc/RPkmuCymL0wRZrqDqfrdTMJvNwVMEqm7QX0RSCZd WNulhfSLhHskd3bXwc4CjRBkNePyNBWQM9lBc= Received: by 10.216.48.6 with SMTP id u6mr13454104web.77.1297071959454; Mon, 07 Feb 2011 01:45:59 -0800 (PST) MIME-Version: 1.0 Sender: mark.j.handley@gmail.com Received: by 10.216.8.15 with HTTP; Mon, 7 Feb 2011 01:45:39 -0800 (PST) In-Reply-To: <201102071009.03576.christoph.paasch@uclouvain.be> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> From: Mark Handley Date: Mon, 7 Feb 2011 09:45:39 +0000 X-Google-Sender-Auth: VKg_j3A8EH8XWVwRsFwExgEgq98 Message-ID: To: multipathtcp@ietf.org, Bob Briscoe Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 09:46:01 -0000 The answer, as with many such things, is "it depends". And as such, the protocol specification must avoid being prescriptive on this issue. As MPTCP is deployed, we'll discover precisely when it does and when it does not make sense to start additional subflows, and how many should be started. There's clearly no point in starting a second subflow if the first subflow will terminate before the second ever sends data. So if either end knows that the flow will end very quickly, a second subflow should either not be initiated or not accepted. - Sometimes the host stack can know (for example, the app used sendfile()). - Often the app can know, but unless it's MPTCP-aware, this doesn't help much. - Sometimes the sysadmin knows, and can configure MPTCP on or off for a particular port that exhibits common behaviour. - Sometimes the host stack can predict (it measures how often it turns out to be useful for a particular port). I'm sure people will profile implementations in different circumstances and come up with different heuristics. For example, MPTCP in a datacentre with a redundant topology will probably want to open a lot more subflows than MPTCP over WiFi and 3G on a smartphone. And in some datacenters, we may wish to disable any fancy subflow authentication mechanisms, because they serve little purpose. This changes the cost of establishing more subflows, so changes the answer. In the end, after much profiling, we may just decide that it's fine to always send the second SYN right after the first SYN/ACK in many enviroments. After all, even with large windows we send one ack every two data packets, which is clearly wasteful. It's just not an overhead we care about optimising out. Or maybe wait for the first subflow to exit slowstart. But that has a significant cost if the first subflow has a much higher RTT than the second, and if we set up the first over 3G and the second would use WiFi, the stack could easily predict that. So, I really want to avoid being prescriptive on this issue in the spec. It's not necessary to standardize this - it doesn't affect interoperability or the safe operation of the network. So we can and should leave it to implementors to make smart choices based on implementation and deployment experience. - Mark From bob.briscoe@bt.com Mon Feb 7 11:07:28 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C73963A6EA2 for ; Mon, 7 Feb 2011 11:07:28 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W2DZbeul9m74 for ; Mon, 7 Feb 2011 11:07:27 -0800 (PST) Received: from smtp4.smtp.bt.com (smtp4.smtp.bt.com [217.32.164.151]) by core3.amsl.com (Postfix) with ESMTP id 7CF053A6E9D for ; Mon, 7 Feb 2011 11:07:27 -0800 (PST) Received: from i2kc06-ukbr.domain1.systemhost.net ([193.113.197.70]) by smtp4.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:07:31 +0000 Received: from cbibipnt05.iuser.iroot.adidom.com ([147.149.196.177]) by i2kc06-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:07:30 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt05.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297105649214; Mon, 7 Feb 2011 19:07:29 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p17J7RhR005451; Mon, 7 Feb 2011 19:07:27 GMT Message-Id: <201102071907.p17J7RhR005451@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 07 Feb 2011 19:07:29 +0000 To: Mark Handley From: Bob Briscoe In-Reply-To: References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 07 Feb 2011 19:07:30.0638 (UTC) FILETIME=[44AB1EE0:01CBC6FA] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 19:07:28 -0000 Mark, I wasn't asking for the draft to be prescriptive. Merely wanting some guidelines from what people have found out so far. And some advice on what is very unlikely to be useful (e.g. opening more sub-flows when the stack knows how much outstanding data there is and that it will likely all have been delivered before a new sub-flow is established). Clearly, from all the emails on this thread so far, there's some experience on this already. There's also views on what ought to be done, some with, some without empirical evidence. Therefore, I don't think the draft should say nothing on this issue. If "it depends", the draft should at least say what it depends on - even repeating what your email says in the draft (under "heuristics") would be much more useful than nothing. Bob At 09:45 07/02/2011, Mark Handley wrote: >The answer, as with many such things, is "it depends". And as such, >the protocol specification must avoid being prescriptive on this >issue. As MPTCP is deployed, we'll discover precisely when it does >and when it does not make sense to start additional subflows, and how >many should be started. > >There's clearly no point in starting a second subflow if the first >subflow will terminate before the second ever sends data. So if >either end knows that the flow will end very quickly, a second subflow >should either not be initiated or not accepted. > > - Sometimes the host stack can know (for example, the app used sendfile()). > - Often the app can know, but unless it's MPTCP-aware, this > doesn't help much. > - Sometimes the sysadmin knows, and can configure MPTCP on or off >for a particular port that exhibits common behaviour. > - Sometimes the host stack can predict (it measures how often it >turns out to be useful for a particular port). > >I'm sure people will profile implementations in different >circumstances and come up with different heuristics. For example, >MPTCP in a datacentre with a redundant topology will probably want to >open a lot more subflows than MPTCP over WiFi and 3G on a smartphone. >And in some datacenters, we may wish to disable any fancy subflow >authentication mechanisms, because they serve little purpose. This >changes the cost of establishing more subflows, so changes the answer. > >In the end, after much profiling, we may just decide that it's fine to >always send the second SYN right after the first SYN/ACK in many >enviroments. After all, even with large windows we send one ack every >two data packets, which is clearly wasteful. It's just not an >overhead we care about optimising out. Or maybe wait for the first >subflow to exit slowstart. But that has a significant cost if the >first subflow has a much higher RTT than the second, and if we set up >the first over 3G and the second would use WiFi, the stack could >easily predict that. > > >So, I really want to avoid being prescriptive on this issue in the >spec. It's not necessary to standardize this - it doesn't affect >interoperability or the safe operation of the network. So we can and >should leave it to implementors to make smart choices based on >implementation and deployment experience. > > - Mark ________________________________________________________________ Bob Briscoe, BT Innovate & Design From bob.briscoe@bt.com Mon Feb 7 11:23:43 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1440E3A6985 for ; Mon, 7 Feb 2011 11:23:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.901 X-Spam-Level: X-Spam-Status: No, score=-2.901 tagged_above=-999 required=5 tests=[AWL=-0.698, BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u4DlgjlNwLGX for ; Mon, 7 Feb 2011 11:23:41 -0800 (PST) Received: from smtp3.smtp.bt.com (smtp3.smtp.bt.com [217.32.164.138]) by core3.amsl.com (Postfix) with ESMTP id 67EEF3A6E91 for ; Mon, 7 Feb 2011 11:23:41 -0800 (PST) Received: from i2kc08-ukbr.domain1.systemhost.net ([193.113.197.71]) by smtp3.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:23:45 +0000 Received: from cbibipnt08.iuser.iroot.adidom.com ([147.149.100.81]) by i2kc08-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:23:44 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt08.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297106623736; Mon, 7 Feb 2011 19:23:43 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p17JNf3A005501; Mon, 7 Feb 2011 19:23:41 GMT Message-Id: <201102071923.p17JNf3A005501@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 07 Feb 2011 19:23:43 +0000 To: , Yoshifumi Nishida , Costin Raiciu From: Bob Briscoe In-Reply-To: <201102071009.03576.christoph.paasch@uclouvain.be> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 07 Feb 2011 19:23:44.0662 (UTC) FILETIME=[893B6360:01CBC6FC] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 19:23:43 -0000 Christoph, Yoshifumi, Costin, What is the original logic behind this... Why=20 should waiting for slow-start to end on one flow=20 be relevant to when to start another sub-flow? I=20 don't see why (in general) slow-start on one path=20 should affect slow start on another. I would have thought the opposite: that opening=20 more sub-flows would be a more friendly way to=20 increase the effective initial window than=20 increasing it all on one sub-flow. Because it=20 spreads out the overshoots over multiple paths=20 (assuming underlying physical path diversity) and=20 over time (assuming RTTs are different). Bob At 09:09 07/02/2011, Christoph Paasch wrote: >Hi, > >this sounds like a reasonable approach, to avoid creating congestion due to >several subflows being in slow-start. >However, I see one problem: >After the initial subflow enters congestion avoidance, we try the next one. >However the SYN's of this new subflow get blocked somewhere on their path. >Then we will be stuck with only the initial=20 >subflow for a long time (until the >TCP-stack gives up retransmitting the SYN's of the second subflow). And= only >then we will try the third subflow. > > >Cheers, >Christoph > >On Monday, February 07, 2011 wrote Yoshifumi Nishida: > > Hi, > > I personally prefer to initiate another subflow after the preceding > > flow enters congestion avoidance. It looks a simple rule and we can > > avoid multiple subflows to perform slow-start at the same time. > > Or, does it hurt performance? > > > > Thanks, > > -- > > Yoshifumi Nishida > > nishida@sfc.wide.ad.jp > > > > 2011/1/31 Bob Briscoe : > > > Alan Costin, Mark, or ANOther, > > > > > > The large majority of TCP flows are very short. If there will only be= one > > > or two packets to send, there will be no point sending off loads of > > > requests to open sub-flows straight after the first 3-way handshake. > > > > > > So I imagined there would be some advice in one of the MPTCP I-Ds on= how > > > long a MPTCP should wait until it attempts to open sub-flows. > > > > > > I can't find this from a quick scan. I would have expected it to be in > > > the heuristics section of draft-ietf-mptcp-multiaddressed: > > >=20 > = > 8> > > > > > > In general, a TCP stack doesn't know how much data the application= might > > > eventually want to send. So I guess the basic advice might be to wait= n_1 > > > RTTs before opening a second subflow, then n_2 RTTs before opening a > > > third and so on. > > > > > > Has anyone done any research on this to find the best tradeoff (ie= best > > > values of n_1, n_2, etc) between improved performance/resilience and > > > wasting signalling packets? I imagine this heuristic would be clocked= by > > > RTTs, not real-time or bytes, because that would likely be the= relevant > > > parameter for the signalling tradeoff. > > > > > > Then, in the API draft, there might be some discussion on whether the > > > application can tell the transport how much data it intends to send,= so > > > the heuristics can be less blind. > > > > > > Thoughts? > > > > > > > > > Bob > > > > > > > > > ________________________________________________________________ > > > Bob Briscoe, BT Innovate & Design > > > _______________________________________________ > > > multipathtcp mailing list > > > multipathtcp@ietf.org > > > https://www.ietf.org/mailman/listinfo/multipathtcp > > > > _______________________________________________ > > multipathtcp mailing list > > multipathtcp@ietf.org > > https://www.ietf.org/mailman/listinfo/multipathtcp > >-- >Christoph Paasch >PhD Student > >IP Networking Lab --- http://inl.info.ucl.ac.be >MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp >Universit=E9 Catholique de Louvain > >www.rollerbulls.be >-- ________________________________________________________________ Bob Briscoe, BT Innovate & Design=20 From bob.briscoe@bt.com Mon Feb 7 11:32:59 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8B83C3A6E51 for ; Mon, 7 Feb 2011 11:32:59 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.25 X-Spam-Level: X-Spam-Status: No, score=-3.25 tagged_above=-999 required=5 tests=[AWL=0.349, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yxd4Wa7-M15S for ; Mon, 7 Feb 2011 11:32:58 -0800 (PST) Received: from smtp2.smtp.bt.com (smtp2.smtp.bt.com [217.32.164.150]) by core3.amsl.com (Postfix) with ESMTP id ADE5F3A6D40 for ; Mon, 7 Feb 2011 11:32:57 -0800 (PST) Received: from i2kc06-ukbr.domain1.systemhost.net ([193.113.197.70]) by smtp2.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:33:01 +0000 Received: from cbibipnt05.iuser.iroot.adidom.com ([147.149.196.177]) by i2kc06-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Mon, 7 Feb 2011 19:33:01 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt05.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297107180218; Mon, 7 Feb 2011 19:33:00 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p17JWxx2005521; Mon, 7 Feb 2011 19:32:59 GMT Message-Id: <201102071932.p17JWxx2005521@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 07 Feb 2011 19:33:00 +0000 To: Costin Raiciu From: Bob Briscoe In-Reply-To: <79CA19E3-7F9C-4F05-8C2B-72C19C84C9A9@cs.ucl.ac.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> <79CA19E3-7F9C-4F05-8C2B-72C19C84C9A9@cs.ucl.ac.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 07 Feb 2011 19:33:01.0177 (UTC) FILETIME=[D4F0D690:01CBC6FD] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Feb 2011 19:32:59 -0000 Costin, These are the sort of results I would take care with. Otherwise the "chattering classes" will start coining "Costin's Law", which mandates "Don't start a second sub-flow until after 150 packets" ;) I don't want to belittle this, as you're the only guy so far who's actually got some empirical evidence. Great! But to handle this sort of advice, I agree with Mark that we need to avoid being prescriptive. We should only using empirical evidence when we can explain why the results occurred. Bob At 10:10 02/02/2011, Costin Raiciu wrote: >Hi, > >The short flows question is a valid one; we don't have any >heuristics yet, but we did some measurements for a recent submission. >We measured file transfer completion times for different file sizes. >The setup was very simple: two computer connected with two gigabit >links. We compared TCP with UCL-BE's Linux kernel implementation of >multipath TCP to understand when it is worth using multipath TCP. >These results (and probably a lot more) could be the starting point >to define the heuristics Bob is asking about. > >Attached is the figure showing the results. > > > > >This is the description with put in the paper. >Should MPTCP be enabled for all TCP connections in a data center? We >connected two dual-interface machines to a gigabit switch, and >measured the time to setup a connection and transfer a short file. >TCP can only use one inter- face; MPTCP can also use the second, but >only after the first subflow has negotiated the use of MPTCP and the >second subflow has been established. Figure 17 shows that TCP is >quicker for files of less than about 150 packets, but much slower >thereafter. Closer analysis shows that our MPTCP implementation >slowstarts slightly slower than TCP. The largest difference is about >two RTTs just as TCP fills the link, and decreases thereafter. This >different does not seem fundamen- tal, and is likely a quirk of >slowstart in our implementation. As it is, it would be better to >delay 5ms until the first subflow leaves slowstart before starting the second. > >Cheers, >Costin > >On 2 Feb 2011, at 11:48, Dominik Kaspar wrote: > > > Hi, > > > > My question might be slightly off topic, but this thread on short > > flows motivated me to ask it anyway. > > > > Are there any existing models to predict the TCP throughput after a > > given elapsed time (e.g., the throughput after 2 seconds, after 5 > > seconds, etc...)? The models I am aware of seem not to be applicable > > because they all estimate a long-term average. I'd be grateful for any > > hints and pointers. > > > > Best regards, > > Dominik > > _______________________________________________ > > multipathtcp mailing list > > multipathtcp@ietf.org > > https://www.ietf.org/mailman/listinfo/multipathtcp ________________________________________________________________ Bob Briscoe, BT Innovate & Design From yoshifumi.nishida@gmail.com Mon Feb 7 22:16:54 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A0F0F3A701A for ; Mon, 7 Feb 2011 22:16:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.977 X-Spam-Level: X-Spam-Status: No, score=-102.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pooRlCLWNhxl for ; Mon, 7 Feb 2011 22:16:53 -0800 (PST) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by core3.amsl.com (Postfix) with ESMTP id C31CF3A7018 for ; Mon, 7 Feb 2011 22:16:52 -0800 (PST) Received: by wwa36 with SMTP id 36so5923330wwa.13 for ; Mon, 07 Feb 2011 22:16:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=HoGLB5SgEFr+KlYt2zjlGkYRF+KftbXj2kghtW5psOY=; b=gTswy6AnEE1l1Kmzo92/6TikSs0HPB88zHBzHaDlhtSShcXwxw7GtM57KWkU8veegU dG2eB7M8oLJWGoPguw66QuoWZMt19BvFPccPmWOOk8Dd5OPBVoph6yvtUr+QNHswntKr 7Mjelr+w7rtxfzhRM97MNjFpcR1aQIuZDpn4I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=xx/d2CxhSS4DUtS1byypTHK19Q4ND3/mfCcL1xYqnLKr2e5cVqWfjJX5NLGE+LhTlX PDyEyPIzdiDSsWBZxqBfxIXkyJ+hiLmmWDgpv+ud/NM87+LoLFymx1T2Ugsf7s7CZEjP 59xtSHnssGRnfnif23JmQLOdxcyVuBK+fe/0w= MIME-Version: 1.0 Received: by 10.216.187.71 with SMTP id x49mr16483747wem.111.1297145817475; Mon, 07 Feb 2011 22:16:57 -0800 (PST) Sender: yoshifumi.nishida@gmail.com Received: by 10.216.89.11 with HTTP; Mon, 7 Feb 2011 22:16:57 -0800 (PST) In-Reply-To: <201102071907.p17J7RhR005451@bagheera.jungle.bt.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> <201102071907.p17J7RhR005451@bagheera.jungle.bt.co.uk> Date: Mon, 7 Feb 2011 22:16:57 -0800 X-Google-Sender-Auth: j0507RZZcPKeYGlHgQH504-ixFQ Message-ID: From: Yoshifumi Nishida To: Bob Briscoe Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 06:16:55 -0000 I'm a bit curious about if this is a part of the standard or an item for implementation guideline. I think some people start working on implementation guideline doc... -- Yoshifumi Nishida nishida@sfc.wide.ad.jp 2011/2/7 Bob Briscoe : > Mark, > > I wasn't asking for the draft to be prescriptive. Merely wanting some > guidelines from what people have found out so far. And some advice on wha= t > is very unlikely to be useful (e.g. opening more sub-flows when the stack > knows how much outstanding data there is and that it will likely all have > been delivered before a new sub-flow is established). > > Clearly, from all the emails on this thread so far, there's some experien= ce > on this already. There's also views on what ought to be done, some with, > some without empirical evidence. > > Therefore, I don't think the draft should say nothing on this issue. > > If "it depends", the draft should at least say what it depends on - even > repeating what your email says in the draft (under "heuristics") would be > much more useful than nothing. > > > Bob > > At 09:45 07/02/2011, Mark Handley wrote: >> >> The answer, as with many such things, is "it depends". =A0And as such, >> the protocol specification must avoid being prescriptive on this >> issue. =A0As MPTCP is deployed, we'll discover precisely when it does >> and when it does not make sense to start additional subflows, and how >> many should be started. >> >> There's clearly no point in starting a second subflow if the first >> subflow will terminate before the second ever sends data. =A0So if >> either end knows that the flow will end very quickly, a second subflow >> should either not be initiated or not accepted. >> >> =A0- Sometimes the host stack can know (for example, the app used >> sendfile()). >> =A0- Often the app can know, but unless it's MPTCP-aware, this doesn't h= elp >> much. >> =A0- Sometimes the sysadmin knows, and can configure MPTCP on or off >> for a particular port that exhibits common behaviour. >> =A0- Sometimes the host stack can predict (it measures how often it >> turns out to be useful for a particular port). >> >> I'm sure people will profile implementations in different >> circumstances and come up with different heuristics. =A0For example, >> MPTCP in a datacentre with a redundant topology will probably want to >> open a lot more subflows than MPTCP over WiFi and 3G on a smartphone. >> And in some datacenters, we may wish to disable any fancy subflow >> authentication mechanisms, because they serve little purpose. =A0This >> changes the cost of establishing more subflows, so changes the answer. >> >> In the end, after much profiling, we may just decide that it's fine to >> always send the second SYN right after the first SYN/ACK in many >> enviroments. =A0After all, even with large windows we send one ack every >> two data packets, which is clearly wasteful. =A0It's just not an >> overhead we care about optimising out. =A0Or maybe wait for the first >> subflow to exit slowstart. =A0But that has a significant cost if the >> first subflow has a much higher RTT than the second, and if we set up >> the first over 3G and the second would use WiFi, the stack could >> easily predict that. >> >> >> So, I really want to avoid being prescriptive on this issue in the >> spec. =A0It's not necessary to standardize this - it doesn't affect >> interoperability or the safe operation of the network. =A0So we can and >> should leave it to implementors to make smart choices based on >> implementation and deployment experience. >> >> =A0- Mark > > ________________________________________________________________ > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0BT Innovate & Design > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp > From yoshifumi.nishida@gmail.com Mon Feb 7 22:18:11 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 094143A7029 for ; Mon, 7 Feb 2011 22:18:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.977 X-Spam-Level: X-Spam-Status: No, score=-102.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7LrM5KhikBoC for ; Mon, 7 Feb 2011 22:18:09 -0800 (PST) Received: from mail-wy0-f172.google.com (mail-wy0-f172.google.com [74.125.82.172]) by core3.amsl.com (Postfix) with ESMTP id 43D913A7027 for ; Mon, 7 Feb 2011 22:18:09 -0800 (PST) Received: by wyf23 with SMTP id 23so5705283wyf.31 for ; Mon, 07 Feb 2011 22:18:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=eXX/FttBNOdzJqg86xjdD9pzUoLFnOr5ZN824oQVY6M=; b=M7ZxtC3JY43XqWhxZ7nrx/76Fj3fzkAd1mkjdMIcgHQQ2xvpDC7Oc3zBKhHBbJsDUg 7yPBVMdymTba7tkSrYF6UMSxm7xhrYlNksrHXed1lLrxaR2drvmslrEDrTwiwpxt9J6y yf+wsNO2T/OIKFahbWgA8POlRob/K8RhPPEEw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=s/H4Ia4Rd0ImwgeLeXRx9a5KhBvwRUm5gvXw2SWTz+HrZG0e0V7z4OMzRdNh4PKimp 2axOyd3TGqgwA3NSj6nSb7ZpZmqDu5UmKR5pkZYeIq1kBG9jvqERpW50mjY8d7pflKOq llChWzbHDZCzIF/ICK5MDfSHjnX+JIa8SB948= MIME-Version: 1.0 Received: by 10.216.170.149 with SMTP id p21mr3396368wel.81.1297145894512; Mon, 07 Feb 2011 22:18:14 -0800 (PST) Sender: yoshifumi.nishida@gmail.com Received: by 10.216.89.11 with HTTP; Mon, 7 Feb 2011 22:18:14 -0800 (PST) In-Reply-To: <201102071923.p17JNf3A005501@bagheera.jungle.bt.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> <201102071923.p17JNf3A005501@bagheera.jungle.bt.co.uk> Date: Mon, 7 Feb 2011 22:18:14 -0800 X-Google-Sender-Auth: pXymuNTg5iYaNkcajgXYa2wWTHw Message-ID: From: Yoshifumi Nishida To: Bob Briscoe Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 06:18:11 -0000 Hello Bob, It's nothing special. My concern is we cannot always presume path diversity. So, I just prefer conservative approach. If we can have good knowledge about network configuration like data center scenarios, I think we won't have to be conservative. But, it looks a bit special case to me. Thanks, -- Yoshifumi Nishida nishida@sfc.wide.ad.jp 2011/2/7 Bob Briscoe : > Christoph, Yoshifumi, Costin, > > What is the original logic behind this... Why should waiting for slow-sta= rt > to end on one flow be relevant to when to start another sub-flow? I don't > see why (in general) slow-start on one path should affect slow start on > another. > > I would have thought the opposite: that opening more sub-flows would be a > more friendly way to increase the effective initial window than increasin= g > it all on one sub-flow. Because it spreads out the overshoots over multip= le > paths (assuming underlying physical path diversity) and over time (assumi= ng > RTTs are different). > > > > Bob > > At 09:09 07/02/2011, Christoph Paasch wrote: >> >> Hi, >> >> this sounds like a reasonable approach, to avoid creating congestion due >> to >> several subflows being in slow-start. >> However, I see one problem: >> After the initial subflow enters congestion avoidance, we try the next >> one. >> However the SYN's of this new subflow get blocked somewhere on their pat= h. >> Then we will be stuck with only the initial subflow for a long time (unt= il >> the >> TCP-stack gives up retransmitting the SYN's of the second subflow). And >> only >> then we will try the third subflow. >> >> >> Cheers, >> Christoph >> >> On Monday, February 07, 2011 wrote Yoshifumi Nishida: >> > Hi, >> > I personally prefer to initiate another subflow after the preceding >> > flow enters congestion avoidance. It looks a simple rule and we can >> > avoid multiple subflows to perform slow-start at the same time. >> > Or, does it hurt performance? >> > >> > Thanks, >> > -- >> > Yoshifumi Nishida >> > nishida@sfc.wide.ad.jp >> > >> > 2011/1/31 Bob Briscoe : >> > > Alan Costin, Mark, or ANOther, >> > > >> > > The large majority of TCP flows are very short. If there will only b= e >> > > one >> > > or two packets to send, there will be no point sending off loads of >> > > requests to open sub-flows straight after the first 3-way handshake. >> > > >> > > So I imagined there would be some advice in one of the MPTCP I-Ds on >> > > how >> > > long a MPTCP should wait until it attempts to open sub-flows. >> > > >> > > I can't find this from a quick scan. I would have expected it to be = in >> > > the heuristics section of draft-ietf-mptcp-multiaddressed: >> > > >> > > > >> > > 8> >> > > >> > > In general, a TCP stack doesn't know how much data the application >> > > might >> > > eventually want to send. So I guess the basic advice might be to wai= t >> > > n_1 >> > > RTTs before opening a second subflow, then n_2 RTTs before opening a >> > > third and so on. >> > > >> > > Has anyone done any research on this to find the best tradeoff (ie >> > > best >> > > values of n_1, n_2, etc) between improved performance/resilience and >> > > wasting signalling packets? I imagine this heuristic would be clocke= d >> > > by >> > > RTTs, not real-time or bytes, because that would likely be the >> > > relevant >> > > parameter for the signalling tradeoff. >> > > >> > > Then, in the API draft, there might be some discussion on whether th= e >> > > application can tell the transport how much data it intends to send, >> > > so >> > > the heuristics can be less blind. >> > > >> > > Thoughts? >> > > >> > > >> > > Bob >> > > >> > > >> > > ________________________________________________________________ >> > > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0BT Innovate & Design >> > > _______________________________________________ >> > > multipathtcp mailing list >> > > multipathtcp@ietf.org >> > > https://www.ietf.org/mailman/listinfo/multipathtcp >> > >> > _______________________________________________ >> > multipathtcp mailing list >> > multipathtcp@ietf.org >> > https://www.ietf.org/mailman/listinfo/multipathtcp >> >> -- >> Christoph Paasch >> PhD Student >> >> IP Networking Lab --- http://inl.info.ucl.ac.be >> MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp >> Universit=E9 Catholique de Louvain >> >> www.rollerbulls.be >> -- > > ________________________________________________________________ > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0BT Innovate & Design > From bob.briscoe@bt.com Tue Feb 8 03:13:01 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0518B3A7112 for ; Tue, 8 Feb 2011 03:13:01 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.668 X-Spam-Level: X-Spam-Status: No, score=-2.668 tagged_above=-999 required=5 tests=[AWL=-0.465, BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gP0nQIfnO77H for ; Tue, 8 Feb 2011 03:12:59 -0800 (PST) Received: from smtp4.smtp.bt.com (smtp4.smtp.bt.com [217.32.164.151]) by core3.amsl.com (Postfix) with ESMTP id 49C183A710E for ; Tue, 8 Feb 2011 03:12:58 -0800 (PST) Received: from i2kc06-ukbr.domain1.systemhost.net ([193.113.197.70]) by smtp4.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 8 Feb 2011 11:13:04 +0000 Received: from cbibipnt05.iuser.iroot.adidom.com ([147.149.196.177]) by i2kc06-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Tue, 8 Feb 2011 11:13:04 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt05.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297163583147; Tue, 8 Feb 2011 11:13:03 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p18BD1iM008475; Tue, 8 Feb 2011 11:13:01 GMT Message-Id: <201102081113.p18BD1iM008475@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 08 Feb 2011 11:13:05 +0000 To: Yoshifumi Nishida From: Bob Briscoe In-Reply-To: References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <201102071009.03576.christoph.paasch@uclouvain.be> <201102071923.p17JNf3A005501@bagheera.jungle.bt.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 08 Feb 2011 11:13:04.0236 (UTC) FILETIME=[27C8A2C0:01CBC781] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 11:13:01 -0000 Yoshifumi, Fair enough. It's only in special cases that a=20 data centre will fiddle around with its OS stack.=20 In most data centres they will just use the default stack. As we're still experimental here anyway, it's=20 fine to be conservative, then people who want to=20 experiment with more aggressive behaviours can do so. Cheers Bob At 06:18 08/02/2011, Yoshifumi Nishida wrote: >Hello Bob, > >It's nothing special. My concern is we cannot always presume path >diversity. So, I just prefer conservative approach. >If we can have good knowledge about network configuration like data >center scenarios, I think we won't have to be conservative. But, it >looks a bit special case to me. > >Thanks, >-- >Yoshifumi Nishida >nishida@sfc.wide.ad.jp > >2011/2/7 Bob Briscoe : > > Christoph, Yoshifumi, Costin, > > > > What is the original logic behind this... Why should waiting for= slow-start > > to end on one flow be relevant to when to start another sub-flow? I= don't > > see why (in general) slow-start on one path should affect slow start on > > another. > > > > I would have thought the opposite: that opening more sub-flows would be= a > > more friendly way to increase the effective initial window than= increasing > > it all on one sub-flow. Because it spreads out the overshoots over= multiple > > paths (assuming underlying physical path diversity) and over time= (assuming > > RTTs are different). > > > > > > > > Bob > > > > At 09:09 07/02/2011, Christoph Paasch wrote: > >> > >> Hi, > >> > >> this sounds like a reasonable approach, to avoid creating congestion= due > >> to > >> several subflows being in slow-start. > >> However, I see one problem: > >> After the initial subflow enters congestion avoidance, we try the next > >> one. > >> However the SYN's of this new subflow get blocked somewhere on their= path. > >> Then we will be stuck with only the initial subflow for a long time= (until > >> the > >> TCP-stack gives up retransmitting the SYN's of the second subflow). And > >> only > >> then we will try the third subflow. > >> > >> > >> Cheers, > >> Christoph > >> > >> On Monday, February 07, 2011 wrote Yoshifumi Nishida: > >> > Hi, > >> > I personally prefer to initiate another subflow after the preceding > >> > flow enters congestion avoidance. It looks a simple rule and we can > >> > avoid multiple subflows to perform slow-start at the same time. > >> > Or, does it hurt performance? > >> > > >> > Thanks, > >> > -- > >> > Yoshifumi Nishida > >> > nishida@sfc.wide.ad.jp > >> > > >> > 2011/1/31 Bob Briscoe : > >> > > Alan Costin, Mark, or ANOther, > >> > > > >> > > The large majority of TCP flows are very short. If there will only= be > >> > > one > >> > > or two packets to send, there will be no point sending off loads of > >> > > requests to open sub-flows straight after the first 3-way= handshake. > >> > > > >> > > So I imagined there would be some advice in one of the MPTCP I-Ds= on > >> > > how > >> > > long a MPTCP should wait until it attempts to open sub-flows. > >> > > > >> > > I can't find this from a quick scan. I would have expected it to be= in > >> > > the heuristics section of draft-ietf-mptcp-multiaddressed: > >> > > > >> > >=20 > = > > >> > > 8> > >> > > > >> > > In general, a TCP stack doesn't know how much data the application > >> > > might > >> > > eventually want to send. So I guess the basic advice might be to= wait > >> > > n_1 > >> > > RTTs before opening a second subflow, then n_2 RTTs before opening= a > >> > > third and so on. > >> > > > >> > > Has anyone done any research on this to find the best tradeoff (ie > >> > > best > >> > > values of n_1, n_2, etc) between improved performance/resilience= and > >> > > wasting signalling packets? I imagine this heuristic would be= clocked > >> > > by > >> > > RTTs, not real-time or bytes, because that would likely be the > >> > > relevant > >> > > parameter for the signalling tradeoff. > >> > > > >> > > Then, in the API draft, there might be some discussion on whether= the > >> > > application can tell the transport how much data it intends to= send, > >> > > so > >> > > the heuristics can be less blind. > >> > > > >> > > Thoughts? > >> > > > >> > > > >> > > Bob > >> > > > >> > > > >> > > ________________________________________________________________ > >> > > Bob Briscoe, BT Innovate & Design > >> > > _______________________________________________ > >> > > multipathtcp mailing list > >> > > multipathtcp@ietf.org > >> > > https://www.ietf.org/mailman/listinfo/multipathtcp > >> > > >> > _______________________________________________ > >> > multipathtcp mailing list > >> > multipathtcp@ietf.org > >> > https://www.ietf.org/mailman/listinfo/multipathtcp > >> > >> -- > >> Christoph Paasch > >> PhD Student > >> > >> IP Networking Lab --- http://inl.info.ucl.ac.be > >> MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptcp > >> Universit=E9 Catholique de Louvain > >> > >> www.rollerbulls.be > >> -- > > > > ________________________________________________________________ > > Bob Briscoe, BT Innovate & Design > > ________________________________________________________________ Bob Briscoe, BT Innovate & Design=20 From prvs=7020fe4000=alan.ford@roke.co.uk Tue Feb 8 03:36:43 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 53F353A7119 for ; Tue, 8 Feb 2011 03:36:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rs9hHTiSAKVY for ; Tue, 8 Feb 2011 03:36:42 -0800 (PST) Received: from gse-mta-29.emailfiltering.com (gse-mta-29-tx.emailfiltering.com [194.116.198.160]) by core3.amsl.com (Postfix) with ESMTP id C53DD3A7115 for ; Tue, 8 Feb 2011 03:36:41 -0800 (PST) Received: from salt-ext.roke.co.uk ([109.207.29.2]) by gse-mta-29.emailfiltering.com with emfmta (version 4.8.0.417) vanilla id 279983532 for multipathtcp@ietf.org; 9e57755f8c213eeb; Tue, 08 Feb 2011 11:36:46 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 8 Feb 2011 11:36:44 -0000 Message-ID: <2181C5F19DD0254692452BFF3EAF1D680C2B26EF@rsys005a.comm.ad.roke.co.uk> In-Reply-To: <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: mptcp-multiaddressed: How long before multipath starts? Thread-Index: AcvElHS2XHiNjQ8xTm6yFIACsSW5bQC7tg/g References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad.roke.co.uk> <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> From: "Ford, Alan" To: "Bob Briscoe" Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 11:36:43 -0000 Bob, all, > At 13:19 04/02/2011, Ford, Alan wrote: >=20 > >However, there are a couple of (smallish) issues with this approach. In > >the client/server model, it is the server who is in a position to know > >its buffer is filling up and thus how many new subflows it wants. But in > >many cases (e.g. client addresses behind NATs) only the client is in a > >position to create these subflows - and the client won't know about the > >amount of data in the server's buffer. >=20 > Good point. >=20 > Then an API at the client *would* be useful... In my first mail I > said that, in the common case of an HTTP GET, the first data packet > from the server usually contains an HTTP header that says how many > octets the object will be. So, as long as the client has a handy > casio calculator (or even a processor) it can work out how many > packets that implies, and decide whether to start more sub-flows. It would be nice if such things weren't dependent on the application having to think about this. And it is important that both ends can be in a position to open MPTCP subflows. This are the kinds of heuristics that should be discovered over experimental deployment. The application-specific aspect is also something worth noting the API draft appendix. > >(Some intelligence could be > >applied, e.g. opening subflows at the client side when the window size > >has stayed static for N packets - this is probably good enough for most > >cases, although adding more than one extra subflow would have to wait > >until both subflow windows have stayed static). >=20 > Ooh. I would assume it wants to be opening sub-flows long before it > would have time to measure such heuristics. The idea of MPTCP is > surely to have all the sub-flows there, and let congestion control > decide how much to use them. Are there any alternative ways of inferring that a server has a lot of data to send? > >The other issue with this approach is that it does not provide resource > >pooling for resilience, only throughput. Therefore, a second heuristic > >about time/number of packets elapsed should be applied, to provide > >resilience to longer-lived flows. >=20 > I assume "this approach" is what you *thought* I said (something to > do with congestion). I believe, what I actually said satisfies your > concern here about resilience. Not entirely. What you were saying, as I understand it now (things got conflated in my brain), is to do with how much data is queued at the sender. In the case of, say, SSH, Instant Messaging, or other low-bandwidth, long-lived flows, this threshold may never be hit, but resilience is of significant interest (indeed, probably more so than for many of the high bandwidth flows). It may be appropriate for such apps to use an API - again something for noting in the API draft. However, I think a long-life heuristic is necessary and also one of the easiest to specify, monitor and tweak. > >Most of this behaviour can be controlled by the endpoints, and does not > >require any protocol changes. Or are there any cases where we may want a > >flag to say "I speak MPTCP, but don't want to use it yet"? >=20 > I am not saying there is any need for a richer wire protocol here. If > I speak MPTCP but don't want to open subflows, surely I just don't > open a sub-flow, or I don't respond when asked to open one. >=20 > But a protocol is more than the wire protocol. It also includes the > algorithms. I'm saying some baseline algo for the decision when to > open sub-flows needs to be in the multi-address spec. But made clear > that it's not mandatory and better heuristics would be encouraged. I think it would be appropriate that we include discussion on this, but should conclude with a "implementations MAY choose" a simple heuristic and pointers to look out for further research. > Otherwise, if most implementers take the easy option, they will open > all possible sub-flows as soon as possible, and we will have storms > of redundant control packets due to all the flows on the Internet > (25%?) that are no more than 3 packets. Quite. Nobody wants this. Regards, Alan From bob.briscoe@bt.com Tue Feb 8 09:10:43 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2B2C23A6820 for ; Tue, 8 Feb 2011 09:10:43 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.25 X-Spam-Level: X-Spam-Status: No, score=-3.25 tagged_above=-999 required=5 tests=[AWL=0.349, BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KP1Ge46pvQfP for ; Tue, 8 Feb 2011 09:10:40 -0800 (PST) Received: from smtp2.smtp.bt.com (smtp2.smtp.bt.com [217.32.164.150]) by core3.amsl.com (Postfix) with ESMTP id 579D03A6805 for ; Tue, 8 Feb 2011 09:10:37 -0800 (PST) Received: from i2kc08-ukbr.domain1.systemhost.net ([193.113.197.71]) by smtp2.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 8 Feb 2011 17:10:43 +0000 Received: from cbibipnt08.iuser.iroot.adidom.com ([147.149.100.81]) by i2kc08-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Tue, 8 Feb 2011 17:10:43 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt08.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297185042852; Tue, 8 Feb 2011 17:10:42 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p18HAfrb009732; Tue, 8 Feb 2011 17:10:41 GMT Message-Id: <201102081710.p18HAfrb009732@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 08 Feb 2011 17:10:45 +0000 To: "Ford, Alan" From: Bob Briscoe In-Reply-To: <2181C5F19DD0254692452BFF3EAF1D680C2B26EF@rsys005a.comm.ad. roke.co.uk> References: <201101311715.p0VHFCcA001378@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C1743A8@rsys005a.comm.ad.roke.co.uk> <201102032345.p13NjUHQ017151@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C204519@rsys005a.comm.ad.roke.co.uk> <201102041753.p14HrZij020929@bagheera.jungle.bt.co.uk> <2181C5F19DD0254692452BFF3EAF1D680C2B26EF@rsys005a.comm.ad.roke.co.uk> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 08 Feb 2011 17:10:43.0293 (UTC) FILETIME=[1E60E0D0:01CBC7B3] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long before multipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 17:10:43 -0000 Alan, I think we are violently agreeing on all the points. You had one question... At 11:36 08/02/2011, Ford, Alan wrote: > > >(Some intelligence could be > > >applied, e.g. opening subflows at the client side when the window >size > > >has stayed static for N packets - this is probably good enough for >most > > >cases, although adding more than one extra subflow would have to wait > > >until both subflow windows have stayed static). Before the question, I have to interject here too... Window stability is not a good metric IMO; If a long flow passes thru a shared bottleneck with other flows coming and going, its window will vary, but this does not mean it's not a long flow. > > > > Ooh. I would assume it wants to be opening sub-flows long before it > > would have time to measure such heuristics. The idea of MPTCP is > > surely to have all the sub-flows there, and let congestion control > > decide how much to use them. > >Are there any alternative ways of inferring that a server has a lot of >data to send? So far Michael Sharf has given the best way I can think of; on the list earlier in this thread: At 17:46 31/01/2011, SCHARF, Michael wrote: >The simpler alternative is to just observe in the TCP stack >how much data the app actually writes and then to use this information >as input to the heuristics. Bob ________________________________________________________________ Bob Briscoe, BT Innovate & Design From philip.eardley@bt.com Tue Feb 8 14:17:51 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D33F13A688C for ; Tue, 8 Feb 2011 14:17:51 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.046 X-Spam-Level: X-Spam-Status: No, score=-102.046 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z5geJ5EEaZbB for ; Tue, 8 Feb 2011 14:17:50 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp63.intersmtp.COM [62.239.224.236]) by core3.amsl.com (Postfix) with ESMTP id 401673A6897 for ; Tue, 8 Feb 2011 14:17:50 -0800 (PST) Received: from EVMHT64-UKRD.domain1.systemhost.net (10.36.3.101) by RDW083A007ED63.smtp-e3.hygiene.service (10.187.98.12) with Microsoft SMTP Server (TLS) id 8.3.106.1; Tue, 8 Feb 2011 22:17:57 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.2.67]) by EVMHT64-UKRD.domain1.systemhost.net ([10.36.3.101]) with mapi; Tue, 8 Feb 2011 22:17:57 +0000 From: To: , Date: Tue, 8 Feb 2011 22:17:56 +0000 Thread-Topic: [multipathtcp] mptcp-multiaddressed: How long beforemultipath starts? Thread-Index: AcvH3gm2XlilXXayQ4WUobFoGe7w/g== Message-ID: <1297204387.2001.7.camel@Nokia-N900-51-1> Accept-Language: en-US, en-GB Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long beforemultipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list Reply-To: philip.eardley@bt.com List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Feb 2011 22:17:51 -0000 U2luY2UgdGhlc2UgaXNzdWVzIGFyZSBmb3IgdGhlIGhldXJpc3RpY3Mgc2VjdGlvbiwgaXQgc2Vl bXMgYXBwcm9wcmlhdGUgdG8gZ2l2ZSBzZXZlcmFsIHBvc3NpYmlsaXRpZXMgYW5kIGFkdmljZSB3 aGVuIGVhY2ggb25lIG1heSBiZSByaWdodC4NCi0tLS0tIE9yaWdpbmFsIG1lc3NhZ2UgLS0tLS0N Cj4gSGVsbG8gQm9iLA0KPg0KPiBJdCdzIG5vdGhpbmcgc3BlY2lhbC4gTXkgY29uY2VybiBpcyB3 ZSBjYW5ub3QgYWx3YXlzIHByZXN1bWUgcGF0aA0KPiBkaXZlcnNpdHkuIFNvLCBJIGp1c3QgcHJl ZmVyIGNvbnNlcnZhdGl2ZSBhcHByb2FjaC4NCj4gSWYgd2UgY2FuIGhhdmUgZ29vZCBrbm93bGVk Z2UgYWJvdXQgbmV0d29yayBjb25maWd1cmF0aW9uIGxpa2UgZGF0YQ0KPiBjZW50ZXIgc2NlbmFy aW9zLCBJIHRoaW5rIHdlIHdvbid0IGhhdmUgdG8gYmUgY29uc2VydmF0aXZlLiBCdXQsIGl0DQo+ IGxvb2tzIGEgYml0IHNwZWNpYWwgY2FzZSB0byBtZS4NCj4NCj4gVGhhbmtzLA0KPiAtLQ0KPiBZ b3NoaWZ1bWkgTmlzaGlkYQ0KPiBuaXNoaWRhQHNmYy53aWRlLmFkLmpwPG1haWx0bzpuaXNoaWRh QHNmYy53aWRlLmFkLmpwPg0KPg0KPiAyMDExLzIvNyBCb2IgQnJpc2NvZSA8Ym9iLmJyaXNjb2VA YnQuY29tPG1haWx0bzpib2IuYnJpc2NvZUBidC5jb20+PjoNCj4gPiBDaHJpc3RvcGgsIFlvc2hp ZnVtaSwgQ29zdGluLA0KPiA+DQo+ID4gV2hhdCBpcyB0aGUgb3JpZ2luYWwgbG9naWMgYmVoaW5k IHRoaXMuLi4gV2h5IHNob3VsZCB3YWl0aW5nIGZvcg0KPiBzbG93LXN0YXJ0DQo+ID4gdG8gZW5k IG9uIG9uZSBmbG93IGJlIHJlbGV2YW50IHRvIHdoZW4gdG8gc3RhcnQgYW5vdGhlciBzdWItZmxv dz8gSQ0KPiBkb24ndA0KPiA+IHNlZSB3aHkgKGluIGdlbmVyYWwpIHNsb3ctc3RhcnQgb24gb25l IHBhdGggc2hvdWxkIGFmZmVjdCBzbG93IHN0YXJ0DQo+IG9uDQo+ID4gYW5vdGhlci4NCj4gPg0K PiA+IEkgd291bGQgaGF2ZSB0aG91Z2h0IHRoZSBvcHBvc2l0ZTogdGhhdCBvcGVuaW5nIG1vcmUg c3ViLWZsb3dzIHdvdWxkDQo+IGJlIGENCj4gPiBtb3JlIGZyaWVuZGx5IHdheSB0byBpbmNyZWFz ZSB0aGUgZWZmZWN0aXZlIGluaXRpYWwgd2luZG93IHRoYW4NCj4gaW5jcmVhc2luZw0KPiA+IGl0 IGFsbCBvbiBvbmUgc3ViLWZsb3cuIEJlY2F1c2UgaXQgc3ByZWFkcyBvdXQgdGhlIG92ZXJzaG9v dHMgb3Zlcg0KPiBtdWx0aXBsZQ0KPiA+IHBhdGhzIChhc3N1bWluZyB1bmRlcmx5aW5nIHBoeXNp Y2FsIHBhdGggZGl2ZXJzaXR5KSBhbmQgb3ZlciB0aW1lDQo+IChhc3N1bWluZw0KPiA+IFJUVHMg YXJlIGRpZmZlcmVudCkuDQo+ID4NCj4gPg0KPiA+DQo+ID4gQm9iDQo+ID4NCj4gPiBBdCAwOTow OSAwNy8wMi8yMDExLCBDaHJpc3RvcGggUGFhc2NoIHdyb3RlOg0KPiA+ID4NCj4gPiA+IEhpLA0K PiA+ID4NCj4gPiA+IHRoaXMgc291bmRzIGxpa2UgYSByZWFzb25hYmxlIGFwcHJvYWNoLCB0byBh dm9pZCBjcmVhdGluZyBjb25nZXN0aW9uDQo+IGR1ZQ0KPiA+ID4gdG8NCj4gPiA+IHNldmVyYWwg c3ViZmxvd3MgYmVpbmcgaW4gc2xvdy1zdGFydC4NCj4gPiA+IEhvd2V2ZXIsIEkgc2VlIG9uZSBw cm9ibGVtOg0KPiA+ID4gQWZ0ZXIgdGhlIGluaXRpYWwgc3ViZmxvdyBlbnRlcnMgY29uZ2VzdGlv biBhdm9pZGFuY2UsIHdlIHRyeSB0aGUNCj4gbmV4dA0KPiA+ID4gb25lLg0KPiA+ID4gSG93ZXZl ciB0aGUgU1lOJ3Mgb2YgdGhpcyBuZXcgc3ViZmxvdyBnZXQgYmxvY2tlZCBzb21ld2hlcmUgb24g dGhlaXINCj4gcGF0aC4NCj4gPiA+IFRoZW4gd2Ugd2lsbCBiZSBzdHVjayB3aXRoIG9ubHkgdGhl IGluaXRpYWwgc3ViZmxvdyBmb3IgYSBsb25nIHRpbWUNCj4gKHVudGlsDQo+ID4gPiB0aGUNCj4g PiA+IFRDUC1zdGFjayBnaXZlcyB1cCByZXRyYW5zbWl0dGluZyB0aGUgU1lOJ3Mgb2YgdGhlIHNl Y29uZCBzdWJmbG93KS4NCj4gQW5kDQo+ID4gPiBvbmx5DQo+ID4gPiB0aGVuIHdlIHdpbGwgdHJ5 IHRoZSB0aGlyZCBzdWJmbG93Lg0KPiA+ID4NCj4gPiA+DQo+ID4gPiBDaGVlcnMsDQo+ID4gPiBD aHJpc3RvcGgNCj4gPiA+DQo+ID4gPiBPbiBNb25kYXksIEZlYnJ1YXJ5IDA3LCAyMDExIHdyb3Rl IFlvc2hpZnVtaSBOaXNoaWRhOg0KPiA+ID4gPiBIaSwNCj4gPiA+ID4gSSBwZXJzb25hbGx5IHBy ZWZlciB0byBpbml0aWF0ZSBhbm90aGVyIHN1YmZsb3cgYWZ0ZXIgdGhlIHByZWNlZGluZw0KPiA+ ID4gPiBmbG93IGVudGVycyBjb25nZXN0aW9uIGF2b2lkYW5jZS4gSXQgbG9va3MgYSBzaW1wbGUg cnVsZSBhbmQgd2UgY2FuDQo+ID4gPiA+IGF2b2lkIG11bHRpcGxlIHN1YmZsb3dzIHRvIHBlcmZv cm0gc2xvdy1zdGFydCBhdCB0aGUgc2FtZSB0aW1lLg0KPiA+ID4gPiBPciwgZG9lcyBpdCBodXJ0 IHBlcmZvcm1hbmNlPw0KPiA+ID4gPg0KPiA+ID4gPiBUaGFua3MsDQo+ID4gPiA+IC0tDQo+ID4g PiA+IFlvc2hpZnVtaSBOaXNoaWRhDQo+ID4gPiA+IG5pc2hpZGFAc2ZjLndpZGUuYWQuanA8bWFp bHRvOm5pc2hpZGFAc2ZjLndpZGUuYWQuanA+DQo+ID4gPiA+DQo+ID4gPiA+IDIwMTEvMS8zMSBC b2IgQnJpc2NvZSA8Ym9iLmJyaXNjb2VAYnQuY29tPG1haWx0bzpib2IuYnJpc2NvZUBidC5jb20+ PjoNCj4gPiA+ID4gPiBBbGFuIENvc3RpbiwgTWFyaywgb3IgQU5PdGhlciwNCj4gPiA+ID4gPg0K PiA+ID4gPiA+IFRoZSBsYXJnZSBtYWpvcml0eSBvZiBUQ1AgZmxvd3MgYXJlIHZlcnkgc2hvcnQu IElmIHRoZXJlIHdpbGwNCj4gb25seSBiZQ0KPiA+ID4gPiA+IG9uZQ0KPiA+ID4gPiA+IG9yIHR3 byBwYWNrZXRzIHRvIHNlbmQsIHRoZXJlIHdpbGwgYmUgbm8gcG9pbnQgc2VuZGluZyBvZmYgbG9h ZHMNCj4gb2YNCj4gPiA+ID4gPiByZXF1ZXN0cyB0byBvcGVuIHN1Yi1mbG93cyBzdHJhaWdodCBh ZnRlciB0aGUgZmlyc3QgMy13YXkNCj4gaGFuZHNoYWtlLg0KPiA+ID4gPiA+DQo+ID4gPiA+ID4g U28gSSBpbWFnaW5lZCB0aGVyZSB3b3VsZCBiZSBzb21lIGFkdmljZSBpbiBvbmUgb2YgdGhlIE1Q VENQIEktRHMNCj4gb24NCj4gPiA+ID4gPiBob3cNCj4gPiA+ID4gPiBsb25nIGEgTVBUQ1Agc2hv dWxkIHdhaXQgdW50aWwgaXQgYXR0ZW1wdHMgdG8gb3BlbiBzdWItZmxvd3MuDQo+ID4gPiA+ID4N Cj4gPiA+ID4gPiBJIGNhbid0IGZpbmQgdGhpcyBmcm9tIGEgcXVpY2sgc2Nhbi4gSSB3b3VsZCBo YXZlIGV4cGVjdGVkIGl0IHRvDQo+IGJlIGluDQo+ID4gPiA+ID4gdGhlIGhldXJpc3RpY3Mgc2Vj dGlvbiBvZiBkcmFmdC1pZXRmLW1wdGNwLW11bHRpYWRkcmVzc2VkOg0KPiA+ID4gPiA+DQo+ID4g PiA+ID4NCj4gPGh0dHA6Ly90b29scy5pZXRmLm9yZy9odG1sL2RyYWZ0LWlldGYtbXB0Y3AtbXVs dGlhZGRyZXNzZWQtMDIjc2VjdGlvbi0zLj47ID4NCj4gPiA+ID4gPiA4Pg0KPiA+ID4gPiA+DQo+ ID4gPiA+ID4gSW4gZ2VuZXJhbCwgYSBUQ1Agc3RhY2sgZG9lc24ndCBrbm93IGhvdyBtdWNoIGRh dGEgdGhlDQo+IGFwcGxpY2F0aW9uDQo+ID4gPiA+ID4gbWlnaHQNCj4gPiA+ID4gPiBldmVudHVh bGx5IHdhbnQgdG8gc2VuZC4gU28gSSBndWVzcyB0aGUgYmFzaWMgYWR2aWNlIG1pZ2h0IGJlIHRv DQo+IHdhaXQNCj4gPiA+ID4gPiBuXzENCj4gPiA+ID4gPiBSVFRzIGJlZm9yZSBvcGVuaW5nIGEg c2Vjb25kIHN1YmZsb3csIHRoZW4gbl8yIFJUVHMgYmVmb3JlDQo+IG9wZW5pbmcgYQ0KPiA+ID4g PiA+IHRoaXJkIGFuZCBzbyBvbi4NCj4gPiA+ID4gPg0KPiA+ID4gPiA+IEhhcyBhbnlvbmUgZG9u ZSBhbnkgcmVzZWFyY2ggb24gdGhpcyB0byBmaW5kIHRoZSBiZXN0IHRyYWRlb2ZmDQo+IChpZQ0K PiA+ID4gPiA+IGJlc3QNCj4gPiA+ID4gPiB2YWx1ZXMgb2Ygbl8xLCBuXzIsIGV0YykgYmV0d2Vl biBpbXByb3ZlZCBwZXJmb3JtYW5jZS9yZXNpbGllbmNlDQo+IGFuZA0KPiA+ID4gPiA+IHdhc3Rp bmcgc2lnbmFsbGluZyBwYWNrZXRzPyBJIGltYWdpbmUgdGhpcyBoZXVyaXN0aWMgd291bGQgYmUN Cj4gY2xvY2tlZA0KPiA+ID4gPiA+IGJ5DQo+ID4gPiA+ID4gUlRUcywgbm90IHJlYWwtdGltZSBv ciBieXRlcywgYmVjYXVzZSB0aGF0IHdvdWxkIGxpa2VseSBiZSB0aGUNCj4gPiA+ID4gPiByZWxl dmFudA0KPiA+ID4gPiA+IHBhcmFtZXRlciBmb3IgdGhlIHNpZ25hbGxpbmcgdHJhZGVvZmYuDQo+ ID4gPiA+ID4NCj4gPiA+ID4gPiBUaGVuLCBpbiB0aGUgQVBJIGRyYWZ0LCB0aGVyZSBtaWdodCBi ZSBzb21lIGRpc2N1c3Npb24gb24gd2hldGhlcg0KPiB0aGUNCj4gPiA+ID4gPiBhcHBsaWNhdGlv biBjYW4gdGVsbCB0aGUgdHJhbnNwb3J0IGhvdyBtdWNoIGRhdGEgaXQgaW50ZW5kcyB0bw0KPiBz ZW5kLA0KPiA+ID4gPiA+IHNvDQo+ID4gPiA+ID4gdGhlIGhldXJpc3RpY3MgY2FuIGJlIGxlc3Mg YmxpbmQuDQo+ID4gPiA+ID4NCj4gPiA+ID4gPiBUaG91Z2h0cz8NCj4gPiA+ID4gPg0KPiA+ID4g PiA+DQo+ID4gPiA+ID4gQm9iDQo+ID4gPiA+ID4NCj4gPiA+ID4gPg0KPiA+ID4gPiA+IF9fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X18NCj4gPiA+ID4gPiBCb2IgQnJpc2NvZSwgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEJUIElubm92YXRlICYgRGVzaWduDQo+ID4g PiA+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4g PiA+ID4gPiBtdWx0aXBhdGh0Y3AgbWFpbGluZyBsaXN0DQo+ID4gPiA+ID4gbXVsdGlwYXRodGNw QGlldGYub3JnPG1haWx0bzptdWx0aXBhdGh0Y3BAaWV0Zi5vcmc+DQo+ID4gPiA+ID4gaHR0cHM6 Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9tdWx0aXBhdGh0Y3ANCj4gPiA+ID4NCj4g PiA+ID4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCj4g PiA+ID4gbXVsdGlwYXRodGNwIG1haWxpbmcgbGlzdA0KPiA+ID4gPiBtdWx0aXBhdGh0Y3BAaWV0 Zi5vcmc8bWFpbHRvOm11bHRpcGF0aHRjcEBpZXRmLm9yZz4NCj4gPiA+ID4gaHR0cHM6Ly93d3cu aWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby9tdWx0aXBhdGh0Y3ANCj4gPiA+DQo+ID4gPiAtLQ0K PiA+ID4gQ2hyaXN0b3BoIFBhYXNjaA0KPiA+ID4gUGhEIFN0dWRlbnQNCj4gPiA+DQo+ID4gPiBJ UCBOZXR3b3JraW5nIExhYiAtLS0gaHR0cDovL2lubC5pbmZvLnVjbC5hYy5iZQ0KPiA+ID4gTXVs dGlQYXRoIFRDUCBpbiB0aGUgTGludXggS2VybmVsIC0tLSBodHRwOi8vaW5sLmluZm8udWNsLmFj LmJlL21wdGNwDQo+ID4gPiBVbml2ZXJzaXTDqSBDYXRob2xpcXVlIGRlIExvdXZhaW4NCj4gPiA+ DQo+ID4gPiB3d3cucm9sbGVyYnVsbHMuYmU8aHR0cDovL3d3dy5yb2xsZXJidWxscy5iZT4NCj4g PiA+IC0tDQo+ID4NCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fDQo+ID4gQm9iIEJyaXNjb2UsICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBCVCBJbm5vdmF0 ZSAmIERlc2lnbg0KPiA+DQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fDQo+IG11bHRpcGF0aHRjcCBtYWlsaW5nIGxpc3QNCj4gbXVsdGlwYXRodGNwQGll dGYub3JnPG1haWx0bzptdWx0aXBhdGh0Y3BAaWV0Zi5vcmc+DQo+IGh0dHBzOi8vd3d3LmlldGYu b3JnL21haWxtYW4vbGlzdGluZm8vbXVsdGlwYXRodGNwDQo+DQoNCg== From mark.j.handley@gmail.com Wed Feb 9 01:58:04 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 46FC03A6962 for ; Wed, 9 Feb 2011 01:58:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.977 X-Spam-Level: X-Spam-Status: No, score=-2.977 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Asl350587y02 for ; Wed, 9 Feb 2011 01:58:00 -0800 (PST) Received: from mail-ww0-f44.google.com (mail-ww0-f44.google.com [74.125.82.44]) by core3.amsl.com (Postfix) with ESMTP id 6AE193A683B for ; Wed, 9 Feb 2011 01:58:00 -0800 (PST) Received: by wwa36 with SMTP id 36so7252993wwa.13 for ; Wed, 09 Feb 2011 01:58:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:from :date:x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=L2o4q+Ooal5Ubg1lrtxS9Mac5Lwd8k0YMpD42HHlhYA=; b=JyNpIZ27RPNqsl5EEbU435K84aPYzgBGPukQtfiqN8g/VLfiYDh384ABLCdczFf2Fi AJtbc07qnfVDumE3SVm7fNe1/01SsDF+yJt0EChIlqYCjuY8uCuOoCSFGcgmCFvbr8Hx nXJgz4NEtUrrX1c8mEjbL6ru0NLQBpYGuLcJk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=oKQn6kxmhzS4W8j+mEnPgzTuD+QDwaPvmRlS+6pizxUqfEUnyGm4q8wz92VdTcCoNK 26i0ZqEiLxaROho4Q2oR5S5kgsuH+KMkyMtTs3BNhtZ30LMA+Jrwkj0HmMhpcXyTmynn spPEgCwOYoQH3P0cxhy1RjDVdvy12ADbXcO/U= Received: by 10.216.86.195 with SMTP id w45mr16233369wee.92.1297245488695; Wed, 09 Feb 2011 01:58:08 -0800 (PST) MIME-Version: 1.0 Sender: mark.j.handley@gmail.com Received: by 10.216.8.15 with HTTP; Wed, 9 Feb 2011 01:57:48 -0800 (PST) In-Reply-To: <1297204387.2001.7.camel@Nokia-N900-51-1> References: <1297204387.2001.7.camel@Nokia-N900-51-1> From: Mark Handley Date: Wed, 9 Feb 2011 09:57:48 +0000 X-Google-Sender-Auth: kDKW8aDTtjOrQ6nQiBcG6XIVft8 Message-ID: To: philip.eardley@bt.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: multipathtcp@ietf.org, "bob.briscoe" Subject: Re: [multipathtcp] mptcp-multiaddressed: How long beforemultipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 09:58:04 -0000 Would it make sense to separate these, and other implementation-related issues into a separate "Advice for MPTCP Implementors" draft? That way the protocol spec would be experimental, and the implementation advice would only be informational. It reduces the bar for revising the advice as we learn. - Mark On 8 February 2011 22:17, philip.eardley wrote: > Since these issues are for the heuristics section, it seems appropriate t= o give several possibilities and advice when each one may be right. > ----- Original message ----- >> Hello Bob, >> >> It's nothing special. My concern is we cannot always presume path >> diversity. So, I just prefer conservative approach. >> If we can have good knowledge about network configuration like data >> center scenarios, I think we won't have to be conservative. But, it >> looks a bit special case to me. >> >> Thanks, >> -- >> Yoshifumi Nishida >> nishida@sfc.wide.ad.jp >> >> 2011/2/7 Bob Briscoe >: >> > Christoph, Yoshifumi, Costin, >> > >> > What is the original logic behind this... Why should waiting for >> slow-start >> > to end on one flow be relevant to when to start another sub-flow? I >> don't >> > see why (in general) slow-start on one path should affect slow start >> on >> > another. >> > >> > I would have thought the opposite: that opening more sub-flows would >> be a >> > more friendly way to increase the effective initial window than >> increasing >> > it all on one sub-flow. Because it spreads out the overshoots over >> multiple >> > paths (assuming underlying physical path diversity) and over time >> (assuming >> > RTTs are different). >> > >> > >> > >> > Bob >> > >> > At 09:09 07/02/2011, Christoph Paasch wrote: >> > > >> > > Hi, >> > > >> > > this sounds like a reasonable approach, to avoid creating congestion >> due >> > > to >> > > several subflows being in slow-start. >> > > However, I see one problem: >> > > After the initial subflow enters congestion avoidance, we try the >> next >> > > one. >> > > However the SYN's of this new subflow get blocked somewhere on their >> path. >> > > Then we will be stuck with only the initial subflow for a long time >> (until >> > > the >> > > TCP-stack gives up retransmitting the SYN's of the second subflow). >> And >> > > only >> > > then we will try the third subflow. >> > > >> > > >> > > Cheers, >> > > Christoph >> > > >> > > On Monday, February 07, 2011 wrote Yoshifumi Nishida: >> > > > Hi, >> > > > I personally prefer to initiate another subflow after the precedin= g >> > > > flow enters congestion avoidance. It looks a simple rule and we ca= n >> > > > avoid multiple subflows to perform slow-start at the same time. >> > > > Or, does it hurt performance? >> > > > >> > > > Thanks, >> > > > -- >> > > > Yoshifumi Nishida >> > > > nishida@sfc.wide.ad.jp >> > > > >> > > > 2011/1/31 Bob Briscoe >: >> > > > > Alan Costin, Mark, or ANOther, >> > > > > >> > > > > The large majority of TCP flows are very short. If there will >> only be >> > > > > one >> > > > > or two packets to send, there will be no point sending off loads >> of >> > > > > requests to open sub-flows straight after the first 3-way >> handshake. >> > > > > >> > > > > So I imagined there would be some advice in one of the MPTCP I-D= s >> on >> > > > > how >> > > > > long a MPTCP should wait until it attempts to open sub-flows. >> > > > > >> > > > > I can't find this from a quick scan. I would have expected it to >> be in >> > > > > the heuristics section of draft-ietf-mptcp-multiaddressed: >> > > > > >> > > > > >> ; > >> > > > > 8> >> > > > > >> > > > > In general, a TCP stack doesn't know how much data the >> application >> > > > > might >> > > > > eventually want to send. So I guess the basic advice might be to >> wait >> > > > > n_1 >> > > > > RTTs before opening a second subflow, then n_2 RTTs before >> opening a >> > > > > third and so on. >> > > > > >> > > > > Has anyone done any research on this to find the best tradeoff >> (ie >> > > > > best >> > > > > values of n_1, n_2, etc) between improved performance/resilience >> and >> > > > > wasting signalling packets? I imagine this heuristic would be >> clocked >> > > > > by >> > > > > RTTs, not real-time or bytes, because that would likely be the >> > > > > relevant >> > > > > parameter for the signalling tradeoff. >> > > > > >> > > > > Then, in the API draft, there might be some discussion on whethe= r >> the >> > > > > application can tell the transport how much data it intends to >> send, >> > > > > so >> > > > > the heuristics can be less blind. >> > > > > >> > > > > Thoughts? >> > > > > >> > > > > >> > > > > Bob >> > > > > >> > > > > >> > > > > ________________________________________________________________ >> > > > > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0BT = Innovate & Design >> > > > > _______________________________________________ >> > > > > multipathtcp mailing list >> > > > > multipathtcp@ietf.org >> > > > > https://www.ietf.org/mailman/listinfo/multipathtcp >> > > > >> > > > _______________________________________________ >> > > > multipathtcp mailing list >> > > > multipathtcp@ietf.org >> > > > https://www.ietf.org/mailman/listinfo/multipathtcp >> > > >> > > -- >> > > Christoph Paasch >> > > PhD Student >> > > >> > > IP Networking Lab --- http://inl.info.ucl.ac.be >> > > MultiPath TCP in the Linux Kernel --- http://inl.info.ucl.ac.be/mptc= p >> > > Universit=E9 Catholique de Louvain >> > > >> > > www.rollerbulls.be >> > > -- >> > >> > ________________________________________________________________ >> > Bob Briscoe, =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0BT Innov= ate & Design >> > >> _______________________________________________ >> multipathtcp mailing list >> multipathtcp@ietf.org >> https://www.ietf.org/mailman/listinfo/multipathtcp >> > > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp > From bob.briscoe@bt.com Wed Feb 9 04:08:16 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 40C063A6984 for ; Wed, 9 Feb 2011 04:08:16 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.622 X-Spam-Level: X-Spam-Status: No, score=-2.622 tagged_above=-999 required=5 tests=[AWL=-0.419, BAYES_00=-2.599, MIME_QP_LONG_LINE=1.396, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JxevR7ak+SPC for ; Wed, 9 Feb 2011 04:08:14 -0800 (PST) Received: from smtp2.smtp.bt.com (smtp2.smtp.bt.com [217.32.164.150]) by core3.amsl.com (Postfix) with ESMTP id 6FC2A3A696A for ; Wed, 9 Feb 2011 04:08:14 -0800 (PST) Received: from i2kc08-ukbr.domain1.systemhost.net ([193.113.197.71]) by smtp2.smtp.bt.com with Microsoft SMTPSVC(6.0.3790.4675); Wed, 9 Feb 2011 12:08:22 +0000 Received: from cbibipnt08.iuser.iroot.adidom.com ([147.149.100.81]) by i2kc08-ukbr.domain1.systemhost.net with Microsoft SMTPSVC(6.0.3790.4675); Wed, 9 Feb 2011 12:08:21 +0000 Received: From bagheera.jungle.bt.co.uk ([132.146.168.158]) by cbibipnt08.iuser.iroot.adidom.com (WebShield SMTP v4.5 MR1a P0803.399); id 1297253300394; Wed, 9 Feb 2011 12:08:20 +0000 Received: from MUT.jungle.bt.co.uk ([10.215.131.97]) by bagheera.jungle.bt.co.uk (8.13.5/8.12.8) with ESMTP id p19C8Iwk013331; Wed, 9 Feb 2011 12:08:18 GMT Message-Id: <201102091208.p19C8Iwk013331@bagheera.jungle.bt.co.uk> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 09 Feb 2011 12:08:16 +0000 To: Mark Handley From: Bob Briscoe In-Reply-To: References: <1297204387.2001.7.camel@Nokia-N900-51-1> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.56 on 132.146.168.158 X-OriginalArrivalTime: 09 Feb 2011 12:08:21.0636 (UTC) FILETIME=[0B857440:01CBC852] Cc: multipathtcp@ietf.org Subject: Re: [multipathtcp] mptcp-multiaddressed: How long beforemultipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 12:08:16 -0000 Mark, Works for me - depends whether the w-g wants to=20 maintain another doc, but I think there was talk of having this doc anyway. Bob At 09:57 09/02/2011, Mark Handley wrote: >Would it make sense to separate these, and other >implementation-related issues into a separate "Advice for MPTCP >Implementors" draft? That way the protocol spec would be >experimental, and the implementation advice would only be >informational. It reduces the bar for revising the advice as we >learn. > > - Mark > >On 8 February 2011 22:17, philip.eardley wrote: > > Since these issues are for the heuristics=20 > section, it seems appropriate to give several=20 > possibilities and advice when each one may be right. > > ----- Original message ----- > >> Hello Bob, > >> > >> It's nothing special. My concern is we cannot always presume path > >> diversity. So, I just prefer conservative approach. > >> If we can have good knowledge about network configuration like data > >> center scenarios, I think we won't have to be conservative. But, it > >> looks a bit special case to me. > >> > >> Thanks, > >> -- > >> Yoshifumi Nishida > >> nishida@sfc.wide.ad.jp > >> > >> 2011/2/7 Bob Briscoe >: > >> > Christoph, Yoshifumi, Costin, > >> > > >> > What is the original logic behind this... Why should waiting for > >> slow-start > >> > to end on one flow be relevant to when to start another sub-flow? I > >> don't > >> > see why (in general) slow-start on one path should affect slow start > >> on > >> > another. > >> > > >> > I would have thought the opposite: that opening more sub-flows would > >> be a > >> > more friendly way to increase the effective initial window than > >> increasing > >> > it all on one sub-flow. Because it spreads out the overshoots over > >> multiple > >> > paths (assuming underlying physical path diversity) and over time > >> (assuming > >> > RTTs are different). > >> > > >> > > >> > > >> > Bob > >> > > >> > At 09:09 07/02/2011, Christoph Paasch wrote: > >> > > > >> > > Hi, > >> > > > >> > > this sounds like a reasonable approach, to avoid creating= congestion > >> due > >> > > to > >> > > several subflows being in slow-start. > >> > > However, I see one problem: > >> > > After the initial subflow enters congestion avoidance, we try the > >> next > >> > > one. > >> > > However the SYN's of this new subflow get blocked somewhere on= their > >> path. > >> > > Then we will be stuck with only the initial subflow for a long time > >> (until > >> > > the > >> > > TCP-stack gives up retransmitting the SYN's of the second subflow). > >> And > >> > > only > >> > > then we will try the third subflow. > >> > > > >> > > > >> > > Cheers, > >> > > Christoph > >> > > > >> > > On Monday, February 07, 2011 wrote Yoshifumi Nishida: > >> > > > Hi, > >> > > > I personally prefer to initiate another subflow after the= preceding > >> > > > flow enters congestion avoidance. It looks a simple rule and we= can > >> > > > avoid multiple subflows to perform slow-start at the same time. > >> > > > Or, does it hurt performance? > >> > > > > >> > > > Thanks, > >> > > > -- > >> > > > Yoshifumi Nishida > >> > > > nishida@sfc.wide.ad.jp > >> > > > > >> > > > 2011/1/31 Bob Briscoe=20 > >: > >> > > > > Alan Costin, Mark, or ANOther, > >> > > > > > >> > > > > The large majority of TCP flows are very short. If there will > >> only be > >> > > > > one > >> > > > > or two packets to send, there will be no point sending off= loads > >> of > >> > > > > requests to open sub-flows straight after the first 3-way > >> handshake. > >> > > > > > >> > > > > So I imagined there would be some advice in one of the MPTCP= I-Ds > >> on > >> > > > > how > >> > > > > long a MPTCP should wait until it attempts to open sub-flows. > >> > > > > > >> > > > > I can't find this from a quick scan. I would have expected it= to > >> be in > >> > > > > the heuristics section of draft-ietf-mptcp-multiaddressed: > >> > > > > > >> > > > > > >>=20 >= ;= > > >> > > > > 8> > >> > > > > > >> > > > > In general, a TCP stack doesn't know how much data the > >> application > >> > > > > might > >> > > > > eventually want to send. So I guess the basic advice might be= to > >> wait > >> > > > > n_1 > >> > > > > RTTs before opening a second subflow, then n_2 RTTs before > >> opening a > >> > > > > third and so on. > >> > > > > > >> > > > > Has anyone done any research on this to find the best tradeoff > >> (ie > >> > > > > best > >> > > > > values of n_1, n_2, etc) between improved= performance/resilience > >> and > >> > > > > wasting signalling packets? I imagine this heuristic would be > >> clocked > >> > > > > by > >> > > > > RTTs, not real-time or bytes, because that would likely be the > >> > > > > relevant > >> > > > > parameter for the signalling tradeoff. > >> > > > > > >> > > > > Then, in the API draft, there might be some discussion on= whether > >> the > >> > > > > application can tell the transport how much data it intends to > >> send, > >> > > > > so > >> > > > > the heuristics can be less blind. > >> > > > > > >> > > > > Thoughts? > >> > > > > > >> > > > > > >> > > > > Bob > >> > > > > > >> > > > > > >> > > > >= ________________________________________________________________ > >> > > > > Bob=20 > Briscoe,=20 > BT Innovate & Design > >> > > > > _______________________________________________ > >> > > > > multipathtcp mailing list > >> > > > > multipathtcp@ietf.org > >> > > > > https://www.ietf.org/mailman/listinfo/multipathtcp > >> > > > > >> > > > _______________________________________________ > >> > > > multipathtcp mailing list > >> > > > multipathtcp@ietf.org > >> > > > https://www.ietf.org/mailman/listinfo/multipathtcp > >> > > > >> > > -- > >> > > Christoph Paasch > >> > > PhD Student > >> > > > >> > > IP Networking Lab --- http://inl.info.ucl.ac.be > >> > > MultiPath TCP in the Linux Kernel ---= http://inl.info.ucl.ac.be/mptcp > >> > > Universit=E9 Catholique de Louvain > >> > > > >> > > www.rollerbulls.be > >> > > -- > >> > > >> > ________________________________________________________________ > >> > Bob=20 > Briscoe,=20 > BT Innovate & Design > >> > > >> _______________________________________________ > >> multipathtcp mailing list > >> multipathtcp@ietf.org > >> https://www.ietf.org/mailman/listinfo/multipathtcp > >> > > > > _______________________________________________ > > multipathtcp mailing list > > multipathtcp@ietf.org > > https://www.ietf.org/mailman/listinfo/multipathtcp > > ________________________________________________________________ Bob Briscoe, BT Innovate & Design=20 From sebastien.barre@uclouvain.be Wed Feb 9 08:09:35 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DF09E3A69A8 for ; Wed, 9 Feb 2011 08:09:35 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -6.299 X-Spam-Level: X-Spam-Status: No, score=-6.299 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-4] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BZlSW+XaSbk2 for ; Wed, 9 Feb 2011 08:09:34 -0800 (PST) Received: from smtp3.sgsi.ucl.ac.be (smtp.sgsi.ucl.ac.be [130.104.5.67]) by core3.amsl.com (Postfix) with ESMTP id 3554A3A6981 for ; Wed, 9 Feb 2011 08:09:33 -0800 (PST) Received: from [130.104.228.80] (marguerite.dhcp.info.ucl.ac.be [130.104.228.80]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sbarre@smtp3.sgsi.ucl.ac.be) by smtp3.sgsi.ucl.ac.be (Postfix) with ESMTPSA id 6A3E71C563E for ; Wed, 9 Feb 2011 17:09:33 +0100 (CET) Message-ID: <4D52BC3D.5020407@uclouvain.be> Date: Wed, 09 Feb 2011 17:09:33 +0100 From: =?ISO-8859-1?Q?S=E9bastien_Barr=E9?= User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); nl; rv:1.9.2.13) Gecko/20101207 Lightning/1.0b2 Thunderbird/3.1.7 MIME-Version: 1.0 To: multipathtcp@ietf.org References: <1297204387.2001.7.camel@Nokia-N900-51-1> <201102091208.p19C8Iwk013331@bagheera.jungle.bt.co.uk> In-Reply-To: <201102091208.p19C8Iwk013331@bagheera.jungle.bt.co.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.96.5-exp at smtp-3.sipr-dc.ucl.ac.be X-Virus-Status: Clean Received-SPF: Pass (sender authenticated); receiver=; client-ip=130.104.228.80; helo= Received-SPF: Pass (sender authenticated); receiver=; client-ip=130.104.228.80; envelope-from= X-Sgsi-Spamcheck: SASL authenticated, X-SGSI-MailScanner-ID: 6A3E71C563E.00000 X-SGSI-MailScanner: Found to be clean X-SGSI-From: sebastien.barre@uclouvain.be X-SGSI-Spam-Status: No Subject: Re: [multipathtcp] mptcp-multiaddressed: How long beforemultipath starts? X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Feb 2011 16:09:36 -0000 Hi, Op 09-02-11 13:08, Bob Briscoe schreef: > Mark, > > Works for me - depends whether the w-g wants to maintain another doc, > but I think there was talk of having this doc anyway. indeed, we are preparing a draft on this, and will try to reflect this discussion in it. We will probably discuss that at next IETF. regards, Sébastien. > > > Bob > > At 09:57 09/02/2011, Mark Handley wrote: >> Would it make sense to separate these, and other >> implementation-related issues into a separate "Advice for MPTCP >> Implementors" draft? That way the protocol spec would be >> experimental, and the implementation advice would only be >> informational. It reduces the bar for revising the advice as we >> learn. >> >> - Mark >> >> On 8 February 2011 22:17, philip.eardley wrote: >> > Since these issues are for the heuristics section, it seems >> appropriate to give several possibilities and advice when each one >> may be right. >> > ----- Original message ----- >> >> Hello Bob, >> >> >> >> It's nothing special. My concern is we cannot always presume path >> >> diversity. So, I just prefer conservative approach. >> >> If we can have good knowledge about network configuration like data >> >> center scenarios, I think we won't have to be conservative. But, it >> >> looks a bit special case to me. >> >> >> >> Thanks, >> >> -- >> >> Yoshifumi Nishida >> >> nishida@sfc.wide.ad.jp >> >> >> >> 2011/2/7 Bob Briscoe >: >> >> > Christoph, Yoshifumi, Costin, >> >> > >> >> > What is the original logic behind this... Why should waiting for >> >> slow-start >> >> > to end on one flow be relevant to when to start another sub-flow? I >> >> don't >> >> > see why (in general) slow-start on one path should affect slow >> start >> >> on >> >> > another. >> >> > >> >> > I would have thought the opposite: that opening more sub-flows >> would >> >> be a >> >> > more friendly way to increase the effective initial window than >> >> increasing >> >> > it all on one sub-flow. Because it spreads out the overshoots over >> >> multiple >> >> > paths (assuming underlying physical path diversity) and over time >> >> (assuming >> >> > RTTs are different). >> >> > >> >> > >> >> > >> >> > Bob >> >> > >> >> > At 09:09 07/02/2011, Christoph Paasch wrote: >> >> > > >> >> > > Hi, >> >> > > >> >> > > this sounds like a reasonable approach, to avoid creating >> congestion >> >> due >> >> > > to >> >> > > several subflows being in slow-start. >> >> > > However, I see one problem: >> >> > > After the initial subflow enters congestion avoidance, we try the >> >> next >> >> > > one. >> >> > > However the SYN's of this new subflow get blocked somewhere on >> their >> >> path. >> >> > > Then we will be stuck with only the initial subflow for a long >> time >> >> (until >> >> > > the >> >> > > TCP-stack gives up retransmitting the SYN's of the second >> subflow). >> >> And >> >> > > only >> >> > > then we will try the third subflow. >> >> > > >> >> > > >> >> > > Cheers, >> >> > > Christoph >> >> > > >> >> > > On Monday, February 07, 2011 wrote Yoshifumi Nishida: >> >> > > > Hi, >> >> > > > I personally prefer to initiate another subflow after the >> preceding >> >> > > > flow enters congestion avoidance. It looks a simple rule and >> we can >> >> > > > avoid multiple subflows to perform slow-start at the same time. >> >> > > > Or, does it hurt performance? >> >> > > > >> >> > > > Thanks, >> >> > > > -- >> >> > > > Yoshifumi Nishida >> >> > > > nishida@sfc.wide.ad.jp >> >> > > > >> >> > > > 2011/1/31 Bob Briscoe >> >: >> >> > > > > Alan Costin, Mark, or ANOther, >> >> > > > > >> >> > > > > The large majority of TCP flows are very short. If there will >> >> only be >> >> > > > > one >> >> > > > > or two packets to send, there will be no point sending off >> loads >> >> of >> >> > > > > requests to open sub-flows straight after the first 3-way >> >> handshake. >> >> > > > > >> >> > > > > So I imagined there would be some advice in one of the >> MPTCP I-Ds >> >> on >> >> > > > > how >> >> > > > > long a MPTCP should wait until it attempts to open sub-flows. >> >> > > > > >> >> > > > > I can't find this from a quick scan. I would have expected >> it to >> >> be in >> >> > > > > the heuristics section of draft-ietf-mptcp-multiaddressed: >> >> > > > > >> >> > > > > >> >> >> ; >> > >> >> > > > > 8> >> >> > > > > >> >> > > > > In general, a TCP stack doesn't know how much data the >> >> application >> >> > > > > might >> >> > > > > eventually want to send. So I guess the basic advice might >> be to >> >> wait >> >> > > > > n_1 >> >> > > > > RTTs before opening a second subflow, then n_2 RTTs before >> >> opening a >> >> > > > > third and so on. >> >> > > > > >> >> > > > > Has anyone done any research on this to find the best >> tradeoff >> >> (ie >> >> > > > > best >> >> > > > > values of n_1, n_2, etc) between improved >> performance/resilience >> >> and >> >> > > > > wasting signalling packets? I imagine this heuristic would be >> >> clocked >> >> > > > > by >> >> > > > > RTTs, not real-time or bytes, because that would likely be >> the >> >> > > > > relevant >> >> > > > > parameter for the signalling tradeoff. >> >> > > > > >> >> > > > > Then, in the API draft, there might be some discussion on >> whether >> >> the >> >> > > > > application can tell the transport how much data it >> intends to >> >> send, >> >> > > > > so >> >> > > > > the heuristics can be less blind. >> >> > > > > >> >> > > > > Thoughts? >> >> > > > > >> >> > > > > >> >> > > > > Bob >> >> > > > > >> >> > > > > >> >> > > > > >> ________________________________________________________________ >> >> > > > > Bob Briscoe, BT Innovate & Design >> >> > > > > _______________________________________________ >> >> > > > > multipathtcp mailing list >> >> > > > > multipathtcp@ietf.org >> >> > > > > https://www.ietf.org/mailman/listinfo/multipathtcp >> >> > > > >> >> > > > _______________________________________________ >> >> > > > multipathtcp mailing list >> >> > > > multipathtcp@ietf.org >> >> > > > https://www.ietf.org/mailman/listinfo/multipathtcp >> >> > > >> >> > > -- >> >> > > Christoph Paasch >> >> > > PhD Student >> >> > > >> >> > > IP Networking Lab --- http://inl.info.ucl.ac.be >> >> > > MultiPath TCP in the Linux Kernel --- >> http://inl.info.ucl.ac.be/mptcp >> >> > > Université Catholique de Louvain >> >> > > >> >> > > www.rollerbulls.be >> >> > > -- >> >> > >> >> > ________________________________________________________________ >> >> > Bob Briscoe, BT Innovate & Design >> >> > >> >> _______________________________________________ >> >> multipathtcp mailing list >> >> multipathtcp@ietf.org >> >> https://www.ietf.org/mailman/listinfo/multipathtcp >> >> >> > >> > _______________________________________________ >> > multipathtcp mailing list >> > multipathtcp@ietf.org >> > https://www.ietf.org/mailman/listinfo/multipathtcp >> > > > ________________________________________________________________ > Bob Briscoe, BT Innovate & Design > _______________________________________________ > multipathtcp mailing list > multipathtcp@ietf.org > https://www.ietf.org/mailman/listinfo/multipathtcp -- Sébastien Barré Researcher, CSE department, UCLouvain, Belgium http://inl.info.ucl.ac.be/sbarre From dreibh@iem.uni-due.de Wed Feb 23 02:20:16 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id F35993A683E for ; Wed, 23 Feb 2011 02:20:15 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.249 X-Spam-Level: X-Spam-Status: No, score=-2.249 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_DE=0.35] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tkc0cqxqweNc for ; Wed, 23 Feb 2011 02:20:14 -0800 (PST) Received: from mailout.uni-due.de (mailout.uni-due.de [132.252.185.19]) by core3.amsl.com (Postfix) with ESMTP id 33FCB3A65A6 for ; Wed, 23 Feb 2011 02:20:13 -0800 (PST) Received: from kappes.localnet ([132.252.151.80]) (authenticated bits=0) by mailout.uni-due.de (8.13.1/8.13.1) with ESMTP id p1NAKrva017128 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 23 Feb 2011 11:20:53 +0100 From: Thomas Dreibholz Organization: University of Duisburg-Essen, Institute for Experimental Mathematics To: multipathtcp@ietf.org Date: Wed, 23 Feb 2011 11:20:46 +0100 User-Agent: KMail/1.13.5 (Linux/2.6.35-25-generic; KDE/4.6.0; x86_64; ; ) References: <9510D26531EF184D9017DF24659BB87F3275F300AE@EMV65-UKRD.domain1.systemhost.net> <7A7B916F-128B-42C4-B535-99ED507882D1@lurchi.franken.de> <396A98EB-D6A3-4492-AB8B-F539D4BAD0C9@cs.ucl.ac.uk> In-Reply-To: <396A98EB-D6A3-4492-AB8B-F539D4BAD0C9@cs.ucl.ac.uk> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart5413000.WofemLyA6A"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <201102231120.52809.dreibh@iem.uni-due.de> X-Virus-Scanned: Clam Anti Virus - http://www.clamav.net X-Spam-Scanned: SpamAssassin: 3.002004 - http://www.spamassassin.org X-Scanned-By: MIMEDefang 2.57 on 132.252.185.19 Cc: Dirceu Cavendish , iccrg Subject: Re: [multipathtcp] [Iccrg] Re: Please review "Coupled Congestion Control for Multipath Transport Protocols" X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Feb 2011 10:20:16 -0000 --nextPart5413000.WofemLyA6A Content-Type: Text/Plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable Dear all, see my comments inline. On Dienstag 01 Februar 2011, Costin Raiciu wrote: > Hi Michael, >=20 > Sorry for the late reply - we haven't slept much lately because of the > Sigcomm deadline. >=20 > When aiming for the same throughput as tcp on the best path, we equalize > rates in bytes per second. >=20 > I have analyzed the formula in the draft again, and it is wrong for that > context. The draft keeps the cwnd in bytes, and the correct formula in > that case completely excludes MSS from the calculation of alpha. This is > also the formula derived in the document you mention. >=20 > The formula for stacks that keep cwnd in packets needs to include MSS, > hence the term cwnd_i *mss_i. I do not understand why you use cwnd_i * mss_i here. If cwnd_i is given in= =20 bytes, then cwnd_i / mss_i gives the cwnd in MSS. I can reconstruct the train of thought for the formula for alpha from=20 "Practical Congestion Control for Multipath Transport Protocols". It is giv= en=20 in MSS instead of bytes. To reconstruct the formula of the I-D, I have =2D replaced \hat{w}_r by (cwnd_i / mss_i) and =2D used i instead of r as index variable. The result is alpha =3D tot_cwnd * ( max_i (sqrt(cwnd_i/mss_i) / rtt_i) / ((sum_i ((cwnd_i/mss_i) / rtt_i))^2) ). To get rid of the square root function, it can be transformed to: alpha =3D tot_cwnd * ( max_i ((cwnd_i / mss_i) / (rtt_i^2)) / ((sum_i ((cwnd_i/mss_i) / rtt_i))^2) ). That is, that fomula in the I-D has to: =2D replace "cwnd_i * mss_i" by "cwnd_i / mss_i" and =2D "mss_i^2" by "mss_i". Also, I am wondering why the fraction term is multiplied by tot_cwnd to=20 compute alpha. The only place alpha is used is to compute the increase: alpha*bytes_acked*mss_i/tot_cwnd Since (tot_cwnd/tot_cwnd)=3D1, tot_cwnd may be simply cancelled out. Best regards =2D-=20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Dr. Thomas Dreibholz University of Duisburg-Essen, Room ES210 Inst. for Experimental Mathematics Ellernstra=DFe 29 Computer Networking Technology Group D-45326 Essen/Germany =2D---------------------------------------------------------------------- E-Mail: dreibh@iem.uni-due.de Homepage: http://www.iem.uni-due.de/~dreibh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --nextPart5413000.WofemLyA6A Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) iEYEABECAAYFAk1k34AACgkQ32BbsHYPLWX8qgCffkUskfi9s21bCfIwSGb+xj9r yBcAniqMnq87lcJicuZy3+D7b1fhXh8e =9pzC -----END PGP SIGNATURE----- --nextPart5413000.WofemLyA6A-- From philip.eardley@bt.com Fri Feb 25 00:39:25 2011 Return-Path: X-Original-To: multipathtcp@core3.amsl.com Delivered-To: multipathtcp@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BE9E23A6928 for ; Fri, 25 Feb 2011 00:39:25 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -101.516 X-Spam-Level: X-Spam-Status: No, score=-101.516 tagged_above=-999 required=5 tests=[AWL=-0.070, BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, J_CHICKENPOX_72=0.6, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x8icjDZW4+IE for ; Fri, 25 Feb 2011 00:39:25 -0800 (PST) Received: from smtpe1.intersmtp.com (smtp61.intersmtp.COM [62.239.224.234]) by core3.amsl.com (Postfix) with ESMTP id C36593A68FE for ; Fri, 25 Feb 2011 00:39:24 -0800 (PST) Received: from EVMHT67-UKRD.domain1.systemhost.net (10.36.3.104) by RDW083A005ED61.smtp-e1.hygiene.service (10.187.98.10) with Microsoft SMTP Server (TLS) id 8.3.106.1; Fri, 25 Feb 2011 08:40:15 +0000 Received: from EMV65-UKRD.domain1.systemhost.net ([169.254.1.86]) by EVMHT67-UKRD.domain1.systemhost.net ([10.36.3.104]) with mapi; Fri, 25 Feb 2011 08:40:15 +0000 From: To: Date: Fri, 25 Feb 2011 08:40:14 +0000 Thread-Topic: MPTCP - Status & agenda planning Thread-Index: AcvUe1HmtXnZzOx3R7eqqDqmbt8WUAASLRJA Message-ID: <9510D26531EF184D9017DF24659BB87F3279EAB75F@EMV65-UKRD.domain1.systemhost.net> Accept-Language: en-US, en-GB Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US, en-GB Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 Subject: [multipathtcp] MPTCP - Status & agenda planning X-BeenThere: multipathtcp@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Multi-path extensions for TCP List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Feb 2011 08:39:25 -0000 V2UgaGF2ZSBiZWVuIGdpdmVuIGEgc2xvdCBvbiBUaHVyc2RheSBhdCAxNzQwLTE5NDAuDQooc2lu Y2Ugd2UgZGlkbuKAmXQgZ2V0IG91ciB1c3VhbCB0d28gc2xvdHMsIGl0IG1heSBiZSB3b3J0aCBh ICdjb3JyaWRvcicgbWVldGluZyBlYXJsaWVyIGluIHRoZSB3ZWVrLCBlc3BlY2lhbGx5IGlmIHRo ZXJlIGFyZSBhbnkgb3BlbiBpc3N1ZXMgdGhhdCBuZWVkIGEgY3ljbGUgb2YgZGlzY3Vzc2lvbiAt PiB0aG91Z2h0IC0+IG1vcmUgZGlzY3Vzc2lvbikgDQoNClN0YXR1czoNCjEuIEFyY2hpdGVjdHVy ZSAtIGluIFJGQyBFZGl0b3IgcXVldWUNCjIuIFRocmVhdHMgLSBpbiBSRkMgRWRpdG9yIHF1ZXVl DQozLiBDb25nZXN0aW9uIGNvbnRyb2wgLSB3YWl0aW5nIGZvciBhdXRob3JzIHRvIHVwZGF0ZSBm b2xsb3dpbmcgcmV2aWV3cy4gV2lsbCB0aGVuIFdHIGxhc3QgY2FsbC4gDQo0LiBBUEkgLSBhIGNv dXBsZSBvZiByZXZpZXdzIHJlcXVlc3RlZCwgZHVlIHNvb24uIEFueSBvcGVuIGlzc3VlcyB0byBi ZSBkaXNjdXNzZWQgaW4gUHJhZ3VlLiBBZnRlciB0YWtpbmcgYWNjb3VudCBvZiB0aGVtLCB3ZSBo b3BlIHdpbGwgYmUgcmVhZHkgZm9yIFdHIGxhc3QgY2FsbA0KNS4gUHJvdG9jb2wgLSBwbGVhc2Ug cmV2aWV3LiBBIGNvdXBsZSBvZiBtaW5vciBpc3N1ZXMgYmVpbmcgZGlzY3Vzc2VkIG9uIHRoZSBN TC4gVGhlIGF1dGhvcnMgaGF2ZSBzb21lIHVwZGF0ZXMgcGxhbm5lZCBhbmQgYSBuZXcgdmVyc2lv biBpbiB0aGUgbmV4dCB3ZWVrIG9yIHR3by4gQXNzdW1pbmcgdGhleSBiZWxpZXZlIGFsbCBvcGVu IGlzc3VlcyBhcmUgY2xvc2VkLCB3ZSB3aWxsIFdHIGxhc3QgY2FsbA0KDQpBZ2VuZGEgcGxhbm5p bmcgZm9yIFByYWd1ZToNCjEuIENvbmdlc3Rpb24gY29udHJvbA0KMi4gQVBJDQozLiBQcm90b2Nv bA0KNC4gSW1wbGVtZW50YXRpb24gc3RhdHVzDQo1LiBEZW1vIG9mIGltcGxlbWVudGF0aW9uDQo2 LiBSZS1jaGFydGVyaW5nDQoNCioqIHBsZWFzZSBsZXQgdXMga25vdyBpZiB5b3Ugd291bGQgbGlr ZSBhIHNsb3QgdG8gdGFsayBhYm91dCBhbnl0aGluZyBlbHNlDQoNCioqIHdlIHNob3VsZCBhbHNv IHRhbGsgYWJvdXQgcmUtY2hhcnRlcmluZywgYXMgd2UncmUgcXVpdGUgY2xvc2UgdG8gZmluaXNo aW5nIG91ciBjaGFydGVyIChleGNlbGxlbnQhKS4gRG8gcGVvcGxlIHdhbnQgdG8gZXh0ZW5kIG91 ciBjaGFydGVyIHRvIGNvbnRpbnVlIHdvcms/IFBvc3NpYmxlIHRvcGljcyBtaWdodCBiZToNCjEu IEFkdmljZSBvbiBpbXBsZW1lbnRhdGlvbiwgaGV1cmlzdGljcw0KMi4gYWx0ZXJuYXRpdmUgc2Vj dXJpdHkNCjMuIGFsdGVybmF0aXZlIGNvbmdlc3Rpb24gY29udHJvbA0KNC4gYWR2YW5jZWQgQVBJ DQo1LiBvdGhlcnM/DQpOb3RlLCB0aGVzZSBhcmUganVzdCBzb21lIHRoaW5ncyB0aGF0IGhhdmUg YmVlbiBtZW50aW9uZWQgYXQgdmFyaW91cyBwb2ludHMgYXMgdGhpbmdzIHdlIG1pZ2h0IHdhbnQg dG8gZG8gaW4gdGhlIGZ1dHVyZS4gV2UgZG8gbm90IGhhdmUgdG8gZG8gYWxsLCBvciBpbmRlZWQg YW55LCBvZiB0aGVtLiBPciB3ZSBjb3VsZCBhZGQgdGhlIGZpcnN0IGl0ZW0gKHNpbmNlIHRoZXJl IGlzIGFscmVhZHkgd29yayBiZWluZyBkb25lIG9uIHRoaXMpIGFuZCBkZWZlciB0aGUgb3RoZXJz IHVudGlsIG5leHQgdGltZSAoYnkgd2hlbiBvdXIgY3VycmVudCBjaGFydGVyIHNob3VsZCBiZSBj b21wbGV0ZSkNCg0KVGhhbmtzDQpQaGlsICYgWW9zaGlmdW1pDQoNCg0KDQotLS0tLU9yaWdpbmFs IE1lc3NhZ2UtLS0tLQ0KRnJvbTogSUVURiBTZWNyZXRhcmlhdCBbbWFpbHRvOmFnZW5kYUBpZXRm Lm9yZ10gDQpTZW50OiAyNCBGZWJydWFyeSAyMDExIDIzOjMzDQpUbzogRWFyZGxleSxQTCxQaGls aXAsREVTOCBSDQpDYzogbmlzaGlkYUBzZmMud2lkZS5hZC5qcDsgaWV0ZmRiaEBjb21jYXN0Lm5l dDsgbGFycy5lZ2dlcnRAbm9raWEuY29tOyB3ZXNAbXRpLXN5c3RlbXMuY29tOyBzZXNzaW9uLXJl cXVlc3RAaWV0Zi5vcmcNClN1YmplY3Q6IE1QVENQIC0gUmVxdWVzdGVkIHNlc3Npb24gaGFzIGJl ZW4gc2NoZWR1bGVkIGZvciBJRVRGIDgwIA0KDQpEZWFyIFBoaWxpcCBFYXJkbGV5LA0KDQpUaGUg c2Vzc2lvbnMgdGhhdCB5b3UgaGF2ZSByZXF1ZXN0ZWQgaGF2ZSBiZWVuIHNjaGVkdWxlZC4NCkJl bG93IGlzIHRoZSBzY2hlZHVsZWQgc2Vzc2lvbiBpbmZvcm1hdGlvbiBmb2xsb3dlZCBieSANCnRo ZSBpbmZvcm1hdGlvbiBvZiBzZXNzaW9ucyB0aGF0IHlvdSBoYXZlIHJlcXVlc3RlZC4NCg0KTVBU Q1AgU2Vzc2lvbiAxICgyIGhvdXJzKQ0KVGh1cnNkYXksIEFmdGVybm9vbiBTZXNzaW9uIElJSSAx NzQwLTE5NDANClJvb20gTmFtZTogQ29uZ3Jlc3MgSGFsbCBJDQotLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCg0KDQpSZXF1ZXN0ZWQgSW5mb3JtYXRpb246 DQoNCg0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tDQpXb3JraW5nIEdyb3VwIE5hbWU6IG1wdGNwDQpBcmVhIE5hbWU6IFRyYW5zcG9ydCBB cmVhDQpTZXNzaW9uIFJlcXVlc3RlcjogUGhpbGlwIEVhcmRsZXkNCg0KTnVtYmVyIG9mIFNlc3Np b25zOiAxDQpMZW5ndGggb2YgU2Vzc2lvbihzKTogIDIgaG91cnMNCiAgICAgICAgICAgICAgICAg ICAgICAgDQogICAgICAgICAgICAgICAgICAgICAgIA0KTnVtYmVyIG9mIEF0dGVuZGVlczogMTAw DQpDb25mbGljdHMgdG8gQXZvaWQ6DQogIEZpcnN0IFByaW9yaXR5OiBtaWYgdHN2d2cgdHN2YXJl YSBjb25leA0KICBTZWNvbmQgUHJpb3JpdHk6ICB0Y3BtIGxlZGJhdCBwY24gYXBwYXJlYSBpbnRh cmVhDQoNClNwZWNpYWwgUmVxdWVzdHM6DQogIA0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQoNCg0K