From mirja.kuehlewind@ikr.uni-stuttgart.de Thu Jan 7 05:08:09 2010 Return-Path: X-Original-To: ledbat@core3.amsl.com Delivered-To: ledbat@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 402513A6810 for ; Thu, 7 Jan 2010 05:08:09 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.249 X-Spam-Level: X-Spam-Status: No, score=-2.249 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_DE=0.35] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4FbfolPHVs2v for ; Thu, 7 Jan 2010 05:08:09 -0800 (PST) Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by core3.amsl.com (Postfix) with ESMTP id 011D33A676A for ; Thu, 7 Jan 2010 05:08:09 -0800 (PST) Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id 57666439F4 for ; Thu, 7 Jan 2010 14:08:07 +0100 (CET) Received: from localhost (inode21 [10.21.18.11]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id 3DB77BC07E for ; Thu, 7 Jan 2010 14:08:07 +0100 (CET) To: ledbat@ietf.org Content-Disposition: inline From: Mirja Kuehlewind Organization: University of Stuttgart (Germany), IKR Date: Thu, 7 Jan 2010 14:08:05 +0100 MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201001071408.05477.mirja.kuehlewind@ikr.uni-stuttgart.de> Subject: [ledbat] Evaluation Results? X-BeenThere: ledbat@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Mailing list of the LEDBAT WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jan 2010 13:08:09 -0000 Hi, at the last meeting in Hiroshima there was this "magic number" discussion. It was stated that evaluation results of the current available implementation show that these numbers are suitable. Are these results somewhere available? Thanks, Mirja From muraris@microsoft.com Wed Jan 13 09:17:44 2010 Return-Path: X-Original-To: ledbat@core3.amsl.com Delivered-To: ledbat@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 206D728C102 for ; Wed, 13 Jan 2010 09:17:44 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.398 X-Spam-Level: X-Spam-Status: No, score=-9.398 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, J_CHICKENPOX_56=0.6, J_CHICKENPOX_62=0.6, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JdppvL9e3xqw for ; Wed, 13 Jan 2010 09:17:28 -0800 (PST) Received: from smtp.microsoft.com (smtp.microsoft.com [131.107.115.214]) by core3.amsl.com (Postfix) with ESMTP id 1E15F3A682B for ; Wed, 13 Jan 2010 09:17:28 -0800 (PST) Received: from TK5EX14HUBC102.redmond.corp.microsoft.com (157.54.7.154) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Wed, 13 Jan 2010 09:18:08 -0800 Received: from TK5EX14MBXC128.redmond.corp.microsoft.com ([169.254.7.2]) by TK5EX14HUBC102.redmond.corp.microsoft.com ([157.54.7.154]) with mapi; Wed, 13 Jan 2010 09:17:24 -0800 From: Murari Sridharan To: "ledbat@ietf.org" Thread-Topic: missing minutes Thread-Index: AcqUdEUCR4wGKYsLS3WAiDv0JMZdwQ== Date: Wed, 13 Jan 2010 17:17:22 +0000 Message-ID: <03520536067DF14395EA93FE7ECFDCF01C08F19F@TK5EX14MBXC128.redmond.corp.microsoft.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Content-Type: multipart/mixed; boundary="_005_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_" MIME-Version: 1.0 Subject: [ledbat] missing minutes X-BeenThere: ledbat@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Mailing list of the LEDBAT WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 17:17:44 -0000 --_005_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_ Content-Type: multipart/alternative; boundary="_000_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_" --_000_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable All, I was fairly sure that the merged minutes from Alissa and Eric's notes= were posted right after Hiroshima before I left. I am attaching the raw no= tes. There are several action items from the meeting that we need to act on= , primarily a document update. In Eric's notes the action items are marked = with a ****. I will try to post the merged notes that I have to the tools site. Thanks --_000_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

All, I was fairly sure that the merged minutes from Al= issa and Eric’s notes were posted right after Hiroshima before I left. I a= m attaching the raw notes. There are several action items from the meeting th= at we need to act on, primarily a document update. In Eric’s notes the action items are marked with a ****.

 

I will try to post the merged notes that I have to the= tools site.

 

Thanks

--_000_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_-- --_005_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_ Content-Type: message/rfc822 Content-Disposition: attachment; creation-date="Wed, 13 Jan 2010 17:17:23 GMT"; modification-date="Wed, 13 Jan 2010 17:17:23 GMT" Received: from TK5-EXHUB-C101.redmond.corp.microsoft.com (157.54.18.48) by TK5EX14HUBC103.redmond.corp.microsoft.com (157.54.86.9) with Microsoft SMTP Server (TLS) id 14.0.639.20; Thu, 12 Nov 2009 18:57:36 -0800 Received: from tk5-exgwy-e802.partners.extranet.microsoft.com (10.251.56.168) by TK5-EXHUB-C101.redmond.corp.microsoft.com (157.54.18.48) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 12 Nov 2009 18:57:32 -0800 Received: from mail181-tx2-R.bigfish.com (65.55.88.114) by mail02.microsoft.com (10.253.140.28) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 12 Nov 2009 18:04:39 -0800 Received: from mail181-tx2 (localhost.localdomain [127.0.0.1]) by mail181-tx2-R.bigfish.com (Postfix) with ESMTP id 06A13C4042F for ; Fri, 13 Nov 2009 02:04:39 +0000 (UTC) X-SpamScore: -18 X-BigFish: ps-18(zz1803O1454K9370J14c3L936eM853kzz1202hzzz2dh6bh62h) X-Spam-TCS-SCL: 1:0 Received: from mail181-tx2 (localhost.localdomain [127.0.0.1]) by mail181-tx2 (MessageSwitch) id 1258077877207011_17510; Fri, 13 Nov 2009 02:04:37 +0000 (UTC) Received: from TX2EHSMHS024.bigfish.com (unknown [10.9.14.244]) by mail181-tx2.bigfish.com (Postfix) with ESMTP id 2FB1B320052 for ; Fri, 13 Nov 2009 02:04:37 +0000 (UTC) Received: from gs19.inmotionhosting.com (66.117.3.189) by TX2EHSMHS024.bigfish.com (10.9.99.124) with Microsoft SMTP Server (TLS) id 14.0.482.32; Fri, 13 Nov 2009 02:04:36 +0000 Received: from host-40-25.meeting.ietf.org ([133.93.40.25]) by gs19.inmotionhosting.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1N8lWW-0000TF-KH; Thu, 12 Nov 2009 18:04:29 -0800 Message-ID: <6BC05803-558C-4D57-8B1F-9E2735FD1270@standardstrack.com> From: Eric Burger To: , Content-Type: text/plain; charset="WINDOWS-1252"; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Subject: LEDBAT minutes Date: Fri, 13 Nov 2009 11:04:30 +0900 X-Mailer: Apple Mail (2.936) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - gs19.inmotionhosting.com X-AntiAbuse: Original Domain - microsoft.com X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - standardstrack.com X-Source: X-Source-Args: X-Source-Dir: X-Reverse-DNS: gs19b.inmotionhosting.com Return-Path: eburger@standardstrack.com X-MS-Exchange-Organization-Antispam-Report: IPOnAllowList X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0 X-MS-Exchange-Organization-SCL: 0 X-MS-Exchange-Organization-AuthSource: tk5-exgwy-e802.partners.extranet.microsoft.com X-MS-Exchange-Organization-AuthAs: Anonymous MIME-Version: 1.0 LEDBAT IETF 76, 13 November 2009 Stanislav Shalunov, Murari Chairs Eric Burger, Scribe =3D=3D=3D=3D=3D Summary Action items. Items are marked with =93****=94 in the narrative. LEDBAT document: Document why no framing in LEDBAT document Need to document need for real randomness in algorithm. Both in terms =20 of implementation (don=92t count on disk rotation latency, etc.) and =20 theory (non-convergence). Document late-comers ADVANTAGE Explain where constants come from. Give a single, specific value with =20 a range. Recommendations document: Split out DIFFSERV into a different document (if people care about =20 DIFFSERV) Be absolutely sure to clarify difference between practices that work =20 around bugs versus practices that are good to follow. Specifically, =20 approaches to work around broken HTTP implementations (e.g., no =20 pipelining) Please send text; formulate things in terms of =93It would be in your =20 interest NOT to do xxx=94 instead of =93You SHOULD do yyy.=94 Low Extra Delay Background Transport: Stanislav Shalunov =3D=3D=3D=3D=3D Framing ----- Rich Woundy: Transport level may or may not have transport =20 identifiers. Depending on transport protocol, you will need to be able =20 to identify stream. Slav: need it for fine-grain monitoring. However, "this is the last =20 latency I have seen" should be good enough. Reording should not be =20 much of an issue. Rich: worth highlighting order issue. If you get stuff in the order =20 3-2-1, you will think packet 3 has great latency, 2 has normal =20 latency, 1 has awful latency. Slav: if measurement error is small, should not impact algorithm. If =20 lots of variance, then ledbat algorithm has failed. Murari: if I am doing framing, why would I make it sequence-number =20 specific? Slav: not a stream-domain protocol. Lars: Not doing framing, but should note how to frame: good, bad, =20 recommended. Should be in the document. Not normative, but =20 suggestions. **** Late comer's (non)-advantage ----- Lars: implementation depends on side effects on user's operating =20 system, and looks like only might work by serendipity. It seems to be =20 just a hack that might not work all the time. Slav: if your machine is so good, then you can introduce noise / =20 random fluctuations in inter-packet transmission time. Stewart: As world moves to flash disk, high-performance CPUs, timers =20 are more and more precise. Cannot depend on the end system introducing =20 random noise. Murari: problem is non-existent on many systems. Rich: Assumption about a backoff time is random. Need to clearly state =20 expectation that things back off with a random component. The random =20 component is an implementation detail. **** Lars: Better to have something explicit in the document. Slav: Need to flush queue to figure out RTT on second stream to really =20 know. Lars: That is too heavy weight (Slav clearly agrees) Bob Briscoe: Late-comer's advantage: please add it to the draft as =20 described on the list. Disadvantage: late-comer does not know base =20 delay. Advantage: if it does know base delay, other flows will not =20 necessarily get out of the way of the new flow. Draft says randomness =20 fixes this, but that means randomness needs to be ensured **** LNS Fairness ----- Bob: Need to write down assumptions. Random walks with a lot of flows =20 competing with each other; do not see how ones will a small amount of =20 bandwidth will ever get large or vice versa. Need to state why =20 randomness helps and importance of true randomness in the algorithm. =20 **** Andrew: randomness protocol adds needs an estimator to know what =20 randomness is available so it knows how much randomness to add. Bob: May be we see randomness in networks because today=92s protocols =20 have randomness built in. If on a ledbat-only network, no randomness. =20 E.g., data center Slav: experience is randomness still there. Stuart: every network does NOT have randomness. Slav: every network I =20 have TRIED has randomness, but not necessarily true for all networks. =20 Stuart: MacOS 8.6 (?) had this problem; throughput cut in half. No =20 randomness, so everyone had the same collisions repeating. Bob: just need to note this in spec. Parameter Values ----- Bob: very uncomfortable about target delay at 25ms Murali: if we are not strict (MUST), then bad actors can set values =20 better for themselves but worse for the network Lars: bad actors can always chose, e.g., low TARGET values Lars: This is the "Magic Numbers" slide. Numbers do not seem to have =20 any justification. OK for experimental protocol. Need a lot of data on =20 what we think will be eventual PS solution. Slav: can get rid of the NOISE_FILTER number; almost always will be 1 =20 packet, which means it is not considered in the calculation. Today, =20 these are ranges because we do NOT know what the value should really =20 be. That is why we are doing the experiment. Jana: these values came from somewhere. How about: =93Here is a =20 mechanism, and here is a constant. Here are the implications of moving =20 the values.=94 Should explain all of the constants this way. **** How =20 can I see if the protocol works if different implementations use =20 different parameters. Where is the failure, if any? Implementers will =20 just copy and paste from spec, so why not give preferred values? Lars: =93It should be this number, but here is the range.=94 Andrew: Bottom 5 numbers are consequences of the choice of the first =20 number. However, the first number seems to be pulled out from nowhere. Slav: Yes, it really IS a magic number. Implementation ----- Rich: Has anyone done an ns2 or ns3 model. Slav: some told in private, but nothing public. Murali: one implementation known of. General Questions ----- Jana: Do we have data we can look at data from BitTorrent =20 implementation? Slav: Can distribute test executable for people. Not open source. Jana: Not using TCP mechanisms. There should be discussion about =20 burstiness considerations. Slav: spent lots of time looking at rate-based, window-based, etc. =20 methods. Not sure how much should be in the spec. Since spec is for =20 implementers, not researchers. Do not need it. Lars: actually, you do need to figure out, e.g., when to send =20 feedback. Perhaps =93Framing Considerations=94? Jana: Need to have some mechanism for spreading out (not bursting) data. Slav: Thought about it a lot. There may be something in their =20 implementation, but not there today. Discuss off-line. Bob: Note: we do not know that BitTorrent implementation of ledbat is =20 ledbat. Slav: Feel free to write an implementation. LEDBAT Applications, Practices, and Recommendations. (Reinaldo Penno) =3D=3D=3D=3D=3D Lars: What do you mean by a control connection? Is it a limited =20 application connection? Reinaldo: It is not a bulk data connection. Vijay: Control connection could be, e.g., SIP; data could be RTP. HTTP =20 does not have that distinction. FTP does. Control connection exchanges =20 parameters to setup bulk data transfer connection. Lars: this is not the obvious place to discuss DIFFSERV. Might want to =20 split out to another document. **** Recommendations ----- Bob: Scope of draft is about application design. One aspect is whether =20 you give user control. Not in document. Could be important to document. Six connections: =93A number between 4 and 8.=94 Jana: How does =20 application know first connections are not saturated. Reinaldo: Because the application does not see packet loss. Stuart: Need to look at studies, but on his MacBook he can saturate =20 his GE interface with a single TCP connection. TCP fast retransmit =20 works better with one connection than with multiple connections. Reinaldo: multiple connections good for head-of-line blocking. =20 Described in the draft. Jana: purpose of multiple connections was for head-of-line blocking, =20 not for getting more-than-fair network bandwidth. Might get more =20 bandwidth, but not guaranteed and not the purpose. Should not be a =20 consideration for this work group. Slav: Set your Firefox browser to 32 connections per server. Looks =20 like win is from parallizing latency from server response. Bob: Would it be better to say things like =93It would be in your =20 interest NOT to xxx=94 instead of saying =93You SHOULD do yyy.=94 **** Jana: +1. Need to wordsmith carefully, as implementors can misconstrue =20 document to say, =93I can always open 6 connections.=94 Reinaldo: Please send text **** Concurrent Connections ---- Stuart: Is question how to generically deal with performance, or how =20 to deal with HTTP servers that cannot do pipelined requests? Good HTTP =20 guidance, but not general guidance. You get better performance with a =20 server that does pipelining with a single connection than multiple =20 connections. Slav: Does no harm Murali: about documenting current practices. Stuart: Important to highlight HTTP issue, not an IP issue. Reinaldo: Important to document why using multiple connections for HTTP. Stuart: Useful to describe what is being done to work around bugs =20 versus truly best practices. **** Bob: Often confusion about problems with half-open connections on =20 operation systems. Murali: Anyone do not want to adopt this document as a WG document? RESULT: silence Murali: Anyone want to adopt this? RESULT: consensus Jana: +1 to Stuart about being careful distinguishing current practice =20 versus recommendations **** Goal: have parameter values by Anaheim (IETF 77). Please review pseudo =20 code. Discuss on list. At least one implementation report by Anaheim. Also: do not have implications of opening multiple connections on =20 devices with low power constraints. --_005_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_ Content-Type: message/rfc822 Content-Disposition: attachment; creation-date="Wed, 13 Jan 2010 17:17:23 GMT"; modification-date="Wed, 13 Jan 2010 17:17:23 GMT" Received: from SVC-EXGWY-E802.partners.extranet.microsoft.com (10.251.24.243) by TK5EX14MLTC103.redmond.corp.microsoft.com (157.54.79.174) with Microsoft SMTP Server (TLS) id 14.0.639.20; Thu, 12 Nov 2009 17:58:51 -0800 Received: from mail38-tx2-R.bigfish.com (65.55.88.111) by mail05.microsoft.com (10.253.160.185) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 12 Nov 2009 17:58:42 -0800 Received: from mail38-tx2 (localhost.localdomain [127.0.0.1]) by mail38-tx2-R.bigfish.com (Postfix) with ESMTP id 53A9A3A82BD for ; Fri, 13 Nov 2009 01:58:19 +0000 (UTC) X-SpamScore: -30 X-BigFish: ps-30(zz146fKef4RaccKa706ladajzz1202hzz1033ILz2dh6bh65h) X-Spam-TCS-SCL: 4:0 Received: from mail38-tx2 (localhost.localdomain [127.0.0.1]) by mail38-tx2 (MessageSwitch) id 1258077494934156_27974; Fri, 13 Nov 2009 01:58:14 +0000 (UTC) Received: from TX2EHSMHS047.bigfish.com (unknown [10.9.14.241]) by mail38-tx2.bigfish.com (Postfix) with ESMTP id E14731640050 for ; Fri, 13 Nov 2009 01:58:14 +0000 (UTC) Received: from mail.maclaboratory.net (209.190.215.232) by TX2EHSMHS047.bigfish.com (10.9.99.147) with Microsoft SMTP Server (TLS) id 14.0.482.32; Fri, 13 Nov 2009 01:58:14 +0000 X-Spam-Status: No, hits=0.0 required=4.0 tests=AWL: 0.000,BAYES_00: -1.665,TOTAL_SCORE: -1.665,autolearn=ham X-Spam-Level: Received: from host-18-93.meeting.ietf.org ([133.93.18.93]) (authenticated user acooper@cdt.org) by mail.maclaboratory.net (using TLSv1/SSLv3 with cipher AES128-SHA (128 bits)); Thu, 12 Nov 2009 20:58:06 -0500 Message-ID: From: Alissa Cooper To: Murari Sridharan , Stanislav Shalunov Content-Type: text/plain; charset="US-ASCII"; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Subject: Raw notes from LEDBAT Date: Fri, 13 Nov 2009 10:58:02 +0900 X-Mailer: Apple Mail (2.936) X-Reverse-DNS: mail.maclaboratory.net Return-Path: acooper@cdt.org X-MS-Exchange-Organization-Antispam-Report: IPOnAllowList X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0 X-MS-Exchange-Organization-SCL: 0 X-MS-Exchange-Organization-AuthSource: SVC-EXGWY-E802.partners.extranet.microsoft.com X-MS-Exchange-Organization-AuthAs: Anonymous X-Message-Flag: Follow up Reply-By: Sun, 15 Nov 2009 00:00:00 +0000 MIME-Version: 1.0 Here you go. -Alissa ------------ LEDBAT Low Extra Delay Background Transport (LEDBAT), Stas Shalunov http://tools.ietf.org/id/draft-ietf-ledbat-congestion-00.txt Stas TCP congestion control designed to maximize throughput, but not address latency if you ignore one characteristic, may be fine, may not be algorithms can be absed on either characteristic or a combination TCP based on throughput Stas approach: keep latency fixed draft has new name: draft-ietf-ledbat-congestion were several open issues, the one non-trivial one was addressed: non- congestive constraint non-congestive constraint: app may be rate-limited itself in this case, congestion may not be encoutered for an extended period, queuing delay is zero congestion control would continue to climb forever if congestion is later encountered, app will have to start from a point where it thinks it can send much more than it should not only will delay be high, but app will encounter loss --> sub-optimal sender should never make assumptions to deviant from what it has done in the past proposed solution: flight size flight size (fs) = amnt of outstanding data (sent but not yet acknowledged by the recipient) fs only meaninful to the sender fs is effective window *right now* keep congestion window on a leash attached to fs cwnd <= ALLOWED_INCREASE + TETHER*fs where TETHER is your leeway parameter tether needs to allow for at least one more packet to allow you to probe, so can't be < 2 but tether < 2 may be too generous once you've ramped up, so we add additive constant (ALLOWED_INCREASE) so tether must be > 1, Stas recommends at least 25% increase ALLOWED_INCREASE and TEHTER are constant A_I must be at least 1 packet, but not more than 3 packets TETHER must be > 1, should be between 1.25 and 2 fs is implemented in all BitTorrent products that use uTP known document issues: framing is out of scope -- discussed during charter discussion need to capture discussion about late comer's advantage (lack thereof) need to discuss whether we have fairness between ledbat flows need to discuss choice of parameter values, e.g., when there are ranges, what tradeoffs are between either end of range check for loose ends in pseudocode and spec -- need new eyes on this framing out of scope: wire representatin is out of scope but need to be able to carry two quantities: time stamp and delay without time stamp, can't measure one-way delay recipient needs to be able to convey info about one-way delay to sender two quantities go in opposite directions (sender->recipient and vice versa) current BT implementation uses UDP, proprietary framing could use this with TCP, but embedding two quantities is not easy in TCP header Rich Woundy: transport level may ahve apcket IDs or it may not, so transport protocol may have to declare which packet it's talking about Stas: you could do that for assurance against extra re-ordering, but you don't have to, can just imply the delay you last saw (delay in each direction is likely similar), so this is probably not a requirement Rich: may not be a req, just worth pointing out Murari: out of order should not matter Rich: worried about delay seesaw-ing (3,2,1 over and over getting mixed up) Stas: re-order can be from sender to receiver or vice versa; former will result in measurement noise and if it's small, should be fine (if large, delay-based protocol will fail anyway); in latter, packet IDs could add extra value; BT implementation has the extra value of ACKs Murari: if I'm framing anyway, why wouldn't I make it sequence number- specific? Stas: but this doesn't need to be a stream transmission protocol Rich: but it may not be TCP. but this is fine. Lars Eggert: point is that we're not doing framing, but can add considerations for what framing protocol needs to do (not reqs, reqs are too heavy) Stas/Murari: agreed late comer's (non) advantage: first ledbat connection builds up a target worth of queue, reaches steady state second connection comes in, starts measuring delay, doesn't know base delay firs connection sees more larger queue, second connection is at advantage of being able to get an extra target's worth of packets two reasons why this doesn't happen in the protocol 1) dips in queue are enough to measure base delay; dips come from machine hiccups 2) even without dips, once first connection backs off, second connection is now exclusive network user and can't be deceived Lars: worried about this. for the good operation of this mechanism we depend on side effects of the computer, or we say even when it goes wrong it doesn't go wrong for long. can we find a mechanism that gives the second connection a better example quicker? e.g., every 5 seconds stop sending (bad suggestion) Stas: or every minute might be feasible. implementors might know what kind of machines you're going to run on. Stuart Cheshire: what if you have a dedicatd special-purpose device with flash-based memory and no disk? data rate can be absolutely constant. can't expect randomness. have discussion of how much randomness is necessary. Murari: position that ICCRG has taken is that if your protocol is only delay-based, they've been very particular about this. Rich: perhaps we cna say there's an assumption of a back-off time but implementation can be based on some randomness. how you back off periodically is not important, just that it happens. Stas: agreed. Murari: can do even better than randomness probably. Lars: having something in the mechanism would be better bc even as implementor you don't know where it will get deployed. Stas: not much choices exists in design space. need to transmit info about base delay through the queue. must empty the queue, and to that you shut off for a RT time. Murari: have to do something periodically. Lars: isn't that too heavyweight? something else we can do? jitter the delay measurement? Bob Briscoe: late comer's disadvantage: has been discussed on list. should be added to the draft. late comer's disavantage = other flows don't get out of the way for new flows. they see increase in delay as does the new flow. Stas: this is the fairness issue, coming up next. fairness between ledbat flows: how do multiple ledbat connections share between themselves? mechanism converges, so there's no probing convergence time is proportional to rate at which bottleneck is sending fairness controller in XCP used to inform this mechanism: add random noise to delay measurement before using it, and suppose that value of radnomness is fixed fraction of target delay something proportional to this fraction will now be distributed each RT time between connections Bob: we need to write down assumptions. with lots of flows competing with each other, no guarantee that random walk will even things out. you only get that when they bounce off either edge. Stas: with constrained random walk, you converge. Andrew McGregor: if bottleneck queue is adding randomness Stas: that's good for redistribution. Andrew: other suggestion is that if randomness is too low, deliberately increase it Stas: have not seen a situation where there isn't enough randomness. reluctant to add an untested code path. suggested mechanism is too smart. Bob: echoing Stuart, we may see randomness bc protocols are designed to put randomness in. what about when protocol is running against itself? what about using ledbat in data center? Stuart: every network does not have randomness. shipped a Mac OS update with this exact problem over ISDN in France. bottleneck was very precisely clocked, TCP safety margin went to zero, triggered retransmission, every packet got sent twice, performance cut in half for 10,000s customers. Stas: understood. should add aritficial randomness. Bob: we are doing this as experimental, so don't have to prove all of it ahead of time. choice of parameter values: current reqs may be unecessarily strict, target and gain both fixed target must be 25 ms or more (and should be < ??) gain must be 1 mss/rtt, but what if you ramp up slower? Bob: very uncomfortable of target delay of 25 ms. once we standardize, if we want to get rid of those delays. Stas: get rid of 8 s of delay first, then we can get rid of last 25 ms. Bob: just need to think about how we can get rid of 25 ms. Murari: if we're not strict, we create unfairness. Stas: yes, allow people to set values that are bad for them if they compete with standard values. Lars: not sure if there will be competition between ledbat flows, but you could win by changing target. Stas: in BT, there is competition, several connections share same bottleneck (home uplink). 4-5 uploading connections split it up based onr andom walk. 2 <= base history <= 10: higher values give better measurements of base delay if neither of the other factors is active but there may be situations to deviate from sweet spot (perfect clocks, no rerouting) 1 packet <= noice filter <= cwnd/2 (should): at low end, you get faster reaction and lower delay in control loop -> more stable, converge faster. probably not a good idea to make noice filter more than 1 cwnd (1 RTT) bc you will double your response time. Lars: I hate magic numbers. these are all magic numbers. would like to elminate some of them or understand better what they should be. eventually we will try to make this a proposed standard, so we will need a lot of data on what these should be set for general internet conditions. might be better to exclude cases where you say "under controlled environment" -- charter discusses deployment on general internet. Stas: that's why we use SHOULD. they're not banned, they're not required. we could get rid of the noise filter, bc in majority of situations you want it to be 1 packet. but others can't really be removed. Lars: it's not only magic numbers, it's also magic ranges. might need experimentation for some of them. Stas: I can't seriously say that I know what the tether value should really be, that's why it has a range. within the ranges, all values would work on internet deployment. Jana Iyengar: one point of experimental protocol is to see if it works, how can we specify a range and see if it works? they will all be using different parameters. Murari: so you want us to pick some value? Stas: SHOULD value that is not a range? Jana: lay down a consideration of the extremes, and then give suggestions of what constants are likely to work. instead of using SHOULD (not as strong). Lars: Jana is saying, it SHOULD be X, and here's what happens when you go lower or higher. Stas: need something between SHOULD and MUST Lars: SHOULD REALLY Andrew: target is a magic number and all the others are consequences of target. but why 25 ms? Stas: we had extensive discussion of this at last meeting. careful proofreading: need implementors to read (esp section 5) best way to read is to implement Rich: has anyone done an NS 2 or NS 3 model? Murari: I will send report to the mailing list of the one that I know of. Rich: might be interesting to take the model and run it against different network topologies. Vijay Gurbani: have implemented BT in NS 3. doesn't have docsis but does have a wireless model. Stas: and don't need to simulate in context of BT, because that has been done already. Q&A Jana: can we look at data about BT? Stas: Bt carries most of the congestive data on the internet. it's a real implementation. Jana: would like to see some data to decide if it works. Stas: client is not open source, but if people need text executable, Stas can give that out. Rich: can we send that to the mailing list? Stas: I will investigate. but you can get it privately. Jana: no discussion of burstiness in algorithm. Stas: tried rate-based vs window-based, but not sure how much of it should be in a spec. can discuss what has been considered and whether it worked, but target audience of the draft is implementor, not researcher. Lars: there are implications on framing. how often do you send the feedback? Murari: may be more than framing. Jana: was talking about burstiness of bulk application. if delay is distributed across multiple queues, that's a reason to spread your data out. pacing or something. not bursting. need to know what amount of burst is reasonable. Stas: discuss offline. Jana: I think it should go in the draft. Bob: we don't know if the BT implementation is a ledbat implementation bc it's not open source. Stas: go ahead. LEDBAT Practices and Recommendations for Managing Multiple Concurrent TCP Connections, Reinaldo Penno http://tools.ietf.org/id/draft-penno-ledbat-app-practices-recommendations-01.txt draft was written because of concern about apps opening multiple TCP connections new draft has recommendations section comments from IETF74 onwards: differentiate between multiple control and data connections DiffServ may or may not be in scope rule of thumb about how many connections are too much? reassembly at app layer. not in spec right now multiple control vs data connections: new section 3 http can open many data connections protocols on top of http open multiple control and data connections BT, Skype use multiple control and data connections Lars: what does control connection mean? Vijay: control connection might be the signaling part in SIP; HTTP doesn't have the differentiator Lars: it's not about one being about xfer and the other not. Vijay: FTP is another example. control connection is there to exchange meta data, data connection to exchange file. DiffServ: section 6.1: recommendation to use RFC 3662 to mark packets for bulk transfer WG input necessary to decide if this should be in there Lars: this document is not an obvious place to put this. could spin it out into stand-alone document or whether the WG will take it on. how many connections are needed: new section 6.3: many recent studies on parallel TCP connections can't make recs for control connections bc they make the protcol work disadvantage due to reassembly on top of TCP at app layer: some studies exist about this, need to add text about this in next rev Bob: draft is about application design. one aspect of design is whether you give the user control over this. should be mentioned. recommendations: rec-1: bulk data xfer should be DiffServ marked rec-2: window should be scaled rec-3: apps should only open more than 6 connections to download the same object if the first hop link is not saturated. browser usually open more than 4, some studies show going up to 8. 6 is a compromise between 4 and 8. Jana: for app to open 6 connections, it should know link is not saturated, how does it know? Reinaldo: app does not see packet drops, so knows link is not saturated. all the studies have constraints. congestion can be near the server. there are lots of reasons to use multiple connections, listed in the draft. Stuart: GB ethernet can be saturated with one connection. browser designers thought TCP fairness was based on flow fairness, not realizing that one pipeline can give better performance. Reinaldo: a lot depends on head of line blocking and whether one persistent connection will slow down everything else or not. TCP usually goes up to 75% of a link. Stas: if you type about:config into Firefox and search for connections and there's a network.http.max-connections and value of default is 15. can change that number to 32 and it will go faster for you. Jana: browsers do not open multiple connections for bandwidth, but for head of line blocking. badnwidth is a side effect. Bob: maybe better than SHOULD is say it's "unlikely to be in your interest" for designers. Murari: purely for throughput reasons, we should not recommending opening multiple connections. Jana: wary of IETF saying "apps can open 6 connections" Reinaldo: suggest text, please. Firefox, Internet Explorer Cloud Four: has 4 domains, asks people to access the page from major mobile browsers some browsers open 40 connections (Opera), next version down to 5 connections Stuart: Safari opens multiple connections because some percentage of webs ervers can't handle pipelines requests. this might be good HTTP guidance, but not sure if it's good for broader applications guidance. Stas: internet is not blowing up because of multiple connections. it's the de facto practice. Stuart: important to distinguish HTTP servers from others Reinaldo: draft has lots of discussion of why browsers do multiple connections Bob: lots of confusions about OS limits on half-open connections. Windows limit used to be 10. RFC 5290 says that these limits exist and it doesn't say they are for half-connections. draft should distinguish between connections and half connections. rec-4: ? to dos: bi-directional HTTP recommendations? reassembly at app level Murari: hum: don't adopt document? result: no hums hum: adopt as WG doc? result: consensus in favor Jana: need to be careful about observing what happens and saying what we recommend Reinaldo: can split into subsections what we observe today and what future recs are Wrap-up: Murari goal for Anaheim IETF: finalize key pieces of the algorithm want to have implementation report on the agenda for the next meeting app practices doesn't discuss power cosndierations for constrained devices (mobile) --_005_03520536067DF14395EA93FE7ECFDCF01C08F19FTK5EX14MBXC128r_-- From muraris@microsoft.com Wed Jan 13 10:09:49 2010 Return-Path: X-Original-To: ledbat@core3.amsl.com Delivered-To: ledbat@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E34A13A69E7 for ; Wed, 13 Jan 2010 10:09:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -9.998 X-Spam-Level: X-Spam-Status: No, score=-9.998 tagged_above=-999 required=5 tests=[AWL=0.600, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BlUoLzrZDmIn for ; Wed, 13 Jan 2010 10:09:47 -0800 (PST) Received: from smtp.microsoft.com (mail3.microsoft.com [131.107.115.214]) by core3.amsl.com (Postfix) with ESMTP id 1C48F3A69CF for ; Wed, 13 Jan 2010 10:09:47 -0800 (PST) Received: from TK5EX14MLTC104.redmond.corp.microsoft.com (157.54.79.159) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Wed, 13 Jan 2010 10:10:27 -0800 Received: from TK5EX14MBXC128.redmond.corp.microsoft.com ([169.254.7.2]) by TK5EX14MLTC104.redmond.corp.microsoft.com ([157.54.79.159]) with mapi; Wed, 13 Jan 2010 10:09:12 -0800 From: Murari Sridharan To: "ledbat@ietf.org" Thread-Topic: missing minutes Thread-Index: AcqUdEUCR4wGKYsLS3WAiDv0JMZdwQABkdxg Importance: low X-Priority: 5 Date: Wed, 13 Jan 2010 18:09:10 +0000 Message-ID: <03520536067DF14395EA93FE7ECFDCF01C09052F@TK5EX14MBXC128.redmond.corp.microsoft.com> References: <03520536067DF14395EA93FE7ECFDCF01C08F19F@TK5EX14MBXC128.redmond.corp.microsoft.com> In-Reply-To: <03520536067DF14395EA93FE7ECFDCF01C08F19F@TK5EX14MBXC128.redmond.corp.microsoft.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: multipart/alternative; boundary="_000_03520536067DF14395EA93FE7ECFDCF01C09052FTK5EX14MBXC128r_" MIME-Version: 1.0 Subject: Re: [ledbat] missing minutes X-BeenThere: ledbat@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Mailing list of the LEDBAT WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 18:09:50 -0000 --_000_03520536067DF14395EA93FE7ECFDCF01C09052FTK5EX14MBXC128r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The minutes have been uploaded as well. Thanks From: ledbat-bounces@ietf.org [mailto:ledbat-bounces@ietf.org] On Behalf Of= Murari Sridharan Sent: Wednesday, January 13, 2010 9:17 AM To: ledbat@ietf.org Subject: [ledbat] missing minutes All, I was fairly sure that the merged minutes from Alissa and Eric's notes= were posted right after Hiroshima before I left. I am attaching the raw no= tes. There are several action items from the meeting that we need to act on= , primarily a document update. In Eric's notes the action items are marked = with a ****. I will try to post the merged notes that I have to the tools site. Thanks --_000_03520536067DF14395EA93FE7ECFDCF01C09052FTK5EX14MBXC128r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

The minutes have been up= loaded as well.

 =

Thanks=

 =

From: ledbat-bounces@ietf.org [mailto:ledbat-bounces@ietf.org] On Behalf Of Murari Sridharan
Sent: Wednesday, January 13, 2010 9:17 AM
To: ledbat@ietf.org
Subject: [ledbat] missing minutes

 

All, I was fairly sure that the merged minutes from Al= issa and Eric’s notes were posted right after Hiroshima before I left. I a= m attaching the raw notes. There are several action items from the meeting th= at we need to act on, primarily a document update. In Eric’s notes the action items are marked with a ****.

 

I will try to post the merged notes that I have to the= tools site.

 

Thanks

--_000_03520536067DF14395EA93FE7ECFDCF01C09052FTK5EX14MBXC128r_-- From muraris@microsoft.com Thu Jan 21 01:27:29 2010 Return-Path: X-Original-To: ledbat@core3.amsl.com Delivered-To: ledbat@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 169DD3A635F for ; Thu, 21 Jan 2010 01:27:29 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.298 X-Spam-Level: X-Spam-Status: No, score=-10.298 tagged_above=-999 required=5 tests=[AWL=0.300, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O5mIT3Wq+Fol for ; Thu, 21 Jan 2010 01:27:25 -0800 (PST) Received: from smtp.microsoft.com (mailb.microsoft.com [131.107.115.215]) by core3.amsl.com (Postfix) with ESMTP id 82AFD3A68CC for ; Thu, 21 Jan 2010 01:27:25 -0800 (PST) Received: from TK5EX14HUBC101.redmond.corp.microsoft.com (157.54.7.153) by TK5-EXGWY-E802.partners.extranet.microsoft.com (10.251.56.168) with Microsoft SMTP Server (TLS) id 8.2.176.0; Thu, 21 Jan 2010 01:28:07 -0800 Received: from TK5EX14MBXC130.redmond.corp.microsoft.com ([169.254.9.211]) by TK5EX14HUBC101.redmond.corp.microsoft.com ([157.54.7.153]) with mapi; Thu, 21 Jan 2010 01:27:20 -0800 From: Murari Sridharan To: Murari Sridharan , "ledbat@ietf.org" Thread-Topic: missing minutes Thread-Index: AcqUdEUCR4wGKYsLS3WAiDv0JMZdwQABkdxgAYBUN8A= Date: Thu, 21 Jan 2010 09:27:18 +0000 Message-ID: <03520536067DF14395EA93FE7ECFDCF02197096C@TK5EX14MBXC130.redmond.corp.microsoft.com> References: <03520536067DF14395EA93FE7ECFDCF01C08F19F@TK5EX14MBXC128.redmond.corp.microsoft.com> <03520536067DF14395EA93FE7ECFDCF01C09052F@TK5EX14MBXC128.redmond.corp.microsoft.com> In-Reply-To: <03520536067DF14395EA93FE7ECFDCF01C09052F@TK5EX14MBXC128.redmond.corp.microsoft.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: Content-Type: multipart/alternative; boundary="_000_03520536067DF14395EA93FE7ECFDCF02197096CTK5EX14MBXC130r_" MIME-Version: 1.0 Subject: Re: [ledbat] missing minutes X-BeenThere: ledbat@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Mailing list of the LEDBAT WG List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2010 09:27:29 -0000 --_000_03520536067DF14395EA93FE7ECFDCF02197096CTK5EX14MBXC130r_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Document authors, we need updated documents based on the minutes before we = can have further discussions. Can you please provide an ETA for the same? From: ledbat-bounces@ietf.org [mailto:ledbat-bounces@ietf.org] On Behalf Of= Murari Sridharan Sent: Wednesday, January 13, 2010 10:09 AM To: ledbat@ietf.org Subject: Re: [ledbat] missing minutes Importance: Low The minutes have been uploaded as well. Thanks From: ledbat-bounces@ietf.org [mailto:ledbat-bounces@ietf.org] On Behalf Of= Murari Sridharan Sent: Wednesday, January 13, 2010 9:17 AM To: ledbat@ietf.org Subject: [ledbat] missing minutes All, I was fairly sure that the merged minutes from Alissa and Eric's notes= were posted right after Hiroshima before I left. I am attaching the raw no= tes. There are several action items from the meeting that we need to act on= , primarily a document update. In Eric's notes the action items are marked = with a ****. I will try to post the merged notes that I have to the tools site. Thanks --_000_03520536067DF14395EA93FE7ECFDCF02197096CTK5EX14MBXC130r_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Document authors, we need updated documents based on the minu= tes before we can have further discussions. Can you please provide an ETA f= or the same?

 

From: ledbat-bounces@ietf.org [mailto:ledbat-bounces@ietf.org] On Behalf Of = Murari Sridharan
Sent: Wednesday, January 13, 2010 10:09 AMTo: ledbat@ietf.org
Subject: Re: [ledbat] missing minute= s
Importance: Low

 

The minutes have been uploaded as well.

 

Thanks

=

 =

From: ledbat-bounces@ietf.org [mailto:= ledbat-bounces@ietf.org] On Behalf Of Murari Sridharan
Sent: Wednesday, January 13, 2010 9:17 AM
To: ledbat@ietf.org
= Subject: [ledbat] missing minutes

 

All, I was fairl= y sure that the merged minutes from Alissa and Eric’s notes were post= ed right after Hiroshima before I left. I am attaching the raw notes. There= are several action items from the meeting that we need to act on, primaril= y a document update. In Eric’s notes the action items are marked with= a ****.

 

I will try to post the merged notes that I have to the tools s= ite.

 

Thanks

= --_000_03520536067DF14395EA93FE7ECFDCF02197096CTK5EX14MBXC130r_--