From richard_woundy@cable.comcast.com  Wed Nov  4 14:29:18 2009
Return-Path: <richard_woundy@cable.comcast.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5103C3A6A0C for <homegate@core3.amsl.com>; Wed,  4 Nov 2009 14:29:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.86
X-Spam-Level: 
X-Spam-Status: No, score=-5.86 tagged_above=-999 required=5 tests=[AWL=2.602,  BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368,  HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kjUrFc7BSePp for <homegate@core3.amsl.com>; Wed,  4 Nov 2009 14:29:15 -0800 (PST)
Received: from pacdcimo01.cable.comcast.com (PacdcIMO01.cable.comcast.com [24.40.8.145]) by core3.amsl.com (Postfix) with ESMTP id E33193A6A0B for <homegate@ietf.org>; Wed,  4 Nov 2009 14:29:14 -0800 (PST)
Received: from ([10.195.246.152]) by pacdcimo01.cable.comcast.com with ESMTP  id 5503620.59654319; Wed, 04 Nov 2009 17:29:34 -0500
Received: from PACDCEXCMB06.cable.comcast.com ([24.40.15.22]) by NJMDCEXCRLY01.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 4 Nov 2009 17:29:35 -0500
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CA5D9E.495E9333"
Date: Wed, 4 Nov 2009 17:29:34 -0500
Message-ID: <8A82D1BFEDDE7E4597978355239BBBCB5CC779@PACDCEXCMB06.cable.comcast.com>
In-Reply-To: <C70E2D1C.158DD%jason_livingood@cable.comcast.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [homegate] Draft BoF Agenda - Your Feedback Needed
Thread-Index: AcpSjdDGIItn0/deIkWdHefSMaV34AFhjVpHAWKAKFA=
References: <C704E876.15051%jason_livingood@cable.comcast.com> <C70E2D1C.158DD%jason_livingood@cable.comcast.com>
From: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>
To: "Livingood, Jason" <Jason_Livingood@cable.comcast.com>, <homegate@ietf.org>
X-OriginalArrivalTime: 04 Nov 2009 22:29:35.0126 (UTC) FILETIME=[49682B60:01CA5D9E]
Subject: Re: [homegate] Draft BoF Agenda - Your Feedback Needed
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Nov 2009 22:29:18 -0000

This is a multi-part message in MIME format.

------_=_NextPart_001_01CA5D9E.495E9333
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

> We could still use someone to speak about congestion issues with home
gateway devices.  Do I have any volunteers?

=20

I put together a single slide of congestion management-related topics
that we could discuss next week.

=20

-- Rich

=20

From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On
Behalf Of Jason Livingood
Sent: Wednesday, October 28, 2009 5:17 PM
To: homegate@ietf.org
Subject: Re: [homegate] Draft BoF Agenda - Your Feedback Needed

=20

Our draft agenda has been uploaded to:
http://www.ietf.org/proceedings/09nov/agenda/homegate.txt

We could still use someone to speak about congestion issues with home
gateway devices.  Do I have any volunteers?

Also, security-related issues - need a volunteer as well.

Thanks
Jason=20


------_=_NextPart_001_01CA5D9E.495E9333
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:x=3D"urn:schemas-microsoft-com:office:excel" =
xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" =
xmlns:a=3D"urn:schemas-microsoft-com:office:access" =
xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" =
xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" =
xmlns:rs=3D"urn:schemas-microsoft-com:rowset" xmlns:z=3D"#RowsetSchema" =
xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" =
xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" =
xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" =
xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" =
xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" =
xmlns:html=3D"http://www.w3.org/TR/REC-html40" =
xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" =
xmlns:rtc=3D"http://microsoft.com/officenet/conferencing" =
xmlns:D=3D"DAV:" xmlns:Repl=3D"http://schemas.microsoft.com/repl/" =
xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/" =
xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml" =
xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" =
xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/" =
xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/" =
xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#" =
xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp" =
xmlns:udc=3D"http://schemas.microsoft.com/data/udc" =
xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema" =
xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/"=
 xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#" =
xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" =
xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/" =
xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance" =
xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap" =
xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" =
xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart" =
xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/" =
xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup" =
xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig" =
xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-sig=
nature" =
xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/2006=
" xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" =
xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationshi=
ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages" =
xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types"=
 =
xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messag=
es" =
xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/=
" =
xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pub=
lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" =
xmlns:st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40">

<head>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)">
<title>Re: [homegate] Draft BoF Agenda - Your Feedback Needed</title>
<style>
<!--
 /* Font Definitions */
 @font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
	{font-family:Tahoma;
	panose-1:2 11 6 4 3 5 4 4 2 4;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0in;
	margin-bottom:.0001pt;
	font-size:12.0pt;
	font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.EmailStyle17
	{mso-style-type:personal-reply;
	font-family:"Calibri","sans-serif";
	color:#1F497D;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-size:10.0pt;}
@page Section1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
	{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
 <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
 <o:shapelayout v:ext=3D"edit">
  <o:idmap v:ext=3D"edit" data=3D"1" />
 </o:shapelayout></xml><![endif]-->
</head>

<body lang=3DEN-US link=3Dblue vlink=3Dpurple>

<div class=3DSection1>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>&gt;</span><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"'>
We could still use someone to speak about congestion issues with home =
gateway
devices. &nbsp;Do I have any volunteers?</span><span =
style=3D'font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I put together a single slide of congestion =
management-related topics
that we could discuss next week.<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-- Rich<o:p></o:p></span></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p>&nbsp;</o:p></span></p>

<div>

<div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt =
0in 0in 0in'>

<p class=3DMsoNormal><b><span =
style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span>=
</b><span
style=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] <b>On =
Behalf Of </b>Jason
Livingood<br>
<b>Sent:</b> Wednesday, October 28, 2009 5:17 PM<br>
<b>To:</b> homegate@ietf.org<br>
<b>Subject:</b> Re: [homegate] Draft BoF Agenda - Your Feedback =
Needed<o:p></o:p></span></p>

</div>

</div>

<p class=3DMsoNormal><o:p>&nbsp;</o:p></p>

<p class=3DMsoNormal><span =
style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif"'>Our
draft agenda has been uploaded to:<br>
<a =
href=3D"http://www.ietf.org/proceedings/09nov/agenda/homegate.txt">http:/=
/www.ietf.org/proceedings/09nov/agenda/homegate.txt</a><br>
<br>
We could still use someone to speak about congestion issues with home =
gateway
devices. &nbsp;Do I have any volunteers?<br>
<br>
Also, security-related issues &#8211; need a volunteer as well.<br>
<br>
Thanks<br>
Jason</span> <o:p></o:p></p>

</div>

</body>

</html>

------_=_NextPart_001_01CA5D9E.495E9333--

From paul.hoffman@vpnc.org  Thu Nov  5 07:01:56 2009
Return-Path: <paul.hoffman@vpnc.org>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3552328C1C0 for <homegate@core3.amsl.com>; Thu,  5 Nov 2009 07:01:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -5.755
X-Spam-Level: 
X-Spam-Status: No, score=-5.755 tagged_above=-999 required=5 tests=[AWL=0.291,  BAYES_00=-2.599, HELO_MISMATCH_COM=0.553, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FxdG+iehbmwP for <homegate@core3.amsl.com>; Thu,  5 Nov 2009 07:01:55 -0800 (PST)
Received: from balder-227.proper.com (Balder-227.Proper.COM [192.245.12.227]) by core3.amsl.com (Postfix) with ESMTP id 32CA73A69D9 for <homegate@ietf.org>; Thu,  5 Nov 2009 07:01:55 -0800 (PST)
Received: from [75.101.18.87] (75-101-30-90.dsl.dynamic.sonic.net [75.101.30.90]) (authenticated bits=0) by balder-227.proper.com (8.14.2/8.14.2) with ESMTP id nA5F2EgC062507 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 5 Nov 2009 08:02:15 -0700 (MST) (envelope-from paul.hoffman@vpnc.org)
Mime-Version: 1.0
Message-Id: <p06240821c718990da5b6@[75.101.18.87]>
In-Reply-To: <C70E2D1C.158DD%jason_livingood@cable.comcast.com>
References: <C70E2D1C.158DD%jason_livingood@cable.comcast.com>
Date: Thu, 5 Nov 2009 07:02:12 -0800
To: Jason Livingood <jason_livingood@cable.comcast.com>, "homegate@ietf.org" <homegate@ietf.org>
From: Paul Hoffman <paul.hoffman@vpnc.org>
Content-Type: text/plain; charset="us-ascii"
Subject: Re: [homegate] Draft BoF Agenda - Your Feedback Needed
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 05 Nov 2009 15:01:56 -0000

At 5:17 PM -0400 10/28/09, Jason Livingood wrote:
>Also, security-related issues - need a volunteer as well.

Sorry for the delayed reply. I can handle that with a brief presentation on two topics: security protocols the firewall should be careful not to thwart, and good security practices for gateway management.

--Paul Hoffman, Director
--VPN Consortium

From iljitsch@muada.com  Mon Nov  9 11:26:26 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1C6FA3A6B9C for <homegate@core3.amsl.com>; Mon,  9 Nov 2009 11:26:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Y9F3FcFEJtoh for <homegate@core3.amsl.com>; Mon,  9 Nov 2009 11:26:25 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 11CB23A692A for <homegate@ietf.org>; Mon,  9 Nov 2009 11:26:24 -0800 (PST)
Received: from [192.168.0.195] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nA9JQfsP064018 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for <homegate@ietf.org>; Mon, 9 Nov 2009 20:26:42 +0100 (CET) (envelope-from iljitsch@muada.com)
From: Iljitsch van Beijnum <iljitsch@muada.com>
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Date: Mon, 9 Nov 2009 20:26:43 +0100
Message-Id: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com>
To: homegate@ietf.org
Mime-Version: 1.0 (Apple Message framework v1076)
X-Mailer: Apple Mail (2.1076)
Subject: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 09 Nov 2009 19:26:26 -0000

To expand a bit on what I said in the jabber room about buffers in  
home gateways:

I've considered two approaches. The first is to see how much buffering  
you need for a single TCP session to achieve 100% utilization. Because  
of the congestion window sawtooth behavior, without any buffering, a  
single session would only be able to use 75% of the link bandwidth.  
Caveats: in practice, we of course have more than one session, there  
are limits to the TCP window size and I might be completely mistaken.  
But anyway:

approx linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the  
linkbw and MSS are in kilobits. For MSS = 12 and RTT = 0.3:

linkbw	buffer size  buffer time
64k	717          90 ms
128k	2867         179 ms
256k	11k          344 ms
1M	171k         1368 ms
10M	17M          13.6 s
100M	1669M        133 s
1G	163G         21.7 m
10G	16T          3.5 h

So basically, trying to go from 75% to 100% isn't worth the trouble as  
soon as you approach speeds of a megabit.

Then what? What's the minimum that TCP needs to be able to function?  
Basically, three packets. With delayed acknowledgments, one ACK  
releases two new segments, and another one if the congestion window  
grew by one packet. If you can't buffer these three packets the  
congestion window can't grow so TCP can't do its job.

Of course if you have three buffers and one, two or three of them are  
filled and then a three packet burst comes in you're going to lose at  
least one packet. So it seems beneficial to have more buffers. But  
that doesn't help, TCP will just fill them up and create a standing  
queue. So the chances of a lost packet remain the same.

Now perhaps buffering only three packets is a bit too radical,  
especially in the absense of simulations or real-world testing. But we  
do have experience with 10 packet buffers, and those seem to work. So  
I'd suggest "no more than 10" as our bottom line for now. And a good  
AQM scheme is highly recommended to avoid the tail drops that happen  
all over the place in everything I talked about so far.

From anyrhine@cs.helsinki.fi  Tue Nov 10 12:31:18 2009
Return-Path: <anyrhine@cs.helsinki.fi>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D790A3A65A6 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 12:31:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Level: 
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BcGf72vUvl1r for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 12:31:17 -0800 (PST)
Received: from mail.cs.helsinki.fi (courier.cs.helsinki.fi [128.214.9.1]) by core3.amsl.com (Postfix) with ESMTP id 395B528C140 for <homegate@ietf.org>; Tue, 10 Nov 2009 12:31:16 -0800 (PST)
Received: from wel-19.cs.helsinki.fi (wel-19.cs.helsinki.fi [128.214.9.86]) (AUTH: PLAIN anyrhine, SSL: TLSv1/SSLv3,128bits,AES128-SHA) by mail.cs.helsinki.fi with esmtp; Tue, 10 Nov 2009 22:31:42 +0200 id 00063E1C.4AF9CDAE.00005635
Date: Tue, 10 Nov 2009 22:31:37 +0200
From: Aki M Nyrhinen <anyrhine@cs.helsinki.fi>
To: Iljitsch van Beijnum <iljitsch@muada.com>
Message-ID: <20091110203137.GA16898@wel-19.cs.helsinki.fi>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
In-Reply-To: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2009 20:31:18 -0000

I don't understand your numbers. I haven't really followed the discussion
well enough, but now feel the urge to provide some input anyway (danger).

The following states my personal opinion which I try to loosely justify in
an unscientific manner. I don't claim this is the only truth, and I'm
sure the details could be debated forever. In any case, I hope some of
this is useful.


On Mon, Nov 09, 2009 at 08:26:43PM +0100, Iljitsch van Beijnum wrote:
> To expand a bit on what I said in the jabber room about buffers in home 
> gateways:
>
> I've considered two approaches. The first is to see how much buffering  
> you need for a single TCP session to achieve 100% utilization. Because  
> of the congestion window sawtooth behavior, without any buffering, a  
> single session would only be able to use 75% of the link bandwidth.  
> Caveats: in practice, we of course have more than one session, there are 
> limits to the TCP window size and I might be completely mistaken. But 
> anyway:
>
> approx linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the  
> linkbw and MSS are in kilobits. For MSS = 12 and RTT = 0.3:
>
> linkbw	buffer size  buffer time
> 64k	717          90 ms
> 128k	2867         179 ms
> 256k	11k          344 ms
> 1M	171k         1368 ms
> 10M	17M          13.6 s
> 100M	1669M        133 s
> 1G	163G         21.7 m
> 10G	16T          3.5 h

If you aim at full utilization when RTT is 0.3 seconds, what exactly is
preventing TCP from achieving full utilization with 0.3 seconds worth of
buffering (that being linkbw * 0.3 in bits or bytes)? It really is this
simple if you know your RTT: TCP will halve its CWND upon a loss, which
occurs when the buffer is full, so if we set the buffer to RTT * BW, the
halved window still exactly saturates the link, leading to full utilization
and has nothing to do with how long it takes to complete a TCP sawtooth
cycle.

> Then what? What's the minimum that TCP needs to be able to function?  
> Basically, three packets. With delayed acknowledgments, one ACK releases 
> two new segments, and another one if the congestion window grew by one 
> packet. If you can't buffer these three packets the congestion window 
> can't grow so TCP can't do its job.

> Of course if you have three buffers and one, two or three of them are  
> filled and then a three packet burst comes in you're going to lose at  
> least one packet. So it seems beneficial to have more buffers. But that 
> doesn't help, TCP will just fill them up and create a standing queue. So 
> the chances of a lost packet remain the same.

> Now perhaps buffering only three packets is a bit too radical,  
> especially in the absense of simulations or real-world testing. But we  
> do have experience with 10 packet buffers, and those seem to work. So  
> I'd suggest "no more than 10" as our bottom line for now. And a good AQM 
> scheme is highly recommended to avoid the tail drops that happen all over 
> the place in everything I talked about so far.

I agree that TCP would suffer incredibly with very small buffers (less
than 3-4 packets), but the "no more than 10 packets" statement is not
reasonable in a longer term either. Nor do I think TCP being able to
function is good enough for high bandwidth uplinks. Such a limit will not
destroy TCP performance but will cause premature termination of the slow
start phase.  Assuming delayed acks and that the access uplink is the
bottleneck, the slow start phase looks like bursts that hit the bottleneck
gateway at 1.5 times available bandwith, meaning its buffer will grow at 0.5
times the link bandwidth, leading to termination of slow start after the
burst length exceeds 20 packets, possibly earlier depending on whether the
buffer has time to drain completely before the last burst starts.


	    | burst |
	RTT | len   | cwnd
	----+-------+------
	  1 |  2    |  2
	  2 |  3    |  3
	  3 |  3    |  4
	  4 |  6    |  6
	  5 |  9    |  9
	  6 | 12    | 13
	  7 | 18    | 19
	  8 | 27    | 28
	  9 | 42    | 42

Looking at this table of slow start induced burst lengths with delayed
acks (i just wrote it, may contain errors), the slow start would end after
8th RTT and result in a congestion window of 15-21 packets which would
then be incremented in congestion avoidance. Assuming an end-to-end RTT
of 100ms somewhat arbitrarily, this is about 2 megabits per second, so
anything significantly faster or further than 100ms away would be forced
to congestion avoidance roughly after the speed of 2 megabits per second.

Assuming quick acks, the bursts would hit the bottleneck at twice available
bandwidth, leading to even smaller CWND and earlier switch to congestion
avoidance.

I see the real difficulty in just picking one number as follows:

	* for a long RTT, striving for full utilization will conflict
	  needs of interactive users.

	* access uplink bandwidth varies, so a fixed number of packets
	  good for one case can be bad for others (0.3s for 5 Mbps is
	  3s for 512 kbps).

	* the (end-to-end!) RTT varies wildly and can be independent of
	  the access link RTT.

I personally can use interactive sessions (that is, ssh) comfortably
upto about 0.25s RTT, beyond which it starts to feel irritatingly
sluggish, and becomes too annoying to use after exceeding 1s. For me,
this means that if I'd tune the buffers for full utilization in the
Internet (say 0.3s end-to-end RTT), I would no longer be happy with
the responsiveness of my interactive sessions. For pathological cases,
things like line-mode could help, but not all programs can be used that
way, and I have yet to see an ssh-client that comes with line-mode
capability.

Consequently, I would pick something as large as possible while keeping
response times for interactive traffic bearable, which boils down to about
0.1s (for me). The reasoning is that if the ISP-side would also choose 0.1s
buffer, this would limit the access link buffer induced RTT increase to
0.2s when bulk transfers saturate both directions, which is still within
my arbitrarily defined pain bound of 0.25s.

Achieving this does require that the gateway adjust its buffer size based
on actual access uplink bandwidth, which is easy for ADSL-type links and
hard for shared medium. But I doubt anything less would be better than
the current randomly sized buffers which people work around by TCP window
limiting and additional policing boxes in front of the gateway.

To summarize, the buffer bound should be IMO set to bound the worst case
damage to response times instead of trying to maximize bulk transfer
throughput for any statistical Internet path, which for actual Internet
RTTs will inevitably lead to unacceptable worst case behavior. But (say)
0.1 seconds worth of buffering at actual access uplink rate seems much
more sensible to me than 10 packets, in terms of both, achieving high
utilization and good response time.

Best regards,

	Aki

From iljitsch@muada.com  Tue Nov 10 14:21:11 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 78F073A6B7C for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 14:21:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bWN8Cgfk3YQ2 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 14:21:10 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id AE8CA3A6975 for <homegate@ietf.org>; Tue, 10 Nov 2009 14:21:09 -0800 (PST)
Received: from [192.168.2.11] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAAMLPqX092674 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Tue, 10 Nov 2009 23:21:26 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <20091110203137.GA16898@wel-19.cs.helsinki.fi>
Date: Tue, 10 Nov 2009 23:21:28 +0100
Content-Transfer-Encoding: 7bit
Message-Id: <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi>
To: Aki M Nyrhinen <anyrhine@cs.helsinki.fi>
X-Mailer: Apple Mail (2.1076)
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2009 22:21:11 -0000

On 10 nov 2009, at 21:31, Aki M Nyrhinen wrote:

> I don't understand your numbers. I haven't really followed the  
> discussion
> well enough, but now feel the urge to provide some input anyway  
> (danger).

I have included the original message in which I arrive at this  
conclusion at the end of the message.

> If you aim at full utilization when RTT is 0.3 seconds, what exactly  
> is
> preventing TCP from achieving full utilization with 0.3 seconds  
> worth of
> buffering (that being linkbw * 0.3 in bits or bytes)?

Let's suppose 10 kiloBIT packets, 10 Mbps and a 0.1 RTT to make the  
calculations nice and simple. That would mean 1 megabit worth of  
buffering as per your suggestion, or 100 packets.

Without buffering TCP would sawtooth between 5 and 10 Mbps, or a  
congestion window of 50 to 100 packets for an average of 75 = 75%  
utilization. What we really want is to cycle between 67 and 133 for an  
average of 100 = 100% utilization.

So we add 100 packets (0.1 s) worth of buffering. Then, when the cwnd  
reaches 101, 100 packets are transmitted and one is buffered. The next  
RTT the cwnd is 102 packets so 100 are transmitted and 2 buffered, so  
the total number of packets in the buffer is 3. So:

roundtrip: queue size
1: 1
2: 3
3: 6
4: 10
5: 15
6: 21
7: 28
8: 36
9: 45
10: 55
11: 66
12: 78
13: 91
14: tail drop

So you don't reach the intended 133 cwnd, but only 114 so you cycle  
between 57 and 114, average 85 = 85% utilization.

So this whole notion of one RTT worth of buffering is an urban myth,  
certainly in the case of of a single TCP session, and I stronly  
suspect also with multiple sessions (I would expect the required  
buffering to go down in a logarithmic or square relationship to the  
number of sessions but haven't been able to figure out the math to  
show that.)

The problem here is that your buffer requirements go up with the  
square of the bandwidth but your transmission rate only increases  
linerarly with bandwith so now your RTT goes up with bandwidth and  
quickly gets out of hand as I showed in my previous message.

> I agree that TCP would suffer incredibly with very small buffers (less
> than 3-4 packets), but the "no more than 10 packets" statement is not
> reasonable in a longer term either.

Why not?

> Nor do I think TCP being able to
> function is good enough for high bandwidth uplinks. Such a limit  
> will not
> destroy TCP performance but will cause premature termination of the  
> slow
> start phase.

You assume that if you put X buffers in a gateway (or switch, but the  
switch people seem to think all of this doesn't apply to them) then a  
fresh TCP session gets to send an X packet burst. That's only true if  
the link was previously idle. I don't think we can assume that. If  
there are ongoing bulk data transfers you're lucky to find two empty  
buffers.

However, you're right that with minimal buffering slow start suffers  
and it takes a long time to get up to speed for long delay sessions.  
The question is where sweet spot is between quick ramping up of slow  
start sessions and the delays caused by standing queues with large  
amounts of buffering. I guess someone should run some experiments or  
simulations.

> I see the real difficulty in just picking one number as follows:

> 	* for a long RTT, striving for full utilization will conflict
> 	  needs of interactive users.

> 	* access uplink bandwidth varies, so a fixed number of packets
> 	  good for one case can be bad for others (0.3s for 5 Mbps is
> 	  3s for 512 kbps).

> 	* the (end-to-end!) RTT varies wildly and can be independent of
> 	  the access link RTT.

Right.

> Consequently, I would pick something as large as possible while  
> keeping
> response times for interactive traffic bearable, which boils down to  
> about
> 0.1s (for me). The reasoning is that if the ISP-side would also  
> choose 0.1s
> buffer, this would limit the access link buffer induced RTT increase  
> to
> 0.2s when bulk transfers saturate both directions, which is still  
> within
> my arbitrarily defined pain bound of 0.25s.

Note that at 128 kbps 0.1 s is only 1 packet...

> Achieving this does require that the gateway adjust its buffer size  
> based
> on actual access uplink bandwidth, which is easy for ADSL-type links  
> and
> hard for shared medium.

Well, originally the TTL field was supposed to count how long packets  
sat in buffers in routers. It wouldn't be inconceivable for a gateway  
to buffer packets and drop them after being buffered for 0.1 s.

> To summarize, the buffer bound should be IMO set to bound the worst  
> case
> damage to response times instead of trying to maximize bulk transfer
> throughput for any statistical Internet path, which for actual  
> Internet
> RTTs will inevitably lead to unacceptable worst case behavior. But  
> (say)
> 0.1 seconds worth of buffering at actual access uplink rate seems much
> more sensible to me than 10 packets, in terms of both, achieving high
> utilization and good response time.

Unfortunately, there doesn't seem to be a magic formula that provides  
an all around good buffer size. There's probably no way around active  
queue management to achieve good overall performance. Certainly if we  
want to address low delay applications such as VoIP.

Iljitsch


The original message:

On 1 aug 2009, at 14:23, Iljitsch van Beijnum wrote:

> One purpose of buffers is as follows. Consider a non-buffering router:

>  --------------
>     /| /| /| /|
>    / |/ |/ |/ |
>   /
>  /
>  --------------

> So with a single flow we cycle between 50% and 100% utilization for  
> an average of 75%. If we now add buffer space to accommodate the  
> missing 25% TCP can run 33% faster and reach full utilization. So  
> the buffering would need to be 33% of an RTT worth of data.

This is incorrect: the space between two teeth in the sawtooth isn't  
an RTT, but the time it takes to grow to the maximum window size from  
the "minimum" window size, so ~ 2/3 * link bandwidth / MSS * RTT.

So for instance, on a 1 Mbps link with a 300 ms RTT a sawtooth cycle  
takes 17 seconds. (The window for 1 Mbps = 85 packets, we cycle  
between 2/3 and 4/3 of that = 57 to 133, which takes 56 RTTs in  
congestion avoidance.)

Half that time we're building up a surplus, which needs to be large  
enough to fill in the deficit during the other half of the cycle. So  
that amount of data is half of a sawtooth wide and and varies between  
zero and a third of a link bandwidth high. So that's 2/3 * linkbw /  
MSS * RTT / 2 * linkbw / 6 = linkbw ^ 2 / MSS * RTT / 18 or approx  
linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the linkbw  
and MSS are in kilobits. For MSS = 12 and RTT = 0.3:

linkbw	buffer size  buffer time
64k	717          90 ms
128k	2867         179 ms
256k	11k          344 ms
1M	171k         1368 ms
10M	17M          13.6 s
100M	1669M        133 s
1G	163G         21.7 m
10G	16T          3.5 h

We already knew that 1 MSS per RTT window growth doesn't do long fat  
pipes any favors, but the above shows how bad it really is. The 1500- 
byte packet size doesn't help, either.

This means that the notion of enough buffering to fill in the valleys  
in the sawtooths is untenable. Then what? A minimum of 3 packets  
allows for a normal 2-packet "burst" after a delayed ACK and then a 1- 
packet window growth. Less than that and TCP will trip over itself  
whenever cwnd grows in congestion avoidance, making it impossible to  
reach a reasonable cwnd.

With multiple sessions at the same time, we would like to give all  
sessions 3 packets worth of buffer space. Simply increasing the buffer  
won't do that because a single session will use up the additional  
buffer space (until it runs out of various windows and buffers in the  
sender or receiver). So in a tail drop scenario there doesn't seem to  
be any advantage in using more than a 3-packet buffer...

From muraris@microsoft.com  Tue Nov 10 15:04:58 2009
Return-Path: <muraris@microsoft.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 092D828C0D0 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:04:58 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.974
X-Spam-Level: 
X-Spam-Status: No, score=-9.974 tagged_above=-999 required=5 tests=[AWL=0.025,  BAYES_00=-2.599, J_CHICKENPOX_22=0.6, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vV4UEhfqc+1i for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:04:56 -0800 (PST)
Received: from smtp.microsoft.com (mail3.microsoft.com [131.107.115.214]) by core3.amsl.com (Postfix) with ESMTP id B00F93A68DE for <homegate@ietf.org>; Tue, 10 Nov 2009 15:04:56 -0800 (PST)
Received: from TK5EX14MLTC101.redmond.corp.microsoft.com (157.54.79.178) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Tue, 10 Nov 2009 15:05:41 -0800
Received: from TK5EX14MBXC130.redmond.corp.microsoft.com ([169.254.9.154]) by TK5EX14MLTC101.redmond.corp.microsoft.com ([157.54.79.178]) with mapi; Tue, 10 Nov 2009 15:05:23 -0800
From: Murari Sridharan <muraris@microsoft.com>
To: Iljitsch van Beijnum <iljitsch@muada.com>, Aki M Nyrhinen <anyrhine@cs.helsinki.fi>
Thread-Topic: [homegate] Buffers in home gateways
Thread-Index: AQHKYXKlCKJs73gN+k6+K7Ps0dVxGJEwTNCAgAAesgD//4ScAA==
Date: Tue, 10 Nov 2009 23:05:21 +0000
Message-ID: <03520536067DF14395EA93FE7ECFDCF009731D12@TK5EX14MBXC130.redmond.corp.microsoft.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
In-Reply-To: <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2009 23:04:58 -0000

Just glanced through this but here are some very useful pointers and a few =
of my points.=20

1. Firstly buffer sizes can be considerably lower than the standard rule of=
 thumb whenever there is enough statmux. Statmux is really the key to lower=
ing buffering requirements by upto a factor of sq.rt of N, where N is the n=
umber of flows. At home that may not be the case. Some interesting reading =
here http://portal.acm.org/citation.cfm?id=3D1163593.1163605
2. Too small buffering can be a serious problem at the edges. Instead of tr=
ying to explain it, here is a quick link http://portal.acm.org/citation.cfm=
?id=3D1198262

We are doing some work on improving perfornmance over low buffered links an=
d will have something to share may be by the next IETF.=20

Thanks
Murari

-----Original Message-----
From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On Behal=
f Of Iljitsch van Beijnum
Sent: Tuesday, November 10, 2009 2:21 PM
To: Aki M Nyrhinen
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways

On 10 nov 2009, at 21:31, Aki M Nyrhinen wrote:

> I don't understand your numbers. I haven't really followed the =20
> discussion
> well enough, but now feel the urge to provide some input anyway =20
> (danger).

I have included the original message in which I arrive at this =20
conclusion at the end of the message.

> If you aim at full utilization when RTT is 0.3 seconds, what exactly =20
> is
> preventing TCP from achieving full utilization with 0.3 seconds =20
> worth of
> buffering (that being linkbw * 0.3 in bits or bytes)?

Let's suppose 10 kiloBIT packets, 10 Mbps and a 0.1 RTT to make the =20
calculations nice and simple. That would mean 1 megabit worth of =20
buffering as per your suggestion, or 100 packets.

Without buffering TCP would sawtooth between 5 and 10 Mbps, or a =20
congestion window of 50 to 100 packets for an average of 75 =3D 75% =20
utilization. What we really want is to cycle between 67 and 133 for an =20
average of 100 =3D 100% utilization.

So we add 100 packets (0.1 s) worth of buffering. Then, when the cwnd =20
reaches 101, 100 packets are transmitted and one is buffered. The next =20
RTT the cwnd is 102 packets so 100 are transmitted and 2 buffered, so =20
the total number of packets in the buffer is 3. So:

roundtrip: queue size
1: 1
2: 3
3: 6
4: 10
5: 15
6: 21
7: 28
8: 36
9: 45
10: 55
11: 66
12: 78
13: 91
14: tail drop

So you don't reach the intended 133 cwnd, but only 114 so you cycle =20
between 57 and 114, average 85 =3D 85% utilization.

So this whole notion of one RTT worth of buffering is an urban myth, =20
certainly in the case of of a single TCP session, and I stronly =20
suspect also with multiple sessions (I would expect the required =20
buffering to go down in a logarithmic or square relationship to the =20
number of sessions but haven't been able to figure out the math to =20
show that.)

The problem here is that your buffer requirements go up with the =20
square of the bandwidth but your transmission rate only increases =20
linerarly with bandwith so now your RTT goes up with bandwidth and =20
quickly gets out of hand as I showed in my previous message.

> I agree that TCP would suffer incredibly with very small buffers (less
> than 3-4 packets), but the "no more than 10 packets" statement is not
> reasonable in a longer term either.

Why not?

> Nor do I think TCP being able to
> function is good enough for high bandwidth uplinks. Such a limit =20
> will not
> destroy TCP performance but will cause premature termination of the =20
> slow
> start phase.

You assume that if you put X buffers in a gateway (or switch, but the =20
switch people seem to think all of this doesn't apply to them) then a =20
fresh TCP session gets to send an X packet burst. That's only true if =20
the link was previously idle. I don't think we can assume that. If =20
there are ongoing bulk data transfers you're lucky to find two empty =20
buffers.

However, you're right that with minimal buffering slow start suffers =20
and it takes a long time to get up to speed for long delay sessions. =20
The question is where sweet spot is between quick ramping up of slow =20
start sessions and the delays caused by standing queues with large =20
amounts of buffering. I guess someone should run some experiments or =20
simulations.

> I see the real difficulty in just picking one number as follows:

> 	* for a long RTT, striving for full utilization will conflict
> 	  needs of interactive users.

> 	* access uplink bandwidth varies, so a fixed number of packets
> 	  good for one case can be bad for others (0.3s for 5 Mbps is
> 	  3s for 512 kbps).

> 	* the (end-to-end!) RTT varies wildly and can be independent of
> 	  the access link RTT.

Right.

> Consequently, I would pick something as large as possible while =20
> keeping
> response times for interactive traffic bearable, which boils down to =20
> about
> 0.1s (for me). The reasoning is that if the ISP-side would also =20
> choose 0.1s
> buffer, this would limit the access link buffer induced RTT increase =20
> to
> 0.2s when bulk transfers saturate both directions, which is still =20
> within
> my arbitrarily defined pain bound of 0.25s.

Note that at 128 kbps 0.1 s is only 1 packet...

> Achieving this does require that the gateway adjust its buffer size =20
> based
> on actual access uplink bandwidth, which is easy for ADSL-type links =20
> and
> hard for shared medium.

Well, originally the TTL field was supposed to count how long packets =20
sat in buffers in routers. It wouldn't be inconceivable for a gateway =20
to buffer packets and drop them after being buffered for 0.1 s.

> To summarize, the buffer bound should be IMO set to bound the worst =20
> case
> damage to response times instead of trying to maximize bulk transfer
> throughput for any statistical Internet path, which for actual =20
> Internet
> RTTs will inevitably lead to unacceptable worst case behavior. But =20
> (say)
> 0.1 seconds worth of buffering at actual access uplink rate seems much
> more sensible to me than 10 packets, in terms of both, achieving high
> utilization and good response time.

Unfortunately, there doesn't seem to be a magic formula that provides =20
an all around good buffer size. There's probably no way around active =20
queue management to achieve good overall performance. Certainly if we =20
want to address low delay applications such as VoIP.

Iljitsch


The original message:

On 1 aug 2009, at 14:23, Iljitsch van Beijnum wrote:

> One purpose of buffers is as follows. Consider a non-buffering router:

>  --------------
>     /| /| /| /|
>    / |/ |/ |/ |
>   /
>  /
>  --------------

> So with a single flow we cycle between 50% and 100% utilization for =20
> an average of 75%. If we now add buffer space to accommodate the =20
> missing 25% TCP can run 33% faster and reach full utilization. So =20
> the buffering would need to be 33% of an RTT worth of data.

This is incorrect: the space between two teeth in the sawtooth isn't =20
an RTT, but the time it takes to grow to the maximum window size from =20
the "minimum" window size, so ~ 2/3 * link bandwidth / MSS * RTT.

So for instance, on a 1 Mbps link with a 300 ms RTT a sawtooth cycle =20
takes 17 seconds. (The window for 1 Mbps =3D 85 packets, we cycle =20
between 2/3 and 4/3 of that =3D 57 to 133, which takes 56 RTTs in =20
congestion avoidance.)

Half that time we're building up a surplus, which needs to be large =20
enough to fill in the deficit during the other half of the cycle. So =20
that amount of data is half of a sawtooth wide and and varies between =20
zero and a third of a link bandwidth high. So that's 2/3 * linkbw / =20
MSS * RTT / 2 * linkbw / 6 =3D linkbw ^ 2 / MSS * RTT / 18 or approx =20
linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the linkbw =20
and MSS are in kilobits. For MSS =3D 12 and RTT =3D 0.3:

linkbw	buffer size  buffer time
64k	717          90 ms
128k	2867         179 ms
256k	11k          344 ms
1M	171k         1368 ms
10M	17M          13.6 s
100M	1669M        133 s
1G	163G         21.7 m
10G	16T          3.5 h

We already knew that 1 MSS per RTT window growth doesn't do long fat =20
pipes any favors, but the above shows how bad it really is. The 1500-=20
byte packet size doesn't help, either.

This means that the notion of enough buffering to fill in the valleys =20
in the sawtooths is untenable. Then what? A minimum of 3 packets =20
allows for a normal 2-packet "burst" after a delayed ACK and then a 1-=20
packet window growth. Less than that and TCP will trip over itself =20
whenever cwnd grows in congestion avoidance, making it impossible to =20
reach a reasonable cwnd.

With multiple sessions at the same time, we would like to give all =20
sessions 3 packets worth of buffer space. Simply increasing the buffer =20
won't do that because a single session will use up the additional =20
buffer space (until it runs out of various windows and buffers in the =20
sender or receiver). So in a tail drop scenario there doesn't seem to =20
be any advantage in using more than a 3-packet buffer...
_______________________________________________
homegate mailing list
homegate@ietf.org
https://www.ietf.org/mailman/listinfo/homegate


From richard_woundy@cable.comcast.com  Tue Nov 10 15:07:08 2009
Return-Path: <richard_woundy@cable.comcast.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 521A928C0DB for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:07:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.981
X-Spam-Level: 
X-Spam-Status: No, score=-1.981 tagged_above=-999 required=5 tests=[AWL=-1.518, BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hh2WFyx4eyQU for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:07:06 -0800 (PST)
Received: from paoakoavas09.cable.comcast.com (paoakoavas09.cable.comcast.com [208.17.35.58]) by core3.amsl.com (Postfix) with ESMTP id 7E9FA3A6960 for <homegate@ietf.org>; Tue, 10 Nov 2009 15:07:06 -0800 (PST)
Received: from ([24.40.15.118]) by paoakoavas09.cable.comcast.com with ESMTP  id KP-NTF18.82227807; Tue, 10 Nov 2009 18:07:20 -0500
Received: from PACDCEXCMB06.cable.comcast.com ([24.40.15.22]) by PACDCEXCSMTP04.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.3959); Tue, 10 Nov 2009 18:07:22 -0500
x-mimeole: Produced By Microsoft Exchange V6.5
x-cr-puzzleid: {325023FF-01B2-4B2D-A8E0-E9B2F565229A}
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
x-cr-hashedpuzzle: CEZl Ek8L Hp3q LjI1 M6FN O8Jm RDjY SeDR Xjy4 XzFK bDnI cRwL cqjI eDxZ g3Aq iNlL; 3; YQBuAHkAcgBoAGkAbgBlAEAAYwBzAC4AaABlAGwAcwBpAG4AawBpAC4AZgBpADsAaABvAG0AZQBnAGEAdABlAEAAaQBlAHQAZgAuAG8AcgBnADsAaQBsAGoAaQB0AHMAYwBoAEAAbQB1AGEAZABhAC4AYwBvAG0A; Sosha1_v1; 7; {325023FF-01B2-4B2D-A8E0-E9B2F565229A}; cgBpAGMAaABhAHIAZABfAHcAbwB1AG4AZAB5AEAAYwBhAGIAbABlAC4AYwBvAG0AYwBhAHMAdAAuAGMAbwBtAA==; Tue, 10 Nov 2009 23:07:12 GMT; UgBFADoAIABbAGgAbwBtAGUAZwBhAHQAZQBdACAAQgB1AGYAZgBlAHIAcwAgAGkAbgAgAGgAbwBtAGUAIABnAGEAdABlAHcAYQB5AHMA
Content-class: urn:content-classes:message
Date: Tue, 10 Nov 2009 18:07:12 -0500
Message-ID: <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com>
In-Reply-To: <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: [homegate] Buffers in home gateways
Thread-Index: AcpiVDLxqnXPHlFDRle9CNMQsbvKKgAA1WgQ
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com><20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
From: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>
To: "Iljitsch van Beijnum" <iljitsch@muada.com>, "Aki M Nyrhinen" <anyrhine@cs.helsinki.fi>
X-OriginalArrivalTime: 10 Nov 2009 23:07:22.0040 (UTC) FILETIME=[8F124B80:01CA625A]
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2009 23:07:08 -0000

Iljitsch,

> So this whole notion of one RTT worth of buffering is an urban myth,
certainly in the case of of a single TCP session

I would consider reading the following Sigcomm 2004 paper, "Sizing
Router Buffers" by Appenzeller, Keslassy, and McKeown,
http://yuba.stanford.edu/~nickm/papers/sigcomm2004.pdf, especially
sections 1.2, 2 and 3.1.

Section 2 shows the mathematics behind why a router (with a bottleneck
link) handling a single long-lived TCP flow ought to have a buffer size
equal to the bandwidth delay product.

Section 3.1 indicates that a relatively small number of long-lived TCP
flows (such as at a home router) will tend to become synchronized, and
therefore the buffer size should still be equal to the bandwidth delay
product.

Section 3.2 and beyond indicate that long lived TCP flows tend to become
de-synchronized at routers in backbones and other places of aggregated
traffic, and suggests that such routers could use much less than the
bandwidth delay product for buffering.

It may take me a while to figure out why your calculations and
conclusions are so different. But I am pretty sure that your
calculations for achieving near 100% utilization for a single TCP flow
at 10 Mbps using 17 MB buffers seem really high.

-- Rich

-----Original Message-----
From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On
Behalf Of Iljitsch van Beijnum
Sent: Wednesday, November 11, 2009 7:21 AM
To: Aki M Nyrhinen
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways

On 10 nov 2009, at 21:31, Aki M Nyrhinen wrote:

> I don't understand your numbers. I haven't really followed the =20
> discussion
> well enough, but now feel the urge to provide some input anyway =20
> (danger).

I have included the original message in which I arrive at this =20
conclusion at the end of the message.

> If you aim at full utilization when RTT is 0.3 seconds, what exactly =20
> is
> preventing TCP from achieving full utilization with 0.3 seconds =20
> worth of
> buffering (that being linkbw * 0.3 in bits or bytes)?

Let's suppose 10 kiloBIT packets, 10 Mbps and a 0.1 RTT to make the =20
calculations nice and simple. That would mean 1 megabit worth of =20
buffering as per your suggestion, or 100 packets.

Without buffering TCP would sawtooth between 5 and 10 Mbps, or a =20
congestion window of 50 to 100 packets for an average of 75 =3D 75% =20
utilization. What we really want is to cycle between 67 and 133 for an =20
average of 100 =3D 100% utilization.

So we add 100 packets (0.1 s) worth of buffering. Then, when the cwnd =20
reaches 101, 100 packets are transmitted and one is buffered. The next =20
RTT the cwnd is 102 packets so 100 are transmitted and 2 buffered, so =20
the total number of packets in the buffer is 3. So:

roundtrip: queue size
1: 1
2: 3
3: 6
4: 10
5: 15
6: 21
7: 28
8: 36
9: 45
10: 55
11: 66
12: 78
13: 91
14: tail drop

So you don't reach the intended 133 cwnd, but only 114 so you cycle =20
between 57 and 114, average 85 =3D 85% utilization.

So this whole notion of one RTT worth of buffering is an urban myth, =20
certainly in the case of of a single TCP session, and I stronly =20
suspect also with multiple sessions (I would expect the required =20
buffering to go down in a logarithmic or square relationship to the =20
number of sessions but haven't been able to figure out the math to =20
show that.)

The problem here is that your buffer requirements go up with the =20
square of the bandwidth but your transmission rate only increases =20
linerarly with bandwith so now your RTT goes up with bandwidth and =20
quickly gets out of hand as I showed in my previous message.

> I agree that TCP would suffer incredibly with very small buffers (less
> than 3-4 packets), but the "no more than 10 packets" statement is not
> reasonable in a longer term either.

Why not?

> Nor do I think TCP being able to
> function is good enough for high bandwidth uplinks. Such a limit =20
> will not
> destroy TCP performance but will cause premature termination of the =20
> slow
> start phase.

You assume that if you put X buffers in a gateway (or switch, but the =20
switch people seem to think all of this doesn't apply to them) then a =20
fresh TCP session gets to send an X packet burst. That's only true if =20
the link was previously idle. I don't think we can assume that. If =20
there are ongoing bulk data transfers you're lucky to find two empty =20
buffers.

However, you're right that with minimal buffering slow start suffers =20
and it takes a long time to get up to speed for long delay sessions. =20
The question is where sweet spot is between quick ramping up of slow =20
start sessions and the delays caused by standing queues with large =20
amounts of buffering. I guess someone should run some experiments or =20
simulations.

> I see the real difficulty in just picking one number as follows:

> 	* for a long RTT, striving for full utilization will conflict
> 	  needs of interactive users.

> 	* access uplink bandwidth varies, so a fixed number of packets
> 	  good for one case can be bad for others (0.3s for 5 Mbps is
> 	  3s for 512 kbps).

> 	* the (end-to-end!) RTT varies wildly and can be independent of
> 	  the access link RTT.

Right.

> Consequently, I would pick something as large as possible while =20
> keeping
> response times for interactive traffic bearable, which boils down to =20
> about
> 0.1s (for me). The reasoning is that if the ISP-side would also =20
> choose 0.1s
> buffer, this would limit the access link buffer induced RTT increase =20
> to
> 0.2s when bulk transfers saturate both directions, which is still =20
> within
> my arbitrarily defined pain bound of 0.25s.

Note that at 128 kbps 0.1 s is only 1 packet...

> Achieving this does require that the gateway adjust its buffer size =20
> based
> on actual access uplink bandwidth, which is easy for ADSL-type links =20
> and
> hard for shared medium.

Well, originally the TTL field was supposed to count how long packets =20
sat in buffers in routers. It wouldn't be inconceivable for a gateway =20
to buffer packets and drop them after being buffered for 0.1 s.

> To summarize, the buffer bound should be IMO set to bound the worst =20
> case
> damage to response times instead of trying to maximize bulk transfer
> throughput for any statistical Internet path, which for actual =20
> Internet
> RTTs will inevitably lead to unacceptable worst case behavior. But =20
> (say)
> 0.1 seconds worth of buffering at actual access uplink rate seems much
> more sensible to me than 10 packets, in terms of both, achieving high
> utilization and good response time.

Unfortunately, there doesn't seem to be a magic formula that provides =20
an all around good buffer size. There's probably no way around active =20
queue management to achieve good overall performance. Certainly if we =20
want to address low delay applications such as VoIP.

Iljitsch


The original message:

On 1 aug 2009, at 14:23, Iljitsch van Beijnum wrote:

> One purpose of buffers is as follows. Consider a non-buffering router:

>  --------------
>     /| /| /| /|
>    / |/ |/ |/ |
>   /
>  /
>  --------------

> So with a single flow we cycle between 50% and 100% utilization for =20
> an average of 75%. If we now add buffer space to accommodate the =20
> missing 25% TCP can run 33% faster and reach full utilization. So =20
> the buffering would need to be 33% of an RTT worth of data.

This is incorrect: the space between two teeth in the sawtooth isn't =20
an RTT, but the time it takes to grow to the maximum window size from =20
the "minimum" window size, so ~ 2/3 * link bandwidth / MSS * RTT.

So for instance, on a 1 Mbps link with a 300 ms RTT a sawtooth cycle =20
takes 17 seconds. (The window for 1 Mbps =3D 85 packets, we cycle =20
between 2/3 and 4/3 of that =3D 57 to 133, which takes 56 RTTs in =20
congestion avoidance.)

Half that time we're building up a surplus, which needs to be large =20
enough to fill in the deficit during the other half of the cycle. So =20
that amount of data is half of a sawtooth wide and and varies between =20
zero and a third of a link bandwidth high. So that's 2/3 * linkbw / =20
MSS * RTT / 2 * linkbw / 6 =3D linkbw ^ 2 / MSS * RTT / 18 or approx =20
linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the linkbw =20
and MSS are in kilobits. For MSS =3D 12 and RTT =3D 0.3:

linkbw	buffer size  buffer time
64k	717          90 ms
128k	2867         179 ms
256k	11k          344 ms
1M	171k         1368 ms
10M	17M          13.6 s
100M	1669M        133 s
1G	163G         21.7 m
10G	16T          3.5 h

We already knew that 1 MSS per RTT window growth doesn't do long fat =20
pipes any favors, but the above shows how bad it really is. The 1500-=20
byte packet size doesn't help, either.

This means that the notion of enough buffering to fill in the valleys =20
in the sawtooths is untenable. Then what? A minimum of 3 packets =20
allows for a normal 2-packet "burst" after a delayed ACK and then a 1-=20
packet window growth. Less than that and TCP will trip over itself =20
whenever cwnd grows in congestion avoidance, making it impossible to =20
reach a reasonable cwnd.

With multiple sessions at the same time, we would like to give all =20
sessions 3 packets worth of buffer space. Simply increasing the buffer =20
won't do that because a single session will use up the additional =20
buffer space (until it runs out of various windows and buffers in the =20
sender or receiver). So in a tail drop scenario there doesn't seem to =20
be any advantage in using more than a 3-packet buffer...
_______________________________________________
homegate mailing list
homegate@ietf.org
https://www.ietf.org/mailman/listinfo/homegate

From muraris@microsoft.com  Tue Nov 10 15:10:11 2009
Return-Path: <muraris@microsoft.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EE7C928C0DD for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:10:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.279
X-Spam-Level: 
X-Spam-Status: No, score=-10.279 tagged_above=-999 required=5 tests=[AWL=0.320, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8qNOku-vjZ8M for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 15:10:08 -0800 (PST)
Received: from smtp.microsoft.com (mailc.microsoft.com [131.107.115.214]) by core3.amsl.com (Postfix) with ESMTP id 79A7228C0E6 for <homegate@ietf.org>; Tue, 10 Nov 2009 15:10:08 -0800 (PST)
Received: from TK5EX14MLTC101.redmond.corp.microsoft.com (157.54.79.178) by TK5-EXGWY-E803.partners.extranet.microsoft.com (10.251.56.169) with Microsoft SMTP Server (TLS) id 8.2.176.0; Tue, 10 Nov 2009 15:10:53 -0800
Received: from TK5EX14MBXC130.redmond.corp.microsoft.com ([169.254.9.154]) by TK5EX14MLTC101.redmond.corp.microsoft.com ([157.54.79.178]) with mapi; Tue, 10 Nov 2009 15:10:27 -0800
From: Murari Sridharan <muraris@microsoft.com>
To: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>, Iljitsch van Beijnum <iljitsch@muada.com>, Aki M Nyrhinen <anyrhine@cs.helsinki.fi>
Thread-Topic: [homegate] Buffers in home gateways
Thread-Index: AQHKYXKlCKJs73gN+k6+K7Ps0dVxGJEwTNCAgAAesgCAAAzHAP//ei7g
Date: Tue, 10 Nov 2009 23:10:25 +0000
Message-ID: <03520536067DF14395EA93FE7ECFDCF009731D4E@TK5EX14MBXC130.redmond.corp.microsoft.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com><20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com>
In-Reply-To: <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Nov 2009 23:10:11 -0000

Another note of caution. In my experience the throughput is typically below=
 the theoretical amount due to quirks in implementation and system issues s=
o at 10G speeds we recently tried out a zero buffer itest, you don't automa=
tically get 75% with new reno its in fact lower than that. Again too much b=
uffering is a problem but there are ways to address that, but too shallow a=
 buffer can seriously hamper throughput. As I said we are hoping to share o=
ur results and shed some light on this problem in general prior to the next=
 IETF. =20

-----Original Message-----
From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On Behal=
f Of Woundy, Richard
Sent: Tuesday, November 10, 2009 3:07 PM
To: Iljitsch van Beijnum; Aki M Nyrhinen
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways

Iljitsch,

> So this whole notion of one RTT worth of buffering is an urban myth,
certainly in the case of of a single TCP session

I would consider reading the following Sigcomm 2004 paper, "Sizing
Router Buffers" by Appenzeller, Keslassy, and McKeown,
http://yuba.stanford.edu/~nickm/papers/sigcomm2004.pdf, especially
sections 1.2, 2 and 3.1.

Section 2 shows the mathematics behind why a router (with a bottleneck
link) handling a single long-lived TCP flow ought to have a buffer size
equal to the bandwidth delay product.

Section 3.1 indicates that a relatively small number of long-lived TCP
flows (such as at a home router) will tend to become synchronized, and
therefore the buffer size should still be equal to the bandwidth delay
product.

Section 3.2 and beyond indicate that long lived TCP flows tend to become
de-synchronized at routers in backbones and other places of aggregated
traffic, and suggests that such routers could use much less than the
bandwidth delay product for buffering.

It may take me a while to figure out why your calculations and
conclusions are so different. But I am pretty sure that your
calculations for achieving near 100% utilization for a single TCP flow
at 10 Mbps using 17 MB buffers seem really high.

-- Rich

-----Original Message-----
From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On
Behalf Of Iljitsch van Beijnum
Sent: Wednesday, November 11, 2009 7:21 AM
To: Aki M Nyrhinen
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways

On 10 nov 2009, at 21:31, Aki M Nyrhinen wrote:

> I don't understand your numbers. I haven't really followed the =20
> discussion
> well enough, but now feel the urge to provide some input anyway =20
> (danger).

I have included the original message in which I arrive at this =20
conclusion at the end of the message.

> If you aim at full utilization when RTT is 0.3 seconds, what exactly =20
> is
> preventing TCP from achieving full utilization with 0.3 seconds =20
> worth of
> buffering (that being linkbw * 0.3 in bits or bytes)?

Let's suppose 10 kiloBIT packets, 10 Mbps and a 0.1 RTT to make the =20
calculations nice and simple. That would mean 1 megabit worth of =20
buffering as per your suggestion, or 100 packets.

Without buffering TCP would sawtooth between 5 and 10 Mbps, or a =20
congestion window of 50 to 100 packets for an average of 75 =3D 75% =20
utilization. What we really want is to cycle between 67 and 133 for an =20
average of 100 =3D 100% utilization.

So we add 100 packets (0.1 s) worth of buffering. Then, when the cwnd =20
reaches 101, 100 packets are transmitted and one is buffered. The next =20
RTT the cwnd is 102 packets so 100 are transmitted and 2 buffered, so =20
the total number of packets in the buffer is 3. So:

roundtrip: queue size
1: 1
2: 3
3: 6
4: 10
5: 15
6: 21
7: 28
8: 36
9: 45
10: 55
11: 66
12: 78
13: 91
14: tail drop

So you don't reach the intended 133 cwnd, but only 114 so you cycle =20
between 57 and 114, average 85 =3D 85% utilization.

So this whole notion of one RTT worth of buffering is an urban myth, =20
certainly in the case of of a single TCP session, and I stronly =20
suspect also with multiple sessions (I would expect the required =20
buffering to go down in a logarithmic or square relationship to the =20
number of sessions but haven't been able to figure out the math to =20
show that.)

The problem here is that your buffer requirements go up with the =20
square of the bandwidth but your transmission rate only increases =20
linerarly with bandwith so now your RTT goes up with bandwidth and =20
quickly gets out of hand as I showed in my previous message.

> I agree that TCP would suffer incredibly with very small buffers (less
> than 3-4 packets), but the "no more than 10 packets" statement is not
> reasonable in a longer term either.

Why not?

> Nor do I think TCP being able to
> function is good enough for high bandwidth uplinks. Such a limit =20
> will not
> destroy TCP performance but will cause premature termination of the =20
> slow
> start phase.

You assume that if you put X buffers in a gateway (or switch, but the =20
switch people seem to think all of this doesn't apply to them) then a =20
fresh TCP session gets to send an X packet burst. That's only true if =20
the link was previously idle. I don't think we can assume that. If =20
there are ongoing bulk data transfers you're lucky to find two empty =20
buffers.

However, you're right that with minimal buffering slow start suffers =20
and it takes a long time to get up to speed for long delay sessions. =20
The question is where sweet spot is between quick ramping up of slow =20
start sessions and the delays caused by standing queues with large =20
amounts of buffering. I guess someone should run some experiments or =20
simulations.

> I see the real difficulty in just picking one number as follows:

> 	* for a long RTT, striving for full utilization will conflict
> 	  needs of interactive users.

> 	* access uplink bandwidth varies, so a fixed number of packets
> 	  good for one case can be bad for others (0.3s for 5 Mbps is
> 	  3s for 512 kbps).

> 	* the (end-to-end!) RTT varies wildly and can be independent of
> 	  the access link RTT.

Right.

> Consequently, I would pick something as large as possible while =20
> keeping
> response times for interactive traffic bearable, which boils down to =20
> about
> 0.1s (for me). The reasoning is that if the ISP-side would also =20
> choose 0.1s
> buffer, this would limit the access link buffer induced RTT increase =20
> to
> 0.2s when bulk transfers saturate both directions, which is still =20
> within
> my arbitrarily defined pain bound of 0.25s.

Note that at 128 kbps 0.1 s is only 1 packet...

> Achieving this does require that the gateway adjust its buffer size =20
> based
> on actual access uplink bandwidth, which is easy for ADSL-type links =20
> and
> hard for shared medium.

Well, originally the TTL field was supposed to count how long packets =20
sat in buffers in routers. It wouldn't be inconceivable for a gateway =20
to buffer packets and drop them after being buffered for 0.1 s.

> To summarize, the buffer bound should be IMO set to bound the worst =20
> case
> damage to response times instead of trying to maximize bulk transfer
> throughput for any statistical Internet path, which for actual =20
> Internet
> RTTs will inevitably lead to unacceptable worst case behavior. But =20
> (say)
> 0.1 seconds worth of buffering at actual access uplink rate seems much
> more sensible to me than 10 packets, in terms of both, achieving high
> utilization and good response time.

Unfortunately, there doesn't seem to be a magic formula that provides =20
an all around good buffer size. There's probably no way around active =20
queue management to achieve good overall performance. Certainly if we =20
want to address low delay applications such as VoIP.

Iljitsch


The original message:

On 1 aug 2009, at 14:23, Iljitsch van Beijnum wrote:

> One purpose of buffers is as follows. Consider a non-buffering router:

>  --------------
>     /| /| /| /|
>    / |/ |/ |/ |
>   /
>  /
>  --------------

> So with a single flow we cycle between 50% and 100% utilization for =20
> an average of 75%. If we now add buffer space to accommodate the =20
> missing 25% TCP can run 33% faster and reach full utilization. So =20
> the buffering would need to be 33% of an RTT worth of data.

This is incorrect: the space between two teeth in the sawtooth isn't =20
an RTT, but the time it takes to grow to the maximum window size from =20
the "minimum" window size, so ~ 2/3 * link bandwidth / MSS * RTT.

So for instance, on a 1 Mbps link with a 300 ms RTT a sawtooth cycle =20
takes 17 seconds. (The window for 1 Mbps =3D 85 packets, we cycle =20
between 2/3 and 4/3 of that =3D 57 to 133, which takes 56 RTTs in =20
congestion avoidance.)

Half that time we're building up a surplus, which needs to be large =20
enough to fill in the deficit during the other half of the cycle. So =20
that amount of data is half of a sawtooth wide and and varies between =20
zero and a third of a link bandwidth high. So that's 2/3 * linkbw / =20
MSS * RTT / 2 * linkbw / 6 =3D linkbw ^ 2 / MSS * RTT / 18 or approx =20
linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the linkbw =20
and MSS are in kilobits. For MSS =3D 12 and RTT =3D 0.3:

linkbw	buffer size  buffer time
64k	717          90 ms
128k	2867         179 ms
256k	11k          344 ms
1M	171k         1368 ms
10M	17M          13.6 s
100M	1669M        133 s
1G	163G         21.7 m
10G	16T          3.5 h

We already knew that 1 MSS per RTT window growth doesn't do long fat =20
pipes any favors, but the above shows how bad it really is. The 1500-=20
byte packet size doesn't help, either.

This means that the notion of enough buffering to fill in the valleys =20
in the sawtooths is untenable. Then what? A minimum of 3 packets =20
allows for a normal 2-packet "burst" after a delayed ACK and then a 1-=20
packet window growth. Less than that and TCP will trip over itself =20
whenever cwnd grows in congestion avoidance, making it impossible to =20
reach a reasonable cwnd.

With multiple sessions at the same time, we would like to give all =20
sessions 3 packets worth of buffer space. Simply increasing the buffer =20
won't do that because a single session will use up the additional =20
buffer space (until it runs out of various windows and buffers in the =20
sender or receiver). So in a tail drop scenario there doesn't seem to =20
be any advantage in using more than a 3-packet buffer...
_______________________________________________
homegate mailing list
homegate@ietf.org
https://www.ietf.org/mailman/listinfo/homegate
_______________________________________________
homegate mailing list
homegate@ietf.org
https://www.ietf.org/mailman/listinfo/homegate


From kojo@cs.helsinki.fi  Tue Nov 10 22:57:57 2009
Return-Path: <kojo@cs.helsinki.fi>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id BB5393A6C10 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 22:57:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Level: 
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hoV0wmp8jaCq for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 22:57:56 -0800 (PST)
Received: from mail.cs.helsinki.fi (courier.cs.helsinki.fi [128.214.9.1]) by core3.amsl.com (Postfix) with ESMTP id B267F3A6C12 for <homegate@ietf.org>; Tue, 10 Nov 2009 22:57:53 -0800 (PST)
Received: from tx3-4.cs.helsinki.fi (host-32-140.meeting.ietf.org [133.93.32.140]) (AUTH: PLAIN cs-relay, TLS: TLSv1/SSLv3,256bits,AES256-SHA) by mail.cs.helsinki.fi with esmtp; Wed, 11 Nov 2009 08:58:18 +0200 id 00063EB0.4AFA608B.00003687
Received: by tx3-4.cs.helsinki.fi (Postfix, from userid 3011) id 8061724A23; Wed, 11 Nov 2009 08:58:14 +0200 (EET)
Received: from localhost (localhost [127.0.0.1]) by tx3-4.cs.helsinki.fi (Postfix) with ESMTP id 730E024A20; Wed, 11 Nov 2009 08:58:14 +0200 (EET)
Date: Wed, 11 Nov 2009 08:58:14 +0200 (EET)
From: Markku Kojo <kojo@cs.helsinki.fi>
To: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
Message-ID: <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 06:57:57 -0000

Hi Iljitsch,

On Tue, 10 Nov 2009, Iljitsch van Beijnum wrote:

> On 10 nov 2009, at 21:31, Aki M Nyrhinen wrote:
> 
> >I don't understand your numbers. I haven't really followed the discussion
> >well enough, but now feel the urge to provide some input anyway (danger).
> 
> I have included the original message in which I arrive at this conclusion at
> the end of the message.
> 
> >If you aim at full utilization when RTT is 0.3 seconds, what exactly is
> >preventing TCP from achieving full utilization with 0.3 seconds worth of
> >buffering (that being linkbw * 0.3 in bits or bytes)?
> 
> Let's suppose 10 kiloBIT packets, 10 Mbps and a 0.1 RTT to make the
> calculations nice and simple. That would mean 1 megabit worth of buffering as
> per your suggestion, or 100 packets.
> 
> Without buffering TCP would sawtooth between 5 and 10 Mbps, or a congestion
> window of 50 to 100 packets for an average of 75 = 75% utilization. What we
> really want is to cycle between 67 and 133 for an average of 100 = 100%
> utilization.

The problem with your 100% avg utilization logic here is that you 
incorrectly assume that by adding buffering you get  > 100% link 
utilization when pkts start accumulating in the buffer!?

When cwnd becomes  > 100 pkts in this example (assuming zero delay for  
the rest of the path), the pkts get buffered at the bottleneck link and 
the link is fully utilized but it still delivers pkts only at max 
link rate!
That is, at times when your cwnd is < 100 pkts your link is underutilized,
and if your link is ever underutilized during your connection (or
observation time period) you can never reach 100 % avg utilization.

After inflating cwnd beyond buf size + link BDP (bandwidth-delay product), 
a pkt drop occurs and cwnd is halved (in CA). Therefore, the cwnd size 
must be at least 2*BPD at the time (before) cwnd gets halved in order to 
allow a TCP sender to continue 100% utilization of the bottleneck link as 
Aki pointed out. But this holds only when the rest of the path delay is 
zero, which is not the case in the real world. In effect, the rest of the 
path delay varies and makes it hard to determine the proper buffer size 
as Aki also pointed out. Longer the rest of the path delay is, less 
buffering is needed to keep up 100% bottleneck link utilization all the 
time. In the example you had,  33 pkt buffer size would be good enough if 
the rest of the path delay is long enough to accomodate buffering 
capacity for 67 pkts. Then the flightsize and cwnd may reach 200 pkts 
before a drop occurs and allow to continue with cwnd of 100 pkts.

Furtehrmore, there is a bunch of other dynamics that affect determining a 
proper buffer size, including multiple flows (statistical multiplexing).
And this makes it even harder. 

Cheers,

/Markku  

From iljitsch@muada.com  Tue Nov 10 23:19:08 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 85FFE3A67B0 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 23:19:08 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nXbxXED3Xs5B for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 23:19:07 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 4C3B83A69D5 for <homegate@ietf.org>; Tue, 10 Nov 2009 23:19:04 -0800 (PST)
Received: from [192.168.2.11] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAB7JKLG097010 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 11 Nov 2009 08:19:21 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <03520536067DF14395EA93FE7ECFDCF009731D12@TK5EX14MBXC130.redmond.corp.microsoft.com>
Date: Wed, 11 Nov 2009 08:19:23 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <A0503305-D6FE-4871-A710-DD4EDE58DF5A@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <03520536067DF14395EA93FE7ECFDCF009731D12@TK5EX14MBXC130.redmond.corp.microsoft.com>
To: Murari Sridharan <muraris@microsoft.com>
X-Mailer: Apple Mail (2.1077)
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 07:19:08 -0000

On 11Nov, 2009, at 0:05 , Murari Sridharan wrote:

> 1. Firstly buffer sizes can be considerably lower than the standard =
rule of thumb

Looks like this rule of thumb was invented in this paper:

High performance tcp in ansnet

http://www.scs.stanford.edu/nyu/04sp/sched/ans-tcp.pdf

However, they provide no justification at all!

> http://portal.acm.org/citation.cfm?id=3D1163593.1163605

This only talks about the core of the network where the number of =
sessions is very large.

> 2. Too small buffering can be a serious problem at the edges. Instead =
of trying to explain it, here is a quick link =
http://portal.acm.org/citation.cfm?id=3D1198262

They tested with 2 buffers... We already established you need at least =
3. They also consider the case where there is more than a single TCP =
flow, which is of course what we need to do ultimately, but at this =
point I'm not even sure we understand the needs of a single TCP flow.

> We are doing some work on improving perfornmance over low buffered =
links and will have something to share may be by the next IETF.=20

Cool! Please send a pointer when you're ready.

Iljitsch=

From iljitsch@muada.com  Tue Nov 10 23:24:32 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 7DE433A6405 for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 23:24:32 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YqIdf3EfoBeZ for <homegate@core3.amsl.com>; Tue, 10 Nov 2009 23:24:31 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 7F9403A68C4 for <homegate@ietf.org>; Tue, 10 Nov 2009 23:24:31 -0800 (PST)
Received: from [192.168.2.11] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAB7OmZB097077 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 11 Nov 2009 08:24:49 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi>
Date: Wed, 11 Nov 2009 08:24:51 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi>
To: Markku Kojo <kojo@cs.helsinki.fi>
X-Mailer: Apple Mail (2.1077)
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 07:24:32 -0000

On 11Nov, 2009, at 7:58 , Markku Kojo wrote:

> The problem with your 100% avg utilization logic here is that you=20
> incorrectly assume that by adding buffering you get  > 100% link=20
> utilization when pkts start accumulating in the buffer!?

> When cwnd becomes  > 100 pkts in this example (assuming zero delay for =
=20
> the rest of the path), the pkts get buffered at the bottleneck link =
and=20
> the link is fully utilized but it still delivers pkts only at max=20
> link rate!
> That is, at times when your cwnd is < 100 pkts your link is =
underutilized,
> and if your link is ever underutilized during your connection (or
> observation time period) you can never reach 100 % avg utilization.

What happens is that during the > 100% periods the excess packets beyond =
100% are buffered. Then, during the < 100% periods, the buffer is =
drained so the 100% rate is maintaned. I calculate the required amount =
of buffering as follows as in my earlier messages:

linkbw ^ 2 / MSS * RTT * 7 for the buffer size in bytes if the linkbw =20=

and MSS are in kilobits

It's really easy to show this in a figure, where you can see that the =
part of the sawtooth above 100% must be buffered to fill in the empty =
part below 100%.

Iljitsch=

From lars.eggert@nokia.com  Wed Nov 11 00:37:36 2009
Return-Path: <lars.eggert@nokia.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A45D13A6822 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:37:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.57
X-Spam-Level: 
X-Spam-Status: No, score=-2.57 tagged_above=-999 required=5 tests=[AWL=0.029,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZFJXQlC2E87s for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:37:36 -0800 (PST)
Received: from mail.fit.nokia.com (mail.fit.nokia.com [195.148.124.195]) by core3.amsl.com (Postfix) with ESMTP id AB8E83A67DD for <homegate@ietf.org>; Wed, 11 Nov 2009 00:37:35 -0800 (PST)
Received: from [IPv6:2001:dfb::24:225:ff:fe45:eccf] ([IPv6:2001:dfb:0:24:225:ff:fe45:eccf]) (authenticated bits=0) by mail.fit.nokia.com (8.14.3/8.14.3) with ESMTP id nAB8boQR033562 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Wed, 11 Nov 2009 10:37:53 +0200 (EET) (envelope-from lars.eggert@nokia.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: multipart/signed; boundary=Apple-Mail-16-448590069; protocol="application/pkcs7-signature"; micalg=sha1
From: Lars Eggert <lars.eggert@nokia.com>
In-Reply-To: <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com>
Date: Wed, 11 Nov 2009 17:37:44 +0900
Message-Id: <984C2F9C-0E5D-4E4A-9FF0-8AD676806AF3@nokia.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi> <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com>
To: Iljitsch van Beijnum <iljitsch@muada.com>
X-Mailer: Apple Mail (2.1077)
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.fit.nokia.com [IPv6:2001:2060:40:1::123]); Wed, 11 Nov 2009 10:37:55 +0200 (EET)
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 08:37:36 -0000

--Apple-Mail-16-448590069
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii

Hi, Iljitsch,

I note that ns2 and ns3 should be readily available to verify the math :-)

Lars
--Apple-Mail-16-448590069
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGbDCCAyUw
ggKOoAMCAQICEAdjk36sXKbnVn15S0/qUp0wDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA5MDYxNTExMjYxNFoXDTEwMDYxNTExMjYx
NFowXDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2Vy
dDEkMCIGCSqGSIb3DQEJARYVbGFycy5lZ2dlcnRAbm9raWEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA7mR8A+Pn0/FsUkMX6Pyjw+FL3IFcJk8GaKV5VJ40TMI0Wh8oq20cqA9X
uqnVDW9WztKwH+o+msJenLwWpprbpJm4TImYGbnUJxYyN8gb81aiX1Bw2xCpJ5z3H2+8DsReJLuY
Rdl4bVvaIxLIL4odmfsRwzPyNkOK8LRtfl6OPcaDOlFWzbikULfIVGGu7BqK4lxQSpYwwpZkOMOB
6nnBSfUOtBEmqO+qZG/nL/JxWFV5vxQgg4XHbsMMTxFf6+ji18BD09BUIfDLTuJoCzFmQhrM9vLT
VuRhHWSL20LoafGjXv6mPt3i9IGJHpVb2dMQUgOgRyWHTKiUJVU/rUTdWwIDAQABo14wXDAqBgUr
ZQEEAQQhMB8CAQAwGjAYAgEEBBNMMnVNeWZmQk5VYk5KSmNkWjJzMCAGA1UdEQQZMBeBFWxhcnMu
ZWdnZXJ0QG5va2lhLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBADUx+67n98wt
I1vydB90HeSZP4Y64VCxxb0NxGGFvfc2+JdVKeHJ/xT+l+ygYKsWNwJJprkPi4WZ5G0crkq4VK1H
5drEJIztpSPVfWI05vPidaaGuuuCR+6MvJMtOTEYEvc/6eovBnkrzRf9x5x5EyuJXAWTeuBADg80
QI3vQ1tZMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTAT
BgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUg
Q29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIG
A1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25h
bC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjEL
MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNV
BAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7
n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAw
QwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJl
ZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRl
TGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9M
Ibj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowg
T2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAxAwggMMAgEBMHYw
YjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq
BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhAHY5N+rFym51Z9eUtP
6lKdMAkGBSsOAwIaBQCgggFvMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF
MQ8XDTA5MTExMTA4Mzc0NFowIwYJKoZIhvcNAQkEMRYEFDTa0gmtZsEvbJLI2sN4AUZ2cGxlMIGF
BgkrBgEEAYI3EAQxeDB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGlu
ZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD
QQIQB2OTfqxcpudWfXlLT+pSnTCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUw
IwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVy
c29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQB2OTfqxcpudWfXlLT+pSnTANBgkqhkiG9w0BAQEF
AASCAQBNZaPIvkpGRcvGQI1Es8a2W7u7vK9/2soaWEpMqtATENAkdEQRufeJjqM0nZ8oJ+IZTqKO
FE/wX0b0qFDwcuPjhbV7id4eMW90In5UqcGHcAnMLKC+0phg7ZRmZdLZU3+wTh1MA+FI0iclaim8
qVqWizpTXuiNcTE0eqhsbTsOVBme/MqxeNtHxbq7hPzaiupyy9O+/466UE4HxcbwGVMVK8CFgAeV
+GEYGgpkeDCgUh4ap6rdGBWsGaQ8CAKg+CZkDErmXSkmsz0YPowZPCwNL+AnHJ7Ke//Bn82883TF
wRvQeO/2++ZvToeFwPZM8ZV2HLz0xKzZpFq8+SBxzcKbAAAAAAAA

--Apple-Mail-16-448590069--

From kojo@cs.helsinki.fi  Wed Nov 11 00:39:00 2009
Return-Path: <kojo@cs.helsinki.fi>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B093A3A67A7 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:39:00 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.599
X-Spam-Level: 
X-Spam-Status: No, score=-6.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2sTzRKlmaEB4 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:39:00 -0800 (PST)
Received: from mail.cs.helsinki.fi (courier.cs.helsinki.fi [128.214.9.1]) by core3.amsl.com (Postfix) with ESMTP id E87C93A6A01 for <homegate@ietf.org>; Wed, 11 Nov 2009 00:38:58 -0800 (PST)
Received: from tx3-4.cs.helsinki.fi (host-32-140.meeting.ietf.org [133.93.32.140]) (AUTH: PLAIN cs-relay, TLS: TLSv1/SSLv3,256bits,AES256-SHA) by mail.cs.helsinki.fi with esmtp; Wed, 11 Nov 2009 10:39:24 +0200 id 00063EB1.4AFA783C.000068CA
Received: by tx3-4.cs.helsinki.fi (Postfix, from userid 3011) id 59D1624A0D; Wed, 11 Nov 2009 10:39:19 +0200 (EET)
Received: from localhost (localhost [127.0.0.1]) by tx3-4.cs.helsinki.fi (Postfix) with ESMTP id 4CD9724A0C; Wed, 11 Nov 2009 10:39:19 +0200 (EET)
Date: Wed, 11 Nov 2009 10:39:19 +0200 (EET)
From: Markku Kojo <kojo@cs.helsinki.fi>
To: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com>
Message-ID: <Pine.LNX.4.64.0911110958410.19508@tx3-4.cs.helsinki.fi>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi> <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 08:39:00 -0000

On Wed, 11 Nov 2009, Iljitsch van Beijnum wrote:

> On 11Nov, 2009, at 7:58 , Markku Kojo wrote:
> 
> > The problem with your 100% avg utilization logic here is that you 
> > incorrectly assume that by adding buffering you get  > 100% link 
> > utilization when pkts start accumulating in the buffer!?
> 
> > When cwnd becomes  > 100 pkts in this example (assuming zero delay for  
> > the rest of the path), the pkts get buffered at the bottleneck link and 
> > the link is fully utilized but it still delivers pkts only at max 
> > link rate!
> > That is, at times when your cwnd is < 100 pkts your link is underutilized,
> > and if your link is ever underutilized during your connection (or
> > observation time period) you can never reach 100 % avg utilization.
> 
> What happens is that during the > 100% periods the excess packets 
> beyond 100% are buffered. Then, during the < 100% periods, the buffer is 
> drained so the 100% rate is maintaned. 

Right, but you need to have enough pkts in the queue to fill the gap,
i.e., the time during which a TCP sender is not sending any new data.

When the cwnd gets halved and the TCP sender fast retransmits, the TCP
sender is not sending any new data pkts until a half of the flightsize 
worth dupacks have arrived at the sender (assuming rate-halving is 
not used). In your example the flightsize (and cwnd) would be 
133 pkts, and after cwnd is halved cwnd becomes ~67 pkts. Now duirng 
the fast recovery the TCP sender needs to wait until 67 dupacks have 
arrived to inflate cwnd beyond 133 pkts and continue injecting new data. 
These dupacks arrive at the same rate as the router buffer at the 
bottleneck link drains. So, with buffer size of 33 pkts there will be a 
gap with no data in the buffer that (roughly) equals to 67-33=34 pkts 
at the link rate --> link becomes underutilized.

/Markku 

From iljitsch@muada.com  Wed Nov 11 00:39:40 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 97F8C3A69AD for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:39:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eTOs-pUUSkL6 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:39:39 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 80E773A67DD for <homegate@ietf.org>; Wed, 11 Nov 2009 00:39:39 -0800 (PST)
Received: from claw.it.uc3m.es (claw.it.uc3m.es [163.117.139.216]) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAB8duwb097780 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 11 Nov 2009 09:39:56 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <984C2F9C-0E5D-4E4A-9FF0-8AD676806AF3@nokia.com>
Date: Wed, 11 Nov 2009 09:39:56 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <1B8E4E37-5402-4011-9DEE-386857896147@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi> <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com> <984C2F9C-0E5D-4E4A-9FF0-8AD676806AF3@nokia.com>
To: Lars Eggert <lars.eggert@nokia.com>
X-Mailer: Apple Mail (2.1077)
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 08:39:40 -0000

On 11Nov, 2009, at 9:37 , Lars Eggert wrote:

> I note that ns2 and ns3 should be readily available to verify the math =
:-)

Should be easy enough to do: just manipulate the amount of router =
buffering with standard TCP.

But I have MPTCP work that I need to get done...

Iljitsch=

From muraris@microsoft.com  Wed Nov 11 00:41:43 2009
Return-Path: <muraris@microsoft.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9B3573A67DD for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:41:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.332
X-Spam-Level: 
X-Spam-Status: No, score=-10.332 tagged_above=-999 required=5 tests=[AWL=0.267, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xVTy0pecU0pj for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 00:41:42 -0800 (PST)
Received: from smtp.microsoft.com (mail1.microsoft.com [131.107.115.212]) by core3.amsl.com (Postfix) with ESMTP id CB7D73A67A7 for <homegate@ietf.org>; Wed, 11 Nov 2009 00:41:42 -0800 (PST)
Received: from TK5EX14HUBC101.redmond.corp.microsoft.com (157.54.7.153) by TK5-EXGWY-E801.partners.extranet.microsoft.com (10.251.56.50) with Microsoft SMTP Server (TLS) id 8.2.176.0; Wed, 11 Nov 2009 00:42:28 -0800
Received: from TK5EX14MBXC130.redmond.corp.microsoft.com ([169.254.9.154]) by TK5EX14HUBC101.redmond.corp.microsoft.com ([157.54.7.153]) with mapi; Wed, 11 Nov 2009 00:42:10 -0800
From: Murari Sridharan <muraris@microsoft.com>
To: Markku Kojo <kojo@cs.helsinki.fi>, Iljitsch van Beijnum <iljitsch@muada.com>
Thread-Topic: [homegate] Buffers in home gateways
Thread-Index: AQHKYXKlCKJs73gN+k6+K7Ps0dVxGJEwTNCAgAAesgCAAJBiAIAAB2+AgAAUz4D//3pBIA==
Date: Wed, 11 Nov 2009 08:42:09 +0000
Message-ID: <03520536067DF14395EA93FE7ECFDCF009732429@TK5EX14MBXC130.redmond.corp.microsoft.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi> <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com> <Pine.LNX.4.64.0911110958410.19508@tx3-4.cs.helsinki.fi>
In-Reply-To: <Pine.LNX.4.64.0911110958410.19508@tx3-4.cs.helsinki.fi>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 08:41:43 -0000

Iljitsch, you should look at the Stevens book there is a nice illustration =
that talks about the gap that markku outlines below.=20

-----Original Message-----
From: homegate-bounces@ietf.org [mailto:homegate-bounces@ietf.org] On Behal=
f Of Markku Kojo
Sent: Wednesday, November 11, 2009 12:39 AM
To: Iljitsch van Beijnum
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways

On Wed, 11 Nov 2009, Iljitsch van Beijnum wrote:

> On 11Nov, 2009, at 7:58 , Markku Kojo wrote:
>=20
> > The problem with your 100% avg utilization logic here is that you=20
> > incorrectly assume that by adding buffering you get  > 100% link=20
> > utilization when pkts start accumulating in the buffer!?
>=20
> > When cwnd becomes  > 100 pkts in this example (assuming zero delay for =
=20
> > the rest of the path), the pkts get buffered at the bottleneck link and=
=20
> > the link is fully utilized but it still delivers pkts only at max=20
> > link rate!
> > That is, at times when your cwnd is < 100 pkts your link is underutiliz=
ed,
> > and if your link is ever underutilized during your connection (or
> > observation time period) you can never reach 100 % avg utilization.
>=20
> What happens is that during the > 100% periods the excess packets=20
> beyond 100% are buffered. Then, during the < 100% periods, the buffer is=
=20
> drained so the 100% rate is maintaned.=20

Right, but you need to have enough pkts in the queue to fill the gap,
i.e., the time during which a TCP sender is not sending any new data.

When the cwnd gets halved and the TCP sender fast retransmits, the TCP
sender is not sending any new data pkts until a half of the flightsize=20
worth dupacks have arrived at the sender (assuming rate-halving is=20
not used). In your example the flightsize (and cwnd) would be=20
133 pkts, and after cwnd is halved cwnd becomes ~67 pkts. Now duirng=20
the fast recovery the TCP sender needs to wait until 67 dupacks have=20
arrived to inflate cwnd beyond 133 pkts and continue injecting new data.=20
These dupacks arrive at the same rate as the router buffer at the=20
bottleneck link drains. So, with buffer size of 33 pkts there will be a=20
gap with no data in the buffer that (roughly) equals to 67-33=3D34 pkts=20
at the link rate --> link becomes underutilized.

/Markku=20
_______________________________________________
homegate mailing list
homegate@ietf.org
https://www.ietf.org/mailman/listinfo/homegate


From iljitsch@muada.com  Wed Nov 11 01:37:35 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1367B28C112 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 01:37:35 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Su2xtMLvlkJR for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 01:37:34 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 12E763A6A6C for <homegate@ietf.org>; Wed, 11 Nov 2009 01:37:33 -0800 (PST)
Received: from claw.it.uc3m.es (claw.it.uc3m.es [163.117.139.216]) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAB9bo5l000521 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 11 Nov 2009 10:37:50 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <03520536067DF14395EA93FE7ECFDCF009732429@TK5EX14MBXC130.redmond.corp.microsoft.com>
Date: Wed, 11 Nov 2009 10:37:53 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <0480304D-CC57-4A7A-A541-3A69762CD1F6@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com> <20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <Pine.LNX.4.64.0911110652230.11003@tx3-4.cs.helsinki.fi> <DA044BE0-2AA3-47B4-8E62-1AF7FDD0FDA7@muada.com> <Pine.LNX.4.64.0911110958410.19508@tx3-4.cs.helsinki.fi> <03520536067DF14395EA93FE7ECFDCF009732429@TK5EX14MBXC130.redmond.corp.microsoft.com>
To: Murari Sridharan <muraris@microsoft.com>
X-Mailer: Apple Mail (2.1077)
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 09:37:35 -0000

On 11Nov, 2009, at 9:42 , Murari Sridharan wrote:

> Iljitsch, you should look at the Stevens book there is a nice =
illustration that talks about the gap that markku outlines below.=20

As I tried to explain, there is no gap, this is why my result is so much =
higher than the bandwith RTT product.

With the numbers used earlier (10 Mbps, 10 kilobit packets, 0.1 s RTT) =
in congestion avoidance it would take 66 RTTs to grow from the minimum =
cwnd of 67 to the maximum of 133, where half of that time we build up a =
surplus that needs to be buffered and the other half of the time there =
is a deficit that we fill in with buffered packets. During 33 RTTs that =
we buffer we start with 1 surplus packet and end with 33 so an average =
of 17. So we need to buffer 33 * 17 =3D 561 packets, nearly 6 times the =
BW RTT product.=

From mark_francisco@cable.comcast.com  Wed Nov 11 02:47:23 2009
Return-Path: <mark_francisco@cable.comcast.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9854128C16A for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 02:47:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.463
X-Spam-Level: 
X-Spam-Status: No, score=-0.463 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n3ipc-UxqQGX for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 02:47:22 -0800 (PST)
Received: from paoakoavas09.cable.comcast.com (paoakoavas09.cable.comcast.com [208.17.35.58]) by core3.amsl.com (Postfix) with ESMTP id AFBF728C168 for <homegate@ietf.org>; Wed, 11 Nov 2009 02:47:22 -0800 (PST)
Received: from ([10.52.116.30]) by paoakoavas09.cable.comcast.com with ESMTP  id KP-NTF18.82243164; Wed, 11 Nov 2009 05:47:39 -0500
Received: from PACDCEXCMB06.cable.comcast.com ([24.40.15.22]) by PAOAKEXCSMTP01.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 11 Nov 2009 05:47:40 -0500
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Wed, 11 Nov 2009 05:47:39 -0500
Message-ID: <0D33B53F71C4134787232BF844B87B870796198B@PACDCEXCMB06.cable.comcast.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: homegate Digest, Vol 6, Issue 4
Thread-Index: AcpinGNYMvwFzJyHSqW1G6eVinLznAAIAAPO
From: "Francisco, Mark" <Mark_Francisco@cable.comcast.com>
To: <homegate@ietf.org>
X-OriginalArrivalTime: 11 Nov 2009 10:47:40.0480 (UTC) FILETIME=[64036C00:01CA62BC]
Subject: Re: [homegate] homegate Digest, Vol 6, Issue 4
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 10:47:23 -0000

UGxlYXNlIGRvLiBDYW4geW91IHJlYWNoIG91dCB0byBTdGV2ZSBSaWNobWFuPw0KTWFyayBGcmFu
Y2lzY28NCkNvbWNhc3QgT2ZmaWNlIG9mIHRoZSBDVE8NCk1hcmtfRnJhbmNpc2NvQGNhYmxlLmNv
bWNhc3QuY29tDQpNb2JpbGUgKDIxNSk0MTYtNTYzMg0KDQotLS0tLSBPcmlnaW5hbCBNZXNzYWdl
IC0tLS0tDQpGcm9tOiBob21lZ2F0ZS1ib3VuY2VzQGlldGYub3JnIDxob21lZ2F0ZS1ib3VuY2Vz
QGlldGYub3JnPg0KVG86IGhvbWVnYXRlQGlldGYub3JnIDxob21lZ2F0ZUBpZXRmLm9yZz4NClNl
bnQ6IFdlZCBOb3YgMTEgMDE6NTc6NTggMjAwOQ0KU3ViamVjdDogaG9tZWdhdGUgRGlnZXN0LCBW
b2wgNiwgSXNzdWUgNA0KDQpJZiB5b3UgaGF2ZSByZWNlaXZlZCB0aGlzIGRpZ2VzdCB3aXRob3V0
IGFsbCB0aGUgaW5kaXZpZHVhbCBtZXNzYWdlDQphdHRhY2htZW50cyB5b3Ugd2lsbCBuZWVkIHRv
IHVwZGF0ZSB5b3VyIGRpZ2VzdCBvcHRpb25zIGluIHlvdXIgbGlzdA0Kc3Vic2NyaXB0aW9uLiAg
VG8gZG8gc28sIGdvIHRvIA0KDQpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZv
L2hvbWVnYXRlDQoNCkNsaWNrIHRoZSAnVW5zdWJzY3JpYmUgb3IgZWRpdCBvcHRpb25zJyBidXR0
b24sIGxvZyBpbiwgYW5kIHNldCAiR2V0DQpNSU1FIG9yIFBsYWluIFRleHQgRGlnZXN0cz8iIHRv
IE1JTUUuICBZb3UgY2FuIHNldCB0aGlzIG9wdGlvbg0KZ2xvYmFsbHkgZm9yIGFsbCB0aGUgbGlz
dCBkaWdlc3RzIHlvdSByZWNlaXZlIGF0IHRoaXMgcG9pbnQuDQoNCg0KDQpTZW5kIGhvbWVnYXRl
IG1haWxpbmcgbGlzdCBzdWJtaXNzaW9ucyB0bw0KCWhvbWVnYXRlQGlldGYub3JnDQoNClRvIHN1
YnNjcmliZSBvciB1bnN1YnNjcmliZSB2aWEgdGhlIFdvcmxkIFdpZGUgV2ViLCB2aXNpdA0KCWh0
dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vaG9tZWdhdGUNCm9yLCB2aWEgZW1h
aWwsIHNlbmQgYSBtZXNzYWdlIHdpdGggc3ViamVjdCBvciBib2R5ICdoZWxwJyB0bw0KCWhvbWVn
YXRlLXJlcXVlc3RAaWV0Zi5vcmcNCg0KWW91IGNhbiByZWFjaCB0aGUgcGVyc29uIG1hbmFnaW5n
IHRoZSBsaXN0IGF0DQoJaG9tZWdhdGUtb3duZXJAaWV0Zi5vcmcNCg0KV2hlbiByZXBseWluZywg
cGxlYXNlIGVkaXQgeW91ciBTdWJqZWN0IGxpbmUgc28gaXQgaXMgbW9yZSBzcGVjaWZpYw0KdGhh
biAiUmU6IENvbnRlbnRzIG9mIGhvbWVnYXRlIGRpZ2VzdC4uLiINCg==

From mark_francisco@cable.comcast.com  Wed Nov 11 03:01:42 2009
Return-Path: <mark_francisco@cable.comcast.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D929628C149 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 03:01:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.463
X-Spam-Level: 
X-Spam-Status: No, score=-4.463 tagged_above=-999 required=5 tests=[AWL=4.000,  BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vCVwIfxn20kx for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 03:01:42 -0800 (PST)
Received: from pacdcimo01.cable.comcast.com (PacdcIMO01.cable.comcast.com [24.40.8.145]) by core3.amsl.com (Postfix) with ESMTP id EABEA28C289 for <homegate@ietf.org>; Wed, 11 Nov 2009 03:01:41 -0800 (PST)
Received: from ([24.40.15.92]) by pacdcimo01.cable.comcast.com with ESMTP  id 5503620.60376740; Wed, 11 Nov 2009 06:02:07 -0500
Received: from PACDCEXCMB06.cable.comcast.com ([24.40.15.22]) by PACDCEXCSMTP03.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.3959); Wed, 11 Nov 2009 06:02:07 -0500
x-mimeole: Produced By Microsoft Exchange V6.5
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Wed, 11 Nov 2009 06:02:06 -0500
Message-ID: <0D33B53F71C4134787232BF844B87B870796198C@PACDCEXCMB06.cable.comcast.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: Please disregard
Thread-Index: Acpivmh+lbs40LrKQbKlHT6xzQpvOw==
From: "Francisco, Mark" <Mark_Francisco@cable.comcast.com>
To: <homegate@ietf.org>
X-OriginalArrivalTime: 11 Nov 2009 11:02:07.0864 (UTC) FILETIME=[6903B780:01CA62BE]
Subject: [homegate] Please disregard
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 11:01:43 -0000

UGxlYXNlIGRpc3JlZ2FyZCBteSBwcmV2aW91cyBtZXNzYWdlLiBJdCB3YXMgYWRkcmVzc2VkIGlu
IGVycm9yLiANCk1hcmsNCk1hcmsgRnJhbmNpc2NvDQpDb21jYXN0IE9mZmljZSBvZiB0aGUgQ1RP
DQpNYXJrX0ZyYW5jaXNjb0BjYWJsZS5jb21jYXN0LmNvbQ0KTW9iaWxlICgyMTUpNDE2LTU2MzIN
Cg==

From iljitsch@muada.com  Wed Nov 11 12:17:53 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8115F3A67A8 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 12:17:53 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level: 
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_23=0.6]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wegmrgt6H5rp for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 12:17:52 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 2A9483A68AA for <homegate@ietf.org>; Wed, 11 Nov 2009 12:17:51 -0800 (PST)
Received: from [192.168.2.11] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nABKI3mV018519 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 11 Nov 2009 21:18:04 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com>
Date: Wed, 11 Nov 2009 21:18:07 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <CABB7AC8-9938-4E21-B96D-1051A0C278D6@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com><20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com>
To: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>
X-Mailer: Apple Mail (2.1077)
Cc: homegate@ietf.org
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Nov 2009 20:17:53 -0000

On 11 nov 2009, at 0:07, Woundy, Richard wrote:

>> So this whole notion of one RTT worth of buffering is an urban myth, =
certainly in the case of of a single TCP session

Ok, this is the part where I have to eat my words and admit that I was =
wrong. I hate it when that happens.

What I overlooked is the ACK clocking. When the link supports a cwnd of =
100 but the cwnd becomes 101, that means one packet needs to be =
buffered. But then when cwnd becomes 102, that means 1 extra packet, not =
2, because the ACKs keep returning at a rate of 50 that ACK two packets =
so two new packets are released except when the window just grew, in =
which case it's 3, which only happens once per RTT. So the buffering =
requirements are linear with bandwidth.

> I would consider reading the following Sigcomm 2004 paper, "Sizing
> Router Buffers" by Appenzeller, Keslassy, and McKeown,
> http://yuba.stanford.edu/~nickm/papers/sigcomm2004.pdf, especially
> sections 1.2, 2 and 3.1.

Thanks. This is much better than the 1994 paper I mentioned earlier. =
However, it looks like they modeled TCP Reno rather than TCP New Reno, =
i.e., they don't account for the fast recovery phase, so they arrive at =
a larger buffering requirement because they assume the sender will stop =
sending after a loss rather than do a fast retransmit and keep going =
with a reduced cwnd.

Putting it all together, it looks like there is some magic number of =
buffers somewhere between 0.5 and 1.0 times BW*RTT (some back of the =
envelope stuff suggests something like BW*RTT-SQRT(BW*RTT*2)) where the =
buffer builds up until there is a tail drop, and then the sending rate =
drops to below the link rate for just long enough to drain the built up =
queue and the cycle restarts.

Too much buffering isn't as disasterous as it would seem. For instance, =
with BW*RTT worth of buffering the cwnd grows to exactly twice the link =
rate, so a loss then kicks it down to the cwnd that matches the link =
rate and the buffer doesn't drain. This means that we almost immediately =
get a tail drop and _then_ the cwnd gets significantly below the link =
rate so the queue will drain for a while.

On the other hand, if there isn't enough buffering then after a single =
drop the cwnd becomes so low that the queue drains completely and then =
the link goes idle for some time before the cwnd reaches the size that =
matches the link rate.

All of the above suggests that any amount of buffering will make TCP =
perform at least a little better than the theoretical 75% without =
buffering and even having too much buffering isn't immediately =
disasterous as there won't be standing queues that cause immediate tail =
drops for all new sessions.

So 10 packets worth of buffering seems a reasonable starting point. It's =
still pretty slow at 128 or 256 kbps, but not to an unworkable degree, =
and assuming a 0.1 RTT 10 packets is BW*RTT at 1200 kbps so full single =
TCP flow performance upto somewhere in the 1 - 2 Mbps range.

(Interesting by the way that we're having a detailed discussion here =
while the congestion control gurus in ICCRG had little or no interest in =
this subject.)

Iljitsch


From lars.eggert@nokia.com  Wed Nov 11 16:49:46 2009
Return-Path: <lars.eggert@nokia.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id CD38D28C15A for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 16:49:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.566
X-Spam-Level: 
X-Spam-Status: No, score=-2.566 tagged_above=-999 required=5 tests=[AWL=0.033,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HnVTk4X1CD4C for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 16:49:46 -0800 (PST)
Received: from mail.fit.nokia.com (mail.fit.nokia.com [195.148.124.195]) by core3.amsl.com (Postfix) with ESMTP id 76DFD28C12D for <homegate@ietf.org>; Wed, 11 Nov 2009 16:49:45 -0800 (PST)
Received: from host-24-27.meeting.ietf.org (host-24-27.meeting.ietf.org [133.93.24.27]) (authenticated bits=0) by mail.fit.nokia.com (8.14.3/8.14.3) with ESMTP id nAC0nvri094980 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 12 Nov 2009 02:50:01 +0200 (EET) (envelope-from lars.eggert@nokia.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: multipart/signed; boundary=Apple-Mail-4-506922484; protocol="application/pkcs7-signature"; micalg=sha1
From: Lars Eggert <lars.eggert@nokia.com>
In-Reply-To: <CABB7AC8-9938-4E21-B96D-1051A0C278D6@muada.com>
Date: Thu, 12 Nov 2009 09:49:56 +0900
Message-Id: <76FBAB64-5CA7-4CE4-9DD3-74E9415F4C40@nokia.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com><20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com> <CABB7AC8-9938-4E21-B96D-1051A0C278D6@muada.com>
To: Iljitsch van Beijnum <iljitsch@muada.com>
X-Mailer: Apple Mail (2.1077)
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.fit.nokia.com [212.213.221.39]); Thu, 12 Nov 2009 02:50:03 +0200 (EET)
Cc: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>, "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Nov 2009 00:49:46 -0000

--Apple-Mail-4-506922484
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

On 2009-11-12, at 5:18, Iljitsch van Beijnum wrote:
> (Interesting by the way that we're having a detailed discussion here =
while the congestion control gurus in ICCRG had little or no interest in =
this subject.)

I dare saying that that is because the conclusion we're reaching here =
aren't exactly new.

Lars=

--Apple-Mail-4-506922484
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGbDCCAyUw
ggKOoAMCAQICEAdjk36sXKbnVn15S0/qUp0wDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA5MDYxNTExMjYxNFoXDTEwMDYxNTExMjYx
NFowXDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2Vy
dDEkMCIGCSqGSIb3DQEJARYVbGFycy5lZ2dlcnRAbm9raWEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA7mR8A+Pn0/FsUkMX6Pyjw+FL3IFcJk8GaKV5VJ40TMI0Wh8oq20cqA9X
uqnVDW9WztKwH+o+msJenLwWpprbpJm4TImYGbnUJxYyN8gb81aiX1Bw2xCpJ5z3H2+8DsReJLuY
Rdl4bVvaIxLIL4odmfsRwzPyNkOK8LRtfl6OPcaDOlFWzbikULfIVGGu7BqK4lxQSpYwwpZkOMOB
6nnBSfUOtBEmqO+qZG/nL/JxWFV5vxQgg4XHbsMMTxFf6+ji18BD09BUIfDLTuJoCzFmQhrM9vLT
VuRhHWSL20LoafGjXv6mPt3i9IGJHpVb2dMQUgOgRyWHTKiUJVU/rUTdWwIDAQABo14wXDAqBgUr
ZQEEAQQhMB8CAQAwGjAYAgEEBBNMMnVNeWZmQk5VYk5KSmNkWjJzMCAGA1UdEQQZMBeBFWxhcnMu
ZWdnZXJ0QG5va2lhLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBADUx+67n98wt
I1vydB90HeSZP4Y64VCxxb0NxGGFvfc2+JdVKeHJ/xT+l+ygYKsWNwJJprkPi4WZ5G0crkq4VK1H
5drEJIztpSPVfWI05vPidaaGuuuCR+6MvJMtOTEYEvc/6eovBnkrzRf9x5x5EyuJXAWTeuBADg80
QI3vQ1tZMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTAT
BgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUg
Q29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIG
A1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25h
bC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjEL
MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNV
BAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7
n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAw
QwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJl
ZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRl
TGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9M
Ibj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowg
T2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAxAwggMMAgEBMHYw
YjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq
BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhAHY5N+rFym51Z9eUtP
6lKdMAkGBSsOAwIaBQCgggFvMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF
MQ8XDTA5MTExMjAwNDk1N1owIwYJKoZIhvcNAQkEMRYEFFFsbsorz66RDtu35B3Rj12BXKJZMIGF
BgkrBgEEAYI3EAQxeDB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGlu
ZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD
QQIQB2OTfqxcpudWfXlLT+pSnTCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUw
IwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVy
c29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQB2OTfqxcpudWfXlLT+pSnTANBgkqhkiG9w0BAQEF
AASCAQCKnCDgP9YHQ266XkbqjC6oRnjc/HHqjqZOJAwZF2pVDKYyuP49zm9u4IU2rNywHVh32kbc
XMEPst10QtMxpRP0ajqWJ5xJrq+KKprOVqQfBFvju4CFA+2RyMwhnJJpx2ARDDMjXKJmuLBx/XKF
HqmEB7OKKuNf/GUDJ+mPCTKcjAU0pN3tka8Vb/Z9fecmS28x8jaaz4YAVgyt0BOsNiy8al94SnHq
8LKCxwUCkz7/Ta+LlC5VAHSvo1c120M1sYB5FOZLUNlQv2LhZ9HybZ2DY3w3uooYhx8wcX6oL76S
eWgdFrNH0FmOnCS0FFeVCc1paGPI1pIMrIaKSnTOgwe9AAAAAAAA

--Apple-Mail-4-506922484--

From iljitsch@muada.com  Wed Nov 11 23:31:17 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id F3E6D28C113 for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 23:31:16 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.574
X-Spam-Level: 
X-Spam-Status: No, score=-2.574 tagged_above=-999 required=5 tests=[AWL=0.025,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6HZD+Mm4SMAP for <homegate@core3.amsl.com>; Wed, 11 Nov 2009 23:31:16 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id 215DB28C0F7 for <homegate@ietf.org>; Wed, 11 Nov 2009 23:31:15 -0800 (PST)
Received: from [192.168.2.11] (static-167-138-7-89.ipcom.comunitel.net [89.7.138.167] (may be forged)) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nAC7VUx9024263 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Thu, 12 Nov 2009 08:31:31 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <76FBAB64-5CA7-4CE4-9DD3-74E9415F4C40@nokia.com>
Date: Thu, 12 Nov 2009 08:31:34 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <00E9CABE-654D-4156-955D-8F53E368FFBF@muada.com>
References: <332B7621-8820-4CA9-8755-AEF7FEC4E1ED@muada.com><20091110203137.GA16898@wel-19.cs.helsinki.fi> <D9DD9B1B-4FA6-470E-B20D-EFDBE2262456@muada.com> <8A82D1BFEDDE7E4597978355239BBBCB5CCD06@PACDCEXCMB06.cable.comcast.com> <CABB7AC8-9938-4E21-B96D-1051A0C278D6@muada.com> <76FBAB64-5CA7-4CE4-9DD3-74E9415F4C40@nokia.com>
To: Lars Eggert <lars.eggert@nokia.com>
X-Mailer: Apple Mail (2.1077)
Cc: "Woundy, Richard" <Richard_Woundy@cable.comcast.com>, "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Buffers in home gateways
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 12 Nov 2009 07:31:17 -0000

On 12 nov 2009, at 1:49, Lars Eggert wrote:

>> (Interesting by the way that we're having a detailed discussion here =
while the congestion control gurus in ICCRG had little or no interest in =
this subject.)

> I dare saying that that is because the conclusion we're reaching here =
aren't exactly new.

Then why is there no RFC providing guidance to vendors about this issue?=

From jason_livingood@cable.comcast.com  Thu Nov 12 17:44:48 2009
Return-Path: <jason_livingood@cable.comcast.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2A6C03A6819 for <homegate@core3.amsl.com>; Thu, 12 Nov 2009 17:44:48 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.009
X-Spam-Level: *
X-Spam-Status: No, score=1.009 tagged_above=-999 required=5 tests=[AWL=-1.992,  BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368,  HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=1.396, RCVD_NUMERIC_HELO=2.067]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o8hWCLSPLJqs for <homegate@core3.amsl.com>; Thu, 12 Nov 2009 17:44:45 -0800 (PST)
Received: from paoakoavas10.cable.comcast.com (paoakoavas10.cable.comcast.com [208.17.35.59]) by core3.amsl.com (Postfix) with ESMTP id 77C133A6358 for <homegate@ietf.org>; Thu, 12 Nov 2009 17:44:45 -0800 (PST)
Received: from ([24.40.15.92]) by paoakoavas10.cable.comcast.com with ESMTP  id KP-TDCH7.72705579; Thu, 12 Nov 2009 20:45:01 -0500
Received: from PACDCEXCMB04.cable.comcast.com ([24.40.15.86]) by PACDCEXCSMTP03.cable.comcast.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 12 Nov 2009 20:45:02 -0500
Received: from 198.137.252.126 ([198.137.252.126]) by PACDCEXCMB04.cable.comcast.com ([24.40.15.86]) via Exchange Front-End Server webmail.comcast.com ([198.137.252.77]) with Microsoft Exchange Server HTTP-DAV ; Fri, 13 Nov 2009 01:45:02 +0000
User-Agent: Microsoft-Entourage/12.20.0.090605
Date: Fri, 13 Nov 2009 10:44:57 +0900
From: Jason Livingood <jason_livingood@cable.comcast.com>
To: "homegate@ietf.org" <homegate@ietf.org>
Message-ID: <C722E929.16C5C%jason_livingood@cable.comcast.com>
Thread-Topic: Draft HomeGate BoF Minutes Posted
Thread-Index: AcpkAud+BH/5QenGtUahhDdcb0ho6Q==
Mime-version: 1.0
Content-type: multipart/alternative; boundary="B_3340953897_771082"
X-OriginalArrivalTime: 13 Nov 2009 01:45:02.0459 (UTC) FILETIME=[EABF2CB0:01CA6402]
Subject: [homegate] Draft HomeGate BoF Minutes Posted
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Nov 2009 01:44:48 -0000

> This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

--B_3340953897_771082
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

Draft minutes are posted at
http://www.ietf.org/proceedings/09nov/minutes/homegate.htm

Please advise of major errors or omissions.  I=B9m taking next week off to se=
e
Japan and when I return to the office, I=B9ll get to work on our next steps
based on feedback at the BoF.

Thanks to everyone who presented at the BoF, spoke at the microphone, or
attended!

Jason

--B_3340953897_771082
Content-type: text/html;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>Draft HomeGate BoF Minutes Posted</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt=
'>Draft minutes are posted at <a href=3D"http://www.ietf.org/proceedings/09nov=
/minutes/homegate.htm">http://www.ietf.org/proceedings/09nov/minutes/homegat=
e.htm</a><BR>
<BR>
Please advise of major errors or omissions. &nbsp;I&#8217;m taking next wee=
k off to see Japan and when I return to the office, I&#8217;ll get to work o=
n our next steps based on feedback at the BoF.<BR>
<BR>
Thanks to everyone who presented at the BoF, spoke at the microphone, or at=
tended!<BR>
<BR>
Jason</SPAN></FONT>
</BODY>
</HTML>


--B_3340953897_771082--


From mbaugher@cisco.com  Fri Nov 13 08:33:58 2009
Return-Path: <mbaugher@cisco.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 733253A67F4 for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 08:33:58 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.598
X-Spam-Level: 
X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EEu0kr6qsu32 for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 08:33:57 -0800 (PST)
Received: from sj-iport-1.cisco.com (sj-iport-1.cisco.com [171.71.176.70]) by core3.amsl.com (Postfix) with ESMTP id 4784A3A67D3 for <homegate@ietf.org>; Fri, 13 Nov 2009 08:33:57 -0800 (PST)
Authentication-Results: sj-iport-1.cisco.com; dkim=neutral (message not signed) header.i=none
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AikFAFAZ/UqrR7Ht/2dsb2JhbACCITC7UJgbhDwE
X-IronPort-AV: E=Sophos;i="4.44,738,1249257600";  d="scan'208,217";a="270829364"
Received: from sj-core-1.cisco.com ([171.71.177.237]) by sj-iport-1.cisco.com with ESMTP; 13 Nov 2009 16:34:27 +0000
Received: from xbh-sjc-221.amer.cisco.com (xbh-sjc-221.cisco.com [128.107.191.63]) by sj-core-1.cisco.com (8.13.8/8.14.3) with ESMTP id nADGYRGE004689; Fri, 13 Nov 2009 16:34:27 GMT
Received: from xfe-sjc-212.amer.cisco.com ([171.70.151.187]) by xbh-sjc-221.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959);  Fri, 13 Nov 2009 08:34:27 -0800
Received: from sjc-mbaugher-8711.cisco.com ([10.19.93.34]) by xfe-sjc-212.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959);  Fri, 13 Nov 2009 08:34:26 -0800
References: <C722E929.16C5C%jason_livingood@cable.comcast.com>
In-Reply-To: <C722E929.16C5C%jason_livingood@cable.comcast.com>
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: multipart/alternative; boundary=Apple-Mail-76-649991691
Message-Id: <03BA788B-2BD3-48F5-B623-1A89BC31E486@cisco.com>
From: Mark Baugher <mbaugher@cisco.com>
Date: Fri, 13 Nov 2009 08:34:25 -0800
To: Jason Livingood <jason_livingood@cable.comcast.com>
X-Mailer: Apple Mail (2.1076)
X-OriginalArrivalTime: 13 Nov 2009 16:34:26.0913 (UTC) FILETIME=[2A70A510:01CA647F]
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Draft HomeGate BoF Minutes Posted
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Nov 2009 16:33:58 -0000

--Apple-Mail-76-649991691
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252;
	format=flowed;
	delsp=yes

Thanks, from the meeting summary, I read:
"
Many problems come from manufacturers that do not care: small, cheap,
and inexperienced. Our work will not influence them. It is important
to reach folks writing RFPs and doing conformance testing.

Can we just advise people as to how to use IETF protocols? Forums,
like Broadband Forum, are doing requirements. However, there is a need
for guidelines."

Much of what gets shipped in the small, cheap routers
is selected by chip vendors whose product is then
picked up by contract manufacturers, mostly in
Asia.  The first step is to publish open source
code, I doubt if guidelines will help - at least for
the low-end products.

Mark
On 12/11/2009, at 5:44 PM, Jason Livingood wrote:

> Draft minutes are posted at =
http://www.ietf.org/proceedings/09nov/minutes/homegate.htm
>
> Please advise of major errors or omissions.  I=92m taking next week =20=

> off to see Japan and when I return to the office, I=92ll get to work =20=

> on our next steps based on feedback at the BoF.
>
> Thanks to everyone who presented at the BoF, spoke at the =20
> microphone, or attended!
>
> Jason
> _______________________________________________
> homegate mailing list
> homegate@ietf.org
> https://www.ietf.org/mailman/listinfo/homegate


--Apple-Mail-76-649991691
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=windows-1252

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
">Thanks, from the meeting summary, I read:<div>"<span =
class=3D"Apple-style-span" style=3D"font-family: Calibri; font-size: =
15px; "><br>Many problems come from manufacturers that do not care: =
small, cheap,&nbsp;<br>and inexperienced. Our work will not influence =
them. It is important&nbsp;<br>to reach folks writing RFPs and doing =
conformance testing.<br><br>Can we just advise people as to how to use =
IETF protocols? Forums,&nbsp;<br>like Broadband Forum, are doing =
requirements. However, there is a need&nbsp;<br>for =
guidelines."<br></span><div><div><br></div><div>Much of what gets =
shipped in the small, cheap routers</div><div>is selected by chip =
vendors whose product is then</div><div>picked up by contract =
manufacturers, mostly in&nbsp;</div><div>Asia. &nbsp;The first step is =
to publish open source</div><div>code, I doubt if guidelines will help - =
at least for</div><div>the low-end =
products.</div><div><br></div><div>Mark</div><div>On 12/11/2009, at 5:44 =
PM, Jason Livingood wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div>
<font face=3D"Calibri, Verdana, Helvetica, Arial"><span =
style=3D"font-size:11pt">Draft minutes are posted at <a =
href=3D"http://www.ietf.org/proceedings/09nov/minutes/homegate.htm">http:/=
/www.ietf.org/proceedings/09nov/minutes/homegate.htm</a><br>
<br>
Please advise of major errors or omissions. &nbsp;I=92m taking next week =
off to see Japan and when I return to the office, I=92ll get to work on =
our next steps based on feedback at the BoF.<br>
<br>
Thanks to everyone who presented at the BoF, spoke at the microphone, or =
attended!<br>
<br>
Jason</span></font>
</div>


_______________________________________________<br>homegate mailing =
list<br><a =
href=3D"mailto:homegate@ietf.org">homegate@ietf.org</a><br>https://www.iet=
f.org/mailman/listinfo/homegate<br></blockquote></div><br></div></body></h=
tml>=

--Apple-Mail-76-649991691--

From iljitsch@muada.com  Fri Nov 13 08:43:43 2009
Return-Path: <iljitsch@muada.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 884CC3A67D7 for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 08:43:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TVnDMhJCGuFr for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 08:43:43 -0800 (PST)
Received: from sequoia.muada.com (unknown [IPv6:2001:1af8:2:5::2]) by core3.amsl.com (Postfix) with ESMTP id B14A63A67A1 for <homegate@ietf.org>; Fri, 13 Nov 2009 08:43:42 -0800 (PST)
Received: from claw.it.uc3m.es (claw.it.uc3m.es [163.117.139.216]) (authenticated bits=0) by sequoia.muada.com (8.13.3/8.13.3) with ESMTP id nADGhsnv058890 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 13 Nov 2009 17:43:55 +0100 (CET) (envelope-from iljitsch@muada.com)
Mime-Version: 1.0 (Apple Message framework v1077)
Content-Type: text/plain; charset=us-ascii
From: Iljitsch van Beijnum <iljitsch@muada.com>
In-Reply-To: <03BA788B-2BD3-48F5-B623-1A89BC31E486@cisco.com>
Date: Fri, 13 Nov 2009 17:43:59 +0100
Content-Transfer-Encoding: 7bit
Message-Id: <F10833AC-98D2-49AE-B447-F4023AC6C9CD@muada.com>
References: <C722E929.16C5C%jason_livingood@cable.comcast.com> <03BA788B-2BD3-48F5-B623-1A89BC31E486@cisco.com>
To: Mark Baugher <mbaugher@cisco.com>
X-Mailer: Apple Mail (2.1077)
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Draft HomeGate BoF Minutes Posted
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Nov 2009 16:43:43 -0000

On 13 nov 2009, at 17:34, Mark Baugher wrote:

> The first step is to publish open source
> code, I doubt if guidelines will help - at least for
> the low-end products.

How do you write the code without the guidelines?

From mbaugher@cisco.com  Fri Nov 13 09:14:46 2009
Return-Path: <mbaugher@cisco.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B8E533A6807 for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 09:14:46 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.598
X-Spam-Level: 
X-Spam-Status: No, score=-6.598 tagged_above=-999 required=5 tests=[AWL=0.001,  BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2Y524wK1BfFJ for <homegate@core3.amsl.com>; Fri, 13 Nov 2009 09:14:46 -0800 (PST)
Received: from sj-iport-2.cisco.com (sj-iport-2.cisco.com [171.71.176.71]) by core3.amsl.com (Postfix) with ESMTP id 1F2BD3A67F4 for <homegate@ietf.org>; Fri, 13 Nov 2009 09:14:46 -0800 (PST)
Authentication-Results: sj-iport-2.cisco.com; dkim=neutral (message not signed) header.i=none
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApoEAOIi/UqrRN+K/2dsb2JhbAC+CZgYhDwE
X-IronPort-AV: E=Sophos;i="4.44,738,1249257600"; d="scan'208";a="224078206"
Received: from sj-core-4.cisco.com ([171.68.223.138]) by sj-iport-2.cisco.com with ESMTP; 13 Nov 2009 17:15:16 +0000
Received: from xbh-sjc-211.amer.cisco.com (xbh-sjc-211.cisco.com [171.70.151.144]) by sj-core-4.cisco.com (8.13.8/8.14.3) with ESMTP id nADHFGWh004370; Fri, 13 Nov 2009 17:15:16 GMT
Received: from xfe-sjc-211.amer.cisco.com ([171.70.151.174]) by xbh-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959);  Fri, 13 Nov 2009 09:15:16 -0800
Received: from sjc-mbaugher-8711.cisco.com ([10.19.93.34]) by xfe-sjc-211.amer.cisco.com with Microsoft SMTPSVC(6.0.3790.3959);  Fri, 13 Nov 2009 09:15:15 -0800
References: <C722E929.16C5C%jason_livingood@cable.comcast.com> <03BA788B-2BD3-48F5-B623-1A89BC31E486@cisco.com> <F10833AC-98D2-49AE-B447-F4023AC6C9CD@muada.com>
In-Reply-To: <F10833AC-98D2-49AE-B447-F4023AC6C9CD@muada.com>
Mime-Version: 1.0 (Apple Message framework v1076)
Content-Type: text/plain; charset=us-ascii; format=flowed
Message-Id: <9BBA35E5-98C8-4AB1-B7AF-896956FF9B63@cisco.com>
Content-Transfer-Encoding: 7bit
From: Mark Baugher <mbaugher@cisco.com>
Date: Fri, 13 Nov 2009 09:15:14 -0800
To: Iljitsch van Beijnum <iljitsch@muada.com>
X-Mailer: Apple Mail (2.1076)
X-OriginalArrivalTime: 13 Nov 2009 17:15:15.0822 (UTC) FILETIME=[DE1A80E0:01CA6484]
Cc: "homegate@ietf.org" <homegate@ietf.org>
Subject: Re: [homegate] Draft HomeGate BoF Minutes Posted
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 13 Nov 2009 17:14:46 -0000

On 13/11/2009, at 8:43 AM, Iljitsch van Beijnum wrote:

> On 13 nov 2009, at 17:34, Mark Baugher wrote:
>
>> The first step is to publish open source
>> code, I doubt if guidelines will help - at least for
>> the low-end products.
>
> How do you write the code without the guidelines?

Or specs.  Yes we need those.  My point is that publishing
guidelines or specifications alone will not get code into
low-end routers.

Mark


From lars.eggert@nokia.com  Wed Nov 18 06:22:26 2009
Return-Path: <lars.eggert@nokia.com>
X-Original-To: homegate@core3.amsl.com
Delivered-To: homegate@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id E99B03A67A6 for <homegate@core3.amsl.com>; Wed, 18 Nov 2009 06:22:26 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YUwW7S4UFJd3 for <homegate@core3.amsl.com>; Wed, 18 Nov 2009 06:22:26 -0800 (PST)
Received: from mail.fit.nokia.com (mail.fit.nokia.com [195.148.124.195]) by core3.amsl.com (Postfix) with ESMTP id DEA4B3A6889 for <homegate@ietf.org>; Wed, 18 Nov 2009 06:22:25 -0800 (PST)
Received: from [IPv6:2001:708:40:fff2:225:ff:fe45:eccf] (lars.local [IPv6:2001:708:40:fff2:225:ff:fe45:eccf] (may be forged)) (authenticated bits=0) by mail.fit.nokia.com (8.14.3/8.14.3) with ESMTP id nAIEMGse035251 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for <homegate@ietf.org>; Wed, 18 Nov 2009 16:22:16 +0200 (EET) (envelope-from lars.eggert@nokia.com)
From: Lars Eggert <lars.eggert@nokia.com>
Content-Type: multipart/signed; boundary=Apple-Mail-62--1073421325; protocol="application/pkcs7-signature"; micalg=sha1
Date: Wed, 18 Nov 2009 16:22:16 +0200
Message-Id: <72659CD9-9906-4220-ACCC-A2B07A7DD1EA@nokia.com>
To: homegate@ietf.org
Mime-Version: 1.0 (Apple Message framework v1077)
X-Mailer: Apple Mail (2.1077)
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.3 (mail.fit.nokia.com [IPv6:2001:708:40:fff1::1]); Wed, 18 Nov 2009 16:22:16 +0200 (EET)
Subject: [homegate] BOF follow-up
X-BeenThere: homegate@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Broadband Home Gateway Discussion <homegate.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/homegate>
List-Post: <mailto:homegate@ietf.org>
List-Help: <mailto:homegate-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/homegate>, <mailto:homegate-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Nov 2009 14:22:27 -0000

--Apple-Mail-62--1073421325
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hi,

I wanted to send a quick follow-up now that the BOF has happened, to let =
you all know where I think we are.

My take away was that the BOF went well, and there is interest and =
energy in the community to take on work in this space. We did have =
people from ISPs, vendors, etc. in the room, which was promising.

It's not crystal clear, however, what exactly the scope of such a =
working group would be. We've made progress on some scoping questions =
(how to handle v4 and v6, etc.), but on other questions we need to do =
some more thinking. We also need to better understand how an IETF effort =
can complement the efforts of other groups in this space.

I'd like to ask the BOF chairs (Jason and Dave) to continue to lead the =
needed follow-up discussion that will get us to the point where HOMEGATE =
can be chartered.

Lars=

--Apple-Mail-62--1073421325
Content-Disposition: attachment;
	filename=smime.p7s
Content-Type: application/pkcs7-signature;
	name=smime.p7s
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGbDCCAyUw
ggKOoAMCAQICEAdjk36sXKbnVn15S0/qUp0wDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UEBhMCWkEx
JTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQ
ZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA5MDYxNTExMjYxNFoXDTEwMDYxNTExMjYx
NFowXDEPMA0GA1UEBBMGRWdnZXJ0MQ0wCwYDVQQqEwRMYXJzMRQwEgYDVQQDEwtMYXJzIEVnZ2Vy
dDEkMCIGCSqGSIb3DQEJARYVbGFycy5lZ2dlcnRAbm9raWEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEA7mR8A+Pn0/FsUkMX6Pyjw+FL3IFcJk8GaKV5VJ40TMI0Wh8oq20cqA9X
uqnVDW9WztKwH+o+msJenLwWpprbpJm4TImYGbnUJxYyN8gb81aiX1Bw2xCpJ5z3H2+8DsReJLuY
Rdl4bVvaIxLIL4odmfsRwzPyNkOK8LRtfl6OPcaDOlFWzbikULfIVGGu7BqK4lxQSpYwwpZkOMOB
6nnBSfUOtBEmqO+qZG/nL/JxWFV5vxQgg4XHbsMMTxFf6+ji18BD09BUIfDLTuJoCzFmQhrM9vLT
VuRhHWSL20LoafGjXv6mPt3i9IGJHpVb2dMQUgOgRyWHTKiUJVU/rUTdWwIDAQABo14wXDAqBgUr
ZQEEAQQhMB8CAQAwGjAYAgEEBBNMMnVNeWZmQk5VYk5KSmNkWjJzMCAGA1UdEQQZMBeBFWxhcnMu
ZWdnZXJ0QG5va2lhLmNvbTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBADUx+67n98wt
I1vydB90HeSZP4Y64VCxxb0NxGGFvfc2+JdVKeHJ/xT+l+ygYKsWNwJJprkPi4WZ5G0crkq4VK1H
5drEJIztpSPVfWI05vPidaaGuuuCR+6MvJMtOTEYEvc/6eovBnkrzRf9x5x5EyuJXAWTeuBADg80
QI3vQ1tZMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTAT
BgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUg
Q29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIG
A1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25h
bC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjEL
MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNV
BAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK
MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7
n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAw
QwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJl
ZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRl
TGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9M
Ibj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowg
T2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCAxAwggMMAgEBMHYw
YjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq
BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhAHY5N+rFym51Z9eUtP
6lKdMAkGBSsOAwIaBQCgggFvMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkF
MQ8XDTA5MTExODE0MjIxNlowIwYJKoZIhvcNAQkEMRYEFEZ7yekP1EX5WLQlK9Q53Lz+rmJZMIGF
BgkrBgEEAYI3EAQxeDB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGlu
ZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD
QQIQB2OTfqxcpudWfXlLT+pSnTCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUw
IwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVy
c29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQB2OTfqxcpudWfXlLT+pSnTANBgkqhkiG9w0BAQEF
AASCAQCndnpDYB/vq9AeiREiBwI3FEp4fKm6IPEC0Gw+tF1tY9YWt56bQLxPJBvKAAl76u5PK1g/
hQSjCM93QZQTuupE4YXmRiEf5j9pcxPARqEx+uia3TF7yoN0vhQ51CZW/COrNOm94a52Jqu5VhAI
shKEixnoQ+6Qb3TZI/r6FIdZJvG53chtxCFqaz/56GZ6pPiRJ4YH11P3kWRLoFn6G9wE9+YTuzRV
b+urDLQgkIkJV0ODo0tufhf+KlLLoKdzNteZ60WyDXOXszIxN26FrCnrCMnv8jB/6Owujal83xqZ
12dBR4LxCKipsVBSQsuJI10IXtOPr8UKIZ46s2C4UzFvAAAAAAAA

--Apple-Mail-62--1073421325--