
From wes@mti-systems.com  Wed Mar 13 19:03:29 2013
Return-Path: <wes@mti-systems.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A29C121F8E46 for <aqm@ietfa.amsl.com>; Wed, 13 Mar 2013 19:03:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.486
X-Spam-Level: 
X-Spam-Status: No, score=-2.486 tagged_above=-999 required=5 tests=[AWL=0.114,  BAYES_00=-2.599]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CJVlHyKcqHwZ for <aqm@ietfa.amsl.com>; Wed, 13 Mar 2013 19:03:29 -0700 (PDT)
Received: from atl4mhob14.myregisteredsite.com (atl4mhob14.myregisteredsite.com [209.17.115.52]) by ietfa.amsl.com (Postfix) with ESMTP id BBA5B21F8E41 for <aqm@ietf.org>; Wed, 13 Mar 2013 19:03:25 -0700 (PDT)
Received: from mailpod.hostingplatform.com (mail.networksolutionsemail.com [205.178.146.50]) by atl4mhob14.myregisteredsite.com (8.14.4/8.14.4) with ESMTP id r2E23Ocq000477 for <aqm@ietf.org>; Wed, 13 Mar 2013 22:03:24 -0400
Received: (qmail 12204 invoked by uid 0); 14 Mar 2013 02:03:23 -0000
Received: from unknown (HELO ?10.5.0.6?) (wes@mti-systems.com@46.21.99.29) by 0 with ESMTPA; 14 Mar 2013 02:03:23 -0000
Message-ID: <51412FD8.8010908@mti-systems.com>
Date: Wed, 13 Mar 2013 22:03:04 -0400
From: Wesley Eddy <wes@mti-systems.com>
Organization: MTI Systems
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130307 Thunderbird/17.0.4
MIME-Version: 1.0
To: aqm@ietf.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Subject: [aqm] TEST
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Mar 2013 02:03:29 -0000

Test of the new list.  Please ignore.

-- 
Wes Eddy
MTI Systems

From fred@cisco.com  Thu Mar 14 04:55:25 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AD8B121F8F2F; Thu, 14 Mar 2013 04:55:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -107.773
X-Spam-Level: 
X-Spam-Status: No, score=-107.773 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tDgQ8m2OdOYO; Thu, 14 Mar 2013 04:55:21 -0700 (PDT)
Received: from rcdn-iport-9.cisco.com (rcdn-iport-9.cisco.com [173.37.86.80]) by ietfa.amsl.com (Postfix) with ESMTP id 650F921F8F22; Thu, 14 Mar 2013 04:55:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=6852; q=dns/txt; s=iport; t=1363262121; x=1364471721; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=dwdOr4tcuxvoFIoCPmTmnmHrI6y2T9QHLZopC/8dHTM=; b=nKhGIx9lteeLOIPlD6Hk+06xML3zKE8tHtHUdQPLITRSwv1+oAB9ku5j Ju6VQ9Rnp4S2c+/riW7iksdzz1wnaCmMo5z1QqC+0bJM2CjH3ut2qCAcX XrmxlB0qtqSc/ZipVJFDysZTR/nRPeZRCbZPycQW3MM3r4VbAG3DgTqXT A=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgAFAH65QVGtJV2a/2dsb2JhbABDxHmBYRZ0gikBAQECAQFsCwIQAgEIDgoKJDIlAgQOBQiIBgbBLY5dAjEHgl9hA4g+ilWUR4FUgTaCKA
X-IronPort-AV: E=Sophos;i="4.84,844,1355097600"; d="scan'208";a="184402837"
Received: from rcdn-core-3.cisco.com ([173.37.93.154]) by rcdn-iport-9.cisco.com with ESMTP; 14 Mar 2013 11:55:20 +0000
Received: from xhc-rcd-x13.cisco.com (xhc-rcd-x13.cisco.com [173.37.183.87]) by rcdn-core-3.cisco.com (8.14.5/8.14.5) with ESMTP id r2EBtK9C029555 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 14 Mar 2013 11:55:20 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-rcd-x13.cisco.com ([173.37.183.87]) with mapi id 14.02.0318.004; Thu, 14 Mar 2013 06:55:10 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: Michael Welzl <michawe@ifi.uio.no>
Thread-Topic: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt
Thread-Index: AQHOIAqa1tJ18EAwSkCm0kPJ4goPbg==
Date: Thu, 14 Mar 2013 11:55:09 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7D1BBA@xmb-rcd-x09.cisco.com>
References: <20130313164833.27324.89314.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D045F@xmb-rcd-x09.cisco.com> <b6d8c6a56b1e4f066019c81da8a202a0.squirrel@www.erg.abdn.ac.uk> <8C48B86A895913448548E6D15DA7553B7D0BD6@xmb-rcd-x09.cisco.com> <fc97d79292a1cfb2efab45753f90351b.squirrel@www.erg.abdn.ac.uk> <4936E58C-A6AE-485D-BD6E-7E6D30493359@ifi.uio.no> <8C48B86A895913448548E6D15DA7553B7D1A35@xmb-rcd-x09.cisco.com>
In-Reply-To: <8C48B86A895913448548E6D15DA7553B7D1A35@xmb-rcd-x09.cisco.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.82.243.95]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <31CD23BE7ECB3444A6BF64A5C1829D6C@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "tsv-area@ietf.org" <TSV-area@ietf.org>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] [tsvwg] New Version Notification for	draft-baker-tsvwg-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Mar 2013 11:55:25 -0000

This will be my last note on the topic on tsvwg. I'd like to believe that b=
y tomorrow we will have all joined aqm@.

On Mar 14, 2013, at 6:54 AM, Fred Baker (fred) <fred@cisco.com> wrote:

>=20
> On Mar 14, 2013, at 12:43 AM, Michael Welzl <michawe@ifi.uio.no> wrote:
>=20
>> Great, I turn on ECN, and that gives me more delay, just what I want!
>> And, even better: the more people use ECN, the more delay everybody gets=
.
>>=20
>> Seriously, I see the incentive idea behind this two-level marking idea, =
but one has to carefully consider the increased delay vs. gained throughput=
 trade-off in such a scheme.
>=20
> I don't understand your comment. Fill me in please?
>=20
> If I have a tail-drop queue, TCP using CUBIC or New Reno seeks to keep th=
e queue full. If the queue has N positions (there are many ways to measure =
a queue, bear with this one for the purpose of the example), worst case del=
ay approximates waiting for N messages, and general case delay is predicted=
 by queuing theory to shoot for N when the link has 95% utilization or what=
ever.
>=20
> With any AQM algorithm, the same queue will accept N messages in the even=
t that it gets a burst, but will start marking/dropping at some point M sig=
nificantly less than N, so that the queue depth tends to approximate M, not=
 N. That's the whole point of any AQM algorithm. How M is chosen or predict=
ed is of course different for various algorithms.
>=20
> The pushback I generally hear about ECN is that people will mark traffic =
as ECN-capable in order to work around AQM algorithms signaling early; to m=
ake them signal later. I am told that people do abuse the EF code point in =
order to make their traffic go into a priority queue, so I can imagine peop=
le doing this as well.
>=20
> What I am suggesting is that the AQM algorithm use the appropriate signal=
 to make queue depth approximate M, whether than is ECN or loss depending o=
n the traffic marking, and in the event of abuse do no worse than present i=
t's-done-everywhere tail drop, in which the queue depth tends towards N.
>=20
> M < N.
>=20
> Remind me how saying that one will use ECN for ECN-capable traffic makes =
the average queue depth deeper? That makes no sense to me.


Following up on the comment about M and N. Let me show you some (old and lo=
w speed) slides that I use as examples in this class of discussion. It's no=
t ECN vs loss, it's AQM vs tail-drop. However, since AQM-using-ECN and AQM-=
using-loss are examples of AQM, and the argument in the draft is to use bot=
h depending on the capabilities of the traffic, I think it's applicable.

    ftp://ftpeng.cisco.com/fred/dpreed/Bufferbloat_Masterclass.pdf

I'm looking at slides 11 and 12. Comments on the rest of the deck are inter=
esting, but I'd suggest taking that private in order to not muddy this disc=
ussion.

The slides were pulled together as a simple and understandable example of w=
hat I said above about M and N. "AQM", in this case, is "RED"; other algori=
thms will display slightly different characteristics, but I assert that the=
y will have the same basic behavior. The value and perhaps appearance of M =
will vary, but not the fact of it.

The test setup is two hosts talking across two low end routers connected wi=
th a 2 MBPS link. I could go faster; the charts would display timings appro=
priate to the line speeds and I would have to generate some multiple of the=
 traffic to present the issue, but the structure of the behavior would be a=
bout the same.=20

     +----+   +------+         +------+   +-----+
     |Left|   |Left  +---------+Right |   |Right|
     |Host|   |Router|         |Router|   |Host |
     +--+-+   +---+--+         +---+--+   +---+-+
        |         |                |          |
    ----+---------+---         ----+----------+---

On the "Left Host", I have a large file, large enough to take perhaps ten s=
econds to move to the "Right Host" in isolation. In one window on "Left", I=
 run a ping -s to measure the RTT of traffic between the two hosts, and by =
extension the changing depth of the queue in "Left Router". In another wind=
ow, I run a script. The script counts from one to fifteen. On each value, i=
t simultaneously opens up that number of FTPs to copy said file to "Right H=
ost". When the last FTP in the set completes, it waits a few seconds, rsh's=
 to delete the now-unneeded copies from "Right", and goes on to the next va=
lue.

Afterwards, I did some simple calculations in Excel. I took successive and =
overlapping sets of ping measurements (ten, IIRC) and calculated the min, m=
ax, average (probably should have been median), and standard deviation of t=
he set. I then plotted the values.

I ran the test twice, the difference being that I used RED or tail drop.

Slide 11 is the tail drop test. You see 15 bumps, measuring traffic from 1.=
.15 FTPs moving said file in parallel. What is pretty obvious is that the v=
ariation in the queue is at the top of the queue; if N was five, we would b=
e discussing variation around five; if it was five hundred, we would be dis=
cussing variation around five hundred. loss-based TCPs work Really Hard to =
fill the queue and keep it full, resulting in the potential queue depth bei=
ng a predictor of end to end delay and variation in delay.

Slide 12 is the RED test. It's pretty obvious where I set min-thresh. I'm n=
ot, in this, suggesting a value for min-thresh, BTW; I set it somewhere to =
demonstrate that where I set it is the average queue depth in the scenario.=
 You want it somewhere else, be my guest. What I observe is wider variation=
 in queue depth, because I am using the configured depth of the queue to ab=
sorb bursts when they happen. But the average queue depth is at min-thresho=
ld. M, which is less than N.

I'll argue, and I think we probably generally agree, that the optimum targe=
t average queue depth is close to zero, and we'd like to use the buffer ava=
ilable to absorb and play out bursts when they happen. It would also be nic=
e if the impact of AQM on the various data flows were more consistent, so t=
hat there was less variation in delay, and that might be true of some algor=
ithms. But that's the target.

Coming back to ECN vs loss, ECN doesn't increase delay, it enables the same=
 signal to be sent end to end, but sent more quickly and reliably (it is ex=
plicit and is known to be about congestion, as opposed to being implicit an=
d possibly triggered by other factors). It targets the same mean queue dept=
h. Signaling without losing traffic is better than losing traffic and calli=
ng that signaling, as the session completes more quickly and reliably and i=
s then out of the way. I don't see the down side.=

From rs@netapp.com  Thu Mar 14 04:57:52 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 40DBA21F8A99 for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 04:57:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -7.773
X-Spam-Level: 
X-Spam-Status: No, score=-7.773 tagged_above=-999 required=5 tests=[RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GhnxwGGAaI5L for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 04:57:51 -0700 (PDT)
Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) by ietfa.amsl.com (Postfix) with ESMTP id C11AC21F8AD8 for <aqm@ietf.org>; Thu, 14 Mar 2013 04:57:48 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,844,1355126400"; d="scan'208";a="30737880"
Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx12-out.netapp.com with ESMTP; 14 Mar 2013 04:57:48 -0700
Received: from vmwexceht02-prd.hq.netapp.com (vmwexceht02-prd.hq.netapp.com [10.106.76.240]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2EBvmFW004445; Thu, 14 Mar 2013 04:57:48 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht02-prd.hq.netapp.com ([10.106.76.240]) with mapi id 14.02.0342.003; Thu, 14 Mar 2013 04:57:48 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: Wesley Eddy <wes@mti-systems.com>, "Fred Baker (fred)" <fred@cisco.com>
Thread-Topic: [tsvwg] Thoughts on AQM
Thread-Index: AQHOHxX3G8oJZ19JKUOZfInmP/1YdpiibgcAgAKpQUA=
Date: Thu, 14 Mar 2013 11:57:47 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24A9C46D@SACEXCMBX02-PRD.hq.netapp.com>
References: <8C48B86A895913448548E6D15DA7553B7CBA3B@xmb-rcd-x09.cisco.com> <513F1C9E.4040506@mti-systems.com>
In-Reply-To: <513F1C9E.4040506@mti-systems.com>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.104.60.116]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] [tsvwg] Thoughts on AQM
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Mar 2013 11:57:52 -0000

FWIW:

I think, such a document should not be based on 2309 in text, but definitel=
y in spirit.


(updates/obsoletes 2309 yes, but not a 2309bis draft).


Richard Scheffenegger


> -----Original Message-----
> From: tsv-area-bounces@ietf.org [mailto:tsv-area-bounces@ietf.org] On
> Behalf Of Wesley Eddy
> Sent: Dienstag, 12. M=E4rz 2013 13:17
> To: Fred Baker (fred)
> Cc: tsv-area@ietf.org
> Subject: Re: [tsvwg] Thoughts on AQM
>=20
> On 3/12/2013 7:37 AM, Fred Baker (fred) wrote:
> >
> > RFC 2309 recommends AQM deployment, but doesn't require a specific
> > algorithm. That is good, because the prototypical algorithm of the day
> > (RED) is difficult to use. Van and Kathy have suggested CoDel, Rong
> > Pan and I are investigating another algorithm called PIE, and there
> > has been ample research on other algorithms. In my opinion,
> > recommendations of specific algorithms are wisely kept off the table;
> > they make future innovation difficult.
> >
>=20
>=20
> I agree, though we certainly would like some known good algorithm(s),
> hopefully free of IPR issues, to be available to implementations that
> don't want to invent their own algorithm.
>=20
> There are also probably some algorithms that are not so good, and it will
> be useful to know how that can be identified by people developing /
> working with them.
>=20
>=20
> > What I think may be of value - and would be willing to contribute to
> > in tsvwg - is a set of recommendations regarding the technical use of
> > AQM and congestion control. That document might recommend, for
> > example, that
> >
> > a) Operational deployments SHOULD use AQM procedures.
> >
> > b) Deployed AQM SHOULD use ECN as well as loss, and set thresholds to
> > mark traffic earlier than it is lost. In this way, congestion control
> > can be achieved in the general case without loss, and loss remains the
> > backup when needed.
> >
> > c) AQM algorithms deployed SHOULD not require operational (manual)
> > tuning.
> >
> > d) AQM algorithms deployed SHOULD be effective on all common Internet
> > traffic, including traffic that uses TCP, SCTP, UDP, and DCCP as
> > transports.
> >
> > e) TCP congestion control algorithms SHOULD maximize their use of
> > available bandwidth without incurring loss or undue delay when
> > possible. In other words, in Jain's language, the TCP/SCTP effective
> > window wants to be greater than or equal to the knee, but less than
> > the cliff.
> >
> > There may also be other recommendations, but these address most of the
> > issues I see.
>=20
>=20
> This is very helpful to hear.  This sounds like 2309bis to me, since
> (a) is basically what 2309 already says, and I see b and c as basically
> incorporating things that have happened since then (ECN and learning that
> auto-tuning is good).  Do you see it as an update to 2309?
>=20
>=20
> --
> Wes Eddy
> MTI Systems

From michawe@ifi.uio.no  Thu Mar 14 05:17:42 2013
Return-Path: <michawe@ifi.uio.no>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 179A321F8E0F for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 05:17:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -99.773
X-Spam-Level: 
X-Spam-Status: No, score=-99.773 tagged_above=-999 required=5 tests=[SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7ACzoyc2cX5T for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 05:17:41 -0700 (PDT)
Received: from mail-out5.uio.no (mail-out5.uio.no [IPv6:2001:700:100:10::17]) by ietfa.amsl.com (Postfix) with ESMTP id 6AD4E21F8AD8 for <aqm@ietf.org>; Thu, 14 Mar 2013 05:17:41 -0700 (PDT)
Received: from mail-mx6.uio.no ([129.240.10.40]) by mail-out5.uio.no with esmtp (Exim 4.80.1) (envelope-from <michawe@ifi.uio.no>) id 1UG76K-0006iC-Om; Thu, 14 Mar 2013 13:17:40 +0100
Received: from [130.129.130.161] by mail-mx6.uio.no with esmtpsa (TLSv1:AES128-SHA:128) user michawe (Exim 4.80.1) (envelope-from <michawe@ifi.uio.no>) id 1UG76K-0002dm-3Q; Thu, 14 Mar 2013 13:17:40 +0100
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
From: Michael Welzl <michawe@ifi.uio.no>
In-Reply-To: <8C48B86A895913448548E6D15DA7553B7D1BBA@xmb-rcd-x09.cisco.com>
Date: Thu, 14 Mar 2013 08:17:38 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <01F8559A-EE2E-4BA3-AB4E-D4D90A608ED5@ifi.uio.no>
References: <20130313164833.27324.89314.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D045F@xmb-rcd-x09.cisco.com> <b6d8c6a56b1e4f066019c81da8a202a0.squirrel@www.erg.abdn.ac.uk> <8C48B86A895913448548E6D15DA7553B7D0BD6@xmb-rcd-x09.cisco.com> <fc97d79292a1cfb2efab45753f90351b.squirrel@www.erg.abdn.ac.uk> <4936E58C-A6AE-485D-BD6E-7E6D30493359@ifi.uio.no> <8C48B86A895913448548E6D15DA7553B7D1A35@xmb-rcd-x09.cisco.com> <8C48B86A895913448548E6D15DA7553B7D1BBA@xmb-rcd-x09.cisco.com>
To: Fred Baker (fred) <fred@cisco.com>
X-Mailer: Apple Mail (2.1499)
X-UiO-SPF-Received: 
X-UiO-Ratelimit-Test: rcpts/h 5 msgs/h 2 sum rcpts/h 13 sum msgs/h 6 total rcpts 2843 max rcpts/h 20 ratelimit 0
X-UiO-Spam-info: not spam, SpamAssassin (score=-5.0, required=5.0, autolearn=disabled, UIO_MAIL_IS_INTERNAL=-5, uiobl=NO, uiouri=NO)
X-UiO-Scanned: 7FAE2EA546850A15EF9BAD69360985E44FB3C6D6
X-UiO-SPAM-Test: remote_host: 130.129.130.161 spam_score: -49 maxlevel 80 minaction 2 bait 0 mail/h: 1 total 1 max/h 1 blacklist 0 greylist 0 ratelimit 0
Cc: aqm@ietf.org
Subject: Re: [aqm] [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Mar 2013 12:17:42 -0000

Hi,

Thanks for all the details, this was an interesting read. Just to avoid =
further confusion:

> Coming back to ECN vs loss, ECN doesn't increase delay, it enables the =
same signal to be sent end to end, but sent more quickly and reliably =
(it is explicit and is known to be about congestion, as opposed to being =
implicit and possibly triggered by other factors). It targets the same =
mean queue depth. Signaling without losing traffic is better than losing =
traffic and calling that signaling, as the session completes more =
quickly and reliably and is then out of the way. I don't see the down =
side.

I know, and of course I agree - this was all a misunderstanding that =
happened because I had mis-read your scheme.

Cheers,
Michael


From fred@cisco.com  Thu Mar 14 06:14:51 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D110421F9067 for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 06:14:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.372
X-Spam-Level: 
X-Spam-Status: No, score=-110.372 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AXXOIUQlf6OS for <aqm@ietfa.amsl.com>; Thu, 14 Mar 2013 06:14:47 -0700 (PDT)
Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id 573E121F9060 for <aqm@ietf.org>; Thu, 14 Mar 2013 06:14:47 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3585; q=dns/txt; s=iport; t=1363266887; x=1364476487; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=Lt+OjTtvt0jJg6rglXxItK8pVyJWd6UvdNYQSS6N2eM=; b=bnQvioZ0yitYxA9oHEF3c/FEx3f818n8rNjv/ykYbA18dpJB3oN4HvU3 svh7tKHWMxcz9SWZVySR4uM9ylUAmbfZbQQxrvZH6xc+Gtui37RvNo7RW 48vQFSdzmSk7puNGcE7Qv/JW4Zl2as4sFj5iUVJR5I0AVg1H8ARe+unxR I=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgEFABHNQVGtJV2Z/2dsb2JhbABDxHuBYhZ0gioBAQEDAW0MBQcEAgEIEQQBAQEKHQcyFAkIAgQOBQiIBgbBKI1GgRcCJgsHBoJZYQOIPp8cgwqCKA
X-IronPort-AV: E=Sophos;i="4.84,845,1355097600"; d="scan'208";a="187420482"
Received: from rcdn-core-2.cisco.com ([173.37.93.153]) by rcdn-iport-7.cisco.com with ESMTP; 14 Mar 2013 13:14:37 +0000
Received: from xhc-aln-x06.cisco.com (xhc-aln-x06.cisco.com [173.36.12.80]) by rcdn-core-2.cisco.com (8.14.5/8.14.5) with ESMTP id r2EDEbTn020119 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 14 Mar 2013 13:14:37 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-aln-x06.cisco.com ([173.36.12.80]) with mapi id 14.02.0318.004; Thu, 14 Mar 2013 08:14:37 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: "Scheffenegger, Richard" <rs@netapp.com>
Thread-Topic: [tsvwg] Thoughts on AQM
Thread-Index: AQHOHxX3pgZSnl4VeEe2kH8X+bXbGpiibgcAgAKpQUCAAGoXgA==
Date: Thu, 14 Mar 2013 13:14:36 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7D1DAF@xmb-rcd-x09.cisco.com>
References: <8C48B86A895913448548E6D15DA7553B7CBA3B@xmb-rcd-x09.cisco.com> <513F1C9E.4040506@mti-systems.com> <012C3117EDDB3C4781FD802A8C27DD4F24A9C46D@SACEXCMBX02-PRD.hq.netapp.com>
In-Reply-To: <012C3117EDDB3C4781FD802A8C27DD4F24A9C46D@SACEXCMBX02-PRD.hq.netapp.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.82.243.94]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <F5F11A58103096409BECFE3A15E6149F@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: Wesley Eddy <wes@mti-systems.com>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] [tsvwg] Thoughts on AQM
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Mar 2013 13:14:51 -0000

On Mar 14, 2013, at 7:57 AM, "Scheffenegger, Richard" <rs@netapp.com>
 wrote:

>=20
> FWIW:
>=20
> I think, such a document should not be based on 2309 in text, but definit=
ely in spirit.
>=20
>=20
> (updates/obsoletes 2309 yes, but not a 2309bis draft).

Works for me. My first instinct was as you suggest, and I took this approac=
h partly at Wes' suggestion and partly because it was something I could hac=
k together in an hour yesterday morning. But if 2309 makes an algorithmic r=
ecommendation, that is at best dated. I'd rather not make a recommendation,=
 but instead agree on a set of requirements that the algorithm someone uses=
 should meet. That way, we leave room for improvements over time.

> Richard Scheffenegger
>=20
>=20
>> -----Original Message-----
>> From: tsv-area-bounces@ietf.org [mailto:tsv-area-bounces@ietf.org] On
>> Behalf Of Wesley Eddy
>> Sent: Dienstag, 12. M=E4rz 2013 13:17
>> To: Fred Baker (fred)
>> Cc: tsv-area@ietf.org
>> Subject: Re: [tsvwg] Thoughts on AQM
>>=20
>> On 3/12/2013 7:37 AM, Fred Baker (fred) wrote:
>>>=20
>>> RFC 2309 recommends AQM deployment, but doesn't require a specific
>>> algorithm. That is good, because the prototypical algorithm of the day
>>> (RED) is difficult to use. Van and Kathy have suggested CoDel, Rong
>>> Pan and I are investigating another algorithm called PIE, and there
>>> has been ample research on other algorithms. In my opinion,
>>> recommendations of specific algorithms are wisely kept off the table;
>>> they make future innovation difficult.
>>>=20
>>=20
>>=20
>> I agree, though we certainly would like some known good algorithm(s),
>> hopefully free of IPR issues, to be available to implementations that
>> don't want to invent their own algorithm.
>>=20
>> There are also probably some algorithms that are not so good, and it wil=
l
>> be useful to know how that can be identified by people developing /
>> working with them.
>>=20
>>=20
>>> What I think may be of value - and would be willing to contribute to
>>> in tsvwg - is a set of recommendations regarding the technical use of
>>> AQM and congestion control. That document might recommend, for
>>> example, that
>>>=20
>>> a) Operational deployments SHOULD use AQM procedures.
>>>=20
>>> b) Deployed AQM SHOULD use ECN as well as loss, and set thresholds to
>>> mark traffic earlier than it is lost. In this way, congestion control
>>> can be achieved in the general case without loss, and loss remains the
>>> backup when needed.
>>>=20
>>> c) AQM algorithms deployed SHOULD not require operational (manual)
>>> tuning.
>>>=20
>>> d) AQM algorithms deployed SHOULD be effective on all common Internet
>>> traffic, including traffic that uses TCP, SCTP, UDP, and DCCP as
>>> transports.
>>>=20
>>> e) TCP congestion control algorithms SHOULD maximize their use of
>>> available bandwidth without incurring loss or undue delay when
>>> possible. In other words, in Jain's language, the TCP/SCTP effective
>>> window wants to be greater than or equal to the knee, but less than
>>> the cliff.
>>>=20
>>> There may also be other recommendations, but these address most of the
>>> issues I see.
>>=20
>>=20
>> This is very helpful to hear.  This sounds like 2309bis to me, since
>> (a) is basically what 2309 already says, and I see b and c as basically
>> incorporating things that have happened since then (ECN and learning tha=
t
>> auto-tuning is good).  Do you see it as an update to 2309?
>>=20
>>=20
>> --
>> Wes Eddy
>> MTI Systems


From fred@cisco.com  Fri Mar 15 18:51:45 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B3B4011E8126 for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 18:51:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.448
X-Spam-Level: 
X-Spam-Status: No, score=-110.448 tagged_above=-999 required=5 tests=[AWL=-0.076, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hIZEbunryh+i for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 18:51:44 -0700 (PDT)
Received: from rcdn-iport-7.cisco.com (rcdn-iport-7.cisco.com [173.37.86.78]) by ietfa.amsl.com (Postfix) with ESMTP id 588B111E80C5 for <aqm@ietf.org>; Fri, 15 Mar 2013 18:51:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=1847; q=dns/txt; s=iport; t=1363398704; x=1364608304; h=from:to:subject:date:message-id:references:in-reply-to: content-id:content-transfer-encoding:mime-version; bh=r6lMj3kgldvWCg22XS8JQWZ4NQ7YdbP3gDLehNXGht4=; b=J15sPgEdHnuGXf4rQtmohG712iJZ5vpEEHaKtvvaf7jHYD3+DBt4IRNf 8WIR/P1YGpQWH3pkE1WVbl5o8mlvL/P4otbNLIVa+bejU6Ko8SScccHGH NHy7LWwJNLEeMajxRNMiEZvuxDz1Wv2lpm6o9wLCS6XiglyepwnsR7wQk k=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgEFANvOQ1GtJV2b/2dsb2JhbABDxSuBZxZtB4ImBAEBAQMBOj0SAgEIIhQQMiUCBBsBC4U1B4ImGAYMw1OOYgI4gl9hA5d6ikyFF4MKgig
X-IronPort-AV: E=Sophos;i="4.84,855,1355097600"; d="scan'208";a="188091468"
Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-7.cisco.com with ESMTP; 16 Mar 2013 01:51:43 +0000
Received: from xhc-aln-x12.cisco.com (xhc-aln-x12.cisco.com [173.36.12.86]) by rcdn-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id r2G1phQV022716 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for <aqm@ietf.org>; Sat, 16 Mar 2013 01:51:43 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-aln-x12.cisco.com ([173.36.12.86]) with mapi id 14.02.0318.004; Fri, 15 Mar 2013 20:51:43 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: "aqm@ietf.org" <aqm@ietf.org>
Thread-Topic: New Version Notification for draft-baker-aqm-recommendation-00.txt
Thread-Index: AQHOIeecW/sgvo3KdUWrIXfSwqOBQJin4Z6A
Date: Sat, 16 Mar 2013 01:51:43 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com>
References: <20130316014305.5419.25287.idtracker@ietfa.amsl.com>
In-Reply-To: <20130316014305.5419.25287.idtracker@ietfa.amsl.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.21.87.233]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C0C61D5184B8664CBBF8D89681C85138@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Mar 2013 01:51:45 -0000

Somewhat with my heart in my hands, given discussion on this list about tex=
t I botched and about RFC 2309's recommendation of RED, I have taken it upo=
n myself to do a wholesale replacement of RFC 2309. I'd appreciate people's=
 comments - especially from the original authors of RFC 2309.

For comparison and review purposes, you may like to look at a diff from RFC=
 2309. If so, go to
    http://tools.ietf.org/rfcdiff
and give it two URLs:
    http://www.rfc-editor.org/rfc/rfc2309.txt
    http://www.ietf.org/id/draft-baker-aqm-recommendation-00.txt

>=20
> A new version of I-D, draft-baker-aqm-recommendation-00.txt
> has been successfully submitted by Fred Baker and posted to the
> IETF repository.
>=20
> Filename:	 draft-baker-aqm-recommendation
> Revision:	 00
> Title:		 IETF Recommendations Regarding Active Queue Management
> Creation date:	 2013-03-15
> Group:		 Individual Submission
> Number of pages: 17
> URL:             http://www.ietf.org/internet-drafts/draft-baker-aqm-reco=
mmendation-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-baker-aqm-recommen=
dation
> Htmlized:        http://tools.ietf.org/html/draft-baker-aqm-recommendatio=
n-00
>=20
>=20
> Abstract:
>   This memo presents recommendations to the Internet community
>   concerning measures to improve and preserve Internet performance.  It
>   presents a strong recommendation for testing, standardization, and
>   widespread deployment of active queue management in routers, to
>   improve the performance of today's Internet.  It also urges a
>   concerted effort of research, measurement, and ultimate deployment of
>   router mechanisms to protect the Internet from flows that are not
>   sufficiently responsive to congestion notification.
>=20
>=20
>=20
>=20
> The IETF Secretariat
>=20


From dave.taht@gmail.com  Fri Mar 15 19:23:06 2013
Return-Path: <dave.taht@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 315B811E812C for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 19:23:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.487
X-Spam-Level: 
X-Spam-Status: No, score=-2.487 tagged_above=-999 required=5 tests=[AWL=-0.113, BAYES_00=-2.599, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SB40aR0sL5DR for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 19:23:05 -0700 (PDT)
Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) by ietfa.amsl.com (Postfix) with ESMTP id 5D46311E80C5 for <aqm@ietf.org>; Fri, 15 Mar 2013 19:23:05 -0700 (PDT)
Received: by mail-ie0-f181.google.com with SMTP id 17so5195510iea.12 for <aqm@ietf.org>; Fri, 15 Mar 2013 19:23:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=FpofMxfrxXI+5e5rMlKkaS1Q0qGgc24t7kPsLOA+15E=; b=QdhvJQFpnsd3lc4LsnKZCqoTZ+QQae7DFSzjRvP+yT+cK3zmT+rZeW6267z2UJw8h/ WHSyNanYVTFELdmakw9k949KphKtliF42SSLmtVCntx0+xgubBOH/j8HYIuaMeyTjSaC jx5Wh1GXQPaLp2PpK239cBaEg8tZ0FyYYPzWldvcs53Dhyp7rTmvRfAdEronzlIRTq6P TWI7IEc+vlnvUcJj2HAYTjCgQQA5aaizOtYubyH3iXKlmfxEvqvTh9XUFdyfcjMDikF2 EkVgG5EoH75YOjT+V7JQ1GxvlsL7f2FxIqUFRhhjAWwpwkop0SNGoj1kKyYMr37/cvhV /bGA==
MIME-Version: 1.0
X-Received: by 10.50.106.199 with SMTP id gw7mr2666335igb.27.1363400585043; Fri, 15 Mar 2013 19:23:05 -0700 (PDT)
Received: by 10.64.132.71 with HTTP; Fri, 15 Mar 2013 19:23:04 -0700 (PDT)
In-Reply-To: <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com>
References: <20130316014305.5419.25287.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com>
Date: Fri, 15 Mar 2013 19:23:04 -0700
Message-ID: <CAA93jw5v1W4rA--2UgVM-rEz-QQD_MVYfhhkK9U0dQdMtkM7VA@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
To: "Fred Baker (fred)" <fred@cisco.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Mar 2013 02:23:06 -0000

On Fri, Mar 15, 2013 at 6:51 PM, Fred Baker (fred) <fred@cisco.com> wrote:
> Somewhat with my heart in my hands, given discussion on this list about t=
ext I botched and about RFC 2309's recommendation of RED, I have taken it u=
pon myself to do a wholesale replacement of RFC 2309. I'd appreciate people=
's comments - especially from the original authors of RFC 2309.
>
> For comparison and review purposes, you may like to look at a diff from R=
FC 2309. If so, go to
>     http://tools.ietf.org/rfcdiff
> and give it two URLs:
>     http://www.rfc-editor.org/rfc/rfc2309.txt
>     http://www.ietf.org/id/draft-baker-aqm-recommendation-00.txt

Fred:

wow. you are fast.

I like your new term "lemmings". I have been using ANTs a lot in my
own talks, so I'd add ants to list of terms. These were defined in a
long discussion on the bloat mailing list a year and a half ago. I'll
find a link.

There is obviously a lot more to revise and add here, but thank you
for making such a great start at it. I'd hoped to have got a chance to
talk to you in person this week, but was buried.

For those newly joining the effort, there is an enormous archive of
discussions going back ~2 years on the "bloat", "codel", and
"cerowrt-devel" mailing lists.

I understand that "aqm" is to be a list discussing standardization
efforts, but I also expect some cross pollination and discussion
crossing over/back and forth from the 400+ people that have been
participating on those lists.

https://lists.bufferbloat.net/pipermail/bloat/

I will do an announcement of this new list over there shortly.

From fred@cisco.com  Fri Mar 15 20:11:56 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B926C1F0D0F for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 20:11:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.448
X-Spam-Level: 
X-Spam-Status: No, score=-110.448 tagged_above=-999 required=5 tests=[AWL=-0.076, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DhcKPyCjrByy for <aqm@ietfa.amsl.com>; Fri, 15 Mar 2013 20:11:56 -0700 (PDT)
Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) by ietfa.amsl.com (Postfix) with ESMTP id 0CF041F0C74 for <aqm@ietf.org>; Fri, 15 Mar 2013 20:11:56 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=606; q=dns/txt; s=iport; t=1363403516; x=1364613116; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=XbWnwXt6W4hg3vzQiMnBJg3hYF+nawMf0D1wMwsSEoM=; b=V36uC64za+HlSllTiGadZwkZZF5X+7gtvb8K48LdLVpSmDbqZXzf1cVv FKVJSeyMBd30d98JLTYos2YNUYID4+jMGpMzDOLFDm5NcqP1ltrtqS500 +DKPRPqsRJpM2dOejHMfVZsSmQKY2KjDmifitDxnaqxEhoJxH1xVNyHzm k=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AkYFALPhQ1GtJV2d/2dsb2JhbABDh2W9QoFkFnSCKgEBAQMBAQJ0AgULAgEIIiQhBwolAgQOBQiHegMJBgy5XA2JVwSMTIIWAjEHgl9hA4g/i15ejUiFGoMKgig
X-IronPort-AV: E=Sophos;i="4.84,855,1355097600"; d="scan'208";a="188080013"
Received: from rcdn-core-6.cisco.com ([173.37.93.157]) by rcdn-iport-2.cisco.com with ESMTP; 16 Mar 2013 03:11:55 +0000
Received: from xhc-aln-x07.cisco.com (xhc-aln-x07.cisco.com [173.36.12.81]) by rcdn-core-6.cisco.com (8.14.5/8.14.5) with ESMTP id r2G3Bt8I014685 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Sat, 16 Mar 2013 03:11:55 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-aln-x07.cisco.com ([173.36.12.81]) with mapi id 14.02.0318.004; Fri, 15 Mar 2013 22:11:55 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: Dave Taht <dave.taht@gmail.com>
Thread-Topic: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
Thread-Index: AQHOIeecW/sgvo3KdUWrIXfSwqOBQA==
Date: Sat, 16 Mar 2013 03:11:54 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7D4A36@xmb-rcd-x09.cisco.com>
References: <20130316014305.5419.25287.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com> <CAA93jw5v1W4rA--2UgVM-rEz-QQD_MVYfhhkK9U0dQdMtkM7VA@mail.gmail.com>
In-Reply-To: <CAA93jw5v1W4rA--2UgVM-rEz-QQD_MVYfhhkK9U0dQdMtkM7VA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.21.91.153]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <A20DC0BAAE477249990FC4C307874B57@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Mar 2013 03:11:56 -0000

On Mar 15, 2013, at 8:23 PM, Dave Taht <dave.taht@gmail.com> wrote:

> I understand that "aqm" is to be a list discussing standardization
> efforts, but I also expect some cross pollination and discussion
> crossing over/back and forth from the 400+ people that have been
> participating on those lists.
>=20
> https://lists.bufferbloat.net/pipermail/bloat/
>=20
> I will do an announcement of this new list over there shortly.

Agreed. That said, please encourage them to join and post here. There are a=
 lot of lists in the world. To do anything focused, we kind of need to pick=
 one.=

From gettysjim@gmail.com  Sun Mar 17 11:40:58 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 55B0A21F8A40 for <aqm@ietfa.amsl.com>; Sun, 17 Mar 2013 11:40:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.862
X-Spam-Level: 
X-Spam-Status: No, score=-2.862 tagged_above=-999 required=5 tests=[AWL=-0.113, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xv4L5fQt1bYu for <aqm@ietfa.amsl.com>; Sun, 17 Mar 2013 11:40:57 -0700 (PDT)
Received: from mail-oa0-f50.google.com (mail-oa0-f50.google.com [209.85.219.50]) by ietfa.amsl.com (Postfix) with ESMTP id 9623E21F88C1 for <aqm@ietf.org>; Sun, 17 Mar 2013 11:40:57 -0700 (PDT)
Received: by mail-oa0-f50.google.com with SMTP id l20so4857331oag.23 for <aqm@ietf.org>; Sun, 17 Mar 2013 11:40:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:date:x-google-sender-auth:message-id :subject:from:to:cc:content-type; bh=1NSzErg0E2YYtmbAjaS5wGc9Y9bRAbWJ/DtJei5Mvb4=; b=DOIfjAxR0GtZoHhZ5+k+0j1YYKroHG2W517ENwco/8GxPjyYlTOUdLysstyVctf0xy rSifOD+fVrfc4C4KMO0TWinq7k8wQmdJ+DHQBWBe03YdrSUe9lt0jpsZ5ev0LplXw/br 0wQ3F2QEMF2px9GjW9ob14T0tB5PbjpLf8kV2ZEXPL+CE3vfl4ePxDhG9vEahab5GG6Q Kvs/RvecSh6+gyiMkKiumr57s/H4yAtWLzs3o8GaephswYjC/6Iu9dVZ2yBFhxV8LMHu txZnujm1wx3tRgD0Zmx7SICrIVcllS7MPdzu04s4G+v8NLiYD+9brimkpPxzTyA6LrjS 589g==
MIME-Version: 1.0
X-Received: by 10.182.2.132 with SMTP id 4mr5867182obu.42.1363545657208; Sun, 17 Mar 2013 11:40:57 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Sun, 17 Mar 2013 11:40:57 -0700 (PDT)
Date: Sun, 17 Mar 2013 14:40:57 -0400
X-Google-Sender-Auth: 3F7xvCNxTQvLd_0Sle7xiWAl4Rk
Message-ID: <CAGhGL2BLYHcrV_dBY3yvQRkeuJJ+y3W_o_DU0Hd1kpkukwMWfw@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Fred Baker <fred@cisco.com>
Content-Type: multipart/alternative; boundary=f46d044519d9887a1904d8233658
Cc: aqm@ietf.org
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Mar 2013 18:40:58 -0000

--f46d044519d9887a1904d8233658
Content-Type: text/plain; charset=ISO-8859-1

I've sent an announcement on the bloat & cerowrt lists encouraging people
to subscribe to the aqm list you established.

The best reason to establish a new list is so that people are aware of the
IETF NoteWell requirements.  Some may also find the language drafting of
IETF documents tedious at times as well.

FWIW: I have draft language I did about 10 days ago to try to motivate why
both a AQM algorithm and flow queuing are needed (at least in the edge of
the network).  As soon as I do another pass on that language I'll post it
here.

I'm doing this language to motivate the description of fq_codel a number of
us have been working on, but it may very well be worthwhile for a section
of the replacement of RFC 2309 that   you posted.

I should have this done later this week.
                               - Jim

--f46d044519d9887a1904d8233658
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I&#39;ve sent an announcement on the bloat &amp; cerowrt l=
ists encouraging people to subscribe to the aqm list you established.<div><=
br></div><div style>The best reason to establish a new list is so that peop=
le are aware of the IETF NoteWell requirements. =A0Some may also find the l=
anguage drafting of IETF documents tedious at times as well.</div>
<div style><br></div><div style>FWIW: I have draft language I did about 10 =
days ago to try to motivate why both a AQM algorithm and flow queuing are n=
eeded (at least in the edge of the network). =A0As soon as I do another pas=
s on that language I&#39;ll post it here.</div>
<div style><br></div><div style>I&#39;m doing this language to motivate the=
 description of fq_codel a number of us have been working on, but it may ve=
ry well be worthwhile for a section of the replacement of RFC 2309 that =A0=
 you posted.</div>
<div style><br></div><div style>I should have this done later this week.</d=
iv><div style>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0- Jim</div><div style><br></div></div>

--f46d044519d9887a1904d8233658--

From gettysjim@gmail.com  Sun Mar 17 12:12:17 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C59B321F8BE7; Sun, 17 Mar 2013 12:12:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.806
X-Spam-Level: 
X-Spam-Status: No, score=-2.806 tagged_above=-999 required=5 tests=[AWL=-0.057, BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5Npz6v+9wPyM; Sun, 17 Mar 2013 12:12:17 -0700 (PDT)
Received: from mail-oa0-f43.google.com (mail-oa0-f43.google.com [209.85.219.43]) by ietfa.amsl.com (Postfix) with ESMTP id D12B321F8BE4; Sun, 17 Mar 2013 12:12:16 -0700 (PDT)
Received: by mail-oa0-f43.google.com with SMTP id l10so4945667oag.16 for <multiple recipients>; Sun, 17 Mar 2013 12:12:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:date:x-google-sender-auth:message-id :subject:from:to:cc:content-type; bh=nwiPFfoeGso64B+Cp0ITsZgWPCEQhbg4fumuJjynZKU=; b=UJIdgXzbRbzhy1YgWTsuDi0ta+BWuviKWwTAzKnScIgG9aAJiHgbXiS+S9QLELhwGy Ea7xAvgr2ix5VFpPw4/3UKgf1TeNYEHsDBSTwmcDlbxWl1ym5bZhmzyBdKM2bdIIvcpq X7xbEe3VF0+qacoJIgq6/05thzNC4xlj1FTrwkFdwLmgm5/M/MydNuzPkLuXNbIQnYld JQ0CmL0ZAagbK8Z0GY+1OTa3cj+wGSYgf8McPs6sPcvCCVFiPeFThTToYxR9VxOB72yK gzYWOkCEq2fcFWtRQ/LZFdMv0ePg+ozVL/BO331R1pItPnW6KY6Wk6fQq8Fjw6RLCmp5 1WIA==
MIME-Version: 1.0
X-Received: by 10.60.8.197 with SMTP id t5mr6029922oea.4.1363547536452; Sun, 17 Mar 2013 12:12:16 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Sun, 17 Mar 2013 12:12:16 -0700 (PDT)
Date: Sun, 17 Mar 2013 15:12:16 -0400
X-Google-Sender-Auth: FTMgJQsj-x_radOXtpS4D2bZ6p8
Message-ID: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Content-Type: multipart/alternative; boundary=e89a8fb1ef428b771c04d823a618
Cc: Dave Taht <dave.taht@gmail.com>, Eric Dumazet <edumazet@google.com>, tsv-area@ietf.org, Fred Baker <fred@cisco.com>, bloat <bloat@lists.bufferbloat.net>, aqm@ietf.org
Subject: [aqm] Flow queuing performance (was: Re: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt)
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Mar 2013 19:12:18 -0000

--e89a8fb1ef428b771c04d823a618
Content-Type: text/plain; charset=ISO-8859-1

To begin with, I tend to use the term 'flow queuing' to distinguish what
we're doing with classic "fair queuing", which is overbound in most
people's minds, and some of what we do is "unfair" by some metrics.

Secondly, an extremely important factoid about why we got so excited about
fq_codel (which is really DRR, in term derived from SFQ, combined with
CoDel along with detecting thin vs. thick flows) is its performance:

If my memory serves me correctly, Eric Dumazet has bench marked fq_codel,
when running on a modern x86 processor with modern NIC's only takes a few
percent of one x86 core at a 10gE rate.  And we get higher total bandwidth
over the link as a result.  Eric, please correct me if I'm wrong, but you
did say I could quote you!

Similarly, Dave Taht reports when using fq_codel on CeroWrt (a 600mhz MIPS
commodity home router), while the cost of running the fq_codel algorithm is
visible in the performance traces, it's well down the list of what
functions in Linux are taking the CPU time on that architecture, with huge
latency benefits.

As a result, at the last Linux Plumber's Conference in August, there was
general consensus that we could replace PFIFO_FAST as the default qdisc in
Linux, awaiting just our confidence that fq_codel (or something very
similar) was mature enough for such a step.

These two results demonstrated that on current general purpose
architectures, an algorithm such as fq_codel is really do-able for the edge
of the Internet.

That is not the same statement as saying you can turn on existing
implementations of various other fair queuing algorithms in existing big
routers or not; just that on general purpose processors we know we can do
with these algorithms immediately.

I think we all need to reset our expectations from the 1990's where we took
away the impression that SFQ/DRR etc were too expensive for deployment:
today's general purpose systems are very different than those we had in
that era, now often dominated by cache misses more than anything else.
 Everything we think we know about performance needs to be revisited in the
light of modern architectures.
             Best regards,
                    - JIm

--e89a8fb1ef428b771c04d823a618
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style>To begin with, I tend to use the term &#39;flow=
 queuing&#39; to distinguish what we&#39;re doing with classic &quot;fair q=
ueuing&quot;, which is overbound in most people&#39;s minds, and some of wh=
at we do is &quot;unfair&quot; by some metrics.</div>
<div><br></div>Secondly, an extremely important factoid about why we got so=
 excited about fq_codel (which is really DRR, in term derived from SFQ, com=
bined with CoDel along with detecting thin vs. thick flows) is its performa=
nce:<br>
<br>If my memory serves me correctly, Eric Dumazet has bench marked fq_code=
l, when running on a modern x86 processor with modern NIC&#39;s only takes =
a few percent of one x86 core at a 10gE rate. =A0And we get higher total ba=
ndwidth over the link as a result. =A0Eric, please correct me if I&#39;m wr=
ong, but you did say I could quote you!<br>
<br>Similarly, Dave Taht reports when using fq_codel on CeroWrt (a 600mhz M=
IPS commodity home router), while the cost of running the fq_codel algorith=
m is visible in the performance traces, it&#39;s well down the list of what=
 functions in Linux are taking the CPU time on that architecture, with huge=
 latency benefits.<br>
<br>As a result, at the last Linux Plumber&#39;s Conference in August, ther=
e was general consensus that we could replace PFIFO_FAST as the default qdi=
sc in Linux, awaiting just our confidence that fq_codel (or something very =
similar) was mature enough for such a step.<br>
<br>These two results demonstrated that on current general purpose architec=
tures, an algorithm such as fq_codel is really do-able for the edge of the =
Internet. =A0<div><br></div><div>That is not the same statement as saying y=
ou can turn on existing implementations of various other fair queuing algor=
ithms in existing big routers or not; just that on general purpose processo=
rs we know we can do with these algorithms immediately.<br>
<br>I think we all need to reset our expectations from the 1990&#39;s where=
 we took away the impression that SFQ/DRR etc were too expensive for deploy=
ment: today&#39;s general purpose systems are very different than those we =
had in that era, now often dominated by cache misses more than anything els=
e. =A0Everything we think we know about performance needs to be revisited i=
n the light of modern architectures.</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0Best regards,<br>=A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 - JIm</div></div>

--e89a8fb1ef428b771c04d823a618--

From stephen@networkplumber.org  Sun Mar 17 12:25:07 2013
Return-Path: <stephen@networkplumber.org>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BF4ED21F8AE3 for <aqm@ietfa.amsl.com>; Sun, 17 Mar 2013 12:25:07 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.372
X-Spam-Level: 
X-Spam-Status: No, score=-3.372 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ocm-0akR2+sY for <aqm@ietfa.amsl.com>; Sun, 17 Mar 2013 12:25:07 -0700 (PDT)
Received: from mail-pb0-f47.google.com (mail-pb0-f47.google.com [209.85.160.47]) by ietfa.amsl.com (Postfix) with ESMTP id 2DFD221F8A9E for <aqm@ietf.org>; Sun, 17 Mar 2013 12:24:54 -0700 (PDT)
Received: by mail-pb0-f47.google.com with SMTP id rp2so5833569pbb.20 for <aqm@ietf.org>; Sun, 17 Mar 2013 12:24:54 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:date:from:to:cc:subject:message-id:in-reply-to :references:x-mailer:mime-version:content-type :content-transfer-encoding:x-gm-message-state; bh=YW5aUZ6tYwbT7pLsSjefh9xnIpp6n23ojEg2bneMmYs=; b=pAalVJwDJqgG9juo5FasrC6/HdNcqh7D9Bskm2IRto0WFMhTGyYOfQSGDZzbC5pjwA xcnDg+8ibsRJ65zfdxg+Q1Gaf7bSTFqeNvvCj3+zjthgGEwk4HqnPOsg/OXr68juhYTX RZUESGZiZUU+yY1nAgIpPZYAzc9QEZLMf9lT6NOF+4LQ2vtZB2wv+DMc8rggA7Bsn1Xn 7kdnSL0D7UgYUpvwO1etGZ/cI5mbv0pA7kXwMyYEIPt0YZk5cip3cvX880cK4f3GtoBN a06/atLvSRO+HKROzXJzxv9f7T9w5rksUwJv5VmHE1rncQ6nOTJnlu3TCurCyyQUEw65 pzsQ==
X-Received: by 10.66.75.193 with SMTP id e1mr6638809paw.202.1363548294564; Sun, 17 Mar 2013 12:24:54 -0700 (PDT)
Received: from nehalam.linuxnetplumber.net (static-50-53-71-109.bvtn.or.frontiernet.net. [50.53.71.109]) by mx.google.com with ESMTPS id pg7sm17496430pbc.5.2013.03.17.12.24.53 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Sun, 17 Mar 2013 12:24:54 -0700 (PDT)
Date: Sun, 17 Mar 2013 12:24:49 -0700
From: Stephen Hemminger <stephen@networkplumber.org>
To: Jim Gettys <jg@freedesktop.org>
Message-ID: <20130317122449.3dce3692@nehalam.linuxnetplumber.net>
In-Reply-To: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com>
References: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com>
X-Mailer: Claws Mail 3.8.1 (GTK+ 2.24.10; x86_64-pc-linux-gnu)
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Gm-Message-State: ALoCoQk3nvwIpRdgGJqaG8AgqCzFL6Xvxbuq9TudeFIjSPOcDRT3CtTTxd9FnhLOzYbGKXS5fEHi
X-Mailman-Approved-At: Sun, 17 Mar 2013 17:59:21 -0700
Cc: Eric Dumazet <edumazet@google.com>, bloat <bloat@lists.bufferbloat.net>, tsv-area@ietf.org, aqm@ietf.org, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] [Bloat] Flow queuing performance (was: Re: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt)
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Mar 2013 19:25:08 -0000

On Sun, 17 Mar 2013 15:12:16 -0400
Jim Gettys <jg@freedesktop.org> wrote:

> As a result, at the last Linux Plumber's Conference in August, there was
> general consensus that we could replace PFIFO_FAST as the default qdisc in
> Linux, awaiting just our confidence that fq_codel (or something very
> similar) was mature enough for such a step.

The only thing holding this back is that some user's still want to have priority queueing
of control packets. I (and others) have done some experiments on layering a traditional
three class priority layer on top of fq_codel, but nothing concrete has been submitted
for review.

From eric.dumazet@gmail.com  Sun Mar 17 12:32:44 2013
Return-Path: <eric.dumazet@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5374221F8B96; Sun, 17 Mar 2013 12:32:44 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.372
X-Spam-Level: 
X-Spam-Status: No, score=-3.372 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Shk7mZSRsnla; Sun, 17 Mar 2013 12:32:43 -0700 (PDT)
Received: from mail-pb0-f43.google.com (mail-pb0-f43.google.com [209.85.160.43]) by ietfa.amsl.com (Postfix) with ESMTP id A1FEF21F8B71; Sun, 17 Mar 2013 12:32:43 -0700 (PDT)
Received: by mail-pb0-f43.google.com with SMTP id md12so5866535pbc.30 for <multiple recipients>; Sun, 17 Mar 2013 12:32:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:message-id:subject:from:to:cc:date:in-reply-to :references:content-type:x-mailer:content-transfer-encoding :mime-version; bh=aPaFOEK+dAqSVElY+LNoMpyEEt5Ha4/Wvua7UbK0I64=; b=psNh+sqzD+R2J9Uxvictan69/IqWRpdgP7YGs4peNWkYHcN1gQyuIoVpnE202OSqW6 AWlIcoxlnqA+b4tI4LHRb2LoDOX36y3i+5luThkYq5d4y67RNumQOl622ys4SXRw8VIl VFCisOsNNt239yci2G6NFWnNE8aVUOWi3hJ7ud4lpzsTuU2kzT152ojoAbwW3HiJtvc5 2uWHJptHCeRhnqSD/Ep/v1otWPlyGsWrf3kn0Qu1hBRAa1iPeh+of8Fqm2RI+S6qvFPx RzNsGivqOlrQ6hQpsJr4IzhyLmbcfjwyQ7xuktO+p9ixqdaW0sh6HdOZZv2ZoD50DOK4 gtIw==
X-Received: by 10.68.75.72 with SMTP id a8mr28970476pbw.143.1363548763334; Sun, 17 Mar 2013 12:32:43 -0700 (PDT)
Received: from [172.19.242.99] ([172.19.242.99]) by mx.google.com with ESMTPS id kl4sm17490158pbc.31.2013.03.17.12.32.41 (version=SSLv3 cipher=RC4-SHA bits=128/128); Sun, 17 Mar 2013 12:32:42 -0700 (PDT)
Message-ID: <1363548760.29475.87.camel@edumazet-glaptop>
From: Eric Dumazet <eric.dumazet@gmail.com>
To: Stephen Hemminger <stephen@networkplumber.org>
Date: Sun, 17 Mar 2013 12:32:40 -0700
In-Reply-To: <20130317122449.3dce3692@nehalam.linuxnetplumber.net>
References: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com> <20130317122449.3dce3692@nehalam.linuxnetplumber.net>
Content-Type: text/plain; charset="UTF-8"
X-Mailer: Evolution 3.2.3-0ubuntu6 
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0
X-Mailman-Approved-At: Sun, 17 Mar 2013 17:59:21 -0700
Cc: Jim Gettys <jg@freedesktop.org>, Eric Dumazet <edumazet@google.com>, tsv-area@ietf.org, bloat <bloat@lists.bufferbloat.net>, aqm@ietf.org, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] [Bloat] Flow queuing performance (was: Re: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt)
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Mar 2013 19:41:30 -0000

On Sun, 2013-03-17 at 12:24 -0700, Stephen Hemminger wrote:
> On Sun, 17 Mar 2013 15:12:16 -0400
> Jim Gettys <jg@freedesktop.org> wrote:
> 
> > As a result, at the last Linux Plumber's Conference in August, there was
> > general consensus that we could replace PFIFO_FAST as the default qdisc in
> > Linux, awaiting just our confidence that fq_codel (or something very
> > similar) was mature enough for such a step.
> 
> The only thing holding this back is that some user's still want to have priority queueing
> of control packets. I (and others) have done some experiments on layering a traditional
> three class priority layer on top of fq_codel, but nothing concrete has been submitted
> for review.


I did a patch on top of fq_codel to add 3 bands (same prio
classification than pfifo_fast).

I added a new attribute at that time to enable this 3 bands (and
disabling the external classifier, as it uses same skb->priority field.)

I probably better clone fq_codel, because if we want a pfifo_fast
replacement, it should run without the help of a "tc" command.




From swmike@swm.pp.se  Sun Mar 17 21:30:39 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3802721F8AE6; Sun, 17 Mar 2013 21:30:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.486
X-Spam-Level: 
X-Spam-Status: No, score=-2.486 tagged_above=-999 required=5 tests=[AWL=-0.113, BAYES_00=-2.599, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h9-e621kZshd; Sun, 17 Mar 2013 21:30:38 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 937B721F8A00; Sun, 17 Mar 2013 21:30:38 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id D62D89E; Mon, 18 Mar 2013 05:30:36 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id BF54D9C; Mon, 18 Mar 2013 05:30:36 +0100 (CET)
Date: Mon, 18 Mar 2013 05:30:36 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: Jim Gettys <jg@freedesktop.org>
In-Reply-To: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1303180512370.2309@uplift.swm.pp.se>
References: <CAGhGL2BdnhRB0kz_AXb9X9=ph3016ODKDc_9kNm5wgO215NCTg@mail.gmail.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: Dave Taht <dave.taht@gmail.com>, Eric Dumazet <edumazet@google.com>, tsv-area@ietf.org, Fred Baker <fred@cisco.com>, bloat <bloat@lists.bufferbloat.net>, aqm@ietf.org
Subject: Re: [aqm] Flow queuing performance (was: Re: [tsvwg] New Version Notification for draft-baker-tsvwg-aqm-recommendation-00.txt)
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Mar 2013 04:30:39 -0000

On Sun, 17 Mar 2013, Jim Gettys wrote:

> Secondly, an extremely important factoid about why we got so excited 
> about fq_codel (which is really DRR, in term derived from SFQ, combined 
> with CoDel along with detecting thin vs. thick flows) is its 
> performance:

I can imagine! One thought, have tests been done that indicate that 
generally an all-TCP-ACK "flow" (as seen in the egress direction) is still 
considered a "thin" flow? In the ADSL case with 20 meg down and 1 meg up, 
I seem to remember that ACKing 20 megabit/s of traffic uses 30-40% of the 
upstream bandwidth so it's a matter of opinion if this is actually a thick 
flow or not?

I'm trying to understand how codel handles the use case of 100 upload 
torrent TCP sessions (all trying to run max upload speed) and a single tcp 
http/ftp download. This is one of the most common complaints I've seen 
over the years about bufferbloat, uploading destroys download performance.

Another thing that I would like to see tested on Linux is the combination 
of fq_codel and shaping the speed. This is for the use case in ETTH 
selling for instance a 100/10 connection, where the 10 upstream connection 
is achieved by policing the bw in the ETTH access switch (the physical 
port is 100/100. It would give the user a better experience if the CPE 
could shape the traffic outgoing to 10 megabit to avoid the packet loss 
caused by the policing. This is something I would like to put in as an RFQ 
requirement, so it would be good if any AQM standard actually defined this 
mechanism and it was part of the description.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From rs@netapp.com  Mon Mar 18 07:07:11 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BC5CA21F8EED for <aqm@ietfa.amsl.com>; Mon, 18 Mar 2013 07:07:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.184
X-Spam-Level: 
X-Spam-Status: No, score=-10.184 tagged_above=-999 required=5 tests=[AWL=0.188, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KcofQjecIEsk for <aqm@ietfa.amsl.com>; Mon, 18 Mar 2013 07:07:11 -0700 (PDT)
Received: from mx1.netapp.com (mx1.netapp.com [216.240.18.38]) by ietfa.amsl.com (Postfix) with ESMTP id 3A91021F8DED for <aqm@ietf.org>; Mon, 18 Mar 2013 07:07:11 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,865,1355126400"; d="scan'208";a="247415141"
Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx1-out.netapp.com with ESMTP; 18 Mar 2013 07:07:11 -0700
Received: from vmwexceht03-prd.hq.netapp.com (vmwexceht03-prd.hq.netapp.com [10.106.76.241]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2IE7AML011073; Mon, 18 Mar 2013 07:07:10 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht03-prd.hq.netapp.com ([10.106.76.241]) with mapi id 14.02.0342.003; Mon, 18 Mar 2013 07:07:10 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: "Fred Baker (fred)" <fred@cisco.com>, "aqm@ietf.org" <aqm@ietf.org>
Thread-Topic: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
Thread-Index: AQHOIeecW/sgvo3KdUWrIXfSwqOBQJin4Z6AgAOc/SA=
Date: Mon, 18 Mar 2013 14:07:09 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24AB8D06@SACEXCMBX02-PRD.hq.netapp.com>
References: <20130316014305.5419.25287.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com>
In-Reply-To: <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.104.60.114]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Mar 2013 14:07:11 -0000

I like the new document.

Only two nits:

In section 2, it states " The next section
   introduces RED, an active queue management mechanism that solves both
   problems listed above (given responsive flows)."=20

But this document really isn't a detailed discussion about RED, but the mot=
ivation of and requirements to an AQM in general..


In section 4, typo ("dos not").


In section 4.6, the last sentence has a question mark to it, albeit the sen=
tence does not pose a question. Introducting additional descriptive traffic=
 types (mice, elephants, lemmings, ants...) in order to have a common under=
standing of use-cases is beneficial... It should just be avoided to end up =
with a zoo of classifications in the end, though. Or is this intended to ma=
rk space for expansion?

Best regards,



Richard Scheffenegger


> -----Original Message-----
> From: aqm-bounces@ietf.org [mailto:aqm-bounces@ietf.org] On Behalf Of Fre=
d
> Baker (fred)
> Sent: Samstag, 16. M=E4rz 2013 02:52
> To: aqm@ietf.org
> Subject: Re: [aqm] New Version Notification for draft-baker-aqm-
> recommendation-00.txt
>=20
> Somewhat with my heart in my hands, given discussion on this list about
> text I botched and about RFC 2309's recommendation of RED, I have taken i=
t
> upon myself to do a wholesale replacement of RFC 2309. I'd appreciate
> people's comments - especially from the original authors of RFC 2309.
>=20
> For comparison and review purposes, you may like to look at a diff from
> RFC 2309. If so, go to
>     http://tools.ietf.org/rfcdiff
> and give it two URLs:
>     http://www.rfc-editor.org/rfc/rfc2309.txt
>     http://www.ietf.org/id/draft-baker-aqm-recommendation-00.txt
>=20
> >
> > A new version of I-D, draft-baker-aqm-recommendation-00.txt
> > has been successfully submitted by Fred Baker and posted to the IETF
> > repository.
> >
> > Filename:	 draft-baker-aqm-recommendation
> > Revision:	 00
> > Title:		 IETF Recommendations Regarding Active Queue Management
> > Creation date:	 2013-03-15
> > Group:		 Individual Submission
> > Number of pages: 17
> > URL:             http://www.ietf.org/internet-drafts/draft-baker-aqm-
> recommendation-00.txt
> > Status:          http://datatracker.ietf.org/doc/draft-baker-aqm-
> recommendation
> > Htmlized:        http://tools.ietf.org/html/draft-baker-aqm-
> recommendation-00
> >
> >
> > Abstract:
> >   This memo presents recommendations to the Internet community
> >   concerning measures to improve and preserve Internet performance.  It
> >   presents a strong recommendation for testing, standardization, and
> >   widespread deployment of active queue management in routers, to
> >   improve the performance of today's Internet.  It also urges a
> >   concerted effort of research, measurement, and ultimate deployment of
> >   router mechanisms to protect the Internet from flows that are not
> >   sufficiently responsive to congestion notification.
> >
> >
> >
> >
> > The IETF Secretariat
> >
>=20
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm

From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 19 01:04:36 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E28D21F86F7; Tue, 19 Mar 2013 01:04:36 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.892
X-Spam-Level: 
X-Spam-Status: No, score=-0.892 tagged_above=-999 required=5 tests=[AWL=-1.357, BAYES_40=-0.185, HELO_EQ_DE=0.35, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0plgJ38qzZnS; Tue, 19 Mar 2013 01:04:36 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 7BD9321F8A7E; Tue, 19 Mar 2013 01:04:35 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id BA624600F5; Tue, 19 Mar 2013 09:04:33 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id 614AE600F4; Tue, 19 Mar 2013 09:04:33 +0100 (CET)
From: Mirja =?iso-8859-1?q?K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
To: iccrg@irtf.org, tcpm@ietf.org, aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Date: Tue, 19 Mar 2013 09:04:32 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
MIME-Version: 1.0
Content-Disposition: inline
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Message-Id: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
Subject: [aqm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 08:04:36 -0000

Hi everybody,

we just published a paper on the state of ECN support and usage on the=20
Internet. We performed a similar study than Bauer et al., probing webserver=
s=20
to get the current status. We found that about 30% of webservers (by Sep'12=
)=20
support ECN. Unfortunately, still we saw a failure rate of 9% when checking
end-2-end path ECN usability (setting CE). Additionally, we analyzed NetFlo=
w=20
data to get a lower bound for the ECN usage which was below 1%. Then we als=
o=20
had a look at IPv6 (47,5% ECN support). At the same time we also monitored =
an=20
increase in general IPv6 support on webserver over the IPv6 launch day last=
=20
year (-> check the paper).

K=FChlewind, M.; Neuner, S.; Trammell, B.: On the state of ECN and TCP Opti=
ons=20
on the Internet. Proceedings of the 14th Passive and Active Measurement=20
conference (PAM 2013), Hong Kong, March 2013.

http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Kue_PAM13_4016=
0.pdf

Do we need to start a campaign to further encourage greater ECN support (al=
so=20
of network providers)? Asking the content providers on the list(s): What ar=
e=20
the reasons to not enable ECN support?

Mirja


From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 19 01:06:35 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 37E7E21F8AA8 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 01:06:35 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.307
X-Spam-Level: 
X-Spam-Status: No, score=-1.307 tagged_above=-999 required=5 tests=[AWL=0.415,  BAYES_00=-2.599, HELO_EQ_DE=0.35, MIME_8BIT_HEADER=0.3, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hSxmemEpco2d for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 01:06:34 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 81A7F21F8AA6 for <aqm@ietf.org>; Tue, 19 Mar 2013 01:06:34 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id D8BC1600F5; Tue, 19 Mar 2013 09:06:33 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id 8DACE600F4; Tue, 19 Mar 2013 09:06:33 +0100 (CET)
From: Mirja =?iso-8859-1?q?K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
To: aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Date: Tue, 19 Mar 2013 09:06:32 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
MIME-Version: 1.0
Content-Disposition: inline
Content-Type: Text/Plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Message-Id: <201303190906.32352.mkuehle@ikr.uni-stuttgart.de>
Subject: [aqm] AQM support in the network
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 08:06:35 -0000

Hi,

I was thinking about a way how to measurement if AQM is used in a network... 
(if someone has ideas please tell me)... but then I though instead of 
spamming the networks with measurement traffic, I might just asked network 
operators directly :-)

Not sure who is subscribed to this list, but as a starting point I would like 
to ask anybody who reads this and is in charge of operating any kind of 
network (isp, cdn provider, or smaller local networks) to post to the list, 
if AQM is used anywhere in the network!

Of course it would also be helpful to tell where AQM is used and which kind of 
AQM also which parameters. I know that people might not be allowed to say, but 
maybe you could at least indicate if it is used somewhere or not. Please also 
tell if you don't use AQM anywhere. Maybe you might also be able to explain 
the reasons why AQM is not used...?

Thanks in advance!
Mirja


From swmike@swm.pp.se  Tue Mar 19 02:22:55 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4C82A21F89D7; Tue, 19 Mar 2013 02:22:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.411
X-Spam-Level: 
X-Spam-Status: No, score=-2.411 tagged_above=-999 required=5 tests=[AWL=-0.112, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id es4du8m7DByD; Tue, 19 Mar 2013 02:22:54 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 5B02721F89AF; Tue, 19 Mar 2013 02:22:54 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 9711C9C; Tue, 19 Mar 2013 10:22:52 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 8C1049A; Tue, 19 Mar 2013 10:22:52 +0100 (CET)
Date: Tue, 19 Mar 2013 10:22:52 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: =?ISO-8859-15?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
In-Reply-To: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
Message-ID: <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: MULTIPART/MIXED; BOUNDARY="-137064504-27937894-1363684972=:2309"
Cc: tcpm@ietf.org, iccrg@irtf.org, aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 09:22:55 -0000

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---137064504-27937894-1363684972=:2309
Content-Type: TEXT/PLAIN; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: 8BIT

On Tue, 19 Mar 2013, Mirja Kühlewind wrote:

> Do we need to start a campaign to further encourage greater ECN support 
> (also of network providers)? Asking the content providers on the 
> list(s): What are the reasons to not enable ECN support?

Let me speculate:

It's not on by default in Windows. There is little reason to turn it on, 
because "nobody" acts on ECN, and nobody acts on ECN because there is no 
demand for AQM, and even if there was AQM, it would most likely not be ECN 
enabled.

I tried enabling ECN back in 2001 but saw major breakage. I re-enabled ECN 
again around 2008 or so <http://seclists.org/nanog/2008/Nov/250> and 
investigated, and at least the vendors I talked to didn't show much 
interest. I've been running with ECN on ever since and have not 
experienced any breakage I could trace back to ECN usage. This is hopeful.

I believe the Bufferbloat discussions and increased interest in AQM on the 
customer access link will increase interest in ECN as well, so the 
important thing is that AQM document includes requirements for AQM, so 
when I request support for "best practice AQM", I get ECN support as well. 
I am looking for RFC6204 (Basic Requirements) style document.

Then we need to advertise ECN benefit to end users, for instance by 
advocating its use through tools like 
<http://www.speedguide.net/tcpoptimizer.php>. The change log says "Changed 
ECN Capability optimal recommendation in General tab based on user 
feedback and issues with some US ISPs" and I fear that the recommendation 
there is to turn ECN off :P. I don't want to install the utility to test.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se
---137064504-27937894-1363684972=:2309--

From swmike@swm.pp.se  Tue Mar 19 02:41:02 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4384421F88FB for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 02:41:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.284
X-Spam-Level: 
X-Spam-Status: No, score=-2.284 tagged_above=-999 required=5 tests=[AWL=-0.212, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GhJzV+QFFegK for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 02:41:01 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 6FB3821F8936 for <aqm@ietf.org>; Tue, 19 Mar 2013 02:41:01 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id B6326A0; Tue, 19 Mar 2013 10:40:59 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id AD4A39E; Tue, 19 Mar 2013 10:40:59 +0100 (CET)
Date: Tue, 19 Mar 2013 10:40:59 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: =?ISO-8859-15?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
In-Reply-To: <201303190906.32352.mkuehle@ikr.uni-stuttgart.de>
Message-ID: <alpine.DEB.2.00.1303191023220.2309@uplift.swm.pp.se>
References: <201303190906.32352.mkuehle@ikr.uni-stuttgart.de>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: MULTIPART/MIXED; BOUNDARY="-137064504-902639590-1363686059=:2309"
Cc: aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] AQM support in the network
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 09:41:02 -0000

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

---137064504-902639590-1363686059=:2309
Content-Type: TEXT/PLAIN; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8BIT

On Tue, 19 Mar 2013, Mirja Kühlewind wrote:

> Not sure who is subscribed to this list, but as a starting point I would 
> like to ask anybody who reads this and is in charge of operating any 
> kind of network (isp, cdn provider, or smaller local networks) to post 
> to the list, if AQM is used anywhere in the network!

I would imagine that most ISPs use AQM on their core links, but this AQM 
is more WRED kind of AQM, where you have a few queues and place the 
traffic into each queue based on DSCP/EXP values. This is mostly used to 
assure certain amount of bandwidth/loss profile to certain services, such 
as the providers own VoIP or IPTV service.

> Of course it would also be helpful to tell where AQM is used and which 
> kind of AQM also which parameters. I know that people might not be 
> allowed to say, but maybe you could at least indicate if it is used 
> somewhere or not. Please also tell if you don't use AQM anywhere. Maybe 
> you might also be able to explain the reasons why AQM is not used...?

Are you defining AQM as anything that is not FIFO?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se
---137064504-902639590-1363686059=:2309--

From fred@cisco.com  Tue Mar 19 09:33:42 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 810B621F8417 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 09:33:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.317
X-Spam-Level: 
X-Spam-Status: No, score=-110.317 tagged_above=-999 required=5 tests=[AWL=0.055, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C65ujqRqO-56 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 09:33:41 -0700 (PDT)
Received: from rcdn-iport-8.cisco.com (rcdn-iport-8.cisco.com [173.37.86.79]) by ietfa.amsl.com (Postfix) with ESMTP id 6538F21F8FB3 for <aqm@ietf.org>; Tue, 19 Mar 2013 09:33:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=4628; q=dns/txt; s=iport; t=1363710820; x=1364920420; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=M8XaPgXWy4vayevAwrbRjNSdfGjI6s3KTKzEbsvw7MA=; b=KxD8DQWbUN6wE9GXi6tNrK+Oikh0nLNH4OUD+lxF9b8elujqckLYAhSl 2BzIEmBDtKb58HfVjkBUuvn5UyWk/CCGajigFSM4gMzuKyTmWyLKVwldb o/O1TvIktWH/65vnnIMCeU9bKL7XFwZHtwklFA64T4qszLV0D3fg9UW5h g=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgEFABmTSFGtJXHA/2dsb2JhbAA5CsUbgVoWdIIkAQEBAwEBAQFrCQIFBwQCAQgRBAEBCx0HJwsUCQgCBA4FCAELh3oGDLIqkB2NVIEHAiYLBwaCWWEDiD+KWoRlikyFF4MKgig
X-IronPort-AV: E=Sophos;i="4.84,872,1355097600"; d="scan'208";a="189145279"
Received: from rcdn-core2-5.cisco.com ([173.37.113.192]) by rcdn-iport-8.cisco.com with ESMTP; 19 Mar 2013 16:33:38 +0000
Received: from xhc-aln-x09.cisco.com (xhc-aln-x09.cisco.com [173.36.12.83]) by rcdn-core2-5.cisco.com (8.14.5/8.14.5) with ESMTP id r2JGXcag032154 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 19 Mar 2013 16:33:38 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-aln-x09.cisco.com ([173.36.12.83]) with mapi id 14.02.0318.004; Tue, 19 Mar 2013 11:33:38 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: "Scheffenegger, Richard" <rs@netapp.com>
Thread-Topic: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
Thread-Index: AQHOJL+B5gPiiK+kKke5nLZZEAKzWA==
Date: Tue, 19 Mar 2013 16:33:37 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7DB761@xmb-rcd-x09.cisco.com>
References: <20130316014305.5419.25287.idtracker@ietfa.amsl.com> <8C48B86A895913448548E6D15DA7553B7D48C1@xmb-rcd-x09.cisco.com> <012C3117EDDB3C4781FD802A8C27DD4F24AB8D06@SACEXCMBX02-PRD.hq.netapp.com>
In-Reply-To: <012C3117EDDB3C4781FD802A8C27DD4F24AB8D06@SACEXCMBX02-PRD.hq.netapp.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.21.100.174]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <2FD1DE0D6DB9304DA41A9486EF56BF98@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] New Version Notification for draft-baker-aqm-recommendation-00.txt
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 16:33:42 -0000

On Mar 18, 2013, at 8:07 AM, "Scheffenegger, Richard" <rs@netapp.com> wrote=
:

>=20
> I like the new document.
>=20
> Only two nits:
>=20
> In section 2, it states " The next section
>   introduces RED, an active queue management mechanism that solves both
>   problems listed above (given responsive flows)."=20
>=20
> But this document really isn't a detailed discussion about RED, but the m=
otivation of and requirements to an AQM in general..

grump. I thought I had found all the references to RED and replaced them wi=
th "AQM". Thanks for pointing that out.

> In section 4, typo ("dos not").
>=20
>=20
> In section 4.6, the last sentence has a question mark to it, albeit the s=
entence does not pose a question. Introducting additional descriptive traff=
ic types (mice, elephants, lemmings, ants...) in order to have a common und=
erstanding of use-cases is beneficial... It should just be avoided to end u=
p with a zoo of classifications in the end, though. Or is this intended to =
mark space for expansion?

I think Map/Reduce applications are an opportunity to expand our thinking, =
and an argument for both maintaining queues as shallow as possible (to incr=
ease the available queue space for bursts when they come) and signaling wit=
hout loss. The fundamental question is (I'm quoting a specific customer use=
 case, although I decline to identify the network): what happens when you a=
sk O(5000) of your best friends a question and they all answer more or less=
 simultaneously with a "mouse"? The particular network maintains standing T=
CP sessions between cache servers and compute servers (to avoid the additio=
nal RTTs of SYN/SYN-ACk and FIN/FIN-ACK), so a drop results in head-of-line=
 blocking at the compute server and *also* denies the TCP session to subseq=
uent possible users until the retransmission happens, which is another form=
 of head-of-line blocking.

I'll take a look at the sentence, however; it sounds like a place where I d=
idn't finish my thought.

> Best regards,
>=20
>=20
>=20
> Richard Scheffenegger
>=20
>=20
>> -----Original Message-----
>> From: aqm-bounces@ietf.org [mailto:aqm-bounces@ietf.org] On Behalf Of Fr=
ed
>> Baker (fred)
>> Sent: Samstag, 16. M=E4rz 2013 02:52
>> To: aqm@ietf.org
>> Subject: Re: [aqm] New Version Notification for draft-baker-aqm-
>> recommendation-00.txt
>>=20
>> Somewhat with my heart in my hands, given discussion on this list about
>> text I botched and about RFC 2309's recommendation of RED, I have taken =
it
>> upon myself to do a wholesale replacement of RFC 2309. I'd appreciate
>> people's comments - especially from the original authors of RFC 2309.
>>=20
>> For comparison and review purposes, you may like to look at a diff from
>> RFC 2309. If so, go to
>>    http://tools.ietf.org/rfcdiff
>> and give it two URLs:
>>    http://www.rfc-editor.org/rfc/rfc2309.txt
>>    http://www.ietf.org/id/draft-baker-aqm-recommendation-00.txt
>>=20
>>>=20
>>> A new version of I-D, draft-baker-aqm-recommendation-00.txt
>>> has been successfully submitted by Fred Baker and posted to the IETF
>>> repository.
>>>=20
>>> Filename:	 draft-baker-aqm-recommendation
>>> Revision:	 00
>>> Title:		 IETF Recommendations Regarding Active Queue Management
>>> Creation date:	 2013-03-15
>>> Group:		 Individual Submission
>>> Number of pages: 17
>>> URL:             http://www.ietf.org/internet-drafts/draft-baker-aqm-
>> recommendation-00.txt
>>> Status:          http://datatracker.ietf.org/doc/draft-baker-aqm-
>> recommendation
>>> Htmlized:        http://tools.ietf.org/html/draft-baker-aqm-
>> recommendation-00
>>>=20
>>>=20
>>> Abstract:
>>>  This memo presents recommendations to the Internet community
>>>  concerning measures to improve and preserve Internet performance.  It
>>>  presents a strong recommendation for testing, standardization, and
>>>  widespread deployment of active queue management in routers, to
>>>  improve the performance of today's Internet.  It also urges a
>>>  concerted effort of research, measurement, and ultimate deployment of
>>>  router mechanisms to protect the Internet from flows that are not
>>>  sufficiently responsive to congestion notification.
>>>=20
>>>=20
>>>=20
>>>=20
>>> The IETF Secretariat
>>>=20
>>=20
>> _______________________________________________
>> aqm mailing list
>> aqm@ietf.org
>> https://www.ietf.org/mailman/listinfo/aqm
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm


From gettysjim@gmail.com  Tue Mar 19 10:26:25 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 238CE21F8D63; Tue, 19 Mar 2013 10:26:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.311
X-Spam-Level: 
X-Spam-Status: No, score=-0.311 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, NO_RELAYS=-0.001, SARE_HTML_USL_OBFU=1.666]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oTYHfZZwzVTn; Tue, 19 Mar 2013 10:26:24 -0700 (PDT)
Received: from mail-ob0-x22b.google.com (mail-ob0-x22b.google.com [IPv6:2607:f8b0:4003:c01::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 5970421F8D34; Tue, 19 Mar 2013 10:26:24 -0700 (PDT)
Received: by mail-ob0-f171.google.com with SMTP id x4so699453obh.16 for <multiple recipients>; Tue, 19 Mar 2013 10:26:23 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=mND3ULikwWoEnVcQ+P37Wl9wGfSIEyyi03wC0s8ZhXI=; b=pmxPKfbCsBhDv0rjVutV1HHULppaVv9aFpZ8Vbst+hEnA3mrD9SU8xNoq4QR0BxlvV M2OX7hNxatsq907Z0U+VpwArYu3Oq1G3wObXvedrti4M/a4xpYKO8qW1hBE50oY1qQu9 KsUSwR+Yuz58lZfvZAYm2pw9ypSelycQL6sTC2Yd6zp1IhrZcGk/ooTaXbGe83OEMpiA c3CI41KKH2GaVLKkf+/FdXh8hhKTvetAQYVOXgwmSzlbznZa6ToUW2nSeizwYEg9zmh2 IfW1dvQqdVzVhevpxQ0d4cKlsKEGzaeHgqy7/RKX3DGnHb8nqlVYVj8ojliU0zvqc/rQ RI0w==
MIME-Version: 1.0
X-Received: by 10.60.3.103 with SMTP id b7mr1850839oeb.86.1363713983726; Tue, 19 Mar 2013 10:26:23 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Tue, 19 Mar 2013 10:26:23 -0700 (PDT)
In-Reply-To: <51487A27.7010904@hp.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com>
Date: Tue, 19 Mar 2013 13:26:23 -0400
X-Google-Sender-Auth: ZkpC693eg7WCg6tnCpl7_b9NK-o
Message-ID: <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Rick Jones <rick.jones2@hp.com>
Content-Type: multipart/alternative; boundary=e89a8ff2514a934f0104d84a670e
Cc: tcpm@ietf.org, aqm@ietf.org, iccrg@irtf.org, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 17:26:25 -0000

--e89a8ff2514a934f0104d84a670e
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

You need to distinguish 1) "Will talk ECN when talked to", from 2) "Will
initiate an ECN conversation".

Linux, for example, will do 1) by default (and has for some time now), but
requires configuration to turn on ECN when initiating a connection.
                                  - Jim



On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <rick.jones2@hp.com> wrote:

> On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
>
>> On Tue, 19 Mar 2013, Mirja K=FChlewind wrote:
>>
>>  Do we need to start a campaign to further encourage greater ECN
>>> support (also of network providers)? Asking the content providers on
>>> the list(s): What are the reasons to not enable ECN support?
>>>
>>
>> Let me speculate:
>>
>> It's not on by default in Windows. There is little reason to turn it on,
>> because "nobody" acts on ECN, and nobody acts on ECN because there is no
>> demand for AQM, and even if there was AQM, it would most likely not be
>> ECN enabled.
>>
>
> Out of curiosity, which stacks have ECN enabled by default?  To my
> recollection, on netperf.org I had to explicitly enable TCP's use of ECN
> in the kernel which ships with Ubuntu 12.04.
>
> rick jones
>
> ______________________________**_________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/**listinfo/tcpm<https://www.ietf.org/mailman=
/listinfo/tcpm>
>

--e89a8ff2514a934f0104d84a670e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">You need to distinguish 1) &quot;Will talk ECN when talked=
 to&quot;, from 2) &quot;Will initiate an ECN conversation&quot;.<div><br><=
/div><div style>Linux, for example, will do 1) by default (and has for some=
 time now), but requires configuration to turn on ECN when initiating a con=
nection.</div>
<div style>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 - Jim</div><div style><br></div></div><div class=3D"gmail_extra"><br><b=
r><div class=3D"gmail_quote">On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <=
span dir=3D"ltr">&lt;<a href=3D"mailto:rick.jones2@hp.com" target=3D"_blank=
">rick.jones2@hp.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div class=3D"im">On 03/19/2013 02:22 AM, Mi=
kael Abrahamsson wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
On Tue, 19 Mar 2013, Mirja K=FChlewind wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Do we need to start a campaign to further encourage greater ECN<br>
support (also of network providers)? Asking the content providers on<br>
the list(s): What are the reasons to not enable ECN support?<br>
</blockquote>
<br>
Let me speculate:<br>
<br>
It&#39;s not on by default in Windows. There is little reason to turn it on=
,<br>
because &quot;nobody&quot; acts on ECN, and nobody acts on ECN because ther=
e is no<br>
demand for AQM, and even if there was AQM, it would most likely not be<br>
ECN enabled.<br>
</blockquote>
<br></div>
Out of curiosity, which stacks have ECN enabled by default? =A0To my recoll=
ection, on <a href=3D"http://netperf.org" target=3D"_blank">netperf.org</a>=
 I had to explicitly enable TCP&#39;s use of ECN in the kernel which ships =
with Ubuntu 12.04.<span class=3D"HOEnZb"><font color=3D"#888888"><br>

<br>
rick jones</font></span><div class=3D"HOEnZb"><div class=3D"h5"><br>
______________________________<u></u>_________________<br>
tcpm mailing list<br>
<a href=3D"mailto:tcpm@ietf.org" target=3D"_blank">tcpm@ietf.org</a><br>
<a href=3D"https://www.ietf.org/mailman/listinfo/tcpm" target=3D"_blank">ht=
tps://www.ietf.org/mailman/<u></u>listinfo/tcpm</a><br>
</div></div></blockquote></div><br></div>

--e89a8ff2514a934f0104d84a670e--

From gettysjim@gmail.com  Tue Mar 19 11:32:33 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B9C5321F8C3E for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 11:32:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.031
X-Spam-Level: 
X-Spam-Status: No, score=-1.031 tagged_above=-999 required=5 tests=[AWL=0.720,  BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jGVrKTIPal25 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 11:32:33 -0700 (PDT)
Received: from mail-ob0-x229.google.com (mail-ob0-x229.google.com [IPv6:2607:f8b0:4003:c01::229]) by ietfa.amsl.com (Postfix) with ESMTP id 42AFD21F8BDE for <aqm@ietf.org>; Tue, 19 Mar 2013 11:32:33 -0700 (PDT)
Received: by mail-ob0-f169.google.com with SMTP id ta14so786899obb.14 for <aqm@ietf.org>; Tue, 19 Mar 2013 11:32:32 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=VHrOjiePcS5JGDybVcJo53p8+oYr0JjUAuWlsXUBM8U=; b=Gq37xxWm10uzSYu1/OiJVWj8qOCv+JrtyUe7VfTNxWXmKLGbn/UwaWyYMYxSgmjiPm pf3WKQAOyfl4a1qnNkjqBzyWAEiU+nJSD+e557wwYV5acx7XE0BlOXB+CtKgsg7NRvnJ k0kx5re50MAIr7R2sBPI5oVT8oVvZNkfvAUhe1iGYek3vDKxTDf7osRsNso1wtgjfGys oX5LxXR46C9wj0tqkw1Tb6Vgw++L7pgD6D//TAqNUYpTi1C1KL/XTZPwFYRH6sxv4OQ+ CCMNIgPvu0rs///iQii//xnZyTJNlBbvjdMIOiaLHCW8J6LT4m7nThJYRda8u1icjiXM OhIQ==
MIME-Version: 1.0
X-Received: by 10.60.8.40 with SMTP id o8mr2062071oea.112.1363717952602; Tue, 19 Mar 2013 11:32:32 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Tue, 19 Mar 2013 11:32:32 -0700 (PDT)
In-Reply-To: <alpine.DEB.2.00.1303191023220.2309@uplift.swm.pp.se>
References: <201303190906.32352.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191023220.2309@uplift.swm.pp.se>
Date: Tue, 19 Mar 2013 14:32:32 -0400
X-Google-Sender-Auth: ydQyyuxTqrr3J9k1lymfx0ajAk0
Message-ID: <CAGhGL2DZ30YKpZfekcXsx4m6OjGinOnJtzvFfFOmVQihnknh4g@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Content-Type: multipart/alternative; boundary=e89a8ff1c7ea238a9804d84b54a4
Cc: =?ISO-8859-1?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>, aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] AQM support in the network
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 18:32:33 -0000

--e89a8ff1c7ea238a9804d84b54a4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 19, 2013 at 5:40 AM, Mikael Abrahamsson <swmike@swm.pp.se>wrote=
:

> On Tue, 19 Mar 2013, Mirja K=FChlewind wrote:
>
>  Not sure who is subscribed to this list, but as a starting point I would
>> like to ask anybody who reads this and is in charge of operating any kin=
d
>> of network (isp, cdn provider, or smaller local networks) to post to the
>> list, if AQM is used anywhere in the network!
>>
>
> I would imagine that most ISPs use AQM on their core links, but this AQM
> is more WRED kind of AQM, where you have a few queues and place the traff=
ic
> into each queue based on DSCP/EXP values. This is mostly used to assure
> certain amount of bandwidth/loss profile to certain services, such as the
> providers own VoIP or IPTV service.


You might imagine so.  But from what I've seen, you'll often be
disappointed.   Your mileage will definitely vary.

Jim

--e89a8ff1c7ea238a9804d84b54a4
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Tue, Mar 19, 2013 at 5:40 AM, Mikael Abrahamsson <span dir=3D"lt=
r">&lt;<a href=3D"mailto:swmike@swm.pp.se" target=3D"_blank">swmike@swm.pp.=
se</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div class=3D"im">On Tue, 19 Mar 2013, Mirja=
 K=FChlewind wrote:<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Not sure who is subscribed to this list, but as a starting point I would li=
ke to ask anybody who reads this and is in charge of operating any kind of =
network (isp, cdn provider, or smaller local networks) to post to the list,=
 if AQM is used anywhere in the network!<br>

</blockquote>
<br></div>
I would imagine that most ISPs use AQM on their core links, but this AQM is=
 more WRED kind of AQM, where you have a few queues and place the traffic i=
nto each queue based on DSCP/EXP values. This is mostly used to assure cert=
ain amount of bandwidth/loss profile to certain services, such as the provi=
ders own VoIP or IPTV service.</blockquote>
<div><br></div><div style>You might imagine so. =A0But from what I&#39;ve s=
een, you&#39;ll often be disappointed. =A0 Your mileage will definitely var=
y.</div><div style><br></div><div style>Jim</div><div style><br></div></div=
>
</div></div>

--e89a8ff1c7ea238a9804d84b54a4--

From swmike@swm.pp.se  Tue Mar 19 11:37:04 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2105C21F8C48 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 11:37:04 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.418
X-Spam-Level: 
X-Spam-Status: No, score=-2.418 tagged_above=-999 required=5 tests=[AWL=-0.046, BAYES_00=-2.599, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R2KSQ7L+F-yp for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 11:37:03 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 9001C21F8B20 for <aqm@ietf.org>; Tue, 19 Mar 2013 11:37:03 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 56AA89C; Tue, 19 Mar 2013 19:37:02 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 4F8969A; Tue, 19 Mar 2013 19:37:02 +0100 (CET)
Date: Tue, 19 Mar 2013 19:37:02 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: Jim Gettys <jg@freedesktop.org>
In-Reply-To: <CAGhGL2DZ30YKpZfekcXsx4m6OjGinOnJtzvFfFOmVQihnknh4g@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1303191934100.2309@uplift.swm.pp.se>
References: <201303190906.32352.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191023220.2309@uplift.swm.pp.se> <CAGhGL2DZ30YKpZfekcXsx4m6OjGinOnJtzvFfFOmVQihnknh4g@mail.gmail.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: =?ISO-8859-15?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>, aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] AQM support in the network
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 18:37:04 -0000

On Tue, 19 Mar 2013, Jim Gettys wrote:

> You might imagine so.  But from what I've seen, you'll often be 
> disappointed.  Your mileage will definitely vary.

It's been a long time since I last saw buffering of more than 50ms on a 
congested core link (I'm talking 1G link or up).

But we're talking about 90% of the traffic being DSCP BE going into a 
single queue with RED dropping most traffic around 50ms of buffer depth.

That's why I was saying that I don't really want to call this AQM. It's 
not very "advanced".

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From ycheng@google.com  Tue Mar 19 13:18:20 2013
Return-Path: <ycheng@google.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D23F621F8DB4 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 13:18:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.678
X-Spam-Level: 
X-Spam-Status: No, score=-101.678 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, MIME_8BIT_HEADER=0.3, NO_RELAYS=-0.001, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8G+qIcFC0mhc for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 13:18:20 -0700 (PDT)
Received: from mail-la0-x234.google.com (mail-la0-x234.google.com [IPv6:2a00:1450:4010:c03::234]) by ietfa.amsl.com (Postfix) with ESMTP id AD4C921F8DE1 for <aqm@ietf.org>; Tue, 19 Mar 2013 13:18:19 -0700 (PDT)
Received: by mail-la0-f52.google.com with SMTP id fs12so1723063lab.25 for <aqm@ietf.org>; Tue, 19 Mar 2013 13:18:18 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:content-transfer-encoding; bh=i6jaxFlFqeOCJNUT8xMntKYkSDYDu3W39alFwdf0Hrw=; b=aMdp+ik84QE/dH4ZmK9UnRCWquiPXxyM9amQpzvyprYdJE6UFGJXXj7PcPmC52JDNy wR2AO0LdZynUuFyW1DC1toCFMgawLn/AwvYlPg3FP3YaBFXAnkzv8MB6OQudkAbghZQu jTMTgoGvw40dZbx7d3s78MrggUjvK9aee+ecs27TMK27REEbIpOJ+hzIRAyBm0e/W1T5 GfgHxjmJGRehRTyYdVz04XI/vBnE3M8mcykNuxd7XsYiK+vMy+btpgHc5WQc3QQm/AAN d8bpqK86g5vgGQhY/Ya3i97Su2eFPJUN7gbvKWw5qO0cydQ4QXlxmXYKQXjzbQBZIDuy O21Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:content-transfer-encoding :x-gm-message-state; bh=i6jaxFlFqeOCJNUT8xMntKYkSDYDu3W39alFwdf0Hrw=; b=MqbZ3jsn2P2sC8qV1vpMs83xrHG4k92sbA+8rFE04CdzM2z8jWEyeDwYmVt/AJA4lq Ihd87EsIie6R+rM4wIT6xtVsQ3Tt5nCq/axMg6+IR8gzWRJ+2PoS2NKGHym8rqk7dLSP oqGuSeIHJrpcKKotfpVAYGEo/6Hn6hmhZBN7jx7BZIl3b1kYPdBjTqY4n4o2cGU58LDP wjGzmA/R5kzWfeeiwPJCxfoAPGLX2MjUFEfdj9DLgbJaOWFiKRA8Fn9JsY6Bk46Q1E04 UrH5v6nrAFhyamXOgxsW5EJbg6tLtIBwhxwFgtJSV54ahZY3yQFottdozpDkKLhz+mM4 3avg==
X-Received: by 10.112.88.35 with SMTP id bd3mr8700119lbb.56.1363724298428; Tue, 19 Mar 2013 13:18:18 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.112.52.48 with HTTP; Tue, 19 Mar 2013 13:17:57 -0700 (PDT)
In-Reply-To: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
From: Yuchung Cheng <ycheng@google.com>
Date: Tue, 19 Mar 2013 13:17:57 -0700
Message-ID: <CAK6E8=dxZbT92K1mgg_E5zo8UQ1HhZe77c8SD8tqu0FXeLzYJw@mail.gmail.com>
To: =?ISO-8859-1?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Gm-Message-State: ALoCoQmtk3qMTeTzjDCIB6UjjRIFlZOkIH4gxbNCiPUViS8uR1Ru6QvBSsEXHkmx+ERXMAXuAV8XDuOLFb9fKBtU1MAbW75sY7mhFceIwuqmgk8FRuTTXuaCfBo2nsRniKUydVOEqBvyowx4dHu/m7mr6KGti7tZYOdaSZB0SQ0XXZ9T6pdTUGc3WDzRbqYBitHxzWGb7X+P
Cc: "tcpm@ietf.org Extensions" <tcpm@ietf.org>, iccrg@irtf.org, aqm@ietf.org, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 20:18:20 -0000

On Tue, Mar 19, 2013 at 1:04 AM, Mirja K=FChlewind
<mirja.kuehlewind@ikr.uni-stuttgart.de> wrote:
> Hi everybody,
>
> we just published a paper on the state of ECN support and usage on the
> Internet. We performed a similar study than Bauer et al., probing webserv=
ers
> to get the current status. We found that about 30% of webservers (by Sep'=
12)
> support ECN. Unfortunately, still we saw a failure rate of 9% when checki=
ng
> end-2-end path ECN usability (setting CE). Additionally, we analyzed NetF=
low
> data to get a lower bound for the ECN usage which was below 1%. Then we a=
lso
> had a look at IPv6 (47,5% ECN support). At the same time we also monitore=
d an
> increase in general IPv6 support on webserver over the IPv6 launch day la=
st
> year (-> check the paper).
>
> K=FChlewind, M.; Neuner, S.; Trammell, B.: On the state of ECN and TCP Op=
tions
> on the Internet. Proceedings of the 14th Passive and Active Measurement
> conference (PAM 2013), Hong Kong, March 2013.
>
> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/Kue_PAM13_40=
160.pdf
Do people on the list know any (good) evaluation of performance impact
of ECN on real and/or emulated networks. E.g., how much faster will
the Web be if ECN is used (correctly).

>
> Do we need to start a campaign to further encourage greater ECN support (=
also
> of network providers)? Asking the content providers on the list(s): What =
are
> the reasons to not enable ECN support?


>
> Mirja
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

From fred@cisco.com  Tue Mar 19 13:35:28 2013
Return-Path: <fred@cisco.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9AE2C21F8DDB; Tue, 19 Mar 2013 13:35:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -109.983
X-Spam-Level: 
X-Spam-Status: No, score=-109.983 tagged_above=-999 required=5 tests=[AWL=0.089, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qAvb25TqASzb; Tue, 19 Mar 2013 13:35:28 -0700 (PDT)
Received: from rcdn-iport-2.cisco.com (rcdn-iport-2.cisco.com [173.37.86.73]) by ietfa.amsl.com (Postfix) with ESMTP id D112521F8CC5; Tue, 19 Mar 2013 13:35:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=331; q=dns/txt; s=iport; t=1363725328; x=1364934928; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=XNMVNWO5Lp5RkLlTmmsAxKTNWN80fB9KJFfGYsZ+O0g=; b=JJX2iytnnT6D/9378RJrPchpb36elKjy/2T0woQzH716TrpUJzH+h9I4 wGK2zT5L2eqowhW9gWRBwCfSW1ALpnrddF/L86J5Q79zlRr6GhaC31i/b eaqhvuZS/gPXfzExUlavXr1EKhR4rKd5HKdZBf/KqObmSmPHq9jmBGF6A k=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: ApMFAC3LSFGtJV2b/2dsb2JhbABDh169P4FcFnSCJAEBAQMBeQULAgEIIiQyJQIEDgUIiAYGsXqQC45bAjEHgl9hA4g/nyKDCoIo
X-IronPort-AV: E=Sophos;i="4.84,873,1355097600"; d="scan'208";a="189218437"
Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by rcdn-iport-2.cisco.com with ESMTP; 19 Mar 2013 20:35:27 +0000
Received: from xhc-rcd-x06.cisco.com (xhc-rcd-x06.cisco.com [173.37.183.80]) by rcdn-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id r2JKZRur019494 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Tue, 19 Mar 2013 20:35:27 GMT
Received: from xmb-rcd-x09.cisco.com ([169.254.9.206]) by xhc-rcd-x06.cisco.com ([173.37.183.80]) with mapi id 14.02.0318.004; Tue, 19 Mar 2013 15:35:27 -0500
From: "Fred Baker (fred)" <fred@cisco.com>
To: =?iso-8859-1?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>
Thread-Topic: [aqm] ECN support and usage on the Internet
Thread-Index: AQHOJOFJiFyhVlNPAEK7GgUpyWANoA==
Date: Tue, 19 Mar 2013 20:35:26 +0000
Message-ID: <8C48B86A895913448548E6D15DA7553B7DC251@xmb-rcd-x09.cisco.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
In-Reply-To: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.21.100.174]
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <9832A55E311E6147AB87E5F5C2F70605@emea.cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "<tcpm@ietf.org>" <tcpm@ietf.org>, "<iccrg@irtf.org>" <iccrg@irtf.org>, "<aqm@ietf.org>" <aqm@ietf.org>, Brian Trammell <trammell@tik.ee.ethz.ch>
Subject: Re: [aqm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 20:35:28 -0000

On Mar 19, 2013, at 1:04 AM, Mirja K=FChlewind <mirja.kuehlewind@ikr.uni-st=
uttgart.de> wrote:

> Do we need to start a campaign to further encourage greater ECN support (=
also=20
> of network providers)? Asking the content providers on the list(s): What =
are=20
> the reasons to not enable ECN support?

probably.=

From gettysjim@gmail.com  Tue Mar 19 15:36:24 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2D17D21F8D12; Tue, 19 Mar 2013 15:36:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 3.733
X-Spam-Level: ***
X-Spam-Status: No, score=3.733 tagged_above=-999 required=5 tests=[AWL=-4.524,  BAYES_50=0.001, CHARSET_FARAWAY_HEADER=3.2, CN_BODY_35=0.339, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, MIME_8BIT_HEADER=0.3, MIME_CHARSET_FARAWAY=2.45, NO_RELAYS=-0.001, SARE_SUB_ENC_GB2312=1.345]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iZ8+yv2nSJvd; Tue, 19 Mar 2013 15:36:20 -0700 (PDT)
Received: from mail-ob0-x229.google.com (mail-ob0-x229.google.com [IPv6:2607:f8b0:4003:c01::229]) by ietfa.amsl.com (Postfix) with ESMTP id ECEDE21F8A98; Tue, 19 Mar 2013 15:36:19 -0700 (PDT)
Received: by mail-ob0-f169.google.com with SMTP id er7so12793obc.0 for <multiple recipients>; Tue, 19 Mar 2013 15:36:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=nokrkxSBgc7Ypd2Gs9HvTnnYp8GAzuhWZVd+vkD5B4U=; b=PosXCTRJXu2PSDDkR1+OKkMYJPe7gwdCOch2xLNDooHdtBSfbygzMv1SlJHjjMkF2E p5X8PtadxbFOxGX9l9fDS5AT1o8ZhoueyZ52fb4D12NfkrpSGsXs+WZ0XdqTXqtcJrJ0 OGbmgqgmlED9tsmKfMea87ras0a+ajAxlY00PeIg4Wv0sgJz388Z/QvE5br1IQtrbWSM MbUQM8ziogMz5zPfT4lfCIAtrCXE3ZDOU5ZMoPdeATX3z8G3UdYUuH0XRTGgM4u+Qnea FfRSlGv5pM1UpCMhd4fVCg/ldpvZAQa5w3St9Kx+LmJ1MgoyNpUcpm3iTespW1CBvr+q B42Q==
MIME-Version: 1.0
X-Received: by 10.60.3.71 with SMTP id a7mr2699262oea.35.1363732579347; Tue, 19 Mar 2013 15:36:19 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Tue, 19 Mar 2013 15:36:19 -0700 (PDT)
In-Reply-To: <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com>
Date: Tue, 19 Mar 2013 18:36:19 -0400
X-Google-Sender-Auth: --9eXhWCqOnC8mjcBV-gR9aVGYg
Message-ID: <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: =?GB2312?B?zOzAvQ==?= <tianlan.lhh@alibaba-inc.com>
Content-Type: multipart/alternative; boundary=e89a8f839d51f5f8c604d84ebb8b
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, Rick Jones <rick.jones2@hp.com>, "aqm@ietf.org" <aqm@ietf.org>, "iccrg@irtf.org" <iccrg@irtf.org>, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] =?gb2312?b?tPC4tDogW2ljY3JnXSBbdGNwbV0gRUNOIHN1cHBvcnQgYW5k?= =?gb2312?b?IHVzYWdlIG9uIHRoZSBJbnRlcm5ldA==?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 22:36:24 -0000

--e89a8f839d51f5f8c604d84ebb8b
Content-Type: text/plain; charset=GB2312
Content-Transfer-Encoding: quoted-printable

Clearly, having a working AQM is a necessary prerequisite for ECN to be
useful; one only overcome in the last year with CoDel/Pie.  Variable
bandwidth is still a challenge, but with CoDel at least, we think it can be
done.  And we've certainly implemented ECN support in the codel/fq_codel
qdisc's (though would be grateful if ECN experts make sure it's been done
correctly).

The incentive structure needs to work here. There may be things we can and
should do to provide incentive for ECN deployment.

For Bell Heads working on wireless cellular transport, the incentive is
really pretty clear: they ***hate*** dropping packets with a passion, not
understanding that either packet drop or ECN are necessary, and it will
probably be easier to convince them to adopt ECN than to drop packets.
 Given how hard they sweat to get packets over the air, I sort of
understand where they come from.

I believe some of the cellular carriers are requiring ECN support in
handsets going forward, and as they classify VOIP into a separate bearer
class anyway, I can see little incentive not to use ECN on their "best
effort" bearer class.

I believe all current operating systems react correctly to ECN marked
packets; the only question is whether they will talk ECN if talked to
(pretty common; Linux has done so now for quite a few years), *and* whether
they enable ECN when establishing new connections or not (no one does right
now by default).

At the moment, Windows is vanishingly small fraction of handsets, being
dominated by Android (Linux) and Apple as it is.

I worry about bufferbloat in the HARQ layer of cellular networks not
reflecting ECN marking into packets, but until that analysis has been done
in real systems, it's just one more thing to worry about. Our Linux
experience is that bufferbloat hides everywhere in current systems: the
analogy in WiFi is the error correction layer in 802.11n, which causes
large amounts of bloat, and fixing WiFi is very much a work in progress
(except that not enough resources are currently available to go do the
work, and it's more complicated that wired systems).  Dave Taht cheered the
first time he got the 802.11 layer to drop a packet, it was so much a
problem.

Right now, there are still some disincentives for ECN: e.g. some very small
fraction of sites that become "black holes", or crashing antique
middelboxes. We have no data on how much this is at this date, and the
database that used to exist documenting middlebox problems has bit-rotted.
 These  middleboxes are likely in the consumer edge, and unlikely to be in
the path of most mobile devices and web sites on 3g/LTE.  Mobile devices on
home WiFi networks are more likely to have problems.

We have ECN enabled in CeroWrt and have not noticed any problems; but 4-5
years ago OpenWrt (with a very much larger user base) still detected
problems to the extent they had to disable initiating ECN connections.

I know Bauer et. al. examined this a few years ago. But we have no data
over the last several years I've seen (though this new paper referenced
may: I've not had a chance to read it yet).

There is one other issue with ECN: when operating at low bandwidth at the
very edge of the net, it can still be advantageous to drop a packet rather
than ECN mark it, to get back the transmission time of the packet.  Dave
Taht's experiments (IIRC) were below 4Mbps for this advantage (when doing
VOIP over that bottleneck competing against other traffic in a best effort
class).

My intuition is to get VOIP reliable over WiFI on a busy network (or when
far away from the AP) we'll want to classify the VOIP packets anyway and
use the right hardware class in WiFi; this is primarily to get priority
access to a transmit opportunity on WiFi rather than the previous reasons
we've had.  Certainly on other media we've seen little reason to have to do
any explicit classification for VOIP anymore once running fq_codel.  Things
generally "just work" without all that hair for VOIP traffic.
                                                  - Jim





On Tue, Mar 19, 2013 at 5:49 PM, =CC=EC=C0=BD <tianlan.lhh@alibaba-inc.com>=
 wrote:

> To let ECN be popular used =A3=ACthe most important is let the connection=
 which
> used =A1=B0ECN=A1=B1 is fairness to the other connection;
> So we should design new algorithm for ECN connection to "prize" those
> people who use ECN;
>
>
>                                          hritian
>
> ________________________________________
> =B7=A2=BC=FE=C8=CB: Jim Gettys [jg@freedesktop.org]
> =B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA3=D4=C220=C8=D5 1:26
> =B5=BD: Rick Jones
> Cc: tcpm@ietf.org; aqm@ietf.org; iccrg@irtf.org; Mikael Abrahamsson
> =D6=F7=CC=E2: Re: [iccrg] [tcpm] ECN support and usage on the Internet
>
> You need to distinguish 1) "Will talk ECN when talked to", from 2) "Will
> initiate an ECN conversation".
>
> Linux, for example, will do 1) by default (and has for some time now), bu=
t
> requires configuration to turn on ECN when initiating a connection.
>                                   - Jim
>
>
>
> On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <rick.jones2@hp.com<mailto:
> rick.jones2@hp.com>> wrote:
> On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
> On Tue, 19 Mar 2013, Mirja K=A8=B9hlewind wrote:
>
> Do we need to start a campaign to further encourage greater ECN
> support (also of network providers)? Asking the content providers on
> the list(s): What are the reasons to not enable ECN support?
>
> Let me speculate:
>
> It's not on by default in Windows. There is little reason to turn it on,
> because "nobody" acts on ECN, and nobody acts on ECN because there is no
> demand for AQM, and even if there was AQM, it would most likely not be
> ECN enabled.
>
> Out of curiosity, which stacks have ECN enabled by default?  To my
> recollection, on netperf.org<http://netperf.org> I had to explicitly
> enable TCP's use of ECN in the kernel which ships with Ubuntu 12.04.
>
> rick jones
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org<mailto:tcpm@ietf.org>
> https://www.ietf.org/mailman/listinfo/tcpm
>
>
> ________________________________
>
> This email (including any attachments) is confidential and may be legally
> privileged. If you received this email in error, please delete it
> immediately and do not copy it or use it for any purpose or disclose its
> contents to any other person. Thank you.
>
>
> =B1=BE=B5=E7=D3=CA(=B0=FC=C0=A8=C8=CE=BA=CE=B8=BD=BC=FE)=BF=C9=C4=DC=BA=
=AC=D3=D0=BB=FA=C3=DC=D7=CA=C1=CF=B2=A2=CA=DC=B7=A8=C2=C9=B1=A3=BB=A4=A1=A3=
=C8=E7=C4=FA=B2=BB=CA=C7=D5=FD=C8=B7=B5=C4=CA=D5=BC=FE=C8=CB=A3=AC=C7=EB=C4=
=FA=C1=A2=BC=B4=C9=BE=B3=FD=B1=BE=D3=CA=BC=FE=A1=A3=C7=EB=B2=BB=D2=AA=BD=AB=
=B1=BE=B5=E7=D3=CA=BD=F8=D0=D0=B8=B4=D6=C6=B2=A2=D3=C3=D7=F7=C8=CE=BA=CE=C6=
=E4=CB=FB=D3=C3=CD=BE=A1=A2=BB=F2=CD=B8=C2=B6=B1=BE=D3=CA=BC=FE=D6=AE=C4=DA=
=C8=DD=A1=A3=D0=BB=D0=BB=A1=A3
>

--e89a8f839d51f5f8c604d84ebb8b
Content-Type: text/html; charset=GB2312
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Clearly, having a working AQM is a necessary prerequisite =
for ECN to be useful; one only overcome in the last year with CoDel/Pie. &n=
bsp;Variable bandwidth is still a challenge, but with CoDel at least, we th=
ink it can be done. &nbsp;And we&#39;ve certainly implemented ECN support i=
n the codel/fq_codel qdisc&#39;s (though would be grateful if ECN experts m=
ake sure it&#39;s been done correctly).<div>
<br></div><div>The incentive structure needs to work here. There may be thi=
ngs we can and should do to provide incentive for ECN deployment.</div><div=
><br></div><div>For Bell Heads working on wireless cellular transport, the =
incentive is really pretty clear: they ***hate*** dropping packets with a p=
assion, not understanding that either packet drop or ECN are necessary, and=
 it will probably be easier to convince them to adopt ECN than to drop pack=
ets. &nbsp;Given how hard they sweat to get packets over the air, I sort of=
 understand where they come from. &nbsp;</div>
<div><br></div><div>I believe some of the cellular carriers are requiring E=
CN support in handsets going forward, and as they classify VOIP into a sepa=
rate bearer class anyway, I can see little incentive not to use ECN on thei=
r &quot;best effort&quot; bearer class.</div>
<div><br></div><div style>I believe all current operating systems react cor=
rectly to ECN marked packets; the only question is whether they will talk E=
CN if talked to (pretty common; Linux has done so now for quite a few years=
), *and* whether they enable ECN when establishing new connections or not (=
no one does right now by default).</div>
<div><br></div><div style>At the moment, Windows is vanishingly small fract=
ion of handsets, being dominated by Android (Linux) and Apple as it is.</di=
v><div><br></div><div>I worry about bufferbloat in the HARQ layer of cellul=
ar networks not reflecting ECN marking into packets, but until that analysi=
s has been done in real systems, it&#39;s just one more thing to worry abou=
t. Our Linux experience is that bufferbloat hides everywhere in current sys=
tems: the analogy in WiFi is the error correction layer in 802.11n, which c=
auses large amounts of bloat, and fixing WiFi is very much a work in progre=
ss (except that not enough resources are currently available to go do the w=
ork, and it&#39;s more complicated that wired systems). &nbsp;Dave Taht che=
ered the first time he got the 802.11 layer to drop a packet, it was so muc=
h a problem.<br>
<div><div><br></div><div style>Right now, there are still some disincentive=
s for ECN: e.g. some very small fraction of sites that become &quot;black h=
oles&quot;, or crashing antique middelboxes. We have no data on how much th=
is is at this date, and the database that used to exist documenting middleb=
ox problems has bit-rotted. &nbsp;These &nbsp;middleboxes are likely in the=
 consumer edge, and unlikely to be in the path of most mobile devices and w=
eb sites on 3g/LTE. &nbsp;Mobile devices on home WiFi networks are more lik=
ely to have problems.</div>
<div style><br></div><div style>We have ECN enabled in CeroWrt and have not=
 noticed any problems; but 4-5 years ago OpenWrt (with a very much larger u=
ser base) still detected problems to the extent they had to disable initiat=
ing ECN connections. &nbsp;</div>
<div style><br></div><div style>I know Bauer et. al. examined this a few ye=
ars ago. But we have no data over the last several years I&#39;ve seen (tho=
ugh this new paper referenced may: I&#39;ve not had a chance to read it yet=
).</div>
</div><div><br></div><div style>There is one other issue with ECN: when ope=
rating at low bandwidth at the very edge of the net, it can still be advant=
ageous to drop a packet rather than ECN mark it, to get back the transmissi=
on time of the packet. &nbsp;Dave Taht&#39;s experiments (IIRC) were below =
4Mbps for this advantage (when doing VOIP over that bottleneck competing ag=
ainst other traffic in a best effort class).</div>
<div style><br></div><div style>My intuition is to get VOIP reliable over W=
iFI on a busy network (or when far away from the AP) we&#39;ll want to clas=
sify the VOIP packets anyway and use the right hardware class in WiFi; this=
 is primarily to get priority access to a transmit opportunity on WiFi rath=
er than the previous reasons we&#39;ve had. &nbsp;Certainly on other media =
we&#39;ve seen little reason to have to do any explicit classification for =
VOIP anymore once running fq_codel. &nbsp;Things generally &quot;just work&=
quot; without all that hair for VOIP traffic.</div>
<div style>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &=
nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;=
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - Jim</div><div style><br></div><div st=
yle><br></div><div style><br></div></div></div><div class=3D"gmail_extra"><=
br><br><div class=3D"gmail_quote">On Tue, Mar 19, 2013 at 5:49 PM, =CC=EC=
=C0=BD <span dir=3D"ltr">&lt;<a href=3D"mailto:tianlan.lhh@alibaba-inc.com"=
 target=3D"_blank">tianlan.lhh@alibaba-inc.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">To let ECN be popular used =A3=ACthe most im=
portant is let the connection which used &ldquo;ECN&rdquo; is fairness to t=
he other connection;<br>

So we should design new algorithm for ECN connection to &quot;prize&quot; t=
hose people who use ECN;<br>
<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp=
; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nb=
sp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &=
nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;=
 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbs=
p; &nbsp; &nbsp; &nbsp; &nbsp;hritian<br>
<br>
________________________________________<br>
=B7=A2=BC=FE=C8=CB: Jim Gettys [<a href=3D"mailto:jg@freedesktop.org">jg@fr=
eedesktop.org</a>]<br>
=B7=A2=CB=CD=CA=B1=BC=E4: 2013=C4=EA3=D4=C220=C8=D5 1:26<br>
=B5=BD: Rick Jones<br>
Cc: <a href=3D"mailto:tcpm@ietf.org">tcpm@ietf.org</a>; <a href=3D"mailto:a=
qm@ietf.org">aqm@ietf.org</a>; <a href=3D"mailto:iccrg@irtf.org">iccrg@irtf=
.org</a>; Mikael Abrahamsson<br>
=D6=F7=CC=E2: Re: [iccrg] [tcpm] ECN support and usage on the Internet<br>
<div class=3D"im"><br>
You need to distinguish 1) &quot;Will talk ECN when talked to&quot;, from 2=
) &quot;Will initiate an ECN conversation&quot;.<br>
<br>
Linux, for example, will do 1) by default (and has for some time now), but =
requires configuration to turn on ECN when initiating a connection.<br>
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp=
; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - Jim<br>
<br>
<br>
<br>
</div><div class=3D"im">On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones &lt;<a=
 href=3D"mailto:rick.jones2@hp.com">rick.jones2@hp.com</a>&lt;mailto:<a hre=
f=3D"mailto:rick.jones2@hp.com">rick.jones2@hp.com</a>&gt;&gt; wrote:<br>
On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:<br>
On Tue, 19 Mar 2013, Mirja K=A8=B9hlewind wrote:<br>
<br>
Do we need to start a campaign to further encourage greater ECN<br>
support (also of network providers)? Asking the content providers on<br>
the list(s): What are the reasons to not enable ECN support?<br>
<br>
Let me speculate:<br>
<br>
It&#39;s not on by default in Windows. There is little reason to turn it on=
,<br>
because &quot;nobody&quot; acts on ECN, and nobody acts on ECN because ther=
e is no<br>
demand for AQM, and even if there was AQM, it would most likely not be<br>
ECN enabled.<br>
<br>
</div>Out of curiosity, which stacks have ECN enabled by default? &nbsp;To =
my recollection, on <a href=3D"http://netperf.org" target=3D"_blank">netper=
f.org</a>&lt;<a href=3D"http://netperf.org" target=3D"_blank">http://netper=
f.org</a>&gt; I had to explicitly enable TCP&#39;s use of ECN in the kernel=
 which ships with Ubuntu 12.04.<br>

<div class=3D"im"><br>
rick jones<br>
<br>
_______________________________________________<br>
tcpm mailing list<br>
</div><a href=3D"mailto:tcpm@ietf.org">tcpm@ietf.org</a>&lt;mailto:<a href=
=3D"mailto:tcpm@ietf.org">tcpm@ietf.org</a>&gt;<br>
<a href=3D"https://www.ietf.org/mailman/listinfo/tcpm" target=3D"_blank">ht=
tps://www.ietf.org/mailman/listinfo/tcpm</a><br>
<br>
<br>
________________________________<br>
<br>
This email (including any attachments) is confidential and may be legally p=
rivileged. If you received this email in error, please delete it immediatel=
y and do not copy it or use it for any purpose or disclose its contents to =
any other person. Thank you.<br>

<br>
=B1=BE=B5=E7=D3=CA(=B0=FC=C0=A8=C8=CE=BA=CE=B8=BD=BC=FE)=BF=C9=C4=DC=BA=AC=
=D3=D0=BB=FA=C3=DC=D7=CA=C1=CF=B2=A2=CA=DC=B7=A8=C2=C9=B1=A3=BB=A4=A1=A3=C8=
=E7=C4=FA=B2=BB=CA=C7=D5=FD=C8=B7=B5=C4=CA=D5=BC=FE=C8=CB=A3=AC=C7=EB=C4=FA=
=C1=A2=BC=B4=C9=BE=B3=FD=B1=BE=D3=CA=BC=FE=A1=A3=C7=EB=B2=BB=D2=AA=BD=AB=B1=
=BE=B5=E7=D3=CA=BD=F8=D0=D0=B8=B4=D6=C6=B2=A2=D3=C3=D7=F7=C8=CE=BA=CE=C6=E4=
=CB=FB=D3=C3=CD=BE=A1=A2=BB=F2=CD=B8=C2=B6=B1=BE=D3=CA=BC=FE=D6=AE=C4=DA=C8=
=DD=A1=A3=D0=BB=D0=BB=A1=A3<br>
</blockquote></div><br></div>

--e89a8f839d51f5f8c604d84ebb8b--

From simon@superduper.net  Tue Mar 19 16:43:27 2013
Return-Path: <simon@superduper.net>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6AE7B21F8D46; Tue, 19 Mar 2013 16:43:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.92
X-Spam-Level: 
X-Spam-Status: No, score=-1.92 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152,  SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DVWy6nOSy+zV; Tue, 19 Mar 2013 16:43:26 -0700 (PDT)
Received: from masada.superduper.net (masada.superduper.net [85.119.82.91]) by ietfa.amsl.com (Postfix) with ESMTP id D48B721F8C5D; Tue, 19 Mar 2013 16:43:25 -0700 (PDT)
Received: from snappy-wlan.parc.xerox.com ([13.1.108.21]) by masada.superduper.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from <simon@superduper.net>) id 1UI6BX-0003Pr-9e; Tue, 19 Mar 2013 23:43:18 +0000
Message-ID: <5148F80F.5000401@superduper.net>
Date: Tue, 19 Mar 2013 16:43:11 -0700
From: Simon Barber <simon@superduper.net>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: Jim Gettys <jg@freedesktop.org>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
In-Reply-To: <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Cc: Rick Jones <rick.jones2@hp.com>, "iccrg@irtf.org" <iccrg@irtf.org>, "tcpm@ietf.org" <tcpm@ietf.org>, =?UTF-8?B?5aSp5r6c?= <tianlan.lhh@alibaba-inc.com>, "aqm@ietf.org" <aqm@ietf.org>, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] =?utf-8?b?562U5aSNOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBh?= =?utf-8?q?nd_usage_on_the_Internet?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 23:43:27 -0000

I'm not sure why people are so obsessed with ECN - the little data I've 
seen about it showed that it's effect on system throughput was in the 
noise. Sure, wireless guys don't like to waste precious wireless 
bandwidth, but they're not dropping packets that have used up the 
precious wireless bandwidth. They get dropped before the wireless link.

Simon

On 03/19/2013 03:36 PM, Jim Gettys wrote:
> Clearly, having a working AQM is a necessary prerequisite for ECN to be
> useful; one only overcome in the last year with CoDel/Pie.  Variable
> bandwidth is still a challenge, but with CoDel at least, we think it can
> be done.  And we've certainly implemented ECN support in the
> codel/fq_codel qdisc's (though would be grateful if ECN experts make
> sure it's been done correctly).
>
> The incentive structure needs to work here. There may be things we can
> and should do to provide incentive for ECN deployment.
>
> For Bell Heads working on wireless cellular transport, the incentive is
> really pretty clear: they ***hate*** dropping packets with a passion,
> not understanding that either packet drop or ECN are necessary, and it
> will probably be easier to convince them to adopt ECN than to drop
> packets.  Given how hard they sweat to get packets over the air, I sort
> of understand where they come from.
>
> I believe some of the cellular carriers are requiring ECN support in
> handsets going forward, and as they classify VOIP into a separate bearer
> class anyway, I can see little incentive not to use ECN on their "best
> effort" bearer class.
>
> I believe all current operating systems react correctly to ECN marked
> packets; the only question is whether they will talk ECN if talked to
> (pretty common; Linux has done so now for quite a few years), *and*
> whether they enable ECN when establishing new connections or not (no one
> does right now by default).
>
> At the moment, Windows is vanishingly small fraction of handsets, being
> dominated by Android (Linux) and Apple as it is.
>
> I worry about bufferbloat in the HARQ layer of cellular networks not
> reflecting ECN marking into packets, but until that analysis has been
> done in real systems, it's just one more thing to worry about. Our Linux
> experience is that bufferbloat hides everywhere in current systems: the
> analogy in WiFi is the error correction layer in 802.11n, which causes
> large amounts of bloat, and fixing WiFi is very much a work in progress
> (except that not enough resources are currently available to go do the
> work, and it's more complicated that wired systems).  Dave Taht cheered
> the first time he got the 802.11 layer to drop a packet, it was so much
> a problem.
>
> Right now, there are still some disincentives for ECN: e.g. some very
> small fraction of sites that become "black holes", or crashing antique
> middelboxes. We have no data on how much this is at this date, and the
> database that used to exist documenting middlebox problems has
> bit-rotted.  These  middleboxes are likely in the consumer edge, and
> unlikely to be in the path of most mobile devices and web sites on
> 3g/LTE.  Mobile devices on home WiFi networks are more likely to have
> problems.
>
> We have ECN enabled in CeroWrt and have not noticed any problems; but
> 4-5 years ago OpenWrt (with a very much larger user base) still detected
> problems to the extent they had to disable initiating ECN connections.
>
> I know Bauer et. al. examined this a few years ago. But we have no data
> over the last several years I've seen (though this new paper referenced
> may: I've not had a chance to read it yet).
>
> There is one other issue with ECN: when operating at low bandwidth at
> the very edge of the net, it can still be advantageous to drop a packet
> rather than ECN mark it, to get back the transmission time of the
> packet.  Dave Taht's experiments (IIRC) were below 4Mbps for this
> advantage (when doing VOIP over that bottleneck competing against other
> traffic in a best effort class).
>
> My intuition is to get VOIP reliable over WiFI on a busy network (or
> when far away from the AP) we'll want to classify the VOIP packets
> anyway and use the right hardware class in WiFi; this is primarily to
> get priority access to a transmit opportunity on WiFi rather than the
> previous reasons we've had.  Certainly on other media we've seen little
> reason to have to do any explicit classification for VOIP anymore once
> running fq_codel.  Things generally "just work" without all that hair
> for VOIP traffic.
>                                                    - Jim
>
>
>
>
>
> On Tue, Mar 19, 2013 at 5:49 PM, 天澜 <tianlan.lhh@alibaba-inc.com
> <mailto:tianlan.lhh@alibaba-inc.com>> wrote:
>
>     To let ECN be popular used ，the most important is let the
>     connection which used “ECN” is fairness to the other connection;
>     So we should design new algorithm for ECN connection to "prize"
>     those people who use ECN;
>
>
>                                                     hritian
>
>     ________________________________________
>     发件人: Jim Gettys [jg@freedesktop.org <mailto:jg@freedesktop.org>]
>     发送时间: 2013年3月20日 1:26
>     到: Rick Jones
>     Cc: tcpm@ietf.org <mailto:tcpm@ietf.org>; aqm@ietf.org
>     <mailto:aqm@ietf.org>; iccrg@irtf.org <mailto:iccrg@irtf.org>;
>     Mikael Abrahamsson
>     主题: Re: [iccrg] [tcpm] ECN support and usage on the Internet
>
>     You need to distinguish 1) "Will talk ECN when talked to", from 2)
>     "Will initiate an ECN conversation".
>
>     Linux, for example, will do 1) by default (and has for some time
>     now), but requires configuration to turn on ECN when initiating a
>     connection.
>                                        - Jim
>
>
>
>     On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <rick.jones2@hp.com
>     <mailto:rick.jones2@hp.com><mailto:rick.jones2@hp.com
>     <mailto:rick.jones2@hp.com>>> wrote:
>     On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
>     On Tue, 19 Mar 2013, Mirja Kühlewind wrote:
>
>     Do we need to start a campaign to further encourage greater ECN
>     support (also of network providers)? Asking the content providers on
>     the list(s): What are the reasons to not enable ECN support?
>
>     Let me speculate:
>
>     It's not on by default in Windows. There is little reason to turn it on,
>     because "nobody" acts on ECN, and nobody acts on ECN because there is no
>     demand for AQM, and even if there was AQM, it would most likely not be
>     ECN enabled.
>
>     Out of curiosity, which stacks have ECN enabled by default?  To my
>     recollection, on netperf.org
>     <http://netperf.org><http://netperf.org> I had to explicitly enable
>     TCP's use of ECN in the kernel which ships with Ubuntu 12.04.
>
>     rick jones
>
>     _______________________________________________
>     tcpm mailing list
>     tcpm@ietf.org <mailto:tcpm@ietf.org><mailto:tcpm@ietf.org
>     <mailto:tcpm@ietf.org>>
>     https://www.ietf.org/mailman/listinfo/tcpm
>
>
>     ________________________________
>
>     This email (including any attachments) is confidential and may be
>     legally privileged. If you received this email in error, please
>     delete it immediately and do not copy it or use it for any purpose
>     or disclose its contents to any other person. Thank you.
>
>     本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件
>     人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或
>     透露本邮件之内容。谢谢。
>
>
>
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>

From rick.jones2@hp.com  Tue Mar 19 07:46:16 2013
Return-Path: <rick.jones2@hp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D2A1C21F8CA7; Tue, 19 Mar 2013 07:46:16 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -108.599
X-Spam-Level: 
X-Spam-Status: No, score=-108.599 tagged_above=-999 required=5 tests=[AWL=2.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FEzOgMtXyXp2; Tue, 19 Mar 2013 07:46:16 -0700 (PDT)
Received: from g1t0027.austin.hp.com (g1t0027.austin.hp.com [15.216.28.34]) by ietfa.amsl.com (Postfix) with ESMTP id 2C0D721F8CA3; Tue, 19 Mar 2013 07:46:16 -0700 (PDT)
Received: from g1t0039.austin.hp.com (g1t0039.austin.hp.com [16.236.32.45]) by g1t0027.austin.hp.com (Postfix) with ESMTP id 161CA38579; Tue, 19 Mar 2013 14:46:09 +0000 (UTC)
Received: from [16.103.148.51] (tardy.usa.hp.com [16.103.148.51]) by g1t0039.austin.hp.com (Postfix) with ESMTP id 933A33454E; Tue, 19 Mar 2013 14:46:00 +0000 (UTC)
Message-ID: <51487A27.7010904@hp.com>
Date: Tue, 19 Mar 2013 07:45:59 -0700
From: Rick Jones <rick.jones2@hp.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2
MIME-Version: 1.0
To: Mikael Abrahamsson <swmike@swm.pp.se>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se>
In-Reply-To: <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Mailman-Approved-At: Tue, 19 Mar 2013 19:00:18 -0700
Cc: tcpm@ietf.org, =?UTF-8?B?TWlyamEgS8O8aGxld2luZA==?= <mirja.kuehlewind@ikr.uni-stuttgart.de>, aqm@ietf.org, iccrg@irtf.org
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 14:46:17 -0000

On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
> On Tue, 19 Mar 2013, Mirja Kühlewind wrote:
>
>> Do we need to start a campaign to further encourage greater ECN
>> support (also of network providers)? Asking the content providers on
>> the list(s): What are the reasons to not enable ECN support?
>
> Let me speculate:
>
> It's not on by default in Windows. There is little reason to turn it on,
> because "nobody" acts on ECN, and nobody acts on ECN because there is no
> demand for AQM, and even if there was AQM, it would most likely not be
> ECN enabled.

Out of curiosity, which stacks have ECN enabled by default?  To my 
recollection, on netperf.org I had to explicitly enable TCP's use of ECN 
in the kernel which ships with Ubuntu 12.04.

rick jones

From gdt@gdt.id.au  Tue Mar 19 16:09:07 2013
Return-Path: <gdt@gdt.id.au>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C096621F867D; Tue, 19 Mar 2013 16:09:07 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.91
X-Spam-Level: 
X-Spam-Status: No, score=-1.91 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_AU=0.377, HOST_MISMATCH_NET=0.311, HTML_MESSAGE=0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B+g11tgLWAzB; Tue, 19 Mar 2013 16:09:07 -0700 (PDT)
Received: from aix.gdt.id.au (eth6445.sa.adsl.internode.on.net [150.101.30.44]) by ietfa.amsl.com (Postfix) with ESMTP id ABF0321F84D9; Tue, 19 Mar 2013 16:09:06 -0700 (PDT)
Received: from thrace.44ansell.gdt.id.au (thrace.44ansell.gdt.id.au [192.168.253.149]) (authenticated bits=0) by aix.gdt.id.au (8.14.2/8.14.2) with ESMTP id r2JN8baD026631 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 20 Mar 2013 09:38:38 +1030
Mime-Version: 1.0 (Apple Message framework v1283)
Content-Type: multipart/alternative; boundary="Apple-Mail=_D3CEF40D-2524-4B60-9F81-2151B0CC81D6"
From: Glen Turner <gdt@gdt.id.au>
In-Reply-To: <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
Date: Wed, 20 Mar 2013 09:38:37 +1030
Message-Id: <BB7258DC-0CC1-403E-B19D-4E86F43143D4@gdt.id.au>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
To: Jim Gettys <jg@freedesktop.org>
X-Mailer: Apple Mail (2.1283)
X-Mailman-Approved-At: Tue, 19 Mar 2013 19:00:18 -0700
Cc: Rick Jones <rick.jones2@hp.com>, "iccrg@irtf.org" <iccrg@irtf.org>, "tcpm@ietf.org" <tcpm@ietf.org>, =?utf-8?B?5aSp5r6c?= <tianlan.lhh@alibaba-inc.com>, "aqm@ietf.org" <aqm@ietf.org>, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] [iccrg]  [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Mar 2013 23:09:07 -0000

--Apple-Mail=_D3CEF40D-2524-4B60-9F81-2151B0CC81D6
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=GB2312


On 20/03/2013, at 9:06 AM, Jim Gettys wrote:
>=20
> We have ECN enabled in CeroWrt and have not noticed any problems; but =
4-5 years ago OpenWrt (with a very much larger user base) still detected =
problems to the extent they had to disable initiating ECN connections. =20=


The issues with accessing web sites from ECN-using hosts were primarily =
due to one software bug in one popular network vendor's popular =
firewall. The software versions with that bug have been end-of-lifed by =
the vendor. So you should have a much better experience with ECN =
reachability these days.

-glen=

--Apple-Mail=_D3CEF40D-2524-4B60-9F81-2151B0CC81D6
Content-Transfer-Encoding: 7bit
Content-Type: text/html;
	charset=GB2312

<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On 20/03/2013, at 9:06 AM, Jim Gettys wrote:</div><blockquote type="cite"><div dir="ltr"><div><div><div style=""><font class="Apple-style-span" color="#000000"><br></font></div><div style="">We have ECN enabled in CeroWrt and have not noticed any problems; but 4-5 years ago OpenWrt (with a very much larger user base) still detected problems to the extent they had to disable initiating ECN connections. &nbsp;</div></div></div></div></blockquote><div><br></div><div>The issues with accessing web sites from ECN-using hosts were primarily due to one software bug in one popular network vendor's popular firewall. The software versions with that bug have been end-of-lifed by the vendor. So you should have a much better experience with ECN reachability these days.</div><div><br></div><div>-glen</div></div></body></html>
--Apple-Mail=_D3CEF40D-2524-4B60-9F81-2151B0CC81D6--

From dpreed@reed.com  Tue Mar 19 20:02:22 2013
Return-Path: <dpreed@reed.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 80ABE21F8CF8 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 20:02:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.371
X-Spam-Level: 
X-Spam-Status: No, score=-3.371 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e18NqN1OtoR8 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 20:02:21 -0700 (PDT)
Received: from smtp176.iad.emailsrvr.com (smtp176.iad.emailsrvr.com [207.97.245.176]) by ietfa.amsl.com (Postfix) with ESMTP id 8D2AD21F8F1C for <aqm@ietf.org>; Tue, 19 Mar 2013 20:02:21 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp57.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id E6A09F84A4 for <aqm@ietf.org>; Tue, 19 Mar 2013 23:02:20 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy2.wa-web.iad1a (legacy2.wa-web.iad1a.rsapps.net [192.168.2.218]) by smtp57.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id C3C30F83AE for <aqm@ietf.org>; Tue, 19 Mar 2013 23:02:20 -0400 (EDT)
Received: from reed.com (localhost [127.0.0.1]) by legacy2.wa-web.iad1a (Postfix) with ESMTP id 9CF52E00B5; Tue, 19 Mar 2013 23:02:20 -0400 (EDT)
Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com)  with HTTP; Tue, 19 Mar 2013 23:02:20 -0400 (EDT)
Date: Tue, 19 Mar 2013 23:02:20 -0400 (EDT)
From: dpreed@reed.com
To: aqm@ietf.org
Cc: aqm@ietf.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_20130319230220000000_71582"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
In-Reply-To: <mailman.5225.1363736608.3432.aqm@ietf.org>
References: <mailman.5225.1363736608.3432.aqm@ietf.org>
Message-ID: <1363748540.64155384@apps.rackspace.com>
X-Mailer: webmail7.0
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 03:02:22 -0000

------=_20130319230220000000_71582
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0AA problem with ECN is that it does not drain an already flooded queue, a=
nd depends (like RED) on the idea that one has a large queue in normal oper=
ation.  It is not obvious that sustaining a deep queue is a desirable state=
, especially when there are highly variable capacities along any path, as i=
s the case where one link is more than 2x slower than the rest of the links=
.=0A =0AIf the dynamics of user loads are fractal, this will cause draining=
 (and latency reduction) not to work.=0A =0AIf user loads are stationary ga=
ussian and so forth, ECN can "tune" to relatively stable situations, but th=
e queueing delay will be quite long end-to-end.=0A =0ASince there is lots o=
f evidence that user loads are fractal, dropping packets during sudden burs=
ts will prevent sustained queueing, whereas ECN will not.=0A =0ASo a proper=
 approach (in my opinion) on the "real" Internet (rather than a bunch of co=
mpeting long-duration FTPs) requires keeping the queues so short that there=
 is no real opportunity to benefit from ECN.=0A =0AHence - we need to reall=
y see what happens with *real* traffic loads, not simplified simulations.=
=0A =0AReal traffic loads are fractal, and fractal loads do not generally o=
bey the Gaussian style "law of large numbers" - multiple fractal flows are =
*rougher* not smoother than a single one.=0A =0ASo check your mathematical =
"intuitions" at the door.  Stop writing equations based on tractable analyt=
ic models, because actual traffic does not follow those models.=0A =0AYeah,=
 the Bell Heads believe in never dropping packets.  So what.  The Internet =
is not about making them comfortable in their misunderstandings.=0A =0A----=
-Original Message-----=0AFrom: aqm-request@ietf.org=0ASent: Tuesday, March =
19, 2013 7:43pm=0ATo: aqm@ietf.org=0ASubject: aqm Digest, Vol 1, Issue 7=0A=
=0A=0A=0AIf you have received this digest without all the individual messag=
e=0Aattachments you will need to update your digest options in your list=0A=
subscription.  To do so, go to =0A=0Ahttps://www.ietf.org/mailman/listinfo/=
aqm=0A=0AClick the 'Unsubscribe or edit options' button, log in, and set "G=
et=0AMIME or Plain Text Digests?" to MIME.  You can set this option=0Agloba=
lly for all the list digests you receive at this point.=0A=0A=0A=0ASend aqm=
 mailing list submissions to=0A aqm@ietf.org=0A=0ATo subscribe or unsubscri=
be via the World Wide Web, visit=0A https://www.ietf.org/mailman/listinfo/a=
qm=0Aor, via email, send a message with subject or body 'help' to=0A aqm-re=
quest@ietf.org=0A=0AYou can reach the person managing the list at=0A aqm-ow=
ner@ietf.org=0A=0AWhen replying, please edit your Subject line so it is mor=
e specific=0Athan "Re: Contents of aqm digest..."=0AToday's Topics:=0A=0A 1=
. Re: [tcpm] ECN support and usage on the Internet (Yuchung Cheng)=0A 2. Re=
: ECN support and usage on the Internet (Fred Baker (fred))=0A 3. Re: ??: [=
iccrg] [tcpm] ECN support and usage on the Internet=0A (Jim Gettys)=0A 4. R=
e: ??: [iccrg] [tcpm] ECN support and usage on the Internet=0A (Simon Barbe=
r)=0A_______________________________________________=0Aaqm mailing list=0Aa=
qm@ietf.org=0Ahttps://www.ietf.org/mailman/listinfo/aqm
------=_20130319230220000000_71582
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"times new roman" size=3D"2"><p style=3D"margin:0;padding:0;">=
A problem with ECN is that it does not drain an already flooded queue, and =
depends (like RED) on the idea that one has a large queue in normal operati=
on.&nbsp; It is not obvious that sustaining a deep queue is a desirable sta=
te, especially when there are highly variable capacities along any path, as=
 is the case where one link is more than 2x slower than the rest of the lin=
ks.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:=
0;padding:0;">If the dynamics of user loads are fractal, this will cause dr=
aining (and latency reduction) not to work.</p>=0A<p style=3D"margin:0;padd=
ing:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">If user loads are sta=
tionary gaussian and so forth, ECN can "tune" to relatively stable situatio=
ns, but the queueing delay will be quite long end-to-end.</p>=0A<p style=3D=
"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">Since t=
here is lots of evidence that user loads are fractal, dropping packets duri=
ng sudden bursts will prevent sustained queueing, whereas ECN will not.</p>=
=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;paddi=
ng:0;">So a proper approach (in my opinion) on the "real" Internet (rather =
than a bunch of competing long-duration FTPs) requires keeping the queues s=
o short that there is no real opportunity to benefit from ECN.</p>=0A<p sty=
le=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">He=
nce - we need to really see what happens with *real* traffic loads, not sim=
plified simulations.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p=
 style=3D"margin:0;padding:0;">Real traffic loads are fractal, and fractal =
loads do not generally obey the Gaussian style "law of large numbers" - mul=
tiple fractal flows are *rougher* not smoother than a single one.</p>=0A<p =
style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;"=
>So check your mathematical "intuitions" at the door.&nbsp; Stop writing eq=
uations based on tractable analytic models, because actual traffic does not=
 follow those models.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<=
p style=3D"margin:0;padding:0;">Yeah, the Bell Heads believe in never dropp=
ing packets.&nbsp; So what.&nbsp; The Internet is not about making them com=
fortable in their misunderstandings.</p>=0A<p style=3D"margin:0;padding:0;"=
>&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">-----Original Message-----<b=
r />From: aqm-request@ietf.org<br />Sent: Tuesday, March 19, 2013 7:43pm<br=
 />To: aqm@ietf.org<br />Subject: aqm Digest, Vol 1, Issue 7<br /><br /></p=
>=0A<div id=3D"SafeStyles1363747882">=0A<p style=3D"margin:0;padding:0;">If=
 you have received this digest without all the individual message<br />atta=
chments you will need to update your digest options in your list<br />subsc=
ription.  To do so, go to <br /><br />https://www.ietf.org/mailman/listinfo=
/aqm<br /><br />Click the 'Unsubscribe or edit options' button, log in, and=
 set "Get<br />MIME or Plain Text Digests?" to MIME.  You can set this opti=
on<br />globally for all the list digests you receive at this point.<br /><=
br /><br /><br />Send aqm mailing list submissions to<br /> aqm@ietf.org<br=
 /><br />To subscribe or unsubscribe via the World Wide Web, visit<br /> ht=
tps://www.ietf.org/mailman/listinfo/aqm<br />or, via email, send a message =
with subject or body 'help' to<br /> aqm-request@ietf.org<br /><br />You ca=
n reach the person managing the list at<br /> aqm-owner@ietf.org<br /><br /=
>When replying, please edit your Subject line so it is more specific<br />t=
han "Re: Contents of aqm digest..."<br />Today's Topics:<br /><br /> 1. Re:=
 [tcpm] ECN support and usage on the Internet (Yuchung Cheng)<br /> 2. Re: =
ECN support and usage on the Internet (Fred Baker (fred))<br /> 3. Re: ??: =
[iccrg] [tcpm] ECN support and usage on the Internet<br /> (Jim Gettys)<br =
/> 4. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet<br /> (S=
imon Barber)<br />_______________________________________________<br />aqm =
mailing list<br />aqm@ietf.org<br />https://www.ietf.org/mailman/listinfo/a=
qm</p>=0A</div></font>
------=_20130319230220000000_71582--


From john@jlc.net  Tue Mar 19 20:56:50 2013
Return-Path: <john@jlc.net>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 61FAD21F8F28 for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 20:56:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.599
X-Spam-Level: 
X-Spam-Status: No, score=-106.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n0BbQp4eZADw for <aqm@ietfa.amsl.com>; Tue, 19 Mar 2013 20:56:49 -0700 (PDT)
Received: from mailhost.jlc.net (mailhost.jlc.net [199.201.159.4]) by ietfa.amsl.com (Postfix) with ESMTP id 9757A21F8F2D for <aqm@ietf.org>; Tue, 19 Mar 2013 20:56:45 -0700 (PDT)
Received: by mailhost.jlc.net (Postfix, from userid 104) id CF1D333C22; Tue, 19 Mar 2013 23:56:45 -0400 (EDT)
Date: Tue, 19 Mar 2013 23:56:45 -0400
From: John Leslie <john@jlc.net>
To: dpreed@reed.com
Message-ID: <20130320035645.GI9569@verdi>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1363748540.64155384@apps.rackspace.com>
User-Agent: Mutt/1.4.1i
Cc: aqm@ietf.org
Subject: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 03:56:50 -0000

dpreed@reed.com <dpreed@reed.com> wrote:
> 
> A problem with ECN is that it does not drain an already flooded queue,

   I presume you mean that ECN doesn't drop the packet it marks.

> and depends (like RED) on the idea that one has a large queue in normal
> operation.

   I don't follow this at all. ECN is pointless without an AQM which
drops or marks well before tail-drop. Were we to run ECN without AQM,
we'd have to drop each packet anyway after marking it.

   Several flavors of AQM mark/drop well enough in advance of tail-drop
that the queue need never fill. Did you mean to say that ECN requires
being able to queue the ECN-marked packet which would otherwise be
dropped by the AQM?

> It is not obvious that sustaining a deep queue is a desirable state,

   Agreed!

> especially when there are highly variable capacities along any path,
> as is the case where one link is more than 2x slower than the rest of
> the links.

   Isn't that _always_ the case?

> If the dynamics of user loads are fractal, this will cause draining
> (and latency reduction) not to work.

   I'm guessing you mean that under "fractal" loads there's a tendency
for ECN-marked packets to clump and introduce latency at the bottleneck.
Recall that this is offset by the quicker response to an ECN mark at
RTT speed instead of having to infer a packet loss.

> If user loads are stationary gaussian and so forth, ECN can "tune"
> to relatively stable situations, but the queueing delay will be quite
> long end-to-end.

   I don't follow... Some AQMs may indeed allow queueing delay to grow
large; but this can only happen if the instantaneous percentage of
marked packets is quite a bit higher than TCP can tolerate.

> Since there is lots of evidence that user loads are fractal, dropping
> packets during sudden bursts will prevent sustained queueing, whereas
> ECN will not.

   I believe there is substantial agreement that ECN-marked packets
will need to be dropped anyway under particularly severe congestion.
At the very least, tail-drop can become necessary...

> So a proper approach (in my opinion) on the "real" Internet (rather
> than a bunch of competing long-duration FTPs) requires keeping the
> queues so short that there is no real opportunity to benefit from ECN.

   Here I quite disagree.

   An AQM could be designed to ECN-mark at a 1 millisecond queue depth;
and that can give the full benefit of ECN. Again, recall that ECN will
deliver the congestion signal in one RTT, instead of waiting to infer
a loss. TCP stops delivering useful bandwidth at one or two percent
packet loss: so it's hard to believe that ECN-marked packets will clog
the path significantly.

> Hence - we need to really see what happens with *real* traffic loads,
> not simplified simulations.

   Agreed! (Now if we could just get enough ECN deployment...)

> Real traffic loads are fractal, and fractal loads do not generally
> obey the Gaussian style "law of large numbers" - multiple fractal
> flows are *rougher* not smoother than a single one.

   I would say rather that real traffic loads _contain_ fractal flows.
Aggregation simply wouldn't work if we never experienced gaussian
smoothing.

> So check your mathematical "intuitions" at the door.  Stop writing
> equations based on tractable analytic models, because actual traffic
> does not follow those models.

   I strongly suspect our models _are_ faulty -- but we need research
results for _actual_ flows, with and without ECN, is order to diagnose
their faults.

--
John Leslie <john@jlc.net>

From swmike@swm.pp.se  Tue Mar 19 21:01:25 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 44DA421F8EF2; Tue, 19 Mar 2013 21:01:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.091
X-Spam-Level: 
X-Spam-Status: No, score=0.091 tagged_above=-999 required=5 tests=[AWL=-2.155,  BAYES_00=-2.599, CHARSET_FARAWAY_HEADER=3.2, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_GB2312=1.345]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pa1z3GoIxTxq; Tue, 19 Mar 2013 21:01:24 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id A749921F8F22; Tue, 19 Mar 2013 21:01:24 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id A878A9C; Wed, 20 Mar 2013 05:01:22 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id A37A79A; Wed, 20 Mar 2013 05:01:22 +0100 (CET)
Date: Wed, 20 Mar 2013 05:01:22 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: Jim Gettys <jg@freedesktop.org>
In-Reply-To: <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1303200451270.2309@uplift.swm.pp.se>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, "aqm@ietf.org" <aqm@ietf.org>, "iccrg@irtf.org" <iccrg@irtf.org>
Subject: Re: [aqm] =?gb2312?b?tPC4tDogW2ljY3JnXSBbdGNwbV0gRUNOIHN1cHBvcnQgYW5k?= =?gb2312?b?IHVzYWdlIG9uIHRoZSBJbnRlcm5ldA==?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 04:01:25 -0000

On Tue, 19 Mar 2013, Jim Gettys wrote:

> I worry about bufferbloat in the HARQ layer of cellular networks not

This might be from just one vendor, but the Radio Link Control (RLC) layer 
on our equipment will buffer up to 400 packets before it does tail-drop. 
There might be additional buffering in GGSN/SGSN doing rate-shaping before 
it's even sent to the RNC (which terminates the RLC layer between the RNC 
and the end user terminal equipment). All this buffering is strict FIFO.

I personally belive that AQM at each of these buffering points would be 
beneficial to the end user equipment.

> My intuition is to get VOIP reliable over WiFI on a busy network (or 
> when far away from the AP) we'll want to classify the VOIP packets 
> anyway and use the right hardware class in WiFi; this is primarily to 
> get priority access to a transmit opportunity on WiFi rather than the 
> previous reasons we've had.  Certainly on other media we've seen little 
> reason to have to do any explicit classification for VOIP anymore once 
> running fq_codel.  Things generally "just work" without all that hair 
> for VOIP traffic.

I agree, fq_codel probably needs some DSCP handling so that VOIP packets 
(EF marked) would end up in a different hardware queue. For mobile, the 
bearer model would mean there would be an L4 inspection engine putting the 
traffic into the correct bearer anyway (that's the 3GPP way), so here 
codel probably wouldn't need to do anything (because there would be 
several codel instances running, one per bearer).

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From ford@isoc.org  Wed Mar 20 04:14:47 2013
Return-Path: <ford@isoc.org>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 843E821F8B8A for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 04:14:47 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -103.372
X-Spam-Level: 
X-Spam-Status: No, score=-103.372 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kzzv4MdYPFnF for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 04:14:46 -0700 (PDT)
Received: from smtp196.iad.emailsrvr.com (smtp196.iad.emailsrvr.com [207.97.245.196]) by ietfa.amsl.com (Postfix) with ESMTP id 4B27D21F84E3 for <aqm@ietf.org>; Wed, 20 Mar 2013 04:14:46 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp59.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id B96693F05B1; Wed, 20 Mar 2013 07:14:45 -0400 (EDT)
X-Virus-Scanned: OK
Received: by smtp59.relay.iad1a.emailsrvr.com (Authenticated sender: ford-AT-isoc.org) with ESMTPSA id 1C5C33F05E9;  Wed, 20 Mar 2013 07:14:44 -0400 (EDT)
Content-Type: text/plain; charset=GB2312
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
From: Matthew Ford <ford@isoc.org>
In-Reply-To: <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
Date: Wed, 20 Mar 2013 11:14:42 +0000
Content-Transfer-Encoding: quoted-printable
Message-Id: <C9DA5524-F6FB-4A8C-9110-B2CC1FA121ED@isoc.org>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com>
To: Jim Gettys <jg@freedesktop.org>
X-Mailer: Apple Mail (2.1503)
Cc: iccrg@irtf.org, aqm@ietf.org
Subject: Re: [aqm] [iccrg] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 11:14:48 -0000

(trimmed the cc list)

On 19 Mar 2013, at 22:36, Jim Gettys <jg@freedesktop.org> wrote:

> There is one other issue with ECN: when operating at low bandwidth at =
the very edge of the net, it can still be advantageous to drop a packet =
rather than ECN mark it, to get back the transmission time of the =
packet.  Dave Taht's experiments (IIRC) were below 4Mbps for this =
advantage (when doing VOIP over that bottleneck competing against other =
traffic in a best effort class).

Hmmm. According to Akamai, global average connection speeds remain =
stubbornly below 4Mbps [1], so if Dave's analysis is correct, an =
unsophisticated ECN-deployment campaign may be counterproductive.

Mat

[1] =
http://www.internetsociety.org/blog/2013/02/internet-connection-speeds-and=
-right-some=

From emmanuel.lochin@isae.fr  Wed Mar 20 04:21:52 2013
Return-Path: <emmanuel.lochin@isae.fr>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2EFEE21F8C82; Wed, 20 Mar 2013 04:21:52 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.249
X-Spam-Level: 
X-Spam-Status: No, score=-2.249 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_FR=0.35]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KkmBbERGp5Gq; Wed, 20 Mar 2013 04:21:51 -0700 (PDT)
Received: from smtpext.isae.fr (smtpext.isae.fr [193.54.120.4]) by ietfa.amsl.com (Postfix) with SMTP id 3A44421F8C7D; Wed, 20 Mar 2013 04:21:51 -0700 (PDT)
Received: from catalpa (unknown [10.4.1.11]) by smtpext.isae.fr (Postfix) with SMTP id C491D22E93C; Wed, 20 Mar 2013 12:21:45 +0100 (CET)
Received: from [10.33.1.1] (pc-lochin.isae.fr [10.33.1.1]) by catalpa (Postfix) with ESMTP id 80197B7D6C; Wed, 20 Mar 2013 12:21:49 +0100 (CET)
Message-ID: <51499BFB.7050202@isae.fr>
Date: Wed, 20 Mar 2013 12:22:35 +0100
From: Emmanuel Lochin <emmanuel.lochin@isae.fr>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: Yuchung Cheng <ycheng@google.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <CAK6E8=dxZbT92K1mgg_E5zo8UQ1HhZe77c8SD8tqu0FXeLzYJw@mail.gmail.com>
In-Reply-To: <CAK6E8=dxZbT92K1mgg_E5zo8UQ1HhZe77c8SD8tqu0FXeLzYJw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Cc: "tcpm@ietf.org Extensions" <tcpm@ietf.org>, =?ISO-8859-1?Q?Mirja_K=FChlewind?= <mirja.kuehlewind@ikr.uni-stuttgart.de>, aqm@ietf.org, iccrg@irtf.org
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 11:21:52 -0000

On 19/03/2013 21:17, Yuchung Cheng wrote:
> Do people on the list know any (good) evaluation of performance impact
> of ECN on real and/or emulated networks. E.g., how much faster will
> the Web be if ECN is used (correctly).

There is : http://dl.acm.org/citation.cfm?id=3D1080091.1080100

EL

--=20
Emmanuel Lochin
Professeur ISAE - OSSI
Institut Sup=E9rieur de l'A=E9ronautique et de l'Espace (ISAE)
Issu du rapprochement SUPAERO et ENSICA
10 avenue Edouard Belin - BP 54032 - 31055 Toulouse cedex 4
Tel : 05 61 33 91 85 - Fax : 05 61 33 91 88
Web : http://personnel.isae.fr/emmanuel-lochin/
---
"This email and any attachments are confidential. They may contain legall=
y privileged information or copyright material. You should not read, copy=
, use or disclose them without authorisation. If you are not an intended =
recipient, please contact us at once by return email and then delete both=
 messages. We do not accept liability in connection with computer virus, =
data corruption, delay, interruption, unauthorised access or unauthorised=
 amendment. This notice should not be removed"


From dave.taht@gmail.com  Wed Mar 20 06:43:34 2013
Return-Path: <dave.taht@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F2D0B1F0CF7 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 06:43:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.543
X-Spam-Level: 
X-Spam-Status: No, score=-2.543 tagged_above=-999 required=5 tests=[AWL=0.057,  BAYES_00=-2.599, NO_RELAYS=-0.001]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zcV1a4EaRffg for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 06:43:31 -0700 (PDT)
Received: from mail-ie0-x232.google.com (mail-ie0-x232.google.com [IPv6:2607:f8b0:4001:c03::232]) by ietfa.amsl.com (Postfix) with ESMTP id CED781F0D05 for <aqm@ietf.org>; Wed, 20 Mar 2013 06:43:31 -0700 (PDT)
Received: by mail-ie0-f178.google.com with SMTP id c13so1995604ieb.23 for <aqm@ietf.org>; Wed, 20 Mar 2013 06:43:26 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type:content-transfer-encoding; bh=/f5/+T8XQasV7ZbfGwmRcyIftvkQrwZ3GqZJ9rAbAdM=; b=wCpzQ3u/EhZ5j67Z99nrPpSNu25gyTXKGa1ObaEtI4ki4SnlDXMmzPiY1NxgadW3DO zdx/NoUuyvXoiNj7sM5Vc7F0R/KR17ag1hNxju0u/1Ud5c3jp39ZjqvUYgSjSi8a+HpT auZWucAYbQwzl5nUFKxqvvuqDZBtTSwiw87L/XeyZcZ4fP/gJlGIMPjbnfMaTUpfOvCr JnOCN9w7VOpYOU0qdH43JNd3CyOUj7pRPKoz5x3I2WLVRhhlKiSK0fo5A/ZTKel+wQJI ILbbYv1P9cpU3S74qNZDVrJCqFrm1IaTYJdLWNfmyErK6SR16vOp0PTdwVs9yBVBB7AE RCGw==
MIME-Version: 1.0
X-Received: by 10.50.41.71 with SMTP id d7mr4107003igl.86.1363787006084; Wed, 20 Mar 2013 06:43:26 -0700 (PDT)
Received: by 10.64.132.71 with HTTP; Wed, 20 Mar 2013 06:43:25 -0700 (PDT)
Date: Wed, 20 Mar 2013 09:43:25 -0400
Message-ID: <CAA93jw63MEHD_q9bsmKTO1jGpmwP7kffHChhrFLML6yJ11EsxQ@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
To: aqm@ietf.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Subject: [aqm] fq_codel, codel, and RRUL
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 13:43:34 -0000

The present state of the research (as opposed to what we've been able
to write down and publish), and some open questions
re codel were described and discussed at stanford a few weeks back.

http://www.youtube.com/watch?v=3DMxoa5Si4Ubw
http://netseminar.stanford.edu/seminars/Inside_Codel_and_Fq_Codel.pdf

There was spirited discussion (kathie nichols, myself, eric dumazet,
and rong) of how codel and fq_codel actually works starting about 24
minutes in, and it ended up dominating the talk so much that I was
unable to talk about the "fq" part of fq_codel very much.

It certainly is my hope to spark further discussion of how these
algorithms work on this list, and I guess I should put together a talk
focusing on the sfq/fq portions of the algo in order to get traction
on it...

Earlier in the stanford and MIT talks I tried to describe how with
tests like the RRUL test how we were starting to look into the
behavior of mice and ants and congestion control of elephants with
aqm.  I also had wanted to talk about the three tier prioritization
scheme I've been fiddling with for over a year, some plots of the
results of that are in the slides.

There was a followup talk at MIT about the problems wireless and wifi
has, I don't have the link handy.

There is still a great deal of research needed on what the right
answers are for data centers, verses what we think is extra-ordinarily
promising on the edge. I note that sfqcodel is available now for ns2
under the bsd license, fq_codel is in ns3 also under the same license,
and the codel codel in linux presently is a bit behind and dual
bad/gpl licensed. fq_codel is GPL.

Please take a look at whatever versions you feel comfortable looking at.

I've been watching the ECN discussion go by and while have some
comments on that thread I'll make tomorrow or so, but I encourage
people to look at the ECN threads on the codel list as a starting
point for thinking about the problems of ECN on the wild and wooly
internet.

--=20
Dave T=E4ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.=
html

From dpreed@reed.com  Wed Mar 20 06:46:40 2013
Return-Path: <dpreed@reed.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 62D7D21F84F2 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 06:46:40 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.485
X-Spam-Level: 
X-Spam-Status: No, score=-3.485 tagged_above=-999 required=5 tests=[AWL=0.113,  BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wLbG5P7Ldj1u for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 06:46:39 -0700 (PDT)
Received: from smtp156.iad.emailsrvr.com (smtp156.iad.emailsrvr.com [207.97.245.156]) by ietfa.amsl.com (Postfix) with ESMTP id 10FBC21F848D for <aqm@ietf.org>; Wed, 20 Mar 2013 06:46:38 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp55.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 7EC472E0622; Wed, 20 Mar 2013 09:46:38 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy3.wa-web.iad1a (legacy3.wa-web.iad1a.rsapps.net [192.168.2.219]) by smtp55.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 65B4D2E01F3; Wed, 20 Mar 2013 09:46:38 -0400 (EDT)
Received: from reed.com (localhost [127.0.0.1]) by legacy3.wa-web.iad1a (Postfix) with ESMTP id 4F06521680AC; Wed, 20 Mar 2013 09:46:38 -0400 (EDT)
Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com)  with HTTP; Wed, 20 Mar 2013 09:46:38 -0400 (EDT)
Date: Wed, 20 Mar 2013 09:46:38 -0400 (EDT)
From: dpreed@reed.com
To: "John Leslie" <john@jlc.net>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_20130320094638000000_73991"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
In-Reply-To: <20130320035645.GI9569@verdi>
References: <mailman.5225.1363736608.3432.aqm@ietf.org>  <1363748540.64155384@apps.rackspace.com> <20130320035645.GI9569@verdi>
Message-ID: <1363787198.319422762@apps.rackspace.com>
X-Mailer: webmail7.0
Cc: aqm@ietf.org
Subject: Re: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 13:46:40 -0000

------=_20130320094638000000_73991
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0AJohn - I understand your response, and I understand the idea of doing EC=
N marking at the 1 msec. level.  That certainly would increase the amount o=
f congestion information delivered to the endpoints.=0A =0AHowever, if you =
think about how long it will take for the receiver to reduce its "window" i=
n typical cases, along with the "reluctance" of router designers to ever dr=
op packets, I think this does not solve the problem.=0A =0AYes, ECN might h=
elp a bit in the case where there are *large* numbers of stationary-process=
 TCP flows traversing a node, leading to nearly full capacity.=0A =0ABut on=
e should focus on the actually difficult problems: e.g. what went wrong in =
DOCSIS 2.0 and early DOCSIS 3 deployments, still in the field, that made qu=
eueing delays close to 1 second typical from a single home.=0A =0ABecause t=
here are relatively small numbers of flows, *way* too much buffering, leadi=
ng to sustained queues that could not drain fast enough under ECN or anythi=
ng else, user experience *sucked*.=0A =0AWhich is why Comcast tried "killin=
g BitTorrent" with Sandvine DPI-based RST-injection, and started claiming i=
t was to "stop music piracy, which was destroying the Internet".=0A =0AWhen=
 Comcast was forced to actually look at why the queues were sustained and u=
nfixable, they realized that the problem was that DOCSIS was *designed* to =
hold packets rather than drop them.  ECN marking would probably not have he=
lped get the queues down under 50 msec, without dropping packets, neither w=
ould RED.  The only way to get the queue down is to prevent it from being c=
reated in the first place....  And since the RTT from US east coast to US w=
est coast is under 45 msec, it does little good to send early congestion si=
gnals.=0A =0AI'm sure you know this.=0A =0AAn analogous problem occurs toda=
y because of the 3-4 *second* buffering in HSPA and LTE networks under load=
.  Having measured it in many American and European deployments of such equ=
ipment, I would suggest that the fundamental problem is *dropping packets f=
airly and quickly*.  You can't get there (to sub-50 msec. queueing) with EC=
N or RED.=0A =0A-----Original Message-----=0AFrom: "John Leslie" <john@jlc.=
net>=0ASent: Tuesday, March 19, 2013 11:56pm=0ATo: dpreed@reed.com=0ACc: aq=
m@ietf.org=0ASubject: Draining queues=0A=0A=0A=0Adpreed@reed.com <dpreed@re=
ed.com> wrote:=0A> =0A> A problem with ECN is that it does not drain an alr=
eady flooded queue,=0A=0A I presume you mean that ECN doesn't drop the pack=
et it marks.=0A=0A> and depends (like RED) on the idea that one has a large=
 queue in normal=0A> operation.=0A=0A I don't follow this at all. ECN is po=
intless without an AQM which=0Adrops or marks well before tail-drop. Were w=
e to run ECN without AQM,=0Awe'd have to drop each packet anyway after mark=
ing it.=0A=0A Several flavors of AQM mark/drop well enough in advance of ta=
il-drop=0Athat the queue need never fill. Did you mean to say that ECN requ=
ires=0Abeing able to queue the ECN-marked packet which would otherwise be=
=0Adropped by the AQM?=0A=0A> It is not obvious that sustaining a deep queu=
e is a desirable state,=0A=0A Agreed!=0A=0A> especially when there are high=
ly variable capacities along any path,=0A> as is the case where one link is=
 more than 2x slower than the rest of=0A> the links.=0A=0A Isn't that _alwa=
ys_ the case?=0A=0A> If the dynamics of user loads are fractal, this will c=
ause draining=0A> (and latency reduction) not to work.=0A=0A I'm guessing y=
ou mean that under "fractal" loads there's a tendency=0Afor ECN-marked pack=
ets to clump and introduce latency at the bottleneck.=0ARecall that this is=
 offset by the quicker response to an ECN mark at=0ARTT speed instead of ha=
ving to infer a packet loss.=0A=0A> If user loads are stationary gaussian a=
nd so forth, ECN can "tune"=0A> to relatively stable situations, but the qu=
eueing delay will be quite=0A> long end-to-end.=0A=0A I don't follow... Som=
e AQMs may indeed allow queueing delay to grow=0Alarge; but this can only h=
appen if the instantaneous percentage of=0Amarked packets is quite a bit hi=
gher than TCP can tolerate.=0A=0A> Since there is lots of evidence that use=
r loads are fractal, dropping=0A> packets during sudden bursts will prevent=
 sustained queueing, whereas=0A> ECN will not.=0A=0A I believe there is sub=
stantial agreement that ECN-marked packets=0Awill need to be dropped anyway=
 under particularly severe congestion.=0AAt the very least, tail-drop can b=
ecome necessary...=0A=0A> So a proper approach (in my opinion) on the "real=
" Internet (rather=0A> than a bunch of competing long-duration FTPs) requir=
es keeping the=0A> queues so short that there is no real opportunity to ben=
efit from ECN.=0A=0A Here I quite disagree.=0A=0A An AQM could be designed =
to ECN-mark at a 1 millisecond queue depth;=0Aand that can give the full be=
nefit of ECN. Again, recall that ECN will=0Adeliver the congestion signal i=
n one RTT, instead of waiting to infer=0Aa loss. TCP stops delivering usefu=
l bandwidth at one or two percent=0Apacket loss: so it's hard to believe th=
at ECN-marked packets will clog=0Athe path significantly.=0A=0A> Hence - we=
 need to really see what happens with *real* traffic loads,=0A> not simplif=
ied simulations.=0A=0A Agreed! (Now if we could just get enough ECN deploym=
ent...)=0A=0A> Real traffic loads are fractal, and fractal loads do not gen=
erally=0A> obey the Gaussian style "law of large numbers" - multiple fracta=
l=0A> flows are *rougher* not smoother than a single one.=0A=0A I would say=
 rather that real traffic loads _contain_ fractal flows.=0AAggregation simp=
ly wouldn't work if we never experienced gaussian=0Asmoothing.=0A=0A> So ch=
eck your mathematical "intuitions" at the door.  Stop writing=0A> equations=
 based on tractable analytic models, because actual traffic=0A> does not fo=
llow those models.=0A=0A I strongly suspect our models _are_ faulty -- but =
we need research=0Aresults for _actual_ flows, with and without ECN, is ord=
er to diagnose=0Atheir faults.=0A=0A--=0AJohn Leslie <john@jlc.net>
------=_20130320094638000000_73991
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"times new roman" size=3D"2"><p style=3D"margin:0;padding:0;">=
John - I understand your response, and I understand the idea of doing ECN m=
arking at the 1 msec. level.&nbsp; That certainly would increase the amount=
 of congestion information delivered to the endpoints.</p>=0A<p style=3D"ma=
rgin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">However, i=
f you think about how long it will take for the receiver to reduce its "win=
dow" in typical cases, along with the "reluctance" of router designers to e=
ver drop packets, I think this does not solve the problem.</p>=0A<p style=
=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">Yes,=
 ECN might help a bit in the case where there are *large* numbers of statio=
nary-process TCP flows traversing a node, leading to nearly full capacity.<=
/p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;pa=
dding:0;">But one should focus on the actually difficult problems: e.g. wha=
t went wrong in DOCSIS 2.0 and early DOCSIS 3 deployments, still in the fie=
ld, that made queueing delays close to 1 second typical from a single home.=
</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;p=
adding:0;">Because there are relatively small numbers of flows, *way* too m=
uch buffering, leading to sustained queues that could not drain fast enough=
 under ECN or anything else, user experience *sucked*.</p>=0A<p style=3D"ma=
rgin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">Which is w=
hy Comcast tried "killing BitTorrent" with Sandvine DPI-based RST-injection=
, and started claiming it was to "stop music piracy, which was destroying t=
he Internet".</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=
=3D"margin:0;padding:0;">When Comcast was forced to actually look at why th=
e queues were sustained and unfixable, they realized that the problem was t=
hat DOCSIS was *designed* to hold packets rather than drop them.&nbsp; ECN =
marking would probably not have helped get the queues down under 50 msec, w=
ithout dropping packets, neither would RED.&nbsp; The only way to get the q=
ueue down is to prevent it from being created in the first place....&nbsp; =
And since the RTT from US east coast to US west coast is under 45 msec, it =
does little good to send early congestion signals.</p>=0A<p style=3D"margin=
:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">I'm sure you k=
now this.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"m=
argin:0;padding:0;">An analogous problem occurs today because of the 3-4 *s=
econd* buffering in HSPA and LTE networks under load.&nbsp; Having measured=
 it in many American and European deployments of such equipment, I would su=
ggest that the fundamental problem is *dropping packets fairly and quickly*=
.&nbsp; You can't get there (to sub-50 msec. queueing) with ECN or RED.</p>=
=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;paddi=
ng:0;">-----Original Message-----<br />From: "John Leslie" &lt;john@jlc.net=
&gt;<br />Sent: Tuesday, March 19, 2013 11:56pm<br />To: dpreed@reed.com<br=
 />Cc: aqm@ietf.org<br />Subject: Draining queues<br /><br /></p>=0A<div id=
=3D"SafeStyles1363786258">=0A<p style=3D"margin:0;padding:0;">dpreed@reed.c=
om &lt;dpreed@reed.com&gt; wrote:<br />&gt; <br />&gt; A problem with ECN i=
s that it does not drain an already flooded queue,<br /><br /> I presume yo=
u mean that ECN doesn't drop the packet it marks.<br /><br />&gt; and depen=
ds (like RED) on the idea that one has a large queue in normal<br />&gt; op=
eration.<br /><br /> I don't follow this at all. ECN is pointless without a=
n AQM which<br />drops or marks well before tail-drop. Were we to run ECN w=
ithout AQM,<br />we'd have to drop each packet anyway after marking it.<br =
/><br /> Several flavors of AQM mark/drop well enough in advance of tail-dr=
op<br />that the queue need never fill. Did you mean to say that ECN requir=
es<br />being able to queue the ECN-marked packet which would otherwise be<=
br />dropped by the AQM?<br /><br />&gt; It is not obvious that sustaining =
a deep queue is a desirable state,<br /><br /> Agreed!<br /><br />&gt; espe=
cially when there are highly variable capacities along any path,<br />&gt; =
as is the case where one link is more than 2x slower than the rest of<br />=
&gt; the links.<br /><br /> Isn't that _always_ the case?<br /><br />&gt; I=
f the dynamics of user loads are fractal, this will cause draining<br />&gt=
; (and latency reduction) not to work.<br /><br /> I'm guessing you mean th=
at under "fractal" loads there's a tendency<br />for ECN-marked packets to =
clump and introduce latency at the bottleneck.<br />Recall that this is off=
set by the quicker response to an ECN mark at<br />RTT speed instead of hav=
ing to infer a packet loss.<br /><br />&gt; If user loads are stationary ga=
ussian and so forth, ECN can "tune"<br />&gt; to relatively stable situatio=
ns, but the queueing delay will be quite<br />&gt; long end-to-end.<br /><b=
r /> I don't follow... Some AQMs may indeed allow queueing delay to grow<br=
 />large; but this can only happen if the instantaneous percentage of<br />=
marked packets is quite a bit higher than TCP can tolerate.<br /><br />&gt;=
 Since there is lots of evidence that user loads are fractal, dropping<br /=
>&gt; packets during sudden bursts will prevent sustained queueing, whereas=
<br />&gt; ECN will not.<br /><br /> I believe there is substantial agreeme=
nt that ECN-marked packets<br />will need to be dropped anyway under partic=
ularly severe congestion.<br />At the very least, tail-drop can become nece=
ssary...<br /><br />&gt; So a proper approach (in my opinion) on the "real"=
 Internet (rather<br />&gt; than a bunch of competing long-duration FTPs) r=
equires keeping the<br />&gt; queues so short that there is no real opportu=
nity to benefit from ECN.<br /><br /> Here I quite disagree.<br /><br /> An=
 AQM could be designed to ECN-mark at a 1 millisecond queue depth;<br />and=
 that can give the full benefit of ECN. Again, recall that ECN will<br />de=
liver the congestion signal in one RTT, instead of waiting to infer<br />a =
loss. TCP stops delivering useful bandwidth at one or two percent<br />pack=
et loss: so it's hard to believe that ECN-marked packets will clog<br />the=
 path significantly.<br /><br />&gt; Hence - we need to really see what hap=
pens with *real* traffic loads,<br />&gt; not simplified simulations.<br />=
<br /> Agreed! (Now if we could just get enough ECN deployment...)<br /><br=
 />&gt; Real traffic loads are fractal, and fractal loads do not generally<=
br />&gt; obey the Gaussian style "law of large numbers" - multiple fractal=
<br />&gt; flows are *rougher* not smoother than a single one.<br /><br /> =
I would say rather that real traffic loads _contain_ fractal flows.<br />Ag=
gregation simply wouldn't work if we never experienced gaussian<br />smooth=
ing.<br /><br />&gt; So check your mathematical "intuitions" at the door.  =
Stop writing<br />&gt; equations based on tractable analytic models, becaus=
e actual traffic<br />&gt; does not follow those models.<br /><br /> I stro=
ngly suspect our models _are_ faulty -- but we need research<br />results f=
or _actual_ flows, with and without ECN, is order to diagnose<br />their fa=
ults.<br /><br />--<br />John Leslie &lt;john@jlc.net&gt;</p>=0A</div></fon=
t>
------=_20130320094638000000_73991--


From rs@netapp.com  Wed Mar 20 09:04:02 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DA39821F8573; Wed, 20 Mar 2013 09:04:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.313
X-Spam-Level: 
X-Spam-Status: No, score=-10.313 tagged_above=-999 required=5 tests=[AWL=0.286, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9TieoXp5OmRN; Wed, 20 Mar 2013 09:04:01 -0700 (PDT)
Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) by ietfa.amsl.com (Postfix) with ESMTP id 3D1A821F8556; Wed, 20 Mar 2013 09:04:01 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,879,1355126400"; d="scan'208";a="32368327"
Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx12-out.netapp.com with ESMTP; 20 Mar 2013 09:04:00 -0700
Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2KG40d0012922; Wed, 20 Mar 2013 09:04:00 -0700 (PDT)
Received: from VMWEXCEHT06-PRD.hq.netapp.com (10.106.77.104) by vmwexceht01-prd.hq.netapp.com (10.106.76.239) with Microsoft SMTP Server (TLS) id 14.2.342.3; Wed, 20 Mar 2013 09:03:59 -0700
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht06-prd.hq.netapp.com ([10.106.77.104]) with mapi id 14.02.0342.003; Wed, 20 Mar 2013 09:03:59 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: Emmanuel Lochin <emmanuel.lochin@isae.fr>, Yuchung Cheng <ycheng@google.com>
Thread-Topic: [tcpm] ECN support and usage on the Internet
Thread-Index: AQHOJN7vLn8i1zQNs0aR6qujDYSanZiu5guA///XOCA=
Date: Wed, 20 Mar 2013 16:03:59 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24AC06C0@SACEXCMBX02-PRD.hq.netapp.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <CAK6E8=dxZbT92K1mgg_E5zo8UQ1HhZe77c8SD8tqu0FXeLzYJw@mail.gmail.com> <51499BFB.7050202@isae.fr>
In-Reply-To: <51499BFB.7050202@isae.fr>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "tcpm@ietf.org Extensions" <tcpm@ietf.org>, "aqm@ietf.org" <aqm@ietf.org>, "iccrg@irtf.org" <iccrg@irtf.org>
Subject: Re: [aqm] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:04:03 -0000

Thanks!

Which fortunately is already an experimental http://tools.ietf.org/html/rfc=
5562=20

But too little experience to follow up on that just yet; Last time I looked=
, I believe it was the AIX 5.3 TCP stack would default to 5562, when ECN is=
 enabled - but ECN is off by default in that OS...


But the findings in that paper don't compare Tail-Drop non-ECN with AQM/ECN=
*, which is probably what you are after Yuchung?


Put differently: With a large enough fraction of AQM deployment, a reasonab=
le number of RTOs should be avoidable (less probability that the last segme=
nt in a session is dropped); with the addition of ECN, even that loss shoul=
d go away...


Do you know what the CCDF for object load times would look like, if there w=
ere virtually no RTOs anymore?


Apparently, the incentive to avoid (costly) RTOs is too small to drive ECN =
deployment, even though that should be something easy to achieve with ECN o=
n TCP.


Richard Scheffenegger



> -----Original Message-----
> From: tcpm-bounces@ietf.org [mailto:tcpm-bounces@ietf.org] On Behalf Of
> Emmanuel Lochin
> Sent: Mittwoch, 20. M=E4rz 2013 12:23
> To: Yuchung Cheng
> Cc: tcpm@ietf.org Extensions; aqm@ietf.org; iccrg@irtf.org
> Subject: Re: [tcpm] ECN support and usage on the Internet
>=20
> On 19/03/2013 21:17, Yuchung Cheng wrote:
> > Do people on the list know any (good) evaluation of performance impact
> > of ECN on real and/or emulated networks. E.g., how much faster will
> > the Web be if ECN is used (correctly).
>=20
> There is : http://dl.acm.org/citation.cfm?id=3D1080091.1080100
>=20
> EL
>=20
> --
> Emmanuel Lochin
> Professeur ISAE - OSSI
> Institut Sup=E9rieur de l'A=E9ronautique et de l'Espace (ISAE) Issu du
> rapprochement SUPAERO et ENSICA
> 10 avenue Edouard Belin - BP 54032 - 31055 Toulouse cedex 4 Tel : 05 61 3=
3
> 91 85 - Fax : 05 61 33 91 88 Web : http://personnel.isae.fr/emmanuel-
> lochin/
> ---
> "This email and any attachments are confidential. They may contain legall=
y
> privileged information or copyright material. You should not read, copy,
> use or disclose them without authorisation. If you are not an intended
> recipient, please contact us at once by return email and then delete both
> messages. We do not accept liability in connection with computer virus,
> data corruption, delay, interruption, unauthorised access or unauthorised
> amendment. This notice should not be removed"
>=20
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm

From rs@netapp.com  Wed Mar 20 09:14:08 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4E7BD21F842A; Wed, 20 Mar 2013 09:14:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.015
X-Spam-Level: 
X-Spam-Status: No, score=-10.015 tagged_above=-999 required=5 tests=[AWL=-0.095, BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_HI=-8, SARE_SUB_ENC_UTF8=0.152, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rwoYnAKiDFs6; Wed, 20 Mar 2013 09:14:07 -0700 (PDT)
Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) by ietfa.amsl.com (Postfix) with ESMTP id 70A7821F8415; Wed, 20 Mar 2013 09:14:05 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,879,1355126400"; d="scan'208";a="32372231"
Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx12-out.netapp.com with ESMTP; 20 Mar 2013 09:14:05 -0700
Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2KGE4l3029309; Wed, 20 Mar 2013 09:14:04 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0342.003; Wed, 20 Mar 2013 09:14:03 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: Simon Barber <simon@superduper.net>, Jim Gettys <jg@freedesktop.org>
Thread-Topic: =?utf-8?B?W2FxbV0g562U5aSNOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBhbmQg?= =?utf-8?Q?usage_on_the_Internet?=
Thread-Index: AQHOJPuQ4YV4mgvTF0iHLgehV1ZAsZiuv+6Q
Date: Wed, 20 Mar 2013 16:14:03 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24AC06F7@SACEXCMBX02-PRD.hq.netapp.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <alpine.DEB.2.00.1303191014021.2309@uplift.swm.pp.se> <51487A27.7010904@hp.com> <CAGhGL2D=qncZZ0YG2jLfXwJLXLrXj4dXns_u66qPNBEVH2ZR4A@mail.gmail.com> <B2CE202A35DFDD4A904DA7497D7D2B9830F98A@CNHZ-EXMAIL-10.ali.com> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com> <5148F80F.5000401@superduper.net>
In-Reply-To: <5148F80F.5000401@superduper.net>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, Rick Jones <rick.jones2@hp.com>, "iccrg@irtf.org" <iccrg@irtf.org>, =?utf-8?B?5aSp5r6c?= <tianlan.lhh@alibaba-inc.com>, "aqm@ietf.org" <aqm@ietf.org>, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] =?utf-8?b?562U5aSNOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBh?= =?utf-8?q?nd_usage_on_the_Internet?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:14:08 -0000

SGkgU2ltb24sDQoNCg0KRm9yIG9uZSwgdGhyb3VnaHB1dCBpcyB0aGUgb25lIG1ldHJpYywgdGhh
dCBzaG91bGQgTk9UIGJlIGluZmx1ZW5jZWQgYnkgRUNOLCBhY3R1YWxseS4gU28gaWYgdGhhdCBp
cyAiaW4gdGhlIG5vaXNlIiwgdGhhdCBpcyBhIHZlcnkgZ29vZCBzaWduOyANCg0KVGhlIG1vcmUg
aW50ZXJlc3RpbmcgbWV0cmljcywgdGhhdCBhcmUgYWRkcmVzc2VkIHdpdGggRUNOIGFyZSBsYXRl
bmN5ICgicGFnZSBsb2FkIHRpbWVzIikgYW5kIG5ldHdvcmsgZWZmaWNpZW5jeSAoZmV3ZXIgcGFj
a2V0cyBhcmUgbmVlZGxlc3NseSBwcm9jZXNzZWQgaW4gaW5pdGlhbCBob3BzLCBqdXN0IHRvIGJl
IGRyb3BwZWQgYnkgdGhlIHBlbnVsdGltYXRlIGhvcDsgdGhhdCBiYW5kd2lkdGggY2FuIGJlIGFs
bG9jYXRlZCB0byBvdGhlciBmbG93cyBldGMpLg0KDQpPdGhlciBzY2hlbWVzIHRoYXQgcmVkdWNl
IGxhdGVuY3kgdXN1YWxseSBkbyBzbyBieSB0cmFkaW5nIGEgZ29vZCBzaGFyZSBvZiBiYW5kd2lk
dGguLi4gKGNyZWRpdCBhbGxvY2F0aW9uIHNjaGVtZSB1c2VkIGluIEZDLCBhbnlvbmU/KQ0KDQoN
CkVDTiBoYXQgYSBodWdlIGlzc3VlIG9mIGRlcGxveWFiaWxpdHkgaXNzdWVzLCB0aGF0IGhhZCBu
b3RoaW5nIHRvIGRvIHdpdGggdGhlIG1lY2hhbmlzbSBpdHNlbGYsIGJ1dCB3aXRoIG1pc2JlaGF2
aW5nIGVxdWlwbWVudCBvbiB0aGUgc2lnbmFsaW5nIC0gYW5kIHRoZSBpbnN0aXR1dGlvbmFsIG1l
bW9yeSBzdGlsbCByZW1lbWJlcnMgdGhvc2UgYWxiZWl0IGFueSBuZXcgc2lnbmFsaW5nIHNjaGVt
ZSBjb3VsZCBydW4gaW50byBzaW1pbGFyIGlzc3VlcyAoaW5kZXBlbmRlbnQgb2YgdGhlIG1lY2hh
bmlzbXMpLg0KDQoNCkllLiBzb21lIHBlb3BsZSBzdGlsbCBoYXZlbid0IGdvdCB0aGUgTWVtbyB0
aGF0IElQdjQgVE9TIHdhcyBkZXByZWNhdGVkIHF1aXRlIHNvbWUgdGltZSBhZ28gKG5vdyBEaWZm
U2Vydi9FQ04pLA0KDQpSZWdhcmRzLA0KDQpSaWNoYXJkIFNjaGVmZmVuZWdnZXINCg0KDQoNCj4g
LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogYXFtLWJvdW5jZXNAaWV0Zi5vcmcg
W21haWx0bzphcW0tYm91bmNlc0BpZXRmLm9yZ10gT24gQmVoYWxmIE9mDQo+IFNpbW9uIEJhcmJl
cg0KPiBTZW50OiBNaXR0d29jaCwgMjAuIE3DpHJ6IDIwMTMgMDA6NDMNCj4gVG86IEppbSBHZXR0
eXMNCj4gQ2M6IFJpY2sgSm9uZXM7IGljY3JnQGlydGYub3JnOyB0Y3BtQGlldGYub3JnOyDlpKnm
vpw7IGFxbUBpZXRmLm9yZzsgTWlrYWVsDQo+IEFicmFoYW1zc29uDQo+IFN1YmplY3Q6IFJlOiBb
YXFtXSDnrZTlpI06IFtpY2NyZ10gW3RjcG1dIEVDTiBzdXBwb3J0IGFuZCB1c2FnZSBvbiB0aGUN
Cj4gSW50ZXJuZXQNCj4gDQo+IEknbSBub3Qgc3VyZSB3aHkgcGVvcGxlIGFyZSBzbyBvYnNlc3Nl
ZCB3aXRoIEVDTiAtIHRoZSBsaXR0bGUgZGF0YSBJJ3ZlDQo+IHNlZW4gYWJvdXQgaXQgc2hvd2Vk
IHRoYXQgaXQncyBlZmZlY3Qgb24gc3lzdGVtIHRocm91Z2hwdXQgd2FzIGluIHRoZQ0KPiBub2lz
ZS4gU3VyZSwgd2lyZWxlc3MgZ3V5cyBkb24ndCBsaWtlIHRvIHdhc3RlIHByZWNpb3VzIHdpcmVs
ZXNzDQo+IGJhbmR3aWR0aCwgYnV0IHRoZXkncmUgbm90IGRyb3BwaW5nIHBhY2tldHMgdGhhdCBo
YXZlIHVzZWQgdXAgdGhlIHByZWNpb3VzDQo+IHdpcmVsZXNzIGJhbmR3aWR0aC4gVGhleSBnZXQg
ZHJvcHBlZCBiZWZvcmUgdGhlIHdpcmVsZXNzIGxpbmsuDQo+IA0KPiBTaW1vbg0KPiANCg0K

From gettysjim@gmail.com  Wed Mar 20 09:25:12 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E07C721F8C71 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:25:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.298
X-Spam-Level: 
X-Spam-Status: No, score=-1.298 tagged_above=-999 required=5 tests=[AWL=1.451,  BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Va0eK8aaCdOh for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:25:10 -0700 (PDT)
Received: from mail-oa0-f45.google.com (mail-oa0-f45.google.com [209.85.219.45]) by ietfa.amsl.com (Postfix) with ESMTP id 5726E21F862A for <aqm@ietf.org>; Wed, 20 Mar 2013 09:25:09 -0700 (PDT)
Received: by mail-oa0-f45.google.com with SMTP id o6so2015417oag.18 for <aqm@ietf.org>; Wed, 20 Mar 2013 09:25:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=AMLVK3HU6xEqvfbYy4S67h5lbUlD48hwo4ZeZCSXoYM=; b=0+kAU8d1lCIp6QBgGhsiZYx544lsiK1ySGLQBTa3plhs+W/Td2zlI73UuNGN6UDXMX qSHaC9ZkRr5NJ0r+8WSNxFzcaQYSZt49DYSwGplGFBi85j6i6Pnbu7RE0IcNmh0WNWDo IpZJBosIyfaqUTIWkIVjmyUpmQUSXZ2zgmjngyTTKFmXKCKjVvqcM8TTHRe/GeAtv7xu 46+i4/G7uBh1+XHVFW2F/HmSI0u4BKYZVRTwgUaMfVq9qksSICJRMjLlX7kpCJrOaFx6 GTno798WJcU7XV5bI0A4CfbjP18WB06sP5/qrQk9Jz5z0/CwZV2Ob2FmntMjwmKiCMwX 1IFA==
MIME-Version: 1.0
X-Received: by 10.60.8.40 with SMTP id o8mr4497862oea.112.1363796706200; Wed, 20 Mar 2013 09:25:06 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Wed, 20 Mar 2013 09:25:06 -0700 (PDT)
In-Reply-To: <1363787198.319422762@apps.rackspace.com>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <20130320035645.GI9569@verdi> <1363787198.319422762@apps.rackspace.com>
Date: Wed, 20 Mar 2013 12:25:06 -0400
X-Google-Sender-Auth: FymHnPXxTiEu9KeC3W49xuxdigk
Message-ID: <CAGhGL2CDY2EYow=J3E7xDr_W_piMNwGJYuRf8pf17P0fs1TcZw@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: David P Reed <dpreed@reed.com>
Content-Type: multipart/alternative; boundary=e89a8ff1c7ea38147104d85daa73
Cc: John Leslie <john@jlc.net>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:25:13 -0000

--e89a8ff1c7ea38147104d85daa73
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Mar 20, 2013 at 9:46 AM, <dpreed@reed.com> wrote:

> John - I understand your response, and I understand the idea of doing ECN
> marking at the 1 msec. level.  That certainly would increase the amount of
> congestion information delivered to the endpoints.
>
>
>
> However, if you think about how long it will take for the receiver to
> reduce its "window" in typical cases, along with the "reluctance" of router
> designers to ever drop packets, I think this does not solve the problem.
>
>
>
> Yes, ECN might help a bit in the case where there are *large* numbers of
> stationary-process TCP flows traversing a node, leading to nearly full
> capacity.
>
>
>
> But one should focus on the actually difficult problems: e.g. what went
> wrong in DOCSIS 2.0 and early DOCSIS 3 deployments, still in the field,
> that made queueing delays close to 1 second typical from a single home.
>
>
>
> Because there are relatively small numbers of flows, *way* too much
> buffering, leading to sustained queues that could not drain fast enough
> under ECN or anything else, user experience *sucked*.
>
>
>
> Which is why Comcast tried "killing BitTorrent" with Sandvine DPI-based
> RST-injection, and started claiming it was to "stop music piracy, which was
> destroying the Internet".
>
>
>
> When Comcast was forced to actually look at why the queues were sustained
> and unfixable, they realized that the problem was that DOCSIS was
> *designed* to hold packets rather than drop them.  ECN marking would
> probably not have helped get the queues down under 50 msec, without
> dropping packets, neither would RED.  The only way to get the queue down is
> to prevent it from being created in the first place....  And since the RTT
> from US east coast to US west coast is under 45 msec, it does little good
> to send early congestion signals.
>
>
>
> I'm sure you know this.
>
>
>
> An analogous problem occurs today because of the 3-4 *second* buffering in
> HSPA and LTE networks under load.  Having measured it in many American and
> European deployments of such equipment, I would suggest that the
> fundamental problem is *dropping packets fairly and quickly*.  You can't
> get there (to sub-50 msec. queueing) with ECN or RED.
>

First off: everyone has made the same mistake: operating system writers,
DSL, and GPON, just to name a few.  Don't single out cable.  We've all
screwed up for a long time; in this sense, we're all "Bell Heads".

With flow queuing such as fq_codel or sfqcodel, the ECN equation changes:
other flows don't get held up the way they do when running a simple AQM
such as RED in a single queue.

And the web's behavior means that any straight AQM such as CoDel, RED or
PIE *cannot* get latency anywhere where it needs to be: too many packets
end up in flight at once to get stuck in any queue; we *have* to do
something to solve this problem: flow queuing makes a huge difference.
 I've measured 100's of milliseconds of transient latency hitting some web
sites as these packet flights hit the broadband edge.  This becomes a
non-issue with fq_codel.

Even so, Dave Taht's experiments (IIRC) were that for low bandwidth, drop
was preferable for ECN as even a single full sized packet takes
considerable time.

Exactly how this is going to play out is really only something we can
figure out by experiment.

I also need to clarify what I mean by "Bell Heads": these people are *not*
restricted to the telecom industry, but everywhere.  In fact, Dave Taht
first encountered "every packet is precious" in working on 802.11, where
the driver writers seemed to think that ever dropping even one packet was
evil, and would retransmit effectively infinitely.  So those of you working
on cellular wireless are in good company, and don't take this personally.
                                                   - Jim



>
> -----Original Message-----
> From: "John Leslie" <john@jlc.net>
> Sent: Tuesday, March 19, 2013 11:56pm
> To: dpreed@reed.com
> Cc: aqm@ietf.org
> Subject: Draining queues
>
>  dpreed@reed.com <dpreed@reed.com> wrote:
> >
> > A problem with ECN is that it does not drain an already flooded queue,
>
> I presume you mean that ECN doesn't drop the packet it marks.
>
> > and depends (like RED) on the idea that one has a large queue in normal
> > operation.
>
> I don't follow this at all. ECN is pointless without an AQM which
> drops or marks well before tail-drop. Were we to run ECN without AQM,
> we'd have to drop each packet anyway after marking it.
>
> Several flavors of AQM mark/drop well enough in advance of tail-drop
> that the queue need never fill. Did you mean to say that ECN requires
> being able to queue the ECN-marked packet which would otherwise be
> dropped by the AQM?
>
> > It is not obvious that sustaining a deep queue is a desirable state,
>
> Agreed!
>
> > especially when there are highly variable capacities along any path,
> > as is the case where one link is more than 2x slower than the rest of
> > the links.
>
> Isn't that _always_ the case?
>
> > If the dynamics of user loads are fractal, this will cause draining
> > (and latency reduction) not to work.
>
> I'm guessing you mean that under "fractal" loads there's a tendency
> for ECN-marked packets to clump and introduce latency at the bottleneck.
> Recall that this is offset by the quicker response to an ECN mark at
> RTT speed instead of having to infer a packet loss.
>
> > If user loads are stationary gaussian and so forth, ECN can "tune"
> > to relatively stable situations, but the queueing delay will be quite
> > long end-to-end.
>
> I don't follow... Some AQMs may indeed allow queueing delay to grow
> large; but this can only happen if the instantaneous percentage of
> marked packets is quite a bit higher than TCP can tolerate.
>
> > Since there is lots of evidence that user loads are fractal, dropping
> > packets during sudden bursts will prevent sustained queueing, whereas
> > ECN will not.
>
> I believe there is substantial agreement that ECN-marked packets
> will need to be dropped anyway under particularly severe congestion.
> At the very least, tail-drop can become necessary...
>
> > So a proper approach (in my opinion) on the "real" Internet (rather
> > than a bunch of competing long-duration FTPs) requires keeping the
> > queues so short that there is no real opportunity to benefit from ECN.
>
> Here I quite disagree.
>
> An AQM could be designed to ECN-mark at a 1 millisecond queue depth;
> and that can give the full benefit of ECN. Again, recall that ECN will
> deliver the congestion signal in one RTT, instead of waiting to infer
> a loss. TCP stops delivering useful bandwidth at one or two percent
> packet loss: so it's hard to believe that ECN-marked packets will clog
> the path significantly.
>
> > Hence - we need to really see what happens with *real* traffic loads,
> > not simplified simulations.
>
> Agreed! (Now if we could just get enough ECN deployment...)
>
> > Real traffic loads are fractal, and fractal loads do not generally
> > obey the Gaussian style "law of large numbers" - multiple fractal
> > flows are *rougher* not smoother than a single one.
>
> I would say rather that real traffic loads _contain_ fractal flows.
> Aggregation simply wouldn't work if we never experienced gaussian
> smoothing.
>
> > So check your mathematical "intuitions" at the door. Stop writing
> > equations based on tractable analytic models, because actual traffic
> > does not follow those models.
>
> I strongly suspect our models _are_ faulty -- but we need research
> results for _actual_ flows, with and without ECN, is order to diagnose
> their faults.
>
> --
> John Leslie <john@jlc.net>
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>
>

--e89a8ff1c7ea38147104d85daa73
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Wed, Mar 20, 2013 at 9:46 AM,  <span dir=3D"ltr">&lt;<a href=3D"=
mailto:dpreed@reed.com" target=3D"_blank">dpreed@reed.com</a>&gt;</span> wr=
ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
-left:1px #ccc solid;padding-left:1ex">
<font face=3D"times new roman"><p style=3D"margin:0;padding:0">John - I und=
erstand your response, and I understand the idea of doing ECN marking at th=
e 1 msec. level.=A0 That certainly would increase the amount of congestion =
information delivered to the endpoints.</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">However, if you think about how long it wil=
l take for the receiver to reduce its &quot;window&quot; in typical cases, =
along with the &quot;reluctance&quot; of router designers to ever drop pack=
ets, I think this does not solve the problem.</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">Yes, ECN might help a bit in the case where=
 there are *large* numbers of stationary-process TCP flows traversing a nod=
e, leading to nearly full capacity.</p>
<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">But one should focus on the actually diffic=
ult problems: e.g. what went wrong in DOCSIS 2.0 and early DOCSIS 3 deploym=
ents, still in the field, that made queueing delays close to 1 second typic=
al from a single home.</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">Because there are relatively small numbers =
of flows, *way* too much buffering, leading to sustained queues that could =
not drain fast enough under ECN or anything else, user experience *sucked*.=
</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">Which is why Comcast tried &quot;killing Bi=
tTorrent&quot; with Sandvine DPI-based RST-injection, and started claiming =
it was to &quot;stop music piracy, which was destroying the Internet&quot;.=
</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">When Comcast was forced to actually look at=
 why the queues were sustained and unfixable, they realized that the proble=
m was that DOCSIS was *designed* to hold packets rather than drop them.=A0 =
ECN marking would probably not have helped get the queues down under 50 mse=
c, without dropping packets, neither would RED.=A0 The only way to get the =
queue down is to prevent it from being created in the first place....=A0 An=
d since the RTT from US east coast to US west coast is under 45 msec, it do=
es little good to send early congestion signals.</p>

<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">I&#39;m sure you know this.</p>
<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">An analogous problem occurs today because o=
f the 3-4 *second* buffering in HSPA and LTE networks under load.=A0 Having=
 measured it in many American and European deployments of such equipment, I=
 would suggest that the fundamental problem is *dropping packets fairly and=
 quickly*.=A0 You can&#39;t get there (to sub-50 msec. queueing) with ECN o=
r RED.</p>
</font></blockquote><div><br></div><div style>First off: everyone has made =
the same mistake: operating system writers, DSL, and GPON, just to name a f=
ew. =A0Don&#39;t single out cable. =A0We&#39;ve all screwed up for a long t=
ime; in this sense, we&#39;re all &quot;Bell Heads&quot;.</div>
<div style><br></div><div style>With flow queuing such as fq_codel or sfqco=
del, the ECN equation changes: other flows don&#39;t get held up the way th=
ey do when running a simple AQM such as RED in a single queue.</div><div st=
yle>
<br></div><div style>And the web&#39;s behavior means that any straight AQM=
 such as CoDel, RED or PIE *cannot* get latency anywhere where it needs to =
be: too many packets end up in flight at once to get stuck in any queue; we=
 *have* to do something to solve this problem: flow queuing makes a huge di=
fference. =A0I&#39;ve measured 100&#39;s of milliseconds of transient laten=
cy hitting some web sites as these packet flights hit the broadband edge. =
=A0This becomes a non-issue with fq_codel.</div>
<div style><br></div><div style>Even so, Dave Taht&#39;s experiments (IIRC)=
 were that for low bandwidth, drop was preferable for ECN as even a single =
full sized packet takes considerable time.</div><div style><br></div><div s=
tyle>
Exactly how this is going to play out is really only something we can figur=
e out by experiment.</div><div style><br></div><div style>I also need to cl=
arify what I mean by &quot;Bell Heads&quot;: these people are *not* restric=
ted to the telecom industry, but everywhere. =A0In fact, Dave Taht first en=
countered &quot;every packet is precious&quot; in working on 802.11, where =
the driver writers seemed to think that ever dropping even one packet was e=
vil, and would retransmit effectively infinitely. =A0So those of you workin=
g on cellular wireless are in good company, and don&#39;t take this persona=
lly.</div>
<div style>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0- Jim</div><div style><br></div><div=
 style><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex"><font face=3D"times new rom=
an"><div>
<div class=3D"h5">
<p style=3D"margin:0;padding:0">=A0</p>
<p style=3D"margin:0;padding:0">-----Original Message-----<br>From: &quot;J=
ohn Leslie&quot; &lt;<a href=3D"mailto:john@jlc.net" target=3D"_blank">john=
@jlc.net</a>&gt;<br>Sent: Tuesday, March 19, 2013 11:56pm<br>To: <a href=3D=
"mailto:dpreed@reed.com" target=3D"_blank">dpreed@reed.com</a><br>
Cc: <a href=3D"mailto:aqm@ietf.org" target=3D"_blank">aqm@ietf.org</a><br>S=
ubject: Draining queues<br><br></p>
<div>
<p style=3D"margin:0;padding:0"><a href=3D"mailto:dpreed@reed.com" target=
=3D"_blank">dpreed@reed.com</a> &lt;<a href=3D"mailto:dpreed@reed.com" targ=
et=3D"_blank">dpreed@reed.com</a>&gt; wrote:<br>&gt; <br>&gt; A problem wit=
h ECN is that it does not drain an already flooded queue,<br>
<br> I presume you mean that ECN doesn&#39;t drop the packet it marks.<br><=
br>&gt; and depends (like RED) on the idea that one has a large queue in no=
rmal<br>&gt; operation.<br><br> I don&#39;t follow this at all. ECN is poin=
tless without an AQM which<br>
drops or marks well before tail-drop. Were we to run ECN without AQM,<br>we=
&#39;d have to drop each packet anyway after marking it.<br><br> Several fl=
avors of AQM mark/drop well enough in advance of tail-drop<br>that the queu=
e need never fill. Did you mean to say that ECN requires<br>
being able to queue the ECN-marked packet which would otherwise be<br>dropp=
ed by the AQM?<br><br>&gt; It is not obvious that sustaining a deep queue i=
s a desirable state,<br><br> Agreed!<br><br>&gt; especially when there are =
highly variable capacities along any path,<br>
&gt; as is the case where one link is more than 2x slower than the rest of<=
br>&gt; the links.<br><br> Isn&#39;t that _always_ the case?<br><br>&gt; If=
 the dynamics of user loads are fractal, this will cause draining<br>&gt; (=
and latency reduction) not to work.<br>
<br> I&#39;m guessing you mean that under &quot;fractal&quot; loads there&#=
39;s a tendency<br>for ECN-marked packets to clump and introduce latency at=
 the bottleneck.<br>Recall that this is offset by the quicker response to a=
n ECN mark at<br>
RTT speed instead of having to infer a packet loss.<br><br>&gt; If user loa=
ds are stationary gaussian and so forth, ECN can &quot;tune&quot;<br>&gt; t=
o relatively stable situations, but the queueing delay will be quite<br>
&gt; long end-to-end.<br><br> I don&#39;t follow... Some AQMs may indeed al=
low queueing delay to grow<br>large; but this can only happen if the instan=
taneous percentage of<br>marked packets is quite a bit higher than TCP can =
tolerate.<br>
<br>&gt; Since there is lots of evidence that user loads are fractal, dropp=
ing<br>&gt; packets during sudden bursts will prevent sustained queueing, w=
hereas<br>&gt; ECN will not.<br><br> I believe there is substantial agreeme=
nt that ECN-marked packets<br>
will need to be dropped anyway under particularly severe congestion.<br>At =
the very least, tail-drop can become necessary...<br><br>&gt; So a proper a=
pproach (in my opinion) on the &quot;real&quot; Internet (rather<br>&gt; th=
an a bunch of competing long-duration FTPs) requires keeping the<br>
&gt; queues so short that there is no real opportunity to benefit from ECN.=
<br><br> Here I quite disagree.<br><br> An AQM could be designed to ECN-mar=
k at a 1 millisecond queue depth;<br>and that can give the full benefit of =
ECN. Again, recall that ECN will<br>
deliver the congestion signal in one RTT, instead of waiting to infer<br>a =
loss. TCP stops delivering useful bandwidth at one or two percent<br>packet=
 loss: so it&#39;s hard to believe that ECN-marked packets will clog<br>
the path significantly.<br><br>&gt; Hence - we need to really see what happ=
ens with *real* traffic loads,<br>&gt; not simplified simulations.<br><br> =
Agreed! (Now if we could just get enough ECN deployment...)<br><br>&gt; Rea=
l traffic loads are fractal, and fractal loads do not generally<br>
&gt; obey the Gaussian style &quot;law of large numbers&quot; - multiple fr=
actal<br>&gt; flows are *rougher* not smoother than a single one.<br><br> I=
 would say rather that real traffic loads _contain_ fractal flows.<br>Aggre=
gation simply wouldn&#39;t work if we never experienced gaussian<br>
smoothing.<br><br>&gt; So check your mathematical &quot;intuitions&quot; at=
 the door.  Stop writing<br>&gt; equations based on tractable analytic mode=
ls, because actual traffic<br>&gt; does not follow those models.<br><br>
 I strongly suspect our models _are_ faulty -- but we need research<br>resu=
lts for _actual_ flows, with and without ECN, is order to diagnose<br>their=
 faults.<br><br>--<br>John Leslie &lt;<a href=3D"mailto:john@jlc.net" targe=
t=3D"_blank">john@jlc.net</a>&gt;</p>

</div></div></div></font><br>______________________________________________=
_<br>
aqm mailing list<br>
<a href=3D"mailto:aqm@ietf.org">aqm@ietf.org</a><br>
<a href=3D"https://www.ietf.org/mailman/listinfo/aqm" target=3D"_blank">htt=
ps://www.ietf.org/mailman/listinfo/aqm</a><br>
<br></blockquote></div><br></div></div>

--e89a8ff1c7ea38147104d85daa73--

From rs@netapp.com  Wed Mar 20 09:31:45 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C2CA521F9040 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:31:45 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.228
X-Spam-Level: 
X-Spam-Status: No, score=-10.228 tagged_above=-999 required=5 tests=[AWL=0.143, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A2yfvl+7vvXG for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:31:44 -0700 (PDT)
Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) by ietfa.amsl.com (Postfix) with ESMTP id 6412321F8FC5 for <aqm@ietf.org>; Wed, 20 Mar 2013 09:31:44 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,879,1355126400"; d="scan'208,217";a="32379274"
Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx12-out.netapp.com with ESMTP; 20 Mar 2013 09:31:44 -0700
Received: from vmwexceht01-prd.hq.netapp.com (vmwexceht01-prd.hq.netapp.com [10.106.76.239]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2KGVi0p019551; Wed, 20 Mar 2013 09:31:44 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0342.003; Wed, 20 Mar 2013 09:31:43 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: "dpreed@reed.com" <dpreed@reed.com>, John Leslie <john@jlc.net>
Thread-Topic: [aqm] Draining queues
Thread-Index: AQHOJR702KCqJKWJbU2rneySNQC+ZJivDcsA//+19WA=
Date: Wed, 20 Mar 2013 16:31:42 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24AC0750@SACEXCMBX02-PRD.hq.netapp.com>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <20130320035645.GI9569@verdi> <1363787198.319422762@apps.rackspace.com>
In-Reply-To: <1363787198.319422762@apps.rackspace.com>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: multipart/alternative; boundary="_000_012C3117EDDB3C4781FD802A8C27DD4F24AC0750SACEXCMBX02PRDh_"
MIME-Version: 1.0
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:31:45 -0000

--_000_012C3117EDDB3C4781FD802A8C27DD4F24AC0750SACEXCMBX02PRDh_
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

V2VsbCwNCg0KdGhlcmUgYXJlIHRob3NlIHdobyBiZWxpZXZlIHRoYXQgbWFya2luZy9kcm9wcGlu
ZyBzaG91bGQgT05MWSBiZSBkb25lIG9uIGVucXVldWUgaW50byBhIHF1ZXVlIChvciBhdCBsZWFz
dCB0aGF0IHRoZXJlIGlzIG5vIGRpZmZlcmVuY2UgYmV0d2VlbiBkb2luZyBpdCBhdCBlbnF1ZXVl
IHZzLiBkZXF1ZXVlKSwNCg0KaHR0cDovL2JyYW1jb2hlbi5jb20vMjAxMi8wNS8wNy90Y3Atc3Vj
a3MjY29tbWVudC02Njk2NDEwNTMNCg0KYW5kIHRoaW5rIGEgcGFja2V0IG9uY2UgcHJvY2Vzc2Vk
IGRlc2VydmVzIGRlbGl2ZXJ5IOKAkyBwb3RlbnRpYWxseSBicmVha2luZywgb3IgYXQgbGVhc3Qg
ZGVsYXlpbmcsIHRoZSBjb250cm9sIGxvb3Agb24gcmVzcG9uc2l2ZSB0cmFuc3BvcnQgcHJvdG9j
b2xzIChUQ1AgYmVpbmcgdGhlIHByaW1lIGV4YW1wbGUpLg0KDQpBcyBNYXR0IE0uIHBvaW50ZWQg
b3V0LCB0aGUgY29udHJvbCBsb29wcyBvZiBhbiBlbmQtdG8tZW5kIHRyYW5zcG9ydCBhcmUgYW4g
ZW50YW5nbGVkIG1lc3MuIFRoZXJlZm9yZSBpdCBpcyBwb2ludGxlc3MgdG8gc3BlY3VsYXRlIHdo
YXQgb25lIG1pbm9yIHR3ZWFrIHdpbGwgaGF2ZSwgd2l0aG91dCBhbHNvIGNvbnNpZGVyaW5nIHRo
ZSBmZWVkYmFjayBlZmZlY3RzIHRoaXMgaGFzIG9uIHRoZSBvdmVyYWxsIHdob2xl4oCmDQoNCkVD
TiBieSBpdHNlbGYgaXMgb25seSBwYXJ0IG9mIHRoZSBzb2x1dGlvbiAoYW5kIHRoZXJlIGFyZSBk
aXNjdXNzaW9uIHRvIGFjaGlldmUgbW9yZSBhY2N1cmF0ZSBFQ04gZmVlZGJhY2sgaW4gVENQLCBm
b3IgYSBudW1iZXIgb2YgcmVhc29ucyksIEVDTiBjYW4gbm90IGxpdmUgd2l0aG91dCBhIHByb3Bl
ciBBUU0sIGFuZCBhIHJlc3BvbnNpdmUgKCEpIHRyYW5zcG9ydCBwcm90b2NvbC4NCg0KQmVzdCBy
ZWdhcmRzLA0KDQpSaWNoYXJkIFNjaGVmZmVuZWdnZXINCg0KDQoNCkZyb206IGFxbS1ib3VuY2Vz
QGlldGYub3JnIFttYWlsdG86YXFtLWJvdW5jZXNAaWV0Zi5vcmddIE9uIEJlaGFsZiBPZiBkcHJl
ZWRAcmVlZC5jb20NClNlbnQ6IE1pdHR3b2NoLCAyMC4gTcOkcnogMjAxMyAxNDo0Nw0KVG86IEpv
aG4gTGVzbGllDQpDYzogYXFtQGlldGYub3JnDQpTdWJqZWN0OiBSZTogW2FxbV0gRHJhaW5pbmcg
cXVldWVzDQoNCg0KSm9obiAtIEkgdW5kZXJzdGFuZCB5b3VyIHJlc3BvbnNlLCBhbmQgSSB1bmRl
cnN0YW5kIHRoZSBpZGVhIG9mIGRvaW5nIEVDTiBtYXJraW5nIGF0IHRoZSAxIG1zZWMuIGxldmVs
LiAgVGhhdCBjZXJ0YWlubHkgd291bGQgaW5jcmVhc2UgdGhlIGFtb3VudCBvZiBjb25nZXN0aW9u
IGluZm9ybWF0aW9uIGRlbGl2ZXJlZCB0byB0aGUgZW5kcG9pbnRzLg0KDQoNCg0KSG93ZXZlciwg
aWYgeW91IHRoaW5rIGFib3V0IGhvdyBsb25nIGl0IHdpbGwgdGFrZSBmb3IgdGhlIHJlY2VpdmVy
IHRvIHJlZHVjZSBpdHMgIndpbmRvdyIgaW4gdHlwaWNhbCBjYXNlcywgYWxvbmcgd2l0aCB0aGUg
InJlbHVjdGFuY2UiIG9mIHJvdXRlciBkZXNpZ25lcnMgdG8gZXZlciBkcm9wIHBhY2tldHMsIEkg
dGhpbmsgdGhpcyBkb2VzIG5vdCBzb2x2ZSB0aGUgcHJvYmxlbS4NCg0KDQoNClllcywgRUNOIG1p
Z2h0IGhlbHAgYSBiaXQgaW4gdGhlIGNhc2Ugd2hlcmUgdGhlcmUgYXJlICpsYXJnZSogbnVtYmVy
cyBvZiBzdGF0aW9uYXJ5LXByb2Nlc3MgVENQIGZsb3dzIHRyYXZlcnNpbmcgYSBub2RlLCBsZWFk
aW5nIHRvIG5lYXJseSBmdWxsIGNhcGFjaXR5Lg0KDQoNCg0KQnV0IG9uZSBzaG91bGQgZm9jdXMg
b24gdGhlIGFjdHVhbGx5IGRpZmZpY3VsdCBwcm9ibGVtczogZS5nLiB3aGF0IHdlbnQgd3Jvbmcg
aW4gRE9DU0lTIDIuMCBhbmQgZWFybHkgRE9DU0lTIDMgZGVwbG95bWVudHMsIHN0aWxsIGluIHRo
ZSBmaWVsZCwgdGhhdCBtYWRlIHF1ZXVlaW5nIGRlbGF5cyBjbG9zZSB0byAxIHNlY29uZCB0eXBp
Y2FsIGZyb20gYSBzaW5nbGUgaG9tZS4NCg0KDQoNCkJlY2F1c2UgdGhlcmUgYXJlIHJlbGF0aXZl
bHkgc21hbGwgbnVtYmVycyBvZiBmbG93cywgKndheSogdG9vIG11Y2ggYnVmZmVyaW5nLCBsZWFk
aW5nIHRvIHN1c3RhaW5lZCBxdWV1ZXMgdGhhdCBjb3VsZCBub3QgZHJhaW4gZmFzdCBlbm91Z2gg
dW5kZXIgRUNOIG9yIGFueXRoaW5nIGVsc2UsIHVzZXIgZXhwZXJpZW5jZSAqc3Vja2VkKi4NCg0K
DQoNCldoaWNoIGlzIHdoeSBDb21jYXN0IHRyaWVkICJraWxsaW5nIEJpdFRvcnJlbnQiIHdpdGgg
U2FuZHZpbmUgRFBJLWJhc2VkIFJTVC1pbmplY3Rpb24sIGFuZCBzdGFydGVkIGNsYWltaW5nIGl0
IHdhcyB0byAic3RvcCBtdXNpYyBwaXJhY3ksIHdoaWNoIHdhcyBkZXN0cm95aW5nIHRoZSBJbnRl
cm5ldCIuDQoNCg0KDQpXaGVuIENvbWNhc3Qgd2FzIGZvcmNlZCB0byBhY3R1YWxseSBsb29rIGF0
IHdoeSB0aGUgcXVldWVzIHdlcmUgc3VzdGFpbmVkIGFuZCB1bmZpeGFibGUsIHRoZXkgcmVhbGl6
ZWQgdGhhdCB0aGUgcHJvYmxlbSB3YXMgdGhhdCBET0NTSVMgd2FzICpkZXNpZ25lZCogdG8gaG9s
ZCBwYWNrZXRzIHJhdGhlciB0aGFuIGRyb3AgdGhlbS4gIEVDTiBtYXJraW5nIHdvdWxkIHByb2Jh
Ymx5IG5vdCBoYXZlIGhlbHBlZCBnZXQgdGhlIHF1ZXVlcyBkb3duIHVuZGVyIDUwIG1zZWMsIHdp
dGhvdXQgZHJvcHBpbmcgcGFja2V0cywgbmVpdGhlciB3b3VsZCBSRUQuICBUaGUgb25seSB3YXkg
dG8gZ2V0IHRoZSBxdWV1ZSBkb3duIGlzIHRvIHByZXZlbnQgaXQgZnJvbSBiZWluZyBjcmVhdGVk
IGluIHRoZSBmaXJzdCBwbGFjZS4uLi4gIEFuZCBzaW5jZSB0aGUgUlRUIGZyb20gVVMgZWFzdCBj
b2FzdCB0byBVUyB3ZXN0IGNvYXN0IGlzIHVuZGVyIDQ1IG1zZWMsIGl0IGRvZXMgbGl0dGxlIGdv
b2QgdG8gc2VuZCBlYXJseSBjb25nZXN0aW9uIHNpZ25hbHMuDQoNCg0KDQpJJ20gc3VyZSB5b3Ug
a25vdyB0aGlzLg0KDQoNCg0KQW4gYW5hbG9nb3VzIHByb2JsZW0gb2NjdXJzIHRvZGF5IGJlY2F1
c2Ugb2YgdGhlIDMtNCAqc2Vjb25kKiBidWZmZXJpbmcgaW4gSFNQQSBhbmQgTFRFIG5ldHdvcmtz
IHVuZGVyIGxvYWQuICBIYXZpbmcgbWVhc3VyZWQgaXQgaW4gbWFueSBBbWVyaWNhbiBhbmQgRXVy
b3BlYW4gZGVwbG95bWVudHMgb2Ygc3VjaCBlcXVpcG1lbnQsIEkgd291bGQgc3VnZ2VzdCB0aGF0
IHRoZSBmdW5kYW1lbnRhbCBwcm9ibGVtIGlzICpkcm9wcGluZyBwYWNrZXRzIGZhaXJseSBhbmQg
cXVpY2tseSouICBZb3UgY2FuJ3QgZ2V0IHRoZXJlICh0byBzdWItNTAgbXNlYy4gcXVldWVpbmcp
IHdpdGggRUNOIG9yIFJFRC4NCg0KDQoNCi0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQpGcm9t
OiAiSm9obiBMZXNsaWUiIDxqb2huQGpsYy5uZXQ8bWFpbHRvOmpvaG5AamxjLm5ldD4+DQpTZW50
OiBUdWVzZGF5LCBNYXJjaCAxOSwgMjAxMyAxMTo1NnBtDQpUbzogZHByZWVkQHJlZWQuY29tPG1h
aWx0bzpkcHJlZWRAcmVlZC5jb20+DQpDYzogYXFtQGlldGYub3JnPG1haWx0bzphcW1AaWV0Zi5v
cmc+DQpTdWJqZWN0OiBEcmFpbmluZyBxdWV1ZXMNCg0KZHByZWVkQHJlZWQuY29tPG1haWx0bzpk
cHJlZWRAcmVlZC5jb20+IDxkcHJlZWRAcmVlZC5jb208bWFpbHRvOmRwcmVlZEByZWVkLmNvbT4+
IHdyb3RlOg0KPg0KPiBBIHByb2JsZW0gd2l0aCBFQ04gaXMgdGhhdCBpdCBkb2VzIG5vdCBkcmFp
biBhbiBhbHJlYWR5IGZsb29kZWQgcXVldWUsDQoNCkkgcHJlc3VtZSB5b3UgbWVhbiB0aGF0IEVD
TiBkb2Vzbid0IGRyb3AgdGhlIHBhY2tldCBpdCBtYXJrcy4NCg0KPiBhbmQgZGVwZW5kcyAobGlr
ZSBSRUQpIG9uIHRoZSBpZGVhIHRoYXQgb25lIGhhcyBhIGxhcmdlIHF1ZXVlIGluIG5vcm1hbA0K
PiBvcGVyYXRpb24uDQoNCkkgZG9uJ3QgZm9sbG93IHRoaXMgYXQgYWxsLiBFQ04gaXMgcG9pbnRs
ZXNzIHdpdGhvdXQgYW4gQVFNIHdoaWNoDQpkcm9wcyBvciBtYXJrcyB3ZWxsIGJlZm9yZSB0YWls
LWRyb3AuIFdlcmUgd2UgdG8gcnVuIEVDTiB3aXRob3V0IEFRTSwNCndlJ2QgaGF2ZSB0byBkcm9w
IGVhY2ggcGFja2V0IGFueXdheSBhZnRlciBtYXJraW5nIGl0Lg0KDQpTZXZlcmFsIGZsYXZvcnMg
b2YgQVFNIG1hcmsvZHJvcCB3ZWxsIGVub3VnaCBpbiBhZHZhbmNlIG9mIHRhaWwtZHJvcA0KdGhh
dCB0aGUgcXVldWUgbmVlZCBuZXZlciBmaWxsLiBEaWQgeW91IG1lYW4gdG8gc2F5IHRoYXQgRUNO
IHJlcXVpcmVzDQpiZWluZyBhYmxlIHRvIHF1ZXVlIHRoZSBFQ04tbWFya2VkIHBhY2tldCB3aGlj
aCB3b3VsZCBvdGhlcndpc2UgYmUNCmRyb3BwZWQgYnkgdGhlIEFRTT8NCg0KPiBJdCBpcyBub3Qg
b2J2aW91cyB0aGF0IHN1c3RhaW5pbmcgYSBkZWVwIHF1ZXVlIGlzIGEgZGVzaXJhYmxlIHN0YXRl
LA0KDQpBZ3JlZWQhDQoNCj4gZXNwZWNpYWxseSB3aGVuIHRoZXJlIGFyZSBoaWdobHkgdmFyaWFi
bGUgY2FwYWNpdGllcyBhbG9uZyBhbnkgcGF0aCwNCj4gYXMgaXMgdGhlIGNhc2Ugd2hlcmUgb25l
IGxpbmsgaXMgbW9yZSB0aGFuIDJ4IHNsb3dlciB0aGFuIHRoZSByZXN0IG9mDQo+IHRoZSBsaW5r
cy4NCg0KSXNuJ3QgdGhhdCBfYWx3YXlzXyB0aGUgY2FzZT8NCg0KPiBJZiB0aGUgZHluYW1pY3Mg
b2YgdXNlciBsb2FkcyBhcmUgZnJhY3RhbCwgdGhpcyB3aWxsIGNhdXNlIGRyYWluaW5nDQo+IChh
bmQgbGF0ZW5jeSByZWR1Y3Rpb24pIG5vdCB0byB3b3JrLg0KDQpJJ20gZ3Vlc3NpbmcgeW91IG1l
YW4gdGhhdCB1bmRlciAiZnJhY3RhbCIgbG9hZHMgdGhlcmUncyBhIHRlbmRlbmN5DQpmb3IgRUNO
LW1hcmtlZCBwYWNrZXRzIHRvIGNsdW1wIGFuZCBpbnRyb2R1Y2UgbGF0ZW5jeSBhdCB0aGUgYm90
dGxlbmVjay4NClJlY2FsbCB0aGF0IHRoaXMgaXMgb2Zmc2V0IGJ5IHRoZSBxdWlja2VyIHJlc3Bv
bnNlIHRvIGFuIEVDTiBtYXJrIGF0DQpSVFQgc3BlZWQgaW5zdGVhZCBvZiBoYXZpbmcgdG8gaW5m
ZXIgYSBwYWNrZXQgbG9zcy4NCg0KPiBJZiB1c2VyIGxvYWRzIGFyZSBzdGF0aW9uYXJ5IGdhdXNz
aWFuIGFuZCBzbyBmb3J0aCwgRUNOIGNhbiAidHVuZSINCj4gdG8gcmVsYXRpdmVseSBzdGFibGUg
c2l0dWF0aW9ucywgYnV0IHRoZSBxdWV1ZWluZyBkZWxheSB3aWxsIGJlIHF1aXRlDQo+IGxvbmcg
ZW5kLXRvLWVuZC4NCg0KSSBkb24ndCBmb2xsb3cuLi4gU29tZSBBUU1zIG1heSBpbmRlZWQgYWxs
b3cgcXVldWVpbmcgZGVsYXkgdG8gZ3Jvdw0KbGFyZ2U7IGJ1dCB0aGlzIGNhbiBvbmx5IGhhcHBl
biBpZiB0aGUgaW5zdGFudGFuZW91cyBwZXJjZW50YWdlIG9mDQptYXJrZWQgcGFja2V0cyBpcyBx
dWl0ZSBhIGJpdCBoaWdoZXIgdGhhbiBUQ1AgY2FuIHRvbGVyYXRlLg0KDQo+IFNpbmNlIHRoZXJl
IGlzIGxvdHMgb2YgZXZpZGVuY2UgdGhhdCB1c2VyIGxvYWRzIGFyZSBmcmFjdGFsLCBkcm9wcGlu
Zw0KPiBwYWNrZXRzIGR1cmluZyBzdWRkZW4gYnVyc3RzIHdpbGwgcHJldmVudCBzdXN0YWluZWQg
cXVldWVpbmcsIHdoZXJlYXMNCj4gRUNOIHdpbGwgbm90Lg0KDQpJIGJlbGlldmUgdGhlcmUgaXMg
c3Vic3RhbnRpYWwgYWdyZWVtZW50IHRoYXQgRUNOLW1hcmtlZCBwYWNrZXRzDQp3aWxsIG5lZWQg
dG8gYmUgZHJvcHBlZCBhbnl3YXkgdW5kZXIgcGFydGljdWxhcmx5IHNldmVyZSBjb25nZXN0aW9u
Lg0KQXQgdGhlIHZlcnkgbGVhc3QsIHRhaWwtZHJvcCBjYW4gYmVjb21lIG5lY2Vzc2FyeS4uLg0K
DQo+IFNvIGEgcHJvcGVyIGFwcHJvYWNoIChpbiBteSBvcGluaW9uKSBvbiB0aGUgInJlYWwiIElu
dGVybmV0IChyYXRoZXINCj4gdGhhbiBhIGJ1bmNoIG9mIGNvbXBldGluZyBsb25nLWR1cmF0aW9u
IEZUUHMpIHJlcXVpcmVzIGtlZXBpbmcgdGhlDQo+IHF1ZXVlcyBzbyBzaG9ydCB0aGF0IHRoZXJl
IGlzIG5vIHJlYWwgb3Bwb3J0dW5pdHkgdG8gYmVuZWZpdCBmcm9tIEVDTi4NCg0KSGVyZSBJIHF1
aXRlIGRpc2FncmVlLg0KDQpBbiBBUU0gY291bGQgYmUgZGVzaWduZWQgdG8gRUNOLW1hcmsgYXQg
YSAxIG1pbGxpc2Vjb25kIHF1ZXVlIGRlcHRoOw0KYW5kIHRoYXQgY2FuIGdpdmUgdGhlIGZ1bGwg
YmVuZWZpdCBvZiBFQ04uIEFnYWluLCByZWNhbGwgdGhhdCBFQ04gd2lsbA0KZGVsaXZlciB0aGUg
Y29uZ2VzdGlvbiBzaWduYWwgaW4gb25lIFJUVCwgaW5zdGVhZCBvZiB3YWl0aW5nIHRvIGluZmVy
DQphIGxvc3MuIFRDUCBzdG9wcyBkZWxpdmVyaW5nIHVzZWZ1bCBiYW5kd2lkdGggYXQgb25lIG9y
IHR3byBwZXJjZW50DQpwYWNrZXQgbG9zczogc28gaXQncyBoYXJkIHRvIGJlbGlldmUgdGhhdCBF
Q04tbWFya2VkIHBhY2tldHMgd2lsbCBjbG9nDQp0aGUgcGF0aCBzaWduaWZpY2FudGx5Lg0KDQo+
IEhlbmNlIC0gd2UgbmVlZCB0byByZWFsbHkgc2VlIHdoYXQgaGFwcGVucyB3aXRoICpyZWFsKiB0
cmFmZmljIGxvYWRzLA0KPiBub3Qgc2ltcGxpZmllZCBzaW11bGF0aW9ucy4NCg0KQWdyZWVkISAo
Tm93IGlmIHdlIGNvdWxkIGp1c3QgZ2V0IGVub3VnaCBFQ04gZGVwbG95bWVudC4uLikNCg0KPiBS
ZWFsIHRyYWZmaWMgbG9hZHMgYXJlIGZyYWN0YWwsIGFuZCBmcmFjdGFsIGxvYWRzIGRvIG5vdCBn
ZW5lcmFsbHkNCj4gb2JleSB0aGUgR2F1c3NpYW4gc3R5bGUgImxhdyBvZiBsYXJnZSBudW1iZXJz
IiAtIG11bHRpcGxlIGZyYWN0YWwNCj4gZmxvd3MgYXJlICpyb3VnaGVyKiBub3Qgc21vb3RoZXIg
dGhhbiBhIHNpbmdsZSBvbmUuDQoNCkkgd291bGQgc2F5IHJhdGhlciB0aGF0IHJlYWwgdHJhZmZp
YyBsb2FkcyBfY29udGFpbl8gZnJhY3RhbCBmbG93cy4NCkFnZ3JlZ2F0aW9uIHNpbXBseSB3b3Vs
ZG4ndCB3b3JrIGlmIHdlIG5ldmVyIGV4cGVyaWVuY2VkIGdhdXNzaWFuDQpzbW9vdGhpbmcuDQoN
Cj4gU28gY2hlY2sgeW91ciBtYXRoZW1hdGljYWwgImludHVpdGlvbnMiIGF0IHRoZSBkb29yLiBT
dG9wIHdyaXRpbmcNCj4gZXF1YXRpb25zIGJhc2VkIG9uIHRyYWN0YWJsZSBhbmFseXRpYyBtb2Rl
bHMsIGJlY2F1c2UgYWN0dWFsIHRyYWZmaWMNCj4gZG9lcyBub3QgZm9sbG93IHRob3NlIG1vZGVs
cy4NCg0KSSBzdHJvbmdseSBzdXNwZWN0IG91ciBtb2RlbHMgX2FyZV8gZmF1bHR5IC0tIGJ1dCB3
ZSBuZWVkIHJlc2VhcmNoDQpyZXN1bHRzIGZvciBfYWN0dWFsXyBmbG93cywgd2l0aCBhbmQgd2l0
aG91dCBFQ04sIGlzIG9yZGVyIHRvIGRpYWdub3NlDQp0aGVpciBmYXVsdHMuDQoNCi0tDQpKb2hu
IExlc2xpZSA8am9obkBqbGMubmV0PG1haWx0bzpqb2huQGpsYy5uZXQ+Pg0K

--_000_012C3117EDDB3C4781FD802A8C27DD4F24AC0750SACEXCMBX02PRDh_
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: base64

PGh0bWwgeG1sbnM6dj0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp2bWwiIHhtbG5zOm89InVy
bjpzY2hlbWFzLW1pY3Jvc29mdC1jb206b2ZmaWNlOm9mZmljZSIgeG1sbnM6dz0idXJuOnNjaGVt
YXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIgeG1sbnM6bT0iaHR0cDovL3NjaGVtYXMubWlj
cm9zb2Z0LmNvbS9vZmZpY2UvMjAwNC8xMi9vbW1sIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcv
VFIvUkVDLWh0bWw0MCI+DQo8aGVhZD4NCjxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIg
Y29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PXV0Zi04Ij4NCjxtZXRhIG5hbWU9IkdlbmVyYXRv
ciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTQgKGZpbHRlcmVkIG1lZGl1bSkiPg0KPHN0eWxl
PjwhLS0NCi8qIEZvbnQgRGVmaW5pdGlvbnMgKi8NCkBmb250LWZhY2UNCgl7Zm9udC1mYW1pbHk6
Q2FsaWJyaTsNCglwYW5vc2UtMToyIDE1IDUgMiAyIDIgNCAzIDIgNDt9DQpAZm9udC1mYWNlDQoJ
e2ZvbnQtZmFtaWx5OlRhaG9tYTsNCglwYW5vc2UtMToyIDExIDYgNCAzIDUgNCA0IDIgNDt9DQov
KiBTdHlsZSBEZWZpbml0aW9ucyAqLw0KcC5Nc29Ob3JtYWwsIGxpLk1zb05vcm1hbCwgZGl2Lk1z
b05vcm1hbA0KCXttYXJnaW46MGNtOw0KCW1hcmdpbi1ib3R0b206LjAwMDFwdDsNCglmb250LXNp
emU6MTIuMHB0Ow0KCWZvbnQtZmFtaWx5OiJUaW1lcyBOZXcgUm9tYW4iLCJzZXJpZiI7fQ0KYTps
aW5rLCBzcGFuLk1zb0h5cGVybGluaw0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6
Ymx1ZTsNCgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30NCmE6dmlzaXRlZCwgc3Bhbi5Nc29I
eXBlcmxpbmtGb2xsb3dlZA0KCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7DQoJY29sb3I6cHVycGxl
Ow0KCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQ0KcA0KCXttc28tc3R5bGUtcHJpb3JpdHk6
OTk7DQoJbXNvLW1hcmdpbi10b3AtYWx0OmF1dG87DQoJbWFyZ2luLXJpZ2h0OjBjbTsNCgltc28t
bWFyZ2luLWJvdHRvbS1hbHQ6YXV0bzsNCgltYXJnaW4tbGVmdDowY207DQoJZm9udC1zaXplOjEy
LjBwdDsNCglmb250LWZhbWlseToiVGltZXMgTmV3IFJvbWFuIiwic2VyaWYiO30NCnAuTXNvQWNl
dGF0ZSwgbGkuTXNvQWNldGF0ZSwgZGl2Lk1zb0FjZXRhdGUNCgl7bXNvLXN0eWxlLXByaW9yaXR5
Ojk5Ow0KCW1zby1zdHlsZS1saW5rOiJCYWxsb29uIFRleHQgQ2hhciI7DQoJbWFyZ2luOjBjbTsN
CgltYXJnaW4tYm90dG9tOi4wMDAxcHQ7DQoJZm9udC1zaXplOjguMHB0Ow0KCWZvbnQtZmFtaWx5
OiJUYWhvbWEiLCJzYW5zLXNlcmlmIjt9DQpzcGFuLkJhbGxvb25UZXh0Q2hhcg0KCXttc28tc3R5
bGUtbmFtZToiQmFsbG9vbiBUZXh0IENoYXIiOw0KCW1zby1zdHlsZS1wcmlvcml0eTo5OTsNCglt
c28tc3R5bGUtbGluazoiQmFsbG9vbiBUZXh0IjsNCglmb250LWZhbWlseToiVGFob21hIiwic2Fu
cy1zZXJpZiI7DQoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6REUtQVQ7fQ0Kc3Bhbi5FbWFpbFN0eWxl
MjANCgl7bXNvLXN0eWxlLXR5cGU6cGVyc29uYWwtcmVwbHk7DQoJZm9udC1mYW1pbHk6IkNhbGli
cmkiLCJzYW5zLXNlcmlmIjsNCgljb2xvcjojMUY0OTdEO30NCi5Nc29DaHBEZWZhdWx0DQoJe21z
by1zdHlsZS10eXBlOmV4cG9ydC1vbmx5Ow0KCWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1z
ZXJpZiI7DQoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVM7fQ0KQHBhZ2UgV29yZFNlY3Rpb24x
DQoJe3NpemU6NjEyLjBwdCA3OTIuMHB0Ow0KCW1hcmdpbjo3MC44NXB0IDcwLjg1cHQgMi4wY20g
NzAuODVwdDt9DQpkaXYuV29yZFNlY3Rpb24xDQoJe3BhZ2U6V29yZFNlY3Rpb24xO30NCi0tPjwv
c3R5bGU+PCEtLVtpZiBndGUgbXNvIDldPjx4bWw+DQo8bzpzaGFwZWRlZmF1bHRzIHY6ZXh0PSJl
ZGl0IiBzcGlkbWF4PSIxMDI2IiAvPg0KPC94bWw+PCFbZW5kaWZdLS0+PCEtLVtpZiBndGUgbXNv
IDldPjx4bWw+DQo8bzpzaGFwZWxheW91dCB2OmV4dD0iZWRpdCI+DQo8bzppZG1hcCB2OmV4dD0i
ZWRpdCIgZGF0YT0iMSIgLz4NCjwvbzpzaGFwZWxheW91dD48L3htbD48IVtlbmRpZl0tLT4NCjwv
aGVhZD4NCjxib2R5IGxhbmc9IkRFLUFUIiBsaW5rPSJibHVlIiB2bGluaz0icHVycGxlIj4NCjxk
aXYgY2xhc3M9IldvcmRTZWN0aW9uMSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90
O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+V2VsbCw8bzpwPjwvbzpwPjwvc3Bhbj48
L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtm
b250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29s
b3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05v
cm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFt
aWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0
OTdEIj50aGVyZSBhcmUgdGhvc2Ugd2hvIGJlbGlldmUgdGhhdCBtYXJraW5nL2Ryb3BwaW5nIHNo
b3VsZCBPTkxZIGJlIGRvbmUgb24gZW5xdWV1ZSBpbnRvIGEgcXVldWUgKG9yIGF0IGxlYXN0IHRo
YXQgdGhlcmUgaXMgbm8gZGlmZmVyZW5jZSBiZXR3ZWVuDQogZG9pbmcgaXQgYXQgZW5xdWV1ZSB2
cy4gZGVxdWV1ZSksPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+
PHNwYW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZx
dW90O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj48
bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBs
YW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2Fs
aWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxhIGhyZWY9
Imh0dHA6Ly9icmFtY29oZW4uY29tLzIwMTIvMDUvMDcvdGNwLXN1Y2tzI2NvbW1lbnQtNjY5NjQx
MDUzIj5odHRwOi8vYnJhbWNvaGVuLmNvbS8yMDEyLzA1LzA3L3RjcC1zdWNrcyNjb21tZW50LTY2
OTY0MTA1MzwvYT48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48
c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1
b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPjxv
OnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxh
bmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxp
YnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+YW5kIHRoaW5r
IGEgcGFja2V0IG9uY2UgcHJvY2Vzc2VkIGRlc2VydmVzIGRlbGl2ZXJ5IOKAkyBwb3RlbnRpYWxs
eSBicmVha2luZywgb3IgYXQgbGVhc3QgZGVsYXlpbmcsIHRoZSBjb250cm9sIGxvb3Agb24gcmVz
cG9uc2l2ZSB0cmFuc3BvcnQgcHJvdG9jb2xzDQogKFRDUCBiZWluZyB0aGUgcHJpbWUgZXhhbXBs
ZSkuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFu
Zz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGli
cmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj48bzpwPiZuYnNw
OzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1V
UyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90
OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMxRjQ5N0QiPkFzIE1hdHQgTS4gcG9pbnRl
ZCBvdXQsIHRoZSBjb250cm9sIGxvb3BzIG9mIGFuIGVuZC10by1lbmQgdHJhbnNwb3J0IGFyZSBh
biBlbnRhbmdsZWQgbWVzcy4gVGhlcmVmb3JlIGl0IGlzIHBvaW50bGVzcyB0byBzcGVjdWxhdGUg
d2hhdCBvbmUgbWlub3INCiB0d2VhayB3aWxsIGhhdmUsIHdpdGhvdXQgYWxzbyBjb25zaWRlcmlu
ZyB0aGUgZmVlZGJhY2sgZWZmZWN0cyB0aGlzIGhhcyBvbiB0aGUgb3ZlcmFsbCB3aG9sZeKApjxv
OnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVO
LVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1
b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286
cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gbGFuZz0iRU4tVVMiIHN0
eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGlicmkmcXVvdDssJnF1
b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj5FQ04gYnkgaXRzZWxmIGlzIG9ubHkg
cGFydCBvZiB0aGUgc29sdXRpb24gKGFuZCB0aGVyZSBhcmUgZGlzY3Vzc2lvbiB0byBhY2hpZXZl
IG1vcmUgYWNjdXJhdGUgRUNOIGZlZWRiYWNrIGluIFRDUCwgZm9yIGEgbnVtYmVyIG9mIHJlYXNv
bnMpLCBFQ04NCiBjYW4gbm90IGxpdmUgd2l0aG91dCBhIHByb3BlciBBUU0sIGFuZCBhIHJlc3Bv
bnNpdmUgKCEpIHRyYW5zcG9ydCBwcm90b2NvbC48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBj
bGFzcz0iTXNvTm9ybWFsIj48c3BhbiBsYW5nPSJFTi1VUyIgc3R5bGU9ImZvbnQtc2l6ZToxMS4w
cHQ7Zm9udC1mYW1pbHk6JnF1b3Q7Q2FsaWJyaSZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7
O2NvbG9yOiMxRjQ5N0QiPjxvOnA+Jm5ic3A7PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250
LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6
IzFGNDk3RCI+QmVzdCByZWdhcmRzLDxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIGNsYXNzPSJN
c29Ob3JtYWwiPjxzcGFuIGxhbmc9IkVOLVVTIiBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250
LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6
IzFGNDk3RCI+PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNw
YW4gbGFuZz0iRU4tVVMiIHN0eWxlPSJmb250LXNpemU6MTEuMHB0O2ZvbnQtZmFtaWx5OiZxdW90
O0NhbGlicmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj48bzpw
PiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3BhbiBzdHls
ZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7LCZxdW90
O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+UmljaGFyZCBTY2hlZmZlbmVnZ2VyPC9z
cGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0NhbGli
cmkmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMUY0OTdEIj48YnI+DQo8YnI+
DQo8YnI+DQo8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48c3Bh
biBzdHlsZT0iZm9udC1zaXplOjExLjBwdDtmb250LWZhbWlseTomcXVvdDtDYWxpYnJpJnF1b3Q7
LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7Y29sb3I6IzFGNDk3RCI+PG86cD4mbmJzcDs8L286cD48
L3NwYW4+PC9wPg0KPGRpdiBzdHlsZT0iYm9yZGVyOm5vbmU7Ym9yZGVyLWxlZnQ6c29saWQgYmx1
ZSAxLjVwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDQuMHB0Ij4NCjxkaXY+DQo8ZGl2IHN0eWxlPSJi
b3JkZXI6bm9uZTtib3JkZXItdG9wOnNvbGlkICNCNUM0REYgMS4wcHQ7cGFkZGluZzozLjBwdCAw
Y20gMGNtIDBjbSI+DQo8cCBjbGFzcz0iTXNvTm9ybWFsIj48Yj48c3BhbiBsYW5nPSJFTi1VUyIg
c3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZx
dW90O3NhbnMtc2VyaWYmcXVvdDsiPkZyb206PC9zcGFuPjwvYj48c3BhbiBsYW5nPSJFTi1VUyIg
c3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGFob21hJnF1b3Q7LCZx
dW90O3NhbnMtc2VyaWYmcXVvdDsiPiBhcW0tYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOmFxbS1i
b3VuY2VzQGlldGYub3JnXQ0KPGI+T24gQmVoYWxmIE9mIDwvYj5kcHJlZWRAcmVlZC5jb208YnI+
DQo8Yj5TZW50OjwvYj4gTWl0dHdvY2gsIDIwLiBNw6RyeiAyMDEzIDE0OjQ3PGJyPg0KPGI+VG86
PC9iPiBKb2huIExlc2xpZTxicj4NCjxiPkNjOjwvYj4gYXFtQGlldGYub3JnPGJyPg0KPGI+U3Vi
amVjdDo8L2I+IFJlOiBbYXFtXSBEcmFpbmluZyBxdWV1ZXM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+
DQo8L2Rpdj4NCjwvZGl2Pg0KPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48
L3A+DQo8cCBzdHlsZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0
eWxlPSJmb250LXNpemU6MTAuMHB0Ij5Kb2huIC0gSSB1bmRlcnN0YW5kIHlvdXIgcmVzcG9uc2Us
IGFuZCBJIHVuZGVyc3RhbmQgdGhlIGlkZWEgb2YgZG9pbmcgRUNOIG1hcmtpbmcgYXQgdGhlIDEg
bXNlYy4gbGV2ZWwuJm5ic3A7IFRoYXQgY2VydGFpbmx5IHdvdWxkIGluY3JlYXNlIHRoZSBhbW91
bnQgb2YgY29uZ2VzdGlvbiBpbmZvcm1hdGlvbiBkZWxpdmVyZWQNCiB0byB0aGUgZW5kcG9pbnRz
LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0
b206LjAwMDFwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiZuYnNwOzxvOnA+PC9v
OnA+PC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAwMDFw
dCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPkhvd2V2ZXIsIGlmIHlvdSB0aGluayBh
Ym91dCBob3cgbG9uZyBpdCB3aWxsIHRha2UgZm9yIHRoZSByZWNlaXZlciB0byByZWR1Y2UgaXRz
ICZxdW90O3dpbmRvdyZxdW90OyBpbiB0eXBpY2FsIGNhc2VzLCBhbG9uZyB3aXRoIHRoZSAmcXVv
dDtyZWx1Y3RhbmNlJnF1b3Q7IG9mIHJvdXRlciBkZXNpZ25lcnMgdG8gZXZlciBkcm9wIHBhY2tl
dHMsIEkgdGhpbmsNCiB0aGlzIGRvZXMgbm90IHNvbHZlIHRoZSBwcm9ibGVtLjxvOnA+PC9vOnA+
PC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAwMDFwdCI+
PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiZuYnNwOzxvOnA+PC9vOnA+PC9zcGFuPjwv
cD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAwMDFwdCI+PHNwYW4gc3R5
bGU9ImZvbnQtc2l6ZToxMC4wcHQiPlllcywgRUNOIG1pZ2h0IGhlbHAgYSBiaXQgaW4gdGhlIGNh
c2Ugd2hlcmUgdGhlcmUgYXJlICpsYXJnZSogbnVtYmVycyBvZiBzdGF0aW9uYXJ5LXByb2Nlc3Mg
VENQIGZsb3dzIHRyYXZlcnNpbmcgYSBub2RlLCBsZWFkaW5nIHRvIG5lYXJseSBmdWxsIGNhcGFj
aXR5LjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1i
b3R0b206LjAwMDFwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiZuYnNwOzxvOnA+
PC9vOnA+PC9zcGFuPjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAw
MDFwdCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPkJ1dCBvbmUgc2hvdWxkIGZvY3Vz
IG9uIHRoZSBhY3R1YWxseSBkaWZmaWN1bHQgcHJvYmxlbXM6IGUuZy4gd2hhdCB3ZW50IHdyb25n
IGluIERPQ1NJUyAyLjAgYW5kIGVhcmx5IERPQ1NJUyAzIGRlcGxveW1lbnRzLCBzdGlsbCBpbiB0
aGUgZmllbGQsIHRoYXQgbWFkZSBxdWV1ZWluZyBkZWxheXMgY2xvc2UgdG8gMSBzZWNvbmQNCiB0
eXBpY2FsIGZyb20gYSBzaW5nbGUgaG9tZS48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBzdHls
ZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJmb250LXNp
emU6MTAuMHB0Ij4mbmJzcDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBzdHlsZT0ibWFyZ2lu
OjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0
Ij5CZWNhdXNlIHRoZXJlIGFyZSByZWxhdGl2ZWx5IHNtYWxsIG51bWJlcnMgb2YgZmxvd3MsICp3
YXkqIHRvbyBtdWNoIGJ1ZmZlcmluZywgbGVhZGluZyB0byBzdXN0YWluZWQgcXVldWVzIHRoYXQg
Y291bGQgbm90IGRyYWluIGZhc3QgZW5vdWdoIHVuZGVyIEVDTiBvciBhbnl0aGluZyBlbHNlLCB1
c2VyIGV4cGVyaWVuY2UNCiAqc3Vja2VkKi48bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBzdHls
ZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJmb250LXNp
emU6MTAuMHB0Ij4mbmJzcDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8cCBzdHlsZT0ibWFyZ2lu
OjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0
Ij5XaGljaCBpcyB3aHkgQ29tY2FzdCB0cmllZCAmcXVvdDtraWxsaW5nIEJpdFRvcnJlbnQmcXVv
dDsgd2l0aCBTYW5kdmluZSBEUEktYmFzZWQgUlNULWluamVjdGlvbiwgYW5kIHN0YXJ0ZWQgY2xh
aW1pbmcgaXQgd2FzIHRvICZxdW90O3N0b3AgbXVzaWMgcGlyYWN5LCB3aGljaCB3YXMgZGVzdHJv
eWluZyB0aGUgSW50ZXJuZXQmcXVvdDsuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgc3R5bGU9
Im1hcmdpbjowY207bWFyZ2luLWJvdHRvbTouMDAwMXB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXpl
OjEwLjBwdCI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgc3R5bGU9Im1hcmdpbjow
Y207bWFyZ2luLWJvdHRvbTouMDAwMXB0Ij48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdCI+
V2hlbiBDb21jYXN0IHdhcyBmb3JjZWQgdG8gYWN0dWFsbHkgbG9vayBhdCB3aHkgdGhlIHF1ZXVl
cyB3ZXJlIHN1c3RhaW5lZCBhbmQgdW5maXhhYmxlLCB0aGV5IHJlYWxpemVkIHRoYXQgdGhlIHBy
b2JsZW0gd2FzIHRoYXQgRE9DU0lTIHdhcyAqZGVzaWduZWQqIHRvIGhvbGQgcGFja2V0cyByYXRo
ZXIgdGhhbiBkcm9wDQogdGhlbS4mbmJzcDsgRUNOIG1hcmtpbmcgd291bGQgcHJvYmFibHkgbm90
IGhhdmUgaGVscGVkIGdldCB0aGUgcXVldWVzIGRvd24gdW5kZXIgNTAgbXNlYywgd2l0aG91dCBk
cm9wcGluZyBwYWNrZXRzLCBuZWl0aGVyIHdvdWxkIFJFRC4mbmJzcDsgVGhlIG9ubHkgd2F5IHRv
IGdldCB0aGUgcXVldWUgZG93biBpcyB0byBwcmV2ZW50IGl0IGZyb20gYmVpbmcgY3JlYXRlZCBp
biB0aGUgZmlyc3QgcGxhY2UuLi4uJm5ic3A7IEFuZCBzaW5jZSB0aGUgUlRUIGZyb20gVVMgZWFz
dA0KIGNvYXN0IHRvIFVTIHdlc3QgY29hc3QgaXMgdW5kZXIgNDUgbXNlYywgaXQgZG9lcyBsaXR0
bGUgZ29vZCB0byBzZW5kIGVhcmx5IGNvbmdlc3Rpb24gc2lnbmFscy48bzpwPjwvbzpwPjwvc3Bh
bj48L3A+DQo8cCBzdHlsZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFu
IHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij4mbmJzcDs8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8
cCBzdHlsZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQiPjxzcGFuIHN0eWxlPSJm
b250LXNpemU6MTAuMHB0Ij5JJ20gc3VyZSB5b3Uga25vdyB0aGlzLjxvOnA+PC9vOnA+PC9zcGFu
PjwvcD4NCjxwIHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAwMDFwdCI+PHNwYW4g
c3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPiZuYnNwOzxvOnA+PC9vOnA+PC9zcGFuPjwvcD4NCjxw
IHN0eWxlPSJtYXJnaW46MGNtO21hcmdpbi1ib3R0b206LjAwMDFwdCI+PHNwYW4gc3R5bGU9ImZv
bnQtc2l6ZToxMC4wcHQiPkFuIGFuYWxvZ291cyBwcm9ibGVtIG9jY3VycyB0b2RheSBiZWNhdXNl
IG9mIHRoZSAzLTQgKnNlY29uZCogYnVmZmVyaW5nIGluIEhTUEEgYW5kIExURSBuZXR3b3JrcyB1
bmRlciBsb2FkLiZuYnNwOyBIYXZpbmcgbWVhc3VyZWQgaXQgaW4gbWFueSBBbWVyaWNhbiBhbmQg
RXVyb3BlYW4gZGVwbG95bWVudHMgb2Ygc3VjaCBlcXVpcG1lbnQsDQogSSB3b3VsZCBzdWdnZXN0
IHRoYXQgdGhlIGZ1bmRhbWVudGFsIHByb2JsZW0gaXMgKmRyb3BwaW5nIHBhY2tldHMgZmFpcmx5
IGFuZCBxdWlja2x5Ki4mbmJzcDsgWW91IGNhbid0IGdldCB0aGVyZSAodG8gc3ViLTUwIG1zZWMu
IHF1ZXVlaW5nKSB3aXRoIEVDTiBvciBSRUQuPG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgc3R5
bGU9Im1hcmdpbjowY207bWFyZ2luLWJvdHRvbTouMDAwMXB0Ij48c3BhbiBzdHlsZT0iZm9udC1z
aXplOjEwLjBwdCI+Jm5ic3A7PG86cD48L286cD48L3NwYW4+PC9wPg0KPHAgc3R5bGU9Im1zby1t
YXJnaW4tdG9wLWFsdDowY207bWFyZ2luLXJpZ2h0OjBjbTttYXJnaW4tYm90dG9tOjEyLjBwdDtt
YXJnaW4tbGVmdDowY20iPg0KPHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQiPi0tLS0tT3Jp
Z2luYWwgTWVzc2FnZS0tLS0tPGJyPg0KRnJvbTogJnF1b3Q7Sm9obiBMZXNsaWUmcXVvdDsgJmx0
OzxhIGhyZWY9Im1haWx0bzpqb2huQGpsYy5uZXQiPmpvaG5AamxjLm5ldDwvYT4mZ3Q7PGJyPg0K
U2VudDogVHVlc2RheSwgTWFyY2ggMTksIDIwMTMgMTE6NTZwbTxicj4NClRvOiA8YSBocmVmPSJt
YWlsdG86ZHByZWVkQHJlZWQuY29tIj5kcHJlZWRAcmVlZC5jb208L2E+PGJyPg0KQ2M6IDxhIGhy
ZWY9Im1haWx0bzphcW1AaWV0Zi5vcmciPmFxbUBpZXRmLm9yZzwvYT48YnI+DQpTdWJqZWN0OiBE
cmFpbmluZyBxdWV1ZXM8bzpwPjwvbzpwPjwvc3Bhbj48L3A+DQo8ZGl2IGlkPSJTYWZlU3R5bGVz
MTM2Mzc4NjI1OCI+DQo8cCBzdHlsZT0ibWFyZ2luOjBjbTttYXJnaW4tYm90dG9tOi4wMDAxcHQi
PjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0Ij48YSBocmVmPSJtYWlsdG86ZHByZWVkQHJl
ZWQuY29tIj5kcHJlZWRAcmVlZC5jb208L2E+ICZsdDs8YSBocmVmPSJtYWlsdG86ZHByZWVkQHJl
ZWQuY29tIj5kcHJlZWRAcmVlZC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+DQomZ3Q7IDxicj4NCiZn
dDsgQSBwcm9ibGVtIHdpdGggRUNOIGlzIHRoYXQgaXQgZG9lcyBub3QgZHJhaW4gYW4gYWxyZWFk
eSBmbG9vZGVkIHF1ZXVlLDxicj4NCjxicj4NCkkgcHJlc3VtZSB5b3UgbWVhbiB0aGF0IEVDTiBk
b2Vzbid0IGRyb3AgdGhlIHBhY2tldCBpdCBtYXJrcy48YnI+DQo8YnI+DQomZ3Q7IGFuZCBkZXBl
bmRzIChsaWtlIFJFRCkgb24gdGhlIGlkZWEgdGhhdCBvbmUgaGFzIGEgbGFyZ2UgcXVldWUgaW4g
bm9ybWFsPGJyPg0KJmd0OyBvcGVyYXRpb24uPGJyPg0KPGJyPg0KSSBkb24ndCBmb2xsb3cgdGhp
cyBhdCBhbGwuIEVDTiBpcyBwb2ludGxlc3Mgd2l0aG91dCBhbiBBUU0gd2hpY2g8YnI+DQpkcm9w
cyBvciBtYXJrcyB3ZWxsIGJlZm9yZSB0YWlsLWRyb3AuIFdlcmUgd2UgdG8gcnVuIEVDTiB3aXRo
b3V0IEFRTSw8YnI+DQp3ZSdkIGhhdmUgdG8gZHJvcCBlYWNoIHBhY2tldCBhbnl3YXkgYWZ0ZXIg
bWFya2luZyBpdC48YnI+DQo8YnI+DQpTZXZlcmFsIGZsYXZvcnMgb2YgQVFNIG1hcmsvZHJvcCB3
ZWxsIGVub3VnaCBpbiBhZHZhbmNlIG9mIHRhaWwtZHJvcDxicj4NCnRoYXQgdGhlIHF1ZXVlIG5l
ZWQgbmV2ZXIgZmlsbC4gRGlkIHlvdSBtZWFuIHRvIHNheSB0aGF0IEVDTiByZXF1aXJlczxicj4N
CmJlaW5nIGFibGUgdG8gcXVldWUgdGhlIEVDTi1tYXJrZWQgcGFja2V0IHdoaWNoIHdvdWxkIG90
aGVyd2lzZSBiZTxicj4NCmRyb3BwZWQgYnkgdGhlIEFRTT88YnI+DQo8YnI+DQomZ3Q7IEl0IGlz
IG5vdCBvYnZpb3VzIHRoYXQgc3VzdGFpbmluZyBhIGRlZXAgcXVldWUgaXMgYSBkZXNpcmFibGUg
c3RhdGUsPGJyPg0KPGJyPg0KQWdyZWVkITxicj4NCjxicj4NCiZndDsgZXNwZWNpYWxseSB3aGVu
IHRoZXJlIGFyZSBoaWdobHkgdmFyaWFibGUgY2FwYWNpdGllcyBhbG9uZyBhbnkgcGF0aCw8YnI+
DQomZ3Q7IGFzIGlzIHRoZSBjYXNlIHdoZXJlIG9uZSBsaW5rIGlzIG1vcmUgdGhhbiAyeCBzbG93
ZXIgdGhhbiB0aGUgcmVzdCBvZjxicj4NCiZndDsgdGhlIGxpbmtzLjxicj4NCjxicj4NCklzbid0
IHRoYXQgX2Fsd2F5c18gdGhlIGNhc2U/PGJyPg0KPGJyPg0KJmd0OyBJZiB0aGUgZHluYW1pY3Mg
b2YgdXNlciBsb2FkcyBhcmUgZnJhY3RhbCwgdGhpcyB3aWxsIGNhdXNlIGRyYWluaW5nPGJyPg0K
Jmd0OyAoYW5kIGxhdGVuY3kgcmVkdWN0aW9uKSBub3QgdG8gd29yay48YnI+DQo8YnI+DQpJJ20g
Z3Vlc3NpbmcgeW91IG1lYW4gdGhhdCB1bmRlciAmcXVvdDtmcmFjdGFsJnF1b3Q7IGxvYWRzIHRo
ZXJlJ3MgYSB0ZW5kZW5jeTxicj4NCmZvciBFQ04tbWFya2VkIHBhY2tldHMgdG8gY2x1bXAgYW5k
IGludHJvZHVjZSBsYXRlbmN5IGF0IHRoZSBib3R0bGVuZWNrLjxicj4NClJlY2FsbCB0aGF0IHRo
aXMgaXMgb2Zmc2V0IGJ5IHRoZSBxdWlja2VyIHJlc3BvbnNlIHRvIGFuIEVDTiBtYXJrIGF0PGJy
Pg0KUlRUIHNwZWVkIGluc3RlYWQgb2YgaGF2aW5nIHRvIGluZmVyIGEgcGFja2V0IGxvc3MuPGJy
Pg0KPGJyPg0KJmd0OyBJZiB1c2VyIGxvYWRzIGFyZSBzdGF0aW9uYXJ5IGdhdXNzaWFuIGFuZCBz
byBmb3J0aCwgRUNOIGNhbiAmcXVvdDt0dW5lJnF1b3Q7PGJyPg0KJmd0OyB0byByZWxhdGl2ZWx5
IHN0YWJsZSBzaXR1YXRpb25zLCBidXQgdGhlIHF1ZXVlaW5nIGRlbGF5IHdpbGwgYmUgcXVpdGU8
YnI+DQomZ3Q7IGxvbmcgZW5kLXRvLWVuZC48YnI+DQo8YnI+DQpJIGRvbid0IGZvbGxvdy4uLiBT
b21lIEFRTXMgbWF5IGluZGVlZCBhbGxvdyBxdWV1ZWluZyBkZWxheSB0byBncm93PGJyPg0KbGFy
Z2U7IGJ1dCB0aGlzIGNhbiBvbmx5IGhhcHBlbiBpZiB0aGUgaW5zdGFudGFuZW91cyBwZXJjZW50
YWdlIG9mPGJyPg0KbWFya2VkIHBhY2tldHMgaXMgcXVpdGUgYSBiaXQgaGlnaGVyIHRoYW4gVENQ
IGNhbiB0b2xlcmF0ZS48YnI+DQo8YnI+DQomZ3Q7IFNpbmNlIHRoZXJlIGlzIGxvdHMgb2YgZXZp
ZGVuY2UgdGhhdCB1c2VyIGxvYWRzIGFyZSBmcmFjdGFsLCBkcm9wcGluZzxicj4NCiZndDsgcGFj
a2V0cyBkdXJpbmcgc3VkZGVuIGJ1cnN0cyB3aWxsIHByZXZlbnQgc3VzdGFpbmVkIHF1ZXVlaW5n
LCB3aGVyZWFzPGJyPg0KJmd0OyBFQ04gd2lsbCBub3QuPGJyPg0KPGJyPg0KSSBiZWxpZXZlIHRo
ZXJlIGlzIHN1YnN0YW50aWFsIGFncmVlbWVudCB0aGF0IEVDTi1tYXJrZWQgcGFja2V0czxicj4N
CndpbGwgbmVlZCB0byBiZSBkcm9wcGVkIGFueXdheSB1bmRlciBwYXJ0aWN1bGFybHkgc2V2ZXJl
IGNvbmdlc3Rpb24uPGJyPg0KQXQgdGhlIHZlcnkgbGVhc3QsIHRhaWwtZHJvcCBjYW4gYmVjb21l
IG5lY2Vzc2FyeS4uLjxicj4NCjxicj4NCiZndDsgU28gYSBwcm9wZXIgYXBwcm9hY2ggKGluIG15
IG9waW5pb24pIG9uIHRoZSAmcXVvdDtyZWFsJnF1b3Q7IEludGVybmV0IChyYXRoZXI8YnI+DQom
Z3Q7IHRoYW4gYSBidW5jaCBvZiBjb21wZXRpbmcgbG9uZy1kdXJhdGlvbiBGVFBzKSByZXF1aXJl
cyBrZWVwaW5nIHRoZTxicj4NCiZndDsgcXVldWVzIHNvIHNob3J0IHRoYXQgdGhlcmUgaXMgbm8g
cmVhbCBvcHBvcnR1bml0eSB0byBiZW5lZml0IGZyb20gRUNOLjxicj4NCjxicj4NCkhlcmUgSSBx
dWl0ZSBkaXNhZ3JlZS48YnI+DQo8YnI+DQpBbiBBUU0gY291bGQgYmUgZGVzaWduZWQgdG8gRUNO
LW1hcmsgYXQgYSAxIG1pbGxpc2Vjb25kIHF1ZXVlIGRlcHRoOzxicj4NCmFuZCB0aGF0IGNhbiBn
aXZlIHRoZSBmdWxsIGJlbmVmaXQgb2YgRUNOLiBBZ2FpbiwgcmVjYWxsIHRoYXQgRUNOIHdpbGw8
YnI+DQpkZWxpdmVyIHRoZSBjb25nZXN0aW9uIHNpZ25hbCBpbiBvbmUgUlRULCBpbnN0ZWFkIG9m
IHdhaXRpbmcgdG8gaW5mZXI8YnI+DQphIGxvc3MuIFRDUCBzdG9wcyBkZWxpdmVyaW5nIHVzZWZ1
bCBiYW5kd2lkdGggYXQgb25lIG9yIHR3byBwZXJjZW50PGJyPg0KcGFja2V0IGxvc3M6IHNvIGl0
J3MgaGFyZCB0byBiZWxpZXZlIHRoYXQgRUNOLW1hcmtlZCBwYWNrZXRzIHdpbGwgY2xvZzxicj4N
CnRoZSBwYXRoIHNpZ25pZmljYW50bHkuPGJyPg0KPGJyPg0KJmd0OyBIZW5jZSAtIHdlIG5lZWQg
dG8gcmVhbGx5IHNlZSB3aGF0IGhhcHBlbnMgd2l0aCAqcmVhbCogdHJhZmZpYyBsb2Fkcyw8YnI+
DQomZ3Q7IG5vdCBzaW1wbGlmaWVkIHNpbXVsYXRpb25zLjxicj4NCjxicj4NCkFncmVlZCEgKE5v
dyBpZiB3ZSBjb3VsZCBqdXN0IGdldCBlbm91Z2ggRUNOIGRlcGxveW1lbnQuLi4pPGJyPg0KPGJy
Pg0KJmd0OyBSZWFsIHRyYWZmaWMgbG9hZHMgYXJlIGZyYWN0YWwsIGFuZCBmcmFjdGFsIGxvYWRz
IGRvIG5vdCBnZW5lcmFsbHk8YnI+DQomZ3Q7IG9iZXkgdGhlIEdhdXNzaWFuIHN0eWxlICZxdW90
O2xhdyBvZiBsYXJnZSBudW1iZXJzJnF1b3Q7IC0gbXVsdGlwbGUgZnJhY3RhbDxicj4NCiZndDsg
Zmxvd3MgYXJlICpyb3VnaGVyKiBub3Qgc21vb3RoZXIgdGhhbiBhIHNpbmdsZSBvbmUuPGJyPg0K
PGJyPg0KSSB3b3VsZCBzYXkgcmF0aGVyIHRoYXQgcmVhbCB0cmFmZmljIGxvYWRzIF9jb250YWlu
XyBmcmFjdGFsIGZsb3dzLjxicj4NCkFnZ3JlZ2F0aW9uIHNpbXBseSB3b3VsZG4ndCB3b3JrIGlm
IHdlIG5ldmVyIGV4cGVyaWVuY2VkIGdhdXNzaWFuPGJyPg0Kc21vb3RoaW5nLjxicj4NCjxicj4N
CiZndDsgU28gY2hlY2sgeW91ciBtYXRoZW1hdGljYWwgJnF1b3Q7aW50dWl0aW9ucyZxdW90OyBh
dCB0aGUgZG9vci4gU3RvcCB3cml0aW5nPGJyPg0KJmd0OyBlcXVhdGlvbnMgYmFzZWQgb24gdHJh
Y3RhYmxlIGFuYWx5dGljIG1vZGVscywgYmVjYXVzZSBhY3R1YWwgdHJhZmZpYzxicj4NCiZndDsg
ZG9lcyBub3QgZm9sbG93IHRob3NlIG1vZGVscy48YnI+DQo8YnI+DQpJIHN0cm9uZ2x5IHN1c3Bl
Y3Qgb3VyIG1vZGVscyBfYXJlXyBmYXVsdHkgLS0gYnV0IHdlIG5lZWQgcmVzZWFyY2g8YnI+DQpy
ZXN1bHRzIGZvciBfYWN0dWFsXyBmbG93cywgd2l0aCBhbmQgd2l0aG91dCBFQ04sIGlzIG9yZGVy
IHRvIGRpYWdub3NlPGJyPg0KdGhlaXIgZmF1bHRzLjxicj4NCjxicj4NCi0tPGJyPg0KSm9obiBM
ZXNsaWUgJmx0OzxhIGhyZWY9Im1haWx0bzpqb2huQGpsYy5uZXQiPmpvaG5AamxjLm5ldDwvYT4m
Z3Q7PG86cD48L286cD48L3NwYW4+PC9wPg0KPC9kaXY+DQo8L2Rpdj4NCjwvZGl2Pg0KPC9ib2R5
Pg0KPC9odG1sPg0K

--_000_012C3117EDDB3C4781FD802A8C27DD4F24AC0750SACEXCMBX02PRDh_--

From touch@isi.edu  Wed Mar 20 09:42:46 2013
Return-Path: <touch@isi.edu>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9E3F421F85ED for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:42:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -104.949
X-Spam-Level: 
X-Spam-Status: No, score=-104.949 tagged_above=-999 required=5 tests=[AWL=1.423, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nXq18RIXnvDR for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:42:46 -0700 (PDT)
Received: from boreas.isi.edu (boreas.isi.edu [128.9.160.161]) by ietfa.amsl.com (Postfix) with ESMTP id 1B06C21F85D6 for <aqm@ietf.org>; Wed, 20 Mar 2013 09:42:46 -0700 (PDT)
Received: from [192.168.1.89] (pool-71-105-87-221.lsanca.dsl-w.verizon.net [71.105.87.221]) (authenticated bits=0) by boreas.isi.edu (8.13.8/8.13.8) with ESMTP id r2KGeuOt016772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 20 Mar 2013 09:41:05 -0700 (PDT)
Message-ID: <5149E699.10403@isi.edu>
Date: Wed, 20 Mar 2013 09:40:57 -0700
From: Joe Touch <touch@isi.edu>
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130307 Thunderbird/17.0.4
MIME-Version: 1.0
To: "Scheffenegger, Richard" <rs@netapp.com>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <20130320035645.GI9569@verdi> <1363787198.319422762@apps.rackspace.com> <012C3117EDDB3C4781FD802A8C27DD4F24AC0750@SACEXCMBX02-PRD.hq.netapp.com>
In-Reply-To: <012C3117EDDB3C4781FD802A8C27DD4F24AC0750@SACEXCMBX02-PRD.hq.netapp.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-ISI-4-43-8-MailScanner: Found to be clean
X-MailScanner-From: touch@isi.edu
Cc: "dpreed@reed.com" <dpreed@reed.com>, John Leslie <john@jlc.net>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:42:46 -0000

On 3/20/2013 9:31 AM, Scheffenegger, Richard wrote:
> ECN by itself is only part of the solution (and there are discussion to
> achieve more accurate ECN feedback in TCP, for a number of reasons), ECN
> can not live without a proper AQM, and a responsive (!) transport protocol

Don't forget a decent scheduler and polling-based processing too; 
latency reduction is a systems issue, not just a networking one.

Joe

From swmike@swm.pp.se  Wed Mar 20 09:55:59 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 48F5311E80A4 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:55:59 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.903
X-Spam-Level: 
X-Spam-Status: No, score=-1.903 tagged_above=-999 required=5 tests=[AWL=0.096,  BAYES_00=-2.599, J_CHICKENPOX_38=0.6]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DMOjt+jmAaOR for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 09:55:58 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id B1C9711E80A2 for <aqm@ietf.org>; Wed, 20 Mar 2013 09:55:58 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 0B8E99C; Wed, 20 Mar 2013 17:55:58 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 00BAF9A for <aqm@ietf.org>; Wed, 20 Mar 2013 17:55:57 +0100 (CET)
Date: Wed, 20 Mar 2013 17:55:57 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: aqm@ietf.org
Message-ID: <alpine.DEB.2.00.1303201746360.2309@uplift.swm.pp.se>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Subject: [aqm] ingress shaping
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 16:55:59 -0000

Hello.

I'd just like to bring attention to another deployment scenario where 
ingress queuing might make a lot of sense:

ISP deploys ETTH on a 100/100 port (100 megabit/s ethernet full duplex). 
Service is 100/10 (100 megabit/s towards the customer (downstream), 10 
megabit/s customer->Internet (upstream)).

ISP solves the traffic problem by putting in a policer upstream that has a 
2 second burst size, and polices at 12 megabit/s. This means TCP will be 
sawtoothing, but it will pass the "speedtest" websites, because effective 
throughput will be ~10 megabit/s upstream.

Now, I feel the customer is better served if there was some 
shaping/buffering instead of this policer (policer is defined as something 
that has no buffer but instead just drops packet that is over bytes/s 
averaged over a certain time). Since the first real buffering point in 
this solution is the customer CPE, to achieve buffering we need to do this 
ingress towards the CPE.

Downstream device layout:

Internet -> ISP -> DR -> AS1 -> AS2 -> AS3 -> CPE

AS = Access switch
DR = Distribution Router
CPE=Customer Premice Equipment

Policing is done egress on AS3. To smoothe out the flows, CPE needs to do 
ingress shaping to ~10 megabit/s.

Would it make sense to apply fq_codel with an artificial bw shaper ingress 
on the CPE? Since the fq_codel would limit queuing to ~5ms and smoothe out 
flows, I don't see how the 12 megabit/s policer at AS3 egress would even 
be hit (unless it's very bursty flows)?

If there is any AQM document that one can do procurement decisions 
against, I'd like to have this deployment scenario included anyway, so 
that device vendors include possibility of ingress shaping as well.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From ietf-secretariat@ietf.org  Wed Mar 20 12:41:58 2013
Return-Path: <ietf-secretariat@ietf.org>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 29F8B11E80E8; Wed, 20 Mar 2013 12:41:54 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.487
X-Spam-Level: 
X-Spam-Status: No, score=-102.487 tagged_above=-999 required=5 tests=[AWL=-0.113, BAYES_00=-2.599, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w0LCHoqfoYNc; Wed, 20 Mar 2013 12:41:53 -0700 (PDT)
Received: from ietfa.amsl.com (localhost [IPv6:::1]) by ietfa.amsl.com (Postfix) with ESMTP id 8B0E011E80CC; Wed, 20 Mar 2013 12:41:53 -0700 (PDT)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: IETF Secretariat <ietf-secretariat@ietf.org>
To: IETF Announcement List <ietf-announce@ietf.org>
X-Test-IDTracker: no
X-IETF-IDTracker: 4.43
Message-ID: <20130320194153.2133.60344.idtracker@ietfa.amsl.com>
Date: Wed, 20 Mar 2013 12:41:53 -0700
Cc: Wes Eddy <wes@mti-systems.com>, aqm@ietf.org, Martin Stiemerling <Martin.Stiemerling@neclab.eu>
Subject: [aqm] New Non-WG Mailing List: aqm -- Active Queue Management
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 19:41:58 -0000

A new IETF non-working group email list has been created.

List address: aqm@ietf.org
Archive: http://www.ietf.org/mail-archive/web/aqm/current/maillist.html
To subscribe: https://www.ietf.org/mailman/listinfo/aqm

Purpose: This list is for discussing requirements, recommendations, algorit=
hms, evaluation techniques, and other topics related to active queue
management algorithms and methods for protecting flows from one
another at a bottleneck.

For additional information, please contact the list administrators.


From dpreed@reed.com  Wed Mar 20 14:26:13 2013
Return-Path: <dpreed@reed.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BB6B711E80F4 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 14:26:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.428
X-Spam-Level: 
X-Spam-Status: No, score=-3.428 tagged_above=-999 required=5 tests=[AWL=-0.057, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nUQGwkumf4e5 for <aqm@ietfa.amsl.com>; Wed, 20 Mar 2013 14:26:13 -0700 (PDT)
Received: from smtp176.iad.emailsrvr.com (smtp176.iad.emailsrvr.com [207.97.245.176]) by ietfa.amsl.com (Postfix) with ESMTP id 0536311E80F2 for <aqm@ietf.org>; Wed, 20 Mar 2013 14:26:12 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp57.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 6EDC5F8240; Wed, 20 Mar 2013 17:26:12 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy16.wa-web.iad1a (legacy16.wa-web.iad1a.rsapps.net [192.168.4.106]) by smtp57.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 4D5CAF8228; Wed, 20 Mar 2013 17:26:12 -0400 (EDT)
Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy16.wa-web.iad1a (Postfix) with ESMTP id 3F3BE2450001; Wed, 20 Mar 2013 17:26:12 -0400 (EDT)
Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com)  with HTTP; Wed, 20 Mar 2013 17:26:12 -0400 (EDT)
Date: Wed, 20 Mar 2013 17:26:12 -0400 (EDT)
From: dpreed@reed.com
To: "Joe Touch" <touch@isi.edu>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_20130320172612000000_97872"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
In-Reply-To: <5149E699.10403@isi.edu>
References: <mailman.5225.1363736608.3432.aqm@ietf.org>  <1363748540.64155384@apps.rackspace.com> <20130320035645.GI9569@verdi>  <1363787198.319422762@apps.rackspace.com>  <012C3117EDDB3C4781FD802A8C27DD4F24AC0750@SACEXCMBX02-PRD.hq.netapp.com> <5149E699.10403@isi.edu>
Message-ID: <1363814772.257213885@apps.rackspace.com>
X-Mailer: webmail7.0
Cc: "Scheffenegger, Richard" <rs@netapp.com>, John Leslie <john@jlc.net>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] Draining queues
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Mar 2013 21:26:13 -0000

------=_20130320172612000000_97872
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0AJoe's right, at least historically: Once upon a time, one of the problem=
s the "receive window" was thought to solve was operating systems that had =
very, very long response times and expensive process switching, so there wa=
s some benefit for the endpoints in "batching" up lots of data before sendi=
ng and before receiving.=0A =0AOne way to think of this is that the bulk of=
 the queueing delay on an end-to-end basis was spent with the packets sitti=
ng in operating system buffers, so one could optimize cost of process switc=
hing and/or deal with time quanta on the order of 50-100 msec. in the endpo=
int schedulers.  It was especially true in the days when the dominant conte=
nt was "character-at-a-time" remote logins and the user could type much fas=
ter than the remote system could echo.=0A =0ASuch systems may still exist -=
 multiprocessing systems where processors are kind of slow, memory is relat=
ively free, etc.=0A =0AI've never been sure that making TCP's design includ=
e these endpoint-buffering features was a good idea, since it encouraged ma=
naging the end-to-end flow with an incredibly slow and variable control res=
ponse.=0A =0APut this kind of buffering in the layer above TCP, at least in=
 a logical sense, by not counting these buffers as part of the "window".  T=
hen you never end up letting the TCP window get big just because the recipi=
ent needs buffering to cover for its weak process scheduler. It's a *transm=
ission* control protocol, not an Operating Systems workload scheduling prot=
ocol.=0A =0A-----Original Message-----=0AFrom: "Joe Touch" <touch@isi.edu>=
=0ASent: Wednesday, March 20, 2013 12:40pm=0ATo: "Scheffenegger, Richard" <=
rs@netapp.com>=0ACc: "dpreed@reed.com" <dpreed@reed.com>, "John Leslie" <jo=
hn@jlc.net>, "aqm@ietf.org" <aqm@ietf.org>=0ASubject: Re: [aqm] Draining qu=
eues=0A=0A=0A=0A=0AOn 3/20/2013 9:31 AM, Scheffenegger, Richard wrote:=0A> =
ECN by itself is only part of the solution (and there are discussion to=0A>=
 achieve more accurate ECN feedback in TCP, for a number of reasons), ECN=
=0A> can not live without a proper AQM, and a responsive (!) transport prot=
ocol=0A=0ADon't forget a decent scheduler and polling-based processing too;=
 =0Alatency reduction is a systems issue, not just a networking one.=0A=0AJ=
oe
------=_20130320172612000000_97872
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"times new roman" size=3D"2"><p style=3D"margin:0;padding:0;">=
Joe's right, at least historically: Once upon a time, one of the problems t=
he "receive window" was thought to solve was operating systems that had ver=
y, very long response times and expensive process switching, so there was s=
ome benefit for the endpoints in "batching" up lots of data before sending =
and before receiving.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<=
p style=3D"margin:0;padding:0;">One way to think of this is that the bulk o=
f the queueing delay on an end-to-end basis was spent with the packets sitt=
ing in operating system buffers, so one could optimize cost of process swit=
ching and/or deal with time quanta on the order of 50-100 msec. in the endp=
oint schedulers.&nbsp; It was especially true in the days when the dominant=
 content was "character-at-a-time" remote logins and the user could type mu=
ch faster than the remote system could echo.</p>=0A<p style=3D"margin:0;pad=
ding:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">Such systems may sti=
ll exist - multiprocessing systems where processors are kind of slow, memor=
y is relatively free, etc.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p=
>=0A<p style=3D"margin:0;padding:0;">I've never been sure that making TCP's=
 design include these endpoint-buffering features was a good idea, since it=
 encouraged managing the end-to-end flow with an incredibly slow and variab=
le control response.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p=
 style=3D"margin:0;padding:0;">Put this kind of buffering in the layer abov=
e TCP, at least in a logical sense, by not counting these buffers as part o=
f the "window".&nbsp; Then you never end up letting the TCP window get big =
just because the recipient needs buffering to cover for its weak process sc=
heduler. It's a *transmission* control protocol, not an Operating Systems w=
orkload scheduling protocol.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;<=
/p>=0A<p style=3D"margin:0;padding:0;">-----Original Message-----<br />From=
: "Joe Touch" &lt;touch@isi.edu&gt;<br />Sent: Wednesday, March 20, 2013 12=
:40pm<br />To: "Scheffenegger, Richard" &lt;rs@netapp.com&gt;<br />Cc: "dpr=
eed@reed.com" &lt;dpreed@reed.com&gt;, "John Leslie" &lt;john@jlc.net&gt;, =
"aqm@ietf.org" &lt;aqm@ietf.org&gt;<br />Subject: Re: [aqm] Draining queues=
<br /><br /></p>=0A<div id=3D"SafeStyles1363814056"><br /><br />On 3/20/201=
3 9:31 AM, Scheffenegger, Richard wrote:<br />&gt; ECN by itself is only pa=
rt of the solution (and there are discussion to<br />&gt; achieve more accu=
rate ECN feedback in TCP, for a number of reasons), ECN<br />&gt; can not l=
ive without a proper AQM, and a responsive (!) transport protocol<br /><br =
/>Don't forget a decent scheduler and polling-based processing too; <br />l=
atency reduction is a systems issue, not just a networking one.<br /><br />=
Joe</div></font>
------=_20130320172612000000_97872--


From rs@netapp.com  Thu Mar 21 08:29:19 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5FF8121F90F5; Thu, 21 Mar 2013 08:29:19 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.22
X-Spam-Level: 
X-Spam-Status: No, score=-10.22 tagged_above=-999 required=5 tests=[AWL=0.079,  BAYES_00=-2.599, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WQU00UssfpT5; Thu, 21 Mar 2013 08:29:18 -0700 (PDT)
Received: from mx2.netapp.com (mx2.netapp.com [216.240.18.37]) by ietfa.amsl.com (Postfix) with ESMTP id 67EE521F9052; Thu, 21 Mar 2013 08:29:18 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,887,1355126400"; d="scan'208";a="16004176"
Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx2-out.netapp.com with ESMTP; 21 Mar 2013 08:29:18 -0700
Received: from vmwexceht01-prd.hq.netapp.com (exchsmtp.hq.netapp.com [10.106.76.239]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2LFTHGr006912; Thu, 21 Mar 2013 08:29:17 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht01-prd.hq.netapp.com ([10.106.76.239]) with mapi id 14.02.0342.003; Thu, 21 Mar 2013 08:29:16 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: =?utf-8?B?5aSp5r6c?= <tianlan.lhh@alibaba-inc.com>, Emmanuel Lochin <emmanuel.lochin@isae.fr>, Yuchung Cheng <ycheng@google.com>
Thread-Topic: [iccrg] [tcpm] ECN support and usage on the Internet
Thread-Index: AQHOJN7vLn8i1zQNs0aR6qujDYSanZiu5guA///XOCCAAW4vAIAAGwOw
Date: Thu, 21 Mar 2013 15:29:16 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24AC54A0@SACEXCMBX02-PRD.hq.netapp.com>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <CAK6E8=dxZbT92K1mgg_E5zo8UQ1HhZe77c8SD8tqu0FXeLzYJw@mail.gmail.com> <51499BFB.7050202@isae.fr>, <012C3117EDDB3C4781FD802A8C27DD4F24AC06C0@SACEXCMBX02-PRD.hq.netapp.com> <B2CE202A35DFDD4A904DA7497D7D2B9830FA99@CNHZ-EXMAIL-10.ali.com>
In-Reply-To: <B2CE202A35DFDD4A904DA7497D7D2B9830FA99@CNHZ-EXMAIL-10.ali.com>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Cc: "tcpm@ietf.org Extensions" <tcpm@ietf.org>, "aqm@ietf.org" <aqm@ietf.org>, "iccrg@irtf.org" <iccrg@irtf.org>
Subject: Re: [aqm] [iccrg] [tcpm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Mar 2013 15:29:19 -0000

SGksDQoNCkknbSBhZnJhaWQgSSB3YXMgbm90IGFibGUgdG8gcGFyc2UgeW91ciBzdGF0ZW1lbnQu
DQoNCk15IGNvbW1lbnQgd2FzIG5vdCBhYm91dCByZWR1Y2luZyB0aGUgUlRPIHRpbWUgKCEpLiBP
biB0aGUgY29udHJhcnksIEkgd2FzIHBvaW50aW5nIG91dCwgdGhhdCB3aXRoIEVDTiAoYW5kIHdp
dGhvdXQgZ2VudWluZSBsb3NzIGR1ZSB0byBjb3JydXB0aW9uKSwgUlRPcyAtIHJlZ2FyZGxlc3Mg
b2YgdGhlIGFjdHVhbCB0aW1lciB2YWx1ZSBhcyBsb25nIGFzIHRoYXQgaXMgcmVhc29uYWJseSBs
YXJnZXIgdGhhbiBSVFQgLSBzaG91bGQgYmVjb21lIGV4Y2VlZGluZ2x5IHJhcmUsIG9uY2UgRUNO
IGlzIGZ1bGx5IGRlcGxveWVkLg0KDQoNCk9uY2UgUlRPIGlzIG5vIGxvbmdlciBuZWNlc3Nhcnkg
dG8gcmVjb3ZlciB+NzAlIG9mIGFsbCBsb3N0IFRDUCBzZWdtZW50cyAoZ29vZ2xlIHdlYnNlcnZl
ciBkYXRhKSwgeW91IGNhbiBhY3R1YWxseSAqaW5jcmVhc2UqIHRoZSBtaW5pbXVtIFJUTyB0aW1l
b3V0IHZhbHVlIHNpbmNlIGl0IHJlYWxseSBpcyBhIGxhc3QgcmVzb3J0LiAoVXNpbmcgYSAibGFz
dCByZXNvcnQiIHJlY292ZXJ5IG1lY2hhbmlzbSBmb3IgNzAlIG9mIHJlY292ZXJpZXMgaXMgcmVh
bGx5IE5PVCAibGFzdCByZXNvcnQiKS4NCg0KDQpCZXN0IHJlZ2FyZHMsDQoNClJpY2hhcmQgU2No
ZWZmZW5lZ2dlcg0KDQoNCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiDl
pKnmvpwgW21haWx0bzp0aWFubGFuLmxoaEBhbGliYWJhLWluYy5jb21dDQo+IFNlbnQ6IERvbm5l
cnN0YWcsIDIxLiBNw6RyeiAyMDEzIDA3OjQ3DQo+IFRvOiBTY2hlZmZlbmVnZ2VyLCBSaWNoYXJk
OyBFbW1hbnVlbCBMb2NoaW47IFl1Y2h1bmcgQ2hlbmcNCj4gQ2M6IHRjcG1AaWV0Zi5vcmcgRXh0
ZW5zaW9uczsgYXFtQGlldGYub3JnOyBpY2NyZ0BpcnRmLm9yZw0KPiBTdWJqZWN0OiDnrZTlpI06
IFtpY2NyZ10gW3RjcG1dIEVDTiBzdXBwb3J0IGFuZCB1c2FnZSBvbiB0aGUgSW50ZXJuZXQNCj4g
DQo+IGFib3V0IFJUTyAsYWNjcm9kaW5nIHRvIG91dCBleHBlcmllbmNlICx1c2Ugc21hbGwgcHJv
YmUgcGFja2V0IGF0IGxlYXN0DQo+IHJlZHVjZSA5MCUgUlRPIHRpbWXvvIhTaW5jZSAyMDEwLG5v
dyBldmVyeSBkYXkgcGVlayBmbG9lIG1vcmUgdGhhbiAxVGJwcyDvvIkNCj4gOyBDb25zaWRlciB0
aGUgbW9iaWxlIG5ldHdvcmsgRXhjZXNzaXZlIHJlZHVjZSBSVE8gdGltZSBpcyB1bnJlYXNvbmFi
bGU7DQo+IA0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IOWP
keS7tuS6ujogU2NoZWZmZW5lZ2dlciwgUmljaGFyZCBbcnNAbmV0YXBwLmNvbV0NCj4g5Y+R6YCB
5pe26Ze0OiAyMDEz5bm0M+aciDIx5pelIDA6MDMNCj4g5YiwOiBFbW1hbnVlbCBMb2NoaW47IFl1
Y2h1bmcgQ2hlbmcNCj4gQ2M6IHRjcG1AaWV0Zi5vcmcgRXh0ZW5zaW9uczsgYXFtQGlldGYub3Jn
OyBpY2NyZ0BpcnRmLm9yZw0KPiDkuLvpopg6IFJlOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9y
dCBhbmQgdXNhZ2Ugb24gdGhlIEludGVybmV0DQo+IA0KPiBUaGFua3MhDQo+IA0KPiBXaGljaCBm
b3J0dW5hdGVseSBpcyBhbHJlYWR5IGFuIGV4cGVyaW1lbnRhbA0KPiBodHRwOi8vdG9vbHMuaWV0
Zi5vcmcvaHRtbC9yZmM1NTYyDQo+IA0KPiBCdXQgdG9vIGxpdHRsZSBleHBlcmllbmNlIHRvIGZv
bGxvdyB1cCBvbiB0aGF0IGp1c3QgeWV0OyBMYXN0IHRpbWUgSQ0KPiBsb29rZWQsIEkgYmVsaWV2
ZSBpdCB3YXMgdGhlIEFJWCA1LjMgVENQIHN0YWNrIHdvdWxkIGRlZmF1bHQgdG8gNTU2Miwgd2hl
bg0KPiBFQ04gaXMgZW5hYmxlZCAtIGJ1dCBFQ04gaXMgb2ZmIGJ5IGRlZmF1bHQgaW4gdGhhdCBP
Uy4uLg0KPiANCj4gDQo+IEJ1dCB0aGUgZmluZGluZ3MgaW4gdGhhdCBwYXBlciBkb24ndCBjb21w
YXJlIFRhaWwtRHJvcCBub24tRUNOIHdpdGgNCj4gQVFNL0VDTiosIHdoaWNoIGlzIHByb2JhYmx5
IHdoYXQgeW91IGFyZSBhZnRlciBZdWNodW5nPw0KPiANCj4gDQo+IFB1dCBkaWZmZXJlbnRseTog
V2l0aCBhIGxhcmdlIGVub3VnaCBmcmFjdGlvbiBvZiBBUU0gZGVwbG95bWVudCwgYQ0KPiByZWFz
b25hYmxlIG51bWJlciBvZiBSVE9zIHNob3VsZCBiZSBhdm9pZGFibGUgKGxlc3MgcHJvYmFiaWxp
dHkgdGhhdCB0aGUNCj4gbGFzdCBzZWdtZW50IGluIGEgc2Vzc2lvbiBpcyBkcm9wcGVkKTsgd2l0
aCB0aGUgYWRkaXRpb24gb2YgRUNOLCBldmVuIHRoYXQNCj4gbG9zcyBzaG91bGQgZ28gYXdheS4u
Lg0KPiANCj4gDQo+IERvIHlvdSBrbm93IHdoYXQgdGhlIENDREYgZm9yIG9iamVjdCBsb2FkIHRp
bWVzIHdvdWxkIGxvb2sgbGlrZSwgaWYgdGhlcmUNCj4gd2VyZSB2aXJ0dWFsbHkgbm8gUlRPcyBh
bnltb3JlPw0KPiANCj4gDQo+IEFwcGFyZW50bHksIHRoZSBpbmNlbnRpdmUgdG8gYXZvaWQgKGNv
c3RseSkgUlRPcyBpcyB0b28gc21hbGwgdG8gZHJpdmUgRUNODQo+IGRlcGxveW1lbnQsIGV2ZW4g
dGhvdWdoIHRoYXQgc2hvdWxkIGJlIHNvbWV0aGluZyBlYXN5IHRvIGFjaGlldmUgd2l0aCBFQ04N
Cj4gb24gVENQLg0KPiANCj4gDQo+IFJpY2hhcmQgU2NoZWZmZW5lZ2dlcg0KPiANCj4gDQo+IA0K
PiA+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+ID4gRnJvbTogdGNwbS1ib3VuY2VzQGll
dGYub3JnIFttYWlsdG86dGNwbS1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYNCj4gPiBPZiBF
bW1hbnVlbCBMb2NoaW4NCj4gPiBTZW50OiBNaXR0d29jaCwgMjAuIE3DpHJ6IDIwMTMgMTI6MjMN
Cj4gPiBUbzogWXVjaHVuZyBDaGVuZw0KPiA+IENjOiB0Y3BtQGlldGYub3JnIEV4dGVuc2lvbnM7
IGFxbUBpZXRmLm9yZzsgaWNjcmdAaXJ0Zi5vcmcNCj4gPiBTdWJqZWN0OiBSZTogW3RjcG1dIEVD
TiBzdXBwb3J0IGFuZCB1c2FnZSBvbiB0aGUgSW50ZXJuZXQNCj4gPg0KPiA+IE9uIDE5LzAzLzIw
MTMgMjE6MTcsIFl1Y2h1bmcgQ2hlbmcgd3JvdGU6DQo+ID4gPiBEbyBwZW9wbGUgb24gdGhlIGxp
c3Qga25vdyBhbnkgKGdvb2QpIGV2YWx1YXRpb24gb2YgcGVyZm9ybWFuY2UNCj4gPiA+IGltcGFj
dCBvZiBFQ04gb24gcmVhbCBhbmQvb3IgZW11bGF0ZWQgbmV0d29ya3MuIEUuZy4sIGhvdyBtdWNo
DQo+ID4gPiBmYXN0ZXIgd2lsbCB0aGUgV2ViIGJlIGlmIEVDTiBpcyB1c2VkIChjb3JyZWN0bHkp
Lg0KPiA+DQo+ID4gVGhlcmUgaXMgOiBodHRwOi8vZGwuYWNtLm9yZy9jaXRhdGlvbi5jZm0/aWQ9
MTA4MDA5MS4xMDgwMTAwDQo+ID4NCj4gPiBFTA0KPiA+DQo+ID4gLS0NCj4gPiBFbW1hbnVlbCBM
b2NoaW4NCj4gPiBQcm9mZXNzZXVyIElTQUUgLSBPU1NJDQo+ID4gSW5zdGl0dXQgU3Vww6lyaWV1
ciBkZSBsJ0HDqXJvbmF1dGlxdWUgZXQgZGUgbCdFc3BhY2UgKElTQUUpIElzc3UgZHUNCj4gPiBy
YXBwcm9jaGVtZW50IFNVUEFFUk8gZXQgRU5TSUNBDQo+ID4gMTAgYXZlbnVlIEVkb3VhcmQgQmVs
aW4gLSBCUCA1NDAzMiAtIDMxMDU1IFRvdWxvdXNlIGNlZGV4IDQgVGVsIDogMDUNCj4gPiA2MSAz
Mw0KPiA+IDkxIDg1IC0gRmF4IDogMDUgNjEgMzMgOTEgODggV2ViIDogaHR0cDovL3BlcnNvbm5l
bC5pc2FlLmZyL2VtbWFudWVsLQ0KPiA+IGxvY2hpbi8NCj4gPiAtLS0NCj4gPiAiVGhpcyBlbWFp
bCBhbmQgYW55IGF0dGFjaG1lbnRzIGFyZSBjb25maWRlbnRpYWwuIFRoZXkgbWF5IGNvbnRhaW4N
Cj4gPiBsZWdhbGx5IHByaXZpbGVnZWQgaW5mb3JtYXRpb24gb3IgY29weXJpZ2h0IG1hdGVyaWFs
LiBZb3Ugc2hvdWxkIG5vdA0KPiA+IHJlYWQsIGNvcHksIHVzZSBvciBkaXNjbG9zZSB0aGVtIHdp
dGhvdXQgYXV0aG9yaXNhdGlvbi4gSWYgeW91IGFyZSBub3QNCj4gPiBhbiBpbnRlbmRlZCByZWNp
cGllbnQsIHBsZWFzZSBjb250YWN0IHVzIGF0IG9uY2UgYnkgcmV0dXJuIGVtYWlsIGFuZA0KPiA+
IHRoZW4gZGVsZXRlIGJvdGggbWVzc2FnZXMuIFdlIGRvIG5vdCBhY2NlcHQgbGlhYmlsaXR5IGlu
IGNvbm5lY3Rpb24NCj4gPiB3aXRoIGNvbXB1dGVyIHZpcnVzLCBkYXRhIGNvcnJ1cHRpb24sIGRl
bGF5LCBpbnRlcnJ1cHRpb24sDQo+ID4gdW5hdXRob3Jpc2VkIGFjY2VzcyBvciB1bmF1dGhvcmlz
ZWQgYW1lbmRtZW50LiBUaGlzIG5vdGljZSBzaG91bGQgbm90IGJlDQo+IHJlbW92ZWQiDQo+ID4N
Cj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0KPiA+
IHRjcG0gbWFpbGluZyBsaXN0DQo+ID4gdGNwbUBpZXRmLm9yZw0KPiA+IGh0dHBzOi8vd3d3Lmll
dGYub3JnL21haWxtYW4vbGlzdGluZm8vdGNwbQ0KPiANCj4gDQo+IF9fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fDQo+IA0KPiBUaGlzIGVtYWlsIChpbmNsdWRpbmcgYW55IGF0dGFjaG1l
bnRzKSBpcyBjb25maWRlbnRpYWwgYW5kIG1heSBiZSBsZWdhbGx5DQo+IHByaXZpbGVnZWQuIElm
IHlvdSByZWNlaXZlZCB0aGlzIGVtYWlsIGluIGVycm9yLCBwbGVhc2UgZGVsZXRlIGl0DQo+IGlt
bWVkaWF0ZWx5IGFuZCBkbyBub3QgY29weSBpdCBvciB1c2UgaXQgZm9yIGFueSBwdXJwb3NlIG9y
IGRpc2Nsb3NlIGl0cw0KPiBjb250ZW50cyB0byBhbnkgb3RoZXIgcGVyc29uLiBUaGFuayB5b3Uu
DQo+IA0KPiDmnKznlLXpgq4o5YyF5ous5Lu75L2V6ZmE5Lu2KeWPr+iDveWQq+acieacuuWvhui1
hOaWmeW5tuWPl+azleW+i+S/neaKpOOAguWmguaCqOS4jeaYr+ato+ehrueahOaUtuS7tuS6uu+8
jOivt+aCqOeri+WNs+WIoOmZpA0KPiDmnKzpgq7ku7bjgILor7fkuI3opoHlsIbmnKznlLXpgq7o
v5vooYzlpI3liLblubbnlKjkvZzku7vkvZXlhbbku5bnlKjpgJTjgIHmiJbpgI/pnLLmnKzpgq7k
u7bkuYvlhoXlrrnjgILosKLosKLjgIINCg==

From g.white@CableLabs.com  Thu Mar 21 09:59:19 2013
Return-Path: <g.white@CableLabs.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A1A9E21F9178; Thu, 21 Mar 2013 09:59:19 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.463
X-Spam-Level: 
X-Spam-Status: No, score=-0.463 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_MODEMCABLE=0.768, HOST_EQ_MODEMCABLE=1.368]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VrBmo6mdY6Lf; Thu, 21 Mar 2013 09:59:19 -0700 (PDT)
Received: from ondar.cablelabs.com (ondar.cablelabs.com [192.160.73.61]) by ietfa.amsl.com (Postfix) with ESMTP id E6C9721F8F44; Thu, 21 Mar 2013 09:59:15 -0700 (PDT)
Received: from kyzyl.cablelabs.com (kyzyl [10.253.0.7]) by ondar.cablelabs.com (8.14.5/8.14.5) with ESMTP id r2LGx4bn003862; Thu, 21 Mar 2013 10:59:04 -0600
Received: from exchange.cablelabs.com (10.5.0.19) by kyzyl.cablelabs.com (F-Secure/fsigk_smtp/407/kyzyl.cablelabs.com); Thu, 21 Mar 2013 10:59:04 -0600 (MDT)
X-Virus-Status: clean(F-Secure/fsigk_smtp/407/kyzyl.cablelabs.com)
Received: from EXCHANGE.cablelabs.com ([fe80::797a:96d1:3c53:18ee]) by EXCHANGE.cablelabs.com ([fe80::797a:96d1:3c53:18ee%11]) with mapi id 14.02.0328.009; Thu, 21 Mar 2013 10:59:03 -0600
From: Greg White <g.white@CableLabs.com>
To: Graham Beneke <graham@apolix.co.za>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Thread-Topic: [tsvwg] how much of a problem is buffer bloat today?
Thread-Index: AQHOH/Zy6Y2ujr6xyky0EcKlL8uw15ikI2qA///Q/gCAAFGcgIAMepqA//+qP4A=
Date: Thu, 21 Mar 2013 16:59:03 +0000
Message-ID: <C38E406F654CD249AB7D30B93B0207F517F5E4AE@EXCHANGE.cablelabs.com>
In-Reply-To: <514B2FE9.90101@apolix.co.za>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
user-agent: Microsoft-MacOutlook/14.3.1.130117
x-originating-ip: [10.4.1.127]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <88FA79DEB375EF42B2E096171593333A@cablelabs.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Approved: ondar
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Mar 2013 16:59:19 -0000

....cross posting to aqm.

On 3/21/13 10:06 AM, "Graham Beneke" <graham@apolix.co.za> wrote:

>On 13/03/2013 19:32, Mikael Abrahamsson wrote:
>> On Wed, 13 Mar 2013, Greg White wrote:
>>=20
>>> Some may consider 100ms a big number.
>
>I've heard the 100ms figure thrown about quite a bit. In the VoIP world
>its considered to be the pain threshold for a reasonable call experience
>and for "general Internet" (aka http browsing) it doesn't normally
>impact the user experience notably.

Web browsing *is* sensitive to RTT (though maybe less so than VoIP or
gaming).  Page load time increases linearly with RTT (at approximately
10x-20x).  So adding 100ms of *unnecessary* buffering latency adds ~1-2
seconds of *unnecessary* delay to page loads.=20


From gettysjim@gmail.com  Thu Mar 21 10:20:22 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4350121F9180; Thu, 21 Mar 2013 10:20:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.54
X-Spam-Level: 
X-Spam-Status: No, score=-1.54 tagged_above=-999 required=5 tests=[AWL=1.209,  BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DI-mCHL8UQsc; Thu, 21 Mar 2013 10:20:21 -0700 (PDT)
Received: from mail-oa0-f50.google.com (mail-oa0-f50.google.com [209.85.219.50]) by ietfa.amsl.com (Postfix) with ESMTP id 3BD2E21F90DD; Thu, 21 Mar 2013 10:20:21 -0700 (PDT)
Received: by mail-oa0-f50.google.com with SMTP id n1so562957oag.9 for <multiple recipients>; Thu, 21 Mar 2013 10:20:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=UwJEIfs/TZlZs6Qfiok8UOccboC4kGhAyF0WB+kR4iE=; b=FbQ+ssP/RZW6jbLphQcJ16uVB3NuP62JyXGbLzsQkC7DIZCAMlmxk0/6BVaqWmqVQV yKV6SksQX0jVV65wtlRFqi5VC9Rt9tiY1NIIBbKMoTv27CTvotNGXnl2wI/XJixVn1oO FUXduYPw75jCUGBSLjlWLgcu5U59zkcIId+aI/vtNt+B7MBm1faWq/TjDgrqtlSM7dy+ zgd9/5JzHwh99XdXwrjxd7vMhv3/ugD8wXSxRc2t9QhUzv8HK0wl/kchA7/v5ttEOIJL j4NJXvV4v5H6BaHWiixzrvKD533YA62JhUPkE/dLiRM6myYlVK/mZW6vMpcq6lesBDmu 3Mhg==
MIME-Version: 1.0
X-Received: by 10.60.10.138 with SMTP id i10mr7495696oeb.95.1363886420730; Thu, 21 Mar 2013 10:20:20 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Thu, 21 Mar 2013 10:20:20 -0700 (PDT)
In-Reply-To: <C38E406F654CD249AB7D30B93B0207F517F5E4AE@EXCHANGE.cablelabs.com>
References: <514B2FE9.90101@apolix.co.za> <C38E406F654CD249AB7D30B93B0207F517F5E4AE@EXCHANGE.cablelabs.com>
Date: Thu, 21 Mar 2013 13:20:20 -0400
X-Google-Sender-Auth: fGLBIzDMN2uNCb2X_0wOjtY3WLU
Message-ID: <CAGhGL2Bzpmrq6wQ2iodtYjd-_S3zMBKaf0qGRo=eYs-Bj+Vfag@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Greg White <g.white@cablelabs.com>
Content-Type: multipart/alternative; boundary=e89a8f5036fc9f2b7004d8728d59
Cc: Graham Beneke <graham@apolix.co.za>, "tsvwg@ietf.org" <tsvwg@ietf.org>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Mar 2013 17:20:22 -0000

--e89a8f5036fc9f2b7004d8728d59
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Mar 21, 2013 at 12:59 PM, Greg White <g.white@cablelabs.com> wrote:

> ....cross posting to aqm.
>
> On 3/21/13 10:06 AM, "Graham Beneke" <graham@apolix.co.za> wrote:
>
> >On 13/03/2013 19:32, Mikael Abrahamsson wrote:
> >> On Wed, 13 Mar 2013, Greg White wrote:
> >>
> >>> Some may consider 100ms a big number.
> >
> >I've heard the 100ms figure thrown about quite a bit. In the VoIP world
> >its considered to be the pain threshold for a reasonable call experience
> >and for "general Internet" (aka http browsing) it doesn't normally
> >impact the user experience notably.
>
> Web browsing *is* sensitive to RTT (though maybe less so than VoIP or
> gaming).  Page load time increases linearly with RTT (at approximately
> 10x-20x).  So adding 100ms of *unnecessary* buffering latency adds ~1-2
> seconds of *unnecessary* delay to page loads.
>

This is one very visible result; but far from the most important, as we
work on trying to make applications like WebRTC viable.

One way I think about latency is that it sets the "diameter" in which
particular applications are viable, as well as causing performance problems
in general.  These are set by human factors and the speed of our neurology.

Variance of latency is just about as bad as the latency itself; you have to
add that to jitter buffers and it increases the latency the system can be
used at.

And you have to think of latency in terms of a budget: it takes of order
75ms to get across the United States by a good unloaded Internet connection
(the same as it did in the 1990's; Einstein + the index of refraction of
glass  makes this hard to change).  So you really have much, much, less
than 100ms you can afford to burn in the edge of the network even by this
metric.

For example, human perception to make interactive "rubber
banding" imperceptible is in the 20-30ms range.  As the latency budget
includes screen refresh time, this one is doable, but you have to work
hard, as we did with the X Window System in the 1980's.  I find it
personally very sad that while X11 used to be usable between continents,
it's not typically usable today (though mistakes we made in the X11
protocol made this worse than it should have been).

If you want to play music with a friend, professional musicians might
tolerate up to 100ms total; but duffers can't tolerate even half of that
latency well.  So if you want to play music in Boston with a friend in NYC,
you've lost 20ms before you spend time in any queuing.

Gamers (and day traders) care about latency down to the single millisecond
range as it changes the *probability* of bad outcomes.

So unpredictable 100ms latency not only hurt web performance in easily
measurable ways as Greg and I have shown, but, more insidiously,  kill a
large set of applications, or reduces the geographic network "diameter"
over which they are practical.

Courtesy of the speed of light, therefore, any unnecessary latency *and*
jitter hurts what the Internet can be used for.
                                                  - Jim

--e89a8f5036fc9f2b7004d8728d59
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small"><br=
></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Ma=
r 21, 2013 at 12:59 PM, Greg White <span dir=3D"ltr">&lt;<a href=3D"mailto:=
g.white@cablelabs.com" target=3D"_blank">g.white@cablelabs.com</a>&gt;</spa=
n> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex">....cross posting to aqm.<br>
<br>
On 3/21/13 10:06 AM, &quot;Graham Beneke&quot; &lt;<a href=3D"mailto:graham=
@apolix.co.za">graham@apolix.co.za</a>&gt; wrote:<br>
<br>
&gt;On 13/03/2013 19:32, Mikael Abrahamsson wrote:<br>
&gt;&gt; On Wed, 13 Mar 2013, Greg White wrote:<br>
&gt;&gt;<br>
&gt;&gt;&gt; Some may consider 100ms a big number.<br>
&gt;<br>
&gt;I&#39;ve heard the 100ms figure thrown about quite a bit. In the VoIP w=
orld<br>
&gt;its considered to be the pain threshold for a reasonable call experienc=
e<br>
&gt;and for &quot;general Internet&quot; (aka http browsing) it doesn&#39;t=
 normally<br>
&gt;impact the user experience notably.<br>
<br>
Web browsing *is* sensitive to RTT (though maybe less so than VoIP or<br>
gaming). =A0Page load time increases linearly with RTT (at approximately<br=
>
10x-20x). =A0So adding 100ms of *unnecessary* buffering latency adds ~1-2<b=
r>
seconds of *unnecessary* delay to page loads.<br></blockquote><div><br></di=
v><div style=3D"font-size:small" class=3D"gmail_default">This is one very v=
isible result; but far from the most important, as we work on trying to mak=
e applications like WebRTC viable.</div>
<div><br></div><div class=3D"gmail_default">One way I think about latency i=
s that it sets the &quot;diameter&quot; in which particular applications ar=
e viable, as well as causing performance problems in general. =A0These are =
set by human factors and the speed of our neurology.</div>
<div class=3D"gmail_default"><br></div><div class=3D"gmail_default"><div cl=
ass=3D"gmail_default">Variance of latency is just about as bad as the laten=
cy itself; you have to add that to jitter buffers and it increases the late=
ncy the system can be used at.</div>
<div><br></div></div><div class=3D"gmail_default" style>And you have to thi=
nk of latency in terms of a budget: it takes of order 75ms to get across th=
e United States by a good unloaded Internet connection (the same as it did =
in the 1990&#39;s; Einstein + the index of refraction of glass =A0makes thi=
s hard to change). =A0So you really have much, much, less than 100ms you ca=
n afford to burn in the edge of the network even by this metric.</div>
<div class=3D"gmail_default"><br></div><div class=3D"gmail_default" style>F=
or example, human perception to make interactive &quot;rubber banding&quot;=
=A0imperceptible is in the 20-30ms range. =A0As the latency budget includes=
 screen refresh time, this one is doable, but you have to work hard, as we =
did with the X Window System in the 1980&#39;s. =A0I find it personally ver=
y sad that while X11 used to be usable between continents, it&#39;s not typ=
ically usable today (though mistakes we made in the X11 protocol made this =
worse than it should have been).</div>
<div class=3D"gmail_default" style><br></div><div class=3D"gmail_default" s=
tyle>If you want to play music with a friend, professional musicians might =
tolerate up to 100ms total; but duffers can&#39;t tolerate even half of tha=
t latency well. =A0So if you want to play music in Boston with a friend in =
NYC, you&#39;ve lost 20ms before you spend time in any queuing.</div>
<div class=3D"gmail_default" style><br></div><div class=3D"gmail_default" s=
tyle>Gamers (and day traders) care about latency down to the single millise=
cond range as it changes the *probability* of bad outcomes.</div><div class=
=3D"gmail_default" style>
<br></div><div class=3D"gmail_default" style>So unpredictable 100ms latency=
 not only hurt web performance in easily measurable ways as Greg and I have=
 shown, but, more=A0insidiously,=A0 kill a large set of applications, or re=
duces the geographic network &quot;diameter&quot; over which they are pract=
ical.</div>
<div class=3D"gmail_default" style><br></div><div class=3D"gmail_default" s=
tyle>Courtesy of the speed of light, therefore, any unnecessary latency *an=
d* jitter hurts what the Internet can be used for.</div><div class=3D"gmail=
_default" style>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0 =A0 =A0 - Jim</div><div class=3D"gmail_default" style><br>=
</div><div class=3D"gmail_default" style><br></div><div style=3D"font-size:=
small" class=3D"gmail_default"><br></div></div></div></div>

--e89a8f5036fc9f2b7004d8728d59--

From swmike@swm.pp.se  Thu Mar 21 21:27:39 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D37921F899E; Thu, 21 Mar 2013 21:27:39 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.378
X-Spam-Level: 
X-Spam-Status: No, score=-2.378 tagged_above=-999 required=5 tests=[AWL=0.221,  BAYES_00=-2.599]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wk7OC8bSSeIK; Thu, 21 Mar 2013 21:27:39 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id D29B621F8733; Thu, 21 Mar 2013 21:27:31 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 28ACA9C; Fri, 22 Mar 2013 05:27:31 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 21CB19A; Fri, 22 Mar 2013 05:27:31 +0100 (CET)
Date: Fri, 22 Mar 2013 05:27:31 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: Jim Gettys <jg@freedesktop.org>
In-Reply-To: <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com>
Message-ID: <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se>
References: <51408BF4.7090304@cisco.com> <514B5AC8.8000502@net.t-labs.tu-berlin.de> <alpine.DEB.2.00.1303212019520.2309@uplift.swm.pp.se> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
Cc: aqm@ietf.org, "tsvwg@ietf.org" <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 04:27:39 -0000

On Thu, 21 Mar 2013, Jim Gettys wrote:

> Every more modern TCP can easily fill any sized buffer given time with a 
> single TCP connection.

I agree with this, I made this discovery myself back in 2004 or so, and 
had to implement Fairqueue and WRED on my home connection to make it 
bearable to use any interactive application while transferring files.

In IETF75 in Stockholm in 2009, I made proposals in both TCP and at open 
mic in one of the sessions, that I would like to see statistics and 
performance numbers on packet loss, delay variation etc from actual 
traffic. The IP stack has great insight in what the network conditions are 
(especially with TCP Timestamping), but as far as I know it's not really 
exported in any usable format to the user. My idea was to have some kind 
of dashboard for the user to show if currently the network was the 
limiting factor, if the tcp window was maxed out etc. Would also be nice 
if there was output that could be cut/pasted and attached to a fault 
report in case the customer talks to customer support. It would be good if 
this was actually a standard so all OSes did the same.

I am not aware of any such work going on, so I'd like to know if anyone 
else is aware of work in this area?

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From garmitage@swin.edu.au  Thu Mar 21 23:00:50 2013
Return-Path: <garmitage@swin.edu.au>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 025C111E809C; Thu, 21 Mar 2013 23:00:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.668
X-Spam-Level: 
X-Spam-Status: No, score=-1.668 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_AU=0.377, HOST_EQ_AU=0.327, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UObxaISMlI6g; Thu, 21 Mar 2013 23:00:49 -0700 (PDT)
Received: from gpo2.cc.swin.edu.au (gpo2.cc.swin.edu.au [136.186.1.31]) by ietfa.amsl.com (Postfix) with ESMTP id 24C5321F8632; Thu, 21 Mar 2013 23:00:48 -0700 (PDT)
Received: from [136.186.229.37] (garmitage.caia.swin.edu.au [136.186.229.37]) by gpo2.cc.swin.edu.au (8.14.3/8.14.3) with ESMTP id r2M60TSX024772 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);  Fri, 22 Mar 2013 17:00:34 +1100
Message-ID: <514BF37D.2080000@swin.edu.au>
Date: Fri, 22 Mar 2013 17:00:29 +1100
From: grenville armitage <garmitage@swin.edu.au>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:16.0) Gecko/20121107 Thunderbird/16.0.2
MIME-Version: 1.0
To: Mikael Abrahamsson <swmike@swm.pp.se>
References: <51408BF4.7090304@cisco.com> <514B5AC8.8000502@net.t-labs.tu-berlin.de> <alpine.DEB.2.00.1303212019520.2309@uplift.swm.pp.se> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se>
In-Reply-To: <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: bloat <bloat@lists.bufferbloat.net>, aqm@ietf.org, "tsvwg@ietf.org" <tsvwg@ietf.org>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 06:00:50 -0000

On 03/22/2013 15:27, Mikael Abrahamsson wrote:
	[..]
> I am not aware of any such work going on, so I'd like to know if
> anyone else is aware of work in this area?

Of possible relevance, recent FreeBSDs have SIFTR -- a kernel resident
module for collecting TCP state-machine stats (like cwnd, rtt estimator,
etc) as packets arrive and depart.

http://www.freebsd.org/cgi/man.cgi?query=siftr

cheers,
gja


From swmike@swm.pp.se  Fri Mar 22 06:23:58 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E72D321F8A8F; Fri, 22 Mar 2013 06:23:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.272
X-Spam-Level: 
X-Spam-Status: No, score=-2.272 tagged_above=-999 required=5 tests=[AWL=0.100,  BAYES_00=-2.599, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1xRtr33oB97l; Fri, 22 Mar 2013 06:23:58 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 5FA8521F8A85; Fri, 22 Mar 2013 06:23:58 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 9E3A99C; Fri, 22 Mar 2013 14:23:56 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 92D019A; Fri, 22 Mar 2013 14:23:56 +0100 (CET)
Date: Fri, 22 Mar 2013 14:23:56 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: grenville armitage <garmitage@swin.edu.au>
In-Reply-To: <514BF37D.2080000@swin.edu.au>
Message-ID: <alpine.DEB.2.00.1303221421180.2309@uplift.swm.pp.se>
References: <51408BF4.7090304@cisco.com> <514B5AC8.8000502@net.t-labs.tu-berlin.de> <alpine.DEB.2.00.1303212019520.2309@uplift.swm.pp.se> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se> <514BF37D.2080000@swin.edu.au>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: "tsvwg@ietf.org" <tsvwg@ietf.org>, aqm@ietf.org, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 13:23:59 -0000

On Fri, 22 Mar 2013, grenville armitage wrote:

>
>
> On 03/22/2013 15:27, Mikael Abrahamsson wrote:
> 	[..]
>> I am not aware of any such work going on, so I'd like to know if
>> anyone else is aware of work in this area?
>
> Of possible relevance, recent FreeBSDs have SIFTR -- a kernel resident
> module for collecting TCP state-machine stats (like cwnd, rtt estimator,
> etc) as packets arrive and depart.
>
> http://www.freebsd.org/cgi/man.cgi?query=siftr

This seems exactly like what I'm after.

On another note, is there software that can get mirrored traffic from 
another host and do TCP analysis of flows when it comes to buffer bloat, 
packet loss etc. If there was, I am in the position of running this 
against a large number of customers over time and can present the results.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From dave.taht@gmail.com  Fri Mar 22 06:31:38 2013
Return-Path: <dave.taht@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8CE4121F8B7E; Fri, 22 Mar 2013 06:31:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.458
X-Spam-Level: 
X-Spam-Status: No, score=-2.458 tagged_above=-999 required=5 tests=[AWL=-0.085, BAYES_00=-2.599, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 782Xg0uJJw08; Fri, 22 Mar 2013 06:31:36 -0700 (PDT)
Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) by ietfa.amsl.com (Postfix) with ESMTP id 358D221F8B5E; Fri, 22 Mar 2013 06:31:36 -0700 (PDT)
Received: by mail-ie0-f180.google.com with SMTP id a11so4650777iee.11 for <multiple recipients>; Fri, 22 Mar 2013 06:31:35 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=n2jqGWzhH9ecVZIqVqFPzH7x8Kh3j2sldc6MLAblHh8=; b=toE3F90kIeGpSdJeXf6FJ0XkIlvK5iyB3TEcGPmAT4XfQy4568JkBjcLbnwVHS3zHQ bu8TRfaQXaR3vrGHq/+DzNEsyS+XJglRaOrmLskQvik1am+4OzafPhP3plaEsqcOiWQs Wqa+u5R2TqHHXfyqRFk1K6QSJ3u8shs8csNoV3Gn4U2T7n6TWpc1X1FdaWGVvkdZvH+J 3NCT8vpVhQkNyV2G4l1VP9CUMpM6seHdlRlGz6AWGgOer5uJvZ61neMHKmTkotsJbvSZ OTyFTcIreYein1qQzV1t5C5z0nZFkx/itpeyovDqoGF0GkuGTVc4Bc3FZ0gCf/OwpwrE yOEg==
MIME-Version: 1.0
X-Received: by 10.50.196.234 with SMTP id ip10mr1178662igc.27.1363959095143; Fri, 22 Mar 2013 06:31:35 -0700 (PDT)
Received: by 10.64.132.71 with HTTP; Fri, 22 Mar 2013 06:31:35 -0700 (PDT)
In-Reply-To: <alpine.DEB.2.00.1303221421180.2309@uplift.swm.pp.se>
References: <51408BF4.7090304@cisco.com> <514B5AC8.8000502@net.t-labs.tu-berlin.de> <alpine.DEB.2.00.1303212019520.2309@uplift.swm.pp.se> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se> <514BF37D.2080000@swin.edu.au> <alpine.DEB.2.00.1303221421180.2309@uplift.swm.pp.se>
Date: Fri, 22 Mar 2013 06:31:35 -0700
Message-ID: <CAA93jw6PMVNQ6pFKdPJcQkHNd1tLDNotAgsEPP_-ciKmavq5xw@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
To: Mikael Abrahamsson <swmike@swm.pp.se>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: bloat <bloat@lists.bufferbloat.net>, aqm@ietf.org, "tsvwg@ietf.org" <tsvwg@ietf.org>, grenville armitage <garmitage@swin.edu.au>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 13:31:38 -0000

On Fri, Mar 22, 2013 at 6:23 AM, Mikael Abrahamsson <swmike@swm.pp.se> wrot=
e:
> On Fri, 22 Mar 2013, grenville armitage wrote:
>
>>
>>
>> On 03/22/2013 15:27, Mikael Abrahamsson wrote:
>>         [..]
>>>
>>> I am not aware of any such work going on, so I'd like to know if
>>> anyone else is aware of work in this area?
>>
>>
>> Of possible relevance, recent FreeBSDs have SIFTR -- a kernel resident
>> module for collecting TCP state-machine stats (like cwnd, rtt estimator,
>> etc) as packets arrive and depart.
>>
>> http://www.freebsd.org/cgi/man.cgi?query=3Dsiftr
>
>
> This seems exactly like what I'm after.
>
> On another note, is there software that can get mirrored traffic from
> another host and do TCP analysis of flows when it comes to buffer bloat,
> packet loss etc. If there was, I am in the position of running this again=
st
> a large number of customers over time and can present the results.

http://web10g.org/ and ihttp://www.web100.org/ are also potentially
very useful. My concern with this used to be that it gets it's fingers
into the tcp hot path, but in practice modern gear is so fast it's
hard to notice.

I used to patch this into cerowrt but it was hard to keep the patches
in sync with the linux mainline code I was also tracking. Might
consider adding it again, but in the general case it seems more useful
on the end points than elsewhere (my use case was against the local
test suites on cero and on the polipo web proxy resident on the
router)

>
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm



--=20
Dave T=E4ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.=
html

From gorry@erg.abdn.ac.uk  Fri Mar 22 07:49:09 2013
Return-Path: <gorry@erg.abdn.ac.uk>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 02C7321F8CDF for <aqm@ietfa.amsl.com>; Fri, 22 Mar 2013 07:49:09 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.486
X-Spam-Level: 
X-Spam-Status: No, score=-106.486 tagged_above=-999 required=5 tests=[AWL=-0.114, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gduP7BxQJZ0K for <aqm@ietfa.amsl.com>; Fri, 22 Mar 2013 07:49:08 -0700 (PDT)
Received: from spey.erg.abdn.ac.uk (spey.erg.abdn.ac.uk [139.133.204.173]) by ietfa.amsl.com (Postfix) with ESMTP id 533CC21F8BA1 for <aqm@ietf.org>; Fri, 22 Mar 2013 07:49:08 -0700 (PDT)
Received: by spey.erg.abdn.ac.uk (Postfix, from userid 5001) id A7DF22B44B4; Fri, 22 Mar 2013 14:49:07 +0000 (GMT)
Received: from gorry-mac.erg.abdn.ac.uk (gorry-mac.erg.abdn.ac.uk [139.133.207.5]) by spey.erg.abdn.ac.uk (Postfix) with ESMTPSA id 0D3FC2B4275 for <aqm@ietf.org>; Fri, 22 Mar 2013 14:49:07 +0000 (GMT)
Message-ID: <514C6F62.5070906@erg.abdn.ac.uk>
Date: Fri, 22 Mar 2013 14:49:06 +0000
From: Gorry Fairhurst <gorry@erg.abdn.ac.uk>
Organization: The University of Aberdeen is a charity registered in Scotland,  No SC013683. 
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:17.0) Gecko/20130107 Thunderbird/17.0.2
MIME-Version: 1.0
To: aqm@ietf.org
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <8C48B86A895913448548E6D15DA7553B7DC251@xmb-rcd-x09.cisco.com>
In-Reply-To: <8C48B86A895913448548E6D15DA7553B7DC251@xmb-rcd-x09.cisco.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Subject: Re: [aqm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: gorry@erg.abdn.ac.uk
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 14:49:09 -0000

On 19/03/2013 20:35, Fred Baker (fred) wrote:
>
> On Mar 19, 2013, at 1:04 AM, Mirja Kühlewind <mirja.kuehlewind@ikr.uni-stuttgart.de> wrote:
>
>> Do we need to start a campaign to further encourage greater ECN support (also
>> of network providers)? Asking the content providers on the list(s): What are
>> the reasons to not enable ECN support?
>
> probably.
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>
I think so too - for years this was just another "technology", it's 
rapidly becoming important to control internet queuing delay and we need 
to consider seriously whether we recommend this and what.

Gorry

From dave.taht@gmail.com  Fri Mar 22 08:04:55 2013
Return-Path: <dave.taht@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BD0EA21F8C8C for <aqm@ietfa.amsl.com>; Fri, 22 Mar 2013 08:04:55 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.664
X-Spam-Level: 
X-Spam-Status: No, score=-1.664 tagged_above=-999 required=5 tests=[AWL=-0.846, BAYES_00=-2.599, FRT_LITTLE=1.555, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gYzYGfeb0TSz for <aqm@ietfa.amsl.com>; Fri, 22 Mar 2013 08:04:55 -0700 (PDT)
Received: from mail-ia0-x22b.google.com (mail-ia0-x22b.google.com [IPv6:2607:f8b0:4001:c02::22b]) by ietfa.amsl.com (Postfix) with ESMTP id 0E56321F8B80 for <aqm@ietf.org>; Fri, 22 Mar 2013 08:04:54 -0700 (PDT)
Received: by mail-ia0-f171.google.com with SMTP id z13so3634385iaz.2 for <aqm@ietf.org>; Fri, 22 Mar 2013 08:04:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=hJw9mYkYUbcAFdlMyY6z1kplxrDeyKD0ztSsYWp8Q1g=; b=Q4lAtzAoZPgJLHg2tKFa8AVH6rPa137Cixx0Rd/vBRgBljOgE+KPqY9lO3WzTDxYvH OE5ExU7ZeCiJ28XEJxeLhe+jCGcKKckx2MbaHMXkbX1dnqMHEcdFWZ8b9psRJo0hHBhg pWUtiHI2nOUDY5+hHIbRwH6fSy0wge9DfPf1Can2bQfDe7tGDDBtn7rN0E6t6R6CuBhE 8FrGmZy4BQUsXEU00SUU5V+FHYiM/buZzaJ2pjjU5SAKde589typ2kwhTjjYLyHJOcb8 oIeJ36c0PAsW+Ivr+FfKJa02nbeUwyfkXhLq0M8+7Bp1bMvJWEiQ8sbv7xrEP+ncF2mx /fDg==
MIME-Version: 1.0
X-Received: by 10.42.247.8 with SMTP id ma8mr1199682icb.1.1363964694681; Fri, 22 Mar 2013 08:04:54 -0700 (PDT)
Received: by 10.64.132.71 with HTTP; Fri, 22 Mar 2013 08:04:54 -0700 (PDT)
In-Reply-To: <514C6F62.5070906@erg.abdn.ac.uk>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <8C48B86A895913448548E6D15DA7553B7DC251@xmb-rcd-x09.cisco.com> <514C6F62.5070906@erg.abdn.ac.uk>
Date: Fri, 22 Mar 2013 08:04:54 -0700
Message-ID: <CAA93jw4nF8RE_f8vqsNFcb+K-V+4L92f6pM9V7mBti179KpSPw@mail.gmail.com>
From: Dave Taht <dave.taht@gmail.com>
To: gorry@erg.abdn.ac.uk
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: Keith Winstein <keithw@mit.edu>, aqm@ietf.org
Subject: Re: [aqm] ECN support and usage on the Internet
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Mar 2013 15:04:55 -0000

This is just sort of a grab bag of reactions to this thread.

ECN on conventional voip is nuts. Another packet will be along soon enough.

It might make sense on videoconferencing and on adaptive voice/video codecs=
.

ECN on egress from a low bandwidth router is dumb. Dropping packets is
much more efficient.

ECN in ingress from a cmts/dslam to a couple short hops inside the home
gateway seems to work pretty well at 20+ mbit. The packet's traversed
the internet, might as well mark it rather than drop.

The Xfq_codel shapers that I've been using to get results like toke's
at iccrg and those on the
http://www.bufferbloat.net/projects/codel/wiki/RRUL_Rogues_Gallery
do ecn in ingress to the home and drop on egress.

ECN however can be seriously gamed and there needs to be an operating
point in an AQM where packets are dropped not marked. I never found
one that worked, but that was in early fq_codel days before
I had tests like RRUL that made sense. (see codel list thread on ECN:

https://lists.bufferbloat.net/pipermail/codel/2012-August/000412.html

I have also successfully been using ecn enablement on wifi routing
traffic, using the congestion indicator as a possible means to explore
new metrics.

That said, marking almost never happens at the stack layer fq_codel
is at now in Linux wifi, but I suspect, if we get separate queues for
wifi packets, notably multicast, it may become useful as an indicator
of (for
example) multicast overload on wifi.

That said, I'd really like there
to be a dropped packet and delay indications from codel information
(via netlink) that a userspace daemon could "see" so as to get more of
that indication, more rapidly, up there, where we could intelligently
(for example) re-route flows.

mosh ( http://mosh.mit.edu) traffic got ecn capability about 6? months
ago, and it was neat.

It worked. :)

What did happen instead, was the weird marking I'd asked the mosh folk
to use (AF42) caused problems on two networks, keith has more
details.

Lastly, as for the lemming problem on incast, I tend to agree with
fred that using TCP as written is not the right thing. :) I liked what
I little I understood of RFC6013 attempts on this front (and on things
like reducing amplification attacks on dns), it's too bad that work
seems to have stalled out, and that option space is so scarce.

--=20
Dave T=E4ht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.=
html

From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 26 07:50:08 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D44921F8AB7 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 07:50:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.98
X-Spam-Level: 
X-Spam-Status: No, score=-0.98 tagged_above=-999 required=5 tests=[AWL=-1.269,  BAYES_20=-0.74, HELO_EQ_DE=0.35, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1xTXV3GyE8qi for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 07:50:07 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 762A421F8735 for <aqm@ietf.org>; Tue, 26 Mar 2013 07:50:03 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id 01AAB60275; Tue, 26 Mar 2013 15:49:59 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id C6A576026E; Tue, 26 Mar 2013 15:49:59 +0100 (CET)
From: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Organization: University of Stuttgart (Germany), IKR
To: aqm@ietf.org
Date: Tue, 26 Mar 2013 15:49:59 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com> <5148F80F.5000401@superduper.net>
In-Reply-To: <5148F80F.5000401@superduper.net>
X-KMail-QuotePrefix: > 
MIME-Version: 1.0
Content-Type: Text/Plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <201303261549.59606.mirja.kuehlewind@ikr.uni-stuttgart.de>
Cc: Simon Barber <simon@superduper.net>
Subject: Re: [aqm] =?utf-8?b?562U5aSNOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBh?= =?utf-8?q?nd_usage_on_the_Internet?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 14:50:08 -0000

Hi,

I don't understand the obsession to not use ECN. The advantage is, you don'=
t=20
have to drop packets. Sure, (core) networks are optimized to have a very lo=
w=20
drop rate anyway, but isn't it better not not drop any packets at all?

Mirja


On Wednesday 20 March 2013 00:43:11 Simon Barber wrote:
> I'm not sure why people are so obsessed with ECN - the little data I've
> seen about it showed that it's effect on system throughput was in the
> noise. Sure, wireless guys don't like to waste precious wireless
> bandwidth, but they're not dropping packets that have used up the
> precious wireless bandwidth. They get dropped before the wireless link.
>
> Simon
>
> On 03/19/2013 03:36 PM, Jim Gettys wrote:
> > Clearly, having a working AQM is a necessary prerequisite for ECN to be
> > useful; one only overcome in the last year with CoDel/Pie.  Variable
> > bandwidth is still a challenge, but with CoDel at least, we think it can
> > be done.  And we've certainly implemented ECN support in the
> > codel/fq_codel qdisc's (though would be grateful if ECN experts make
> > sure it's been done correctly).
> >
> > The incentive structure needs to work here. There may be things we can
> > and should do to provide incentive for ECN deployment.
> >
> > For Bell Heads working on wireless cellular transport, the incentive is
> > really pretty clear: they ***hate*** dropping packets with a passion,
> > not understanding that either packet drop or ECN are necessary, and it
> > will probably be easier to convince them to adopt ECN than to drop
> > packets.  Given how hard they sweat to get packets over the air, I sort
> > of understand where they come from.
> >
> > I believe some of the cellular carriers are requiring ECN support in
> > handsets going forward, and as they classify VOIP into a separate bearer
> > class anyway, I can see little incentive not to use ECN on their "best
> > effort" bearer class.
> >
> > I believe all current operating systems react correctly to ECN marked
> > packets; the only question is whether they will talk ECN if talked to
> > (pretty common; Linux has done so now for quite a few years), *and*
> > whether they enable ECN when establishing new connections or not (no one
> > does right now by default).
> >
> > At the moment, Windows is vanishingly small fraction of handsets, being
> > dominated by Android (Linux) and Apple as it is.
> >
> > I worry about bufferbloat in the HARQ layer of cellular networks not
> > reflecting ECN marking into packets, but until that analysis has been
> > done in real systems, it's just one more thing to worry about. Our Linux
> > experience is that bufferbloat hides everywhere in current systems: the
> > analogy in WiFi is the error correction layer in 802.11n, which causes
> > large amounts of bloat, and fixing WiFi is very much a work in progress
> > (except that not enough resources are currently available to go do the
> > work, and it's more complicated that wired systems).  Dave Taht cheered
> > the first time he got the 802.11 layer to drop a packet, it was so much
> > a problem.
> >
> > Right now, there are still some disincentives for ECN: e.g. some very
> > small fraction of sites that become "black holes", or crashing antique
> > middelboxes. We have no data on how much this is at this date, and the
> > database that used to exist documenting middlebox problems has
> > bit-rotted.  These  middleboxes are likely in the consumer edge, and
> > unlikely to be in the path of most mobile devices and web sites on
> > 3g/LTE.  Mobile devices on home WiFi networks are more likely to have
> > problems.
> >
> > We have ECN enabled in CeroWrt and have not noticed any problems; but
> > 4-5 years ago OpenWrt (with a very much larger user base) still detected
> > problems to the extent they had to disable initiating ECN connections.
> >
> > I know Bauer et. al. examined this a few years ago. But we have no data
> > over the last several years I've seen (though this new paper referenced
> > may: I've not had a chance to read it yet).
> >
> > There is one other issue with ECN: when operating at low bandwidth at
> > the very edge of the net, it can still be advantageous to drop a packet
> > rather than ECN mark it, to get back the transmission time of the
> > packet.  Dave Taht's experiments (IIRC) were below 4Mbps for this
> > advantage (when doing VOIP over that bottleneck competing against other
> > traffic in a best effort class).
> >
> > My intuition is to get VOIP reliable over WiFI on a busy network (or
> > when far away from the AP) we'll want to classify the VOIP packets
> > anyway and use the right hardware class in WiFi; this is primarily to
> > get priority access to a transmit opportunity on WiFi rather than the
> > previous reasons we've had.  Certainly on other media we've seen little
> > reason to have to do any explicit classification for VOIP anymore once
> > running fq_codel.  Things generally "just work" without all that hair
> > for VOIP traffic.
> >                                                    - Jim
> >
> >
> >
> >
> >
> > On Tue, Mar 19, 2013 at 5:49 PM, =E5=A4=A9=E6=BE=9C <tianlan.lhh@alibab=
a-inc.com
> > <mailto:tianlan.lhh@alibaba-inc.com>> wrote:
> >
> >     To let ECN be popular used =EF=BC=8Cthe most important is let the
> >     connection which used =E2=80=9CECN=E2=80=9D is fairness to the othe=
r connection;
> >     So we should design new algorithm for ECN connection to "prize"
> >     those people who use ECN;
> >
> >
> >                                                     hritian
> >
> >     ________________________________________
> >     =E5=8F=91=E4=BB=B6=E4=BA=BA: Jim Gettys [jg@freedesktop.org <mailto=
:jg@freedesktop.org>]
> >     =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2013=E5=B9=B43=E6=9C=8820=E6=
=97=A5 1:26
> >     =E5=88=B0: Rick Jones
> >     Cc: tcpm@ietf.org <mailto:tcpm@ietf.org>; aqm@ietf.org
> >     <mailto:aqm@ietf.org>; iccrg@irtf.org <mailto:iccrg@irtf.org>;
> >     Mikael Abrahamsson
> >     =E4=B8=BB=E9=A2=98: Re: [iccrg] [tcpm] ECN support and usage on the=
 Internet
> >
> >     You need to distinguish 1) "Will talk ECN when talked to", from 2)
> >     "Will initiate an ECN conversation".
> >
> >     Linux, for example, will do 1) by default (and has for some time
> >     now), but requires configuration to turn on ECN when initiating a
> >     connection.
> >                                        - Jim
> >
> >
> >
> >     On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <rick.jones2@hp.com
> >     <mailto:rick.jones2@hp.com><mailto:rick.jones2@hp.com
> >     <mailto:rick.jones2@hp.com>>> wrote:
> >     On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
> >     On Tue, 19 Mar 2013, Mirja K=C3=BChlewind wrote:
> >
> >     Do we need to start a campaign to further encourage greater ECN
> >     support (also of network providers)? Asking the content providers on
> >     the list(s): What are the reasons to not enable ECN support?
> >
> >     Let me speculate:
> >
> >     It's not on by default in Windows. There is little reason to turn it
> > on, because "nobody" acts on ECN, and nobody acts on ECN because there =
is
> > no demand for AQM, and even if there was AQM, it would most likely not =
be
> > ECN enabled.
> >
> >     Out of curiosity, which stacks have ECN enabled by default?  To my
> >     recollection, on netperf.org
> >     <http://netperf.org><http://netperf.org> I had to explicitly enable
> >     TCP's use of ECN in the kernel which ships with Ubuntu 12.04.
> >
> >     rick jones
> >
> >     _______________________________________________
> >     tcpm mailing list
> >     tcpm@ietf.org <mailto:tcpm@ietf.org><mailto:tcpm@ietf.org
> >     <mailto:tcpm@ietf.org>>
> >     https://www.ietf.org/mailman/listinfo/tcpm
> >
> >
> >     ________________________________
> >
> >     This email (including any attachments) is confidential and may be
> >     legally privileged. If you received this email in error, please
> >     delete it immediately and do not copy it or use it for any purpose
> >     or disclose its contents to any other person. Thank you.
> >
> >     =E6=9C=AC=E7=94=B5=E9=82=AE(=E5=8C=85=E6=8B=AC=E4=BB=BB=E4=BD=95=E9=
=99=84=E4=BB=B6)=E5=8F=AF=E8=83=BD=E5=90=AB=E6=9C=89=E6=9C=BA=E5=AF=86=E8=
=B5=84=E6=96=99=E5=B9=B6=E5=8F=97=E6=B3=95=E5=BE=8B=E4=BF=9D=E6=8A=A4=E3=80=
=82=E5=A6=82=E6=82=A8=E4=B8=8D=E6=98=AF=E6=AD=A3=E7=A1=AE=E7=9A=84=E6=94=B6=
=E4=BB=B6
> >     =E4=BA=BA=EF=BC=8C=E8=AF=B7=E6=82=A8=E7=AB=8B=E5=8D=B3=E5=88=A0=E9=
=99=A4=E6=9C=AC=E9=82=AE=E4=BB=B6=E3=80=82=E8=AF=B7=E4=B8=8D=E8=A6=81=E5=B0=
=86=E6=9C=AC=E7=94=B5=E9=82=AE=E8=BF=9B=E8=A1=8C=E5=A4=8D=E5=88=B6=E5=B9=B6=
=E7=94=A8=E4=BD=9C=E4=BB=BB=E4=BD=95=E5=85=B6=E4=BB=96=E7=94=A8=E9=80=94=E3=
=80=81=E6=88=96
> >     =E9=80=8F=E9=9C=B2=E6=9C=AC=E9=82=AE=E4=BB=B6=E4=B9=8B=E5=86=85=E5=
=AE=B9=E3=80=82=E8=B0=A2=E8=B0=A2=E3=80=82
> >
> >
> >
> >
> > _______________________________________________
> > aqm mailing list
> > aqm@ietf.org
> > https://www.ietf.org/mailman/listinfo/aqm
>
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm



=2D-=20
=2D------------------------------------------------------------------
Dipl.-Ing. Mirja K=C3=BChlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: mirja.kuehlewind@ikr.uni-stuttgart.de
web: www.ikr.uni-stuttgart.de
=2D------------------------------------------------------------------

From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 26 08:02:37 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 52E7221F8C77 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 08:02:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.501
X-Spam-Level: 
X-Spam-Status: No, score=-1.501 tagged_above=-999 required=5 tests=[AWL=0.521,  BAYES_00=-2.599, HELO_EQ_DE=0.35, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z8k1v2eWLNbV for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 08:02:36 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 50E3321F8C4C for <aqm@ietf.org>; Tue, 26 Mar 2013 08:02:33 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id 6FCD960275; Tue, 26 Mar 2013 16:02:32 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id 4AA326026E; Tue, 26 Mar 2013 16:02:32 +0100 (CET)
From: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Organization: University of Stuttgart (Germany), IKR
To: aqm@ietf.org
Date: Tue, 26 Mar 2013 16:02:31 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com>
In-Reply-To: <1363748540.64155384@apps.rackspace.com>
X-KMail-QuotePrefix: > 
MIME-Version: 1.0
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de>
Cc: dpreed@reed.com
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 15:02:37 -0000

Hi,

it is actually a feature of AQM/ECN that you have small standing queues whe=
n=20
having persistent flows but short flow are able to fill the queue (for a=20
short period only) without getting packet losses immediately!

Mirja


On Wednesday 20 March 2013 04:02:20 dpreed@reed.com wrote:
> A problem with ECN is that it does not drain an already flooded queue, and
> depends (like RED) on the idea that one has a large queue in normal
> operation.  It is not obvious that sustaining a deep queue is a desirable
> state, especially when there are highly variable capacities along any pat=
h,
> as is the case where one link is more than 2x slower than the rest of the
> links.
>
> If the dynamics of user loads are fractal, this will cause draining (and
> latency reduction) not to work.
>
> If user loads are stationary gaussian and so forth, ECN can "tune" to
> relatively stable situations, but the queueing delay will be quite long
> end-to-end.
>
> Since there is lots of evidence that user loads are fractal, dropping
> packets during sudden bursts will prevent sustained queueing, whereas ECN
> will not.
>
> So a proper approach (in my opinion) on the "real" Internet (rather than a
> bunch of competing long-duration FTPs) requires keeping the queues so sho=
rt
> that there is no real opportunity to benefit from ECN.
>
> Hence - we need to really see what happens with *real* traffic loads, not
> simplified simulations.
>
> Real traffic loads are fractal, and fractal loads do not generally obey t=
he
> Gaussian style "law of large numbers" - multiple fractal flows are
> *rougher* not smoother than a single one.
>
> So check your mathematical "intuitions" at the door.  Stop writing
> equations based on tractable analytic models, because actual traffic does
> not follow those models.
>
> Yeah, the Bell Heads believe in never dropping packets.  So what.  The
> Internet is not about making them comfortable in their misunderstandings.
>
> -----Original Message-----
> From: aqm-request@ietf.org
> Sent: Tuesday, March 19, 2013 7:43pm
> To: aqm@ietf.org
> Subject: aqm Digest, Vol 1, Issue 7
>
>
>
> If you have received this digest without all the individual message
> attachments you will need to update your digest options in your list
> subscription.  To do so, go to
>
> https://www.ietf.org/mailman/listinfo/aqm
>
> Click the 'Unsubscribe or edit options' button, log in, and set "Get
> MIME or Plain Text Digests?" to MIME.  You can set this option
> globally for all the list digests you receive at this point.
>
>
>
> Send aqm mailing list submissions to
>  aqm@ietf.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>  https://www.ietf.org/mailman/listinfo/aqm
> or, via email, send a message with subject or body 'help' to
>  aqm-request@ietf.org
>
> You can reach the person managing the list at
>  aqm-owner@ietf.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of aqm digest..."
> Today's Topics:
>
>  1. Re: [tcpm] ECN support and usage on the Internet (Yuchung Cheng)
>  2. Re: ECN support and usage on the Internet (Fred Baker (fred))
>  3. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
>  (Jim Gettys)
>  4. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
>  (Simon Barber)
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm



=2D-=20
=2D------------------------------------------------------------------
Dipl.-Ing. Mirja K=FChlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: mirja.kuehlewind@ikr.uni-stuttgart.de
web: www.ikr.uni-stuttgart.de
=2D------------------------------------------------------------------

From gettysjim@gmail.com  Tue Mar 26 09:18:29 2013
Return-Path: <gettysjim@gmail.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2189921F872E for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 09:18:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.75
X-Spam-Level: 
X-Spam-Status: No, score=-1.75 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FM_FORGED_GMAIL=0.622, HTML_MESSAGE=0.001, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id w1GkNm-Wx6uy for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 09:18:28 -0700 (PDT)
Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) by ietfa.amsl.com (Postfix) with ESMTP id DFF1F21F8721 for <aqm@ietf.org>; Tue, 26 Mar 2013 09:18:27 -0700 (PDT)
Received: by mail-ob0-f181.google.com with SMTP id ni5so7303517obc.12 for <aqm@ietf.org>; Tue, 26 Mar 2013 09:18:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=q/FAqxFCTFDYYobgY9aKWn7cFOe/Dlyow9+IbT6xo64=; b=ijuaSyPi/E2cGza6CrmARJz0tg40AYXhXMDK5ngYfB31EROjexiAjEA7iI9aLdx5lN xYX/rKfYPZpcM8oOZWpLLYUlZC2Ng6OxLcdsDnz3C0EQnQHoYakJVAlqU6Nf7fNoBwZR G2KPTGMRrVy9cvPihm4pflc3mwlZt2QjnZJF2qytoQRtIA9n/zDOoVU6rl/hXZ2dot+j iwShaAciruYmgwtuRbclJ0+j2AGRJMI/Qe2lBK8tkPVk46+XJAABSfC5gW5K1+IGVDZo 5t4idVTtRounNTqMhcR/lRb8RI4SWKb0uew3OhqgQPGKu/uajFl839wyhTCRO71j7S49 MFkg==
MIME-Version: 1.0
X-Received: by 10.60.8.197 with SMTP id t5mr14269074oea.4.1364314707301; Tue, 26 Mar 2013 09:18:27 -0700 (PDT)
Sender: gettysjim@gmail.com
Received: by 10.76.22.193 with HTTP; Tue, 26 Mar 2013 09:18:27 -0700 (PDT)
In-Reply-To: <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de>
Date: Tue, 26 Mar 2013 12:18:27 -0400
X-Google-Sender-Auth: Yuodsw_WEzdCXzEG9Em5-miMBHY
Message-ID: <CAGhGL2ArSLkDhSgAHGfAx=oHrBSKQBDPBNTGz444U_k5-3kPhw@mail.gmail.com>
From: Jim Gettys <jg@freedesktop.org>
To: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Content-Type: multipart/alternative; boundary=e89a8fb1ef427d9d1404d8d6459d
Cc: David P Reed <dpreed@reed.com>, "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 16:18:29 -0000

--e89a8fb1ef427d9d1404d8d6459d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Mar 26, 2013 at 11:02 AM, Mirja Kuehlewind <
mirja.kuehlewind@ikr.uni-stuttgart.de> wrote:

> Hi,
>
> it is actually a feature of AQM/ECN that you have small standing queues
> when
> having persistent flows but short flow are able to fill the queue (for a
> short period only) without getting packet losses immediately!
>
> Mirja
>
>
Exactly what the tradeoffs we're going to have when we throw flow queuing
into the mix isn't clear to me as yet.  It's one thing when the packet's
latency hurts every flow; different when it's just the flow building the
queue.

I don't think anyone has looked into this at all.
                                    - Jim


>
> On Wednesday 20 March 2013 04:02:20 dpreed@reed.com wrote:
> > A problem with ECN is that it does not drain an already flooded queue,
> and
> > depends (like RED) on the idea that one has a large queue in normal
> > operation.  It is not obvious that sustaining a deep queue is a desirab=
le
> > state, especially when there are highly variable capacities along any
> path,
> > as is the case where one link is more than 2x slower than the rest of t=
he
> > links.
> >
> > If the dynamics of user loads are fractal, this will cause draining (an=
d
> > latency reduction) not to work.
> >
> > If user loads are stationary gaussian and so forth, ECN can "tune" to
> > relatively stable situations, but the queueing delay will be quite long
> > end-to-end.
> >
> > Since there is lots of evidence that user loads are fractal, dropping
> > packets during sudden bursts will prevent sustained queueing, whereas E=
CN
> > will not.
> >
> > So a proper approach (in my opinion) on the "real" Internet (rather tha=
n
> a
> > bunch of competing long-duration FTPs) requires keeping the queues so
> short
> > that there is no real opportunity to benefit from ECN.
> >
> > Hence - we need to really see what happens with *real* traffic loads, n=
ot
> > simplified simulations.
> >
> > Real traffic loads are fractal, and fractal loads do not generally obey
> the
> > Gaussian style "law of large numbers" - multiple fractal flows are
> > *rougher* not smoother than a single one.
> >
> > So check your mathematical "intuitions" at the door.  Stop writing
> > equations based on tractable analytic models, because actual traffic do=
es
> > not follow those models.
> >
> > Yeah, the Bell Heads believe in never dropping packets.  So what.  The
> > Internet is not about making them comfortable in their misunderstanding=
s.
> >
> > -----Original Message-----
> > From: aqm-request@ietf.org
> > Sent: Tuesday, March 19, 2013 7:43pm
> > To: aqm@ietf.org
> > Subject: aqm Digest, Vol 1, Issue 7
> >
> >
> >
> > If you have received this digest without all the individual message
> > attachments you will need to update your digest options in your list
> > subscription.  To do so, go to
> >
> > https://www.ietf.org/mailman/listinfo/aqm
> >
> > Click the 'Unsubscribe or edit options' button, log in, and set "Get
> > MIME or Plain Text Digests?" to MIME.  You can set this option
> > globally for all the list digests you receive at this point.
> >
> >
> >
> > Send aqm mailing list submissions to
> >  aqm@ietf.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >  https://www.ietf.org/mailman/listinfo/aqm
> > or, via email, send a message with subject or body 'help' to
> >  aqm-request@ietf.org
> >
> > You can reach the person managing the list at
> >  aqm-owner@ietf.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of aqm digest..."
> > Today's Topics:
> >
> >  1. Re: [tcpm] ECN support and usage on the Internet (Yuchung Cheng)
> >  2. Re: ECN support and usage on the Internet (Fred Baker (fred))
> >  3. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
> >  (Jim Gettys)
> >  4. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
> >  (Simon Barber)
> > _______________________________________________
> > aqm mailing list
> > aqm@ietf.org
> > https://www.ietf.org/mailman/listinfo/aqm
>
>
>
> --
> -------------------------------------------------------------------
> Dipl.-Ing. Mirja K=FChlewind
> Institute of Communication Networks and Computer Engineering (IKR)
> University of Stuttgart, Germany
> Pfaffenwaldring 47, D-70569 Stuttgart
>
> tel: +49(0)711/685-67973
> email: mirja.kuehlewind@ikr.uni-stuttgart.de
> web: www.ikr.uni-stuttgart.de
> -------------------------------------------------------------------
> _______________________________________________
> aqm mailing list
> aqm@ietf.org
> https://www.ietf.org/mailman/listinfo/aqm
>

--e89a8fb1ef427d9d1404d8d6459d
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-size:small"><br=
></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Tue=
, Mar 26, 2013 at 11:02 AM, Mirja Kuehlewind <span dir=3D"ltr">&lt;<a href=
=3D"mailto:mirja.kuehlewind@ikr.uni-stuttgart.de" target=3D"_blank">mirja.k=
uehlewind@ikr.uni-stuttgart.de</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex">Hi,<br>
<br>
it is actually a feature of AQM/ECN that you have small standing queues whe=
n<br>
having persistent flows but short flow are able to fill the queue (for a<br=
>
short period only) without getting packet losses immediately!<br>
<br>
Mirja<br>
<div class=3D""><div class=3D"h5"><br></div></div></blockquote><div><br></d=
iv><div class=3D"gmail_default">Exactly what the tradeoffs we&#39;re going =
to have when we throw flow queuing into the mix isn&#39;t clear to me as ye=
t. =A0It&#39;s one thing when the packet&#39;s latency hurts every flow; di=
fferent when it&#39;s just the flow building the queue.</div>
<div class=3D"gmail_default"><br></div><div class=3D"gmail_default">I don&#=
39;t think anyone has looked into this at all.</div><div class=3D"gmail_def=
ault">=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 - Jim</div><div style=3D"font-size:small" class=3D"gmail_default">
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:s=
olid;padding-left:1ex"><div class=3D""><div class=3D"h5">
<br>
On Wednesday 20 March 2013 04:02:20 <a href=3D"mailto:dpreed@reed.com">dpre=
ed@reed.com</a> wrote:<br>
&gt; A problem with ECN is that it does not drain an already flooded queue,=
 and<br>
&gt; depends (like RED) on the idea that one has a large queue in normal<br=
>
&gt; operation. =A0It is not obvious that sustaining a deep queue is a desi=
rable<br>
&gt; state, especially when there are highly variable capacities along any =
path,<br>
&gt; as is the case where one link is more than 2x slower than the rest of =
the<br>
&gt; links.<br>
&gt;<br>
&gt; If the dynamics of user loads are fractal, this will cause draining (a=
nd<br>
&gt; latency reduction) not to work.<br>
&gt;<br>
&gt; If user loads are stationary gaussian and so forth, ECN can &quot;tune=
&quot; to<br>
&gt; relatively stable situations, but the queueing delay will be quite lon=
g<br>
&gt; end-to-end.<br>
&gt;<br>
&gt; Since there is lots of evidence that user loads are fractal, dropping<=
br>
&gt; packets during sudden bursts will prevent sustained queueing, whereas =
ECN<br>
&gt; will not.<br>
&gt;<br>
&gt; So a proper approach (in my opinion) on the &quot;real&quot; Internet =
(rather than a<br>
&gt; bunch of competing long-duration FTPs) requires keeping the queues so =
short<br>
&gt; that there is no real opportunity to benefit from ECN.<br>
&gt;<br>
&gt; Hence - we need to really see what happens with *real* traffic loads, =
not<br>
&gt; simplified simulations.<br>
&gt;<br>
&gt; Real traffic loads are fractal, and fractal loads do not generally obe=
y the<br>
&gt; Gaussian style &quot;law of large numbers&quot; - multiple fractal flo=
ws are<br>
&gt; *rougher* not smoother than a single one.<br>
&gt;<br>
&gt; So check your mathematical &quot;intuitions&quot; at the door. =A0Stop=
 writing<br>
&gt; equations based on tractable analytic models, because actual traffic d=
oes<br>
&gt; not follow those models.<br>
&gt;<br>
&gt; Yeah, the Bell Heads believe in never dropping packets. =A0So what. =
=A0The<br>
&gt; Internet is not about making them comfortable in their misunderstandin=
gs.<br>
&gt;<br>
&gt; -----Original Message-----<br>
&gt; From: <a href=3D"mailto:aqm-request@ietf.org">aqm-request@ietf.org</a>=
<br>
&gt; Sent: Tuesday, March 19, 2013 7:43pm<br>
&gt; To: <a href=3D"mailto:aqm@ietf.org">aqm@ietf.org</a><br>
&gt; Subject: aqm Digest, Vol 1, Issue 7<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; If you have received this digest without all the individual message<br=
>
&gt; attachments you will need to update your digest options in your list<b=
r>
&gt; subscription. =A0To do so, go to<br>
&gt;<br>
&gt; <a href=3D"https://www.ietf.org/mailman/listinfo/aqm" target=3D"_blank=
">https://www.ietf.org/mailman/listinfo/aqm</a><br>
&gt;<br>
&gt; Click the &#39;Unsubscribe or edit options&#39; button, log in, and se=
t &quot;Get<br>
&gt; MIME or Plain Text Digests?&quot; to MIME. =A0You can set this option<=
br>
&gt; globally for all the list digests you receive at this point.<br>
&gt;<br>
&gt;<br>
&gt;<br>
&gt; Send aqm mailing list submissions to<br>
&gt; =A0<a href=3D"mailto:aqm@ietf.org">aqm@ietf.org</a><br>
&gt;<br>
&gt; To subscribe or unsubscribe via the World Wide Web, visit<br>
&gt; =A0<a href=3D"https://www.ietf.org/mailman/listinfo/aqm" target=3D"_bl=
ank">https://www.ietf.org/mailman/listinfo/aqm</a><br>
&gt; or, via email, send a message with subject or body &#39;help&#39; to<b=
r>
&gt; =A0<a href=3D"mailto:aqm-request@ietf.org">aqm-request@ietf.org</a><br=
>
&gt;<br>
&gt; You can reach the person managing the list at<br>
&gt; =A0<a href=3D"mailto:aqm-owner@ietf.org">aqm-owner@ietf.org</a><br>
&gt;<br>
&gt; When replying, please edit your Subject line so it is more specific<br=
>
&gt; than &quot;Re: Contents of aqm digest...&quot;<br>
&gt; Today&#39;s Topics:<br>
&gt;<br>
&gt; =A01. Re: [tcpm] ECN support and usage on the Internet (Yuchung Cheng)=
<br>
&gt; =A02. Re: ECN support and usage on the Internet (Fred Baker (fred))<br=
>
&gt; =A03. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet<br>
&gt; =A0(Jim Gettys)<br>
&gt; =A04. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet<br>
&gt; =A0(Simon Barber)<br>
&gt; _______________________________________________<br>
&gt; aqm mailing list<br>
&gt; <a href=3D"mailto:aqm@ietf.org">aqm@ietf.org</a><br>
&gt; <a href=3D"https://www.ietf.org/mailman/listinfo/aqm" target=3D"_blank=
">https://www.ietf.org/mailman/listinfo/aqm</a><br>
<br>
<br>
<br>
</div></div><span class=3D""><font color=3D"#888888">--<br>
-------------------------------------------------------------------<br>
Dipl.-Ing. Mirja K=FChlewind<br>
Institute of Communication Networks and Computer Engineering (IKR)<br>
University of Stuttgart, Germany<br>
Pfaffenwaldring 47, D-70569 Stuttgart<br>
<br>
tel: <a href=3D"tel:%2B49%280%29711%2F685-67973" value=3D"+4971168567973">+=
49(0)711/685-67973</a><br>
email: <a href=3D"mailto:mirja.kuehlewind@ikr.uni-stuttgart.de">mirja.kuehl=
ewind@ikr.uni-stuttgart.de</a><br>
web: <a href=3D"http://www.ikr.uni-stuttgart.de" target=3D"_blank">www.ikr.=
uni-stuttgart.de</a><br>
-------------------------------------------------------------------<br>
</font></span><div class=3D""><div class=3D"h5">___________________________=
____________________<br>
aqm mailing list<br>
<a href=3D"mailto:aqm@ietf.org">aqm@ietf.org</a><br>
<a href=3D"https://www.ietf.org/mailman/listinfo/aqm" target=3D"_blank">htt=
ps://www.ietf.org/mailman/listinfo/aqm</a><br>
</div></div></blockquote></div><br></div></div>

--e89a8fb1ef427d9d1404d8d6459d--

From dpreed@reed.com  Tue Mar 26 09:46:31 2013
Return-Path: <dpreed@reed.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7D47A21F8A3E for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 09:46:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.371
X-Spam-Level: 
X-Spam-Status: No, score=-3.371 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4xP7lQQLlYIF for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 09:46:30 -0700 (PDT)
Received: from smtp196.iad.emailsrvr.com (smtp196.iad.emailsrvr.com [207.97.245.196]) by ietfa.amsl.com (Postfix) with ESMTP id 39B2821F8A41 for <aqm@ietf.org>; Tue, 26 Mar 2013 09:46:29 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp49.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id DB9C7190727; Tue, 26 Mar 2013 12:46:27 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy17.wa-web.iad1a (legacy17.wa-web.iad1a.rsapps.net [192.168.4.107]) by smtp49.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id C215E19070E; Tue, 26 Mar 2013 12:46:27 -0400 (EDT)
Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy17.wa-web.iad1a (Postfix) with ESMTP id ACD023B8050; Tue, 26 Mar 2013 12:46:27 -0400 (EDT)
Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com)  with HTTP; Tue, 26 Mar 2013 12:46:27 -0400 (EDT)
Date: Tue, 26 Mar 2013 12:46:27 -0400 (EDT)
From: dpreed@reed.com
To: "Mirja Kuehlewind" <mirja.kuehlewind@ikr.uni-stuttgart.de>
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_20130326124627000000_35170"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
In-Reply-To: <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de>
References: <mailman.5225.1363736608.3432.aqm@ietf.org>  <1363748540.64155384@apps.rackspace.com>  <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de>
Message-ID: <1364316387.705515611@apps.rackspace.com>
X-Mailer: webmail7.0
Cc: aqm@ietf.org
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 16:46:31 -0000

------=_20130326124627000000_35170
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0AAnd I would ask: why is this a feature?  It's a bug, IMHO.   The motivat=
ion, I understand.  The idea seems to be that the standing queue allows a "=
stateless" measurement to have enough information to assess what it might d=
o.   But why not just remember what packets had  passed through and had bee=
n dropped.  That would be like a "queue" without a queue's bad side-effect =
properties of holding up traffic and slowing recovery from the congested st=
ate.=0A =0AIt's like saying that to manage highway traffic congestion, firs=
t we must create a sustained traffic jam that never goes away, but adds sig=
nificant latency to every driver's trip - solely to measure congestion - be=
fore it becomes self-sustaining!=0A =0AThe problem with many actual situati=
ons (the ones where bufferbloat happens) is that there are processes that c=
reate self-sustaining congestion.=0A =0AThe best queue is the shortest poss=
ible queue, and there are techniques to achieve that - fq_codel with aggres=
sive dropping does not need to build up a standing queue to do its work.  E=
CN does. RED does.  They are therefore inferior on a key quality of experie=
nce metric.=0A =0AIn passing, let me point out what many have pointed out: =
 Optimizing TCP goodput is NOT the correct measure to apply to the Internet=
.   Yes, ECN is good for that.  But it does NOT fix the real user problems.=
  This is what Raj Jain didn't seem to understand: the Internet is about la=
tency, above all, because it is about both human-computer interaction and c=
omputer-computer (distributed systems) interaction.  It is not a set of "tu=
bes" that are unto themselves and measurable by goodput.=0A =0AThe optimal =
state of a highway system in terms of "cars per hour" (optimum goodput) is,=
 unfortunately, really bad for the cars - because it is a highway system in=
 a state of constant traffic jam, and where the latency of any particular c=
ar is not minimized - it is actually far, far from the minimum.=0A =0AThat =
being said, the Internet, though different from the highway system, shares =
this problem.  If you tune for "goodput", you will have a completely conges=
ted system, where users will *walk away* and abandon interactions leaving t=
hem in the queues.   We see that in HSPA and LTE networks today, live.  It'=
s not "interference" that causes this on wireless networks, it's been demon=
strated that it is queueing delay that is allowed to get too large, because=
 "goodput" is the only metric used.  Users don't want the last 10% of goodp=
ut, if it causes them a factor of 100 in response time.  (easily happening =
if there are 10 handshakes in a protocol step, and each is 10x longer from =
a latency perspective).=0A =0A-----Original Message-----=0AFrom: "Mirja Kue=
hlewind" <mirja.kuehlewind@ikr.uni-stuttgart.de>=0ASent: Tuesday, March 26,=
 2013 11:02am=0ATo: aqm@ietf.org=0ACc: dpreed@reed.com=0ASubject: Re: [aqm]=
 aqm Digest, Vol 1, Issue 7=0A=0A=0A=0AHi,=0A=0Ait is actually a feature of=
 AQM/ECN that you have small standing queues when =0Ahaving persistent flow=
s but short flow are able to fill the queue (for a =0Ashort period only) wi=
thout getting packet losses immediately!=0A=0AMirja=0A=0A=0AOn Wednesday 20=
 March 2013 04:02:20 dpreed@reed.com wrote:=0A> A problem with ECN is that =
it does not drain an already flooded queue, and=0A> depends (like RED) on t=
he idea that one has a large queue in normal=0A> operation.  It is not obvi=
ous that sustaining a deep queue is a desirable=0A> state, especially when =
there are highly variable capacities along any path,=0A> as is the case whe=
re one link is more than 2x slower than the rest of the=0A> links.=0A>=0A> =
If the dynamics of user loads are fractal, this will cause draining (and=0A=
> latency reduction) not to work.=0A>=0A> If user loads are stationary gaus=
sian and so forth, ECN can "tune" to=0A> relatively stable situations, but =
the queueing delay will be quite long=0A> end-to-end.=0A>=0A> Since there i=
s lots of evidence that user loads are fractal, dropping=0A> packets during=
 sudden bursts will prevent sustained queueing, whereas ECN=0A> will not.=
=0A>=0A> So a proper approach (in my opinion) on the "real" Internet (rathe=
r than a=0A> bunch of competing long-duration FTPs) requires keeping the qu=
eues so short=0A> that there is no real opportunity to benefit from ECN.=0A=
>=0A> Hence - we need to really see what happens with *real* traffic loads,=
 not=0A> simplified simulations.=0A>=0A> Real traffic loads are fractal, an=
d fractal loads do not generally obey the=0A> Gaussian style "law of large =
numbers" - multiple fractal flows are=0A> *rougher* not smoother than a sin=
gle one.=0A>=0A> So check your mathematical "intuitions" at the door.  Stop=
 writing=0A> equations based on tractable analytic models, because actual t=
raffic does=0A> not follow those models.=0A>=0A> Yeah, the Bell Heads belie=
ve in never dropping packets.  So what.  The=0A> Internet is not about maki=
ng them comfortable in their misunderstandings.=0A>=0A> -----Original Messa=
ge-----=0A> From: aqm-request@ietf.org=0A> Sent: Tuesday, March 19, 2013 7:=
43pm=0A> To: aqm@ietf.org=0A> Subject: aqm Digest, Vol 1, Issue 7=0A>=0A>=
=0A>=0A> If you have received this digest without all the individual messag=
e=0A> attachments you will need to update your digest options in your list=
=0A> subscription.  To do so, go to=0A>=0A> https://www.ietf.org/mailman/li=
stinfo/aqm=0A>=0A> Click the 'Unsubscribe or edit options' button, log in, =
and set "Get=0A> MIME or Plain Text Digests?" to MIME.  You can set this op=
tion=0A> globally for all the list digests you receive at this point.=0A>=
=0A>=0A>=0A> Send aqm mailing list submissions to=0A>  aqm@ietf.org=0A>=0A>=
 To subscribe or unsubscribe via the World Wide Web, visit=0A>  https://www=
.ietf.org/mailman/listinfo/aqm=0A> or, via email, send a message with subje=
ct or body 'help' to=0A>  aqm-request@ietf.org=0A>=0A> You can reach the pe=
rson managing the list at=0A>  aqm-owner@ietf.org=0A>=0A> When replying, pl=
ease edit your Subject line so it is more specific=0A> than "Re: Contents o=
f aqm digest..."=0A> Today's Topics:=0A>=0A>  1. Re: [tcpm] ECN support and=
 usage on the Internet (Yuchung Cheng)=0A>  2. Re: ECN support and usage on=
 the Internet (Fred Baker (fred))=0A>  3. Re: ??: [iccrg] [tcpm] ECN suppor=
t and usage on the Internet=0A>  (Jim Gettys)=0A>  4. Re: ??: [iccrg] [tcpm=
] ECN support and usage on the Internet=0A>  (Simon Barber)=0A> ___________=
____________________________________=0A> aqm mailing list=0A> aqm@ietf.org=
=0A> https://www.ietf.org/mailman/listinfo/aqm=0A=0A=0A=0A-- =0A-----------=
--------------------------------------------------------=0ADipl.-Ing. Mirja=
 K=C3=BChlewind=0AInstitute of Communication Networks and Computer Engineer=
ing (IKR)=0AUniversity of Stuttgart, Germany=0APfaffenwaldring 47, D-70569 =
Stuttgart=0A=0Atel: +49(0)711/685-67973=0Aemail: mirja.kuehlewind@ikr.uni-s=
tuttgart.de=0Aweb: www.ikr.uni-stuttgart.de=0A-----------------------------=
--------------------------------------
------=_20130326124627000000_35170
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"times new roman" size=3D"2"><p style=3D"margin:0;padding:0;">=
And I would ask: why is this a feature?&nbsp; It's a bug, IMHO.&nbsp;&nbsp;=
 The motivation, I understand.&nbsp; The idea seems to be that the standing=
 queue allows a "stateless" measurement to have enough information to asses=
s what it might do.&nbsp;&nbsp; But why not just remember what packets had&=
nbsp; passed through and had been dropped.&nbsp; That would be like a "queu=
e" without a queue's bad side-effect properties of holding up traffic and s=
lowing recovery from the congested state.</p>=0A<p style=3D"margin:0;paddin=
g:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">It's like saying that t=
o manage highway traffic congestion, first we must create a sustained traff=
ic jam that never goes away, but adds significant latency to every driver's=
 trip - solely to measure congestion - before it becomes self-sustaining!</=
p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;pad=
ding:0;">The problem with many actual situations (the ones where bufferbloa=
t happens) is that there are processes that create self-sustaining congesti=
on.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:=
0;padding:0;">The best queue is the shortest possible queue, and there are =
techniques to achieve that - fq_codel with aggressive dropping does not nee=
d to build up a standing queue to do its work.&nbsp; ECN does. RED does.&nb=
sp; They are therefore inferior on a key quality of experience metric.</p>=
=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;paddi=
ng:0;">In passing, let me point out what many have pointed out:&nbsp; Optim=
izing TCP goodput is NOT the correct measure to apply to the Internet.&nbsp=
;&nbsp; Yes, ECN is good for that.&nbsp; But it does NOT fix the real user =
problems.&nbsp; This is what Raj Jain didn't seem to understand: the Intern=
et is about latency, above all, because it is about both human-computer int=
eraction and computer-computer (distributed systems) interaction.&nbsp; It =
is not a set of "tubes" that are unto themselves and measurable by goodput.=
</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margin:0;p=
adding:0;">The optimal state of a highway system in terms of "cars per hour=
" (optimum goodput) is, unfortunately, really bad for the cars - because it=
 is a highway system in a state of constant traffic jam, and where the late=
ncy of any particular car is not minimized - it is actually far, far from t=
he minimum.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D=
"margin:0;padding:0;">That being said, the Internet, though different from =
the highway system, shares this problem.&nbsp; If you tune for "goodput", y=
ou will have a completely congested system, where users will *walk away* an=
d abandon interactions leaving them in the queues.&nbsp;&nbsp; We see that =
in HSPA and LTE networks today, live.&nbsp; It's not "interference" that ca=
uses this on wireless networks, it's been demonstrated that it is queueing =
delay that is allowed to get too large, because "goodput" is the only metri=
c used.&nbsp; Users don't want the last 10% of goodput, if it causes them a=
 factor of 100 in response time.&nbsp; (easily happening if there are 10 ha=
ndshakes in a protocol step, and each is 10x longer from a latency perspect=
ive).</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p style=3D"margi=
n:0;padding:0;">-----Original Message-----<br />From: "Mirja Kuehlewind" &l=
t;mirja.kuehlewind@ikr.uni-stuttgart.de&gt;<br />Sent: Tuesday, March 26, 2=
013 11:02am<br />To: aqm@ietf.org<br />Cc: dpreed@reed.com<br />Subject: Re=
: [aqm] aqm Digest, Vol 1, Issue 7<br /><br /></p>=0A<div id=3D"SafeStyles1=
364315554">=0A<p style=3D"margin:0;padding:0;">Hi,<br /><br />it is actuall=
y a feature of AQM/ECN that you have small standing queues when <br />havin=
g persistent flows but short flow are able to fill the queue (for a <br />s=
hort period only) without getting packet losses immediately!<br /><br />Mir=
ja<br /><br /><br />On Wednesday 20 March 2013 04:02:20 dpreed@reed.com wro=
te:<br />&gt; A problem with ECN is that it does not drain an already flood=
ed queue, and<br />&gt; depends (like RED) on the idea that one has a large=
 queue in normal<br />&gt; operation.  It is not obvious that sustaining a =
deep queue is a desirable<br />&gt; state, especially when there are highly=
 variable capacities along any path,<br />&gt; as is the case where one lin=
k is more than 2x slower than the rest of the<br />&gt; links.<br />&gt;<br=
 />&gt; If the dynamics of user loads are fractal, this will cause draining=
 (and<br />&gt; latency reduction) not to work.<br />&gt;<br />&gt; If user=
 loads are stationary gaussian and so forth, ECN can "tune" to<br />&gt; re=
latively stable situations, but the queueing delay will be quite long<br />=
&gt; end-to-end.<br />&gt;<br />&gt; Since there is lots of evidence that u=
ser loads are fractal, dropping<br />&gt; packets during sudden bursts will=
 prevent sustained queueing, whereas ECN<br />&gt; will not.<br />&gt;<br /=
>&gt; So a proper approach (in my opinion) on the "real" Internet (rather t=
han a<br />&gt; bunch of competing long-duration FTPs) requires keeping the=
 queues so short<br />&gt; that there is no real opportunity to benefit fro=
m ECN.<br />&gt;<br />&gt; Hence - we need to really see what happens with =
*real* traffic loads, not<br />&gt; simplified simulations.<br />&gt;<br />=
&gt; Real traffic loads are fractal, and fractal loads do not generally obe=
y the<br />&gt; Gaussian style "law of large numbers" - multiple fractal fl=
ows are<br />&gt; *rougher* not smoother than a single one.<br />&gt;<br />=
&gt; So check your mathematical "intuitions" at the door.  Stop writing<br =
/>&gt; equations based on tractable analytic models, because actual traffic=
 does<br />&gt; not follow those models.<br />&gt;<br />&gt; Yeah, the Bell=
 Heads believe in never dropping packets.  So what.  The<br />&gt; Internet=
 is not about making them comfortable in their misunderstandings.<br />&gt;=
<br />&gt; -----Original Message-----<br />&gt; From: aqm-request@ietf.org<=
br />&gt; Sent: Tuesday, March 19, 2013 7:43pm<br />&gt; To: aqm@ietf.org<b=
r />&gt; Subject: aqm Digest, Vol 1, Issue 7<br />&gt;<br />&gt;<br />&gt;<=
br />&gt; If you have received this digest without all the individual messa=
ge<br />&gt; attachments you will need to update your digest options in you=
r list<br />&gt; subscription.  To do so, go to<br />&gt;<br />&gt; https:/=
/www.ietf.org/mailman/listinfo/aqm<br />&gt;<br />&gt; Click the 'Unsubscri=
be or edit options' button, log in, and set "Get<br />&gt; MIME or Plain Te=
xt Digests?" to MIME.  You can set this option<br />&gt; globally for all t=
he list digests you receive at this point.<br />&gt;<br />&gt;<br />&gt;<br=
 />&gt; Send aqm mailing list submissions to<br />&gt;  aqm@ietf.org<br />&=
gt;<br />&gt; To subscribe or unsubscribe via the World Wide Web, visit<br =
/>&gt;  https://www.ietf.org/mailman/listinfo/aqm<br />&gt; or, via email, =
send a message with subject or body 'help' to<br />&gt;  aqm-request@ietf.o=
rg<br />&gt;<br />&gt; You can reach the person managing the list at<br />&=
gt;  aqm-owner@ietf.org<br />&gt;<br />&gt; When replying, please edit your=
 Subject line so it is more specific<br />&gt; than "Re: Contents of aqm di=
gest..."<br />&gt; Today's Topics:<br />&gt;<br />&gt;  1. Re: [tcpm] ECN s=
upport and usage on the Internet (Yuchung Cheng)<br />&gt;  2. Re: ECN supp=
ort and usage on the Internet (Fred Baker (fred))<br />&gt;  3. Re: ??: [ic=
crg] [tcpm] ECN support and usage on the Internet<br />&gt;  (Jim Gettys)<b=
r />&gt;  4. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet<b=
r />&gt;  (Simon Barber)<br />&gt; ________________________________________=
_______<br />&gt; aqm mailing list<br />&gt; aqm@ietf.org<br />&gt; https:/=
/www.ietf.org/mailman/listinfo/aqm<br /><br /><br /><br />-- <br />--------=
-----------------------------------------------------------<br />Dipl.-Ing.=
 Mirja K=C3=BChlewind<br />Institute of Communication Networks and Computer=
 Engineering (IKR)<br />University of Stuttgart, Germany<br />Pfaffenwaldri=
ng 47, D-70569 Stuttgart<br /><br />tel: +49(0)711/685-67973<br />email: mi=
rja.kuehlewind@ikr.uni-stuttgart.de<br />web: www.ikr.uni-stuttgart.de<br /=
>-------------------------------------------------------------------</p>=0A=
</div></font>
------=_20130326124627000000_35170--


From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 26 10:06:29 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9D3DC21F8BD7 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:06:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.675
X-Spam-Level: 
X-Spam-Status: No, score=-1.675 tagged_above=-999 required=5 tests=[AWL=0.347,  BAYES_00=-2.599, HELO_EQ_DE=0.35, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xMzMHess93Xc for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:06:28 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 5D8B721F8BBD for <aqm@ietf.org>; Tue, 26 Mar 2013 10:06:28 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id A002560275; Tue, 26 Mar 2013 18:06:26 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id 1A2976026E; Tue, 26 Mar 2013 18:06:26 +0100 (CET)
From: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Organization: University of Stuttgart (Germany), IKR
To: aqm@ietf.org
Date: Tue, 26 Mar 2013 18:06:25 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de> <1364316387.705515611@apps.rackspace.com>
In-Reply-To: <1364316387.705515611@apps.rackspace.com>
X-KMail-QuotePrefix: > 
MIME-Version: 1.0
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <201303261806.25456.mirja.kuehlewind@ikr.uni-stuttgart.de>
Cc: dpreed@reed.com
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 17:06:29 -0000

Hi,

you claim that the Internet is only about latency. That's is as wrong as to=
=20
say it's only about goodput. It really depends on the service you want to u=
se=20
over the Internet. Some might care about latency only and some about goodpu=
t=20
(as few as possible packet drops only). Today's Internet is mainly optimize=
d=20
for goodput though. To realize both requirement at the same time is actuall=
y=20
hard. What's the solution? Do we need two Internets?

Mirja


On Tuesday 26 March 2013 17:46:27 dpreed@reed.com wrote:
> And I would ask: why is this a feature?  It's a bug, IMHO.   The
> motivation, I understand.  The idea seems to be that the standing queue
> allows a "stateless" measurement to have enough information to assess what
> it might do.   But why not just remember what packets had  passed through
> and had been dropped.  That would be like a "queue" without a queue's bad
> side-effect properties of holding up traffic and slowing recovery from the
> congested state.
>
> It's like saying that to manage highway traffic congestion, first we must
> create a sustained traffic jam that never goes away, but adds significant
> latency to every driver's trip - solely to measure congestion - before it
> becomes self-sustaining!
>
> The problem with many actual situations (the ones where bufferbloat
> happens) is that there are processes that create self-sustaining
> congestion.
>
> The best queue is the shortest possible queue, and there are techniques to
> achieve that - fq_codel with aggressive dropping does not need to build up
> a standing queue to do its work.  ECN does. RED does.  They are therefore
> inferior on a key quality of experience metric.
>
> In passing, let me point out what many have pointed out:  Optimizing TCP
> goodput is NOT the correct measure to apply to the Internet.   Yes, ECN is
> good for that.  But it does NOT fix the real user problems.  This is what
> Raj Jain didn't seem to understand: the Internet is about latency, above
> all, because it is about both human-computer interaction and
> computer-computer (distributed systems) interaction.  It is not a set of
> "tubes" that are unto themselves and measurable by goodput.
>
> The optimal state of a highway system in terms of "cars per hour" (optimum
> goodput) is, unfortunately, really bad for the cars - because it is a
> highway system in a state of constant traffic jam, and where the latency =
of
> any particular car is not minimized - it is actually far, far from the
> minimum.
>
> That being said, the Internet, though different from the highway system,
> shares this problem.  If you tune for "goodput", you will have a complete=
ly
> congested system, where users will *walk away* and abandon interactions
> leaving them in the queues.   We see that in HSPA and LTE networks today,
> live.  It's not "interference" that causes this on wireless networks, it's
> been demonstrated that it is queueing delay that is allowed to get too
> large, because "goodput" is the only metric used.  Users don't want the
> last 10% of goodput, if it causes them a factor of 100 in response time.=
=20
> (easily happening if there are 10 handshakes in a protocol step, and each
> is 10x longer from a latency perspective).
>
> -----Original Message-----
> From: "Mirja Kuehlewind" <mirja.kuehlewind@ikr.uni-stuttgart.de>
> Sent: Tuesday, March 26, 2013 11:02am
> To: aqm@ietf.org
> Cc: dpreed@reed.com
> Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
>
>
>
> Hi,
>
> it is actually a feature of AQM/ECN that you have small standing queues
> when having persistent flows but short flow are able to fill the queue (f=
or
> a short period only) without getting packet losses immediately!
>
> Mirja
>
> On Wednesday 20 March 2013 04:02:20 dpreed@reed.com wrote:
> > A problem with ECN is that it does not drain an already flooded queue,
> > and depends (like RED) on the idea that one has a large queue in normal
> > operation.  It is not obvious that sustaining a deep queue is a desirab=
le
> > state, especially when there are highly variable capacities along any
> > path, as is the case where one link is more than 2x slower than the rest
> > of the links.
> >
> > If the dynamics of user loads are fractal, this will cause draining (and
> > latency reduction) not to work.
> >
> > If user loads are stationary gaussian and so forth, ECN can "tune" to
> > relatively stable situations, but the queueing delay will be quite long
> > end-to-end.
> >
> > Since there is lots of evidence that user loads are fractal, dropping
> > packets during sudden bursts will prevent sustained queueing, whereas E=
CN
> > will not.
> >
> > So a proper approach (in my opinion) on the "real" Internet (rather than
> > a bunch of competing long-duration FTPs) requires keeping the queues so
> > short that there is no real opportunity to benefit from ECN.
> >
> > Hence - we need to really see what happens with *real* traffic loads, n=
ot
> > simplified simulations.
> >
> > Real traffic loads are fractal, and fractal loads do not generally obey
> > the Gaussian style "law of large numbers" - multiple fractal flows are
> > *rougher* not smoother than a single one.
> >
> > So check your mathematical "intuitions" at the door.  Stop writing
> > equations based on tractable analytic models, because actual traffic do=
es
> > not follow those models.
> >
> > Yeah, the Bell Heads believe in never dropping packets.  So what.  The
> > Internet is not about making them comfortable in their misunderstanding=
s.
> >
> > -----Original Message-----
> > From: aqm-request@ietf.org
> > Sent: Tuesday, March 19, 2013 7:43pm
> > To: aqm@ietf.org
> > Subject: aqm Digest, Vol 1, Issue 7
> >
> >
> >
> > If you have received this digest without all the individual message
> > attachments you will need to update your digest options in your list
> > subscription.  To do so, go to
> >
> > https://www.ietf.org/mailman/listinfo/aqm
> >
> > Click the 'Unsubscribe or edit options' button, log in, and set "Get
> > MIME or Plain Text Digests?" to MIME.  You can set this option
> > globally for all the list digests you receive at this point.
> >
> >
> >
> > Send aqm mailing list submissions to
> >  aqm@ietf.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >  https://www.ietf.org/mailman/listinfo/aqm
> > or, via email, send a message with subject or body 'help' to
> >  aqm-request@ietf.org
> >
> > You can reach the person managing the list at
> >  aqm-owner@ietf.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of aqm digest..."
> > Today's Topics:
> >
> >  1. Re: [tcpm] ECN support and usage on the Internet (Yuchung Cheng)
> >  2. Re: ECN support and usage on the Internet (Fred Baker (fred))
> >  3. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
> >  (Jim Gettys)
> >  4. Re: ??: [iccrg] [tcpm] ECN support and usage on the Internet
> >  (Simon Barber)
> > _______________________________________________
> > aqm mailing list
> > aqm@ietf.org
> > https://www.ietf.org/mailman/listinfo/aqm



=2D-=20
=2D------------------------------------------------------------------
Dipl.-Ing. Mirja K=FChlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: mirja.kuehlewind@ikr.uni-stuttgart.de
web: www.ikr.uni-stuttgart.de
=2D------------------------------------------------------------------

From mirja.kuehlewind@ikr.uni-stuttgart.de  Tue Mar 26 10:25:50 2013
Return-Path: <mirja.kuehlewind@ikr.uni-stuttgart.de>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 086F921F8B45; Tue, 26 Mar 2013 10:25:50 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.762
X-Spam-Level: 
X-Spam-Status: No, score=-1.762 tagged_above=-999 required=5 tests=[AWL=0.260,  BAYES_00=-2.599, HELO_EQ_DE=0.35, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zx+qpugt7QSG; Tue, 26 Mar 2013 10:25:49 -0700 (PDT)
Received: from mailsrv.ikr.uni-stuttgart.de (mailsrv.ikr.uni-stuttgart.de [129.69.170.2]) by ietfa.amsl.com (Postfix) with ESMTP id 2ECB221F8A3F; Tue, 26 Mar 2013 10:25:48 -0700 (PDT)
Received: from netsrv1.ikr.uni-stuttgart.de (netsrv1-c [10.11.12.12]) by mailsrv.ikr.uni-stuttgart.de (Postfix) with ESMTP id 00A5660277; Tue, 26 Mar 2013 18:25:48 +0100 (CET)
Received: from vpn-2-cl195 (vpn-2-cl195 [10.41.21.195]) by netsrv1.ikr.uni-stuttgart.de (Postfix) with ESMTP id BA47360276; Tue, 26 Mar 2013 18:25:47 +0100 (CET)
From: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Organization: University of Stuttgart (Germany), IKR
To: aqm@ietf.org
Date: Tue, 26 Mar 2013 18:25:47 +0100
User-Agent: KMail/1.9.10 (enterprise35 0.20101217.1207316)
References: <51408BF4.7090304@cisco.com> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se>
In-Reply-To: <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se>
X-KMail-QuotePrefix: > 
MIME-Version: 1.0
Content-Type: Text/Plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <201303261825.47601.mirja.kuehlewind@ikr.uni-stuttgart.de>
Cc: bloat <bloat@lists.bufferbloat.net>, "tsvwg@ietf.org" <tsvwg@ietf.org>, Mikael Abrahamsson <swmike@swm.pp.se>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 17:25:50 -0000

Hi,

+1. Would be nice to have such statictics available for the user in every O=
S!

Mirja


On Friday 22 March 2013 05:27:31 Mikael Abrahamsson wrote:
> On Thu, 21 Mar 2013, Jim Gettys wrote:
> > Every more modern TCP can easily fill any sized buffer given time with a
> > single TCP connection.
>
> I agree with this, I made this discovery myself back in 2004 or so, and
> had to implement Fairqueue and WRED on my home connection to make it
> bearable to use any interactive application while transferring files.
>
> In IETF75 in Stockholm in 2009, I made proposals in both TCP and at open
> mic in one of the sessions, that I would like to see statistics and
> performance numbers on packet loss, delay variation etc from actual
> traffic. The IP stack has great insight in what the network conditions are
> (especially with TCP Timestamping), but as far as I know it's not really
> exported in any usable format to the user. My idea was to have some kind
> of dashboard for the user to show if currently the network was the
> limiting factor, if the tcp window was maxed out etc. Would also be nice
> if there was output that could be cut/pasted and attached to a fault
> report in case the customer talks to customer support. It would be good if
> this was actually a standard so all OSes did the same.
>
> I am not aware of any such work going on, so I'd like to know if anyone
> else is aware of work in this area?



=2D-=20
=2D------------------------------------------------------------------
Dipl.-Ing. Mirja K=FChlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: mirja.kuehlewind@ikr.uni-stuttgart.de
web: www.ikr.uni-stuttgart.de
=2D------------------------------------------------------------------

From swmike@swm.pp.se  Tue Mar 26 10:27:38 2013
Return-Path: <swmike@swm.pp.se>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AC80221F8BF6 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:27:38 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.372
X-Spam-Level: 
X-Spam-Status: No, score=-2.372 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7b1hw64pnAlO for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:27:38 -0700 (PDT)
Received: from uplift.swm.pp.se (ipv6.swm.pp.se [IPv6:2a00:801::f]) by ietfa.amsl.com (Postfix) with ESMTP id 21AEA21F8BCD for <aqm@ietf.org>; Tue, 26 Mar 2013 10:27:37 -0700 (PDT)
Received: by uplift.swm.pp.se (Postfix, from userid 501) id 7E4BD9C; Tue, 26 Mar 2013 18:27:36 +0100 (CET)
Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 7776C9A for <aqm@ietf.org>; Tue, 26 Mar 2013 18:27:36 +0100 (CET)
Date: Tue, 26 Mar 2013 18:27:36 +0100 (CET)
From: Mikael Abrahamsson <swmike@swm.pp.se>
To: aqm@ietf.org
In-Reply-To: <1364316387.705515611@apps.rackspace.com>
Message-ID: <alpine.DEB.2.00.1303261817321.11255@uplift.swm.pp.se>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de> <1364316387.705515611@apps.rackspace.com>
User-Agent: Alpine 2.00 (DEB 1167 2008-08-23)
Organization: People's Front Against WWW
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 17:27:38 -0000

On Tue, 26 Mar 2013, dpreed@reed.com wrote:

> The optimal state of a highway system in terms of "cars per hour" 
> (optimum goodput) is, unfortunately, really bad for the cars - because 
> it is a highway system in a state of constant traffic jam, and where the 
> latency of any particular car is not minimized - it is actually far, far 
> from the minimum.

Analogies are usually hard, but I have been told by a traffic system 
professor that the optimal cars/hour speed is ~70 km/h, that's when the 
flow is the highest. Faster than that, and cars increase the inter-car 
distance more than the cars/second increases, slower than that and flow 
suffers. So as traffic increases from an nearly empty road, speed 
decreases until everybody drives ~70 km/h, then any small event causes the 
whole system to break down and traffic starts to back up.

Now, I don't have a good analogy to this and fq_codel but giving every 
flow "fair access" to the medium sounds to me like a good solution. I 
don't really care if I for some combination of flows and latencies can 
only use 80-90% of theoretical maximum, it's better to keep performance 
under high load fairly good than to wreck interactive flows in name of 
100% usage for aggressive flows.

I have 10 years experience in trying to fault find performance problems 
for customers in real life. I have been using fair-queue for 8 years on my 
home connection because I don't like bufferbloat (200ms of buffering was 
quite common even on my fairly high capacity connection). It uses quite a 
lot of CPU on cisco routers, so fq_codel sounds to me to be a great 
compromise between simplicity and still getting the job done. However, I'd 
imagine it's the "fq" part that is of more importance than "codel", but 
together they sound like a great combination.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

From rs@netapp.com  Tue Mar 26 10:42:10 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9407E21F8C04 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:42:10 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -9.886
X-Spam-Level: 
X-Spam-Status: No, score=-9.886 tagged_above=-999 required=5 tests=[AWL=0.486,  BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id q1VawRUrmr5o for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 10:42:09 -0700 (PDT)
Received: from mx1.netapp.com (mx1.netapp.com [216.240.18.38]) by ietfa.amsl.com (Postfix) with ESMTP id 5273121F8B7E for <aqm@ietf.org>; Tue, 26 Mar 2013 10:42:09 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,913,1355126400"; d="scan'208";a="248947981"
Received: from smtp2.corp.netapp.com ([10.57.159.114]) by mx1-out.netapp.com with ESMTP; 26 Mar 2013 10:42:08 -0700
Received: from vmwexceht05-prd.hq.netapp.com (vmwexceht05-prd.hq.netapp.com [10.106.77.35]) by smtp2.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2QHg76g006714; Tue, 26 Mar 2013 10:42:07 -0700 (PDT)
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht05-prd.hq.netapp.com ([10.106.77.35]) with mapi id 14.02.0342.003; Tue, 26 Mar 2013 10:42:07 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: "dpreed@reed.com" <dpreed@reed.com>, Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
Thread-Topic: [aqm] aqm Digest, Vol 1, Issue 7
Thread-Index: AQHOJRdZaSSArZKizU6GFDk3nJ5O25i4kQqAgAAdCoD//5D4wA==
Date: Tue, 26 Mar 2013 17:42:06 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24ACFA57@SACEXCMBX02-PRD.hq.netapp.com>
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de> <1364316387.705515611@apps.rackspace.com>
In-Reply-To: <1364316387.705515611@apps.rackspace.com>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Cc: "aqm@ietf.org" <aqm@ietf.org>
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 17:42:10 -0000

SGksDQoNCg0KPiBGcm9tOiBhcW0tYm91bmNlc0BpZXRmLm9yZyBbbWFpbHRvOmFxbS1ib3VuY2Vz
QGlldGYub3JnXSBPbiBCZWhhbGYgT2YgZHByZWVkQHJlZWQuY29tDQo+DQo+DQo+IEFuZCBJIHdv
dWxkIGFzazogd2h5IGlzIHRoaXMgYSBmZWF0dXJlP8KgIEl0J3MgYSBidWcsIElNSE8uwqDCoCAN
Cg0KWW91IG1pZ2h0IGhhdmUgdGhlIHdyb25nIG1ldGFwaG9yIGluIG1pbmQgdG8gY29tZSB0byB0
aGlzIGNvbmNsdXNpb24uLi4NCg0KPiBUaGUgbW90aXZhdGlvbiwgSSANCj4gdW5kZXJzdGFuZC7C
oCBUaGUgaWRlYSBzZWVtcyB0byBiZSB0aGF0IHRoZSBzdGFuZGluZyBxdWV1ZSBhbGxvd3MgYSAi
c3RhdGVsZXNzIg0KPiBtZWFzdXJlbWVudCB0byBoYXZlIGVub3VnaCBpbmZvcm1hdGlvbiB0byBh
c3Nlc3Mgd2hhdCBpdCBtaWdodCBkby7CoMKgIA0KDQpObywgYSBzbWFsbCBzdGFuZGluZyBxdWV1
ZSAoYW5kIHNtYWxsIG1lYW5zIGFuIGF2ZXJhZ2UgcXVldWUgbGVuZ3RoIG9mIDErLTEgcGFja2V0
KSBpcyBuZWNlc3NhcnkgdG8gZnVsbHkgdXRpbGl6ZSBhIHBhY2tldCBuZXR3b3JrLg0KDQo+IEJ1
dCB3aHkgbm90IGp1c3QgDQo+IHJlbWVtYmVyIHdoYXQgcGFja2V0cyBoYWTCoCBwYXNzZWQgdGhy
b3VnaCBhbmQgaGFkIGJlZW4gZHJvcHBlZC7CoCBUaGF0IHdvdWxkIGJlIGxpa2UgYSANCj4gInF1
ZXVlIiB3aXRob3V0IGEgcXVldWUncyBiYWQgc2lkZS1lZmZlY3QgcHJvcGVydGllcyBvZiBob2xk
aW5nIHVwIHRyYWZmaWMgYW5kIHNsb3dpbmcgDQo+IHJlY292ZXJ5IGZyb20gdGhlIGNvbmdlc3Rl
ZCBzdGF0ZS4NCg0KSSB0aGluayB0aGlzIGNvbmNlcHQgaXMgY2FsbGVkIHZpcnR1YWwgcXVldWUg
LSB5b3UgZHJhaW4gYSBzaGFkb3cgKHZpcnR1YWwpIHF1ZXVlIGF0IHNsaWdodGx5IGxvd2VyIHJh
dGUgKGllLiA5NS05OSUpIHRoYW4gdGhlIHBoeXNpY2FsIG9uZSwgdGh1cyBsZWdhY3kgQVFNIGxp
a2UgUkVEIGNhbiByZWFjdCAqYmVmb3JlKiB0aGUgYWN0dWFsIHF1ZXVlIHN0YXJ0cyBidWlsZGlu
Zy4gSG93ZXZlciwgdGhpcyBwcm9wZXJ0eSBpcyBib3VnaHQgYnkgbGVzcyB0aGFuIGZ1bGwgdXRp
bGl6YXRpb24gKGluIHN0ZWFkeSBzdGF0ZSkgb2YgdGhlIG5ldHdvcmsuLi4NCg0KPsKgDQo+IEl0
J3MgbGlrZSBzYXlpbmcgdGhhdCB0byBtYW5hZ2UgaGlnaHdheSB0cmFmZmljIGNvbmdlc3Rpb24s
IGZpcnN0IHdlIG11c3QgY3JlYXRlIGEgDQo+IHN1c3RhaW5lZCB0cmFmZmljIGphbSB0aGF0IG5l
dmVyIGdvZXMgYXdheSwgYnV0IGFkZHMgc2lnbmlmaWNhbnQgbGF0ZW5jeSB0byBldmVyeSANCj4g
ZHJpdmVyJ3MgdHJpcCAtIHNvbGVseSB0byBtZWFzdXJlIGNvbmdlc3Rpb24gLSBiZWZvcmUgaXQg
YmVjb21lcyBzZWxmLXN1c3RhaW5pbmchDQo+DQoNCkFzIFZhbiBwb2ludGVkIG91dCwgY2FycyBv
biBhIGhpZ2h3YXkgYXJlIG5vdCBuZWNlc3NhcmlseSB0aGUgY29ycmVjdCBtZXRhcGhvciB0byBn
cmFzcCB3aGF0cyBnb2luZyBvbiBpbiBwYWNrZXQgbmV0d29ya3MuDQoNCmh0dHA6Ly93d3cuaWV0
Zi5vcmcvcHJvY2VlZGluZ3MvODQvc2xpZGVzL3NsaWRlcy04NC10c3ZhcmVhLTQucGRmDQoNCj4N
Cj4gVGhlIHByb2JsZW0gd2l0aCBtYW55IGFjdHVhbCBzaXR1YXRpb25zICh0aGUgb25lcyB3aGVy
ZSBidWZmZXJibG9hdCBoYXBwZW5zKSBpcyB0aGF0IA0KPiB0aGVyZSBhcmUgcHJvY2Vzc2VzIHRo
YXQgY3JlYXRlIHNlbGYtc3VzdGFpbmluZyBjb25nZXN0aW9uLg0KDQpXZWxsLCBvbmUgY2FuIGFs
c28gc3RhdGUsIHRoYXQgaWYgdGhlc2UgcHJvY2Vzc2VzIGFyZSByZWFjdGl2ZSwgdGhleSBkb24n
dCBnZXQgdGltZWx5IGZlZWRiYWNrIHRvIGF2b2lkIHRoaXMgZnJvbSBoYXBwZW5pbmcuIChUaGVy
ZSB3aWxsIGFsd2F5cyBiZSBub24tcmVzcG9uc2l2ZSBVRFAgYmxhc3RzIHRoYXQgY2Fubid0IGJl
IGN1cmVkIHJlYWxseSkuDQoNCg0KPiBUaGUgYmVzdCBxdWV1ZSBpcyB0aGUgc2hvcnRlc3QgcG9z
c2libGUgcXVldWUsIGFuZCB0aGVyZSBhcmUgdGVjaG5pcXVlcyB0byBhY2hpZXZlIHRoYXQNCj4g
LSBmcV9jb2RlbCB3aXRoIGFnZ3Jlc3NpdmUgZHJvcHBpbmcgZG9lcyBub3QgbmVlZCB0byBidWls
ZCB1cCBhIHN0YW5kaW5nIHF1ZXVlIHRvIGRvDQo+IGl0cyB3b3JrLsKgIEVDTiBkb2VzLiBSRUQg
ZG9lcy7CoCBUaGV5IGFyZSB0aGVyZWZvcmUgaW5mZXJpb3Igb24gYSBrZXkgcXVhbGl0eSBvZg0K
PiBleHBlcmllbmNlIG1ldHJpYy4NCg0KRUNOIGJ5IGl0c2VsZiBpcyBub3QgYSBBUU0gc2NoZW1l
IGJ1dCBvbmx5IGEgc2lnbmFsaW5nIHNjaGVtZS4gQWxzbywgdGhlIEFRTSBzY2hlbWUgZG9lc24n
dCBwcmV2ZW50IHRoZSBidWlsZHVwIC0gaXQncyBhIHJlYWN0aXZlIHNlbmRlciAod2hpY2ggcmVh
Y3RzIHByb3Blcmx5IHRvIGEgc2lnbmFsIGZyb20gYW4gQVFNKSB0aGF0IHJlc3VsdHMgaW4gbG93
IHF1ZXVlIG9jY3VwYW5jeS4NCg0KTm93LCB3ZSBjYW4gYXJndWUgaWYgZm9yd2FyZC0gKEVDTiBs
b29wKSBvciBiYWNrLXNpZ25hbGluZyAoODAyLjFRIENOKSB0aGUgY29uZ2VzdGlvbiBzaWduYWwg
aXMgYmV0dGVyLiBCdXQgd2l0aG91dCBhIHJlYWN0aXZlIHNlbmRlciAoc3VjaCBhcyBUQ1AsIFND
VFAsIERDQ1AsIExlZGJhdCwgLi4uKSBhbGwgdGhhdCBpcyBpbiB2YWluLCBhbmQgdGhlIEFRTSB3
aWxsIGhhdmUgdG8gZHJvcCAoZXhjZXNzaXZlKSBhbW91bnRzIG9mIHBhY2tldHMuDQoNCsKgDQo+
IEluIHBhc3NpbmcsIGxldCBtZSBwb2ludCBvdXQgd2hhdCBtYW55IGhhdmUgcG9pbnRlZCBvdXQ6
wqAgT3B0aW1pemluZyBUQ1AgZ29vZHB1dCBpcyBOT1QgDQo+IHRoZSBjb3JyZWN0IG1lYXN1cmUg
dG8gYXBwbHkgdG8gdGhlIEludGVybmV0LsKgwqAgWWVzLCBFQ04gaXMgZ29vZCBmb3IgdGhhdC7C
oCBCdXQgaXQgZG9lcyANCj4gTk9UIGZpeCB0aGUgcmVhbCB1c2VyIHByb2JsZW1zLsKgIFRoaXMg
aXMgd2hhdCBSYWogSmFpbiBkaWRuJ3Qgc2VlbSB0byB1bmRlcnN0YW5kOiB0aGUgDQo+IEludGVy
bmV0IGlzIGFib3V0IGxhdGVuY3ksIGFib3ZlIGFsbCwgYmVjYXVzZSBpdCBpcyBhYm91dCBib3Ro
IGh1bWFuLWNvbXB1dGVyIGludGVyYWN0aW9uDQo+IGFuZCBjb21wdXRlci1jb21wdXRlciAoZGlz
dHJpYnV0ZWQgc3lzdGVtcykgaW50ZXJhY3Rpb24uwqAgSXQgaXMgbm90IGEgc2V0IG9mICJ0dWJl
cyIgdGhhdA0KPiBhcmUgdW50byB0aGVtc2VsdmVzIGFuZCBtZWFzdXJhYmxlIGJ5IGdvb2RwdXQu
DQoNCkFncmVlZCAtIGZpeGluZyBUQ1AgZ29vZHB1dCBieSBpdHNlbGYgKHVzZSBvZiBFQ04gd2l0
aCBzb21lIGxlZ2FjeSAiQSJRTSBzY2hlbWUgbGlrZSBSRUQgb3IgIk1hcmsgYWJvdmUgdGhyZXNo
b2xkIikgZG9lc24ndCBoZWxwLiBCdXQgZHJpdmluZyBkb3duIHN0YW5kaW5nIHF1ZXVlcyBjbG9z
ZSB0byB0aGUgb3B0aW11bSAoMSstMSBwYWNrZXQpIG9wZXJhdGluZyBwb2ludCBkb2VzLi4uIEVD
TiBpcyBhIG1lYW5zLCBub3QgYW4gZW5kLg0KDQoNCj4gVGhlIG9wdGltYWwgc3RhdGUgb2YgYSBo
aWdod2F5IHN5c3RlbSBpbiB0ZXJtcyBvZiAiY2FycyBwZXIgaG91ciIgKG9wdGltdW0gZ29vZHB1
dCkgaXMsIA0KPiB1bmZvcnR1bmF0ZWx5LCByZWFsbHkgYmFkIGZvciB0aGUgY2FycyAtIGJlY2F1
c2UgaXQgaXMgYSBoaWdod2F5IHN5c3RlbSBpbiBhIHN0YXRlIG9mIA0KPiBjb25zdGFudCB0cmFm
ZmljIGphbSwgYW5kIHdoZXJlIHRoZSBsYXRlbmN5IG9mIGFueSBwYXJ0aWN1bGFyIGNhciBpcyBu
b3QgbWluaW1pemVkIC0gaXQgDQo+IGlzIGFjdHVhbGx5IGZhciwgZmFyIGZyb20gdGhlIG1pbmlt
dW0uDQoNCkFnYWluLCB3cm9uZyBtZXRhcGhvci4gQWxzbywgdGhpcyBwaWN0dXJlIGlzIHdyb25n
IGluc29mYXIsIGFzIHRoZSBzYW1lIG51bWJlciBvZiBjYXJzIHBlciB1bml0IHRpbWUgY2FuIHBh
c3MgYSBnaXZlbiBtYXJrIGFzIHZhc3RseSBkaWZmZXJlbnQgc3BlZWRzIChzYXksIDUga3BoIGdy
aWRsb2NrLCBhbmQgODAga3BoIGZsb3dpbmcgdHJhZmZpYzsgZGlzdGFuY2UgYmV0d2VlbiBjYXJz
IGEgbWVhZ2VyIDNtIGZpcnN0LCBhbmQgZ3JhY2lvdXMgMTIwbSBpbiB0aGUgb3RoZXIgY2FzZSku
DQoNCkFzIEZyZWQgc3RhdGVkIGVhcmxpZXIsIHRoZSBvcHRpbXVtIHN0YXRlIG9mIGEgcGFja2V0
IG5ldHdvcmsgaXMgY2xvc2UgdG8gdGhlIGtuZWUgKHNtYWxsIHN0YW5kaW5nIHF1ZXVlcyksIG5v
dCBjbG9zZSB0byB0aGUgY2xpZmYuIEl0J3MgdGhlIGpvYiBvZiBhIGdvb2QgQVFNLCB0byBzaWdu
YWwsIGJ5IG1lYW5zIG9mIGxvc3MsIGRlbGF5IGFuZCBFQ04sIHRoYXQga25lZSBhcmVhIHRvIGEg
cmVhY3RpdmUgc2VuZGVyLCBhbmQgcHJvdGVjdCB0aGUgbmV0d29yayBmcm9tIG5vbi1yZWFjdGl2
ZSBzZW5kZXJzLi4uDQoNCj7CoA0KPiBUaGF0IGJlaW5nIHNhaWQsIHRoZSBJbnRlcm5ldCwgdGhv
dWdoIGRpZmZlcmVudCBmcm9tIHRoZSBoaWdod2F5IHN5c3RlbSwgc2hhcmVzIHRoaXMgDQo+IHBy
b2JsZW0uwqAgSWYgeW91IHR1bmUgZm9yICJnb29kcHV0IiwgeW91IHdpbGwgaGF2ZSBhIGNvbXBs
ZXRlbHkgY29uZ2VzdGVkIHN5c3RlbSwgd2hlcmUgDQo+IHVzZXJzIHdpbGwgKndhbGsgYXdheSog
YW5kIGFiYW5kb24gaW50ZXJhY3Rpb25zIGxlYXZpbmcgdGhlbSBpbiB0aGUgcXVldWVzLsKgwqAg
DQoNCkkgZGlzYWdyZWUuIElmIHlvdSB0dW5lIGZvciB0aG91Z2hwdXQsIHlvdSBlbmQgdXAgd2l0
aCBhIGNvbXBsZXRlbHkgY29uZ2VzdGVkIHN5c3RlbSwgd2l0aCBlZmZlY3RpdmUgemVybyBnb29k
cHV0LiBJZiB5b3UgdHVuZSBmb3IgZ29vZHB1dCwgeW91IGFsc28gYWRkcmVzcyBsYXRlbmN5IGFu
ZCB1dGlsaXphdGlvbiBpc3N1ZXMuDQoNCj4gV2Ugc2VlIHRoYXQNCj4gaW4gSFNQQSBhbmQgTFRF
IG5ldHdvcmtzIHRvZGF5LCBsaXZlLsKgIEl0J3Mgbm90ICJpbnRlcmZlcmVuY2UiIHRoYXQgY2F1
c2VzIHRoaXMgb24gDQo+IHdpcmVsZXNzIG5ldHdvcmtzLCBpdCdzIGJlZW4gZGVtb25zdHJhdGVk
IHRoYXQgaXQgaXMgcXVldWVpbmcgZGVsYXkgdGhhdCBpcyBhbGxvd2VkIHRvIA0KPiBnZXQgdG9v
IGxhcmdlLCBiZWNhdXNlICJnb29kcHV0IiBpcyB0aGUgb25seSBtZXRyaWMgdXNlZC7CoCBVc2Vy
cyBkb24ndCB3YW50IHRoZSBsYXN0IDEwJSANCj4gb2YgZ29vZHB1dCwgaWYgaXQgY2F1c2VzIHRo
ZW0gYSBmYWN0b3Igb2YgMTAwIGluIHJlc3BvbnNlIHRpbWUuwqAgKGVhc2lseSBoYXBwZW5pbmcg
aWYgDQo+IHRoZXJlIGFyZSAxMCBoYW5kc2hha2VzIGluIGEgcHJvdG9jb2wgc3RlcCwgYW5kIGVh
Y2ggaXMgMTB4IGxvbmdlciBmcm9tIGEgbGF0ZW5jeSANCj4gcGVyc3BlY3RpdmUpLg0KDQpVbmZv
cnR1bmF0ZWx5LCAiVXNlcnMiIHVzZSB0b29scyBsaWtlIHNwZWVkIHRlc3QsIHdoaWNoIGFyZSBj
b21wbGV0ZWx5IGFnbm9zdGljIHRvIGxhdGVuY3kgZWZmZWN0cyBhbmQgZ28gZm9yIHRocm91Z2hw
dXQgKG5vdCBnb29kcHV0KS4NCg0KSSBiZWxpZXZlIEppbSB3YW50ZWQgdG8gYWRkcmVzcyB0aGUg
c2ltdWxhdGVub3VzIG1lYXN1cmVtZW50IG9mIHRocm91Z2hwdXQgKGJpdHMvcykgdG9nZXRoZXIg
d2l0aCBsYXRlbmN5IHdoaWxlIHRlc3RpbmcuLi4gQWxsIHBvcHVsYXIgdG9vbHMgSSBrbm93IHRv
ZGF5IG1lYXN1cmUgKGljbXAgcGluZykgbGF0ZW5jeSBzZXBhcmF0ZSBmcm9tIHRoZSAodGNwL3Vk
cCkgdHJhZmZpYyBmbG93Li4uDQoNClJlZ2FyZHMsDQogICBSaWNoYXJkDQoNCsKgDQotLS0tLU9y
aWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogIk1pcmphIEt1ZWhsZXdpbmQiIDxtaXJqYS5rdWVo
bGV3aW5kQGlrci51bmktc3R1dHRnYXJ0LmRlPg0KU2VudDogVHVlc2RheSwgTWFyY2ggMjYsIDIw
MTMgMTE6MDJhbQ0KVG86IGFxbUBpZXRmLm9yZw0KQ2M6IGRwcmVlZEByZWVkLmNvbQ0KU3ViamVj
dDogUmU6IFthcW1dIGFxbSBEaWdlc3QsIFZvbCAxLCBJc3N1ZSA3DQpIaSwNCg0KaXQgaXMgYWN0
dWFsbHkgYSBmZWF0dXJlIG9mIEFRTS9FQ04gdGhhdCB5b3UgaGF2ZSBzbWFsbCBzdGFuZGluZyBx
dWV1ZXMgd2hlbiANCmhhdmluZyBwZXJzaXN0ZW50IGZsb3dzIGJ1dCBzaG9ydCBmbG93IGFyZSBh
YmxlIHRvIGZpbGwgdGhlIHF1ZXVlIChmb3IgYSANCnNob3J0IHBlcmlvZCBvbmx5KSB3aXRob3V0
IGdldHRpbmcgcGFja2V0IGxvc3NlcyBpbW1lZGlhdGVseSENCg0KTWlyamENCg0KDQpPbiBXZWRu
ZXNkYXkgMjAgTWFyY2ggMjAxMyAwNDowMjoyMCBkcHJlZWRAcmVlZC5jb20gd3JvdGU6DQo+IEEg
cHJvYmxlbSB3aXRoIEVDTiBpcyB0aGF0IGl0IGRvZXMgbm90IGRyYWluIGFuIGFscmVhZHkgZmxv
b2RlZCBxdWV1ZSwgYW5kDQo+IGRlcGVuZHMgKGxpa2UgUkVEKSBvbiB0aGUgaWRlYSB0aGF0IG9u
ZSBoYXMgYSBsYXJnZSBxdWV1ZSBpbiBub3JtYWwNCj4gb3BlcmF0aW9uLiBJdCBpcyBub3Qgb2J2
aW91cyB0aGF0IHN1c3RhaW5pbmcgYSBkZWVwIHF1ZXVlIGlzIGEgZGVzaXJhYmxlDQo+IHN0YXRl
LCBlc3BlY2lhbGx5IHdoZW4gdGhlcmUgYXJlIGhpZ2hseSB2YXJpYWJsZSBjYXBhY2l0aWVzIGFs
b25nIGFueSBwYXRoLA0KPiBhcyBpcyB0aGUgY2FzZSB3aGVyZSBvbmUgbGluayBpcyBtb3JlIHRo
YW4gMnggc2xvd2VyIHRoYW4gdGhlIHJlc3Qgb2YgdGhlDQo+IGxpbmtzLg0KPg0KPiBJZiB0aGUg
ZHluYW1pY3Mgb2YgdXNlciBsb2FkcyBhcmUgZnJhY3RhbCwgdGhpcyB3aWxsIGNhdXNlIGRyYWlu
aW5nIChhbmQNCj4gbGF0ZW5jeSByZWR1Y3Rpb24pIG5vdCB0byB3b3JrLg0KPg0KPiBJZiB1c2Vy
IGxvYWRzIGFyZSBzdGF0aW9uYXJ5IGdhdXNzaWFuIGFuZCBzbyBmb3J0aCwgRUNOIGNhbiAidHVu
ZSIgdG8NCj4gcmVsYXRpdmVseSBzdGFibGUgc2l0dWF0aW9ucywgYnV0IHRoZSBxdWV1ZWluZyBk
ZWxheSB3aWxsIGJlIHF1aXRlIGxvbmcNCj4gZW5kLXRvLWVuZC4NCj4NCj4gU2luY2UgdGhlcmUg
aXMgbG90cyBvZiBldmlkZW5jZSB0aGF0IHVzZXIgbG9hZHMgYXJlIGZyYWN0YWwsIGRyb3BwaW5n
DQo+IHBhY2tldHMgZHVyaW5nIHN1ZGRlbiBidXJzdHMgd2lsbCBwcmV2ZW50IHN1c3RhaW5lZCBx
dWV1ZWluZywgd2hlcmVhcyBFQ04NCj4gd2lsbCBub3QuDQo+DQo+IFNvIGEgcHJvcGVyIGFwcHJv
YWNoIChpbiBteSBvcGluaW9uKSBvbiB0aGUgInJlYWwiIEludGVybmV0IChyYXRoZXIgdGhhbiBh
DQo+IGJ1bmNoIG9mIGNvbXBldGluZyBsb25nLWR1cmF0aW9uIEZUUHMpIHJlcXVpcmVzIGtlZXBp
bmcgdGhlIHF1ZXVlcyBzbyBzaG9ydA0KPiB0aGF0IHRoZXJlIGlzIG5vIHJlYWwgb3Bwb3J0dW5p
dHkgdG8gYmVuZWZpdCBmcm9tIEVDTi4NCj4NCj4gSGVuY2UgLSB3ZSBuZWVkIHRvIHJlYWxseSBz
ZWUgd2hhdCBoYXBwZW5zIHdpdGggKnJlYWwqIHRyYWZmaWMgbG9hZHMsIG5vdA0KPiBzaW1wbGlm
aWVkIHNpbXVsYXRpb25zLg0KPg0KPiBSZWFsIHRyYWZmaWMgbG9hZHMgYXJlIGZyYWN0YWwsIGFu
ZCBmcmFjdGFsIGxvYWRzIGRvIG5vdCBnZW5lcmFsbHkgb2JleSB0aGUNCj4gR2F1c3NpYW4gc3R5
bGUgImxhdyBvZiBsYXJnZSBudW1iZXJzIiAtIG11bHRpcGxlIGZyYWN0YWwgZmxvd3MgYXJlDQo+
ICpyb3VnaGVyKiBub3Qgc21vb3RoZXIgdGhhbiBhIHNpbmdsZSBvbmUuDQo+DQo+IFNvIGNoZWNr
IHlvdXIgbWF0aGVtYXRpY2FsICJpbnR1aXRpb25zIiBhdCB0aGUgZG9vci4gU3RvcCB3cml0aW5n
DQo+IGVxdWF0aW9ucyBiYXNlZCBvbiB0cmFjdGFibGUgYW5hbHl0aWMgbW9kZWxzLCBiZWNhdXNl
IGFjdHVhbCB0cmFmZmljIGRvZXMNCj4gbm90IGZvbGxvdyB0aG9zZSBtb2RlbHMuDQo+DQo+IFll
YWgsIHRoZSBCZWxsIEhlYWRzIGJlbGlldmUgaW4gbmV2ZXIgZHJvcHBpbmcgcGFja2V0cy4gU28g
d2hhdC4gVGhlDQo+IEludGVybmV0IGlzIG5vdCBhYm91dCBtYWtpbmcgdGhlbSBjb21mb3J0YWJs
ZSBpbiB0aGVpciBtaXN1bmRlcnN0YW5kaW5ncy4NCj4NCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdl
LS0tLS0NCj4gRnJvbTogYXFtLXJlcXVlc3RAaWV0Zi5vcmcNCj4gU2VudDogVHVlc2RheSwgTWFy
Y2ggMTksIDIwMTMgNzo0M3BtDQo+IFRvOiBhcW1AaWV0Zi5vcmcNCj4gU3ViamVjdDogYXFtIERp
Z2VzdCwgVm9sIDEsIElzc3VlIDcNCj4NCj4NCj4NCj4gSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhp
cyBkaWdlc3Qgd2l0aG91dCBhbGwgdGhlIGluZGl2aWR1YWwgbWVzc2FnZQ0KPiBhdHRhY2htZW50
cyB5b3Ugd2lsbCBuZWVkIHRvIHVwZGF0ZSB5b3VyIGRpZ2VzdCBvcHRpb25zIGluIHlvdXIgbGlz
dA0KPiBzdWJzY3JpcHRpb24uIFRvIGRvIHNvLCBnbyB0bw0KPg0KPiBodHRwczovL3d3dy5pZXRm
Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL2FxbQ0KPg0KPiBDbGljayB0aGUgJ1Vuc3Vic2NyaWJlIG9y
IGVkaXQgb3B0aW9ucycgYnV0dG9uLCBsb2cgaW4sIGFuZCBzZXQgIkdldA0KPiBNSU1FIG9yIFBs
YWluIFRleHQgRGlnZXN0cz8iIHRvIE1JTUUuIFlvdSBjYW4gc2V0IHRoaXMgb3B0aW9uDQo+IGds
b2JhbGx5IGZvciBhbGwgdGhlIGxpc3QgZGlnZXN0cyB5b3UgcmVjZWl2ZSBhdCB0aGlzIHBvaW50
Lg0KPg0KPg0KPg0KPiBTZW5kIGFxbSBtYWlsaW5nIGxpc3Qgc3VibWlzc2lvbnMgdG8NCj4gYXFt
QGlldGYub3JnDQo+DQo+IFRvIHN1YnNjcmliZSBvciB1bnN1YnNjcmliZSB2aWEgdGhlIFdvcmxk
IFdpZGUgV2ViLCB2aXNpdA0KPiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZv
L2FxbQ0KPiBvciwgdmlhIGVtYWlsLCBzZW5kIGEgbWVzc2FnZSB3aXRoIHN1YmplY3Qgb3IgYm9k
eSAnaGVscCcgdG8NCj4gYXFtLXJlcXVlc3RAaWV0Zi5vcmcNCj4NCj4gWW91IGNhbiByZWFjaCB0
aGUgcGVyc29uIG1hbmFnaW5nIHRoZSBsaXN0IGF0DQo+IGFxbS1vd25lckBpZXRmLm9yZw0KPg0K
PiBXaGVuIHJlcGx5aW5nLCBwbGVhc2UgZWRpdCB5b3VyIFN1YmplY3QgbGluZSBzbyBpdCBpcyBt
b3JlIHNwZWNpZmljDQo+IHRoYW4gIlJlOiBDb250ZW50cyBvZiBhcW0gZGlnZXN0Li4uIg0KPiBU
b2RheSdzIFRvcGljczoNCj4NCj4gMS4gUmU6IFt0Y3BtXSBFQ04gc3VwcG9ydCBhbmQgdXNhZ2Ug
b24gdGhlIEludGVybmV0IChZdWNodW5nIENoZW5nKQ0KPiAyLiBSZTogRUNOIHN1cHBvcnQgYW5k
IHVzYWdlIG9uIHRoZSBJbnRlcm5ldCAoRnJlZCBCYWtlciAoZnJlZCkpDQo+IDMuIFJlOiA/Pzog
W2ljY3JnXSBbdGNwbV0gRUNOIHN1cHBvcnQgYW5kIHVzYWdlIG9uIHRoZSBJbnRlcm5ldA0KPiAo
SmltIEdldHR5cykNCj4gNC4gUmU6ID8/OiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBhbmQg
dXNhZ2Ugb24gdGhlIEludGVybmV0DQo+IChTaW1vbiBCYXJiZXIpDQo+IF9fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IGFxbSBtYWlsaW5nIGxpc3QNCj4g
YXFtQGlldGYub3JnDQo+IGh0dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vYXFt
DQoNCg0KDQotLSANCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCkRpcGwuLUluZy4gTWlyamEgS8O8aGxld2luZA0KSW5z
dGl0dXRlIG9mIENvbW11bmljYXRpb24gTmV0d29ya3MgYW5kIENvbXB1dGVyIEVuZ2luZWVyaW5n
IChJS1IpDQpVbml2ZXJzaXR5IG9mIFN0dXR0Z2FydCwgR2VybWFueQ0KUGZhZmZlbndhbGRyaW5n
IDQ3LCBELTcwNTY5IFN0dXR0Z2FydA0KDQp0ZWw6ICs0OSgwKTcxMS82ODUtNjc5NzMNCmVtYWls
OiBtaXJqYS5rdWVobGV3aW5kQGlrci51bmktc3R1dHRnYXJ0LmRlDQp3ZWI6IHd3dy5pa3IudW5p
LXN0dXR0Z2FydC5kZQ0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K

From rs@netapp.com  Tue Mar 26 10:49:54 2013
Return-Path: <rs@netapp.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DB74621F85A4; Tue, 26 Mar 2013 10:49:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.048
X-Spam-Level: 
X-Spam-Status: No, score=-10.048 tagged_above=-999 required=5 tests=[AWL=0.324, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K0dOHS7A+lIK; Tue, 26 Mar 2013 10:49:50 -0700 (PDT)
Received: from mx12.netapp.com (mx12.netapp.com [216.240.18.77]) by ietfa.amsl.com (Postfix) with ESMTP id D881421F85A1; Tue, 26 Mar 2013 10:49:50 -0700 (PDT)
X-IronPort-AV: E=Sophos;i="4.84,913,1355126400"; d="scan'208";a="34224530"
Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx12-out.netapp.com with ESMTP; 26 Mar 2013 10:49:50 -0700
Received: from vmwexceht03-prd.hq.netapp.com (vmwexceht03-prd.hq.netapp.com [10.106.76.241]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id r2QHnoef006408; Tue, 26 Mar 2013 10:49:50 -0700 (PDT)
Received: from VMWEXCEHT06-PRD.hq.netapp.com (10.106.77.104) by vmwexceht03-prd.hq.netapp.com (10.106.76.241) with Microsoft SMTP Server (TLS) id 14.2.342.3; Tue, 26 Mar 2013 10:49:50 -0700
Received: from SACEXCMBX02-PRD.hq.netapp.com ([169.254.1.222]) by vmwexceht06-prd.hq.netapp.com ([10.106.77.104]) with mapi id 14.02.0342.003; Tue, 26 Mar 2013 10:49:49 -0700
From: "Scheffenegger, Richard" <rs@netapp.com>
To: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>, "aqm@ietf.org" <aqm@ietf.org>
Thread-Topic: [tsvwg] [aqm] [Bloat] how much of a problem is buffer bloat today?
Thread-Index: AQHOKkb4bz1RAbJaNkyIcjxYVOA1Q5i4PtEA
Date: Tue, 26 Mar 2013 17:49:49 +0000
Message-ID: <012C3117EDDB3C4781FD802A8C27DD4F24ACFA86@SACEXCMBX02-PRD.hq.netapp.com>
References: <51408BF4.7090304@cisco.com> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se> <201303261825.47601.mirja.kuehlewind@ikr.uni-stuttgart.de>
In-Reply-To: <201303261825.47601.mirja.kuehlewind@ikr.uni-stuttgart.de>
Accept-Language: de-AT, en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.106.53.53]
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "tsvwg@ietf.org" <tsvwg@ietf.org>, bloat <bloat@lists.bufferbloat.net>
Subject: Re: [aqm] [tsvwg] [Bloat] how much of a problem is buffer bloat	today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 17:49:54 -0000

Hmm...


TCP measures RTT;=20

one could create a global histogram of all measured RTTs by TCP (whenever a=
 valid measurement is taken), and export that with "netstat -sp tcp"... Of =
course, vastly different paths would be gobbled up together, but when inves=
tigating specific paths, that should be good enough as a high level startin=
g point.




Richard Scheffenegger


> -----Original Message-----
> From: tsvwg-bounces@ietf.org [mailto:tsvwg-bounces@ietf.org] On Behalf Of
> Mirja Kuehlewind
> Sent: Dienstag, 26. M=E4rz 2013 18:26
> To: aqm@ietf.org
> Cc: bloat; tsvwg@ietf.org
> Subject: Re: [tsvwg] [aqm] [Bloat] how much of a problem is buffer bloat
> today?
>=20
> Hi,
>=20
> +1. Would be nice to have such statictics available for the user in every
> OS!
>=20
> Mirja
>=20
>=20
> On Friday 22 March 2013 05:27:31 Mikael Abrahamsson wrote:
> > On Thu, 21 Mar 2013, Jim Gettys wrote:
> > > Every more modern TCP can easily fill any sized buffer given time
> > > with a single TCP connection.
> >
> > I agree with this, I made this discovery myself back in 2004 or so,
> > and had to implement Fairqueue and WRED on my home connection to make
> > it bearable to use any interactive application while transferring files=
.
> >
> > In IETF75 in Stockholm in 2009, I made proposals in both TCP and at
> > open mic in one of the sessions, that I would like to see statistics
> > and performance numbers on packet loss, delay variation etc from
> > actual traffic. The IP stack has great insight in what the network
> > conditions are (especially with TCP Timestamping), but as far as I
> > know it's not really exported in any usable format to the user. My
> > idea was to have some kind of dashboard for the user to show if
> > currently the network was the limiting factor, if the tcp window was
> > maxed out etc. Would also be nice if there was output that could be
> > cut/pasted and attached to a fault report in case the customer talks
> > to customer support. It would be good if this was actually a standard s=
o
> all OSes did the same.
> >
> > I am not aware of any such work going on, so I'd like to know if
> > anyone else is aware of work in this area?
>=20
>=20
>=20
> --
> -------------------------------------------------------------------
> Dipl.-Ing. Mirja K=FChlewind
> Institute of Communication Networks and Computer Engineering (IKR)
> University of Stuttgart, Germany Pfaffenwaldring 47, D-70569 Stuttgart
>=20
> tel: +49(0)711/685-67973
> email: mirja.kuehlewind@ikr.uni-stuttgart.de
> web: www.ikr.uni-stuttgart.de
> -------------------------------------------------------------------

From touch@isi.edu  Tue Mar 26 11:07:08 2013
Return-Path: <touch@isi.edu>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A0F2E21F8BFD for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 11:07:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -101.885
X-Spam-Level: 
X-Spam-Status: No, score=-101.885 tagged_above=-999 required=5 tests=[AWL=0.486, BAYES_00=-2.599, SARE_SUB_OBFU_Q1=0.227, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e1XUDWjPcPwx for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 11:07:07 -0700 (PDT)
Received: from darkstar.isi.edu (darkstar.isi.edu [128.9.128.127]) by ietfa.amsl.com (Postfix) with ESMTP id CF8D421F8BD7 for <aqm@ietf.org>; Tue, 26 Mar 2013 11:07:07 -0700 (PDT)
Received: from [128.9.184.109] ([128.9.184.109]) (authenticated bits=0) by darkstar.isi.edu (8.13.8/8.13.8) with ESMTP id r2QI6hYQ021420 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 26 Mar 2013 11:06:43 -0700 (PDT)
Message-ID: <5151E3B5.2010409@isi.edu>
Date: Tue, 26 Mar 2013 11:06:45 -0700
From: Joe Touch <touch@isi.edu>
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130307 Thunderbird/17.0.4
MIME-Version: 1.0
To: dpreed@reed.com
References: <mailman.5225.1363736608.3432.aqm@ietf.org> <1363748540.64155384@apps.rackspace.com> <201303261602.32076.mirja.kuehlewind@ikr.uni-stuttgart.de> <1364316387.705515611@apps.rackspace.com>
In-Reply-To: <1364316387.705515611@apps.rackspace.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-ISI-4-43-8-MailScanner: Found to be clean
X-MailScanner-From: touch@isi.edu
Cc: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>, aqm@ietf.org
Subject: Re: [aqm] aqm Digest, Vol 1, Issue 7
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 18:07:08 -0000

> The optimal state of a highway system in terms of "cars per hour"
> (optimum goodput) is, unfortunately, really bad for the cars - because
> it is a highway system in a state of constant traffic jam, and where the
> latency of any particular car is not minimized - it is actually far, far
> from the minimum.

http://www.paleale.eecs.berkeley.edu/~varaiya/papers_ps.dir/accessF05v2.pdf

Max cars-per-second happens at a variety of speeds, but min driving time 
happens at the highest speed only. See Figure 1.

I'm not sure if it's in this study but I also remember hearing that "max 
cars per second" in most cities is around 45MPH, whereas in LA it's 
closer to 65MPH. LA drivers tend to travel in tighter "packs".

Joe



From dpreed@reed.com  Tue Mar 26 12:34:03 2013
Return-Path: <dpreed@reed.com>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B918621F8CB1 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 12:34:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.295
X-Spam-Level: 
X-Spam-Status: No, score=-3.295 tagged_above=-999 required=5 tests=[AWL=-0.076, BAYES_00=-2.599, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-1, SARE_SUB_ENC_UTF8=0.152, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zvRufDz-mJYJ for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 12:34:02 -0700 (PDT)
Received: from smtp116.iad.emailsrvr.com (smtp116.iad.emailsrvr.com [207.97.245.116]) by ietfa.amsl.com (Postfix) with ESMTP id 79FA821F8CAE for <aqm@ietf.org>; Tue, 26 Mar 2013 12:34:02 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp41.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id A1FE029163E for <aqm@ietf.org>; Tue, 26 Mar 2013 15:34:01 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy23.wa-web.iad1a (legacy23.wa-web.iad1a.rsapps.net [192.168.4.109]) by smtp41.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 1E44E2909CA for <aqm@ietf.org>; Tue, 26 Mar 2013 15:34:00 -0400 (EDT)
Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy23.wa-web.iad1a (Postfix) with ESMTP id C89BD10A0003 for <aqm@ietf.org>; Tue, 26 Mar 2013 15:34:00 -0400 (EDT)
Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com)  with HTTP; Tue, 26 Mar 2013 15:34:00 -0400 (EDT)
Date: Tue, 26 Mar 2013 15:34:00 -0400 (EDT)
From: dpreed@reed.com
To: aqm@ietf.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_20130326153400000000_70654"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
Message-ID: <1364326440.82022390@apps.rackspace.com>
X-Mailer: webmail7.0
Subject: Re: [aqm] =?utf-8?q?real_traffic=2C_network_and_highway_=28was_aqm_Di?= =?utf-8?q?gest=2C_Vol_1=2C_Issue_16=29?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 19:34:03 -0000

------=_20130326153400000000_70654
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0A =0A=0AA small standing queue of approximately 1/2 packet for the bottle=
neck link would not work for either ECN or RED, and neither can get close t=
o that - because by the time either one actually signals congestion (bit or=
 drop), there are far more than one packet in the queue.=0A =0AAs far as a =
highway operating at high speed maximizing cars per hour, that's plain wron=
g.   But here's the mistake I think you are making.  You are assuming all c=
ars are already on the highway, it is infinitely long, and *there are no on=
ramps and offramps*, probably.=0A =0ABut *real* traffic in real cities is N=
EVER in that state (OK, maybe once in a million years by accident).  Severa=
l have quoted to me things about traffic that are manifestly *not* true in =
complex city highway systems at any time of the day as "truth".  Only in a =
trivial model system - used for teaching simple principles, not reflecting =
actual traffic.=0A =0ASo here's the deal regarding network traffic: you all=
 seem to focus on lots of "steady state" flows, and also assume that the va=
riations are "stationary random processes", typically with Gaussian-family =
distributions.=0A =0ANo computer programs behave like that.  No interactive=
 systems behave like that.  Even YouTube uploads and downloads don't behave=
 like that!  (people don't all start their uploads at 7 pm, and go have din=
ner - they generally have a very fat-tailed distribution of starts and stop=
s).=0A =0AOne might think that "fractal" or non-stationary distributions ar=
e the rare case - because that is how elementary probability is *taught*.  =
 But they are the common case (except in casinos, which is the rare excepti=
on, not the model system)!=0A =0ASo, we can have a conversation about the r=
eal world and the real Internet, OR we can have a conversation about "bench=
mark land"  (the favorite networking hangout of academia, and most IETFers)=
.=0A =0AAs an engineer, I prefer to discuss the real world and the real Int=
ernet. Whether ECN or RED work well in simplified model systems with statio=
nary random processes (and no applications or humans) is a nice topic for a=
 theoretical PhD thesis.  But a better *engineering* PhD thesis would actua=
lly work on the Internet as it is, and those who actually build equipment s=
hould only work on such problems.=0A =0AIn the old days, ECN might have bee=
n OK, because the end-to-end delays due to running on 50 Kb/sec ARPANET wer=
e quite long.  But today the end-to-end delays are under 50 msec, so little=
 queues at bottleneck links really, really matter.=0A =0A =0A-----Original =
Message-----=0AFrom: aqm-request@ietf.org=0ASent: Tuesday, March 26, 2013 3=
:00pm=0ATo: aqm@ietf.org=0ASubject: aqm Digest, Vol 1, Issue 16=0A=0A=0A=0A=
If you have received this digest without all the individual message=0Aattac=
hments you will need to update your digest options in your list=0Asubscript=
ion.  To do so, go to =0A=0Ahttps://www.ietf.org/mailman/listinfo/aqm=0A=0A=
Click the 'Unsubscribe or edit options' button, log in, and set "Get=0AMIME=
 or Plain Text Digests?" to MIME.  You can set this option=0Aglobally for a=
ll the list digests you receive at this point.=0A=0A=0A=0ASend aqm mailing =
list submissions to=0A aqm@ietf.org=0A=0ATo subscribe or unsubscribe via th=
e World Wide Web, visit=0A https://www.ietf.org/mailman/listinfo/aqm=0Aor, =
via email, send a message with subject or body 'help' to=0A aqm-request@iet=
f.org=0A=0AYou can reach the person managing the list at=0A aqm-owner@ietf.=
org=0A=0AWhen replying, please edit your Subject line so it is more specifi=
c=0Athan "Re: Contents of aqm digest..."=0AToday's Topics:=0A=0A 1. Re: aqm=
 Digest, Vol 1, Issue 7 (Mikael Abrahamsson)=0A 2. Re: aqm Digest, Vol 1, I=
ssue 7 (Scheffenegger, Richard)=0A 3. Re: [tsvwg] [Bloat] how much of a pro=
blem is buffer=09bloat=0A today? (Scheffenegger, Richard)=0A 4. Re: aqm Dig=
est, Vol 1, Issue 7 (Joe Touch)=0A_________________________________________=
______=0Aaqm mailing list=0Aaqm@ietf.org=0Ahttps://www.ietf.org/mailman/lis=
tinfo/aqm
------=_20130326153400000000_70654
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"times new roman" size=3D"2"><p style=3D"margin:0;padding:0;">=
&nbsp;</p>=0A<p style=3D"margin:0;padding:0;"><span style=3D"font-family: t=
imes new roman; font-size: 10pt;">=0A<p style=3D"margin:0;padding:0;margin:=
 0; padding: 0;">A small standing queue of approximately 1/2 packet for the=
 bottleneck link would not work for either ECN or RED, and neither can get =
close to that - because by the time either one actually signals congestion =
(bit or drop), there are far more than one packet in the queue.</p>=0A<p st=
yle=3D"margin:0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"=
margin:0;padding:0;margin: 0; padding: 0;">As far as a highway operating at=
 high speed maximizing cars per hour, that's plain wrong.&nbsp;&nbsp; But h=
ere's the mistake I think you are making.&nbsp; You are assuming all cars a=
re already on the highway, it is infinitely long, and *there are no onramps=
 and offramps*, probably.</p>=0A<p style=3D"margin:0;padding:0;margin: 0; p=
adding: 0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;margin: 0; padding:=
 0;">But *real* traffic in real cities is NEVER in that state (OK, maybe on=
ce in a million years by accident).&nbsp; Several have quoted to me things =
about traffic that are manifestly *not* true in complex city highway system=
s at any time of the day as "truth".&nbsp; Only in a trivial model system -=
 used for teaching simple principles, not reflecting actual traffic.</p>=0A=
<p style=3D"margin:0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p styl=
e=3D"margin:0;padding:0;margin: 0; padding: 0;">So here's the deal regardin=
g network traffic: you all seem to focus on lots of "steady state" flows, a=
nd also assume that the variations are "stationary random processes", typic=
ally with Gaussian-family distributions.</p>=0A<p style=3D"margin:0;padding=
:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;marg=
in: 0; padding: 0;">No computer programs behave like that.&nbsp; No interac=
tive systems behave like that.&nbsp; Even YouTube uploads and downloads don=
't behave like that!&nbsp; (people don't all start their uploads at 7 pm, a=
nd go have dinner - they generally have a very fat-tailed distribution of s=
tarts and stops).</p>=0A<p style=3D"margin:0;padding:0;margin: 0; padding: =
0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;margin: 0; padding: 0;">One=
 might think that "fractal" or non-stationary distributions are the rare ca=
se - because that is how elementary probability is *taught*.&nbsp;&nbsp; Bu=
t they are the common case (except in casinos, which is the rare exception,=
 not the model system)!</p>=0A<p style=3D"margin:0;padding:0;margin: 0; pad=
ding: 0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;margin: 0; padding: 0=
;">So, we can have a conversation about the real world and the real Interne=
t, OR we can have a conversation about "benchmark land"&nbsp; (the favorite=
 networking hangout of academia, and most IETFers).</p>=0A<p style=3D"margi=
n:0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"margin:0;pad=
ding:0;margin: 0; padding: 0;">As an engineer, I prefer to discuss the real=
 world and the real Internet. Whether ECN or RED work well in simplified mo=
del systems with stationary random processes (and no applications or humans=
) is a nice topic for a theoretical PhD thesis.&nbsp; But a better *enginee=
ring* PhD thesis would actually work on the Internet as it is, and those wh=
o actually build equipment should only work on such problems.</p>=0A<p styl=
e=3D"margin:0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"ma=
rgin:0;padding:0;margin: 0; padding: 0;">In the old days, ECN might have be=
en OK, because the end-to-end delays due to running on 50 Kb/sec ARPANET we=
re quite long.&nbsp; But today the end-to-end delays are under 50 msec, so =
little queues at bottleneck links really, really matter.</p>=0A<p style=3D"=
margin:0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"margin:=
0;padding:0;margin: 0; padding: 0;">&nbsp;</p>=0A<p style=3D"margin:0;paddi=
ng:0;margin: 0; padding: 0;">-----Original Message-----<br />From: aqm-requ=
est@ietf.org<br />Sent: Tuesday, March 26, 2013 3:00pm<br />To: aqm@ietf.or=
g<br />Subject: aqm Digest, Vol 1, Issue 16<br /><br /></p>=0A<div id=3D"Sa=
feStyles1364325262">=0A<p style=3D"margin:0;padding:0;margin: 0; padding: 0=
;">If you have received this digest without all the individual message<br /=
>attachments you will need to update your digest options in your list<br />=
subscription.  To do so, go to <br /><br />https://www.ietf.org/mailman/lis=
tinfo/aqm<br /><br />Click the 'Unsubscribe or edit options' button, log in=
, and set "Get<br />MIME or Plain Text Digests?" to MIME.  You can set this=
 option<br />globally for all the list digests you receive at this point.<b=
r /><br /><br /><br />Send aqm mailing list submissions to<br /> aqm@ietf.o=
rg<br /><br />To subscribe or unsubscribe via the World Wide Web, visit<br =
/> https://www.ietf.org/mailman/listinfo/aqm<br />or, via email, send a mes=
sage with subject or body 'help' to<br /> aqm-request@ietf.org<br /><br />Y=
ou can reach the person managing the list at<br /> aqm-owner@ietf.org<br />=
<br />When replying, please edit your Subject line so it is more specific<b=
r />than "Re: Contents of aqm digest..."<br />Today's Topics:<br /><br /> 1=
. Re: aqm Digest, Vol 1, Issue 7 (Mikael Abrahamsson)<br /> 2. Re: aqm Dige=
st, Vol 1, Issue 7 (Scheffenegger, Richard)<br /> 3. Re: [tsvwg] [Bloat] ho=
w much of a problem is buffer=09bloat<br /> today? (Scheffenegger, Richard)=
<br /> 4. Re: aqm Digest, Vol 1, Issue 7 (Joe Touch)<br />_________________=
______________________________<br />aqm mailing list<br />aqm@ietf.org<br /=
>https://www.ietf.org/mailman/listinfo/aqm</p>=0A</div>=0A</span></p></font=
>
------=_20130326153400000000_70654--


From simon@superduper.net  Tue Mar 26 13:51:42 2013
Return-Path: <simon@superduper.net>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 8D69C21F8910 for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 13:51:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 0.494
X-Spam-Level: 
X-Spam-Status: No, score=0.494 tagged_above=-999 required=5 tests=[BAYES_40=-0.185, MIME_8BIT_HEADER=0.3, SARE_SUB_ENC_UTF8=0.152,  SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BiPxohdtjANS for <aqm@ietfa.amsl.com>; Tue, 26 Mar 2013 13:51:41 -0700 (PDT)
Received: from masada.superduper.net (masada.superduper.net [85.119.82.91]) by ietfa.amsl.com (Postfix) with ESMTP id 2ED1E21F89C7 for <aqm@ietf.org>; Tue, 26 Mar 2013 13:51:41 -0700 (PDT)
Received: from snappy-wlan.parc.xerox.com ([13.1.108.21]) by masada.superduper.net with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from <simon@superduper.net>) id 1UKaq9-00025v-3a; Tue, 26 Mar 2013 20:51:31 +0000
Message-ID: <51520A4E.7070308@superduper.net>
Date: Tue, 26 Mar 2013 13:51:26 -0700
From: Simon Barber <simon@superduper.net>
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20130308 Thunderbird/17.0.4
MIME-Version: 1.0
To: Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>
References: <201303190904.32180.mkuehle@ikr.uni-stuttgart.de> <CAGhGL2AbSUvm-XcgH6y2gRy_hYj4mc4YU+u7LeuiCp0DzfsRDg@mail.gmail.com> <5148F80F.5000401@superduper.net> <201303261549.59606.mirja.kuehlewind@ikr.uni-stuttgart.de>
In-Reply-To: <201303261549.59606.mirja.kuehlewind@ikr.uni-stuttgart.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Cc: aqm@ietf.org
Subject: Re: [aqm] =?utf-8?b?562U5aSNOiBbaWNjcmddIFt0Y3BtXSBFQ04gc3VwcG9ydCBh?= =?utf-8?q?nd_usage_on_the_Internet?=
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 20:51:42 -0000

I can see advantages for time sensitive applications, like VoIP, or for 
protocols with longer, slow timeouts (TCP SYN, SYN-ACK). I don't see 
any real advantage for TCP data, the extra traffic caused by the losses 
is insignificant, compared to the complexity of getting ECN to work in 
today's networks.

Simon


On Tue 26 Mar 2013 07:49:59 AM PDT, Mirja Kuehlewind wrote:
> Hi,
>
> I don't understand the obsession to not use ECN. The advantage is, you don't
> have to drop packets. Sure, (core) networks are optimized to have a very low
> drop rate anyway, but isn't it better not not drop any packets at all?
>
> Mirja
>
>
> On Wednesday 20 March 2013 00:43:11 Simon Barber wrote:
>> I'm not sure why people are so obsessed with ECN - the little data I've
>> seen about it showed that it's effect on system throughput was in the
>> noise. Sure, wireless guys don't like to waste precious wireless
>> bandwidth, but they're not dropping packets that have used up the
>> precious wireless bandwidth. They get dropped before the wireless link.
>>
>> Simon
>>
>> On 03/19/2013 03:36 PM, Jim Gettys wrote:
>>> Clearly, having a working AQM is a necessary prerequisite for ECN to be
>>> useful; one only overcome in the last year with CoDel/Pie.  Variable
>>> bandwidth is still a challenge, but with CoDel at least, we think it can
>>> be done.  And we've certainly implemented ECN support in the
>>> codel/fq_codel qdisc's (though would be grateful if ECN experts make
>>> sure it's been done correctly).
>>>
>>> The incentive structure needs to work here. There may be things we can
>>> and should do to provide incentive for ECN deployment.
>>>
>>> For Bell Heads working on wireless cellular transport, the incentive is
>>> really pretty clear: they ***hate*** dropping packets with a passion,
>>> not understanding that either packet drop or ECN are necessary, and it
>>> will probably be easier to convince them to adopt ECN than to drop
>>> packets.  Given how hard they sweat to get packets over the air, I sort
>>> of understand where they come from.
>>>
>>> I believe some of the cellular carriers are requiring ECN support in
>>> handsets going forward, and as they classify VOIP into a separate bearer
>>> class anyway, I can see little incentive not to use ECN on their "best
>>> effort" bearer class.
>>>
>>> I believe all current operating systems react correctly to ECN marked
>>> packets; the only question is whether they will talk ECN if talked to
>>> (pretty common; Linux has done so now for quite a few years), *and*
>>> whether they enable ECN when establishing new connections or not (no one
>>> does right now by default).
>>>
>>> At the moment, Windows is vanishingly small fraction of handsets, being
>>> dominated by Android (Linux) and Apple as it is.
>>>
>>> I worry about bufferbloat in the HARQ layer of cellular networks not
>>> reflecting ECN marking into packets, but until that analysis has been
>>> done in real systems, it's just one more thing to worry about. Our Linux
>>> experience is that bufferbloat hides everywhere in current systems: the
>>> analogy in WiFi is the error correction layer in 802.11n, which causes
>>> large amounts of bloat, and fixing WiFi is very much a work in progress
>>> (except that not enough resources are currently available to go do the
>>> work, and it's more complicated that wired systems).  Dave Taht cheered
>>> the first time he got the 802.11 layer to drop a packet, it was so much
>>> a problem.
>>>
>>> Right now, there are still some disincentives for ECN: e.g. some very
>>> small fraction of sites that become "black holes", or crashing antique
>>> middelboxes. We have no data on how much this is at this date, and the
>>> database that used to exist documenting middlebox problems has
>>> bit-rotted.  These  middleboxes are likely in the consumer edge, and
>>> unlikely to be in the path of most mobile devices and web sites on
>>> 3g/LTE.  Mobile devices on home WiFi networks are more likely to have
>>> problems.
>>>
>>> We have ECN enabled in CeroWrt and have not noticed any problems; but
>>> 4-5 years ago OpenWrt (with a very much larger user base) still detected
>>> problems to the extent they had to disable initiating ECN connections.
>>>
>>> I know Bauer et. al. examined this a few years ago. But we have no data
>>> over the last several years I've seen (though this new paper referenced
>>> may: I've not had a chance to read it yet).
>>>
>>> There is one other issue with ECN: when operating at low bandwidth at
>>> the very edge of the net, it can still be advantageous to drop a packet
>>> rather than ECN mark it, to get back the transmission time of the
>>> packet.  Dave Taht's experiments (IIRC) were below 4Mbps for this
>>> advantage (when doing VOIP over that bottleneck competing against other
>>> traffic in a best effort class).
>>>
>>> My intuition is to get VOIP reliable over WiFI on a busy network (or
>>> when far away from the AP) we'll want to classify the VOIP packets
>>> anyway and use the right hardware class in WiFi; this is primarily to
>>> get priority access to a transmit opportunity on WiFi rather than the
>>> previous reasons we've had.  Certainly on other media we've seen little
>>> reason to have to do any explicit classification for VOIP anymore once
>>> running fq_codel.  Things generally "just work" without all that hair
>>> for VOIP traffic.
>>>                                                     - Jim
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 19, 2013 at 5:49 PM, 天澜 <tianlan.lhh@alibaba-inc.com
>>> <mailto:tianlan.lhh@alibaba-inc.com>> wrote:
>>>
>>>      To let ECN be popular used ，the most important is let the
>>>      connection which used “ECN” is fairness to the other connection;
>>>      So we should design new algorithm for ECN connection to "prize"
>>>      those people who use ECN;
>>>
>>>
>>>                                                      hritian
>>>
>>>      ________________________________________
>>>      发件人: Jim Gettys [jg@freedesktop.org <mailto:jg@freedesktop.org>]
>>>      发送时间: 2013年3月20日 1:26
>>>      到: Rick Jones
>>>      Cc: tcpm@ietf.org <mailto:tcpm@ietf.org>; aqm@ietf.org
>>>      <mailto:aqm@ietf.org>; iccrg@irtf.org <mailto:iccrg@irtf.org>;
>>>      Mikael Abrahamsson
>>>      主题: Re: [iccrg] [tcpm] ECN support and usage on the Internet
>>>
>>>      You need to distinguish 1) "Will talk ECN when talked to", from 2)
>>>      "Will initiate an ECN conversation".
>>>
>>>      Linux, for example, will do 1) by default (and has for some time
>>>      now), but requires configuration to turn on ECN when initiating a
>>>      connection.
>>>                                         - Jim
>>>
>>>
>>>
>>>      On Tue, Mar 19, 2013 at 10:45 AM, Rick Jones <rick.jones2@hp.com
>>>      <mailto:rick.jones2@hp.com><mailto:rick.jones2@hp.com
>>>      <mailto:rick.jones2@hp.com>>> wrote:
>>>      On 03/19/2013 02:22 AM, Mikael Abrahamsson wrote:
>>>      On Tue, 19 Mar 2013, Mirja Kühlewind wrote:
>>>
>>>      Do we need to start a campaign to further encourage greater ECN
>>>      support (also of network providers)? Asking the content providers on
>>>      the list(s): What are the reasons to not enable ECN support?
>>>
>>>      Let me speculate:
>>>
>>>      It's not on by default in Windows. There is little reason to turn it
>>> on, because "nobody" acts on ECN, and nobody acts on ECN because there is
>>> no demand for AQM, and even if there was AQM, it would most likely not be
>>> ECN enabled.
>>>
>>>      Out of curiosity, which stacks have ECN enabled by default?  To my
>>>      recollection, on netperf.org
>>>      <http://netperf.org><http://netperf.org> I had to explicitly enable
>>>      TCP's use of ECN in the kernel which ships with Ubuntu 12.04.
>>>
>>>      rick jones
>>>
>>>      _______________________________________________
>>>      tcpm mailing list
>>>      tcpm@ietf.org <mailto:tcpm@ietf.org><mailto:tcpm@ietf.org
>>>      <mailto:tcpm@ietf.org>>
>>>      https://www.ietf.org/mailman/listinfo/tcpm
>>>
>>>
>>>      ________________________________
>>>
>>>      This email (including any attachments) is confidential and may be
>>>      legally privileged. If you received this email in error, please
>>>      delete it immediately and do not copy it or use it for any purpose
>>>      or disclose its contents to any other person. Thank you.
>>>
>>>      本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件
>>>      人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或
>>>      透露本邮件之内容。谢谢。
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> aqm mailing list
>>> aqm@ietf.org
>>> https://www.ietf.org/mailman/listinfo/aqm
>>
>> _______________________________________________
>> aqm mailing list
>> aqm@ietf.org
>> https://www.ietf.org/mailman/listinfo/aqm
>
>
>

From hagen@jauu.net  Tue Mar 26 13:02:24 2013
Return-Path: <hagen@jauu.net>
X-Original-To: aqm@ietfa.amsl.com
Delivered-To: aqm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B866521F8D34; Tue, 26 Mar 2013 13:02:24 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.373
X-Spam-Level: 
X-Spam-Status: No, score=-2.373 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, NO_RELAYS=-0.001, SARE_SUB_OBFU_Q1=0.227]
Received: from mail.ietf.org ([12.22.58.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3OA5wmIyEskR; Tue, 26 Mar 2013 13:02:24 -0700 (PDT)
Received: from Chamillionaire.breakpoint.cc (Chamillionaire.breakpoint.cc [IPv6:2001:4d88:1ffa:82:880:aa0:9009:64ae]) by ietfa.amsl.com (Postfix) with ESMTP id CF69421F8D06; Tue, 26 Mar 2013 13:02:23 -0700 (PDT)
Received: from pfeifer by Chamillionaire.breakpoint.cc with local (Exim 4.72) (envelope-from <hagen@jauu.net>) id 1UKa4X-000555-E8; Tue, 26 Mar 2013 21:02:17 +0100
Date: Tue, 26 Mar 2013 21:02:16 +0100
From: Hagen Paul Pfeifer <hagen@jauu.net>
To: "Scheffenegger, Richard" <rs@netapp.com>
Message-ID: <20130326200216.GB9410@virgo.local>
References: <51408BF4.7090304@cisco.com> <CAGhGL2DHrNUEnVF-Qn6OeeD9LvST06c0QaTxSJ+zbKdpQq2Y7g@mail.gmail.com> <alpine.DEB.2.00.1303220521370.2309@uplift.swm.pp.se> <201303261825.47601.mirja.kuehlewind@ikr.uni-stuttgart.de> <012C3117EDDB3C4781FD802A8C27DD4F24ACFA86@SACEXCMBX02-PRD.hq.netapp.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <012C3117EDDB3C4781FD802A8C27DD4F24ACFA86@SACEXCMBX02-PRD.hq.netapp.com>
X-Key-Id: 98350C22
X-Key-Fingerprint: 490F 557B 6C48 6D7E 5706 2EA2 4A22 8D45 9835 0C22
X-GPG-Key: gpg --recv-keys --keyserver wwwkeys.eu.pgp.net 98350C22
User-Agent: Mutt/1.5.21 (2010-09-15)
X-Mailman-Approved-At: Wed, 27 Mar 2013 18:55:04 -0700
Cc: bloat <bloat@lists.bufferbloat.net>, Mirja Kuehlewind <mirja.kuehlewind@ikr.uni-stuttgart.de>, "aqm@ietf.org" <aqm@ietf.org>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Subject: Re: [aqm] [Bloat] [tsvwg] how much of a problem is buffer bloat today?
X-BeenThere: aqm@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: "Discussion list for active queue management and flow isolation." <aqm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/aqm>, <mailto:aqm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/aqm>
List-Post: <mailto:aqm@ietf.org>
List-Help: <mailto:aqm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/aqm>, <mailto:aqm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Mar 2013 20:02:25 -0000

* Scheffenegger, Richard | 2013-03-26 17:49:49 [+0000]:

>TCP measures RTT; 
>
>one could create a global histogram of all measured RTTs by TCP (whenever a
>valid measurement is taken), and export that with "netstat -sp tcp"... Of
>course, vastly different paths would be gobbled up together, but when
>investigating specific paths, that should be good enough as a high level
>starting point.

For Linux tcp-probe[1] is your friend. The last colum is the smoothed round trip
time. The port filter is optional and can be 0 to disable filtering. Not sure
if Linux Distribution ship this module by default. But via tcp-probe you get
accurate per-path rtt statistics without any effort (just a few CPU cycles).


Hagen

[1] http://www.linuxfoundation.org/collaborate/workgroups/networking/tcpprobe

