
From acmorton@att.com  Mon Dec 10 09:58:11 2012
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7478B21F8504 for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 09:58:11 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -105.313
X-Spam-Level: 
X-Spam-Status: No, score=-105.313 tagged_above=-999 required=5 tests=[AWL=0.686, BAYES_00=-2.599, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GcnmXkRRNQ+S for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 09:58:10 -0800 (PST)
Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id AF7B121F8552 for <bmwg@ietf.org>; Mon, 10 Dec 2012 09:58:10 -0800 (PST)
Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id 2b226c05.0.320884.00-212.881612.nbfkord-smmo05.seg.att.com (envelope-from <acmorton@att.com>);  Mon, 10 Dec 2012 17:58:10 +0000 (UTC)
X-MXL-Hash: 50c622b214aa3620-084f667325e6a69a35a50e793837936d92962652
Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBAHw9mm001767 for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:58:10 -0500
Received: from sflint01.pst.cso.att.com (sflint01.pst.cso.att.com [144.154.234.228]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBAHw1Ew001677 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:58:02 -0500
Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by sflint01.pst.cso.att.com (RSA Interceptor) for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:57:42 -0500
Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBAHvHo9009230 for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:57:17 -0500
Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBAHv09W008778 for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:57:13 -0500
Received: from lt-hp1044652.att.com (vpn-135-70-235-242.vpn.east.att.com[135.70.235.242](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121210175652gw10063215e>; Mon, 10 Dec 2012 17:56:53 +0000
X-Originating-IP: [135.70.235.242]
Message-Id: <7.0.1.0.0.20121210121952.04b67b60@att.com>
X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0
Date: Mon, 10 Dec 2012 12:54:51 -0500
To: bmwg@ietf.org
From: Al Morton <acmorton@att.com>
In-Reply-To: <7.0.1.0.0.20121111171100.04d17eb0@att.com>
References: <7.0.1.0.0.20121111082840.09070110@att.com> <7.0.1.0.0.20121111171100.04d17eb0@att.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
X-RSA-Inspected: yes
X-RSA-Classifications: public
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)]
X-MAIL-FROM: <acmorton@att.com>
X-SOURCE-IP: [144.160.20.145]
X-AnalysisOut: [v=2.0 cv=GNm/5JxK c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a]
X-AnalysisOut: [=soI6yPYPFSYA:10 a=W2hlYBC8_kYA:10 a=ofMgfj31e3cA:10 a=BLc]
X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=e1xCkD5h]
X-AnalysisOut: [TCUA:10 a=48vgC7mUAAAA:8 a=-UBTm5eQ76_y43_Ub-MA:9 a=CjuIK1]
X-AnalysisOut: [q_8ugA:10 a=AM14pv5mMGwA:10 a=qg_hPROt29YA:10 a=Hz7IrDYlS0]
X-AnalysisOut: [cA:10 a=lZB815dzVvQA:10 a=N1M-K4NG7HYy8Apn:21 a=0GY8_tT5yP]
X-AnalysisOut: [b9z72a:21]
Cc: worley@ariadne.com
Subject: Re: [bmwg] WGLC on SIP Benchmarking Drafts (04)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 10 Dec 2012 17:58:11 -0000

BMWG:

This is the last day to review the SIP drafts under WGLC.

As part of document shepherd review, I noted the following
gripes-worth-fixing from the nit-checker routine,
(no offense to the nits-checker)

These comments are for -term-06

   == No 'Intended status' indicated for this document; assuming Proposed
      Standard
(need to add Informational, like all BMWG's drafts and RFCs. MUST fix.)

   == Outdated reference: draft-ietf-soc-overload-design has been published as
      RFC 6357

   == Outdated reference: A later version (-11) exists of
      draft-ietf-soc-overload-control-10

Also, in section 2.1, one of the bullets says:
...This is because our goal is to determine the
maximum throughput of the device or system under test, that is the
number of simultaneous SIP sessions that the device or system can
support.

I suggest s/throughput/capacity/ because we already have a
special definition of "throughput" in BMWG, and capacity
(or some other term you choose, is a better metric for size here.


Last full sentence before section 3.2:

  The EA must make that  count available for viewing ad recording.
s/ad/and/

Section 3.2.3: typo
... service.he SIP-Aware Stateful Firewall
s/.he/. The/

-meth comments to follow,
Al


At 05:12 PM 11/11/2012, Al Morton wrote:

>Re-sending to Dale's new address.
>
>At 08:45 AM 11/11/2012, Al Morton wrote:
>
>>TO: BMWG,
>>CC: RAI Dir Reviewer Dale Worley, sipcore wg,
>>
>>A WG Last Call period for the Internet-Drafts on SIP Device
>>Benchmarking:
>>
>>   http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/
>>   http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-meth/
>>
>>will be open from 11 Nov 2012 through 10 Dec 2012.
>>
>>These drafts are continuing the BMWG Last Call Process. See
>>http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
>>The first WGLC was completed on 5 April 2010 with comments.
>>The second WGLC was completed on 18 May 2012 with comments.
>>
>>Please read and express your opinion on whether or not these
>>Internet-Drafts should be forwarded to the Area Directors for
>>publication as Informational RFCs.  Send your comments
>>to this list or acmorton@att.com
>>
>>Al
>>bmwg chair
>>_______________________________________________
>>bmwg mailing list
>>bmwg@ietf.org
>>https://www.ietf.org/mailman/listinfo/bmwg
>
>_______________________________________________
>bmwg mailing list
>bmwg@ietf.org
>https://www.ietf.org/mailman/listinfo/bmwg


From acmorton@att.com  Mon Dec 10 10:28:16 2012
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 52D2021F8593 for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 10:28:16 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -105.484
X-Spam-Level: 
X-Spam-Status: No, score=-105.484 tagged_above=-999 required=5 tests=[AWL=0.515, BAYES_00=-2.599, J_CHICKENPOX_72=0.6, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7GiFBPqb4Xc6 for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 10:28:15 -0800 (PST)
Received: from nbfkord-smmo05.seg.att.com (nbfkord-smmo05.seg.att.com [209.65.160.92]) by ietfa.amsl.com (Postfix) with ESMTP id 73ABD21F858F for <bmwg@ietf.org>; Mon, 10 Dec 2012 10:28:15 -0800 (PST)
Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo05.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id fb926c05.0.336020.00-411.925077.nbfkord-smmo05.seg.att.com (envelope-from <acmorton@att.com>);  Mon, 10 Dec 2012 18:28:15 +0000 (UTC)
X-MXL-Hash: 50c629bf5e9d777c-7b7b13a5400a691a35b296cbf8c79b4b5b5cb166
Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBAISE4L015947 for <bmwg@ietf.org>; Mon, 10 Dec 2012 13:28:15 -0500
Received: from sflint01.pst.cso.att.com (sflint01.pst.cso.att.com [144.154.234.228]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBAISBE6015935 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <bmwg@ietf.org>; Mon, 10 Dec 2012 13:28:12 -0500
Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by sflint01.pst.cso.att.com (RSA Interceptor) for <bmwg@ietf.org>; Mon, 10 Dec 2012 13:27:57 -0500
Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBAIRuXt005932 for <bmwg@ietf.org>; Mon, 10 Dec 2012 13:27:56 -0500
Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBAIRnSG005788 for <bmwg@ietf.org>; Mon, 10 Dec 2012 13:27:52 -0500
Received: from lt-hp1044652.att.com (vpn-135-70-235-242.vpn.east.att.com[135.70.235.242](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121210182741gw10063218e>; Mon, 10 Dec 2012 18:27:42 +0000
X-Originating-IP: [135.70.235.242]
Message-Id: <7.0.1.0.0.20121210130738.04b681c8@att.com>
X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0
Date: Mon, 10 Dec 2012 13:25:40 -0500
To: bmwg@ietf.org
From: Al Morton <acmorton@att.com>
In-Reply-To: <7.0.1.0.0.20121210121952.04b67b60@att.com>
References: <7.0.1.0.0.20121111082840.09070110@att.com> <7.0.1.0.0.20121111171100.04d17eb0@att.com> <7.0.1.0.0.20121210121952.04b67b60@att.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
X-RSA-Inspected: yes
X-RSA-Classifications: public
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)]
X-MAIL-FROM: <acmorton@att.com>
X-SOURCE-IP: [144.160.20.145]
X-AnalysisOut: [v=2.0 cv=GNm/5JxK c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a]
X-AnalysisOut: [=soI6yPYPFSYA:10 a=W2hlYBC8_kYA:10 a=ofMgfj31e3cA:10 a=BLc]
X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=e1xCkD5h]
X-AnalysisOut: [TCUA:10 a=48vgC7mUAAAA:8 a=sEI5Qq5O355QfX2c3r8A:9 a=CjuIK1]
X-AnalysisOut: [q_8ugA:10 a=qg_hPROt29YA:10 a=Hz7IrDYlS0cA:10 a=lZB815dzVv]
X-AnalysisOut: [QA:10 a=CQfGPnxrh8xVlibJ:21 a=bqamFyAgfE8z6l_T:21]
Cc: worley@ariadne.com
Subject: Re: [bmwg] WGLC on SIP Benchmarking Drafts (04)
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 10 Dec 2012 18:28:16 -0000

shepherd's review of -meth-06:

Add Intended Status: Informational  in the draft header.  MUST fix.

Section 5.1, Note 3:  It seems that you may have intended a more
report-like formatting for the last two lines:
...
    The Test Organization interested in knowing how many times
    the EA was intended to send a REGISTER as distinct from how many
    times the EA wound up actually sending a REGISTER may wish to record
    the following data as well: Number of responses of the following
    type: 401: _____________ (if authentication turned on; N/A otherwise)
    407: _____________ (if authentication turned on; N/A otherwise)

maybe
    the following data as well: Number of responses of the following
    type:
    401: _____________ (if authentication turned on; N/A otherwise)
    407: _____________ (if authentication turned on; N/A otherwise)



At 12:54 PM 12/10/2012, Al Morton wrote:
>BMWG:
>
>This is the last day to review the SIP drafts under WGLC.
>
>As part of document shepherd review, I noted the following
>gripes-worth-fixing from the nit-checker routine,
>(no offense to the nits-checker)
>
>These comments are for -term-06
>
>   == No 'Intended status' indicated for this document; assuming Proposed
>      Standard
>(need to add Informational, like all BMWG's drafts and RFCs. MUST fix.)
>
>   == Outdated reference: draft-ietf-soc-overload-design has been published as
>      RFC 6357
>
>   == Outdated reference: A later version (-11) exists of
>      draft-ietf-soc-overload-control-10
>
>Also, in section 2.1, one of the bullets says:
>...This is because our goal is to determine the
>maximum throughput of the device or system under test, that is the
>number of simultaneous SIP sessions that the device or system can
>support.
>
>I suggest s/throughput/capacity/ because we already have a
>special definition of "throughput" in BMWG, and capacity
>(or some other term you choose, is a better metric for size here.
>
>
>Last full sentence before section 3.2:
>
>  The EA must make that  count available for viewing ad recording.
>s/ad/and/
>
>Section 3.2.3: typo
>... service.he SIP-Aware Stateful Firewall
>s/.he/. The/
>
>-meth comments to follow,
>Al
>
>
>At 05:12 PM 11/11/2012, Al Morton wrote:
>
>>Re-sending to Dale's new address.
>>
>>At 08:45 AM 11/11/2012, Al Morton wrote:
>>
>>>TO: BMWG,
>>>CC: RAI Dir Reviewer Dale Worley, sipcore wg,
>>>
>>>A WG Last Call period for the Internet-Drafts on SIP Device
>>>Benchmarking:
>>>
>>>   http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-term/
>>>   http://datatracker.ietf.org/doc/draft-ietf-bmwg-sip-bench-meth/
>>>
>>>will be open from 11 Nov 2012 through 10 Dec 2012.
>>>
>>>These drafts are continuing the BMWG Last Call Process. See
>>>http://www1.ietf.org/mail-archive/web/bmwg/current/msg00846.html
>>>The first WGLC was completed on 5 April 2010 with comments.
>>>The second WGLC was completed on 18 May 2012 with comments.
>>>
>>>Please read and express your opinion on whether or not these
>>>Internet-Drafts should be forwarded to the Area Directors for
>>>publication as Informational RFCs.  Send your comments
>>>to this list or acmorton@att.com
>>>
>>>Al
>>>bmwg chair
>>>_______________________________________________
>>>bmwg mailing list
>>>bmwg@ietf.org
>>>https://www.ietf.org/mailman/listinfo/bmwg
>>
>>_______________________________________________
>>bmwg mailing list
>>bmwg@ietf.org
>>https://www.ietf.org/mailman/listinfo/bmwg


From curtis@occnc.com  Mon Dec 10 12:59:51 2012
Return-Path: <curtis@occnc.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D9FBD21F8689 for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 12:59:51 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.495
X-Spam-Level: 
X-Spam-Status: No, score=-0.495 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_COM=0.553,  RDNS_NONE=0.1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4xuUwXG0lf9A for <bmwg@ietfa.amsl.com>; Mon, 10 Dec 2012 12:59:51 -0800 (PST)
Received: from gateway1.orleans.occnc.com (unknown [173.9.106.132]) by ietfa.amsl.com (Postfix) with ESMTP id A8FB321F85E0 for <bmwg@ietf.org>; Mon, 10 Dec 2012 12:59:50 -0800 (PST)
Received: from harbor1.ipv6.occnc.com (harbor1.ipv6.occnc.com [IPv6:2001:470:1f07:1545::2:819]) (authenticated bits=0) by gateway1.orleans.occnc.com (8.14.5/8.14.5) with ESMTP id qBAKwNo8046934; Mon, 10 Dec 2012 15:58:23 -0500 (EST) (envelope-from curtis@occnc.com)
Message-Id: <201212102058.qBAKwNo8046934@gateway1.orleans.occnc.com>
To: "Jay Karthik (jakarthi)" <jakarthi@cisco.com>
From: Curtis Villamizar <curtis@occnc.com>
In-reply-to: Your message of "Wed, 28 Nov 2012 02:18:53 GMT." <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.com>
Date: Mon, 10 Dec 2012 15:58:22 -0500
Cc: Al Morton <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Subject: Re: [bmwg] Benchmarking MPLS Forwarding
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: curtis@occnc.com
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 10 Dec 2012 20:59:52 -0000

Jay, Al,

Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
the comments.

See inline.

Curtis

In message <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.com>
"Jay Karthik (jakarthi)" writes:
> 
> Al,
>  
> I took a look at this draft and in my opinion, most (if not all) of
> the clarifications in section 1.1 would need to be benchmarked
> appropriately.  Many of these scenarios described are beyond the scope
> of previous work (MPLS Forwarding Compliance and Performance
> Requirements/RFC5695) which focused on label stack depth of 1.

Label stack depth is one variable that has not been considered so far
in BMWG.

Another is a look at packet forwarding rate vs packet size with a fine
granularity of packet size.  When doing so, an sawtooth is observed.
In my experience, the sawtooth is almost always due to memory width
and an optimization toward packets of with 64 byte payloads (payload
padded in ethernet, plus header yielding on the order of 84 byte
times) and then multiples of some power of two, often 64.  It is
common to store an internal representation of destination in a
relative small number of bytes, such as 16, and then store the payload
separately in other banks of RAM.  The result is that the worst
performance is that experienced by packts with payloads that are a
little larger than 64 bytes.  This is bad news if carrying something
like pseudowires that may have small payloads, but a little larger
than 64 bytes.

> The entire section 4 (Forwarding Compliance and Performance Testing)
> could potentially be a new work item.

I agree but also think it might be worth breaking this up functionally
into more than one work item.  There has also been discussion off list
aimed at expanding on the pseudowire requirements and adding mpls-tp
oam forwarding requirements (currently the impact on POP and counters
is called out, but not the OAM itself).

It is nice to hear that there is interest in BMWG.  It might be best
though to wait for this work to settle a bit in MPLS so as not to be
working toward too much of a moving target.

> Here are some comments on draft-villamizar-mpls-forwarding-00.

I'm reading -01-preview1, but I think it is the same here.

> If T#1 in section 4 fails, the max supported depth should be
> determined for a given implementation.

The current text says "While packets with more than a dozen label
entriess are unlikely to be used in any practical scenario today, it
is useful to know if limitations exists."

So we appear to be in full agreement.  If this is not clear enough,
please suggest new wording.

> Subcase T#3b needs to be modified. While benchmarking, a device is
> either a tester/tool or a DUT. Specifically, the procedure described
> should apply to any DUT and not a router that has certain capability.

I think the issue is the sentence "One way to accomplish this is to
use a router with higher priority set on the interfaces on which small
packets are sent to it."  That router is not the DUT.  It is another
router front ending the DUT and providing the merged flow of traffic.
I can make that more clear by moving the entire suggested test setup
to an appendix with an ascii line drawing and referring to it from the
main body of the document.

The point here is that router A (front end) gets packets on more than
one interface and sends them out one interface.  Router B, the DUT
receives the packets complete with bursts of small packets on one
interface.  Router A has two or more forwarding engines working on the
problem and therefore is more likely to not bog down the its
forwarding engines.  The output side of router A could also be a
bottleneck but only if the issue is purely output side buffering and
is not related to the decision engine.  Clearly more than a paragraph
in the main part of the document is needed.

> T#5 is a generic functional validation that could apply to most cases
> (eg T#1). Same comment applies to other cases, which validates
> functionality or tests conformance (eg T#10). These tests could be
> part of the validation procedure but may not represent a benchmark.

The point in T#5 is that we've seen equipment in which the on-chip
counters are small enough that if all traffic went to the same LSP,
the counters would have to be polled in under 2 seconds, under 1
second if you prefer to avoid a rollover by half the counter capacity.
Since you don't know apriori which LSP is going to have a long burst
of activity, a brute force solution would require polling all of the
counters every second to update the 64 bit external DRAM version.
Some architectures are close to unworkable due to very small on chip
counters.  This really is a performance issue.  There is no reason
that the on-chip counters must be 64 bit, as long as there is an
efficient way to offload the counters and get them into 64 bit
counters in DRAM for the purpose of returning MIB values.

I can see your argument that T#10 is really a basic conformance test,
however it is now under "Multipath Capabilities and Performance" and
while it is conformance, it is conformance specific to multipath.  I
think T#10 is fine where it is.

> A few more test scenarios could be created to measure the max label
> stack size with PW control word, flow label and Entropy label. This is
> different from T#1 where the focus is to measure in general, the
> maximum stack size supported by DUT, instead of characterizing the max
> supported in different scenarios.

Andy Malis and I have had a long email thread off list.  Expect to see
a lot more in the 01 draft on pseudowire.

> -Jay
>  
> On 11/8/12 9:51 AM, "Al Morton" <acmorton@att.com> wrote:
>  
> >Folks,
> >
> >Curtis suggested I take a look at this (I did),
> >http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-11
> >and I think others should take a look, too.
> >
> >Food for thought (at least) as we consider new work.
> >
> >regards,
> >Al
> >bmwg chair 

Thanks again for the comments.  It would be best to keep this on the
MPLS mailing list for now and return to BMWG later.  That said, the
MPLS WG list has been silent on this but I have been getting quite a
few comments in private email.  (Mostly from service providers.)

Curtis

From Internet-Drafts@ietf.org  Thu Dec 13 11:00:32 2012
Return-Path: <Internet-Drafts@ietf.org>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 40A8E21F8980; Thu, 13 Dec 2012 11:00:32 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.532
X-Spam-Level: 
X-Spam-Status: No, score=-102.532 tagged_above=-999 required=5 tests=[AWL=0.067, BAYES_00=-2.599, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1+wat-6-2qdQ; Thu, 13 Dec 2012 11:00:31 -0800 (PST)
Received: from ietfa.amsl.com (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id AE20C21F8699; Thu, 13 Dec 2012 11:00:31 -0800 (PST)
MIME-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
From: Internet-Drafts@ietf.org
To: i-d-announce@ietf.org
X-Test-IDTracker: no
X-IETF-IDTracker: 4.36
Message-ID: <20121213190031.26658.63819.idtracker@ietfa.amsl.com>
Date: Thu, 13 Dec 2012 11:00:31 -0800
Cc: bmwg@ietf.org
Subject: [bmwg] I-D ACTION:draft-ietf-bmwg-imix-genome-04.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Dec 2012 19:00:32 -0000

--NextPart

A new Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Benchmarking Methodology Working Group of the IETF.

    Title         : IMIX Genome: Specification of variable packet sizes for additional testing
    Author(s)     : A. Morton
    Filename      : draft-ietf-bmwg-imix-genome
    Pages         : 10 
    Date          : Dec. 13, 2012 
    
   Benchmarking Methodologies have always relied on test conditions with
   constant packet sizes, with the goal of understanding what network
   device capability has been tested.  Tests with constant packet size
   reveal device capabilities but differ significantly from the
   conditions encountered in operational deployment, and so additional
   tests are sometimes conducted with a mixture of packet sizes, or
   &quot;IMIX&quot;.  The mixture of sizes a networking device will encounter is
   highly variable and depends on many factors.  An IMIX suited for one
   networking device and deployment will not be appropriate for another.
   However, the mix of sizes may be known and the tester may be asked to
   augment the fixed size tests.  To address this need, and the
   perpetual goal of specifying repeatable test conditions, this draft
   defines a way to specify the exact repeating sequence of packet sizes
   from the usual set of fixed sizes, and other forms of mixed size
   specification.



A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-bmwg-imix-genome-04.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Message/External-body; name="draft-ietf-bmwg-imix-genome";
	site="ftp.ietf.org"; access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <2012-12-13110031.I-D@ietf.org>


--NextPart--

From acmorton@att.com  Thu Dec 13 12:21:32 2012
Return-Path: <acmorton@att.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6736321F8AD2 for <bmwg@ietfa.amsl.com>; Thu, 13 Dec 2012 12:21:32 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -106.307
X-Spam-Level: 
X-Spam-Status: No, score=-106.307 tagged_above=-999 required=5 tests=[AWL=0.292, BAYES_00=-2.599, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IHp7Wa-iA08m for <bmwg@ietfa.amsl.com>; Thu, 13 Dec 2012 12:21:31 -0800 (PST)
Received: from nbfkord-smmo08.seg.att.com (nbfkord-smmo08.seg.att.com [209.65.160.95]) by ietfa.amsl.com (Postfix) with ESMTP id A570721F8A99 for <bmwg@ietf.org>; Thu, 13 Dec 2012 12:21:31 -0800 (PST)
Received: from unknown [144.160.20.145] (EHLO mlpd192.enaf.sfdc.sbc.com) by nbfkord-smmo08.seg.att.com(mxl_mta-6.11.0-12) over TLS secured channel with ESMTP id bc83ac05.0.1591250.00-232.4410784.nbfkord-smmo08.seg.att.com (envelope-from <acmorton@att.com>);  Thu, 13 Dec 2012 20:21:31 +0000 (UTC)
X-MXL-Hash: 50ca38cb02107fd5-277bea5384bf49fc7b6a9f0f3eb2419157732b3f
Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBDKLUeq027107 for <bmwg@ietf.org>; Thu, 13 Dec 2012 15:21:30 -0500
Received: from sflint01.pst.cso.att.com (sflint01.pst.cso.att.com [144.154.234.228]) by mlpd192.enaf.sfdc.sbc.com (8.14.5/8.14.5) with ESMTP id qBDKLRYj027101 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for <bmwg@ietf.org>; Thu, 13 Dec 2012 15:21:27 -0500
Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by sflint01.pst.cso.att.com (RSA Interceptor) for <bmwg@ietf.org>; Thu, 13 Dec 2012 15:21:19 -0500
Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBDKLJqJ018624 for <bmwg@ietf.org>; Thu, 13 Dec 2012 15:21:19 -0500
Received: from dns.maillennium.att.com (maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id qBDKLCOx018406 for <bmwg@ietf.org>; Thu, 13 Dec 2012 15:21:12 -0500
Received: from lt-hp1044652.att.com (dn135-16-251-78.dhcpn.ugn.att.com[135.16.251.78](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20121213202101gw1006324re>; Thu, 13 Dec 2012 20:21:01 +0000
X-Originating-IP: [135.16.251.78]
Message-Id: <7.0.1.0.0.20121213151427.08c4fb98@att.com>
X-Mailer: QUALCOMM Windows Eudora Version 7.0.1.0
Date: Thu, 13 Dec 2012 15:19:23 -0500
To: bmwg@ietf.org, Ron Bonica <rbonica@juniper.net>
From: Al Morton <acmorton@att.com>
In-Reply-To: <20121213190031.26658.63819.idtracker@ietfa.amsl.com>
References: <20121213190031.26658.63819.idtracker@ietfa.amsl.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"; format=flowed
X-RSA-Inspected: yes
X-RSA-Classifications: public
X-Spam: [F=0.2000000000; CM=0.500; S=0.200(2010122901)]
X-MAIL-FROM: <acmorton@att.com>
X-SOURCE-IP: [144.160.20.145]
X-AnalysisOut: [v=2.0 cv=F8zggNdN c=1 sm=0 a=ZRNLZ4dFUbCvG8UMqPvVAA==:17 a]
X-AnalysisOut: [=DwyFDWaKD4wA:10 a=gmwGGNEp0e0A:10 a=ofMgfj31e3cA:10 a=BLc]
X-AnalysisOut: [eEmwcHowA:10 a=kj9zAlcOel0A:10 a=zQP7CpKOAAAA:8 a=eZ5YdXnF]
X-AnalysisOut: [AuUA:10 a=48vgC7mUAAAA:8 a=fpRFpa1pSrSnpb7KmUsA:9 a=CjuIK1]
X-AnalysisOut: [q_8ugA:10 a=AM14pv5mMGwA:10 a=lZB815dzVvQA:10]
Subject: Re: [bmwg] I-D ACTION:draft-ietf-bmwg-imix-genome-04.txt
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Dec 2012 20:21:32 -0000

BMWG,

I've revised this twice since the meeting, to address

- Scott Bradner's comments at the mic in Atlanta

- additional off-list comments, resulting in some clarifications

We'll have a WGLC on this in Jan 2013, but feel free to give it
a look now and put your opinion on advancement or comments in
a message to the list.

Ron Bonica, you'll be calling consensus on this, as agreed when we adopted
the draft.

regards,
Al


At 02:00 PM 12/13/2012, Internet-Drafts@ietf.org wrote:
>     Title         : IMIX Genome: Specification of variable packet 
> sizes for additional testing
>     Author(s)     : A. Morton
>     Filename      : draft-ietf-bmwg-imix-genome
>     Pages         : 10
>     Date          : Dec. 13, 2012
>
>...
>A URL for this Internet-Draft is:
>http://www.ietf.org/internet-drafts/draft-ietf-bmwg-imix-genome-04.txt


From cpignata@cisco.com  Sat Dec 15 14:30:17 2012
Return-Path: <cpignata@cisco.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5F30821F845A for <bmwg@ietfa.amsl.com>; Sat, 15 Dec 2012 14:30:17 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.599
X-Spam-Level: 
X-Spam-Status: No, score=-110.599 tagged_above=-999 required=5 tests=[AWL=0.000, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bICyoyQ7JJ3o for <bmwg@ietfa.amsl.com>; Sat, 15 Dec 2012 14:30:16 -0800 (PST)
Received: from rcdn-iport-5.cisco.com (rcdn-iport-5.cisco.com [173.37.86.76]) by ietfa.amsl.com (Postfix) with ESMTP id F011021F8441 for <bmwg@ietf.org>; Sat, 15 Dec 2012 14:30:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=8321; q=dns/txt; s=iport; t=1355610616; x=1356820216; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=tgIFqPXLyAOqQy7hFoqyhzc9zvfQcFz3qM9R1iLOXLE=; b=dSLnZuJDA2/iMNZmZGIepNzpNDLiXbD3UHLFtb5Qw2Tr+Vk+XKCgsatW 9coiPxE02476NwhZNnVWqkrwu6M+FpmQ/+RigqZPNJuhptYS62jsRPUyI Clh6XQ+USvcC4HRKXgL2fuBKrCK0ztAJjS1e/yiv6TO6mEoUixEEUBYlO A=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgAFAJz4zFCtJXG+/2dsb2JhbABFvlMWc4IXBwEBAQMBAQEBNzQLBQsCAQgYChQQJwslAgQOBQgSAYUuB4I9Bgy7AIxdCgERg0ZhA5JXhE+PLIJzgWYHFwYY
X-IronPort-AV: E=Sophos;i="4.84,291,1355097600"; d="scan'208";a="153370210"
Received: from rcdn-core2-3.cisco.com ([173.37.113.190]) by rcdn-iport-5.cisco.com with ESMTP; 15 Dec 2012 22:30:15 +0000
Received: from xhc-rcd-x11.cisco.com (xhc-rcd-x11.cisco.com [173.37.183.85]) by rcdn-core2-3.cisco.com (8.14.5/8.14.5) with ESMTP id qBFMUFLR014063 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Sat, 15 Dec 2012 22:30:15 GMT
Received: from xmb-aln-x02.cisco.com ([169.254.5.91]) by xhc-rcd-x11.cisco.com ([173.37.183.85]) with mapi id 14.02.0318.004; Sat, 15 Dec 2012 16:30:14 -0600
From: "Carlos Pignataro (cpignata)" <cpignata@cisco.com>
To: "curtis@occnc.com" <curtis@occnc.com>
Thread-Topic: [bmwg] Benchmarking MPLS Forwarding
Thread-Index: AQHN2xPAnc6KQ7H9tUaiqk4it/yafg==
Date: Sat, 15 Dec 2012 22:30:14 +0000
Message-ID: <95067C434CE250468B77282634C96ED3227D0971@xmb-aln-x02.cisco.com>
References: <201212102058.qBAKwNo8046934@gateway1.orleans.occnc.com>
In-Reply-To: <201212102058.qBAKwNo8046934@gateway1.orleans.occnc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.117.115.57]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <AFE5517661AD2B4DB4550A386CAEE66A@cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: Al Morton <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Subject: Re: [bmwg] Benchmarking MPLS Forwarding
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 15 Dec 2012 22:30:17 -0000

Curtis,

This is a very nicely written document that calls out important topics. Ple=
ase find some bmwg-focused comments inline.

On Dec 10, 2012, at 3:58 PM, Curtis Villamizar <curtis@occnc.com> wrote:

>=20
> Jay, Al,
>=20
> Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
> the comments.

Currently the document filename is draft-villamizar-mpls-forwarding, from w=
hich it could be extrapolated (1id-guidelines) that it targets the mpls wg,=
 with a document name of "forwarding". However, there seem to be two multip=
le to this document: clarification on misconceptions, and conformance and p=
erformance requirements (and benchmarking?). More on this below.

>=20
> See inline.
>=20
> Curtis
>=20
> In message <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.co=
m>
> "Jay Karthik (jakarthi)" writes:
>>=20
>> Al,
>>=20
>> I took a look at this draft and in my opinion, most (if not all) of
>> the clarifications in section 1.1 would need to be benchmarked
>> appropriately.  Many of these scenarios described are beyond the scope
>> of previous work (MPLS Forwarding Compliance and Performance
>> Requirements/RFC5695) which focused on label stack depth of 1.
>=20
> Label stack depth is one variable that has not been considered so far
> in BMWG.
>=20
> Another is a look at packet forwarding rate vs packet size with a fine
> granularity of packet size.  When doing so, an sawtooth is observed.
> In my experience, the sawtooth is almost always due to memory width
> and an optimization toward packets of with 64 byte payloads (payload
> padded in ethernet, plus header yielding on the order of 84 byte
> times) and then multiples of some power of two, often 64.  It is
> common to store an internal representation of destination in a
> relative small number of bytes, such as 16, and then store the payload
> separately in other banks of RAM.  The result is that the worst
> performance is that experienced by packts with payloads that are a
> little larger than 64 bytes.  This is bad news if carrying something
> like pseudowires that may have small payloads, but a little larger
> than 64 bytes.
>=20
>> The entire section 4 (Forwarding Compliance and Performance Testing)
>> could potentially be a new work item.
>=20
> I agree but also think it might be worth breaking this up functionally
> into more than one work item.  There has also been discussion off list
> aimed at expanding on the pseudowire requirements and adding mpls-tp
> oam forwarding requirements (currently the impact on POP and counters
> is called out, but not the OAM itself).
>=20
> It is nice to hear that there is interest in BMWG.  It might be best
> though to wait for this work to settle a bit in MPLS so as not to be
> working toward too much of a moving target.

I also tend to thing that S4 is a bit its own document (or plural if extend=
ed in scope). I wonder also how much of a continuation of RFC 5695 some par=
ts should be.=20

>=20
>> Here are some comments on draft-villamizar-mpls-forwarding-00.
>=20
> I'm reading -01-preview1, but I think it is the same here.
>=20
>> If T#1 in section 4 fails, the max supported depth should be
>> determined for a given implementation.
>=20
> The current text says "While packets with more than a dozen label
> entriess are unlikely to be used in any practical scenario today, it
> is useful to know if limitations exists."
>=20
> So we appear to be in full agreement.  If this is not clear enough,
> please suggest new wording.
>=20
>> Subcase T#3b needs to be modified. While benchmarking, a device is
>> either a tester/tool or a DUT. Specifically, the procedure described
>> should apply to any DUT and not a router that has certain capability.
>=20
> I think the issue is the sentence "One way to accomplish this is to
> use a router with higher priority set on the interfaces on which small
> packets are sent to it."  That router is not the DUT.  It is another
> router front ending the DUT and providing the merged flow of traffic.
> I can make that more clear by moving the entire suggested test setup
> to an appendix with an ascii line drawing and referring to it from the
> main body of the document.
>=20
> The point here is that router A (front end) gets packets on more than
> one interface and sends them out one interface.  Router B, the DUT
> receives the packets complete with bursts of small packets on one
> interface.  Router A has two or more forwarding engines working on the
> problem and therefore is more likely to not bog down the its
> forwarding engines.  The output side of router A could also be a
> bottleneck but only if the issue is purely output side buffering and
> is not related to the decision engine.  Clearly more than a paragraph
> in the main part of the document is needed.
>=20

One thought is that perhaps defining up front a topology and methodology ca=
n make these easier to follow and more mechanic.


>> T#5 is a generic functional validation that could apply to most cases
>> (eg T#1). Same comment applies to other cases, which validates
>> functionality or tests conformance (eg T#10). These tests could be
>> part of the validation procedure but may not represent a benchmark.
>=20
> The point in T#5 is that we've seen equipment in which the on-chip
> counters are small enough that if all traffic went to the same LSP,
> the counters would have to be polled in under 2 seconds, under 1
> second if you prefer to avoid a rollover by half the counter capacity.

While I like testing for this, and like this test, it might appear that it =
is a bit specific as compared to others. For example, these is no mention o=
f spreading or dispersing the forwarding among multiple LSPs versus sending=
 all the traffic in one LSP for forwarding performance; however there is th=
is mention in the context of counters.

> Since you don't know apriori which LSP is going to have a long burst
> of activity, a brute force solution would require polling all of the
> counters every second to update the 64 bit external DRAM version.
> Some architectures are close to unworkable due to very small on chip
> counters.  This really is a performance issue.  There is no reason
> that the on-chip counters must be 64 bit, as long as there is an
> efficient way to offload the counters and get them into 64 bit
> counters in DRAM for the purpose of returning MIB values.
>=20
> I can see your argument that T#10 is really a basic conformance test,
> however it is now under "Multipath Capabilities and Performance" and
> while it is conformance, it is conformance specific to multipath.  I
> think T#10 is fine where it is.

The location of T#10 makes sense in terms of multipath testing. However, it=
 is not necessarily obvious or clear the methodology to make such determina=
tion (especially as it is proving a negative).

>=20
>> A few more test scenarios could be created to measure the max label
>> stack size with PW control word, flow label and Entropy label. This is
>> different from T#1 where the focus is to measure in general, the
>> maximum stack size supported by DUT, instead of characterizing the max
>> supported in different scenarios.
>=20
> Andy Malis and I have had a long email thread off list.  Expect to see
> a lot more in the 01 draft on pseudowire.
>=20
>> -Jay
>>=20
>> On 11/8/12 9:51 AM, "Al Morton" <acmorton@att.com> wrote:
>>=20
>>> Folks,
>>>=20
>>> Curtis suggested I take a look at this (I did),
>>> http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-11
>>> and I think others should take a look, too.
>>>=20
>>> Food for thought (at least) as we consider new work.
>>>=20
>>> regards,
>>> Al
>>> bmwg chair=20
>=20
> Thanks again for the comments.  It would be best to keep this on the
> MPLS mailing list for now and return to BMWG later.  That said, the
> MPLS WG list has been silent on this but I have been getting quite a
> few comments in private email.  (Mostly from service providers.)
>=20
> Curtis
> _______________________________________________
> bmwg mailing list
> bmwg@ietf.org
> https://www.ietf.org/mailman/listinfo/bmwg
>=20

Thanks,

-- Carlos.


From curtis@occnc.com  Wed Dec 19 11:57:29 2012
Return-Path: <curtis@occnc.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 67D7E21F8750 for <bmwg@ietfa.amsl.com>; Wed, 19 Dec 2012 11:57:29 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.495
X-Spam-Level: 
X-Spam-Status: No, score=-0.495 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_COM=0.553,  RDNS_NONE=0.1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6ihVVBd3O5k5 for <bmwg@ietfa.amsl.com>; Wed, 19 Dec 2012 11:57:28 -0800 (PST)
Received: from gateway1.orleans.occnc.com (unknown [173.9.106.132]) by ietfa.amsl.com (Postfix) with ESMTP id 075B121F8742 for <bmwg@ietf.org>; Wed, 19 Dec 2012 11:57:27 -0800 (PST)
Received: from harbor1.ipv6.occnc.com (harbor1.ipv6.occnc.com [IPv6:2001:470:1f07:1545::2:819]) (authenticated bits=0) by gateway1.orleans.occnc.com (8.14.5/8.14.5) with ESMTP id qBJJu0HM007184; Wed, 19 Dec 2012 14:56:00 -0500 (EST) (envelope-from curtis@occnc.com)
Message-Id: <201212191956.qBJJu0HM007184@gateway1.orleans.occnc.com>
To: "Carlos Pignataro (cpignata)" <cpignata@cisco.com>
From: Curtis Villamizar <curtis@occnc.com>
In-reply-to: Your message of "Sat, 15 Dec 2012 22:30:14 GMT." <95067C434CE250468B77282634C96ED3227D0971@xmb-aln-x02.cisco.com>
Date: Wed, 19 Dec 2012 14:56:00 -0500
Cc: Al Morton <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Subject: Re: [bmwg] Benchmarking MPLS Forwarding
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: curtis@occnc.com
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 19 Dec 2012 19:57:29 -0000

Carlos,

Thanks for the comments.  Please keep in mind that this discussion is
a "heads up" from Al to BMWG about work in the MPLS WG.  The MPLS work
is more about requirements and is not about precise specification of
benchmarks which would be better dealt with in BMWG.

See inline.

In message <95067C434CE250468B77282634C96ED3227D0971@xmb-aln-x02.cisco.com>
"Carlos Pignataro (cpignata)" writes:
> 
> Curtis,
>  
> This is a very nicely written document that calls out important
> topics. Please find some bmwg-focused comments inline.
>  
> On Dec 10, 2012, at 3:58 PM, Curtis Villamizar <curtis@occnc.com> wrote:
>  
> > 
> > Jay, Al,
> > 
> > Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
> > the comments.
>  
> Currently the document filename is draft-villamizar-mpls-forwarding,
> from which it could be extrapolated (1id-guidelines) that it targets
> the mpls wg, with a document name of "forwarding". However, there seem
> to be two multiple to this document: clarification on misconceptions,
> and conformance and performance requirements (and benchmarking?). More
> on this below.

You are correct.  draft-villamizar-mpls-forwarding is work intended
for MPLS WG about MPLS forwarding.

There is a bit of text in the draft on motivation for this work (a few
chip vendors got it very wrong, a few with a common theme - hence the
clarification on misconceptions).

> > See inline.
> > 
> > Curtis
> > 
> > In message <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.com>
> > "Jay Karthik (jakarthi)" writes:
> >> 
> >> Al,
> >> 
> >> I took a look at this draft and in my opinion, most (if not all) of
> >> the clarifications in section 1.1 would need to be benchmarked
> >> appropriately.  Many of these scenarios described are beyond the scope
> >> of previous work (MPLS Forwarding Compliance and Performance
> >> Requirements/RFC5695) which focused on label stack depth of 1.
> > 
> > Label stack depth is one variable that has not been considered so far
> > in BMWG.
> > 
> > Another is a look at packet forwarding rate vs packet size with a fine
> > granularity of packet size.  When doing so, an sawtooth is observed.
> > In my experience, the sawtooth is almost always due to memory width
> > and an optimization toward packets of with 64 byte payloads (payload
> > padded in ethernet, plus header yielding on the order of 84 byte
> > times) and then multiples of some power of two, often 64.  It is
> > common to store an internal representation of destination in a
> > relative small number of bytes, such as 16, and then store the payload
> > separately in other banks of RAM.  The result is that the worst
> > performance is that experienced by packts with payloads that are a
> > little larger than 64 bytes.  This is bad news if carrying something
> > like pseudowires that may have small payloads, but a little larger
> > than 64 bytes.
> > 
> >> The entire section 4 (Forwarding Compliance and Performance Testing)
> >> could potentially be a new work item.
> > 
> > I agree but also think it might be worth breaking this up functionally
> > into more than one work item.  There has also been discussion off list
> > aimed at expanding on the pseudowire requirements and adding mpls-tp
> > oam forwarding requirements (currently the impact on POP and counters
> > is called out, but not the OAM itself).
> > 
> > It is nice to hear that there is interest in BMWG.  It might be best
> > though to wait for this work to settle a bit in MPLS so as not to be
> > working toward too much of a moving target.
>  
> I also tend to thing that S4 is a bit its own document (or plural if
> extended in scope). I wonder also how much of a continuation of RFC
> 5695 some parts should be.

Section 4 is intended to be an outline of things to test for, without
specifying a methodology for testing in the way that BMWG does.

I should reference RFC 5695 as existing work.

However, RFC 5695, like other BMWG work misses the sawtooth that can
only be discovered using smaller increments in packet size, typically
large powers of two starting with 64 bytes.  This is highly
*counterproductive* since chip vendors may then design for the
benchmark and put forward designs which are poorly suited for real
world traffic.

There is a tendency going all the way back to the Bradner IP tests in
the early 1990s (hi Scott, sorry to bring this up again, but..) in
BMWG to precisely specify benchmarks whish prove inadequate and are
therefore counterproductive.

IMHO this tendency can be corrected by first considering the
requirements in greater detail, and *then* specifying a set of
benchmarks.  The focus must be less on making the benchmarks quick and
easy to perform and report and more on making the benchmarks
adequately reflect real world network requirements.

Doing work on requirements in MPLS WG first, then work on benchmarks
in BMWG *might* better accomplish this.

> >> Here are some comments on draft-villamizar-mpls-forwarding-00.
> > 
> > I'm reading -01-preview1, but I think it is the same here.
> > 
> >> If T#1 in section 4 fails, the max supported depth should be
> >> determined for a given implementation.
> > 
> > The current text says "While packets with more than a dozen label
> > entriess are unlikely to be used in any practical scenario today, it
> > is useful to know if limitations exists."
> > 
> > So we appear to be in full agreement.  If this is not clear enough,
> > please suggest new wording.
> > 
> >> Subcase T#3b needs to be modified. While benchmarking, a device is
> >> either a tester/tool or a DUT. Specifically, the procedure described
> >> should apply to any DUT and not a router that has certain capability.
> > 
> > I think the issue is the sentence "One way to accomplish this is to
> > use a router with higher priority set on the interfaces on which small
> > packets are sent to it."  That router is not the DUT.  It is another
> > router front ending the DUT and providing the merged flow of traffic.
> > I can make that more clear by moving the entire suggested test setup
> > to an appendix with an ascii line drawing and referring to it from the
> > main body of the document.
> > 
> > The point here is that router A (front end) gets packets on more than
> > one interface and sends them out one interface.  Router B, the DUT
> > receives the packets complete with bursts of small packets on one
> > interface.  Router A has two or more forwarding engines working on the
> > problem and therefore is more likely to not bog down the its
> > forwarding engines.  The output side of router A could also be a
> > bottleneck but only if the issue is purely output side buffering and
> > is not related to the decision engine.  Clearly more than a paragraph
> > in the main part of the document is needed.
> > 
>  
> One thought is that perhaps defining up front a topology and
> methodology can make these easier to follow and more mechanic.

The goal of the MPLS work is *not* to precisely specify a set of
benchmarks, although BMWG is welcome to do so.

There was some text in Subcase T#3b regarding the feasibility of
testing (for very high speed interfaces such as 100Bb/s interfaces)
which was too terse to be clear (as Jay pointed out).

That text should be either removed entirely or moved to an appendix
and expanded on.  If expanded on it should be made clear that the text
only exists to indicate a means of inserting bursts of small packets
of known burst durection into streams of larger packets.  That text
was entirely for the benefit of anyone that might wonder how it would
be feasible to do this and maybe should just be removed.

> >> T#5 is a generic functional validation that could apply to most cases
> >> (eg T#1). Same comment applies to other cases, which validates
> >> functionality or tests conformance (eg T#10). These tests could be
> >> part of the validation procedure but may not represent a benchmark.
> > 
> > The point in T#5 is that we've seen equipment in which the on-chip
> > counters are small enough that if all traffic went to the same LSP,
> > the counters would have to be polled in under 2 seconds, under 1
> > second if you prefer to avoid a rollover by half the counter capacity.
>  
> While I like testing for this, and like this test, it might appear
> that it is a bit specific as compared to others. For example, these is
> no mention of spreading or dispersing the forwarding among multiple
> LSPs versus sending all the traffic in one LSP for forwarding
> performance; however there is this mention in the context of counters.

In the update that I've been working on after IETF-85, there is more
coverage of large flow and large number of flow issues.  That section
was prompted by a conversation just after the MPLS WG presentation and
some follow up private email.

The counter problem has been overlooked by some chip vendors
accounting for it getting into the list of tests before other large
flow and large number of flow issues.

> > Since you don't know apriori which LSP is going to have a long burst
> > of activity, a brute force solution would require polling all of the
> > counters every second to update the 64 bit external DRAM version.
> > Some architectures are close to unworkable due to very small on chip
> > counters.  This really is a performance issue.  There is no reason
> > that the on-chip counters must be 64 bit, as long as there is an
> > efficient way to offload the counters and get them into 64 bit
> > counters in DRAM for the purpose of returning MIB values.
> > 
> > I can see your argument that T#10 is really a basic conformance test,
> > however it is now under "Multipath Capabilities and Performance" and
> > while it is conformance, it is conformance specific to multipath.  I
> > think T#10 is fine where it is.
>  
> The location of T#10 makes sense in terms of multipath
> testing. However, it is not necessarily obvious or clear the
> methodology to make such determination (especially as it is proving a
> negative).

Currently T#10 reads:

       T#10  Determine whether reserved labels are included in the
             label stack hash.  They MUST NOT be included.

The MUST NOT is new and the change from SHOULD NOT is due to many
comments asking for change.  It is needed for OAM to work right.

I could change the wording to

       T#10  Determine whether reserved labels are excluded from the
             label stack hash.  They MUST be excluded.

The way to make this determination is send traffic and then send the
same traffic with a router-alert or GAL label down the stack a ways.
If the traffic splits differently with the reserved label, then you
have identified a problem.

You could also just carry multiple MPLS-TP LSP inside MPLS LSP without
EL (TP label must be at the very bottom) and use the TP LM OAM and see
if it comes out right.  If the counts are off a little , the reserved
labels were used and there was a little bit of reordering.

[ Or you could wait for it to happen in the field and then deal with
the complaints.  just kidding. ]

> >> A few more test scenarios could be created to measure the max label
> >> stack size with PW control word, flow label and Entropy label. This is
> >> different from T#1 where the focus is to measure in general, the
> >> maximum stack size supported by DUT, instead of characterizing the max
> >> supported in different scenarios.
> > 
> > Andy Malis and I have had a long email thread off list.  Expect to see
> > a lot more in the 01 draft on pseudowire.
> > 
> >> -Jay
> >> 
> >> On 11/8/12 9:51 AM, "Al Morton" <acmorton@att.com> wrote:
> >> 
> >>> Folks,
> >>> 
> >>> Curtis suggested I take a look at this (I did),
> >>> http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-11
> >>> and I think others should take a look, too.
> >>> 
> >>> Food for thought (at least) as we consider new work.
> >>> 
> >>> regards,
> >>> Al
> >>> bmwg chair 
> > 
> > Thanks again for the comments.  It would be best to keep this on the
> > MPLS mailing list for now and return to BMWG later.  That said, the
> > MPLS WG list has been silent on this but I have been getting quite a
> > few comments in private email.  (Mostly from service providers.)
> > 
> > Curtis
> > _______________________________________________
> > bmwg mailing list
> > bmwg@ietf.org
> > https://www.ietf.org/mailman/listinfo/bmwg
> > 
>  
> Thanks,
>  
> -- Carlos.

Thanks again for the comments.  

I will do the following: 1) point out the applicability of RFC 5695,
2) reword T#10 to eliminate the double negative, 3) do something
(exactly what is TBD) about the extraneous methodology hints in T#3b
that Jay pointed out.

Curtis

From cpignata@cisco.com  Thu Dec 20 07:30:38 2012
Return-Path: <cpignata@cisco.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 560E021F88BE for <bmwg@ietfa.amsl.com>; Thu, 20 Dec 2012 07:30:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -110.073
X-Spam-Level: 
X-Spam-Status: No, score=-110.073 tagged_above=-999 required=5 tests=[AWL=0.526, BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8, USER_IN_WHITELIST=-100]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FAsw06CY4gtd for <bmwg@ietfa.amsl.com>; Thu, 20 Dec 2012 07:30:37 -0800 (PST)
Received: from rcdn-iport-4.cisco.com (rcdn-iport-4.cisco.com [173.37.86.75]) by ietfa.amsl.com (Postfix) with ESMTP id 404DC21F88B4 for <bmwg@ietf.org>; Thu, 20 Dec 2012 07:30:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=13875; q=dns/txt; s=iport; t=1356017436; x=1357227036; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-id:content-transfer-encoding: mime-version; bh=qTOt1yF7wLF8EkyKdQ3ussyeI3d1pD1yikXs/CdnpVU=; b=CjwyKXoDrt/JQKQ6i5gyKJgx8yjwsMFmZPorFWmfbXEq3QKbK6AGxjZf RQqlrl3DQ5OLlcoTr26LXa9KiY8/EbQ5kqvSP6zC66/SkF/yI055Zcau/ 6A5Tsppmv1mMf7O55jNsDahH8yIi2kZf4/ro4/0Y7wMB0yykMdEF5UmTE I=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AgAFAPkt01CtJV2c/2dsb2JhbABEvW4Wc4IXBwEBAQMBAQEBNzQLBQsCAQgYChQQJwslAgQOBQgSAYUuB4I9Bgy5PoxSCgERg0ZhA5JYhE+PLIJ0gWYHFwYY
X-IronPort-AV: E=Sophos;i="4.84,324,1355097600"; d="scan'208";a="155138257"
Received: from rcdn-core-5.cisco.com ([173.37.93.156]) by rcdn-iport-4.cisco.com with ESMTP; 20 Dec 2012 15:30:35 +0000
Received: from xhc-aln-x13.cisco.com (xhc-aln-x13.cisco.com [173.36.12.87]) by rcdn-core-5.cisco.com (8.14.5/8.14.5) with ESMTP id qBKFUZIS006060 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL); Thu, 20 Dec 2012 15:30:35 GMT
Received: from xmb-aln-x02.cisco.com ([169.254.5.159]) by xhc-aln-x13.cisco.com ([173.36.12.87]) with mapi id 14.02.0318.004; Thu, 20 Dec 2012 09:30:35 -0600
From: "Carlos Pignataro (cpignata)" <cpignata@cisco.com>
To: "<curtis@occnc.com>" <curtis@occnc.com>
Thread-Topic: [bmwg] Benchmarking MPLS Forwarding
Thread-Index: AQHN3iLnQd/k8zqy2k+ZcVjSipCuLJgiNjWA
Date: Thu, 20 Dec 2012 15:30:34 +0000
Message-ID: <95067C434CE250468B77282634C96ED322810070@xmb-aln-x02.cisco.com>
References: <201212191956.qBJJu0HM007184@gateway1.orleans.occnc.com>
In-Reply-To: <201212191956.qBJJu0HM007184@gateway1.orleans.occnc.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-originating-ip: [10.150.52.103]
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E37F81C4FD0C68479CD2F9CBD1289891@cisco.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: Al Morton <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Subject: Re: [bmwg] Benchmarking MPLS Forwarding
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Dec 2012 15:30:38 -0000

Thanks for the detailed response, Curtis.

A couple of follow-ups inline.

On Dec 19, 2012, at 2:56 PM, Curtis Villamizar <curtis@occnc.com> wrote:

>=20
> Carlos,
>=20
> Thanks for the comments.  Please keep in mind that this discussion is
> a "heads up" from Al to BMWG about work in the MPLS WG.  The MPLS work
> is more about requirements and is not about precise specification of
> benchmarks which would be better dealt with in BMWG.
>=20
> See inline.
>=20
> In message <95067C434CE250468B77282634C96ED3227D0971@xmb-aln-x02.cisco.co=
m>
> "Carlos Pignataro (cpignata)" writes:
>>=20
>> Curtis,
>>=20
>> This is a very nicely written document that calls out important
>> topics. Please find some bmwg-focused comments inline.
>>=20
>> On Dec 10, 2012, at 3:58 PM, Curtis Villamizar <curtis@occnc.com> wrote:
>>=20
>>>=20
>>> Jay, Al,
>>>=20
>>> Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
>>> the comments.
>>=20
>> Currently the document filename is draft-villamizar-mpls-forwarding,
>> from which it could be extrapolated (1id-guidelines) that it targets
>> the mpls wg, with a document name of "forwarding". However, there seem
>> to be two multiple to this document: clarification on misconceptions,
>> and conformance and performance requirements (and benchmarking?). More
>> on this below.
>=20
> You are correct.  draft-villamizar-mpls-forwarding is work intended
> for MPLS WG about MPLS forwarding.
>=20
> There is a bit of text in the draft on motivation for this work (a few
> chip vendors got it very wrong, a few with a common theme - hence the
> clarification on misconceptions).
>=20
>>> See inline.
>>>=20
>>> Curtis
>>>=20
>>> In message <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.=
com>
>>> "Jay Karthik (jakarthi)" writes:
>>>>=20
>>>> Al,
>>>>=20
>>>> I took a look at this draft and in my opinion, most (if not all) of
>>>> the clarifications in section 1.1 would need to be benchmarked
>>>> appropriately.  Many of these scenarios described are beyond the scope
>>>> of previous work (MPLS Forwarding Compliance and Performance
>>>> Requirements/RFC5695) which focused on label stack depth of 1.
>>>=20
>>> Label stack depth is one variable that has not been considered so far
>>> in BMWG.
>>>=20
>>> Another is a look at packet forwarding rate vs packet size with a fine
>>> granularity of packet size.  When doing so, an sawtooth is observed.
>>> In my experience, the sawtooth is almost always due to memory width
>>> and an optimization toward packets of with 64 byte payloads (payload
>>> padded in ethernet, plus header yielding on the order of 84 byte
>>> times) and then multiples of some power of two, often 64.  It is
>>> common to store an internal representation of destination in a
>>> relative small number of bytes, such as 16, and then store the payload
>>> separately in other banks of RAM.  The result is that the worst
>>> performance is that experienced by packts with payloads that are a
>>> little larger than 64 bytes.  This is bad news if carrying something
>>> like pseudowires that may have small payloads, but a little larger
>>> than 64 bytes.
>>>=20
>>>> The entire section 4 (Forwarding Compliance and Performance Testing)
>>>> could potentially be a new work item.
>>>=20
>>> I agree but also think it might be worth breaking this up functionally
>>> into more than one work item.  There has also been discussion off list
>>> aimed at expanding on the pseudowire requirements and adding mpls-tp
>>> oam forwarding requirements (currently the impact on POP and counters
>>> is called out, but not the OAM itself).
>>>=20
>>> It is nice to hear that there is interest in BMWG.  It might be best
>>> though to wait for this work to settle a bit in MPLS so as not to be
>>> working toward too much of a moving target.
>>=20
>> I also tend to thing that S4 is a bit its own document (or plural if
>> extended in scope). I wonder also how much of a continuation of RFC
>> 5695 some parts should be.
>=20
> Section 4 is intended to be an outline of things to test for, without
> specifying a methodology for testing in the way that BMWG does.
>=20
> I should reference RFC 5695 as existing work.
>=20
> However, RFC 5695, like other BMWG work misses the sawtooth that can
> only be discovered using smaller increments in packet size, typically
> large powers of two starting with 64 bytes.  This is highly
> *counterproductive* since chip vendors may then design for the
> benchmark and put forward designs which are poorly suited for real
> world traffic.
>=20
> There is a tendency going all the way back to the Bradner IP tests in
> the early 1990s (hi Scott, sorry to bring this up again, but..) in
> BMWG to precisely specify benchmarks whish prove inadequate and are
> therefore counterproductive.
>=20
> IMHO this tendency can be corrected by first considering the
> requirements in greater detail, and *then* specifying a set of
> benchmarks.  The focus must be less on making the benchmarks quick and
> easy to perform and report and more on making the benchmarks
> adequately reflect real world network requirements.
>=20
> Doing work on requirements in MPLS WG first, then work on benchmarks
> in BMWG *might* better accomplish this.
>=20
>>>> Here are some comments on draft-villamizar-mpls-forwarding-00.
>>>=20
>>> I'm reading -01-preview1, but I think it is the same here.
>>>=20
>>>> If T#1 in section 4 fails, the max supported depth should be
>>>> determined for a given implementation.
>>>=20
>>> The current text says "While packets with more than a dozen label
>>> entriess are unlikely to be used in any practical scenario today, it
>>> is useful to know if limitations exists."
>>>=20
>>> So we appear to be in full agreement.  If this is not clear enough,
>>> please suggest new wording.
>>>=20
>>>> Subcase T#3b needs to be modified. While benchmarking, a device is
>>>> either a tester/tool or a DUT. Specifically, the procedure described
>>>> should apply to any DUT and not a router that has certain capability.
>>>=20
>>> I think the issue is the sentence "One way to accomplish this is to
>>> use a router with higher priority set on the interfaces on which small
>>> packets are sent to it."  That router is not the DUT.  It is another
>>> router front ending the DUT and providing the merged flow of traffic.
>>> I can make that more clear by moving the entire suggested test setup
>>> to an appendix with an ascii line drawing and referring to it from the
>>> main body of the document.
>>>=20
>>> The point here is that router A (front end) gets packets on more than
>>> one interface and sends them out one interface.  Router B, the DUT
>>> receives the packets complete with bursts of small packets on one
>>> interface.  Router A has two or more forwarding engines working on the
>>> problem and therefore is more likely to not bog down the its
>>> forwarding engines.  The output side of router A could also be a
>>> bottleneck but only if the issue is purely output side buffering and
>>> is not related to the decision engine.  Clearly more than a paragraph
>>> in the main part of the document is needed.
>>>=20
>>=20
>> One thought is that perhaps defining up front a topology and
>> methodology can make these easier to follow and more mechanic.
>=20
> The goal of the MPLS work is *not* to precisely specify a set of
> benchmarks, although BMWG is welcome to do so.
>=20
> There was some text in Subcase T#3b regarding the feasibility of
> testing (for very high speed interfaces such as 100Bb/s interfaces)
> which was too terse to be clear (as Jay pointed out).
>=20
> That text should be either removed entirely or moved to an appendix
> and expanded on.  If expanded on it should be made clear that the text
> only exists to indicate a means of inserting bursts of small packets
> of known burst durection into streams of larger packets.  That text
> was entirely for the benefit of anyone that might wonder how it would
> be feasible to do this and maybe should just be removed.
>=20
>>>> T#5 is a generic functional validation that could apply to most cases
>>>> (eg T#1). Same comment applies to other cases, which validates
>>>> functionality or tests conformance (eg T#10). These tests could be
>>>> part of the validation procedure but may not represent a benchmark.
>>>=20
>>> The point in T#5 is that we've seen equipment in which the on-chip
>>> counters are small enough that if all traffic went to the same LSP,
>>> the counters would have to be polled in under 2 seconds, under 1
>>> second if you prefer to avoid a rollover by half the counter capacity.
>>=20
>> While I like testing for this, and like this test, it might appear
>> that it is a bit specific as compared to others. For example, these is
>> no mention of spreading or dispersing the forwarding among multiple
>> LSPs versus sending all the traffic in one LSP for forwarding
>> performance; however there is this mention in the context of counters.
>=20
> In the update that I've been working on after IETF-85, there is more
> coverage of large flow and large number of flow issues.  That section
> was prompted by a conversation just after the MPLS WG presentation and
> some follow up private email.
>=20
> The counter problem has been overlooked by some chip vendors
> accounting for it getting into the list of tests before other large
> flow and large number of flow issues.
>=20
>>> Since you don't know apriori which LSP is going to have a long burst
>>> of activity, a brute force solution would require polling all of the
>>> counters every second to update the 64 bit external DRAM version.
>>> Some architectures are close to unworkable due to very small on chip
>>> counters.  This really is a performance issue.  There is no reason
>>> that the on-chip counters must be 64 bit, as long as there is an
>>> efficient way to offload the counters and get them into 64 bit
>>> counters in DRAM for the purpose of returning MIB values.
>>>=20
>>> I can see your argument that T#10 is really a basic conformance test,
>>> however it is now under "Multipath Capabilities and Performance" and
>>> while it is conformance, it is conformance specific to multipath.  I
>>> think T#10 is fine where it is.
>>=20
>> The location of T#10 makes sense in terms of multipath
>> testing. However, it is not necessarily obvious or clear the
>> methodology to make such determination (especially as it is proving a
>> negative).
>=20
> Currently T#10 reads:
>=20
>       T#10  Determine whether reserved labels are included in the
>             label stack hash.  They MUST NOT be included.
>=20
> The MUST NOT is new and the change from SHOULD NOT is due to many
> comments asking for change.  It is needed for OAM to work right.
>=20
> I could change the wording to
>=20
>       T#10  Determine whether reserved labels are excluded from the
>             label stack hash.  They MUST be excluded.
>=20
> The way to make this determination is send traffic and then send the
> same traffic with a router-alert or GAL label down the stack a ways.
> If the traffic splits differently with the reserved label, then you
> have identified a problem.

Yes. But if the traffic does not split differently, it does not mean you do=
 not have a problem. How much precision do you feel is needed in the specif=
ication of testing cases like this? Should all testings use the same method=
ology to perform the determination in T#10?

>=20
> You could also just carry multiple MPLS-TP LSP inside MPLS LSP without
> EL (TP label must be at the very bottom) and use the TP LM OAM and see
> if it comes out right.  If the counts are off a little , the reserved
> labels were used and there was a little bit of reordering.
>=20
> [ Or you could wait for it to happen in the field and then deal with
> the complaints.  just kidding. ]
>=20
>>>> A few more test scenarios could be created to measure the max label
>>>> stack size with PW control word, flow label and Entropy label. This is
>>>> different from T#1 where the focus is to measure in general, the
>>>> maximum stack size supported by DUT, instead of characterizing the max
>>>> supported in different scenarios.
>>>=20
>>> Andy Malis and I have had a long email thread off list.  Expect to see
>>> a lot more in the 01 draft on pseudowire.
>>>=20
>>>> -Jay
>>>>=20
>>>> On 11/8/12 9:51 AM, "Al Morton" <acmorton@att.com> wrote:
>>>>=20
>>>>> Folks,
>>>>>=20
>>>>> Curtis suggested I take a look at this (I did),
>>>>> http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-1=
1
>>>>> and I think others should take a look, too.
>>>>>=20
>>>>> Food for thought (at least) as we consider new work.
>>>>>=20
>>>>> regards,
>>>>> Al
>>>>> bmwg chair=20
>>>=20
>>> Thanks again for the comments.  It would be best to keep this on the
>>> MPLS mailing list for now and return to BMWG later.  That said, the
>>> MPLS WG list has been silent on this but I have been getting quite a
>>> few comments in private email.  (Mostly from service providers.)
>>>=20
>>> Curtis
>>> _______________________________________________
>>> bmwg mailing list
>>> bmwg@ietf.org
>>> https://www.ietf.org/mailman/listinfo/bmwg
>>>=20
>>=20
>> Thanks,
>>=20
>> -- Carlos.
>=20
> Thanks again for the comments. =20

Thank you!

>=20
> I will do the following: 1) point out the applicability of RFC 5695,
> 2) reword T#10 to eliminate the double negative, 3) do something
> (exactly what is TBD) about the extraneous methodology hints in T#3b
> that Jay pointed out.
>=20

Sounds good. Thanks,

-- Carlos.


> Curtis
>=20


From curtis@occnc.com  Thu Dec 20 10:12:10 2012
Return-Path: <curtis@occnc.com>
X-Original-To: bmwg@ietfa.amsl.com
Delivered-To: bmwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3250221F8A87 for <bmwg@ietfa.amsl.com>; Thu, 20 Dec 2012 10:12:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.495
X-Spam-Level: 
X-Spam-Status: No, score=-0.495 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599, FH_RELAY_NODNS=1.451, HELO_MISMATCH_COM=0.553,  RDNS_NONE=0.1]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3U3pFmt0pqjN for <bmwg@ietfa.amsl.com>; Thu, 20 Dec 2012 10:12:08 -0800 (PST)
Received: from gateway1.orleans.occnc.com (unknown [173.9.106.132]) by ietfa.amsl.com (Postfix) with ESMTP id 9FF9621F8866 for <bmwg@ietf.org>; Thu, 20 Dec 2012 10:11:58 -0800 (PST)
Received: from harbor1.ipv6.occnc.com (harbor1.ipv6.occnc.com [IPv6:2001:470:1f07:1545::2:819]) (authenticated bits=0) by gateway1.orleans.occnc.com (8.14.5/8.14.5) with ESMTP id qBKIAbrF019741; Thu, 20 Dec 2012 13:10:37 -0500 (EST) (envelope-from curtis@occnc.com)
Message-Id: <201212201810.qBKIAbrF019741@gateway1.orleans.occnc.com>
To: "Carlos Pignataro (cpignata)" <cpignata@cisco.com>
From: Curtis Villamizar <curtis@occnc.com>
In-reply-to: Your message of "Thu, 20 Dec 2012 15:30:34 GMT." <95067C434CE250468B77282634C96ED322810070@xmb-aln-x02.cisco.com>
Date: Thu, 20 Dec 2012 13:10:36 -0500
Cc: Al Morton <acmorton@att.com>, "bmwg@ietf.org" <bmwg@ietf.org>
Subject: Re: [bmwg] Benchmarking MPLS Forwarding
X-BeenThere: bmwg@ietf.org
X-Mailman-Version: 2.1.12
Precedence: list
Reply-To: curtis@occnc.com
List-Id: Benchmarking Methodology Working Group <bmwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/bmwg>, <mailto:bmwg-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/bmwg>
List-Post: <mailto:bmwg@ietf.org>
List-Help: <mailto:bmwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/bmwg>, <mailto:bmwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Dec 2012 18:12:10 -0000

Carlos,

Inline.

In message <95067C434CE250468B77282634C96ED322810070@xmb-aln-x02.cisco.com>
"Carlos Pignataro (cpignata)" writes:
> 
> Thanks for the detailed response, Curtis.
>  
> A couple of follow-ups inline.
>  
> On Dec 19, 2012, at 2:56 PM, Curtis Villamizar <curtis@occnc.com> wrote:
>  
> > 
> > Carlos,
> > 
> > Thanks for the comments.  Please keep in mind that this discussion is
> > a "heads up" from Al to BMWG about work in the MPLS WG.  The MPLS work
> > is more about requirements and is not about precise specification of
> > benchmarks which would be better dealt with in BMWG.
> > 
> > See inline.
> > 
> > In message <95067C434CE250468B77282634C96ED3227D0971@xmb-aln-x02.cisco.com>
> > "Carlos Pignataro (cpignata)" writes:
> >> 
> >> Curtis,
> >> 
> >> This is a very nicely written document that calls out important
> >> topics. Please find some bmwg-focused comments inline.
> >> 
> >> On Dec 10, 2012, at 3:58 PM, Curtis Villamizar <curtis@occnc.com> wrote:
> >> 
> >>> 
> >>> Jay, Al,
> >>> 
> >>> Thanks for looking at the mpls-forwarding draft.  Jay - thank you for
> >>> the comments.
> >> 
> >> Currently the document filename is draft-villamizar-mpls-forwarding,
> >> from which it could be extrapolated (1id-guidelines) that it targets
> >> the mpls wg, with a document name of "forwarding". However, there seem
> >> to be two multiple to this document: clarification on misconceptions,
> >> and conformance and performance requirements (and benchmarking?). More
> >> on this below.
> > 
> > You are correct.  draft-villamizar-mpls-forwarding is work intended
> > for MPLS WG about MPLS forwarding.
> > 
> > There is a bit of text in the draft on motivation for this work (a few
> > chip vendors got it very wrong, a few with a common theme - hence the
> > clarification on misconceptions).
> > 
> >>> See inline.
> >>> 
> >>> Curtis
> >>> 
> >>> In message <C6009DF6226EF44B90DD04DC2DFF768B1509971F@xmb-aln-x10.cisco.com>
> >>> "Jay Karthik (jakarthi)" writes:
> >>>> 
> >>>> Al,
> >>>> 
> >>>> I took a look at this draft and in my opinion, most (if not all) of
> >>>> the clarifications in section 1.1 would need to be benchmarked
> >>>> appropriately.  Many of these scenarios described are beyond the scope
> >>>> of previous work (MPLS Forwarding Compliance and Performance
> >>>> Requirements/RFC5695) which focused on label stack depth of 1.
> >>> 
> >>> Label stack depth is one variable that has not been considered so far
> >>> in BMWG.
> >>> 
> >>> Another is a look at packet forwarding rate vs packet size with a fine
> >>> granularity of packet size.  When doing so, an sawtooth is observed.
> >>> In my experience, the sawtooth is almost always due to memory width
> >>> and an optimization toward packets of with 64 byte payloads (payload
> >>> padded in ethernet, plus header yielding on the order of 84 byte
> >>> times) and then multiples of some power of two, often 64.  It is
> >>> common to store an internal representation of destination in a
> >>> relative small number of bytes, such as 16, and then store the payload
> >>> separately in other banks of RAM.  The result is that the worst
> >>> performance is that experienced by packts with payloads that are a
> >>> little larger than 64 bytes.  This is bad news if carrying something
> >>> like pseudowires that may have small payloads, but a little larger
> >>> than 64 bytes.
> >>> 
> >>>> The entire section 4 (Forwarding Compliance and Performance Testing)
> >>>> could potentially be a new work item.
> >>> 
> >>> I agree but also think it might be worth breaking this up functionally
> >>> into more than one work item.  There has also been discussion off list
> >>> aimed at expanding on the pseudowire requirements and adding mpls-tp
> >>> oam forwarding requirements (currently the impact on POP and counters
> >>> is called out, but not the OAM itself).
> >>> 
> >>> It is nice to hear that there is interest in BMWG.  It might be best
> >>> though to wait for this work to settle a bit in MPLS so as not to be
> >>> working toward too much of a moving target.
> >> 
> >> I also tend to thing that S4 is a bit its own document (or plural if
> >> extended in scope). I wonder also how much of a continuation of RFC
> >> 5695 some parts should be.
> > 
> > Section 4 is intended to be an outline of things to test for, without
> > specifying a methodology for testing in the way that BMWG does.
> > 
> > I should reference RFC 5695 as existing work.
> > 
> > However, RFC 5695, like other BMWG work misses the sawtooth that can
> > only be discovered using smaller increments in packet size, typically
> > large powers of two starting with 64 bytes.  This is highly
> > *counterproductive* since chip vendors may then design for the
> > benchmark and put forward designs which are poorly suited for real
> > world traffic.
> > 
> > There is a tendency going all the way back to the Bradner IP tests in
> > the early 1990s (hi Scott, sorry to bring this up again, but..) in
> > BMWG to precisely specify benchmarks whish prove inadequate and are
> > therefore counterproductive.
> > 
> > IMHO this tendency can be corrected by first considering the
> > requirements in greater detail, and *then* specifying a set of
> > benchmarks.  The focus must be less on making the benchmarks quick and
> > easy to perform and report and more on making the benchmarks
> > adequately reflect real world network requirements.
> > 
> > Doing work on requirements in MPLS WG first, then work on benchmarks
> > in BMWG *might* better accomplish this.
> > 
> >>>> Here are some comments on draft-villamizar-mpls-forwarding-00.
> >>> 
> >>> I'm reading -01-preview1, but I think it is the same here.
> >>> 
> >>>> If T#1 in section 4 fails, the max supported depth should be
> >>>> determined for a given implementation.
> >>> 
> >>> The current text says "While packets with more than a dozen label
> >>> entriess are unlikely to be used in any practical scenario today, it
> >>> is useful to know if limitations exists."
> >>> 
> >>> So we appear to be in full agreement.  If this is not clear enough,
> >>> please suggest new wording.
> >>> 
> >>>> Subcase T#3b needs to be modified. While benchmarking, a device is
> >>>> either a tester/tool or a DUT. Specifically, the procedure described
> >>>> should apply to any DUT and not a router that has certain capability.
> >>> 
> >>> I think the issue is the sentence "One way to accomplish this is to
> >>> use a router with higher priority set on the interfaces on which small
> >>> packets are sent to it."  That router is not the DUT.  It is another
> >>> router front ending the DUT and providing the merged flow of traffic.
> >>> I can make that more clear by moving the entire suggested test setup
> >>> to an appendix with an ascii line drawing and referring to it from the
> >>> main body of the document.
> >>> 
> >>> The point here is that router A (front end) gets packets on more than
> >>> one interface and sends them out one interface.  Router B, the DUT
> >>> receives the packets complete with bursts of small packets on one
> >>> interface.  Router A has two or more forwarding engines working on the
> >>> problem and therefore is more likely to not bog down the its
> >>> forwarding engines.  The output side of router A could also be a
> >>> bottleneck but only if the issue is purely output side buffering and
> >>> is not related to the decision engine.  Clearly more than a paragraph
> >>> in the main part of the document is needed.
> >>> 
> >> 
> >> One thought is that perhaps defining up front a topology and
> >> methodology can make these easier to follow and more mechanic.
> > 
> > The goal of the MPLS work is *not* to precisely specify a set of
> > benchmarks, although BMWG is welcome to do so.
> > 
> > There was some text in Subcase T#3b regarding the feasibility of
> > testing (for very high speed interfaces such as 100Bb/s interfaces)
> > which was too terse to be clear (as Jay pointed out).
> > 
> > That text should be either removed entirely or moved to an appendix
> > and expanded on.  If expanded on it should be made clear that the text
> > only exists to indicate a means of inserting bursts of small packets
> > of known burst durection into streams of larger packets.  That text
> > was entirely for the benefit of anyone that might wonder how it would
> > be feasible to do this and maybe should just be removed.
> > 
> >>>> T#5 is a generic functional validation that could apply to most cases
> >>>> (eg T#1). Same comment applies to other cases, which validates
> >>>> functionality or tests conformance (eg T#10). These tests could be
> >>>> part of the validation procedure but may not represent a benchmark.
> >>> 
> >>> The point in T#5 is that we've seen equipment in which the on-chip
> >>> counters are small enough that if all traffic went to the same LSP,
> >>> the counters would have to be polled in under 2 seconds, under 1
> >>> second if you prefer to avoid a rollover by half the counter capacity.
> >> 
> >> While I like testing for this, and like this test, it might appear
> >> that it is a bit specific as compared to others. For example, these is
> >> no mention of spreading or dispersing the forwarding among multiple
> >> LSPs versus sending all the traffic in one LSP for forwarding
> >> performance; however there is this mention in the context of counters.
> > 
> > In the update that I've been working on after IETF-85, there is more
> > coverage of large flow and large number of flow issues.  That section
> > was prompted by a conversation just after the MPLS WG presentation and
> > some follow up private email.
> > 
> > The counter problem has been overlooked by some chip vendors
> > accounting for it getting into the list of tests before other large
> > flow and large number of flow issues.
> > 
> >>> Since you don't know apriori which LSP is going to have a long burst
> >>> of activity, a brute force solution would require polling all of the
> >>> counters every second to update the 64 bit external DRAM version.
> >>> Some architectures are close to unworkable due to very small on chip
> >>> counters.  This really is a performance issue.  There is no reason
> >>> that the on-chip counters must be 64 bit, as long as there is an
> >>> efficient way to offload the counters and get them into 64 bit
> >>> counters in DRAM for the purpose of returning MIB values.
> >>> 
> >>> I can see your argument that T#10 is really a basic conformance test,
> >>> however it is now under "Multipath Capabilities and Performance" and
> >>> while it is conformance, it is conformance specific to multipath.  I
> >>> think T#10 is fine where it is.
> >> 
> >> The location of T#10 makes sense in terms of multipath
> >> testing. However, it is not necessarily obvious or clear the
> >> methodology to make such determination (especially as it is proving a
> >> negative).
> > 
> > Currently T#10 reads:
> > 
> >       T#10  Determine whether reserved labels are included in the
> >             label stack hash.  They MUST NOT be included.
> > 
> > The MUST NOT is new and the change from SHOULD NOT is due to many
> > comments asking for change.  It is needed for OAM to work right.
> > 
> > I could change the wording to
> > 
> >       T#10  Determine whether reserved labels are excluded from the
> >             label stack hash.  They MUST be excluded.
> > 
> > The way to make this determination is send traffic and then send the
> > same traffic with a router-alert or GAL label down the stack a ways.
> > If the traffic splits differently with the reserved label, then you
> > have identified a problem.
>  
> Yes. But if the traffic does not split differently, it does not mean
> you do not have a problem. How much precision do you feel is needed in
> the specification of testing cases like this? Should all testings use
> the same methodology to perform the determination in T#10?

Use far more than one label stack in the test so that you can't
possibly get a false indication of success.  But this MPLS document is
not about testing methodology.  My response is just part of a
discussion with you on feasibility of doing this sort of testing.
This could end up in a BMWG document *later* but doesn't belong in
the MPLS document.

> > 
> > You could also just carry multiple MPLS-TP LSP inside MPLS LSP without
> > EL (TP label must be at the very bottom) and use the TP LM OAM and see
> > if it comes out right.  If the counts are off a little , the reserved
> > labels were used and there was a little bit of reordering.
> > 
> > [ Or you could wait for it to happen in the field and then deal with
> > the complaints.  just kidding. ]
> > 
> >>>> A few more test scenarios could be created to measure the max label
> >>>> stack size with PW control word, flow label and Entropy label. This is
> >>>> different from T#1 where the focus is to measure in general, the
> >>>> maximum stack size supported by DUT, instead of characterizing the max
> >>>> supported in different scenarios.
> >>> 
> >>> Andy Malis and I have had a long email thread off list.  Expect to see
> >>> a lot more in the 01 draft on pseudowire.
> >>> 
> >>>> -Jay
> >>>> 
> >>>> On 11/8/12 9:51 AM, "Al Morton" <acmorton@att.com> wrote:
> >>>> 
> >>>>> Folks,
> >>>>> 
> >>>>> Curtis suggested I take a look at this (I did),
> >>>>> http://tools.ietf.org/html/draft-villamizar-mpls-forwarding-00#page-11
> >>>>> and I think others should take a look, too.
> >>>>> 
> >>>>> Food for thought (at least) as we consider new work.
> >>>>> 
> >>>>> regards,
> >>>>> Al
> >>>>> bmwg chair 
> >>> 
> >>> Thanks again for the comments.  It would be best to keep this on the
> >>> MPLS mailing list for now and return to BMWG later.  That said, the
> >>> MPLS WG list has been silent on this but I have been getting quite a
> >>> few comments in private email.  (Mostly from service providers.)
> >>> 
> >>> Curtis
> >>> _______________________________________________
> >>> bmwg mailing list
> >>> bmwg@ietf.org
> >>> https://www.ietf.org/mailman/listinfo/bmwg
> >>> 
> >> 
> >> Thanks,
> >> 
> >> -- Carlos.
> > 
> > Thanks again for the comments.  
>  
> Thank you!
>  
> > 
> > I will do the following: 1) point out the applicability of RFC 5695,
> > 2) reword T#10 to eliminate the double negative, 3) do something
> > (exactly what is TBD) about the extraneous methodology hints in T#3b
> > that Jay pointed out.
> > 
>  
> Sounds good. Thanks,
>  
> -- Carlos.

Thanks again for the comments and followup.  I will send you diffs off
list when the next authors 01-preview comes out and if you like you
can check to see if your comments are adequately addressed before we
send out an 01 version.

Curtis
