Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 7920B3A699A; Tue, 23 Nov 2010 09:45:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.471 X-Spam-Level: X-Spam-Status: No, score=-102.471 tagged_above=-999 required=5 tests=[AWL=0.128, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3tHKpykeOpQp; Tue, 23 Nov 2010 09:45:03 -0800 (PST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 57FB328C167; Tue, 23 Nov 2010 09:45:02 -0800 (PST) MIME-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" From: Internet-Drafts@ietf.org To: i-d-announce@ietf.org X-Test-IDTracker: no X-IETF-IDTracker: 3.09 Message-ID: <20101123174502.24017.91688.idtracker@localhost> Date: Tue, 23 Nov 2010 09:45:02 -0800 Cc: pmol@ietf.org Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-06.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 17:45:04 -0000 --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF. Title : Guidelines for Considering New Performance Metric Development Author(s) : A. Clark, B. Claise Filename : draft-ietf-pmol-metrics-framework-06.txt Pages : 22 Date : 2010-11-23 This document describes a framework and a process for developing performance metrics of protocols and applications transported over over IETF-specified protocols, and that can be used to characterize traffic on live networks and services. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-06.txt Internet-Drafts are also available by anonymous FTP at: ftp://ftp.ietf.org/internet-drafts/ Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Message/External-body; name="draft-ietf-pmol-metrics-framework-06.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2010-11-23094259.I-D@ietf.org> --NextPart-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8F4813A6990 for ; Tue, 23 Nov 2010 09:44:24 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.584 X-Spam-Level: X-Spam-Status: No, score=-2.584 tagged_above=-999 required=5 tests=[AWL=0.014, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E9TZRUeRgq40 for ; Tue, 23 Nov 2010 09:44:23 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 7F8BD28C0D9 for ; Tue, 23 Nov 2010 09:44:23 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oANHjKlO027680 for ; Tue, 23 Nov 2010 18:45:20 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oANHjKFB028985 for ; Tue, 23 Nov 2010 18:45:20 +0100 (CET) Message-ID: <4CEBFDB0.3000007@cisco.com> Date: Tue, 23 Nov 2010 18:45:20 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "pmol@ietf.org" Content-Type: multipart/alternative; boundary="------------090203090707030406070107" Subject: [PMOL] Fwd: New Version Notification for draft-ietf-pmol-metrics-framework-06 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 17:44:24 -0000 This is a multi-part message in MIME format. --------------090203090707030406070107 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Dear all, Here is the new version of the PMOL framework, which takes into account the WGLC feedbacks. Regards, Benoit. -------- Original Message -------- Subject: New Version Notification for draft-ietf-pmol-metrics-framework-06 Date: Tue, 23 Nov 2010 09:42:59 -0800 (PST) From: IETF I-D Submission Tool To: bclaise@cisco.com CC: alan.d.clark@telchemy.com A new version of I-D, draft-ietf-pmol-metrics-framework-06.txt has been successfully submitted by Benoit Claise and posted to the IETF repository. Filename: draft-ietf-pmol-metrics-framework Revision: 06 Title: Guidelines for Considering New Performance Metric Development Creation_date: 2010-11-23 WG ID: pmol Number_of_pages: 22 Abstract: This document describes a framework and a process for developing performance metrics of protocols and applications transported over over IETF-specified protocols, and that can be used to characterize traffic on live networks and services. The IETF Secretariat. --------------090203090707030406070107 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit Dear all,

Here is the new version of the PMOL framework, which takes into account the WGLC feedbacks.

Regards, Benoit.

-------- Original Message --------
Subject: New Version Notification for draft-ietf-pmol-metrics-framework-06
Date: Tue, 23 Nov 2010 09:42:59 -0800 (PST)
From: IETF I-D Submission Tool <idsubmission@ietf.org>
To: bclaise@cisco.com
CC: alan.d.clark@telchemy.com 


A new version of I-D, draft-ietf-pmol-metrics-framework-06.txt has been successfully submitted by Benoit Claise and posted to the IETF repository.

Filename:	 draft-ietf-pmol-metrics-framework
Revision:	 06
Title:		 Guidelines for Considering New Performance Metric Development
Creation_date:	 2010-11-23
WG ID:		 pmol
Number_of_pages: 22

Abstract:
This document describes a framework and a process for developing
performance metrics of protocols and applications transported over
over IETF-specified protocols, and that can be used to characterize
traffic on live networks and services.
                                                                                  


The IETF Secretariat.
--------------090203090707030406070107-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id F343F3A691B for ; Tue, 23 Nov 2010 03:38:15 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.151 X-Spam-Level: X-Spam-Status: No, score=-2.151 tagged_above=-999 required=5 tests=[AWL=-0.437, BAYES_00=-2.599, HTML_FONT_FACE_BAD=0.884, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zp2Z5dttQ2q2 for ; Tue, 23 Nov 2010 03:38:14 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 924A43A6901 for ; Tue, 23 Nov 2010 03:38:14 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oANBJ87m005976; Tue, 23 Nov 2010 12:19:09 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oANBJ8DH024168; Tue, 23 Nov 2010 12:19:08 +0100 (CET) Message-ID: <4CEBA32C.7080003@cisco.com> Date: Tue, 23 Nov 2010 12:19:08 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Romascanu, Dan (Dan)" References: <4CEA8AED.9050307@cisco.com> <4CEB8DB4.7050502@cisco.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------090902060704010507040602" Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 11:38:16 -0000 This is a multi-part message in MIME format. --------------090902060704010507040602 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Done. Thanks for your review. Regards, Benoit. > Yes, this looks OK. > Just as an editing comment, I would rather: > s/The IETF does have current and completed activities related to the > reporting of application performance metrics: for example/Previous > IETF work related to reporting of application performance metrics > includes/ > Thanks and Regards, > Dan > (speaking as contributor) > > ------------------------------------------------------------------------ > *From:* Benoit Claise [mailto:bclaise@cisco.com] > *Sent:* Tuesday, November 23, 2010 11:48 AM > *To:* Romascanu, Dan (Dan) > *Cc:* Alan Clark; pmol@ietf.org > *Subject:* Re: [PMOL] I-D > Action:draft-ietf-pmol-metrics-framework-05.txt > > Dan, >> >> >>> -----Original Message----- >>> From: Benoit Claise [mailto:bclaise@cisco.com] >>> Sent: Monday, November 22, 2010 5:23 PM >>> To: Alan Clark >>> Cc: Romascanu, Dan (Dan); Al Morton;pmol@ietf.org >>> Subject: Re: [PMOL] I-D >>> Action:draft-ietf-pmol-metrics-framework-05.txt >>> >>> Alan, >>>> Apologies for my long absence!! >>>> >>>> A few comments >>>> >>>> 1. Introduction - last sentence >>>> >>>> Standardized metrics help to ensure that performance measurement is >>>> implemented consistently and facilitate interpretation and >>> comparison >>> Done. >>>> 1.1 Background - paragraph 5 >>>> >>>> Suggest the addition of RFC3611 and (soon to be) RFC6035 as >>> these are >>>> widely used IETF protocols for VoIP performance reporting. >>> The RAQMON >>>> description reads a little like an advert - so maybe >>>> >>>> "for example, the RTP Control Protocol Extended Reports >>> (RTCP XR) [RFC3611]. >>>> the Real -time Application Quality-of-Service Monitoring (RAQMON) >>>> Framework [RFC4710] and the SIP RTCP Summary Report >>> Protocol [RC6035] >>>> are protocols that support the real-time reporting of Voice over IP >>>> and other applications running on devices such as IP phones >>> and mobile handsets." >>> I propose to keep the RAQMON section, as it deals with >>> application performance metrics, but to add a new paragraph >>> >>> The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] >>> and the SIP RTCP Summary Report Protocol [RC6035] are >>> protocols that support the real-time reporting of Voice over >>> IP and other applications running on devices such as IP >>> phones and mobile handsets. >>> >> RAQMON also supports real-time reporting. > What we have right now is in the to-be-posted version: > > The IETF does have current and completed > activities related to the > reporting of application performance metrics: for > example the Real-time > Application Quality-of-Service Monitoring (RAQMON) > Framework [RFC4710], > which extends the remote network monitoring > (RMON) family of specifications to allow > _real-time_ quality-of-service (QoS) > monitoring of various applications that run on > devices such as IP phones, > pagers, Instant Messaging clients, mobile phones, > and various other > handheld computing devices. Furthermore, the RTP > Control Protocol Extended > Reports (RTCP XR) [RFC3611] and the SIP RTCP > Summary Report Protocol > [RC6035] are protocols that support the > real-time reporting of Voice over IP and other > applications running on > devices such as IP phones and mobile handsets. > > Please let me know if this is fine with you you. > > Regards, Benoit. > >> Dan >> (speaking as contributor) >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol > > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol --------------090902060704010507040602 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Done.

Thanks for your review.

Regards, Benoit.
Yes, this looks OK.
 
Just as an editing comment, I would rather:
 
s/The IETF does have current and completed activities related to the reporting of application performance metrics: for example/Previous IETF work related to reporting of application performance metrics includes/
 
Thanks and Regards,
 
Dan
(speaking as contributor)

From: Benoit Claise [mailto:bclaise@cisco.com]
Sent: Tuesday, November 23, 2010 11:48 AM
To: Romascanu, Dan (Dan)
Cc: Alan Clark; pmol@ietf.org
Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt

Dan, 
 


-----Original Message-----
From: Benoit Claise [mailto:bclaise@cisco.com] 
Sent: Monday, November 22, 2010 5:23 PM
To: Alan Clark
Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org
Subject: Re: [PMOL] I-D 
Action:draft-ietf-pmol-metrics-framework-05.txt

Alan,

Apologies for my long absence!!

A few comments

1. Introduction - last sentence

Standardized metrics help to ensure that performance measurement is 
implemented consistently and facilitate interpretation and 

comparison
Done.

1.1 Background - paragraph 5

Suggest the addition of RFC3611 and (soon to be) RFC6035 as 

these are 

widely used IETF protocols for VoIP performance reporting.  

The RAQMON 

description reads a little like an advert - so maybe

"for example, the RTP Control Protocol Extended Reports 

(RTCP XR) [RFC3611].

the Real -time Application Quality-of-Service Monitoring (RAQMON) 
Framework [RFC4710] and the SIP RTCP Summary Report 

Protocol [RC6035] 

are protocols that support the real-time reporting of Voice over IP 
and other applications running on devices such as IP phones 

and mobile handsets."
I propose to keep the RAQMON section, as it deals with 
application performance metrics, but to add a new paragraph

The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] 
and the SIP RTCP Summary Report Protocol [RC6035] are 
protocols that support the real-time reporting of Voice over 
IP and other applications running on devices such as IP 
phones and mobile handsets.


RAQMON also supports real-time reporting.
What we have right now is in the to-be-posted version:

                The IETF does have current and completed activities related to the
                reporting of application performance metrics: for example the Real-time
                Application Quality-of-Service Monitoring (RAQMON) Framework [RFC4710],
                which extends the remote network monitoring
                (RMON) family of specifications to allow real-time quality-of-service (QoS)
                monitoring of various applications that run on devices such as IP phones,
                pagers, Instant Messaging clients, mobile phones, and various other
                handheld computing devices. Furthermore, the RTP Control Protocol Extended
                Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol
                [RC6035] are protocols that support the
                 real-time reporting of Voice over IP and other applications running on
                devices such as IP phones and mobile handsets.

Please let me know if this is fine with you you.

Regards, Benoit.

Dan
(speaking as contributor)
_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol



_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------090902060704010507040602-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id F07963A68F0 for ; Tue, 23 Nov 2010 02:42:04 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.593 X-Spam-Level: X-Spam-Status: No, score=-2.593 tagged_above=-999 required=5 tests=[AWL=0.005, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wzr14yY36sgw for ; Tue, 23 Nov 2010 02:42:02 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 0DF583A68EB for ; Tue, 23 Nov 2010 02:42:01 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAN9lXb0022713; Tue, 23 Nov 2010 10:47:33 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAN9lWSi021843; Tue, 23 Nov 2010 10:47:32 +0100 (CET) Message-ID: <4CEB8DB4.7050502@cisco.com> Date: Tue, 23 Nov 2010 10:47:32 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Romascanu, Dan (Dan)" References: <4CEA8AED.9050307@cisco.com> In-Reply-To: Content-Type: multipart/alternative; boundary="------------060604090702070809030309" Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 10:42:05 -0000 This is a multi-part message in MIME format. --------------060604090702070809030309 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Dan, > > >> -----Original Message----- >> From: Benoit Claise [mailto:bclaise@cisco.com] >> Sent: Monday, November 22, 2010 5:23 PM >> To: Alan Clark >> Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org >> Subject: Re: [PMOL] I-D >> Action:draft-ietf-pmol-metrics-framework-05.txt >> >> Alan, >>> Apologies for my long absence!! >>> >>> A few comments >>> >>> 1. Introduction - last sentence >>> >>> Standardized metrics help to ensure that performance measurement is >>> implemented consistently and facilitate interpretation and >> comparison >> Done. >>> 1.1 Background - paragraph 5 >>> >>> Suggest the addition of RFC3611 and (soon to be) RFC6035 as >> these are >>> widely used IETF protocols for VoIP performance reporting. >> The RAQMON >>> description reads a little like an advert - so maybe >>> >>> "for example, the RTP Control Protocol Extended Reports >> (RTCP XR) [RFC3611]. >>> the Real -time Application Quality-of-Service Monitoring (RAQMON) >>> Framework [RFC4710] and the SIP RTCP Summary Report >> Protocol [RC6035] >>> are protocols that support the real-time reporting of Voice over IP >>> and other applications running on devices such as IP phones >> and mobile handsets." >> I propose to keep the RAQMON section, as it deals with >> application performance metrics, but to add a new paragraph >> >> The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] >> and the SIP RTCP Summary Report Protocol [RC6035] are >> protocols that support the real-time reporting of Voice over >> IP and other applications running on devices such as IP >> phones and mobile handsets. >> > RAQMON also supports real-time reporting. What we have right now is in the to-be-posted version: The IETF does have current and completed activities related to the reporting of application performance metrics: for example the Real-time Application Quality-of-Service Monitoring (RAQMON) Framework [RFC4710], which extends the remote network monitoring (RMON) family of specifications to allow _real-time_ quality-of-service (QoS) monitoring of various applications that run on devices such as IP phones, pagers, Instant Messaging clients, mobile phones, and various other handheld computing devices. Furthermore, the RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol [RC6035] are protocols that support the real-time reporting of Voice over IP and other applications running on devices such as IP phones and mobile handsets. Please let me know if this is fine with you you. Regards, Benoit. > Dan > (speaking as contributor) > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol --------------060604090702070809030309 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Dan, 
 


-----Original Message-----
From: Benoit Claise [mailto:bclaise@cisco.com] 
Sent: Monday, November 22, 2010 5:23 PM
To: Alan Clark
Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org
Subject: Re: [PMOL] I-D 
Action:draft-ietf-pmol-metrics-framework-05.txt

Alan,

Apologies for my long absence!!

A few comments

1. Introduction - last sentence

Standardized metrics help to ensure that performance measurement is 
implemented consistently and facilitate interpretation and 

comparison
Done.

1.1 Background - paragraph 5

Suggest the addition of RFC3611 and (soon to be) RFC6035 as 

these are 

widely used IETF protocols for VoIP performance reporting.  

The RAQMON 

description reads a little like an advert - so maybe

"for example, the RTP Control Protocol Extended Reports 

(RTCP XR) [RFC3611].

the Real -time Application Quality-of-Service Monitoring (RAQMON) 
Framework [RFC4710] and the SIP RTCP Summary Report 

Protocol [RC6035] 

are protocols that support the real-time reporting of Voice over IP 
and other applications running on devices such as IP phones 

and mobile handsets."
I propose to keep the RAQMON section, as it deals with 
application performance metrics, but to add a new paragraph

The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] 
and the SIP RTCP Summary Report Protocol [RC6035] are 
protocols that support the real-time reporting of Voice over 
IP and other applications running on devices such as IP 
phones and mobile handsets.


RAQMON also supports real-time reporting.
What we have right now is in the to-be-posted version:

                The IETF does have current and completed activities related to the
                reporting of application performance metrics: for example the Real-time
                Application Quality-of-Service Monitoring (RAQMON) Framework [RFC4710],
                which extends the remote network monitoring
                (RMON) family of specifications to allow real-time quality-of-service (QoS)
                monitoring of various applications that run on devices such as IP phones,
                pagers, Instant Messaging clients, mobile phones, and various other
                handheld computing devices. Furthermore, the RTP Control Protocol Extended
                Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol
                [RC6035] are protocols that support the
                 real-time reporting of Voice over IP and other applications running on
                devices such as IP phones and mobile handsets.

Please let me know if this is fine with you you.

Regards, Benoit.

Dan
(speaking as contributor)
_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------060604090702070809030309-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4B2A63A68C1 for ; Tue, 23 Nov 2010 02:03:10 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.533 X-Spam-Level: X-Spam-Status: No, score=-102.533 tagged_above=-999 required=5 tests=[AWL=0.065, BAYES_00=-2.599, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h-zItPtcwZ7x for ; Tue, 23 Nov 2010 02:03:09 -0800 (PST) Received: from co300216-co-outbound.net.avaya.com (co300216-co-outbound.net.avaya.com [198.152.13.100]) by core3.amsl.com (Postfix) with ESMTP id F32B93A68BF for ; Tue, 23 Nov 2010 02:03:08 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApADAGYg60yHCzI1/2dsb2JhbACCAJIejkRxo2ICmRuFSwSOBg X-IronPort-AV: E=Sophos;i="4.59,241,1288584000"; d="scan'208,217";a="251412540" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by co300216-co-outbound.net.avaya.com with ESMTP; 23 Nov 2010 05:04:05 -0500 X-IronPort-AV: E=Sophos;i="4.59,241,1288584000"; d="scan'208,217";a="543696540" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.12]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 23 Nov 2010 05:04:05 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CB8AF5.C10F01B5" Date: Tue, 23 Nov 2010 11:04:00 +0100 Message-ID: In-Reply-To: <4CEB8DB4.7050502@cisco.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt Thread-Index: AcuK9IIHtqnnlZ+TSfiodpfVqY26wgAAQcyw References: <4CEA8AED.9050307@cisco.com> <4CEB8DB4.7050502@cisco.com> From: "Romascanu, Dan (Dan)" To: "Benoit Claise" Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Nov 2010 10:03:10 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01CB8AF5.C10F01B5 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Yes, this looks OK.=20 =20 Just as an editing comment, I would rather:=20 =20 s/The IETF does have current and completed activities related to the reporting of application performance metrics: for example/Previous IETF work related to reporting of application performance metrics includes/ =20 Thanks and Regards, =20 Dan (speaking as contributor) ________________________________ From: Benoit Claise [mailto:bclaise@cisco.com]=20 Sent: Tuesday, November 23, 2010 11:48 AM To: Romascanu, Dan (Dan) Cc: Alan Clark; pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt =09 =09 Dan,=20 =20 =09 -----Original Message----- From: Benoit Claise [mailto:bclaise@cisco.com]=20 Sent: Monday, November 22, 2010 5:23 PM To: Alan Clark Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org Subject: Re: [PMOL] I-D=20 Action:draft-ietf-pmol-metrics-framework-05.txt =09 Alan, Apologies for my long absence!! =09 A few comments =09 1. Introduction - last sentence =09 Standardized metrics help to ensure that performance measurement is=20 implemented consistently and facilitate interpretation and=20 comparison Done. 1.1 Background - paragraph 5 =09 Suggest the addition of RFC3611 and (soon to be) RFC6035 as=20 these are=20 widely used IETF protocols for VoIP performance reporting. =20 The RAQMON=20 description reads a little like an advert - so maybe =09 "for example, the RTP Control Protocol Extended Reports=20 (RTCP XR) [RFC3611]. the Real -time Application Quality-of-Service Monitoring (RAQMON)=20 Framework [RFC4710] and the SIP RTCP Summary Report=20 Protocol [RC6035]=20 are protocols that support the real-time reporting of Voice over IP=20 and other applications running on devices such as IP phones=20 and mobile handsets." I propose to keep the RAQMON section, as it deals with=20 application performance metrics, but to add a new paragraph =09 The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611]=20 and the SIP RTCP Summary Report Protocol [RC6035] are=20 protocols that support the real-time reporting of Voice over=20 IP and other applications running on devices such as IP=20 phones and mobile handsets. =09 RAQMON also supports real-time reporting.=20 What we have right now is in the to-be-posted version: =09 The IETF does have current and completed activities related to the reporting of application performance metrics: for example the Real-time Application Quality-of-Service Monitoring (RAQMON) Framework [RFC4710],=20 which extends the remote network monitoring (RMON) family of specifications to allow real-time quality-of-service (QoS) monitoring of various applications that run on devices such as IP phones, pagers, Instant Messaging clients, mobile phones, and various other handheld computing devices. Furthermore, the RTP Control Protocol Extended=20 Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol=20 [RC6035] are protocols that support the=20 real-time reporting of Voice over IP and other applications running on=20 devices such as IP phones and mobile handsets. =09 Please let me know if this is fine with you you. =09 Regards, Benoit. =09 =09 Dan (speaking as contributor) _______________________________________________ PMOL mailing list PMOL@ietf.org https://www.ietf.org/mailman/listinfo/pmol ------_=_NextPart_001_01CB8AF5.C10F01B5 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Yes,=20 this looks OK.
 
Just=20 as an editing comment, I would rather:
 
s/The IETF does=20 have current and completed activities related to the reporting of = application=20 performance metrics: for example/Previous IETF work related to reporting = of=20 application performance metrics includes/
 
Thanks=20 and Regards,
 
Dan
(speaking as contributor)

From: Benoit Claise = [mailto:bclaise@cisco.com]=20
Sent: Tuesday, November 23, 2010 11:48 AM
To: = Romascanu,=20 Dan (Dan)
Cc: Alan Clark; pmol@ietf.org
Subject: = Re:=20 [PMOL] I-D=20 Action:draft-ietf-pmol-metrics-framework-05.txt

Dan,=20 
=20


-----Original Message-----
From: Benoit Claise [mailto:bclaise@cisco.com]=20
Sent: Monday, November 22, 2010 5:23 PM
To: Alan Clark
Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org
Subject: Re: [PMOL] I-D=20
Action:draft-ietf-pmol-metrics-framework-05.txt

Alan,

Apologies for my long =
absence!!

A few comments

1. Introduction - last sentence

Standardized metrics help to ensure that performance measurement is=20
implemented consistently and facilitate interpretation and=20

comparison
Done.

1.1 Background - =
paragraph 5

Suggest the addition of RFC3611 and (soon to be) RFC6035 as=20

these are=20

widely used IETF =
protocols for VoIP performance reporting. =20

The RAQMON=20

description reads a =
little like an advert - so maybe

"for example, the RTP Control Protocol Extended Reports=20

(RTCP XR) [RFC3611].

the Real -time =
Application Quality-of-Service Monitoring (RAQMON)=20
Framework [RFC4710] and the SIP RTCP Summary Report=20

Protocol [RC6035]=20

are protocols that =
support the real-time reporting of Voice over IP=20
and other applications running on devices such as IP phones=20

and mobile handsets."
I propose to keep the RAQMON section, as it deals with=20
application performance metrics, but to add a new paragraph

The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611]=20
and the SIP RTCP Summary Report Protocol [RC6035] are=20
protocols that support the real-time reporting of Voice over=20
IP and other applications running on devices such as IP=20
phones and mobile handsets.


RAQMON also supports real-time =
reporting.=20
What we have right now is in the to-be-posted=20 version:

        =    =20     The IETF does have current and completed activities = related=20 to the
           =20     reporting of application performance metrics: for = example=20 the Real-time
        =    =20     Application Quality-of-Service Monitoring (RAQMON)=20 Framework [RFC4710],=20 =
           &nb= sp;   =20 which extends the remote network monitoring
   =20             (RMON) family = of=20 specifications to allow real-time quality-of-service=20 (QoS)
           =20     monitoring of various applications that run on = devices such=20 as IP phones,
        =    =20     pagers, Instant Messaging clients, mobile phones, = and=20 various other
        =    =20     handheld computing devices. Furthermore, the RTP = Control=20 Protocol Extended=20 =
           &nb= sp;   =20 Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol=20 =
           &nb= sp;=20    [RC6035] are protocols that support the=20 =
           &nb= sp;    =20 real-time reporting of Voice over IP and other applications running on = =
           &nb= sp;   =20 devices such as IP phones and mobile handsets.

Please let me = know if=20 this is fine with you you.

Regards, Benoit.

Dan
(speaking as contributor)
_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/=
mailman/listinfo/pmol

------_=_NextPart_001_01CB8AF5.C10F01B5-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 39C9E3A6A9D; Mon, 22 Nov 2010 08:15:34 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -10.599 X-Spam-Level: X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hdCp53XXWnGU; Mon, 22 Nov 2010 08:15:33 -0800 (PST) Received: from ams-iport-1.cisco.com (ams-iport-1.cisco.com [144.254.224.140]) by core3.amsl.com (Postfix) with ESMTP id A2AF63A6A9F; Mon, 22 Nov 2010 08:15:32 -0800 (PST) Authentication-Results: ams-iport-1.cisco.com; dkim=neutral (message not signed) header.i=none X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: At4DAA8m6kyQ/khNgWdsb2JhbACDSp4xYhUBARYiIqIBijqQa4EigzZzBI1y X-IronPort-AV: E=Sophos;i="4.59,236,1288569600"; d="scan'208";a="69836449" Received: from ams-core-4.cisco.com ([144.254.72.77]) by ams-iport-1.cisco.com with ESMTP; 22 Nov 2010 16:16:06 +0000 Received: from xbh-ams-201.cisco.com (xbh-ams-201.cisco.com [144.254.75.7]) by ams-core-4.cisco.com (8.14.3/8.14.3) with ESMTP id oAMGG6C4011801; Mon, 22 Nov 2010 16:16:06 GMT Received: from xmb-ams-107.cisco.com ([144.254.74.82]) by xbh-ams-201.cisco.com with Microsoft SMTPSVC(6.0.3790.4675); Mon, 22 Nov 2010 17:16:06 +0100 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Date: Mon, 22 Nov 2010 17:16:05 +0100 Message-ID: <6674CF9A9F682245A590204F819F78AD03378F97@XMB-AMS-107.cisco.com> In-Reply-To: <4CE252AC.8080509@cisco.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [bmwg] Repeated WGLC on draft-ietf-pmol-metrics-framework-05 Thread-Index: AcuFcwaoIQCpsngETWehFp5u5uilBgE7Vx0w References: <6674CF9A9F682245A590204F819F78AD02ED90BE@XMB-AMS-107.cisco.com> <4CE252AC.8080509@cisco.com> From: "Jan Novak (janovak)" To: "Benoit Claise (bclaise)" X-OriginalArrivalTime: 22 Nov 2010 16:16:06.0739 (UTC) FILETIME=[912DEA30:01CB8A60] Cc: pmol@ietf.org, bmwg@ietf.org Subject: Re: [PMOL] [bmwg] Repeated WGLC on draft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2010 16:15:34 -0000 SGkgYWdhaW4sDQoNCuKeoiBMZXQgbWUga25vdyBpZiB0aGlzIHdvcmtzIGZvciB5b3UNCg0KT2ss IHRoYW5rcyBmb3IgY2xhcmlmaWNhdGlvbnMgLSBpdHMgZmluZS4NCg0KSmFuDQoNCg0KDQpUaGUg Y2xpbWF0ZSBvZiBFZGluYnVyZ2ggaXMgc3VjaCB0aGF0IHRoZSB3ZWFrIHN1Y2N1bWIgeW91bmcg Li4uLiANCmFuZCB0aGUgc3Ryb25nIGVudnkgdGhlbS4NCiAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIERyLiBKb2huc29uDQoNCg0KRnJvbTogQmVub2l0IENsYWlzZSAoYmNsYWlzZSkg DQpTZW50OiAxNiBOb3ZlbWJlciAyMDEwIDA5OjQ1DQpUbzogSmFuIE5vdmFrIChqYW5vdmFrKQ0K Q2M6IGJtd2dAaWV0Zi5vcmc7IHBtb2xAaWV0Zi5vcmc7IEFsIE1vcnRvbg0KU3ViamVjdDogUmU6 IFtibXdnXSBSZXBlYXRlZCBXR0xDIG9uIGRyYWZ0LWlldGYtcG1vbC1tZXRyaWNzLWZyYW1ld29y ay0wNQ0KDQpIaSBKYW4sDQoNCkhpLA0KDQpBIGJpdCBsYXRlIGJ1dCBqdXN0IGZldyBjb21tZW50 cyBvbiB0aGUgZHJhZnQ6DQoNCkp1c3QgYSBudW1wdHkgaW4gdGhlIHRleHQ6DQoNClRoZSBrZXkg YXJlYXMgb2YgdmFyaWF0aW9uIGJldHdlZW4gZGlmZmVyZW50IG1ldHJpYyB1c2VycyBpbmNsdWRl Og0KICAgICBvICBTdWl0YWJpbGl0eSBvZiBwYXNzaXZlIG1lYXN1cmVtZW50cyBvZiBsaXZlIHRy YWZmaWMsIG9yIGFjdGl2ZQ0KICAgICAgICBtZWFzdXJlbWVudHMgdXNpbmcgZGVkaWNhdGVkIHRy YWZmaWMNCg0KICAgICBvICBNZWFzdXJlbWVudCBpbiBsYWJvcmF0b3J5IGVudmlyb25tZW50LCBv ciBvbiBhIG5ldHdvcmsgb2YNCmRlcGxveWVkDQogICAgICAgIGRldmljZXMNCg0KICAgICBvICBT dWl0YWJpbGl0eSBvZiBwYXNzaXZlIG1lYXN1cmVtZW50cyBvZiBsaXZlIHRyYWZmaWMsIG9yIGFj dGl2ZQ0KICAgICAgICBtZWFzdXJlbWVudHMgdXNpbmcgZGVkaWNhdGVkIHRyYWZmaWMNCg0KICAg ICBvICBNZWFzdXJlbWVudCBpbiBsYWJvcmF0b3J5IGVudmlyb25tZW50LCBvciBvbiBhIG5ldHdv cmsgb2YNCmRlcGxveWVkDQogICAgICAgIGRldmljZXMNCg0KIEluIHRoZSBjb250ZXh0IG9mIHRo aXMgZHJhZnQgLSBzZWN0aW9uIDUuMSBhbmQgNS40LjIgbW9zdCBvZiB0aGUgdGVybXMgDQogdXNl ZCBhcmUgc2VsZiBleHBsYW5hdGl2ZSwgYnV0IHdvdWxkbid0IGJlIHNvIHN1cmUgYWJvdXQgdGhl IG1lYW5pbmcgb2YNCiJNZWFzdXJlbWVudCBQb2ludCIgYW5kICJ0aGUgbWVhc3VyZW1lbnQNCiAg ICAgZG9tYWluIiAtIHNvdW5kcyBmYW1pbGlhciBmcm9tIElQRklYIGJ1dCBub3QgaGVyZSAuLi4N CkkgY2xhcmlmaWVkIHRoZXNlIGFzOg0KT0xEOg0KICAgSWYgdGhlIG1lYXN1cmVtZW50IGlzIHNw ZWNpZmljIHRvIGEgbWVhc3VyZW1lbnQgcG9pbnQgdGhpcyBTSE9VTEQgYmUNCiAgIGRlZmluZWQu ICBUaGUgbWVhc3VyZW1lbnQgZG9tYWluIE1BWSBhbHNvIGJlIGRlZmluZWQuICBTcGVjaWZpY2Fs bHksDQogICBpZiBtZWFzdXJlbWVudCBwb2ludHMgYXJlIHNwcmVhZCBhY3Jvc3MgZG9tYWlucywg dGhlIG1lYXN1cmVtZW50DQogICBkb21haW4gKGludHJhLSwgaW50ZXItKSBpcyBhbm90aGVyIGZh Y3RvciB0byBjb25zaWRlci4NCg0KTkVXOg0KICAgSWYgdGhlIG1lYXN1cmVtZW50IGlzIHNwZWNp ZmljIHRvIGEgbWVhc3VyZW1lbnQgcG9pbnQgdGhpcyBTSE9VTEQgYmUNCiAgIGRlZmluZWQuICBU aGUgbWVhc3VyZW1lbnQgZG9tYWluIE1BWSBhbHNvIGJlIGRlZmluZWQuICBUaGUgbWVhc3VyZW1l bnQgDQogICBkb21haW4gaXMgZGVmaW5lZCBhcyBhIHNldCBvZiBtZWFzdXJlbWVudCBwb2ludCB0 aGF0IG9wZXJhdGUgdW5kZXIgdGhlIA0KICAgdGhlIHNhbWUgYWRtaW5pc3RyYXRpb24uIFNwZWNp ZmljYWxseSwNCiAgIGlmIG1lYXN1cmVtZW50IHBvaW50cyBhcmUgc3ByZWFkIGFjcm9zcyBkb21h aW5zLCB0aGUgbWVhc3VyZW1lbnQNCiAgIGRvbWFpbiAoaW50cmEtLCBpbnRlci0pIGlzIGFub3Ro ZXIgZmFjdG9yIHRvIGNvbnNpZGVyLg0KTGV0IG1lIGtub3cgaWYgdGhpcyB3b3JrcyBmb3IgeW91 Lg0KDQoNCiBTZWN0aW9uIDUuNC40IC0gZG9lc24ndCB0aGUgdGVtcGxhdGUgbWlzcyB0aGUgbm9y bWF0aXZlIHBvaW50ICh2KQ0KTWVhc3VyZW1lbnQgUG9pbnQocykNCiAgICAgICAgICAgICAgICAg IGZyb20gNS40LjIgPz8NCldlbGwgc3BvdC4gSSBtYWRlIHN1cmUgdGhhdCBhbGwgZW50cmllcyB3 ZXJlIGFsaWduZWQuDQpPTEQ6DQo1LjQuNC4gIFBlcmZvcm1hbmNlIE1ldHJpYyBEZWZpbml0aW9u IFRlbXBsYXRlDQoNCg0KICAgTm9ybWF0aXZlDQoNCiAgIG8gIE1ldHJpYyBOYW1lDQoNCiAgIG8g IE1ldHJpYyBEZXNjcmlwdGlvbg0KDQogICBvICBNZXRob2QNCg0KICAgbyAgVW5pdHMgb2YgbWVh c3VyZW1lbnQNCg0KICAgbyAgTWVhc3VyZW1lbnQgVGltaW5nDQoNCiAgIEluZm9ybWF0aXZlDQoN CiAgIG8gIEltcGxlbWVudGF0aW9uIEd1aWRlbGluZXMNCg0KICAgbyAgVmVyaWZpY2F0aW9uDQoN CiAgIG8gIFVzZSBhbmQgQXBwbGljYXRpb25zDQoNCiAgIG8gIFJlcG9ydGluZyBNb2RlbA0KDQpO RVc6DQogICBOb3JtYXRpdmUNCg0KICAgbyAgTWV0cmljIE5hbWUNCg0KICAgbyAgTWV0cmljIERl c2NyaXB0aW9uDQoNCiAgIG8gIE1ldGhvZCBvZiBNZWFzdXJlbWVudCBvciBDYWxjdWxhdGlvbg0K DQogICBvICBVbml0cyBvZiBtZWFzdXJlbWVudA0KDQogICBvICBNZWFzdXJlbWVudCBQb2ludChz KSB3aXRoIHBvdGVudGlhbCBNZWFzdXJlbWVudCBEb21haW4NCg0KICAgbyAgTWVhc3VyZW1lbnQg VGltaW5nDQoNCiAgIEluZm9ybWF0aXZlDQoNCiAgIG8gIEltcGxlbWVudGF0aW9uDQoNCiAgIG8g IFZlcmlmaWNhdGlvbg0KDQogICBvICBVc2UgYW5kIEFwcGxpY2F0aW9ucw0KDQogICBvICBSZXBv cnRpbmcgTW9kZWwNCg0KUmVnYXJkcywgQmVub2l0Lg0KDQoNCiBKYW4NCg0KDQpUaGUgY2xpbWF0 ZSBvZiBFZGluYnVyZ2ggaXMgc3VjaCB0aGF0IHRoZSB3ZWFrIHN1Y2N1bWIgeW91bmcgLi4uLiAN CmFuZCB0aGUgc3Ryb25nIGVudnkgdGhlbS4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIERyLiBKb2huc29uDQoNCkFuZCB5ZXQgdGhlIHBsYWNlIGVzdGFibGlzaGVzIGFuIGludGVy ZXN0IGluIHBlb3BsZSdzIGhlYXJ0czsNCmdvIHdoZXJlIHRoZXkgd2lsbCwgdGhleSBmaW5kIG5v IGNpdHkgb2YgdGhlIHNhbWUgZGlzdGluY3Rpb247DQpnbyB3aGVyZSB0aGV5IHdpbGwsIHRoZXkg dGFrZSBhIHByaWRlIGluIHRoZWlyIG9sZCBob21lLg0KDQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICBSTFMNCg0KDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fXw0KYm13ZyBtYWlsaW5nIGxpc3QNCmJtd2dAaWV0Zi5vcmcNCmh0dHBzOi8vd3d3 LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vYm13Zw0KDQo= Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B7B203A6AA9 for ; Mon, 22 Nov 2010 08:12:32 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.531 X-Spam-Level: X-Spam-Status: No, score=-102.531 tagged_above=-999 required=5 tests=[AWL=0.068, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NZqt7jDvXDWX for ; Mon, 22 Nov 2010 08:12:31 -0800 (PST) Received: from de307622-de-outbound.net.avaya.com (de307622-de-outbound.net.avaya.com [198.152.71.100]) by core3.amsl.com (Postfix) with ESMTP id 10E8D3A6AA4 for ; Mon, 22 Nov 2010 08:12:30 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhIDAB4l6kyHCzI1/2dsb2JhbACUFo5HcaQVApkQhUsEjgY X-IronPort-AV: E=Sophos;i="4.59,236,1288584000"; d="scan'208";a="219719526" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by de307622-de-outbound.net.avaya.com with ESMTP; 22 Nov 2010 11:13:25 -0500 X-IronPort-AV: E=Sophos;i="4.59,236,1288584000"; d="scan'208";a="543321231" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.12]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 22 Nov 2010 11:13:23 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Mon, 22 Nov 2010 17:13:04 +0100 Message-ID: In-Reply-To: <4CEA8AED.9050307@cisco.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt Thread-Index: AcuKWUt6NgR2W2GVScGBupKgGYTeMQABrSZA References: <4CEA8AED.9050307@cisco.com> From: "Romascanu, Dan (Dan)" To: "Benoit Claise" , "Alan Clark" Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2010 16:12:32 -0000 =20 > -----Original Message----- > From: Benoit Claise [mailto:bclaise@cisco.com]=20 > Sent: Monday, November 22, 2010 5:23 PM > To: Alan Clark > Cc: Romascanu, Dan (Dan); Al Morton; pmol@ietf.org > Subject: Re: [PMOL] I-D=20 > Action:draft-ietf-pmol-metrics-framework-05.txt >=20 > Alan, > > Apologies for my long absence!! > > > > A few comments > > > > 1. Introduction - last sentence > > > > Standardized metrics help to ensure that performance measurement is=20 > > implemented consistently and facilitate interpretation and=20 > comparison > Done. > > 1.1 Background - paragraph 5 > > > > Suggest the addition of RFC3611 and (soon to be) RFC6035 as=20 > these are=20 > > widely used IETF protocols for VoIP performance reporting. =20 > The RAQMON=20 > > description reads a little like an advert - so maybe > > > > "for example, the RTP Control Protocol Extended Reports=20 > (RTCP XR) [RFC3611]. > > the Real -time Application Quality-of-Service Monitoring (RAQMON)=20 > > Framework [RFC4710] and the SIP RTCP Summary Report=20 > Protocol [RC6035]=20 > > are protocols that support the real-time reporting of Voice over IP=20 > > and other applications running on devices such as IP phones=20 > and mobile handsets." > I propose to keep the RAQMON section, as it deals with=20 > application performance metrics, but to add a new paragraph >=20 > The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611]=20 > and the SIP RTCP Summary Report Protocol [RC6035] are=20 > protocols that support the real-time reporting of Voice over=20 > IP and other applications running on devices such as IP=20 > phones and mobile handsets. >=20 RAQMON also supports real-time reporting.=20 Dan (speaking as contributor) Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 7857428C10E for ; Mon, 22 Nov 2010 08:09:06 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.593 X-Spam-Level: X-Spam-Status: No, score=-2.593 tagged_above=-999 required=5 tests=[AWL=0.005, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J62xgK4dPB7n for ; Mon, 22 Nov 2010 08:09:02 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id CB66E28C113 for ; Mon, 22 Nov 2010 08:08:58 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMG67kZ012808; Mon, 22 Nov 2010 17:06:07 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMG621p013752; Mon, 22 Nov 2010 17:06:03 +0100 (CET) Message-ID: <4CEA94EA.9030803@cisco.com> Date: Mon, 22 Nov 2010 17:06:02 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Alan Clark References: In-Reply-To: Content-Type: multipart/alternative; boundary="------------050008040609090805000603" Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2010 16:09:06 -0000 This is a multi-part message in MIME format. --------------050008040609090805000603 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Alan, Actually, I would prefer to keep the E.800 definition as it is. As Al suggested... > ...I don't intend to start a debate of the QoE > definition here, and if debate is wanted, it should take place > in the ITU-T Study Group responsible for this definition. > > Al Regards, Benoit. > I'd agree with the spirit of the QoE definition but the use of quantitative > vs qualitative could be stated a little differently. > > QoE is a measure of the quality of an application as perceived by a person. > Externally observable behaviors of the application interact with any > psychoacoustic, psychovisual, memory and other effects and with the > conditioning and expectations of the person for the application, leading to > an expression of quality that is (or would be) stated by the person. > Observable behaviors of the application may be objectively measured or > inferred based on measurements of impairments that affect application > performance in a known way. QoE may be directly measured using subjective > testing or may be estimated through the use of appropriate perceptual > models. > > So:- > (i) if there is an (objectively measurable or quantitative) impairment, this > may be masked or enhanced by some psychoacoustic or psychovisual effect. For > example a macroblock error may be very visible in a video scene with smooth > panning but not noticeable in a video scene with high motion. > (ii) QoE is affected by the expectations of the user/ person. SD video may > have an "excellent" QoE if the user's expectations are based on SD video and > there are no impairments other than image resolution. SD video may have a > "fair" QoE if the user's expectations are based on a mix of SD and HD video. > We find the same with narrowband vs wideband speech. Historically > subjective tests were done with all narrowband or all wideband and hence the > results were relative to speech samples of the same type. This leads to the > sometimes confusing result that a wideband speech sample may have lower MOS > than a narrowband sample, even though it sounds better. There has been work > done in ITU using subjective tests with mixed narrowband and wideband > speech. > > > Alan > > > On 11/17/10 11:53 AM, "Al Morton" wrote: > >> At 04:15 AM 11/16/2010, Benoit Claise wrote: >>>> Aamer wrote: >>>> section 2.3 >>>> >>>> "The quantitative component can be influenced by the >>>> complete end-to-end system effects (network infrastructure).". >>>> >>>> Not clear what this means. >>>> >>>> What is the difference between application and network >>>> infrastructure? Where do network transcoders sit? >>>> >>>> Is HD vs. SD video a qualitative or quantitative component? >>> If I recall correctly, Al wrote this one. >> I didn't write it, though I think I supplied the reference and text >> from an existing Recommendation, E.800. >> >>> Al, can you shed some light. >> Part of the problem is that this definition wording does not >> stand on its own beyond the context of the Recommendation >> it comes from. Maybe we can clarify: >> >> NOTE 2 - QoE has two main components: quantitative and >> qualitative. The quantitative component can be influenced by the >> complete end-to-end system effects (including user devices and >> network infrastructure). >> >> where the "complete end-to-end system" is all communications equipment. >> I'm avoiding the application/network split. As you pointed out, >> transcoders would be part of network infrastructure. >> HD vs. SD is quantitative in this view, a configuration option >> of the e2e system. >> >> This (QoE) has been a difficult term to define, it appears >> to subsume part of the existing definition of QoS in some >> people's minds, but it adds the opportunity to include context >> more directly. I don't intend to start a debate of the QoE >> definition here, and if debate is wanted, it should take place >> in the ITU-T Study Group responsible for this definition. >> >> Al >> >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol > --------------050008040609090805000603 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Alan,

Actually, I would prefer to keep the E.800 definition as it is.
As Al suggested...
> ...I don't intend to start a debate of the QoE
> definition here, and if debate is wanted, it should take place
> in the ITU-T Study Group responsible for this definition.
>
> Al
Regards, Benoit.
I'd agree with the spirit of the QoE definition but the use of quantitative
vs qualitative could be stated a little differently.

QoE is a measure of the quality of an application as perceived by a person.
Externally observable behaviors of the application interact with any
psychoacoustic, psychovisual, memory and other effects and with the
conditioning and expectations of the person for the application, leading to
an expression of quality that is (or would be) stated by the person.
Observable behaviors of the application may be objectively measured or
inferred based on measurements of impairments that affect application
performance in a known way. QoE may be directly measured using subjective
testing or may be estimated through the use of appropriate perceptual
models. 

So:-
(i) if there is an (objectively measurable or quantitative) impairment, this
may be masked or enhanced by some psychoacoustic or psychovisual effect. For
example a macroblock error may be very visible in a video scene with smooth
panning but not noticeable in a video scene with high motion.
(ii) QoE is affected by the expectations of the user/ person.  SD video may
have an "excellent" QoE if the user's expectations are based on SD video and
there are no impairments other than image resolution. SD video may have a
"fair" QoE if the user's expectations are based on a mix of SD and HD video.
We find the same with narrowband vs wideband speech.  Historically
subjective tests were done with all narrowband or all wideband and hence the
results were relative to speech samples of the same type.  This leads to the
sometimes confusing result that a wideband speech sample may have lower MOS
than a narrowband sample, even though it sounds better.  There has been work
done in ITU using subjective tests with mixed narrowband and wideband
speech.


Alan


On 11/17/10 11:53 AM, "Al Morton" <acmorton@att.com> wrote:


At 04:15 AM 11/16/2010, Benoit Claise wrote:

Aamer wrote:
section 2.3

"The quantitative component can be influenced by the
   complete end-to-end system effects (network infrastructure).".

Not clear what this means.

What is the difference between application and network
infrastructure? Where do network transcoders sit?

Is HD vs. SD video a qualitative or quantitative component?

If I recall correctly, Al wrote this one.

I didn't write it, though I think I supplied the reference and text
from an existing Recommendation, E.800.


Al, can you shed some light.

Part of the problem is that this definition wording does not
stand on its own beyond the context of the Recommendation
it comes from. Maybe we can clarify:

    NOTE 2 - QoE has two main components: quantitative and
    qualitative.  The quantitative component can be influenced by the
    complete end-to-end system effects (including user devices and
    network infrastructure).

where the "complete end-to-end system" is all communications equipment.
I'm avoiding the application/network split. As you pointed out,
transcoders would be part of network infrastructure.
HD vs. SD is quantitative in this view, a configuration option
of the e2e system.

This (QoE) has been a difficult term to define, it appears
to subsume part of the existing definition of QoS in some
people's minds, but it adds the opportunity to include context
more directly.  I don't intend to start a debate of the QoE
definition here, and if debate is wanted, it should take place
in the ITU-T Study Group responsible for this definition.

Al

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------050008040609090805000603-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A2FE33A69D2 for ; Mon, 22 Nov 2010 07:22:38 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.594 X-Spam-Level: X-Spam-Status: No, score=-2.594 tagged_above=-999 required=5 tests=[AWL=0.005, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lL00XAo+ZcPe for ; Mon, 22 Nov 2010 07:22:37 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 3A5523A6A97 for ; Mon, 22 Nov 2010 07:22:37 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMFNQHZ006837; Mon, 22 Nov 2010 16:23:26 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMFNP0q014599; Mon, 22 Nov 2010 16:23:26 +0100 (CET) Message-ID: <4CEA8AED.9050307@cisco.com> Date: Mon, 22 Nov 2010 16:23:25 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Alan Clark References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2010 15:22:38 -0000 Alan, > Apologies for my long absence!! > > A few comments > > 1. Introduction - last sentence > > Standardized metrics help to ensure that performance measurement is > implemented consistently and facilitate interpretation and comparison Done. > 1.1 Background - paragraph 5 > > Suggest the addition of RFC3611 and (soon to be) RFC6035 as these are widely > used IETF protocols for VoIP performance reporting. The RAQMON description > reads a little like an advert - so maybe > > "for example, the RTP Control Protocol Extended Reports (RTCP XR) [RFC3611]. > the Real -time Application Quality-of-Service Monitoring (RAQMON) Framework > [RFC4710] and the SIP RTCP Summary Report Protocol [RC6035] are protocols > that support the real-time reporting of Voice over IP and other applications > running on devices such as IP phones and mobile handsets." I propose to keep the RAQMON section, as it deals with application performance metrics, but to add a new paragraph The RTP Control Protocol Extended Reports (RTCP XR) [RFC3611] and the SIP RTCP Summary Report Protocol [RC6035] are protocols that support the real-time reporting of Voice over IP and other applications running on devices such as IP phones and mobile handsets. > 2.2 Quality of Service > > Remove "The" and correct language - so "Quality of Service (QoS) is defined > in a similar way to the...." done. > 2.3 Quality of Experience > > Remove "The" - so "Quality of Experience (QoE) is defined in a similar way > to the...." > Done. > References > > Add RFC3611 and RFC6035 Done, as informative reference. However, when do you expect the AUTH48 to be finished for RFC6035? Regards, Benoit. > > > Alan > > > > > On 10/10/10 4:45 PM, "org" wrote: > >> A New Internet-Draft is available from the on-line Internet-Drafts >> directories. >> This draft is a work item of the Performance Metrics for Other Layers Working >> Group of the IETF. >> >> >> Title : Guidelines for Considering New Performance Metric >> Development >> Author(s) : A. Clark, B. Claise >> Filename : draft-ietf-pmol-metrics-framework-05.txt >> Pages : 21 >> Date : 2010-10-10 >> >> This document describes a framework and a process for developing >> performance metrics of protocols and applications transported over >> over IETF-specified protocols, and that can be used to characterize >> traffic on live networks and services. >> >> A URL for this Internet-Draft is: >> http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-05.txt >> >> Internet-Drafts are also available by anonymous FTP at: >> ftp://ftp.ietf.org/internet-drafts/ >> >> Below is the data which will enable a MIME compliant mail reader >> implementation to automatically retrieve the ASCII version of the >> Internet-Draft. >> _______________________________________________ >> PMOL mailing list >> PMOL@ietf.org >> https://www.ietf.org/mailman/listinfo/pmol > Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9B7A128C10D for ; Mon, 22 Nov 2010 07:07:16 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.593 X-Spam-Level: X-Spam-Status: No, score=-2.593 tagged_above=-999 required=5 tests=[AWL=0.006, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7n2ize44lF-6 for ; Mon, 22 Nov 2010 07:07:15 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 058A628C104 for ; Mon, 22 Nov 2010 07:07:14 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMF4Mv7004158; Mon, 22 Nov 2010 16:04:23 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAMF4M5P020368; Mon, 22 Nov 2010 16:04:22 +0100 (CET) Message-ID: <4CEA8676.6080802@cisco.com> Date: Mon, 22 Nov 2010 16:04:22 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Al Morton References: <4CB226D7.2060503@cisco.com> <7F298ACC76CC154F832B6D02852D169F02F41471@XMB-RCD-101.cisco.com> <4CE24BB7.5050202@cisco.com> <201011171653.oAHGrW3L017451@alpd052.aldc.att.com> In-Reply-To: <201011171653.oAHGrW3L017451@alpd052.aldc.att.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Nov 2010 15:07:17 -0000 Al, > At 04:15 AM 11/16/2010, Benoit Claise wrote: >>> Aamer wrote: >>> section 2.3 >>> >>> "The quantitative component can be influenced by the >>> complete end-to-end system effects (network infrastructure).". >>> >>> Not clear what this means. >>> >>> What is the difference between application and network >>> infrastructure? Where do network transcoders sit? >>> >>> Is HD vs. SD video a qualitative or quantitative component? >> If I recall correctly, Al wrote this one. > > I didn't write it, though I think I supplied the reference and text > from an existing Recommendation, E.800. > >> Al, can you shed some light. > > Part of the problem is that this definition wording does not > stand on its own beyond the context of the Recommendation > it comes from. Maybe we can clarify: > > NOTE 2 - QoE has two main components: quantitative and > qualitative. The quantitative component can be influenced by the > complete end-to-end system effects (including user devices and > network infrastructure). I replaced the current NOTE 2 with the NOTE 2 above. > > where the "complete end-to-end system" is all communications equipment. > I'm avoiding the application/network split. As you pointed out, > transcoders would be part of network infrastructure. > HD vs. SD is quantitative in this view, a configuration option > of the e2e system. > > This (QoE) has been a difficult term to define, it appears > to subsume part of the existing definition of QoS in some > people's minds, but it adds the opportunity to include context > more directly. I don't intend to start a debate of the QoE > definition here, and if debate is wanted, it should take place > in the ITU-T Study Group responsible for this definition. Fully agree. Regards, Benoit. > > Al > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 7D9D93A69A3 for ; Wed, 17 Nov 2010 11:58:31 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lEBJydzF2qHv for ; Wed, 17 Nov 2010 11:58:30 -0800 (PST) Received: from mail01.netplexity.net (mail01.netplexity.net [66.129.85.135]) by core3.amsl.com (Postfix) with ESMTP id 615313A695F for ; Wed, 17 Nov 2010 11:58:30 -0800 (PST) Received: from UnknownHost [97.67.102.65] by mail01.netplexity.net with SMTP; Wed, 17 Nov 2010 14:59:11 -0500 User-Agent: Microsoft-Entourage/12.14.0.081024 Date: Wed, 17 Nov 2010 14:59:01 -0500 From: Alan Clark To: Al Morton , Benoit Claise , "Aamer Akhter (aakhter)" Message-ID: Thread-Topic: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 Thread-Index: AcuGkeDLt3hKg3Q0JUyHS0NLeUnbkg== In-Reply-To: <201011171653.oAHGrW3L017451@alpd052.aldc.att.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 19:58:31 -0000 I'd agree with the spirit of the QoE definition but the use of quantitative vs qualitative could be stated a little differently. QoE is a measure of the quality of an application as perceived by a person. Externally observable behaviors of the application interact with any psychoacoustic, psychovisual, memory and other effects and with the conditioning and expectations of the person for the application, leading to an expression of quality that is (or would be) stated by the person. Observable behaviors of the application may be objectively measured or inferred based on measurements of impairments that affect application performance in a known way. QoE may be directly measured using subjective testing or may be estimated through the use of appropriate perceptual models. So:- (i) if there is an (objectively measurable or quantitative) impairment, this may be masked or enhanced by some psychoacoustic or psychovisual effect. For example a macroblock error may be very visible in a video scene with smooth panning but not noticeable in a video scene with high motion. (ii) QoE is affected by the expectations of the user/ person. SD video may have an "excellent" QoE if the user's expectations are based on SD video and there are no impairments other than image resolution. SD video may have a "fair" QoE if the user's expectations are based on a mix of SD and HD video. We find the same with narrowband vs wideband speech. Historically subjective tests were done with all narrowband or all wideband and hence the results were relative to speech samples of the same type. This leads to the sometimes confusing result that a wideband speech sample may have lower MOS than a narrowband sample, even though it sounds better. There has been work done in ITU using subjective tests with mixed narrowband and wideband speech. Alan On 11/17/10 11:53 AM, "Al Morton" wrote: > At 04:15 AM 11/16/2010, Benoit Claise wrote: >>> Aamer wrote: >>> section 2.3 >>> >>> "The quantitative component can be influenced by the >>> complete end-to-end system effects (network infrastructure).". >>> >>> Not clear what this means. >>> >>> What is the difference between application and network >>> infrastructure? Where do network transcoders sit? >>> >>> Is HD vs. SD video a qualitative or quantitative component? >> If I recall correctly, Al wrote this one. > > I didn't write it, though I think I supplied the reference and text > from an existing Recommendation, E.800. > >> Al, can you shed some light. > > Part of the problem is that this definition wording does not > stand on its own beyond the context of the Recommendation > it comes from. Maybe we can clarify: > > NOTE 2 - QoE has two main components: quantitative and > qualitative. The quantitative component can be influenced by the > complete end-to-end system effects (including user devices and > network infrastructure). > > where the "complete end-to-end system" is all communications equipment. > I'm avoiding the application/network split. As you pointed out, > transcoders would be part of network infrastructure. > HD vs. SD is quantitative in this view, a configuration option > of the e2e system. > > This (QoE) has been a difficult term to define, it appears > to subsume part of the existing definition of QoS in some > people's minds, but it adds the opportunity to include context > more directly. I don't intend to start a debate of the QoE > definition here, and if debate is wanted, it should take place > in the ITU-T Study Group responsible for this definition. > > Al > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C630F3A693C for ; Wed, 17 Nov 2010 08:52:56 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.591 X-Spam-Level: X-Spam-Status: No, score=-105.591 tagged_above=-999 required=5 tests=[AWL=0.205, BAYES_00=-2.599, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fvt9fxm30I-g for ; Wed, 17 Nov 2010 08:52:56 -0800 (PST) Received: from mail120.messagelabs.com (mail120.messagelabs.com [216.82.250.83]) by core3.amsl.com (Postfix) with ESMTP id EF35E3A6936 for ; Wed, 17 Nov 2010 08:52:55 -0800 (PST) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-8.tower-120.messagelabs.com!1290012820!53633941!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 1416 invoked from network); 17 Nov 2010 16:53:41 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-8.tower-120.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 17 Nov 2010 16:53:41 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oAHGrx1T022069 for ; Wed, 17 Nov 2010 11:53:59 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oAHGrsti021990 for ; Wed, 17 Nov 2010 11:53:54 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oAHGrZR6017579 for ; Wed, 17 Nov 2010 11:53:35 -0500 Received: from mailgw1.maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oAHGrW3L017451 for ; Wed, 17 Nov 2010 11:53:32 -0500 Message-Id: <201011171653.oAHGrW3L017451@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-144-172.vpn.mwst.att.com[135.70.144.172](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20101117165331gw1004lkf6e>; Wed, 17 Nov 2010 16:53:31 +0000 X-Originating-IP: [135.70.144.172] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 17 Nov 2010 11:53:57 -0500 To: Benoit Claise , "Aamer Akhter (aakhter)" From: Al Morton In-Reply-To: <4CE24BB7.5050202@cisco.com> References: <4CB226D7.2060503@cisco.com> <7F298ACC76CC154F832B6D02852D169F02F41471@XMB-RCD-101.cisco.com> <4CE24BB7.5050202@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 16:52:56 -0000 At 04:15 AM 11/16/2010, Benoit Claise wrote: >>Aamer wrote: >>section 2.3 >> >>"The quantitative component can be influenced by the >> complete end-to-end system effects (network infrastructure).". >> >>Not clear what this means. >> >>What is the difference between application and network >>infrastructure? Where do network transcoders sit? >> >>Is HD vs. SD video a qualitative or quantitative component? >If I recall correctly, Al wrote this one. I didn't write it, though I think I supplied the reference and text from an existing Recommendation, E.800. >Al, can you shed some light. Part of the problem is that this definition wording does not stand on its own beyond the context of the Recommendation it comes from. Maybe we can clarify: NOTE 2 - QoE has two main components: quantitative and qualitative. The quantitative component can be influenced by the complete end-to-end system effects (including user devices and network infrastructure). where the "complete end-to-end system" is all communications equipment. I'm avoiding the application/network split. As you pointed out, transcoders would be part of network infrastructure. HD vs. SD is quantitative in this view, a configuration option of the e2e system. This (QoE) has been a difficult term to define, it appears to subsume part of the existing definition of QoS in some people's minds, but it adds the opportunity to include context more directly. I don't intend to start a debate of the QoE definition here, and if debate is wanted, it should take place in the ITU-T Study Group responsible for this definition. Al Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C25993A6808 for ; Tue, 16 Nov 2010 20:07:19 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.599 X-Spam-Level: X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3C8ukp70MqSh for ; Tue, 16 Nov 2010 20:07:18 -0800 (PST) Received: from mail01.netplexity.net (mail01.netplexity.net [66.129.85.135]) by core3.amsl.com (Postfix) with ESMTP id 890513A67DF for ; Tue, 16 Nov 2010 20:07:18 -0800 (PST) Received: from c-24-98-71-91.hsd1.ga.comcast.net [24.98.71.91] by mail01.netplexity.net with SMTP; Tue, 16 Nov 2010 23:07:34 -0500 User-Agent: Microsoft-Entourage/12.14.0.081024 Date: Tue, 16 Nov 2010 23:07:50 -0500 From: Alan Clark To: Benoit Claise , "Dan (Dan)" , Al Morton Message-ID: Thread-Topic: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt Thread-Index: AcuGDP/T8jcOzbISQkOROB/8RkVPQw== In-Reply-To: <20101010204501.C32B03A6858@core3.amsl.com> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Cc: pmol@ietf.org Subject: Re: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-05.txt X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2010 04:07:19 -0000 Apologies for my long absence!! A few comments 1. Introduction - last sentence Standardized metrics help to ensure that performance measurement is implemented consistently and facilitate interpretation and comparison 1.1 Background - paragraph 5 Suggest the addition of RFC3611 and (soon to be) RFC6035 as these are widely used IETF protocols for VoIP performance reporting. The RAQMON description reads a little like an advert - so maybe "for example, the RTP Control Protocol Extended Reports (RTCP XR) [RFC3611]. the Real -time Application Quality-of-Service Monitoring (RAQMON) Framework [RFC4710] and the SIP RTCP Summary Report Protocol [RC6035] are protocols that support the real-time reporting of Voice over IP and other applications running on devices such as IP phones and mobile handsets." 2.2 Quality of Service Remove "The" and correct language - so "Quality of Service (QoS) is defined in a similar way to the...." 2.3 Quality of Experience Remove "The" - so "Quality of Experience (QoE) is defined in a similar way to the...." References Add RFC3611 and RFC6035 Alan On 10/10/10 4:45 PM, "org" wrote: > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > This draft is a work item of the Performance Metrics for Other Layers Working > Group of the IETF. > > > Title : Guidelines for Considering New Performance Metric > Development > Author(s) : A. Clark, B. Claise > Filename : draft-ietf-pmol-metrics-framework-05.txt > Pages : 21 > Date : 2010-10-10 > > This document describes a framework and a process for developing > performance metrics of protocols and applications transported over > over IETF-specified protocols, and that can be used to characterize > traffic on live networks and services. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-05.txt > > Internet-Drafts are also available by anonymous FTP at: > ftp://ftp.ietf.org/internet-drafts/ > > Below is the data which will enable a MIME compliant mail reader > implementation to automatically retrieve the ASCII version of the > Internet-Draft. > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 8F32B3A6DB6 for ; Tue, 16 Nov 2010 02:08:23 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.571 X-Spam-Level: X-Spam-Status: No, score=-2.571 tagged_above=-999 required=5 tests=[AWL=0.027, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9YcsmCgN8eKs for ; Tue, 16 Nov 2010 02:08:18 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id B68403A6C56 for ; Tue, 16 Nov 2010 02:08:17 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9Wf7k015244; Tue, 16 Nov 2010 10:32:41 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9WebN024150; Tue, 16 Nov 2010 10:32:40 +0100 (CET) Message-ID: <4CE24FB8.3060301@cisco.com> Date: Tue, 16 Nov 2010 10:32:40 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Schmoll, Carsten" References: <70524A4436C03E43A293958B505008B60308A02A@EXCHSRV.fokus.fraunhofer.de> In-Reply-To: <70524A4436C03E43A293958B505008B60308A02A@EXCHSRV.fokus.fraunhofer.de> Content-Type: multipart/alternative; boundary="------------010801070205010800030406" Cc: pmol@ietf.org Subject: Re: [PMOL] Feedback on draft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Nov 2010 10:08:23 -0000 This is a multi-part message in MIME format. --------------010801070205010800030406 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Carsten, Thanks for your review. > Dear authors of draft-ietf-pmol-metrics-framework-05, > > there is only very few comments I can make on the draft -- it is in very good > shape (content- and English-wise)! > > > here are my few (hopefully helpful) observations: > > (quoted text is taken from the draft, then afterwards I note a possible > change or addition in each of the following paragraphs) > > > ######################################################################### > > "The key areas of variation between different metric users include: > > * Required reporting formats" > > > Maybe write here instead: > > " * Required reporting formats and periods" Done. > ######################################################################### > > "Some metrics may not be measured directly, but may be composed from > base metrics that have been measured." > > I am not sure about the double use of "may" here. Maybe "can be composed" > would be better?!? Done. > > ######################################################################### > > "Some parameters linked to the method MAY also be reported, in order > to fully interpret the Performance Metric. For example, the time > interval, the load, the minimum packet loss, etc..." > > I think it would be useful to note also that it makes sense to report the > possible measurement errors and or the accuracy of results. > e.g. "potential measurement errors and their sources should be noted. The > attainable accuracy of the metric (e.g. +-0,1) should be reported as well." OLD: Some parameters linked to the method MAY also be reported, in order to fully interpret the Performance Metric. For example, the time interval, the load, the minimum packet loss, etc... NEW: Some parameters linked to the method MAY also be reported, in order to fully interpret the Performance Metric. For example, the time interval, the load, the minimum packet loss, the potential measurement errors and their sources, the attainable accuracy of the metric (e.g. +/-0,1) etc.. > ######################################################################### > > "The objective of this part of the metric definition is to assist > implementers to achieve a consistent result." > > Change to plural ( "consistent results" ) ? Done. > ######################################################################### > > "(iii) Use and Applications > > The use and applications description is intended to assist the "user" > to understand how, when and where the metric can be applied, and what > significance the value range for the metric may have. This MAY > include a definition of the "typical" and "abnormal" range of the > Performance Metric, if this was not apparent from the nature of the > metric. > " > > Maybe something could be added like > "The description may (should??) also include information about the influence > of extreme measurement values, i.e. if the metric is very sensitive to > outliers." OLD: The use and applications description is intended to assist the "user" to understand how, when and where the metric can be applied, and what significance the value range for the metric may have. This MAY include a definition of the "typical" and "abnormal" range of the Performance Metric, if this was not apparent from the nature of the metric. NEW: The use and applications description is intended to assist the "user" to understand how, when and where the metric can be applied, and what significance the value range for the metric may have. This MAY include a definition of the "typical" and "abnormal" range of the Performance Metric, if this was not apparent from the nature of the metric. _The description MAY include information about the influence of extreme measurement values, i.e. if the Performance Metric is sensitive to outliers._ > ######################################################################### > > Space between "Metric" and the dot: > > "of the Performance Metric ." -> "of the Performance Metric." Done. > > ######################################################################### > > "Existing mailing lists SHOULD be used however a dedicated mailing list MAY > be initiated if necessary to facilitate work on a draft." > > I recommend to add a comma after "used". Done. Thanks again for your feedback. Regards, Benoit > ######################################################################### > > Best regards > Carsten > > > > > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol --------------010801070205010800030406 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Carsten,

Thanks for your review.
Dear authors of draft-ietf-pmol-metrics-framework-05,
 
there is only very few comments I can make on the draft – it is in very good
shape (content- and English-wise)!
 
 
here are my few (hopefully helpful) observations:
 
(quoted text is taken from the draft, then afterwards I note a possible
change or addition in each of the following paragraphs)


#########################################################################

“The key areas of variation between different metric users include:
 
* Required reporting formats”
 
 
Maybe write here instead:
 
“ * Required reporting formats and periods”
Done.
######################################################################### 
 
"Some metrics may not be measured directly, but may be composed from
   base metrics that have been measured."
 
I am not sure about the double use of “may” here. Maybe “can be composed”
would be better?!?
Done.
 
#########################################################################
 
"Some parameters linked to the method MAY also be reported, in order
   to fully interpret the Performance Metric.  For example, the time
   interval, the load, the minimum packet loss, etc..."
 
I think it would be useful to note also that it makes sense to report the
possible measurement errors and or the accuracy of results.
e.g. “potential measurement errors and their sources should be noted. The
attainable accuracy of the metric (e.g. +-0,1) should be reported as well.”
OLD:
   Some parameters linked to the method MAY also be reported, in order
   to fully interpret the Performance Metric.  For example, the time
   interval, the load, the minimum packet loss, etc...


NEW:
          Some parameters linked to the method MAY also be reported, in
          order to fully interpret the Performance Metric. For example, the
          time interval, the load, the minimum packet loss, the potential measurement
          errors and their sources, the attainable accuracy of the metric (e.g. +/-0,1)
          etc..
         
######################################################################### 
 
"The objective of this part of the metric definition is to assist
implementers to achieve a consistent result."
 
Change to plural ( “consistent results” ) ?
Done.
#########################################################################
   
"(iii) Use and Applications
 
   The use and applications description is intended to assist the "user"
   to understand how, when and where the metric can be applied, and what
   significance the value range for the metric may have.  This MAY
   include a definition of the "typical" and "abnormal" range of the
   Performance Metric, if this was not apparent from the nature of the
   metric.
"

Maybe something could be added like
“The description may (should??) also include information about the influence
of extreme measurement values, i.e. if the metric is very sensitive to
outliers.”
OLD:
             The use and applications description is intended to assist the "user"
              to understand how, when and where the metric can be applied, and what
              significance the value range for the metric may have. This MAY
              include a definition of the "typical" and "abnormal" range of the
              Performance Metric, if this was not apparent from the nature of the metric.

NEW:
            The use and applications description is intended to assist the "user"
              to understand how, when and where the metric can be applied, and what
              significance the value range for the metric may have. This MAY
              include a definition of the "typical" and "abnormal" range of the
              Performance Metric, if this was not apparent from the nature of the metric.
              The description MAY include information about the influence of extreme
              measurement values, i.e. if the Performance Metric is sensitive to
              outliers.
######################################################################### 
 
Space between “Metric” and the dot:
 
"of the Performance Metric ." -> "of the Performance Metric."
Done.
 
######################################################################### 
 
“Existing mailing lists SHOULD be used however a dedicated mailing list MAY
be initiated if necessary to facilitate work on a draft.”
 
I recommend to add a comma after “used”.
Done.

Thanks again for your feedback.

Regards, Benoit
#########################################################################
 
Best regards
Carsten
 
               
 


_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------010801070205010800030406-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5A5353A6C9C; Tue, 16 Nov 2010 01:55:53 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.57 X-Spam-Level: X-Spam-Status: No, score=-2.57 tagged_above=-999 required=5 tests=[AWL=0.028, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rSFQ6NFqz-PY; Tue, 16 Nov 2010 01:55:52 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id DF1663A6C56; Tue, 16 Nov 2010 01:55:51 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9jHui017068; Tue, 16 Nov 2010 10:45:17 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9jGDR008139; Tue, 16 Nov 2010 10:45:16 +0100 (CET) Message-ID: <4CE252AC.8080509@cisco.com> Date: Tue, 16 Nov 2010 10:45:16 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Jan Novak (janovak)" References: <6674CF9A9F682245A590204F819F78AD02ED90BE@XMB-AMS-107.cisco.com> In-Reply-To: <6674CF9A9F682245A590204F819F78AD02ED90BE@XMB-AMS-107.cisco.com> Content-Type: multipart/alternative; boundary="------------040905000509090100070804" Cc: pmol@ietf.org, bmwg@ietf.org Subject: Re: [PMOL] [bmwg] Repeated WGLC on draft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Nov 2010 09:55:53 -0000 This is a multi-part message in MIME format. --------------040905000509090100070804 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Jan, > Hi, > > A bit late but just few comments on the draft: > > Just a numpty in the text: > > The key areas of variation between different metric users include: > o Suitability of passive measurements of live traffic, or active > measurements using dedicated traffic > > o Measurement in laboratory environment, or on a network of > deployed > devices > > o Suitability of passive measurements of live traffic, or active > measurements using dedicated traffic > > o Measurement in laboratory environment, or on a network of > deployed > devices > > In the context of this draft - section 5.1 and 5.4.2 most of the terms > used are self explanative, but wouldn't be so sure about the meaning of > "Measurement Point" and "the measurement > domain" - sounds familiar from IPFIX but not here ... I clarified these as: OLD: If the measurement is specific to a measurement point this SHOULD be defined. The measurement domain MAY also be defined. Specifically, if measurement points are spread across domains, the measurement domain (intra-, inter-) is another factor to consider. NEW: If the measurement is specific to a measurement point this SHOULD be defined. The measurement domain MAY also be defined. The measurement domain is defined as a set of measurement point that operate under the the same administration. Specifically, if measurement points are spread across domains, the measurement domain (intra-, inter-) is another factor to consider. Let me know if this works for you. > Section 5.4.4 - doesn't the template miss the normative point (v) > Measurement Point(s) > from 5.4.2 ?? Well spot. I made sure that all entries were aligned. OLD: 5.4.4. Performance Metric Definition Template Normative o Metric Name o Metric Description o Method o Units of measurement o Measurement Timing Informative o Implementation Guidelines o Verification o Use and Applications o Reporting Model NEW: Normative o Metric Name o Metric Description o Method of Measurement or Calculation o Units of measurement o Measurement Point(s) with potential Measurement Domain o Measurement Timing Informative o Implementation o Verification o Use and Applications o Reporting Model Regards, Benoit. > Jan > > > The climate of Edinburgh is such that the weak succumb young .... > and the strong envy them. > Dr. Johnson > > And yet the place establishes an interest in people's hearts; > go where they will, they find no city of the same distinction; > go where they will, they take a pride in their old home. > > RLS > > > _______________________________________________ > bmwg mailing list > bmwg@ietf.org > https://www.ietf.org/mailman/listinfo/bmwg --------------040905000509090100070804 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi Jan,
Hi,

A bit late but just few comments on the draft:

Just a numpty in the text:

The key areas of variation between different metric users include:
     o  Suitability of passive measurements of live traffic, or active
        measurements using dedicated traffic

     o  Measurement in laboratory environment, or on a network of
deployed
        devices

     o  Suitability of passive measurements of live traffic, or active
        measurements using dedicated traffic

     o  Measurement in laboratory environment, or on a network of
deployed
        devices

 In the context of this draft - section 5.1 and 5.4.2 most of the terms 
 used are self explanative, but wouldn't be so sure about the meaning of
"Measurement Point" and "the measurement
     domain" - sounds familiar from IPFIX but not here ...
I clarified these as:
OLD:
   If the measurement is specific to a measurement point this SHOULD be
   defined.  The measurement domain MAY also be defined.  Specifically,
   if measurement points are spread across domains, the measurement
   domain (intra-, inter-) is another factor to consider.

NEW:
   If the measurement is specific to a measurement point this SHOULD be
   defined.  The measurement domain MAY also be defined.  The measurement 
   domain is defined as a set of measurement point that operate under the 
   the same administration. Specifically,
   if measurement points are spread across domains, the measurement
   domain (intra-, inter-) is another factor to consider.
Let me know if this works for you.
 Section 5.4.4 - doesn't the template miss the normative point (v)
Measurement Point(s)
                  from 5.4.2 ??
Well spot. I made sure that all entries were aligned.
OLD:
5.4.4.  Performance Metric Definition Template


   Normative

   o  Metric Name

   o  Metric Description

   o  Method

   o  Units of measurement

   o  Measurement Timing

   Informative

   o  Implementation Guidelines

   o  Verification

   o  Use and Applications

   o  Reporting Model

NEW:

   Normative

   o  Metric Name

   o  Metric Description

   o  Method of Measurement or Calculation

   o  Units of measurement

   o  Measurement Point(s) with potential Measurement Domain

   o  Measurement Timing

   Informative

   o  Implementation

   o  Verification

   o  Use and Applications

   o  Reporting Model

Regards, Benoit.
 Jan


The climate of Edinburgh is such that the weak succumb young .... 
and the strong envy them.
                                 Dr. Johnson

And yet the place establishes an interest in people's hearts;
go where they will, they find no city of the same distinction;
go where they will, they take a pride in their old home.

                                 RLS


_______________________________________________
bmwg mailing list
bmwg@ietf.org
https://www.ietf.org/mailman/listinfo/bmwg

--------------040905000509090100070804-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id CE1E23A6DB3 for ; Tue, 16 Nov 2010 01:30:40 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.569 X-Spam-Level: X-Spam-Status: No, score=-2.569 tagged_above=-999 required=5 tests=[AWL=0.029, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nh2Jc29-HBiN for ; Tue, 16 Nov 2010 01:30:38 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 573393A6DA2 for ; Tue, 16 Nov 2010 01:30:37 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9Facm012617 for ; Tue, 16 Nov 2010 10:15:36 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG9FZRA003430; Tue, 16 Nov 2010 10:15:35 +0100 (CET) Message-ID: <4CE24BB7.5050202@cisco.com> Date: Tue, 16 Nov 2010 10:15:35 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: "Aamer Akhter (aakhter)" References: <4CB226D7.2060503@cisco.com> <7F298ACC76CC154F832B6D02852D169F02F41471@XMB-RCD-101.cisco.com> In-Reply-To: <7F298ACC76CC154F832B6D02852D169F02F41471@XMB-RCD-101.cisco.com> Content-Type: multipart/alternative; boundary="------------080508090600050708090702" Cc: pmol@ietf.org Subject: Re: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Nov 2010 09:30:41 -0000 This is a multi-part message in MIME format. --------------080508090600050708090702 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Thanks Aamer for your review > > Hi Benoit, > > Please find my comments below. Overall I found the guidelines very > much on the right path – and (unsurprisingly) do mirror the internal > work discussions we have done in our definitions process. > > section 1.1 > > The motivation does not seem to cleanly resolve whether the generation > of upper layer performance metrics needs to be separated from the > monitoring and transport frameworks (RMON, IPFIX). Unless this is > done, there may be overlapping efforts across the WGs. This is > somewhat talked about in section 6, but can we have clear guidance > between separation between metrics definition, transport, and > verification/validation. The verification/validation being something > perhaps we can extend in BMWGs charter. > In section 1.1, after this paragraph The IETF does have current and completed activities related to the reporting of application performance metrics: for example the Real- time Application Quality-of-Service Monitoring (RAQMON) FrameworkRFC 4710 [RFC4710 ], which extends the remote network monitoring (RMON) family of specifications to allow real-time quality-of-service (QoS) monitoring of various applications that run on devices such as IP phones, pagers, Instant Messaging clients, mobile phones, and various other handheld computing devices. I would like to add: Another example is the IPFIX protocol [RFC5101], which exports flow records composed of IPFIX information elements, initially specified in the IPFIX Information Model [RFC5102]. New IPFIX information elements can be requested to IPFIX IANA [IPFIX-IANA]. It is anticipated that some new information elements will contain performance metrics. In this document, the generation of upper layer performance metrics need to be separated from the monitoring and transport frameworks (such MIB variable with SNMP or information element with IPFIX). Does it solve your issue? Note: I'm not too sure how to translate this into a specification in section 5 or section 6. > > section 2.3 > > "The quantitative component can be influenced by the > > complete end-to-end system effects (network infrastructure).". > > Not clear what this means. > > What is the difference between application and network infrastructure? > Where do network transcoders sit? > > Is HD vs. SD video a qualitative or quantitative component? > If I recall correctly, Al wrote this one. Al, can you shed some light. > > section 2.4 > > "specific to an IETF-specified > > protocol or specific to an application transported over an IETF- > > specified protocol." then "database logging time" > > does this restrict the evaluation of database logging time to IETF > protocols (TCP, HTTP), and not any deeper? > Deeper is also allowed. > > section 3. > > Are active and passive measurements widely understood terms? Maybe > just a reference. > I was unable to find a good reference. Let me propose OLD: The framework in this document applies to performance metrics derived from both active and passive measurements. NEW: The framework in this document applies to performance metrics derived from both active and passive measurements. Active measurements, also referred to as synthetic traffic, injects specific traffic in the network to order to measure its performance (example: one-way active measurement [RFC 4656] or two-way active measurements [RFC 5357], while passive measurements analyses the live traffic to measure its performance > section 5.1 > > "The key areas of variation between different metric > users..." > > Not sure if this covered by the exiting cases but > please add a statement regarding the performance context that is being > considered. For example, the network operator may find it convenient > to look at RTP flows as an indicator of network health rather than a > specific interest in the health of the RTP flow. In that regard, some > of the QoE aspects specific to the application may not matter at all. > Can you please propose some text. Alternatively, a 5 minutes discussion would clarify. > > is time range part of the existing list (real-time, delayed reporting, > summarized reporting, etc) > Yes. Proposal: add * time range (real-time, delayed reporting, summarized reporting, etc) > > section 5.4.2 > > " Performance Metric names MUST be unique within the set of metrics > > being defined and MAY be descriptive." > > Unclear why a metric name would _not_ be descriptive. Perhaps an > example is needed? > I don't recall why we made that distinction. Unless I hear otherwise, I will change it to " Performance Metric names MUST be unique within the set of metrics being defined and MUST be descriptive." > > section 5.4.2 > > "If the measurement is specific to a measurement point > this SHOULD be > > defined. The measurement domain MAY also be defined. Specifically, > > if measurement points are spread across domains, the measurement > > domain (intra-, inter-) is another factor to consider. > > In some cases, the measurement requires multiple measurement points: > > all measurement points SHOULD be defined, including the measurement > > domain(s)." > > It would be helpful to have references to more discussion on what > defines a measurement domain. Is a measurement domain an > administrative domain? Is the network portion of a flow leg a > measurement domain? What if a NAT is involved? Are there two domains then? > OLD: If the measurement is specific to a measurement point this SHOULD be defined. The measurement domain MAY also be defined. Specifically, if measurement points are spread across domains, the measurement domain (intra-, inter-) is another factor to consider. NEW: If the measurement is specific to a measurement point this SHOULD be defined. The measurement domain MAY also be defined. The measurement domain is defined as a set of measurement point that operate under the the same administration. Specifically, if measurement points are spread across domains, the measurement domain (intra-, inter-) is another factor to consider. > section 5.4.2 > > Slightly related to measurement timing is the issue of granularity of > measurement needed to compute the metric. For example it is possible > to sample RTP flows to determine packet loss. However it is not > possible to sample packets from a single RTP flow to determine packet > loss. > Propose to add your sentence as a third paragraph to section 5.4.2 "measurement timing" Related to measurement timing is the issue of granularity of measurement needed to compute the metric. For example it is possible to sample RTP flows to determine packet loss. However it is not possible to sample packets from a single RTP flow to determine packet loss. > section 6.2 > > "repeatable" > > I think for application performance this might mean a > complete description of several workloads to be tested against. How > will this be done? ie are we talking about a pcap file-- if so what > would that pcap file need to look like? How would it be distributed? > Propose to add a paragraph at the end of section 6.2 Note that for the "repeatable" qualification, the performance metric might have to be qualified with some extra information related t othe environment during the measurement. For example, the workload, the existing flow, etc... in the measurement domain Please let me know what you think Thanks again for your review. Regards, Benoit. > > *From:*pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] *On Behalf > Of *Benoit Claise (bclaise) > *Sent:* Sunday, October 10, 2010 4:49 PM > *To:* pmol@ietf.org > *Subject:* [PMOL] Fwd: New Version Notification > fordraft-ietf-pmol-metrics-framework-05 > > Dear all, > > Here is a new version of the framework draft. > What is new? > - improve the Performance Metric definition > - change the title to RFC5706 > - PM Entity defined as a directorate > > No more open issues in the draft. This should be ready for WGLC. > > Regards, Benoit. > > -------- Original Message -------- > > *Subject: * > > > > New Version Notification for draft-ietf-pmol-metrics-framework-05 > > *Date: * > > > > Sun, 10 Oct 2010 13:38:40 -0700 (PDT) > > *From: * > > > > IETF I-D Submission Tool > > > *To: * > > > > bclaise@cisco.com > > *CC: * > > > > alan.d.clark@telchemy.com > > A new version of I-D, draft-ietf-pmol-metrics-framework-05.txt has been successfully submitted by Benoit Claise and posted to the IETF repository. > > Filename: draft-ietf-pmol-metrics-framework > Revision: 05 > Title: Guidelines for Considering New Performance Metric Development > Creation_date: 2010-10-10 > WG ID: pmol > Number_of_pages: 21 > > Abstract: > This document describes a framework and a process for developing > performance metrics of protocols and applications transported over > over IETF-specified protocols, and that can be used to characterize > traffic on live networks and services. > > > > The IETF Secretariat. > --------------080508090600050708090702 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit Thanks Aamer for your review

Hi Benoit,

 

Please find my comments below. Overall I found the guidelines very much on the right path – and (unsurprisingly) do mirror the internal work discussions we have done in our definitions process.

 

 

section 1.1

 

The motivation does not seem to cleanly resolve whether the generation of upper layer performance metrics needs to be separated from the monitoring and transport frameworks (RMON, IPFIX). Unless this is done, there may be overlapping efforts across the WGs. This is somewhat talked about in section 6, but can we have clear guidance between separation between metrics definition, transport, and verification/validation. The verification/validation being something perhaps we can extend in BMWGs charter.


In section 1.1, after this paragraph

   The IETF does have current and completed activities related to the
   reporting of application performance metrics: for example the Real-
   time Application Quality-of-Service Monitoring (RAQMON) Framework RFC
   4710 [RFC4710], which extends the remote network monitoring (RMON)
   family of specifications to allow real-time quality-of-service (QoS)
   monitoring of various applications that run on devices such as IP
   phones, pagers, Instant Messaging clients, mobile phones, and various
   other handheld computing devices.

I would like to add:
Another example is the IPFIX protocol [RFC5101], which exports flow records
composed of IPFIX information elements, initially specified in the IPFIX Information Model [RFC5102].
New IPFIX information elements can be requested to IPFIX IANA [IPFIX-IANA]. It is anticipated that
some new information elements will contain performance metrics.

In this document, the generation of upper layer performance metrics need
to be separated from the monitoring and transport frameworks (such MIB variable
with SNMP or information element with IPFIX).

Does it solve your issue?

Note: I'm not too sure how to translate this into a specification in section 5 or section 6.

 

section 2.3

 

"The quantitative component can be influenced by the

   complete end-to-end system effects (network infrastructure).".

 

Not clear what this means.

 

What is the difference between application and network infrastructure? Where do network transcoders sit?

Is HD vs. SD video a qualitative or quantitative component?

If I recall correctly, Al wrote this one.
Al, can you shed some light.

 

section 2.4

 

"specific to an IETF-specified

   protocol or specific to an application transported over an IETF-

   specified protocol." then "database logging time"

 

does this restrict the evaluation of database logging time to IETF protocols (TCP, HTTP), and not any deeper?

Deeper is also allowed.

 

section 3.

 

Are active and passive measurements widely understood terms? Maybe just a reference.

I was unable to find a good reference.
Let me propose

OLD:
   The framework in this document applies to performance metrics derived
   from both active and passive measurements.
NEW:
   The framework in this document applies to performance metrics derived
   from both active and passive measurements.
   Active measurements, also referred to as synthetic traffic, injects specific traffic 
   in the network to order to measure its performance (example: one-way active measurement
   [RFC 4656] or two-way active measurements [RFC 5357], while passive measurements
   analyses the live traffic to measure its performance

 

section 5.1

 

                "The key areas of variation between different metric users..."

 

                Not sure if this covered by the exiting cases but please add a statement regarding the performance context that is being considered. For example, the network operator may find it convenient to look at RTP flows as an indicator of network health rather than a specific interest in the health of the RTP flow. In that regard, some of the QoE aspects specific to the application may not matter at all.

Can you please propose some text.
Alternatively, a 5 minutes discussion would clarify.

 

is time range part of the existing list (real-time, delayed reporting, summarized reporting, etc)

Yes.
Proposal: add
   *   time range (real-time, delayed reporting, summarized reporting, etc)

 

section 5.4.2

"   Performance Metric names MUST be unique within the set of metrics

   being defined and MAY be descriptive."

 

   Unclear why a metric name would _not_ be descriptive. Perhaps an example is needed?

I don't recall why we made that distinction.
Unless I hear otherwise, I will change it to

"   Performance Metric names MUST be unique within the set of metrics

   being defined and MUST be descriptive."

  

section 5.4.2

 

                "If the measurement is specific to a measurement point this SHOULD be

   defined.  The measurement domain MAY also be defined.  Specifically,

   if measurement points are spread across domains, the measurement

   domain (intra-, inter-) is another factor to consider.

   In some cases, the measurement requires multiple measurement points:

   all measurement points SHOULD be defined, including the measurement

   domain(s)."

  

   It would be helpful to have references to more discussion on what defines a measurement domain. Is a measurement domain an administrative domain? Is the network portion of a flow leg a measurement domain? What if a NAT is involved? Are there two domains then?

OLD:
   If the measurement is specific to a measurement point this SHOULD be
   defined.  The measurement domain MAY also be defined.  Specifically,
   if measurement points are spread across domains, the measurement
   domain (intra-, inter-) is another factor to consider.

NEW:
   If the measurement is specific to a measurement point this SHOULD be
   defined.  The measurement domain MAY also be defined.  The measurement 
   domain is defined as a set of measurement point that operate under the 
   the same administration. Specifically,
   if measurement points are spread across domains, the measurement
   domain (intra-, inter-) is another factor to consider.

  

section 5.4.2

 

Slightly related to measurement timing is the issue of granularity of measurement needed to compute the metric. For example it is possible to sample RTP flows to determine packet loss. However it is not possible to sample packets from a single RTP flow to determine packet loss.

Propose to add your sentence as a third paragraph to section 5.4.2 "measurement timing"

Related to measurement timing is the issue of granularity of measurement needed to compute the metric. For example it is possible to sample RTP flows to determine packet loss. However it is not possible to sample packets from a single RTP flow to determine packet loss.


 

section 6.2

 

                "repeatable"

               

                I think for application performance this might mean a complete description of several workloads to be tested against. How will this be done? ie are we talking about a pcap file-- if so what would that pcap file need to look like? How would it be distributed?

               

Propose to add a paragraph at the end of section 6.2
Note that for the "repeatable" qualification, the performance metric might have to be qualified with some extra information related t othe environment during the measurement. For example, the workload, the existing flow, etc... in the measurement domain

Please let me know what you think
Thanks again for your review.

Regards, Benoit.

         

               

 

 

 

From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Benoit Claise (bclaise)
Sent: Sunday, October 10, 2010 4:49 PM
To: pmol@ietf.org
Subject: [PMOL] Fwd: New Version Notification fordraft-ietf-pmol-metrics-framework-05

 

Dear all,

Here is a new version of the framework draft.
What is new?
    - improve the Performance Metric definition
    - change the title to RFC5706
    - PM Entity defined as a directorate

No more open issues in the draft. This should be ready for WGLC.

Regards, Benoit.

-------- Original Message --------

Subject:

New Version Notification for draft-ietf-pmol-metrics-framework-05

Date:

Sun, 10 Oct 2010 13:38:40 -0700 (PDT)

From:

IETF I-D Submission Tool <idsubmission@ietf.org>

To:

bclaise@cisco.com

CC:

alan.d.clark@telchemy.com

 

A new version of I-D, draft-ietf-pmol-metrics-framework-05.txt has been successfully submitted by Benoit Claise and posted to the IETF repository.
 
Filename:      draft-ietf-pmol-metrics-framework
Revision:      05
Title:         Guidelines for Considering New Performance Metric Development
Creation_date:  2010-10-10
WG ID:         pmol
Number_of_pages: 21
 
Abstract:
This document describes a framework and a process for developing
performance metrics of protocols and applications transported over
over IETF-specified protocols, and that can be used to characterize
traffic on live networks and services.
                                                                                  
 
 
The IETF Secretariat.
 

--------------080508090600050708090702-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id DACD73A6D81; Tue, 16 Nov 2010 00:06:07 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.569 X-Spam-Level: X-Spam-Status: No, score=-2.569 tagged_above=-999 required=5 tests=[AWL=0.030, BAYES_00=-2.599] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4t4kjn+J3tH9; Tue, 16 Nov 2010 00:06:06 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 56C063A6DB0; Tue, 16 Nov 2010 00:06:06 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG86k6C002807; Tue, 16 Nov 2010 09:06:46 +0100 (CET) Received: from [10.55.43.54] (ams-bclaise-8715.cisco.com [10.55.43.54]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAG86fcl019311; Tue, 16 Nov 2010 09:06:42 +0100 (CET) Message-ID: <4CE23B91.1080607@cisco.com> Date: Tue, 16 Nov 2010 09:06:41 +0100 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Yaakov Stein References: <201010111228.o9BCS55n016546@alpd052.aldc.att.com> <48E7911F78327A449A9FB95637667286A8219E35@exrad4.ad.rad.co.il> In-Reply-To: <48E7911F78327A449A9FB95637667286A8219E35@exrad4.ad.rad.co.il> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: IETF IPPM WG , "pmol@ietf.org" Subject: Re: [PMOL] Repeated WGLC on draft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Nov 2010 08:06:08 -0000 Hi Yaakov, Now that the WGLC is finished and that the IETF week is over, let me cover all the WGLC comments. > I think there are important ideas here. > > Three minor comments: > > 1) The acronym PM is used by the OAM community for "Performance Management" > (in contrast to FM which stands for "Fault Management"). > Re-use for "Performance Metrics" in a related field > (especially considering the weak intercommunications between the communities) > could lead to confusion. Make sense to me. I removed the PM acronym > 2) I understand that the entity/directorate will be reviewing documents > that define performance metrics. Who determines which documents need to be reviewed ? > For example, what about an ID from some WG in transport or security that in passing > defines a measured parameter ? (Some directorates look at every document that makes it to the IESG) The way I see this is a strong link between the OPS directorate and this new Performance Metrics Entity/directorate). Yes, the OPS directorate reviews all documents. Then they could direct the ones that have performance metrics to this new directorate. In practice I don't see any issues. For example, both Al and I are part of OPS directorate. Do you believe some text should be updated in the draft? Where? > 3) I am not a strong supporter of meta-documents - > I much prefer RFCs with meaningful technical content to those that dictate internal processes. > I see this document as a "Tao" for the PMOL and IPPM communities. Yes. Hence the name "Guidelines for Considering New Performance Metric Development" > I remember that 2418 mentions the concept of directorate, > and found a list of directorates here - http://www.ietf.org/iesg/directorate/ > but do not recall RFCs for the security, transport, ... directorates, > except for the quoted 5706. > > Since I was not involved in the discussions in PMOL, I would like to know if alternative mechanisms > for making this information available (e.g., a Performance Metric wiki) > have been discussed. The alternative that was discussed is that the PMOL WG would contain all the performance metrics related drafts. Some issues: we could see an explosion of those drafts, while we might be lacking the technology expertise. A WIKI, I don't believe too much in that unless there is a procedure in place Regards, Benoit. > Y(J)S > > > -----Original Message----- > From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Al Morton > Sent: Monday, October 11, 2010 14:28 > To: pmol@ietf.org > Cc: bmwg@ietf.org; ippm@ietf.org > Subject: [PMOL] Repeated WGLC on draft-ietf-pmol-metrics-framework-05 > > PMOL WG, > cc IPPM and BMWG WGs, > > This message begins a PMOL WG Last Call on the following draft: > > Title: Guidelines for Considering New Performance Metric Development > Author(s) : A. Clark, B. Claise > Filename : draft-ietf-pmol-metrics-framework-05.txt > Pages : 21 > Date : 2010-10-10 > > This memo describes a framework and guidelines for the development of > performance metrics that are beyond the scope of existing working > group charters in the IETF. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-05.txt > > There has been discussion of this draft for over 3 years > on the PMOL list and at meetings, both formal and informal. > The previous WGLC closed on January 2, 2009. > The Authors believe all comments to date have been addressed. > > Please weigh-in on whether or not this Draft should be forwarded > to the Area Directors for publication as a Standards Track RFC. > Send your comments to the PMOL list or the co-chairs. > > The Last Call will be open till November 8, 2010. > > thanks for your review and comments, > Al Morton > co-chair, PMOL WG > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 56AA63A69D9 for ; Thu, 11 Nov 2010 01:57:49 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.598 X-Spam-Level: X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SmCirdOmi0Bk for ; Thu, 11 Nov 2010 01:57:48 -0800 (PST) Received: from mail-qw0-f44.google.com (mail-qw0-f44.google.com [209.85.216.44]) by core3.amsl.com (Postfix) with ESMTP id 658943A6A38 for ; Thu, 11 Nov 2010 01:57:48 -0800 (PST) Received: by qwb7 with SMTP id 7so1733536qwb.31 for ; Thu, 11 Nov 2010 01:58:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=XLB6/k7a3Yu8KM7LzIJ68hPHj/6P9kuSnZaksu3lzLQ=; b=ZWgKAZT7AvVUKegDh0rNpQY9v9yKn/PxEa6KlERgiEmZaCHgiebiRMNvl5go/6m5/u 4JhZUKTfJlyledTUXvIiUsmbgMUR1f5Gc2tG8CXt3GYk8UVnFaed3qfs5reD68oGFBt2 jaLdmVhOK8RcktCO3DlyG3S630qT3dyondHCs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=dRrvQC4A9qpcnipVIXdIIm6f3VfW6+ZG39aNDgm2/ryHbW2vvu2OZ1Hg1zbU9RRjUy 5Sg9g5R/+lMOaH0NJCO0H4GBYCV8vx+dD/EEOI84jz9Bmpwp8cvYbxPKFwncsJ2gwliS 52mW5t3jBpLEFR1muvm8zhGMxvoCaGfj+QpmM= MIME-Version: 1.0 Received: by 10.224.202.200 with SMTP id ff8mr716474qab.211.1289469496984; Thu, 11 Nov 2010 01:58:16 -0800 (PST) Received: by 10.220.200.77 with HTTP; Thu, 11 Nov 2010 01:58:14 -0800 (PST) In-Reply-To: <4CDB4BE5.9020502@cisco.com> References: <4CDB4BE5.9020502@cisco.com> Date: Thu, 11 Nov 2010 15:28:14 +0530 Message-ID: From: Vinayak Hegde To: Benoit Claise Content-Type: multipart/alternative; boundary=20cf300faf8b5133770494c403f2 Cc: pmol@ietf.org Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 09:57:49 -0000 --20cf300faf8b5133770494c403f2 Content-Type: text/plain; charset=ISO-8859-1 Since there is no quorum, I am not scheduling the meeting then. We can possibly meet at the next IETF. -- Vinayak On Thu, Nov 11, 2010 at 7:20 AM, Benoit Claise wrote: > Vinayak, > > I'm afraid I'm not available. > > Regards, Benoit. > > Hi all, > > How about having an informal meeting at IETF 79 During Friday lunchtime ? > If enough people are available, I can send an update on the location. > > -- Vinayak > > > _______________________________________________ > PMOL mailing list > PMOL@ietf.orghttps://www.ietf.org/mailman/listinfo/pmol > > > --20cf300faf8b5133770494c403f2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Since there is no quorum, I am not scheduling the meeting then. We can poss= ibly meet at the next IETF.

-- Vinayak

On Thu, Nov 11, 2010 at 7:20 AM, Benoit Claise <= bclaise@cisco.com> wrote= :
=20 =20 =20 =20
Vinayak,

I'm afraid I'm not available.

Regards, Benoit.
Hi all,

How about having an informal meeting at IETF 79 During Friday lunchtime ? If enough people are available, I can send an update on the location.

-- Vinayak

_______________________________________________
PMOL mailing list

PMOL@i=
etf.org
ht=
tps://www.ietf.org/mailman/listinfo/pmol


--20cf300faf8b5133770494c403f2-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2A53A3A6918 for ; Wed, 10 Nov 2010 18:01:48 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.559 X-Spam-Level: X-Spam-Status: No, score=-2.559 tagged_above=-999 required=5 tests=[AWL=0.039, BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dVwt7ILAcD-E for ; Wed, 10 Nov 2010 18:01:37 -0800 (PST) Received: from av-tac-bru.cisco.com (weird-brew.cisco.com [144.254.15.118]) by core3.amsl.com (Postfix) with ESMTP id 0F8993A6905 for ; Wed, 10 Nov 2010 18:01:36 -0800 (PST) X-TACSUNS: Virus Scanned Received: from strange-brew.cisco.com (localhost.cisco.com [127.0.0.1]) by av-tac-bru.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAB1onIx022353; Thu, 11 Nov 2010 02:50:49 +0100 (CET) Received: from [10.21.88.149] (sjc-vpn5-149.cisco.com [10.21.88.149]) by strange-brew.cisco.com (8.13.8+Sun/8.13.8) with ESMTP id oAB1oUl3007422; Thu, 11 Nov 2010 02:50:39 +0100 (CET) Message-ID: <4CDB4BE5.9020502@cisco.com> Date: Thu, 11 Nov 2010 09:50:29 +0800 From: Benoit Claise User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: Vinayak Hegde References: In-Reply-To: Content-Type: multipart/alternative; boundary="------------080403080106080600060401" Cc: pmol@ietf.org Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 02:01:48 -0000 This is a multi-part message in MIME format. --------------080403080106080600060401 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Vinayak, I'm afraid I'm not available. Regards, Benoit. > Hi all, > > How about having an informal meeting at IETF 79 During Friday > lunchtime ? If enough people are available, I can send an update on > the location. > > -- Vinayak > > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol --------------080403080106080600060401 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Vinayak,

I'm afraid I'm not available.

Regards, Benoit.
Hi all,

How about having an informal meeting at IETF 79 During Friday lunchtime ? If enough people are available, I can send an update on the location.

-- Vinayak

_______________________________________________
PMOL mailing list
PMOL@ietf.org
https://www.ietf.org/mailman/listinfo/pmol

--------------080403080106080600060401-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3C6D23A690F for ; Tue, 9 Nov 2010 00:41:11 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.186 X-Spam-Level: X-Spam-Status: No, score=-105.186 tagged_above=-999 required=5 tests=[AWL=0.010, BAYES_00=-2.599, J_CHICKENPOX_33=0.6, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fzO1N+QNblg4 for ; Tue, 9 Nov 2010 00:41:09 -0800 (PST) Received: from mail167.messagelabs.com (mail167.messagelabs.com [216.82.253.179]) by core3.amsl.com (Postfix) with ESMTP id 1AA343A680C for ; Tue, 9 Nov 2010 00:41:09 -0800 (PST) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-2.tower-167.messagelabs.com!1289292091!16403649!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 2989 invoked from network); 9 Nov 2010 08:41:32 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-2.tower-167.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 9 Nov 2010 08:41:32 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oA98fnkT031893 for ; Tue, 9 Nov 2010 03:41:49 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oA98fh0K031841 for ; Tue, 9 Nov 2010 03:41:43 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oA98fONG020403 for ; Tue, 9 Nov 2010 03:41:24 -0500 Received: from mailgw1.maillennium.att.com (mailgw1.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oA98fJ8g020317 for ; Tue, 9 Nov 2010 03:41:19 -0500 Message-Id: <201011090841.oA98fJ8g020317@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-239-200.vpn.east.att.com[135.70.239.200](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20101109084118gw1004lkpge>; Tue, 9 Nov 2010 08:41:19 +0000 X-Originating-IP: [135.70.239.200] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 09 Nov 2010 03:41:43 -0500 To: Vinayak Hegde , pmol@ietf.org From: Al Morton In-Reply-To: <201011090815.oA98F6mh016075@alpd052.aldc.att.com> References: <201011090815.oA98F6mh016075@alpd052.aldc.att.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 08:41:11 -0000 Hi Vin, One comment on your draft: 3.2. Connection Setup Time Once the IP address has been received by the application, it will try to setup a connection. The connection time is the difference between the instant from when the first SYN packet is sent to the destination host to the time when the final ACK is received by the sender host This definition requires a measurement at two interfaces, or both ends of the connection. A decent approximation of the setup time can be derived at the server interface, see below. The "Time to complete ANY TCP 3-way Handshake" would be the interval from the time when the first TCP SYN is received (any address family), to the time when the ACK for the Server's SYN is received, all observations made at the Server. DNS Server Client (Alice?) Server (Bob?) | | | 1. |<--www.example.com A?-----| | 2. |<--www.example.com AAAA?--| | 3. |---192.0.2.1------------->| | 4. |---2001:dba::1----------->| | 5. | | | 6. | |--TCP SYN, IPv6--->X |<*********** 7. | |--TCP SYN, IPv6--->X | | 8. | |--TCP SYN, IPv6--->X | TCP 3wHS 9. | | | Time 10. | |--TCP SYN, IPv4------->|(any family) 11. | |<-TCP SYN+ACK, IPv4----| | 12. | |--TCP ACK, IPv4------->|<*********** Figure 1: Existing behavior message flow This figure is from the Wing & Yourtchenko "Happy Eyeballs" draft. hope this helps, Al At 03:15 AM 11/9/2010, Al Morton wrote: >At 02:33 AM 11/9/2010, Vinayak Hegde wrote: >>Hi all, >> >>How about having an informal meeting at IETF 79 During Friday >>lunchtime ? If enough people are available, I can send an update on >>the location. > >I'm available, and I think we should nail down what we want >to talk about. I assume: >http://tools.ietf.org/html/draft-vinayak-http-latency-00 >or an updated version... > >Al > >_______________________________________________ >PMOL mailing list >PMOL@ietf.org >https://www.ietf.org/mailman/listinfo/pmol Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3111D3A695E for ; Tue, 9 Nov 2010 00:19:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.462 X-Spam-Level: X-Spam-Status: No, score=-102.462 tagged_above=-999 required=5 tests=[AWL=0.137, BAYES_00=-2.599, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id touL+YXBIcEQ for ; Tue, 9 Nov 2010 00:19:53 -0800 (PST) Received: from postgirl.ripe.net (postgirl.ipv6.ripe.net [IPv6:2001:610:240:11::c100:1342]) by core3.amsl.com (Postfix) with ESMTP id B2FEF3A690A for ; Tue, 9 Nov 2010 00:19:52 -0800 (PST) Received: from dodo.ripe.net ([193.0.23.4]) by postgirl.ripe.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.63) (envelope-from ) id 1PFjR3-0001Cw-KZ for pmol@ietf.org; Tue, 09 Nov 2010 09:20:15 +0100 Received: from [193.0.21.124] (helo=dhcp-4418.meeting.ietf.org) by dodo.ripe.net with esmtp (Exim 4.63) (envelope-from ) id 1PFjR2-0003of-VV for pmol@ietf.org; Tue, 09 Nov 2010 09:20:09 +0100 Message-ID: <4CD90436.1080708@ripe.net> Date: Tue, 09 Nov 2010 09:20:06 +0100 From: Henk Uijterwaal Organization: RIPE NCC User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-GB; rv:1.9.2.12) Gecko/20101027 Lightning/1.0b2 Thunderbird/3.1.6 MIME-Version: 1.0 To: pmol@ietf.org References: <201011090815.oA98F6mh016075@alpd052.aldc.att.com> In-Reply-To: <201011090815.oA98F6mh016075@alpd052.aldc.att.com> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-RIPE-Signature: e0cdef1f45f89a40ad608d255b27e7d5bc928a2bbfd4ca7cb66e1b3d4455719c X-RIPE-Spam-Level: -- X-RIPE-Signature: e0cdef1f45f89a40ad608d255b27e7d5bc928a2bbfd4ca7cb66e1b3d4455719c Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 08:19:54 -0000 On 09/11/2010 09:15, Al Morton wrote: > At 02:33 AM 11/9/2010, Vinayak Hegde wrote: >> Hi all, >> >> How about having an informal meeting at IETF 79 During Friday lunchtime ? If >> enough people are available, I can send an update on the location. I'm not available, unless you want to meet in the China Southern lounge on the airport. Henk > > I'm available, and I think we should nail down what we want > to talk about. I assume: > http://tools.ietf.org/html/draft-vinayak-http-latency-00 > or an updated version... > > Al > > _______________________________________________ > PMOL mailing list > PMOL@ietf.org > https://www.ietf.org/mailman/listinfo/pmol > -- ------------------------------------------------------------------------------ Henk Uijterwaal Email: henk.uijterwaal(at)ripe.net RIPE Network Coordination Centre http://www.xs4all.nl/~henku P.O.Box 10096 Singel 258 Phone: +31.20.5354414 1001 EB Amsterdam 1016 AB Amsterdam Fax: +31.20.5354445 The Netherlands The Netherlands Mobile: +31.6.55861746 ------------------------------------------------------------------------------ I confirm today what I denied yesterday. Anonymous Politician. Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id C73DC3A6914 for ; Tue, 9 Nov 2010 00:14:54 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -105.331 X-Spam-Level: X-Spam-Status: No, score=-105.331 tagged_above=-999 required=5 tests=[AWL=0.465, BAYES_00=-2.599, MSGID_FROM_MTA_HEADER=0.803, RCVD_IN_DNSWL_MED=-4, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id scK1gfpSaN7Y for ; Tue, 9 Nov 2010 00:14:52 -0800 (PST) Received: from mail129.messagelabs.com (mail129.messagelabs.com [216.82.250.147]) by core3.amsl.com (Postfix) with ESMTP id 9F3193A690F for ; Tue, 9 Nov 2010 00:14:52 -0800 (PST) X-VirusChecked: Checked X-Env-Sender: acmorton@att.com X-Msg-Ref: server-15.tower-129.messagelabs.com!1289290515!35273049!1 X-StarScan-Version: 6.2.9; banners=-,-,- X-Originating-IP: [144.160.20.145] Received: (qmail 20601 invoked from network); 9 Nov 2010 08:15:15 -0000 Received: from sbcsmtp6.sbc.com (HELO mlpd192.enaf.sfdc.sbc.com) (144.160.20.145) by server-15.tower-129.messagelabs.com with DHE-RSA-AES256-SHA encrypted SMTP; 9 Nov 2010 08:15:15 -0000 Received: from enaf.sfdc.sbc.com (localhost.localdomain [127.0.0.1]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oA98FX35016183 for ; Tue, 9 Nov 2010 03:15:33 -0500 Received: from alpd052.aldc.att.com (alpd052.aldc.att.com [130.8.42.31]) by mlpd192.enaf.sfdc.sbc.com (8.14.4/8.14.4) with ESMTP id oA98FT36016141 for ; Tue, 9 Nov 2010 03:15:29 -0500 Received: from aldc.att.com (localhost.localdomain [127.0.0.1]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oA98FAii016179 for ; Tue, 9 Nov 2010 03:15:10 -0500 Received: from dns.maillennium.att.com (dns.maillennium.att.com [135.25.114.99]) by alpd052.aldc.att.com (8.14.4/8.14.4) with ESMTP id oA98F6mh016075 for ; Tue, 9 Nov 2010 03:15:07 -0500 Message-Id: <201011090815.oA98F6mh016075@alpd052.aldc.att.com> Received: from acmt.att.com (vpn-135-70-239-200.vpn.east.att.com[135.70.239.200](misconfigured sender)) by maillennium.att.com (mailgw1) with SMTP id <20101109081505gw1004lkpde>; Tue, 9 Nov 2010 08:15:06 +0000 X-Originating-IP: [135.70.239.200] X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 09 Nov 2010 03:15:30 -0500 To: Vinayak Hegde , pmol@ietf.org From: Al Morton In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 08:14:55 -0000 At 02:33 AM 11/9/2010, Vinayak Hegde wrote: >Hi all, > >How about having an informal meeting at IETF 79 During Friday >lunchtime ? If enough people are available, I can send an update on >the location. I'm available, and I think we should nail down what we want to talk about. I assume: http://tools.ietf.org/html/draft-vinayak-http-latency-00 or an updated version... Al Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B7B023A696D for ; Tue, 9 Nov 2010 00:05:34 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -102.543 X-Spam-Level: X-Spam-Status: No, score=-102.543 tagged_above=-999 required=5 tests=[AWL=0.055, BAYES_00=-2.599, HTML_MESSAGE=0.001, USER_IN_WHITELIST=-100] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Pp7Evu0Pry13 for ; Tue, 9 Nov 2010 00:05:34 -0800 (PST) Received: from co300216-co-outbound.net.avaya.com (co300216-co-outbound.net.avaya.com [198.152.13.100]) by core3.amsl.com (Postfix) with ESMTP id D3C3D3A6866 for ; Tue, 9 Nov 2010 00:05:33 -0800 (PST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAG2P2EyHCzI1/2dsb2JhbACiHnGjEAKZWIVKBI16 X-IronPort-AV: E=Sophos;i="4.59,174,1288584000"; d="scan'208,217";a="249135186" Received: from unknown (HELO p-us1-erheast.us1.avaya.com) ([135.11.50.53]) by co300216-co-outbound.net.avaya.com with ESMTP; 09 Nov 2010 03:05:56 -0500 X-IronPort-AV: E=Sophos;i="4.59,174,1288584000"; d="scan'208,217";a="535698438" Received: from unknown (HELO 307622ANEX5.global.avaya.com) ([135.64.140.14]) by p-us1-erheast-out.us1.avaya.com with ESMTP; 09 Nov 2010 03:05:56 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CB7FE4.E4193217" Date: Tue, 9 Nov 2010 09:05:35 +0100 Message-ID: In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [PMOL] Informal Lunch meeting at IETF 79 Thread-Index: Act/4H6h/+tzlQVoRVOAJFL0IkstmwABE1Nw References: From: "Romascanu, Dan (Dan)" To: "Vinayak Hegde" Cc: pmol@ietf.org Subject: Re: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 08:05:34 -0000 This is a multi-part message in MIME format. ------_=_NextPart_001_01CB7FE4.E4193217 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable If no other events preempt I may be able to join.=20 =20 Dan =20 ________________________________ From: pmol-bounces@ietf.org [mailto:pmol-bounces@ietf.org] On Behalf Of Vinayak Hegde Sent: Tuesday, November 09, 2010 9:34 AM To: pmol@ietf.org Subject: [PMOL] Informal Lunch meeting at IETF 79 =09 =09 Hi all, =09 How about having an informal meeting at IETF 79 During Friday lunchtime ? If enough people are available, I can send an update on the location. =09 -- Vinayak =09 ------_=_NextPart_001_01CB7FE4.E4193217 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
If no=20 other events preempt I may be able to join.
 
Dan
 

From: pmol-bounces@ietf.org=20 [mailto:pmol-bounces@ietf.org] On Behalf Of Vinayak=20 Hegde
Sent: Tuesday, November 09, 2010 9:34 AM
To: = pmol@ietf.org
Subject: [PMOL] Informal Lunch meeting at IETF = 79

Hi all,

How about having an informal meeting at IETF = 79=20 During Friday lunchtime ? If enough people are available, I can send = an update=20 on the location.

-- Vinayak
------_=_NextPart_001_01CB7FE4.E4193217-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 81D563A67FC for ; Mon, 8 Nov 2010 23:33:37 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.598 X-Spam-Level: X-Spam-Status: No, score=-2.598 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HTML_MESSAGE=0.001] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xJQ56Kqcynmj for ; Mon, 8 Nov 2010 23:33:36 -0800 (PST) Received: from mail-vw0-f44.google.com (mail-vw0-f44.google.com [209.85.212.44]) by core3.amsl.com (Postfix) with ESMTP id C17223A67A1 for ; Mon, 8 Nov 2010 23:33:36 -0800 (PST) Received: by vws3 with SMTP id 3so3013892vws.31 for ; Mon, 08 Nov 2010 23:33:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=5vOkibhGr91smfh+RSwOIfJpLB6e/V/ym13RoNXRtoI=; b=Tdb/Ze/v+OhOl5vp9Oq43lrBTCH5iFDBg2wLtljckSpQgdedPMiL3oUiUGq42BGCmt 8CAMn/xzPboxZS8+zaGbiHwmTgGN0cPXZrdadiB9KYsGsOQbE0ImQbYVAZGPuok+xyUo z7HLZqZCp4Xenaa6wBioovK7KA3GGz+FjDlhg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=wL6pMfBue1VFTnEXvL0zY9lhbGkW3BVqUDZSDZ7b5BYzk1guQZsUGgORgB3PD/xxF9 RiuwzJcrCBXHODiDQbbb0J6SeD8mM9lDkuaRD4tWNn36NxxCdEJNB5Sle9paUmKQgMFp iFootPInJqFUENeXoJwyi6jZsbkM8TcemKccY= MIME-Version: 1.0 Received: by 10.224.11.66 with SMTP id s2mr4668563qas.311.1289288039098; Mon, 08 Nov 2010 23:33:59 -0800 (PST) Received: by 10.220.200.77 with HTTP; Mon, 8 Nov 2010 23:33:58 -0800 (PST) Date: Tue, 9 Nov 2010 13:03:58 +0530 Message-ID: From: Vinayak Hegde To: pmol@ietf.org Content-Type: multipart/alternative; boundary=0015175cda049568c7049499c365 Subject: [PMOL] Informal Lunch meeting at IETF 79 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2010 07:33:37 -0000 --0015175cda049568c7049499c365 Content-Type: text/plain; charset=ISO-8859-1 Hi all, How about having an informal meeting at IETF 79 During Friday lunchtime ? If enough people are available, I can send an update on the location. -- Vinayak --0015175cda049568c7049499c365 Content-Type: text/html; charset=ISO-8859-1 Hi all,

How about having an informal meeting at IETF 79 During Friday lunchtime ? If enough people are available, I can send an update on the location.

-- Vinayak
--0015175cda049568c7049499c365-- Return-Path: X-Original-To: pmol@core3.amsl.com Delivered-To: pmol@core3.amsl.com Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0D58E28C0FA for ; Wed, 3 Nov 2010 09:31:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -3.599 X-Spam-Level: X-Spam-Status: No, score=-3.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_LOW=-1] Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aaIz-1-P6t1Z for ; Wed, 3 Nov 2010 09:31:29 -0700 (PDT) Received: from relay01-haj2.antispameurope.com (relay01-haj2.antispameurope.com [83.246.65.51]) by core3.amsl.com (Postfix) with ESMTP id 0191F3A6AC9 for ; Wed, 3 Nov 2010 09:31:28 -0700 (PDT) Received: by relay01-haj2.antispameurope.com (ASE-Secure-MTA, from userid 1000) id 8D6FA160034; Wed, 3 Nov 2010 17:31:34 +0100 (CET) Received: from pluto.fokus.fraunhofer.de (pluto.fokus.fraunhofer.de [195.37.77.164]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relay01-haj2.antispameurope.com (ASE-Secure-MTA) with ESMTP id DB455160011 for ; Wed, 3 Nov 2010 17:31:33 +0100 (CET) Received: from EXCHSRV.fokus.fraunhofer.de (bohr.fokus.fraunhofer.de [10.147.9.231]) by pluto.fokus.fraunhofer.de (8.14.2/8.14.2) with SMTP id oA3GVX7E010425 for ; Wed, 3 Nov 2010 17:31:33 +0100 (MET) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Wed, 3 Nov 2010 17:31:33 +0100 Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_036F_01CB7B7C.F5975D50" Message-ID: <70524A4436C03E43A293958B505008B60308A02A@EXCHSRV.fokus.fraunhofer.de> X-MS-Has-Attach: yes X-MS-TNEF-Correlator: Thread-Topic: Feedback on draft-ietf-pmol-metrics-framework-05 Thread-Index: Act7dJPO3Z9U4NijSYSA5JlHZ3wlFA== From: "Schmoll, Carsten" To: Subject: [PMOL] Feedback on draft-ietf-pmol-metrics-framework-05 X-BeenThere: pmol@ietf.org X-Mailman-Version: 2.1.9 Precedence: list List-Id: Performance Metrics at Other Layers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Nov 2010 16:31:31 -0000 This is a multi-part message in MIME format. ------=_NextPart_000_036F_01CB7B7C.F5975D50 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Dear authors of draft-ietf-pmol-metrics-framework-05, =A0 there is only very few comments I can make on the draft =96 it is in = very good shape (content- and English-wise)! =A0 =A0 here are my few (hopefully helpful) observations: =A0 (quoted text is taken from the draft, then afterwards I note a possible change or addition in each of the following paragraphs) #########################################################################= =93The key areas of variation between different metric users include: =A0 * Required reporting formats=94 =A0 =A0 Maybe write here instead: =A0 =93 * Required reporting formats and periods=94 #########################################################################= =A0 =A0 "Some metrics may not be measured directly, but may be composed from =A0=A0 base metrics that have been measured." =A0 I am not sure about the double use of =93may=94 here. Maybe =93can be = composed=94 would be better?!? =A0 #########################################################################= =A0 "Some parameters linked to the method MAY also be reported, in order =A0=A0 to fully interpret the Performance Metric.=A0 For example, the = time =A0=A0 interval, the load, the minimum packet loss, etc..." =A0 I think it would be useful to note also that it makes sense to report = the possible measurement errors and or the accuracy of results. e.g. =93potential measurement errors and their sources should be noted. = The attainable accuracy of the metric (e.g. +-0,1) should be reported as = well.=94 #########################################################################= =A0 =A0 "The objective of this part of the metric definition is to assist implementers to achieve a consistent result." =A0 Change to plural ( =93consistent results=94 ) ? #########################################################################= =A0=A0=20 "(iii) Use and Applications =A0 =A0=A0 The use and applications description is intended to assist the = "user" =A0=A0 to understand how, when and where the metric can be applied, and = what =A0=A0 significance the value range for the metric may have.=A0 This MAY =A0=A0 include a definition of the "typical" and "abnormal" range of the =A0=A0 Performance Metric, if this was not apparent from the nature of = the =A0=A0 metric. " Maybe something could be added like =93The description may (should??) also include information about the = influence of extreme measurement values, i.e. if the metric is very sensitive to outliers.=94 #########################################################################= =A0 =A0 Space between =93Metric=94 and the dot: =A0 "of the Performance Metric ." -> "of the Performance Metric." =A0 #########################################################################= =A0 =A0 =93Existing mailing lists SHOULD be used however a dedicated mailing = list MAY be initiated if necessary to facilitate work on a draft.=94 =A0 I recommend to add a comma after =93used=94. #########################################################################= =A0 Best regards Carsten =A0 =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=20 =A0 --=20 "With great power comes great responsibility!=94=A0 What power, you = ask?=A0 UNIX root privileges. =A0 Carsten Schmoll Dipl.-Ing. Project Manager Network Research +49 (0) 30 3463-7136 =A0 Fraunhofer FOKUS www.fokus.fraunhofer.de =A0 ------=_NextPart_000_036F_01CB7B7C.F5975D50 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIYDzCCA58w ggKHoAMCAQICASYwDQYJKoZIhvcNAQEFBQAwcTELMAkGA1UEBhMCREUxHDAaBgNVBAoTE0RldXRz Y2hlIFRlbGVrb20gQUcxHzAdBgNVBAsTFlQtVGVsZVNlYyBUcnVzdCBDZW50ZXIxIzAhBgNVBAMT GkRldXRzY2hlIFRlbGVrb20gUm9vdCBDQSAyMB4XDTk5MDcwOTEyMTEwMFoXDTE5MDcwOTIzNTkw MFowcTELMAkGA1UEBhMCREUxHDAaBgNVBAoTE0RldXRzY2hlIFRlbGVrb20gQUcxHzAdBgNVBAsT FlQtVGVsZVNlYyBUcnVzdCBDZW50ZXIxIzAhBgNVBAMTGkRldXRzY2hlIFRlbGVrb20gUm9vdCBD QSAyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqwujNeCLKRSxFIWvPBDkOW81XUqu 3ephjZVJ9G9koxpgZqSpQCKE2dSl5XiTDmgBrblNXDrO07ioQkDfz6O6gllqkhusHJraCCslJ/lp I0fx4Ossepv1EwLQfjR8wp48AFmr9doM9TI8K6xQ2tbD3oOUyqgMmTIOCEhWW2r72uFYWAFJX3JB PBUGAY5draq4k7TNnuun6GotUjTbOu9cdVHa2/Mx+e5xmDLEVBVEDPmbVe2t3xgIoKOGiknuUwWP GUzV3lh5m9JqHEKrxdWnz2gPluThYZh2YciRfNY+AOKRUIfhnQrmrZfSHcY6fcu82gM01Y5bAfVq B7cWtm5KfwIDAQABo0IwQDAdBgNVHQ4EFgQUMcN5G7r1U9cX4Il6LRdsCrMrnTMwDwYDVR0TBAgw BgEB/wIBBTAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQEFBQADggEBAJRkWa05ZOcp6xP+WsOL E1fIBCTwdHfAYONn++mJpoO/loJ8btTDPe+egG67KbSYerE7VOs5F0d+Go4L/B8xWTEEss4X8yzH YjZV4iLYiVW0mEiqZPrWHDbYRHhaWiM6V5f1ejBPrp9qTEsrjqAD4z7gqdTSe9KzqOJyPK2e/4BZ 5JtFtPY7sM05GZgy5eohYZDkMSGONLH3LzVKhRDa54o3Ib5ZY+DyhYgxU9RUFIVwefQuBncndS8f uIr5/sW62Dbkg+znZbe/Y1rzRq+BlDfUQYzWI9Yez/VoG0Rjolq6pzVZoeVwBZsOI1eZlAptujlj KIaS8xiE2PvRzwVWZFcwggQuMIIDFqADAgECAgIBDDANBgkqhkiG9w0BAQUFADBxMQswCQYDVQQG EwJERTEcMBoGA1UEChMTRGV1dHNjaGUgVGVsZWtvbSBBRzEfMB0GA1UECxMWVC1UZWxlU2VjIFRy dXN0IENlbnRlcjEjMCEGA1UEAxMaRGV1dHNjaGUgVGVsZWtvbSBSb290IENBIDIwHhcNMDcxMjA1 MTUxODU4WhcNMTkwNjMwMjM1OTU5WjBnMQswCQYDVQQGEwJERTETMBEGA1UEChMKRnJhdW5ob2Zl cjEhMB8GA1UECxMYRnJhdW5ob2ZlciBDb3Jwb3JhdGUgUEtJMSAwHgYDVQQDExdGcmF1bmhvZmVy IFJvb3QgQ0EgMjAwNzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMM9HslljVuIN/c+ wSG2VN6wOlcbGhzumAh0AVpvtd82VytERc7zRhNldpWYVMeohRVuEqQX6AvhIacxPSnd/6PWT9K0 if1jStqrU+9ZBbRP4jsqkZnAy2VXgaMwk5AOyb39FTIyCKRIfFYF8rs2ieJIwm9LkgwlRb6tPtTH Ae2xyBK0MAhup29/eiCnu7TlNYBULBmP2UvAHatD+GEUlPuwHV23kexiwJHRXt2nC7MtH2Y5GDyu BBTl0n022bLNpxOwzEWdVgRRcTM2XC9dUROeT0iQDpdaS3DbuXnZodaUyJLQpftgvy2tnnEAF4bA /UhmnRRMBU5M0XFEC9B6D7ECAwEAAaOB2TCB1jAfBgNVHSMEGDAWgBQxw3kbuvVT1xfgiXotF2wK syudMzAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFC9FQh4xBYDVcNj4HVfLW3rVPZz3MBIGA1Ud EwEB/wQIMAYBAf8CAQEwcAYDVR0fBGkwZzBloGOgYYZfaHR0cDovL3BraS50ZWxlc2VjLmRlL2Nn aS1iaW4vc2VydmljZS9hZl9Eb3dubG9hZEFSTC5jcmw/LWNybF9mb3JtYXQ9WF81MDkmLWlzc3Vl cj1EVF9ST09UX0NBXzIwDQYJKoZIhvcNAQEFBQADggEBABq3THo85d3CY7LehiVORWJM9PKPX13X 5Y3ODtf1lOGLkzE2p5oj52fhUsFaGNaZ4YBkOA9KNTNdZnoo3Dh/N3LNUnVuEI5sfYz3Ket3wrkZ BdS3PWG66AUS1FBiU+8iVGL8TQHDXtQNg3RpUdU8nqzbpCt8bYSW03FNz9UtcaOSxD9Vz5s9I3cH V+nIzh2XtjP7kZdglg/39t5vJZASqxlH1UQjrsGSNSi/KkNeD+oHXdJE0IWC4xK8R+osrfjwQVJ9 Nroin3qgMu9LvPk6B7Ypxn04XzVVfjjyP3yz7i1uIXhfuRPP795lgMgl9WYtVEqtztkuDjDPgDOn ixly6kEwggTGMIIDrqADAgECAgphHTMZAAAAAAADMA0GCSqGSIb3DQEBBQUAMGcxCzAJBgNVBAYT AkRFMRMwEQYDVQQKEwpGcmF1bmhvZmVyMSEwHwYDVQQLExhGcmF1bmhvZmVyIENvcnBvcmF0ZSBQ S0kxIDAeBgNVBAMTF0ZyYXVuaG9mZXIgUm9vdCBDQSAyMDA3MB4XDTA3MTIxMjE0NDkzNVoXDTE5 MDYzMDIzNTk1OVowZzELMAkGA1UEBhMCREUxEzARBgNVBAoTCkZyYXVuaG9mZXIxITAfBgNVBAsT GEZyYXVuaG9mZXIgQ29ycG9yYXRlIFBLSTEgMB4GA1UEAxMXRnJhdW5ob2ZlciBVc2VyIENBIDIw MDcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDCdBBRy3XRgfErNZ1agjqvkq7NAvyu yPsgdw7/aBIVkEmnZ3xoJF/6DiYaZT5KngkK04dPMxGKa1WiS02lT1Yb48d4efijC95VRmSH2owP PU6JYS8JPppdvVW6leZy4Ta0sHWctee2+nQruSX38b6TucFwp/eWf0y8i3XrsYFR7PzJ4XM75gWw UUvjsPVKCp/j2b8mJlBHTh4PHAOTxhWkT/avPKu9+FZ5SPluPsuwaDecLbqla6lm26TkNrB2KPa4 r92/seVnbDK1YnihgKUP3Tb4Aeb7rgHIUZ7ADWR4KQIjwsz+/0BWGKpNrp3XMRF8D4KO9qInwd7k Gd8Rzlf3AgMBAAGjggFyMIIBbjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBRPHa+Iym24 qhwJ+cXREe1ZtJP6CzAOBgNVHQ8BAf8EBAMCAQYwHwYDVR0jBBgwFoAUL0VCHjEFgNVw2PgdV8tb etU9nPcwdQYDVR0fBG4wbDBqoGigZoYxaHR0cDovL2NybC5wa2kuZnJhdW5ob2Zlci5kZS9maGct cm9vdC1jYS0yMDA3LmNybIYxaHR0cDovL2NybC5mcmF1bmhvZmVyLXBraS5kZS9maGctcm9vdC1j YS0yMDA3LmNybDCBkAYIKwYBBQUHAQEEgYMwgYAwPgYIKwYBBQUHMAKGMmh0dHA6Ly9jZXJ0LnBr aS5mcmF1bmhvZmVyLmRlL2ZoZy1yb290LWNhLTIwMDcuY2VyMD4GCCsGAQUFBzAChjJodHRwOi8v Y2VydC5mcmF1bmhvZmVyLXBraS5kZS9maGctcm9vdC1jYS0yMDA3LmNlcjANBgkqhkiG9w0BAQUF AAOCAQEAAHdsjDP57lcEJcrLiDnmTkykzsKWSxmGTbhEV8KVqRA37DBVMV/1nrW7tWtaoXfseksl 2OFdQs3BQzhhd8AO28iFf+ncgudyPZbmU48vc41LIy2bnA+I3jY2t0BLt/bELGLLhNzE3xRSYK/z XgBTB1ZJBxsd5VV2/hdyd/vcXlx5P8Af8iCnf0SIjDE0+Pmvq2tjEwrGAHuJICdL89k12BjfLbo4 woLFFG8HyQvcKzogiTDSTNYu7FFalSiKLMFu4eUKVlDbtROEQLAxM+3XqVwmGgCMD6axCJK9HROt 86h4ZjOHoEu4hpgXzUKIVDuoyTljXCfXSManNHP+FlB4PDCCBa4wggSWoAMCAQICCiQbfogAAAAA LvkwDQYJKoZIhvcNAQEFBQAwZzELMAkGA1UEBhMCREUxEzARBgNVBAoTCkZyYXVuaG9mZXIxITAf BgNVBAsTGEZyYXVuaG9mZXIgQ29ycG9yYXRlIFBLSTEgMB4GA1UEAxMXRnJhdW5ob2ZlciBVc2Vy IENBIDIwMDcwHhcNMDgwNzMwMDg0ODQxWhcNMTQwNzI5MDg0ODQxWjBdMQswCQYDVQQGEwJERTET MBEGA1UEChMKRnJhdW5ob2ZlcjEOMAwGA1UECxMFRk9LVVMxDzANBgNVBAsTBlBlb3BsZTEYMBYG A1UEAxMPQ2Fyc3RlbiBTY2htb2xsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqCYI VA+4RIuFi8eDG5HaAy9D8nwzf6VEMcGRIMBUcfWtqlg1RxCJRXUwAaBtNBYVyyYQVw9Ns8JCjR/A J+k8FjEngEI5s7CQDs5X0lrH6La2pL8B6qp5FH5IytJX9K0i7AT9O5mvWr0XZq2iX5T0jtFVRPLM iRaQE/eO/db/OOdWBupMIccs8Ogi/AfLIFcvUqmIyxL8AmxL6dIqUD9IxoqZU/OBX2wYBE9rVxcb jjFhoERgWaXPI9u7oCVvmr2EgKix1E+bGDSqthLd3mCufcSPYR3ruiIr6oJ7aGRI314vKjGyDB4z lc2AjlVEnBVIscd51+VaNpjFnFwOZppHxwIDAQABo4ICZDCCAmAwDgYDVR0PAQH/BAQDAgbAMC4G A1UdEQQnMCWBI2NhcnN0ZW4uc2NobW9sbEBmb2t1cy5mcmF1bmhvZmVyLmRlMB0GA1UdDgQWBBRW H2ey9Dgiuz+McTYdYXoPjfuHxTAfBgNVHSMEGDAWgBRPHa+Iym24qhwJ+cXREe1ZtJP6CzB1BgNV HR8EbjBsMGqgaKBmhjFodHRwOi8vY3JsLnBraS5mcmF1bmhvZmVyLmRlL2ZoZy11c2VyLWNhLTIw MDcuY3JshjFodHRwOi8vY3JsLmZyYXVuaG9mZXItcGtpLmRlL2ZoZy11c2VyLWNhLTIwMDcuY3Js MIIBCgYIKwYBBQUHAQEEgf0wgfowPgYIKwYBBQUHMAKGMmh0dHA6Ly9jZXJ0LnBraS5mcmF1bmhv ZmVyLmRlL2ZoZy11c2VyLWNhLTIwMDcuY2VyMD4GCCsGAQUFBzAChjJodHRwOi8vY2VydC5mcmF1 bmhvZmVyLXBraS5kZS9maGctdXNlci1jYS0yMDA3LmNlcjA7BggrBgEFBQcwAYYvaHR0cDovL2Zo Zy11c2VyLWNhLTIwMDcub2NzcC5wa2kuZnJhdW5ob2Zlci5kZS8wOwYIKwYBBQUHMAGGL2h0dHA6 Ly9maGctdXNlci1jYS0yMDA3Lm9jc3AuZnJhdW5ob2Zlci1wa2kuZGUvMBMGA1UdJQQMMAoGCCsG AQUFBwMEMEQGA1UdIAQ9MDswOQYLKwYBBAGGClADAQEwKjAoBggrBgEFBQcCARYcaHR0cDovL3Br aS5mcmF1bmhvZmVyLmRlL2NwLzANBgkqhkiG9w0BAQUFAAOCAQEAKCSUKN9DtFwl/8JK3k2PJdt+ U888hm4xWLeMtdDzlcNwoHB0tn/+Q3VnxnGqFefEB7mn4ozxU4id684LvzQyJPEM1qKEbDPafQTu gpvCZ4rldWRauit8SQ44LJYJway02q/BgklCDqWKl8vIzzOXZZxbSwwA/HserIoGkUvLKl2c+V/T IDIJdSncfXehNZOZ1+hzWrHUq3qrQmmIWEy+LWKWO0CUUForJBXM6JrZVfTj9IqbYwixiT3Rf/ht Yv2IiiSVlP68egAqJmbzBqxJVei26XXztEbP9CpAQMxgjOnB8U90k+ZLFtY4mDFxxV6+RC+E3GQm qdOTXnuyei3XeTCCBbowggSioAMCAQICCiQbex0AAAAALvgwDQYJKoZIhvcNAQEFBQAwZzELMAkG A1UEBhMCREUxEzARBgNVBAoTCkZyYXVuaG9mZXIxITAfBgNVBAsTGEZyYXVuaG9mZXIgQ29ycG9y YXRlIFBLSTEgMB4GA1UEAxMXRnJhdW5ob2ZlciBVc2VyIENBIDIwMDcwHhcNMDgwNzMwMDg0ODQw WhcNMTQwNzI5MDg0ODQwWjBdMQswCQYDVQQGEwJERTETMBEGA1UEChMKRnJhdW5ob2ZlcjEOMAwG A1UECxMFRk9LVVMxDzANBgNVBAsTBlBlb3BsZTEYMBYGA1UEAxMPQ2Fyc3RlbiBTY2htb2xsMIIB IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApK2s+9MlCtIh1ePVIHMIMEoQRppMqJOE4L6S WU3kzAMBQ/zvqni7ygsy48DJIYq7S+JjbP8eE5oKRHZuekfaS8+wEKG6teHT7F52wkZYLiHHAkoT 8gQjvTHL0HB+eNYOX5si7IbSIbmGPaNFFswbdEUW9FNZ3CXaekJsOotn3EmBmh8+/Xg50fes5pOz hhD5ybQVae7qzZRci1o6dolcv4s5P5qSfmyOpMRp7rXBLBRUkAZhzbwPzhlrtKz0WIQJMIQhQHYK RjuJtBePTkGYu5s6srOOMIfLTKDNg4ip6U9WLg/bBv0p2tKqTZ9U7OkMoPf13JXhLaDPGCBU4/MD TwIDAQABo4ICcDCCAmwwDgYDVR0PAQH/BAQDAgQwMC4GA1UdEQQnMCWBI2NhcnN0ZW4uc2NobW9s bEBmb2t1cy5mcmF1bmhvZmVyLmRlMB0GA1UdDgQWBBSDvYGXqr0uzn83nTw+eOC0+Sw1hTAfBgNV HSMEGDAWgBRPHa+Iym24qhwJ+cXREe1ZtJP6CzB1BgNVHR8EbjBsMGqgaKBmhjFodHRwOi8vY3Js LnBraS5mcmF1bmhvZmVyLmRlL2ZoZy11c2VyLWNhLTIwMDcuY3JshjFodHRwOi8vY3JsLmZyYXVu aG9mZXItcGtpLmRlL2ZoZy11c2VyLWNhLTIwMDcuY3JsMIIBCgYIKwYBBQUHAQEEgf0wgfowPgYI KwYBBQUHMAKGMmh0dHA6Ly9jZXJ0LnBraS5mcmF1bmhvZmVyLmRlL2ZoZy11c2VyLWNhLTIwMDcu Y2VyMD4GCCsGAQUFBzAChjJodHRwOi8vY2VydC5mcmF1bmhvZmVyLXBraS5kZS9maGctdXNlci1j YS0yMDA3LmNlcjA7BggrBgEFBQcwAYYvaHR0cDovL2ZoZy11c2VyLWNhLTIwMDcub2NzcC5wa2ku ZnJhdW5ob2Zlci5kZS8wOwYIKwYBBQUHMAGGL2h0dHA6Ly9maGctdXNlci1jYS0yMDA3Lm9jc3Au ZnJhdW5ob2Zlci1wa2kuZGUvMB8GA1UdJQQYMBYGCisGAQQBgjcKAwQGCCsGAQUFBwMEMEQGA1Ud IAQ9MDswOQYLKwYBBAGGClADAQEwKjAoBggrBgEFBQcCARYcaHR0cDovL3BraS5mcmF1bmhvZmVy LmRlL2NwLzANBgkqhkiG9w0BAQUFAAOCAQEAkA63yPk5+08ii49lnTe3mqNmftSg0NVICyHckMOC n9l9yIAF7x+M9/Dwo3zd5EVUWZmzTFQt+lawU1uApvMp7CfGvAPIXTgW22/RLRVZbswD8V4C8QP5 yoLXiZaz/MoJwoSmehYR025XszMbZHDZLa8ZbKnnMA0JNhFwnDWcRJ7zTl6M0Z/CgF4bF4Gu5SKz NEWN6sn2MmuWdqjrsrmAT3DrBvCAYYNVwkifTweLm5Ok0/VdJU8rx2lXQDn/qnVH/RZzqvcYKK/n eXGO5Nm4MZhKhQHUJlveKJikH26Q0qaVL8DAlP3ifLuq098DxmBi/d2GH04Qudk8mACB5/QJ2DGC A3YwggNyAgEBMHUwZzELMAkGA1UEBhMCREUxEzARBgNVBAoTCkZyYXVuaG9mZXIxITAfBgNVBAsT GEZyYXVuaG9mZXIgQ29ycG9yYXRlIFBLSTEgMB4GA1UEAxMXRnJhdW5ob2ZlciBVc2VyIENBIDIw MDcCCiQbfogAAAAALvkwCQYFKw4DAhoFAKCCAdYwGAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAc BgkqhkiG9w0BCQUxDxcNMTAxMTAzMTYzMTMzWjAjBgkqhkiG9w0BCQQxFgQUcRlJqCkJNy6xBylk VIXLHLTY6x4wZwYJKoZIhvcNAQkPMVowWDAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYI KoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwBwYFKw4DAhowCgYIKoZIhvcNAgUw gYQGCSsGAQQBgjcQBDF3MHUwZzELMAkGA1UEBhMCREUxEzARBgNVBAoTCkZyYXVuaG9mZXIxITAf BgNVBAsTGEZyYXVuaG9mZXIgQ29ycG9yYXRlIFBLSTEgMB4GA1UEAxMXRnJhdW5ob2ZlciBVc2Vy IENBIDIwMDcCCiQbex0AAAAALvgwgYYGCyqGSIb3DQEJEAILMXegdTBnMQswCQYDVQQGEwJERTET MBEGA1UEChMKRnJhdW5ob2ZlcjEhMB8GA1UECxMYRnJhdW5ob2ZlciBDb3Jwb3JhdGUgUEtJMSAw HgYDVQQDExdGcmF1bmhvZmVyIFVzZXIgQ0EgMjAwNwIKJBt7HQAAAAAu+DANBgkqhkiG9w0BAQEF AASCAQAceVBIr+SfF5WZ8LhXS4NEkezwS9xEHazCnvT6e1/MjHDd145Tr2rqfJ/njvO+06xF6agt ilpec63FoA9oew2vJJK99RlPHpwr/96+T7cWkFhh4z9VawDBD81/hhG46PP3Rr/NlNfeJZqvp0KP jQp3lBB0VU7WUt0Pf0MZTp90p+HyDb2jwaecDmBkR8vyypmXRvpYwb6h2WPlLD4bO3cfWc84N1+I Q27cjR2jTN7o6yMOsWLJtwpywsPQsxqIbrOhAI0Y/mwxTeOaupicVkyf2vahu0L4kzUvR5uqVvHg rEXvG44KiSyA9jPNlYn1evhd3P+s78CQD/OotecCooqCAAAAAAAA ------=_NextPart_000_036F_01CB7B7C.F5975D50--