Subject: [PMOL] PMOL directorate: meeting at the IETF?

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Oct 2012 18:29:13 -0400

Dear PMOL directorate members,

During my review of the latest AVTCORE and  XRBLOCK drafts ( 
draft-ietf-xrblock-rtcp-xr-pdv and draft-ietf-avtcore-monarch), I came 
to the conclusion that we have an issue in terms of performance metrics 
at the IETF, and actually in the industry.

As background information, here is my DISCUSS on the two drafts

    ----------------------------------------------------------------------
    DISCUSS:
    ----------------------------------------------------------------------

    My entire point is more a DISCUSS-DISCUSS, for both
    draft-ietf-avtcore-monarch-19 and draft-ietf-xrblock-rtcp-xr-pdv-05.txt.
    Sorry to pick on these two drafts, but we need to have an IESG
    performance metrics discussion.
    Where does the list of performance metric definitions come from at the
    IETF?
    We have multiple sources:
    - IPPM for IP performance metrics
    - RTCP for RTP performance metrics:
       Definitions in the document themselves or potentially referencing some
    other SDOs
       Example:http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-pdv-05
        bits 014-011
                 0: MAPDV2, Clause 6.2.3.2 of [G.1020],
                 1: 2-point PDV, Clause 6.2.4 of [Y.1540].
    - PMOL: Performance Metrics at Other Layers, with
       RFC 6076 on Basic Telephony SIP End-to-End Performance Metrics
    - IPFIX will one day or the other exports performance metrics.
       I see for example
       http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03
       It's again a redefinition, and it should not be!

    My concerns are that we start to define performance metrics in different
    parts of the IETF, without consistency.

    We have defined RFC 6390 on "Guidelines for Considering New Performance
    Metric Development", which ask for specific definition
    Seehttp://tools.ietf.org/html/rfc6390#section-5.4.4

    I believe that the IETF should at least:
    - define the performance metrics in a consistent way according to
    RFC6390.
    - document those performance metrics in a single location

    So my questions are:
    - are we defining the performance metrics the right way?
    - where is this shared repository of performance metrics (at least for
    the ones created in the IETF)?
    - is the PMOL directorate (RFC 6390) used effectively?

After discussing with Dan Romascanu, we came to this conclusion

    I had a discussion with Dan Romascanu, and we settled on:
    - RFC 6390 template is required for new perf metric definition
    - RFC 6390 template is a nice-to-have when we refer to an existing perf metric
    Nice-to-have because the performance metric reference doesn't always include
    all the required information about: measurement points, measurement timing, use
    and applications, reporting model, etc... but focus only on the "Method of
    Measurement or Calculation"

I would like to have a meeting during the IETF, with the following agenda

- are we defining the performance metrics the right way?
- where is this shared repository of performance metrics (at least for
the ones created in the IETF)?
- is the PMOL directorate (RFC 6390) used effectively?
- conclusion discussed with Dan

Here is a doodle invite. Please let me know if/when you are available.
Your feedback on this mailer is also welcome.

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Mon, 22 Oct 2012 18:48:24 -0400

With doodle this time http://www.doodle.com/fz3gxri76ngzqygv
Thanks Carlos

B.
> to the conclusion that we have an issue in terms of performance 
> metrics at the IETF, and actually in the industry.
>

---

From: "Romascanu, Dan (Dan)" <[email-masked]>
Date: Wed, 24 Oct 2012 14:02:52 +0200

Hi Benoit, 

Wednesday at 3:40PM does not seem right, as it enters plenary time,
unless we believe that we shall not need more than 20 min. 

Dan

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Benoit Claise
Sent: Tuesday, October 23, 2012 12:48 AM
To: Benoit Claise
Cc: [email-masked]; Ron Bonica;
[email-masked]; [email-masked]; Wesley Eddy
Subject: Re: [PMOL] PMOL directorate: meeting at the IETF?

B.

	

	 
	My entire point is more a DISCUSS-DISCUSS, for both
	draft-ietf-avtcore-monarch-19 and
draft-ietf-xrblock-rtcp-xr-pdv-05.txt.
	Sorry to pick on these two drafts, but we need to have an IESG
	performance metrics discussion.
	Where does the list of performance metric definitions come from
at the
	IETF?
	We have multiple sources:
	- IPPM for IP performance metrics
	- RTCP for RTP performance metrics: 
	  Definitions in the document themselves or potentially
referencing some
	other SDOs
	  Example:
http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-pdv-05
	   bits 014-011
	            0: MAPDV2, Clause 6.2.3.2 of [G.1020],
	            1: 2-point PDV, Clause 6.2.4 of [Y.1540].
	- PMOL: Performance Metrics at Other Layers, with 
	  RFC 6076 on Basic Telephony SIP End-to-End Performance Metrics
	- IPFIX will one day or the other exports performance metrics.
	  I see for example
	
http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03
<http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03> 
	  It's again a redefinition, and it should not be!
	 
	My concerns are that we start to define performance metrics in
different
	parts of the IETF, without consistency.
	 
	We have defined RFC 6390 on "Guidelines for Considering New
Performance
	Metric Development", which ask for specific definition
	See http://tools.ietf.org/html/rfc6390#section-5.4.4
	 
	I believe that the IETF should at least:
	- define the performance metrics in a consistent way according
to
	RFC6390.
	- document those performance metrics in a single location
	 
	So my questions are:
	- are we defining the performance metrics the right way?
	- where is this shared repository of performance metrics (at
least for
	the ones created in the IETF)?
	- is the PMOL directorate (RFC 6390) used effectively?

	I had a discussion with Dan Romascanu, and we settled on:
	- RFC 6390 template is required for new perf metric definition
	- RFC 6390 template is a nice-to-have when we refer to an
existing perf metric
	Nice-to-have because the performance metric reference doesn't
always include
	all the required information about: measurement points,
measurement timing, use
	and applications, reporting model, etc... but focus only on the
"Method of
	Measurement or Calculation"

	I would like to have a meeting during the IETF, with the
following agenda

	- where is this shared repository of performance metrics (at
least for
	 
	Here is a doodle invite. Please let me know if/when you are
available.
	Your feedback on this mailer is also welcome.

	Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Sun, 28 Oct 2012 20:07:46 -0500

Hi Dan,

You're right. Daylight savings yesterday in Belgium + daylight saving 
next Sunday in Atlanta + a little bit of distraction of my side = my 
mistake. I was unable to remove the entry, so I marked myself 
unavailable for that slot.

All, please don't forgot to insert your availabilities.

Regards, Benoit
> *From:[email-masked] [mailto:[email-masked]] *On Behalf 
> Of *Benoit Claise
> *Sent:* Tuesday, October 23, 2012 12:48 AM
> *To:* Benoit Claise
> *Cc:* [email-masked]; Ron Bonica; 
> *Subject:* Re: [PMOL] PMOL directorate: meeting at the IETF?
>
>     draft-ietf-xrblock-rtcp-xr-pdv and draft-ietf-avtcore-monarch), I
>     came to the conclusion that we have an issue in terms of
>     performance metrics at the IETF, and actually in the industry.
>
>        http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03  <http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03>
>

---

From: Benoit Claise <[email-masked]>
Date: Tue, 30 Oct 2012 19:40:10 -0500

Based on doodle, this meeting will take place on Wed Nov 7th at 1440 - 
1540 in the IESG breakout room, to be confirmed.
I hope to see you all.

Regards, Benoit

---

From: "Aamer Akhter (aakhter)" <[email-masked]>
Date: Wed, 31 Oct 2012 15:59:20 +0000

Just as an FYI:

"  http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03<http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03>"

Explicitly does NOT define any performance metrics-just the IPFIX transport. There are references to other documents (the XRBLOCK, RFC3550, the perf-mon methodology draft, etc.) that do define the metrics. Both this draft and the perf-mon methodology draft (it's expired currently, and I need to update it), are using the templates defined RFC6390. There might be some mistakes here and there, but that is the intent anyway and I'm hopeful that further reviews will make these drafts sharper.

==

That said, I do not disagree with the premise below that the entire performance monitoring (IMHO) creation and dissemination is (frankly) a mess. I would like to propose an enforced clean separation between metrics definition (the methodology and usage etc-which should follow the templates in RFC6390) from the transport-expression of these metrics (RTCP, MIB, IPFIX, syslog whatever), which refer back to the methodology.

Unfortunately, I've got other commitments and will not be in ATL, but very interested in moving this discussion along.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Benoit Claise (bclaise)
Sent: Monday, October 22, 2012 6:29 PM
To: [email-masked]
Cc: [email-masked]; Ron Bonica; [email-masked]; Wesley Eddy
Subject: [PMOL] PMOL directorate: meeting at the IETF?

During my review of the latest AVTCORE and  XRBLOCK drafts ( draft-ietf-xrblock-rtcp-xr-pdv and draft-ietf-avtcore-monarch), I came to the conclusion that we have an issue in terms of performance metrics at the IETF, and actually in the industry.

DISCUSS:

IETF?

other SDOs

  Example: http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-pdv-05

   bits 014-011

  I see for example

  http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03<http://tools.ietf.org/html/draft-akhter-opsawg-perfmon-ipfix-03>

RFC6390.

So my questions are: