Subject: [ippm] Proposed charter updates for IPPM

From: Brian Trammell <[email-masked]>
Date: Sun, 9 Jul 2017 17:23:57 +0200

Greetings, all,

Bill and I met briefly last week to discuss a proposed change to the charter, as a way to resolve the rough consensus to adopt the IOAM work within IPPM with concerns that have been raised that that work doesn't really fit on the charter. We arrived at a minimal suggested set of edits. Though inspired by IOAM, these aren't so much focused on bringing IOAM on charter as they are making the charter reflect the direction work in IPPM has been going over the five years since we last updated the charter.

We propose only to touch three paragraphs, as below:

---

OLD Paragraph 1:

The IP Performance Metrics (IPPM) Working Group develops and maintains standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services and applications running over transport layer protocols (e.g. TCP, UDP) over IP. Specifying network or lower layer OAM mechanisms is out of scope of the IPPM charter. It also develops and maintains protocols for the measurement of these metrics. These metrics are designed such that they can be used by network operators, end users, or independent testing groups. Metrics developed by the IPPM WG are intended to provide unbiased quantitative performance measurements and not a value judgement.

NEW Paragraph 1:

The IP Performance Measurement (IPPM) Working Group develops and maintains standard metrics that can be applied to the quality, performance, and reliability of Internet data delivery services and applications running over transport layer protocols (e.g. TCP, UDP) over IP. It also develops and maintains methodologies and protocols for the measurement of these metrics. These metrics are designed such that they can be used by network operators, end users, or independent testing groups. Metrics developed by the IPPM WG are intended to provide unbiased quantitative performance measurements and not a value judgement.

Two basic edits here:

(1) Rename working group from IP Performance Metrics to IP Performance Measurement, and add "methodologies" to the set of things we do. In our opinion, this reflects work that has been happening on charter for a while: MBM, PDM, and alt-mark are all methodology work as opposed to metric work.

(2) Remove the sentence "Specifying network or lower layer OAM mechanisms is out of scope of the IPPM charter". In my personal opinion, this isn't really necessary to bring the IOAM work on, since though IOAM can be carried on lower-layer headers, it doesn't really specify an OAM mechanism as I understand that term to be defined in the routing area. However, this restriction on the IPPM doesn't appear to do anything useful beyond reinforcing an artificial silo dividing data-plane and control-plane measurement, make it difficult for us to work together with other (e.g. INT or RTG area) working groups on performance measurement, and cause confusion as to whether we can do a thing called "IOAM" in IPPM.

---

OLD Paragraph 6:

The WG has produced protocols for communication among test equipment to enable the measurement of the one- and two-way metrics (OWAMP and TWAMP respectively). These protocols will be advanced along the standards track. The work of the WG will take into account the suitability of measurements for automation, in order to support large-scale measurement efforts. This may result in further developments in protocols such as OWAMP and TWAMP. Agreement about the definitions of metrics and methods of measurement enables accurate, reproducible, and equivalent results across different implementations. To this end, the WG will define and maintain a registry of metric definitions. The WG encourages work which assesses the comparability of measurements of IPPM metrics with metrics developed elsewhere.

NEW Paragraph 6:

The WG has produced protocols for communication among test equipment to enable the measurement of the one- and two-way metrics (OWAMP and TWAMP respectively). These protocols will be advanced along the standards track. The work of the WG will take into account the suitability of measurements for automation, in order to support large-scale measurement efforts. This may result in further developments in protocols such as OWAMP and TWAMP. Agreement about the definitions of metrics and methods of measurement enables accurate, reproducible, and equivalent results across different implementations. To this end, the WG defines and maintains a registry of metric definitions. The WG encourages work which assesses the comparability of measurements of IPPM metrics with metrics developed elsewhere.

This is a purely editorial change: move definition and maintenance of registry to present tense, since we're doing it.

---

OLD Paragraph 7:

The WG also encourages work which improves the availability of information about the context in which measurements were taken.

NEW Paragraph 7:

The WG also encourages work which improves the availability of information about the context in which measurements were taken, for example, measurement implementation information, conditions on the networks on which measurements are taken, or information about the data-plane topology of the measured network.

This is an arguably editorial change. It makes it clear that work targeted at IPPM that focuses on measurement on the spatial characteristics of paths as context for metrics (e.g. parts of IOAM, but also work like draft-amf-ippm-route, which attempts to answer the question "what about traceroute" applied to the registry).

---

We additionally propose to strike the (out of date) near term milestones from the charter text; milestones are separately managed.

Beyond these updates, we propose to leave the charter unchanged. We've got some time to have this discussion on the agenda in Prague, but in the meantime please send your thoughts on this proposal to the mailing list.

Many thanks, best regards,

Brian (for the chairs)

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 9 Jul 2017 17:22:57 +0000

Hi Brian and Bill,

Thanks for preparing a new charter proposal and organizing
this message to highlight the changes. I agree with your
proposals.

I have one further proposal for the last sentence of
paragraph 1.

OLD (and your NEW):
Metrics developed by the IPPM WG are intended to provide unbiased 
quantitative performance measurements and not a value judgement.

I propose to delete/replace the last phrase, which I think has been present
since the very first charter. IIRC, the concern was that contributions 
might attempt to establish service acceptance thresholds, or other
numerical performance requirements/objectives using the metrics.
The work of setting numerical objectives proceeded elsewhere
(ITU-T Rec Y.1541), and the Earth continued to rotate.

If we want to ensure the status quo, we could say:
s/and not a value judgement/but numerical thresholds for the metrics are out-of-scope/

I also suggest this change with MBM in mind, where we have
testing outcomes of Pass/Fail/Inconclusive (PFI). Of course, the user
supplies the target rate, RTT, and/or loss ratio numerical values,
not the working group.

So we could express an explicit restriction that does not hamper MBM PFI:

NEW^2:
Metrics developed by the IPPM WG are intended to provide unbiased 
quantitative performance measurements, but numerical thresholds for 
the metrics are out-of-scope.

Or, we could remove the phrase, and refer any contributions on 
numerical thresholds to ITU-T SG 12, Question 17, to avoid 
overlapping work with another SDO (coordination with this SDO
should also be mentioned somewhere).

NEW^3:
Metrics developed by the IPPM WG are intended to provide unbiased 
quantitative performance measurements. 

thanks for considering this proposal, and regards,
Al

> -----Original Message-----
> From: ippm [mailto:[email-masked]] On Behalf Of Brian Trammell
> Sent: Sunday, July 09, 2017 11:24 AM
> To: IETF IPPM WG
> Subject: [ippm] Proposed charter updates for IPPM
> 
> Greetings, all,
> 
> Bill and I met briefly last week to discuss a proposed change to the
> charter, as a way to resolve the rough consensus to adopt the IOAM work
> within IPPM with concerns that have been raised that that work doesn't
> really fit on the charter. We arrived at a minimal suggested set of
> edits. Though inspired by IOAM, these aren't so much focused on bringing
> IOAM on charter as they are making the charter reflect the direction
> work in IPPM has been going over the five years since we last updated
> the charter.
> 
> The IP Performance Metrics (IPPM) Working Group develops and maintains
> standard metrics that can be applied to the quality, performance, and
> reliability of Internet data delivery services and applications running
> over transport layer protocols (e.g. TCP, UDP) over IP. Specifying
> network or lower layer OAM mechanisms is out of scope of the IPPM
> charter. It also develops and maintains protocols for the measurement of
> these metrics. These metrics are designed such that they can be used by
> network operators, end users, or independent testing groups. Metrics
> developed by the IPPM WG are intended to provide unbiased quantitative
> performance measurements and not a value judgement.
> 
> NEW Paragraph 1:
> 
> The IP Performance Measurement (IPPM) Working Group develops and
> maintains standard metrics that can be applied to the quality,
> performance, and reliability of Internet data delivery services and
> applications running over transport layer protocols (e.g. TCP, UDP) over
> IP. It also develops and maintains methodologies and protocols for the
> measurement of these metrics. These metrics are designed such that they
> can be used by network operators, end users, or independent testing
> groups. Metrics developed by the IPPM WG are intended to provide
> unbiased quantitative performance measurements and not a value
> judgement.
> 
> 
> Two basic edits here:
> 
> (1) Rename working group from IP Performance Metrics to IP Performance
> Measurement, and add "methodologies" to the set of things we do. In our
> opinion, this reflects work that has been happening on charter for a
> while: MBM, PDM, and alt-mark are all methodology work as opposed to
> metric work.
> 
> (2) Remove the sentence "Specifying network or lower layer OAM
> mechanisms is out of scope of the IPPM charter". In my personal opinion,
> this isn't really necessary to bring the IOAM work on, since though IOAM
> can be carried on lower-layer headers, it doesn't really specify an OAM
> mechanism as I understand that term to be defined in the routing area.
> However, this restriction on the IPPM doesn't appear to do anything
> useful beyond reinforcing an artificial silo dividing data-plane and
> control-plane measurement, make it difficult for us to work together
> with other (e.g. INT or RTG area) working groups on performance
> measurement, and cause confusion as to whether we can do a thing called
> "IOAM" in IPPM.
> 
> ---
> 
> OLD Paragraph 6:
> 
> The WG has produced protocols for communication among test equipment to
> enable the measurement of the one- and two-way metrics (OWAMP and TWAMP
> respectively). These protocols will be advanced along the standards
> track. The work of the WG will take into account the suitability of
> measurements for automation, in order to support large-scale measurement
> efforts. This may result in further developments in protocols such as
> OWAMP and TWAMP. Agreement about the definitions of metrics and methods
> of measurement enables accurate, reproducible, and equivalent results
> across different implementations. To this end, the WG will define and
> maintain a registry of metric definitions. The WG encourages work which
> assesses the comparability of measurements of IPPM metrics with metrics
> developed elsewhere.
> 
> NEW Paragraph 6:
> 
> The WG has produced protocols for communication among test equipment to
> enable the measurement of the one- and two-way metrics (OWAMP and TWAMP
> respectively). These protocols will be advanced along the standards
> track. The work of the WG will take into account the suitability of
> measurements for automation, in order to support large-scale measurement
> efforts. This may result in further developments in protocols such as
> OWAMP and TWAMP. Agreement about the definitions of metrics and methods
> of measurement enables accurate, reproducible, and equivalent results
> across different implementations. To this end, the WG defines and
> maintains a registry of metric definitions. The WG encourages work which
> assesses the comparability of measurements of IPPM metrics with metrics
> developed elsewhere.
> 
> 
> This is a purely editorial change: move definition and maintenance of
> registry to present tense, since we're doing it.
> 
> ---
> 
> OLD Paragraph 7:
> 
> The WG also encourages work which improves the availability of
> information about the context in which measurements were taken.
> 
> NEW Paragraph 7:
> 
> The WG also encourages work which improves the availability of
> information about the context in which measurements were taken, for
> example, measurement implementation information, conditions on the
> networks on which measurements are taken, or information about the data-
> plane topology of the measured network.
> 
> 
> This is an arguably editorial change. It makes it clear that work
> targeted at IPPM that focuses on measurement on the spatial
> characteristics of paths as context for metrics (e.g. parts of IOAM, but
> also work like draft-amf-ippm-route, which attempts to answer the
> question "what about traceroute" applied to the registry).
> 
> ---
> 
> We additionally propose to strike the (out of date) near term milestones
> from the charter text; milestones are separately managed.
> 
> 
> Beyond these updates, we propose to leave the charter unchanged. We've
> got some time to have this discussion on the agenda in Prague, but in
> the meantime please send your thoughts on this proposal to the mailing
> list.
> 
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Sun, 9 Jul 2017 22:02:09 +0000

Thank you Brian for proactively bringing this discussion on the list before Prague.

All the proposed changes below bring the charter (i.e., the contract between the WG and the IETF) closer to reality and remove potential ambiguities and sources of misinterpretation. I believe that all these edits serve the higher purpose of charter hygiene and forward-looking polishing, and are not narrowly written solely with iOAM in mind.

As such, I very much support this proposal.

Related, and likely a question for Benoit, but curious about the PM-Directorate and whether this is OBE (I've not seen traffic for a couple years):
https://www.ietf.org/iesg/directorate/performance-metrics.html

I also agree with Al and support the removal (instead of rewrite) of the extra sentence.

Please find a couple small follow-ups inline.

Sent from my iPad

On Jul 9, 2017, at 1:23 PM, MORTON, ALFRED C (AL) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Brian and Bill,

OLD (and your NEW):

NEW^2:

NEW^3:

+1 to this one.

-----Original Message-----
From: ippm [mailto:[email-masked]] On Behalf Of Brian Trammell
Sent: Sunday, July 09, 2017 11:24 AM
To: IETF IPPM WG
Subject: [ippm] Proposed charter updates for IPPM

Greetings, all,

---

OLD Paragraph 1:

NEW Paragraph 1:

Two basic edits here:

Measurement,

This is a very important change. Thanks for raising it.

and add "methodologies" to the set of things we do. In our
opinion, this reflects work that has been happening on charter for a
while: MBM, PDM, and alt-mark are all methodology work as opposed to
metric work.

I agree.

Editorial question: where it says: "These metrics are designed such that..." should that be: "These metrics, protocols, and methodologies are designed such that..."?

---

OLD Paragraph 6:

NEW Paragraph 6:

---

OLD Paragraph 7:

NEW Paragraph 7:

Good clarification.

The "for example" speaks to these, but I would add that context is not limited to this enumeration.

---

Thanks!!!

Carlos.

Brian (for the chairs)

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 11 Jul 2017 14:03:59 +0200

hi Al,

Briefly, inline below.

> On 09 Jul 2017, at 19:22, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Hi Brian and Bill,
> 

Thanks for this history... this change (NEW^3) seems to make sense to me, and I'd propose we do it as well.

Cheers,

Brian

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Tue, 11 Jul 2017 14:17:03 +0200

hi Carlos,

followups to followups inline; I've noted all your points inline to which I don't reply directly...

> On 10 Jul 2017, at 00:02, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 

I believe the directorate still exists, and the guidelines in 6390 remain (though our work on the registry has augmented them), so I think the charter should continue to mention it. Though, on second read, I  I think we could also strike the " It is possible that new measurement protocols will be needed to support new metrics; if this is the case, the working group will be rechartered to develop these protocols" since this proposal, to some extent, is that recharter.

<snip>

> Editorial question: where it says: "These metrics are designed such that..." should that be: "These metrics, protocols, and methodologies are designed such that..."?

Yes, though that makes the sentence a little unwieldy. Let's take the input that this clause should not be limited to metrics only and wordsmith the paragraph later.

<snip>

> The "for example" speaks to these, but I would add that context is not limited to this enumeration.

so, "for example (but not limited to)..."? Yep, seems good to me.

Cheers,

Brian

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Tue, 11 Jul 2017 12:20:24 +0000

Hi Brian,

Thanks for the responses — Ack to all of your comments below.

— Carlos.

On Jul 11, 2017, at 8:17 AM, Brian Trammell (IETF) <[email-masked]<mailto:[email-masked]>> wrote:

hi Carlos,

On 10 Jul 2017, at 00:02, Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:

<snip>

<snip>

OLD Paragraph 7:

NEW Paragraph 7:

Good clarification.

Cheers,

Brian

—
Carlos Pignataro, [email-masked]<mailto:[email-masked]>

“Sometimes I use big words that I do not fully understand, to make myself sound more photosynthesis."

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Sun, 16 Jul 2017 13:02:01 +0200

Greetings, all,

Following up on a discussion at Saturday's Applied Networking Research Workshop, it also seems to me that IPPM could provide a useful home for the standardization of measurement result information and data models. Since one of the main gaps in practice here is metadata, this work would hang nicely off of Paragraph 7, and further propose to add one more sentence to this paragraph:

OLD Paragraph 7:

NEW NEW Paragraph 7:

The WG also encourages work which improves the availability of information about the context in which measurements were taken, for example (but not limited to) measurement implementation information, conditions on the network(s) on which measurements are taken, and/or information about the data-plane topology of these network(s). It may foster this work by defining information and data models for storage and dissemination of measurement data.

Cheers,

Brian

> On 09 Jul 2017, at 17:23, Brian Trammell <[email-masked]> wrote:
> 
> Greetings, all,
> 
> [email-masked]

---

From: Joachim Fabini <[email-masked]>
Date: Mon, 17 Jul 2017 08:32:17 +0200

Hi Brian, ippm,

I fully support the charter addition and work that you propose. Main
focus of the ippm WG is on defining metrics and methodologies to enable
accurate measurements. Measurement results, as well as their
interpretation, depend to a large extent on parameters that ippm defines
and uses. Some metric and methodology definitions even recommend/mandate
the documentation of measurement parameters, but there is no guidance on
how to do this. So imo it's straight-forward for ippm to define data
models, too, that support the dissemination of data and the formal
description of the acquisition process (i.e, context).

Detailed knowledge on how exactly a specific dataset was acquired, as
well as a recommended data format helps all data users, i.e., people who
later on post-process the data. I'd go even further to mandate any
dataset that is released to the community/public to be accompanied by a
formal (ideally machine-readable?) detailed description. Ippm can
support by developing and recommending standard formats/frameworks for
disseminating measurement data.

Two minor issues wrt wording:
1. One could emphasize the need for metadata even more ("It may foster
this work by defining information and data models for storage and
dissemination of measurement data _and_context_").
2. The terms "measurement methodology" and "measurement parameters" are
hidden in your wording inside "measurement implementation". I'd mention
these two explicitly.

thanks,
Joachim