Subject: [pm-dir] Performance metrics doctors generated email

From: Benoit Claise <[email-masked]>
Date: Mon, 4 Feb 2013 12:53:18 +0100

Dear all,

This is an automatically generated email.  
It lists the IETF internet-drafts that reference the PMOL RFC 6390, as a normative or informative reference.
It also lists all the IETF internet-drafts that contain "performance metric".

Regards, Benoit

===========================================================

Normative References
--------------------
draft-ietf-xrblock-rtcp-xr-discard-rle-metrics-05 Active	
    
Informative References
----------------------
draft-ietf-xrblock-rtcp-xr-burst-gap-discard-10   In IESG processing - ID Tracker state <Publication Requested>	
draft-ietf-xrblock-rtcp-xr-burst-gap-loss-08      In IESG processing - ID Tracker state <Publication Requested>	
draft-ietf-xrblock-rtcp-xr-concsec-03             Active	
draft-ietf-xrblock-rtcp-xr-decodability-07        Active	
draft-ietf-xrblock-rtcp-xr-discard-11             Active	
draft-ietf-xrblock-rtcp-xr-jb-07                  Active	
draft-ietf-xrblock-rtcp-xr-loss-conceal-04        Active	
draft-ietf-xrblock-rtcp-xr-qoe-03                 Active	

drafts containing performance metric
------------------------------------
draft-ietf-alto-deployments-05                    Active	
draft-ietf-alto-protocol-13                       In IESG processing - ID Tracker state <AD Evaluation::Revised ID Needed>	
draft-ietf-ippm-rate-problem-02                   Active	
draft-ietf-ippm-testplan-rfc2680-01               Active	
draft-ietf-karp-threats-reqs-07                   In IESG processing - ID Tracker state <RFC Ed Queue>	
draft-ietf-manet-olsrv2-mib-05                    Active	
draft-ietf-manet-smf-mib-06                       Active	
draft-ietf-mmusic-media-loopback-27               In IESG processing - ID Tracker state <RFC Ed Queue>	
draft-ietf-nvo3-framework-01                      Active	
draft-ietf-opsawg-oam-overview-08                 In IESG processing - ID Tracker state <Waiting for AD Go-Ahead::Revised ID Needed>	
draft-ietf-rtcweb-rtp-usage-05                    Active	
draft-ietf-rtgwg-cl-framework-02                  Active	
draft-ietf-xrblock-rtcp-xr-burst-gap-discard-10   In IESG processing - ID Tracker state <Publication Requested>	
draft-ietf-xrblock-rtcp-xr-burst-gap-loss-08      In IESG processing - ID Tracker state <Publication Requested>	
draft-ietf-xrblock-rtcp-xr-concsec-03             Active	
draft-ietf-xrblock-rtcp-xr-decodability-07        Active	
draft-ietf-xrblock-rtcp-xr-discard-11             Active	
draft-ietf-xrblock-rtcp-xr-discard-rle-metrics-05 Active	
draft-ietf-xrblock-rtcp-xr-jb-07                  Active	
draft-ietf-xrblock-rtcp-xr-loss-conceal-04        Active	
draft-ietf-xrblock-rtcp-xr-qoe-03                 Active	
draft-ietf-xrblock-rtcp-xr-summary-stat-06        In IESG processing - ID Tracker state <Waiting for AD Go-Ahead>	
draft-ietf-xrblock-rtcp-xr-synchronization-02     Active

---

From: Benoit Claise <[email-masked]>
Date: Mon, 04 Feb 2013 13:06:56 +0100

Dear all,

This is a new version of the script, provided by Alan Dekok. Many thanks 
Alan.
The important improvement is that we also list the drafts that contain a 
specific keyword.
In this case, "performance metric". We could add some more...
With this new script, we can then engage sooner with the people who 
don't know about this directorate and RFC 6390.

Potential small optimization for the script: remove the drafts that 
contain "performance metric" and that already reference RFC 6390

Regards, Benoit
> _______________________________________________
> pm-dir mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pm-dir
>
>

---

From: Paul Aitken <[email-masked]>
Date: Mon, 04 Feb 2013 12:33:19 +0000

Benoit, All,

How is pm-dir tracking these drafts? ie, how do we know whether someone 
is currently reviewing, or has already reviewed, each of these drafts?

How are reviewers selected? By relevance, skill, luck, or misfortune?

Also, it'd be useful to know whether the previous version was already 
reviewed, and what the feedback was:

     * issues were raised, recommendations were made, changes were 
expected -> some re-review should be done

     * no issues were raised -> a quick review of the delta may be 
sufficient, to check for any new issues

     * previous version wasn't reviewed -> a thorough review may be 
required.

I'm thinking it'd be useful to have a pm-dir wiki listing each of the 
drafts below. eg, one page per draft, with one section per version, 
containing reviewer comments pertinent to that version, even if the 
comments simply say, "reviewed, no issues".

Hopefully I missed something, and most (if not all?) of this is already 
in place? :-)

Thanks,
P.

---

From: Benoit Claise <[email-masked]>
Date: Mon, 04 Feb 2013 13:36:53 +0100

Hi Paul,

Yes, all these questions are important.
Yes a WIKI is the right solution IMHO.

Regards, Benoit
> Benoit, All,
>
> How is pm-dir tracking these drafts? ie, how do we know whether 
> someone is currently reviewing, or has already reviewed, each of these 
> drafts?
>
> Hopefully I missed something, and most (if not all?) of this is 
> already in place? :-)
>
> Thanks,
> P.
>
>
>> draft-ietf-xrblock-rtcp-xr-burst-gap-discard-10   In IESG processing 
>> - ID Tracker state <Publication Requested>
>> draft-ietf-xrblock-rtcp-xr-burst-gap-loss-08      In IESG processing 
>> - ID Tracker state <Publication Requested>
>> draft-ietf-alto-protocol-13                       In IESG processing 
>> - ID Tracker state <AD Evaluation::Revised ID Needed>
>> draft-ietf-opsawg-oam-overview-08                 In IESG processing 
>> - ID Tracker state <Waiting for AD Go-Ahead::Revised ID Needed>
>> draft-ietf-xrblock-rtcp-xr-burst-gap-discard-10   In IESG processing 
>> - ID Tracker state <Publication Requested>
>> draft-ietf-xrblock-rtcp-xr-burst-gap-loss-08      In IESG processing 
>> - ID Tracker state <Publication Requested>
>> draft-ietf-xrblock-rtcp-xr-summary-stat-06        In IESG processing 
>> - ID Tracker state <Waiting for AD Go-Ahead>

---

From: Paul Aitken <[email-masked]>
Date: Mon, 04 Feb 2013 12:49:55 +0000

Benoit,

> Yes a WIKI is the right solution IMHO.

Great. Where can it be hosted?

eg, is http://trac.tools.ietf.org/wg/pmol/trac/wiki a good starting 
place? Or is there a pm-dir page?

P.

>> someone is currently reviewing, or has already reviewed, each of 
>> these drafts?
>>

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Mon, 4 Feb 2013 08:22:54 -0500

Hi Paul,

Our generally accepted mode of operation is in the message
appended below.  We need to make a few changes now that the
script is finding more drafts than there are Directorate members.

We could choose to expand our membership, that's a question I 
asked the Directorate a few weeks ago, but with no feedback there
was no action.
	When the new list was announced, several people tried to join.
	We should probably have a call for new members soon, 
	if the current Directorate members agree.  Also, if anyone would 
	like to retire from the Performance Directorate (for example,
	if you didn't find time to perform reviews in 2012, then
	you might consider the question in that light), please let me know.

I agree that we need a more accessible and now, more dynamic means of 
organization.  A Wiki will work, as long as everyone has access to it
and can update draft status as they complete reviews, etc.

Once we handle the initial rush of drafts from the 
script, I would suggest that the script keep last week's list and
highlight the new arrivals to facilitate assignment, rather than 
leave this step for manual checking.

regards,
Al
PM Dir admin

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
At 09:47 AM 1/16/2012, Al Morton wrote:
As this new year goes into full swing, I'd like to offer some
thoughts on how we might operate as a Directorate and
hear from others.

As our web page says: "RFC 6390 is used as a reference for the technical 
and process issues." Have your copy handy when reviewing a draft.
A lot of interactions may be streamlined by pointing the authors to 6390,
and mentioning that it is a BCP.

I feel that our reviews should be *early* in the draft development
process, and that no performance-related draft should start IETF Last Call 
without previous review against 6390, unless it truly slipped by us
and other folks who know about the Directorate.  

There are two ways we can collect drafts for review:

- WG Chairs and ADs can/should suggest drafts to us

- Directorate Members will see drafts, hear about drafts with performance
work, etc.  We are the eyes and ears of the Directorate. We all have friends
working in other areas of IETF, let them know what we're looking for.

I suggest to split the early review into two categories:

1. Quick Scan - If you find a draft, do a quick scan or suggest that
others take a look with a message to pmol-list.  Finding a draft does
not mean that you will be designated to do a review (see below).

2. WG Candidate/Chartered Item - When a draft appears to have enough value
and interest to become a WG item, or the WG Chairs involved recommend a 
Performance Metrics Directorate review on their own, then we do a more
complete review against 6390.  

If these categories are useful, then I'd like to suggest that *two* 
reviewers are assigned to WG Candidate/Chartered drafts and they follow
the draft after early review. This may mean looking at the draft again
in WG or IETF Last Call, assuming it goes that far. I suggest 2 reviewers
while we're all new at this, and to help ensure that one is still around
years later when Last Calls are done.

All drafts identified, quick scans, and reviews should be sent to the
pmol-list and the authors (and possibly the relevant WG chairs)

let us all know what you think,
Al

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Paul Aitken
> Sent: Monday, February 04, 2013 7:50 AM
> To: Benoit Claise
> Cc: [email-masked]
> Subject: Re: [pm-dir] Performance metrics doctors generated email
> 
> Benoit,
>

---

From: Benoit Claise <[email-masked]>
Date: Wed, 13 Feb 2013 14:01:47 +0100

Al, All,
> and can update draft status as they complete reviews, etc.
Agreed.
> leave this step for manual checking.
That would nice, but we would need a kind of database to keep track of 
the already assigned drafts.
It starts to be a more complex script...

Note: Alan (in the copy list) is improving the script so that the drafts 
that maps a keyword (performance metric in this case) and that have a 
reference to RFC 6390 doesn't appear twice.

Regards, Benoit

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Wed, 13 Feb 2013 11:02:29 -0500

It doesn't have to keep track of assignments, just last week's list.

IOW, highlight the drafts that are new this week, an automated "uniq" list

at the top of the mail...

great, that's a good start.

I'm not requesting any volunteers right now,

with the draft deadlines looming...

Al

From: Benoit Claise [mailto:[email-masked]]
Sent: Wednesday, February 13, 2013 8:03 AM
To: MORTON JR., ALFRED (AL)
Subject: Fwd: Re: [pm-dir] Performance metrics doctors generated email

Hi Al,

Can you please take care of setting up a WIKI.
Thanks in advance.

Regards, Benoit

-------- Original Message --------
Subject:

Re: [pm-dir] Performance metrics doctors generated email

Date:

Wed, 13 Feb 2013 14:01:47 +0100

From:

Benoit Claise <[email-masked]><mailto:[email-masked]>

To:

MORTON JR., ALFRED (AL) <[email-masked]><mailto:[email-masked]>

CC:

Paul Aitken <[email-masked]><mailto:[email-masked]>, "[email-masked]"<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, Alan DeKok <[email-masked]><mailto:[email-masked]>

Al, All,

Agreed.

Regards, Benoit

>> From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf

>> Cc: [email-masked]<mailto:[email-masked]>

>>>>> It lists the IETF internet-drafts that reference the PMOL RFC 6390,

>>>>> as a normative or informative reference.

>>>>> It also lists all the IETF internet-drafts that contain "performance

>>>>> metric".

>>>>> draft-ietf-karp-threats-reqs-07                   In IESG processing

>>>>> - ID Tracker state <RFC Ed Queue>

>>>>> draft-ietf-mmusic-media-loopback-27               In IESG processing

>>>>> - ID Tracker state <RFC Ed Queue>

>>>>> [email-masked]<mailto:[email-masked]>

>>>> [email-masked]<mailto:[email-masked]>

>> [email-masked]<mailto:[email-masked]>

---

From: Paul Aitken <[email-masked]>
Date: Fri, 15 Feb 2013 11:11:32 +0000

Benoit,

How often is the script run? I haven't seen any more postings these last 
10 days.

If it's run regularly - eg, weekly or nightly - then it'd be useful as 
Al said, to send a delta from the previous run so we can easily see what 
changed.

I saved the current(?) list at http://trac.tools.ietf.org/wg/pmol/trac/wiki
You could imagine additional columns, eg reviewer, comments, status, so 
the status and history can be seen at a glance.

Is this useful?

P.

On 04/02/13 12:06, Benoit Claise wrote:
> Dear all,
>
> This is a new version of the script, provided by Alan Dekok. Many 
> thanks Alan.
> The important improvement is that we also list the drafts that contain 
> a specific keyword.

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Fri, 15 Feb 2013 08:20:17 -0500

Hi Paul,

Thanks for continuing the discussion on tracking
draft review assignments and status.  I've been discussing 
this off-list with Benoit. Let me put my position on
the table here.

I'd prefer a solution where:
 - everyone can view the page
 - directorate reviewers can edit the page
 - all editing is WYSIWYG
 - no collisions between editors
  (if two people are editing, they can see each other and 
   pressing save does not overwrite a nested editing session)

Unfortunately, the last two points are beyond Trac's capability.
I think that I could create a doc on Google Drive that
would meet all the above criteria.
  
But, all Directorate members who are active/performing reviews
would need to have an account on Google and be willing to use
it for this purpose. I suspect that many of us already have
a gmail account.

If there is another solution that meets all the points above,
we could consider that as well.

Let's hear from all Directorate members on this, it's a cross-road
that will affect how we work together in the future.

regards,
Al

> Sent: Friday, February 15, 2013 6:12 AM
> To: Benoit Claise
> Cc: [email-masked]; Alan DeKok
> Subject: Re: [pm-dir] Performance metrics doctors generated email - new
> version.
> 
> Benoit,
> 
> I saved the current(?) list at
> http://trac.tools.ietf.org/wg/pmol/trac/wiki

---

From: Benoit Claise <[email-masked]>
Date: Tue, 19 Feb 2013 16:10:11 +0100

Hi Al,
> would meet all the above criteria.
My experience with google doc is not great, but I'm ready to give it a 
try...

Regards, Benoit

---

From: Benoit Claise <[email-masked]>
Date: Tue, 19 Feb 2013 16:09:09 +0100

Paul,
> Benoit,
>
> How often is the script run? 
Weekly, on Sundays
> I haven't seen any more postings these last 10 days.
One was sent on Feb 10th

> Al said, to send a delta from the previous run so we can easily see 
> what changed.
>
> You could imagine additional columns, eg reviewer, comments, status, 
> so the status and history can be seen at a glance.
>
> Is this useful?
Yes, if we keep updating it.

Regards, Benoit
>> The important improvement is that we also list the drafts that 
>> contain a specific keyword.

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Wed, 20 Feb 2013 09:36:11 -0500

Ok Benoit, 
I have a version of the doc going, but probably won't finish
till after the revised deadline expires...

Before going further, it would be useful to know if anyone
in the directorate objects to using or establishing a gmail
account in order to register their review assignments and status.

regards,
Al

> Of Benoit Claise
> Sent: Tuesday, February 19, 2013 10:10 AM
> Cc: [email-masked]; Paul Aitken; Alan DeKok
...

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 20 Feb 2013 14:59:35 +0000

> account in order to register their review assignments and status.

No objection. I think it's a great solution.

Thanks,

-- Carlos.

---

From: Paul Aitken <[email-masked]>
Date: Wed, 20 Feb 2013 15:46:51 +0000

> No objection. I think it's a great solution.

+1

P.

---

From: Yaakov Stein <[email-masked]>
Date: Thu, 21 Feb 2013 07:06:39 +0000

Sorry for perhaps not catching on, 
but are you suggesting setting up a single account for all of the directorate
and using "Google Tasks" to aggregate the review statuses ?

While this could work, I think there are perhaps better options.

The simplest would be just maintaining a table with draft-name + reviewer + status + date
on a wiki.

As a reviewer for several academic journals I know they have tools
(much too advanced for our present needs)
that select the most appropriate reviewer(s), send them emails,
receive their acceptance/rejection of the task, 
send email reminders when the due date approaches, 
input the review, etc.

Y(J)S 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Carlos Pignataro (cpignata)
Sent: Wednesday, February 20, 2013 17:00
To: MORTON JR., ALFRED (AL)
Cc: Benoit Claise (bclaise); [email-masked]; Paul Aitken (paitken); Alan DeKok
Subject: Re: [pm-dir] Performance metrics doctors generated email - new version.

Thanks,

-- Carlos.

---

From: "MORTON JR., ALFRED (AL)" <[email-masked]>
Date: Thu, 21 Feb 2013 07:36:22 -0500

Not Google Tasks (which I haven't investigated),
just a Google doc with the table:
draft-name + reviewer + status + date

Which all reviewers can edit WYSIWYG
I'll own the doc and share it with the everyone in the 
directorate (who actually does reviews :-) ).

And, that doc won't be ready till after the draft deadline!

Al

> From: Yaakov Stein [mailto:[email-masked]]
> Sent: Thursday, February 21, 2013 2:07 AM
> Subject: RE: [pm-dir] Performance metrics doctors generated email - new
> version.
> 
> Of Carlos Pignataro (cpignata)