Subject: [apps-discuss] Responding to Applications Area Directorate reviews

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Apr 2012 02:25:36 -0700

Hello,

I'll comment on the template used for Applications Area Directorate 
(AppsDir) reviews as it may help authors to respond to the reviews.

   "Please resolve these comments along with any other Last Call comments
    you may receive. Please wait for direction from your document shepherd
    or AD before posting a new version of the draft."

In plain English, this means that the comments in the review does not 
bear more weight than a comment submitted by an individual during a Last Call.

The Applications Area Directors may or may not agree with the 
comments of the AppsDir reviewer.  The  Applications Area Directorate 
does not have any veto power on a draft.  It is suggested that the 
author(s) asks the document shepherd (RFC 4858) or the Area Director 
who sponsored the draft for guidance about IETF process, changes that 
should be made, whether to post a new revision, etc.

A review generally list issues as major or minor.  I'll use a review 
performed by Ted Hardie as an example ( 
http://www.ietf.org/mail-archive/web/apps-discuss/current/msg04847.html ).

The reviewer explained why he considered some issues as major.  The 
author discussed about the issues with the reviewer ( 
http://www.ietf.org/mail-archive/web/apps-discuss/current/msg04859.html 
).  There isn't a clear-cut rule on what is a major or minor 
issue.  It's up to the reviewer to make a judgement call.  It is up 
to the author(s), or working group, to decide about whether these 
issues should be resolved and how to do so.

Some AppsDir reviews may list nits.  These nits are generally 
editorial issues.  For example:

  "I personally found the Introduction somewhat hard to parse"

That doesn't mean that the author has to make changes to the 
Introduction section.  It's an editorial decision.  It's up to the 
authors to determine whether the text should be changed so that the 
reader can understand it.

Regards,
S. Moonesamy

---

From: Dave Crocker <[email-masked]>
Date: Wed, 11 Apr 2012 08:39:33 -0700

On 4/11/2012 2:25 AM, S Moonesamy wrote:
> 
> In plain English, this means that the comments in the review does not bear more
> weight than a comment submitted by an individual during a Last Call.

SM,

It might be worth adding simple language like this to our material, including
the apps area review team web page AND the template.

The essence of directorate-based reviews is perspective, not authority.

In our case, it supplies a review from someone with an applications-oriented
perspective.  What should matter, therefore, are the merits of the review and
not the body "authorizing" it.

d/
-- 

  Dave Crocker
  Brandenburg InternetWorking
  bbiw.net

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Apr 2012 16:27:44 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Crocker
> Sent: Wednesday, April 11, 2012 8:40 AM
> To: S Moonesamy
> Cc: [email-masked]; [email-masked]
> Subject: Re: [apps-discuss] [appsdir] Responding to Applications Area Directorate reviews
> 
> It might be worth adding simple language like this to our material,
> including the apps area review team web page AND the template.
> 
> In our case, it supplies a review from someone with an applications-
> oriented perspective.  What should matter, therefore, are the merits of
> the review and not the body "authorizing" it.

I think that's a laudable goal, but there are constant reminders that directorate reviews at least appear to carry more weight since their association with authority (the ADs in particular) is difficult to obscure.  Often I see DISCUSSes posted based solely on the fact that a review team or directorate did a review but it was not addressed.  Further, those reviews are referenced in the DISCUSS as (for example) "the review done by Murray Kucherawy of the Applications Directorate", rather than just by name.

It's true that we're volunteers, but being appointed to the directorate does mean someone thinks you're an expert at something.

This paragraph of SM's message captures it better, I think, though I would change the word "veto" to something else:

The Applications Area Directors may or may not agree with the comments of the AppsDir reviewer.  The  Applications Area Directorate does not have any veto power on a draft.  It is suggested that the
author(s) asks the document shepherd (RFC 4858) or the Area Director who sponsored the draft for guidance about IETF process, changes that should be made, whether to post a new revision, etc.

-MSK

---

From: Claudio Allocchio <[email-masked]>
Date: Wed, 11 Apr 2012 22:52:47 +0200 (CEST)

> not the body "authorizing" it.

+1

I write (in private) exactly the same idea this morning, and as I seem not 
to be alone with it, it might be a good idea!

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Apr 2012 13:47:03 -0700

>the apps area review team web page AND the template.

I'll add it the AppsDir Wiki.

>not the body "authorizing" it.

Yes.

At 09:27 11-04-2012, Murray S. Kucherawy wrote:
>I think that's a laudable goal, but there are constant reminders 
>that directorate reviews at least appear to carry more weight since 
>their association with authority (the ADs in

The guidance at 
http://trac.tools.ietf.org/area/app/trac/wiki/AppsDirReview says:

  "What to Look For

   The most important item is to give the Apps ADs a sense of how important it
   is that they pay attention to the document."

That applies for reviews performed by the Application Area Directorate.

>  particular) is difficult to obscure.  Often I see DISCUSSes posted 
> based solely on the
>  fact that a review team or directorate did a review but it was not 
> addressed.  Further, those reviews are referenced in the DISCUSS as 
> (for example) "the review done by Murray Kucherawy of the 
> Applications Directorate", rather than just by name.

If someone from a review team or directorate did a review, it is 
likely because some AD sees a benefit is getting their 
perspective.  If the issues pointed out in the review have not been 
addressed and an AD thinks that it is worthy of a DISCUSS, it is 
within their prerogative to do so.

Maybe the "Applications Directorate" part was added to acknowledge the effort.

>It's true that we're volunteers, but being appointed to the 
>directorate does mean someone thinks you're an expert at something.

If a person joins the directorate, it means:

  (a) that the person will perform document reviews as requested by the
      Application Area Directors.

  (b) that the person implicitly acknowledges that he can comment
      on a particular subject and provide relevant technical arguments.

As the reviews are posted to an IETF mailing list which is open to 
everyone who cares to join it, the community can assess the expertise 
of the person (see point (b)).

If a person are still on the Application Area Directorate, it means:

  (i)  that the person can still commit your time to perform document reviews
       as requested by the Application Area Directors.

  (ii) Someone thinks that the person have contributed the expertise and
       experience he said he had through AppsDir reviews.

>This paragraph of SM's message captures it better, I think, though I 
>would change the word "veto" to something else:

I chose "veto" instead of an euphemism so that the reader does not 
have to second-guess what I meant.

Regards,
S. Moonesamy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Apr 2012 21:12:07 +0000

> From: S Moonesamy [mailto:[email-masked]]
> Sent: Wednesday, April 11, 2012 1:47 PM
> To: [email-masked]; Murray S. Kucherawy
> Cc: [email-masked]; [email-masked]
> Subject: Re: [appsdir] Responding to Applications Area Directorate reviews
> 
> If someone from a review team or directorate did a review, it is likely
> because some AD sees a benefit is getting their perspective.  If the
> issues pointed out in the review have not been addressed and an AD
> thinks that it is worthy of a DISCUSS, it is within their prerogative
> to do so.
> [...]

I think most of what you said is geared toward those of us that are already part of AppsDir.  I'm talking from the perspective of people who receive these reviews, especially those of us less familiar with the process.

And from my own experience as an author, for reasons I mentioned earlier, it certainly seems as if directorate reviews get more attention than individual ones even though we disclaim that in the boilerplate at the top.

-MSK

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 11 Apr 2012 23:42:30 +0200

> And from my own experience as an author, for reasons I mentioned earlier, it certainly seems as if directorate reviews get more attention than individual ones even though we disclaim that in the bo…

And, like for any kind of cross-area review they *should* get good attention, please!
So the de-facto status is not a bug, but a feature.

We should just make sure to make very clear that there is no formal status, so the WG is as free to process (as, in, heed or discard) advice as for any other kind of more or less informed advice, including direct advice from an AD outside the formal ballot process.

Grüße, Carsten

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 11 Apr 2012 21:45:55 +0000

> From: Carsten Bormann [mailto:[email-masked]]
> Sent: Wednesday, April 11, 2012 2:43 PM
> To: Murray S. Kucherawy

I totally agree.  I just don't want us to go too far claiming they won't get special attention when they often do.  It seems misleading at best.

-MSK

---

From: Dave Crocker <[email-masked]>
Date: Wed, 11 Apr 2012 15:01:37 -0700

On 4/11/2012 9:27 AM, Murray S. Kucherawy wrote:
> I think that's a laudable goal, but there are constant reminders that
> directorate reviews at least appear to carry more weight since their
> association with authority (the ADs in particular) is difficult to obscure.
> Often I see DISCUSSes posted based solely on the fact that a review team or
> directorate did a review but it was not addressed.  Further, those reviews
> are referenced in the DISCUSS as (for example) "the review done by Murray
> Kucherawy of the Applications Directorate", rather than just by name.
...
On 4/11/2012 2:12 PM, Murray S. Kucherawy wrote:
> And from my own experience as an author, for reasons I mentioned earlier, it
> certainly seems as if directorate reviews get more attention than individual
> ones even though we disclaim that in the boilerplate at the top.

Note that when an IAB person speaks in a working group, the fact of their
affiliation does weight heavily in psychological terms, but they have no
/formal/ status in the wg.

I think the same applies here.

Yes, it's "notable" that the review is from a directorate, but the formal status
is that it's just another review.  That some folk like ADs pay special attention
might well be true, but it might be good and it might be problematic. It's a
separable issue that's their own choice and not a de jure formality.

Those of us writing and pursuing our reviews need to keep in mind that the
formal status is that it has not formal status.  The review is our personal
comments.

d/
-- 

  Dave Crocker

---

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Apr 2012 17:34:36 -0700

Hi Murray,
At 14:12 11-04-2012, Murray S. Kucherawy wrote:
>And from my own experience as an author, for reasons I mentioned 
>earlier, it certainly seems as if directorate reviews get more 
>attention than individual ones even though we disclaim that in the 
>boilerplate at the top.

I agree that directorate reviews get more attention.  Dave commented 
on the psychological aspect.

The candid answer is at 
http://www.ietf.org/mail-archive/web/apps-discuss/current/msg04899.html 
If the Apps ADs ask me about this matter, I will give them the same 
answer.  If an author of a draft which has been reviewed by AppsDir 
considers the answer as inappropriate, I suggest that he/she files a 
complaint with the Apps ADs as that's a problem for them to address 
in any manner they wish.

Regards,
S. Moonesamy

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 14 Apr 2012 18:04:58 +0100

On 11/04/2012 10:25, S Moonesamy wrote:
> The Applications Area Directors may or may not agree with the comments 
> of the AppsDir reviewer.  The  Applications Area Directorate does not 
> have any veto power on a draft.  It is suggested that the author(s) 
> asks the document shepherd (RFC 4858) or the Area Director who 
> sponsored the draft for guidance about IETF process, changes that 
> http://www.ietf.org/mail-archive/web/apps-discuss/current/msg04859.html ).  
> There isn't a clear-cut rule on what is a major or minor issue.  It's 
> up to the reviewer to make a judgement call.  It is up to the 
> author(s), or working group, to decide about whether these issues 
> should be resolved and how to do so.
For me "major" is something I would have put a DISCUSS for.
> reader can understand it. 

SM, it is not clear if you are trying to suggest some changes to the 
boilerplate used in AppsDir reviews or not. Can you clarify?

---

From: S Moonesamy <[email-masked]>
Date: Mon, 16 Apr 2012 05:14:11 -0700

Hi Alexey,
>For me "major" is something I would have put a DISCUSS for.

Yes, as long as it is accompanied by supporting arguments.

>boilerplate used in AppsDir reviews or not. Can you clarify?

The boilerplate can be improved.  I don't want to turn it into a long 
disclaimer though.  I suggest to wait and see whether the information 
for document authors published at 
http://trac.tools.ietf.org/area/app/trac/wiki/ApplicationsAreaDirectorate 
ends up being useful.  The boilerplate can be fine-tuned if there is 
significant feedback from people affected by AppsDir reviews.

Regards,
S. Moonesamy

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 17 Apr 2012 17:53:18 +0900

> For me "major" is something I would have put a DISCUSS for.

Then this is a difficult call for somebody who never was an AD :-).

About once per draft, I find myself in the situation that I want to say 
"you would really better have done this in that other way", but don't 
really expect the authors/WG to actually do it, because it may be too 
late to change. What I hope is that the authors/WG learn something, and 
do it the better way next time.

I usually classify these as major; I hope that's okay.

Regards,   Martin.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 17 Apr 2012 10:37:27 +0100

On 17 Apr 2012, at 09:53, "Martin J. Dürst" <[email-masked]> wrote:

> About once per draft, I find myself in the situation that I want to say "you would really better have done this in that other way", but don't really expect the authors/WG to actually do it, because it may be too late to change. What I hope is that the authors/WG learn something, and do it the better way next time.
> 

It is all a bit subjective, but there is IESG DISCUSS Criteria document which can be used as a guideline.

---

From: S Moonesamy <[email-masked]>
Date: Tue, 17 Apr 2012 04:11:37 -0700

Hi Martin,
>Then this is a difficult call for somebody who never was an AD :-).

The DISCUSS criteria is documented at 
http://www.ietf.org/iesg/statement/discuss-criteria.html

>About once per draft, I find myself in the 
>situation that I want to say "you would really 
>better have done this in that other way", but 
>don't really expect the authors/WG to actually 
>do it, because it may be too late to change. 
>What I hope is that the authors/WG learn 
>something, and do it the better way next time.
>

One of the reasons why AppsDir performs early 
reviews is so that changes can suggested before it is too late to change.

If your advice is that "this would really have 
been done in that other way", it's fine to raise 
it as an issue.  The informality of AppsDir 
reviews allows AppsDir reviewers wide 
discretion.  As such, such issues can be classified as major.

Regards,
S. Moonesamy