Subject: [Rfcplusplus] Conversation as metaphor

From: Ted Hardie <[email-masked]>
Date: Mon, 9 Jul 2018 08:24:10 -0700

In a different thread, Eric made a statement about the RFC Series being in
conversation with other publications:

The RFC series (and also I-Ds) have an important role to play here, but it
> also exists in conversation with a lot of other publication venues, and I
> think that's healthy.
>

While I agree with him, I think the metaphor of "conversation" is even more
useful in describing both the current series and the question before us.
>From my personal perspective, the primary reason we use "RFC" as a series
identifier is to identify a specific set of technical documents as part of
a common "conversation".  The adoption of the term and series by the IETF
was a signal about the conversation their documents were to be part of;
choosing a different document  series (like ANSI, ISO, or minting a new
one) would have sent a signal about a different technical community with
whom the IETF was in dialog.

When the idea of different streams and stream managers gelled, we kept the
same series identifier for all of them.  I think, personally, we did that
because we wanted to be clear that all of the documents continued to be
part of a larger conversation about the development of Internet
technologies.

One way to understand the problem motivating this BoF is also through the
metaphor of conversation: many outside the community simply don't recognize
that there are multiple voices inside that conversation.  They see all of
the documents as utterances by a single, somewhat nebulous group.  That can
cause problems.  Among those named earlier were the academic community's
failure to value the output of the IRTF; vendors or customers not
distinguishing consensus output from proprietary alternatives; and even a
few efforts to get rejected ideas to appear to have been accepted ones.

The question before us could be cast as:  is it more important now to
highlight the different voices that the streams and statuses currently
convey, so that others understand them as disctinct?

As Eric points out, there are other ways to maintain a conversation among
different groups than to make their output part of a single series.  There
are also other ways we could try to make sure that we highlight that
distinction more fully (using STD numbers for all IETF standards documents,
for example, from proposed standard onward).  But I think this is the core
of the tension, shorn of discussion of brand or history:  how to we get to
the right balance of maintaining the conversation while improving the
understanding that these are individual voices within it?

My thoughts as individual,

regards,

Ted

---

From: Eliot Lear <[email-masked]>
Date: Mon, 9 Jul 2018 23:14:59 +0200

Hi Ted,

I like your metaphor of a conversation, but it seems to me that we have
to bound what we mean.  This is not a conversation between friends, nor
a @Tweet, but a very formal conversation in which a well supported
specification or point of view is articulated.  Each contribution to
that conversation, as far as RFCs are concerned, should be sufficiently
clear so as to be consumable, and each should be important, no matter
which stream they come from.

As to voices, one of the perhaps lost aspects of RFCs is that Jon always
encouraged dissenting views on important topics to be published, so long
as they were well reasoned and reasonably well written.  Personally I'd
like to see more dissenting views coming out of independent
submissions.  I know I have quite a few to share ;-)  I offer that only
in as much as having a few dissenting views might make clear that we
really do speak with multiple voices.

Anyway, I won't be in Montreal, so here's my own bottom line:  if the
IAB are serious about actually using different labels, ask yourselves
what the plan is to make those new series successful.  How will the new
series be promoted?  What will you attract readership?  And what will
the rules of the road be for the series?  If you do not have a plan,
then let's not be under any illusions that the new series will be
successful.

Eliot

On 09.07.18 17:24, Ted Hardie wrote:
> In a different thread, Eric made a statement about the RFC Series
> being in conversation with other publications:
>
>     The RFC series (and also I-Ds) have an important role to play
>     here, but it also exists in conversation with a lot of other
>     publication venues, and I think that's healthy.
>
>
> While I agree with him, I think the metaphor of "conversation" is even
> more useful in describing both the current series and the question
> before us.  From my personal perspective, the primary reason we use
> "RFC" as a series identifier is to identify a specific set of
> technical documents as part of a common "conversation".  The adoption
> of the term and series by the IETF was a signal about the conversation
> their documents were to be part of; choosing a different document 
> series (like ANSI, ISO, or minting a new one) would have sent a signal
> about a different technical community with whom the IETF was in dialog. 
>
> When the idea of different streams and stream managers gelled, we kept
> the same series identifier for all of them.  I think, personally, we
> did that because we wanted to be clear that all of the documents
> continued to be part of a larger conversation about the development of
> Internet technologies.
>
> One way to understand the problem motivating this BoF is also through
> the metaphor of conversation: many outside the community simply don't
> recognize that there are multiple voices inside that conversation. 
> They see all of the documents as utterances by a single, somewhat
> nebulous group.  That can cause problems.  Among those named earlier
> were the academic community's failure to value the output of the IRTF;
> vendors or customers not distinguishing consensus output from
> proprietary alternatives; and even a few efforts to get rejected ideas
> to appear to have been accepted ones.
>
> As Eric points out, there are other ways to maintain a conversation
> among different groups than to make their output part of a single
> series.  There are also other ways we could try to make sure that we
> highlight that distinction more fully (using STD numbers for all IETF
> standards documents, for example, from proposed standard onward).  But
> I think this is the core of the tension, shorn of discussion of brand
> or history:  how to we get to the right balance of maintaining the
> conversation while improving the understanding that these are
> individual voices within it?
>
> _______________________________________________
> Rfcplusplus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfcplusplus

---

From: Ted Hardie <[email-masked]>
Date: Tue, 10 Jul 2018 10:13:50 -0700

On Mon, Jul 9, 2018 at 2:14 PM, Eliot Lear <[email-masked]> wrote:

> encouraged dissenting views on important topics to be published, so long as
> they were well reasoned and reasonably well written.  Personally I'd like
> to see more dissenting views coming out of independent submissions.  I know
> I have quite a few to share ;-)  I offer that only in as much as having a
> few dissenting views might make clear that we really do speak with multiple
> voices.
>
I personally don't think the dissenting opinions are fewer now, but fewer
of them are expressed in RFCs.  The critiques and dissents come, as Eric
mentioned, in other series or other forms.

Anyway, I won't be in Montreal, so here's my own bottom line:  if the IAB
> are serious about actually using different labels, ask yourselves what the
> plan is to make those new series successful.  How will the new series be
> promoted?  What will you attract readership?  And what will the rules of
> the road be for the series?  If you do not have a plan, then let's not be
> under any illusions that the new series will be successful.
>
> I think you're quite right that making a successful change of this type
will take work, and that promoting the output of the different streams will
be part of that work.  What's been interesting for me is that the thought
of how different the audiences are for that promotion.  Reaching the folks
who need to understand what the IRTF's output is and means is a very
different enterprise than reaching those who might need to understand what
the Internet Architecture Board has to say.  Given the acronym overload
there, a portion of that has to be making sure which IAB is talking, an
impact that the IAB or ISE would not feel.    But I think that actually
illustrates the problem folks have brought forward here:  if we were
promoting these voices, we would do so at least partly to very different
audiences.  That's may be an important signal.

Ted

> more useful in describing both the current series and the question before
> us.  From my personal perspective, the primary reason we use "RFC" as a
> series identifier is to identify a specific set of technical documents as
> part of a common "conversation".  The adoption of the term and series by
> the IETF was a signal about the conversation their documents were to be
> part of; choosing a different document  series (like ANSI, ISO, or minting
> a new one) would have sent a signal about a different technical community
> with whom the IETF was in dialog.
>
> Rfcplusplus mailing [email-masked]://www.ietf.org/mailman/listinfo/rfcplusplus
>
>
>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 10 Jul 2018 11:15:22 -0700

Howdy,

On Tue, Jul 10, 2018 at 11:06 AM, Eric C Rosen <[email-masked]> wrote:

> On 7/9/2018 11:24 AM, Ted Hardie wrote:
>
>> the academic community's failure to value the output of the IRTF
>>
>
> I don't understand the relevance of this issue to anything being discussed
> here.
>
> If a RG wishes to publish its output in a respected academic journal, I
> don't think there is anything to stop them.
>
> I meant specifically "output as RFCs", sorry that wasn't as clear as it
needed to be.

> Maintaining the current publication process for RG output but changing the
> name from "RFC" to "IRTF-Masterpiece" is not going to make any difference
> in how the academic community values the work.
>
>
That's not what others are saying.  The problem as it has been explained to
me is that tenure committees and similar academic assessments look at the
output of the individual in part by looking at where the output has
appeared.  When it appears in a series which looks to them to be primarily
engineering specifications, it is not valued as highly.  A differently
labeled series which was entirely composed of research output might be
valued differently.

I agree with Eliot's earlier point that this is a "might", and that the
value ascribed to it depends not just on its output but on the work done to
highlight the stream's output.

regards,

Ted

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 10 Jul 2018 14:06:35 -0400

> the academic community's failure to value the output of the IRTF

I don't understand the relevance of this issue to anything being 
discussed here.

Maintaining the current publication process for RG output but changing 
the name from "RFC" to "IRTF-Masterpiece" is not going to make any 
difference in how the academic community values the work.

---

From: Bob Hinden <[email-masked]>
Date: Tue, 10 Jul 2018 12:19:58 -0700

> On Jul 10, 2018, at 11:06 AM, Eric C Rosen <[email-masked]> wrote:
> 
> I don't understand the relevance of this issue to anything being discussed here.
> 
> If a RG wishes to publish its output in a respected academic journal, I don't think there is anything to stop them.
> 
> Maintaining the current publication process for RG output but changing the name from "RFC" to "IRTF-Masterpiece" is not going to make any difference in how the academic community values the work.

Worse, I suspect it will make it harder.  The RFC Series is well known, the “IRTF-Masterpice” series will be new and not trusted.

Bob

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 10 Jul 2018 15:28:10 -0400

On Tue, Jul 10, 2018 at 12:19:58PM -0700, Bob Hinden wrote:
> Worse, I suspect it will make it harder.

Of course, as people have pointed out in other threads, we actually
just have no real data about this and our gut feelings about what
might happen are not a good guide.

> The RFC Series is well known, the “IRTF-Masterpice” series will be
> new and not trusted.
 
One thing we _do_ have some data about is the low regard in which the
RFC series is held in many academic contexts, because it is not
considered a peer reviewed publication and because it is often
considered a series for technical standards rather than for academic
research.  That is data that has been provided by various IRTF
participants over several years, and with few exceptions I'm unaware
of that having changed in any large degree recently.

A different series with a different publication standard might have an
easier time of getting acceptance for this role than an established
one that "everyone knows" isn't a peer-reviewed publication.
Certainly in the mists of time when I was still an aspiring academic,
it was easier to establish a new journal's reputation than it was to
rectify the reputation of a journal that was widely regarded as having
passed its peak.  Rather than speculating, however, I'd be inclined to
do some research on journal ranking in the past decade among computer
science people.  This is again something that I doubt can be done
without some investment.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Bob Hinden <[email-masked]>
Date: Tue, 10 Jul 2018 12:51:27 -0700

Andrew,

> On Jul 10, 2018, at 12:28 PM, Andrew Sullivan <[email-masked]> wrote:
> 

Fair point.

I think that’s a good summary for the whole topic of the BOF.   We don’t understand the problem (or even agree there is a problem), nor have we done any real research on solutions.

I continue to think a better way to approach these issues, is to hand them to the RFC Editor and let her come back with a recommendation.

Bob

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 10 Jul 2018 15:58:47 -0400

On 7/10/2018 3:28 PM, Andrew Sullivan wrote:
> One thing we_do_  have some data about is the low regard in which the
> research.

If the IRTF determines that it would prefer a different publication 
venue, I certainly have no problem with that.  We shouldn't force 
researchers to publish their work as RFCs.  (I didn't think we did so, 
but ...)  If I were a researcher, I'd probably want to publish in a 
reputable academic journal, but perhaps my academic experience is a bit 
dated.

But that's really irrelevant to the main topic of this discussion. What 
we are primarily discussing is whether we should prohibit folks from 
publishing RFCs when they want to do so and find it valuable to do so.  
That's a completely different question.

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 10 Jul 2018 13:03:16 -0700

> mentioned, in other series or other forms.

It's perhaps worthwhile to distinguish between a number of different types
of this kind of meta-RFC material. Here's a non-exhaustive list

- Alternate proposals that didn't get accepted
- "Inside" critiques (some of the examples Eliot gave)
- "Outside" critiques
- Academic papers on a given protocol (e.g., the examples I gave for TLS
1.3)

In my experience, the first two of these often get published as RFCs, but
sometimes don't (though not always, see
https://hueniverse.com/oauth-2-0-and-the-road-to-hell-8eec45921529).
Conversely the last two almost never appear as RFCs. Critiques are
typically blog posts, twitter, etc., with academic papers appearing in
journals, conferences, etc.

-Ekr

>> In a different thread, Eric made a statement about the RFC Series being
>> in conversation with other publications:
>>
>> The RFC series (and also I-Ds) have an important role to play here, but
>>> it also exists in conversation with a lot of other publication venues, and
>>> I think that's healthy.
>>>
>>
>> the same series identifier for all of them.  I think, personally, we did
>> that because we wanted to be clear that all of the documents continued to
>> be part of a larger conversation about the development of Internet
>> technologies.
>>
>> Rfcplusplus mailing [email-masked]://www.ietf..org/mailman/listinfo/rfcplusplus <https://www.ietf.org/mailman/listinfo/rfcplusplus>
>>
>>
>>
>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 10 Jul 2018 13:09:18 -0700

On Tue, Jul 10, 2018 at 12:58 PM, Eric C Rosen <[email-masked]> wrote:

>
> If the IRTF determines that it would prefer a different publication venue,
> I certainly have no problem with that.  We shouldn't force researchers to
> publish their work as RFCs.  (I didn't think we did so, but ...)  If I were
> a researcher, I'd probably want to publish in a reputable academic journal,
> but perhaps my academic experience is a bit dated.
>
> But that's really irrelevant to the main topic of this discussion. What we
> are primarily discussing is whether we should prohibit folks from
I must respectfully disagree that this is an irrelevant question.  The
proposed experiment would establish new identifiers for the work of the
IRTF currently published as a stream within the RFC series.  That output
might (and I acknowledge again that it is only "might") change the academic
assessment of that output while retaining its other characteristics.  I
personally believe this question is thus on topic for a discussion of the
experiment.

regards,

Ted

---

From: "Aaron Falk" <[email-masked]>
Date: Tue, 10 Jul 2018 16:23:42 -0400

On 10 Jul 2018, at 16:09, Ted Hardie wrote:

>> venue,

As the IRTF chair when this stream was formalized I should point out 
that an important reason for RGs to publish RFCs is the same reason that 
they are RGs in the first place: to have their work be adjacent, 
influence, and possibly migrate to the IETF. Of course, that is not the 
only reason we have RGs but it is a good reason to have non-standards 
track output in similar format & repositories as the IETF.

IIRC, the copyrights for IRTF stream RFCs default to permit publication 
elsewhere to allow for easy submission in more-recognized forums.

Finally, I haven't seen anyone present that helping IRTF output get more 
recognition is a goal of this activity.  Is someone asking for this?

--aaron

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 10 Jul 2018 16:45:12 -0400

On Tue, Jul 10, 2018 at 03:58:47PM -0400, Eric C Rosen wrote:

> certainly have no problem with that.  We shouldn't force researchers to
> publish their work as RFCs.  (I didn't think we did so, but ...)

Indeed.

> are primarily discussing is whether we should prohibit folks from publishing
> RFCs when they want to do so and find it valuable to do so.  That's a
> completely different question.

It's a distinct question, but (as Ted argued) I think it's still
on-topic.

But I think your way of describing this issue has a certain merit,
because you are in effect asserting that there is no benefit to the
Internet community in having a gateway function to the RFC series.
One proposed such gateway is IETF consensus, and the argument for that
is that some significant population of people seem to believe already
that the IETF is who is responsible for RFCs (see the remark I posted
yesterday about Googling this).  So, the argument goes, we should
align what we claim with the way things are on the Internet, just as
we might do in the case of our own protocols.

Others seem to be arguing that the RFC series should in fact welcome
anything on any topic, and as long as they can find someone willing to
let them in, there's no reason they shouldn't be in the RFC series too
(i.e. we shouldn't prohibit people "from publishing RFCs when they
want to do so and find it valuable to do so").  It is an interesting
question whether, if I wanted to publish an RFC about Wittgenstein's
remarks on colour, whether the ISE would consider it.  We do have the
April 1st ones, after all.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Ted Hardie <[email-masked]>
Date: Tue, 10 Jul 2018 14:13:05 -0700

> elsewhere to allow for easy submission in more-recognized forums.

Well, you're the author of the relevant doc, so I hesitate to cite it to
you, but the text is in RFC 5743 and in
https://trustee.ietf.org/docs/IETF-Trust-License-Policy.pdf.  Basically,
though, the individuals retain their own copyright (along with the Trust
Copyright, managed by the IETF Trust at the request of the IRTF).  They can
re-submit.

> recognition is a goal of this activity. Is someone asking for this?

It's been a discussion between IRTF folk and the IAB for some time.  It
could, of course, be a separate activity from this experiment.

Ted

---

From: "Aaron Falk" <[email-masked]>
Date: Tue, 10 Jul 2018 17:24:31 -0400

> Copyright, managed by the IETF Trust at the request of the IRTF).  

Thanks for confirming my memory is still of some use. ;)

> could, of course, be a separate activity from this experiment.

Indeed.  I think it is separate for the reason in my last message you 
chose not to cite:

>  an important reason for RGs to publish RFCs is the same reason that 
> influence, and possibly migrate to the IETF. Of course, that is not 
> the only reason we have RGs but it is a good reason to have 
> non-standards track output in similar format & repositories as the 
> IETF.

To be explicit: while there might be value in creating a peer-reviewed 
"IRTF Journal" or somesuch (and it has been proposed many times), many 
IRTF documents would be candidates for it.  Compared the IRTF RFC 
Stream, the criteria for publication would be quite different.

--aaron

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 10 Jul 2018 17:29:21 -0400

On 7/10/2018 4:45 PM, Andrew Sullivan wrote:
> you are in effect asserting that there is no benefit to the

What I would say is  that there is no benefit to the Internet community 
in making a major change to the existing gateway function.

> It is an interesting
> remarks on colour, whether the ISE would consider it

I don't think that has any relevance to Internet engineering.

---

From: "Aaron Falk" <[email-masked]>
Date: Tue, 10 Jul 2018 17:29:43 -0400

On 10 Jul 2018, at 17:24, Aaron Falk wrote:

> IRTF documents would

*not* (sigh)

> be candidates for it.  Compared the IRTF RFC Stream, the criteria for 
> publication would be quite different.
>
> --aaron

---

From: Melinda Shore <[email-masked]>
Date: Tue, 10 Jul 2018 13:49:18 -0800

> track output in similar format & repositories as the IETF.

It's also probably worth mentioning that academics produce output
other than research papers, and we've got at least one instance of
a researcher who's got a protocol spec they'd like to see published
as an RFC through the IRTF.  It's already been submitted as a draft
and that draft has led to the start of several independent
implementations, and may be used as foundational technology for
other work.  There's at least some value to academics in having
an I-related publication stream available, independent of whether
or not an RFC carries the same stature as peer-reviewed publication.

Melinda

---

From: "Aaron Falk" <[email-masked]>
Date: Tue, 10 Jul 2018 17:51:10 -0400

On 10 Jul 2018, at 17:49, Melinda Shore wrote:

>> the

Indeed.  I would say this is a very common case.  Researchers know the 
RFC series isn't the place to publish original research.

--aaron

---

From: "RFC ISE (Adrian Farrel)" <[email-masked]>
Date: Tue, 10 Jul 2018 15:20:12 -0700

> April 1st ones, after all.

If the question is so interesting you could, of course, ask the ISE and
get an answer. :-)

To save you the trouble...

RFC 4846 provides some good guidance that I consider a little loose. The
list of document types in Section 2 starts well mentioning "Internet" or
"IETF" in every bullet. But then it stops doing that (perhaps the authors
got tired of typing the same thing?).

In my opinion all IS publications must be related to the Internet and/or
the workings of the IETF. That applies even to "Satirical material,"
"Eulogies," and "Meeting notes and reports".

I'm open to a discussion of that, and the easiest way to test it is to
write your draft and ask for publication. Until then, the only connection
to the Internet that I can see is that "Remarks on Colour is considered
difficult on account of its fragmentation," [1] suggesting that
Wittgenstein may have had opinions on what to do when the link MTU was
exceeded.

Of course, I have not been in harness long enough for anyone to establish
a pattern for what documents I consider in or out of scope. I will always
be interested to hear when a document I am considering or publication is
considered out of scope.

Thanks,
Adrian

[1] McGinn, M. (October 1991). "Wittgenstein's Remarks on Colour".
Philosophy. 66 (258): 435–453. doi:10.1017/S0031819100065104. JSTOR
3751218.

-- 
Adrian Farrel (ISE),
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 11:46:19 +1200

> labeled series which was entirely composed of research output might be

That's highly unlikely. Although the whole business of journal impact factors
and the value of conference publications is mainly pseudo-science, and the
whole question of open-access journals (where the author's institution pays)
versus traditional journals (where the reader's institution pays) is very
contentious, the chance of a brand new series from the IRTF breaking into
that game is vanishingly small. (IMNSHO, of course)

RFCs have a 50 year history and are in fact widely cited in academic
publications. My own 2nd highest citation count** is 422 for RFC1958
(unfortunately, most indexes have me as author, not editor). My highest
non-RFC citation count is 153.

**according to Google Scholar.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 14:04:45 +1200

> That's a completely different question.

Indeed, and that (to repeat myself) is something that the IETF and its
committees simply cannot do, since the RFC series belongs to a much wider
community. In that sense, the entire discussion is out of order.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 14:10:35 +1200

> personally believe this question is thus on topic for a discussion of the

Ted,

It would be on topic if there was a proposal inside the IRTF to change
the publication venue for IRTF output. But this is an IETF BOF so all
we can do is discuss how IETF stream documents are published.

I know this is an inconvenient truth for some people, but there it is.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 14:14:49 +1200

On 11/07/2018 08:45, Andrew Sullivan wrote:
...
> anything on any topic,

Really? I haven't seen that and it seems like a strawman to me.

The Independent stream has criteria that have already been mentioned
here; relevance to the Internet and surviving peer review are among them.

> and as long as they can find someone willing to

But even they are tested for relevance to the Internet and are peer
reviewed, just not in public for fairly obvious reasons.

   Brian

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 11 Jul 2018 10:58:03 +0200

'Morning, all, pseudoacademic here (waves).

> On 11 Jul 2018, at 01:46, Brian E Carpenter <[email-masked]> wrote:
> 

It depends on the committee and the question before them, but it's not quite so cut-and-dried, at least outside the US. Engineering work is valued (as impact) when there's also "more serious" academic work next to or behind it; exclusively engineering work is generally not.

> and the value of conference publications is mainly pseudo-science,

Indeed, one of the nice things about Google Scholar (for us pseudoacademics who have been fortunate enough to be able to play in the space between Internet science and Internet engineering) is that it's good enough for these pseudoscientific purposes (i.e. people who want a number that allows them to classify a person as "a serious academic"), while being quite a bit less picky about the label on the publication or on the publications citing it.

> and the

The case of RFCs-as-peer-reviewed-academic-work is one on which someone could almost do academic work on. (Indeed, Brian co-authored an [article on this for CCR][1], though it's an editorial note and therefore, strictly speaking in the manner of caring about the difference between Standards Track and Informational RFCs, does not count).

On the one hand, they are usually often cited, because they're often used as a citation when an academic work needs to refer to an Internet protocol, which they often do. My own most-cited non-patent in [Google Scholar][2] is an tutorial that was designed to be a more useful cite than 5101/7011 for IPFIX and than 3954 for Netflow in these contexts; number four is RFC 5103, usually cited as "this study uses bidirectional flow data [RFC5103] as input...", i.e. it's most often used as academic citation shorthand for the *problem* 5103 addresses, not the particular solution therein described.

On the other hand, protocol specifications are generally not academically rigorous, in the sense that the decisions that they document are sometimes (often?) made for accidental reasons as opposed to essential ones (i.e. tradition or politics, not a rigorous theoretical or empirical evaluation of the solution space). The criteria for evaluation of a protocol are different than those for most academic work in networking. (Whether *that's* a good thing is another question, and one that in part MAPRG was set up to provide a venue to address.)

On a, um, third hand, in academia there's a complete lack of understanding (or interest in understanding) the work that goes into an RFC. They're often thought of as equivalent to tech reports (maybe in part because that's how you cite them in bibtex), but academics I've explained the process to generally comment with surprise something like "oh, so it's kind of like a big journal article", at least in the terms of the amount of review.

> **according to Google Scholar.

The ** here is, if I recall correctly, the work of Heather, who worked to get RFCs correctly indexed in Scholar. (Thanks, Heather!!)

Back toward being on topic, what does the value of the brand mean in academia? Different things to different academics. Do I support the use of alternate stream labels and/or alternate publication venues for IRTF output? Depends on the RG and the type of output, but I don't see that a new label would automatically be a less-useful brand than RFC.

(With respect to the IETF use of alternate labels specifically and the future of the stream more generally, ISTM that the proposed experiment has started a a useful conversation, and that converging on a problem statement or statements would be a good outcome for Monday's BoF. I'm still forming my own opinions on the relative importance of the various problems with the series. "Unsuitability of RFCs as a publication stream for Internet-related research" is not a big one, though.)

[1]:http://delivery.acm.org/10.1145/1680000/1672315/p31-carpenter.pdf
[2]:https://scholar.google.ch/citations?user=vTA2dL0AAAAJ&hl=en

---

From: Ted Hardie <[email-masked]>
Date: Wed, 11 Jul 2018 08:07:55 -0700

> I know this is an inconvenient truth for some people, but there it is.

If  you have a suggestion for a label that would help communicate that this
meeting involves all of the streams, I think we can get it applied and
communicated quickly.  Practically speaking, though, the co-location of
this meeting with the IETF, the ANRW/IRTF, and the IAB gives us  a large
number of contributors across many streams.  Adrian, as ISE, will also be
present.  The conflicts with this meeting within IETF 102 were strictly
minimized, and  I'm not sure how we could get a better opportunity to
discuss it in a face-to-face meeting.  Of course, the BoF has also given us
a chance to have this discussion on a dedicated mailing list. (In a reply
to a question which is almost certain to arise, Heather asked that we not
hold this on rfc-interest, because a large number of RFC format issues were
expected to be discussed at roughly the same time, and she did not think
interleaving them would be a good idea).

regards,

Ted

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 11 Jul 2018 10:03:00 -0600

On 7/10/18 8:14 PM, Brian E Carpenter wrote:
> ....

In what sense are Independent Stream documents peer reviewed?

At a higher level, is there a good definition of "peer review" so that
we can understand how documents from the various streams map to that
definition?

Peter

---

From: "RFC ISE (Adrian Farrel)" <[email-masked]>
Date: Wed, 11 Jul 2018 09:19:28 -0700

> we can understand how documents from the various streams map to that

Peter, "what is a peer and what constitutes a peer review?" are questions
that have bothered me for a while even for tier one journals. I have
received some frankly appalling journal reviews over the years (and I say
that not just because I have disagreed with them) where the reviewers have
clearly not understood the fundamentals of the material. I suspect we have
all read journal article where we have wondered about the source of the
substance being smoked.

So you might be asking: what review is performed on the Independent
Submissions stream? There is no secret about that, so here is a summary in
no particular order:

- The ISE reviews. This might not be a subject matter expert review as the
ISE is a generalist with some areas of detailed knowledge. The review is
often more for consistency and readability.

- The ISE commissions two or three reviews from persons suggested by the
authors but who are not directly involved in the work. It is evident to
the ISE from these reviews whether the reviewer has sufficient subject
knowledge to be classed as a peer. Note that failure to get sufficient
reviews in this class is gating on publication - if even the authors can't
find people to review their work then it really is of limited interest.

- The ISE requests reviews from other experts in the IETF community. These
are often drawn from the ISEB (names available on the web), but may be
people that the ISE can see have a close interest in the subject matter.
(If you would like to be more involved in this sort of review then I will
bite your hand off!)

- The RFC 5742 review by the IESG often draws review comments, although
the IESG is not required or expected to provide such a review.

Cheers,
Adrian

PS. Some I-Ds pass through WG and IETF last call with no comments at all
(possibly suggesting no one is reviewing them) and get relatively light
review in the IESG. Saying that is not an attack - just a comparison.

-- 
Adrian Farrel (ISE),
[email-masked]

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 11 Jul 2018 10:24:08 -0600

> the IESG is not required or expected to provide such a review.

Thanks for the clear summary of process and criteria.

> review in the IESG. Saying that is not an attack - just a comparison.

Most definitely.

We as a community seem to be confused about what level of "peer review"
we think is important, appropriate, necessary, or sufficient. We're
engineers, not academics, so comparing our publications to academic
journals might not be relevant.

Peter

---

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Jul 2018 10:38:01 -0700

Hi Peter,
>journals might not be relevant.

Do IETF participants need a license [1] in the practice of 
engineering?  There are academics who have authored RFCs in some of 
the Streams.

There is an IAB RFC about citability of for scholarly 
publications.  Why would academic journals not be relevant?

There has to be some level of review or else there wouldn't be any 
quality control.  I agree that there is some confusion on the level 
of review which is required.  It has been an issue over the past 
decade.  Some of the discussions about RFCs have a strong correlation 
to "IETF problems".

Regards,
S. Moonesamy

1. https://www.nytimes.com/2017/04/30/business/traffic-light-fine.html

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 11 Jul 2018 12:07:25 -0600

Hi SM!

On 7/11/18 11:38 AM, S Moonesamy wrote:
> Do IETF participants need a license [1] in the practice of engineering? 

s/engineers/technologists/ ;-)

> There are academics who have authored RFCs in some of the Streams.

Indeed. We could ask them what their goals were in doing so and what the
results of RFC publication were for them.

> There is an IAB RFC about citability of for scholarly publications.  Why
> would academic journals not be relevant?

Publishing an RFC is different from publishing a paper in an academic
journal, so applying the same criteria might not be appropriate. As far
as I can see we're not actively catering to an academic audience (either
as authors or readers).

> quality control.  I agree that there is some confusion on the level of
> review which is required. 

And different levels or kinds of review across the different streams.
This returns to the question of audiences for the streams. Vendors and
operators seem to differ quite a bit from researchers and academics
(naturally there is always overlap).

Peter

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 11 Jul 2018 11:54:57 -0700

On Wed, Jul 11, 2018 at 11:07 AM, Peter Saint-Andre <[email-masked]>
wrote:

> journal, so applying the same criteria might not be appropriate.

Indeed. For instance, novelty is one of the first questions one asks about
a scientific
paper. And yet here at IETF we routinely publish documents with no new
scientific
content at all. Indeed, one might argue that it's better if they don't
because you have
more confidence it will work

-Ekr

As far

---

From: Eliot Lear <[email-masked]>
Date: Wed, 11 Jul 2018 21:08:47 +0200

Hi Eric,

On 11.07.18 20:54, Eric Rescorla wrote:
>
> Indeed. For instance, novelty is one of the first questions one asks
> about a scientific

I agree.  On projects I've worked on, we've generally made it a point to
*not* start at the IETF, precisely because we'd rather kink out the bugs
elsewhere. Also, it's a pretty tough venue to get stuff through.  That
having been said, RFCs are excellent references for how things are
supposed to work.  Those aren't scientific observations and
interpretations, but rather engineering specifications.

Eliot

---

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Jul 2018 17:04:05 -0700

Hi Peter,
>s/engineers/technologists/ ;-)

:-)

>results of RFC publication were for them.

Yes.

>as I can see we're not actively catering to an academic audience (either

RFCs from the IETF Submissions Stream are relatively closer to the 
industry in comparison with an academic journal.

>(naturally there is always overlap).

The level of review is not apparent to the external reader.  The 
breadth of review is subsumed in what the IETF describes as 
"consensus"; other Streams use a different "measure".  The external 
reader is not interested in the review process.  One of the intrinsic 
property of a RFC is availability.  Another property, if I can 
describe it as such, is that a RFC may be viewed as authoritative.

Regards,
S. Moonesamy

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Jul 2018 12:20:28 +1200

On 12/07/2018 12:04, S Moonesamy wrote:
<snip>

> describe it as such, is that a RFC may be viewed as authoritative.

But falsely so, according to the boilerplate in many RFCs. However,
I'm not sure this is different from academic articles: whether or
not they are authoritative depends on many factors, and each reader
has to make their own judgment. Being labelled "RFC", or being published
in "IEEE Transactions", is only a starting point.

    Brian

---

From: Martin Thomson <[email-masked]>
Date: Thu, 12 Jul 2018 12:09:21 +1000

> in "IEEE Transactions", is only a starting point.

I have come to a slightly different conclusion to this.  The value of
a thing that is primarily a specification derives largely from how
closely it accurately describes reality.  As Brian (T) observes, an
academic publication might only be concerned with a superficial
treatment of the subject and so there are plenty of cases where any
reference will do (I theorize that this is what dramatically inflates
citation counts on certain documents).

For a document to be valuable beyond that simple purpose, it needs to
describe reality accurately.  To the extent that the IETF publishes
documents that are valuable, their congruence with reality is the
primary source of that value.  In many cases, we don't really get to
be sure about that until afterwards, because the documents are written
as a promise.  But one strength of our process is that the delivery on
promises is pretty high.

For many years, the value of certain specifications that I worked on
was diminished by neglect.  The draft said one thing, but practical
constraints meant that you had to do something else completely.
Whether that was to achieve interoperability, the expected level of
security, or whatever the reason, that had the effect of undermining
the credibility of those RFCs.

I realize that I'm about to expand the problem scope further, but...
We tend to hit more problems when we have aspirational protocol
specifications.  That is, protocols without a promise.  We're not
consistently able to distinguish between objectively good designs and
designs that will be deployed, partly because that requires the use of
time travel.  We do better with protocols that deploy before
publication, for example.

This is something that discussions of the level of review don't really
help with.  I consider the value of the RFC series to derive primarily
from its ability to accurately describe some part of the state of the
Internet.  Peer review in the sense of what an academic publication
demands doesn't serve that goal.  Current IETF processes and culture
are somewhat geared toward that outcome, which is what I believe has
produced this reputation.

That's not necessary a shared goal across the streams.

I suspect that the outcome here may depend more on whether you think
that the RFC series is a venue or a descriptor.  Conceived of as a
journal or publication outlet, I can see how people would conclude
there being no problem with the current situation.  This is merely
another place that you go to witness or participate in a discussion.
However, as a label or descriptor, the current situation isn't ideal.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 11 Jul 2018 22:19:13 -0400

At least in the areas I work in, RFCs when published describe protocols 
which have seen limited implementation or deployment.  Even when we are 
revising widely deployed protocols, the revisions or enhancements have 
usually experienced limtitd implementation or deployment.  We like to 
have some experience (IDR for example requires two implementations).  We 
rarely have large numbers of vendors implementing, since most vendors 
wait for the document to stabilize and be published as an RFC before 
they implement.

So I am very confused by your text describing RFCs that "describe reality".

Yours,
Joel

---

From: Martin Thomson <[email-masked]>
Date: Thu, 12 Jul 2018 12:47:46 +1000

> wait for the document to stabilize and be published as an RFC before

Yes, this is common.  What we did with HTTP/2 and TLS is perhaps weird
in that we deployed pretty widely well before the RFC was published.
My point was that the ability for the RFC to predict and ultimately
describe the reality of the Internet is grounded in either the actual
deployment (ideal) or the strength of the commitment of those who
participate in the process (the process you describe).  That RFCs end
up describing reality with a non-trivial proability means that we are
moderately successful in identifying which protocols will be deployed.

In other words, running code and all that.

---

From: Eliot Lear <[email-masked]>
Date: Thu, 12 Jul 2018 08:26:36 +0200

> in that we deployed pretty widely well before the RFC was published.

In interdisciplinary work of some form, the standard is really a
starting gun for implementation.  Also true when purchasing managers are
looking for a combination of interoperability and stability.

> moderately successful in identifying which protocols will be deployed.

Yes.  This goes to a rule that Brian Kantor of UCSD laid out very early
on (1992ish) with the series: RFCs provide their greatest utility when
they document existing practice (yes, he predated Clark on that).  I
would point out that value can be found in the independent stream and in
what we label "Informational" documents that contain specifications
perhaps as often as can be found in nascent standards.  Good examples
from the past include IAX, NFS, syslog, PGP, BGP4, and MPLS-VPNs. 
Perhaps today Chacha20,   Arguably all of these were de facto standards
long before the IETF touched them (so was IPv4).

At a cursory level that would argue for labeling not based on whether
something has achieved rough consensus in the IETF, but rather based on
how well fielded an approach is.  That would require two things: a
measure of adoption, something we have greatly struggled with in the
past, and relabeling from time to time.  Imagine for a moment, charging
the RFC Editor with establishing such a methodology for ALL documents. 
Now that would really get interesting ;-)  STD was *sort of* intended to
cover this ground, but it is very coarse grain, and requires a very high
bar, and to me has somewhat more political connotations for us than is
any real signal to a market.

Eliot

---

From: S Moonesamy <[email-masked]>
Date: Thu, 12 Jul 2018 02:13:12 -0700

Hi Joel, Martin, Peter,
At 07:19 PM 11-07-2018, Joel M. Halpern wrote:
>At least in the areas I work in, RFCs when published describe 
>protocols which have seen limited implementation or 
>deployment.  Even when we are revising widely deployed protocols, 
>the revisions or enhancements have usually experienced limtitd 
>implementation or deployment.  We like to have some experience (IDR 
>for example requires two implementations).  We rarely have large 
>numbers of vendors implementing, since most vendors wait for the 
>document to stabilize and be published as an RFC before they implement.
>

The conversation is getting back to what was known as (IETF) Proposed 
Standard.  The audience (re. Peter's comments} is vendors which have 
not participated in the review process as they are better placed to 
determine whether the specification can be clearly understood.  There 
is also the deployment angle, i.e. the practices are determined by 
the people running operations.

Regards,
S. Moonesamy

---

From: "Aaron Falk" <[email-masked]>
Date: Thu, 12 Jul 2018 12:44:45 -0400

On 12 Jul 2018, at 2:26, Eliot Lear wrote:

> past, and relabeling from time to time.

Elliot! This seems to me to be very interesting, useful, and difficult. 
(YMMV, of course)  MAPRG is touching on some aspects of this.

--aaron