Subject: [Rfcplusplus] RFC 1796 revisited

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jun 2018 09:58:42 -0600

The BoF proposal references RFC 1796, which I'd never looked at. The RFC
makes for interesting reading and takes us back to a time (1995) when
the Internet was being actively invented but was not all that usable for
online publishing. For instance, the document suggests that we:

      Utilize the "web" hypertext technology to publicize the state of
      the standardization process.

This newfangled "web" thing sure has taken off, hasn't it?

The following claim also has not aged well:

   The IAB believes that the community benefitted significantly from
   having a single archival document series.  Documents are easy to find
   and to retrieve, and file servers are easy to organize.  This has
   been very important over the long term.  Experience of the past shows
   that subseries, or series of limited scope, tend to vanish from the
   network.

Many document series are now published on the web and have been for
years, even including things like IESG and IAB statements. Yes, some of
what's been published has vanished from the network, but the more
valuable series are actively maintained. From this angle, there is no
absolute necessity for, say, IRTF documents to be published as RFCs
(e.g., the IRTF could curate a separate journal). And many independent
submissions could be submitted to technology journals, presented as
conference talks, or posted to weblogs. The Internet has made possible a
robust publication ecosystem, so an RFC monoculture makes less sense now
than it might have in 1995.

The mailing list posts from Barry and Adam led me to believe that the
primary concern here is customer confusion. The more significant
confusion is one mentioned in, but denied by, RFC 1796:

   It is a regrettably well spread misconception that publication as an
   RFC provides some level of recognition.  It does not, or at least not
   any more than the publication in a regular journal.  In fact, each
   RFC has a status, relative to its relation with the Internet
   standardization process: Informational, Experimental, or Standards
   Track (Proposed Standard, Draft Standard, Internet Standard), or
   Historic.  This status is reproduced on the first page of the RFC
   itself, and is also documented in the periodic "Internet Official
   Protocols Standards" RFC (STD 1).  But this status is sometimes
   omitted from quotes and references, which may feed the confusion.

The claim about recognition might have been true in 1995; however, due
to years of hard work in defining and implementing high quality
standards for IETF documents in particular, nowadays publication as an
RFC does indeed provide "some level of recognition" (actually a great
deal of it).

As I understand things now, the problem folks have in mind for this BoF
is that we continue to pursue a one-size-fits-all publication strategy
when at the least that is no longer necessary and at the most that might
be actively harmful because of widespread confusion about the nature and
goals of the RFC Series or the status of a particular document.

If I'm off-base, corrections are welcome.

Peter

---

From: Ted Hardie <[email-masked]>
Date: Wed, 20 Jun 2018 09:36:41 -0700

> If I'm off-base, corrections are welcome.

I would put this slightly differently.  We have formalized a system that
has multiple streams and multiple stream managers.  Each stream has
different input characteristics and its output should be judged, in part,
by that input.  To take the IAB example, IAB documents represent the
consensus of the IAB.  That consensus takes into account community
commentary, but the output can easily be the IAB's (sometimes unpopular)
opinion rather than the consensus opinion.  An IETF RFC on the same topic
might easily be very different and it would represent a different, larger
community's consensus.

At the moment, readers distinguish between the streams using in band
signals (boiler plate and status).  For any reader who is used to the
series, those are significant and likely sufficient.   It is clear, though,
that those are less effective signals for those not used to the series.

The BoF has two questions:  is this problem, now 25 years old or so, worth
tackling?  If so, would a method that moved to a much more explicit
signaling mechanism (e.g. a new name for the output of the IAB stream) help?

Additional questions that are part of exploring the second question are:
how would you test that the method has helped?  If you decide it has not or
decide it has damaged the visibility of a stream's output, how would you
revert to the status quo ante?

My take as an individual,

Ted

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jun 2018 11:40:02 -0600

On 6/20/18 10:36 AM, Ted Hardie wrote:
> On Wed, Jun 20, 2018 at 8:58 AM, Peter Saint-Andre <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> 
>      
> different input characteristics and its output should be judged, in
> part, by that input. 

Right. Another way to put this is that we have different document
creation processes (with different goals, levels of review, etc.)
resulting in publication to the same document series.

How is that working out for us and the readers of these documents?

A quick look at rfc-editor.org reveals the following data:

IETF stream - 5782 RFCs
IRTF stream - 71 RFCs
IAB stream - 105 RFCs
Independent stream - 326 RFCs
"Legacy" stream (pre RFC 3600 in 2003) - 2012 RFCs

So 92%+ of RFCs in the last 15 years were produced by the IETF. That
might be relevant to our thinking about what customers we're serving.

> To take the IAB example, IAB documents represent
> the consensus of the IAB.  That consensus takes into account community
> opinion rather than the consensus opinion.  An IETF RFC on the same
> topic might easily be very different and it would represent a different,
> larger community's consensus.
> 
> series, those are significant and likely sufficient.   It is clear,
> though, that those are less effective signals for those not used to the
> series.

Indeed.

> The BoF has two questions:  is this problem, now 25 years old or so,
> worth tackling?  If so, would a method that moved to a much more
> explicit signaling mechanism (e.g. a new name for the output of the IAB
> stream) help?

As hinted, I suggest that we also explore different output methods. For
instance, the IAB could publish documents at its website, which for some
kinds of documents it already does:

https://www.iab.org/documents/correspondence-reports-documents/

Perhaps IAB and IRTF folks have already explored such avenues?

> Additional questions that are part of exploring the second question
> are:  how would you test that the method has helped? 

Because we are working with a paucity of data, the testing methods are
not yet clear to me.

> If you decide it
> has not or decide it has damaged the visibility of a stream's output,
> how would you revert to the status quo ante?

Given the customer base mentioned above, I suggest that the key
documents here are those produced by the IETF (~92% compared to IAB at
~1%, IRTF at ~2%, and independent at ~5%).

Is visibility the right measure? And visibility to whom? Do we have a
clear picture of the user base for IAB or IRTF or independent stream
documents (or, for that matter, IETF stream documents)? It's not hard to
imagine that the audience for IRTF documents might skew somewhat
differently, and that the IAB audience overlaps significantly with the
IETF audience. The audience for independent stream documents strikes me
as the least well defined.

Are measures other than visibility relevant? (For instance: stability,
accessibility, distribution.)

Peter

---

From: "Andrew G. Malis" <[email-masked]>
Date: Wed, 20 Jun 2018 14:32:36 -0400

I’ve yet to see any suggestions on how we objectively measure the current
“problem”. All we’ve been discussing are anecdotal experiences. We
certainly can’t measure the results of changes if we can’t measure what we
want to change.

That’s why a low-cost, safe, approach that certainly can’t make anything
“worse” (assuming that it needs improvement) is to make the boilerplate
text very clear to a naive reader regarding the status of a particular RFC.
As Sue suggested, we should be able to trust the editors to do their job
and produce suitable wording.

Cheers,
Andy

On Wed, Jun 20, 2018 at 1:40 PM, Peter Saint-Andre <[email-masked]>
wrote:

> _______________________________________________
> Rfcplusplus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfcplusplus
>
>

---

From: Bob Hinden <[email-masked]>
Date: Wed, 20 Jun 2018 11:41:27 -0700

Peter,

> On Jun 20, 2018, at 10:40 AM, Peter Saint-Andre <[email-masked]> wrote:
> 

Given these numbers, I am finding it difficult to think the purported problem is very serious.  After all, as you report, more than 92% of the RFCs are from the IETF stream.  Could 326 RFCs from the Independent Stream be causing that much of a problem?

Bob

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Wed, 20 Jun 2018 11:53:26 -0700

On 6/20/18 11:32 AM, Andrew G. Malis wrote:
> I’ve yet to see any suggestions on how we objectively measure the 
> current “problem”. All we’ve been discussing are anecdotal 
> experiences. We certainly can’t measure the results of changes if we 
> can’t measure what we want to change.
>
> That’s why a low-cost, safe, approach that certainly can’t make 
> anything “worse” (assuming that it needs improvement) is to make the 
> boilerplate text very clear to a naive reader regarding the status of 
> a particular RFC. As Sue suggested, we should be able to trust the 
> editors to do their job and produce suitable wording.

If we as a community decide that this is a serious enough problem to 
assign resources to it (which means either deprioritizing other work or 
throwing more money at the problem), I think we need to start with a 
basic market research effort. I know this community tends to prefer 
using community-based resources, but market research is a whole field of 
endeavor that has people who are actually experts in that area. 
Marketing is often a dirty word to engineers, but it has its place and 
would give us the kind of baseline we need to figure out if and how to 
make changes to the Series in a data-driven way.

-Heather

---

From: "Paul Hoffman" <[email-masked]>
Date: Wed, 20 Jun 2018 12:46:03 -0700

> might be relevant to our thinking about what customers we're serving.

...but maybe not specific enough. One of the components of the proposed 
experiment is to publish informational, procedural, and experimental 
documents not in the main stream. Of that 92%, a not-insignificant 
number are informational, procedural, or experimental.

Just to be clear, I'm not proposing that this is a good idea. If the 
IESG wants to segregate out all their non-standards documents into 
separate publishing streams (hopefully well-collected with long-lived 
URLs), that should be their choice. As author of a bunch of those 
documents, I would prefer being able to keep the future ones in the RFC 
series, but it really is up to them.

--Paul Hoffman

---

From: Peter Saint-Andre <[email-masked]>
Date: Wed, 20 Jun 2018 14:14:53 -0600

> number are informational, procedural, or experimental.

Good point, Paul. I was thinking about the creation process (IETF
consensus or at least last call, IESG review, etc.), not the document
types within the IETF stream. I'm not yet sure which of these dimensions
is relevant.

Peter