Subject: [Rfcplusplus] draft-thomson-rfcplusplus-label-00

From: Brian E Carpenter <[email-masked]>
Date: Tue, 3 Jul 2018 20:14:25 +1200

Hi,

My main comment on this draft is that it proposes an experiment
without first explaining properly what the problem is and why the
proposed experiment might fix it.

It isn't sufficient to say there is misunderstanding of the
various RFC types and statuses. It isn't sufficient to give a
couple of examples of non-IETF-standard RFCs that might be
implemented and deployed in parallel with IETF standards.
Where is the damage to users, where are the numbers, how do
we assess the seriousness of the damage? Isn't it sometimes
better to have multiple solutions? Isn't it *always* better
to document something which is deployed, whether it's good
or bad? Isn't the IETF solution sometimes wrong? How much of
the perceived problem is just a matter of damaged pride? How
does this problem compare with others that have been known about
for just as long? Why is this one worth solving (disruptively)
when the others aren't?

Right now I just don't see a problem statement that answers those
questions.

I'm tempted to stop there, but here are some more detailed
comments:

> RFC 1796 [RFCS] was published in April of 1995.  At that time, it was
> clear that there was an "regrettably well spread misconception" that
> the label "RFC" implied "some level of recognition".

True. Also at that time (in fact a few months before, since the
RFC Editor had already reacted), the RFC boilerplate was modified
to avoid misconception for anyone who bothered to read it.

> Not a lot has
> changed in the 23 years since that statement.

Evidence, please.

I'd say that the evidence is that people don't take much notice
of the label and the boilerplate; they go straight to the text.
(OK, insert the obligatory anecdote of Avian Carriers being cited
in an RFP, but one swallow doesn't make a summer.)

> The common perception of the significance of the "RFC" label is
> simple.

That may be so, but since the draft doesn't state what that 
perception is, the reader cannot guess what is intended.

For example, my perception of the significance is that an RFC
is a request for comments. Then I look for its status. Then
I look for its contents. Isn't that what we should assume serious
engineers & product managers do?

> This document suggests that this view to some degree underestimates
> the value of a label (or "brand").

Or, equally plausibly, it overestimates the value of the label.
My belief is that, the occasional dishonest salesman apart, it's
the content that people value. If they know anything, they know
that *all* RFCs have had genuine peer review. (Unlike many
academic papers, see "Some science journals that claim to peer
review papers do not do so", The Economist, June 23, 2018.)

> This document proposes an
> experiment that limits the use of the "RFC" label to the product of
> that process.

I have to re-iterate that this is not a choice the IETF can make,
since the IETF and its committees do not own the RFC series. The
IETF can't kick the other streams out.

> Capturing the nuance required to properly understand a protocol is
> difficult, and a large number of documents fail to properly convey
> that status.

Correct. However, this is a hard problem that the IETF has
essentially refused to solve, because the solution is hard
work, both for WGs and for the IESG.
draft-klensin-std-numbers and draft-ietf-newtrk-sample-isd
were both put in the "too hard" bucket by previous IESGs,
for example.

> ZRTP [RFC6189] was published as an informational RFC on
> the IETF stream after the IETF reached consensus to develop DTLS-SRTP
> [RFC5764] for the same use case.

The document makes no claim to be a standard and it gives some
explanation of how it differs from RFC5764. It isn't clear
to me what is unclear to the reader.

> HTTP Live Streaming (HLS) [RFC8216] was published on the
> Independent Submissions Stream in defiance of a standardized
> protocol, MPEG-DASH [DASH]

The word "defiance" seems quite inappropriate. Firstly, the RFC
doesn't mention MPEG-DASH at all.  Secondly, the document makes
no claim whatever to be a standard.

> Both describe de-facto standards, each of which are implemented in
> more than one product and deployed.

Right. It has long been considered a good idea to document stuff that
is widely deployed, for everybody's benefit.

> Each captures a range of design decisions that differ from the
> commonly-accepted view.

How often in the history of engineering has "the commonly-accepted
view" turned out to be wrong? How often in the history of the
Internet has something unexpected turned out to be a winner?

> If nothing else, the deployment of these protocols
> could adversely affect interoperability relative to implementations
> of their more widely accepted alternatives.

That's true, of course, but the Internet has always worked by
survival of the fittest. I'm not seeing a tragedy here.

> Information about the status of the document is contained entirely in
> the "Status of this Note" section.  For instance, ZRTP is published
> with IETF consensus, so the significance of it not being an "Internet
> Standards Track specification" is easily lost.

Easily if you don't read the words, yes.

> That it also limits
> its mention of DTLS-SRTP to comparative criticisms makes it possible
> to interpret the document as it represents itself: a newer, superior
> technique.

Yes, the authors believe that their way is better. That's
hardly a surprise.

> There are numerous examples of RFCs that make an honest
> representation of their status in more than just the boilerplate.

Indeed. And this is an aspect that peer reviewers should look
out for.

> So why is the publication of an RFC so keenly sought, when the
> document doesn't capture the output of the Internet Standards
> Process?

I don't know. Did you try asking the authors? Speculation
doesn't help us much. For my own independent submissions,
it's because it's the appropriate way to reach the Internet
technical community in general (not just the IETF as you
suggest), and another motivation is to get constructive review
(as opposed to the essentially random or sometimes bogus review
you get from academic journals and conferences). Most of the
Internet technical community doesn't read academic journals, by
the way: apart from anything else they're too expensive, and
most of the articles are too... academic.

> This documents proposes reserving the "RFC" label for those documents
> that are the product of the Internet Standards Process [RFC2026].

And again, in case anybody missed it: the IETF can't do
that because it doesn't own the RFC Series.

What the IETF can do is change the requirements for its own
stream of RFCs, or stop calling its documents RFCs at all.

The second one is a terrible idea, for reasons I gave at
https://tools.ietf.org/html/draft-carpenter-request-for-comments-00#section-4

The first one is absolutely reasonable, and a good
first step is at
https://tools.ietf.org/html/draft-klensin-std-numbers-02
With the right typography in the new RFC format, this
could be very effective.

> 4.1.  New Labels for Other Documents

This section is ambiguous. But most of it is out of IETF
scope anyway, because the other users of the RFC series
will label their RFCs as they wish. I wouldn't be against
the header lines being more explicit.

> The "STD" label hasn't been especially effective in distinguishing
> those documents that attain the rare status of Full Internet
> Standard.  The BCP subseries has been more effective, perhaps because
> of the greater familiarity of its primary audience with its meaning.

Exactly why we should apply the STD series label to *all* standards
track documents, as proposed many years ago and more than once since then.
(Also why I'm a fan of
https://tools.ietf.org/html/draft-loughney-newtrk-one-size-fits-all-01 )

> However, the choice of rendering is not
> part of the canonical XML document.  Alternative renderings could
> legitimately erase that information.

They could drop special choices of font & font size, true, but they
couldn't drop the actual status etc. if it's part of the document
text. In any case, most people will use and refer to the primary
HTML rendering, which the RFC Editor will generate.

> That leads to the conclusion that clearer marking for status, while
> possibly helpful, would not be sufficiently effective in addressing
> the problem.

That may in fact be true, since the fundamental problem is that humans
see what they want to see, and readily skip boring stuff like document
labels and status. This leads to the conclusion that adjusting the labels
won't address the perceived problem: if people have a good reason to
ignore today's labels, they will ignore tomorrow's as well.

> A term of 3 years is proposed as being long enough to provide enough
> evidence to assess the effect of a change of labels.

I question whether that's long enough. Some product cycles are faster,
but many are slower than that. How many cases per year do we have
of RFC-label-confusion causing significant problems to users? Unless
that's a large number, and we can use those cases to assess the time lag
between RFC publication and the occurrence of problems for users,
I really don't know how long a test would be needed.

(The proposed metrics about publication rates are pretty useless
except perhaps for a sociologist, because they don't seem to
be relevant to the alleged problem. And in any case, as noted,
the datasets will be tiny and most of what we'll see will be
statistical noise.)

> 6.  Security and Privacy Considerations
> 
>    It is believed that there are none.

Given how many of the Independent Series RFCs are about
crypto algorithms and the like, potentially interfering with
that publication channel seems to pose a risk.

Regards
   Brian Carpenter

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 3 Jul 2018 10:06:45 -0400

Hi,

Sent from my mobile device

> On Jul 3, 2018, at 4:14 AM, Brian E Carpenter <[email-masked]> wrote:
> 
> Hi,
> 

I agree that we haven’t done a good job of promoting standards and this could solve the issue at hand.  If we are more consistent and then use the references for STD, this could result in reducing the confusion.  It’s a matter of discipline and willingness to do the work to assign the numbers and use them.  Perhaps this should be the first step.

> ignore today's labels, they will ignore tomorrow's as well.
+1
> that publication channel seems to pose a risk.
Agreed.  There are numerous drafts published for crypto where the independent stream is absolutely the right choice.  If it’s specific to a region, there’s no need for (and is likely inappropriate) IETF review and publication via AD sponsorship or a WG process.

Regards,
Kathleen 

> Regards
>   Brian Carpenter
> 
> 
> _______________________________________________
> Rfcplusplus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfcplusplus

---

From: Ted Hardie <[email-masked]>
Date: Tue, 3 Jul 2018 09:14:02 -0700

On Tue, Jul 3, 2018 at 1:14 AM, Brian E Carpenter <
[email-masked]> wrote:

> Isn't it *always* better to document something which is deployed, whether
> it's good
> or bad?
>

I believe that a number of your questions are ill-formed, and that the one
I highlight above illustrates this problem best.  The question is not
"should X, which is deployed, be documented?", but "Should the RFC Series
contain the documentation for X, which is deployed?".

If X was specified by the IETF, then the answer is yes, it should be
contained in the IETF stream of the RFC Series.  In the small number of
cases where it was specified by the IRTF, then it should be contained by
the IRTF stream of the RFC Series.  The IAB stream could also theoretically
produce an protocol specification that was deployed (though current
practice is to bring any work of that type back into the IETF).  Those
streams are, in other words, set aside as publication outlets for specific
bodies.

The Independent stream is very different, in that the ISE is meant to
exercise editorial discretion for publication and clearly could not publish
documentation of everything deployed even if it were made available to the
stream for publication.  To illustrate that impossibility, note that there
are approximately 2 million apps in the Apple app store, according to
published reports; just documenting which ones used what variants of HTTPS
*alone* would dwarf the output of the ISE (240 pages in 2017 outside of
April 1st RFCs, according to email from Adrian).

Eliot's pointer earlier today to the FYI series is instructive--the FYIs
reused the RFC series in part because the distribution mechanism was well
understood and well developed:

   There are several reasons why the FYIs are part of the larger RFC
   series of notes.  The formost reason is that the distribution
   mechanisms for RFCs are tried and true.  Anyone who can get an RFC,
   can automatically get an FYI.  More importantly, anyone who knows of
   the RFC series, can easily find out about the FYIs.
Given that there are many other distribution mechanisms available
today, we do not have to treat the RFC series as
the only place where such documentation can take place.

regards,

Ted

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 3 Jul 2018 14:14:21 -0400

Not only am I in complete agreement with everything that Brian has said
here, but I would like to add that section 5 makes this draft one of the
most cynical documents I have read in a very long time. It makes the
inference that one of the primary reasons people publish
non-standards-track RFCs is to DELIBERATELY TAKE ADVANTAGE of any confusion
in the reader’s mind regarding the status of RFCs. By measuring “success”
as a reduction in the rate of non-standards-track documents once they’re no
longer labelled “RFC”s, the draft implies that the current rate of
non-standards-track RFCs is ONLY because they are being published as
“RFC"s. I find this cynical in the extreme, and insulting to current and
past non-standards-track RFC authors, and would object to any such
“experiment” just on those grounds.

Cheers,
Andy

---

From: Ted Hardie <[email-masked]>
Date: Tue, 3 Jul 2018 11:28:57 -0700

On Tue, Jul 3, 2018 at 11:14 AM, Andrew G. Malis <[email-masked]> wrote:

> in the reader’s mind regarding the status of RFCs.
>

That's not quite what the document says.  Here's the text:

   One hypothesis this experiment proposes to test is the degree to
   which the "RFC" label motivates authors seeking publication.  Though
   numbers are unlikely to provide a clear answer when so much of the
   problem is subjective, measuring the rate of submission and
   publication for affected documents could provide some insight.

There are a variety of reasons that could go into that motivation; in some
companies, as an example, publishing an RFC is rewarded financially in a
way that an internal technical report is not.

Joel has written on some of the difficulties in working out what to measure
here, and I appreciate any consideration you can give to that; it's not an
easy problem, but if there is no data to work from forward motion on the
basis of anything other than impressionistic anecdotes will be difficult.

regards,

Ted

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 3 Jul 2018 14:46:15 -0400

Ted,

That is true, but the rest of the draft talks about the "common perception
of the significance of the "RFC” label”, how “misconceptions about the
significance of publication as an RFC is [sic] commonplace.”, and how
“there are numerous examples of RFCs that make an honest representation of
their status”, meaning there are some that do not. There is also "For the
examples given, a principled reason for publication is well justified.”,
again implying that there are unprincipled reasons for publication as well.

I just just adding 1+1 to make 2.

Regarding your example of authors being financially rewarded for RFC
publication, I’m sure that would extend to any IETF publication type if the
IETF were to run this experiment, thus this particular publication
motivation would be unaffected by the proposed experiment.

Cheers,
Andy

On Tue, Jul 3, 2018 at 2:28 PM, Ted Hardie <[email-masked]> wrote:

> Joel has written on some of the difficulties in working out what to
> measure here, and I appreciate any consideration you can give to that; it's
> not an easy problem, but if there is no data to work from forward motion on
> the basis of anything other than impressionistic anecdotes will be
> difficult.
>
> regards,
>
> Ted
>
>

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 3 Jul 2018 14:50:36 -0400

On Tue, Jul 3, 2018 at 2:28 PM, Ted Hardie <[email-masked]> wrote:
>> inference that one of the primary reasons people publish non-standards-track
>> RFCs is to DELIBERATELY TAKE ADVANTAGE of any confusion in the reader’s mind
>> regarding the status of RFCs.
>
>

Section 4 talks about the meaning of RFC, but this is written as the
meaning to the author of this draft perhaps?  Here's the text and the
rest of the section follows suit with the authors idea of what RFC
means without stating it:

   Reserving the RFC Series for the publication of products of the
   Internet Standards Process would ensure clarity about what "RFC"
   means.

RFC expands to Request For Comments.  I don't see why we wouldn't want
to receive comments on experimental or informational drafts as well as
for ISE, IRTF, and IAB stream documents.  Comments on ISE or IRTF RFCs
could lead to additional work that may generalize work for broader
applicability (vendor specific, narrow use case supported, crypto
algorithms, applications of research to develop supporting protocols,
etc.).  The author seems to take RFC to mean blessed standard.  That's
an argument for using STD in addition to RFC and making that work to
solve the problem.  The idea behind RFCs is important to collaborative
work and the work style of the IETF.

STD hasn't been effective as many don't bother to take the time to
change the status on documents.  Put a concerted effort around that
with a design team and I think we'd have a valuable experiment.
Additionally, why don't we make it easier to publish documents as STD
from the start?  That's a key difference with BCP, there is no
separate step to get this designation.  If there are 2 interoperable
implementations at time of publication, why not have an STD assigned?
Errata can still be verified.  Right now, it runs through the
publication process again and errata is included, this adds extra work
that may be part of the reason this is not used more.  People have
also pushed back on doing this forklift to help with perception.  It
seems it could have the desired impact to meet the stated goals in
this draft.

Responding here as I also agree there is bias in the document.

In section 5, why is the number of documents published important to
this experiment if the objective is about clarity on streams?  These
seem to be different objectives.

Best regards,
Kathleen

> companies, as an example, publishing an RFC is rewarded financially in a way
> that an internal technical report is not.
>

-- 

Best regards,
Kathleen

---

From: Bob Hinden <[email-masked]>
Date: Tue, 3 Jul 2018 12:06:03 -0700

Ted,

> On Jul 3, 2018, at 9:14 AM, Ted Hardie <[email-masked]> wrote:
> 
> On Tue, Jul 3, 2018 at 1:14 AM, Brian E Carpenter <[email-masked]> wrote:
> Isn't it *always* better to document something which is deployed, whether it's good
> or bad?
> 
> I believe that a number of your questions are ill-formed, and that the one I highlight above illustrates this problem best.  The question is not "should X, which is deployed, be documented?", but "Should the RFC Series contain the documentation for X, which is deployed?”.

My answer to this question is emphatically YES.  It is useful the Internet to publish deployed protocols in the RFC series.

I also note that in a number of cases publishing an RFC for a deployed protocol has led to that work being brought into the IETF and a new IETF standard version was produced.  I suspect that if this proposal goes forward, we will see less of that.  No one will know what an FOOnnn means.   Isn’t our real goal of our work to foster Interoperability?   Or do we only care about interoperability of protocols on the IETF stream?

What are we trying to do here?  Are we trying to make the “tent” smaller, or make our “tent” as big as possible?   This effort seems like the former.   I had thought we were also trying (via IASA 2.0) to make it easier to get more support for the IETF.  Moving work to other series that the larger community has never seen before, will have exactly the opposite effect.

Bob

> 
> If X was specified by the IETF, then the answer is yes, it should be contained in the IETF stream of the RFC Series.  In the small number of cases where it was specified by the IRTF, then it should be contained by the IRTF stream of the RFC Series.  The IAB stream could also theoretically produce an protocol specification that was deployed (though current practice is to bring any work of that type back into the IETF).  Those streams are, in other words, set aside as publication outlets for specific bodies.
> 
> The Independent stream is very different, in that the ISE is meant to exercise editorial discretion for publication and clearly could not publish documentation of everything deployed even if it were made available to the stream for publication.  To illustrate that impossibility, note that there are approximately 2 million apps in the Apple app store, according to published reports; just documenting which ones used what variants of HTTPS *alone* would dwarf the output of the ISE (240 pages in 2017 outside of April 1st RFCs, according to email from Adrian).
> 
> Eliot's pointer earlier today to the FYI series is instructive--the FYIs reused the RFC series in part because the distribution mechanism was well understood and well developed:
> Given that there are many other distribution mechanisms available today, we do not have to treat the RFC series as

---

From: Ted Hardie <[email-masked]>
Date: Tue, 3 Jul 2018 12:48:17 -0700

On Tue, Jul 3, 2018 at 11:46 AM, Andrew G. Malis <[email-masked]> wrote:

> IETF were to run this experiment,
>

Do you believe it would also be true for IRTF publications if the
experiment were run?  ISE publications?

regards,

Ted

> thus this particular publication motivation would be unaffected by the
> proposed experiment.
>
> Cheers,
> Andy
>
>
>> There are a variety of reasons that could go into that motivation; in
>> some companies, as an example, publishing an RFC is rewarded financially in
>> a way that an internal technical report is not.
>>

---

From: Ted Hardie <[email-masked]>
Date: Tue, 3 Jul 2018 13:08:22 -0700

Hi Bob,

On Tue, Jul 3, 2018 at 12:06 PM, Bob Hinden <[email-masked]> wrote:

> > I believe that a number of your questions are ill-formed, and that the
> one I highlight above illustrates this problem best.  The question is not
> contain the documentation for X, which is deployed?”.
>
> My answer to this question is emphatically YES.  It is useful the Internet
> to publish deployed protocols in the RFC series.
>
> As I pointed out in my answer to Brian, that task is likely beyond our
means.  There are about double the number of apps in the collected Android
app stores as the number I cited for the Apple app store; documenting what
they use (even if it is shared for apps which appear in both) at a level
sufficient to create an interoperable implementation would require an
effort beyond than can work with the RFC Series' methods.

We current cite documents from multiple SDOs and certainly use documents
from a wide variety of sources.  The correct question for us, then, isn't
really:  "is the world a better place if this is documented", but "is the
world a better place if this is a documented as an RFC (as distinct from
somewhere else)?"  I think having an ISE who curates the independent series
represents our current answer that the number of times it should be
documented as an RFC is limited.

I think the question before this BoF, though, is different.  It's about
whether calling all the streams' outputs "RFCs" is serving us well.  As I
pointed out before, one question leading up to this was whether the IRTF
output being called RFCs is serving the academic community well.  That same
might be asked for the IAB stream.

Martin's draft puts a stake in the ground about which one would still be
called "RFC" if we ran the experiment of using other identifiers for the
streams.  We could certainly discuss what would happen if the IETF stream
output was called "Internet Standards" instead of his proposal, but that
didn't get documented in time for this BoF.

regards,

Ted

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 3 Jul 2018 13:42:37 -0700

> that publication channel seems to pose a risk.

Hi Brian,

I agree that many of the IS documents are security-related, but I
don't think that failing to publish them in as RFCs would have a
significant negative impact on security.

Looking back on the past 5 years or so of IS RFCs, the following
seem to be "crypto algorithms and the like" [0]

  RFC 7076: P6R's Secure Shell Public Key Subsystem
  RFC 7091: GOST R 34.10-2012: Digital Signature Algorithm
  RFC 7906: NSA's Cryptographic Message Syntax (CMS) Key Management
Attributes
  RFC 8235: Schnorr Non-interactive Zero-Knowledge Proof
  RFC 8236: J-PAKE: Password-Authenticated Key Exchange by Juggling

RFC 7076 and 7906 look to me to document protocols which are in use by
some organization but which are not in wide use.  This is one of the
4846-defined uses of the Stream (Informational publication of
vendor-specific protocols) but I don't think that the security of the
Internet would be materially harmed if they had not been published.
Neither is cited by any I-D or RFC.

RFCs 7906, 8235, and 8236 document algorithms that the IETF has not
shown significant interest in using.

- 7091 is a national signature algorithm, and the IETF has instead
  converged on ECDSA and EdDSA.
- 8235 is a ZK-proof system which isn't employed by any IETF
  protocol (there is one Informative reference in an individual
  I-D, but it's just an example).
- 8236 is a PAKE system, but not one that the CFRG is converging
  on, and again not used by any IETF protocol (cited as an
  example by the same I-D).

In my opinion, it would not be a material loss to the security of the
Internet if these technologies had not been published as RFCs.

I was actually a bit surprised at how few algorithm drafts had been
published recently. Going back a further, it seems to me that the
algorithm drafts fall into two main categories:

- Documenting algorithms which are in wide use but just didn't
  have a formal specification (e.g., SHA-1, Base-64).
- Documenting algorithms which the IETF didn't have much interest
  in and haven't seen much adoption (GOST, Brainpool, Rabbit,
  ARIA).

In the former case, I think we now have ample mechanisms for
publishing those documents without the ISE. Specifically, we now use
the CFRG both to review and to publish those documents (cf. X25519 and
ChaCha/Poly1305). Amusingly, I wasn't even aware that there *was* a
SHA-1 RFC; TLS, for instance, cites the FIPS rather than this RFC.

In the latter case, I don't think that having these documents
published in the RFC series significantly improves security over some
other stable reference, at least in part because the IETF would to
some extent prefer that they not be implemented in IETF protocols (as
opposed to the algorithms we do recommend).

Arguably, taking these algorithms out of the RFC Series makes matters
somewhat better  because it reduces some pressure to implement them in
libraries. I guess that might be a useful thing to say?

-Ekr

[0] Ignoring stuff like DMARC which seems to me to be in a different
category.

---

From: Bob Hinden <[email-masked]>
Date: Tue, 3 Jul 2018 15:02:52 -0700

Hi Ted,

> On Jul 3, 2018, at 1:08 PM, Ted Hardie <[email-masked]> wrote:
> 
> Hi Bob,
> 
> As I pointed out in my answer to Brian, that task is likely beyond our means.  There are about double the number of apps in the collected Android app stores as the number I cited for the Apple app store; documenting what they use (even if it is shared for apps which appear in both) at a level sufficient to create an interoperable implementation would require an effort beyond than can work with the RFC Series' methods.

I don’t see how we went from, for example, publishing the specs for NFS, to publishing documents for all Android and Apple app store applicaitons.  That doesn’t seem like something that is likely to happen.  Did someone submit millions of docs to the IS Editor?

> 
> We current cite documents from multiple SDOs and certainly use documents from a wide variety of sources.  The correct question for us, then, isn't really:  "is the world a better place if this is documented", but "is the world a better place if this is a documented as an RFC (as distinct from somewhere else)?"  I think having an ISE who curates the independent series represents our current answer that the number of times it should be documented as an RFC is limited.

In some cases it does make the world a better place (e.g, NFS protocols), and others it does not.   Given the number of documents on the IS stream, this doesn’t seem like a significant issue.  The ISE can easily reject documents that don’t fit into the series.

> 
> I think the question before this BoF, though, is different.  It's about whether calling all the streams' outputs "RFCs" is serving us well.  As I pointed out before, one question leading up to this was whether the IRTF output being called RFCs is serving the academic community well.  That same might be asked for the IAB stream.

I think is a more complicated question.   It also should include if the cost of fixing this perceived problem, is more or less the benefit gained.

> 
> Martin's draft puts a stake in the ground about which one would still be called "RFC" if we ran the experiment of using other identifiers for the streams.  We could certainly discuss what would happen if the IETF stream output was called "Internet Standards" instead of his proposal, but that didn't get documented in time for this BoF.

I think the BOF would have been better structured, if it had been limited to discussing to what the problem is, and if it is serious enough to fix.  Then we would be having a different discussion.

I note that RFC 5434 goes to great length to have a BOF focus on the problem, and not get into solutions.   I continue to be surprised that this was approved in it’s current form.

Bob

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 3 Jul 2018 15:15:39 -0700

On Tue, Jul 3, 2018 at 3:02 PM, Bob Hinden <[email-masked]> wrote:

> I note that RFC 5434 goes to great length to have a BOF focus on the
> problem, and not get into solutions.   I continue to be surprised that this
> was approved in it’s current form.
>

Hi Bob,

RFC 5434 is an Informational document, not a BCP or a PS. So, while it may
have valuable advice, it doesn't really bear on whether the IESG should
have approved the BOF. Incidentally, I had to look up the RFC to see its
status, which seems to illustrate how this  is a source of confusion.

-Ekr

Bob

---

From: Martin Thomson <[email-masked]>
Date: Wed, 4 Jul 2018 11:41:39 +1000

> proposed experiment might fix it.

I agree that it's a weak point, though we might have different
expectations about the degree of evidence required.  We seem to
disagree on the magnitude of the change, which means that we
consequently are unable to agree on evidence.

Stepping back a little, it seems that we are all collectively
comfortable with "document that describes protocol X" existing, no
matter the venue.  Indeed, that is generally of benefit.  Where the
disagreement arises is in the (sometimes emotional, judging by some of
the feedback here) connection to the label and what that label has
come to mean to different people.

"RFC" clearly means different things to different people.  We could
say "just a document about the Internet", and ignore the association
with the 50-year history of the series and the aspects of existing
documents that define our understanding of its significance.  Or we
could attempt to define the meaning more clearly.

The draft in question (somewhat ineptly) attempted to describe a
narrower definition than in our current understanding.  I personally
think that it corresponds more closely with how the series is
understood outside of the inner circle, but have no firm evidence of
that.  I also believe that greater clarity of purpose has benefits
beyond simply addressing the potential for misconception, as I
separately described, but I'm happy to engage further on the question
of defining what the label means first.

It's probably necessary to get past the misunderstanding that this
isn't attempting to suppress publications of non-standard things.  If
publishing cryptographic algorithms is considered important[1],
nothing would change other than the name.

[1] I don't believe it is, for all of the reasons ekr described.
Also, iacr.org is a superior venue in many ways.

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Jul 2018 21:56:23 -0400

I understand that your suggestion (proposal? idea?) is not attempting to 
suppress publication in any broad sense.

However, you are proposing making a change to existing practice.  You 
are proposing making this change to address a perceived problem.  You 
are proposing that the change is an experiment.

It seems to me that the onus is therefore upon you to clearly spell out 
what the problem is and how the proposed experiment relates to the problem.

You agree below that the problem is not well-described.
And it seems clear that the proposed measurements of the experiment do 
not relate to the problem that you are attempting to solve.
Arguing that "it is a small change, so we should do it anyway" does not 
seem to me to meet the minimum bar for making a change to IETF process. 
Even if we agreed that it was a small change.

Yours,
Joel

PS: Your text indicating that you consider some of the responses to be 
"emotional" seems to me to be irrelevant and inappropriate.

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 3 Jul 2018 21:01:56 -0500

> "emotional" seems to me to be irrelevant and inappropriate.

Is it?

If one party sees an other party as generating responses driven by emotion
(and thus, one presumes, not by reason), that seems highly indicative of a
failure to properly communicate, regardless of whether the responses are
actually driven by emotion.  We are unlikely to make progress if we are
talking past each other or getting annoyed by the other party's
"irrationality", and recognizing that there is a difference of perception
is a good start to resuming effective communication.

Given the above, I'm not sure what argument would support a claim of
inappropriateness.

Of course, I should probably disclaim that some of the responses on this
thread did seem (to me) to be at least partially driven by emotion; I must
acknowledge the possibility that I am also in error.

-Ben

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 3 Jul 2018 22:07:31 -0400

> On Jul 3, 2018, at 10:01 PM, Benjamin Kaduk <[email-masked]> wrote:
> 
When your starting point on a thread is a document written with bias, responses are bound to be emotional. Having a clear problem statement seems to be a very reasonable request for a starting point.

Kathleen

---

From: Alissa Cooper <[email-masked]>
Date: Tue, 3 Jul 2018 22:18:58 -0400

Hi Andy,

> On Jul 3, 2018, at 2:46 PM, Andrew G. Malis <[email-masked]> wrote:
> 
> Regarding your example of authors being financially rewarded for RFC publication, I’m sure that would extend to any IETF publication type if the IETF were to run this experiment,

Why are you sure? I’m curious because this aspect has come up in the IESG’s discussions of the proposed experiment, namely the notion that a document with the label “RFC” is or would be considered more valuable than a document with a different label. (FWIW, I agree with Bob’s view that “RFC” is a very strong brand.)

Thanks,
Alissa

> thus this particular publication motivation would be unaffected by the proposed experiment.
> 
> Cheers,
> Andy
> 
> 
> On Tue, Jul 3, 2018 at 2:28 PM, Ted Hardie <[email-masked] <mailto:[email-masked]>> wrote:
> On Tue, Jul 3, 2018 at 11:14 AM, Andrew G. Malis <[email-masked] <mailto:[email-masked]>> wrote:
> Not only am I in complete agreement with everything that Brian has said here, but I would like to add that section 5 makes this draft one of the most cynical documents I have read in a very long time. It makes the inference that one of the primary reasons people publish non-standards-track RFCs is to DELIBERATELY TAKE ADVANTAGE of any confusion in the reader’s mind regarding the status of RFCs.
> 
> There are a variety of reasons that could go into that motivation; in some companies, as an example, publishing an RFC is rewarded financially in a way that an internal technical report is not.  
> 
> Joel has written on some of the difficulties in working out what to measure here, and I appreciate any consideration you can give to that; it's not an easy problem, but if there is no data to work from forward motion on the basis of anything other than impressionistic anecdotes will be difficult.
> 
> regards,
> 
> Ted
> 
>

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 3 Jul 2018 19:25:28 -0700

On Tue, Jul 3, 2018 at 7:07 PM, Kathleen Moriarty <
[email-masked]> wrote:

> When your starting point on a thread is a document written with bias,
> responses are bound to be emotional. Having a clear problem statement seems
> to be a very reasonable request for a starting point.
>

I'm a little puzzled by the use of the term "bias" here.

I take Martin's document to be a piece of advocacy for something that ought
to happen. I wouldn't expect it to represent a neutral point of view and
I'm not sure why people would be upset that it didn't.

-Ekr

Kathleen

---

From: Kathleen Moriarty <[email-masked]>
Date: Tue, 3 Jul 2018 22:34:46 -0400

> On Jul 3, 2018, at 10:25 PM, Eric Rescorla <[email-masked]> wrote:
> 
> 
> 
>> On Tue, Jul 3, 2018 at 7:07 PM, Kathleen Moriarty <[email-masked]> wrote:
>> 
>> 
> I take Martin's document to be a piece of advocacy for something that ought to happen. I wouldn't expect it to represent a neutral point of view and I'm not sure why people would be upset that it didn't.
> 
I was responding to 3 people on the IESG complaining that responses from the community were emotional with an explanation, nothing more.

Kathleen

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Jul 2018 16:53:59 +1200

On 04/07/2018 04:14, Ted Hardie wrote:
...
> the only place where such documentation can take place.

That is true. This is why the ION series was kept separate from
the RFC series - because distribution-by-web-page seemed necessary
and sufficient for the purpose, at the time the ION series was
invented. Web pages weren't an option when the FYI series started.

However, I think it's rather orthogonal to the question of what
types of document are suitable for inclusion in the RFC series.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Jul 2018 17:05:50 +1200

> contain the documentation for X, which is deployed?".

Of course that's the final question. But before that there's another
question: what is the audience that we (for some value of "we" != "the IETF")
are trying to reach? In my opinion it's the entire technical
community, most of whom will never attend an IETF meeting or even
join an IETF mailing list. Who needs to know about some widely
deployed and totally obnoxious protocol that is different from
the IETF standard? Everybody deploying or operating a network
where this protocol might appear. In most cases, these people simply
will not care about the document's formal status or who approved
its publication. They will, however, be grateful for all such
documents being in a similar format and obtainable from the same
place.

However, I full accept that my litany of questions was careless
and probably incomplete. After all, my point is that we simply
haven't done a problem analysis, and here we are asked to debate
a point solution.
 
> stream for publication.

Indeed. But I think that the universe of potential submitters to the
Independent stream must do a lot of self-censorship, because the reality
is that most drafts that show up for review (i.e. have got past the
ISE's sniff test) are at least related to layers 2-4 or to an application
protocol that's recognisably general in nature.

   Brian

> To illustrate that impossibility, note that there

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Jul 2018 17:17:54 +1200

Eric,

Thanks for that substantive answer. To be frank I'd rather
leave the response to someone who understands more about
crypto deployment than I do. It does seem to me that for
national algorithms, they will be in the libraries anyway because
they will be mandatory procurement requirements. For algorithms
that are not in procurement requirements, it clearly doesn't matter
in the same way.

Regards
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Jul 2018 17:21:32 +1200

> status, which seems to illustrate how this  is a source of confusion.

iirc 5434 was published as an RFC after an early life as a random web
page, but not made a BCP because although it set community expectations
it just didn't feel like it should become a set of rules. Of course
the reason it was written was a history of too many noisy, unproductive
BOFs, especially ones with solutions presented before people agreed on
a problem.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 4 Jul 2018 17:27:31 +1200

> Why are you sure? I’m curious because this aspect has come up in the IESG’s discussions of the proposed experiment, namely the notion that a document with the label “RFC” is or would be considered…

Predicting how corporate management will react is always hard, but if I
was still in the job I had at a 3-letter company, I could imagine writing
something like this to the bosses:

'By the way, the IETF has just decided to rename some of the documents
in "RFC" series as "FOO" or "BAR" to reduce confusion about document
status. There will be no changes in the document review processes.
I suggest that for bonus calculations, we can ignore this administrative
change.'

But really, we can't predict.

   Brian

---

From: Eliot Lear <[email-masked]>
Date: Wed, 4 Jul 2018 08:34:13 +0200

Hi Ben,

Since you asked...

>> "emotional" seems to me to be irrelevant and inappropriate.

My view..,

With a  general statement, it certainly can be a form of poisoning the
well or tone policing, and should generally be avoided in
argumentation.  That's not to excuse appeal to emotion itself, but one
should be specific (e.g., "the above argument doesn't stand on its own
because it is an appeal to emotion").  Nor should we excuse sloppy
arguments.

It is, I think, also ok to point out that a debate is emotional, but
that point must be made without being pejorative.  To lay claim or imply
that all those opposing are arguing irrationally because they are
emotional is pejorative.  Alissa's note, I think, demonstrates how to
get that right.

> Given the above, I'm not sure what argument would support a claim of

I hope I have ;-)

Eliot

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Jul 2018 02:58:23 -0400

--On Wednesday, July 4, 2018 16:53 +1200 Brian E Carpenter
<[email-masked]> wrote:

>...
>> Given that there are many other distribution mechanisms
>> available today, we do not have to treat the RFC series as
> the RFC series - because distribution-by-web-page seemed
> necessary and sufficient for the purpose, at the time the ION
> series was invented. Web pages weren't an option when the FYI
> series started.

Another piece of that issue is that we've gone to some lengths
over the years to make sure the RFC Series has reasonably good
properties from an archival standpoint.  That is not true for
may, or most, of those "other distribution mechanisms" (noting
as a particular example the density of bad links on the IETF's
own web pages).  If I encounter some popular, but badly designed
and implemented, piece of protocol on the Internet, it is good
to be able to find a description of it, in a known format, and
to know that the description will be equally accessible some
years down the line. 

If the thing is a hazard, but out there, the Internet is, IMO,
still better off if "we" encourage documentation and make it
available, accompanied, if it is appropriate and worth the
effort, by Applicability Statements that explain why using it is
a Really Bad Idea, rather than pretending it does not exist and
hoping that, if we ignore it long enough, it will disappear.
>From that point of view, the right remedy for a terrible
protocol idea --especially one that is either widely-deployed or
that seems likely to get traction-- is publication alongside a
well-reasoned critique or denunciation, not attempts to suppress
documentation or discussion of it.

The Internet's technical community, via the RFC Editor, has long
felt some obligation to provide a consistent, curated,
collection of such protocol documentation, in addition to
documentation of its own work (standards-related or not) as a
service to the Internet.  

If we are really having a conversation about giving up that role
and responsibility and letting the RFC Editor Function publish
only those things the IETF either develops or likes, let's have
that conversation and not disguise it as a discussion of how we
label things.  I would hope that such a discussion would
consider the possibility that at least part of the reason the
IETF has been as successful and influential as it has been is
that we have felt, and demonstrated, that level of
responsibility for the protocols of the whole Internet, rather
than being just another standards body doing its own thing and
pushing its own work.  Lots of those out there and most of them
are easier to push work through than the IETF.

I would also hope that people who would prefer to eliminate or
severely constrain the Independent Stream would recognize that
"less workload for the IESG" should not be an objective.  Today,
the Independent Stream acts as a sort of safety valve for
documents the IETF does not find of sufficient interest and, by
using its own editorial review and selection process, avoids
putting that workload on the IESG and the IETF as a whole.
Eliminate that mechanism and the number of requests for AD
Sponsorship will, I think inevitably, go up and, in the absence
of that safety valve, it is likely that the number of appeals of
AD decisions to not sponsor documents will go up too, probably
followed by community insistence that every AD decision to
sponsor (or not sponsor) a document be well-documented with
reasoning, etc.   Certainly that does not reduce the workload on
the IESG.   Because every AD sponsored document goes through
IETF Last Call and IESG balloting, it would also increase the
workload on IETF participants as well.  I think "be careful what
you wish for" might be in order.

The comments above are rather different from worrying about
descriptions of particular applications on particular platforms.
In addition to the self-filtering that others have described, we
have almost never been in the business (either in the IETF or as
Independent Submissions) of developing or publishing
descriptions of actual applications, platform-specific or not.
We do mail protocols and formats but, at least AFAIK, have never
published a description of an MUA.  We do calendar protocols and
data formats, but don't do actual calendaring programs.  We do
PGP and S/MIME, and specify or publish the cryptographic
algorithms and procedures that underlie them, but have stayed
away from specifying (or publishing in the Independent Stream)
the UIs for those protocols.   So I see a discussion of the
number of applications on, e.g., the iPhone or iPad, as fairly
close to a strawman.

> However, I think it's rather orthogonal to the question of
> what types of document are suitable for inclusion in the RFC
> series.

Yes.  And it seems to me that even that question is rather
different from anything I see in the BOF proposal, which brings
me back to the question of what we are talking about.

best,
   john

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Jul 2018 03:13:58 -0400

--On Wednesday, July 4, 2018 17:27 +1200 Brian E Carpenter
<[email-masked]> wrote:

>> Why are you sure? I'm curious because this aspect has come
>> up in the IESG's discussions of the proposed experiment,
>> namely the notion that a document with the label "RFC" is
>> or would be considered more valuable than a document with a
>> different label. (FWIW, I agree with Bob's view that
>> "RFC" is a very strong brand.)
> 
> Predicting how corporate management will react is always hard,
> but if I was still in the job I had at a 3-letter company, I
> could imagine writing something like this to the bosses:
> 
> 'By the way, the IETF has just decided to rename some of the
> documents in "RFC" series as "FOO" or "BAR" to reduce
> confusion about document status. There will be no changes in
> the document review processes. I suggest that for bonus
> calculations, we can ignore this administrative change.'
> 

You think?  Let me predict anyway.  

There are several mechanisms out there called "impact scores".
There is also a lot of literature about why they are lousy
measures of what they allegedly measure, but they are wildly
popular among counters of beans, academic promotion committees,
etc., precisely because they provide quantitative,
interval-scale, measures that are easily compared.   By those
measures, the RFC Series does fairly well if only because there
are lots and lots of references around to technical standards
for the Internet.  In a way, they are the quantitative
reflection of "strong brand".  Now if one splits things off into
different pieces, maybe the scores for the RFC Series actually
go up because documents that might be important but are not
widely references are somewhere else.  However, if one examines
those other collections three years down the line, they are
likely to have scores in the "why should we bother with that or
support our employees in writing such things" range.  There may
be good (and bad) reasons for breaking things up, but at least
on that dimension, I'd predict negative outcomes for any new and
more narrowly focused series.

best,
   john

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 4 Jul 2018 05:40:31 -0700

On Tue, Jul 3, 2018 at 10:17 PM, Brian E Carpenter <
[email-masked]> wrote:

> crypto deployment than I do.

Fair enough.

It does seem to me that for
> they will be mandatory procurement requirements.

That's certainly an understandable prediction, but I haven't found it to be
true in practice. For instance, NSS doesn't implement GOST or SM, and to
the best of my knowledge neither does BoringSSL. And if we did decide to
implement those algorithms for some market reason, having them documented
in an RFC would not make our lives substantially easier.

Thanks,
-Ekr

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 5 Jul 2018 19:41:18 -0500

Hi Eliot,

> emotional is pejorative.  Alissa's note, I think, demonstrates how to

It does, especially so compared to mine.

> > Given the above, I'm not sure what argument would support a claim of

I think you have, yes.  Thanks!

-Ben

---

From: Kathleen Moriarty <[email-masked]>
Date: Fri, 6 Jul 2018 07:18:10 -0400

> On Jul 4, 2018, at 8:40 AM, Eric Rescorla <[email-masked]> wrote:
> 
> 
> 
>> On Tue, Jul 3, 2018 at 10:17 PM, Brian E Carpenter <[email-masked]> wrote:
>> Eric,
>> 
> That's certainly an understandable prediction, but I haven't found it to be true in practice. For instance, NSS doesn't implement GOST or SM, and to the best of my knowledge neither does BoringSSL. And if we did decide to implement those algorithms for some market reason, having them documented in an RFC would not make our lives substantially easier.

I work for Dell (EMC, VMWare, RSA, Pivotal) and we have needed to implement several national algorithms.  I verified this with the team that has done at least 3 of them and they used the RFCs to implement.

Best,
Kathleen

---

From: Eliot Lear <[email-masked]>
Date: Fri, 6 Jul 2018 13:38:35 +0200

Hi Kathleen:

On 06.07.18 13:18, Kathleen Moriarty wrote:
> I work for Dell (EMC, VMWare, RSA, Pivotal) and we have needed to
> implement several national algorithms.  I verified this with the team
> that has done at least 3 of them and they used the RFCs to implement.

That's good information.  It seems to me that we seem to have an overt
concern and a less overt concern.  The overt concern is that we are
committing resources to something that does not have community
consensus, for one reason or another.  The less-than-overt concern is
that the specifications are faulty or not well vetted.  Add to this the
RFC cachet that attaches to the work.  In the case of national
algorithms, I'm going to claim that the cachet doesn't matter: national
algorithms are going to get implemented where it is necessary for market
entrance, even if the label we attach is "MASSIVELY-STUPID-IDEA-0001".

In other words, this cut both ways in this discussion: we can use the
RFC label or some other label, the production costs are going to be the
same, and people will just implement when they need to.  On the other
hand, there isn't really brand dilution, and so who really cares if RFC
is used?

The one benefit of the ISE doing any of this is that we get to have a
say about the work.  That's not nothing.  In addition, the work can be
legible and freely available.  But, as discussed with EKR, other venues
can serve that function.

Eliot

---

From: Kathleen Moriarty <[email-masked]>
Date: Fri, 6 Jul 2018 08:57:50 -0400

I’d like to add that crypto algorithms are used in more than just TLS.  We should not be scoping use of these algorithms by a few libraries of one (very broadly used) protocol.

Kathleen 

> On Jul 6, 2018, at 7:38 AM, Eliot Lear <[email-masked]> wrote:
> 
> Hi Kathleen:
>

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 6 Jul 2018 06:01:41 -0700

On Fri, Jul 6, 2018 at 5:57 AM, Kathleen Moriarty <
[email-masked]> wrote:

> I’d like to add that crypto algorithms are used in more than just TLS.

Indeed. In fact, Firefox uses crypto algorithms for more than just TLS. And
in fact, libraries such as NSS, BoringSSL, and OpenSSL (despite the names)
are widely used to provide crypto for other protocols.

We should not be scoping use of these algorithms by a few libraries of one
> (very broadly used) protocol.
>

I don't think I'm suggesting otherwise. I'm merely reporting that the
question of RFC status does in fact come up in discussions of whether to
implement specific algorithms in crypto libraries and products which I am
familiar with.

-Ekr

---

From: Kathleen Moriarty <[email-masked]>
Date: Fri, 6 Jul 2018 09:05:29 -0400

> On Jul 6, 2018, at 9:01 AM, Eric Rescorla <[email-masked]> wrote:
> 
>> On Fri, Jul 6, 2018 at 5:57 AM, Kathleen Moriarty <[email-masked]> wrote:
> Indeed. In fact, Firefox uses crypto algorithms for more than just TLS. And in fact, libraries such as NSS, BoringSSL, and OpenSSL (despite the names) are widely used to provide crypto for other protocols.
> 
> 
>> We should not be scoping use of these algorithms by a few libraries of one (very broadly used) protocol.
> 
> I don't think I'm suggesting otherwise. I'm merely reporting that the question of RFC status does in fact come up in discussions of whether to implement specific algorithms in crypto libraries and products which I am familiar with.

Sure, understandable.  For those not in this space, having insight to places where national algorithms are implemented may factor into their thoughts on the usefulness of these algorithms being published in the RFC series.  We are just one company headquartered in the US, I’m sure others implement them as well.  

Best,
Kathleen

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Jul 2018 09:33:57 -0400

--On Friday, July 6, 2018 13:38 +0200 Eliot Lear
<lear=[email-masked]> wrote:

>...
> The one benefit of the ISE doing any of this is that we get to
> have a say about the work.  That's not nothing.  In
> addition, the work can be legible and freely available.  But,
> as discussed with EKR, other venues can serve that function.

There is one other benefit that I think we should not lose sight
of.   Over the years (including well before there was an IETF)
"we" have presented the RFC Series as the go-to place for
technical specifications about what is used on the Internet.  I
suggest that role is one of the reasons the RFC brand is as
strong as it is.  To the extent to which specifications are
adopted, deployed widely, and used (even if only within
countries where they are mandated) but that are not included in
the RFC series, that reputation and brand suffers. 

In a way, being explicit that no specification will be published
in the RFC Series unless it is approved of (favored) by the IETF
(or the IESG) is an advantage because it explicitly drops the
"what is used on the Internet" assumption for "what the IETF
approves of".  But that is likely to reduce the value of the
brand even more, especially for organizations (and governments)
for which the IETF's endorsement is less important that market
pressures, their own evaluations, etc.   

>From that perspective, your "get to have a say" observation is
even more important because it both preserves the IETF's
reputation for making decisions based on technical merit and
openness to ideas not invented here.   The best way to preserve
and support that reputation is to go ahead and publish things we
don't like and explain why we don't like them.  For the latter,
"the IETF does not recommend this because XYZ was deemed
superior because" is far stronger than "the IETF does not
recommend this because, when sniffed, it let off a bad odor" but
either is better than ignoring a spec that is widely-deployed
and used because we don't like it.

Let me give an example that lies a bit outside the crypto realm
and that may therefore be easier for the rest of us to
understand.  Suppose the IETF adopts a protocol that is
carefully designed for maximum privacy.   Now suppose that
something comes along that is faster, lighter-weight, supported
or mandated by some governments but that protects privacy from
everyone but those governments (but leaves information
completely exposed to them).    We can certainly ignore the
latter as just bad news but I contend we protect our reputation
and the privacy-safety of users, and make the Internet better,
far more by publishing its specs, explaining the vulnerabilities
it creates, and recommending against it unless one is actually
required to use it than by ignoring it.

Does that require extra work?  Of course it does, but pushing it
off into the Independent Submission process mitigates some of
that.  Maybe we can't do it or need to take shortcuts with the
level of detail in our explanations.  But that isn't without
costs because every further step we take toward ignoring such
specifications brings us a step further away from the image of
an engineering organization that produces standards when that is
appropriate and toward just another standards body that arrives
at its conclusions by obscure means and then pushes them.
Precisely because we have never wanted to (and cannot) claim
credibility on the basis of the companies, established national
bodies, or governments who agree to make the standards, that
engineering orientation and commitment to the welfare and
improvement of the Internet is all we have to stand on.

best,
    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 06 Jul 2018 09:53:33 -0400

--On Friday, July 6, 2018 06:01 -0700 Eric Rescorla
<[email-masked]> wrote:

>> We should not be scoping use of these algorithms by a few
>> libraries of one (very broadly used) protocol.
 
> I don't think I'm suggesting otherwise. I'm merely reporting
> that the question of RFC status does in fact come up in
> discussions of whether to implement specific algorithms in
> crypto libraries and products which I am familiar with.

Sure.  As well it should.   But, if the people participating in
the discussions and doing the evaluation have any sense and
knowledge at all (as Mozilla and Dell/EMC certainly do and would
even without you or Kathleen), then "RFC status" is precisely
what should be examined.  That certainly includes the difference
between Experimental and Informational.  If we took maturity
levels as seriously as RFC 2026 anticipated, it would include
the difference between Proposed and Full Standards.  And, if we
used Applicability Statements as often as some of us thought we
should (or at least more often than near-zero), they would be
part of the picture too.  In most cases, it should include the
difference between, e.g., a WG Informational document and an
Informational Document published by the ISE for the information
of the community with no endorsement by anyone other than the
authors (not even endorsement by the ISE).

However "question of RFC status" and "discussions of whether to
implement" are about people who are actually reading the
documents.  It seems to me that whether a document is flagged
with one series code or label or another is likely to make very
little difference to them because they are reading it anyway.
The people who are arguably protected by label changes are those
who are not knowledgeably "discussing" or "evaluating"
specifications but who are relying entirely on marketing claims
or government assertions.   Many years ago, a colleague of my
was fond of quoting his father as saying that, while one can
reasonably aspire to make a system fool proof, making one d*mn
fool proof was probably an unreasonable goal.

best,
    john

---

From: Kathleen Moriarty <[email-masked]>
Date: Fri, 6 Jul 2018 09:59:54 -0400

On Fri, Jul 6, 2018 at 9:53 AM, John C Klensin <[email-masked]> wrote:
> what should be examined.

Our teams implementing fully understand the difference without any
explanation needed.  When I asked the question to find out what
national algorithms we had implemented, I received a very thorough
response including the algorithms, RFC#s and an explanation in short
order.  They monitor various RFCs and I at times get educated
questions on status in terms of publication and path for publication.

Best,
Kathleen

> That certainly includes the difference

-- 

Best regards,
Kathleen

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 6 Jul 2018 07:33:46 -0700

On Fri, Jul 6, 2018 at 6:33 AM, John C Klensin <[email-masked]> wrote:

> technical specifications about what is used on the Internet.

Hmm... This seems to me to be true only if one has a pretty narrow
definition of "on the Internet". Here are a few examples of things which
get used on the Internet on a regular basis that aren't RFCs:

- HTML (WHATWG and/or W3C)
- JavaScript (TC39)
- H.264 (MPEG)
- JPEG
- AES, P-256 (NIST)
- Bittorrent
- Tor

I spent a bunch of time trying to think of a characterization of the role
of the RFC series (or the IETF's) role that was succinct, but in truth I
think we're part of a broad coaltion of people publishing Internet protocol
specificationswith somewhat fuzzily defined boundaries between the areas of
work.

> required to use it than by ignoring it.

This hypothetical contains a number of things "we" might do (publish the
spec, critique it, recommend against it). I don't think it's actually a
particular service to the world for us to publish the spec as long as it's
readily found by other means. It's not clear to me how that helps. I
certainly think critiquing it and recommending against it are good ideas
(though depending on the technology, perhaps making something that had the
good properties without the bad would be even better). It's not clear to me
that having those critiques in the RFC series is that helpful, though
having the IETF recommend against the protocol (whether that recommendation
was in an RFC or not) might well be.

With that said, this hypothetical isn't *that* far off the mark. Tor is a
privacy protocol better than anything in the IETF (though to the best of my
knowledge it doesn't have the negative properties of the protocol in your
hypothetical). Its protocol is mostly defined by the implementation
although there's a not-particularly clear spec. There are a pile of papers
describing potential attacks. To the best of my knowledge, none of this is
in the RFC series. From my perspective as a protocol implementer:

- Having a good spec published would be valuable. Having it in the RFC
series would not be particularly helpful, though it would not be harmful [0]
- Having an IETF standard would probably be helpful depending on whether it
improved Tor. It would likely lead to a clearer spec, which would be good.
- Having the various attacks/critiques published in the RFC series would
not be helpful. Currently, they are in peer-reviewed venues (e.g., IEEE
Oakland) which do a better job of review than we do. Moreover, the academic
community is where the conversation about the security of this type of
protocol takes place.

So, in this particular example, I don't actually think that the RFC series
-- as opposed to the IETF -- is bringing that much to the party,

-Ekr

[0] One might argue that the process of turning the spec into an RFC would
make it clearer. I'm note sure that's true; the spec is at about -00 level
now. But in any case, if we think it would be good for specs to be clearer,
we could pay to have them copy-edited inside or outside the RFC series.

---

From: Eric Rescorla <[email-masked]>
Date: Fri, 6 Jul 2018 07:39:56 -0700

> - HTML (WHATWG and/or W3C)

I should have footnoted this: there is an HTML spec in the RFC series
[RFC1866] but it's prehistoric and if you try to use it on the Web you're
going to have a bad day.

-Ekr

- JavaScript (TC39)
> - Having an IETF standard would probably be helpful depending on whether
> it improved Tor. It would likely lead to a clearer spec, which would be
> good.

---

From: Ted Hardie <[email-masked]>
Date: Fri, 6 Jul 2018 15:56:32 +0000

On Wed, Jul 4, 2018 at 7:13 AM, John C Klensin <[email-masked]> wrote:

> interval-scale, measures that are easily compared.

Just to add to this that there may be more than one impact score associated
with the same series.  As was noted earlier, the impact of the RFC Series
is not that high in academic circles, and that has a follow-on impact on
the IRTF.

Ted

> By those

---

From: Richard Barnes <[email-masked]>
Date: Fri, 6 Jul 2018 17:04:23 -0400

On Fri, Jul 6, 2018 at 10:40 AM Eric Rescorla <[email-masked]> wrote:

> [RFC1866] but it's prehistoric
>

I don't **think** that's one of the labels we have today.  But there's a
lot of them, so I might be wrong.

--Richard

> and if you try to use it on the Web you're going to have a bad day.
>
> -Ekr
>
>
> - JavaScript (TC39)
>> - H.264 (MPEG)
>> - JPEG
>> - AES, P-256 (NIST)
>> - Bittorrent
>> - Tor
>>
>> So, in this particular example, I don't actually think that the RFC
>> series -- as opposed to the IETF -- is bringing that much to the party,
>>
>> -Ekr
>>
>> [0] One might argue that the process of turning the spec into an RFC
>> would make it clearer. I'm note sure that's true; the spec is at about -00
>> level now. But in any case, if we think it would be good for specs to be
>> clearer, we could pay to have them copy-edited inside or outside the RFC
>> series.
>>
>

---

From: John C Klensin <[email-masked]>
Date: Sat, 07 Jul 2018 08:19:32 -0400

--On Friday, July 6, 2018 07:33 -0700 Eric Rescorla
<[email-masked]> wrote:

>> There is one other benefit that I think we should not lose
>> sight of.   Over the years (including well before there was
>> an IETF) "we" have presented the RFC Series as the go-to
>> place for technical specifications about what is used on the
>> Internet.
> 
> 
> Hmm... This seems to me to be true only if one has a pretty
> narrow definition of "on the Internet". Here are a few
> examples of things which get used on the Internet on a regular
> basis that aren't RFCs:

Sure.   I would agree if you had just said that presentation is
something of a myth, but suggest it is a myth that works in our
favor.   We definitely don't have a monopoly.  If we ever did,
it probably hasn't been since the late 70s or early 80s.   In
particular...

> - HTML (WHATWG and/or W3C)

As you mention, there was an early version (or two?) of HTML
that we did publish in the RFC series.   Then, after W3C decided
it wanted to be in the standards business after all, there was a
fairly explicit agreement and handoff.

> - JavaScript (TC39)

And ECMA (or whatever it is called now), and several other
places.

> - JPEG

Content types, which we have rather explicitly said we don't
want beyond registering their names and pointers to their
descriptions.

> - AES, P-256 (NIST)

Yes.  And incorporated into various of our protocols by
reference (and referenced from them).

> - Tor

Both, IIR, things that, at one point or another, we decided we
didn't want.

You left at least one thing off your list that is arguably as
important to today's Internet as several of those put together,
which is the internal protocols and mechanisms used by various
CDNs and other providers to 

> I spent a bunch of time trying to think of a characterization
> of the role of the RFC series (or the IETF's) role that was
> succinct, but in truth I think we're part of a broad coaltion
> of people publishing Internet protocol specificationswith
> somewhat fuzzily defined boundaries between the areas of work.

I would agree with that.   I don't think it changes my
underlying point any more than your list above does, but YMMD.

>> something comes along that is faster, lighter-weight,
>> supported or mandated by some governments but that protects
>> privacy from everyone but those governments (but leaves
>> information completely exposed to them).    We can certainly
>> ignore the latter as just bad news but I contend we protect
>> our reputation and the privacy-safety of users, and make the
>> Internet better, far more by publishing its specs, explaining
>> the vulnerabilities it creates, and recommending against it
>> unless one is actually required to use it than by ignoring it.

> This hypothetical contains a number of things "we" might do
> (publish the spec, critique it, recommend against it). I don't
> think it's actually a particular service to the world for us
> to publish the spec as long as it's readily found by other
> means. It's not clear to me how that helps.

I think that depends a lot on the same issues that drove my
earlier note on which you asked for more details (haven't
forgotten that, just haven't been able to make time yet).   "Not
actually a particular service" implies that everyone who might
be interested can follow links with ease, that there are no
advantages to a uniform format and style, that either one
doesn't care about these things for the long term (e.g., for
understanding the history of various protocol developments and
choices some years on) or that links never go bad, etc.  I think
those things are important and have learned to be very
conservative (largely in the "conservation" sense) about them.
YMMD.

> I certainly think
> critiquing it and recommending against it are good ideas
> (though depending on the technology, perhaps making something
> that had the good properties without the bad would be even
> better).

If the bad stuff is already in wide use and especially if the
newer version is better but not obviously and overwhelmingly
better, both may be needed.     More important, "X is better
than Y" is a much more absolute statement than many of the
design choices we face.  In many situations, we are less likely
to have "the good properties without the bad" as an option but
find ourselves dealing with tradeoffs in which optimizing one
thing pessimizes another and we have to make tradeoffs.  I
suggest that, at least in recent years, the IETF has been fairly
poor about being explicit about those choices, preferring to
either leave the information out in the hope that its choices
will be more accepted or just not wanting to take the time to
document them.   I think that is unfortunate for those who need
to make choices in the marketplace, for our reputation as an
engineering-oriented body that is consistently trying to make
good overall choices for the Internet, and for the academic
considerations you mention in another context below.  Again,
YMMD.

> It's not clear to me that having those critiques in
> the RFC series is that helpful, though having the IETF
> recommend against the protocol (whether that recommendation

There are, as far as I know, only two ways to recommend against
a protocol.   One is to do so on the basis of some clear
critique the other is to say some form of "you shouldn't use
this because we are the IETF, we know more about these things
than you do, and we are all-seeing and all-knowing".  To me, the
latter is even worse that "we recommend this and against
everything else because we are a collection of government
representatives and we not only know more than you do but can
make you".  And, btw, in case it hasn't be clear, I think
critiques of whatever the IETF decides on are at least as
important as IETF critiques as everything else.  If nothing
else, if they are done well, they keep us honest and help
clarify the tradeoffs we are making.

> With that said, this hypothetical isn't *that* far off the
> mark. Tor is a privacy protocol better than anything in the
> IETF (though to the best of my knowledge it doesn't have the
> negative properties of the protocol in your hypothetical). Its
> protocol is mostly defined by the implementation although
> there's a not-particularly clear spec. There are a pile of
> papers describing potential attacks. To the best of my
> knowledge, none of this is in the RFC series. From my
> perspective as a protocol implementer:
> 
> - Having a good spec published would be valuable. Having it in
> the RFC series would not be particularly helpful, though it
> would not be harmful [0] - Having an IETF standard would
> probably be helpful depending on whether it improved Tor. It
> would likely lead to a clearer spec, which would be good. -
> Having the various attacks/critiques published in the RFC
> series would not be helpful. Currently, they are in
> peer-reviewed venues (e.g., IEEE Oakland) which do a better
> job of review than we do. Moreover, the academic community is
> where the conversation about the security of this type of
> protocol takes place.

And should therefore probably stay there.  My arguments would be
more relevant if the IETF decided to develop and standardize a
TORjunior and TORbis protocol and push it instead of the
original.  I also suggest that the decision to use tor rather
than more traditional routing is also something that involves
tradeoffs, something of which at least parts of that research
community are aware, but which appear to be under-documented.
Probably that is ok in their context, but I'd hope that, when we
produce standards, we don't set such things aside.  

Incidentally, if you believe that other peer-reviewed venues,
such as IEEE Oakland in your example, consistently do a better
job of review than we do, that is a fairly good argument, not
for changing the labels on RFCs or constraining the ISE, but for
just stopping the efforts to make the RFC series more
academically credible and for abandoning the IRTF entirely.  If
syntax models like HTML are better done elsewhere than in the
IETF, we should recognize that and hand them off (as we have,
explicitly).  If specialized routing models like tor are better
explored and developed in parts of the academic community, then
we should encourage, not, resist that.   From the other side, if
some protocol is developed and widely deployed in industry --for
their own reasons rather than as a way of doing an end run
around IETF review while seeking IETF endorsement anyway--
decides that a particular protocol or syntax is better further
developed and curated by the IENT rather than continuing as an
industry-proprietary mechanism, then I think we should embrace
the transfer if it is within our normal scope.    But, if you
believe our review mechanisms are not effective, or even just
not as effective as others that are out there, it is time to ask
much deeper questions that ones about the labels on documents.

best,
    john

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 7 Jul 2018 20:17:37 -0700

> favor

Well, I'm not sure I agree that it's something "we" have generally
presented (though I'm sure that some people have said this at
one time or another.) But in any case, it sounds like we agree
that it's not true, and I don't think most people working
directly with protocol standards are likely to be confused
on this point.

> We definitely don't have a monopoly.  If we ever did,

> fairly explicit agreement and handoff.

Yes.

> And ECMA (or whatever it is called now), and several other

TC39 is an ECMA technical committee.

> want beyond registering their names and pointers to their

Well, we standardized OPUS, RFC 2083 describes OPUS,
and we currently have a videocodec WG, so I'm not sure
I agree we don't want media formats.

> Both, IIR, things that, at one point or another, we decided we

Potentially, though I don't recall this.

> CDNs and other providers to

It wasn't intended to be an exhaustive list.

> conservative (largely in the "conservation" sense) about them.

Indeed. I don't think there are *no* advantages, but I think
they're pretty modest. Moreover, the uniformity is pretty
superficial (7-bit ASCII, etc.) One need only look at the
QUIC specs to see an example of this, with TLS-style
PDU definitions nestled next to ASCII art bit diagrams.

> than you do, and we are all-seeing and all-knowing".

I'm not sure what you're responding to here. My position isn't
that the documents ought not to provide detailed critiques,
but rather that it's not obviously important that they be
in the RFC series. It's not clear to me that that's a relevant
factor.

> academically credible and for abandoning the IRTF entirely.

Well, I wasn't offering it as an argument for anything in particular,
just stating my opinion -- albeit one supported by considerable
evidence -- about where the relevant work in critiquing a certain
class of protocols is published. Those critiques then get fed
into the IETF process. A good example of this is the development
of TLS 1.3, which was done in close collaboration with the academic
community and led to quite a few academic papers which then influenced
the design of the protocol. Appendix E of the TLS draft provides
citations to quite a bit of this work:

https://tools.ietf.org/html/draft-ietf-tls-tls13-28#appendix-E

With that said, at least in my field, I think the IRTF (specifically
the CFRG) is serving an important function in terms of liaising
between the academic community and the IETF protocol development
community, so I don't think it would be good to shut down that
piece of the IRTF. Despite that, the academics who do cryptography
tend to publish that work in academic conferences and then
later it may get transcribed into an RFC in IRTF.

> If
> we should encourage, not, resist that.

I fear you may be misunderstanind me. For context, here's what
I said:

My point is not that things like Tor are better developed in the
academic community but rather that the development of protocol
standards -- at least in the area of communication security --
involves a number of different types of technical work
(protocol specification, analysis, verification, prototyping, etc.
That work is done by a number of communities that overlap to
some extent, and each community has its conventional venues for
sharing its work [0]. So, when those communities collaborate,
it involves publication in multiple venues.

To return to the TLS 1.3 example:

- The protocol development itself happened in conventional IETF
  venues (mailing lists, meetings, etc.) but also on Github.
- Implementation coordination was done over Slack, Github,
  hackathons, etc.
- Analysis was done in academic-type venues (eprint, conferences,
  workshops, etc.)

The RFC series (and also I-Ds) have an important role to play here,
but it also exists in conversation with a lot of other publication
venues, and I think that's healthy.

-Ekr

[0] To some extent this is just path dependence, but it's also
the case that different kinds of work demand different kinds of
tools. To give just one example, cryptographic papers are
full of symbols and almost invariably written in LaTeX, and
for obvious reasons, cryptographers aren't excited about
using the IETF's tools.