Subject: [Rfcplusplus] Outlining the experiment

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jul 2018 17:32:39 +1000

It's been good to see the level of engagement on this list.  I've been
reading with some interest.

I've submitted a draft that attempts to summarize the proposed
experiment: https://datatracker.ietf.org/doc/draft-thomson-rfcplusplus-label/
[0]

I had hoped to get this out sooner, but the urgency of other tasks
meant that I barely met the draft submission deadline, and it's very
rough still.  Hopefully that draft clarifies some of the questions
people had about the scope of the proposed experiment.

Obviously, this is just my view, and I don't think that everything in
there is correct (Adrian already found one hole).  As Ted said, the
IAB are generally of the view that this is a topic worth discussing,
but as a group we don't agree on a particular solution.  This is, more
than anything else, a convenient strawman.

However, based on the feedback thus far, it seems like it might be
good to examine some of the more fundamental issues:

1. What is the problem?

I think that the problem is well understood.  The common perception of
"RFC" encompasses a far simpler notion than the reality.

The debate seems to be centred more on the question of whether this -
of the multitude of problems that we might tackle - is the one that is
worth addressing.  Several people have suggested that the broader
issues might be tackled more comprehensively.  I'm personally not
opposed to that, though the thought was to keep the focus narrow.

Brian cites the ISD proposal [1], which is much more ambitious, but
it's been a few years since anything like this has been attempted,
maybe there is enough pent-up demand for change that we could find a
more comprehensive set of changes.

2. Why this problem?

It's a relatively small problem, with a (proposed) small solution.

There are some potential advantages that might be later realized from
a looser coupling between different sources.  There are certain
uniform expectations about publication as RFC that mean that certain
streams tend to affect the outcome of others, sometimes in ways that
are less than ideal.

For instance, we see that all RFCs tend to have a security
considerations section, even if the notion is a little ridiculous (BCP
78 says as much).  Security considerations make sense for standards,
but less so for other types of document.  Besides, I think that they
exist solely for the comfort of SEC ADs now, we've a pretty strongly
enculturated understanding of the importance of security in our
designs [2].

Similarly, one of the ideas floated early in this process was that
this might allow the RFC 5742 conflict review process to be removed.
That was intended to allow documents to proceed with fewer explicit
controls.  Feedback from the ISE and ISEB indicated that they found
this process extremely valuable and the IESG don't see it as a burden,
so that's unlikely to be something that comes out of this, but I
mention it to illustrate the potential.

3. Why now?

Why not?  I don't really have a good answer for this, other than the
fact that there seemed to be sufficient interest in doing something
and that this was a reasonable candidate.

4. This is so unscientific!

Yeah.  That's the nature of the problem.  Alissa articulated this
neatly in her email, it's a human problem.

We should definitely explore deeper, because this is small, but it
isn't simple.  My personal view though is that if this gets to the
point where we're employing market research companies, something has
gone seriously wrong.  Going that deep, or dragging in a laundry list
of other unsolved problems, might be a good way to ensure continuation
of the status quo.

5. Why not erect a bigger sign?

The current boilerplate in RFCs could be improved, but - as the draft
says - that doesn't really get to the crux of the issue.

Even if there was a huge banner on every RFC that wasn't standards
track that said "DON'T IMPLEMENT OR DEPLOY THIS PROTOCOL.
IMPLEMENTING THIS PROTOCOL WILL HURT THE INTERNET", it won't stop
someone from receiving conflicting advice.

At the risk of being the first to invoke what will eventually be
verboten under the terms of Godwin's Law, RFC 7974 is the best example
I have of this.  It bears a warning nearly that strong and yet I have
received or witnessed strong endorsements of its virtues on several
occasions.

[0] You can find a more readable and updated version here:
https://martinthomson.github.io/is-the-future/draft-thomson-rfcplusplus-label.html
[1] https://tools.ietf.org/html/draft-ietf-newtrk-sample-isd-00
[2] This is as it should be, alongside all the other things we might
consider as constraints on designs: privacy, deployment, operations,
usability, internationalization, human rights, economic effects,
intellectual property, etc...

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 3 Jul 2018 08:46:07 -0400

I have several concerns with the draft as presented.

1) The draft seems to imply that confusion about status is principally 
or significantly a matter of labels, and a matter of Standards Track vs 
Information or Experimental RFCs.  This is grounded in anecdotal 
evidence, since that is all we have.  There is at least as much 
anecdotal evidence that suggests that labels are not an effectie 
solution at all.  We have seen press, buyer, and vendor confusion 
treating Internet-Drafts (whcih clearly have a very different label) as 
if they were standards.  This suggests at least to me that we have the 
wrong end of the stick.

2) The description of measuring the experiment seems very odd in several 
different ways.  First, it states some measurements without stating 
whether increase or decrease in those measurements would constitute 
success.  Second, one of the measures is the amount of standards track 
documents produced, which seems very odd.  Third, 3 years is a long time 
for an experiment, but a very short time in the life cycle of produce 
useful documents at the IETF.  (I know folks don't like that fact, but 
it is a fact.)  Most importantly, there does not seem to be any 
measurement that relates the experiment to the problem it aims to solve. 
  I understand that measuring that is VERY difficult.  Heck, I have no 
clue how to measure it.  But it seems very odd to measure an experiment 
by things which have nothing to do with the goal of the experiment.

3) I did not see any discussion of the issue of downrefs.  If I were 
constructing a strawman to pretend that informational and experimental 
publication diluted the clairty of standards, the first thing I would 
point to as a cause of confusion is our practice of allowing downrefs to 
such documents.  I actually think such downrefs are a good idea.  In the 
absence of comment, I assume that the intention is to allow such 
normative downrefs when appropriate, as we currently do.  That seems to 
weaken the very dstinction of labeling that the 'experiment" is 
attempting to perform."

4) As a relatively minor matter, the description of the BPC documents is 
wrong.  The description in the document says that they are for IETF 
process documents.  While BCP is used, with good reason, for IETF 
process documents, it is also used for other things.  Inf act, there is 
significant ambiguity about what it should be used for, and it has been 
used for a range of protocol behavioral descriptions and operational 
practice descriptions.

Yours,
Joel

On 7/3/18 3:32 AM, Martin Thomson wrote:
> _______________________________________________
> Rfcplusplus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfcplusplus
>

---

From: Eliot Lear <[email-masked]>
Date: Tue, 3 Jul 2018 15:01:00 +0200

Hi Joel,

On 03.07.18 14:46, Joel M. Halpern wrote:
> point to as a cause of confusion is our practice of allowing downrefs
> to such documents.  I actually think such downrefs are a good idea. 
> In the absence of comment, I assume that the intention is to allow
> such normative downrefs when appropriate, as we currently do.  That
> seems to weaken the very dstinction of labeling that the 'experiment"
> is attempting to perform."

To recapitulate part of an earlier conversation, to me, downrefs and in
particular approved downrefs to what are now RFCs are really standards
unto themselves.  One approach, using what I think Martin and Adrian
were suggesting earlier would be to republish downrefs as RFCs.  This
violates Brian's principle of attributes versus names, to be fair, but
holds true to the intent of the experiment (I think).

Again, I am neutral on the experiment itself.

Eliot

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 03 Jul 2018 09:58:46 -0700

On 3 Jul 2018, at 0:32, Martin Thomson wrote:

> worth addressing.

The accompanying question that many of us keep asking is: what are the 
negative effects of this problem? Are they even worth addressing if the 
cost of the experiment is high?

So far, we've heard the main place where the problem appears is with 
customer requirements of vendors for things that are in RFCs that are 
not standards.

1) Vendor A might be losing sales to Vendor B because the latter 
includes support for things in RFCs that are not standards.

2) Vendor A's influential customer is demanding that they add support 
for something in an RFC that is not on standards track.

Neither of these should be that important to the IETF or the RFC Series. 
Vendor A has a responsibility to educate its potential customers about 
its products and the features in them. That education inherently 
involves explaining things about RFCs beyond just "is it a standard", 
particularly about which SHOULD-level requirements were not implemented, 
and how interoperability with other vendors who also implement that RFC 
has been tested.

Unless someone can show that the negative effects of the mis-perceptions 
that we have been handling just fine for 25 years are significant, there 
is no reason to try any experiment that might have negative effects if 
the experiment fails. As other folks discussing this draft have pointed 
out, there are plenty of long-term negative effects of backing out of 
this experiment.

--Paul Hoffman

---

From: <[email-masked]>
Date: Wed, 4 Jul 2018 06:04:45 +0000

Hi Martin, 

You said: 

===
RFC 7974 is the best example
I have of this.  It bears a warning nearly that strong and yet I have
received or witnessed strong endorsements of its virtues on several
occasions.
====

I'm puzzled with this statement for many reasons, but will focus only on two: 
* You seem to assume that a reader has not to have his/her OWN analysis on the merits/"virtues" of a technical proposal, but must adhere to whatever an IESG note says.
* Why not considering the problem from another perspective: wouldn't the "strong endorsements" be a signal to question a note itself?

There is no shame in including Experimental or information RFCs in RFPs or in strongly claiming the merits of an RFC published under the ISE stream! 
This is the responsibility of the one asking for it.  

The use of another stream or label is IMHO unlikely to change the situation. Consider these sample examples: 
- suppose that DC-TCP is published as EXP8257, instead of current RFC 8257
- suppose that TFO is published as EXP7413, instead of current RFC 7413
- or RFC 6234 is published under INF6234

If one of these features is needed in an RFP, the corresponding FOO/BAR documents will be listed. 

How can this be measured in the proposed experiment?

Cheers,
Med

> -----Message d'origine-----
> De : Rfcplusplus [mailto:[email-masked]] De la part de Martin
> Thomson
> Envoyé : mardi 3 juillet 2018 09:33
> À : [email-masked]
> Objet : [Rfcplusplus] Outlining the experiment
>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 4 Jul 2018 16:17:43 +1000

> * Why not considering the problem from another perspective: wouldn't the "strong endorsements" be a signal to question a note itself?

They could be.  You can make that assessment for yourself, but the
IESG don't make these statements lightly, and I respect their
conclusions.  FWIW, Section 1.2 of that document is pretty frank about
the purpose of the document, so the question of dishonesty clearly
doesn't apply, just the potential for confusion.

But as an author of that document, what value do you see that
publication as RFC provides in this case?  As opposed to publication
as FOO 7974, that is.  Maybe also consider a comparison with
publishing an internet-draft.

> How can this be measured in the proposed experiment?

Poorly, most likely.  But if that is the standard we insist upon,
we're probably done here already.

---

From: <[email-masked]>
Date: Wed, 4 Jul 2018 06:38:52 +0000

Re-,

Please see inline. 

Cheers,
Med

> De : Martin Thomson [mailto:[email-masked]]
> Envoyé : mercredi 4 juillet 2018 08:18
> À : BOUCADAIR Mohamed IMT/OLN
> Cc : [email-masked]
> Objet : Re: [Rfcplusplus] Outlining the experiment
> 
> > * Why not considering the problem from another perspective: wouldn't the
> "strong endorsements" be a signal to question a note itself?
> 

[Med] There might be confusion if one doesn't read the document. 

That document clearly indicate: 

* Its is an "Independent Submission"
* It is an "Informational" document 

And, it includes: 
* an ISE note about the intent of the document
* a note from the IESG
* a scope section (1.2)

> publication as RFC provides in this case?

[Med] We have already answered to this in Section 1.2 of that RFC, but I can cite the following: 

* Good technical reviews in both the IETF and ISE streams
* Encourage for more interoperable implementations

I do still think that RFC7974 is superior to RFC 7239 (which is published in the Standard Track). 

  As opposed to publication
> as FOO 7974, that is.

[Med] I'm not sure about the possible implication on documents review, but I don't see much difference except reducing the page count as the notes mentioned above are likely to be deleted because this will be implied by the FOO label itself. 

  Maybe also consider a comparison with

---

From: John C Klensin <[email-masked]>
Date: Wed, 04 Jul 2018 11:41:04 -0400

Martin and Med,

Let me build a bit on Paul's summary of vendor customer
requirements for things that are not standards.  Inline below. 

--On Wednesday, July 4, 2018 06:04 +0000
[email-masked] wrote:

> I have of this.  It bears a warning nearly that strong and yet
> I have received or witnessed strong endorsements of its
> virtues on several occasions.

and 

--On Wednesday, July 4, 2018 16:17 +1000 Martin Thomson
<[email-masked]> wrote:

>> * Why not considering the problem from another perspective:
>> wouldn't the "strong endorsements" be a signal to question a
>> note itself?
> 
> They could be.  You can make that assessment for yourself, but
> the IESG don't make these statements lightly, and I respect
> their conclusions.  FWIW, Section 1.2 of that document is
> pretty frank about the purpose of the document, so the
> question of dishonesty clearly doesn't apply, just the
> potential for confusion.

I respect the IESG's conclusions too and IETF informed consensus
conclusions about the right way to go even more so.

But, when a vendor decides to require support for some spec that
is not on the standards track (even one the IESG has warned
against in some detail), I think there are three possibilities.

(i) They have reviewed the spec and concluded it is a good idea
for their purposes, even though the IETF has reached an opposite
conclusion.

(b) The spec is widely-deployed enough that they have customer
requirements to interoperate with it.

(c) They are idiots or excessively influenced by aggressive
marketing and have therefore uncritically accepted the
specification probably without reading either it or the notes
attached to it.

I don't think we can solve the third problem.  Different labels
are unlikely to make a difference, disclaimers that are not read
because the document isn't might as well not be there, and, were
the spec published in the Journal of Irreproducible Results
rather than the RFC Series, that likely wouldn't make any
difference either.  We can only hope that the marketplace deals
appropriately with such companies.  So, moving on to the other
two...

Much as we might wish it otherwise, neither the IESG nor even
IETF consensus decisions are endowed with either infallibility
or perfect foresight.  Sometimes the marketplace is going to
make decisions, either broadly or in particular areas, that are
inconsistent with our preferences and decisions.  The broader
Internet community knows that and, if we carry looking at a
protocol, data structure, crypto method, or something else some
or all of us don't like, and saying "nasty, nasty, unapproved"
very forcefully when marketplace decision-makers are making
other decisions, we hurt the IETF's credibility in a way that no
amount of fussing with labels or "statements" of various sorts
is going to fix.

If we see a particular specification as a significant problem,
it may be that the very best thing we can do is to encourage it
to be completely and adequately documented and then publish (in
the same series) an Applicability Statement that carefully
describes why we think it is a bad idea.  That may not work
either, but it at least has some hope of solving the problme...
which more statements that are easily construed as "you
shouldn't do this because we said so and we are the
all-powerful, all-knowing, and all seeing IETF" is definitely
not going to help much.

    john

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 5 Jul 2018 16:43:26 +1200

> amount of fussing with labels or "statements" of various sorts

There is of course a poster child for exactly that, and in some
ways the IETF is lucky to have survived it. I refer to RFC 1597
and RFC 1631. We threw our toys out of the playpen (e.g. RFC 2993)
for years, but to no avail. BCP 5 even managed to avoid any
reference whatever to address translation.

I doubt that the history of NAT would have been much different
if those had been INDEP 1597, INDEP 1631 and IAB 2993.

    Brian