Subject: [Rfcplusplus] Sunk cost + not about us

From: Richard Barnes <[email-masked]>
Date: Mon, 9 Jul 2018 18:17:00 -0400

Hey all,

I'm still catching up on the many emails on this list, so I might be
missing something, but I wanted to surface two arguments that seem to me to
be pretty dispositive that there is a problem here to solve:

1. A recommendation to hold is a recommendation to buy / sunk cost fallacy

Suppose someone came to you with the following problem statement:

- We have 16 (or so) types of document that we want to publish
- We need readers of one of these documents to be clear on which type
they're reading

Is there anyone here who would look at that problem statement and say, "The
ideal way to do that is to throw all the documents into one linear series,
and have the reader look for distinguishing marks in the text of the
document"?  We shouldn't keep supporting a system we wouldn't build today.

2. This is not about us

There are a total of a little over 30 unique senders on this list.  The
person closest to being a newcomer is Joe Hall, whose first IETF meeting
was IETF 89, more than four years ago.  This discussion isn't about us --
it's about the much larger group of people who read these documents and try
to build products off of them that work.

In an ideal world, some of those folks would be participating here.  Until
that happens, we should rely on what data we can gather.  We've already
head several anecdotes, and they're pretty much all on one side -- the
current arrangement causes confusion and makes RFCs less useful.

To try to be slightly more systematic, I sent a survey out over the weekend
to a bunch of communities I participate in that are "IETF-adjacent".  It
got 115 responses, and the data [1] are consistent with the anecdata:

- 52% of respondents thought that an RFC was "A document published by the
IETF that defines a technical standard for the Internet"

- While 93% of respondents agreed that the IETF can publish RFCs, the other
streams came in much farther behind; the IAB was second-most recognized, at
only 26%

- Nobody has any idea that about the taxonomy of RFCs.  A majority of
people said they thought that there were 5 types, but I would conjecture
that's just an artifact of the options I put in there.

Based on those observations, I hope it's clear to folks that there is a
problem to be solved here.  The survey data, sketchy as they are, also
point toward the solution, which is to refine the RFC label to have a much
more limited semantic, probably only IETF and possibly only standards.

Hope that helps,

--Richard

[1]
https://docs.google.com/forms/d/e/1FAIpQLSeMoeR0TBWkZNpBKXJN3Am6nUL04Vr4-12T2VgEbiRdBwzngQ/viewanalytics

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 9 Jul 2018 18:32:28 -0400

This formulation assumes that change does not have a cost.  It does.  I 
agree that not changing has some cost.  However, absent indication that 
the changes will actually address the claimed problem, paying the 
various direct and indirect costs of changing things is not inherently a 
good thing.

If we really want to change something that causes confusion in a way 
that actually causes us process problems, maybe we should actually 
tackle one?  In particular, in looking at kinds of confusion, one of the 
ones that causes the IETF real and significant problems is when outside 
groups look at individual Internet Drafts, and treat them like working 
group adopted products.  (I would like to fix folks treating adotped 
I-Ds as if they were finished RFCs, but given that those already have 
clearly distinct labels, that does not seem possible.)
In one sense, fixing that would likely require changing quite a few 
moving parts in our process.  It is clearly something the IETF owns.  It 
is something that causes myriad issues between us and other standards 
bodies and between us an users.  We often have to actually put in work 
pushing back on folks mis-characterizing individual I-Ds as IETF work.

Yours,
Joel

PS: I do not see how we can draw any conclusion from the very informal 
survey.  We have been lectured, with good reason, in the past about the 
dangers of drawing conclusions from even carefully formulated and 
carefully distributed surveys.

On 7/9/18 6:17 PM, Richard Barnes wrote:
> missing something, but I wanted to surface two arguments that seem to me 
> to be pretty dispositive that there is a problem here to solve:
> 
> 
> Is there anyone here who would look at that problem statement and say, 
> "The ideal way to do that is to throw all the documents into one linear 
> series, and have the reader look for distinguishing marks in the text of 
> the document"?  We shouldn't keep supporting a system we wouldn't build 
> today.
> 
> 
> 2. This is not about us
> 
> was IETF 89, more than four years ago.  This discussion isn't about us 
> -- it's about the much larger group of people who read these documents 
> and try to build products off of them that work.
> 
> In an ideal world, some of those folks would be participating here.  
> Until that happens, we should rely on what data we can gather.  We've 
> already head several anecdotes, and they're pretty much all on one side 
> -- the current arrangement causes confusion and makes RFCs less useful.
> 
> To try to be slightly more systematic, I sent a survey out over the 
> weekend to a bunch of communities I participate in that are 
> "IETF-adjacent".  It got 115 responses, and the data [1] are consistent 
> with the anecdata:
> 
> - 52% of respondents thought that an RFC was "A document published by 
> the IETF that defines a technical standard for the Internet"
> 
> - While 93% of respondents agreed that the IETF can publish RFCs, the 
> other streams came in much farther behind; the IAB was second-most 
> recognized, at only 26%
> 
> problem to be solved here..  The survey data, sketchy as they are, also 
> point toward the solution, which is to refine the RFC label to have a 
> much more limited semantic, probably only IETF and possibly only standards.
> 
> Hope that helps,
> 
> --Richard
> 
> [1] 
> _______________________________________________
> Rfcplusplus mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rfcplusplus
>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 Jul 2018 12:30:39 +1200

On 10/07/2018 10:32, Joel M. Halpern wrote:
...
> carefully distributed surveys.

It's worse than that. The survey was hastily designed and included
at least one ad hominem entry, and I think anyone with experience
of surveys and their analysis would simply junk the results. See
below for something more concrete on that.

...
> much more limited semantic, probably only IETF and possibly only standards.

That's a pretty perfect example of confirmation bias. Sorry, but
even if we accept that there's a problem worth solving, the "data"
don't point to any particular solution.

I hope that as scientists and engineers, we can do better than this.

By the way, the full results are at
https://docs.google.com/forms/d/e/1FAIpQLSeMoeR0TBWkZNpBKXJN3Am6nUL04Vr4-12T2VgEbiRdBwzngQ/viewanalytics

The last question is interesting.:
"Does an RFC published by the IETF require IETF consensus?"
got 58.8% "no", which is the correct answer. However, this
seems very unlikely as the preferred response from outsiders.
Only insiders know that only some IETF stream RFCs require
IETF consensus. So I don't think the data can reasonably be
assumed to represent the opinion of outsiders.

    Brian

---

From: Richard Barnes <[email-masked]>
Date: Tue, 10 Jul 2018 10:21:15 -0400

On Mon, Jul 9, 2018 at 6:32 PM Joel M. Halpern <[email-masked]> wrote:

> the changes will actually address the claimed problem...

People are presenting indications.  Attach what caveats you need to my
little study; it's still real data from a relevant population.  Do you have
better data?

> tackle one?

Without commenting on whether this I-D problem is a worthwhile problem to
solve, it is not the one this list is here to discuss.

--Richard

> In particular, in looking at kinds of confusion, one of the

---

From: Richard Barnes <[email-masked]>
Date: Tue, 10 Jul 2018 10:34:05 -0400

> below for something more concrete on that.

I'll admit it was hastily conceived; the idea came to me as I was composing
my message to the list on Friday.  I won't admit it's ad-hominem -- it's
simply fact that the process entitles Adrian can approve RFCs as he likes.

But in the real world, when you're trying to make a decision, you either
work with dirty data or you collect better data.  I don't see anyone here
doing any better survey work, so throwing out what data we have seems
counterproductive.

> That's a pretty perfect example of confirmation bias.
>

If you have another interpretation for the responses to questions 1 and 2,
I would love to hear it.

> I hope that as scientists and engineers, we can do better than this.

I have no doubt that we could do better, but so far, nobody's done the work.

> assumed to represent the opinion of outsiders.

I think this is more likely read as people just not understanding the
question, and the role consensus plays in our process. If you think the
respondents were insiders, how do you explain the abundantly "incorrect"
responses to question 2?

I actually asked people let me know when they filled out the survey, and
got around 20 notifications, including some IETF newcomers, several senior
developers for major projects, and a few newbie developers.  Taking that
together with the Q2 observation above, I doubt the sample was dominated by
insiders.

--Richard

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 10 Jul 2018 10:45:25 -0400

Hi,

On Mon, Jul 09, 2018 at 06:32:28PM -0400, Joel M. Halpern wrote:
> agree that not changing has some cost.  However, absent indication that the
> changes will actually address the claimed problem, paying the various direct
> and indirect costs of changing things is not inherently a good thing.

I think this argument would be stronger if we understood what your
estimation of the costs are.  The analysis underpinning the proposal
(in the BoF request), as near as I can tell, is that the costs are the
using up of a few letters (for new series names) plus the cost of
possibly reserving RFC numbers for the alternative series publications
(in case they turn out not to work after 3 or more years), plus the
costs to tooling changes.  Are there other costs you can think of?

Keep in mind, the proposal is for an experiment.  Experiments do
indeed result in some costs, and I think it is entirely reasonable to
ask what those costs really would be.  But I also think we ought to
keep in mind the potential benefit of the experiment, which is to do
an empirical evaluation of whether the long-debated confusion can be
clarified (at least, in this way).

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Eliot Lear <[email-masked]>
Date: Tue, 10 Jul 2018 16:59:16 +0200

On 10.07.18 16:34, Richard Barnes wrote:

    I have no doubt that we could do better, but so far, nobody's done
    the work.

That seems like a far better place to start in all of this.  In fact, it
would be quite good if this group could even agree on what goo looks
like.  That will also help clarify what measurements are important.

Eliot

---

From: Eliot Lear <[email-masked]>
Date: Tue, 10 Jul 2018 17:01:13 +0200

Erm...

On 10.07.18 16:59, Eliot Lear wrote:
> That seems like a far better place to start in all of this.  In fact,
> it would be quite good if this group could even agree on what goo
> looks like.  That will also help clarify what measurements are important.

s/goo/good/

Please send no textual descriptions of goo.

Eliot

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 10 Jul 2018 11:20:40 -0400

> costs to tooling changes.  Are there other costs you can think of?

1) Human costs for all authors for pointing to the new series while the 
experiment is running.

2) Human costs of arguing about how to measure the successfulness of the 
experiment.

3) If the experiment ends in failure (likely because we can't evaluate 
if there is less confusion with the new series than in the current 
setup), we will either have a bunch of small vestigial series with a 
bunch of RFCs pointing to them, or we have to update all of the RFCs 
that have pointers to the new series to instead point to the RFC 
numbers.

> clarified (at least, in this way).

We have not seen any suggestion of how such an empirical evaluation 
would take place. We have not seen any suggestion for, even if we could 
measure the problem today and a reduction of the measured problem, how 
much reduction would be labeled "success".

--Paul Hoffman

---

From: "Adrian Farrel" <[email-masked]>
Date: Tue, 10 Jul 2018 17:04:43 +0100

All,

> I'll admit it was hastily conceived; the idea came to me as I was
> composing my message to the list on Friday.  I won't admit it's 
> ad-hominem -- it's simply fact that the process entitles Adrian 
> can approve RFCs as he likes.

I can accept that it was not intentionally an attack on me or an attempt to divert the argument from the real issues by focusing on me, and I have spoken with Richard about it privately. Nevertheless, if it was not carefully formulated and it definitely used my name rather than my role as Independent Submissions Editor. I suspect that, despite it being in the public domain that I am the ISE, it is not public knowledge. Thus, if the responders truly knew little of the IETF (as requested in the preamble) they should have had no idea who I am or why my name was listed. Had the question used "Independent Submissions Editor" instead, the answers might have been different.

But who cares? Twitter is a wild park where idiots play. I am not offended by anything that is said there, nor do I take seriously any "results" of surveys or opinion storms that happen there. I may draw my own conclusions about those who pay too much attention to Twitter, but that is between me and my analyst.

> doing any better survey work, so throwing out what data we have seems

Ah, "the real world". What are you trying to say, Richard? Is Twitter the real world? Does Brian not live in the real world? 

May I suggest that it does not help advance the discussion in any way to proceed like this? I shall not be going to the pub tonight to ask three drunks, the barmaid, and the landlord's dog. That would give us some more data, but it, too, would have no value.

Adrian (Finding it hard to keep separate my personal opinions and my role as ISE when others confuse them, but still attempting to respond from this address as an individual participant in the IETF.)

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Jul 2018 13:01:32 -0400

--On Tuesday, July 10, 2018 10:34 -0400 Richard Barnes
<[email-masked]> wrote:

>...
> But in the real world, when you're trying to make a decision,
> you either work with dirty data or you collect better data.  I
> don't see anyone here doing any better survey work, so
> throwing out what data we have seems counterproductive.
>...

Richard, 

As someone who used to teach survey design, survey data
analysis, and threats to validity of such analyses, I beg to
disagree.  

To summarize a much longer tutorial... If one has data that are
known to reflect sample biases relative to the population of
interest (and some of this discussion leads me to believe we
don't even have consensus about what that population is) or
whose collection of questions cannot be clearly and
unambiguously interpreted, show biases toward certain types of
answers or are otherwise prone to confirmation bias, etc., then
the results are likely to be of little or no value at all.  

That is very different from "dirty data".  "Dirty data", while
not a precise technical term, usually refers to data whose
biases (statistical or substantive) we understand well enough to
at least partially de-bias or adjust for its inadequacies.

Bad data in which the biases are impossible to evaluate other
than knowing that they are present and potentially serious (and
as distinct from "dirty data" as above) may not only be worse
than "no data" but may be quite a bit worse than someone
standing up and saying "this is true because I said so".  At
least in the latter case, one has some reasonably chance of
guessing at the expertise, prejudices, preconceptions, and
general reliability of the source and calibrating the assertion
accordingly.

     best,
       john

---

From: "Andrew G. Malis" <[email-masked]>
Date: Tue, 10 Jul 2018 13:07:58 -0400

Paul’s statement:

> We have not seen any suggestion of how such an empirical evaluation would
> take place. We have not seen any suggestion for, even if we could measure
> the problem today and a reduction of the measured problem, how much
> reduction would be labeled "success".

Really says it all - other than anecdotes and opinions, there has been no
hard evidence of a “problem” that a reasonable third party could look at
and evaluate, nor have there been any proposals as how to reasonably
determine the results of the experiment in three years, were it to be held.

I think the most of productive use of our time during the BoF session will
be to discuss how to gather real evidence of a problem, and how to evaluate
the effects of any resulting IETF actions. Only after we have agreement on
the methodology, and carry it out, will it be time to discuss what we
should or should not do based on the results.

Cheers,
Andy

---

From: Eric C Rosen <[email-masked]>
Date: Tue, 10 Jul 2018 13:59:16 -0400

On 7/9/2018 6:17 PM, Richard Barnes wrote:
> "The ideal way to do that is to throw all the documents into one 
> linear series, and have the reader look for distinguishing marks in 
> the text of the document"?  We shouldn't keep supporting a system we 
> wouldn't build today.

The bias here is in the statement of the problem.

One could equally frame the problem as:

- We have a long-standing and well-recognized series of documents 
containing a lot of useful information about how the Internet works, and 
about how one can interoperate with existing Internet deployments.

- There are a number of different procedures for getting a document 
published in that series.

- Sometimes it is of interest to the readers to know which procedure 
resulted in the publication of a particular document.

How would you solve this problem?

Well, you could add some text to each document saying which procedure 
resulted in the publication of the document.

Alternatively, you can say, "Well, we need to break the series up into 
16 different series, each with its own name, and with no apparent 
relation to the pre-existing series.  Then when one wants to understand 
some aspect of the Internet, one has 15 more places to look.  And before 
one can find out how to interoperate with an existing deployment, one 
has to figure out which of the series is likely to have the documents 
one needs.  Of course, to see all the relevant documents, one may have 
to look at two or three series, good luck finding them."

I can't see that the latter alternative has much to recommend it. The 
former alternative seems much more sensible.

BTW, the statement "we need readers of these documents to be clear on 
which type they're reading" is not a correct statement of the problem 
anyway.  Most of the anecdotes about confusion have to do with folks 
that have never read the documents and never will.  The RFP-writers who 
list RFC numbers without having a clue what's in those RFCs will still 
generate confusing and non-implementable RFPs.  The need to educate the 
customer could be regarded as a waste of time, but really it's just 
another aspect of sales support.

Even the folks who do read the documents don't necessarily have to be 
clear on the procedure that caused the document to be published. If 
you're reading a protocol spec, it's probably because you need to 
interoperate with some existing deployments, and whether the protocol 
went through some particular process is irrelevant.

> "IETF-adjacent".  It got 115 responses, and the data [1] are 
> consistent with the anecdata:

I think this pseudo-survey has been fully debunked by others.

If one wants to show that there is a problem, one should provide 
evidence that there have been many cases where folks have implemented 
and deployed the wrong protocol, because they got confused about the 
status of the RFCs that specify the protocol. The fact that folks not 
involved in the publication process cannot say which procedures caused 
the documents to be published is really of no importance at all.

---

From: John C Klensin <[email-masked]>
Date: Tue, 10 Jul 2018 15:37:47 -0400

Eric,

Thanks for, IMO, a very good summary.   One addition: because
the BOF request starts from a very old description of/assertions
about the supposed confusion problem, it seems to me to be
incumbent on those who claim there is still a problem serious
enough to require drastic action to demonstrate that the many
changes we have introduced to clarify the status of documents
since then have been ineffective.

thanks again,
   john

--On Tuesday, July 10, 2018 13:59 -0400 Eric C Rosen
<[email-masked]> wrote:

>> - We need readers of one of these documents to be clear on
>> which type  they're reading
>> 
>> Is there anyone here who would look at that problem statement
>> and say,  "The ideal way to do that is to throw all the
>> documents into one  linear series, and have the reader look
>> for distinguishing marks in  the text of the document"?  We
>> shouldn't keep supporting a system we  wouldn't build today.
> 
> - We have a long-standing and well-recognized series of
> documents containing a lot of useful information about how the
> Internet works, and about how one can interoperate with
> existing Internet deployments.
> 
> - There are a number of different procedures for getting a
> document published in that series.
> 
> - Sometimes it is of interest to the readers to know which
> procedure resulted in the publication of a particular document.
> 
> Well, you could add some text to each document saying which
> procedure resulted in the publication of the document.
> 
> Alternatively, you can say, "Well, we need to break the series
> up into 16 different series, each with its own name, and with
> no apparent relation to the pre-existing series.  Then when
> one wants to understand some aspect of the Internet, one has
> 15 more places to look.  And before one can find out how to
> interoperate with an existing deployment, one has to figure
> out which of the series is likely to have the documents one
> needs.  Of course, to see all the relevant documents, one may
> have to look at two or three series, good luck finding them."
> 
> I can't see that the latter alternative has much to recommend
> it. The former alternative seems much more sensible.
> 
> BTW, the statement "we need readers of these documents to be
> clear on which type they're reading" is not a correct
> statement of the problem anyway.  Most of the anecdotes about
> confusion have to do with folks that have never read the
> documents and never will.  The RFP-writers who list RFC
> numbers without having a clue what's in those RFCs will still
> generate confusing and non-implementable RFPs.  The need to
> educate the customer could be regarded as a waste of time, but
> really it's just another aspect of sales support.
> 
> Even the folks who do read the documents don't necessarily
> have to be clear on the procedure that caused the document to
> be published. If you're reading a protocol spec, it's probably
> because you need to interoperate with some existing
> deployments, and whether the protocol went through some
> particular process is irrelevant.
> 
>> 
>> To try to be slightly more systematic, I sent a survey out
>> over the  weekend to a bunch of communities I participate in
>> that are  "IETF-adjacent".  It got 115 responses, and the
>> data [1] are  consistent with the anecdata:
> 
> If one wants to show that there is a problem, one should
> provide evidence that there have been many cases where folks
> have implemented and deployed the wrong protocol, because they
> got confused about the status of the RFCs that specify the
> protocol. The fact that folks not involved in the publication
> process cannot say which procedures caused the documents to be
> published is really of no importance at all.
> 
>

---

From: Bob Hinden <[email-masked]>
Date: Tue, 10 Jul 2018 13:11:54 -0700

Richard,

> On Jul 10, 2018, at 7:21 AM, Richard Barnes <[email-masked]> wrote:
> 
> 
> 
> On Mon, Jul 9, 2018 at 6:32 PM Joel M. Halpern <[email-masked] <mailto:[email-masked]>> wrote:
> People are presenting indications.  Attach what caveats you need to my little study; it's still real data from a relevant population.  Do you have better data?

When I saw the survey, after I filled it in, I noticed that I could do it again. There didn’t appear to be a mechanism to keep anyone from taking it multiple times.   Based on this, I don’t think one can draw any conclusions.

Bob

---

From: Richard Barnes <[email-masked]>
Date: Tue, 10 Jul 2018 16:56:18 -0400

On Tue, Jul 10, 2018 at 4:11 PM Bob Hinden <[email-masked]> wrote:

> When I saw the survey, after I filled it in, I noticed that I could do it
> again. There didn’t appear to be a mechanism to keep anyone from taking it
> multiple times.   Based on this, I don’t think one can draw any conclusions.
>

Do you ever use telemetry from fielded products?  How do you know your
competitors aren't feeding you bad data?

--Richard

---

From: Bob Hinden <[email-masked]>
Date: Tue, 10 Jul 2018 14:51:23 -0700

> On Jul 10, 2018, at 1:56 PM, Richard Barnes <[email-masked]> wrote:
> 
> 
> 
> On Tue, Jul 10, 2018 at 4:11 PM Bob Hinden <[email-masked] <mailto:[email-masked]>> wrote:
> Richard,
> 
>> On Jul 10, 2018, at 7:21 AM, Richard Barnes <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 
> Do you ever use telemetry from fielded products?  How do you know your competitors aren't feeding you bad data?

You point is?  That your flawed survey is OK because there are other flawed surveys?

Bob

---

From: Richard Barnes <[email-masked]>
Date: Tue, 10 Jul 2018 18:03:18 -0400

> You point is?  That your flawed survey is OK because there are other

Since Eden, we have lived in a fallen, flawed world, and we get along with
flawed surveys.  We extract what meaning we can.

In any case, as I think I said elsewhere in this thread, I have talked to a
few dozen real, live, honest respondents.  So I've got pretty good
confident that the survey is not massively tainted by abuse.

--Richard

---

From: Joseph Lorenzo Hall <[email-masked]>
Date: Tue, 10 Jul 2018 18:05:41 -0400

> You point is?  That your flawed survey is OK because there are other
I have a hard time seeing any survey research design that would convince
this group, and I think that Richard's survey is a non-trivial effort to
get some granted ugly data.

As Richard pointed out, I guess I'm the noob here on the list, having been
attending IETFs since IETF 89 in London, about 4.5 years ago. I must admit
I still find so much of IETF a bit baffling... now that I'm approaching my
IETF adolescence, I'm starting to understand why certain things are the way
they are at IETF, and I'm thankful that it does what it does well and
thoroughly. (I've even chipped in on the IASA 2.0 effort to help adapt
IETF's administrative structure for the future.)

However, the document series still baffles me, and I know a lot of folks
that squint at IETF and don't understand any of the distinctions we do (I
would have answered the survey question as there being only 4 document
series). Alissa had the unenviable task at one point of sitting me down and
telling me the difference between and ID and RFC, even after I had
thoroughly read the Tao and been to a couple IETFs. (The ID we've been
working on for a while I even named (incorrectly) when setting up a github
repository as an RFC: https://github.com/josephlhall/rfc-censorship-tech ).

I'll throw my hat in and say I'd like to see data, qualitative or
quantitative. I'm a mixed methods researcher and have experience in both. I
know a number of us have had enough informal discussions with "liminal"
IETF participants that there is enough confusion here to explore doing
something about it. That may not warrant an immediate document identifier
experiment, but I think we could settle on some sort of research design
that would provide essentially something like "running code" we could work
with. cheers, Joe

-- 
Joseph Lorenzo Hall
Chief Technologist, Center for Democracy & Technology [https://www.cdt.org]
1401 K ST NW STE 200, Washington DC 20005-3497
e: [email-masked], p: 202.407.8825, pgp: https://josephhall.org/gpg-key
Fingerprint: 3CA2 8D7B 9F6D DBD3 4B10  1607 5F86 6987 40A9 A871

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 10 Jul 2018 18:18:01 -0400

Thank you Joseph.Yes, it is confusing.
And it is confusing in many different ways for many different reasons.

It is very strange to have a BoF about such confusion, and then be told 
that various specific forms of confusion (such as I-D vs RFC whcih you 
cite, or different kinds of I-Ds which I mentioned) are out of scope. 
Apparently, we know which problems are important to the community before 
we have the discussion.
usually, we get that information from the email list discussion with the 
community before a BoF is requested.

Yours,
Joel

On 7/10/18 6:05 PM, Joseph Lorenzo Hall wrote:
> 
> 
> On Tue, Jul 10, 2018 at 5:51 PM Bob Hinden <[email-masked] 
> <mailto:[email-masked]>> wrote:
> 
> 
>>     On Jul 10, 2018, at 1:56 PM, Richard Barnes <[email-masked]
>>     <mailto:[email-masked]>> wrote:
>>
>>
>>
>>>         On Mon, Jul 9, 2018 at 6:32 PM Joel M. Halpern
>>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>>         People are presenting indications.  Attach what caveats you
>>>         need to my little study; it's still real data from a relevant
>>>         population.  Do you have better data?
>>
>>
>>         When I saw the survey, after I filled it in, I noticed that I
>>         could do it again. There didn’t appear to be a mechanism to
>>         keep anyone from taking it multiple times.   Based on this, I
>>         don’t think one can draw any conclusions.
>>
>>
>>     Do you ever use telemetry from fielded products?  How do you know
>>     your competitors aren't feeding you bad data?
> 
> As Richard pointed out, I guess I'm the noob here on the list, having 
> been attending IETFs since IETF 89 in London, about 4.5 years ago. I 
> must admit I still find so much of IETF a bit baffling... now that I'm 
> approaching my IETF adolescence, I'm starting to understand why certain 
> things are the way they are at IETF, and I'm thankful that it does what 
> it does well and thoroughly. (I've even chipped in on the IASA 2.0 
> effort to help adapt IETF's administrative structure for the future.)
> 
> that squint at IETF and don't understand any of the distinctions we do 
> (I would have answered the survey question as there being only 4 
> document series). Alissa had the unenviable task at one point of sitting 
> me down and telling me the difference between and ID and RFC, even after 
> I had thoroughly read the Tao and been to a couple IETFs. (The ID we've 
> been working on for a while I even named (incorrectly) when setting up a 
> github repository as an RFC: 
> https://github.com/josephlhall/rfc-censorship-tech ).
> 
> quantitative. I'm a mixed methods researcher and have experience in 
> both. I know a number of us have had enough informal discussions with 
> "liminal" IETF participants that there is enough confusion here to 
> explore doing something about it. That may not warrant an immediate 
> document identifier experiment, but I think we could settle on some sort 
> of research design that would provide essentially something like 
> "running code" we could work with. cheers, Joe
> -- 
> Joseph Lorenzo Hall
> e: [email-masked] <mailto:[email-masked]>, p: 202.407.8825, pgp: 
> https://josephhall.org/gpg-key

---

From: Alissa Cooper <[email-masked]>
Date: Tue, 10 Jul 2018 21:36:09 -0400

Hi Joel,

> On Jul 10, 2018, at 6:18 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> It is very strange to have a BoF about such confusion, and then be told that various specific forms of confusion (such as I-D vs RFC whcih you cite, or different kinds of I-Ds which I mentioned) are out of scope.

In my experience arguing about the scope of the problem is pretty typical BOF fodder, so in that respect this one does not seem unusual thus far.

It’s true that the BOF description keyed off of the problem statement in RFC 1796, which doesn’t mention Internet-drafts. I don’t think that makes confusion about what I-Ds are or whether they are standards out of scope. These also aren’t mutually exclusive problems; tackling more than one of them seems like a valid outcome if that’s what the community wants to do.

> Apparently, we know which problems are important to the community before we have the discussion.

I’m not sure who you mean by “we,” but I don’t feel like I could draw such a conclusion at this point. 

Alissa

> usually, we get that information from the email list discussion with the community before a BoF is requested.
> 
> Yours,
> Joel
> 
>> On Tue, Jul 10, 2018 at 5:51 PM Bob Hinden <[email-masked] <mailto:[email-masked]>> wrote:
>> I have a hard time seeing any survey research design that would convince this group, and I think that Richard's survey is a non-trivial effort to get some granted ugly data.
>> As Richard pointed out, I guess I'm the noob here on the list, having been attending IETFs since IETF 89 in London, about 4.5 years ago. I must admit I still find so much of IETF a bit baffling... now that I'm approaching my IETF adolescence, I'm starting to understand why certain things are the way they are at IETF, and I'm thankful that it does what it does well and thoroughly. (I've even chipped in on the IASA 2.0 effort to help adapt IETF's administrative structure for the future.)
>> However, the document series still baffles me, and I know a lot of folks that squint at IETF and don't understand any of the distinctions we do (I would have answered the survey question as there being only 4 document series). Alissa had the unenviable task at one point of sitting me down and telling me the difference between and ID and RFC, even after I had thoroughly read the Tao and been to a couple IETFs. (The ID we've been working on for a while I even named (incorrectly) when setting up a github repository as an RFC: https://github.com/josephlhall/rfc-censorship-tech ).
>> I'll throw my hat in and say I'd like to see data, qualitative or quantitative. I'm a mixed methods researcher and have experience in both. I know a number of us have had enough informal discussions with "liminal" IETF participants that there is enough confusion here to explore doing something about it. That may not warrant an immediate document identifier experiment, but I think we could settle on some sort of research design that would provide essentially something like "running code" we could work with. cheers, Joe
>> -- 
>> Joseph Lorenzo Hall
>> e: [email-masked] <mailto:[email-masked]>, p: 202.407.8825, pgp: https://josephhall.org/gpg-key

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 14:35:24 +1200

> If you have another interpretation for the responses to questions 1 and 2,

That people are confused, or simply don't read the boilerplate, isn't news.

What has not been demonstrated is whether this matters enough to be worth fixing
in a disruptive way.

     Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Jul 2018 14:47:51 +1200

Andrew,

On 11/07/2018 02:45, Andrew Sullivan wrote:
> Keep in mind, the proposal is for an experiment.

Andrew,

It purports to be an experiment, but it's one that is hard to unwind.
My own excursion into solutionism
https://mailarchive.ietf.org/arch/msg/rfcplusplus/7GIxREzdIYAZFAHRSqCyp6S1JMo
is, if implemented experimentally, much easier to unwind because it
adds some wrapping to the archival series, and that wrapping could
later be removed. However, what I'm getting clearly from the last
24 hours' email is that since we don't yet understand the problem,
it's way too soon to seriously design an (experimental) solution.

    Brian

> Experiments do

---

From: S Moonesamy <[email-masked]>
Date: Wed, 11 Jul 2018 01:28:08 -0700

Hi Joseph,
At 03:05 PM 10-07-2018, Joseph Lorenzo Hall wrote:
>However, the document series still baffles me, and I know a lot of 
>folks that squint at IETF and don't understand any of the 
>distinctions we do (I would have answered the survey question as 
>there being only 4 document series). Alissa had the unenviable task 
>at one point of sitting me down and telling me the difference 
>between and ID and RFC, even after I had thoroughly read the Tao and 
>been to a couple IETFs. (The ID we've been working on for a while I 
>even named (incorrectly) when setting up a github repository as an 
>RFC: https://github.com/josephlhall/rfc-censorship-tech ).

The IETF changed its position on Internet-Drafts a few years 
ago.  There are cases where Internet-Drafts have been referenced as RFCs.

As an off-topic comment, the repository states that the discussions 
of the work occurs on the censorship working group mailing list.  I 
could not find that IETF working group.

Regards,
S. Moonesamy

---

From: Joseph Lorenzo Hall <[email-masked]>
Date: Wed, 11 Jul 2018 07:02:50 -0400

> could not find that IETF working group.

There is certainly no WG for that; I suspect a typo or relic of the
template used.

Joseph Lorenzo Hall

---

From: John C Klensin <[email-masked]>
Date: Wed, 11 Jul 2018 09:49:43 -0400

--On Wednesday, July 11, 2018 14:47 +1200 Brian E Carpenter
<[email-masked]> wrote:

>...
> However, what I'm getting clearly from the last
> 24 hours' email is that since we don't yet understand the
> problem, it's way too soon to seriously design an
> (experimental) solution.

Normally, I'd say that was true.

But the situation is not normal.  We are in a situation in which
it seems clear that the thinking that went into
draft-thomson-rfcplusplus-label-00 was key to the BOF proposal
and that both it and the BOF proposal posit a very specific
solution.  While I accept Martin's explanation and hence have no
reason to believe the posting of that draft at the last minute
was intended to disadvantage any other proposed solutions, it
does have that effect, an effect that is aggravated by there
still being no posted agenda for a BOF that occurs on the first
day of the meeting.  That leaves us without knowing whether the
intention is to discuss the problem or to assume, as the BOF
proposal does, that there is a problem that is in need of
solutions and that we are discussing solutions (or one specific
solution).  Perhaps the plan is "both", but, especially with
only one possible solution available in I-D form before Monday
morning and the amount of discussion about a variety of topics
on the list, 90 minutes seems very compressed.

Alissa, given that state of affairs and assuming that possible
solutions are not going to be ruled out of order in Monday's
discussion, can we get blanket permission for late-posting
(before Monday) of any draft related to this BOF?  I think it is
generally accepted that discussing I-Ds, even hastily prepared
ones, is usually more productive than discussing partial
proposals that go by on mailing lists.   And, while the posting
cutoff was intended, in part, to allow people adequate time to
read and study documents before IETF begins, it seems to me that
giving people the weekend (and possibly even part of tomorrow
and Friday) is preferable to either giving them zero time
(solutions introduced at the BOF itself) or well under 24 hours
between whenever I-D postings are allowed again on Monday (the
"important dates" page appears to no longer show whether that
is, e.g., 00:01 UTC or the starting time of the meeting).

best,
   john

---

From: Alissa Cooper <[email-masked]>
Date: Wed, 11 Jul 2018 09:58:55 -0400

Hi John,

> On Jul 11, 2018, at 9:49 AM, John C Klensin <[email-masked]> wrote:
> 
> 
> 
> (before Monday) of any draft related to this BOF?  

As has been my practice pretty much since I joined the IESG, I’m fine with doing manual postings of late-arriving drafts between now and when the submissions window opens, if people send them to me. And of course there is nothing stopping people from sending pointers to documents they’ve written that haven’t yet been posted to the I-D repo, as has become more common in some WGs lately.

Alissa

> I think it is