Subject: Re: [Tools-discuss] Last Call: <draft-sheffer-running-code-04.txt> (Improving Awareness of Running Code: the Implementation Status Section) to Experimental RFC

From: Fred Baker <[email-masked]>
Date: Thu, 25 Apr 2013 15:09:18 -0700

On Apr 12, 2013, at 2:57 PM, The IESG <[email-masked]> wrote:

> Abstract
> 
> 
>   This document describes a simple process that allows authors of
>   Internet-Drafts to record the status of known implementations by
>   including an Implementation Status section.  This will allow
>   reviewers and working groups to assign due consideration to documents
>   that have the benefit of running code, by considering the running
>   code as evidence of valuable experimentation and feedback that has
>   made the implemented protocols more mature.
> 
>   The process in this document is offered as an experiment.  Authors of
>   Internet-Drafts are encouraged to consider using the process for
>   their documents, and working groups are invited to think about
>   applying the process to all of their protocol specifications.  The
>   authors of this document intend to collate experiences with this
>   experiment and to report them to the community.

I have read the draft. I like the concept. It applies primarily to protocols and procedures, as opposed to white papers. It, however, puts emphasis on the experience behind the usefulness of a protocol or procedure, which is good.

BTW, once upon a time we required implementation reports for routing protocols, which I thought was a good thing and am concerned about the loss of in recent times. That resulted in the publication of sets like RFCs 1245, 1246, and 1247, which told about the protocol, its operational characteristics, and the testing it went through (and by implication, the implementations done of it) on its way to RFC-dom.

In 2013, I personally would accomplish this a little differently, however. A section in an internet draft, which gets frozen when the draft is published, is perhaps useful for the working group and IESG review processes. On the other hand, it requires implementers to communicate with the draft author and the draft author to update the draft in response to their input, which can be a logistical mess. It ceases being useful once the draft is published. If a new implementation is done, there is no report. If and old one is abandoned, nobody knows. It is dated information, potentially true at a point in time but largely irrelevant two minutes later.

I would think we want something associated with the data tracker page - another web page, perhaps implemented as a wiki - that enables an implementer to identify himself and indicate the current status of the implementation. Ideally, that might be coupled with a ticket system in which issues are raised and closed, and comments are discussed. Ideally, this would continue into the life of an RFC, with implementations being identified ("The protocol in RFC 12345 is implemented in Andy Systems releases 22.70 and later") and associated with errata ("but we really wish that the parameter FOO had been specified").

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 26 Apr 2013 12:12:18 +0300

Hi Fred,

Thanks for your review. Responding to you, and to other similar comments 
on the list:

The draft refers to two "styles" of documenting implementation: in-line 
in the Internet draft, and by a reference to (presumably) a database or 
a wiki. The draft is also quite clear that the Implementation Status 
section should be removed from the I-D before publication. We do not 
specify what happens with Implementation wikis that are referenced by 
the I-D, presumably in most cases they will be abandoned.

The in-line style is obviously simpler to set up (no infrastructure is 
needed) and in fact, this was the path taken by the 4 I-Ds that are 
using this option to date.

There are two problems with long-term maintenance of implementation data:

- The technical infrastructure (wiki, registration, etc.) should be set 
up. This is not too difficult, and there are lots of people who'd be 
happy to implement such a thing.

- There should be long-term commitment to maintain the data. I think we 
simply don't have such processes in place, and personally I don't want 
to even try to deal with this problem. I suspect that we'd have to 
eventually use paid help if we are serious about keeping the information 
current, and I don't even think it would be worth the cost.

Thanks,
	Yaron

On 2013-04-26 01:09, Fred Baker wrote:
> _______________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 26 Apr 2013 16:07:37 +0000

On Apr 26, 2013, at 2:12 AM, Yaron Sheffer <[email-masked]>
 wrote:

> - There should be long-term commitment to maintain the data. I think we simply don't have such processes in place, and personally I don't want to even try to deal with this problem. I suspect that we'd have to eventually use paid help if we are serious about keeping the information current, and I don't even think it would be worth the cost.

Understood. That said, we already have working group wikis and errata. I don't want to trivialize the investment, but it seems like we have at least part of the infrastructure already. I'm asking what will be the best for IETF discussion and for maintenance of the information. I suspect it's something we can do if we choose to.

---

From: Dave Crocker <[email-masked]>
Date: Fri, 26 Apr 2013 09:16:25 -0700

Given this thread on the ietf list, I guess I should forward the 
following, which was done as a solicited Apps Area review for this draft:

-------- Original Message --------
Subject: Review of:  draft-sheffer-running-code
Date: Wed, 24 Apr 2013 06:38:24 -0700
From: Dave Crocker <[email-masked]>
To: [email-masked],
[email-masked],  [email-masked]

Howdy.

I have been selected as the Applications Area Review Team reviewer for
this draft (for background on apps-review, please see
http://www.apps.ietf.org/content/applications-area-review-team).

Please resolve these comments along with any other Last Call comments
you may receive. Please wait for direction from your document shepherd
or AD before posting a new version of the draft.

Review
------

Title:    Improving Awareness of Running Code: the Implementation Status
Section

I-D:      draft-sheffer-running-code-04

Reviewer:     D. Crocker
Review Date:  24 April 2013

Summary:

Given an organizational motto of "Rough consensus and running code",
implementation experience has a value built into the IETF's DNA.
However formal IETF reliance on running code for specifications has been
minimal, with its only universal occurrence being in consideration of
advanced standards status; it is notably /not/ an organizational
requirement for entry-level Proposed Standard status, if only for the
very high barrier to publication this imposes on document versions that
were intended to be preliminary.  The current draft proposes an
experiment to optionally include documentation of implementations as
part of document development, prior to issuance of a specification as an
RFC.

Within the broader perspective of pre- and post-standardization efforts,
it is quite common to document available implementations of a
specification.  (As a convenient example, see http://dkim.org/deploy.)
In effect, the current proposal merely seeks to have a common IETF-based
means of recording a type of information that industry has already
frequently formulated.

In their current form, the draft and the proposal seem likely to be
useful.  However, both could be improved.

The draft includes some language and assertions that would benefit from
being a bit more carefully written; questions, comments and suggestions
are in the detailed comments below.

The proposal itself would benefit from directly paying more attention to
existing industry practice, which is implied by the "alternatives"
section; that is, that implementation lists already happen and are
maintained after IETF document processing. The proposal should do a
better job of integrating this fact into its design.

One approach could be a relatively simple document change, to
distinguish between specifying the information to be reported, as
distinct from the means of reporting it -- think of it as payload vs.
mechanism...  Specify them separately.  Then define both how to
incorporate the information directly and the form of an external
citation to it; this would eliminate the document's section on
'alternatives' and treat the original and alternative methods as equal.

d/

Detailed Comments:

> 1.  Introduction
>
> Most IETF participants are familiar with the saying, "rough
> consensus and running code" [Tao], and can identify with its
> pragmatic approach.  However, there are many examples of
> Internet-Drafts containing protocol specification that have gone
> through to publication as Proposed Standard RFCs without
> implementation.  Some of them may never get implemented.

The "however" implies that it is wrong or problematic for Proposed to be
assigned to unimplemented protocols.  It isn't.  I suggest a softer
tone, while still noting the relevantfact. Something like:

      It is not a requirement of Proposed Standard status to have any
implementations; by the time of reaching that status, some
specifications have been implemented and some have not. Indeed, some
never get implemented.

> Over time, a variety of policies have been implemented within the

implemented -> applied

> IETF to consider running code.  In the Routing Area it used to be a
> requirement that one or more implementations must exist before an
> Internet-Draft could be published as a Proposed Standard RFC
> [RFC1264].  That RFC was later obsoleted and the requirement for
> implementation was lifted, but each working group was given the
> authority to impose its own implementation requirements [RFC4794]
> and at least one working group (IDR) continues to require two
> independent implementations.
>
> The hypothesis behind this document is that there are benefits to
> the

this -> the current

> IETF standardization process of producing implementations of
> protocol specifications before publication as RFCs.  These benefits,
> which

I suspect that that isn't its "hypothesis" at all, since the document
does nothing to specify or directly affect policies for requiring or
encouraging implementation.  Perhaps there is a hope that the proposed
notation about implementations will serve as some sort of encouragement,
but that's pretty indirect; success or failure of the proposed
experiment won't validate or invalidate whether there are benefits to
early implementation.  And providing for a notation convention hardly
constitutes an 'incentive'.

Rather, I believe the premise of the document is that the community is
aided by /more widely knowing about/ early implementations.

> include determining that the specification is comprehensible and
> that there is sufficient interest to implement, are further discussed
> in Section 4.
>

process -> mechanism

> Internet-Drafts to record the status of known implementations, by
> including an Implementation Status section.  The document defines
> (quite informally) the contents of this section, to ensure that the
> relevant information is included.  This will allow reviewers and
> working groups to assign due consideration to documents that have
> the benefit of running code, by considering the running code as
> evidence of valuable experimentation and feedback that has made the
> implemented protocols more mature.

The above last sentence seems to be the actual claimed value proposition
for the experiment.

> This document provides a mechanism to record and publicize the

This sentence seems entirely redundant.

> existence of running code.  It is up to the individual working
> groups to use this information as they see fit, but one result might
> be the preferential treatment of documents resulting in them being
> processed more rapidly.

I think it won't be the working group that 'uses' the information, but
rather IETF management and the broader community?

> The process in this document is offered as an experiment.  Authors
> of Internet-Drafts are encouraged to consider using the process for
> applying the process to all of their protocol specifications.
>
> The scope of the intended experiment is all Internet-Drafts whether

Probably not all I-Ds, since they do not all contain implementable
specifications.

>
>
> Sheffer & Farrel        Expires October 14, 2013                [Page
> 3]  Internet-Draft                Running Code
> April 2013
>
>
> produced within IETF working groups, outside working groups but

What I-Ds are produced by "outside working groups"?  I can't tell who
this refers to.It's probably just as well to remove the attempted case
analysis for groups and just say that the scope is all specifications
issued as I-Ds.

> intended for IETF consensus, or for publication on the Independent
> Stream.  However, it is expected that most benefit from the

that most benefit from the experiment will
->
the greatest benefit in the experiment will

> experiment will be seen with Standards Track documents developed
> within working groups.

Here's my guess:

1. Those offering comments or making decisions about the status of
document will be aided by this added information.

2. Those deciding about whether to write code or deploy it will be aided.

I think the latter is likely to be the greater benefit.

> The authors of this document intend to collate experiences with this
> 2.  The "Implementation Status" Section
>
> Each Internet-Draft may contain a section entitled "Implementation
> Status".  This section, if it appears, should be located just before
> the "Security Considerations" section and contain, for each existing
> implementation:
>
> o  The implementation's name and/or a link to a web page describing
> the implementation. o  A brief general description. o  The
> implementation's level of maturity: research, prototype, alpha, beta,
> production, widely used etc. o  Coverage: which parts of the protocol
> specification are implemented and which versions of the
> Internet-Draft were implemented. o  Licensing: the terms under which
> the implementation can be used. For example: proprietary, royalty
> licensing, freely distributable with acknowledgement (BSD style),
> freely distributable with requirement to redistribute source (GPL
> style), other (specify). o  Contact information: ideally a person's
> name and email address, but possibly just a URL or mailing list.

List the organization that did the implementation and contact
information for the proper place in the organization.

> In addition, this section can contain information about the
> interoperability of any or all of the implementations.
>
> Since this information is necessarily time-dependent, it is
> inappropriate for inclusion in a published RFC.  The authors should

Way to bury the lead.  This stuff won't be in the published document?
It's only for pre-publication?

> include a note to the RFC Editor requesting that the section be
> removed before publication.
>
> 2.1.  Introductory Text
>
> The following boilerplate text is proposed to head the
> Implementation Status section:
>
>
>
>
>
>
>
> Sheffer & Farrel        Expires October 14, 2013                [Page
> 4]  Internet-Draft                Running Code
> April 2013
>
>
> This section records the status of known implementations of the
> protocol defined by this specification at the time of posting of this
> Internet-Draft, and is based on a proposal described in [RFC Editor:
> replace by a reference to this document].  According to [RFC Editor:
> replace by a reference to this document], "this will allow reviewers
> and working groups to assign due consideration to documents that have
> the benefit of running code, by considering the running code as
> evidence of valuable experimentation and feedback that has made the
> implemented protocols more mature.  It is up to the individual
> working groups to use this information as they see fit".
>
> Authors are requested to add a note to the RFC Editor at the top of
> this section, advising the Editor to remove the entire section
> before publication, as well as the reference to [RFC Editor: replace
> by a reference to this document].
>
>
> 3.  Alternative Formats

Yes!

> Sometimes it can be advantageous to publish the implementation
> status separately from the base Internet-Draft, e.g. on the IETF
> wiki:
>
> o  When the Implementation Status section becomes too large to be
> conveniently managed within the document. o  When a working group
> decides to have implementors, rather than authors, keep the status of
> their implementations current. o  When a working group already
> maintains an active wiki and prefers to use it for this purpose.
>
> It is highly desirable for all readers of the Internet-Draft to be
> made aware of this information.  Initially this can be done by
> replacing the Implementation Status section's contents with a URL
> pointing to the wiki.  Later, the IETF Tools may support this
> functionality, e.g. by including such a link from the HTMLized draft
> version, similar to the IPR link.
>
> If the implementation status is published separately from the I-D,
> then this information needs to be openly available without requiring
> authentication, registration or access controls if it is to have any
> useful effects.
>
>
> 4.  Benefits
>
> Publishing the information about implementations provides the
> working group with several benefits:
>
>
>
>
> Sheffer & Farrel        Expires October 14, 2013                [Page
> 5]  Internet-Draft                Running Code
> April 2013
>
>
> o  Participants are motivated to implement protocol proposals, which
> helps in discovering protocol flaws at an early stage.

The psychological premise seems to be that getting cited in this list
will somehow serve as an incentive to implementers. I can imagine that
they'll like being listed, but find it extremely difficult to believe
that it will affect whether they do the work; not that I think it's a
deciding factor, but note that the ego value is further reduced by the
information's being ephemeral, since it it removed prior to RFC publication!

> o  Other participants can use the software, to evaluate the
> usefulness of protocol features, its correctness (to some degree) and
> other properties, such as resilience and scalability.

Efficacy of the spec.

> o  WG members may choose to perform interoperability testing with
> known implementations, especially when they are publicly available.

Interoperability of the spec.

> o  In the case of open source, people may want to study the code to
> better understand the protocol and its limitations, determine if the
> implementation matches the protocol specification, and whether the
> protocol specification has omissions or ambiguities.

Reference implementation as exemplar.

> o  Working group chairs and ADs may use the information provided to
> help prioritize the progress of I-Ds in some way.

What does "prioritize the progress" mean?

> o  Similarly, the information is useful when deciding whether the
> document should be progressed on a different track (individual
> submission, Experimental etc.).
Assess spec maturity.  (This seems redundant with the combination of
Efficacy and Interoperability?)

> o  And lastly, some protocol features may be hard to understand, and
> for such features, the mere assurance that they can be implemented is
> beneficial.

mumble.  Might highlight problems with the spec language. Might
undermine the spec text by causing code to be used in place of it?

> We do not specify here whether and to what degree working groups are
> expected to prefer proposals that have "running code" associated
> with them, over others that do not.
>
>
> 5.  Process Experiment
>
> The current proposal is proposed as an experiment.  The inclusion of
> "Implementation Status" sections in Internet-Drafts is not
> mandatory, but the authors of this document wish to encourage authors
> of other Internet-Drafts to try out this simple process to discover
> whether it is useful.  Working group chairs are invited to suggest
> this process to document editors in their working groups, and to draw
> the attention of their working group participants to "Implementation
> Status" sections where they exist.
>
> Following a community discussion, it was concluded that [RFC3933] is
> not an appropriate framework for this experiment, primarily because
> no change is required to any existing process.

It defines a document meta-data encoding mechanism. That's not really a
"process".

> 7.  Security Considerations
>
> This is a process document and therefore, it does not have a direct
> effect on the security of any particular IETF protocol.  Better

Better -> However, better

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: "Adrian Farrel" <[email-masked]>
Date: Sun, 28 Apr 2013 12:22:08 +0100

Hi Fred,

I'm in complete agreement with you, but... :-)

Before investing in a common set of tools to archive implementation information,
I wanted to see whether there was *any* intention to make that information
available. 

Thus, this is a baby-step towards the end result that you and I wold like to
see. If, after our 18 month experiment, it turns out that no-one wants to record
implementation information, we will know where we stand (or sit). OTOH, if there
is good support for the idea, we can move to the next stage.

Adrian

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Fred
> Baker (fred)
> Sent: 26 April 2013 17:08
> To: Yaron Sheffer
> Cc: <[email-masked]>; [email-masked] Discussion
> Subject: Re: [Tools-discuss] Last Call: <draft-sheffer-running-code-04.txt>
> (Improving Awareness of Running Code: the Implementation Status Section) to
> Experimental RFC
> 
> 
> simply don't have such processes in place, and personally I don't want to even
try
> to deal with this problem. I suspect that we'd have to eventually use paid
help if
> we are serious about keeping the information current, and I don't even think
it
> would be worth the cost.
> 
> Understood. That said, we already have working group wikis and errata. I don't
> want to trivialize the investment, but it seems like we have at least part of
the
> infrastructure already. I'm asking what will be the best for IETF discussion
and for
> maintenance of the information. I suspect it's something we can do if we
choose
> to.=

---

From: "Adrian Farrel" <[email-masked]>
Date: Sun, 28 Apr 2013 12:22:08 +0100

Hi John,

Seems consistent with what is in the I-D at the moment. See section 3.

Thus,  those who want to record the info in the I-D can do that, while those who
want to go straight to a wiki can do that (although we ask that the I-D has a
pointer to the wiki).

Cheers,
Adrian

> From: [email-masked] [mailto:[email-masked]] On Behalf Of John C
> Klensin
> Sent: 26 April 2013 17:51
> To: Fred Baker (fred); Yaron Sheffer
> --On Friday, April 26, 2013 16:07 +0000 "Fred Baker (fred)"
> <[email-masked]> wrote:
> 
> >
> > On Apr 26, 2013, at 2:12 AM, Yaron Sheffer
> > <[email-masked]>  wrote:
> >
> >> - There should be long-term commitment to maintain the data.
> >> I think we simply don't have such processes in place, and
> >> personally I don't want to even try to deal with this
> >> problem. I suspect that we'd have to eventually use paid help
> >> if we are serious about keeping the information current, and
> >> I don't even think it would be worth the cost.
> >
> > Understood. That said, we already have working group wikis and
> > errata. I don't want to trivialize the investment, but it
> > seems like we have at least part of the infrastructure
> > already. I'm asking what will be the best for IETF discussion
> > and for maintenance of the information. I suspect it's
> > something we can do if we choose to.
> 
> Fred,
> 
> First, I agree with both the above and with your prior note
> agreeing with the general idea and suggesting something more
> "live" than a section of an I-D.  Second, while I certainly see
> the value, I would get nervous if we were to move significantly
> toward a long-term, IETF-supported, official statement or
> compendium of implementation status.  At least unless pursued
> with great caution [1], such a thing would raise some of the
> same issues that going into the conformance testing business
> does in terms of the perception of guarantees that a given
> implementation is somehow "IETF approved".
> 
> Perhaps the right model would be to keep this material in I-Ds
> (as the proposal suggests) to support the evolution and review
> of specification documents, then to move it to a wiki or
> equivalent that was clearly identified as unofficial and for the
> convenience of the community and that was "maintained by the
> IETF" only to the extent needed to minimize spam, libel, and
> other nonsense.
> 
> It also occurs to me that an alternative to part of the
> experiment (still consistent with it, IMO) would be to start the
> wiki process earlier and use the I-Ds merely to snapshot the
> wiki at various points to help in the review process.  That
> would give both the advantages of a continually-evolving list
> and those of periodic stable snapshots.
> 
> Just a thought or two.
> 
>    john
> 
> 
> 
> [1] Images of dragging along as small pack of lawyers,
> albatross-like, are probably in order.

---

From: Yaron Sheffer <[email-masked]>
Date: Sun, 28 Apr 2013 18:03:56 +0300

Hi Dave,

Thank you for your thorough review. While I accept many of your
comments, I must say I disagree with you on a few points, which together
go to the core of our motivation in writing this document. Thank you for
helping me clarify these points to myself :-)

- Our goal is much *less* ambitious than defining a framework for
documenting implementations for long-term protocol projects. Our primary
goal is to aid working group members in making informed decisions about
the quality of specifications. I believe this narrow focus is much more
realistic.

- As a result of the above: 1. We define the information as pre-RFC
only; 2. The main medium is the I-D, as opposed to Web pages/wikis/CMS
(although we see them as valid alternatives); 3. We focus on WG
documents, as opposed to "all" IETF documents.

- Despite the glowing counterexample of DKIM, it is rarely practical to
maintain implementation status information, keeping it current for years 
(put bluntly, you need to pay someone to do that). The IETF certainly 
does not have the processes or mechanisms to do so. And we do not try to 
establish such processes here.

Please see my detailed comments below.

Thanks,
	Yaron

> aided by /more widely knowing about/ early implementations.

This is splitting hairs: there would be no benefit in informing the 
community if there wasn't any benefit in the actual early 
implementations. But fine.

> The above last sentence seems to be the actual claimed value proposition

I agree.

> rather IETF management and the broader community?

For most documents, AFAIK, most work is done by the WG, and a lot less 
by the rest of the community. Hence the above.

> analysis for groups and just say that the scope is all specifications

In fact others have commented that we should exclude individual 
submissions for process reasons.

> It's only for pre-publication?

Very clearly YES.

> information's being ephemeral, since it it removed prior to RFC

Actually not (subtly so). The premise is making this information public 
and an explicit part of WG deliberations will incentivize 
implementations (i.e. not the ego factor at all).

> What does "prioritize the progress" mean?

When there are multiple competing proposals to solve some problem.

> undermine the spec text by causing code to be used in place of it?

I'm with you on both mumbles.

---

From: Dave Crocker <[email-masked]>
Date: Sun, 28 Apr 2013 15:06:08 -0700

On 4/28/2013 8:03 AM, Yaron Sheffer wrote:
> the quality of specifications.

Forgive me, but working group members seem like the /least/ interesting 
target audience, since it is already the most-informed group, by virtue 
of its direct activity over the course of developing the specification.

The people who are most likely to benefit from pointers to 
implementation details are:

    1.  Reviewers, who want a sense of implementation history to 
'season' their analysis;

    2.  IESG members, who are deciding whether to vote approval for the 
specification; and

    3.  Those considering adoption of the technology for use, such as 
other implementers and service operators.

That said, what you propose certainly accommodates #1 and #2, without 
needing to agree on their being a target audience...

[However the small discussion, farther down, about being better able to 
evaluate competing proposals, does provide an example of benefit 
/within/ the working group...]

> documents, as opposed to "all" IETF documents.

Right.  And here my suggestions were merely to clarify this earlier in 
the proposal document.

> (put bluntly, you need to pay someone to do that).

Would that I'd been getting paid...  But yeah,it  takes continuing 
effort.  (On the other hand, there isn't much on-going work, after 
initial bursts of development activity.)

At any rate, rather than insisting that the list be external and 
on-going, my intention was to suggest designing it to comfortably 
/allow/ the wg to make the choice of either, gracefully, including an 
easy transition from one to the other.

  - - - - -

> by the rest of the community. Hence the above.

IETF Working groups do not create market demand and do not do develop or 
deployment.  All of that is quite explicitly outside the scope of the 
IETF, and at industry-scale, they represent more aggregate effort than 
is put in by the IETF working group...

> submissions for process reasons.

Right.  That suggests writing the scoping text a bit more tightly.

> implementations (i.e. not the ego factor at all).

The statements you are making about "incentives" to implementers really 
are in terms of their ego, since you have not stated any other benefit 
that will accrue to /them/.  There will be benefits to the community at 
large, but the question in this section of text is what the motivation 
of the implementer will be.

> When there are multiple competing proposals to solve some problem.

I think you are trying to say something like:

    By having early implementation examples for competing proposals, the 
working group can better evaluate choices among them.

d/

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Apr 2013 12:50:44 -0400

Fred,

Just a thought or two.

   john

---

From: John C Klensin <[email-masked]>
Date: Sun, 28 Apr 2013 10:35:47 -0400

--On Sunday, April 28, 2013 12:22 +0100 Adrian Farrel
<[email-masked]> wrote:

> Thus,  those who want to record the info in the I-D can do
> that, while those who want to go straight to a wiki can do
> that (although we ask that the I-D has a pointer to the wiki).

Indeed.  I was trying to make that point in a different way.  To
be more explicit, I think the I-D is fine, precisely because it
allows some reasonable flexibility about that sort of option.
I'd encourage the experimental evaluation process to compare
those two options in practice (e.g., by including statistics
about which method is used in the Summary Report (Section 5.2)
and commenting on relative utility if there is anything to say).
But there is nothing in the I-D that would prevent that and I'm
not convinced that more specifics in this I-D would be worth the
trouble.

   john