Subject: Re: [Tools-team] Secretariat Comments on draft-ietf-tools-draft-submission-07

From: "Alex Rousskov" <[email-masked]>
Date: Tue, 05 Apr 2005 17:35:45 -0600

On Wed, 2005/03/16 (MST), <[email-masked]> wrote:

>>> Comments on "Requirements for IETF Draft Submission Toolset"
>>> draft-ietf-tools-draft-submission-07
>>> 1. Section 3, "Scope"
>>> Notification and expiration/archiving of old draft versions may be out
>>> scope for this toolset.  However, the toolset must be compatible with  
>>> the Secretariat's tools for sending notifications and creating  
>>> tombstones.
>>
>> Agreed.
>
> Ok

Not sure what this means in practice because the Toolset will be created  
by the Secretariat. So, the Secretariat will make the Toolset compatible  
or adjust other tools (or both), whatever they feel is the best approach.  
Are there any specific rules we should add to address the above comment? A  
"must be compatible with existing tools" seems like a too rigid rule  
because the existing tools can be changed if that's the best way to  
achieve compatibility.

>>> 2. Section 4, "Notation and Terminology"
>>> WG draft:  All drafts that are WG drafts begin with "draft-ietf."   
>>> However,
>>> not all drafts that begin with "draft-ietf" are working group drafts.   
>>> (Also see comment #6b.)
>>
>> The usage of this definition within the requirements document means that
>> the document does not require that the tool recognize WG drafts on any  
>> other basis than the "draft-ietf-" prefix.  If you feel that it is  
>> necessary to use a more complex lookup or algorithm to recognize WG  
>> drafts, I think that would be OK; however I'm not sure that it would  
>> actually be necessary for this tool to have that ability.
>>
>> Maybe it would be better to use some other term than "WG Draft" here -  
>> maybe we should call it "A draft following the WG draft naming  
>> convention" ...
>> although that is maybe a bit long ,:-)
>
> Agreed to try to find another expression than "WG drafts" for drafts  
> which are named "draft-ietf-", and update the document accordingly.

It looks like we failed to find a better expression, right? I have  
suggested "Group draft", but that seems like a bad solution after I  
re-read the above comment. Should I just add a note saying that there are  
WG drafts that are not considered WG drafts according to our definition,  
and that this document is not going to address that "problem"? Other  
solutions?

>>> 3. Section 6, "Overall Toolset Operation"
>>> (a) Manual adjustment and posting
>>> It is not clear why manual adjustment of meta-data and manual posting  
>>> are
>>> linked.  That is, why is it *mandatory* that manual adjustment of  
>>> meta-data
>>> be followed by manual posting by the Secretariat?  The option to post
>>> manually should always be available.  However, the option to adjust  
>>> metadata
>>> and post automatically should also be available.
>>
>> Ok, if this is not clear then it needs more explanation in the draft;  
>> but if I briefly try to explain how we have been thinking, then maybe  
>> we can see if this requirement makes sense or not.  Below:
>> <cases snipped>
>>
>> Agreed, this makes sense and is what we also had in mind.

So no changes are needed, right?

> The tool needs to make clear that the secretariat will not accept meta-
> data which is different than those given in the draft, so if the
> extracted data needs tweaking, and there is no extraction error, the
> data should be corrected in the draft, and the draft re-submitted.

I have added the following text:

    The manual validation and posting route is a Toolset bypass mechanism
    (see Section 15) not meant for fixing problems with the draft itself.
    The Secretariat does not generally correct submitted drafts.  If the
    draft needs tweaking to match submitter's intent, then the draft
    should be corrected by the submitter and resubmitted.

>>> (b) Hosting the toolset at an open-source-friendly project management  
>>> site.
>>> Should "toolset" be changed to "source code"?  It does not make sense  
>>> for the toolset to be hosted at such a site since the Secretariat will  
>>> need to modify the toolset from time to time to accommodate changes in  
>>> procedure, requests for new features, etc.
>>
>> Quite correct, the intention was to have the source code open, not to  
>> host the tool elsewhere.

The draft now says:

     Hint: Hosting the Toolset sources at an
     open-source-friendly project management site like SourceForge.net would
     provide the IETF community with decent, ready-to-use interface to  
access
     the code, documentation, bug reports, and discussion forums.

Note that this is an informal hint only! We do not require any hosting at  
all.

>>> 4. Section 7, "Upload Page"
>>> The Secretariat does not currently accept XML submissions.  The  
>>> Secretariat
>>> can move towards accepting XML submissions in the future.
>>
>> Yes.  The tools team see this as a policy decision, which has to be  
>> made by the IESG I believe, before it can be put into effect.
>
> Agreed.  This goes to the IESG.

No changes in the draft. The fact that accepting XML is a policy change is  
already documented in Appendix A.

>>> 5. Section 8, "Check Action"
>>> (a) The third paragraph states that "It is an error to submit a draft  
>>> which has neither plain text nor XML sources format (R68/a)."  The  
>>> "/a" implies that XML format will be included in the phase "a"  
>>> development.  However,
>>> Section 17, "Implementation Stages," states that development phase "a"
>>> "focuses on plain text draft submission via the web interface."   
>>> Focusing
>>> *only* on plain text draft submission in phase "a" seems appropriate.
>>
>> Yes, agreed, this was the intention.
>
> Ok

I am guessing that no changes are needed because /a does not imply any  
special action for XML sources when XML sources are not supported (or we  
would have to rewrite many rules to add "except when X is not yet  
supported"!).

>>> (b) The third paragraph also states that "PDF- or PostScript-only  
>>> drafts cannot be posted without Secretariat involvement."  This  
>>> statement is not clear.  First of all, if "it is an error to submit a  
>>> draft that has neither plain text nor XML sources format," then PDF-  
>>> or PS-only submissions are not allowed, and such drafts cannot be  
>>> posted with or without Secretariat
>>> involvement.
>>> The Secretariat currently does not accept "PDF- or PostScript-only"  
>>> drafts. A text version is *always* required.
>>
>> Ok.  This point was not clear to us.  The text should be corrected
>> accordingly.
>
> Ok. Correction needed.

Not sure what to do here. The fact that a submission has validation errors  
does not mean that the draft cannot be posted. It only means that  
automated posting is not possible. See the Bypass section. Secretariat can  
reject manual submissions for a variety of reasons not documented in our  
draft. On the other hand, I agree that we should not imply that manual  
submission of PDF-only drafts might work. Here is a polished text. Please  
let me know if any more changes are needed:

    It is an error to submit a draft which has neither plain text nor XML
    sources format (R68/a).  XML source is acceptable without
    accompanying plain text only if the Toolset successfully generates a
    draft in plain text format from the XML source, as a part of the
    processing step documented below (R69/b).  These rules imply that
    PDF- or PostScript-only drafts cannot be auto-posted.  Moreover, even
    manual Secretariat involvement cannot help with posting these drafts
    per IETF policy to always require a plain text format in addition to
    PDF or PostScript.  Furthermore, drafts containing PDF or Postscript
    format must not be auto-posted until the Toolset can validate that
    their content matches plain text format (R143/a).

>>> Furthermore, the Secretariat currently
>>> does not review PDF or PS submissions to ensure that they are  
>>> identical to
>>> the text version.  Rather, the staff posts them "as is."  If the rules
>>> change, and PDF- or PS-only drafts are allowed, then they should be
>>> auto-posted.  If PDF and PS drafts continue to be allowed as auxilliary
>>> formats, AND if the procedures require that the content of these  
>>> drafts is
>>> identical to the content of the text versions, then the comparisons  
>>> should be
>>> automated as well.
>>
>> No objections, makes sense to me.
>
> Ok.  Check if clarification needed.

I think the description above does not contradict and is mostly covered by  
the above draft text, including R143/a. We do not say that the Secretariat  
does not review PDF or PS formats, and I think that is OK. We only say  
that if your submission includes PDF, then you have to go manual route.  
The gray area is a result of a conflict between Secretariat abilities to  
compare PDF with text and IETF rules requiring submission of both. I do  
not think we have to address that problem in this draft.

>>> 6. Section 8.4, "Extraction"
>>> (a) The first sentence states that "The toolset extracts meta-data  
>>> from the
>>> following stored draft formats: plain text, XML, and other."  If  
>>> either (i) a
>>> text version of a draft, or (ii) an XML version that can be converted  
>>> to text
>>> is always required, then why is it necessary to be able to extract  
>>> data from
>>> any format other than text?  Only if the rules change, e.g., if PDF- or
>>> PS-only submissions are permitted, would it be necessary to provide  
>>> this feature.
>>
>> Correct.  One completely valid way of extracting the meta-data from the  
>> XML file would be to convert it to text and do the extraction there.   
>> We don't want to *require* that it be done this way, however - it's  
>> just as fine from a requirement viewpoint to do extraction directly  
>> from XML.  Up to the
>> implementor, really.
>
> Ok.  Check if clarification needed.

We require extraction from all formats because we want to validate that  
all metadata matches across all formats (in addition to and before we can  
validate that the contents matches as well). Once the Secretariat starts  
implementing this, they will hit this normative text:

    1.  All available meta-data entries must match across all submitted
        draft formats (R18/a).  For example, if the interpreter managed
        to extract a draft title from plain text and PDF format, both
        titles must match.  This requirement prevents accidental
        submission of mismatching formats.

There are other rules that clarify that you do not have to consider  
yet-unsupported formats in initial Toolset implementations.

I agree that it is sometimes possible to extract XML metadata by  
generating a plain-text version and extracting metadata from there. This  
hack can be a good starting point. However, I hope that mature Toolset  
versions will not use that hack. I am sure one can come up with a test  
case that exposes the dangers of this approach. And in any case, note that  
this hack is still much better than just trusting the plain text version  
submitted along with XML!

>>> (b) WG ID and WG flag: The word "tools" in
>>> draft-ietf-tools-draft-submission-07 is not a WG ID.  The tools team  
>>> is not a working group.  The WG flag would be "False" for this  
>>> document.
>>
>> Agreed.
>
> Ok.  Check if clarification needed.

Hmm... For the purposes of this document, Tools team is equivalent to a  
working group, but I have changed draft-ietf-tools-draft-submission to  
draft-ietf-sieve-vacation

>>> (c) creation date: Why is it necessary to note the "creation date"?   
>>> The "submission date" is the date that is relevant in this process.   
>>> The
>>> submission date is assigned by the system.  It is not extracted from  
>>> the draft.  (Also see comment #7)
>>
>> The idea is to have this part of the document meta-data be in agreement  
>> with the system's view of how long the document is valid.  Currently,  
>> there is no check on this, with the result that it is simply not  
>> possible to rely on the expiration date you see when you read a draft -  
>> and this is unfortunate...
>>
>> We should probably change the document to indicate that it's not really  
>> the creation date, but rather the Expiration date which is given in the  
>> draft which it is desirable to have in reasonable agreement with the  
>> system's
>> view of expiration date. [...]
>
> Ok.  Clarification needed in draft?

IMHO, the current draft requires the right thing and the Secretariat tools  
need to be adjusted (eventually) to support the requirement.

We should strive to base our requirements on the original/source data and  
not on the derivatives. Expiration date is a derivative of the draft date.
The draft date is encoded in the draft. If the system tries to assign its  
own date, then there will be a mismatch between draft metadata and draft  
text.

We can argue whether the draft date is the Creation Date, Submission Date,  
or Posting Date, but current requirements make such arguments silly  
because they ensure that all those dates are close enough. Thus, the only  
thing that matters is the match between draft text and metadata. Allowing  
the system to assign dates leads to mismatches. Requiring the Toolset to  
validate dates ensures a match.

>>> (d) expiration date: The expiration date is calculated by the system  
>>> .  It
>>> is not extracted from the draft.
>>
>> Ok.
>
> Ok.

Again, the current approach is broken IMO. The expiration date should be  
extracted from the draft, validated, and rejected if needed. See above for  
supporting arguments.

>>> 7. Section 8.5.1, "Absolute Requirements"
>>> Item 7.  Once again, why are we concerned about when the draft was  
>>> created?
>>
>> See above.

>>> 8. Section 8.5.4, "WG Approval"
>>> WG chair approval is mandatory for posting of version 00 WG drafts.
>>> Therefore, a mechanism for obtaining WG chair approval for posting of  
>>> these drafts should be included in this toolset.
>>
>> My personal viewpoint: Absolutely.
>
> Ok.

There is a logical disconnect here. Just because something is mandatory  
does not imply that the draft has to document the mechanism for  
implementing that something. For example, the draft has no requirements  
related to checking the language of the submission.

>> The team could not agree on exactly what the IETF requirements were on  
>> this point, if any, and has left this open to the implementor, to  
>> decide exactly how to implement this mechanism.

And I still think it was an acceptible decision. The implementor can  
document what they have implemented, and the IESG can verify that the  
implementation intends to do the right thing. Alternatively, somebody may  
write another draft with requirements specific for chair approval.  
Moreover, first versions of the toolset may require manual submission for  
00 versions of WG drafts.

>>> 9. Section 9.1, "External Meta-Data"
>>> (a) Submitter email address.  If the submitter is not a "lawful  
>>> submitter,"
>>> then why send the draft to the Secretariat?  What action would the
>>> Secretariat take?  Currently, the Secretariat does not make decisions  
>>> on the
>>> the lawfulness of a submitter.  If the tool determines that a  
>>> submitter is
>>> not lawful based on rules established by the IESG, then the submission  
>>> should
>>> be rejected.

What if a tool was wrong? For example, perhaps the tool failed to  
correctly extract author's info.

>> Ok.  We believed that the secretariat verified (for non -00 drafts)  
>> that the submitter was either one of the authors, or one of the WG  
>> chairs or ADs.  If this is incorrect, the IESG will have to decide  
>> whether such a check should be done before this is incorporated in the  
>> tool.
>
> Harald and Brian in the meeting indicated that they would like to not  
> change the rules for this at this point.  For now, there is no such  
> concept as a
> lawful submitter.  The draft needs to be updated, and this requirement  
> removed or described as a possible future enhancement.

I am afraid we need to revisit this. I misunderstood the implications when  
reading this two weeks ago. If we remove requirements related to the  
submitter identity, does it mean that I can post any new version of any  
IETF draft out there? What would stop me? Or is the plan to still say that  
the submitter has to be one of the authors, but not call it "lawful"?  
Today, we have the Secretariat to at least slow down those who submit fake  
drafts. With the web interface, a kid can write a script to update all  
drafts with a "funny" picture.  Please clarify!

>>> (b) List of drafts obsoleted by this draft.  An RFC can be "obsoleted"  
>>> by another RFC.  However, an Internet-Draft cannot be "obsoleted" by  
>>> another Internet-Draft.  It can only be "replaced" by another  
>>> Internet-Draft.
>>
>> Right.  I believe this is just bad terminology in the draft - should  
>> probably be re-written to use "removed" and/or "replaced.
>
> Ok.

IIRC, this 3rd-phase feature was requested by somebody. Changed to  
"replaced".

>>> (c) Indicating that an Internet-Draft replaces another Internet-Draft
>>> should not be optional...it should be mandatory.  The submitter must  
>>> indicate if an Internet-Draft replaces another Internet-Draft.
>>
>> Agreed.
>
> Ok.

I am not sure I follow. We do not have any requirements for submitters,  
only for the Toolset. Since not all drafts replace other drafts, it seems  
that the information is optional from the Toolset point of view. What am I  
missing here?

>>> 11. Section 13, "Post Manually Action"
>>> The "post manually" action should always be available, and should be  
>>> used if the automated posting process is not functioning properly.   
>>> However, the
>>> Secretariat does not make exceptions with respect to the posting of
>>> Internet-Drafts.  If an Internet-Draft is non-compliant, then the staff
>>> rejects it.  Therefore, if an Internet-Draft is non-compliant, then the
>>> submitter should not attempt to circumvent rejection by sending the  
>>> draft directly to the Secretariat.  As long as the toolset is working,  
>>> an
>>> Internet-Draft can be posted automatically.  If the toolset is not  
>>> working,
>>> then all Internet-Drafts must be posted manually...and the Secretariat  
>>> should
>>> receive lots of bug reports!!
>>
>> Agreed.  This agrees with the intention of the requirements document,  
>> but may be insufficiently clearly stated there.
>
> Ok.  Check if clarification needed in document.

I think there was a general misunderstanding on the part of the  
Secretariat that the Toolset somehow requires them to fix drafts or post  
broken drafts. I hope that is cleared now. The manual posting route is one  
of the most efficient bug reporting interfaces I can think of; if  
implemented correctly, it can supply the Secretariat with all the vital  
information in one bundle: what the submitter thinks should be happening,  
what the toolset says, and what the draft is. If you can think of specific  
ways to improve the draft in this context, please let me know.

Thank you,

Alex.

_______________________________________________
Tools-team mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tools-team

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 06 Apr 2005 09:26:01 +0200

> because the existing tools can be changed if that's the best way to  

Agreed.  Maybe add a note saying pretty much what the secretariat comment
says, to the scope section?

> and that this document is not going to address that "problem"? Other  

"WG-named draft" ?

> So no changes are needed, right?

Right.

Possibly you could add something which Brian Carpenter pointed out, 
which is that before letting the toolset go live, we have a corpus
of thousands of recently submitted drafts with which we can test
the meta-data extraction algorithms, to be sure that meta-data
extraction will succeed in almost all cases, and thus avoid a large
percentage of drafts being sent on for manual handling because of
meta-data extraction failure.

>     should be corrected by the submitter and resubmitted.

Looks very good to me.

> Note that this is an informal hint only! We do not require any hosting at  

This still uses the 'hosting' word, which I think is unnecessary and
makes some people nervous - what about "Placing the toolset source
repository on an open-source friendly ..." ?

I think we want to give a stronger requirement than a hint about the open
source repository, so we don't need to back off this much in that
regard. The important thing is that it's clear that we're not talking
about hosting the tool itself there.

> already documented in Appendix A.

Right.

> would have to rewrite many rules to add "except when X is not yet  

Logically, I agree.  Since they have picked on this particular item,
it might still be good to add a note at this point.

>     their content matches plain text format (R143/a).

s/per IETF policy/as the IETF policy is/

Maybe s!(R68/a)!(R68/a) (XML is only an option in stage b.)!

> not think we have to address that problem in this draft.

There may still be a gap between the requirement that the toolset be able
to verify the pdf / ps drafts before permitting auto-posting, but no
verification being required on manual posting.  Not sure how to handle that
best.

> submitted along with XML!

Agreed.  Maybe add a para. pointing out the logic in the first para.
of your comment above?

> validate dates ensures a match.

I think there is still one side of this which you don't touch here, which
might warrant a clarification, which is that the draft expires 153 days
after it was posted, not 153 days after it was created or submitted.  In
this respect, the system date is more correct than the date in the draft
text; and we don't want to simply note the creation date - we want to 
ensure that the meta-data match - i.e. posting date and expiration date
as given in the draft aren't too much out of whack.  

> extracted from the draft, validated, and rejected if needed. See above for  

Agreed.  But since we haven't managed to communicate this well enough, some
kind of fix to the text seems to be needed.

> 00 versions of WG drafts.

Sure.  We don't need to change this.

Agreed.

My suggestion here would be to remove the concept of lawful submitter,
but require a successful verification of the submitter for automated posting,
for reasons such as the one you give above.

Ok.

> that the information is optional from the Toolset point of view. What am I  

Nothing.  The draft is good as is.

> ways to improve the draft in this context, please let me know.

Does the draft currently suggest or require a "Why manual posting is
deemed necessary" field?  (I think so but don't remember clearly).  If
so, maybe point out also in the draft that this is an excellent field
for reporting bugs :-)

	Henrik