Subject: [xml2rfc] Development version dislikes <!ENTITY RFC2119 PUBLIC ...

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Jul 2011 09:33:45 +1200

What's going on with the development version at http://xml.resource.org/experimental.html?
It barfs on what I thought was a bog standard format since Day One of xml2rfc:

<!ENTITY RFC2119 PUBLIC '' 'http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119'>

gives

ERROR: unable to parse the XML document:
   INPUT: StartTag: invalid element name, line 18, column 2

The production version is fine.

Regards
   Brian Carpenter

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Jul 2011 00:01:36 +0200

> The production version is fine.

Isn't there a ".xml" missing?

Best regards, Julian

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 25 Jul 2011 23:26:36 +0100

rumour has it (source= RFC Editor desk at ietf81) that what you are 
seeing is the redeveloped code.  Truly bleeding edge.  But I only heard 
this about 1 hour ago and I haven't done my testing yet.

/Elwyn

Brian E Carpenter wrote:
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc
>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Jul 2011 00:35:57 +0200

On 2011-07-26 00:26, Elwyn Davies wrote:
> ...

Well,

   http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119

returns 404. If this worked before, it was an ugly hack.

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Mon, 25 Jul 2011 18:57:47 -0400

Yes, xml.resource.org/experimental.html is now pointing at the new 
python-based xml2rfc code, as of the code sprint on Saturday. (Up until 
Saturday, experimental.html did an auto-redirect to index.html.)

In my own testing, about 1/6 of the current internet drafts that XML 
pass through with no issues whatsoever. The new code is *much* pickier 
about certain things, even pickier than the "strict mode checking" that 
has been the default setting on the standard page since March.

One of the conversations I had yesterday with a number of people was: of 
the things that it's being picky about, which are things we really 
*want* it to be picky about? Julian notes that the ENTITY reference is 
missing the trailing .xml, which I'm surprised actually worked. Is that 
something that *should* be fixed? My feeling is that the answer should 
be "no".

     Tony Hansen

PS. Note that, as Elwyn puts it, this truly is bleeding edge. In 
addition to the pickiness mentioned above, a variety of the input files 
cause exceptions, all of which need to be fixed.

On 7/25/2011 6:26 PM, Elwyn Davies wrote:
> seeing is the redeveloped code.  Truly bleeding edge.  But I only 
> heard this about 1 hour ago and I haven't done my testing yet.
>
> /Elwyn
>
> Brian E Carpenter wrote:
>> What's going on with the development version at 
>> http://xml.resource.org/experimental.html?
>> <!ENTITY RFC2119 PUBLIC '' 
>> 'http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119'>
>>
>> gives
>>

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Jul 2011 11:56:17 +1200

> Isn't there a ".xml" missing?

It's missing in most draft*.xml files I have made since 2005.
However, fixing this does not change the error message. A few
experiments show that it is the name RFC2119 (or anything I
have tried replacing it with) that causes the barfing.

A sample file xml2rfc-1.xml that Elwyn provided with his original
tutorial in 2006 doesn't show this problem, although it uses the
identical form of ENTITY declaration. So the problem is hiding
elsewhere in my file.

On 2011-07-26 10:57, Tony Hansen wrote:

> In my own testing, about 1/6 of the current internet drafts that XML pass through with no issues whatsoever. The new code is *much* pickier about certain things, even pickier than the "strict mode checking" that has been the default setting on the standard page since March. 

Which is really not OK; I should have specified that I use "Fast" mode which is supposed
to relax the checking.

Right now this version is nowhere near being a candidate for production use.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Jul 2011 14:39:55 +1200

It appears that the new code requires ENTITY definitions
to be embedded within the DOCTYPE declaration. This has never been
the case before. Thus:

<!DOCTYPE rfc SYSTEM "rfc2629.dtd" [

<!ENTITY RFC2119 PUBLIC ''
  'http://xml.resource.org/public/rfc/bibxml/reference.RFC.2119.xml'>

]>

I suspect a lot of existing source files would be invalidated by this
requirement.

Regards
   Brian

> On 2011-07-26 11:56, Brian E Carpenter wrote:

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Jul 2011 15:01:56 +1200

It appears that the new code requires <list></list> to be embedded
in <t></t>, even in "fast" checking mode. The old version tolerates
bare <list></list> (in "fast" mode).

That's another significant compatibility problem for older source files.

With that fix and the previously noted <!ENTITY fix, my source file
now works with the new version.

Regards
   Brian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 26 Jul 2011 13:14:42 +0900

Hello Brian,

> I suspect a lot of existing source files would be invalidated by this

Sorry, but that would actually be a good thing. Free-standing ENTITY 
declarations are not allowed by XML. Please see:

http://www.w3.org/TR/REC-xml/#NT-doctypedecl
http://www.w3.org/TR/REC-xml/#NT-EntityDecl

This slipped through the cracks up to now because the Tcl implementation 
was very ad-hoc. The new implementation uses a parser library, I guess 
(would be crazy if it didn't).

Of all things, we don't want to give the impression that the IETF is 
calling something XML that isn't.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 26 Jul 2011 13:18:06 +0900

On 2011/07/26 12:01, Brian E Carpenter wrote:
> It appears that the new code requires<list></list>  to be embedded
> in<t></t>, even in "fast" checking mode. The old version tolerates
> bare<list></list>  (in "fast" mode).
>

It's not only a problem for older files, it's also a problem for people 
who don't know xml2rfc specifics and are used to other, more 
straightforward markup. I very much encourage the team to evolve the DTD 
so that there is no need anymore for <t><list><t><list> nesting.
(I can give help with the DTD if necessary.)

Regards,    Martin.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 26 Jul 2011 16:37:09 +1200

On 2011-07-26 16:14, Martin J. Dürst wrote:
> Sorry, but that would actually be a good thing. 

Yes, if it is carefully announced and introduced at a specific date.
I have no trouble with fixing my source files, but I need to be told.
(The diagnostic was quite helpful in that it pointed to exactly
the line that was causing the problem, but I could only isolate the
problem by a line-by-line comparison with Elwyn's future-proof
tutorial example. That's not satisfactory for the entire user
population.)

    Brian

> Free-standing ENTITY

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Jul 2011 00:24:39 -0500

On Mon, Jul 25, 2011 at 11:18 PM, "Martin J. Dürst"
<[email-masked]> wrote:
> It's not only a problem for older files, it's also a problem for people who
> don't know xml2rfc specifics and are used to other, more straightforward
> markup. I very much encourage the team to evolve the DTD so that there is no
> need anymore for <t><list><t><list> nesting.

I've been doing this right for years, so I won't be affected, but,
hear hear!  It's quite annoying to have to embed list tags in t tags.

Nico
--

---

From: Tony Hansen <[email-masked]>
Date: Tue, 26 Jul 2011 10:05:31 -0400

> Right now this version is nowhere near being a candidate for 

*Absolutely* not ready. But we needed to be able to start doing tests of it.

     Tony Hansen

---

From: Tony Hansen <[email-masked]>
Date: Tue, 26 Jul 2011 10:53:34 -0400

> tutorial example. That's not satisfactory for the entire user

One of the things we need to do before rolling out the new version is to 
start figuring out what some of those common "errors" are. For those 
that can be mechanically converted, it might be worth having a tool to 
help with that. If nothing else, we definitely need to collect a list of 
the changes that have been found to be required, with before and after 
examples.

To that end, did your code just not have the DOCTYPE declaration before? 
Or was it there but just not surrounding the ENTITY declarations?

     Tony Hansen

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 26 Jul 2011 16:21:03 +0100

Hi.

I just tried out an old draft that I know contains some unusual features.

Unfortunately it didn't get as far as checking on these.

I get a lot of error messages related to entity references:
My imported references are in the form

*<!ENTITY* RFC3471 SYSTEM "http://xml.resource.org/public/rfc/bibxml/reference.RFC.3471.xml"*>

The error messages are of the form:
*ERROR: Unable to validate the XML document: INPUT
  Line 1406: IDREF attribute target references an unknown ID "RFC3471"

The problem is doubtless that there are a number of different forms for these ENTITY definitions!

Regards,
Elwyn

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 26 Jul 2011 16:24:12 +0100

Hi.

Should have said draft is 
http://tools.ietf.org/id/draft-irtf-routing-reqs-11.xml

*<!ENTITY* RFC3471 SYSTEM 
"http://xml.resource.org/public/rfc/bibxml/reference.RFC.3471.xml"*>

The problem is doubtless that there are a number of different forms for 
these ENTITY definitions!

Regards,
Elwyn

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 26 Jul 2011 18:14:15 +0100

Yes. I checked that..

/E
Brian Rosen wrote:
> Are your ENTITYs in DOCTYPE:
> That is what Brian Carpenter found was needed.
>
> Brian
>
> On Jul 26, 2011, at 11:24 AM, Elwyn Davies wrote:
>
>   
>> Hi.
>>
>> Should have said draft is http://tools.ietf.org/id/draft-irtf-routing-reqs-11.xml
>>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Jul 2011 20:01:38 +0200

> these ENTITY definitions!

That may be an actual bug; my validators do not see any (serious) 
problems with that source file.

Best regards, Julian

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 26 Jul 2011 19:27:55 +0100

> problems with that source file.
:-)
Well it certainly used to go through earlier versions of xml2rfc! And 
the RFC Editor was happy with it!

The main reason for trying that draft was that it uses a numbered list 
that spans many sections which is a relatively unusual feature.

I know there is already a set of drafts/rfcs as test docs... Do we have 
a document that provides an (arcane) test of as many features as we 
think of?

Maybe the project plan includes a set of automated regression tests of 
this kind - if not I think that would be a useful idea.  Perhaps I ought 
to expand my  example file that I did when I did the xml2rfc tutorial.  
Would that be helpful or has it been done already?

Regards,
Elwyn

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Jul 2011 20:32:29 +0200

> a document that provides an (arcane) test of as many features as we

I have: <http://greenbytes.de/tech/webdav/rfc2629xslt/testcase.html>

Best regards, Julian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jul 2011 10:33:38 +1200

> hear hear!  It's quite annoying to have to embed list tags in t tags.

In some cases it's worse than annoying, because you have to resort
to an ugly <vspace> construct to get "paragraph" breaks in numbered
lists containing embedded lists.

Yes please somebody fix this in the DTD!

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Jul 2011 10:40:09 +1200

On 2011-07-27 02:53, Tony Hansen wrote:
...
> Or was it there but just not surrounding the ENTITY declarations?

The second case - the ENTITY declarations stood alone, below the
DOCTYPE. To someone whose brain is not wired with XML, there's no
logic in requiring the embedding (and I've been using SGML in one form
or another for about 25 years, so this isn't exactly my first run-in
with picky syntax).

And where did I copy it from? I can't say; probably from somebody
else's xm2rfc source around 2006.

I agree that logging these issues is the first step towards dealing
with them when the new version is considered ready for release.

   Brian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 27 Jul 2011 11:22:09 +0900

Hello Elwyn,

>>> http://tools.ietf.org/id/draft-irtf-routing-reqs-11.xml

I'm a bit confused here. I have looked at this, and it only contains
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">, no ENTITY declarations. Can you 
check again?

Regards,    Martin.

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 28 Jul 2011 11:22:38 +0200

On 26 July 2011 06:18, "Martin J. Dürst" wrote:

> I very much encourage the team to evolve the DTD so
> that there is no need anymore for <t><list><t><list>
> nesting.

Above all there should be a "stable forever" DTD
published in a 2629bis for the purpose of its
registration as public IETF ID in the IANA registry,
where absolutely nobody can touch a single erroneous
comma in it without involving the IAB.

> (I can give help with the DTD if necessary.)

-Frank

---

From: Frank Ellermann <[email-masked]>
Date: Thu, 28 Jul 2011 11:43:21 +0200

On 26 July 2011 06:14, "Martin J. Dürst" wrote:

> Of all things, we don't want to give the impression
> that the IETF is calling something XML that isn't.

IIRC the W3C is calling something "URL" in newer XML
versions that is no STD 66 URL (or at least 3987 IRI).

...but of course DTD subsets should follow XML rules.
This used to work as expected in 2008 (xml2rfc happy,
w3 validator happy, author happy).

-Frank