Subject: [xml2rfc] include processing directive

From: "Jim Schaad" <[email-masked]>
Date: Sun, 10 Nov 2013 19:50:31 -0800

There are currently a number of restrictions on the current system for the
include processing directive.  I am wondering how many of these should be
kept and which should be fixed.

1.       They cannot be recursive

2.       They will incorrectly deal with entities that are declared in the
main document

3.       Included files MUST be have a single root node

The only one that I think really needs to get fixed is #1 - to deal with
ticket #106, but I would be interested in opinions on the others.

I do expect to file a tracker issue as soon as I am sure my list is
reasonably exhaustive.

Jim

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Nov 2013 10:14:49 +0100

On 2013-11-11 04:50, Jim Schaad wrote:
> There are currently a number of restrictions on the current system for
> the include processing directive.  I am wondering how many of these
> should be kept and which should be fixed.
>
> 1.They cannot be recursive
>
> 2.They will incorrectly deal with entities that are declared in the main
> document
>
> 3.Included files MUST be have a single root node
>
> The only one that I think really needs to get fixed is #1 – to deal with

PIs are a hack. We shouldn't extend them anymore.

Please collect use cases for inclusions, and then have a proper 
mechanism for dealing with them (see Erik Wilde's proposal from a few 
days ago).

Best regards, Julian

---

From: Yaron Sheffer <[email-masked]>
Date: Mon, 11 Nov 2013 11:34:38 +0200

Hi Jim,

I have a good counter-example for #3: an external tool (coming up...) 
generates a list of references as a separate file, and they get included 
into the main document.

Thanks,
Yaron

On 11/11/2013 05:50 AM, Jim Schaad wrote:
> 2.They will incorrectly deal with entities that are declared in the 
> main document
>
> The only one that I think really needs to get fixed is #1 – to deal 
> with ticket #106, but I would be interested in opinions on the others.
>

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 11 Nov 2013 13:18:47 +0000

Hi.

#3  as stated strikes me as leading to "really difficult to explain to
user" errors.  I would have thought (without looking in detail at the
code) that if you fix #1 by switching input streams in the feed to the
etree parser then #3 is automatically 'solved'.

As regards #2, it seems that undefining the entities from the parent(s)
would be almost more trouble than making them usable.  The issue might
be whether entities defined in included files should remain available in
the parents.  My feeling would be that they should.  However there might
ber an issue with redefinitions.

Regards,
Elwyn

On Mon, 2013-11-11 at 10:14 +0100, Julian Reschke wrote:
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Julian Reschke <[email-masked]>
Date: Mon, 11 Nov 2013 14:43:32 +0100

On 2013-11-11 14:18, Elwyn Davies wrote:
> ...

XML already has one inclusion system (external entities), plus xinclude 
defines a second one. Can we please stop re-inventing the wheel?

Best regards, Julian

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 11 Nov 2013 14:01:09 +0000

> defines a second one. Can we please stop re-inventing the wheel?

If it was really reinvention I'd be entirely in agreement.  Regrettably
this is a matter of backwards compatibility I believe.

/Elwyn

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 11 Nov 2013 21:05:55 -0800

Does this work today with the version 1 tool?  If not then I would not
expect it to work in the future.

I note that this would not work with either an entity reference or with
xinclude and does not work with the include PI in xml2rfc v2 today.

Jim

> -----Original Message-----
> From: Yaron Sheffer [mailto:[email-masked]]
> Sent: Monday, November 11, 2013 1:35 AM
> To: Jim Schaad; [email-masked]
> Subject: Re: [xml2rfc] include processing directive
> 
> Hi Jim,
> 
into the
> main document.
> > The only one that I think really needs to get fixed is #1 - to deal

---

From: "Jim Schaad" <[email-masked]>
Date: Mon, 11 Nov 2013 21:06:57 -0800

Given that we are using a package to do the XML parsing, it does not seem
that switching streams in mid-stream seems to be an option.

Jim

> From: Elwyn Davies [mailto:[email-masked]]
> Sent: Monday, November 11, 2013 5:19 AM
> To: Julian Reschke
> Cc: Jim Schaad; [email-masked]
user" errors.
> I would have thought (without looking in detail at the
etree
> parser then #3 is automatically 'solved'.
> 
would
> be almost more trouble than making them usable.  The issue might be
> whether entities defined in included files should remain available in the
> parents.  My feeling would be that they should.  However there might ber
an
> issue with redefinitions.
> 
> Regards,
> Elwyn
> 
> > > There are currently a number of restrictions on the current system
> > > for the include processing directive.  I am wondering how many of
> > > these should be kept and which should be fixed.
> > >

---

From: Yaron Sheffer <[email-masked]>
Date: Tue, 12 Nov 2013 08:05:58 +0200

Hi Jim,

I'm probably missing what you're trying to do in xml2rfc (I couldn't 
find documentation for an "include" in xml2rfc). But with XInclude, I 
can do this to include multiple "reference" elements:

           <xi:include href="refs.xml" xpointer='xpointer(*/*)'/>

Works for me with xmllint, where the included file is a valid XML with a 
root node. The root node is ignored with this syntax.

Thanks,
	Yaron

---

From: Tony Hansen <[email-masked]>
Date: Tue, 12 Nov 2013 00:53:39 -0500

> mechanism for dealing with them (see Erik Wilde's proposal from a few

For v2, we should do enough work to make v2 work the same as v1. But
extensions beyond that should normally be out of scope, except in the
rare case where it might be the only way to achieve parity with v1 for
some reason.

For v3, we can certainly rethink how inclusions are accomplished. I
think Erik's proposals look interesting.

    Tony

---

From: Nico Williams <[email-masked]>
Date: Tue, 12 Nov 2013 00:26:26 -0600

On Mon, Nov 11, 2013 at 3:14 AM, Julian Reschke <[email-masked]> wrote:
> Please collect use cases for inclusions, and then have a proper mechanism
> for dealing with them (see Erik Wilde's proposal from a few days ago).

Agreed.  One use case I've seen a few times is where you have a large
ASN.1/XDR/whatever module that needs to go in an appendix but select
pieces of which need to be interspersed with the prose for explanatory
purposes.  Hand-coding this is extremely error prone.  Here the idea
would be to markup sub-sections of the module but otherwise have it in
one piece, then reference specific sub-sections of the module as
desired.  Another possibility is to spread the module throughout the
source doc and have functionality to pull all the bits, concatenate
them, then make the result the appendix -- IMO this second approach is
terrible, but the first approach requires a fairly radical new
approach to "artwork".

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 12 Nov 2013 00:35:00 -0600

If examples of RFCs with large modules half-strewn over the RFC and
fully duplicated in an appendix are needed, let me know.  I can think
of quite a few:

 - PKIX-related RFCs
 - LDAP-related RFCs
 - TLS-related RFCs
 - Kerberos-related RFCs
 - GSS-API-related RFCs
 - XDR and NFS-related RFCs

For each of the above, without looking at *any* of them, just the ones
I know for a fact are a) not obsolete/historic, and b) have this
problem, there's at least eight, but probably more than a dozen (if I
looked).  A co-author of at least of them complained to me about the
difficulty of managing this and explained how he used nroff and
scripts to manage the mess.  I myself have wanted for a decent
solution to this.  I am pretty sure at least one of those RFCs has
some inconsistencies between code that appears twice in it.

One look at RFCs 5280 or 5661 should be sufficient to prove the point.
 We urgently need a way to avoid hte errors that come with duplicating
ASN.1/XDR/ABNF/whatever (TLS' ad-hoc syntax) text.

Nico
--

---

From: Erik Wilde <[email-masked]>
Date: Mon, 11 Nov 2013 22:40:45 -0800

hello nico.

> terrible, but the first approach requires a fairly radical new

not really. unless you plan on making <artwork> itself inclusion-enabled 
(which i think would not be such a great way to go), you simply use 
XInclude and you're done. this is a working snippet from 
https://github.com/dret/I-D/blob/master/xml-patch/draft-wilde-xml-patch-08.xml, 
which requires XInclude processing (&files; is defined as 
"draft-wilde-xml-patch-08-files" in the XML):

<artwork type="xml"><xi:include parse="text" 
href="&files;/patch-document.xml"/></artwork>

the current downside of XIPr (the XInclude processor i am using, 
https://github.com/dret/XIPr) is that you can only include complete text 
files, and not fragments. XInclude 1.1 fixes that, but i haven't 
implemented it yet. i have no idea how widely 1.1 is supported, but i 
would guess that most current XInclude implementations are 1.0 only. 1.1 
support would allow something like this:

<artwork type="xml"><xi:include parse="text" 
href="&files;/patch-document.xml#line=10,12"/></artwork>

one problem is that if you support 1.1 (i.e., fragments for textual 
inclusion), once you insert one line at the start of your included file, 
you have to adjust all fragment identifiers or they'll "get out of sync" 
(integrity checks http://tools.ietf.org/html/rfc5147#section-3.1 can be 
used to detect this situation, but they won't fix it). there's no easy 
way around this. but apart from that, it's certainly much better than 
what i was doing before, i.e. copy/pasting files into the XML source.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked]  -  tel:+1-510-2061079 |
            | UC Berkeley  -  School of Information (ISchool) |
            | http://dret.net/netdret http://twitter.com/dret |

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Nov 2013 07:49:33 +0100

> terrible, but the first approach requires a fairly radical new

The second is what we do in HTTPbis (and it works just fine using a 
simple shell script that updates the appendix).

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Tue, 12 Nov 2013 00:53:33 -0600

On Tue, Nov 12, 2013 at 12:40 AM, Erik Wilde <[email-masked]> wrote:
>> purposes.  [...]
> (which i think would not be such a great way to go), you simply use XInclude
> and you're done. this is a working snippet from

I guess I could live with that.  I'd rather have something somewhat
better integrated, but I suppose we've been hacking our way around
this so far reasonably well -- why stop?

> files, and not fragments. XInclude 1.1 fixes that, but i haven't implemented
> it yet. i have no idea how widely 1.1 is supported, but i would guess that
> most current XInclude implementations are 1.0 only. 1.1 support would allow
> something like this:
>
> inclusion), once you insert one line at the start of your included file, you
> have to adjust all fragment identifiers or they'll "get out of sync"
> (integrity checks http://tools.ietf.org/html/rfc5147#section-3.1 can be used
> to detect this situation, but they won't fix it). there's no easy way around
> this. but apart from that, it's certainly much better than what i was doing
> before, i.e. copy/pasting files into the XML source.

Yeah.  No, that's awful.  I want fragment IDs.  In general all these
code modules are written in languages that permit comments, so the
obvious thing to do is to put the whole module in one file (well,
where it makes sense) and then mark it up with comment directives, and
extract fragments to Xinclude (or whatever) into the prose.  That
kinda means hacking it up every time, or at least once per-language.

Nico
--

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 12 Nov 2013 14:27:48 +0000

I am not deeply skilled with lxml but it appears to have the hooks
needed, e.g., http://lxml.de/parsing.html#parsers

PI events seem to give one a handle on how to do this possibly combined
either with a suitable file-like object or a feed parser.

/Elwyn

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Nov 2013 15:42:53 +0100

> either with a suitable file-like object or a feed parser.

Please be very careful.

In the end we need a language that can be processed by more than a 
single tool. Just because you are able to get it working in a specific 
code base doesn't mean that it is a sane design.

Best regards, Julian

---

From: Erik Wilde <[email-masked]>
Date: Tue, 12 Nov 2013 10:12:22 -0800

hello nico.

> this so far reasonably well -- why stop?

keep in mind that for this to work, all i need is local pre-processing 
(well, and then removing the xmlns:xi namespace declaration which gets 
rejected as an unknown attribute by xml2rfc).

> kinda means hacking it up every time, or at least once per-language.

what could be hacked would be fragment ids for languages, rather than 
the ones that are defined for text/plain. it might be interesting to see 
how this could plug into an XInclude implementation such as XIPr and 
XInclude in general. i guess you could use the "parse" attribute to 
invoke behavior beyond standard text/xml processing:

http://www.w3.org/TR/xinclude-11/#include_element

<artwork type="abnf"><xi:include parse="abnf" 
href="&files;/patch-grammar.txt#rule=5,9"/></artwork>

cheers,

dret.