Subject: Re: [Json] [rfc-i] v3imp #8 Fragment tagging on sourcecode

From: Sean Leonard <[email-masked]>
Date: Wed, 28 Jan 2015 14:31:06 -0800

On 1/28/2015 9:32 AM, Nico Williams wrote:
> On Tue, Jan 27, 2015 at 09:16:37PM -0800, Sean Leonard wrote:
>> Overall I still stand by my proposition that the RFC is the module
>> for ABNF purposes. Honestly it just makes things a lot simpler. To
>> the extent that you need to split things inside the RFC, you can
>> refer to specific sections. Specific comments below.
> Well, draft-ietf-json-text-sequence-13 will break this proposition.

Hello--I took a look at draft-ietf-json-text-sequence-13.

There are two typos:
Abstract:

    ASCII Record Separator (0x1E), and each ending with an ASCII Line
    Feed character (0x1A).
                   ^^^^^^

should be 0x0A.

Section 2.2

    Note that on some systems it's possible to input RS by typing
    'ctrl-^'; on some system or applications the correct sequence may be
    'ctrl-v crtl-^'.  This is helpful when constructing a sequence
            ^^^^^^

should be ctrl-^.

An editorial point: why define LF in this I-D when it's already defined 
in RFC 5234 Appendix B? Just sayin'.

> It's on the RFC-Editor queue.  We could "fix" the "problem" in AUTH48,
> but I've no intention to.

Well you might want to look at the problems above...

> I doubt it's the first RFC that will break your assumption.
It's not an RFC yet. :-)

First of all, I don't see a "problem" because there are no "modules" in 
draft-ietf-json-text-sequence, or any other Internet-Draft. It's all a 
bunch of plain text. I am not trying to be flippant; it's simply the way 
it is. I-Ds are just large reams of plain text punctuated with FF codes. 
That is why extracting data out of RFCs is convoluted and fraught with 
error.

I am not stating an assumption so much as I am stating a workable 
proposition that makes sense for authoring future RFCs in a structured 
format, i.e., a format where ABNF is clearly distinguished from 
specification text or other things.

I too dealt with this exact problem in draft-josefsson-pkix-textual-10 
(also in the RFC-Editor queue, btw). In that soon-to-be-RFC, I renamed 
the rules to:
textualmsg              ; what most extant parsers and generators will 
work with
stricttextualmsg     ; what compliant generators SHOULD emit
laxtextualmsg         ; the most relaxed input that parsers MAY accept

Thus, consider renaming the rules to distinguish between inputs and 
outputs, such as {JSON-sequence-in} and {JSON-sequence-out}. It is 
clearer and more accurate. I note that the "meat" parts are named 
differently: {possible-JSON} vs. {JSON-text}, presumably to indicate the 
difference in strictness between generators and parsers.

As a broader useful point (not directed specifically to 
draft-ietf-json-text-sequence-13), I think it would be nice if future 
RFC ABNF can assume that the symbols in RFC 20 for %x00-20 / %x7F are 
rules that can be used as-is. Hence NUL = %x00, SUB = %x1A, DEL = %x7F, etc.

> Not that it matters.  Suppose all RFCs with ABNF only had a *single*
> ABNF module, so that each module could be said to be named rfc1234.abnf.
> Now what?  You'd still be missing the machine-readable imports.

Uh I already laid this out...I no longer ascribe to the premise of "ABNF 
modules". It's just a bunch of ABNF text.

You know, other symbolic systems such as ASN.1 have modules...and ASN.1 
is seen as way too complicated. I doubt that anyone is going to 
"compile" the ABNF in draft-ietf-json-text-sequence-13 anyway--they are 
just going to hunt for RS bytes, which is the whole point.

> We could say that every module exports every rule by default, so we
> could get away with not having exports.  But imports are absolutely
> necessary in order to validate ABNF modules that import rules from
> others.

Unless you don't have ABNF modules...

> We already have a mess on our hands.  ABNF needs a module system.  An
> ad-hoc module system won't do.

No modules = no messes. I would say that all the crazy importing and 
exporting in ASN.1 modules (an in particular, the fact that the 
definitions have changed over time, witness PKCS #7 1.0 -> PKCS #7 1.5 
-> CMS '99 -> CMS '02 -> CMS '04 -> CMS '09) is much messier. 
Significant tool support is required, which is one significant reason 
why people hate on ASN.1 and are trying to do things with JSON instead.

Sean

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Jan 2015 17:02:31 -0600

> Hello--I took a look at draft-ietf-json-text-sequence-13.

Thanks.  Those have already been noted and will be fixed before
publication.

> An editorial point: why define LF in this I-D when it's already
> defined in RFC 5234 Appendix B? Just sayin'.

Because RS isn't, and both RS and LF come from STD 80 (RFC 20).

> Well you might want to look at the problems above...

Yes, but I'm asking about a more specific problem.

> >I doubt it's the first RFC that will break your assumption.

But this issue (two ABNF rules with the same name) was raised earlier,
and no one thought it was a problem.  Now there's a question about ABNF
modules: must ABNF-using RFCs have a single ABNF "module" consisting of
all ABNF rules extracted from their figures??

I don't think "yes" is really the right answer, but it would be
convenient if no other RFC deviates from that.

> First of all, I don't see a "problem" because there are no "modules"
> in draft-ietf-json-text-sequence, or any other Internet-Draft. It's

There are no ABNF modules in *any* RFCs.  Because there are no ABNF
modules at all.

> all a bunch of plain text. I am not trying to be flippant; it's

No, there's figures.

> simply the way it is. I-Ds are just large reams of plain text
> punctuated with FF codes. That is why extracting data out of RFCs is
> convoluted and fraught with error.

This is an I-D on the RFC-Editor queue.  It will soon be an RFC.

> proposition that makes sense for authoring future RFCs in a
> structured format, i.e., a format where ABNF is clearly
> distinguished from specification text or other things.
> 
> I too dealt with this exact problem in
> draft-josefsson-pkix-textual-10 (also in the RFC-Editor queue, btw).
> In that soon-to-be-RFC, I renamed the rules to:
> textualmsg              ; what most extant parsers and generators
> will work with

If ABNF has no modules and we don't insist on an 'RFCs contain at most
one ABNF "module"' rule, then I don't need to consider this proposal at
all.

I'd rather we defined ABNF modules.

A rule that RFCs must have at most one ABNF module does not get us past
the need to know what rules get imported from what other modules.  And
it's not elegant.

> differently: {possible-JSON} vs. {JSON-text}, presumably to indicate
> the difference in strictness between generators and parsers.

I'm inclined to continue rejecting this proposal.

> draft-ietf-json-text-sequence-13), I think it would be nice if
> future RFC ABNF can assume that the symbols in RFC 20 for %x00-20 /
> %x7F are rules that can be used as-is. Hence NUL = %x00, SUB = %x1A,
> DEL = %x7F, etc.

I agree.  These should be added to RFC5234 (i.e., we should publish an
update RFC listing them).

> Uh I already laid this out...I no longer ascribe to the premise of
> "ABNF modules". It's just a bunch of ABNF text.

But I'd like to have first-class ABNF modules.  "Just a bunch of [ABNF]
text" is not enough: because we can't validate it.

> You know, other symbolic systems such as ASN.1 have modules...and
> ASN.1 is seen as way too complicated. I doubt that anyone is going
> to "compile" the ABNF in draft-ietf-json-text-sequence-13
> anyway--they are just going to hunt for RS bytes, which is the whole
> point.

We like to machine-validate ABNF modules.  How should we do this as to
imported rules?

> [...]

No modules -> No need to rename the conflicting rules in draft-ietf-json-text-sequence.

Thanks,

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Thu, 29 Jan 2015 10:25:34 +1100

>> for ABNF purposes. Honestly it just makes things a lot simpler.

> But this issue (two ABNF rules with the same name) was raised earlier, and no one thought it was a problem.

The issued was raised because someone did think it was a problem.

--
James Manger

---

From: Nico Williams <[email-masked]>
Date: Wed, 28 Jan 2015 17:29:32 -0600

> The issued was raised because someone did think it was a problem.

I typed too quickly.  I mean that no one thought the change had to be
made.

I still don't think so, not on account of a heretofore non-existent (and
inelegant) rule that there be at most one ABNF "module" per-RFC.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 29 Jan 2015 07:11:54 +0100

>I typed too quickly.  I mean that no one thought the change had to be

I very strongly urge to change this.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
D-10243 Berlin · PGP Pub. KeyID: 0xA4357E78 · http://www.bjoernsworld.de
 Available for hire in Berlin (early 2015)  · http://www.websitedev.de/

---

From: Sean Leonard <[email-masked]>
Date: Thu, 29 Jan 2015 03:04:01 -0800

> I very strongly urge to change this.

Yes. Let's just look at this from a readability standpoint.

Suppose you have some standard that defines "newline" in two different 
illustrations as:

newline   = CR
...
newline   = LF

is that really clear? It's pretty unclear. I would call that an 
editorial problem (at least). It would be much clearer to say:

newlineMac  = CR
...
newlineUnix  = LF

anynewline = newlineMac / newlineUnix

i.e., with different rule names. Then you can say things in the 
specification text, like "in UNIX environments, use newlineUnix...but 
when reading data from an unknown source, use anynewline..." and be 
unambiguous.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Fri, 30 Jan 2015 03:31:38 -0800

> update RFC listing them).

I would be willing to write up/work on a concise RFC (Internet-Draft) 
that does this, prior to IETF 92. I would not mind adding a few 
additional rules to the Core Rules if there is near-unanimous support 
for such rules. (Hard to think of any in particular, except maybe some 
generic UTF8 character rules. Last I checked, ABNF is still 
octet-oriented and US-ASCII focused.)

While I would not say "precondition", it is reasonable to require that 
RFCs after the publication of such an RFC follow it, much like RFCs past 
5234 use RFC 5234 as the basis.

Sean

---

From: Sean Leonard <[email-masked]>
Date: Sat, 31 Jan 2015 07:13:16 -0800

On 1/30/2015 3:32 PM, Bjoern Hoehrmann wrote:
> * Sean Leonard wrote:
> It is integer-oriented, you have to define the alphabet in prose and are
> free to say it's "octets" or "Unicode scalar values" or whatever else.

Ok. I was not aware of that, however it is stated clearly enough in RFC 
5234 Section 2.3.

Many specifications use ABNF as octets, and define UTF-8 data in terms 
of octets. E.g., LDAP [RFC4512]; see "UTFMB". It makes equal sense (in 
some contexts, perhaps better sense) to use ABNF in integer form for 
integers 0 - 0x10FFFF, i.e., Unicode scalar values.

However those folks who want to define formal import syntaxes will have 
to grapple with this issue--namely that integer values do not represent 
the same thing across different specifications. I see this as another 
reason not to pursue import syntaxes.

> I do not think an RFC that updates RFC 5234 just to add names for three
> symbols is a good idea.

*Only* the following symbols are being proposed:
NUL = %d0
SOH = %d1
STX = %d2
ETX = %d3
EOT = %d4
ENQ = %d5
ACK = %d6
BEL = %d7
BS = %d8
HT = %d9 ; also defined as HTAB
LF = %d10 ; already defined
VT = %d11
FF = %d12 ; (literally used in every RFC)
CR = %d13 ; already defined
SO = %d14
SI = %d15
DLE = %d16
DC1 = %d17
DC2 = %d18
DC3 = %d19
DC4 = %d20
NAK = %d21
SYN = %d22
ETB = %d23
CAN = %d24
EM = %d25
SUB = %d26
ESC = %d27
FS = %d28
GS = %d29
RS = %d30
US = %d31
SP = %d32 ; already defined
DEL = %d127

These are all taken from RFC 20, and (mercifully) have the same symbol 
names in Unicode.

> It's very rare that the particular characters
> are useful,

NUL is used more often than you (or many others) might care to admit.

draft-ietf-json-text-sequence-13 makes use of RS, and that usage is very 
appropriate given the engineering problem (that the WG has probably 
already explored).

ESC is used in ANSI escape codes, which form an integral part of a 
variety of standards (maybe few IETF standards, but ISO and 
industry-specific standards such as those in the healthcare industry) 
and are used in modern command-lines and consoles.

SO and SI are used in ISO 2022 character sets.

There are probably a number of historical examples.

>   and ordinarily you will have to define other rules on your
> own aswell;

Well yes, but now you will not need to define as many 
rules--particularly rules for unprintable characters.

I note also that XML 1.0 prohibits C0 control characters except for HT, 
LF, CR, SP, and DEL. Why the standard prohibits FF and BS, but lets DEL 
in, is rather mysterious to me. However if you want to represent these 
characters for their semantics, say, in xml2rfc V3, RS is possible but 
the actual "0x1E" is a harder sell.

XML 1.1 permits all C0 control characters except NUL. Therefore, a XML 
fragment streaming format akin to draft-ietf-json-text-sequence could 
easily use NUL to separate records on the wire.

>   on the software side you would then have ABNF tools that do
> not know the new Core rules, I would have to update my Parse::ABNF Perl
> module for instance,
"lazy"? :) Laziness is not a valid reason to prohibit new work...

For some of these rules (e.g., DC1) perhaps the principal reason to 
include them in the Core Rules is to discourage future spec writers from 
using that rule name to define something else.

>   and there is the issue that some specifications may
> already be using the symbol names, and dealing with clashes with Core
> rules is a bit nasty.

As with "lazy" there may be clashes with the Core Rules, but that can be 
taken as evidence that the Core Rules should have been extended (or 
defined) earlier--not as evidence to prohibit new work. As it is, if you 
use a Core Rule (as draft-josefsson-pkix-textual does), you can't take 
the reference for granted; the IESG will ask you to reference [RFC5234].

The only spec that I am aware of that would cause problems is LDAP's RFC 
4512, which defines ESC = %x5C ("\" backslash). However, RFC 4512 also 
proposes a lot of completely unnecessary rule names (NULL, SPACE, 
DOLLAR, USCORE, EQUALS, DOT), restates at least one Core Rule (DQUOTE), 
and has a bona-fide conflict with existing Core Rules (SP and WSP). 
Since RFC 5234 was defined after 4512, this is evidence in support of 
standardization for future RFCs.

Cheers,

Sean

---

From: John Cowan <[email-masked]>
Date: Sat, 31 Jan 2015 14:53:03 -0500

Sean Leonard scripsit:

> I note also that XML 1.0 prohibits C0 control characters except for
> HT, LF, CR, SP, and DEL. Why the standard prohibits FF and BS, but
> lets DEL in, is rather mysterious to me. 

Carelessness, basically.

> XML 1.1 permits all C0 control characters except NUL. Therefore, a
> XML fragment streaming format akin to draft-ietf-json-text-sequence
> could easily use NUL to separate records on the wire.

XML 1.1 (says the one who initiated its development and fought for it)
is dead, and in fact died a-borning.

-- 
John Cowan          http://www.ccil.org/~cowan        [email-masked]
They tried to pierce your heart with a Morgul-knife that remains in
the wound.  If they had succeeded, you would become a wraith under the
domination of the Dark Lord.         --Gandalf