Subject: [Json] ABNF nits -- LAST CHANCE ON PROPOSALS

From: Paul Hoffman <[email-masked]>
Date: Tue, 11 Jun 2013 10:32:48 -0700

Greetings again. We currently have one rolled-up proposal for ABNF nit fixes for RFC 4627. If you have other nits to fix, please do so now on this thread. The ABNF below is separated by the text sections where it appears.

Note that these are fixes that apply only to RFC 4627. Other proposals change some of the ABNF, but those are being dealt with in their own threads.

--Paul Hoffman

JSON-text = object / array

begin-array     = ws %x5B ws  ; [ left square bracket
begin-object    = ws %x7B ws  ; { left curly bracket
end-array       = ws %x5D ws  ; ] right square bracket
end-object      = ws %x7D ws  ; } right curly bracket
name-separator  = ws %x3A ws  ; : colon
value-separator = ws %x2C ws  ; , comma

ws = *(
     %x20 /              ; Space
     %x09 /              ; Horizontal tab
     %x0A /              ; Line feed or New line
     %x0D                ; Carriage return
     )

value = false / null / true / object / array / number / string
false = %x66.61.6c.73.65   ; false
null  = %x6e.75.6c.6c      ; null
true  = %x74.72.75.65      ; true

object = begin-object [ member *( value-separator member ) ]
         end-object
member = string name-separator value

array = begin-array [ value *( value-separator value ) ] end-array

number = [ minus ] int [ frac ] [ exp ]
decimal-point = %x2E       ; .
digit1-9 = %x31-39         ; 1-9
e = %x65 / %x45            ; e E
exp = e [ minus / plus ] 1*DIGIT
frac = decimal-point 1*DIGIT
int = zero / ( digit1-9 *DIGIT )
minus = %x2D               ; -
plus = %x2B                ; +
zero = %x30                ; 0

string = quotation-mark *char quotation-mark
char = unescaped /
    escape (
        %x22 /          ; "    quotation mark  U+0022
        %x5C /          ; \    reverse solidus U+005C
        %x2F /          ; /    solidus         U+002F
        %x62 /          ; b    backspace       U+0008
        %x66 /          ; f    form feed       U+000C
        %x6E /          ; n    line feed       U+000A
        %x72 /          ; r    carriage return U+000D
        %x74 /          ; t    tab             U+0009
        %x75 4HEXDIG )  ; uXXXX                U+XXXX
escape = %x5C              ; \
quotation-mark = %x22      ; "
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

HEXDIG = DIGIT / %x41-46 / %x61-66   ; 0-9, A-F, or a-f
       ; HEXDIG equivalent to HEXDIG rule in [RFC5234]
DIGIT = %x30-39            ; 0-9
      ; DIGIT equivalent to DIGIT rule in [RFC5234]

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 12 Jun 2013 08:19:20 +0200

The efforts to real-world-test the ABNF are ongoing; so far everything looks fine (unsurprisingly).

From a stylistic point of view, I have two comments on the ABNF:

1) the production for char relies on the operator precedence, i.e., concatenation binds closer than alternative.

      char = unescaped /
          escape (

This is nicely layed out to suggest this priority, and given the precedence defined in 5234, it *is* unambiguous.

Still, section 3.10 of 5234 rightly notes:

   Use of the alternative operator, freely mixed with concatenations,
   can be confusing.

      Again, it is recommended that the grouping operator be used to
      make explicit concatenation groups.

So the canonical form of writing this might be:

      char = unescaped / (
          escape (
              (%x75 4HEXDIG) ) )  ; uXXXX                U+XXXX

2) the character literals use hex in favor of "x" notation.

begin-array = ws "[" ws
begin-object = ws "{" ws
end-array = ws "]" ws
end-object = ws "}" ws
name-separator = ws ":" ws
value-separator = ws "," ws

would generally be considered slightly more readable.
Productions like minus, zero, plus, would not really be needed.
(Hex notation is indeed needed for quotation-mark, for case-sensitive productions like false, and for the ranges, so using just that instead of a mix with quoted "char-val"s is a simplification for people new to ABNF.)

Now, changing the ABNF even on a stylistic level could make it look like something changed syntactically between 4627 and 4627bis.
On the other hand, I'm not aware of any problems that have been caused by the two stylistic peculiarities.

So I'm not actually proposing to make any of these changes now, but I would like to make sure the WG considered and consciously decided against them because they will inevitably come up at IESG review time.

(If we do decide to change the ABNF for other reasons, these items might be reconsidered.)

Grüße, Carsten

---

From: Stefan Drees <[email-masked]>
Date: Wed, 12 Jun 2013 08:52:29 +0200

On 2013-06-12 08:19 CEST, Carsten Bormann wrote:
> 1) the production for char relies on the operator precedence, i.e.,
> concatenation binds closer than alternative.
>
>        char = unescaped /
>            escape (
> This is nicely layed out to suggest this priority, and given the
> precedence defined in 5234, it *is* unambiguous.
>

I second the proposal, maybe with the last row knit slightly different.

> (Hex notation is indeed needed for quotation-mark, for
> case-sensitive productions like false, and for the ranges, so using just
> that instead of a mix with quoted "char-val"s is a simplification for
> people new to ABNF.)
>

or would not ;-) especially as the target language (JSON) of the grammar 
uses itself the double quote as token, it is IMO distracting to spread 
ABNF double quote tokens more than absolutely necessary.

In the cases under 2) above I see the task of the "rule spell out" as 
follows:

A) What is the rule?
B) What is that whitespace separated character token?

In random ordering I have two comments on this:

Firstly I think the cultural concept of lower and upper case in the 
realm of punctuation mixes (I presume) badly for people coming from 
other cultures where this concept of eg. 'a is somehow like A' does not 
apply and also with most keyboard layouts. Thus the rewrite of the 
rules as in 2 above may trigger extra work for the reader like:
Double quotes matches two characters per slot, ah, no this character has 
no "upper" sister.

Secondly I think that the "other" proposed form seems to transport the 
rules covering both aspects A and B clearly readable (giving the 
"printed" representation and common english name of the character in the 
comment.

All personal and opinion only, everyone will see this from a different 
angle, but that's why /I/ write it ;-)

> Now, changing the ABNF even on a stylistic level could make it look
> like something changed syntactically between 4627 and 4627bis.
> On the other hand, I'm not aware of any problems that have been
> caused  by the two stylistic peculiarities.
>
> So I'm not actually proposing to make any of these changes now, but
> I would like to make sure the WG considered and consciously decided
> against them because they will inevitably come up at IESG review time.

So I am all for the first proposed change above and at the same time 
opposed to applying the second proposed change based on the above reasoning.

> (If we do decide to change the ABNF for other reasons, these items

ditto.

All the best,

Stefan.

---

From: Stefan Drees <[email-masked]>
Date: Wed, 12 Jun 2013 17:19:13 +0200

On 2013-06-11 19:32 CEST, Paul Hoffman wrote:
> ...
> ...

allthough I am used to ABNF stanzas like the above, as a grammar, it is 
only "paper ware" (IMO not useful for an optimal parser generator) and I 
would like to hereby ask all participants in this endeavour if it is 
possible to list the allowed "white space" without defining such a mad 
token like "ws" is defined above. (I think a grammar should clearly 
focus on real tokens and not name "nothing")

Why now and not earlier? Well, as Carsten in a separate sub thread (of 
this "ABNF nits last proposal" thread) suggested a better readable ABNF 
of this part and I opposed to it, I went back and looked at the above 
proposed version again and of course disliked the "ws"'s sprinkled all 
over the place that are in fact optional (zero or more) but which is 
buried inside the ws rule.

Wouldn't it be better to explicitely state the following:

"""

begin-array     = *ws %x5B *ws  ; [ left square bracket
begin-object    = *ws %x7B *ws  ; { left curly bracket
end-array       = *ws %x5D *ws  ; ] right square bracket
end-object      = *ws %x7D *ws  ; } right curly bracket
name-separator  = *ws %x3A *ws  ; : colon
value-separator = *ws %x2C *ws  ; , comma

ws =  (
       %x20 /              ; Space
       %x09 /              ; Horizontal tab
"""

instead of the grammar part cited on top of this mail?

Then a "ws" truly would be a single white space.
Also the optionality (zero or more) of this white space surrounding the 
punctuation tokens would clearly stand out.

What do you think? Based on that, would it be helpful, to deliver a 
complete new proposal? I thought yes, and thus integrated Carstens first 
partial proposal on better grouping on the right hand side of the "char" 
rule.

So **this** below is my proposal unification effort. (I named it 
proposal 2, as I did not see another complete proposal. If I overlooked 
one, please excuse and renumber the below one accordingly.)

Proposal 2
==========

begin-array     = *ws %x5B *ws  ; [ left square bracket
begin-object    = *ws %x7B *ws  ; { left curly bracket
end-array       = *ws %x5D *ws  ; ] right square bracket
end-object      = *ws %x7D *ws  ; } right curly bracket
name-separator  = *ws %x3A *ws  ; : colon
value-separator = *ws %x2C *ws  ; , comma

ws = (
      %x20 /              ; Space
      %x09 /              ; Horizontal tab

char = unescaped / (
     escape (

escape = %x5C              ; \

""" end of proposal 2.

Thanks a lot,
Stefan.

---

From: Norbert Lindenberg <[email-masked]>
Date: Wed, 12 Jun 2013 09:13:20 -0700

>        %x75 4HEXDIG )  ; uXXXX                U+XXXX

The last line isn't quite correct - \uXXXX can not only represent U+XXXX, but also be part of U+YYYYYY, where YYYYYY depends on a preceding or following \uXXXX sequence.

Proposal: Replace "U+XXXX" with "U+XXXX or part of U+YYYYYY".

Norbert

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 12 Jun 2013 20:16:36 +0200

On Jun 12, 2013, at 17:19, Stefan Drees <[email-masked]> wrote:

> Wouldn't it be better

Well, I apologize for making editorial comments on the ABNF.  
We have more important things to decide right now; I was incited by the "LAST CHANCE" message.

The single-level grammars created by RFC 5234 ABNF are generally a lot better for describing protocols than the scanner/parser grammar pairs used by programming language standards (like ECMAscript).  ABNF tends to be a bit verbose around the optional whitespace, but I think Douglas has hit this nail on the head with the single exception that he maybe should have called ws ows (optional white space).
For the reasons I gave, changing this is only worth it if we are changing a lot of the rest.

Norbert's observation of course is spot on; I must admit I didn't read the comments at all.
(Maybe we can avoid writing the somewhat disconnected U+YYYYYY, though, and instead point to the fourth paragraph of 2.5, "U+XXXX or part of a surrogate pair, see Section 2.5".)

Grüße, Carsten