Subject: [Json] ABNF nits

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 00:46:22 +0000

Some things I found futzing with the ABNF:

- normalize the indentation (makes it easier to pull out)
- Hex literals need to be uppercase.  The false and null rules are wrong.
- DIGIT and HEXDIG are neither defined nor imported, as far as I can tell.
 I suggest:

DIGIT = %x30-39                        ; 0-9
HEXDIG = %x30-39 / %x41-46 / %x61-66   ; 0-9, A-F, or a-f

-- 
Joe Hildebrand

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 12:24:47 +1000

> - Hex literals need to be uppercase.  The false and null rules are

Lowercase is allowed in hex literals.
It links back to HEXDIG in RFC5234 (or RFC 4234).
You have to remember the sneaky ABNF rule that quoted strings (eg "A") are case insensitive. "A" is equivalent to %x41 / %x61.

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 02:39:24 +0000

On 6/6/13 8:24 PM, "Manger, James H" <[email-masked]>
wrote:

>You have to remember the sneaky ABNF rule that quoted strings (eg "A")
>are case insensitive. "A" is equivalent to %x41 / %x61.

Crap.  I didn't see that because it's in a "Note:" in section 2.3.
Regardless, we should change  both of those rules for consistency with the
rest of the doc.

>> HEXDIG = %x30-39 / %x41-46 / %x61-66   ; 0-9, A-F, or a-f

I still recommend being explicit about this, like the rule for e.

-- 
Joe Hildebrand

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 7 Jun 2013 13:15:48 +1000

> Crap.  I didn't see that because it's in a "Note:" in section 2.3.

I didn't call it "sneaky" for nothing ;)

> Regardless, we should change  both of those rules for consistency with
> the rest of the doc.
> 

I agree.
Add DIGIT and HEXDIG to the doc so all the ABNF is present.
Use your definition of HEXDIG above, not the RFC5234 one, to avoid the sneaky ABNF rule.
Optionally add a note "; DIGIT and HEXDIG are equivalent to rules with the same names in [RFC5234]".

--
James Manger

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 08:31:48 +0200

On 2013-06-07 05:15, James H. Manger wrote:
> I didn't call it "sneaky" for nothing ;) ...

lucky JSON, it only needs short tokens (cowardly avoiding the word 
single character here ;-)

If defining a grammar with real case-sensitive "word"-ish tokens in it, 
ABNF is IMO barely usable, unless you extend it with say single quotes 
as case-sensitve string delimiters. But then dealing with two kinds of 
strings also has it's catches =(

> Optionally add a note "; DIGIT and HEXDIG are equivalent to rules with the same names in [RFC5234]".

+1 on your proposal James (including the ABNF comment).

Stefan.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 07 Jun 2013 17:14:42 +0900

> strings also has it's catches =(

Another way to do this is to just note that you are using ABNF *except 
that literals are case-sensitive*. I'm sure there's an RFC or two which 
have done so.

Regards,   Martin.

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 11:08:39 +0200

On 2013-06-07 10:14, Martin J. Dürst wrote:
> that literals are case-sensitive*.

Piece of cake, if all words are case sensitive, but the devil is in the 
mix. As I stated above "with ... case-sensitive ... tokens in it" on 
purpose, and not "with ... case-sensitive ... tokens only" ;-)

That is a situation where I prefer eg. ANTLR stating an easy to parse
tokenCS: 'CaseSensitive' nicely different from the more varying
tokenCIS: [Cc][Aa][Ss][Ee][Ii][Nn][Ss][Ee][Nn][Ss][Ii][Tt][Ii][Vv][Ee]

But I think readability is in the eye of the reader and for JSON the 
grammar is set to be ABNF which is also fully sufficient.

> I'm sure there's an RFC or two which have done so. ...

To the enlightment of the readers, when feeding such a grammar into an 
ABNF tool =( I presume ...

In Open Data we settled in the ABNF URL construction rules[1] for the 
upcoming version 4 on an explicit differentiating strategy and also 
provided a tool (adaption) [2] for translating into real ABNF.

References:

[1]: 
https://tools.oasis-open.org/version-control/browse/wsvn/odata/trunk/spec/ABNF/odata-abnf-construction-rules.txt?rev=356&sc=1
[2]: https://github.com/SAP/abnf-test-tool

All the best,
Stefan.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Fri, 07 Jun 2013 12:49:39 +0200

>that literals are case-sensitive*. I'm sure there's an RFC or two which 

Sadly.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Matt Miller (mamille2)" <[email-masked]>
Date: Fri, 7 Jun 2013 14:10:27 +0000

Would one of you be so kind as to put all of this into a proposal we can all get our heads wrapped around?

kthxbye,

- m&m

Matt Miller < [email-masked] >
Cisco Systems, Inc.

On Jun 7, 2013, at 12:31 AM, Stefan Drees <[email-masked]> wrote:

> lucky JSON, it only needs short tokens (cowardly avoiding the word single character here ;-)
> 
> If defining a grammar with real case-sensitive "word"-ish tokens in it, ABNF is IMO barely usable, unless you extend it with say single quotes as case-sensitve string delimiters. But then dealing with two kinds of strings also has it's catches =(
> 
> 
> _______________________________________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/json

---

From: Stefan Drees <[email-masked]>
Date: Fri, 07 Jun 2013 16:28:18 +0200

> all get our heads wrapped around?

I will try for an estimated consensual definition of all and this ;-)

Proposal:

Replace the closing ABNF rules inside section 2.4. Numbers

OLD:
"""
       number = [ minus ] int [ frac ] [ exp ]

       decimal-point = %x2E       ; .

       digit1-9 = %x31-39         ; 1-9

       e = %x65 / %x45            ; e E

       exp = e [ minus / plus ] 1*DIGIT

       frac = decimal-point 1*DIGIT

       int = zero / ( digit1-9 *DIGIT )

       minus = %x2D               ; -

       plus = %x2B                ; +

       zero = %x30                ; 0
"""

with

NEW:
"""
       number = [ minus ] int [ frac ] [ exp ]

       decimal-point = %x2E       ; .

       digit1-9 = %x31-39         ; 1-9

       e = %x65 / %x45            ; e E

       exp = e [ minus / plus ] 1*DIGIT

       frac = decimal-point 1*DIGIT

       int = zero / ( digit1-9 *DIGIT )

       DIGIT = %x30-39            ; 0-9
             ; DIGIT equivalent to DIGIT rule in [RFC5234]

       minus = %x2D               ; -

       plus = %x2B                ; +

       zero = %x30                ; 0

"""

and in section 2.5 Strings (as there is a 4HEXDIG, and no HEXDIG 
declared) also the ABNF rules section replace:

OLD:
"""
       string = quotation-mark *char quotation-mark

       char = unescaped /
           escape (
               %x22 /          ; "    quotation mark  U+0022
               %x5C /          ; \    reverse solidus U+005C
               %x2F /          ; /    solidus         U+002F
               %x62 /          ; b    backspace       U+0008
               %x66 /          ; f    form feed       U+000C
               %x6E /          ; n    line feed       U+000A
               %x72 /          ; r    carriage return U+000D
               %x74 /          ; t    tab             U+0009
               %x75 4HEXDIG )  ; uXXXX                U+XXXX

       escape = %x5C              ; \

       quotation-mark = %x22      ; "

       unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

"""

with:
NEW:
"""
       string = quotation-mark *char quotation-mark

       char = unescaped /
           escape (
               %x22 /          ; "    quotation mark  U+0022
               %x5C /          ; \    reverse solidus U+005C
               %x2F /          ; /    solidus         U+002F
               %x62 /          ; b    backspace       U+0008
               %x66 /          ; f    form feed       U+000C
               %x6E /          ; n    line feed       U+000A
               %x72 /          ; r    carriage return U+000D
               %x74 /          ; t    tab             U+0009
               %x75 4HEXDIG )  ; uXXXX                U+XXXX

       escape = %x5C              ; \

       quotation-mark = %x22      ; "

       unescaped = %x20-21 / %x23-5B / %x5D-10FFFF

       HEXDIG = DIGIT / %x41-46 / %x61-66   ; 0-9, A-F, or a-f
              ; HEXDIG equivalent to HEXDIG rule in [RFC5234]

"""

Did it catch everything? Or do we want to explain, that 4HEXDIG means 
exactly 4 hex digits? I guess no, as the comment is depicting this nice 
and it is common ABNF practice to deal with multiplicity.

All the best and second to top posting as this is a summarizing job ;-)
Stefan

---

From: "Joe Hildebrand (jhildebr)" <[email-masked]>
Date: Fri, 7 Jun 2013 16:56:53 +0000

On 6/7/13 8:28 AM, "Stefan Drees" <[email-masked]> wrote:
>Did it catch everything?

indentation, which is less of a nit in ABNF than other syntaxes.

>Or do we want to explain, that 4HEXDIG means

See section 3.7 of 5234.  4HEXDIG means exactly 4 HEXDIG.

-- 
Joe Hildebrand

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 10 Jun 2013 22:09:07 +0200

On Jun 7, 2013, at 10:14, Martin J. Dürst <[email-masked]> wrote:

> Another way to do this is to just note that you are using ABNF *except that literals are case-sensitive*. I'm sure there's an RFC or two which have done so.

And, for your merriment, here is the right way to do this (RFC 4997):

   ; case-sensitive literals
   C            = %d67
   COMPRESSED   = %d67.79.77.80.82.69.83.83.69.68
   CONTROL      = %d67.79.78.84.82.79.76
   DEFAULT      = %d68.69.70.65.85.76.84
   ENFORCE      = %d69.78.70.79.82.67.69
   INITIAL      = %d73.78.73.84.73.65.76
   LENGTH       = %d76.69.78.71.84.72
   THIS         = %d84.72.73.83
   U            = %d85
   UNCOMPRESSED = %d85.78.67.79.77.80.82.69.83.83.69.68
   VALUE        = %d86.65.76.85.69
   VARIABLE     = %d86.65.82.73.65.66.76.69
   false        = %d102.97.108.115.101
   true         = %d116.114.117.101

(Yes, I automatically generated that ABNF from the list of literals.)

Grüße, Carsten

---

From: Stefan Drees <[email-masked]>
Date: Mon, 10 Jun 2013 22:57:50 +0200

On 2013-06-10 22:09 CEST, Carsten Bormann wrote:
> On Jun 7, 2013, at 10:14, Martin J. Dürst ... wrote:
>
>> Another way to do this is to just note that you are using ABNF
>> *except that literals are case-sensitive*. I'm sure there's an RFC or
>> two which have done so.
>

shouldn't it say instead in productions 3 - 6:

      CONTROL      = %d73.78.73.84.73.65.76
      DEFAULT      = %d67.79.78.84.82.79.76
      ENFORCE      = %d68.69.70.65.85.76.84
      INITIAL      = %d69.78.70.79.82.67.69

No, only joking, I feel quite cheerful now :-)

Stefan.