Subject: [Json] more on schemas, JCR -03

From: Andrew Newton <[email-masked]>
Date: Wed, 15 Oct 2014 15:54:50 -0400

All,

Continuing the "discussion we are not having", I've updated JSON
Content Rules (JCR) based on feedback I have received from people who
are using it and others. The draft is here:

URL:
http://www.ietf.org/internet-drafts/draft-newton-json-content-rules-03.txt
Status:
https://datatracker.ietf.org/doc/draft-newton-json-content-rules/
Htmlized:       http://tools.ietf.org/html/draft-newton-json-content-rules-03
Diff:
http://www.ietf.org/rfcdiff?url2=draft-newton-json-content-rules-03

While JCR is not JSON, its sole intent is to model JSON. There is no
abstraction: it is composed of 5 basic rules, the first four being
tied to JSON syntax and the fifth being a grouping rule.

Specifically regarding JCR, one of the comments is that it is
"cryptic". The new draft introduces an alternative syntax that will
probably be less cryptic to readers unfamiliar with JCR. Personally I
find this to be a matter of taste where TL;DR can also be an issue,
but I am sympathetic to the argument that some readers will prefer
words over symbols.

While a prose based approach does have its place, sometimes
conciseness and compactness is a virtue. For example, if you take a
look at draft-ietf-weirds-json-response
(http://tools.ietf.org/html/draft-ietf-weirds-json-response-10) you
will see that there are a lot of JSON structures that are re-used
inside of other structures. In my opinion, description of this type of
JSON benefits from a more compact and concise schema language, and had
draft-ietf-weirds-json-response had the benefit of JCR it might not
need to rely so heavily on copious examples. This is illustrated in
Appendix B of the JCR draft.

Another valuable trait of JCR in describing IETF documents
specifically is the ability to describe values as IP addresses, domain
names, and other types used by protocol documents. While a small
benefit, anything that reduces the tedium of specification writing
will hopefully reduce mistakes. To a larger point, the same hope
applies to a syntax that is concise and compact.

With regards to schema languages in general, Tim has repeatedly said
that we should concentrate on protocol test suites. I agree, but I do
not see schema languages as being mutually exclusive of test suites.
In fact, they aid in the construction of test suites. With JCR, it is
possible to have JCR validators call custom code based on rule names
for evaluation of JSON in the context of the protocol and specific to
the needs of the test suite. I've already written a JCR parser in
hopes of getting a validator written, and that is one of my intended
features.

I welcome feedback on this draft, and encourage those who have not
looked at it to do so.

-andy

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 15 Oct 2014 16:27:59 -0400

Hi Andrew,

> I welcome feedback on this draft, and encourage those who have not

Looks good. The draft I have in the tzdist WG (draft-ietf-tzdist-service)
uses JCR to describe the JSON responses that the protocol uses. I have 
found the JCR syntax pretty simple to use and understand and it looks to me 
like a good way to describe JSON.

There is an open issue in the tzdist WG about how best to describe the JSON 
responses - my preference is to stick with JCR and thus I would like to see 
JCR get published as an RFC suitable for normative reference in a preposed 
standard.

-- 
Cyrus Daboo

---

From: "Andrew Biggs (adb)" <[email-masked]>
Date: Wed, 15 Oct 2014 20:49:18 +0000

Hi Andrew,

I've only had a chance to briefly scan the new draft, but it looks good so
far.  Will pass on more detailed comments after I've had a chance to dig
in more.

I've been working with the previous revisions of the JCR for over a year
now, to document JSON based APIs and API design guidelines, and have
really appreciated the conciseness and simplicity of the syntax.  The
symbolic approach actually becomes pretty natural after a short time as
evidenced by the feedback I've received from readers of my docs.

Andrew

On 10/15/14, 1:54 PM, "Andrew Newton" <[email-masked]> wrote:

>Htmlized:       
>http://tools.ietf.org/html/draft-newton-json-content-rules-03
>Diff:
>_______________________________________________
>json mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/json

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 15 Oct 2014 17:11:13 -0400

I think a tool for producing specs and JSON parsing code is a good
idea. And I have been using one for quite a while.

The main difference between Andy's scheme and mine is that I have a
description element that allows each entry to be documented and I have
elements that allow messages to be grouped together as transactions
and transactions into services.

There are some differences in the syntax but the main features I
didn't add were to specify maximum/minimum integer values. I can see
the point in specifying 1,2,4 or 8 bit integers and signed or
unsigned. But going deeper seems unnecessary.

If we specify any limits it should be on overall message sizes.

The part that I think the group could usefully look at is the
composition problem. Lets say we have Spec A/1.0 and we want to use
that inside Spec B/1.0, how should we guide people to do that in a way
that maximizes code reuse and all that goodness and also means that we
are clear when we do A/1.1 and B/1.1 and they both introduce new code
points.

This is what XML purported to solve but turned out not to.

---

From: Andrew Newton <[email-masked]>
Date: Wed, 15 Oct 2014 17:47:28 -0400

> and transactions into services.

By messages, transactions, services do you mean each message is a JSON
"document", and the transactions and services are protocol concepts
beyond the JSON serialization? Otherwise I do not understand. Is this
in an I-D?

> unsigned. But going deeper seems unnecessary.

To each his own, I guess. I can certainly think of use cases where
min/max are useful.

> are clear when we do A/1.1 and B/1.1 and they both introduce new code

There is an include directive in JCR that does this, if I correctly
understand your point.

> This is what XML purported to solve but turned out not to.

I think XML, and more specifically XML Schema, is what gives a lot
people pause when discussing JSON schemas.

-andy

---

From: "Paul E. Jones" <[email-masked]>
Date: Thu, 16 Oct 2014 23:16:24 +0000

Andy,

This spec is being used to help craft the syntax in some work ITU-T SG16 
is currently undertaking, so I'm glad to see some forward motion on it.  
I do have a few comments.

I really appreciate seeing an enumeration in there.  That will be very 
helpful.

You introduce the idea of "object mixins", but wasn't that syntax 
already possible?  Did you just introduce the word "mixins" or is this 
really new syntax?

I can appreciate the motivation for "# include", but we cannot automate 
processing of "Section 3 of RFC XXXX".  I guess this wasn't necessarily 
intended, anyway, but I still think it would be more appropriate to do 
it this way:

# include "http://uri/to/some/file.jcr" ; Section 3 of RFC XXXX

I do prefer the original syntax over the proposed alternative.  I 
appreciate the argument that syntax is a matter of taste.  However, the 
purpose of this draft is to define content rules for JSON.  So, those 
making use of it will know JSON.  I don't think the alternative is going 
to make that any easier or useful.  (Of course since I said that, it 
will now become widely used throughout the industry.)

Also, I noted the ABNF was removed from this version.  I think having 
the ABNF there is immensely helpful, as there are many syntax constructs 
that are not immediately obvious by just looking at the limited number 
of examples.  I do hope to see the ABNF return in a subsequent version.

Paul

------ Original Message ------
From: "Andrew Newton" <[email-masked]>
To: "JSON WG" <[email-masked]>
Sent: 10/15/2014 3:54:50 PM
Subject: [Json] more on schemas, JCR -03

>http://www.ietf.org/rfcdiff?url2=draft-newton-json-content-rules-03
<snip>

---

From: Andrew Newton <[email-masked]>
Date: Fri, 17 Oct 2014 10:44:31 -0400

Paul,

Thanks for your comments and review.

On Thu, Oct 16, 2014 at 7:16 PM, Paul E. Jones <[email-masked]> wrote:
> You introduce the idea of "object mixins", but wasn't that syntax already
> possible?  Did you just introduce the word "mixins" or is this really new
> syntax?

That syntax was already present with the group rule, it just wasn't
obvious that mixins were possible with it. I hope you find the concept
useful.

> intended, anyway, but I still think it would be more appropriate to do it
> this way:
>

I agree. Your approach is better.

> I do prefer the original syntax over the proposed alternative.  I appreciate
> the argument that syntax is a matter of taste.  However, the purpose of this
> draft is to define content rules for JSON.  So, those making use of it will
> know JSON.  I don't think the alternative is going to make that any easier
> or useful.  (Of course since I said that, it will now become widely used
> throughout the industry.)

I prefer the original syntax as well. If nobody finds the alternative
syntax useful, it should be removed.

> Also, I noted the ABNF was removed from this version.  I think having the
> ABNF there is immensely helpful, as there are many syntax constructs that
> are not immediately obvious by just looking at the limited number of
> examples.  I do hope to see the ABNF return in a subsequent version.

I plan to put it back once syntax issues are settled.

Thanks.
-andy

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 17 Oct 2014 13:49:29 -0400

Since Andy asked, here is a rough draft describing my alternative approach:

https://tools.ietf.org/id/draft-hallambaker-protogen-00

To see specifications generated using the tool take a look at:

http://prismproof.org/resources.html#specifications

All the JSON specifications are generated from schemas described with
protogen. All the examples are produced by code generated from the same
code. This ensures that the code and data are always locked.

On the topic of syntax, I really don't care very much. I am going to use my
occam style syntax which was adopted by python. The existing parser does
accept braces as an alternative to indentation for control.

So this is my preferred style:

    Structure Site
    Description
    |A site location
    String Country
    Description
    |ISO ALPHA-2 Country Code.
    String precision
    Decimal Latitude
    Decimal Longitude
    String Address
    String City
    String State
    String Zip

But the following is accepted:

Structure Site {
    String Country {
                   ...
                   }
                ....
                }

If people really like I can make the lexer ignore semicolons but the parser
does not need them. Unlike my comp sci undergrad lecturer I knew that
Chomsky's LR(1) model is designed to describe human languages.

I have from time to time considered writing a parser that would accept
either RFC 4627 JSON syntax or a python style syntax. all it would require
is a modification to the lexer. Which is fairly straightforward as that is
generated using a tool.

Whether the syntax specifies <type> <identifier> or <identifier> <type> is
something I keep changing my mind on. It is certainly more consistent to
use the latter but C switches between the first and the second.

---

From: "Andrew Biggs (adb)" <[email-masked]>
Date: Wed, 22 Oct 2014 21:19:30 +0000

>I welcome feedback on this draft, and encourage those who have not

Comments below on draft -03.  In general, I really appreciated the
additional explanatory text introduced in this draft, particularly in the
introduction.  

1. Introduction

* s/more concise/more concise and unambiguous/

* s/the rules name/the rule's name/

* The bullet list names 5 types of rules, but the subsequent text of this
section only mentions the first 4.

* Figure 4 shows capitalized attributes, though some of the corresponding
member rules of figure 5 are not (e.g. "width").

* I find figure 6 takes much less effort to grok than figure 7.  I¹m not a
big fan of the alternate syntax.

3. Rules

* I had to read the first paragraph a few times before I really understood
it.  Wondering if it might help to coin the terms "named rule" vs
"anonymous rule" to differentiate between the case where a rule name is
used as part of another rule's definition, versus a definition embedded
directly within a definition.  Seems like this could make it easier to
explain.

* s/rule names of other appropriate types of rules/rule names of other
rules of appropriate type/

* I¹m guessing rule names are case sensitive, it may be worth mentioning
one way or the other.

3.2 Member Rules

* First sentence, I would think value rules are the simplest.

* s/may be substituted for/may be (substituted with | replaced by)/

3.3 Member Rules 

* Regarding the '&' token, if I have something like this:

	response { location_uri & referrer_uri }

Does this mean that the location_uri must be present (since there's no '?'
before location_uri)?  If so, then would rewriting it this way:

	response { ?location_uri & referrer_uri }

mean "location_uri is optional, and referrer_uri may be present iff
location_uri is present" or does it mean "location_uri is optional, and
referrer_uri must be present iff location_uri is present"?  Seems like we
could actually represent either of these conditions, almost as concisely,
using just the existing syntax for groups:

	response { ?( location_uri, ?referrer_uri ) }
	response { ?( location_uri, referrer_uri ) }

So I'm wondering, do we actually need '&'?

3.4 Array Rules 

* I'm not exactly advocating this, but if '&' is supported for member
rules, should it have an analogous place within array rules?

3.6 Any Value and Any Member

* Regarding the exception that allows repetition of member rules with ^""
as the member name, would these represent valid syntax:

	any_member_any_value ^"" : any
	object_of_three_to_six_members_that_are_anything { 3*6
any_member_any_value }

	any_member_string_value ^"" : string
	object_of_three_or_more_members_that_are_strings { 3*
any_member_string_value }

3.5 Group Rules

* Can the Œ?¹ token be used in group rules?  I have actually been using it
this way in practice, it seems useful.

4.1 ignore-unknown-members

The ignore-unknown-members directive seems to be more consistent with
common practice in JSON than the alternative.  Would it make sense for
this to be regarded as the default, and instead define a directive such as
"pedantic" or "no-unknown-members" for the case where the presence of
unknown members invalidates a JSON object?

4.3 all-members-optional

With the all-members-optional directive, would it be an error to use this
directive with rules that explicitly mark some members as optional?

Cheers,
Andrew

---

From: Andrew Newton <[email-masked]>
Date: Fri, 24 Oct 2014 09:41:49 -0400

> big fan of the alternate syntax.

I'm beginning to think the alternative syntax is not a good idea.
Nobody seems to like it.

> directly within a definition.  Seems like this could make it easier to

Good point.

> using just the existing syntax for groups:

It means location_uri is optional, and referrer_uri can only be
present if location_uri is present.

>         response { ?( location_uri, ?referrer_uri ) }

Or it could be written that way.

> So I'm wondering, do we actually need '&'?

Good point. We can probably remove it.

> rules, should it have an analogous place within array rules?

Given your point above, we shouldn't do this.

> any_member_string_value }

Yes.

> this way in practice, it seems useful.

Given your point above, I think it is a useful addition.

> unknown members invalidates a JSON object?

That's a good idea.

> directive with rules that explicitly mark some members as optional?

I don't think it should be an error, but I do see how it could cause
confusion. Perhaps this directive should be removed. What do you
think?

-andy

---

From: "Andrew Biggs (adb)" <[email-masked]>
Date: Fri, 24 Oct 2014 13:57:44 +0000

>confusion. Perhaps this directive should be removed. What do you

+1, I could see how this directive might create more confusion than it¹s
worth.

Andrew

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 24 Oct 2014 15:25:04 -0400

> I'm beginning to think the alternative syntax is not a good idea.

I think an alternative syntax for the schema language is not the best idea.

But back on the JSON list a while back there was a proposal for a less
verbose version of the JSON syntax to make it easier to use it for writing
admin config files.

So if you have something like

{ "name": "Fred",
  "type": "Integer",
  "constrants": { "max": 10, "min", 5}
  }

A small change to the lexer allows us to write that as follows:

name=Fred, type=Integer, constraints {
 max=10, min=5}

Which is a lot less noisy. But what you would really want is something more
like

Fred Integer {max=10, min=5}

Since name and type are required parameters we can elide the tags for them
if the parser is schema aware.

The ability to do that sort of thing is one of the main reasons to have a
schema. It makes it very easy to convert data from human readable format to
wireline.

---

From: Andrew Newton <[email-masked]>
Date: Fri, 24 Oct 2014 16:31:20 -0400

> if the parser is schema aware.

Interesting. I'll note that in JCR this is

"Fred" : integer 5..10

-andy

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 25 Oct 2014 09:55:46 +0200

> I'm beginning to think the alternative syntax is not a good idea.

+1

Instead, it might be more useful to invest in a critical review of the main syntax.
E.g., JCR repeats JSON’s mistake of using “,” as a separator, not as terminator.
(Before I say more about this, I need an ABNF to look at; I’m not sure the one in -02 is still current, clearly needing some review.)

I also would like to see someone spend time on comparing JCR to CDDL and possibly learning how and why they are different...

Grüße, Carsten