Subject: [Json] Nudging the English-language vs. formalisms discussion forward

From: Paul Hoffman <[email-masked]>
Date: Wed, 19 Feb 2014 07:31:49 -0800

[[ Phill is the only one who has responded with a proposal to speak. Hearing whether others want to would be useful. --Paul Hoffman ]]

It's been a week, and yet I can't imagine that everything has been said with respect to our proposed charter item that now stands at:

  A set of natural-language terms and/or phrases for use in future specifications
  that use JSON. This explicitly excludes schema languages and similar formalisms.

After that, a bunch of people started talking about formalisms and actual schemas again. In order to get this decided, we need more discussion and then agreement. To that end, and to put our 90 minutes on Friday afternoon in London to best use, I will ask for at least three people to present their views in 10-minute presentations at the meeting. However, in order to cause this to not be the normal IETF "let's wait for the meeting" game, the presentations need to be done by next Monday, Feb. 24. That gives people on the list a preview of what will be said, time to argue about it, and time for the presenters to hone their slides if they want.

Let me know online or offline if you want to do a presentation. If you're not going to be at the meeting but want to say something, you need to find some like-minded soul with whom to work on the presentation.

--Paul Hoffman
_______________________________________________
json mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/json

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 07:49:07 -0800

For the record, I think the draft language is clear and correct and we
should move forward with it.

On Wed, Feb 19, 2014 at 7:31 AM, Paul Hoffman <[email-masked]> wrote:

> [[ Phill is the only one who has responded with a proposal to speak.
> Hearing whether others want to would be useful. --Paul Hoffman ]]
>
> It's been a week, and yet I can't imagine that everything has been said
> with respect to our proposed charter item that now stands at:
>
> After that, a bunch of people started talking about formalisms and actual
> schemas again. In order to get this decided, we need more discussion and
> then agreement. To that end, and to put our 90 minutes on Friday afternoon
> in London to best use, I will ask for at least three people to present
> their views in 10-minute presentations at the meeting. However, in order to
> cause this to not be the normal IETF "let's wait for the meeting" game, the
> presentations need to be done by next Monday, Feb. 24. That gives people on
> the list a preview of what will be said, time to argue about it, and time
> for the presenters to hone their slides if they want.
>
> Let me know online or offline if you want to do a presentation. If you're
> not going to be at the meeting but want to say something, you need to find
> some like-minded soul with whom to work on the presentation.
>
> --Paul Hoffman

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 10:56:38 -0500

Perhaps we could do it a different way

What features/non features would people like from a formalism for
describing protocols of which JSON is at least one example?

If people want a feature and it is compatible with other feature proposals
then I can add it to my open source tool. However it must be noted that the
strongest feature request is 'simplicity'.

Are people writing tools because they want to write tools or because they
want a feature in a tool? If that feature can be described as a requirement
rather than an implementation, that is even easier.

Since my JSON Web Service has a HTTP binding, I have to have a minimal HTTP
client in the library as an option. While knocking up a simple HTTP header
parser, I suddenly discovered that the HTTP headers are regular enough that
they can be described using the same schema as JSON.

When we did the KEYPROV working group, the main spec is in XML but a group
said 'all our stuff is in ASN.1, we want a version in that' since the group
in question were the IETF Chair and a Security AD, this was a pretty strong
request.

So even though my view is that JSON is the data model I intend to use for
all future protocols (unless something else comes along), It seems to me
that if we design a formal documentation/prototyping tool we should have it
be capable of dumping out the equivalent XML and ASN.1 schemas. But since
the capabilities of XML and ASN.1 schema are a superset of the capabilities
of JSON, this is pretty easy to do.

-- 
Website: http://hallambaker.com/

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 08:05:17 -0800

I feel that successful Internet interoperation is achieved at the level of
syntax; clear specification of which bytes should be sent and which
returned.  Many times over the years I’ve heard assertions that if you get
the data model right, then the syntax is ephemeral: JSON or XML or ASN.1,
who cares? I’ve never found that notion remotely plausible, as there always
impedence mismatches.  You end up having to fight over another syntax to
describe the model before you can fight about the syntax you’re going to
send over the wire.

This is why I deplored the recent (but vanishing, thank goodness) trend to
offer APIs in XML-or-JSON-take-your-pick.

So, I find Phil’s arguments here (as with most schema-centric arguments)
entirely unpersuasive.

On Wed, Feb 19, 2014 at 7:56 AM, Phillip Hallam-Baker <[email-masked]>wrote:

> Since my JSON Web Service has a HTTP binding, I have to have a minimal
> HTTP client in the library as an option. While knocking up a simple HTTP
> header parser, I suddenly discovered that the HTTP headers are regular
> enough that they can be described using the same schema as JSON.
>

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 10:06:16 -0600

> of JSON, this is pretty easy to do.

Sure that's doable.  Quick question: should a JSON schema also follow
the model of defining "types", like ASN.1?  Or do you have something
else in mind?

Nico
--

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Feb 2014 11:30:59 -0500

Tim Bray scripsit:

> returned.  

  It took me years to realise how deep and important the divide is between
  wanting an SDK and wanting to know the underlying protocol. Too much of
  our biz can only see one of these realities. I grew up with networked
  minicomputers and (mostly) Unix, and maybe that's why, in the final
  analysis, I always want to see the bits on the wire, because in the
  final analysis, given any programmable device, I can work with them.
        --Tim Bray, some time before 2001

-- 
All Gaul is divided into three parts: the part          John Cowan
that cooks with lard and goose fat, the part            http://ccil.org/~cowan
that cooks with olive oil, and the part that            [email-masked]
cooks with butter. --David Chessler

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 11:52:17 -0500

On Wed, Feb 19, 2014 at 11:05 AM, Tim Bray <[email-masked]> wrote:

> returned.  Many times over the years I've heard assertions that if you get
> who cares? I've never found that notion remotely plausible, as there always
> describe the model before you can fight about the syntax you're going to
> send over the wire.
>

The point is to focus the discussion on the data going over the wire rather
than the syntax.

When the discussion is about syntax, that is usually a specification smell.
Take HTTP's weak e-tags

ETag: W/"who/ordered/this"

Now syntax is the very least of the issues that ETags raise. But the resort
to this particular syntax should have at least been a hint of the problems
to come.

Another peculiarity of the HTTP spec, there are two encodings for dates
that parsers must support. Both contain completely useless information (why
does a machine need to know the day of the week anyway). Both are
needlessly verbose but we are talking about compressing headers rather than
not sending irrelevant data. If people had not got into the weeds on syntax
we might have come up with a better caching model.

I don't see the value in having multiple encodings for a Web Service. But I
would much rather do that than end up with two separate specifications
because one group wanted ASN.1 and the other XML or some people want JSON
and others must have XML.

XML Web services have a whole ecology that some people are very heavily
bought into. They really can't use a specification that doesn't play nice
with that system.

JSON is the future for new work. But just as the PKIX people had a point
when they asked for an ASN.1 version of KEYPROV, there are going to be
people whose existing infrastructure is XML who want to make use of work
being done by a WG producing a JSON spec.

The point is that the function of this group and any other platform level
group is to support the work of the IETF WGs and the other groups that want
to build on top. If we can provide them with a mechanism that allows them
to provide a smooth transition path to the common encoding syntax, that
helps them meet their goals.

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 09:18:59 -0800

On Wed, Feb 19, 2014 at 8:52 AM, Phillip Hallam-Baker <[email-masked]>wrote:

>
> The point is to focus the discussion on the data going over the wire
> rather than the syntax.
>
>
> When the discussion is about syntax, that is usually a specification
> smell. Take HTTP's weak e-tags
>
> Now syntax is the very least of the issues that ETags raise. But the
> resort to this particular syntax should have at least been a hint of the
> problems to come.
>
> I don't see the value in having multiple encodings for a Web Service. But
> I would much rather do that than end up with two separate specifications

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 11:20:03 -0600

> offer APIs in XML-or-JSON-take-your-pick.

First this objection: indeed, no one should want that.  The only such
conversion that makes sense is JSON->HTML for BUIs, and that should
happen mostly on the client side (except for dumb clients, if the
service wants to reach them).

Even if the conversion can be completely automatic.  Why would anyone
want to burn cycles/energy running such conversions (except perhaps
for the JSON->HTML for dumb clients case)?

> describe the model before you can fight about the syntax you’re going to

The syntax is no more ephemeral than the RFC is (where there's an RFC).

Sometimes there's just no need for an RFC.  Most services out there
that define HTTP+JSON APIs don't bother publishing I-Ds for them, much
less RFCs, _even when then promise API stability_.  Yet even then,
syntax can help others implement clients for those APIs -- that's the
point of publishing an API: to help others implement clients for them,
to increase the popularity of the service.

> So, I find Phil’s arguments here (as with most schema-centric arguments)

Phil proposed that conversions to ASN.1 be feasible to please certain
people, not to allow services to serve JSON-or-DER or anything of the
sort.  Your argument about the latter is therefore a non-sequitur.

My proposal is that the WG take all comers for JSON schema languages
as Informational and leave it at that (well, all proposals for which
there's enough authors and reviewers, enough interest).  That can't
even be an irritant for you: you can ignore them...

...unless you think that formal languages used by others will impair
your ability to understand their specs, unless you really prefer
English prose to the max.  That'd be an argument I'd want to hear, if
you were making it.  I'm prepared to buy into a strong such argument,
even though you can probably tell that biased in favor of simple
schema languages.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 09:22:30 -0800

[pardon the mangled email. Blame my cat]

> The point is to focus the discussion on the data going over the wire

Here is where we disagree absolutely. The point is to specify the syntax
clearly and unambiguously, and the semantics of its payload.  Trying to
come at it from the other direction (specifying data structures and then
extracting syntax) leads to huge mistakes like CORBA and WS-*.

You go on to provide examples of badly-designed protocol syntax.  I don’t
think we have any disagreement that bad design is possible and should be
avoided.

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Feb 2014 12:25:11 -0500

> extracting syntax) leads to huge mistakes like CORBA and WS-*.

I agree absolutely with Tim, if it wasn't clear before.  Semantics is vague
and fuzzy (my semantics of your JSON may consist solely in counting the
number of fields in the top-level object, for example).  Syntax is not.
Agreement on syntax promotes interoperation; agreement on semantics
takes forever, pushing syntax to the rear, thus tending to create bad syntax.

-- 
What asininity could I have uttered     John Cowan <[email-masked]>
that they applaud me thus?              http://www.ccil.org/~cowan
        --Phocion, Greek orator

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 09:30:20 -0800

On Wed, Feb 19, 2014 at 9:20 AM, Nico Williams <[email-masked]>wrote:

> even be an irritant for you: you can ignore them...

No harm in that, although I also perceive little benefit.

> you were making it.

I think schemas can be useful (but designing good schema languages is
horribly hard, and so easy to get wrong).

I think clear English prose is *essential*, the one thing a specification
must have. Thus, schemas can be actively harmful if arguing over them
distracts attention from crafting the prose properly.  This is particularly
the case when the schema language is a flawed tool, which so many of them
are.

I also think that for most protocols, an open-source validator is immensely
more useful than a schema. Validators can check semantic constraints that
are in principle inaccessible to schema languages, especially simple ones.

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 11:30:50 -0600

>         --Tim Bray, some time before 2001

I'll grant that SDK-only protocols are bad in this context: because
the protocols then are proprietary (whereas we're the IETF, and we do
open protocols).  I don't see what the problem is with open protocols
with abstract APIs... or, for that matter, message descriptions in
some formal (or even ad-hoc but not just English prose) syntaxes.

Nor do I see the relevance of the above quote to this thread: it's yet
another non-sequitur.

I get that some don't want any JSON schemas, to which my answer is: so
don't use them.  But please refrain from fallacious arguments.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 11:40:05 -0600

> No harm in that, although I also perceive little benefit.

So ignore them.  Advise against their use even.

> horribly hard, and so easy to get wrong).

Hard?  Nah, it's just an exercise in re-inventing the same wheel
again.  XDR is really a subset of ASN.1, with different syntax, but
clearly it's possible to map XDR on ASN.1 trivially.  Ditto DCE RPC's
IDL.  And so on.

I really want any JSON schema to be expressed in JSON itself: we all
have JSON parsers, so at least the first step is done and we won't
have to worry about how difficult it is to write LALR(1) grammars for
ASN.1 or whatever.

> the case when the schema language is a flawed tool, which so many of them

English prose cannot be avoided.  For me it's about finding the
sweet-spot between formal language and English prose.  We could
publish code (remember SDL?) as a specification, but no one seriously
proposes _that_ (certainly not I).

> are in principle inaccessible to schema languages, especially simple ones.

What do you want to validate?  Form?  Then use schema; a form
validator not based on schema... is still a schema of sorts.

Or do you want to validate semantics?  That requires test-suites.
Perhaps we should publish test-suites, yes -- not a bad idea, except
for the enormous impact I think that would have on RFC authoring costs
and publication timelines.

Nico
--

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 19 Feb 2014 17:49:24 -0000

If the IETF does choose to promote JSON schemas I recommend that such 
languages be designed with the knowledge that their principle use is likely 
to be in RFCs (or similar) documents.  Hence they need to be able to be 
interleaved in with narrative text without limiting the structure of the 
text.  This can be achieved in ways similar to how PHP is included in HTML. 
For example you could do something like:

------------Example start--------------------

2.3.2 Connection Address Type
The ConnectionAddressType represents an IP address and port part that can be 
used to create an IP connection.

<?JSONSchemaA

ConnectionAddressType ::= object {
    IP4Address address;
    IPPort port;
    String { UDP / TCP / ... } protocol;
}
?>

- address specifies the IP4 address of the connection.
- port specifies the 16-bit port of the connection
- protocol specifies whether the connection should be made over UDP or TCP.
------------Example end---------------------

This fixes the problem of having narrative and protocol description 
separated by a long way.

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
----- Original Message ----- 
From: "Tim Bray" <[email-masked]>
To: "Nico Williams" <[email-masked]>
Cc: "Phillip Hallam-Baker" <[email-masked]>; "Paul Hoffman" 
<[email-masked]>; "JSON WG" <[email-masked]>
Sent: Wednesday, February 19, 2014 5:30 PM
Subject: Re: [Json] Nudging the English-language vs. formalisms discussion 
forward

> are in principle inaccessible to schema languages, especially simple ones.

--------------------------------------------------------------------------------

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 19 Feb 2014 18:04:08 -0000

----- Original Message From: "Nico Williams"

> have to worry about how difficult it is to write LALR(1) grammars for

Personally I think it's more important that the schemas are easier to read 
and write by a human.  Compare:

    int<0..65535>? port;

to:

    [ {
        "name": "port",
        "type": "int",
        "minOccurs": 0,
        "maxOccurs": 1,
        "minInclusive": 0,
        "maxInclusive": 65535
        } ]

Comments are also nice in schema!

An important consideration in designing a schema like the above is to have 
some kind of 'escape' mechanism for extending the definition such as 
attributes and annotations in C# and Java.

Pete Cordell
Codalogic Ltd

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 10:10:32 -0800

On Wed, Feb 19, 2014 at 9:40 AM, Nico Williams <[email-masked]>wrote:

> I really want any JSON schema to be expressed in JSON itself

Reasonable sounding, but the resounding success of RelaxNG’s non-XML syntax
(which surprised me, for one) makes this sort of idea less self-evidently
true.

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 12:22:04 -0600

> (which surprised me, for one) makes this sort of idea less self-evidently

It's compact form, but yes, yours and Pete Cordell's are great points.
 Fair enough.  I'd be fine with any syntax that maps onto JSON (like
RNG compact onto XML) and is easy to write LALR(1) grammars for.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 13:43:33 -0500

> > extracting syntax) leads to huge mistakes like CORBA and WS-*.

They are only mistakes because they are huge. Microsoft and IBM wanted to
build a consulting business which meant that they liked a specification
that was overly complex.

The whole ProtoGen system is less than about 5,000 lines of code and it has
equivalent functionality to the WS-* stack.

XML isn't the farce that SGML was and JSON is even simpler than XML without
loss of power (though it does not make a very good document markup).

C# and Java are not the screwups that C++ was either. And C is not ADA.

The fact that the B-crew has botched a job in the past does not mean the
approach is wrong. The collapse of ADA did not prove that high level
languages were a bad idea. Though there were folk on USENet making that
argument at the time. My college tutor was known for having walked out of
the ADA design meetings but he still worked on many other programming
languages afterwards.

> takes forever, pushing syntax to the rear, thus tending to create bad

JSON has no semantics, All the JSON syntax has is a mapping to an implicit
data model.

Semantics can be defined very precisely and most modern computer science
courses teach methods of doing that Z, VDM and the rest are all very
precise ways of specifying the behavior of a protocol with as high a degree
of precision as LR(0) parsers allow for syntax.

What I have found is that making use of syntax to encode semantics is
always a mistake. For example we might have a specification that says that
a Date field can either be an RFC3337 DateTime or an integer giving the
offset in seconds from the current time. That is certainly possible but we
now require the parser to be able to recognize the format of the data and
select the representation appropriately. Maybe 95% of people will do that
right but at least 5% will do it wrong and if one of those implementations
becomes popular we end up with a specification that now has three different
cases to code, the two in the spec and the real life situation that isn't
documented.

Using different tags for "DateTime" (=String) and "DeltaTime" (=Integer)
avoids that whole business.

When people reach for Regular Expressions they are almost always doing
something that is better done without.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 19 Feb 2014 10:47:57 -0800

On Wed, Feb 19, 2014 at 10:43 AM, Phillip Hallam-Baker <[email-masked]>wrote:

>
>
> On Wed, Feb 19, 2014 at 12:25 PM, John Cowan <[email-masked]>wrote:
>
>> Tim Bray scripsit:
>>
> The whole ProtoGen system is less than about 5,000 lines of code and it
> has equivalent functionality to the WS-* stack.
>
> XML isn't the farce that SGML was and JSON is even simpler than XML
> without loss of power (though it does not make a very good document markup).
>

I agree completely with above.

-+ Tatu +-

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 19 Feb 2014 21:02:42 +0100

At the danger of repeating myself and others here, I’ll try to summarize:

— We have quite good experience with using a single, standard (!) ABNF in IETF protocols.
  ABNF is a production system with well-understood semantics.
  It is somewhat idiosyncratic in the world of EBNF, but that has caused *zero* problems in practice.
  (People have just written ABNF parsers.  You still have the second half of the afternoon to do something else after that.)

— A production system that generates JSON (at the data model level) is easy to do.
  (But we have to find people who have the background and can commit the time.)  

— Relax NG compact is a nice existence proof that a production system like this can work and be highly usable.
  A JSON version could be even simpler.

— Previous attempts at trying to express XML or JSON data model syntax in the formalism of XML or JSON itself provide incontrovertible proof that this approach does not work.
  Don’t do that then.
  (It is so much easier to spend half the afternoon writing the parser for the workable syntax.
   We can even spec it in ABNF!)

— If you want to cover 100 % of the “syntax” of an XML document, you need to add Schematron to Relax NG compact.
  a) Don’t do that then for JSON — stay with an 80 % solution like Relax NG (which is more like 90 %) or maybe even simpler.
  b) Choose some form of “Jpath” to complement the production system with assertions.

(Note that choice b can always be added later, on a separate timeline from doing a production system.)

My proposals following from this little exercise in fact finding:

— I would propose that we try to find energy for doing the production system.
(In addition to the Web people, we may want to tap the YANG people for some recent experience in doing this kind of work.)

— I would propose that we stay open to adding a Jpath/Schematron approach (choice “b”), *if* somebody brings a credible one to the table.

— I also would propose that we collect a small number of benchmarks that we use for demonstrating that proposals for the production system are useful.
  My first suggestion: RFC 7071.  But we need a couple of different ones.  RFC 7095?  Maybe too ambitious.

Grüße, Carsten

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Feb 2014 15:06:10 -0500

> don't use them.  But please refrain from fallacious arguments.

We tried not using XSD schemas.  Didn't work.

-- 
John Cowan  [email-masked]  http://ccil.org/~cowan
If I have seen farther than others, it is because I am surrounded by dwarves.
        --Murray Gell-Mann

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Feb 2014 15:47:08 -0500

Phillip Hallam-Baker scripsit:

> Semantics can be defined very precisely and most modern computer
> science courses teach methods of doing that Z, VDM and the rest are
> all very precise ways of specifying the behavior of a protocol with as
> high a degree of precision as LR(0) parsers allow for syntax.

That's behavioral semantics (what to do), not document semantics (what
is meant).  It is impossible to define document semantics without
reference to the recipient: indeed, as a member of the pragmaticist
tradition I define the semantics of a document as its effect on the
recipient.  A book written in Georgian has no semantics to me, for I
cannot understand a single word of it.

> always a mistake.

See, if I didn't know that you were talking about behavioral semantics,
that would be incomprehensible.  This very email uses syntax (English
syntax) to encode the semantics of what I am (I hope) communicating to
you.  Indeed, there is no other way for me to communicate it.

> Using different tags for "DateTime" (=String) and "DeltaTime"
> (=Integer) avoids that whole business.

That is, of course, a syntactic distinction.

> something that is better done without.

"I have a problem.  I know!  I'll use a computer.  Now I have 100,000
problems." --me

-- 
John Cowan              [email-masked]          http://www.ccil.org/~cowan
Historians aren't constantly confronted with people who carry on
self-confidently about the rule against adultery in the sixth amendment to
the Declamation of Independence, as written by Benjamin Hamilton. Computer
scientists aren't always having to correct people who make bold assertions
about the value of Objectivist Programming, as examplified in the HCNL
entities stored in Relaxational Databases.  --Mark Liberman

---

From: Barry Leiba <[email-masked]>
Date: Wed, 19 Feb 2014 17:04:35 -0500

Using Carsten's "summary" note as an anchor, let me back up to where
this proposal came from.

I'm interested in providing a recommended way (through a non-binding,
Informational document that describes it) in which protocol elements
that are JSON things can be described in documents.  I'm hoping we can
keep this work item to that limited scope, and not go too far in
trying to devise something mega-general that does everything but
eat.[1]

That means that I'm looking for something that addresses the 90% case
nicely and cleanly.  The 10% of the messier cases should be workable,
but can be messier, I think.  As an example of what made me think
about this, look at these:
1. Section 6.1.1 of http://tools.ietf.org/html/draft-ietf-repute-media-type-12
2. Section 6.2.2 of http://tools.ietf.org/html/draft-ietf-repute-media-type-13

Note that the attempt to reproduce the JSON ABNF, in version -12, had
a number of problems.  I think the mechanism in version -13 is much
cleaner... and it's simple and easy to explain.  Nested and complex
structures can be handled as they are in ABNF, by naming the nested or
complex bit and then expanding it in another "rule".  I think this
sort of mechanism meets the "addresses the 90% case nicely" desire.  I
like it because it's easy to write and, more importantly, easy to read
and understand.

We could possibly come up with something more general; we could
consider Andy Newton's draft, or variations of that (as Andy
suggested, perhaps with changes to make it more readable).  But I'd
really like us to avoid getting so general that we end up with
something that's precise and covers everything... but is difficult to
get right and hard to read.

Barry

[1] Points to you if you know the reference:
http://mindprod.com/jgloss/debe.html

On Wed, Feb 19, 2014 at 3:02 PM, Carsten Bormann <[email-masked]> wrote:
> At the danger of repeating myself and others here, I'll try to summarize:
>
> -- We have quite good experience with using a single, standard (!) ABNF in IETF protocols.
> -- A production system that generates JSON (at the data model level) is easy to do.
> -- Relax NG compact is a nice existence proof that a production system like this can work and be highly usable.
> -- Previous attempts at trying to express XML or JSON data model syntax in the formalism of XML or JSON itself provide incontrovertible proof that this approach does not work.
>   Don't do that then.
> -- If you want to cover 100 % of the "syntax" of an XML document, you need to add Schematron to Relax NG compact.
>   a) Don't do that then for JSON -- stay with an 80 % solution like Relax NG (which is more like 90 %) or maybe even simpler.
>   b) Choose some form of "Jpath" to complement the production system with assertions.
>
> -- I would propose that we try to find energy for doing the production system.
> -- I would propose that we stay open to adding a Jpath/Schematron approach (choice "b"), *if* somebody brings a credible one to the table.
>
> -- I also would propose that we collect a small number of benchmarks that we use for demonstrating that proposals for the production system are useful.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Feb 2014 09:12:00 +1100

Agreed. 

Sent from my iPhone

> On 20 Feb 2014, at 2:49 am, Tim Bray <[email-masked]> wrote:
> 
> For the record, I think the draft language is clear and correct and we should move forward with it.
> 
>

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 17:23:43 -0500

> We tried not using XSD schemas.  Didn't work.

That is because XML markup requires the schema to be specified in the
document because the extensibility scheme depends on it (i.e. the mechanism
is bust)

What I hate about XML is that schemas don't compose properly. If I have a
schema that imports definitions from XMLSignature and XMLEncryption, this
is going to be visible in the final spec. Like Frankenstein's monster, the
stitching is always visible. This might make sense in a world where a
program could see that XMLSignature was being loaded and hand off security
decisions to the relevant module but that is not something I could expect
to work.

The problem with XML is that it was designed to be a less awful version of
SGML while allowing the HTML document markup that already existed to be
supported which meant support for most existing DTDs. So XML Schema has to
have a lot of features that are unnecessary and complex so that all the
horrors of SGML could be supported.

ASN.1 has a similar problem, it is not really as abstract as the title
suggests. Instead of accepting the fact that there would be some overhead
due to encoding, the BER encoding tries to avoid using more bytes than
necessary with horrors like IMPLICIT tags and bit fields and the rest.

Both are cases of someone or rather a committee trying to define a
specification for mapping bits on the wire to the abstract data model
rather than mapping the abstract data model to bits on the wire.

If we do the first then we have ASN.1 people saying 'hey, why do I need
this type tag here when I know that the optional tag X will always have an
integer field? And then what could have been a very easy to parse data
encoding suddenly becomes a very complicated one because the decoder now
has to know the schema to map the data on the wire to an abstract data
structure.

Doing things the second way round is much more regular. We decide that we
have an API that has to pass three bits of data in a message (in or out,
does not matter), one is a Boolean, another is an Integer, another is a
string:

Message Test
    Integer    ANumber
    Boolean   TruthValue
    String      SomeText

The obvious way to map these to the bits on the wire are to use a JSON
number value, true/false/null literals and a string value. And that is just
how the tool works. But give that same problem to some people to solve with
ABNF and they will produce a syntax 42+<Betcha can't guess what the false
value would be>. Which is why I don't like giving people ABNF to work with.

Leaving aside the fact that there are umpteen internal representations of
numbers, I can't think of an intrinsic type that isn't a boolean, a number
or a string. And as far as bits on the wire goes, there are no sets or bags
either, there are only structures and lists.

Where schemas get into trouble is trying to define features to support
ad-hoc sub-syntax that isn't actually an interesting general case. I have
added DateTime, DNS labels and URIs to my schema as intrinsic types because
those are existing sub-types that are very general occurring in most
network protocols and have very distinct semantics that are best handled by
a single encoder/decoder rather than have multiple handlers throughout the
code. But thats where I see the 90/10 rule cutting in.

---

From: Andrew Newton <[email-masked]>
Date: Wed, 19 Feb 2014 17:23:55 -0500

> the case when the schema language is a flawed tool, which so many of them

I agree, clear English prose are essential. At the moment, I am
evaluating two competing security protocols, both open standards
specified with XML Schema of which one is the product of an IETF
working group. The IETF standard is much clearer to understand because
it offers prose on top of the XSD. I cannot help but think that the
reason the IETF standard stands out is simply because it is an IETF
standard; along the way to RFC it was reviewed and reviewed and people
simply would not have let it pass had it been only an XSD. Therefore I
do not think we need to worry about IETF specifications being harmed
by schemas.

-andy

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 17:28:16 -0500

On Wed, Feb 19, 2014 at 5:04 PM, Barry Leiba <[email-masked]>wrote:

> 1. Section 6.1.1 of
> http://tools.ietf.org/html/draft-ietf-repute-media-type-12
> 2. Section 6.2.2 of
> http://tools.ietf.org/html/draft-ietf-repute-media-type-13
>

It is obviously possible to create an ABNF description of JSON (call this X)

It is thus possible to create an ABNF description of a Web Service message
as an ABNF description. (Call this Y)

What I think is going to be very hard is proving that a given Y is a subset
of X. And if we do that we risk having specifications that are not actually
JSON but only JSON-like.

Unless that is we start off with a tool that generates Y in a fashion that
makes it easy to check that it is a subset of X.

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 19 Feb 2014 22:29:32 -0000

I agree with Carsten's suggestion of looking for something to do the 80%. 
The JSON group's biggest blessing might be that it dissolves before it can 
add sufficient esoterica to the schema to make it useless.

Things I've learnt while working with schema is that a schema needs to:

- Be able to be explicit about which one (or many) constructs serve as the 
top-level construct,

- Expecting a 'schema' processor to miraculously import stuff over the 
network is hard,

- Being able to extend a schema via a third-party document is important 
(especially for the IETF).

Also, cardinality and integer ranges, and string lengths are 80% of 
constraints a schema needs on types.

I thought I'd throw down n gauntlet as a baseline for something to be beat. 
So I started with something C like, and stole stuff from other languages and 
came up with the following

import "something.jsc"
    // import just tells the schema processor to issue a warning if
    // the specified filename does not appear on the command-line.
    // It's not a directive to find the specified file.

namespace com.ietf.person;
    // Namespaces are handy for composing vocabularies across
    // multiple domains.  Switching to another namespace then output
    // a new namespace directive.

start struct com.ietf.person
    // The 'start' keyword marks one or more root constructs
    // struct maps to an object of unordered members
{
    string<1..255> name;
        // Constraints on a type are in angled brackets.  In the case
        // of a string it is the string length, here 1 to 255 characters.
    string<1..255>[?] alias;
        // Cardinality is expressed in square brackets.  The Kleene
        // operators are used for the common cases.  More complex
        // cases might be [5], [1..2], [1..*]
    ShortString[?] maidenName;
    int<!0..!65536>[1..10] scores;
        // Constraints on integers are the number range.  Inclusive
        // range might be <0..65535>, exclusive ranges <!0..!10>
    bool isEligible;
    Car[1..*] cars;
    House[*] houses;
    Sport[*] sports;
    com.ietf.other.job job;
        // Referring to a type in another namespace
};

string<1..255> ShortString;
    // Defining a simple type

struct Car
    // Defining a compound type
{
    string make;
    string model;
    int<1900..9999> year;
};

struct House
{
    string name;
    string street;
    string town;
    string country;
};

union Sport
{
    null track;
    null racket;
    null water;
    null water;
    null motor;
};

// Allow extensions that can plug components into
// type defined elsewhere
plug into Car, House
{
    float[?] cost;
};

plug into Sport
{
    null insane;
};

// If required to return to the global namespace:
namespace ;

In many respects I think the problem we will have is not that the problem is 
too hard, but that it is too easy and we all have an opinion on it, making 
hard to come to agreement!

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: "Carsten Bormann" <[email-masked]>
To: "JSON WG" <[email-masked]>
Sent: Wednesday, February 19, 2014 8:02 PM
Subject: Re: [Json] Nudging the English-language vs. formalisms discussion 
forward

— We have quite good experience with using a single, standard (!) ABNF in 
IETF protocols.
  ABNF is a production system with well-understood semantics.
  It is somewhat idiosyncratic in the world of EBNF, but that has caused 
*zero* problems in practice.
  (People have just written ABNF parsers.  You still have the second half of 
the afternoon to do something else after that.)

— A production system that generates JSON (at the data model level) is easy 
to do.
  (But we have to find people who have the background and can commit the 
time.)

— Relax NG compact is a nice existence proof that a production system like 
this can work and be highly usable.
  A JSON version could be even simpler.

— Previous attempts at trying to express XML or JSON data model syntax in 
the formalism of XML or JSON itself provide incontrovertible proof that this 
approach does not work.
  Don’t do that then.
  (It is so much easier to spend half the afternoon writing the parser for 
the workable syntax.
   We can even spec it in ABNF!)

— If you want to cover 100 % of the “syntax” of an XML document, you need to 
add Schematron to Relax NG compact.
  a) Don’t do that then for JSON — stay with an 80 % solution like Relax NG 
(which is more like 90 %) or maybe even simpler.
  b) Choose some form of “Jpath” to complement the production system with 
assertions.

(Note that choice b can always be added later, on a separate timeline from 
doing a production system.)

— I would propose that we try to find energy for doing the production 
system.
(In addition to the Web people, we may want to tap the YANG people for some 
recent experience in doing this kind of work.)

— I would propose that we stay open to adding a Jpath/Schematron approach 
(choice “b”), *if* somebody brings a credible one to the table.

— I also would propose that we collect a small number of benchmarks that we 
use for demonstrating that proposals for the production system are useful.
  My first suggestion: RFC 7071.  But we need a couple of different ones. 
RFC 7095?  Maybe too ambitious.

Grüße, Carsten

---

From: Barry Leiba <[email-masked]>
Date: Wed, 19 Feb 2014 17:32:15 -0500

> makes it easy to check that it is a subset of X.

Oh, I absolutely think that tools to support this are a big help!

Barry

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 19 Feb 2014 23:57:19 +0100

>makes it easy to check that it is a subset of X.

The main problem with using ABNF for Y is that it is tedious to write it
out properly, they want to write "example" and not a complex grammar for
all possible "\uXXXX" escape sequences. This is a common problem with
URI scheme specifications, for instance. Beyond that it is not really a
problem for a tool to compare two grammars as would be needed here (the
problem is undecidable in the most general case, but even then a regular
approximation could be used; I will probably release a tool later this
year that does something like this).
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Cowan <[email-masked]>
Date: Wed, 19 Feb 2014 18:04:49 -0500

Phillip Hallam-Baker scripsit:

> document

Not so, unless you mean DTDs.

> because the extensibility scheme depends on it (i.e. the mechanism
> is bust)

Not so.

> supported

Not so.

> which meant support for most existing DTDs.

Not so.

> So XML Schema has to

Not so.

> rather than mapping the abstract data model to bits on the wire.

Not so.

> I can't think of an intrinsic type that isn't a boolean, a number

All I can say is, your imagination appears deficient.

-- 
Verbogeny is one of the pleasurettes    John Cowan <[email-masked]>
of a creatific thinkerizer.             http://www.ccil.org/~cowan
   --Peter da Silva

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 17:15:08 -0600

> do not think we need to worry about IETF specifications being harmed

So you agree that prose is needed (no one here yet disagrees), but you
don't think schemas are harmful because we generally require prose
(which we do) and that's good.  Good!

We do need prose-mostly descriptions of protocols.  We need formal
languages to avoid accidents and to convey concisely and precisely
things that can be difficult to do in prose (in any natural language).
 Prose is needed for semantics -- formal alternatives for that (e.g.,
SDL) so far haven't worked well.

What I want to avoid:

 - TLS-style tool-less inconsistent ad-hoc syntaxes
 - SSHv2-style tool-less inconsistent ad-hoc syntaxes
 - 100% prose-only (e.g., SASL)

I also do not want an ASN.1 where the only way to implement is either
a) spend years building tools first, or b) ad-hoc manual coding based
off a syntax that's full of nuances (since this leads to accidents).

Whatever schema(s) we go with has to be simple enough that ad-hoc
manual coding off of the syntax is less likely to cause accidents than
ad-hoc manual coding off of prose, while also being usable with
automatic tooling.

Since we're talking about JSON we don't need to worry about silly
warts like DER/BER/CER tagging.  No need to mention such nastiness :)

Things I'd consider, and maybe propose:

 - just pattern-matching validation rules (e.g., using the jq or other
similar language that can do pattern matching);

 - a schema with describe-by-example-mostly metaschema, with special
names denoting "types" defined separately, something like:

{
    "message": { "sender": "_sender_type, "receiver":
"_receiver_type", "payload": "_json_string"}
    "sender": { ... },
    ...
}

 - a schema like compact RelaxNG that can be parsed and converted into
one of the above;

or anything else that's relatively simple and from which code can be
generated (or which can be interpreted) to at least do validation (for
testing, not necessarily at run-time in production), and preferably
more (e.g., RPC-like stubs, programming language types, ...).

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 17:20:02 -0600

On Wed, Feb 19, 2014 at 4:57 PM, Bjoern Hoehrmann <[email-masked]> wrote:
> out properly, [...]

Some problems with ABNF beyond the ones you mention:

 - ABNF is a grammar for strings, but we're dealing with
already-parsed (notionally) JSON texts.  Augmenting the JSON ABNF to
specify a JSON message sounds painful.

 - Generating anything more than a recognizer/validator from ABNF is
difficult or difficult to use the results.

ABNF has its uses; this isn't one of them.

Nico
--

---

From: "Manger, James" <[email-masked]>
Date: Thu, 20 Feb 2014 11:18:42 +1100

One more example of an existing practice for describing JSON values appears in recent draft specs from the FIDO Alliance <http://fidoalliance.org/specifications/download>. It seems to be based on Web IDL with ECMAScript bindings <http://www.w3.org/TR/WebIDL/#ecmascript-binding>.  The following example comes from fido-uaf-protocol-v1.0-rd-20140209.pdf:

dictionary Version {
  int mj; // Mandatory.
  int mn; // Mandatory.
}

dictionary OperationHeader {
  Version upv; // Mandatory.
  DOMString op; // Mandatory. Must be “Reg”, “Auth” or “Dereg”
  DOMString appID; // Mandatory. string[1..512].
  DOMString serverData; // Optional, string[1..1536]
  Extension[] exts; // Optional.
}

dictionary Extension {
  DOMString id; // Mandatory. string[1..32].
  DOMString data; // Mandatory. base64url(byte[1..8192]).
  boolean fail_if_unknown;// Mandatory.
}

I don’t particularly like it; it doesn’t feel like JSON (eg "dictionary" vs "object"); constraints are all in comments; and IDL is aimed at APIs as opposed to protocol messages. But it sort-of works: it is fairly easy to guess what the JSON has to look like.

--
James Manger

---

From: Tim Bray <[email-masked]>
Date: Wed, 19 Feb 2014 16:24:51 -0800

Hm, obvious when you think about it, since JSON’s roots are in JavaScript
and JSON docs are parseable as such, a bagful of JS assertions (yeah, I
know JS doesn’t have an “assert” primitive, but still) could be a
not-terrible stand-in for a schema. Sort of Schematron-like.

On Wed, Feb 19, 2014 at 4:18 PM, Manger, James <
[email-masked]> wrote:

> One more example of an existing practice for describing JSON values
> appears in recent draft specs from the FIDO Alliance <
> http://fidoalliance.org/specifications/download>. It seems to be based on
> Web IDL with ECMAScript bindings <
> http://www.w3.org/TR/WebIDL/#ecmascript-binding>.  The following example
> comes from fido-uaf-protocol-v1.0-rd-20140209.pdf:
>
> dictionary Version {
>   int mj; // Mandatory.
>   int mn; // Mandatory.
> }
>
> I don’t particularly like it; it doesn’t feel like JSON (eg "dictionary"
> vs "object"); constraints are all in comments; and IDL is aimed at APIs as
> opposed to protocol messages. But it sort-of works: it is fairly easy to
> guess what the JSON has to look like.
>
> --
> James Manger

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 18:31:15 -0600

On Wed, Feb 19, 2014 at 6:24 PM, Tim Bray <[email-masked]> wrote:
> and JSON docs are parseable as such, a bagful of JS assertions (yeah, I know
> JS doesn’t have an “assert” primitive, but still) could be a not-terrible
> stand-in for a schema. Sort of Schematron-like.

Yes, pattern matching type rules would WFM as one option.  Note that
the stuff you're responding to here looks not too unlike... XDR, or
even ASN.1.  The metaphor seems to be: named types.

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 19 Feb 2014 20:21:31 -0500

This is what I was referring to:

<saml:Assertion   xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
ID="b07b804c-7c29-ea16-7300-4f3d6f7928ac"   Version="2.0"
IssueInstant="2004-12-05T09:22:05">
   <saml:Issuer>https://idp.example.org/SAML2</saml:Issuer>
   <ds:Signature
xmlns:ds="http://www.w3.org/2000/09/xmldsig#">...</ds:Signature>
   <saml:Subject>

Interpreting those namespaces correctly requires a huge amount of complex
logic that is completely unnecessary.

Since we had Eve Maler on the TC, I think we were doing the XML schema
right the second time round...

If XML Schema had been better designed the namespaces would never appear on
the wire

Though since the post had no content other than 'oh no it isn't' the poster
might have had a different idea.

Clearly composition of schema specs is hard. Does it have to be solved?

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 20 Feb 2014 12:32:11 +1100

+1 to the hm. ;)

I wouldn't object if someone wanted to establish a WebIDL profile, etc., or just refer to it in their spec.

I would have a problem if we started requiring people to use it.

Cheers,

On 20 Feb 2014, at 11:24 am, Tim Bray <[email-masked]> wrote:

> Hm, obvious when you think about it, since JSON’s roots are in JavaScript and JSON docs are parseable as such, a bagful of JS assertions (yeah, I know JS doesn’t have an “assert” primitive, but still) could be a not-terrible stand-in for a schema. Sort of Schematron-like.
> 
> 
> On Wed, Feb 19, 2014 at 4:18 PM, Manger, James <[email-masked]> wrote:

--
Mark Nottingham   http://www.mnot.net/

---

From: Nico Williams <[email-masked]>
Date: Wed, 19 Feb 2014 19:39:07 -0600

> I would have a problem if we started requiring people to use it.

Absolutely, there should be no requirement that people use it (or any
alternative).  I don't think anyone has proposed such a requirement.
The only proposal is that the WG charter permit the WG to take on work
items for schema documents.

Nico
--

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 19 Feb 2014 21:05:47 -0500

Hi Pete,

--On February 19, 2014 at 10:29:32 PM +0000 Pete Cordell 
<[email-masked]> wrote:

> I thought I'd throw down n gauntlet as a baseline for something to be
> beat. So I started with something C like, and stole stuff from other
> languages and came up with the following
>

Please take a look at Andrew Newton's spec 
(<https://datatracker.ietf.org/doc/draft-newton-json-content-rules/>).

I have used that in two specs of my own:

<http://tools.ietf.org/html/draft-douglass-timezone-service-10#section-7>
<http://tools.ietf.org/id/draft-daboo-aggregated-service-discovery-03.txt> 
Appendix B

I found it to be easy to use and provide the necessary "commentary" for the 
JSON format chosen for those specs. I know several others who have 
implemented the timezone service protocol and we had no issues with 
interoperability related to the JSON format (beyond some poor text 
descriptions on my part).

As Andrew noted earlier, his spec could probably be simplified some more - 
but we certainly don't need anything more complicated than what is there 
now.

-- 
Cyrus Daboo

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 13:01:54 -0000

----- Original Message From: "Cyrus Daboo"
> (<https://datatracker.ietf.org/doc/draft-newton-json-content-rules/>).

Thanks Cyrus.  It certainly looks like another option.  I would suggest that 
the authors could look at the following to improve it:

- It doesn't seem to have much in the way of supporting modularity.  It 
would be attractive if a schema could pull in and use definitions defined in 
other documents.  Many IETF protocols consist of suites of components and 
support for that would be handy.

- It doesn't seem to support third-party extensions to a core protocol.  For 
example, HTTP and SIP are defined in a core RFC but there are then a number 
of additional RFC that define extensions.  Support for expressing how a new 
extension can extend an existing external extension would be good.  (Just 
about every schema language I've seen fails miserably at this!)

- The plethora of data types are a problem for schema languages.  I've come 
to the conclusion that the best option is to declare a "microformat" 
pseudo-type that basically says this is a string encoded in a particular 
way.  The 'particular way' can then be defined in narrative form as required 
(80-20 etc).  This avoids having to define the kitchen sink of data types up 
front.  e.g.:

struct Asset
{
    GPSLocation location;
};

microformat GPSLocation;
    // A GPSLocation is a pair of comma separated floating-point
    // numbers representing longitude and latitude.
    // e.g. "location": "0.0,51.5"

- Does it allow multiple roots to be defined?  Many protocols have 
request/response type pairs so it's nice to define a protocol that has the 
form:

"request": {...} and "response": {...}

Pete Cordell
Codalogic Ltd
----- Original Message ----- 
From: "Cyrus Daboo" <[email-masked]>
To: "Pete Cordell" <[email-masked]>; "Carsten Bormann" 
<[email-masked]>; "JSON WG" <[email-masked]>
Sent: Thursday, February 20, 2014 2:05 AM
Subject: Re: [Json] Nudging the English-language vs. formalisms discussion 
forward

> I found it to be easy to use and provide the necessary "commentary" for 
> the JSON format chosen for those specs. I know several others who have

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Feb 2014 09:36:17 -0500

On Thu, Feb 20, 2014 at 8:01 AM, Pete Cordell <[email-masked]>wrote:

>  Please take a look at Andrew Newton's spec (<https://datatracker.ietf.
>> org/doc/draft-newton-json-content-rules/>).
>>
>
> Thanks Cyrus.  It certainly looks like another option.  I would suggest
> that the authors could look at the following to improve it:
>
> would be attractive if a schema could pull in and use definitions defined
> in other documents.  Many IETF protocols consist of suites of components
> and support for that would be handy.
>

That is probably the right decision at this point.

IETF has not decided how to make JSON protocols composable. We have JOSE
but that may be a poor example because signature and encryption are
wrappers rather than composition. JOSE does not actually encrypt JSON, it
encrypts an octet stream that may contain JSON which is a different thing
entirely.

Now that might in part be due to lack of having a schema so it is not
currently possible to say 'structure X from RFC9666 goes here'.

But what does modularity look like? Is it importing structures defined in
one spec into another?

There are bad precedents here. QNames in XML exist in the ABNF but using
them in schemas has some very odd and unexpected effects that we only
discovered after trying to use them. It seems obvious that if you have
<prefix:tag prefix2:attr="foo"> then you can have attribute values <tag
attr="prefix:foo"> but this requires the application to be aware of the
namespace fiasco that is usually hidden from view in platforms like .NET
etc.

So just pointing to a syntactic production in another spec and making use
of it may not be acceptable because there may be a subtle change in context.

I certainly don't want anything as broken as XML Namespaces in JSON. Nor
can I see a need for them.

> - It doesn't seem to support third-party extensions to a core protocol.
>  For example, HTTP and SIP are defined in a core RFC but there are then a
> number of additional RFC that define extensions.  Support for expressing
> how a new extension can extend an existing external extension would be
> good.  (Just about every schema language I've seen fails miserably at this!)
>

Also true and again this may be the right choice in this case.

But remember than in JSON, every structure can be extended just by adding a
tag.

> - The plethora of data types are a problem for schema languages.  I've
> come to the conclusion that the best option is to declare a "microformat"
> way.  The 'particular way' can then be defined in narrative form as
> required (80-20 etc).  This avoids having to define the kitchen sink of
> data types up front.  e.g.:
>

I disagree, I don't think the schema should address sub-syntax at all
except for a small number of encodings that are RFCs such as RFC3339
timestamps, URIs and DNS labels.

>    // e.g. "location": "0.0,51.5"

These can and should be encoded in JSON:

Structure GPSLocation
    Float X
    Float Y

Message Foo
    GPSLocation location

"location" : {"X" 0.0, "Y", 51.5}

Parsing floating point numbers is complicated. I want that complexity to be
in the JSON parser and nowhere else. Though this example is possibly a sign
that decimal fractions should be an intrinsic type since we use decimal
fractions in protocols but floats very rarely.

Perhaps I will add a Decimal type to my schema that would map to a INT64
structure with a multiplier of 1,000,000,000. That would allow for numbers
up to 17,000,000,000 with nine digits of decimal precision. They would map
to JSON decimal fractions.

When people reach for regular expressions it is almost always a sign that
they are doing something that they should not.

This is also a problem for my binary version of JSON (and I suspect CBOR).
I added the binary float values to JSON-B because changing from binary to
decimal fractions results in a loss of precision. Here we have a decimal
fraction being forced to binary. This is a big problem for currency
transactions and spatial coordinates.

Right now the encoding scores are

Microformat - 10 bytes
JSON   - 18 bytes
JSON-C  - 21 bytes

The compressed JSON encoding actually requires more bytes because the only
encoding for floats is Real64 (8bytes). And despite the number of bits we
have a precision loss because of a decimal->binary fraction conversion.

Adding a decimal fraction tag allows the number of bytes for each number to
be reduced to 3 and the locations would only require 8 bytes each. Though
this is very much a corner case, the text tags are short so we don't get
any leverage from compression and real GPS coordinates would have far more
decimal places.

In most real world examples you are going to get far more leverage by
sticking to only encoding in JSON data model and then finding an efficient
way to encode JSON rather than inventing ad-hoc microformats that are
neither JSON nor JSON data model, are going to require a custom parser and
are not going to compress.

---

From: John Cowan <[email-masked]>
Date: Thu, 20 Feb 2014 09:52:49 -0500

Phillip Hallam-Baker scripsit:

> Interpreting those namespaces correctly requires a huge amount of
> complex logic that is completely unnecessary.

That is untrue (just to vary myself a bit).

> right the second time round...

Well, your snippet of an *instance* document contains a declaration of
the XSD *schema* namespace, which makes no sense whatever.  I don't know
who wrote that instance (which appears in Wikipedia and elsewhere), but
I doubt it was Eve.

> If XML Schema had been better designed the namespaces would never
> appear on the wire

Namespaces long predated XML Schema.

> Though since the post had no content other than 'oh no it isn't' the
> poster might have had a different idea.

My point was that you were defending an argument I consider to be
incorrect using claims that any XML Schema user (certainly including
Eve) would certainly know were false, but which the present audience
might not recognize as such.

I am not attacking JSON or defending XML Schema.  I'm attacking the
claim of "semantic-first" designs to be the obviously Right Thing.
Indeed, XML Schema itself is a magnificent example of just such a
semantics-first design, with the result that it is damned hard for
mere mortals to decipher actual schemas without tools.

-- 
John Cowan   http://ccil.org/~cowan   [email-masked]
'My young friend, if you do not now, immediately and instantly, pull
as hard as ever you can, it is my opinion that your acquaintance in the
large-pattern leather ulster' (and by this he meant the Crocodile) 'will
jerk you into yonder limpid stream before you can say Jack Robinson.'
        --the Bi-Coloured-Python-Rock-Snake

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 15:22:52 -0000

----- Original Message From: "Phillip Hallam-Baker"
> That is probably the right decision at this point.

I'm not sure what "That" refers to here.

> IETF has not decided how to make JSON protocols composable.

Isn't that why we're here?

> We have JOSE

Yes, QNames are broken, so don't do that.  Think more packages in Java, 
namespaces in C#.  As simple as doing something like:

    int port;
    com.ietf.sip.contact contact;

yields JSON of:

    "port": 25, "contact": "...whatever..."

We don't have to make things complicated!

> can I see a need for them.

Be assured that XML Namespaces are irrelevant to the discussion.

> But remember than in JSON, every structure can be extended just by adding 

Yes.  The question is, how do we exploit that in the schema language?

> "location" : {"X" 0.0, "Y", 51.5}

I disagree with this.  Dates alone illustrate that there are 'microformats' 
that can result in a much more compact, meaningful and useful format than 
say:

    { "date": 25, "month": 12, "year": 2015, "hour": 12, "min": 0 }

Another location format might be:

    56°14'23.45"N,18°5'16.65"W

Forcing such formats to be JSONified may cause as much error as forcing IEEE 
754 numbers to be decimalised.

So let's learn from the past and recognise that we can't build them all in 
upfront and design our format accordingly.

> are not going to compress.

I'm not interested in compression beyond zip and co.  It may sound harsh of 
me, but my feeling is if this or the errors in floating point numbers is 
critical to you, then use something else.  We don't need something that is 
all things, to all people.

Pete Cordell
Codalogic Ltd

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Feb 2014 11:06:25 -0500

(trimmed)

> Isn't that why we're here?

It is why the group is here. But my point is that the group should decide
how to compose protocols and the people developing schemas should support
those tropes that are chosen.

So it is a separate problem. If people decide on how to do this, I will be
happy to extend my tool. But I don't think how this problem is solved
should be a way to decide between tools.

> We don't have to make things complicated!

That is a model that I could very easily support as my tool is written in
C#. In fact it already supports that kind of type label. All I would have
to do is add in 'Using' specifiers at the top.

The only thing it doesn't support are those semicolons. I find a carriage
return is enough.

Incidentally, this is what I meant when I said that discussion of the
schema ideas and merger might be more useful than more proposals.

>> timestamps, URIs and DNS labels
>
>

> I disagree with this.  Dates alone illustrate that there are
> 'microformats' that can result in a much more compact, meaningful and
> useful format than say:
>

But we already have an RFC for Date - RFC 3339. (Actually we have others
but we will forget those).

So this is one of the microformats that I define in the tool as intrinsic.
This also allows for the use of Integer encodings for DateTimes using
seconds since the start of some epoch.

>    56°14'23.45"N,18°5'16.65"W

{ [56,14,23,45], [-18,-5,-16,-65]}

Forcing such formats to be JSONified may cause as much error as forcing
> IEEE 754 numbers to be decimalised.
>

It is certainly possible to do the job wrong but the above does not result
in any loss of precision.

The only case where I would feel comfortable with the microformat above is
if the protocol was interfacing to some application where that format was
already defined and translation between the formats would be needed.

> upfront and design our format accordingly.

There may be a need for a microformat. But that should be rare and should
probably be specified in ABNF rather than the JSON schema.

> I'm not interested in compression beyond zip and co.  It may sound harsh
> of me, but my feeling is if this or the errors in floating point numbers is

If someone is saying 'I only want text, space isn't important' then fine,
just use text.

But if someone is saying 'lets use JSON but then lets avoid making it too
verbose using these microformats to shave a few bytes' then I would much
prefer to just use an efficient encoding.

Unless dealing with text documents, LZ compression of an inefficient
encoding is not as effective as an efficient encoding in the first place.

If the threat of a binary encoding causes someone to withdraw an ad-hoc
microformat then its done its job...

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 16:55:01 -0000

----- Original Message From: "Phillip Hallam-Baker" <[email-masked]>
> should be a way to decide between tools.

As it's framed at the moment, nobody has to decide between tools (at least 
not in the JSON working group).

> The only thing it doesn't support are those semicolons. I find a carriage

The details of how it's done are not important right now.  I only presented 
an example syntax as a way of illustrating what I'm proposing.

> seconds since the start of some epoch.

My position is that, having recognised that Dates represent a case where 
microformats are useful, perhaps we should not assume that these are the 
only cases.  IP addresses?  Crypto OIDs?  Dates on Mars?

>>    56°14'23.45"N,18°5'16.65"W

Exactly.  That's the use-case - where the format is already standard for the 
domain in question.

> probably be specified in ABNF rather than the JSON schema.

Exactly.  It's saying there's something special here that is beyond my (the 
JSON schema's) ability to define.  It's also admitting that it's a rare case 
and the JSON schema doesn't want to be able to define it.  However, just 
because it doesn't want to do it, doesn't mean it wants to prevent you from 
defining something if it's useful to you.  EBNF with narrative in an RFC 
would be a suitable way to define it's format.

> prefer to just use an efficient encoding.

Microformats are nothing to do with minimising byte count.  They are about 
representing data in a natural form for the application domain.

Pete Cordell
Codalogic Ltd

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Feb 2014 12:23:19 -0500

> only cases.  IP addresses?  Crypto OIDs?  Dates on Mars?

All I am saying is that it is a slippery slope.

There are two ways to go down a slippery slope, not at all or on skis.

Skis: I will add a new type, 'Format <Name>' that declares a string with
the named sub-format which can be documented elsewhere in ABNF.

This also solves my 50 shades of yuck problem trying to parse HTTP headers
with all the mindboggling syntax baroqueness. Sometimes a label has quotes,
sometimes not, sometimes angle braces. There is no logic at all just like
there is no logic to the choice of attributes or elements in XML2RFC format.

> JSON schema's) ability to define.  It's also admitting that it's a rare
> case and the JSON schema doesn't want to be able to define it.  However,
> just because it doesn't want to do it, doesn't mean it wants to prevent you
> from defining something if it's useful to you.  EBNF with narrative in an
> RFC would be a suitable way to define it's format.
>

Or a reference to some other spec where it was predefined.

That would also provide a way to deal with enumerations which I have
currently punted on.

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Feb 2014 11:22:12 -0600

> only cases.  IP addresses?  Crypto OIDs?  Dates on Mars?

There are two ways to deal with alternative representations:

 - convert to/from a canonical representation and use that one for interchange
 - use a discriminated union (XDR) / CHOICE (ASN.1, same thing)

I think any decent schema will need to allow for the latter, and not
just because of types that have multiple possible representations.

As for the former, it's always an option, schema or not.  And it's
more efficient to have receiver-makes-right in this case (dates &
timestamps), though that does require having a fixed set of
representations that can be sent, else a negotiation would be
required.

This seems obvious enough that we shouldn't dwell on it.

The real question is: which schema(s) should we be copying/learning
from?  So far the answer seems to be down to:

 - Schematron
 - RelaxNG compact
 - Web IDL

I think that's a fine starting point.

If you want to talk about requirements, I think we should consider things like:

 - which grammar parsing algorithms we want to support: LR, LALR(1),
LALR(k), GLR, ...

 - the basic metaphor:
    - types! (a-la ASN.1, only without the tags, no IOS, ...)
    - pattern matching rules! (something like collections of
XPath-like expressions)
    - something else

The metaphor thing gets to, in part, the purpose of the schema:

 - documentation for sure, validation no doubt, but, code generation
(into C, Java, JS, C#, ...)?  (IMO: yes)

Finally:

 - extensibility (meta: to have it or not; if yes, how)

 - modularity (meta: to have it or not; if yes, how)

Oh, and:

 - one schema language, or more?  (IMO: inevitably we'll end up with more)

I may have missed some things.  But I think that's a decent set of
attributes from which to start a schema requirements discussion.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Feb 2014 11:52:03 -0600

> There are two ways to go down a slippery slope, not at all or on skis.

This sounds to me like an argument about how to handle extensibility.
We should first consider whether we want schema extensibility and if
we do, whether we want it to be explicit (like the ... extensibility
marker in ASN.1) or implicit (like implicit ... ASN.1 extensibility
markers in every type).

My position: we should want extensibility, and it should be explicit
(because having every type be implicitly extensible is a PITA for code
generation).

Nico
--

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Feb 2014 13:30:59 -0500

> just because of types that have multiple possible representations.

In the code as it is right now it is possible to specify different formats
as different tags:

ConvolutedTime Structure
    DateTime                Date
    Format RFC1123     Date1
    Format RFC822       Date2

Would allow for any of the following:
    {"Date":"2002-10-02T10:00:00Z"}
    {"Date1":"Thursday, 20 Feb 2014 18:14:16 GMT"}
    {"Date2":"Thu, 20 Feb 2014 18:14:16 GMT"}

What I don't support at the moment is a constraint that says that only one
of the "Date", "Date1" and "Date2" tags is permitted. This could be
specified as:

ConvolutedTime Choice
    DateTime                Date
    Format RFC1123     Date1
    Format RFC822       Date2

[Protogen does allow braces to be used instead of indentation to denote
blocks but does not need end of statement or statement separators. If
people really insist on semicolons I can add a production to the lexer so
they are just treated like whitespace.]

> LALR(k), GLR, ...

I would hope we make the syntax so simple that we don't need the power of
LR parsers. They are models of human languages rather than computer
languages.

>     - pattern matching rules! (something like collections of

Since the target languages are likely to be C#, JS and Java, I suggest that
we use dot separated tags and array indexes as path extractors.

Given:

{"first" : {"second" : {"third" : [{ "fourth": 1}, { "fourth": 42}, {
"fourth": 666}]}}}

first.second.third[1].fourth = 42

> (into C, Java, JS, C#, ...)?  (IMO: yes)

I have code generation for C# and C. I can add Java without difficulty.

It is not clear to me that code generation is necessary or particularly
useful for scripting languages with late binding like JS. In those
situations you can just say

X = JSON_Parse (text)

Answer = first.second.third[1].fourth

>  - extensibility (meta: to have it or not; if yes, how)

Hopefully formats are enough.

I am adding a Using directive right now. It seems to do the trick.

>  - one schema language, or more?  (IMO: inevitably we'll end up with more)

There will be multiple schemas. I think the progression will work like it
did for JSON itself. People will choose the one they like and there will
eventually be a convergence.

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Feb 2014 12:38:23 -0600

On Thu, Feb 20, 2014 at 12:30 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> I would hope we make the syntax so simple that we don't need the power of LR
> parsers. They are models of human languages rather than computer languages.

Oops, I meant left recursive, not LR.  Apologies for the thinko.

> first.second.third[1].fourth = 42

Sure, where rules need to be expressed, using something XPath-like for
expressing paths would be nice.

> situations you can just say

For them the important thing may be to have XPath-like facilities.

> Hopefully formats are enough.

I'm not sure what you mean.

> I am adding a Using directive right now. It seems to do the trick.

What about namespacing?

Nico
--

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 19:42:42 -0000

> marker in ASN.1) or implicit (like implicit ... ASN.1 extensibility

I assume you're taling about extensibility of the vocabularies defined by a 
schema rather than extensibility of the schema language itself?

> (because having every type be implicitly extensible is a PITA for code

What aspect do you find a PITA?  Supporting both implicit extensibility and 
explicit extensibility is surely harder than just supporting implicit 
extensibility?

Pete Cordell
Codalogic Ltd

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 20 Feb 2014 14:55:03 -0500

On Thu, Feb 20, 2014 at 2:42 PM, Pete Cordell <[email-masked]>wrote:

> What aspect do you find a PITA?  Supporting both implicit extensibility
> and explicit extensibility is surely harder than just supporting implicit
> extensibility?

The elephant in this room is the X.509 CRITICAL flag.

The idea of the critical flag is that if there is a certificate extension
that an RP MUST understand to understand the certificate correctly it can
be marked critical and processors will reject the certificate rather than
accept it and do the wrong thing.

The problem came when people mistakenly interpreted CRITICAL as being
'important' rather than 'so important that if it is not understood you
should break backwards compatibility'.

So right now there is an X.509 extension that the PKIX specification says
MUST be marked critical and the CABForum certificate specification
explicitly states MAY be marked critical or not critical at the discretion
of the issuing CA.

I faced the same problem in the design of TASS which became the SAML
assertion format with only minor changes. Since I knew that I would not get
people to agree to a critical flag I instead introduced an element
<Conditions> and the rule that an RP MUST understand every condition to
rely on the assertion. It is the exact same mechanism but has not created
any problems as far as I know.

So what I would suggest for JSON is that we apply the same approach:

Anyone can add tags into any structure they like and tags that are not
understood are ignored unless the structure is tagged 'Splunge' in the
schema.

Splunge can be any word people like except 'Critical'. Right now the
metaschema says splunge.

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 21:40:40 -0000

> On Thu, Feb 20, 2014 at 2:42 PM, Pete Cordell :
>

Sadly I don't think you can write an RFC to prevent stupid people being 
stupid!

> Splunge can be any word people like except 'Critical'. Right now the

I like SIP's "Require" header, and have used the scheme in a number of 
protocols.  The advantage is that you can get that basic level of 
compatibility testing out the way in the early stages of processing.  An 
example in a JSON message might be:

    "Require": [ "TLS", "deferal#higson|standard" ]

The other benefit is that can be handled at the protocol design level and 
doesn't impact the schema language!

Pete Cordell
Codalogic Ltd

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Feb 2014 15:53:51 -0600

> schema rather than extensibility of the schema language itself?

Yes.  See ASN.1's extensibility marker.

Briefly, if you end a structure/set/choice type with '...' then
decoders are supposed to not fail just because unknown fields are
included in the input; the decoders.  (There's more to it, but I'm
simplifying.)

> explicit extensibility is surely harder than just supporting implicit

Let's say you're generating C structs from the schema, and a given
type is intended to be extensible, that it's an object.  So now you
need to decide what your parser will do when faced with names in this
object that were not known when the C code was generated.  There's two
options: ignore them, or make them available as a C representation of
an object with all the unexpected names.  The latter can be important
at times (e.g., when you're eventually going to re-encode the inputs
and you want to preserve those unknown bits).

So far so good.  But if you make every type extensible by default,
then you're polluting those C structs and the generated code even if
mostly you don't need extensibility -- this is the PITA.  IMO
extensibility needs to be requested explicitly, but then the problem
you run into is people forgetting to do so.

Nico
--

---

From: "Pete Cordell" <[email-masked]>
Date: Thu, 20 Feb 2014 22:24:36 -0000

>> and

A PITA, true, but not as big a PITA as when somebody forgets it or convinces 
themselves that they're never going to need it.  In my experience change is 
the only constant in data structures!

Pete Cordell
Codalogic Ltd

---

From: Nico Williams <[email-masked]>
Date: Thu, 20 Feb 2014 16:28:43 -0600

> the only constant in data structures!

Maybe so, but then you need to make sure that people don't extend
structures and expect extensions to be critical.  One way or another
you depend on ppl knowing what they're doing when they extend
protocols.  Anyways, I'm convincing myself that this (extensibility,
explicit or implicit) is a minor detail.

Nico
--

---

From: "Pete Cordell" <[email-masked]>
Date: Fri, 21 Feb 2014 12:06:15 -0000

> Anyways, I'm convincing myself that this (extensibility,

If you've had to live with XML Schema for any length of time, you'll know 
extensibility is only a minor detail if you actually have it.

I hope for this group it is a minor detail because it is a given.

Pete Cordell
Codalogic Ltd