Subject: [Json] Encoding

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 11:37:41 -0700

I think the section on encoding is not saying anything useful and should 
be completely removed.

---

From: Tim Bray <[email-masked]>
Date: Wed, 5 Jun 2013 12:12:13 -0700

You mean http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-00#section-3?

This is the only place that says default UTF-8, unless we adopt the
previous proposal that says so in the definition of String.  I don’t think
it’s OK to drop the UTF-8 mandate.

The language about encoding recognition is not actively harmful, but you’re
right, doesn’t add much value. -T

On Wed, Jun 5, 2013 at 11:37 AM, Douglas Crockford <[email-masked]>wrote:

> ______________________________**_________________
> json mailing list
> [email-masked]
> https://www.ietf.org/mailman/**listinfo/json<https://www.ietf.org/mailman/listinfo/json>
>

---

From: Stephan Beal <[email-masked]>
Date: Wed, 5 Jun 2013 21:27:21 +0200

On Wed, Jun 5, 2013 at 9:12 PM, Tim Bray <[email-masked]> wrote:

> The language about encoding recognition is not actively harmful, but
> you’re right, doesn’t add much value. -T
>

(and this time to the list...)

FWIW: the implication of the recognition possibilities didn't occur to me
until reading that bit, so i found it helpful. To be clear, i understand
you to be referring to the 2nd paragraph of the link:

Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020 <http://tools.ietf.org/html/rfc0020>], it is
possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal

---

From: Jacob Davies <[email-masked]>
Date: Wed, 5 Jun 2013 13:08:29 -0700

be completely removed.

I agree with a caveat about UTF-8 as the default, because it is possible
for two compliant implementations to communicate using any text encoding
(e.g. ASCII) provided they agree on escaping characters it cannot represent
literally, as long as it can literally represent all of
[a-zA-Z0-9+-.{}[",:/\]\\]. It's reasonable to specify UTF-8 as the default
where a content-type but no encoding is given, but otherwise leave it
unspecified.

The part on detecting encoding is of historical interest but not really
relevant anymore.

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 13:11:17 -0700

On 6/5/2013 12:12 PM, Tim Bray wrote:
> You mean 
> http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-00#section-3 ?
>
> previous proposal that says so in the definition of String. I don’t 
> think it’s OK to drop the UTF-8 mandate.
>
Not harmful, but not important. I added section 3 originally because at 
that time the web was knee deep in charsets and Unicode adoption was 
quite slow. So I insisted that the local charsets not be allowed so that 
I could avoid putting charset declarations into JSON.

Now, 12 years later, Unicode is now well established. So the thing that 
Section 3 was trying to do is no longer needed. I don't think the format 
needs to mandate the encoding. Perhaps that belongs in the best 
practices doc.

---

From: "Pete Cordell" <[email-masked]>
Date: Wed, 5 Jun 2013 21:04:40 +0100

Original Message From: "Douglas Crockford"

>I think the section on encoding is not saying anything useful and should be 
>completely removed.

As a humble hacker I found section 3 adds the most helpful additional 
pragmatic information beyond the diagrams at json.org.

Also, I'd be surprised if Unicode assumes the first characters will be 
ASCII, so I wouldn't expect to find encoding deduction rules along similar 
lines there, so the section seems to add some value.

A more explicit statement on whether BOMs are or are not allowed would be 
helpful also.

Humbly yours,

Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 5 Jun 2013 13:18:30 -0700

> where a content-type but no encoding is given, but otherwise leave it
Allowing character encodings other than existing ones (UTF-8/16/32) would
be a breaking change -- compliant libraries currently can count on simple
and reliable auto-detection -- and I can not see how use of other encodings
(except for special case of ASCII, as that simply falls within UTF-8
umbrella) would be considered compliant currently.

Is there a reason to expand set of allowed encodings? That would remove one
nice simplicity aspect, for little or no benefit.

> The part on detecting encoding is of historical interest but not really
I thought it is among more useful parts, as it points out that
auto-detection is easy to do. It is similar to XML spec's
(non-authoritative) part in how XML encoding is to be detected.

-+ Tatu +-

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 5 Jun 2013 13:22:14 -0700

On Wed, Jun 5, 2013 at 1:11 PM, Douglas Crockford <[email-masked]>wrote:

>> You mean http://tools.ietf.org/html/**draft-ietf-json-rfc4627bis-00#**
>> section-3<http://tools.ietf.org/html/draft-ietf-json-rfc4627bis-00#section-3>?
>>
> that time the web was knee deep in charsets and Unicode adoption was quite
> slow. So I insisted that the local charsets not be allowed so that I could
> avoid putting charset declarations into JSON.
>
> needs to mandate the encoding. Perhaps that belongs in the best practices
> doc.
>
>
If limitation on allowed encodings was lifted or removed, what would the
replacement then be? Unlike XML that specifies an in-document way of
specifying encoding, JSON has no equivalent mechanism. And use of
externally provided encoding declaration would either be mandated, or
suggested along with specifying default encoding (UTF-8) that must be
assumed in absence of such information.

I think it would be better to leave the limitation in place; if one
absolutely must work with non-compliant content, they are free to transcode
content from unsupported encoding into canonical one(s). This is what
happens currently with tools in Java space.

-+ Tatu +-

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 13:22:30 -0700

On 6/5/2013 1:18 PM, Tatu Saloranta wrote:
> On Wed, Jun 5, 2013 at 1:08 PM, Jacob Davies <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Wed, Jun 5, 2013 at 11:37 AM, Douglas Crockford
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     > I think the section on encoding is not saying anything useful
>     and should be completely removed.
>
>     I agree with a caveat about UTF-8 as the default, because it is
>     possible for two compliant implementations to communicate using
>     any text encoding (e.g. ASCII) provided they agree on escaping
>     characters it cannot represent literally, as long as it can
>     literally represent all of [a-zA-Z0-9+-.{}[",:/\]\\]. It's
>     reasonable to specify UTF-8 as the default where a content-type
>     but no encoding is given, but otherwise leave it unspecified.
>
>
> Allowing character encodings other than existing ones (UTF-8/16/32) 
> would be a breaking change -- compliant libraries currently can count 
> on simple and reliable auto-detection -- and I can not see how use of 
> other encodings (except for special case of ASCII, as that simply 
> falls within UTF-8 umbrella) would be considered compliant currently.
>
> Is there a reason to expand set of allowed encodings? That would 
> remove one nice simplicity aspect, for little or no benefit.
>
>     The part on detecting encoding is of historical interest but not
>     really relevant anymore.
>
>
I am not aware of any implementation that does that detection. Everyone 
uses UTF-8, at least on the wire. That is a practice that is so well 
established now that I don't think the format needs to mention it.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 5 Jun 2013 13:30:18 -0700

On Wed, Jun 5, 2013 at 1:22 PM, Douglas Crockford <[email-masked]>wrote:

>> On Wed, Jun 5, 2013 at 1:08 PM, Jacob Davies <[email-masked] <mailto:
>> [email-masked]>> wrote:
>>
>>     <[email-masked] <mailto:[email-masked]>**> wrote:
>> Is there a reason to expand set of allowed encodings? That would remove
>> one nice simplicity aspect, for little or no benefit.
>>

For JSON, Jackson (the leading Java JSON lib) has done this from beginning,
and actually supports all legal encodings, including UTF-32. Most other
libraries just punt the problem by requiring caller to pass in Reader that
does byte->char decoding; but that seems odd given how easy auto-detection
and handling is.

For XML all serious Java libs do auto-detection; Xerces, Woodstox and Aalto
to name some. Non-normative rules from XML spec are very valuable to get
this part right; and it works well.

But I do not see how one could omit specification of how encoding is to be
detected, given that a HUGE number of interoperability problems are
directly caused by mismatch between encoding writer used and reader
assumes. Relying on contextual encoding information may work for HTTP (but
I doubt even that, helping developers on daily basis); but there is more to
JSON usage than HTTP exchange.

-+ Tatu +-

---

From: "Markus Lanthaler" <[email-masked]>
Date: Wed, 5 Jun 2013 22:45:45 +0200

On Wednesday, June 05, 2013 10:23 PM, Douglas Crockford wrote:
> > [...]

The reason why it is so well established (for JSON at least) is because it
was in the RFC for a decade. I see no harm in leaving it in.

I had the charset discussion more than once and pointing people to the RFC
was by far the easiest way to convince them that e.g., no charset media type
param is necessary.

--
Markus Lanthaler
@markuslanthaler

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 Jun 2013 22:52:53 +0200

On Jun 5, 2013, at 22:11, Douglas Crockford <[email-masked]> wrote:

> I don't think the format needs to mandate the encoding.

So what about the text in "encoding considerations" in section 6?

Note that application/json does not have a charset parameter, so "not mandating the encoding" isn't really an option here.

On Jun 5, 2013, at 22:22, Douglas Crockford <[email-masked]> wrote:

> Everyone uses UTF-8, at least on the wire.

+1

Wouldn't it be nice if the specification reflected reality here with a MUST or at least a SHOULD.

(If that is not possible, maybe it would be sufficient to include that precious sentence in the bis document...)

Grüße, Carsten

---

From: Jacob Davies <[email-masked]>
Date: Wed, 5 Jun 2013 14:16:21 -0700

The point is that the entire discussion of encoding is irrelevant other
than assuming UTF-8 if no encoding is specified. The language should not be
concerned with the binary representation of the characters it is written. I
can write a perfectly valid JSON text on paper with a pen, including
non-ASCII Unicode characters in a string. Am I a non-compliant
implementation?

I think you can simply talk about JSON as a language consisting of a series
of characters drawn from the basic JSON alphabet, except inside quoted
string values that may not contain a small set of whitespace and control
characters, but may contain other literal characters. Reading a JSON text
produces a model containing nulls, booleans, numbers (of unspecified
representation), Unicode strings, arrays, and objects. Worrying about
whether the input text contained the literal bytes f09f98bf instead of
d83dde3f or a hand-drawn crying cat face on paper isn't necessary (and
irrelevant in those environments where the encoding is interpreted at a
level below that of JSON parsing).

I do not believe it has any effect on compliance or interoperability since
there is no existing requirement that all encodings or all possible
documents be accepted. (I doubt many implementations handle UTF-32 in the
absence of an external encoding specification, for instance.) All that the
current wording on encoding winds up saying is that a bunch of potentially
useful things that unambiguously work are not compliant (e.g. two parties
agreeing to put JSON in EBCDIC text columns in DB2 as long as they escape
everything outside the basic JSON alphabet).

The requirement that you assume UTF-8 unless you have agreed on another
encoding by some external channel should be enough to unambiguously
interpret any series of bytes that claims to be JSON without an encoding
specified. If you agree on some other binary encoding with your partner,
that's not the business of the specification.

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 5 Jun 2013 18:21:22 -0700

On Jun 5, 2013, at 11:37 AM, Douglas Crockford <[email-masked]> wrote:

> I think the section on encoding is not saying anything useful and should be completely removed.

The section is not saying anything useful, and it actually disagrees with the text from Section 6:
     JSON may be represented using UTF-8, UTF-16, or UTF-32.  When JSON
     is written in UTF-8, JSON is 8bit compatible.  When JSON is
     written in UTF-16 or UTF-32, the binary content-transfer-encoding
     must be used.

Proposal:

Change the title of Section 3 to "Encoding JSON on the Internet", and make the contents of the section:

JSON texts sent on the Internet MUST be encoded as UTF-8 unless there is a definitive way for a recipient to definitively determine the encoding.

--Paul Hoffman

---

From: Douglas Crockford <[email-masked]>
Date: Wed, 05 Jun 2013 18:50:21 -0700

> JSON texts sent on the Internet MUST be encoded as UTF-8 unless there is a definitive way for a recipient to definitively determine the encoding.
I think this belongs in the best practices doc.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 06 Jun 2013 13:29:54 +0900

Hello Douglas, others,

> established now that I don't think the format needs to mention it.

I'm not exactly a JSON expert, but think about how you would react if 
somebody said something like:

"In JSON, everybody uses '[' and ']' for arrays, and '{' and '}' for 
objects. This is a practice that is so well established now that I don't 
think the format needs to mention it."

As I hope this example shows, even if something is extremely well 
established, if it is part of interoperability, it better be clearly 
documented.

I agree with the assessment that character encoding is less of a mess 
these days than it was when the first RFC was written, but I'd rather 
have one nail too many in the coffin of that mess than one nail too few!

Regards,    Martin.

---

From: Tatu Saloranta <[email-masked]>
Date: Wed, 5 Jun 2013 21:35:52 -0700

On Wed, Jun 5, 2013 at 2:16 PM, Jacob Davies <[email-masked]> wrote:

> would be a breaking change -- compliant libraries currently can count on
> simple and reliable auto-detection -- and I can not see how use of other
> encodings (except for special case of ASCII, as that simply falls within
> UTF-8 umbrella) would be considered compliant currently.
>
If starting from scratch, I would agree. But we are not.

Existing JSON specification explicitly mandates that any byte sequence
serialization of JSON MUST be in one of sanctioned unicode encodings. Your
pen and paper case is not covered by JSON specification; I don't care how
you'd do it. But when that JSON goes on the wire, or stored on file system,
it better use one of legal Unicode encodings and nothing else.
This simplifies handling a lot, based on my work on writing tools (parser,
generators, databinding) and on supporting developers who do. Amount of
encoding issues with JSON is remarkably low, all things considered.

> I think you can simply talk about JSON as a language consisting of a
> series of characters drawn from the basic JSON alphabet, except inside
> quoted string values that may not contain a small set of whitespace and
> control characters, but may contain other literal characters. Reading a
> JSON text produces a model containing nulls, booleans, numbers (of
> unspecified representation), Unicode strings, arrays, and objects. Worrying
> about whether the input text contained the literal bytes f09f98bf instead
> of d83dde3f or a hand-drawn crying cat face on paper isn't necessary (and
You could, but this is a big backwards-incompatible change that affects
_existing_ libraries and use cases. If one can not assume simple and
reliable encoding detection, encoding must now be provided as external
metadata.

Decoding characters from byte stream is not necessarily below parser (for
performance reasons one often wants to bundle things); but more importantly
something has to handle it.
And with redefinition that task becomes bigger issue.

> that's not the business of the specification.

I think assumption that the default encoding (in absence of other
information) is UTF-8 would be sensible, regardless of other questions.

-+ Tatu +-

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Thu, 06 Jun 2013 11:55:50 +0200

>I think the section on encoding is not saying anything useful and should 

The specification has to define how to handle resources such as

  <data:application/json,%00%5B%00%31%00%5D>

which RFC 4627 defines to be a properly formed application/json resource
consisting of an array with the number `1` as only element and there are
quite a number of implementations that treat it as such. Same goes for

  <data:application/json,%EF%BB%BF%5B%31%5D>

which RFC 4627 defines to be malformed junk, but many implementations do
treat it as an array with the number `1` as only element.

Removing the whole Encoding section without replacement would make it a
lot less clear how to handle cases like the two above, so that is not an
option.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/