Subject: [Json] Sensible limits for tags

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 24 Jan 2014 16:50:20 -0500

I am just working on converting my JSON parser / schema package to C.

One major change in moving to C is that there is no string package and so
managing infinitely extensible strings becomes a lot more expensive than
for C#.

I am considering putting limits on the lengths of tags. Setting the max tag
length to 255 octets improves efficiency and 'should be sufficient for
anyone'. (insert 640 byte joke here).

Since I only support UTF8 encoding (anyone needing another encoding can
convert externally) this would support 128 Unicode characters which seems
generous.

Has anyone come up against this before and come up with a particular limit?

-- 
Website: http://hallambaker.com/

---

From: John Cowan <[email-masked]>
Date: Fri, 24 Jan 2014 17:49:23 -0500

> convert externally) this would support 128 Unicode characters which seems

255 bytes supports, worst-case, 63 Unicode codepoints, if none of them
are in the range U+0000 to U+FFFF (unlikely, but someone may want names
entirely in Gothic).

-- 
John Cowan   http://ccil.org/~cowan    [email-masked]
In might the Feanorians / that swore the unforgotten oath
brought war into Arvernien / with burning and with broken troth.
and Elwing from her fastness dim / then cast her in the waters wide,
but like a mew was swiftly borne, / uplifted o'er the roaring tide.
        --the Earendillinwe

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sat, 25 Jan 2014 15:49:16 +0900

> I am just working on converting my JSON parser / schema package to C.

First, I'd only do that with really strong reasons, but I'd guess you 
have been through that already.

> One major change in moving to C is that there is no string package

It's just a wild guess, but I'd think there should be such packages 
somewhere out there.

and so
> managing infinitely extensible strings becomes a lot more expensive than

In terms of programmer effort, definitely.

> anyone'. (insert 640 byte joke here).

Any such limit (even a lower one or a higher one) is going to work until 
it fails some day. At that point, it will become apparent that you 
implementation is simply not conforming.

I'd not go with a fully variable implementation, but use a fixed length 
(possibly shorter than 255 bytes) for efficiency in the general case, 
and something extensible for the exceptions.

> convert externally) this would support 128 Unicode characters which seems

John already corrected this. It would be 255 ASCII only, 127 
Cyrillic/Arabic/Greek/Hebrew,..., 85 for all Indic Scripts, Thai,..., 
and CJK.

> Has anyone come up against this before and come up with a particular limit?

No, I didn't. Long names should be really rare (a really thin long tail 
:-), but they can't be excluded.

Regards,   Martin.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Sat, 25 Jan 2014 14:14:18 +0100

>Has anyone come up against this before and come up with a particular limit?

It is very common to have arbitrarily long strings as keys in objects in
JSON documents especially when they don't come from a finite enumeration
of pre-defined keys. As a trivial example, take `sort | uniq -c` which
would create a histogram for repeated lines, and it would be natural to
represent the lines as object keys in JSON, and frequencies as values.
And it is obviously common that lines of text exceed 255 UTF-8 octets,
like when paragraphs are not hard-wrapped. I would regard anything below
64KB as a joke in a general-purpose parser.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: John Cowan <[email-masked]>
Date: Sat, 25 Jan 2014 19:23:31 -0500

"Martin J. Dürst" scripsit:

> Any such limit (even a lower one or a higher one) is going to work
> until it fails some day. At that point, it will become apparent that
> you implementation is simply not conforming.

It is in fact conforming, thanks to the escape in Section 4 that allows
implementations to set limits on things.  Indeed, no implementations
exist that allow *unlimited* strings.

> I'd not go with a fully variable implementation, but use a fixed
> length (possibly shorter than 255 bytes) for efficiency in the
> general case, and something extensible for the exceptions.

Fixed 255 length would be very wasteful.  I think the implication
is that counted strings are being used, with one-byte counts.

-- 
An observable characteristic is not necessarily         John Cowan
a functional requirement.  --John Hudson                [email-masked]

---

From: "John Levine" <[email-masked]>
Date: 26 Jan 2014 02:23:16 -0000

>It's just a wild guess, but I'd think there should be such packages 

There are a gazillion packages for counted dynamically allocated
strings.  Some are quite good.  The problem is that none of them are
in the standard C or POSIX libraries, so you can't count on having any
of them available.

R's,
John

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 25 Jan 2014 22:15:25 -0500

> in the standard C or POSIX libraries, so you can't count on having any

Precisely.

I would be very happy if the C world could detach itself from the C++ world
and define a set of simple libraries for lists, bound checked array and
strings.

I don't ever want C++ because the last thing I need is a badly designed
object system when I can choose C# or Java or possibly even the Google
scheme on its way.

Looking at most C packages, they typically come with their own memory
management and string handling schemes. Which means that when you have
three or four packages, there is redundancy and inconsistency.

Anyway, I came up with a solution that should suffice. If the labels grow
over 256 bytes I just feed them into SHA256 and thats done with them.

One reason I am rolling my own is that I am writing crypto code and none of
the libraries I have looked at is appropriate. Shotgun parsing isn't
acceptable in security code. The rule is that you first check the integrity
of the bits received, iff and only iff they are correct you put the checked
bits and only the checked bits through the parser and then the code works
on the parsed data and only the parsed data.

So the parser needs to be schema driven and the decoded structures synced
to the schema.

One of the advantages of JSON is that it avoids some of the security
problems that most IETF protocols have managed to create because it does
not have nested definite length delimited structures.

---

From: Stephan Beal <[email-masked]>
Date: Sun, 26 Jan 2014 10:06:44 +0100

> in the standard C or POSIX libraries, so you can't count on having any

The C JSON libs i've looked at or written tend to provide their own String
struct wrapper, not because the world needs yet another String struct for
C, but because it's the only portable way to do it in C without pulling in
3rd-party libs.

In the off chance that this is of any help to the original poster, mine
looks like this:

struct json_string {
    unsigned int length;
};

Notice that there is no (char *). The struct instances can be allocated
with N (==strlen) +1 (NUL) extra bytes and that can be used to store the
string bytes:

json_string * json_string_new(char const * src, unsigned int n);

An accessor function can be used to fetch the contents. Mine looks
something like (eliding the special-case optimization for zero-length
strings):

char const * json_string_cstr(json_string const * js){
    return (char const *)(js+1);
}

char const * json_string_cstr2(json_string const * js, unsigned int * len ){
    if(len) *len = js->length;
    return (char const *)(js+1);
}

and the public API provides no mechanism for modifying them.

-- 
----- stephan beal
http://wanderinghorse.net/home/stephan/
http://gplus.to/sgbeal
"Freedom is sloppy. But since tyranny's the only guaranteed byproduct of
those who insist on a perfect world, freedom will have to do." -- Bigby Wolf