Subject: Re: [Json] JSL: Clarifying purpose, and renaming it to JDDF

From: Ulysse Carion <[email-masked]>
Date: Sun, 1 Sep 2019 15:19:39 -0700

> JDDF offers "type":"uint8" but very common languages, such as Java, don't have an unsigned 8-bit type. So code generation might use a short (16-bit signed integer) to handle the [0, 255] range. That seems to put the code in a very similar situation to using a long when the JDDF says "int54". In both cases the native type can hold values that the schema doesn't allow, requiring extra parse/stringify functions if you really want to enforce the schema's rules.

This isn't quite right. Yes -- Java only has "byte", but by convention
you can store both signed and unsigned 8-bit numbers in these bytes.
Protobufs does this:

> [1] In Java, unsigned 32-bit and 64-bit integers are represented using their signed counterparts, with the top bit simply being stored in the sign bit.

Source: https://developers.google.com/protocol-buffers/docs/proto#scalar

Furthermore, the Java standard library has methods that explicitly let
you do the conversion of a Java byte-as-uint8 into a Java int or long:

https://docs.oracle.com/javase/8/docs/api/java/lang/Byte.html#toUnsignedInt-byte-
https://docs.oracle.com/javase/8/docs/api/java/lang/Byte.html#toUnsignedLong-byte-

If bytes-as-uint8 weren't conventional, then why does java.lang.Byte
have methods that enable such a style?

This is why, in my view, it's reasonable to say that uint8 has a clear
correspondence to Java's byte.

>> I didn't see any support for binary data

> Suggested extra types:
> * "type": "intString" -- an integer exchanged in decimal notation as a JSON string, with no decimal point or exponent. (maybe only offer "int64String" instead, if it is only 64-bit values that typically use this format)
> * "type": "uintB64u" -- a (non-negative) integer (potentially huge, as often used in cryptography) exchanged as a JSON string holding the base64url-encoding (without padding) of an unsigned big-endian representation of the integer. (may need a comment about gracefully accepting leading 0 bytes).
> * "type": "bytesB64" -- a byte array exchanged as a JSON string holding its base64-encoding
> * "type": "bytesB64u" -- a byte array exchanged as a JSON string holding its base64url-encoding (without padding)

I think I'm opposed to this suggestion. Adding support for arbitrary
binary data is going to be challenging. I think I prefer that we just
stick to {"type":"string"}, and then folks can write their own
extensions to JDDF to achieve their specific needs. After awhile, the
community may perhaps converge on common extensions, or patterns may
arise and we can update the JDDF spec.

Anything we add to the spec now would have to be supported by all
implementations. So I'd prefer we give out a minimal spec to the
world, and see where the real world takes it. If we try to solve all
problems right away, we may make it too hard for implementations to
crop up, ultimately solving no problems for anyone.

On Thu, Aug 29, 2019 at 12:42 AM Manger, James
<[email-masked]> wrote:
>
> > https://tools.ietf.org/html/draft-ucarion-jddf-00
> > ...
> > I fear we may be at loggerheads on the question of {"type":"int53"}.
>
>
> JDDF offers "type":"uint8" but very common languages, such as Java, don't have an unsigned 8-bit type. So code generation might use a short (16-bit signed integer) to handle the [0, 255] range. That seems to put the code in a very similar situation to using a long when the JDDF says "int54". In both cases the native type can hold values that the schema doesn't allow, requiring extra parse/stringify functions if you really want to enforce the schema's rules.
>
> >> I didn't see any support for binary data
>
> Suggested extra types:
> * "type": "intString" -- an integer exchanged in decimal notation as a JSON string, with no decimal point or exponent. (maybe only offer "int64String" instead, if it is only 64-bit values that typically use this format)
> * "type": "uintB64u" -- a (non-negative) integer (potentially huge, as often used in cryptography) exchanged as a JSON string holding the base64url-encoding (without padding) of an unsigned big-endian representation of the integer. (may need a comment about gracefully accepting leading 0 bytes).
> * "type": "bytesB64" -- a byte array exchanged as a JSON string holding its base64-encoding
> * "type": "bytesB64u" -- a byte array exchanged as a JSON string holding its base64url-encoding (without padding)
>
> --
> James Manger

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 1 Sep 2019 15:27:57 -0700

Fixed-length strings and arrays both sound like things one could
implement with extensions, no?

Tuples are a very sensible suggestion. But I wonder if we can live
without them? I very much want to keep JDDF as small as possible.

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Sep 2019 18:41:50 -0400

Section 2.3 says that extensions can't be validation-related, so a length
limitation would most certainly be a matter of validation.

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 1 Sep 2019 16:03:31 -0700

Sure -- but you could use the length validation as a hint to code
generators, and a warning to data producers that the application will,
post JDDF-level validation, do additional length checks.

I'm not sure the need for length validation is strong enough to merit
its inclusion in this version of the spec.

On Sun, Sep 1, 2019 at 3:42 PM John Cowan <[email-masked]> wrote:
>
> Section 2.3 says that extensions can't be validation-related, so a length limitation would most certainly be a matter of validation.
>

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Sep 2019 19:14:28 -0400

The argument for it is precisely in situations where static allocation is
required along with static typing.  If you know the (maximum) length, you
can safely allocate a fixed amount of memory for your internal
representation, which you cannot do otherwise.   Languages with static
types and garbage collection (allowing dynamic allocation) are just one
particular kind of languages, and I think you need to accommodate all
widely used kinds of languages.  I wouldn't object to skipping bounded
arrays and strings in favor of just fixed-length ones.

See < http://www.catb.org/esr/microjson/>, a JSON-subset parser written in
C that imposes exactly this restriction so that it can run safely on
embedded systems with only small amounts of RAM.

On Sun, Sep 1, 2019 at 7:03 PM Ulysse Carion <[email-masked]> wrote:

> > Section 2.3 says that extensions can't be validation-related, so a
> length limitation would most certainly be a matter of validation.
> >

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 1 Sep 2019 16:24:24 -0700

How is such a thing supposed to work portably? What is the "length" of
a JSON string, and does this have any correspondence to the memory
representation of that string, post-parsing?

On Sun, Sep 1, 2019 at 4:14 PM John Cowan <[email-masked]> wrote:
>
> The argument for it is precisely in situations where static allocation is required along with static typing.  If you know the (maximum) length, you can safely allocate a fixed amount of memory for your internal representation, which you cannot do otherwise.   Languages with static types and garbage collection (allowing dynamic allocation) are just one particular kind of languages, and I think you need to accommodate all widely used kinds of languages.  I wouldn't object to skipping bounded arrays and strings in favor of just fixed-length ones.
>
> See < http://www.catb.org/esr/microjson/>, a JSON-subset parser written in C that imposes exactly this restriction so that it can run safely on embedded systems with only small amounts of RAM.
>
>

---

From: John Cowan <[email-masked]>
Date: Sun, 1 Sep 2019 19:38:33 -0400

I agree that that is somewhat arbitrary, but I'd say the length of a string
is its length in bytes in UTF-8 encoding, given that UTF-8 is the standard
encoding of JSON.

On Sun, Sep 1, 2019 at 7:24 PM Ulysse Carion <[email-masked]> wrote:

> > The argument for it is precisely in situations where static allocation
> is required along with static typing.  If you know the (maximum) length,
> you can safely allocate a fixed amount of memory for your internal
> > See < http://www.catb.org/esr/microjson/>, a JSON-subset parser written
> in C that imposes exactly this restriction so that it can run safely on

---

From: Ulysse Carion <[email-masked]>
Date: Sun, 1 Sep 2019 16:47:32 -0700

This seems like another example of a call I'd rather not have to make,
nor insert into the core spec.

Again -- nothing stops folks who have specific limitations to add
specific extensions. And just because something isn't validated in
JDDF itself doesn't mean it can't be validated at all.

On Sun, Sep 1, 2019 at 4:38 PM John Cowan <[email-masked]> wrote:
>
> I agree that that is somewhat arbitrary, but I'd say the length of a string is its length in bytes in UTF-8 encoding, given that UTF-8 is the standard encoding of JSON.
>

---

From: "Manger, James" <[email-masked]>
Date: Mon, 2 Sep 2019 01:43:46 +0000

>This isn't quite right. Yes -- Java only has "byte", but by convention you can store both signed and unsigned 8-bit numbers in these bytes.
>Protobufs does this:

That is certainly 1 reasonable choice for a code generator.

>Furthermore, the Java standard library has methods that explicitly let you do the conversion of a Java byte-as-uint8 into a Java int or long:
>
>If bytes-as-uint8 weren't conventional, then why does java.lang.Byte have methods that enable such a style?

Did you notice the "Since 1.8" annotation on those methods? This support for byte-as-uint8 was only added in Java 8.
And even these additions don't make Java's standard sorting or printing work for uint8. Generated code cannot merely using byte for uint8; it also needs to add code so a byte-as-uint8 is serialized differently to a normal byte-as-int8.

Short-as-uint8 has different properties: sorting & serializing works as "normal", but it takes twice as much memory.

If I saw uint8 in JDDF, I'd almost certainly want control over whether it became a byte or short in my Java code, depending on the situation.

>> * "type": "intString" ...
>> * "type": "uintB64u" ...
>> * "type": "bytesB64" ...
>> * "type": "bytesB64u" ...

> I think I'm opposed to this suggestion. Adding support for arbitrary binary data is going to be challenging. I think I prefer that we just stick to {"type":"string"}, and then folks can write their own extensions to JDDF to achieve their specific needs. After awhile, the community may perhaps converge on common extensions, or patterns may arise and we can update the JDDF spec.
>
>Anything we add to the spec now would have to be supported by all implementations. So I'd prefer we give out a minimal spec to the world, and see where the real world takes it. If we try to solve all problems right away, we may make it too hard for implementations to crop up, ultimately solving no problems for anyone.

But if the spec is too minimal it isn't worthwhile.

--
James Manger

---

From: Ulysse Carion <[email-masked]>
Date: Mon, 2 Sep 2019 19:02:09 -0700

> And even these additions don't make Java's standard sorting or printing work for uint8. Generated code cannot merely using byte for uint8; it also needs to add code so a byte-as-uint8 is serialized…

It's true that the standard sorting and printing stuff doesn't work
as-is. It's untrue that one cannot merely use byte for uint8, or int
for uint32. gRPC does this. It's quite common to use []byte in a
context where the byte is known, beyond the type system, to represent
unsigned numbers.

Serializing an int back to JSON is a stronger point. But code
generation can deal with this by overriding the relevant
fromJSON/toJSON or equivalent. Both gson and Jackson have such
functionality. This is unpleasant, but handled entirely by the code
generator. But from the user's perspective they have a POJO they can
immediately use, and which behaves correctly with respect to SerDe.
That's what matters most for a code generating tool, in my view.

Having to write special serialization logic happens already for the
discriminator form, and for many languages the timestamp data type.
But the important thing is that the codegen'd code can contain this
nastiness, and automate away having to write such code.

> If I saw uint8 in JDDF, I'd almost certainly want control over whether it became a byte or short in my Java code, depending on the situation.

This is a very sensible position. Sounds like jddf-codegen ought to
support such an option, and equivalents for uint16 and uint32.

> But if the spec is too minimal it isn't worthwhile.

Yes, but as it is I don't think the spec meets this "too minimal" threshold.