Subject: [apps-discuss] Standardizing protocol buffers

From: "Rex Fernando (rex)" <[email-masked]>
Date: Thu, 11 Oct 2012 17:21:40 +0000

Hi,

We have posted a new draft to standardize the protocol buffers data serialization format and to request a MIME type for it. The draft is located here:
http://www.ietf.org/id/draft-rfernando-protocol-buffers-00.txt

Given the charter of this group, it seems like a good place to process this draft. 

Please read and provide comments.

Regards,
- Rex

---

From: Martin Thomson <[email-masked]>
Date: Thu, 11 Oct 2012 10:38:01 -0700

> http://www.ietf.org/id/draft-rfernando-protocol-buffers-00.txt

Can you explain why this is superior to ASN.1 and PER?

This is not explained in the draft, or on the website
(https://developers.google.com/protocol-buffers/docs/overview).  Which
is where I ended up when I was trying to understand how this encoding
actually worked.

s/Proposed Standard/Informational/

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 11 Oct 2012 11:41:01 -0600

On 10/11/12 11:38 AM, Martin Thomson wrote:
>> We have posted a new draft to standardize the protocol buffers
>> data serialization format and to request a MIME type for it. The
>> draft is located here: 

Does it need to be superior in order to be registered?

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: James M Snell <[email-masked]>
Date: Thu, 11 Oct 2012 10:49:07 -0700

I'll second Martin's suggestion that this should be Information rather than
Proposed Standard.

Also, I noticed that there is absolutely zero mention of IPR/Licensing
either in the draft or on the referenced Google-hosted website. There is
zero indication as to the license under which Protocol buffers can be used.
As an individual implementor I would have no idea if this can be safely
implemented and used.

- James

On Thu, Oct 11, 2012 at 10:21 AM, Rex Fernando (rex) <[email-masked]> wrote:

>
> Hi,
>
> We have posted a new draft to standardize the protocol buffers data
> serialization format and to request a MIME type for it. The draft is
> located here:
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Barry Leiba <[email-masked]>
Date: Thu, 11 Oct 2012 14:17:54 -0400

> zero indication as to the license under which Protocol buffers can be used.

Well, I'll note that the draft says this:

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

That means that the authors are asserting that any IPR they are aware
of has been disclosed, according to the requirements of BCP 79.

The authors can (should), of course, confirm this....

Barry

---

From: James M Snell <[email-masked]>
Date: Thu, 11 Oct 2012 11:22:35 -0700

> The authors can (should), of course, confirm this....
+1... I don't anticipate that there is any issue here, I just get nervous
when seeing something from a specific vendor that has no specific license
mentioned on the original spec. :-)

- James

---

From: Martin Thomson <[email-masked]>
Date: Thu, 11 Oct 2012 11:27:29 -0700

> Does it need to be superior in order to be registered?

Oh no, I was just interested, that's all.  The website explains why
this is better than XML for a certain class of application.

If I were naive, I'd assume that something that requires this much
effort to design and implement would have to have some sort of reason
as justification.

---

From: James M Snell <[email-masked]>
Date: Thu, 11 Oct 2012 11:40:04 -0700

Yep! That definitely helps. Basically, it just needs to be clear that any
one is free to implement the protobufs without any doubts... a statement
buried in the FAQ works, but you could make the statement a bit more
directly on the landing page.. just a suggestion. :-)

- James

On Thu, Oct 11, 2012 at 11:33 AM, Stephen Stuart <[email-masked]> wrote:

> The FAQ notes that we haven't filed any patents on protocol buffers:
>
>     https://developers.google.com/protocol-buffers/docs/faq
>
> We published an implementation here,
> http://code.google.com/p/protobuf/, under the "Nw BSD license"
> (http://opensource.org/licenses/BSD-3-Clause).
>
> Does that help?
>
> Stephen
>
> On Thu, Oct 11, 2012 at 10:49 AM, James M Snell <[email-masked]> wrote:
> located
> >> here:

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 11 Oct 2012 22:26:33 +0200

> http://www.ietf.org/id/draft-rfernando-protocol-buffers-00.txt

Hey, I can do that, too.

http://www.ietf.org/id/draft-bormann-apparea-bpack-00.txt

How many of these do we need?

(I'm actually quite serious with binarypack, as there is a lot of stuff out there that uses msgpack, and it probably would be a good idea to do a version with IETF change control.  Area of application for me is mostly constrained node/network REST, but of course it is a quite universal hammer.  Not the impact wrench that protocol buffers is :-)

Grüße, Carsten

---

From: "Manger, James H" <[email-masked]>
Date: Fri, 12 Oct 2012 12:49:24 +1100

Carsten,

Binarypack [draft-bormann-apparea-bpack] looks fairly simple and efficient.

One limitation is that you have to know the length of a string, the number of items in an array, and the number of key/value pairs in a table (aka object or map) before you start encoding them.

JSON does not have this limitation.

This is likely to make binarypack unsuitable for really large values, for values created in a streaming mode, or for tools that want to filter a stream as it passes.

One solution would be to allow values of these types to be sent in pieces. A new "partial" representation for these types would be defined (using the same syntax as the representation that uses a 4-byte length field). Any number of those can be concatenated to convey a full value (terminated with a non-partial representation of the same type, or defining a special terminator).

So "Hello, World!" could be encoded as:
  D9 0000000D 48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21
Or (picking D7 to indicate a "partial" string)
  D7 00000005 48 65 6C 6C 6F
  D7 00000008 2C 20 57  6F 72 6C 64 21
  B0

P.S. I believe protocol buffers have the same scalability limitation.

P.S. ASN.1 BER supports indefinite-length encodings to overcome this limitation.

P.S. Being able to encode parts of a larger field is also useful for occasional manual debugging as you can modify a value (add/remove bytes, characters, array items, or key/value pairs) without having to recalculate nested length fields all the way up to the root of the message.

--
James Manger

---

From: Stephen Stuart <[email-masked]>
Date: Thu, 11 Oct 2012 11:33:58 -0700

Does that help?

Stephen

On Thu, Oct 11, 2012 at 10:49 AM, James M Snell <[email-masked]> wrote:
>> serialization format and to request a MIME type for it. The draft is located
>> here:

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 12 Oct 2012 15:13:06 -0400

+1 and how does all this relate to JSON?

Also we should note that we have another very similar scheme inside TLS.

There are good reasons to prefer binary encoding over JSON and vice versa.
But it would be rather nice if we had a way to abstract the encoding choice
from the protocol.

For example, JSON does not support binary types which makes it rather
inefficient as a wrapper for video streams or the like. Another problem
with JSON is that it uses an escaped encoding for strings and those raise
security issues and performance issues. [And no, telling people not to make
mistakes is not a security solution]

There are also advantages to JSON of course but which one is best is going
to depend on the target application. So I think that is a choice that
should be made in deployment rather than by the protocol designer.

I think we should try to minimize the number of packing options. Most
decisions do not matter at all, some matter a lot but there is a very clear
choice (don't repeat DER) and some depend on circumstances.

As far as the details go, there are only a few useful design choices in a
packing scheme. I am certain this proposal has some of them wrong.

1) Integer encoding

The encoding of negative integers needs to be much better thought out. As
should the fact that an integer might well be larger than 64 bits.

The Varint scheme uses the top bit of each integer to encode the run
length. This means that if we AND each octet of an integer with 0x80 we
will get one of the following

0x00
0x80 0x00
0x80 0x80 0x00
...

The problem with this scheme is that it requires each integer to be packed
into 7 bit form which is kind of inefficient. Lots of rotates and so on. A
more time efficient scheme is to follow the ASN1 approach and pack the run
length into the first byte in some fashion. So if the top byte is:

0xxx xxxx length = 1
10xx xxxx length = 2
110x xxxx length = 3
1110 xxxx length = 4
1111 nnnn length = 4 + nnnn

One of the big advantages of this approach is that it avoids the need for
the 7 bit shifting which is not only inefficient, it requires special case
fixup for negative integers. A twos compliment number can be fitted in very
easily.

2) Tagged or untagged data

JSON and XML data is tagged which means that it is very flexible. Adding an
extra data field does not cause chaos. It was not immediately apparent to
me which approach this proposal takes.

ASN.1 data gives a choice on a per field basis. This does not seem to be a
helpful approach (to say the least).

3) Typed or untyped

This is not the same as tagged or untagged. A data structure {"a" : 1} has
different semantics from {
"b" : 1}. The tags may imply a type but they are not themselves a type
declaration.

JSON data is implicitly typed by the syntax. ASN.1 is not.

Any conversion from a binary format to JSON needs to have the type
information. That may come from either a schema or be explicit in the data
stream as in BER encoding.

4) Can it be implemented as a linear traversal?

The really insane part of ASN.1 is actually two lines buried in the X.500
documentation defining DER. According to the DER encoding rules, an object
is represented using the definite length encoding. So each object is
prefixed by the total length of the data fields.

This is a problem because the included data fields may also be objects and
the length field is encoded using variable length encoding. So emitting DER
encoding requires a totally pointless double recursion down the tree and
back again, keeping track of multiple values. It is just stupid, stupid,
stupid.

Oh and there is absolutely no security reason for DER encoding, none, zip,
nada.

Again, the draft seems pretty vague on this point as well.

On Thu, Oct 11, 2012 at 4:26 PM, Carsten Bormann <[email-masked]> wrote:

> (I'm actually quite serious with binarypack, as there is a lot of stuff
> out there that uses msgpack, and it probably would be a good idea to do a
> version with IETF change control.  Area of application for me is mostly
> constrained node/network REST, but of course it is a quite universal
> hammer.  Not the impact wrench that protocol buffers is :-)
>
> Grüße, Carsten
>

-- 
Website: http://hallambaker.com/

---

From: Martin Thomson <[email-masked]>
Date: Fri, 12 Oct 2012 13:07:46 -0700

> +1 and how does all this relate to JSON?

Maybe the authors of the draft under discussion can comment on what
their intent is.  Are they intending to bring work for the IETF to
work in (as would be hinted at by the Proposed Standard status), or is
this an attempt to provide a stable informational reference for
something that is widely deployed.

I assumed it was the latter.  In that sense, a more relevant question
to debate is whether this information, if published as an RFC, would
improve the Internet in any meaningful way.

If this were the former, then I'd be more inclined to engage in some
sort of debate about the merits of ASN.1 and the various encoding
rules as they relate to the protocol buffers design.  Carsten's draft
is clearly in this camp.

--Martin

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 12 Oct 2012 22:11:07 +0200

On Oct 12, 2012, at 03:49, "Manger, James H" <[email-masked]> wrote:

> indefinite-length

I have too many scars from BER to be too happy about indefinite-length encoding.
I do see the advantages that you mention, for certain applications.
They don't really apply to my constrained node use case.
But they are needed to enable completely different ones ("streaming").

My main concern really is that many representation formats have too many ways to say the same thing, which leads to a number of issues (interop testing becomes harder, this is a covert channel, etc.).  msgpack of course already has three ways to express the same data for things with 0..15 elements, so maybe this already is a lost cause.

One interesting alternative that I was looking at before deciding to go for msgpack was bencode, which has exactly one encoding for a given data object (i.e., it is a DER, actually more like CER).
For its structured types (and integers!), this entirely relies on push-pop style delimiters (but then it doesn't have chunking for strings, which would destroy the DERness).
In the end, I was quite happy with the definiteness of msgpack -- works well with constrained nodes.
It seems a lot of msgpack users are happy with this, too (their main application so far appears to be RPC marshaling, which also works well with definite length).
Adding a "chunking" code for "concatenate the next few data elements until you get a zero length one" is a very small extension in the number of words of the spec, but a big one in the number of lines of code.
(One could also add a code for "concatenate this to the previous data element", but this would lose the self-delimiting property of msgpack.)

Another way of handling this requirement is making the chunking code optional.
I.e., essentially splitting the format into two variants (as there is no other way to handle this optionality).
If it is possible to carve up the use cases cleanly into two groups, this might even make sense.

In summary: good point, but no obvious resolution springs to mind.

(The other big user requirement neither JSON nor msgpack address is internal referencing, i.e. deviations from the pure tree model.
I believe JSON was right in not addressing this, and I don't really want to go there.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 12 Oct 2012 22:29:02 +0200

On Oct 12, 2012, at 22:07, Martin Thomson <[email-masked]> wrote:

> Carsten's draft

I'm not quite sure in which camp I am.

msgpack is widely deployed.
binarypack is an interesting small tweak on this (that I didn't even invent, but that happens to solve my specific problem with msgpack).

I think is it highly appropriate for the IETF to take widely deployed, relevant protocols, and add some value to them:

1) creating an unambiguous reference

2) picking up and solving specific requirements that the IETF is interested in, while maintaining the value of the base spec.

(For a [much larger and much more valuable] example, compare this to SSL and TLS, if you want.)

Of course, I'm happy going the JSON way here (JSON is an informational RFC that is widely used as a normative reference) :-)

As regards the protocol buffers spec: 
This is really a Google thing, so I'd expect this to become another informational document titled "Google's ...".
Maybe my view is too narrow.
(In case this isn't clear: I believe protocol buffers is definitely worth documenting.  Certainly more than, say, RFC 3072.)

Grüße, Carsten

---

From: "Rex Fernando (rex)" <[email-masked]>
Date: Fri, 12 Oct 2012 23:50:19 +0000

Martin, Carsten,

| -----Original Message-----
| From: [email-masked] [mailto:apps-discuss-
| [email-masked]] On Behalf Of Carsten Bormann
| Sent: Friday, October 12, 2012 1:29 PM
| To: Martin Thomson
| Cc: [email-masked]; [email-masked]
| Subject: Re: [apps-discuss] Standardizing protocol buffers
| 
| On Oct 12, 2012, at 22:07, Martin Thomson <[email-masked]>
| wrote:
| 
| > Carsten's draft
| > is clearly in this camp.
| 
| I'm not quite sure in which camp I am.
| 
| msgpack is widely deployed.
| binarypack is an interesting small tweak on this (that I didn't even invent, but
| that happens to solve my specific problem with msgpack).
| 
| I think is it highly appropriate for the IETF to take widely deployed, relevant
| protocols, and add some value to them:
| 
| 1) creating an unambiguous reference
| 
| 2) picking up and solving specific requirements that the IETF is interested in,
| while maintaining the value of the base spec.
| 
| (For a [much larger and much more valuable] example, compare this to SSL
| and TLS, if you want.)
| 
| Of course, I'm happy going the JSON way here (JSON is an informational RFC
| that is widely used as a normative reference) :-)
| 
| As regards the protocol buffers spec:
| This is really a Google thing, so I'd expect this to become another
| informational document titled "Google's ...".
| Maybe my view is too narrow.
| (In case this isn't clear: I believe protocol buffers is definitely worth
| documenting.  Certainly more than, say, RFC 3072.)

Agree with the points being made.

Our intention is to create an unambiguous and stable reference for implementers of protocol buffers.  The intention is not to supplant existing encoding formats.

We'll fix the 'Intended Status' to be  'Informational' if that helps clear things a bit.

- Rex
 
| Grüße, Carsten
| 
| _______________________________________________
| apps-discuss mailing list
| [email-masked]
| https://www.ietf.org/mailman/listinfo/apps-discuss