Subject: [quicwg/base-drafts] Don't use bitmap frames to describe varint structures (#3115)

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 17 Oct 2019 12:56:39 -0700

Many of the QUIC drafts use fixed 32bit maps to "illustrate" the layout of various parts of a message. In some cases, that is useful up to the point where the frame bits end. In other cases, the illustration consists of a list of arbitrary varint values, for which the 32bit illustration is both horribly misleading and a waste of space (and reader's time). Please stop.

I prefer C-style data structures for descriptive ordering, but YMMV. The simplest alternative is to just list the variable names (and types) in order. The only place where I would like to see a bitmap presentation would be an actual example, which shows the variable ints in the actual positions that the example values would cause them to be placed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115

---

From: David Schinazi <[email-masked]>
Date: Sat, 19 Oct 2019 13:56:25 -0700

I disagree, I personally quite like the current syntax.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-544196651

---

From: ianswett <[email-masked]>
Date: Sat, 19 Oct 2019 14:32:10 -0700

I also prefer the current syntax because I find it easier to understand how everything packs together.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-544199011

---

From: Mike Bishop <[email-masked]>
Date: Wed, 23 Oct 2019 11:22:49 -0700

It's a fair point that when our structures are no longer 32-bit-aligned, the graphics are somewhat misleading.  However, for many of the frames, they're still more useful as a mental model than a C struct style, at least for many readers.  We've adopted conventions to indicate variable-sized fields; if you see places those aren't used properly, please flag them.

We previously had a principle that we don't illustrate frames/objects that contain a single varint.  The "inconsistency" resulted in them being added.  Ultimately, this is a question of editorial style and input on what readers find useful.  Your feedback is welcome and will be considered.

(I'll also note that this seems to oppose the discussion about moving the transport params from TLS-style description language to wire-frames.)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-545574047

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 25 Oct 2019 13:08:55 -0700

Honestly, I think people are responding based on "I don't want to change my imagination of what the diagrams show" versus any sensible understanding of the examples in use:

https://quicwg.org/base-drafts/draft-ietf-quic-http.html#rfc.section.6.2

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Stream Type (i)                      ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

```
https://quicwg.org/base-drafts/draft-ietf-quic-http.html#rfc.section.6.2.2

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           0x01 (i)                          ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Push ID (i)                        ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

```
https://quicwg.org/base-drafts/draft-ietf-quic-http.html#rfc.section.7.2

```
 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Type (i)                          ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Length (i)                         ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Frame Payload (*)                     ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

There is absolutely no value to these diagrams as a "mental model", in any way shape or form. They are simply misleading because they look like they contain information when they actually contain a list of ordered names of unknown length. The prose already contains that.

These are not the only ones -- there are eight more useless diagrams in that draft alone. There are some good diagrams of the same thing in other QUIC drafts, basically the ones containing bitfields, but even those lose sensibility as soon as a varint appears.

They would ALL be fine if they illustrated actual examples values and how their 0/1 bits were laid out. But they don't. They don't even come close in that middle example above.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-546494948

---

From: ekr <[email-masked]>
Date: Sun, 27 Oct 2019 16:04:58 -0700

I agree with Roy. I see how we got here, but I think at this point some kind of specification language (TLS or otherwise) would be far better.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-546744331

---

From: Lucas Pardue <[email-masked]>
Date: Sun, 27 Oct 2019 17:05:46 -0700

As the person that created many of the HTTP/3 frame diagrams I apologise for raising your ire :)

As someone that mainly came into IETF via HTTP/2, my main motivation for diagramming the frames in HTTP/3 was for consistency when switching back and forth between drafts. For example, single variable frames are diagrammed in RFC 7540 so it did seem inconsistent to skip them (to @MikeBishop's point). I'd experienced several occasions (myself and others) where the lack of a diagram had led to some oversight when implementing the specification - even if that is not their intended purpose because they are duplicatory of text.

That said, I do see the point you are making. For instance, I opened https://github.com/quicwg/base-drafts/issues/2338 to point out the problem with the diagram in section 6.2.2. I think the diagram is a problem but Mike closed it as no-one seemed to care at the time.

However, I'd like to highlight that RFC 7540 has, in part, the problem you are describing where fields of unknown length are represented by the `*` character.

https://tools.ietf.org/html/rfc7540#section-6.2
```
    +---------------+
    |Pad Length? (8)|
    +-+-------------+-----------------------------------------------+
    |E|                 Stream Dependency? (31)                     |
    +-+-------------+-----------------------------------------------+
    |  Weight? (8)  |
    +-+-------------+-----------------------------------------------+
    |                   Header Block Fragment (*)                 ...
    +---------------------------------------------------------------+
    |                           Padding (*)                       ...
    +---------------------------------------------------------------+

                      Figure 7: HEADERS Frame Payload
```

Is your complaint about the QUIC specs caused by the top line of each diagram that show the 32-bit space, or do you think we got it wrong with H2 also?

FWIW I think there could be merit in defining a more structured syntax for Frames and Streams - especially as we look forward to new versions and extensions of QUIC because we should make it pretty straightforward for new contributors to pick up and go. However, I see a change at this stage being a bunch of editorial work.
2) 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-546749401

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 27 Oct 2019 17:35:52 -0700

I think I agree with @LPardue that for HTTP/3 it is beneficial to use same notation as in HTTP/2. I'd assume that we would be seeing extensions that cover both HTTP/2 and /3. It would be confusing to the readers of those specifications to understand two different notations.

I do not have a strong opinion on the transport draft.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-546752173

---

From: Nick Harper <[email-masked]>
Date: Mon, 28 Oct 2019 11:25:07 -0700

My preference is something closer to the TLS presentation language for specifying packet formats (at least in the transport draft).

For example, the initial packet header format is currently described as 

```
+-+-+-+-+-+-+-+-+
|1|1| 0 |R R|P P|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Version (32)                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| DCID Len (8)  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|               Destination Connection ID (0..160)            ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| SCID Len (8)  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Source Connection ID (0..160)               ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Token Length (i)                    ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Token (*)                        ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Length (i)                        ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Packet Number (8/16/24/32)               ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                          Payload (*)                        ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
```

Only one of fields drawn to be 32 bits wide is actually 32 bits (the version) - for the rest of them that drawing style (even though there's a `...` instead of a `|` at the end of the line) isn't particularly helpful.

Here's an example of what that would look like done in something like the TLS presentation language:

```
struct {
  uint8 flags;
  uint32 version;
  uint8 dcid_len;
  blob[dcid_len] dcid;
  uint8 scid_len;
  blob[scid_len] scid;
  varint token_len;
  blob[token_len] token;
  varint length;
  blob[pn_len] packet_number;
  blob[length-pn_len] payload;
} InitialPacket;
```

The only place where this struct loses clarity compared to the diagram is the `flags` field (and regardless of whether a struct or a diagram is used, prose is still needed to describe what the bits are). For the rest of this struct layout, it's a list of fields, which is essentially what the diagram is, but the types and/or lengths of the fields are described in better detail.

If there's interest in making such a change to the drafts, I volunteer to help make those changes.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547082729

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 28 Oct 2019 11:35:23 -0700

But blob is in reality strongly typed - for example a packet contains a STREAM frame that contains a set of H3 frames (and each frame has the same frame header). what is an effective way of describing the data encapsulated in such structures?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547086841

---

From: Nick Harper <[email-masked]>
Date: Mon, 28 Oct 2019 14:31:41 -0700

Continuing from my example above, instead of `blob[length-pn_len] payload;`, perhaps that could be `Packet[length-pn_len] payload;`. (Now clarifying some of the syntax I'm making up or copying from TLS: `Type[len] name;` means that the field named `name` has `len` bytes and is of type `Type`.)

Then a StreamFrame might be defined as something like the following:

```
struct {
  varint stream_id;
  optional varint offset;
  optional varint length;
  blob[length] data;
} StreamFrame : Frame;

struct {
  varint frame_type;
  Frame frame;
} GenericFrame;

struct Packet {
  repeated GenericFrame frames;
};
```

Note that I'm basically making things up right now, although this is an interesting exercise in what features a description language would need for QUIC packet format.

I think the transport draft would need to stop at having a `blob[length] data` as the contents of a StreamFrame, and the application (H3) could pick up writing similar definitions.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547155059

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 28 Oct 2019 14:37:43 -0700

Yeah sorry I should have recognised you were providing a simple example not the whole shebang. I will point out that this format begins to resemble @rmarx qlog definition in TypeScript notation. If we really did want to change things, he'd probably be interested in aligning them if possible.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547157123

---

From: Martin Thomson <[email-masked]>
Date: Mon, 28 Oct 2019 21:07:57 -0700

I'd just like to point out that the only sane approach here is to use ASN.1 with PER so that we can get bit-level alignment.  I believe that with care it shouldn't matter whether we pick the aligned or unaligned variant.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547247427

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 28 Oct 2019 21:17:48 -0700

In addition to my previous point that the notations of HTTP/2 and HTTP/3 should better be aligned, I'm inclined to argue that using three different notations in three related drafts (QUIC, TLS, HTTP/3) would be a nightmare.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547249150

---

From: Marten Seemann <[email-masked]>
Date: Mon, 28 Oct 2019 22:23:49 -0700

#3108 removes the TLS presentation language, so at least we're down to two different notations.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547261505

---

From: Robin Marx <[email-masked]>
Date: Tue, 29 Oct 2019 06:17:06 -0700

> Yeah sorry I should have recognised you were providing a simple example not the whole shebang. I will point out that this format begins to resemble @rmarx qlog definition in TypeScript notation. If…

For what it's worth, @nharper's example would look a bit like this in TypeScript notation:

```
class StreamFrame implements Frame {
 stream_id:varint;
 offset?:varint;
 length?:varint;
 data:Array<byte> = new Array<byte>(length);
}

interface GenericFrame {
 frame_type:varint,
 frame:Frame
}

interface Packet {
 frames:Array<GenericFrame>
}
```

I've been using this in qlog mainly for convenience but am willing to move to another format down the line if qlog actually gets adopted and this type of flexible format is used in other QUIC drafts. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3115#issuecomment-547414052