Subject: Re: [dispatch] Dispatching WebTransport

From: Peter Thatcher <[email-masked]>
Date: Mon, 1 Jul 2019 17:10:27 -0700

On Mon, Jun 17, 2019 at 7:13 AM T H Panton <[email-masked]> wrote:

>
>
> On 17 Jun 2019, at 14:35, Victor Vasiliev <
> vasilvv=[email-masked]> wrote:
>
> Hello friendly IETF dispatchers,
>
> I am writing about new work I want to bring to IETF.  The proposal is
> called WebTransport.  It’s a combination of a Web API currently under
> development in W3C WICG [0], a protocol framework and some protocols that
> fit into that framework.  Combined, they would allow web applications to
> establish WebSocket-like connections that instead of ordered reliable
> messages use multiple streams and datagrams (datagrams are unreliable and
> streams do not have head-of-line blocking).  This is highly useful for
> real-time and other latency sensitive applications.
>
>
> Establish connections to where ? Between eachother P2P ? Or to any server?
> Or just to the originating server only?
>

Any server or p2p.

>
>
> # Background
>
> Historically, the only networking operations available to the Web
> applications were sending HTTP requests and receiving HTTP responses.  That
> model does not fit all applications well, so over time, more mechanisms
> were added.  The two most relevant here are WebSockets (RFC 6455) and RTC
> Data Channels (draft-ietf-rtcweb-data-channel).  WebSockets are a way for
> Web applications to do bidirectional communication over a TCP connection;
> they work great if TCP fits your transport needs, but perform poorly if
> your application is latency sensitive and would, in non-Web context, use a
> UDP-based protocol.  There are many different kinds of applications like
> that, but I would like to highlight two major categories which I to some
> extent surveyed when coming up with this proposal:
>
>    1. Custom client-server chat/multimedia protocols (faster-than-DASH
>    video streaming, game streaming, etc).  Those are usually developed by
>    teams with a good amount of resources, and they are interested in tailoring
>    the setup for their use case.
>    2. Game developers.  Online games are commonly real-time in nature and
>    benefit dramatically from ability to give up on transmitting old
>    information.  They usually use some in-house UDP-based protocol, and often
>    need to run on unusual platforms.
>
> WebRTC Data Channels are a mechanism that provides a WebSocket-like
> interface with unreliable delivery features.  On the wire, it’s
> SCTP-over-DTLS, established using ICE and SDP.  In theory, this provides
> users with enough functionality to build anything they need, since SCTP
> messages can be unreliable and unordered.  In practice, while
> RtcDataChannel is fairly straightforward to use for browser-to-browser
> peer-to-peer communication, it has seen much lower adoption than WebSockets
> in the client-server scenario, even considering the fact that its use cases
> is naturally more niche.
>
> The main reason for this is the incredible complexity of the WebRTC
> stack.  WebSockets are a fairly straightforward overlay on top of TCP and
> TLS; there is a wide variety of implementations out there, and it's
> fairly easy to write a new one (I wrote on myself in less than 1,000 lines
> of C++).  With data channels, however, once there is no browser to abstract
> all of the complexity away, the web developers are required to understand
> and implement (or at least integrate) SDP, ICE, STUN, DTLS and userspace
> SCTP.  While a lot of those have simplifications for this use case (ICE
> Lite) and some protocols listed have a variety of implementations widely
> available (DTLS), the entire system still requires going through hundreds
> of pages of RFCs in order to understand it well enough to implement.  This
> complexity barrier has precluded Data Channel adoption by communities of
> smaller developers who don’t have resources to implement them, notably game
> developers (see [1] and [2] for some discussion).
>
>
> Much, but not all, of the complexity of the webRTC stack is due to the NAT
> traversal and security commitments you need to do P2P.
>
If you exclude the need to do NAT traversal and P2P
> then it _could_ be much simpler. But it also gives you E2E encryption and
> low latency as side effects!
>
>
 In client/server mode, WebTransport doesn't require any of that, so it is
a lot more simple.

Work is in progress decoupling the APIs from the (dreaded) SDP and
> simplifying the developer experience.
>
>
What work?

> That said, it is by no means un-managable - I've implemented pretty much
> the whole stack from scratch for small devices and I know of 2 other
> implementations which don't use libusrsctp
>
>
Maybe not for you, but we regularly get emails from people saying things
like "It would be great to delete the mess of the WebRTC stack if there's a
simpler protocol available. ICE/DTLS/SCTP all scare me".

>
>
> Even among the people who got past the complexity barrier, the feedback I
> heard almost universally is that WebRTC Data Channels are hard to work
> with.  From the feedback I gathered, the main problem is usually around the
> transport protocol itself.  Userspace SCTP is essentially a monoculture:
> virtually all implementations use libusrsctp, a 80,000-line adaptation of
> FreeBSD SCTP implementation.  This lack of tool choice is fairly painful
> since latency-sensitive real-time applications often require quite a bit of
> tuning on the transport side to get the best performance (custom congestion
> control, etc).  In addition, the limitations on the message size stemming
> from both the API itself and the lack of widespread support for message
> interleaving (RFC 8260) means that the developers have to roll their own
> framing on top of SCTP messages if they want to avoid head-of-line-blocking
> (this is particularly bad because the framing overhead in data channels is
> already large as-is).
>
>
> SCTP is well defined nice protocol that suffers (IMHO) from a lack of
> 'coolness' due to being used by the telcos for most of it's life.
> The head-of-line issue is a problem, but entirely fixable with small
> tweaks to SCTP protocol. (read it wasn't in the original requirements - so
> it doesn't do it).
>

It's been fixable for a very long time.  I have my doubts about when it
will actually get fixed.

>
>
> In summary, we have a system that technically provides what everyone
> wants, but that nobody is happy with, and that is not usable by all but the
> most well-resourced users.
>
>
> I'd point out that 2.5bn endpoints support it right now - including
> _every_ smartphone shipped in the last 2 years. Last I heard DataChannel
> traffic was 0.1% of chrome's traffic
> which is _astonishingly_ high
>

>
> # Proposal
>
> Our initial idea for fixing this was to take QUIC and do what WebSocket
> did to TCP: add security features that would make it safe to expose on the
> Web (by adding origin checks, etc), but otherwise expose it as-is.  This
> would get us out of libusrsctp monoculture (QUIC is not yet finished, but
> it already has a fairly diverse implementation ecosystem, see [3]), and
> remove all P2P-related complexity involving SDP and ICE.  The original
> proposal for that was called QuicTransport; we showed it to various people,
> and the feedback we got is that (1) the API should not be tied to a
> particular transport (since we already switched once from SCTP to QUIC,
> tying it to QUIC specificially would not be wise), and (2) it shouldn’t
> fail hard when QUIC is unavailable.
>
>
> It would be _great_ if that abstraction could _also_ work over SCTP data
> channels. That way you could cover the P2P use cases too.
>

QUIC can be p2p.  In fact, we already have it implemented in Chrome (in an
original trial).

The WebTransport API could run on top of SCTP, and you could probably make
a JS polyfill for it.  But I'm not sure why anyone would bother.

> As a result of that feedback, we abstracted it into a general-purpose
> framework called WebTransport.  The overview draft,
>
>   https://tools.ietf.org/html/draft-vvv-webtransport-overview-00
>
> describes the framework itself, mainly the requirements the transport
> protocols have to satisfy to be usable on the web through the API.  Within
> this framework, we propose the following protocols:
>
>    - QuicTransport -- a simple WebSocket-like adaptation of QUIC,
>    described in https://tools.ietf.org/html/draft-vvv-webtransport-quic-00
>    - Http3Transport -- a mechanism that allows creating custom non-HTTP
>    streams within an HTTP/3 session, described in
>    https://tools.ietf.org/html/draft-vvv-webtransport-http3-00..  This is
>    sort of a version of RFC 8441 for QuicTransport.
>    - FallbackTransport -- a TCP-based transport with multiplexed streams
>    that can be used when QUIC is not available (e.g. on network that require
>    CONNECT proxy).  We don’t have a draft specifically for this, and there are
>    at least two approaches we could take here: either reusing HTTP/2 as a
>    transport (i.e. just use draft-kinnear-httpbis-http2-transport), or
>    building a protocol with QUIC-like semantics on top of WebSockest.  The
>    earlier is a more straightforward way; the latter has the advantage of
>    being fully polyfillable in JavaScript.
>
>
> # Discussion
>
> At this point, I am fairly certain that there is a problem here that needs
> to be addressed.  I am formally requesting ART area to take this problem on.
>
>
> I'd really like to be clear what that problem is.
>
>
Web developers routinely ask for an unreliable websocket, or something
close to UDP (but it needs to be secure).   It turns out that ICE/DTLS/SCTP
is too complex to fill that roll.

I believe the drafts above would be a good starting point for discussion.
> The design that they describe went through several iterations based on the
> feedback I got when I discussed this work within a more narrow audience
> (mostly people in QUIC working group), so we’re hopefully at least looking
> in the right direction here.  I am requesting feedback on this proposal,
> both on the overall plan and the specifics described in the drafts.  I hope
> to discuss this in depth in Montreal, both at dispatch and (in more depth)
> at a side-meeting.
>
>
> Tim.
>
>
> Thanks,
>   Victor.
>
> [0] https://github.com/WICG/web-transport
> [1] https://discourse.wicg.io/t/webtransport-proposal/3508/9
> [2] https://news.ycombinator.com/item?id=13266692
> [3] https://github.com/quicwg/base-drafts/wiki/Implementations
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch
>
>
> _______________________________________________
> dispatch mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dispatch
>

---

From: Peter Thatcher <[email-masked]>
Date: Mon, 1 Jul 2019 17:13:28 -0700

On Mon, Jun 17, 2019 at 7:52 AM Sergio Garcia Murillo <
[email-masked]> wrote:

> From the draft:
>
>    WebTransport [OVERVIEW <https://tools.ietf.org/html/draft-vvv-webtransport-quic-00#ref-OVERVIEW>] is a protocol framework that enables clients
>    constrained by the Web security model to communicate with a remote
>    server using a secure multiplexed transport.
>
>
> Although while I agree with your analysis regarding current webrtc dc and
> sctp status, if this is a pure client-to-server transport, why is webrtc
> mentioned in the background at all? shouldn't this be a replacement of
> websockets and NOT of the webrtc datachannels?
>

It's both a better WebSocket and a better RTCDataChannel.  Take your pick
of how you look at it.

> Best regards
> Sergio
>
>
> On 17/06/2019 15:35, Victor Vasiliev wrote:
>
>    https://tools.ietf.org/html/draft-vvv-webtransport-http3-00.  This is
> dispatch mailing [email-masked]://www.ietf.org/mailman/listinfo/dispatch
>
>

---

From: westhawk <[email-masked]>
Date: Tue, 2 Jul 2019 09:22:45 +0100

(Top post  - the quote level was getting unwieldy so I’ve trimmed quite a bit from the discussion - hopefully it still makes sense)

> On 2 Jul 2019, at 01:10, Peter Thatcher <[email-masked]> wrote:
> 
> 
> 
> On Mon, Jun 17, 2019 at 7:13 AM T H Panton <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
>> On 17 Jun 2019, at 14:35, Victor Vasiliev <vasilvv=[email-masked] <mailto:vasilvv=[email-masked]>> wrote:
>> 
>> I am writing about new work I want to bring to IETF.  The proposal is called WebTransport.  It’s a combination of a Web API currently under development in W3C WICG [0], a protocol framework and some protocols that fit into that framework.  Combined, they would allow web applications to establish WebSocket-like connections that instead of ordered reliable messages use multiple streams and datagrams (datagrams are unreliable and streams do not have head-of-line blocking).  This is highly useful for real-time and other latency sensitive applications.
> 
> Establish connections to where ? Between eachother P2P ? Or to any server? Or just to the originating server only? 
> 
> Any server or p2p.
>  

Ok, great, but the requirements are different, and the OP seemed to be claiming simplifications that to my mind only work
client - server.

> 
>  In client/server mode, WebTransport doesn't require any of that, so it is a lot more simple. 

Right. See above. But that becomes a null statement if you do want P2P support .

> 
> Work is in progress decoupling the APIs from the (dreaded) SDP and simplifying the developer experience.
> 
> 
> What work?   

Slow, slow, adoption of the ORTC interfaces. 

>  
> That said, it is by no means un-managable - I've implemented pretty much the whole stack from scratch for small devices and I know of 2 other implementations which don't use libusrsctp 
> 
> 
> Maybe not for you, but we regularly get emails from people saying things like "It would be great to delete the mess of the WebRTC stack if there's a simpler protocol available. ICE/DTLS/SCTP all scare me”.

Yeah, but not as much as ICE/QUIC-RT scares me. (which by the way isn’t documented _anywhere_).

>  
> 
>> 
>> Even among the people who got past the complexity barrier, the feedback I heard almost universally is that WebRTC Data Channels are hard to work with.  From the feedback I gathered, the main problem is usually around the transport protocol itself.  Userspace SCTP is essentially a monoculture: virtually all implementations use libusrsctp, a 80,000-line adaptation of FreeBSD SCTP implementation.  This lack of tool choice is fairly painful since latency-sensitive real-time applications often require quite a bit of tuning on the transport side to get the best performance (custom congestion control, etc).  In addition, the limitations on the message size stemming from both the API itself and the lack of widespread support for message interleaving (RFC 8260) means that the developers have to roll their own framing on top of SCTP messages if they want to avoid head-of-line-blocking (this is particularly bad because the framing overhead in data channels is already large as-is).
> 
> SCTP is well defined nice protocol that suffers (IMHO) from a lack of 'coolness' due to being used by the telcos for most of it's life.
> The head-of-line issue is a problem, but entirely fixable with small tweaks to SCTP protocol. (read it wasn't in the original requirements - so it doesn't do it).
> 
> It's been fixable for a very long time.  I have my doubts about when it will actually get fixed.

With that being the view from a browser team you can see why people choose not to donate time to submit patches that fix it.

>  
> 
>> 
>> In summary, we have a system that technically provides what everyone wants, but that nobody is happy with, and that is not usable by all but the most well-resourced users.
> 
> I'd point out that 2.5bn endpoints support it right now - including _every_ smartphone shipped in the last 2 years. Last I heard DataChannel traffic was 0.1% of chrome's traffic 
>> Our initial idea for fixing this was to take QUIC and do what WebSocket did to TCP: add security features that would make it safe to expose on the Web (by adding origin checks, etc), but otherwise expose it as-is.  This would get us out of libusrsctp monoculture (QUIC is not yet finished, but  it already has a fairly diverse implementation ecosystem, see [3]), and remove all P2P-related complexity involving SDP and ICE.  The original proposal for that was called QuicTransport; we showed it to various people, and the feedback we got is that (1) the API should not be tied to a particular transport (since we already switched once from SCTP to QUIC, tying it to QUIC specificially would not be wise), and (2) it shouldn’t fail hard when QUIC is unavailable.
>> 
> 
> It would be _great_ if that abstraction could _also_ work over SCTP data channels. That way you could cover the P2P use cases too.
> 
> QUIC can be p2p.  In fact, we already have it implemented in Chrome (in an original trial).

On top of ICE with a laughable security model no media and no RFC - not a good basis (yet) for others to implement.

> 
> The WebTransport API could run on top of SCTP, and you could probably make a JS polyfill for it.  But I'm not sure why anyone would bother.

2.5 billion shipped endpoints. near 100% penetration in smartphones today.

> Web developers routinely ask for an unreliable websocket, or something close to UDP (but it needs to be secure).   It turns out that ICE/DTLS/SCTP is too complex to fill that roll.  

I agree. If they want unreliable websockets to their servers, then we can cut out most of the layers and produce something based on QUIC
very simply. If they want P2P and want it to carry media (especially audio) then there is a _lot_ of work to do to ensure it is usable, secure etc.

It seems to me the right way forward is to define a data-only interface and PoC implementation that:

1) works with QUIC to the originating server almost immediately on one or 2 of the QUIC stacks
2) has a polyfill on top of data channels so it can be used P2P immediately
3) a companion GOLang/python server side implementation on top of the existing webRTC (non usrsctp) libraries.
4) serves as a target/requiements for the necessary protocol work on Quic-RT

The realtime media can be added later.

T.

---

From: westhawk <[email-masked]>
Date: Tue, 2 Jul 2019 10:46:13 +0100

> Maybe not for you, but we regularly get emails from people saying things like "It would be great to delete the mess of the WebRTC stack if there's a simpler protocol available. ICE/DTLS/SCTP all sc…

To be clear, I utterly sympathise with this viewpoint - no one in their right minds wants to pull libwebrtc into their server codebase.

I took one look at it and decided to write a fresh implementation that better suited my needs.
The fact that I could do that is due to the existence of RFCs for all of the  webRTC stack layers.

When implementing the datachannel I did ask a few questions of the experts, their replies were pointers
to the relevant bits of the RFCs. QUIC-RT simply isn’t in that position yet.

T.

---

From: Peter Thatcher <[email-masked]>
Date: Mon, 8 Jul 2019 11:47:39 -0700

On Tue, Jul 2, 2019 at 1:22 AM westhawk <[email-masked]> wrote:

> (Top post  - the quote level was getting unwieldy so I’ve trimmed quite a
> bit from the discussion - hopefully it still makes sense)
>
>> Establish connections to where ? Between eachother P2P ? Or to any
>> server? Or just to the originating server only?
>>
>
> Any server or p2p.
>
>
>
> Ok, great, but the requirements are different, and the OP seemed to be
> claiming simplifications that to my mind only work
> client - server.
>
>

> simpler protocol available. ICE/DTLS/SCTP all scare me”.
>
>
> With that being the view from a browser team you can see why people choose
> not to donate time to submit patches that fix it.
>
>
>
>>
>>
> On top of ICE with a laughable security model no media and no RFC - not a
> good basis (yet) for others to implement.
>
>
It's the same security model as the RTCDataChannel uses, and it can
transfer media just fine (transport doesn't care if it's media or not).  As
for RFCs, there are ICE and QUIC RFCs.  What more do you need?  If you
tried to write a "ICE + QUIC" RFC, I'm pretty sure it would be all
boilerplate and no actual text.

> I agree. If they want unreliable websockets to their servers, then we can
> cut out most of the layers and produce something based on QUIC
> very simply. If they want P2P and want it to carry media (especially
> audio) then there is a _lot_ of work to do to ensure it is usable, secure
> etc.
>
>
When you say "a lot" of work, what do you mean?  You can already send audio
on top of the QUIC-based WebTransport we have in origin trial and it's
usable and secure.  It would be nice to have low-level audio codec APIs in
the browser (which we are working on), but it's feasible to do audio codecs
in WASM in conjunction with WebAudio.

The only thing that seems like an issue is proving out whether or not one
can implement a good audio jitter buffer in WASM/JS (or port NetEq).  That
is work left to be done, indeed.  Is that what you were referring to?

> 1) works with QUIC to the originating server almost immediately on one or

I'm not completely sure what you mean here by "one or two of the QUIC
stacks", but see #3 below for code that can (hopefully soon) can do the
server side.

 3) a companion GOLang/python server side implementation on top of the
existing webRTC (non usrsctp) libraries.

I have one in progress:
https://github.com/pthatcherg/quic-go/blob/gquic/example/webtransport/server.go
.
I don't quite have it working, but it's close.

And there is also this one:
https://github.com/pion/webrtc/blob/736e0bb5062c473a61d9aae0dbb3bac7f32db377/quictransport.go