Subject: Re: [rtcweb] Open data channel issues

From: Randell Jesup <[email-masked]>
Date: Sun, 02 Mar 2014 04:37:00 -0500

On 2/26/2014 2:50 AM, Magnus Westerlund wrote:
> Thanks Michael,
>
> WG, please consider these open issues and try to form your own position
> on them. They are intended to be discussed at the meeting. If you have
> proposals on how to resolve them or other input you are very welcome to
> provide that on the list.

One more big issue.  I realize this is very late for pre-meeting 
discussion; I'd hoped to hash this out earlier but for various reasons 
(including power outages and my own workload) this didn't happen.

We discussed a way to deal with the issues surrounding maximum message 
sizes at the last IETF.  Right now we have a proposal in the MMUSIC 
draft for limiting the maximum message size via the SDP.

There is a problem with this: it's at odds with the definition of 
DataChannels in the W3 and with the "duck-typing" of DataChannels to 
work largely as a superset of WebSockets (outside of channel creation), 
and the WebAPI folk at Mozilla I talked to don't like the direction 
we're taking.

I've been having talks with the WebAPI people at Mozilla, in particular 
Jonas Sicking, our WebAPI lead, and they strongly dislike encouraging 
applications to try to implement their large-data/blob transfer 
protocols; browsers have considerably more tools available to them to 
avoid memory hits and to make use of off-main-thread resources than the 
JS apps do.  "Having Send(blob) fail for any size of blob is crazy and 
non-sensical" was one comment made when I described the impacts of the 
current plan.

Manual chunking in the application means poorly-implemented congestion 
control in the app to keep the channel running efficiently (the only 
feedback available directly is either having the far-end ack at the user 
level, or trying to estimate sleep times via setTimeout() and 
bufferedAmount() (which is simply not a great solution), or simply 
dumping a large amount of smaller transfers into Send() and causing the 
browser to have to buffer them in memory).  Also GC or other pauses in 
JS execution may cause hiccups in the transfer and mis-estimation of 
available bandwidth.  And of course this is being run over a 
congestion-controlled channel in the first place.

Unless and until the W3 side makes DataChannels (and by extension, 
PeerConnection) APIs available from JS workers (and this is 
implemented), there will be compromises with packet-level protocols in 
JS.  One of those will be "it's hard to implement your own congestion 
control well".  Even with worker support, considerable extension of the 
APIs would be needed to make it work really well there.  I'll also note 
that DataChannels-from-worker support is nowhere near implementation in 
browsers.

Another BIG problem as it's currently defined is that there's no lower 
bound for this limit, so all DataChannel users will need to implement 
manual chunking even if they use small fixed messages to guarantee spec 
compliance.  Of course they won't do so...  and even if they did, they 
wouldn't test it (another big problem).  You might say "ok, fine, lets 
set some small lower bound on this value, say 2 or 4 or 16K".  That 
doesn't really help much either.  Many will send variable-sized messages 
(because it's easy), and again won't test what happens when the messages 
trip over the spec limit (or the actual browser implementation limit!)  
Those with fixed-size messages larger than the spec lower-bound won't 
test the against that; they'll test against what Firefox and Chrome 
implement at the moment.  So the net result is they'll ship applications 
that can break randomly in the field for no obvious reason (say if IE 
implements and uses 16K when Chrome used 32K and Firefox used 100MB).

Why hand the application a footgun?

Jonas Sicking suggested if the IETF insists on not supporting arbitrary 
Send(blob), we'll need to push in the W3 for a W3 protocol run on top of 
IETF DataChannels that handles fragmentation and reassembly in order to 
preserve the JS API for Send().  We can do this, but part of the whole 
partnership between the IETF and W3 on WebRTC was to try to avoid having 
the W3 define network protocols and keep them in the IETF where they belong.

Note: abandoning Send(blob) in W3 doesn't help much, as the comments I 
made above about arbitrary limits and almost-certain lack of testing of 
messages violating the negotiated size would still apply.  Send(blob) 
just makes it easier to trip over the problem (and in fact more likely 
that the application will test very large sizes).

Our options are:

A) Just accept this complexity and just hope that people write good code 
or use good libraries. (See above about testing...)
Note: we'd need to set *some* lower bound for the value.

B) Make the W3 API implementation add a level of protocol on top of the 
underlying IETF network protocol. This protocol could then deal with 
fragmenting messages on the sending side and reassembling them on the 
receiving side.

C) Convince IETF WG to support arbitrarily sized messages at a protocol 
level, at least in the WebRTC context, similar to WebSockets.

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: Justin Uberti <[email-masked]>
Date: Sun, 2 Mar 2014 13:27:40 -0800

I'm not sure I understand the actual problem here. We *are* going to handle
arbitrary-sized messages once ndata gets done. The max-msg-size stuff is
just a solution until ndata is deployed. (To cite your options, this is A
followed by C).

That said, Send(blob) seems like a bit of a footgun to me anyway; I think
apps are going to typically avoid it, a) to avoid memory bloat, and b) to
get progress indications. The only way out of that situation is to provide
stream semantics, which seems like a lot to take on given that WebSockets
hasn't gone that route. I also agree that we shouldn't try to go the (B)
route that you mention.

So I still think manual chunking seems like a reasonable thing to support.
I don't quite get the congestion control concerns; just because there is a
max chunk size doesn't mean the impl can't buffer multiple chunks in
bufferedAmount; the app could let that fill up to a degree to avoid having
to poll constantly to prevent underrun.

On Sun, Mar 2, 2014 at 1:37 AM, Randell Jesup <[email-masked]>wrote:

> sizes at the last IETF.  Right now we have a proposal in the MMUSIC draft
> for limiting the maximum message size via the SDP.
>
> DataChannels in the W3 and with the "duck-typing" of DataChannels to work
> largely as a superset of WebSockets (outside of channel creation), and the
> WebAPI folk at Mozilla I talked to don't like the direction we're taking.
>
> applications to try to implement their large-data/blob transfer protocols;
> browsers have considerably more tools available to them to avoid memory
> hits and to make use of off-main-thread resources than the JS apps do.
>  "Having Send(blob) fail for any size of blob is crazy and non-sensical"
> was one comment made when I described the impacts of the current plan.
>
>
> bufferedAmount() (which is simply not a great solution), or simply dumping
> a large amount of smaller transfers into Send() and causing the browser to
> have to buffer them in memory).  Also GC or other pauses in JS execution
> may cause hiccups in the transfer and mis-estimation of available
> bandwidth.  And of course this is being run over a congestion-controlled
> channel in the first place.
>
> PeerConnection) APIs available from JS workers (and this is implemented),
> there will be compromises with packet-level protocols in JS.  One of those
> will be "it's hard to implement your own congestion control well".  Even
> with worker support, considerable extension of the APIs would be needed to
> make it work really well there.  I'll also note that
> DataChannels-from-worker support is nowhere near implementation in browsers.
>
>
> wouldn't test it (another big problem).  You might say "ok, fine, lets set
> some small lower bound on this value, say 2 or 4 or 16K".  That doesn't
> really help much either.  Many will send variable-sized messages (because
> it's easy), and again won't test what happens when the messages trip over
> the spec limit (or the actual browser implementation limit!)  Those with
> fixed-size messages larger than the spec lower-bound won't test the against
> that; they'll test against what Firefox and Chrome implement at the moment.
>  So the net result is they'll ship applications that can break randomly in
> the field for no obvious reason (say if IE implements and uses 16K when
> Chrome used 32K and Firefox used 100MB).
>
> messages violating the negotiated size would still apply.  Send(blob) just
> makes it easier to trip over the problem (and in fact more likely that the
> application will test very large sizes).
>
>
> Our options are:
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Randell Jesup <[email-masked]>
Date: Sun, 02 Mar 2014 21:42:21 -0500

On 3/2/2014 4:27 PM, Justin Uberti wrote:
> I'm not sure I understand the actual problem here. We *are* going to 
> handle arbitrary-sized messages once ndata gets done. The max-msg-size 
> stuff is just a solution until ndata is deployed. (To cite your 
> options, this is A followed by C).

So, here's where I think there may be a disconnect (and if I'm wrong, 
great):

ndata solves the monopolization issue between streams, allowing packets 
for different streams to be interleaved on the wire.  It does not (so 
far as I know) relax the usrsctp library's returning EMSGSIZE if the 
sendto() is larger than available buffer space, and the alternative (EOR 
mode) doesn't allow for interleaving of messages when sending at all 
(and I'm not sure it allows interleaving on reception either, but I 
didn't check right now).

Now, this is an implementation/API issue which could be fixed with some 
work, but so far as I know no work has been done on it.  So ndata will 
allow us to expand the max-sending size to min(SCTP_BUFFER_SIZE - N, 
other-sides-max-receive-size).  It does not allow us to expand the size 
to any useful Blob size.

>
> That said, Send(blob) seems like a bit of a footgun to me anyway; I 
> think apps are going to typically avoid it, a) to avoid memory bloat, 
> and b) to get progress indications. The only way out of that situation 
> is to provide stream semantics, which seems like a lot to take on 
> given that WebSockets hasn't gone that route. I also agree that we 
> shouldn't try to go the (B) route that you mention.

Well, Streams would be a good thing once they're done. Progress 
indication could be added at the W3 level without much problem.  The 
memory issue should be resolvable in a number of ways, per my discussion 
with Sicking.  Note that application chunking has a serious memory issue 
today in that the File Writer API hasn't been agreed to; I think there's 
progress towards eventually adopting a version of the Filesystem API 
(with changes), but that will be a while.  Again, Stream should help - 
and note that the semantics for Blob reception allow it to be written to 
disk as it's received and not held entirely in memory when it's handed 
to the application, which is NOT possible today for application chunking.

>
> So I still think manual chunking seems like a reasonable thing to 
> support. I don't quite get the congestion control concerns; just 
> because there is a max chunk size doesn't mean the impl can't buffer 
> multiple chunks in bufferedAmount; the app could let that fill up to a 
> degree to avoid having to poll constantly to prevent underrun.

On a slow link this will work if the browser isn't janky.  On a fast 
link GC pauses and other things may cause the buffer to drain out and go 
idle for significant periods.

   Randell

>
>
> On Sun, Mar 2, 2014 at 1:37 AM, Randell Jesup <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     discussion; I'd hoped to hash this out earlier but for various
>     reasons (including power outages and my own workload) this didn't
>     happen.
>
>
>     We discussed a way to deal with the issues surrounding maximum
>     message sizes at the last IETF.  Right now we have a proposal in
>     the MMUSIC draft for limiting the maximum message size via the SDP.
>
>     DataChannels in the W3 and with the "duck-typing" of DataChannels
>     to work largely as a superset of WebSockets (outside of channel
>     creation), and the WebAPI folk at Mozilla I talked to don't like
>     the direction we're taking.
>
>     I've been having talks with the WebAPI people at Mozilla, in
>     particular Jonas Sicking, our WebAPI lead, and they strongly
>     dislike encouraging applications to try to implement their
>     large-data/blob transfer protocols; browsers have considerably
>     more tools available to them to avoid memory hits and to make use
>     of off-main-thread resources than the JS apps do.  "Having
>     Send(blob) fail for any size of blob is crazy and non-sensical"
>     Manual chunking in the application means poorly-implemented
>     congestion control in the app to keep the channel running
>     efficiently (the only feedback available directly is either having
>     the far-end ack at the user level, or trying to estimate sleep
>     times via setTimeout() and bufferedAmount() (which is simply not a
>     great solution), or simply dumping a large amount of smaller
>     transfers into Send() and causing the browser to have to buffer
>     them in memory).  Also GC or other pauses in JS execution may
>     cause hiccups in the transfer and mis-estimation of available
>     bandwidth.  And of course this is being run over a
>     implemented), there will be compromises with packet-level
>     protocols in JS.  One of those will be "it's hard to implement
>     your own congestion control well".  Even with worker support,
>     considerable extension of the APIs would be needed to make it work
>     really well there.  I'll also note that DataChannels-from-worker
>     support is nowhere near implementation in browsers.
>
>
>     Another BIG problem as it's currently defined is that there's no
>     lower bound for this limit, so all DataChannel users will need to
>     implement manual chunking even if they use small fixed messages to
>     guarantee spec compliance.  Of course they won't do so...  and
>     even if they did, they wouldn't test it (another big problem).
>      You might say "ok, fine, lets set some small lower bound on this
>     value, say 2 or 4 or 16K".  That doesn't really help much either.
>      Many will send variable-sized messages (because it's easy), and
>     again won't test what happens when the messages trip over the spec
>     limit (or the actual browser implementation limit!)  Those with
>     fixed-size messages larger than the spec lower-bound won't test
>     the against that; they'll test against what Firefox and Chrome
>     implement at the moment.  So the net result is they'll ship
>     applications that can break randomly in the field for no obvious
>     reason (say if IE implements and uses 16K when Chrome used 32K and
>     Firefox used 100MB).
>
>     Jonas Sicking suggested if the IETF insists on not supporting
>     arbitrary Send(blob), we'll need to push in the W3 for a W3
>     protocol run on top of IETF DataChannels that handles
>     fragmentation and reassembly in order to preserve the JS API for
>     Send().  We can do this, but part of the whole partnership between
>     the IETF and W3 on WebRTC was to try to avoid having the W3 define
>     network protocols and keep them in the IETF where they belong.
>
>     Note: abandoning Send(blob) in W3 doesn't help much, as the
>     comments I made above about arbitrary limits and almost-certain
>     lack of testing of messages violating the negotiated size would
>     still apply.  Send(blob) just makes it easier to trip over the
>     problem (and in fact more likely that the application will test
>     very large sizes).
>
>
>     Our options are:
>
>     A) Just accept this complexity and just hope that people write
>     good code or use good libraries. (See above about testing...)
>     B) Make the W3 API implementation add a level of protocol on top
>     of the underlying IETF network protocol. This protocol could then
>     deal with fragmenting messages on the sending side and
>     reassembling them on the receiving side.
>
>     C) Convince IETF WG to support arbitrarily sized messages at a
>     protocol level, at least in the WebRTC context, similar to
>     WebSockets.
>
>
>
>     -- 
>     [email-masked] <mailto:[email-masked]>

---

From: Martin Thomson <[email-masked]>
Date: Sun, 2 Mar 2014 22:56:54 -0800

On 2 March 2014 18:42, Randell Jesup <[email-masked]> wrote:
> ndata solves the monopolization issue between streams, allowing packets for
> different streams to be interleaved on the wire.  It does not (so far as I
> know) relax the usrsctp library's returning EMSGSIZE if the sendto() is
> larger than available buffer space, and the alternative (EOR mode) doesn't
> allow for interleaving of messages when sending at all (and I'm not sure it
> allows interleaving on reception either, but I didn't check right now).
>
> work, but so far as I know no work has been done on it.  So ndata will allow
> us to expand the max-sending size to min(SCTP_BUFFER_SIZE - N,
> other-sides-max-receive-size).  It does not allow us to expand the size to
> any useful Blob size.

Sounds like the complaint would be best addressed by fixing the SCTP
implementations.  I would hope that ndata would allow for transmission
of messages far in excess of what the implementation is willing to
buffer, that seems arbitrarily restrictive.

(A streaming API wounds like a good idea too.)

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 06:53:59 +0000

On 03 Mar 2014, at 02:42, Randell Jesup <[email-masked]> wrote:

>> I'm not sure I understand the actual problem here. We *are* going to handle arbitrary-sized messages once ndata gets done. The max-msg-size stuff is just a solution until ndata is deployed. (To cite your options, this is A followed by C).
> 
> So, here's where I think there may be a disconnect (and if I'm wrong, great):
> 
> ndata solves the monopolization issue between streams, allowing packets for different streams to be interleaved on the wire.  It does not (so far as I know) relax the usrsctp library's returning 
Correct.
> EMSGSIZE if the sendto() is larger than available buffer space, and the alternative (EOR mode) doesn't allow for interleaving of messages when sending at all (and I'm not sure it allows interleaving on reception either, but I didn't check right now).
If you want to send messages larger than the socket send buffer, you must use the explicit
EOR more. This is independent from using NDATA or not.
If you use NDATA, you can have on each outgoing stream a single message (order or unordered)
being sent using explicit EOR mode and the calls can be interleaved. I don't think I have
tested this up to know, but this is how it should work.

rrs@: Just to double check: Am I right? Please note that RTCWeb doesn't use ordered and
      unordered on the same stream, and you don't need to send more than one message with
      explicit EOR on the same stream at the same time. Therefore the above simplification...
> 
> Now, this is an implementation/API issue which could be fixed with some work, but so far as I know no work has been done on it.  So ndata will allow us to expand the max-sending size to 
This API work is part of getting NDATA implemented. Randall has implemented it, we do have
code, but it needs some testing and cleaning up in the API which supports the scheduling...
> min(SCTP_BUFFER_SIZE - N, other-sides-max-receive-size).  It does not allow us to expand the size to any useful Blob size.
Please note that you never have a limitation of the receive buffer send size, since SCTP
would use partial delivery. So other-sides-max-receive-size is only relevant if there are
buffering limits on top of SCTP.

Best regards
Michael
> 
>> 
>> That said, Send(blob) seems like a bit of a footgun to me anyway; I think apps are going to typically avoid it, a) to avoid memory bloat, and b) to get progress indications. The only way out of that situation is to provide stream semantics, which seems like a lot to take on given that WebSockets hasn't gone that route. I also agree that we shouldn't try to go the (B) route that you mention.
> 
> Well, Streams would be a good thing once they're done. Progress indication could be added at the W3 level without much problem.  The memory issue should be resolvable in a number of ways, per my discussion with Sicking.  Note that application chunking has a serious memory issue today in that the File Writer API hasn't been agreed to; I think there's progress towards eventually adopting a version of the Filesystem API (with changes), but that will be a while.  Again, Stream should help - and note that the semantics for Blob reception allow it to be written to disk as it's received and not held entirely in memory when it's handed to the application, which is NOT possible today for application chunking.
> 
>> 
>> So I still think manual chunking seems like a reasonable thing to support. I don't quite get the congestion control concerns; just because there is a max chunk size doesn't mean the impl can't buffer multiple chunks in bufferedAmount; the app could let that fill up to a degree to avoid having to poll constantly to prevent underrun.
> 
> On a slow link this will work if the browser isn't janky.  On a fast link GC pauses and other things may cause the buffer to drain out and go idle for significant periods.
> 
>   Randell
> 
>> 
>> 
>> On Sun, Mar 2, 2014 at 1:37 AM, Randell Jesup <[email-masked]> wrote:
>> One more big issue.  I realize this is very late for pre-meeting discussion; I'd hoped to hash this out earlier but for various reasons (including power outages and my own workload) this didn't happen.
>> 
>> 
>> We discussed a way to deal with the issues surrounding maximum message sizes at the last IETF.  Right now we have a proposal in the MMUSIC draft for limiting the maximum message size via the SDP.
>> 
>> There is a problem with this: it's at odds with the definition of DataChannels in the W3 and with the "duck-typing" of DataChannels to work largely as a superset of WebSockets (outside of channel creation), and the WebAPI folk at Mozilla I talked to don't like the direction we're taking.
>> 
>> I've been having talks with the WebAPI people at Mozilla, in particular Jonas Sicking, our WebAPI lead, and they strongly dislike encouraging applications to try to implement their large-data/blob transfer protocols; browsers have considerably more tools available to them to avoid memory hits and to make use of off-main-thread resources than the JS apps do.  "Having Send(blob) fail for any size of blob is crazy and non-sensical" was one comment made when I described the impacts of the current plan.
>> 
>> 
>> Manual chunking in the application means poorly-implemented congestion control in the app to keep the channel running efficiently (the only feedback available directly is either having the far-end ack at the user level, or trying to estimate sleep times via setTimeout() and bufferedAmount() (which is simply not a great solution), or simply dumping a large amount of smaller transfers into Send() and causing the browser to have to buffer them in memory).  Also GC or other pauses in JS execution may cause hiccups in the transfer and mis-estimation of available bandwidth.  And of course this is being run over a congestion-controlled channel in the first place.
>> 
>> Unless and until the W3 side makes DataChannels (and by extension, PeerConnection) APIs available from JS workers (and this is implemented), there will be compromises with packet-level protocols in JS.  One of those will be "it's hard to implement your own congestion control well".  Even with worker support, considerable extension of the APIs would be needed to make it work really well there.  I'll also note that DataChannels-from-worker support is nowhere near implementation in browsers.
>> 
>> 
>> Another BIG problem as it's currently defined is that there's no lower bound for this limit, so all DataChannel users will need to implement manual chunking even if they use small fixed messages to guarantee spec compliance.  Of course they won't do so...  and even if they did, they wouldn't test it (another big problem).  You might say "ok, fine, lets set some small lower bound on this value, say 2 or 4 or 16K".  That doesn't really help much either.  Many will send variable-sized messages (because it's easy), and again won't test what happens when the messages trip over the spec limit (or the actual browser implementation limit!)  Those with fixed-size messages larger than the spec lower-bound won't test the against that; they'll test against what Firefox and Chrome implement at the moment.  So the net result is they'll ship applications that can break randomly in the field for no obvious reason (say if IE implements and uses 16K when Chrome used 32K and Firefox used 100MB).
>> 
>> Jonas Sicking suggested if the IETF insists on not supporting arbitrary Send(blob), we'll need to push in the W3 for a W3 protocol run on top of IETF DataChannels that handles fragmentation and reassembly in order to preserve the JS API for Send().  We can do this, but part of the whole partnership between the IETF and W3 on WebRTC was to try to avoid having the W3 define network protocols and keep them in the IETF where they belong.
>> 
>> Note: abandoning Send(blob) in W3 doesn't help much, as the comments I made above about arbitrary limits and almost-certain lack of testing of messages violating the negotiated size would still apply.  Send(blob) just makes it easier to trip over the problem (and in fact more likely that the application will test very large sizes).
>> 
>> 
>> Our options are:
>> 
>> A) Just accept this complexity and just hope that people write good code or use good libraries. (See above about testing...)
>> B) Make the W3 API implementation add a level of protocol on top of the underlying IETF network protocol. This protocol could then deal with fragmenting messages on the sending side and reassembling them on the receiving side.
>> 
>> C) Convince IETF WG to support arbitrarily sized messages at a protocol level, at least in the WebRTC context, similar to WebSockets.
>> 
>> 
>> 
>> --

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 07:01:28 +0000

> buffer, that seems arbitrarily restrictive.
As said in my previous mail: Using NDATA will allow to have on each
stream a single ordered or unordered message being sent in explicit
EOR mode. The send calls can be interleaved.

Best regards
Michael

---

From: Martin Thomson <[email-masked]>
Date: Mon, 3 Mar 2014 02:39:01 -0800

> EOR mode. The send calls can be interleaved.

That's great.  So the protocol can and is being fixed.  I think that
part of Randell's concern comes from the implementations thereof.
Those would seem to be even easier to fix.

---

From: Stefan Håkansson LK <[email-masked]>
Date: Mon, 3 Mar 2014 10:45:46 +0000

> (A streaming API wounds like a good idea too.)

There is some ongoing work: 
http://lists.w3.org/Archives/Public/public-webapps/2014JanMar/0218.html

---

From: Martin Thomson <[email-masked]>
Date: Mon, 3 Mar 2014 02:57:26 -0800

> http://lists.w3.org/Archives/Public/public-webapps/2014JanMar/0218.html

I was thinking more specific work applying that to thewebsocketapi.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 3 Mar 2014 03:18:42 -0800

On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup <[email-masked]>wrote:

> stuff is just a solution until ndata is deployed. (To cite your options,
> this is A followed by C).
>
>
> for different streams to be interleaved on the wire.  It does not (so far
> as I know) relax the usrsctp library's returning EMSGSIZE if the sendto()
> is larger than available buffer space, and the alternative (EOR mode)
> doesn't allow for interleaving of messages when sending at all (and I'm not
> sure it allows interleaving on reception either, but I didn't check right
> now).
>

Based on the discussion of ndata/eor in followup messages, I think this
problem will be addressed in the near future.

> think apps are going to typically avoid it, a) to avoid memory bloat, and
> b) to get progress indications. The only way out of that situation is to
> provide stream semantics, which seems like a lot to take on given that
> WebSockets hasn't gone that route. I also agree that we shouldn't try to go
> the (B) route that you mention.
>
>
> Well, Streams would be a good thing once they're done. Progress indication
> could be added at the W3 level without much problem.  The memory issue
> should be resolvable in a number of ways, per my discussion with Sicking.
> Note that application chunking has a serious memory issue today in that the
> File Writer API hasn't been agreed to; I think there's progress towards
> eventually adopting a version of the Filesystem API (with changes), but
> that will be a while.  Again, Stream should help - and note that the
> semantics for Blob reception allow it to be written to disk as it's
> received and not held entirely in memory when it's handed to the
> application, which is NOT possible today for application chunking.
>

Agree Streams will be useful when done, but that's probably post-1.0 (as
Martin says, we should first let it be applied to WebSockets). However I
don't think app chunking will have any memory problems - the chunks will be
small enough that they can be easily spooled to disk as they come in.

> support. I don't quite get the congestion control concerns; just because
> there is a max chunk size doesn't mean the impl can't buffer multiple
> chunks in bufferedAmount; the app could let that fill up to a degree to
> avoid having to poll constantly to prevent underrun.
>
>
> On a slow link this will work if the browser isn't janky.  On a fast link
> GC pauses and other things may cause the buffer to drain out and go idle
> for significant periods.
>

Since the amount buffered is up to the app, the app can just increase the
buffer size to provide enough data to cover the next second or so of
transfer.

Note that app chunking will clearly be needed in some of the more
interesting cases we want to support, such as where the transfer is not
1:1. For swarm-style downloading, we need to make sure app chunking works
well (and I think it does).

> rtcweb mailing [email-masked]://www.ietf.org/mailman/listinfo/rtcweb
>
>
>
> --

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Mar 2014 06:25:16 -0500

> EOR more. This is independent from using NDATA or not.

right.

>        explicit EOR on the same stream at the same time. Therefore the above simplification...

If this is the case, this should deal with the issue I believe, with one 
caveat for the W3 layer:

If we can only send one at a time, we get an issue where we call 
channel.send(blob1); channel.send(blob2);

the JS interface to send() is synchronous and can't block.  This implies 
that if send(blob1) hasn't completed, the send(blob2) can't start, and 
all following send()s must also be buffered in memory. For a blob, since 
a blob reference is read-only, this is likely ok (need to verify what 
happens if we haven't read the blob into memory - if this isn't the 
case, we have a problem).  For sending an ArrayBuffer (or string), we'll 
have to snapshot the buffer at send() time if it's queued since the 
referenced data may change.  However, this really isn't anything that 
new, since if the SCTP buffers are full you have to do something similar 
anyways, even without any messages using fragmentation.

I presume from your comment (since it wasn't entirely clear) that if I 
have channel.send(blob1); channel.send("foo") that "foo" can't be sent 
until blob1 has been fully transmitted (has hit EOR at the send side), 
even though "foo" doesn't require fragmentation, and so all send()s for 
a channel during a fragmented send() must be copied and queued.

> code, but it needs some testing and cleaning up in the API which supports the scheduling...

So the last issue would be backwards compatibility.  If today an app 
must be written to query (via an unspecified W3 property) the max size 
you can reliably send() to the other side, then it must buy into (and 
test!) that functionality now and until it's pretty certain that all 
non-compliant implementations are gone.  This can take a while, even if 
you assume people update promptly, due to things like Firefox "ESR" 
releases (Extended Support for businesses, schools, etc).  Thes are 
valid for most of a year, and if we land ndata support right after one 
is created it could be closer to 1-1.5 years before if even starts to 
reach clients - and it may take even more time for all ESR users to 
approve/test it and update.

The arguments about having all the apps be required to implement their 
own fragmentation, etc for a considerable period adds lots of complexity 
to the apps.  Implementing the PPID chunking as the stopgap (and 
retiring it once ndata is universal) means the API to the application 
remains stable and simple.  Since all DataChannel implementations must 
implement queuing of JS send()s (since the SCTP bufferspace may be 
full), it's easy to support PPID fragmentation if you don't worry about 
memory-use optimization).  ndata not being available yet highlights this 
issue.

> buffering limits on top of SCTP.

I was referring to the SDP value for "minimum value I guarantee receiving".

We would need to specify that if ndata is agreed to (I presume we know 
that) that the application would be told an "infinite" value for 
max-I-can-receive (since we can't use the value from the SDP, as that 
predates the two sides creating the association and negotiating ndata).  
I don't want to renegotiate just to remove the max SCTP receive size.

   Randell

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Mar 2014 06:44:44 -0500

On 3/3/2014 6:18 AM, Justin Uberti wrote:
>
>
> On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup <[email-masked] 
>>     I'm not sure I understand the actual problem here. We *are* going
>>     to handle arbitrary-sized messages once ndata gets done. The
>>     max-msg-size stuff is just a solution until ndata is deployed.
>>     (To cite your options, this is A followed by C).
>
>     ndata solves the monopolization issue between streams, allowing
>     packets for different streams to be interleaved on the wire.  It
>     does not (so far as I know) relax the usrsctp library's returning
>     EMSGSIZE if the sendto() is larger than available buffer space,
>     and the alternative (EOR mode) doesn't allow for interleaving of
>     messages when sending at all (and I'm not sure it allows
>     interleaving on reception either, but I didn't check right now).
>
>     Now, this is an implementation/API issue which could be fixed with
>     some work, but so far as I know no work has been done on it.  So
>     ndata will allow us to expand the max-sending size to
>     min(SCTP_BUFFER_SIZE - N, other-sides-max-receive-size).  It does
>     not allow us to expand the size to any useful Blob size.
>
>
> Based on the discussion of ndata/eor in followup messages, I think 
> this problem will be addressed in the near future.

Hopefully so, though I hate forcing apps to include a bunch of code for 
backwards compat for a year or two (and many won't, or won't test it, 
etc).  That hurts the feature.  And today no one has ndata.

>
>>     That said, Send(blob) seems like a bit of a footgun to me anyway;
>>     I think apps are going to typically avoid it, a) to avoid memory
>>     bloat, and b) to get progress indications. The only way out of
>>     that situation is to provide stream semantics, which seems like a
>>     lot to take on given that WebSockets hasn't gone that route. I
>>     also agree that we shouldn't try to go the (B) route that you
>>     mention.
>
>     indication could be added at the W3 level without much problem. 
>     The memory issue should be resolvable in a number of ways, per my
>     discussion with Sicking.  Note that application chunking has a
>     serious memory issue today in that the File Writer API hasn't been
>     agreed to; I think there's progress towards eventually adopting a
>     version of the Filesystem API (with changes), but that will be a
>     while.  Again, Stream should help - and note that the semantics
>     for Blob reception allow it to be written to disk as it's received
>     and not held entirely in memory when it's handed to the
> Agree Streams will be useful when done, but that's probably post-1.0 
> (as Martin says, we should first let it be applied to WebSockets). 
> However I don't think app chunking will have any memory problems - the 
> chunks will be small enough that they can be easily spooled to disk as 
> they come in.

Agree on Streams (post 1.0).  Chunking does have memory problems unless 
you have a File Writer API (we don't) or a Filesystem API (we don't, 
though my understanding is that we believe that with some changes this 
can be adoptable in the future).  You have two choices today for app 
chunking (and I assume using blob slices for sending chunks):

1) create a blob from each received chunk then then create a new blob 
from the concatenation of all the sub-blobs.  This requires a large 
memory hit, though I suppose in theory maybe it could be done without a 
hit (painfully).  Even if the memory hit is avoided there will be a 
large disk IO hit to merge all those files and write a new file from them.

2) hold all the parts in memory, then write the entire thing as a blob 
after the last part is received.  Large memory and disk IO hit on 
reception of final part.

>>     because there is a max chunk size doesn't mean the impl can't
>>     buffer multiple chunks in bufferedAmount; the app could let that
>>     fill up to a degree to avoid having to poll constantly to prevent
>>     underrun.
>
>     On a slow link this will work if the browser isn't janky.  On a
>     fast link GC pauses and other things may cause the buffer to drain
>     out and go idle for significant periods.
>
>
> Since the amount buffered is up to the app, the app can just increase 
> the buffer size to provide enough data to cover the next second or so 
> of transfer.

How does the app increase the size of the internal SCTP buffers? (And 
that buffer is shared by all channels, of course).

> interesting cases we want to support, such as where the transfer is 
> not 1:1. For swarm-style downloading, we need to make sure app 
> chunking works well (and I think it does).

Agreed, we need good solutions for application chunking (and thus my 
interest in resolving the Filesystem API issues in the W3).  My problem 
is that we don't have those today for the reception side - but this (for 
apps that *need* chunking) is largely a W3 issue.

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 12:10:38 +0000

On 03 Mar 2014, at 11:25, Randell Jesup <[email-masked]> wrote:

> If this is the case, this should deal with the issue I believe, with one caveat for the W3 layer:
> 
> If we can only send one at a time, we get an issue where we call channel.send(blob1); channel.send(blob2);
> 
> the JS interface to send() is synchronous and can't block.  This implies that if send(blob1) hasn't completed, the send(blob2) can't start, and all following send()s must also be buffered in memory. For a blob, since a blob reference is read-only, this is likely ok (need to verify what happens if we haven't read the blob into memory - if this isn't the case, we have a problem).  For sending an ArrayBuffer (or string), we'll have to snapshot the buffer at send() time if it's queued since the referenced data may change.  However, this really isn't anything that new, since if the SCTP buffers are full you have to do something similar anyways, even without any messages using fragmentation.
> 
> I presume from your comment (since it wasn't entirely clear) that if I have channel.send(blob1); channel.send("foo") that "foo" can't be sent until blob1 has been fully transmitted (has hit EOR at the send side), even though "foo" doesn't require fragmentation, and so all send()s for a channel during a fragmented send() must be copied and queued.
Correct. And if you are on an ordered channel, there is no alternative, right? On an unordered,
you can't interleave messages on a single stream...
> So the last issue would be backwards compatibility.  If today an app must be written to query (via an unspecified W3 property) the max size you can reliably send() to the other side, then it must buy into (and test!) that functionality now and until it's pretty certain that all non-compliant implementations are gone.  This can take a while, even if you assume people update promptly, due to things like Firefox "ESR" releases (Extended Support for businesses, schools, etc).  Thes are valid for most of a year, and if we land ndata support right after one is created it could be closer to 1-1.5 years before if even starts to reach clients - and it may take even more time for all ESR users to approve/test it and update.
> 
> The arguments about having all the apps be required to implement their own fragmentation, etc for a considerable period adds lots of complexity to the apps.  Implementing the PPID chunking as the stopgap (and retiring it once ndata is universal) means the API to the application remains stable and simple.  Since all DataChannel implementations must implement queuing of JS send()s (since the SCTP bufferspace may be full), it's easy to support PPID fragmentation if you don't worry about memory-use optimization).  ndata not being available yet highlights this issue.
> 
Understood. My point was the following:

If NDATA is not yet supported, the sender can choose a message limit to avoid
monopolisation. It is a sender side thing.

So any limit the receiver is enforcing and communicating via SDP is for different
reasons. So if we agree on support for arbitrary size messages when we have
NDATA, then there is ne need to negotiate a message size limit. The sender just
enforces whatever he things is appropriate to mitigate monopolisation.
> 
> We would need to specify that if ndata is agreed to (I presume we know that) that the application would be told an "infinite" value for max-I-can-receive (since we can't use the value from the SDP, as that predates the two sides creating the association and negotiating ndata).  I don't want to renegotiate just to remove the max SCTP receive size.
Correct. See my above statement. Why negotiate at all if we agree on the receiver being
able to handle arbitrary size messages?

Best regards
Michael

---

From: Justin Uberti <[email-masked]>
Date: Mon, 3 Mar 2014 05:28:54 -0800

On Mon, Mar 3, 2014 at 3:44 AM, Randell Jesup <[email-masked]>wrote:

>> indication could be added at the W3 level without much problem.  The memory
>> issue should be resolvable in a number of ways, per my discussion with
>> Sicking.  Note that application chunking has a serious memory issue today
>> in that the File Writer API hasn't been agreed to; I think there's progress
>> towards eventually adopting a version of the Filesystem API (with changes),
>> but that will be a while.  Again, Stream should help - and note that the
> you have a File Writer API (we don't) or a Filesystem API (we don't, though
> my understanding is that we believe that with some changes this can be
> adoptable in the future).  You have two choices today for app chunking (and
> I assume using blob slices for sending chunks):
>
> 1) create a blob from each received chunk then then create a new blob from
> the concatenation of all the sub-blobs.  This requires a large memory hit,
> though I suppose in theory maybe it could be done without a hit
> (painfully).  Even if the memory hit is avoided there will be a large disk
> IO hit to merge all those files and write a new file from them.
>
> after the last part is received.  Large memory and disk IO hit on reception
> of final part.
>
>
>
>>
> the buffer size to provide enough data to cover the next second or so of
> transfer.
>
>

What I meant is the app can make multiple calls to send() at a time, each
with a chunk, which will be buffered in the api layer and increase
bufferedAmount.

> interest in resolving the Filesystem API issues in the W3).  My problem is
> that we don't have those today for the reception side - but this (for apps
> that *need* chunking) is largely a W3 issue.
>
>
>
> --

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Mar 2014 08:51:44 -0500

On 3/3/2014 8:28 AM, Justin Uberti wrote:
> On Mon, Mar 3, 2014 at 3:44 AM, Randell Jesup <[email-masked] 
>>     On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>>     So I still think manual chunking seems like a reasonable thing
>>>     to support. I don't quite get the congestion control concerns;
>>>     just because there is a max chunk size doesn't mean the impl
>>>     can't buffer multiple chunks in bufferedAmount; the app could
>>>     let that fill up to a degree to avoid having to poll constantly
>>>     to prevent underrun.
>>
>>         On a slow link this will work if the browser isn't janky.  On
>>         a fast link GC pauses and other things may cause the buffer
>>         to drain out and go idle for significant periods.
>>
>>
>>     Since the amount buffered is up to the app, the app can just
>>     increase the buffer size to provide enough data to cover the next
>>     second or so of transfer.
>
>     How does the app increase the size of the internal SCTP buffers? 
>     (And that buffer is shared by all channels, of course).
>
>
> What I meant is the app can make multiple calls to send() at a time, 
> each with a chunk, which will be buffered in the api layer and 
> increase bufferedAmount.

Aha - that explains "increase the buffer size".  Yes, that will work, 
modulo that the app will have to track the actual bandwidth by either 
monitoring how fast the bufferedAmount drains, or via a JS API to report 
bandwidth ala the bug I opened on JS interaction with congestion 
control.  I still feel this pushes some level of congestion control into 
the JS app (and done like trying to thread needles with oven mitts on - 
ok, mild exaggeration) or the app will simply ignore this issue (likely) 
and get sub-optimal performance, perhaps significantly.  So I think more 
thought would be useful here, especially at the W3/JS level.

If we have the bandwidth reporting, having the app choose a reasonable 
amount of buffering would be much easier/simpler.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 3 Mar 2014 05:53:57 -0800

On 3 March 2014 03:44, Randell Jesup <[email-masked]> wrote:
> backwards compat for a year or two (and many won't, or won't test it, etc).
> That hurts the feature.  And today no one has ndata.

Since WebRTC is behind a flag, I don't find this particularly
convincing.  We can implement ndata.

I think that we could even remove our temporary length restriction.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 3 Mar 2014 06:37:13 -0800

On Mon, Mar 3, 2014 at 5:51 AM, Randell Jesup <[email-masked]>wrote:

>>> fast link GC pauses and other things may cause the buffer to drain out and
>>> go idle for significant periods.
>>>
>>
> each with a chunk, which will be buffered in the api layer and increase
> bufferedAmount.
>
>
> bandwidth ala the bug I opened on JS interaction with congestion control.
> I still feel this pushes some level of congestion control into the JS app
> (and done like trying to thread needles with oven mitts on - ok, mild
> exaggeration) or the app will simply ignore this issue (likely) and get
> sub-optimal performance, perhaps significantly.  So I think more thought
> would be useful here, especially at the W3/JS level.
>

In the simplest case, you could just have a medium-sized buffer of size
(max expected transfer rate * worst-case poll interval); 10 MB would be
plenty for almost all cases.

If you wanted to be smarter, you could start with 1MB buffer, poll every
100 ms, and then adjust the buffer size to be at least ((data sent since
last poll / time since last poll) * worst-case poll interval), where
worst-case poll interval, accounting for GC, is somewhere around 1 second.

I don't think this pushes CC onto apps, it just pushes spooling onto them,
which is the consequence of not having streams yet.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 3 Mar 2014 06:41:08 -0800

I should have said prefix.

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 14:41:46 +0000

On 26 Feb 2014, at 22:21, Christer Holmberg <[email-masked]> wrote:

> Hi,
> 
>>>> * Section 6.5
>>>> 
>>>> This section contains:
>>> 
>>>    Data channels can be opened by using internal or external
>>>    negotiation.  The details are out of scope of this document.
>>> 
>>>    A simple protocol for internal negotiation is specified in
>>>    [I-D.ietf-rtcweb-data-protocol] and MUST be supported.
>>> 
>>>> But internal and external negotiation are not defined in this document.
>>>> 
>>> 
>>>> I *thought* that internal negotiation was by definition negotiation 
>>> by use of  > rtcweb-data-protocol. 
>>> (draft-ejzak-mmusic-data-channel-sdpneg-00
>>> thinks so too,
>>>> but calls it "in-band negotiation".) There should be a good 
>>> definition of these  > terms, or reference to one. And more discussion 
>>> if there can be other kinds of  > internal negotiation. (If so, how 
>>> would one be chosen?)
>>> 
>>> Well, the text above says that ietf-rtcweb-data-protocol specifies an 
>>> internal negotiation protocol, so it's not that far off.  The split 
>>> does allow someone to use an alternative negotiation protocol 
>>> (internal or external).
>>> 
>>> How about:
>>> 
>>>    Data channels can be opened by using negotiation within the SCTP 
>>> association or external
>>>    negotiation.  External negotiation is defined as any method which 
>>> results in an agreement
>>>    as to the parameters of a channel and the creation thereof.
>>>    The details are out of scope of this document.
>>> 
>>>    A simple protocol for negotiation within the SCTP association is 
>>> specified in
>>>    [I-D.ietf-rtcweb-data-protocol] and MUST be supported.
>>> 
>>> Perhaps someone can wordsmith "external negotiation" better.
>> 
>> ISTM the issue is that Internal vs. External isn't the key distinction. 
>> What is key is that there is one method that MUST be *supported* but need not be the only one >used. Other mechanisms may be used. It doesn't really matter whether they are conducted >internally (over the SCTP association) or externally (anything else).
> 
> I think the MUST-be-supported-method is a separate question.
> 
>> E.g., I could use DCEP to establish one channel, and then use my own private protocol over that to >negotiate other channels. Would that be internal or external negotiation? Does it matter?
> 
> I guess they would both be in-band methods. Maybe using in-band and out-of-band terminology would be more clear - and more aligned with existing terminology.
OK, changed to in-band and out-of-band.

Best regards
Michael
> Regards,
> Christer

---

From: "Karl Stahl" <[email-masked]>
Date: Mon, 3 Mar 2014 16:06:22 +0100

Från: rtcweb [mailto:[email-masked]] För Justin Uberti
Skickat: den 3 mars 2014 15:37
Till: Randell Jesup
Kopia: [email-masked]
Ämne: Re: [rtcweb] Open data channel issues

On Mon, Mar 3, 2014 at 5:51 AM, Randell Jesup <[email-masked]> wrote:

On Mon, Mar 3, 2014 at 3:44 AM, Randell Jesup <[email-masked]> wrote:

On Sun, Mar 2, 2014 at 6:42 PM, Randell Jesup <[email-masked]> wrote:

Since the amount buffered is up to the app, the app can just increase the buffer size to provide enough data to cover the next second or so of transfer. 

How does the app increase the size of the internal SCTP buffers?  (And that buffer is shared by all channels, of course).

What I meant is the app can make multiple calls to send() at a time, each with a chunk, which will be buffered in the api layer and increase bufferedAmount.

Aha - that explains "increase the buffer size".  Yes, that will work, modulo that the app will have to track the actual bandwidth by either monitoring how fast the bufferedAmount drains, or via a JS API to report bandwidth ala the bug I opened on JS interaction with congestion control.  I still feel this pushes some level of congestion control into the JS app (and done like trying to thread needles with oven mitts on - ok, mild exaggeration) or the app will simply ignore this issue (likely) and get sub-optimal performance, perhaps significantly.  So I think more thought would be useful here, especially at the W3/JS level.

If we have the bandwidth reporting, having the app choose a reasonable amount of buffering would be much easier/simpler.

In the simplest case, you could just have a medium-sized buffer of size (max expected transfer rate * worst-case poll interval); 10 MB would be plenty for almost all cases.

If you wanted to be smarter, you could start with 1MB buffer, poll every 100 ms, and then adjust the buffer size to be at least ((data sent since last poll / time since last poll) * worst-case poll interval), where worst-case poll interval, accounting for GC, is somewhere around 1 second. 

I don't think this pushes CC onto apps, it just pushes spooling onto them, which is the consequence of not having streams yet.

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Mar 2014 10:26:04 -0500

> convincing.  We can implement ndata.

I don't understand the "behind a flag" comment; an app using 
DataChannels for anything variable-length (unless with low bounds) must 
care until all old implementations are gone.  Since we already have 
DataChannel impls in  ESR releases, there's no avoiding this if we rely 
on that (even worse, since this undefined property doesn't exist yet 
(let alone in ESR), the app has to handle that case as well).  Or add 
browser and version sniffing to all the apps...

> I think that we could even remove our temporary length restriction.

Sure, we can.  This will reduce or eliminate source changes for apps 
that don't care about interleaving, which is a plus.  It will cause 
total stall of all other DataChannels during  transmission of one large 
message, which some applications would be ok with, and others would not, 
and so those that would not would have to somehow deal with it.

So, applications that are ok with blocking other channels with large 
sends could be allowed to send arbitrary-sized messages, and receivers 
would be required to receive them (without chunking). Applications that 
do care about other channels would need to restrict their sending size 
when ndata isn't available at both ends.  We really only need to know if 
ndata is available and expose that fact to the application, not a 
receive size from the receiver.

So this plan would be:
1) We expose ndata being agreed to (currently == false);
2) We turn off the PPID chunking in Firefox (and live with ESR24 and 
current revs of Firefox not being compatible; see 2a)
2a) we make sure the app can detect if the local client has the ndata 
property at all (supports this solution).
3) We make sure Chrome and Firefox both implement both EOR sending and 
EOR reception and unlimited (or virtually) send/receive sizes.

For bonus points, on EOR reception of a blob, spool it to disk in the 
browser (if over some limit perhaps), and on EOR send of a blob, 
optimize it to avoid send-side memory hits and performance issues.

This will break large send/receive between Chrome and current/older 
Firefox - but those are broken today by Chrome not implement PPID 
chunking and Firefox doing so.  Firefox can still fall back on PPID 
chunking if it knows it's talking to another Firefox until we decide 
that this no longer matters.  I could be wrong, but I think Chrome 
doesn't support EOR sending currently; I know Firefox doesn't since we 
were using PPID chunking, so we'd need to add that.

Once ndata is supported, all will work well, and applications that 
didn't care about interleaving don't have to change, and those that do 
will trigger on ndata and switch (and they can test ndata==true mode 
reasonably well by forcing it in the app, since send(large) will work).

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 15:38:50 +0000

On 26 Feb 2014, at 21:02, Randell Jesup <[email-masked]> wrote:

> On 2/26/2014 2:50 PM, Paul Kyzivat wrote:
>> Michael,
>> 
>> There have been no replies to my comments:
>> 
>> http://www.ietf.org/mail-archive/web/rtcweb/current/msg11518.html
>> http://www.ietf.org/mail-archive/web/rtcweb/current/msg11519.html
>> 
>> The first of those is about channels, the 2nd about DCEP. One of my issues is that the partitioning between those two drafts is awkward. At the least some things from DCEP should be moved to the channel draft. It might be better to just merge those two drafts, or do a serious refactoring.
> 
> There seem to have been a number of responses to your second posting at least.
> 
>> How about:
> 
>   Each SCTP user message contains a so called Payload Protocol
>   Identifier (PPID) that is passed to SCTP by the data channel layer
>   and sent to its peer.  This value is used to multiplex WebRTC Data
>   Channel Establishment Protocol messages (defined in [I-D.ietf-rtcweb-
>   data-protocol]) with messages containing user data on a data
>   channel. The PPID is also used to distinguish UTF-8 encoded user
>   data and binary encoded user data.
> 
> That seems reasonable and clearer.
> 
> 
>> * Section 6.5
>> 
>> This section contains:
> 
> > I *thought* that internal negotiation was by definition negotiation by use of
> > rtcweb-data-protocol. (draft-ejzak-mmusic-data-channel-sdpneg-00 thinks so too,
> > but calls it "in-band negotiation".) There should be a good definition of these
> > terms, or reference to one. And more discussion if there can be other kinds of
> > internal negotiation. (If so, how would one be chosen?)
> 
> Well, the text above says that ietf-rtcweb-data-protocol specifies an internal negotiation protocol,
> so it's not that far off.  The split does allow someone to use an alternative negotiation protocol
> (internal or external).
> 
> How about:
> 
>   Data channels can be opened by using negotiation within the SCTP association or external
>   negotiation.  External negotiation is defined as any method which results in an agreement
>   A simple protocol for negotiation within the SCTP association is specified in
Taken, but with in-band and out-of-band...
> 
> 
>> * Section 6.6:
>> 
>> Say:
> 
>   All data sent on a Channel in both directions MUST be sent over the
>   underlying stream using the reliability defined when the Channel was
>   opened unless the options are changed, or per-message options are
>   specified by a higher level.
> 
> > Is the recipient to consider it an error if messages are received with options different from those
> > defined for the channel? Also, is it an error if messages are received with a PPID that isn't specified
> > in Section 8? (And what about PPID 50?)
> 
>> How are such channel errors to be treated?
> 
> 
> I'd propose that if messages are received with options that don't match the initial definition, that fact should be ignored and the message processed.
> If messages with an unknown PID are received, those messages should be ignored.
I'll bring that up at the WebRTC session.

Best regards
Michael
> 
> In both cases, logging of such an event to the application (or onerror calls in WebRTC W3 JS layers) is allowed but not required. Note that the onerror callback is currently under-defined, so this might change.
> 
> --

---

From: Martin Thomson <[email-masked]>
Date: Mon, 3 Mar 2014 09:26:50 -0800

On 3 March 2014 07:26, Randell Jesup <[email-masked]> wrote:
> I don't understand the "behind a flag" comment; an app using DataChannels
> for anything variable-length (unless with low bounds) must care until all
> old implementations are gone.

My point was that these people who you want to protect have already
renounced rights to protection by using the prefixed API.  Don't use
that as a basis for argument.  We should concentrate on building the
functionality we want to have.

---

From: Justin Uberti <[email-masked]>
Date: Mon, 3 Mar 2014 09:54:28 -0800

On Mon, Mar 3, 2014 at 7:26 AM, Randell Jesup <[email-masked]>wrote:

>>> etc).
> old implementations are gone.  Since we already have DataChannel impls in
>  ESR releases, there's no avoiding this if we rely on that (even worse,
> since this undefined property doesn't exist yet (let alone in ESR), the app
> has to handle that case as well).  Or add browser and version sniffing to
> all the apps...
>
>
> Sure, we can.  This will reduce or eliminate source changes for apps that
> don't care about interleaving, which is a plus.  It will cause total stall
> of all other DataChannels during  transmission of one large message, which
> some applications would be ok with, and others would not, and so those that
> would not would have to somehow deal with it.
>
> So, applications that are ok with blocking other channels with large sends
> could be allowed to send arbitrary-sized messages, and receivers would be
> required to receive them (without chunking). Applications that do care
> about other channels would need to restrict their sending size when ndata
> isn't available at both ends.  We really only need to know if ndata is
> available and expose that fact to the application, not a receive size from
> the receiver.
>
> So this plan would be:
> 3) We make sure Chrome and Firefox both implement both EOR sending and EOR
> reception and unlimited (or virtually) send/receive sizes.
>
> browser (if over some limit perhaps), and on EOR send of a blob, optimize
> it to avoid send-side memory hits and performance issues.
>
> Firefox - but those are broken today by Chrome not implement PPID chunking
> and Firefox doing so.  Firefox can still fall back on PPID chunking if it
> knows it's talking to another Firefox until we decide that this no longer
> matters.  I could be wrong, but I think Chrome doesn't support EOR sending
> currently; I know Firefox doesn't since we were using PPID chunking, so
> we'd need to add that.
>
> Once ndata is supported, all will work well, and applications that didn't
> care about interleaving don't have to change, and those that do will
> trigger on ndata and switch (and they can test ndata==true mode reasonably
> well by forcing it in the app, since send(large) will work).

While I agree with the final outcome of #3, I don't think going down the
1/2/2a path makes sense, since it leads to an additional intermediate state
over the current situation. i.e. we end up with 3 states:
a) today: max safe send is 16 KB
b) soon: ndata exists, but large sends lock out other sends
c) final: ndata exists, and large sends work OK

I would rather maintain a) or similar for now (using the max-message-size
param) and jump to c) when we have the right stuff in SCTP.

---

From: "Makaraju, Maridi Raju (Raju)" <[email-masked]>
Date: Mon, 3 Mar 2014 22:19:23 +0000

Sorry to jump in from middle of nowhere in this interesting conversation. I couldn’t resist providing this feedback and I hope it is useful.

First, on “EOR solving monopolization issue”:
I could be wrong, but I am not sure how EOR based approach avoids SCTP link monopolization?! Per my understanding EOR is a local send () utility rather than receive side utility and it is implemented on top of base SCTP RFC (so no indication of special EOR in SCTP headers other than base SCTP RFC B/E fragment indication bits). Once a send () with EOR set to false is called, on a stream id, the SCTP link is monopolized until send () on same stream id with EOR set to true is called; so same limitation exist.  EOR allows sending side to start sending chunks without waiting for complete data. EOR has no impact on receiving side, the SCTP stack waits for last chunk (which is the result of sending side setting EOR=true) and delivers entire message to app.

New PC approach:
If an application is sensitive to SCTP link monopolisation then there is another option it can consider – use of separate new PeerConnection (PC) for the data channel(s) which may monopolize the link.
Depending on how an application is designed, using a new PC could be simpler than having an application level chunking protocol (fragmentation and reassembly).  If app needs such “heavy” data channel infrequently then it can create new PC as needed; if it is needed frequently then a dedicated PC can be allocated and keep it open all the time.

One disadvantage with application level chunking protocol is both the peers need to use it; so it may not work between clients of different types or same clients with different versions. New PC based approach obviously does not have this issue as no chunking is involved.

At first, new PC approach may appear heavy weight wrt # of PCs in use (which I don’t think an issue on browsers), ICE overhead for new PC etc. But in comparision to application level chunking new PC approach may not be that bad.

Please see additional comments inline.

-Raju

From: rtcweb [mailto:[email-masked]] On Behalf Of Justin Uberti
Sent: Monday, March 03, 2014 11:54 AM
To: Randell Jesup
Cc: [email-masked]
Subject: Re: [rtcweb] Open data channel issues

On Mon, Mar 3, 2014 at 7:26 AM, Randell Jesup <[email-masked]<mailto:[email-masked]>> wrote:
On 3/3/2014 8:53 AM, Martin Thomson wrote:
On 3 March 2014 03:44, Randell Jesup <[email-masked]<mailto:[email-masked]>> wrote:
Hopefully so, though I hate forcing apps to include a bunch of code for
backwards compat for a year or two (and many won't, or won't test it, etc).
That hurts the feature.  And today no one has ndata.
Since WebRTC is behind a flag, I don't find this particularly
convincing.  We can implement ndata.

I don't understand the "behind a flag" comment; an app using DataChannels for anything variable-length (unless with low bounds) must care until all old implementations are gone.  Since we already have DataChannel impls in  ESR releases, there's no avoiding this if we rely on that (even worse, since this undefined property doesn't exist yet (let alone in ESR), the app has to handle that case as well).  Or add browser and version sniffing to all the apps...

Sure, we can.  This will reduce or eliminate source changes for apps that don't care about interleaving, which is a plus.  It will cause total stall of all other DataChannels during  transmission of one large message, which some applications would be ok with, and others would not, and so those that would not would have to somehow deal with it.

So, applications that are ok with blocking other channels with large sends could be allowed to send arbitrary-sized messages, and receivers would be required to receive them (without chunking). Applications that do care about other channels would need to restrict their sending size when ndata isn't available at both ends.  We really only need to know if ndata is available and expose that fact to the application, not a receive size from the receiver.

So this plan would be:
1) We expose ndata being agreed to (currently == false);
2) We turn off the PPID chunking in Firefox (and live with ESR24 and current revs of Firefox not being compatible; see 2a)
2a) we make sure the app can detect if the local client has the ndata property at all (supports this solution).
3) We make sure Chrome and Firefox both implement both EOR sending and EOR reception and unlimited (or virtually) send/receive sizes.
[Raju] This requires PeerConnection API to indicate EOR during send(). Right? Which may be useful in some cases where app wants to send partial data as it is available/ready. But as explained above, in my opinion EOR won’t solve SCTP link monopolization and unncessarily introduces overhead at application.

For bonus points, on EOR reception of a blob, spool it to disk in the browser (if over some limit perhaps), and on EOR send of a blob, optimize it to avoid send-side memory hits and performance issues.

This will break large send/receive between Chrome and current/older Firefox - but those are broken today by Chrome not implement PPID chunking and Firefox doing so.  Firefox can still fall back on PPID chunking if it knows it's talking to another Firefox until we decide that this no longer matters.  I could be wrong, but I think Chrome doesn't support EOR sending currently; I know Firefox doesn't since we were using PPID chunking, so we'd need to add that.

Once ndata is supported, all will work well, and applications that didn't care about interleaving don't have to change, and those that do will trigger on ndata and switch (and they can test ndata==true mode reasonably well by forcing it in the app, since send(large) will work).

While I agree with the final outcome of #3, I don't think going down the 1/2/2a path makes
sense, since it leads to an additional intermediate state over the current situation. i.e. we end up with 3 states:
a) today: max safe send is 16 KB
b) soon: ndata exists, but large sends lock out other sends
c) final: ndata exists, and large sends work OK

I would rather maintain a) or similar for now (using the max-message-size param) and jump to c) when we have the right stuff in SCTP.
[Raju] With 16KB safe size won’t applications have to have their own chunking protocol? as per my understanding EOR won’t solve the monopolization issue, so it adds no value.

To avoid applications coming up with their own chunking protocol, an attractive alternative would be to use the PPID chunking that’s already mentioned abovr and implemented by Firefox. Then the issue of interworking pushed down to browser from application area. But this obviously requires browsers on both the ends to support this PPID based chunking.

-Raju

--
Randell Jesup -- rjesup a t mozilla d o t com
_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 3 Mar 2014 22:45:36 +0000

> I could be wrong, but I am not sure how EOR based approach avoids SCTP link monopolization?! Per my understanding EOR is a local send () utility rather than receive side utility and it is implement…
You are right, if you consider RFC 4960. But using
http://tools.ietf.org/html/draft-ietf-tsvwg-sctp-ndata-00
overcomes this limitation. You can have on each stream a message being sent in explicit EOR mode.
Using an interleaving scheduler, they get interleaved on the wire. So you need the NDATA extension...
> At first, new PC approach may appear heavy weight wrt # of PCs in use (which I don’t think an issue on browsers), ICE overhead for new PC etc. But in comparision to application level chunking new P…
I think the large message support with NDATA is sufficient to do whatever RTCWeb wants...
> On Mon, Mar 3, 2014 at 7:26 AM, Randell Jesup <[email-masked]> wrote:
At the API to the SCTP stack you need EOR support. But this API is not exposed to the JS user.
The JS user can just call send(bigmessage) and it works...
> To avoid applications coming up with their own chunking protocol, an attractive alternative would be to use the PPID chunking that’s already mentioned abovr and implemented by Firefox. Then the iss…
... correct. And at the last RTCWeb session it was decided to avoid this intermediate step and
just use NDATA...

Best regards
Michael

---

From: Randell Jesup <[email-masked]>
Date: Mon, 03 Mar 2014 22:38:16 -0500

On 3/3/2014 12:54 PM, Justin Uberti wrote:
>     DataChannels for anything variable-length (unless with low bounds)
>     must care until all old implementations are gone.  Since we
>     already have DataChannel impls in  ESR releases, there's no
>     avoiding this if we rely on that (even worse, since this undefined
>     property doesn't exist yet (let alone in ESR), the app has to
>     handle that case as well).  Or add browser and version sniffing to
>     all the apps...
>
>
>     Sure, we can.  This will reduce or eliminate source changes for
>     apps that don't care about interleaving, which is a plus.  It will
>     cause total stall of all other DataChannels during  transmission
>     of one large message, which some applications would be ok with,
>     and others would not, and so those that would not would have to
>     somehow deal with it.
>
>     So, applications that are ok with blocking other channels with
>     large sends could be allowed to send arbitrary-sized messages, and
>     receivers would be required to receive them (without chunking).
>     Applications that do care about other channels would need to
>     restrict their sending size when ndata isn't available at both
>     ends.  We really only need to know if ndata is available and
>     expose that fact to the application, not a receive size from the
>     receiver.
>
>     So this plan would be:
>     2) We turn off the PPID chunking in Firefox (and live with ESR24
>     and current revs of Firefox not being compatible; see 2a)
>     2a) we make sure the app can detect if the local client has the
>     ndata property at all (supports this solution).
>     3) We make sure Chrome and Firefox both implement both EOR sending
>     and EOR reception and unlimited (or virtually) send/receive sizes.
>
>     For bonus points, on EOR reception of a blob, spool it to disk in
>     the browser (if over some limit perhaps), and on EOR send of a
>     blob, optimize it to avoid send-side memory hits and performance
>     issues.
>
>     This will break large send/receive between Chrome and
>     current/older Firefox - but those are broken today by Chrome not
>     implement PPID chunking and Firefox doing so.  Firefox can still
>     fall back on PPID chunking if it knows it's talking to another
>     Firefox until we decide that this no longer matters.  I could be
>     wrong, but I think Chrome doesn't support EOR sending currently; I
>     know Firefox doesn't since we were using PPID chunking, so we'd
>     need to add that.
>
>     didn't care about interleaving don't have to change, and those
>     that do will trigger on ndata and switch (and they can test
>     ndata==true mode reasonably well by forcing it in the app, since
>     send(large) will work).
>
>
> While I agree with the final outcome of #3, I don't think going down 
> the 1/2/2a path makes sense, since it leads to an additional 
> intermediate state over the current situation. i.e. we end up with 3 
> states:
> I would rather maintain a) or similar for now (using the 
> max-message-size param) and jump to c) when we have the right stuff in 
> SCTP.

So, is there any significant advantage to using a max-message-size param 
(with >16K) as an interim step?  You get 3 similar steps (though perhaps 
with a smaller lockout).

To take your suggestion:
a) applications assume 16KB if ndata wasn't negotiated (per the current 
suggestion)
b) implement ndata and EOR support in the browsers, and drop the 16KB 
limit when ndata is negotiated.
All this needs is "is ndata supported by this association" exposed to 
the app; no sdp changes needed.  (One less dependency!)

---

From: Martin Thomson <[email-masked]>
Date: Tue, 4 Mar 2014 07:00:18 +0000

On 4 March 2014 03:38, Randell Jesup <[email-masked]> wrote:
> b) implement ndata and EOR support in the browsers, and drop the 16KB limit
> when ndata is negotiated.
> All this needs is "is ndata supported by this association" exposed to the
> app; no sdp changes needed.  (One less dependency!)

Or, to avoid vestigal crap in the API in the end state, when you
install the 16K limitation, add an indicator that applications can
check.  Once you remove the cap, remove the indicator.

if (dataChannel.isNotVeryGood) {
  chunkMessage(message);
} else {
  dataChannel.send(message);
}

---

From: Randell Jesup <[email-masked]>
Date: Tue, 04 Mar 2014 02:30:11 -0500

>    dataChannel.send(message);

Yup, that works for me.  dataChannel.requiresFragmentation would be the 
wordy version or mustFragmentAt16K for self-documentation.

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 4 Mar 2014 07:55:25 +0000

On 04 Mar 2014, at 07:30, Randell Jesup <[email-masked]> wrote:

> Yup, that works for me.  dataChannel.requiresFragmentation would be the wordy version or mustFragmentAt16K for self-documentation.
Or you return a number which gives you the limit if you don't want to specify the
16K limit...

Best regards
Michael

---

From: Randell Jesup <[email-masked]>
Date: Tue, 04 Mar 2014 04:00:39 -0500

>>>    dataChannel.send(message);

Wait, the reason why we had it be 16K as a default was that current 
impls don't have the property.  So datachannel.isntFixed (ok, better 
names can be found - but that's W3 anyways.

---

From: Martin Thomson <[email-masked]>
Date: Tue, 4 Mar 2014 09:09:36 +0000

On 4 March 2014 09:00, Randell Jesup <[email-masked]> wrote:
> Wait, the reason why we had it be 16K as a default was that current impls
> don't have the property.  So datachannel.isntFixed (ok, better names can be
> found - but that's W3 anyways.

Some current implementations don't have a limit either.

We could spend lots of our precious cycles on fixing shit that's both
broken and unchangeable.  Or, we could try to design the thing that we
actually want.

---

From: "Makaraju, Maridi Raju (Raju)" <[email-masked]>
Date: Tue, 4 Mar 2014 11:17:19 +0000

Please see inline comments.

> -----Original Message-----
> From: Michael Tuexen [mailto:[email-masked]]
> Sent: Monday, March 03, 2014 4:46 PM
> To: Makaraju, Maridi Raju (Raju)
> Cc: Justin Uberti; Randell Jesup; [email-masked]
> On 03 Mar 2014, at 22:19, Makaraju, Maridi Raju (Raju)
> <[email-masked]> wrote:
> 
> > Sorry to jump in from middle of nowhere in this interesting conversation.
> I couldn't resist providing this feedback and I hope it is useful.
> >
> > First, on "EOR solving monopolization issue":
> > I could be wrong, but I am not sure how EOR based approach avoids SCTP
> link monopolization?! Per my understanding EOR is a local send () utility
> rather than receive side utility and it is implemented on top of base SCTP
> RFC (so no indication of special EOR in SCTP headers other than base SCTP
> RFC B/E fragment indication bits). Once a send () with EOR set to false is
> called, on a stream id, the SCTP link is monopolized until send () on same
> stream id with EOR set to true is called; so same limitation exist.  EOR
> allows sending side to start sending chunks without waiting for complete
> data. EOR has no impact on receiving side, the SCTP stack waits for last
> chunk (which is the result of sending side setting EOR=true) and delivers
> entire message to app.

[Raju] 
Thanks for clarifying. Earlier it wasn't clear to me as EOR was referred to be EOR+NDATA. I was reading EOR as just EOR support without NDATA.
Yes, NDATA support is a must to avoid monopolization.
[/Raju]

> >
> > New PC approach:
> > If an application is sensitive to SCTP link monopolisation then there is
> another option it can consider - use of separate new PeerConnection (PC) for
> the data channel(s) which may monopolize the link.
> > Depending on how an application is designed, using a new PC could be
> simpler than having an application level chunking protocol (fragmentation
> and reassembly).  If app needs such "heavy" data channel infrequently then
> it can create new PC as needed; if it is needed frequently then a dedicated
> PC can be allocated and keep it open all the time.
> >
> > One disadvantage with application level chunking protocol is both the
> peers need to use it; so it may not work between clients of different types
> or same clients with different versions. New PC based approach obviously
> does not have this issue as no chunking is involved.
> >
> > At first, new PC approach may appear heavy weight wrt # of PCs in use
> (which I don't think an issue on browsers), ICE overhead for new PC etc. But
> in comparision to application level chunking new PC approach may not be that
> bad.

[Raju] 
Yes, I agree that NDATA is the final solution. But in the interim, application chunking per 16K size requires some chunking protocol at app.
Do we know the timeframe for NDATA support from browsers?
Until NDATA is available, to avoid monopolization one can use new PC as an alternative to application level chunking (which also require some kind of flow/congestion control).
Appreictae any comments on this approach.
[/Raju]

Best Regards
Raju

---

From: "Makaraju, Maridi Raju (Raju)" <[email-masked]>
Date: Tue, 4 Mar 2014 11:39:53 +0000

> From: rtcweb [mailto:[email-masked]] On Behalf Of Randell Jesup
> Sent: Tuesday, March 04, 2014 3:01 AM
> the

[Raju] 
NDATA requires support from both the ends. Is browser going to update datachannel.isntFixed/ndata after SCTP is up? Per my reading of PeerConnection onopen() won't be called before SCTP is up (however may be called before DCEP, if used, is done). So, app knows NDATA support before sending first message.

Thanks
Raju