Subject: Re: (SCTP) draft-ietf-tsvwg-sctpsocket-04.txt and non-blocking mode

From: Phillip Conrad <[email-masked]>
Date: Wed, 25 Sep 2002 11:26:12 -0400

La Monte and Randy raise two points that I suspected were true; I paraphrase these below (with an acknowledged "spin" that is my own, not theirs).   I would argue that this calls for at least an "implemenation/usage note"in the sockets API draft.

(I trust La Monte and Randy will speak up if I have misunderstood anything)

(1) that there is also some extra buffering that will happen in the sockets layer after the rwnd closes... this buffering has to also be exhausted before a sendmsg"() will block because of a zero rwnd.

(2) that one association's "personal problem" (i.e. a zero rwnd causing data to back up in the socket buffer) can affect the ability to send data on other associations through a shared UDP-socket.   

#2 is a new twist on the "fate sharing" principle that applies to UDP-style sockets, and I would suggest it is something we should either prevent (not likely to be feasible) or that users should be warned about.   

I suggest we just document it, because "fixing it" might require architecturally-prohibited, transport-specific  mucking around in the socket layer internals (per assoication socket buffers instead of per-socket socket buffers).    Randy is quick to remind me that such ugly hacks are unlikely to be accepted by the defenders of the faith (and rightly so.)

So here's some suggested text for the next version of the sockets API draft (where it would go in the outline is left as an exercise to the reader.)

[****start of suggested text for sockets api draft****]

Implementation and Usage Note for UDP-style sockets: 

Implementors of the sockets API are free to implement a socket serving multiple associations (e.g., a UDP-style socket) with a single socket buffer shared across all associations.   This design has consequences that users of the socket API need to take into account.

When a peer advertises a zero rwnd on a particular association, further attempts to send data to that association can eventually cause the socket's output buffer to fill up.     When this happens, any attempt to send data over that socket will either block, or return an error (depending on whether the socket is blocking or non-blocking).    If the socket is shared across multiple associations, the inability to send further data affects all associations on that socket, not just the association with the zero rwnd. 

Programmers may want to consider using a single socket per association in any case where it is important to avoid a zero rwnd on one association impacting the ability to send data on another association.

[****end of suggested text for sockets api draft****]

Cheers,
Phill

PS: If this seems like too much advice for a document that is just supposed to specify a standard for an API, I can sympathize with this point of view.  I don't think the sockets API should spell out "every little trap" that a programmer might fall into using SCTP.   We are not in the business of teaching network programming.  In particular, anything that is common to programmers experience with TCP and/or UDP is something we don't need to talk about.   

BUT... when SCTP opens up some entirely "new" kind of trap that no previous experience would prepare a programmer to avoid, I think we have a "duty to inform".   
Backpressure on a socket from a zero rwnd causing writing to block is "nothing new"... it has happened in TCP for years...  but backpressure on one association/connection affecting another association/connection on the same socket is a "new animal".   Nothing like this occurs with "plain old UDP" sockets, because there is no rwnd.     That's why I think it deserves special mention in the draft.

PPS:  Here's a question: is it even necessary for there to be a closed rwnd for the socket buffer backup problem to occur?  What if we are spraying data at high volume over a large number of associations, each of which is in slow start... could that also cause the socket buffer to gum up?  I suppose the difference is that with slow start, as long as the rwnd doesn't fill up and the network is behaving reasonably well, within an RTT or so the buffer will eventually start emptying out at some rate.  This is not true of a full rwnd on a peer that is stubbornly refusing to read, perhaps because of a bug in the receiving ULP; in that case, the backup can last indefinitely.

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 25 Sep 2002 13:36:20 -0500

>So here's some suggested text for the next version of the sockets API draft (where it would go in the outline is left as an exercise to the reader.)

Hmm I think you have it mostly right... I do want to clearify that each 
association usually has a socketbuffers worth
of buffering.. the key is that in the blocking model if I send to 
assoc-a and it blocks even though assoc-b would NOT
block (at that percise moment if I sent to it) I am stuck since .. I am 
blocked :-0

This is where the non-blocking mode would suit better... i.e. get a 
EAGAIN back and then one could still write
to asoc-b ...

>[****end of suggested text for sockets api draft****]
I think some mention of the NON-BLOCKING mode API as another solution 
would also be good here.. just saying
go to a single socket per association  does not do justice to the fact 
that there is another solution as well... aka set
non-blocking...

>PS: If this seems like too much advice for a document that is just supposed to specify a standard for an API, I can sympathize with this point of view.  I don't think the sockets API should spell ou…

Somehow I knew a BUT was comming :>

>PPS:  Here's a question: is it even necessary for there to be a closed rwnd for the socket buffer backup problem to occur?  What if we are spraying data at high volume over a large number of associa…
One could envision a socketbuffer being set to 2k and the rwnd being set 
(by the peer) to 32k. In such a situation it
becomes quite easy to fill up the socket buffer at any given time... 
even besides your "blast" of data idea... which
is also a valid way it can happen.. by the way it happens today in TCP 
too, even with a single connection.

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: "K. Poon" <[email-masked]>
Date: Wed, 25 Sep 2002 12:13:48 -0700 (PDT)

--- Phillip Conrad <[email-masked]> wrote:

[Some notes deleted.]

> Implementors of the sockets API are free to implement a socket
> serving multiple associations (e.g., a UDP-style socket) with a
> single socket buffer shared across all associations.   This design
> has consequences that users of the socket API need to take into
> account.
> 
> When a peer advertises a zero rwnd on a particular association,
> further attempts to send data to that association can eventually
> cause the socket's output buffer to fill up.     When this happens,
> any attempt to send data over that socket will either block, or
> return an error (depending on whether the socket is blocking or
> non-blocking).    If the socket is shared across multiple
> associations, the inability to send further data affects all
> associations on that socket, not just the association with the zero
> rwnd. 
> 
> Programmers may want to consider using a single socket per
> association in any case where it is important to avoid a zero rwnd on
> one association impacting the ability to send data on another
> association.
> 
> PS: If this seems like too much advice for a document that is just
> supposed to specify a standard for an API, I can sympathize with this
> point of view.  I don't think the sockets API should spell out "every
> little trap" that a programmer might fall into using SCTP.   We are
> not in the business of teaching network programming.  In particular,
> anything that is common to programmers experience with TCP and/or UDP
> is something we don't need to talk about.   

Another thing I don't like is the mention of the internal working
of an implementation and the protocol (rwnd).  I think the draft
should stay away from that.

> BUT... when SCTP opens up some entirely "new" kind of trap that no
> previous experience would prepare a programmer to avoid, I think we
> have a "duty to inform".   
> Backpressure on a socket from a zero rwnd causing writing to block is
> "nothing new"... it has happened in TCP for years...  but
> backpressure on one association/connection affecting another
> association/connection on the same socket is a "new animal".  
> Nothing like this occurs with "plain old UDP" sockets, because there
> is no rwnd.     That's why I think it deserves special mention in the
> draft.

Another way is to convey this is to stress the point of blocking
and non-blocking of socket call, and the return error code.  This
is an API draft, so we should stick to an API specification.

> PPS:  Here's a question: is it even necessary for there to be a
> closed rwnd for the socket buffer backup problem to occur?  What if
> we are spraying data at high volume over a large number of
> associations, each of which is in slow start... could that also cause
> the socket buffer to gum up?  I suppose the difference is that with
> slow start, as long as the rwnd doesn't fill up and the network is
> behaving reasonably well, within an RTT or so the buffer will
> eventually start emptying out at some rate.  This is not true of a
> full rwnd on a peer that is stubbornly refusing to read, perhaps
> because of a bug in the receiving ULP; in that case, the backup can
> last indefinitely. 

Well, this is a bug in the app's protocol...  I guess we cannot
do anything.  This can happen right now with any TCP app too.

=====
K. Poon.				[email-masked]

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com