Subject: (SCTP) draft-ietf-tsvwg-sctpsocket-04.txt and non-blocking mode

From: Marco Molteni <[email-masked]>
Date: Sat, 21 Sep 2002 17:15:30 +0200

[Bcc to [email-masked], please follow-up on [email-masked]]

Hi all, the following refers to the SCTP/KAME implementation.

The SCTP socket API draft in section "3.3 Non-blocking mode" says:

    Once all bind() calls are complete on a UDP-style socket, the
    application must set the non-blocking option by a fcntl() (such as
    O_NONBLOCK).

I read that as meaning that the default mode is blocking. So I wrote my
test program for a UDP-style SCTP socket, I called recvmsg() and I got
back EAGAIN! I started trying all kinds of modifications to the program,
and then dig into the SCTP/KAME source code.

After two long days, I found some comments in sctp.h that say that
blocking is the default for TCP-style, and non-blocking is the default
for UDP-style.

Please update the draft to reflect this behaviour.

Also, the code example in the draft will loop forever, since it expects
a blocking socket.

Also, what is the proper way to set blocking an UDP-style socket?
What I did is the following:

    if (fcntl(fd, F_SETFL, 0) < 0) { 
        err(1, "fcntl"); 
    } 

which is ugly. Or maybe, given the semantics, it simply doesn't make sense
to have a blocking UDP-style socket?

marco

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Sat, 21 Sep 2002 12:32:18 -0500

> Please update the draft to reflect this behaviour.
Not sure.. maybe we should update the implementation :>

>     if (fcntl(fd, F_SETFL, 0) < 0) {

I think that you can also do

int off=0;

ioctl(sd,FIONBIO,&off);

or to turn it on

int on=1;

ioctl(sd,FIONBIO,&on);

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: Marco Molteni <[email-masked]>
Date: Mon, 23 Sep 2002 09:44:32 +0200

> Marco Molteni wrote:

[..]

> > I read that as meaning that the default mode is blocking. So I wrote
> > my test program for a UDP-style SCTP socket, I called recvmsg() and
> > I got back EAGAIN! I started trying all kinds of modifications to
> > the program, and then dig into the SCTP/KAME source code.
> > 
> > blocking is the default for TCP-style, and non-blocking is the
> > default for UDP-style.
> > 

Hi Randall,

sure, I made that suggestion because I though that the implementation
was more updated than the draft.

Thanks for the examples on how to set the socket blocking.

ciao
Marco

---

From: "K. Poon" <[email-masked]>
Date: Mon, 23 Sep 2002 12:38:47 -0700 (PDT)

--- Marco Molteni <[email-masked]> wrote:

> my

So it seems to be a bug in the SCTP/KAME implementation.  It
should be in blocking mode by default.

> Please update the draft to reflect this behaviour.

I guess instead of updating the draft, a bug should be filed
against the SCTP/KAME implementation...

> to have a blocking UDP-style socket?

I guess the reason for blocking mode by default is to follow
the traditional semantics of all socket calls, which are all
in blocking mode.  And whether a blcoking or non-blocking mode
makes more sense really depends on the context.  So the simple
solution is to just pick one which conforms to all other socket
calls.

=====
K. Poon.				[email-masked]

__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Mon, 23 Sep 2002 18:57:59 -0500

K:

Here is a question for you... can you make a UDP send() call block?

R

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Mon, 23 Sep 2002 22:10:05 -0500

Of course!  If you consume the buffer space available to a given UDP
socket, it should block until the IP layer has cleared some buffer
space.

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Mon, 23 Sep 2002 22:15:55 -0500

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Tue, 24 Sep 2002 06:36:52 -0500

Hmmm... I don't think that any buffer space is consumed by
a udp send call. It is put on the wire when the call is made
and never is it added to the output socket buffer.

The only way I can imagine it every blocking is ONLY if one chews up
all the MBUF's (say in some tcp connections) and then trys the send. I 
believe
the MGET() call is setup to block in this case....

But of course if you chew up all the mbufs in a system things are so 
sick I am
not to sure the system would survive :-0

R

---

From: Marco Molteni <[email-masked]>
Date: Tue, 24 Sep 2002 13:58:06 +0200

> Here is a question for you... can you make a UDP send() call block?

Randall,

just to be sure, in my previous email I was talking about revcmsg(), not
sendmsg().

Marco

-- 
Computers are like air conditioners.
They stop working when you open Windows.

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Tue, 24 Sep 2002 07:28:04 -0500

Marco:

I figured you were talking about recv-msg.. but this is the real crux of 
the matter.
In practice UDP sockets WILL block on a recvmsg but WON'T block on a 
sendmsg.

The reason being that of course you block when you read.. but unless the 
system is under
sever conditions (out of mbufs really does not happen unless something 
is very very wrong)
a sendto() will never block. This is kind of an issue...IMO in that in a 
TCP app I would expect
to block in a sendto() possibly.. but will this blocking surprise any 
UDP style model?

I don't know the answer but it is the reason I made the default UDP side 
do a non-blocking
model (which is a one line code change by the way :>)

R

---

From: Marco Molteni <[email-masked]>
Date: Tue, 24 Sep 2002 14:55:58 +0200

On Tue, 24 Sep 2002, "Randall Stewart (cisco)" <[email-masked]> wrote:

> Marco:
> 
> I figured you were talking about recv-msg.. but this is the real crux
> of the matter.
> The reason being that of course you block when you read.. but unless
> the system is under sever conditions (out of mbufs really does not happen
> unless something is very very wrong) a sendto() will never block.
> This is kind of an issue...IMO in that in a TCP app I would expect

Randall,

I may be missing something, but what's wrong with the following:

If I set a socket (who cares if TCP/UDP/SCTP) in blocking mode, I
do _not_ expect a read/write syscall to block just because the socket is
in blocking mode, what I expect is a read/write syscall to block _only_ if
there isn't data available to read or if there isn't buffer space to
write.

This is what happens with BSD kernels for UDP sockets, as you said: a
write, in normal system operation, is always successful without blocking.

> I don't know the answer but it is the reason I made the default UDP
> side do a non-blocking

Marco

---

From: Jon Grimm <[email-masked]>
Date: Tue, 24 Sep 2002 08:14:43 -0500

Hi Randy/Marco/La Monte/Everyone:

> to block in a sendto() possibly.. but will this blocking surprise any

Or stated in another way... will receiving an EAGAIN error from 
sendto() suprise any UDP-style model in the face of SCTP output buffers 
filling up?   ;-)   
  
Its not a clean fit either way...SCTP is so much more than UDP that its
difficult to
get the absolute behavior of SOCK_DGRAM (Is this why SOCK_DGRAM was not
used?)   

One thing we touched on a bit out at the Bakeoff and should probably
spin into its own thread if folks think interesting is that of the
UDP-style API itself:  Is the UDP-style 
API goal really towards a programming model or towards SOCK_DGRAM
transparency?  
(or likely even a little of both).    

> model (which is a one line code change by the way :>)

IIRC, we default to blocking behavior since that is normal default
sockets configuration.  

Best Regards,
Jon

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Tue, 24 Sep 2002 08:43:17 -0500

>there isn't data available to read or if there isn't buffer space to
If in TCP you have blocking turned on and you do a send/write call you 
WILL block
if there is not room in the socket buffer. This does not occur in a UDP 
model
since the data is never kept in the socket buffer and hits the wire 
right away (or
is dropped).

So this IS fundamentally different in a Udp style..

R

---

From: Phillip Conrad <[email-masked]>
Date: Tue, 24 Sep 2002 10:30:26 -0400

Randy,
  Your question (can a UDP socket ever block on a write() ) misses an important point, and illustrates the problem with the use of the term "UDP style" socket.    A true UDP socket is "many to one" (many associations, one socket).  This is the main similarity with a so-called UDP-style SCTP socket.    BUT that UDP socket also is unreliable and not flow controlled.

With a *true* UDP socket, there is no flow control, and there is no guarantee of reliability.   It is reasonable to say that there is really no good reason for a UDP socket to block on a write(), because if there is no buffer space left, there is no problem with just throwing the write away!   (After all, the protocol is "best effort"... if there is no buffer space, oh well!)    This may or may not be what real UDP implementations do, but if they did, there would be no  mismatch between the API and the protocol semantics.

But in a reliable, flow controlled protocol, such as SCTP, a write() should either block or fail if there is no rwnd() space.   This is what provides backpressure to applications to let them know to stop sending when the receiver is not consuming data fast enough and is an important part of what allows the protocol to be "reliable". 

Now, of course, that is much easier to apply in the "TCP model" where you have one socket for one association, and where there is an implicit understanding that things are always reliable.

With an SCTP "UDP-style" socket, we have a new animal: a many-to-one socket (many associations, one socket) that is also reliable and flow controlled.    We cannot just apply  "UDP-style" thinking here.  

 There are two choices: 

   (1) make the socket ALWAYS non blocking, in which case a send () directed towards an association that has a zero rwnd should return an error (perhaps EWOULDBLOCK or EAGAIN)?

   (2) allow blocking behavior... in particular, if you write() on socket s1 directed towards an association a1 with a zero rwnd,  that socket should block even if there are other associations a2 through aN that have rwnd space.   The user may not like being blocked, but that was part of the bargain he/she made when you chose a blocking UDP-style socket; to avoid that situation, use a non-blocking socket, or use the one-to-one model (currently called the TCP model.)

Either way,  it is not only a question of running out of mbufs.  The interaction of UDP-style SCTP sockets with rwnd needs to be taken into account. 

Cheers,
Phill

At 06:36 AM 9/24/2002 -0500, Randall Stewart (cisco) wrote:
>WARNING: The remainder of this message has not been transferred.
>The estimated size of this message is 4510 bytes.
>Click on the Retrieve From Server icon above and check mail again to get the 
>whole thing. (If you're reading this in the preview pane, you'll need to 
>open the message to see the icon.)  If the Retrieve From Server icon is not 
>showing, then this message is no longer on the server.

---

From: FUJITA Ken <[email-masked]>
Date: Tue, 24 Sep 2002 17:06:17 -0700

Phillip, Randy, and Folks

 >  There are two choices:
 >
 >    (1) make the socket ALWAYS non blocking, in which case a send ()
 > directed towards an association that has a zero rwnd should return an 
 > error (perhaps EWOULDBLOCK or EAGAIN)?
 >
 >    (2) allow blocking behavior... in particular, if you write() on
 > socket s1 directed towards an association a1 with a zero rwnd,  that
 > socket should block even if there are other associations a2 through aN
 > that have rwnd space.   The user may not like being blocked, but that
 > was part of the bargain he/she made when you chose a blocking
 > UDP-style socket; to avoid that situation, use a non-blocking socket,
 > or use the one-to-one model (currently called the TCP model.)
 >

Or we can use sctp_peeloff() to branch off the association into a 
separate socket/file descriptor. In this case, if one UDP-style 
association becomes blocking, I believe the other association still 
works without blocking.
In my feeling, option 2 is better.

Thanks,

-- 
FUJITA Ken
mailto:[email-masked]

Phillip Conrad wrote:
 >    (1) make the socket ALWAYS non blocking, in which case a send () 
directed towards an association that has a zero rwnd should return an 
error (perhaps EWOULDBLOCK or EAGAIN)?
 >
 >    (2) allow blocking behavior... in particular, if you write() on 
socket s1 directed towards an association a1 with a zero rwnd,  that 
socket should block even if there are other associations a2 through aN 
that have rwnd space.   The user may not like being blocked, but that 
was part of the bargain he/she made when you chose a blocking UDP-style 
socket; to avoid that situation, use a non-blocking socket, or use the 
one-to-one model (currently called the TCP model.)

---

From: Phillip Conrad <[email-masked]>
Date: Tue, 24 Sep 2002 22:20:06 -0400

>association becomes blocking, I believe the other association still 

I'm a little bit lost after reading this last sentence, because I generally think only in terms of UDP-style "sockets", not UDP-style "associations".    But I think I *might* understand what you are saying... let me make things a little more specific, and tell me if I have understood your meaning, as well as the intent of the sockets API internet draft.

Assume
  * there is only one UDP-style socket s1
  * association a1 is active on socket s1, 
  * other associations a2 through aN are also active on s1.
  * association a1 currently has a zero rwnd.

If you now do a sendmsg directed towards association a1 on socket s1, then I suggest  that a reasonable semantics for "blocking socket" is that sendmsg should block until there is sufficient rwnd space on association a1 for that operation to complete.

If that is not the meaning of 'blocking socket' for the so-called "UDP-style syntax", then there probably shouldn't be any reason to have a blocking socket for UDP (for reasons alluded to earlier by Randy Stewart.)

(Note: in my earlier message I said "write()", where I should have said sendto() or sendmsg(), I suppose... write() doesn't make much sense on a socket that has multiple associations.)

Note, you can't do a sctp_peeloff until that blocked sendmsg() call returns.  When you do the sctp_peeloff on socket s1 for association a1, then the result is a brand new TCP-style socket s2 for association a1, and association a1 no longer has any relationship to socket s1.  

Is this what you mean when you say: "In this case, if one UDP-style association becomes blocking, I believe the other association still works without blocking?"    Or are you saying something different?

>In my feeling, option 2 is better.

On this point, I agree totally: option 2 allows the programmer the choice of non-blocking or blocking behavior.    

Many programmers may prefer to use non-blocking mode for a "UDP-style" SCTP socket, so that if there is no rwnd space to send some data, the sendmsg() returns immediately with EAGAIN, so that the program can go on to some other task while it is waiting for rwnd space to open up...  but it really ought to be up to the programmer.     

There may be cases where being able to "go to sleep" (i.e. block) until a message can be sent on a particular association is the only logical course of action, and it would be nice to not HAVE TO do an sctp_peeloff() (thus moving over to the TCP-style socket) to have that option available.    

Come to think of it, maybe that is what you were trying to say... that if I don't allow for blocking UDP-style sockets, I can just call sctp_peeloff after getting the EAGAIN error, and then call write() or send() on my brand new TCP-style socket for the same association.   But I think it would be better if this were not the only way to direct the process to "go to sleep until the rwnd opens up on association a".

Cheers,
Phill Conrad

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Wed, 25 Sep 2002 06:16:59 -0500

Phil:

I too am just a bit confused by what Ken is getting at here..

comments below

>Many programmers may prefer to use non-blocking mode for a "UDP-style" SCTP socket, so that if there is no rwnd space to send some data, the sendmsg() returns immediately with EAGAIN, so that the pr…
One little note.. it is not just rwnd space that has to run out.. but it 
is also the space in the socket-buffer. TCP and
SCTP allow this behavior.. i.e. when you send and the rwnd is 0 you will 
be able to continue
to send until you hit some max associated with the socketbuffer... now 
note that if the peers rwnd (initially)
is the same value as your socketbuffer than your statements are 
completely correct... but it is unlikely
that this will occur.

Also note, I think in past discussion with Jon Grim on some list, we 
have talked about a 'udp style' socket
having socket buffer space allocated to it per association (not per 
socket)... Not sure how that all works :>
and its on my to-do list.. but any way.. back to the main discussion..

>Come to think of it, maybe that is what you were trying to say... that if I don't allow for blocking UDP-style sockets, I can just call sctp_peeloff after getting the EAGAIN error, and then call write() or send() on my brand new TCP-style socket for the same association.   But I think it would be better if this were not the only way to direct the process to "go to sleep until the rwnd opens up on association a"
>
>  
>
Hmm.. I am not sure either which ken meant here... A second note is that 
I don't think the
issue Marco reported is that you can't make things blocking (this works 
I have tested it).. it is
just the default in our implementation is not blocking...

No matter which way we make it (for default) I think one must allow the 
app writter to make
a call to toggle things blocking/non-blocking..

The point I was trying to make (and I think I did) is that things are 
quite the same when comparing
UDP style vs UDP... since once you block in a sendto() on one assoc you 
may not have blocked on
another assoc (as you point out) .. Also there is no way to recvfrom() 
on just a particular association. So
a recvfrom() blocking style will return the first message up from any 
association (I know you realize
this.. just reminding everyone else on the list).

R

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Wed, 25 Sep 2002 07:08:54 -0500

Phillip Conrad <[email-masked]> writes:
> I'm a little bit lost after reading this last sentence, because I
> generally think only in terms of UDP-style "sockets", not UDP-style
> "associations".    But I think I *might* understand what you are
> saying... let me make things a little more specific, and tell me if
> I have understood your meaning, as well as the intent of the sockets
> API internet draft.
> 
> Assume

There is an extra buffer involved--zero rwnd alone should not be
sufficient to cause blocking.

From socket(7):

       SO_SNDBUF
              Sets or gets the  maximum  socket  send  buffer  in
              bytes.    The   default   value   is   set  by  the
              wmem_default sysctl and the maximum  allowed  value
              is set by the wmem_max sysctl.

This buffer is PER SOCKET rather than per association.  Furthermore,
it should feed all the associations equally.  So if one association
runs out of rwnd for sending more data, the socket buffer will
eventually fill (with messages for that association).  Once the socket
buffer fills, ALL send calls should block--this is completely
independant of the underlying protocol.

---

From: "K. Poon" <[email-masked]>
Date: Wed, 25 Sep 2002 11:12:52 -0700 (PDT)

> and never is it added to the output socket buffer.

When the data is copied from user land to kernel land in
some OS, memory is needed...

> not to sure the system would survive :-0

Actually no.  It also depends on the OS.  For example, some
OS can have a call like alloc (NO_WAIT) and alloc(HARDER).
The semantics of the first one can be that if there is nothing
in the cache (assuming there is an internal kernel memory
hierarchy), just return with an error.  This can be used for
non-blocking socket call.  The semantics of the second one
can be that the kernel will ask all the subsystema to do some
garbage collection and return their own memory cache to the
main kernel memory pool.  After the garbage collection, the
kernel can return memory to the alloc(HARDER) call.  The
system is still running happiliy, just that the alloc(HARDER)
will block for a while.

As I mentioned in the earlier mail, it all depends...  So I 
guess the easy solution is to just pick a default and live 
with it (-:

=====

---

From: "K. Poon" <[email-masked]>
Date: Wed, 25 Sep 2002 11:28:58 -0700 (PDT)

> (or likely even a little of both).    

Some background...

When the current draft was first written, only the TCP style
socket mapping was designed towards compatibility.  We had a
somewhat "heated" phone conference about what the socket mapping
should be, as I recalled...  The end result was to have 2 different
mappings, TCP and UDP style.  I guess Randy can answer his 
intention behind the UDP style socket.

=====

---

From: "K. Poon" <[email-masked]>
Date: Wed, 25 Sep 2002 12:00:17 -0700 (PDT)

> and its on my to-do list.. but any way.. back to the main

I think we have talked about this funny socket buffer issue 
before, if I remember correctly...  One issue with the per
association socket buffer allocation with a UDP style socket
is that the OS needs to have another kind of resource
control.  Right now, I believe all OSes impose a limit on
the number of sockets a user can open, and the maximum
buffer size each socket can use.  This cannot control the
per association socket buffer allocation scheme.  Well,
it is just code, but one important rationale behind the
draft is that the socket layer in most OSes does not need
to be fundamentally changed, if possible.

> a call to toggle things blocking/non-blocking..

Agree.

=====

---

From: Jon Grimm <[email-masked]>
Date: Wed, 25 Sep 2002 14:04:06 -0500

> intention behind the UDP style socket.

Yes, this is in part how the discussion arose, I think Phil brought up 
that the UDP-style was really a different beast than UDP (SOCK_DGRAM) 
programming.

One option is to change the name.. Phil had been proposing, IIRC, 
Explicit vs Implicit SCTP sockets, but this didn't seem needed since 
TCP-style really does try to have API compatibility with 
SOCK_STREAM/TCP, so the terminology of "Explicit" only adds to the 
confusion.

One idea brought out at the bakeoff would be to make the default 
UDP-style behavior more SOCK_DGRAM application friendly:
1) Auto-listen Enabled
2) Auto-connect Enabled (already a default)
3) Events Disabled.
4) AUTO-CLOSE Enabled.

Or...Another approach would be just to claim that UDP-style is a 
completely different beast and application writers using it must 
understand how and why it is different than SOCK_DGRAM programming. 
Having default behavior that break application compatibility helps force 
the application writer to come up to speed.  ;-)

Thanks,
jon