Subject: Partial delivery API unclear

From: Anders Torger <[email-masked]>
Date: Tue, 17 Apr 2007 22:13:17 +0200

It is not clear to me how SCTP partial delivery using recvmsg() should 
work.

It seems to be two approaches, and the socket API draft is not clear on 
which one it should be:

 A) On the same association, a partial delivery should not be
    interleaved.

 B) Partial deliveries can be interleaved with other complete or partial
    deliveries with no restrictions.

Take a Solaris 10 stack from a build November 11 2006 for example, if 
recvmsg() returns a fragment of an ordered message, the next call that 
delivers a message or fragment of a message will be one of:

 1) a fragment (possibly the last) of the partially delivered message.
 2) a fragment or full message from another assocation (one-to-many
    sockets only).
 3) a fragment or full message from the same association but from
    another stream.
 4) a fragment or full message from the same association and same stream
    but unordered.

Clearly, this is according to model B). Looking in the latest socket API 
draft (14), section 3.1.4 says:

 "If the SCTP stack is running low on buffers, it may partially
  deliver a message.  In this case, MSG_EOR will not be set, and more
  calls to recvmsg() will be necessary to completely consume the
  message.  Only one message at a time per stream can be partially
  delivered."

The last sentence states almost model B), but with the restriction that 
an unordered message cannot be partitially delivered on the same stream 
while an ordered is being partially delivered and the other way around. 
But it could be interleaved with a message on the same stream as long 
as it is delivered in full (naturally with the unordered flag if 
partially deliverey is ordered and vice versa). 

Actually, Solaris 10 seems to never deliver unordered messages partially 
(at least if the input buffer is large enough), if you feed it with 
unordered messages a bit larger than SO_RCVBUF is, the association will 
indeed hang, and if tried a few times a crash can be provoked, so I 
guess that the actual plan is to deliver unordered messages partially 
and that it does not is a bug. (Test performed: send a 103001 byte 
unordered message from a Linux 2.6.10 machine to a Solaris 10 machine 
with default SO_RCVBUF size of 102400 bytes, association will hang. Ok, 
it does not make sense with that large unordered messages, but I'm here 
to stress the API... I would guess the test can be done between two 
Solaris machines too, just increase SO_SNDBUF on the sending machine).

However, the newer socket API drafts also add the 
SCTP_FRAGMENT_INTERLEAVE socket option:

"7.1.24.  Get or set fragmented interleave (SCTP_FRAGMENT_INTERLEAVE)

   This options will at a minimum specify if the implementation is doing
   fragmented interleave.  Fragmented interleave, for a one to many
   socket, is when subsequent calls to receive a message may return
   parts of messages from different associations.  Some implementations
   may allow you to turn this value on or off.  If so, when turned off,
   no fragment interleave will occur (which will cause a head of line
   blocking amongst multiple associations sharing the same one to many
   socket).  When this option is turned on, then each receive call may
   come from a different association (thus the user must receive data
   with the extended calls (e.g. sctp_recvmsg) to keep track of which
   association each receive belongs to."

The text here seems to suggest that the API should work according to 
the A) model, since it talks about its use only in relation to 
one-to-many sockets, whilst model B) would mean that it is also 
applicable to one-to-one sockets. In other words, it seems like the 
above text is written with the assumption that there is no interleave 
on a one-to-one socket.

Thus, as far as I can see, the socket API draft indirectly contradicts 
itself, and leaves me confused.

As an SCTP application programmer, this 'detail' is important. With 
model A), it means that an application only needs to hold a buffer for 
one message at a time, but with model B) there is potentially a need to 
hold one buffer per stream times two (one for ordered and one for 
unordered messages). Thus B) is a bit cumbersome, but it has an 
advantage which is that it does not introduce head-of-line blocking when 
dealing with large user messages, which model A) does. But one could 
also argue that only a poorly designed upper layer protocol would 
employ very large user messages and streams or unordered messages 
simultaneously, and therefore the A) model is better since it makes the 
life easier for the API user.

Anyone who knows where the draft is going - interleave or not, and how 
are other stack implementations working in this area?

/Anders Torger

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 17 Apr 2007 14:43:27 -0600

Anders,

> are other stack implementations working in this area?

The OpenSS7 stacks both follow model B.

However, if you are concerned with what is coming next, why not
use MSG_PEEK?

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Vlad Yasevich <[email-masked]>
Date: Tue, 17 Apr 2007 17:12:06 -0400

>     deliveries with no restrictions.

Hi Anders

Hm...  looking at LKSCTP, I am really not sure what the default mode
of operation is.  It's leaning towards A, but not strictly.

>  1) a fragment (possibly the last) of the partially delivered message.

same in LKSCTP

>  2) a fragment or full message from another assocation (one-to-many

Only if SCTP_FRAGMENT_INTERLEAVE is enabled.  The option is disabled
by default.

>  3) a fragment or full message from the same association but from

Nope, not in lksctp.

>  4) a fragment or full message from the same association and same stream

Actually, only a full message (ie. it was never fragmented) that was
unordered has a chance to interleave with PD in lksctp
implementation.  Looking at this one further, this might actually be
a bug in the implementation.

You are right.  There are definitely some wiggle room in the
interpretation; however, I don't think that following model A is
wrong.

-vlad

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 00:06:06 +0200

Hi Anders,

I'm CCing Kacheong, since you most likely point out a bug in the
Solaris implementation below. I'm sure Kacheong wants to know about
it such that Sun can fix it.

Comments in-line.

Best regards
Michael

> partially deliverey is ordered and vice versa).
I think this is not intended. You should not deliver a complete ordered
user message with sid = n and ssn = m while partially delivering
a message with sid = n and ssn = m - 1. This violates the rule that
all ordered messages within each stream is delivered in order.
> Solaris machines too, just increase SO_SNDBUF on the sending machine).
I have not verified this, but this seems to be a bug in the SCTP  
implementation.
> above text is written with the assumption that there is no interleave
Yes, this text was written to overcome the problem, that one association
should not be able to block others. This is not a problem for 1-to-1,
but someone (an attacker) could start sending a long fragmented message
such the the partial delivery kicks off. If the attacker never sends
the last fragment and fragment interleave is not enabled the application
would never receive messages from the other associations. This allows
a DOS attack. To avoid this, this option was introduced.
> life easier for the API user.
I think it is important to do the interleave between the different  
associations
of 1-to-many sockets to avoid the DOS attack described above.

I do see that (B) is possible, it defers more tasks to the upper  
layer and
minimizes the time parts of the messages a kept inside the SCTP layer.

But how likely will this situation occur? I would assume that a  
message which
is being partially delivered is also transmitted in multiple DATA  
chunks.
On the wire these fragmented user messages can not be "interleaved".  
They
must have consecutive TSNs. So I'm not sure that you gain much, by  
providing
(B) instead of (A). So I think an implementation providing (A) is  
sufficient
to write good applications.

But I see your point: For an application write it is important to know
what he has to expect. I think I would prefer (A) as an application  
writer.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 17 Apr 2007 16:54:39 -0600

Michael,

I think you are confusing the API with the protocol.  Although the
protocol (unnecessarily as it may be) requires that a ordered message be
fully delivered, with sequential TSNs, before another message (ordered
or not) in the same stream, that is not a required on the API.

Consider that, regardless of the size or fragmentation on the wire, one,
say 8192 byte ordered message arrives (completely).  The user reads 4096
bytes of this message.  Before the next read a 32 byte unordered message
for the same stream arrives.  A subsequent read could read the unordered
message before reading the remainder of the ordered message.

For SIGTRAN expediting unordered messages (which are more time
sensitive) is desirable.

--brian

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 01:13:05 +0200

Brian,

comments in-line.

Best regards
Michael

> or not) in the same stream, that is not a required on the API.
No I'm not confusing it.
My point is only:
- Interleaving messages from different associations is necessary to
   protect against a DOS attack.
- Due to the limitations of the TSN assignment, the benefit of partial
   deliver multiple fragmented users messages within one association
   seems to be limited. This is just my opinion.

I do understand that one could do (B), the user then has to do more...
> message before reading the remainder of the ordered message.
Whether or not unordered messages (fragmented or not) are delivered
"faster" than ordered ones is not specified.
> For SIGTRAN expediting unordered messages (which are more time
As far as I know, none of the SIGTRAN messages are required to
be delivered unordered. And the delivery of the unordered user
messages is not specified. So you can not assume, that they are
delivered "faster".

---

From: Anders Torger <[email-masked]>
Date: Wed, 18 Apr 2007 07:44:12 +0200

Thank you for the reply.

I just have access to the "Solaris 10 11/06 s10s_u3wos_10 SPARC" and I 
think there are patches since then, so perhaps the problem is already 
fixed in later patches. However, I can provide a test program and do 
some further tests if Kacheong is interested. The company I work for is 
in any case interested in stabilizing the SCTP stacks out there.

If I understood things correctly, model A) exists solely to make it 
easier for the application programmer, not for the stack implementors. 
I would guess that model A) actually make implementing a stack harder. 
So, then there should be a real value with model A) for the programmer.

However, I realized that the A) model probably does not really help the 
programmer that much. To do that, the programmer must assume that there 
is a 1-to-1 mapping between SCTP user messages and upper layer protocol 
messages. If there is no 1-to-1 mapping, the ULP message could be 
fragmented into multiple SCTP user messages and thus be delivered in 
parts anyway.

A server must probably expect that a client could use "tcp compatibility 
mode" and is doing write()/send() instead of sendmsg(). Of course, the 
most natural way is to make a single write() per ULP message even in 
such application, and it is most likely that a single write() will lead 
to a single SCTP message, and thus a 1-to-1 mapping is quite likely. 
However, if the client has a separate buffered I/O layer the 1-to-1 
mapping may not be there, or, the ULP may have large messages which 
will be split due to exceeding SO_SNDBUF.

Take the Diameter protocol as an example. The largest message size there 
is 16 megabytes, messages never get that large in the normal case, but 
it is not too unlikely that they exceed SO_SNDBUF for some client. A 
client using non-blocking write() would thus sometimes split a Diameter 
message over several SCTP user messages.

Say that we require clients being properly implemented using sendmsg(), 
then we have the problem that most(?) stacks do not support partial 
sending. I've been informed by Randall Stewart that BSD has some EEOR 
mode such that you can actually send messages of any size, but I don't 
think this is supported in for example Linux or Solaris.

Thus, if an ULP message exceeds SO_SNDBUF, the sender is generally 
forced to split the message into several SCTP messages.

When there is no widespread API to send messages exceeding SO_SNDBUF in 
size, then I don't see any strong value that a stack does model A), 
since an application programmer would need to manage buffers per stream 
anyway (which for Diameter is a bit cumbersome, since the max message 
size is a bit large), unless the ULP max message size is so small one 
can assume that no client has an SO_SNDBUF that is smaller.

/Anders Torger

On Wednesday 18 April 2007 00:06, Michael Tuexen wrote:
> > It is not clear to me how SCTP partial delivery using recvmsg()
> > should work.
> >
> > Take a Solaris 10 stack from a build November 11 2006 for example,
> > if recvmsg() returns a fragment of an ordered message, the next
> > call that delivers a message or fragment of a message will be one
> > of:
> >
> >  1) a fragment (possibly the last) of the partially delivered
> > message. 2) a fragment or full message from another assocation
> > (one-to-many sockets only).
> >   deliver a message.  In this case, MSG_EOR will not be set, and
> > more calls to recvmsg() will be necessary to completely consume the
> > But it could be interleaved with a message on the same stream as
> > long as it is delivered in full (naturally with the unordered flag
> > if partially deliverey is ordered and vice versa).
>
> I think this is not intended. You should not deliver a complete
> ordered user message with sid = n and ssn = m while partially
> delivering a message with sid = n and ssn = m - 1. This violates the
> rule that all ordered messages within each stream is delivered in
> order.
>
> > guess that the actual plan is to deliver unordered messages
> > partially and that it does not is a bug. (Test performed: send a
> > 103001 byte unordered message from a Linux 2.6.10 machine to a
> > Solaris 10 machine with default SO_RCVBUF size of 102400 bytes,
> > association will hang. Ok,
> >    no fragment interleave will occur (which will cause a head of
> > line blocking amongst multiple associations sharing the same one to
> > many socket).  When this option is turned on, then each receive
> > call may come from a different association (thus the user must
> > receive data with the extended calls (e.g. sctp_recvmsg) to keep
> > track of which association each receive belongs to."
> >
> > The text here seems to suggest that the API should work according
> > to the A) model, since it talks about its use only in relation to
> > above text is written with the assumption that there is no
> > interleave on a one-to-one socket.
>
> Yes, this text was written to overcome the problem, that one
> association should not be able to block others. This is not a problem
> for 1-to-1, but someone (an attacker) could start sending a long
> fragmented message such the the partial delivery kicks off. If the
> attacker never sends the last fragment and fragment interleave is not
> enabled the application would never receive messages from the other
> associations. This allows a DOS attack. To avoid this, this option
> was introduced.
>
> > Thus, as far as I can see, the socket API draft indirectly
> > contradicts itself, and leaves me confused.
> >
> > model A), it means that an application only needs to hold a buffer
> > for one message at a time, but with model B) there is potentially a
> > need to
> > dealing with large user messages, which model A) does. But one
> > could also argue that only a poorly designed upper layer protocol
> > would employ very large user messages and streams or unordered
> > messages simultaneously, and therefore the A) model is better since
> > it makes the
> But I see your point: For an application write it is important to
> know what he has to expect. I think I would prefer (A) as an
> application writer.
>
> > Anyone who knows where the draft is going - interleave or not, and
> > how are other stack implementations working in this area?
> >
> > /Anders Torger

---

From: Anders Torger <[email-masked]>
Date: Wed, 18 Apr 2007 09:04:01 +0200

MSG_PEEK does not really help, since you cannot force the stack to 
deliver things in different order than it is planning.

However, I am not that concerned, but I need to understand how things do 
work, and how things are supposed to work. I'm actually working on 
documentation, a guideline how to use the SCTP socket API when writing 
carrier grade soft-realtime telecom applications. In the process I'm 
trying to understand the ideas behind the API, and where it is heading. 
And specifically looking a bit closer at Solaris and Linux 
implementations which is the main interest of the company I work for.

What I see concerning partial delivery is that it seems like the socket 
API designers have identified that it is easier to handle one message 
at a time, than having to deal with (=buffer) fragments of multiple 
messages. Therefore some parts of the API draft indicate that partial 
delivery should only happen on one message at at time, at least per 
association.

However, this is not the way all implementations work, one clear example 
of this is the Solaris implementation, which probably is one of the 
most 'important' implementations out there, since Solaris is very 
popular in telecom.

In addition to this, the API draft is not 100% clear in this area. The 
SCTP_FRAGMENT_INTERLEAVE option do indicate that interleaved message 
delivery should be a special case that have to actively be enabled. But 
in another place in the document, it says that interleave happens 
between streams.

My assumption here is that avoiding interleave makes it harder to 
implement the stack, thus the only reason for avoiding it must be to 
make it easier to use the socket API.

Then the natural follow-on question is - does it really make things 
easier for the application programmer or not? To me it seems like it 
becomes simpler only when possible to enforce a 1-to-1 mapping between 
ULP messages and SCTP user messages. To support that case, I think it 
is important to provide means in the socket API to decide the size of 
the message sent. To do that, partial sending must be supported, and 
that seems to be rare.

Another aspect is that doing partial delivery on only one message at a 
time within an association seems to introduce head-of-line blocking in 
the API between streams although that is not appearent in the protocol. 
From a low-delay telecom signaling perspective this seems like a bad 
idea.

So if I had the power to say how the socket API should be made, I'd say:

 - interleave whenever possible to avoid head-of-line blocking, or at
   least make it possible to activate, that is SCTP_FRAGMENT_INTERLEAVE
   should also enable interleaving within an association between streams
   and unordered messages.

 - make it clear that it must be possible to do partial sending, such
   that the user gets full control over SCTP user message size.

I know that I have kind of advocated the non-interleaved mode, but I 
think I'm changing my mind. After some consideration, I simply don't 
think that the non-interleave mode simplifies things significantly 
enough to be really valuable. It usually ends up with should we buffer 
per association (non-interleave) or per stream + unordered 
(interleaved). Per association buffering is indeed a bit simpler, it 
provides the following advantages:

 - One can always offer 65535 streams to the client, since buffer space
   is not dependent on the number of streams.
 - In many cases it becomes practical to pre-allocate a buffer of
   ULP max message size, which could become unpractical if we need one
   buffer per stream.

But apart from that, stream buffering follows the same principles and 
does not necessarily result in more complex code, except from possibly 
that one cannot pre-allocate max size buffers for each stream, or that 
one have to limit the number of inbound streams.

It would be interesting to know the opinions on this from the 
participants on this list.

On Tuesday 17 April 2007 22:43, Brian F. G. Bidulock wrote:
> >  B) Partial deliveries can be interleaved with other complete or
> > partial deliveries with no restrictions.
> >

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 18 Apr 2007 22:46:01 +0800

[Comments in-line]

>   message.  Only one message at a time per stream can be partially

I do not remember if I wrote the above part, it may be...  Sorry
for the confusion.  I think the intention is that message delivery
should not be blocked unless it violates the in sequence delivery.
This is in line with the reason why SCTP supports multiple streams
in one  association.  So (B) is the intention.  The only restriction
the API should have is that an app doing a recvmsg() should always
receive messages from the same stream in an association in sequence.
So when partial delivery is on for message Y on stream X, the next
message returned from recvmsg() can either be the next part of
the message X, or a message from another stream, or an unordered
message from stream X.  In the case of 1-N style socket, the next
message returned can also be from a different association.

> partially deliverey is ordered and vice versa). 

Yes, I think the above is the intention.

> above text is written with the assumption that there is no interleave 

I think this is specific to 1-N style socket.  Some
implementations may not allow such interleaving.  So here
is an option to let an app know if the underlying stack
supports it or not.  But I think the intention is to have
an API which does this interleaving by default.

> life easier for the API user.

I think it depends on the app.  The receiver of the app
probably knows if the sender will send unordered message
or not.  So if the sender will never do that, the receiver
does not need to handle the complicated case.  Given the
fact that SCTP supports multiple streams, I think most apps
do not need to use unordered message.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 18 Apr 2007 23:15:57 +0800

>> But it could be interleaved with a message on the same stream as long
       ^^

> all ordered messages within each stream is delivered in order.

I think the "it" above refers to an unordered message, not
ordered message.

> But how likely will this situation occur? I would assume that a message
> which
> is being partially delivered is also transmitted in multiple DATA chunks.
> On the wire these fragmented user messages can not be "interleaved". They

I think the issue is whether a message from stream X of
an association can be delivered while a partial delivery of
a message from stream Y is going on.  I think it is better
to allow this as this is in line with the reason why SCTP
supports multiple streams.  For example, message A of stream
X takes two and a half IP packets.  The other half of the third
packet belongs to message B of stream Y.  Suppose the partial
delivery point is set to one packet worths of data.  After the
stack gets the first and third packets, it can partially
deliver message A and also completely deliver message B without
waiting for the second packet.  IMHO, there is no special need
why the stack needs to defer delivering message B.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 18 Apr 2007 23:44:40 +0800

>    should also enable interleaving within an association between streams

I think this is the original intention.

>    that the user gets full control over SCTP user message size.

In some sense, an app programmer has full control over SCTP
message size as long as it is not too big :-)  Partial sending
is a "relatively" new addition to the draft.  I think this was
not in the draft because it was not expected that an app had a
need to send a message larger than the socket buffer size.  This
followed the UDP socket model which was known to be sufficient.
It seems that with SCTP, app programmers are now trying to go
beyond what has been available.

I expect that as time goes on, all stacks will fully support
the API draft.

-- 

						K. Poon.
						[email-masked]

---

From: Anders Torger <[email-masked]>
Date: Wed, 18 Apr 2007 20:43:33 +0200

> why the stack needs to defer delivering message B.

I can see one reason why one should not deliver it which I did not think 
about earlier. This has to do with receive buffer design in an 
application.

Typically one would want to have a complete ULP message in contiguous 
memory before parsing it. More advanced parsers could of handle scatter 
arrays, but I guess that is not too common.

Additionally, one would want to avoid/minimize copying or re-allocating 
memory to maximize efficiency. One simple example would be that one 
provides the start of an ULP max size buffer to recvmsg(), and when a 
complete message is in the buffer, it is passed along to the parser.

Combining your and my example, that would mean that the ULP buffer would 
(in the 'worst' case) after the first recvmsg() contain partial message 
A, and then when providing the appropriate offset into the buffer for 
the next recvmsg() we get message B. Argh! We were expecting the 
continuation of message B! The problem here is that we cannot know on 
what stream (and association if one-to-many) the next recvmsg() will 
return. Interestingly enough a feature has been added in the later 
drafts containing an sctp_extrcvinfo which contains information about 
what the next recvmsg() will return, I wonder if this has been done to 
solve this problem. However, it only partly solves the problem, since 
the stack could partially deliver message A, have no more to deliver 
(next info is empty), and the B arrives from the network before the 
next fragment for A.

The application programmer would never need to think about this problem 
if the stack employs a strict model to only partially deliver one 
message at a time.

There is a relatively easy way to solve the problem with an interleaved 
stack though, when partially delivery arrives, make a best guess what 
the next recvmsg() will return - if you have sctp_extrcvinfo use that 
information, if that is empty (or sctp_extrcvinfo is not implemented) 
simply guess that the next fragment will be the continuation of the 
message (which will be the most common case), and if it isn't, 
re-allocate to a new buffer, there will never be a huge amount of 
memory to move anyway.

So I still think that the "interleave without restrictions to avoid 
introducing any head-of-line blocking"-approach like the Solaris stack 
employs is the right one. But perhaps it needs to be made clearer in 
socket API draft, since there are obviously differences in how stacks 
are implemented, and as I said in an earlier mail, I don't think that 
the stricter one-message-at-a-time model has evolved naturally out of 
stack design, but is rather a feature added to make things easier for 
the application programmer.

In any case, a socket API that behave differently on different platforms 
in this rather fundamental aspect does not make things easier for us 
programmers that want to write portable code...

It would be interesting to hear the opinion from Randall Stewart, since 
I've sort of got the idea that he prefers the one-message-at-a-time 
model, but I could be wrong.

/Anders Torger

---

From: Vlad Yasevich <[email-masked]>
Date: Wed, 18 Apr 2007 15:19:55 -0400

On additional potential issue that I just thought off is
that with default stream interleaving (scenario B), upper
layer protocols would need specify the stream information.
In this case I am not talking about specific sigtran protocols,
but more of ports of say SSH, or FTP, or even HTTP to run over SCTP
with multi-streaming capabilities.

My concert was that if a client was ported to use the multi-streaming
feature, while the server may not have.  In this case, it's possible
for the server to run into major interoperability issues if the association
just happens to enter partial delivery.

I guess one could consider this a server _bug_, but without something
that says that says multi-streaming should be used, it might be too common.

So, this whole message interleaving during partial delivery may be something
that needs to be explicitly enabled or disabled???  Just thinking outloud...

Thanks
-vlad

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 22:21:29 +0200

Hi Anders,

the protocols I know of, which use SCTP as their transport layer require
that an upper layer message is one SCTP user message and is fragmented
by SCTP if it is too  large. And I think this is the way SCTP is  
designed
to transport messages. There is no protocol limit which limits the
size of user messages.

I do see the problem you are pointing out, that if an implementation  
(not
the protocol) limits the size of messages, this limits the size of the
upper layer messages. But this is something the implementation has to
overcome. So an SCTP implementation should be able to handle messages
of the size of its upper layer correctly. This especially applies
to the send() calls, in particular to the handling of signals during
the send() call.

So I think the stacks should handle sending messages larger than
the send buffer, if they want to support applications which are
sending large messages.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 22:31:15 +0200

Hi Kacheong,

I think the intention of the text in the socketAPI id is more
like (A). The interleave is there to avoid a DOS. This requires
only interleaving between different associations.

Please have also a look at the example code at the end of
the socket API ID. I think it only works correctly, if the
stack does (A), not (B). Or am I wrong?

Best regards
Michael

On Apr 18, 2007, at 4:46 PM, Kacheong Poon wrote:

>> should
>> message.
>> for

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 22:41:09 +0200

.. more thinking...

The code in the example breaks as soon as you do any interleave.
But I think it is important to interleave between different  
associations,
so the code needs an update for the 1-to-many model.
It needs to do a partial write when getting a partial message.

Best regards
Michael

On Apr 18, 2007, at 10:31 PM, Michael Tuexen wrote:

>>> more
>>> long
>>> line

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 18 Apr 2007 22:47:37 +0200

> if the stack employs a strict model to only partially deliver one
But this allows a DOS attack. A stack must be able to interleave  
messages
from different associations. Therefore the program must be prepared
to handle this.

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 18:41:49 -0400

Vlad:

I have not yet finished reading this thread.. but I wanted
to comment on yours.. below..

>>  1) a fragment (possibly the last) of the partially delivered message.
  same for BSD.
> Only if SCTP_FRAGMENT_INTERLEAVE is enabled.  The option is disabled

Same for BSD
>>  3) a fragment or full message from the same association but from

Nope NOT in BSD.

> a bug in the implementation.

Nope..

Basically BSD takes approach <A>. With the exception that
fragment interleave, if turned on , allows you to interleave
other association messages with ones doing a PD-API.

Now, as to this whole discussion... it was discussed in
detail on either this list or the tsvwg one (can't remember which
one) when fragment interleave was added. At that time the
conclusion was that.

A) fragment interleave was off by default.
B) Turning it on got you msg's from other associations NOT this one.
C) We did discuss if interleaving other msg's from other streams
    was worth doing. And the basic thread, as I remember it, was that
    no, once you start a PD-API you get just that streams msg until
    its all read... If I remember right I did suggest that one could
    deliver another streams msg in the same asoc (even talked about
    having an option to enable/disable this). But folks argued that
    it was not needed and it just added more complication to the
    app.

Consider where PD-API is used. Its used when the peer sends a message
larger than the socket send buffer (I will discount the local ability
to set the partial delivery point). Now if the peer sends a
1 meg message.. PD-API begins. We send up 1/2 Meg for the app
to read .. it reads it.. When finally we get the rest we send
it up.. Eventually we will get the next TSN after the fragmented
big message. Here you could have some other message from another
stream. By that time you either have put all the msg up.. OR you
have a loss preventing you from sending up the last of the PD-API.
If you have it all. Then its no big deal.. its up and the user
should read it. Loss is going to be a rare case.. if it happens
you do want the loss recovered anyway..

Anyway.. I will go read the rest of the thread now ;-)

R

-- 
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 803-317-4952 (cell)

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 18:45:40 -0400

> why the stack needs to defer delivering message B.

So in your example.. if stream X's message is also partially deliverable
only.. i.e. there is a missing piece too. Do you start that
partial delivery as well?

How many partial deliveries does one allow to start all at
once (not that this should happen.. but once you allow a subsequent
partial delivery.. I think you must allow all streams to go
in to a partial delivery state)... This may have been the reason
we did not add this..

I need to go back and find that old thread :-D

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 18:53:35 -0400

Anders Torger wrote:
> solve this problem.

Yes :-D

  However, it only partly solves the problem, since
> (next info is empty), and the B arrives from the network before the 

Or if you allow multiple partial deliveries (which you probabaly
should if you do one).

Then you could have in queue msgs for:

--Str1-> Str2 -> Str3
    (p)    (p)     (c)

(p) = partial msg
(c) = complete msg

Now I read all in str1 ext-info tells me str 2 is next
I read all in strm2 ext info tells me str 3 is next.. and
at the time of the read of str2, str1 has nothing. By
the next user-to-kernel transition more as arrived on
str1.. so the kernel will send back strm1 data .. since
it needs to clear that out first and can't know what
the previous return was...

Very ugly.. I don't think mixing this stuff is a
good idea.. once you start paritial delivery you should
keep with that stream.

Otherwise the app becomes very very complicated to try
to write.

I might be able to see why an unordered complete msg
could be delivered (kind of a priority thing).. but
even that will mess up your buffering..

> if the stack employs a strict model to only partially deliver one 

And this is what BSD does (unless you turn on fragment interleave
which could get you a msg from a different assoc).
> re-allocate to a new buffer, there will never be a huge amount of 
And as I said.. you may NOT get what was foretold in the the
extrcvinfo structure since the circumstances may have changed
by the time you read... the next to deliver may be back to
str 1..

> the application programmer.

I don't know if thats a good idea...

Where this interleave things is really going to come in is only
when huge messages are being sent.

and no matter what when you send a huge message you ARE going
to block other stream messages until all those TSN's can
get across...

R

---

From: Vlad Yasevich <[email-masked]>
Date: Wed, 18 Apr 2007 19:54:29 -0400

>> a bug in the implementation.

Just fixed it to a "nope" for lksctp as well.  LKSCTP will be a
strict <A> approach as of kernel 2.6.22.  I'll see if I can push this
fix to stable as well.

> Anyway.. I will go read the rest of the thread now ;-)

Here it is:

http://www1.ietf.org/mail-archive/web/tsvwg/current/msg05826.html

It's a fun thread to read again... enjoy ;)

-vlad

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 20:34:10 -0400

> Yes, I think the above is the intention.

Well I wrote the sentence and it WAS NOT what I intended....
in either case we need clarification I think :-D

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 20:42:45 -0400

> only interleaving between different associations.

Exactly.. any other interpretation is more an accident of
my very poor English skills :-D

> stack does (A), not (B). Or am I wrong?

I have not looked at the code recently.. but the intention
is (A).

We had this discussion like I said.. I need to go
find that thread and have a look at it.

I am not adverse to changing to (B).. but the intention
is (A). Note also that we want to do the thing that
makes programming the easier for the application. Not harder.
To me (B) makes it much harder, we can do this, but its
much more difficult for the app.. If we do (B) we need
to have a switch to enable/disable the behavior I think.
And even more so, either way, we need to better document
this behavior.

I vote for (A) .. just because its easier... coding wise
it makes little difference to the stack.. but it will
make a huge difference to the app writer.

a) the extinfo information may be wrong (as I illustrated)
b) multiple partial deliveries will happen.

All this due to a set of cases that will occur only
on message loss (which we usually think of as rare)
or huge message being sent with maybe loss as well.

In the case of huge messages, has I told the class I taught
today, if you send large messages you are GOING to have
head of line blocking.. since all the TSN's must be sequential
for reassembly.

I think we would be best to better define the intention (A) and
not do (B).. but as Kacheong as said, its just code.. I am
open to either method.

Let me also go back and find that other thread :-D

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 20:43:32 -0400

Michael Tuexen wrote:
> But I think it is important to interleave between different associations,
But I think the default behavior is fragment interleave OFF.
so with defaults the code does not break.

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 18 Apr 2007 21:50:08 -0400

Vlad:

Thanks for the pointer.. 11/28/05 - 12/??/05 is the
time frame here .. for you that keep tsvwg mail around (like
I do)..

And looking at this thread it convinces me that

Option <A> is correct
and
that we DO need to touch up the mis-leading wording that
I put in 3.14 that talks about one per stream.. and put
the wording BACK to the way it was.

The Fragment Interleave solves the DOS attack problem.. and
it is incorrect (as Mark Butler pointed out in Posix semantics)
to interleave different messages.

So, <A> is the right approach.. with the only exception of
Fragment Interleave aka different assoc's in a 1-2-many socket
may have interleaved delivery.. as the socket document says..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 18 Apr 2007 20:24:43 -0600

Randall,

> to interleave different messages.

Mark misinterpreted the POSIX specifications.

> may have interleaved delivery.. as the socket document says..

<B> is equally valid.

--brian

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 12:09:53 +0800

> layer protocols would need specify the stream information.

I think not necessarily.  If no stream is used and no unordered
message is sent, meaning that only send() and recv() calls are
used, (B) still works fine.  Ordered message are delivered in
sequence.  When only one stream is used, it means that while
partial delivery is going on, the next recv() call can only
receive the next part of the message, no other message can be
delivered.

> just happens to enter partial delivery.

I am not sure I understand the above.  Suppose a client makes
use of multiple streams and sends data to different streams in
an association, and the server receiving data from the client
does not understand the different streams.  Even without partial
delivery, it is unclear how the server can work in this case.
While data from every streams will arrive in sequence, but data
from different streams may interleave.  So how does it work?

> that says that says multi-streaming should be used, it might be too common.

If one side of an association uses different streams, I think
the other must also understand it.  Otherwise it cannot work.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 13:04:34 +0800

> stack does (A), not (B). Or am I wrong?

I think it depends on the client code, which is not
shown.  If only one stream is used, the code works
fine with (B).

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 13:08:52 +0800

> It needs to do a partial write when getting a partial message.

Maybe we don't need a sample piece of code at the end of the
draft.  The example does not show everything the API is capable
to do anyway.  I'd suggest remove it and then have several (or
many many :-) examples available from Randy's www.sctp.org.
Interested people can contribute examples also.  This may be
more useful.  Just a thought.

-- 

						K. Poon.
						[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 18 Apr 2007 23:08:34 -0600

Kacheong,

> the other must also understand it.  Otherwise it cannot work.

The server that cannot handle multiple streams can set the maximum
number of inbound streams to 1 when the association is formed.

--brian

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 07:13:30 +0200

If you take the Diameter protocol, the RFC says like "oh, well, you 
could use this new cool SCTP if you want, and streams seems to be 
good", but it does not state anything about message borders, if one is 
allowed to use the unordered flag etc.

And then we have all old apps that are written with TCP and then just 
the socket call is changed to get SCTP. In this cases I don't find it 
unlikely that a 200k ULP message will be split over several write() or 
send() calls, so a Diameter server will probably have to be prepared to 
handle Diameter messages that are split over several SCTP user 
messages. It is unlikely that there would be message border mismatch or 
bundling, but you never know what strange I/O they have at the client 
side ;-).

So I guess one have to adapt to the actual situation among clients when 
writing an application server.

On Wednesday 18 April 2007 22:21, Michael Tuexen wrote:
> Hi Anders,
>
> the protocols I know of, which use SCTP as their transport layer
> require that an upper layer message is one SCTP user message and is
> fragmented by SCTP if it is too  large. And I think this is the way
> SCTP is designed
> the protocol) limits the size of messages, this limits the size of
> the upper layer messages. But this is something the implementation
> has to overcome. So an SCTP implementation should be able to handle
> messages of the size of its upper layer correctly. This especially
> applies to the send() calls, in particular to the handling of signals
> during the send() call.
>
> > I just have access to the "Solaris 10 11/06 s10s_u3wos_10 SPARC"
> > and I think there are patches since then, so perhaps the problem is
> > already fixed in later patches. However, I can provide a test
> > program and do some further tests if Kacheong is interested. The
> > company I work for is
> > easier for the application programmer, not for the stack
> > implementors. I would guess that model A) actually make
> > implementing a stack harder. So, then there should be a real value
> > with model A) for the programmer.
> >
> > fragmented into multiple SCTP user messages and thus be delivered
> > in parts anyway.
> >
> > mode" and is doing write()/send() instead of sendmsg(). Of course,
> > the most natural way is to make a single write() per ULP message
> > even in such application, and it is most likely that a single
> > write() will lead
> > to a single SCTP message, and thus a 1-to-1 mapping is quite
> > likely. However, if the client has a separate buffered I/O layer
> > the 1-to-1 mapping may not be there, or, the ULP may have large
> > messages which will be split due to exceeding SO_SNDBUF.
> >
> > is 16 megabytes, messages never get that large in the normal case,
> > but it is not too unlikely that they exceed SO_SNDBUF for some
> > client. A client using non-blocking write() would thus sometimes
> > split a Diameter
> > Say that we require clients being properly implemented using
> > sendmsg (),
> > sending. I've been informed by Randall Stewart that BSD has some
> > EEOR mode such that you can actually send messages of any size, but
> > I don't think this is supported in for example Linux or Solaris.
> >
> > anyway (which for Diameter is a bit cumbersome, since the max
> > message size is a bit large), unless the ULP max message size is so
> > small one can assume that no client has an SO_SNDBUF that is
> > smaller.
> >
> > /Anders Torger
> >
> >> Solaris implementation below. I'm sure Kacheong wants to know
> >> about it such that Sun can fix it.
> >>
> >> Comments in-line.
> >>
> >> Best regards
> >> Michael
> >>
> >>> Take a Solaris 10 stack from a build November 11 2006 for
> >>> example, if recvmsg() returns a fragment of an ordered message,
> >>> the next call that delivers a message or fragment of a message
> >>> will be one of:
> >>>
> >>> more calls to recvmsg() will be necessary to completely consume
> >>> the message.  Only one message at a time per stream can be
> >>> partially delivered."
> >>>
> >>> long as it is delivered in full (naturally with the unordered
> >>> flag if partially deliverey is ordered and vice versa).
> >>
> >> delivering a message with sid = n and ssn = m - 1. This violates
> >> the rule that all ordered messages within each stream is delivered
> >> in order.
> >>
> >>> (at least if the input buffer is large enough), if you feed it
> >>> with unordered messages a bit larger than SO_RCVBUF is, the
> >>> association will
> >>> indeed hang, and if tried a few times a crash can be provoked, so
> >>> I guess that the actual plan is to deliver unordered messages
> >>> to stress the API... I would guess the test can be done between
> >>> two Solaris machines too, just increase SO_SNDBUF on the sending
> >>> machine).
> >>
> >>>    fragmented interleave.  Fragmented interleave, for a one to
> >>> many socket, is when subsequent calls to receive a message may
> >>> return parts of messages from different associations.  Some
> >>> implementations
> >>> line blocking amongst multiple associations sharing the same one
> >>> to many socket).  When this option is turned on, then each
> >>> receive call may come from a different association (thus the user
> >>> must receive data with the extended calls (e.g. sctp_recvmsg) to
> >>> keep track of which association each receive belongs to."
> >>>
> >>> applicable to one-to-one sockets. In other words, it seems like
> >>> the above text is written with the assumption that there is no
> >> association should not be able to block others. This is not a
> >> problem for 1-to-1, but someone (an attacker) could start sending
> >> a long fragmented message such the the partial delivery kicks off.
> >> If the attacker never sends the last fragment and fragment
> >> interleave is not enabled the application would never receive
> >> messages from the other associations. This allows a DOS attack. To
> >> avoid this, this option was introduced.
> >>
> >>> As an SCTP application programmer, this 'detail' is important.
> >>> With model A), it means that an application only needs to hold a
> >>> buffer for one message at a time, but with model B) there is
> >>> potentially a need to
> >>> messages simultaneously, and therefore the A) model is better
> >>> since it makes the
> >>> Anyone who knows where the draft is going - interleave or not,
> >>> and how are other stack implementations working in this area?
> >>>
> >>> /Anders Torger

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 13:29:08 +0800

> in to a partial delivery state)... This may have been the reason

Yes, I think the idea is to allow one partial delivery
per stream.  So if there are multiple streams, partial
delivery can happen for all of them.

> I need to go back and find that old thread :-D

I don't remember this thread at all...  Did I say
the opposite idea??

-- 

						K. Poon.
						[email-masked]

---

From: Mark Butler <[email-masked]>
Date: Wed, 18 Apr 2007 23:33:48 -0600

Hello all,

Comments inline.

>>> It seems to be two approaches, and the socket API draft is not clear 
>>> on which one it should be:
>>>

I hope it is clear that if you don't set SCTP_FRAGMENT_INTERLEAVE, you 
are (generally speaking) not going to get any interleaving at all.  That 
is relatively benign on a 1:1 socket, but may lead to serious stalling 
problems on a 1:M socket.

Since the draft 14 document for the SCTP_FRAGMENT_INTERLEAVE option 
doesn't say anything about interleaving messages on different streams on 
the same association, it seems to me that a neutral reading of the draft 
is that by default you typically get behavior (C) - no interleaving, and 
when SCTP_FRAGMENT_INTERLEAVE is turned on you get behavior (A).

However, it seems to me there are cases where both behavior (A) and 
behavior (B) have real world benefits. So I suggest that consideration 
be given to having an additional value for the SCTP_FRAGMENT_INTERLEAVE 
option as follows:

     0 - interleave off
     1 - interleave across different associations only
     2 - interleave across different streams within the same association

The good part about such a specification is that higher levels are 
backward compatible to lower levels.  An SCTP application can request 
any higher level, and the SCTP implementation can deliver behavior 
corresponding to any lower level without breaking anything.

The rationale for supporting level 0 is backward compatibility.

The rationale for supporting level 1 is that it simplifies application 
programming to know in some cases that you only need to handle one 
message per association at a time, and of course it eliminates the most 
serious inter-association blocking problems.

The rationale for supporting level 2 is that it allows multiple-stream 
applications to recover from single packet loss stutations more smoothly 
than otherwise - such that in many cases only messages on the affected 
stream(s) are delayed instead of messages on all streams. Plus the 
inter-association benefits of level 1 of course.

 - Mark B.

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 08:52:46 +0200

> from different associations. Therefore the program must be prepared

Yes, you're right, I was thinking in one-to-one terms and forgetting 
about the one-to-many case.

(I prefer one-to-one sockets such that I get flow control per assocation 
and can use poll() to check ready-for-writing in a meaningful way. From 
my analysis, the only real reason for one-to-many is to be able to have 
huge amount of associations without wasting sockets, and therefore I 
think that one-to-one should generally be the preferred way for an 
application programmer.)

/Anders

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 04:08:20 -0400

> I don't remember this thread at all...  Did I say
You said nothing.. but previous to the version 12 spec
the interleaving of ANY message was specifically prohibited.
The strm addition to 3.1.4 that Anders points out came in
version 12 of the spec.. and was made by me by mistake as
I was working through all the comments.

In the end we added FRAGMENT-INTERLEAVE and I forgot to
take back out that little "per stream" I added and of course
mention an exception to the rule for "per association" when
FRAGMENT_INTERLEAVE is on.

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 04:02:14 -0400

> receive the next part of the message, no other message can be

You forget about the setsockopt() to change the default
stream. I can do just plain send()/recv().. more correctly
sendto()/recvfrom() [since this is the one-to-many model] and
sue the socket option to change the stream number.

The naive receiver will break when the interleaved
messages begins.. go see that very long thread from
Nov of 2005...

Note also that is when the little caveat in 3.1.4 crept in. In
the middle of the discussion. Previously it stated only
one message may be partially delivered.

Version -11 of the spec said:
"
    If the SCTP stack is running low on buffers, it may partially deliver
    a message.  In this case, MSG_EOR will not be set, and more calls to
    recvmsg() will be necessary to completely consume the message.  Only
    one message at a time can be partially delivered.
"

This small change was my error in the middle of this discussion that
resulted in Fragment Interleave being put in. The next version will
change this back to what it was before since I should never
have made the change but it WILL add the note pointing to
Fragment Interleave ...

> delivery, it is unclear how the server can work in this case.
How so?

I recv msg's until I see an MSG_EOR. When I see that I have
a complete message. It does not matter to the receiver
at this point that the messages came in a different
stream.. The receiver can be completely oblivious to
the fact that the sender was using different streams
to parallelize the transaction requests being made.

> from different streams may interleave.  So how does it work?

If msg in stream 1 are always about transaction 1, and messages
in stream 2 are always about transaction 2, I have no problem
I read until I see MSG_EOR, process the message .. it has me
advance T-1 in some fashion. I read the next msg until MSG_EOR
and I advance T-2 then I read the next message and advance
T-1. As long as the message contains a transaction ID there
is no problem whatsoever. There are many many telco apps
that do this (or there used to be when I worked for the
bell system way way back :-D)

> the other must also understand it.  Otherwise it cannot work.

Nope.. you are completely incorrect. The receiver DOES have
to understand what transaction a message is associated with but
it DOES NOT have to understand it was sent in a different stream.
The stream assures me that the transactions themselves advance
in the correct order.

Doing interleaving of PD-API is a sure way to break such an
application.

If we want this added.. which I don't think we do.. then it
MUST have a socket option to turn it on and off and it
MUST be off by default. This is why the SCTP_FRAGMENT_INTERLEAVE
ended up off by default.

I think in general this should NOT be done..

Please go read all the threads .. there were many
messages on tsvwg..

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 04:06:12 -0400

> think that one-to-one should generally be the preferred way for an 
You are also forgetting that:

1) One-to-Many also allows you to send data on the 3rd leg
    of the handshake.
2) One-to-Many can also free you of any protocol state giving you
    more of a UDP like feel.

Its true if you are going to be sending large messages and filling
up your socket send buffers it may not be the right model for
you (since you cannot select on write).. but there are ways
around this.. and by the way .. at least in BSD .. you get
flow-control per association. I don't know how other implementations
have been written but BSD works that way no matter if you are
one-2-many or one-2-one.

So the only thing different is you can't do select on write.. which
can be gotten around with threads if you really want that.

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 04:22:24 -0400

comments in-line..

>     2 - interleave across different streams within the same association

I like this idea...

> The rationale for supporting level 0 is backward compatibility.

Exactly.. and a naive receiver can leave it set at the default
of 0.. and it will not break..

> serious inter-association blocking problems.

And of course many applications are dealing with small messages
anyway.. so they can set the PD-API point down to there small
message value and if they get something from such an attacker
they can abort its association.

> inter-association benefits of level 1 of course.

Yes, I can see some advantages in doing 2, but an app really
must know it is doing it. I would also say the app must have
enabled the SEND_RECV_INFO events to get level 2 on as well.
Otherwise it will NOT get the stream information it needs.
Level 0/1 don't have that problem but level 2 surely does.

Now I would imagine then this also means the app, if using
level 2, must also include the ability to have MULTIPLE
partial delivery's running on MULTIPLE streams from the
same association.

Now I know this is an extreme case, but it must be considered.
In fact in the general case, if you have no message loss
going on at all (and even some minor msg loss for that matter)
you are not going to see the PD-API happen.. unless of course
you are sending large messages.. (or set the PD-API point down
to a low value as I mentioned above) and in such a case there is
a HOL blocking that is going to happen amongst the streams
anyway.

I tell all my students when I give the socket API class I give
on occasion (like yesterday for example ;D) that they must
be aware that if they send a 2Meg message, and then shortly
there after send a 100 byte message. The 100 byte message
will get blocked by the TSN chunking of the 2Meg message.
This even includes if you had a level 2 set... since
all the data has to get across in TSN order, in general,
and so a large part of that message (several RTT's worth)
will need to be sent across.. assuming that the rwnd
is < 2Meg of course.

But in general I like your idea. And of course implementations
that are unwilling to support level 1 or level 2 can refuse
to let you set it .. giving a ENOSUPPORT... I would hope
everyone will support at LEAST level 0 and 1..

What do other think? If everyone agrees I will make a
wording pass for the next version of this...

R

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 16:55:23 +0800

[Attached is some typos I found in 5.2.3 of the draft
 related to sctp_extrcvinfo.]

> I read all in strm2 ext info tells me str 3 is next.. and
                   ^

What does the m denote?  So when the app reads in
partial message M1 from stream 1 and checks the
sctp_extrcvinfo, it find that the next read will
return data from stream 2.  Then it does a read
and gets back partial message M2 from stream 2.
And the sctp_extrcvinfo there says that the next
read will return data from stream 3.  Are you
referring to the above sequence of events?

> the previous return was...

Is sctp_extrcvinfo supposed to work as above?  The
intuitive way I think is that sctp_extrcvinfo tells
exactly what the next read will return if the flag is
SCTP_NEXT_MSG_AVAIL.  So in the above sequence of
events, when the app reads M2 and the sctp_extrcvinfo
shows that the next message is from stream 3, I think
the socket layer needs to deliver data from stream 3.
The app will be surprised otherwise.  And the socket
layer does not really need to know what has been
returned in sctp_extrcvinfo if all data is maintained
in sequence.  It is both natural and easy to do.

> Otherwise the app becomes very very complicated to try

It is really not that complicated.  An app needs to
handle multiple streams anyway if it uses this feature.
What it means is that for every read, the app needs to check
which stream the message belongs to.  For partial delivery,
it only means that the app needs to keep a buffer (per
stream) holding the partial data.  It is not adding
more complexity if the app can already handle multiple
streams.  What is the complexity referred to above?

> even that will mess up your buffering..

I think unordered message is always a problem, whether
there is partial delivery or not.

..

> by the time you read... the next to deliver may be back to

I think this is reducing the usefulness of
sctp_exrsndrcvinfo and a surprise to app programmers.  I
don't see the reason why this actually makes things easier
in the socket layer or the app.

> when huge messages are being sent.

This is no longer true after the introduction of partial
delivery point.  I think it is important to those apps which
make use of partial delivery point.

> to block other stream messages until all those TSN's can

It is not the case anymore.

-- 

						K. Poon.
						[email-masked]

Some typos.

> 5.2.3.  Extended SCTP Header Information Structure (SCTP_EXTRCV)
> 
>    This cmsghdr structure specifies SCTP options for SCTP header
>    information about a received message via recvmsg().  Note that this
>    structure is an extended version of SCTP_SNDRCV and will only be
                                         ^^^^^^^^^^^

Should be "sctp_sndrcvinfo"?

>    received if the user has set the socket option SCTP_USE_EXT_RCVINFO
>    to true in addition to any event subscription needed to receive
>    ancillary data..
> 
>    cmsg_level    cmsg_type      cmsg_data[]
>    ------------  ------------   ----------------------
>    IPPROTO_SCTP  SCTP_SNDRCV    struct sctp_sndrcvinfo
                   ^^^^^^^^^^^           ^^^^^^^^^^^^^^^

Should be "SCTP_EXTRCV" and "sctp_extrcvinfo"?

>    Here is the definition of sctp_sndrcvinfo:
                               ^^^^^^^^^^^^^^^

Should be "sctp_extrcvinfo"?

>    struct sctp_extrcvinfo {
>       uint16_t sinfo_stream;
>       uint16_t sinfo_ssn;
>       uint16_t sinfo_flags;
>       uint32_t sinfo_ppid;
>       uint32_t sinfo_context;
>       uint32_t sinfo_timetolive;
>       uint32_t sinfo_tsn;
>       uint32_t sinfo_cumtsn;
>       sctp_assoc_t sinfo_assoc_id;
>       uint16_t next_flags;
>       uint16_t next_stream;
>       uint32_t next_asocid;
>       uint32_t next_length;
>       uint32_t next_ppid;
>    };

I'm wondering if it should be something like

	struct sctp_extrcvinfo {
		struct sctp_sndrcvinfo	serinfo_sinfo;
		uint16_t		serinfo_next_flags;
		uint16_t		serinfo_next_stream;
		uint32_t		serinfo_next_aid;
		uint32_t		serinfo_next_length;
		uint32_t		serinfo_next_ppid;
	};

to be consistent with other structures.

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 17:11:38 +0800

> And even more so, either way, we need to better document

I still do not understand the reason why programming
with (B) is harder if an app already makes use of the
multiple stream feature.  In fact, I'd say that doing
(A) limits the flexibility of app programming.  Suppose
a sender app has multiple threads, each tied to a
specific stream sending data.  And the receiver uses a
dispatcher to send the received messages to different
threads based on the stream number.  Because of this,
it sets a convenient partial delivery point to avoid
longer than necessary pause.  If partial delivery in
one stream blocks delivery of messages from other streams,
it becomes a performance issue with the above programming
model.

> make a huge difference to the app writer.

I don't think it makes a huge difference to an app
writer if the app requires the use of multiple streams.
And to apps which do not send big messages nor set partial
delivery point, (A) and (B) are just the same.  I suspect
that this is the largest set of apps.

> a) the extinfo information may be wrong (as I illustrated)

Note that even with (A), if the socket layer works as
described in the last mail, the app has exactly the same
problem with 1-N style socket if it wants to avoid the
DoS attack.  So a robust app needs to handle it anyway.
But I'd say that the socket layer's handling of
sctp_extsndrcvinfo should not behave like what was described
in the last mail.  It is not intuitive to app programmer
and does not make programming in both the socket layer and
the app layer easier.

> not do (B).. but as Kacheong as said, its just code.. I am

I'd like to have (B) for its flexibility and for making
the use of partial delivery point more effective.  And I
don't think it actually makes app programming nor stack
implementation harder.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 17:53:31 +0800

> may have interleaved delivery.. as the socket document says..

In reading the thread, it seems to me that the reason for
the thread is exactly this sentence "only one message at a
time can be partially delivered" as it causes blocking.
So why do we still want to keep this blocking behavior?

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 18:25:35 +0800

> corresponding to any lower level without breaking anything.

I think the above suggestion is good.

-- 

						K. Poon.
						[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 18:27:11 +0800

> wording pass for the next version of this...

And please also correct the section on sctp_extsndrcvinfo.

-- 

						K. Poon.
						[email-masked]

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 12:17:08 +0200

On Thursday 19 April 2007 10:06, Randall Stewart wrote:
> > (I prefer one-to-one sockets such that I get flow control per
> > assocation and can use poll() to check ready-for-writing in a
> > meaningful way. From my analysis, the only real reason for
> > one-to-many is to be able to have huge amount of associations
> > without wasting sockets, and therefore I think that one-to-one
> > should generally be the preferred way for an application
> > programmer.)
>

I'm wandering a bit off topic here, but anyway...

I guess what you mean by flow control per association in one-to-many is
flow control enforced by the kernel, but it cannot be enforced by the
application per association anyway, right? Either you read from the
socket, or you don't read. If I don't want any more data for the moment
from a particular association, because I want to prioritize another one, 
I stop reading on that one-to-one socket, but continue reading on 
another.

On a one-to-many socket, I don't get to choose which association that 
will deliver data to me, which may be important in refined overload 
controls.

This is what I mean with that flow control is lost on a one-to-many
socket.

On the write side, when I have a message ready for association A but not
for B, and A is stalled, but not B, that will lead to a problem with
one-to-many, poll() says ready for writing, and sendmsg() says EAGAIN.
However, you say one can solve that problem with threads, but I don't
understand how?

If one needs to send data in the connection setup, it indicates that 
we're not using long-lived associations, and then the flow control 
issue becomes less important too. So one-to-many seems to be the choice 
for short-lived associations.

/Anders Torger

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 07:14:16 -0400

Anders:

Let me change the subject line then :-D

Anders Torger wrote:
> application per association anyway, right? 

The kernel always enforces flow control using the send and recv buffer
as far as the app is concerned. These are translated into an rwnd
to give to a peer... so both SCTP endpoints cooperate using the
rwnd to do flow-control.

Either you read from the
> I stop reading on that one-to-one socket, but continue reading on 

Ahh.. yes.. For this type of application flow control you can do
that very thing. I had not thought of that aspect...

Of course there would be nothing stopping a implementation from
also adding the ability to specify which association (or stream
for that matter) to read from.. I  have hesitated  in doing that in BSD
since this has some drastic consequences  after all.. just like
the application not reading from its socket. Since this means
you are sucking up kernel resources to do your flow control. It
is far better IMO, if you need that type of granular control, to
have a way to do this app - 2 -app.. and not eat up all your
mbuf/skbuf what ever you call them... for the app doing flow-control.

You can even envision a situation where if you had 20 sockets open
and 19 of them you did not read from.. you could get the 19 * the rwnd
mbufs in.. which stops you from getting things from the 1 you want...
and in such a situation there would be nothing the kernel could do.. it
could not reneg on the data in the 19 sockets because it already
cum-acked it all and sent it up to the application to read who is
refusing to do so. Such a method of flow control is very bad practice
and can lead to disaster .. especially if you are trying to write
a high availability telco app. I spent many many years doing telco
apps and we would never do such a thing.. since the consequences of it
are not pleasant.. :-o

> will deliver data to me, which may be important in refined overload 
But in such an overload situation you have a disaster waiting
to happen. Oh sure, most of the time it will work.. but if
the overload is big enough, your kernel will run out of
mbufs/skbufs etc... and then your one or two association you want to
read from you will get nothing on.. and the others will hold all
sorts of data you are refusing to read.

And of course it will only hit you on mothers-day or some other
very very bad situation... the worst actually..

Poor design choice IMO..

> This is what I mean with that flow control is lost on a one-to-many

And I see that now.. and I see how I did not even think about this
as well... its because it is IMO such a bad choice.. it is this
same reason that I have not suggested we let the receiver fill
in the assoc-id or the stream number to read specific things
from the socket buffer... its an easy thing to do.. but dangerous
and encourages this type of mis-guided flow control ;-(

> However, you say one can solve that problem with threads, but I don't

select and poll are a poor mans thread. It takes a single
threaded application and makes it "multi-threaded". One can easily
setup a worker thread to accept the data and manage getting it
down.. blocking if needed. You can create a "write" thread
per association if you so desired (not that I know I would do
this). The sender thread does a blocking write to that association.
And thus blocks. The other worker thread is still free to write
the data down the other association.

Of course this requires a bit more work.. I have actually been
thinking on writing such a library for folks though. I have
an MPI app that does non-blocking writes and tends to slam
a machine over and over again... in some situations.. such
a worker thread arrangement would solve two problems, yours
and his :-D

> for short-lived associations.

True.. which if you look at it, most connections in TCP on the
Internet are this way :-D

Blocking also only happens when you have large volumes of data
all at once that you want to send..

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 07:25:07 -0400

>  related to sctp_extrcvinfo.]

Great.. I will fix these...

>> I read all in strm2 ext info tells me str 3 is next.. and

It was early for me here out in Ft Collins when I wrote
that.. I just typed an extra letter for str = strm = stream.

> referring to the above sequence of events?
Yep.. it gets mis-directed since when it next
reads it gets data for the earlier PD-API going
on.
> SCTP_NEXT_MSG_AVAIL.  

Correct.

So in the above sequence of
> the socket layer needs to deliver data from stream 3.

And how will it know that the previous call even
asked to see the ext-rcvinfo?

> in sequence.  It is both natural and easy to do.

But this is exactly what you are advocating
to do.. deliver out-of-order

Str-1 is ahead of Str-3 in your socket buffer.
Str-1 had nothing at the last read.. but Str-3 did.
By the time the user cycles back around to read,
Str-1 has data, its the first entry in your socket
buffer.. so you will return it.

You would have to track additional state.. in order
to know which read to go get next.. overriding
the natural order of the socket buffer..

This is my point.

>> Otherwise the app becomes very very complicated to try

And the app that does level 2 for reading will need
to be very complicated  I think...:-D

> streams.  What is the complexity referred to above?

See one of Anders earlier posts about contiguous memory..

> there is partial delivery or not.

not sure I agree  here...
> in the socket layer or the app.

I agree that your concept of out of order stream
delivery does this.. aka level = 2 as Mark put it.

> make use of partial delivery point.

And so something like Mark's level=0, 1 or 2. .. might
be the right choice.

If you change the PD-API point, then you probably want
to set the level to 2.

(unless you are emulating a stream of bytes by having
  EEOR mode on on the sender side of course).

> It is not the case anymore.
It may not be the case.

But I don't think you should be at level 2 unless you explicitly
ask for it.

I can support the feature only if its an option as Mark proposed
and the default is 0.

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 07:26:17 -0400

> and does not make programming in both the socket layer and

Not true.. the address in the recvfrom() tells you it
came from a different association. I don't need the snd-rcv-info
structure to tell me that.

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 07:27:40 -0400

> And please also correct the section on sctp_extsndrcvinfo.
But of course :-D

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 07:28:19 -0400

So if I hear no objections by tomorrow, I will work on
this on the plane going back to Dulles.

R

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 19 Apr 2007 19:38:29 +0800

> asked to see the ext-rcvinfo?

The socket layer does not need to know.  The data is in a
queue so whatever it is next, the socket layer just returns
it.  That's why I think sctp_extsndrcvinfo should always be
able to return a correct info for the next stream, assoc_id
because it is already there in the queue.  But a partial
message may become complete when the app does the actual read.

> to do.. deliver out-of-order

It is not.  We are talking about what is being returned in
sctp_extrcvinfo should match the next read.  It does not
matter whether partial delivery is going on or not.

> buffer.. so you will return it.

No.  The sequence of event is that the app reads in the
partial delivered message M1 from stream 1.  In the
sctp_extsndrcvinfo which comes with M1, it indicates that
the next message is from stream 2.  Then when the app does
another read to read M2 from streams 2, the sctp_extsndrcvinfo
indicates that the next message if from stream 3.  Now
the rest of M1 comes in, and it will be put in the end of
the socket buffer, behind message M3 from stream 3.  When the
app does the next read, it will get M3, not the rest of M1.

> the natural order of the socket buffer..

I don't think it is necessary.

> See one of Anders earlier posts about contiguous memory..

In the same message, he wrote down as easy way to solve
it :-)  I don't think it is complicated.

> I can support the feature only if its an option as Mark proposed

OK.

-- 

						K. Poon.
						[email-masked]

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 14:23:54 +0200

Randall,

comments inline:

On Thursday 19 April 2007 13:14, you wrote:
> >> flow-control per association. I don't know how other
> >> implementations have been written but BSD works that way no matter
> >> if you are one-2-many or one-2-one.
> >>
> >> So the only thing different is you can't do select on write..
> >> which can be gotten around with threads if you really want that.
> >
> > I guess what you mean by flow control per association in
> > one-to-many is flow control enforced by the kernel, but it cannot
> > be enforced by the application per association anyway, right?
>
> The kernel always enforces flow control using the send and recv
> buffer as far as the app is concerned. These are translated into an
> rwnd to give to a peer... so both SCTP endpoints cooperate using the
> rwnd to do flow-control.
>
>
> Either you read from the
>
> > socket, or you don't read. If I don't want any more data for the
> > moment from a particular association, because I want to prioritize
> > another one, I stop reading on that one-to-one socket, but continue
> > reading on another.
>
> for that matter) to read from.. I  have hesitated  in doing that in
> BSD since this has some drastic consequences  after all.. just like
> the application not reading from its socket. Since this means you are
> sucking up kernel resources to do your flow control. It is far better
> IMO, if you need that type of granular control, to have a way to do
> this app - 2 -app.. and not eat up all your mbuf/skbuf what ever you
> call them... for the app doing flow-control.

Sorry, I don't understand the "this app - 2 -app..." part, so I'm not 
sure what you're saying.

Anyway, it seems to me that it makes little sense to provide this 
possibility for one-to-many sockets. If you only want to read from one 
association, why not use one-to-one socket? Only if the stack can scale 
association count much more than the kernel can scale socket count this 
seems to be reasonable.

Specifying which *stream* to read from is broken, since SCTP has no flow 
control per stream. The idea here is to make the application indirectly 
control affect the flow control on the protocol level (just as for 
TCP), not make selective reads from a single association.

If you want send data with the association setup, there's the trick to 
have a one-to-many socket for making association setups, and as soon as 
it is set up, you do sctp_peeloff() and continue with one-to-one mode.

> and 19 of them you did not read from.. you could get the 19 * the
> rwnd mbufs in.. which stops you from getting things from the 1 you
> want... and in such a situation there would be nothing the kernel
> could do.. it could not reneg on the data in the 19 sockets because
> it already cum-acked it all and sent it up to the application to read
> who is refusing to do so. Such a method of flow control is very bad
> practice and can lead to disaster .. especially if you are trying to
> write a high availability telco app. I spent many many years doing
> telco apps and we would never do such a thing.. since the
> consequences of it are not pleasant.. :-o

In a telco app I never really want to get into flow control anyway since 
that blocks potentially high priority ULP requests. So for a telco app 
this is perhaps a bit theoretical...

However, it seems from your description above that all sockets share the 
same buffer space, that is those 19 sockets eat up all buffer space, 
including that for the remaining? It sounds really strange to me that 
one socket can eat up buffer space for another socket (and thus reduce 
its rwnd). Is this the way BSD is implemented?

How can multiple bulk transfers using TCP work at all if this is the 
way, were direct or indirect application flow control is a natural part 
of the deal?

> > On a one-to-many socket, I don't get to choose which association
> > that will deliver data to me, which may be important in refined
> > overload controls.
>

I'm wondering here if this is due to poor design in the BSD kernel 
rather than the application ;-). I don't think you can run out of 
skbufs in Linux in the way you describe (or at least the way I 
interpret you description), but please correct me if I'm wrong.

When kernel fill up its SO_RCVBUF, which is per association, the rwnd 
will go down to zero, and the peer will send no more data, and if it 
would do anyway, the kernel would drop the message. So I don't see how 
it could lead to a disaster.

> and encourages this type of mis-guided flow control ;-(

Association could make some sense, but then use one-to-one instead. 
Stream number never make sense, since there's no flow control per 
stream in the protocol.

> > On the write side, when I have a message ready for association A
> > but not for B, and A is stalled, but not B, that will lead to a
> > problem with one-to-many, poll() says ready for writing, and
> > sendmsg() says EAGAIN. However, you say one can solve that problem
> > with threads, but I don't understand how?
>

Having I/O worker threads is a common design. But if you do blocking 
I/O, you would want to have one I/O thread per association, and if you 
have that, you would use one-to-one sockets. But having one thread per 
association is bad news scalability-wise, so it is better to have a 
fixed-size pool of I/O threads, which each do poll() and non-blocking 
I/O. And then in most cases it is shown that it is usually not much 
idea to have more than one I/O thread (or separate input and output 
into own threads).

Soon we're going to get into POSIX AIO... aio_read(), aio_write()... ;-)

/Anders Torger

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 09:09:19 -0400

> Sorry, I don't understand the "this app - 2 -app..." part, so I'm not 

What I am saying is you are best to have the client/server do flow
control between themselves.. or peer to peer .. or whatever you
application is..

> TCP), not make selective reads from a single association.

You would be surprised how many times I have had folks ask
for it though..

I think the idea is they want to put a thread on each
stream reading.. so you don't have a flow control issue
really.. but it always bothered me that you could have one
stream (or even assoc) not get read from ..

> it is set up, you do sctp_peeloff() and continue with one-to-one mode.
true.
> this is perhaps a bit theoretical...

of course.. but you still have to allow for it.. thats why SS7 had
gapping if I remember the term right... there are other means
that have been used as well..
> one socket can eat up buffer space for another sock

I think this is the way ALL kernels work. Think about what
you are doing. You have a limited number of mbufs in BSD, skbufs in
linux.. or whatever other O/S's call them. These are special purpose
memory that is used to hold data from the network. They are flexible
but there is a limited number of these critters..

Its a shared pool that ALL sockets end up drawing from.

> way, were direct or indirect application flow control is a natural part 

Because applications don't stop reading in general. The flow
control is to help with small application hickups.. not to
be used IMO has the general flow control of all apps.

If I look at my BSD system I am typing this on and
do netstat -m I see:

256/134/390/25600 mbuf clusters in use (current/cache/total/max)

A cluster is what really holds the data.. and not the max.

So my machine can have 25,600 2k chunks of data pending
into my various TCP and SCTP associations.. yes TCP and
SCTP draw on the same pool.

In fact a standard ethernet driver, grabs a mbuf/cluster
dumps the frame off the wire into it and then kicks it
up to IP.. IP sees its either SCTP or TCP  or UDP and kicks it
to the right one. If you need to reassemble you hold
mbufs (or skbufs in linux) to do it.

When you deliver to a socket, you give up the skbuf.. or mbuf
which is read by the socket read calls and copied out
of the mbuf (where it was put by the driver) into the
user memory. It gives you efficiency since you are not
making multiple copies of the data over and over again.
This is what kills performance when you are trying to
achieve 1 or 10 Gig ... extra copies.. just ask the iSCSI
folks :-D

> rather than the application ;-). 

No go look at ANY kernel. BSD linux.. any that you can
look at the source and you will see this behavior.

The poor design choice is the app expecting to use kernel
memory to do its buffering because it wants to
flow control someone. That is just bad design IMO

I don't think you can run out of
> interpret you description), but please correct me if I'm wrong.

You are wrong.. you will eventually run out of skbufs. No system
has unlimited resources to throw at things.. it does not happen
often but when it does it either effects the app or can echo
to something far worse.

> it could lead to a disaster.

How many connections are you going to support. A big web server
may want 10,000 connections or more. If you give each of them
a receive buffer of 64k in the kernel (not an unreasonable number
and it probably needs to be much higher to get Gig speed) then
you are talking about 625 Megabytes of memory. Now I will grant
you we have come a long way in getting memory in your machines,
but no one lets that much memory be dedicated to the network.

When you get a so buffer reservation it does not reserve the
memory for you, its just like the airlines.. your seat can
be given up if you don't show up..

> Stream number never make sense, since there's no flow control per 

Go tell that to all the people that have asked me about it.. as I
said they are all thinking multi-threading I am sure.. I have never
supported the idea.. but its what folks have asked for :-D

> have that, you would use one-to-one sockets. 

Maybe in app space, but not necessarily in kernel space..

But having one thread per
> idea to have more than one I/O thread (or separate input and output 

So are you saying there are limits on threads too? Just like mbufs or
skbufs?

I agree.. but I don't expect a 1 to many socket to have 10,000 assoc's
under it either :-D

> Soon we're going to get into POSIX AIO... aio_read(), aio_write()... ;-)

won't that be fun :-D

R

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 15:51:52 +0200

On Thursday 19 April 2007 15:09, Randall Stewart wrote:
> > However, it seems from your description above that all sockets
> > share the same buffer space, that is those 19 sockets eat up all
> > buffer space, including that for the remaining? It sounds really
> > strange to me that one socket can eat up buffer space for another
> > sock
>

Yes, I understand, but I thought the overcommitting idea was only a BSD 
thing... I have experienced running out of mbufs on FreeBSD, but never 
run out of skbufs in Linux. But I really don't know much of the 
implementation, it's just my experience. Perhaps I've been nicer to my 
precious Linux machines.

I shall investigate in detail how this work in Linux and Solaris. 
Overcommitting kernel buffers in a carrier grade system sounds like a 
rather bad idea to me. But as you say, 10000 sockets will eat up a lot 
of rarely used RAM if overcommitting is not allowed.

Thank you for pointing this out to me.

> > fixed-size pool of I/O threads, which each do poll() and
> > non-blocking I/O. And then in most cases it is shown that it is
> > usually not much idea to have more than one I/O thread (or separate
> > input and output into own threads).
>
> I agree.. but I don't expect a 1 to many socket to have 10,000
> assoc's under it either :-D

I'm saying that having threads per session, threads per socket, threads 
per X is a bad idea in general. What happens is that if you increase 
load on that system in a way such that more threads are created, the 
performance will degenerate and eventually there'll be some sort of 
trashing, one will do more work switching threads than doing actual 
work. In a carrier grade application server the number of threads 
should be moderate and static.

/Anders Torger

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 10:06:35 -0400

> implementation, it's just my experience. Perhaps I've been nicer to my 

Or perhaps the defaults are much higher as the limit. These
have raised in BSD lately I think.. you used to configure
them in the kernel... now they are done as a percentage
of memory I think.. this is probably what linux has been
doing as well... which means if you have a lot of memory
you are LESS likely to see the problem.

Also remember that most apps don't use the kernel in this
way.

A web server writes to his clients.. until he gets
all the data sent.

The receiver reads and display's it as fast as possible
to give a user a "good feeling" on how the web pictures
are downloading. None of them stop and do flow control by
not reading.. the flow control the kernel provides IMO is more
designed for when the web app is taking a bit more to
render a picture... this gives a temporary buffer for it
to get the jpg displayed (or whatever) and get back and
read more...

> of rarely used RAM if overcommitting is not allowed.

I don't think ANY system I have ever looked at (and I have looked
at a lot of kernel source code) does not do over-committing. Its
a sound principle as long as apps don't start designing all there
flow control in by making the kernel hold all their data.

> Thank you for pointing this out to me.

No problem ;-D Its an interesting discussion.. as these
things normally are...

> should be moderate and static.

Maybe.. depends on the types of threads actually being used.. and
how much it costs to context switch threads vs proc's and a
whole host of other things. Often times with a threaded model you
can avoid the overhead of a process context switch... which is
usually ( but not always) more expensive. A select/poll always
gives you a context switch... of course often times recv/send will
do the same .. but it depends on how the threading model has been
built into the kernel of course :-D

R

---

From: "Peter Lei (peterlei)" <[email-masked]>
Date: Thu, 19 Apr 2007 10:14:45 -0500

> inter-association benefits of level 1 of course.

I like this proposal:  it allows an app to explicitly specifying
what behavior an application desires/expects, and the default case
being all interleave off.  Depending on what HOL blocking I care
about, I can move up the level.

thanks,
--peter

---

From: Vlad Yasevich <[email-masked]>
Date: Thu, 19 Apr 2007 11:26:51 -0400

> inter-association benefits of level 1 of course.

This is a good way forward.  I like this proposal.

-vlad

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 17:32:38 +0200

Randall,

I've been unclear in my reasoning. The question here is not about 
letting the kernel do the buffering for you. One should always read as 
soon as possible, I don't want to suggest anything else although I may 
have done so in my babbling about flow control. So I agree with you 
totally that it is bad design.

The question on the receive side is actually about priority.

During overload, in this case meaning that we get more incoming data 
than we can read, I want to have the possibility to prioritize 
associations differently.

That is, I may have a high priority client that I want to always treat 
first. Thus I choose to read from that socket, and I can if I want to 
even have strict priority (although that is probably a bad idea), 
because as soon as there is no data on that socket, I will read from 
any other socket with data available.

This means that the rate we return mbufs/skbufs to the kernel will be 
equal, even though we don't read from some of the sockets. So there's 
nothing wrong with the design here.

That is, the example with 20 sockets and not reading from 19 of them, 
this only happens when we are so heavily overloaded that we only have 
time to read from the high prio socket, and there's always data on it. 
Having a round-robin dispatching or use a one-to-many socket would not 
change anything, we would still consume kernel buffers at the same 
rate.

So, with one-to-one sockets, I can do prioritization between 
associations in overload situations, and favour it concerning response 
time in normal but bursty situations. This is not possible with 
one-to-many sockets. Oh, well, it is, since one will have a queue where 
read-out messages are put before being parsed, but using one-to-one the 
hidden scheduling artifacts is moved farther out.

I'm the first to confess that the value of this prioritization is not 
that easy to quantify, and probably is rather limited.

So I am not claiming this is a strong argument in favour of one-to-one 
sockets.

What actually is the main reason why I favour one-to-one sockets is not
this priority thing on the receive side, but on the sending side, to 
make my application robust against stalls at the peers.

Say my app is a server which has 2 peers connected with long-lived
assocations.

Then I want to send data to only one of them as urgently as possible. 
But that peer happens to be stalled, and has rwnd down at zero. 
However, the other one is not.

In this situation I want to wait until rwnd increases at the stalled 
peer, and then send. With a one-to-one socket, I simply put it into 
poll(), and wait until poll() says its ready for writing.

With a one-to-many socket on the other hand, I'm doomed ;-).

Well, the strategy with one-to-many would be to assume that stalling 
peers is not common (probably true), and if that would happen, one 
either makes a sleep/retry-polling loop, or peels it off to a 
one-to-one socket.

Peeling off to a one-to-one socket for that case only really makes sense
if it is possible to put it back to the one-to-many socket again when
there is no stall any longer, but there's no API function for that. It
also makes sense if we peel it off just to send an error message and 
then close the connection (for example if we say stalled peer is an 
error condition, but it is probably too aggressive to peel off and 
close at the first EAGAIN). But having some connections on one-to-one 
and others on one-to-many doing the same processing does not make 
sense.

The sleep/retry-polling loop would work, but will introduce some 
latency, unless we do away with the sleep and do busy wait, but that is 
not very attractive.

Thus, this is the main reason I recommend using one-to-one sockets. I 
can change my mind though, but I haven't come up with any strong 
arguments for one-to-many, except possibly for short-lived 
associations. But then again, I come mainly from a TCP world, not UDP. 
And if I was a UDP programmer to begin with (more common in telco I 
guess), I guess I could favour one-to-many because it is more similar 
to UDP.

/Anders Torger

On Thursday 19 April 2007 16:06, Randall Stewart wrote:
> >> you are doing. You have a limited number of mbufs in BSD, skbufs
> >> in linux.. or whatever other O/S's call them. These are special
> >> purpose memory that is used to hold data from the network. They
> >> are flexible but there is a limited number of these critters..
> >
> > Yes, I understand, but I thought the overcommitting idea was only a
> > BSD thing... I have experienced running out of mbufs on FreeBSD,
> > but never run out of skbufs in Linux. But I really don't know much
> > of the implementation, it's just my experience. Perhaps I've been
> > nicer to my precious Linux machines.
>
> > Overcommitting kernel buffers in a carrier grade system sounds like
> > a rather bad idea to me. But as you say, 10000 sockets will eat up
> > a lot of rarely used RAM if overcommitting is not allowed.
>
> >>> association is bad news scalability-wise, so it is better to have
> >>> a fixed-size pool of I/O threads, which each do poll() and
> >>> usually not much idea to have more than one I/O thread (or
> >>> separate input and output into own threads).
> >>
> >> So are you saying there are limits on threads too? Just like mbufs
> >> or skbufs?
> >>
> > I'm saying that having threads per session, threads per socket,
> > threads per X is a bad idea in general. What happens is that if you
> > increase load on that system in a way such that more threads are
> > created, the performance will degenerate and eventually there'll be
> > some sort of trashing, one will do more work switching threads than
> > doing actual work. In a carrier grade application server the number
> > of threads should be moderate and static.
>

---

From: Anders Torger <[email-masked]>
Date: Thu, 19 Apr 2007 17:36:03 +0200

From my point of view, this seems to be a very sensible approach.

It should also be made clear that option 2 is applicable to one-to-one 
sockets. That is, one-to-many either 0, 1, or 2, and for one-to-one, 0 
or 2.

The default being 0 for both types.

/Anders

On Thursday 19 April 2007 07:33, Mark Butler wrote:
> >>> It seems to be two approaches, and the socket API draft is not
> >>> clear on which one it should be:
> >>>
> I hope it is clear that if you don't set SCTP_FRAGMENT_INTERLEAVE,
> you are (generally speaking) not going to get any interleaving at
> all.  That is relatively benign on a 1:1 socket, but may lead to
> serious stalling problems on a 1:M socket.
>
> doesn't say anything about interleaving messages on different streams
> on the same association, it seems to me that a neutral reading of the
> draft is that by default you typically get behavior (C) - no
> interleaving, and when SCTP_FRAGMENT_INTERLEAVE is turned on you get
> behavior (A).
>
> behavior (B) have real world benefits. So I suggest that
> consideration be given to having an additional value for the
> SCTP_FRAGMENT_INTERLEAVE option as follows:
>
>      0 - interleave off
> The rationale for supporting level 1 is that it simplifies
> application programming to know in some cases that you only need to
> handle one message per association at a time, and of course it
> eliminates the most serious inter-association blocking problems.
>
> The rationale for supporting level 2 is that it allows
> multiple-stream applications to recover from single packet loss
> stutations more smoothly than otherwise - such that in many cases
> only messages on the affected stream(s) are delayed instead of
> messages on all streams. Plus the inter-association benefits of level
> 1 of course.
>
>  - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 18:29:49 -0400

> So I am not claiming this is a strong argument in favour of one-to-one 
Ok I understand a bit better..

> With a one-to-many socket on the other hand, I'm doomed ;-).

I don't think I can remember using select/poll on write
more than the fingers on my right hand times in my to
many years of writing code..

I understand why one would want this.. but on the other hand
I don't see it as that valuable..
> either makes a sleep/retry-polling loop, or peels it off to a 

Yes.. and one could peel-off (thats one of the reasons peel-off
exists) and also one could start an explict thread to deal with
the one association (a blocking write to it) and go back to
doing the sending and reading with the other thread..

There is always more than one way to skin a cat :-D
> there is no stall any longer, but there's no API function for that. 

I believe we discussed it but had no demand for it :-D

Also it might be quite messy to do.. then again maybe not :-o

It
> latency, unless we do away with the sleep and do busy wait, but that is 

Nope.. a busy wait is bad.. you are better to make the assoc blocking
(since you must be non-blocking) have a thread do a blocking write
and then make the socket non-blocking..

The one thread will block with the send and at completion of its
write exit.

> associations. But then again, I come mainly from a TCP world, not UDP. 
Which I had guessed ;-D

> guess), I guess I could favour one-to-many because it is more similar 

Yep..

R

---

From: Randall Stewart <[email-masked]>
Date: Thu, 19 Apr 2007 18:50:32 -0400

> The default being 0 for both types.

Ack..

I will work on this in the plane tomorrow am and see if I
can't put out some drafty text here shortly.

R

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 20 Apr 2007 09:06:17 +0200

Dear all,

I like the idea of having a socket option to support multiple modes.

2 should be called
interleave between different associations and also between diferent  
streams of the same association.

Mode 0 and 2 apply to 1-to-1 sockets, 0 being the default.

Mode 0, 1, 2 apply to 1-to-many models. But zero provides a security  
risk. So I'm
wondering if it would be better to make 1 he default of 1-to-many  
sockets. So the
application writer has to set a socket option and provide code for  
not being
affected by the security risk.
If we make 0 the default we have to put a warning in and provide text  
in the
security section. But I'm not sure if everyone will read the text of the
socketoption when he's not going to use it.

So I'm supporting the socket option, it is a nice thing. But I also  
vote the 1
being the default for 1-to-many style sockets.

Best regards
Michael

On Apr 19, 2007, at 7:33 AM, Mark Butler wrote:

> doesn't say anything about interleaving messages on different  
> streams on the same association, it seems to me that a neutral  
> reading of the draft is that by default you typically get behavior  
> (C) - no interleaving, and when SCTP_FRAGMENT_INTERLEAVE is turned  
> on you get behavior (A).
>
> backward compatible to lower levels.  An SCTP application can  
> request any higher level, and the SCTP implementation can deliver  
> behavior corresponding to any lower level without breaking anything.
>
> The rationale for supporting level 2 is that it allows multiple- 
> stream applications to recover from single packet loss stutations  
> more smoothly than otherwise - such that in many cases only  
> messages on the affected stream(s) are delayed instead of messages  
> on all streams. Plus the inter-association benefits of level 1 of  
> course.
>
> - Mark B.
>

---

From: "Henrik Jensen (TED)" <[email-masked]>
Date: Fri, 20 Apr 2007 13:45:58 +0200

Mark Butler wrote:
> So I suggest that consideration be given to having an additional value 
> for the SCTP_FRAGMENT_INTERLEAVE option as follows:
>
>     0 - interleave off
Question: Is it clear whether or not SCTP_FRAGMENT_INTERLEAVE=2 still 
limits the number of concurrent partial deliveries per association to 1 
only?

If not, I believe this change in SCTP_FRAGMENT_INTERLEAVE does also 
require a modification to the
SCTP_PARTIAL_DELIVERY_EVENT notification. The notification is only able 
to inform about the abortion of partial delivery per association - not 
which stream the abort is concerning.

I dont think it is acceptable to let the SCTP_PARTIAL_DELIVERY_ABORT 
abort all the current ongoing partial deliveries of the association. In 
some cases (e.g. the support of the partial reliability functionality) 
not _all_ the ongoing partial deliveries should be aborted.

As an alternative, we could still limit the number of concurrent partial 
deliveries per association to 1 only.

Regards
Henrik

-- 
Henrik Jensen                Phone   Job: +45 3091 8461
Ericsson Telebit A/S             Private: +45 8628 1084
Skanderborgvej 232           Email   Job: [email-masked]
DK-8260 Viby J,  Denmark         Private: [email-masked]

---

From: Mark Butler <[email-masked]>
Date: Fri, 20 Apr 2007 19:15:47 -0600

Comments in-line.

Henrik Jensen (TED) wrote:

> Mark Butler wrote:
>
>> So I suggest that consideration be given to having an additional 
>> value for the SCTP_FRAGMENT_INTERLEAVE option as follows:
>>
>>     0 - interleave off
> limits the number of concurrent partial deliveries per association to 
> 1 only?

No.  I believe there are many cases where support for multiple 
concurrent partial deliveries on different streams of the same 
association would be beneficial.

> SCTP_PARTIAL_DELIVERY_EVENT notification. The notification is only 
> able to inform about the abortion of partial delivery per association 
> - not which stream the abort is concerning.
>
> abort all the current ongoing partial deliveries of the association. 
> In some cases (e.g. the support of the partial reliability 
> functionality) not _all_ the ongoing partial deliveries should be 
> aborted.

Good point.  I suggest that a "uint16_t pdapi_stream" field be added to 
the sctp_pdapi_event structure for that very purpose. 

I might add as a further suggestion, that the cleanest way to handle 
messages that expire during partial delivery is to adopt the convention 
of terminating them with a zero length read with both MSG_EOR and 
MSG_TRUNC set.   There are several advantages to that convention - it is 
simpler, it is easier to program, and naive applications will be less 
likely to run messages together.  Proper compatibility could be 
maintained by only delivering such semantics to applications that do 
_not_ subscribe to partial delivery events. 

 - Mark B.

---

From: Kacheong Poon <[email-masked]>
Date: Sat, 21 Apr 2007 13:13:34 +0800

> _not_ subscribe to partial delivery events.

What is the original reason for SCTP_PARTIAL_DELIVERY_EVENT?
It seems to me that using MSG_TRUNC is probably a better
indication.  And the app handling code is simpler.

-- 

						K. Poon.
						[email-masked]

---

From: Mark Butler <[email-masked]>
Date: Sat, 21 Apr 2007 01:08:35 -0600

>indication.  And the app handling code is simpler.

I don't know the original reason, but I anticipate a couple of minor 
objections: POSIX specifies that recvmsg() should return zero after "no 
messages are available to be received and the peer has performed an 
orderly shutdown" (such as a TCP half close for example).  All though 
POSIX does not require it, I believe some people assume the reverse - 
that such an "end-of-file" condition is the only time recvmsg ever 
returns zero bytes read.  This convention, of course, would require such 
a zero length read in many cases.

POSIX also specifies that MSG_TRUNC be set on SOCK_DGRAM style sockets 
when the available data exceeds the buffer size.  It does not anticipate 
protocol initiated truncation, and using MSG_TRUNC in this way would be 
an extension of POSIX semantics.  It appears to be a clean extension to 
me - either way MSG_TRUNC means you do not have the whole message.  

Of course we do not currently support SOCK_DGRAM style read semantics, 
but if we did there would be a slight ambiguity when the number of bytes 
available on a PR-SCTP terminated message was exactly equal to the 
provided buffer size. Although I can't imagine what difference it would 
make in practice, strictly speaking we would need an additional flag 
added to sctp_sndrcvinfo to tell the difference in that case.

 - Mark B.

---

From: Anders Torger <[email-masked]>
Date: Sat, 21 Apr 2007 09:21:50 +0200

It seems to me that for SCTP (since we have partial delivery) the only 
time I will ever receive MSG_TRUNC is for an aborted partial delivery. 
This is good if it is true, because it makes the case easy to handle.

Thus, for an aborted partial delivery I would simply look for MSG_TRUNC, 
I don't need to check for MSG_EOR in that case if I don't want to.

Additionally, I think it also would be possible that recvmsg() returns > 
0 and MSG_TRUNC as well?

Would this example code work today (Solaris, Linux my primary 
interests), or do we have to listen to the SCTP_PARTIAL_DELIVERY_EVENT 
to catch this case?

count = recvmsg(sock, &msg, 0);
switch (count) {
case -1:
    ...
case 0:
    /* 
       Either a clean shutdown, or we got a partial delivery abort,
       which can happen when PR-SCTP is used or when an association is
       aborted.
     */
    if ((msg.msg_flags & MSG_TRUNC) == 0) {
        return PARTIAL_DELIVERY_ABORT;
    }
    return CLEAN_SHUTDOWN;
default:
    if ((msg.msg_flags & MSG_TRUNC) == 0) {
        return PARTIAL_DELIVERY_ABORT;
    }
    ...
}

/Anders Torger

On Saturday 21 April 2007 07:13, Kacheong Poon wrote:
> Mark Butler wrote:
> > Good point.  I suggest that a "uint16_t pdapi_stream" field be
> > added to the sctp_pdapi_event structure for that very purpose.
> > I might add as a further suggestion, that the cleanest way to
> > handle messages that expire during partial delivery is to adopt the
> > convention of terminating them with a zero length read with both
> > MSG_EOR and MSG_TRUNC set.   There are several advantages to that
> > convention - it is simpler, it is easier to program, and naive
> > applications will be less likely to run messages together.  Proper
> > compatibility could be maintained by only delivering such semantics
> > to applications that do _not_ subscribe to partial delivery events.
>

---

From: Mark Butler <[email-masked]>
Date: Sat, 21 Apr 2007 02:30:14 -0600

>Additionally, I think it also would be possible that recvmsg() returns > 
If the convention were adopted, yes.   Some applications might benefit 
from getting the remainder of a partial message, but it would likely be 
implementation dependent whether any such data was available.

>interests), or do we have to listen to the SCTP_PARTIAL_DELIVERY_EVENT 
No. I don't think anyone implements this yet.

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Sat, 21 Apr 2007 07:15:04 -0400

> association would be beneficial.

I agree, once you have multi-stream delivery you can have
each one being partially delivered.

> the sctp_pdapi_event structure for that very purpose.

Yep.. we will need that.. I will circle back and add it.

R

---

From: Randall Stewart <[email-masked]>
Date: Sat, 21 Apr 2007 07:40:03 -0400

Mark Butler wrote:
>> Thus, for an aborted partial delivery I would simply look for 
>> MSG_TRUNC, I don't need to check for MSG_EOR in that case if I don't 
>> want to.
>>
>> Additionally, I think it also would be possible that recvmsg() returns 
>> > 0 and MSG_TRUNC as well?
>>  
>>
Actually you are incorrect ;-)

BSD supports the pd-api event.. And by the way it also
supports sending a MSG_TRUNC with the last bit of data
being read. I believe there has to be data though..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 21 Apr 2007 08:44:17 -0600

Randall,

> being read. I believe there has to be data though..

OpenSS7 sockets stack returns MSG_TRUNC on aborted partial delivery.  If data
has already been written to the user on the call at the time of the abort,
read(), recv(), recvfrom(), recvmsg() return non-zero.  If the user was
blocked at the time of the abort and not data was yet transferred, read(),
recv(), recvfrom(), recvmsg() return 0, (recvmsg() with MSG_TRUNC set).

So I think Anders' code will work as is.

--brian

---

From: Anders Torger <[email-masked]>
Date: Sat, 21 Apr 2007 17:28:53 +0200

On Saturday 21 April 2007 16:44, Brian F. G. Bidulock wrote:
> OpenSS7 sockets stack returns MSG_TRUNC on aborted partial delivery. 
> If data has already been written to the user on the call at the time
> of the abort, read(), recv(), recvfrom(), recvmsg() return non-zero. 
> If the user was blocked at the time of the abort and not data was yet
> transferred, read(), recv(), recvfrom(), recvmsg() return 0,
> (recvmsg() with MSG_TRUNC set).
>

Except for one thing...

if ((msg.msg_flags & MSG_TRUNC) == 0) 
                                ^^
should of course be !=, or like true C hackers want it, just 

if (msg.msg_flags & MSG_TRUNC)

..but I kind of like to have a comparison there. I have a tendency to 
put in the wrong logical operator though... I guess that's why I'm 
doing design rather than coding these days :-)

/Anders Torger

---

From: Mark Butler <[email-masked]>
Date: Sat, 21 Apr 2007 13:29:32 -0600

Randall Stewart wrote:

>> Anders Torger wrote:
>>
>>> It seems to me that for SCTP (since we have partial delivery) the 
>>> only time I will ever receive MSG_TRUNC is for an aborted partial 
>>> delivery. This is good if it is true, because it makes the case easy 
>>> to handle.
>>>
>>> interests), or do we have to listen to the 
>>> SCTP_PARTIAL_DELIVERY_EVENT to catch this case?
>>
> ...
>
> being read. I believe there has to be data though.

I am glad to hear there are a couple of precedents.

Does anyone see a problem with returning MSG_TRUNC (and MSG_EOR) with a 
zero length read when necessary?  That would certainly be the usual case. 

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Sun, 22 Apr 2007 06:58:37 -0400

> zero length read when necessary?  That would certainly be the usual case.
The only issue I would think on this is that many apps
(as you said) consider a 0 length read to mean you get
no more data from the peer.. This would usually be the
case if a PD-API is aborted.. due to the peer itself
aborted/shutdown. However the case where this event got
added was for the PR_SCTP feature. If you have a partially
reliable message it could get "skipped" via timeout etc.
In such a case, the read returns 0 does NOT mean the
peer has gone away... don't know if this will confuse
apps or not.

R

---

From: Mark Butler <[email-masked]>
Date: Sun, 22 Apr 2007 23:29:47 -0600

Randall Stewart wrote:

> Mark Butler wrote:
>
>> ...
>> Does anyone see a problem with returning MSG_TRUNC (and MSG_EOR) with 
>> a zero length read when necessary?  That would certainly be the usual 
>> case.
>> - Mark B.
>>

Wouldn't it be fair to say that those apps are currently using a 
different protocol from SCTP, and that if they were to be adapted to 
support SCTP, changing them to recognize a zero length read w/ MSG_TRUNC 
would be a trivial consideration  compared to the other changes 
necessary to take advantage of SCTP's advanced features?

To me it seems that it is impossible to properly support a record 
oriented protocol without supporting zero length reads and writes to 
terminate messages when there is no more data.  The very fact that POSIX 
documents MSG_EOR implies support for splitting a message into multiple 
writes.   And any application that generates a message on the fly is 
going to have situations where there is no more data in the current 
message.  And the only available means POSIX provides for terminating a 
message in such a case is a zero length write with MSG_EOR set.

I understand there are protocol level restrictions on interleaving 
messages on send (on the same association), but nonetheless it seems to 
me that it would be a considerable improvement to add an socket option 
to allow multiple writes terminated by MSG_EOR on the sending side as 
well.   Call it SCTP_PARTIAL_WRITE.   At a minimum it would solve the 
inverse head of line blocking problem, where a 1:M style socket is tied 
up sending a large message on one association, preventing all of the 
others from being serviced, by allowing writes to be interleaved across 
associations.  Of course, that would also improve the symmetry of the 
whole interface.

Comments?

 - Mark B.

---

From: Anders Torger <[email-masked]>
Date: Mon, 23 Apr 2007 09:02:52 +0200

It seems to me that if the stack supports MSG_TRUNC, it is not necessary 
to subscribe for SCTP_PARTIAL_DELIVERY_EVENT. Actually, since the 
partial delivery event currently lacks a stream field, MSG_TRUNC is 
required in cases where the stack can have more than one partial 
delivery going on per association.

The worst case is thus a stack supporting PR-SCTP, supporting multiple 
partial deliveries per association, but not supporting MSG_TRUNC, 
meaning that when the application gets SCTP_PARTIAL_DELIVERY_EVENT it 
does not know which partial delivery that aborted.

I've read comments on this regarding many stacks in this thread, and 
they seem ok. However, I haven't heard how the implementation in 
Solaris works. It seems to me that SCTP on Solaris could currently be 
in this state, that is if multiple partial deliveries are going on an 
one of them is aborted, the user cannot know which one. Or does the 
Solaris stack support MSG_TRUNC?

(Unfortunately I currently have not set up any good packet loss tools so 
it is a bit hard for me to test these cases.)

/Anders Torger

On Saturday 21 April 2007 09:08, Mark Butler wrote:
> >> handle messages that expire during partial delivery is to adopt
> >> the convention of terminating them with a zero length read with
> >> both MSG_EOR and MSG_TRUNC set.   There are several advantages to
> >> that convention - it is simpler, it is easier to program, and
> >> naive applications will be less likely to run messages together. 
> >> Proper compatibility could be maintained by only delivering such
> >> semantics to applications that do _not_ subscribe to partial
> >> delivery events.
> >
> objections: POSIX specifies that recvmsg() should return zero after
> "no messages are available to be received and the peer has performed
> an orderly shutdown" (such as a TCP half close for example).  All
> though POSIX does not require it, I believe some people assume the
> reverse - that such an "end-of-file" condition is the only time
> recvmsg ever returns zero bytes read.  This convention, of course,
> would require such a zero length read in many cases.
>
> POSIX also specifies that MSG_TRUNC be set on SOCK_DGRAM style
> sockets when the available data exceeds the buffer size.  It does not
> anticipate protocol initiated truncation, and using MSG_TRUNC in this
> way would be an extension of POSIX semantics.  It appears to be a
> clean extension to me - either way MSG_TRUNC means you do not have
> the whole message.
>
> Of course we do not currently support SOCK_DGRAM style read
> semantics, but if we did there would be a slight ambiguity when the
> number of bytes available on a PR-SCTP terminated message was exactly
> equal to the provided buffer size. Although I can't imagine what
> difference it would make in practice, strictly speaking we would need
> an additional flag added to sctp_sndrcvinfo to tell the difference in
> that case.
>
>  - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Mon, 23 Apr 2007 05:07:22 -0400

> associations.  Of course, that would also improve the symmetry of the 

Please go look at the current socket api document...
check out SCTP_EXPLICIT_EOR

Note that BSD supports this.. but when I implemented it
I did not expect a user NOT to know it had hit the end
of record without data to write... i.e. you have to
have data to send down in that last write.. It seems
to me you don't realize after the fact that you were
all out of data for that record... but when you are
writing the last piece.. thats when you realize.. hey
this is the last of this record..

R

---

From: Mark Butler <[email-masked]>
Date: Mon, 23 Apr 2007 09:42:47 -0600

Randall Stewart wrote:

>> terminate messages when there is no more data.  The very fact that 
>> POSIX documents MSG_EOR implies support for splitting a message into 
>> multiple writes.   And any application that generates a message on 
>> the fly is going to have situations where there is no more data in 
>> the current message.  And the only available means POSIX provides for 
>> terminating a message in such a case is a zero length write with 
>> MSG_EOR set.
>>
>> ...
>>
>

Wonderful!  I must have missed this - thanks for pointing me in the 
right direction.

> this is the last of this record..

I don't think that is a realistic expectation.  A typical streaming data 
interface (CGI for example) will have calls to write data that are 
distinct from a final "all done" call.  If the "all done" indication 
must be attached to a non-zero length write, it forces the application 
to hold the last one to see if there is going to be a following end of 
record in all cases, which is a considerable complication.  No one would 
expect "close" to have to be accompanied by a "write" operation.

And as you say, PR-SCTP faces the same problem on read - if a message 
expires mid-delivery, there generally will not be any more data to 
deliver.  If you do not send MSG_TRUNC and MSG_EOR with a zero length 
read in such cases, setting MSG_TRUNC on any other occasion is 
superfluous, because the application is forced to rely on partial 
delivery abort events instead, which makes the interface unnecessarily 
complicated.

There is a further precedent - RFC2960 states that a "DATA chunk with no 
user data field will have Length set to 16 (indicating 16 bytes).", 
implying that even if you cannot have a zero length message, you can 
have a message terminating DATA chunk with no actual data.   This is 
greatly to be desired, because otherwise the SCTP implementation would 
have to hold the last write waiting to see if it was going to be the end 
of record or not, even if the application was not forced to.

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Mon, 23 Apr 2007 12:20:30 -0400

> expect "close" to have to be accompanied by a "write" operation.

Ok, then this means there will be an extreme cost to an application
doing this..

Consider

a) SCTP cannot send a data chunk with 0 bytes.
b) If a SCTP endpoint begins to start sending data, in response
    to your users write.. most likely it will get all of this
    data chunkifed and sent.
c) Now your app sends down an "all done" 0 byte write with a MSG_EOR.

Now if SCTP has sent all the data previously, there is no way to close
the message. Otherwise, you will get the association aborted.

So the only way a stack can maintain your semantic of allowing a
0 byte MSG_EOR write to close a message, is to hold the last
byte always.. in the event that the user is going to do this.. which
means you will often have a 1 byte chunk sent to close a message.. a
very poor poor use of the network and quite an overhead to the
performance as well..

I think it is worse to push this down to the kernel for some
applications that do this.. and thus penalize all SCTP users.
It would be far better from a performance perspective to push
this back in the app.

In fact I don't see the benefit of pushing to the kernel to be
exact.. it makes no sense..

You can't do this in UDP..

If you want a streaming semantic, then either TCP is the write
on.. or you turn on explict EOR mode and set the PD-API to 1 and
then treat the single message as a TCP stream.. then you can
have the same semantic as TCP.. i.e. you can do a close() of
the association to terminate the MSG_EOR.. and this will work
fine with SCTP..

The key thing here is you are trying to put streaming semantics
on a MSG oriented delivery.. if you want that enable the options
as I said above.. and you will have it.

> delivery abort events instead, which makes the interface unnecessarily 

And there is a protocol mechanism to handle this.. aka FWD-TSN that will
cause the PD-API to abort and thats fine..

In the case of the user not sending data and wanting to communicate
a end of a segmented message, there is no protocol mechanism for
this...

> user data field will have Length set to 16 (indicating 16 bytes).", 

Thats been fixed in 4460.. you cannot send a zero byte message. The
protocol requires an ABORT.

I suggest looking at the bis document.. its much easier to follow.

> have a message terminating DATA chunk with no actual data. 

You cannot do this.. sorry.

   This is
> of record or not, even if the application was not forced to.

This is my point.. you are attempting to transfer a stream oriented
concept to message semantics. If you want stream oriented semantics
you either do the socket options as I have said and only send
one message (the entire transfer).. or you, the app, hold
the last byte to make the determination that yep.. I am all done
and then set MSG_EOR.

R

---

From: Mark Butler <[email-masked]>
Date: Mon, 23 Apr 2007 12:45:18 -0600

Randall Stewart wrote:

..

> protocol requires an ABORT.

I am aware that one cannot send a zero length message.  However, the 
prohibition of zero length message _fragments_ in the 2960bis draft 
seems like an unnecessary restriction to me.  Zero length message 
terminators are very useful.

The case where you want to send part of a message before you know if 
there will be any more is very common.  And you cannot do that if you 
cannot terminate a message after the fact.   Holding back a single byte 
is ugly, and will often eliminate the benefit of sending the preceding 
fragment in the first place.
..

> the last byte to make the determination that yep.. I am all done

In order to reliably send and receive arbitrary size messages, it is a 
technical necessity to break up messages into multiple reads and writes, 
and use a streaming, non-atomic interface style.  On send for example, 
you cannot send large messages unless SCTP_EXPLICIT_EOR is enabled and 
you let the stack take as much data as it can handle.  An atomic send of 
a message larger than the send buffer is impossible to provide, due to 
issues with signals.

So given that this MSG_EOR oriented streaming programming style is a 
necessity, and the semantic equivalent of record terminating zero length 
reads is a necessity (for PR-SCTP), why should the protocol be 
artificially restricted to prohibit record terminating zero length 
writes?  Are there any benefits to such a prohibition?

The alternative is to say that SCTP does not natively support dynamic 
generation of large messages, which eliminates a key advantage over TCP 
for a considerable class of applications.

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Mon, 23 Apr 2007 19:51:09 -0400

> fragment in the first place.

sorry.. you don't change the protocol to adapt to the API...
> a message larger than the send buffer is impossible to provide, due to 

So use EOR mode and set the EOR point when you are done.. not a
problem..

> reads is a necessity (for PR-SCTP), 
I disagree.. you are adding that requirement to fit the
stack.

why should the protocol be
> for a considerable class of applications.

And I disagree with this last.. You can generate a dynamically large
message any time you want.. and if you don't know where it will
end then fine, always hang on to the last byte until you know.

R

---

From: "yanrong shen" <[email-masked]>
Date: Tue, 24 Apr 2007 01:33:54 +0000

Please remove my e-mail from this list.

Thanks.

---

From: Mark Butler <[email-masked]>
Date: Mon, 23 Apr 2007 21:25:51 -0600

Randall Stewart wrote:

>> cannot terminate a message after the fact.   Holding back a single 
>> byte is ugly, and will often eliminate the benefit of sending the 
>> preceding fragment in the first place.
>

It seems to me that the only reason the protocol exists is to serve 
common application requirements.  This is a purely artificial constraint 
that has no added value whatsoever that is only clearly expressed in a 
document that is still in draft form.  Implementations would actually 
get simpler if it were removed.  

However, if no one else feels that it would be change worth having, I 
won't belabor the point.

 - Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Tue, 24 Apr 2007 05:16:54 -0400

> However, if no one else feels that it would be change worth having, I 
Mark:

The sigtran group, at one time, had a "zero length message" allowed..
and we actually used this to "skip msgs" like PR-SCTP. In the final
IESG comments.. this was taken out... can't remember why.

We have discussed this in the past.. I don't see the value.. and
for the majority case I don't think its a problem.

I have been developing applications for almost 30 years now.. and
very very rarely has there ever been an app that I did not
know where the message ended a priori.. so I find your scenario
rather strange IMO..

I will admit most of my app development as been in the telco space
but I have been associated with a LOT of non-telco projects as
well... and it just seems strange to me.

If you truly have a HUGE message that you have no idea where
the end of the message is, then maybe TCP is a better fit for
it then SCTP..

R

---

From: Anders Torger <[email-masked]>
Date: Tue, 24 Apr 2007 12:35:58 +0200

On Monday 23 April 2007 20:45, Mark Butler wrote:
> generation of large messages, which eliminates a key advantage over
> TCP for a considerable class of applications.

Really?

I think I agree with Randall here, I don't really see how the code is  
constructed in order *not* to know that the last fragment you send is 
the last of a particular ULP message. 

Since that case would not only require extra system calls to terminate 
messages, it also requires sending zero size message over the protocol, 
it seems to be an inefficient way to send dynamically sized messages. 
Thus one would not want to design code that way.

Do you have an enlightning example where the application do not know 
that the last fragment it sends is the last of an ULP message?

/Anders

---

From: Mark Butler <[email-masked]>
Date: Tue, 24 Apr 2007 10:20:33 -0600

>that the last fragment it sends is the last of an ULP message?
Here is a historical example of a considerable complication that could 
have been avoided. (It is relatively long, so I inserted section headers 
for clarity):

INTRODUCTION

HTTP supports on-the-fly page generation, which of course is very 
common.  There is an enormous benefit from starting to transmit and 
starting to render a document as soon as possible, and so those are all 
done on-the-fly as m well, transmitting before the content has been 
completely generated, and rendering before it has been completely 
received, subject to formatting constraints.

In the beginning, it was necessary to open and close a separate HTTP TCP 
connection for every document requested, and that was woefully 
inefficient, notably because a single page often had a very large number 
of included icons and other images, each of which required a separate 
request.

So it was determined that there should be a "Keepalive" option with some 
sort of inactivity timeout to allow multiple requests to be issued on 
the same connection.  That only worked if the HTTP content length was 
known in advance, because the convention for terminating a dynamically 
generated document was to omit the "Content-length:" header and just 
close the TCP connection when the document was finished.

Lacking any sort of message boundary support, the solution for HTTP/1.1 
was to invent something called "Chunked transfer coding", which is 
nothing less than a application level transport protocol.  In order to 
generate dynamic documents, instead of completing an HTTP response with 
a binary dump of the document, the web server emits a transfer coding 
header, followed by a series of data bearing chunks (each with its its 
own header), followed by a trailer that indicates the transfer is 
complete. 

Now if TCP had record boundaries, all this extra complexity just to 
support dynamically generated documents would be superflous.  There 
would only need to be a convention that each response is composed of a 
single record, terminated by a record boundary. So when the requester 
read MSG_EOR it would know that the document was done, and it could go 
on to the next response.

DOCUMENT GENERATION INTERFACES

Web document generation interfaces don't ever combine an operation to 
write a piece of data with the end document operation because it is 
enormously artificial constraint.  The prototypical example, CGI, just 
has the page generation program issue writes on a pipe and close and 
exit when the page is complete.  The "write" and the "close" are never 
combined.  Other alternatives (such as the internal interfaces to Apache 
and IIS) work in a similar way, a series of write operations (e.g. 
ap_fwrite, ap_fprintf, etc.) followed by an implicit or explicit end 
document operation.

In the TCP case, the web server then has to buffer and merge the write 
operations into efficiently sized chunks, add a chunk transfer header 
for each one, and then write a trailer after it receives an end 
document.  And the web browser has to perform the inverse chunk header 
parsing and stripping operation on the other end.

With SCTP this extra overhead could all eliminated by letting the 
transport protocol do what it is good it - transmitting distinct 
messages, one message per response.  If you do more than one message per 
response, you are back to the same inefficiences that the TCP solution 
has, more or less defeating the benefit of having record boundary 
capability in the first place.

RATIONALE

With all of these interfaces, the web server never knows, and the 
generating application or module often doesn't know, which of these data 
generation calls will be the last in a document.  So if you are writing 
out one record per response (to gain any benefit in the first place), if 
there is not an independent way to terminate a record, the web server 
would have to always hold back some portion of the data.

That adds unnecessary complexity to the web server, but there is another 
problem.  In order to make progress rendering a document, the web 
browser needs as much data as soon as possible.  If, for example, the 
last angle bracket on a "</table>" closing tag is left off for 
artificial reasons, the renderer will just sit there until it comes 
along, which stalls the process unnecessarily.  

Similar logic applies to database protocols, where you have a series of 
queries each generating a response of arbitrary length, where the query 
engine generates and transmits row data long before it knows how many 
rows or how much row data there is going to be, and also where putting 
message boundaries at the query response level is much cleaner and more 
efficient than artificially wasting them at the row level, or chunking 
rows together with yet another application level transport protocol.

So the rationale is that allowing zero length message terminators maps 
naturally to the way large dynamically sized messages are typically 
generated, i.e. on the fly, with separate content generation and end 
message operations.  And that further, disallowing zero length message 
terminators at the protocol level just adds application level complexity 
and processing delays with no corresponding benefit whatsoever.  And of 
course SCTP stacks would actually get smaller.

EFFICIENCY CONSIDERATIONS

There is nothing that prevents the transport protocol implementation 
from merging writes that belong to the same message together.  In the 
common case, whenever there is still message data waiting to be 
transmitted, and an independent message terminator (i.e. a zero length 
write with MSG_EOR set)  is received, the stack can just flip a bit on 
the last data chunk, and there will be no extra protocol level overhead 
at all.

The only case where a message terminator would need to be transmitted 
independently would be when all the data for that message was already 
sent.   That would be relatively rare, and would only be incurred by 
applications that benefit accordingly.  And of course removing the 
artificial restriction at the implementation level is trivial.

- Mark B.

---

From: Randall Stewart <[email-masked]>
Date: Wed, 25 Apr 2007 06:22:53 -0400

So as I understand what you wrote here you have something like this:

                   ,'''''''''''''''|
                   |  Renderer     |
                   |               |
                   '---------------'
                          |
                          v
                   ,''''''''''''''`.
                   |  Web Chunker  |
                   |               |
                   '`'''''''''''''''
                          |
                          v
                   ,---------------.
                   |     TCP       |
                   |               |
                   '`''''''''''''''

The "web chunker" is an artificial entity that you would
like to eliminate .. and its sole purpose is to "chunk up" the
messages and make it so multiple msg's from renderer's can
be sent/shared over the TCP connection..

The web chunker is complicated by having to send control
"start msg"/data/"end msg" to its peer... who then schedules
the data out to the proper renderer...

So, there is a very simple solution for this that does not
require a "zero length" write. You simply take your web chunker
and make it do only one thing.. When the "renderer" does its
close.. which is done say after the last write of data.. the
web chunk then sends a 1 byte message with "msg eor". The web
chunk on the opposite side then becomes very simple.. all it has
to do is to watch for MSG_EOR and when it sees this, it takes off
the last byte of the message.. and knows its next msg in that
stream will be associated with some other renderer.

R