Subject: Re: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about

From: Randall Stewart <[email-masked]>
Date: Thu, 03 Oct 2002 19:16:21 -0500

Venkat:

This seems supicious..

If you do a recvmsg() and then do a subsequent recvmsg() you
don't necessarily get the address and control info again...
Not until the EOR is set...

R

venkat venkatsubra wrote:
> venkat venkatsubra wrote:
> 
> 
>>Randall,
>>
>>
>>>--__--__--
>>>
>>>Message: 10
>>>Date: Wed, 02 Oct 2002 19:45:25 -0500
>>>From: Randall Stewart <[email-masked]>
>>>To: venkat venkatsubra <[email-masked]>
>>>CC: [email-masked], [email-masked]
>>>Subject: Re: [Tsvwg] A SCTP sockets/sctp_peeloff() issue to think about
>>>
>>>
>>>So how do you tell which association goes with which data... do you
>>>keep back pointers and such? If you use a mbuf like structure do
>>>you have EVERY chained mbuf assured of having this information..
>>
>>We attach a sctp_sndrcvinfo structure with each message
>>in the socket buffer. We made this a mbuf type of MT_CONTROL.
>>And the notification stuffs (SHUTDOWN, ASSOC_CHANGE, etc.)
>>we keep it as a MT_SONAME mbuf type.
>>So, getting the assoc_id and moving the messages or
>>notifications of a particular assoc_id is not an issue.
>>As Kacheong had pointed out you need it anyway
>>to pass to recvmsg.
>>
>>
>>>It is also possible to read 100 bytes (of a 1000 byte message) and then
>>>do the peel off... in the case of a BSD system this may well loose the
>>>M_PKTHDR (which is where any back pointers could possibly be kept) and
>>>thus you would have data left on the socket with no way to tell
>>>which association went with that data for the partially read message...
>>
>>To support partial delivery api we insert the
>>"control" structure back again at the head
>>of the message after the user does a partial read.
>>
>>
>>>
>>>R
>>>
>>>
>>>>>R
>>>>>
>>>>>--
>>>>>Randall R. Stewart
>>>>>[email-masked] 815-342-5222 (cell phone)
>>>>
>>Venkat
> 
> 
> _______________________________________________
> tsvwg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/tsvwg
> 
> 

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Jon Grimm <[email-masked]>
Date: Fri, 04 Oct 2002 00:04:09 -0500

Hi Randy,

> Not until the EOR is set...

Why do you say this?  Why not?  I don't recall anything in the I-D
calling out this behavior for subsequent reads. I interpreted this as 
Venkat and Kavitha did and assume that every message would
have msg_name (address) and/or CMSG data, even if only
reading a subportion (MSG_EOR not set) of the message.

It may be redundant information (as it will not change), but I can't
see a good reason that the application shouldn't be able to get
that information either.

Hmmm.. what happens on your TCP-style impl.  What happens on
recvmsg/recvfrom?  (Assuming not from the primary address) do you fill 
in msg_name for every read even when it takes a consequent read(s) to 
pull up the rest of the message?

Which, in turn, reminds me to ask another dumb question, why the wording
about the primary address? For example section 4.1.8:

"When receiving, if a message is not received from the primary
  address, the SCTP stack will fill in the msg_name field on return so
  that the application can retrieve the source address information of
  the received message."

This seems rather an odd exclusion; the primary address may never even
be the address that the peer sources the message from (assuming 
multihoming of course.)  Why have different behavior to just confuse the 
  application writer? I'd suggest filling in msg_name whether its the 
primary or not.

Maybe another question to ask is whether SOCK_STREAM server applications
(multiple sockets) would expect to the msg_name to always be filled in 
when they request it? I'm pretty sure Linux SOCK_STREAM/TCP will do
so at least.

Best Regards,
Jon

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Fri, 04 Oct 2002 05:39:07 -0500

> reading a subportion (MSG_EOR not set) of the message.

I have thought about this and here is the reason I think it is bad

Lets say I am using recvfrom instead of recvmsg()

In this case I get no MSG_EOR.. I can't see it... I realize this is 
unwise ... but many
application implementors are not going to go the extra mile to learn 
recvmsg... the folks
coming from UDP land WILL know recvfrom though...

Now if I get a message with a sockaddr filled in... I know it is the 
beginning of a
message...

If the next read has NO sockaddr... guess what I can determine that its 
a continuation
of the previous message... up until I read a subsequent message with a 
sockaddr setup.. maybe
to the same guy maybe different..

But if you do this copy.. you have taken away the only indication I have 
that this is
a second part of a message...

> see a good reason that the application shouldn't be able to get

Just the opposite.. you are losing information.. record boundaries..

> pull up the rest of the message?

Good question.. I would hope to see it with an address to mark the beginning
of a record and no addresses to mark subsequent parts of the same 
message.. that way
the TCP style would ALSO get some sort of indication where the record 
boundaries
are...

>  that the application can retrieve the source address information of

I agree.. not sure where it came from :-0

> multihoming of course.)  Why have different behavior to just confuse 
> the  application writer? I'd suggest filling in msg_name whether its 
> the primary or not.

I think that paragraph should be killed.. don't know why it is there...

> when they request it? I'm pretty sure Linux SOCK_STREAM/TCP will do
That would surprise me.. most BSD implementations do NOT give you a 
sockaddr at ALL
in sock_stream mode.. on the theory that during the accept you got
the address and now you can equate the socket-descriptor to the
address...

But I do think more wording is in order ...

1) to specify that the address can be used as a message delinator (and 
lack of ... as I state above)
2) recomend that it be done in the TCP style too
3) get rid of that stray paragraph :>

R

-- 
Randall R. Stewart
ITD
Cisco Systems Inc.
[email-masked] 815-342-5222(cell) or 815-477-2127(office)

---

From: "La Monte Henry Piggy Yarroll" <[email-masked]>
Date: Fri, 04 Oct 2002 07:19:14 -0500

On Fri, 04 Oct 2002 05:39:07 -0500, "Randall Stewart (cisco)" <[email-masked]> w
> I think that paragraph should be killed.. don't know why it is there...

This is my fault--I should have removed it when I put in assoc_id.

---

From: Jon Grimm <[email-masked]>
Date: Fri, 04 Oct 2002 08:27:48 -0500

> the TCP style would ALSO get some sort of indication where the record

A couple comments:

1) If TCP style apps care about msg boundary they should use recvmsg,
looking
at msg_name isn't natural for them either.  Using it as a poor man's
MSG_EOR 
seems a little hackish.  
2) How about UDP-style applications, above you talk that UDP
applications
(which I will admit right now that UDP != UDP-style but there was some
attempt at similarity) know how to use recvfrom: Will they be expecting
msg_name at all times?
3) Depending on ones perspective, is it also interesting to ask what a 
(non-SCTP) SOCK_SEQPACKET does?  Looking at the Linux DECNET code,
they'll
fill it msg_name in on every subread of a record.   I don't have POSIX
docs
handy to check there.

> the address and now you can equate the socket-descriptor to the

BTW, I double checked and you are correct sock_stream does NOT fill in
msg_name.
The Linux man page for recvmsg says that: 

"If from is not NULL, and the socket is not connection-oriented,  the  
source  address  of the message is filled in."

So now thinking in reverse I question if a SOCK_STREAM app will ever use
the poor man's MSG_EOR trick at all.   In general, I think app. writers
will need to use sendmsg/recvmsg if they want to exploit the
interesting features of SCTP anyway (well I guess that several sockopts
have been introduced that don't make this a _completely_ true
statement),
to where this trick isn't needed. 

Thanks, Jon

---

From: Randall Stewart <[email-masked]>
Date: Fri, 04 Oct 2002 09:29:31 -0500

> at msg_name isn't natural for them either.  Using it as a poor man's

I would not call it hackish.. at least it is a clue

> attempt at similarity) know how to use recvfrom: Will they be expecting

Well UDP uses PR-ATOMIC symantics. This means that
if you don't read it all in one shot you LOOSE the message.
If we want to go that route... then the whole issue becomes
mute. Qiaobing suggested that... Of course that means
large messages just can't be sent (larger > 64k)

> fill it msg_name in on every subread of a record.   I don't have POSIX

Hmm... don't know.. I don't think I have seen any SOCK_SEQPACKET in
any of the BSD's other than SCTP.... of course I have not looked either.

Note that currently there is no way in the sockets layer of BSD to
do the add a sockaddr * to every read ...

Not without hacking up the soread function .. which is exactly
what Venkat did... he added this copy the mbuf with the
address back to the mbuf chain if you don't read it all..

That is very difficult to justify to the BSD folks.. doing anything
within that layer if a stretch..

Now that being said I think this IS rather NEW and uncharted teritory.
i.e. a type of socket that does not use PR_ATOMIC (UDP's choice) and is
not all one big record (tcp)..

I do like having some hint that this record just read is a continuation
from the last read.. and I don't see it has a hack..

After all if nothing else I can use it as a clue that I got a
big record.. .and do some exception processing... if I have no
clue.. then the app will get two "seperate records" in its
view that are one...

Not a good thing..

R

---

From: venkat venkatsubra <[email-masked]>
Date: Fri, 04 Oct 2002 10:23:17 -0500

Randall,

> Now if I get a message with a sockaddr filled in... I know it is the

In case of UDP i thought the remaining portion of the
message will be truncated. It won't be returned on
a subsequent recvfrom(). In fact, the socket code
always expects the first mbuf is a MT_SONAME
containing the address.

> to the same guy maybe different..

I am not sure if the UDP recvfrom works this way
currently. In my understanding it truncates the message
if the user buffer is smaller.

> a second part of a message...

I dont't think such an indication exists now
(at least in our UDP socket implementation).
I can verify.

Venkat

---

From: venkat venkatsubra <[email-masked]>
Date: Fri, 04 Oct 2002 10:30:13 -0500

> any of the BSD's other than SCTP.... of course I have not looked either.

XNS might have SOCK_SEQPACKET. I could be wrong.

> within that layer if a stretch..

Since BSD allows you have your own protocol specific
socket level receive, send functions (soreceive, sosend)
we used separate functions for SCTP. So whatever we
did, did not affect anybody else.

Venkat

---

From: "Randall Stewart (cisco)" <[email-masked]>
Date: Fri, 04 Oct 2002 11:16:56 -0500

Venkat:

As I said in my earlier post..

PR_ATOMIC is set on UDP.. this means you trucnate
things... you NEVER will get any partial mbufs in UDP that
do not have the header..

R

---

From: Qiaobing Xie <[email-masked]>
Date: Fri, 04 Oct 2002 16:22:05 -0500

> This is my fault--I should have removed it when I put in assoc_id.

As I am coding RSERPOOL, my biggest wish now is to make sure that this
assoc_id is supplied by all SCTP stacks and is kept contant throughout
the life of the association.

(before assoc_id, we used to rely on a source addr sockaddr_in * to tell
apart the association. Unfortunately, when the peer is multihomed and
when "Add-ip" is in play, the sockaddr_in * cannot be kept constant for
a given association, which becomes a big headache to the app).

So pls kill the paragraph :-)

-Q

---

From: Randall Stewart <[email-masked]>
Date: Fri, 04 Oct 2002 18:33:46 -0500

> So pls kill the paragraph :-)

Don't worry my friend..

It is already dead in the next pre-version :>

R