Subject: Re: FW: ssthressh

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 15 Jul 2005 16:56:09 -0600

Janardhan,

On Fri, 15 Jul 2005, Janardhan Iyengar wrote:

> Brian,
> 
> >> A couple of years ago, we introduced the term CMT as Concurrent 
> >> Multipath Transfer - sending new data concurrently to the multiple 
> >> destination addresses of a multihomed receiver.
> >
> > Well, if you want to define CMT as something else, go ahead.  But, what
> > I'm doing is transmitting to multiple destinations to achieve efficient
> > load balancing over multiple destinations in contrast to the stock
> > RFC 2960 primary/alternate approach.
> 
> That is what we had intended when we introduced the term "CMT". Glad to 
> see that you use the term in the same sense.

Well, am I using it the same way you are then, or not?

> 
> > Yes, I looked at these last year.  It has taken your research some time
> > to catch up with practice.
> 
> What practice? I do not understand what you mean by this comment - I 
> certainly do not know of any CMT using SCTP in practice.

The OpenSS7 implementation has been using it in practice for years.

> You claim to do CMT in your code.

It's not a claim, it's what started this discussion.

> I do not know how you do it.. I have asked you 
> repeatedly, in vain, to share your algos in the past. I see that you have 
> given more info in the email to Randy - comments below.

The selection algorithm was last posted to tsvwg, 31 Mar 2005, under the
thread "SCTP load balancing policy - reg".

The OpenSS7 implementation code (both sockets and STREAMS) has been
available for your perusal and comments since early 2001 at
http://www.openss7.org/download.html

> 
> > Look at my last note to Randall to see the two little other things that 
> > I do to get rid of those effects.
> 
> I did see your note. And I have a couple of comments. So, I'll copy and 
> paste from the other email:
> 
> > If you place some persistence on the "best" destination into the
> > algorithm, you might not only find the bottleneck behavior similar to
> > what I report, but you should find that the gap report generation at the
> > receiver reduced.
> 
> I still do not understand how your sending algo works. When do you stop 
> sending to "the best destination" and when do you start sending to others?

It's simple: When I go to send, I choose the "best" destination and then I
send packets to it as long as packets are in queue for transmission and I can
continue to send to that destination given normal window calculations.  If I
can no longer send to that destination due to window constraints, but still
have data for transmission in queue, I choose the next "best" destination, and
so on, until I run out of queued data for transmission, or there is no
destination that will accept further data transmission due to windowing
considerations.

See function sctp_transmit_wakeup in the code.

> 
> 
> > You could also try my two tricks for gap reports on CMT:
> >
> > 1. When the sender gets a gap report, I only perform FR steps when the
> >   receiver is not "making progress".  The receiver is "making progress"
> >   when it is contigiuosly filling a gap.  That is, a gap is shrinking.
> >   This is far simpler that trying to run a multiple queuing model.
> >   This trick does not break the rules: one is allowed to be more
> >   conservative than the FR rules allow, per RFC 2960.  I don't think
> >   one really has to even do FR, however, it has great benefits.
> 
> I'm guessing that you are assuming a certain sending mechanism. Unless you 
> work with a multiple queueing model, IMHO, you cannot handle any sender 
> scheduling mechanisms. In any case, a multiple queueing model, IMHO, is 
> not complicated at all. The code to implement it is quite simple and 
> requires delta changes.

Actually, I lied.  Looking at the code, this is done at the receiver to handle
sacks.  So, let me describe things again.

1. At the receiver, when I receive a new TSN and while performing gap
   algorithm I determine whether the sender is making "progress", where
   progress is defined as a shrinking or collapsing gap.  When the
   receiver goes to send a SACK due to gap reports, I suppress the SACK
   generation (and fall back to delayed SACK) iff the sender was making
   progress.  (See function sctp_recv_data).

2. At the receiver, when I go to send a SACK, I don't if there are more
   received packets to process in the socket's receive queue (packets
   that passed checksum and vtags checks and arrived at the socket while
   I was processing the current packet.  Due to queuing delays, this
   normally only occurs in implementation when a packet is received from
   an interface which is independent from the interface for the packet
   that is currently being processed.  (See functions sctp_bundle_more
   and sctp_bundle_sack.)

3. At the sender, when I receive a SACK acknowledging data, and I have
   gaps, I mark all destinations with messages outstanding in gaps as
   "eligible for FR".  Then, if the SACK acknowledges data within a
   gap, I mark the destination to which that data belonged as
   "ineligible for FR".  Then, when I go to do FR calculations on
   messages unacknowledged in the gap, I skip the calculation for
   messages sent to destinations marked "ineligible for FR".  This not
   only works for CMT but makes FR less sensitive to networks that
   reorder messages (even for single-homed associations).  As it is less
   aggressive than RFC 2960, it is probably OK.  Also, it does not change
   the situation that if a block of messages is consistently reported
   as missing in SACK gap reports, FR will be performed on those
   messages.  I should point out, however, that this is a more recent
   improvement, I used to to the same thing as in (1.) above at both
   the receiver and sender and that older method is what the performance
   results posted at http://www.openss7.org/sctp_perf.html were base on.
   (See function sctp_recv_sack.)

> 
> 
> >> The experiments you point to assume several things; a significant one
> >> being that the bandwidth and delay on the two paths used are the same.
> >> These results cannot be generalized.
> >
> > As I said, tests were also performed in competition with TCP, which results
> > in paths that do not have the same bandwidth-delay product.  The same
> > performance is achieved.
> 
> 
> >> It would really help if you could present your algo to the list. Even I
> >> have only a vague memory of what it is. If you are interested, we could
> >> take the discussion from there.
> >
> > Check with Armando, I have sent it to him many times, starting in 2001.  I
> > have been sending it to you for 4 years now...
> 
> You sent it *once* to Armando four years ago.
> 
> In any case, I do remember looking at what you sent, and it was just a 
> destination selection algorithm based on cwnd, RTT, among other things. I 
> did not, and still do not understand what it does, and why it is the 
> panacea for all CMT concerns.
> 
> Armando is currently on vacation, and will be for a little while longer. 
> Maybe you could repost what you sent, this time *to the list* ?

This list does not accept attachments.  Check the tsvwg thread I cited
above.  (In fact this note is probably too large to make it to the list
already.)

> 
> Brian, I have pointed out the papers that present CMT concerns, and how we 
> deal with them. I am trying to discern what you do from your quizzical 
> responses. If you think it is important to you, and if you would like to 
> share your ideas with the community at large, I would really appreciate a 
> simple document that clearly talks about what you do.

The last time we discussed this (October of last year) you were going to
write (update) an internet-draft on which I was looking forward to
commenting.  How is that coming along?

> A simple outline would contain your sending algo, algo to deal with
> reordering, and a simple analysis describing how these algos work, and
> show why the shared bottleneck problem does not exist. Maybe some
> experimental results even.
> 
> Until then, if I said anything else, I would probably be repeating myself.

I'm sorry but I do not want to spend too much time walking you through
all of the details, when you appear to refuse to download the code and
look at it yourself, and when you are funded to do this work and I
currently am not.

> 
> regards,
> jana
> 
> ---------------------------------------------------------------
> Janardhan R. Iyengar           http://www.cis.udel.edu/~iyengar
> Protocol Engineering Lab  --   CIS   --  University Of Delaware
> ---------------------------------------------------------------

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦