Subject: SCTP failover

From: "Anatoly Khusid" <[email-masked]>
Date: Fri, 11 Feb 2005 12:29:36 -0500

Hello,

 I am using Linux SCTP implementation (LKSCTP) SLES9 (2.6.5-7.111.19-smp)
distribution.
 I have a client application that is sending data to a server on a remote
 machine.  The machines are connected over two private LANs.
 When I disconnect a primary interface, I expect SCTP to start using an
 alternative LAN as soon as possible.  Well, it takes about one minute for
 LKSCTP to detect that LAN is down, before it starts to transmit data
 messages on another LAN.  I don’t have any data messages lost, but I have a
 1-minute delay during which time no data is received by a server, after
 about one minute, the data transmission resumes through the alternative
 interface.
 I am curious why it takes so long to detect a LAN failure?  I am using all
 the defaults for SCTP provisioning. (In fact I used getsockopt() to verify
 that the defaults match SCTP specs).
 Based on this section in SCTP RFC I would expect the switchover to be in a
 matter of seconds the most.  I am using Ulticom’s SCTP implementation and
 the switchover only takes about 1 second.  Could anyone please shed some
 light on this?

 Section 6.4 of SCTP RFC 2960:
 Furthermore, when its peer is multi-homed, an endpoint SHOULD try to
 retransmit a chunk to an active destination transport address that is
 different from the last destination address to which the DATA chunk was
 sent.

 Thanks,

 Anatoly Khusid
 Ulticom Inc.
  Senior Software Engineer

---

From: Randall Stewart <[email-masked]>
Date: Fri, 11 Feb 2005 13:08:06 -0500

Anatoly:

Shridhar can probably shed more light on this than I .. but
I have a couple of ideas.

In theory the sctp stack SHOULD send retransmissions to the
alternate address.. I would think this is NOT happening.. Even
if it does happen for some time you will still have some
delays in messages... Think of it this way

--msg-1--->
--msg---->
--msg-n-->
T.O (1sec)
--rsend- msg1-n to alternate
--msg-n+1-->
--msg-n+..->
--msg-n+m-->
T.O (2sec)
--resend msg(n-m) to alternate
--msg+m-1-->

etc..

until you get

TO's
1sec
2sec
4sec
8sec
16sec
32sec
---
or about 63 seconds after failure the primary will be delcared dead
and new transmissions will go to the alternate (which is now
the primary).

In between the break and the 6 timeouts, messages will be delayed
anywhere from 1 - 32 seconds .. but they should still get through.

The fact that you don't see any messages for 60 or so second indicates
to me that maybe the retransmit to alternate is not working in lk-sctp,
or maybe there is an option to turn it on??

In any event the only way to keep the network failover time
down is to set RTO.Max to a lower value.. that would make things
faster... To have a 1 second failover I would imagine that
Ulticom's stack is setting both RTO.Min and RTO.Max to a
lower value... aka that adds up to a total of 1 second..
I.e. something like 50ms RTO.Min and 400ms RTO.Max

Now, as discused on the tsv, when you do this you need to make
sure the receiver is also cranking down its delayed sack timer
to be smaller than RTO.Min .. otherwise you are going to get
T3-Timeouts on normal sack delay when only one TSN is sent
and there is nothing else to send..

Hope that helps..

R

-- 
Randall Stewart
ITD
803-345-0369 <or> 815-342-5222

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 11 Feb 2005 11:50:34 -0700

Anatoly,

One thing that can impact failover performance is the source
address and/or transmitting interface on retransmitted messages.

If the transmitting interface does not change for retransmitted
messages (because there exists a route from each interface to
each peer transport address, or due to an error), then
retransmitted messages will never appear.  This sounds possibly
like what is happening.  You can play with the routing table a
bit to make sure that you have two completely independent
subnets described for diagnosis.

If the retransmitting interface changes, but the source address
placed in the retransmitted packets does not change, then the
receiver is likely attempting to send SACKs on the failed path
(if following the rule that SACKs are sent to the source address
of the data they are acknowledging).  So, if the data is one-way
and the sender does not change the source address to match the
actual outgoing interface on retransmitted messages, SACKs go to
the wrong place and it takes multiple retransmissions for the
receiver to detect duplicates and suspect that its SACKs are not
getting through.  If the receiver does not send SACKs to a
different destination after receiving duplicates, or gets
otherwise confused by the source address in the DATA packets
then failover will be slowed.  But this doesn't sound like what
is happening because you said that you received no data at the
receiver.

Your description makes be believe that the former is the case:
that, although it is retransmitting to an alternate destination
address, the source address and originating interface is not
changing.  Set up your routing tables so that each destination
address can only be reached by one interface and see if the
problem disappears.  Also watch out for default routes.  You can
try removing default routes for the purposes of testing.  If the
problem disappears it is a routing problem on retransmissions.

Aside from that, slap Ethereal on both interfaces on both hosts
and you should see what is happening.

A better solution than cranking down RTO.min for increasing
failover performance is CMT.  At moderate data rates, failover
is smooth and immediate and queuing delays impacted only by
milliseconds on fast connections.  Only a proportion of data is
affected by interface/single-path failures and the data on the
other paths remains completely unaffected.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart <[email-masked]>
Date: Fri, 11 Feb 2005 14:18:40 -0500

Brian:

Good point.. this could be an issue with a default
route or non-subnet routes in place..  much like
what happens the first few days of the inter-op when
folks forget to setup explicit network routes for
the two different sub-nets..

Only problem with that theory is that he would not
get any traffic ever through unless the interface
failed and the route was actually removed. As long as an
interface was good .. then the route would have stayed
in place and no traffic would have gotten there.. but
it is a possiblity.. depending on what type of failure is
involved..

R

---

From: David Lehmann <[email-masked]>
Date: Fri, 11 Feb 2005 14:08:35 -0500

> I.e. something like 50ms RTO.Min and 400ms RTO.Max

Nope.  When the error count is greater than 0 on the
primary destination, Ulticom uses an alternate destination,
until the error is cleared on the primary destination.
So, if the user is using the default values, the user
will only see a blip for one second.
(...assuming an RTO of 1 second)

IMHO, waiting around for 1 minute while data trickles through
the association is not acceptable and does not give the user
of a sense of transparent network redundancy.

IMHO, the language in section 6.4 should be changed from:
    By default, an endpoint SHOULD always transmit to the primary path,
    unless the SCTP user explicitly specifies the destination transport
    address (and possibly source transport address) to use.
to something like:
    By default, an endpoint SHOULD always transmit to the primary path,
    unless the error count on the destination is greater than zero or the
    SCTP user explicitly specifies the destination transport address (and
    possibly source transport address) to use.

-- 

David Lehmann                          Ulticom, Inc.
AOL/Yahoo IM: davidULCM                1020 Briggs Road
1-856-787-2729                         Mt. Laurel, NJ 08054   USA

---

From: David Lehmann <[email-masked]>
Date: Fri, 11 Feb 2005 14:33:41 -0500

> (...assuming an RTO of 1 second)

I need to correct my former post.  Assuming a steady flow
of data chunks, the switch-over with Ulticom's stack will
actually be in a fraction of a second since the FR will
notice the problem long before the RTO.

--

---

From: Ryan W Bickhart <[email-masked]>
Date: Fri, 11 Feb 2005 15:23:06 -0500

I'm not sure switching over to the alternate after seeing a single FR is
the best approach for all cases though. Imagine a situation where the
alternate is significantly slower or less desirable than the primary. A
random FR on the primary may not necessarily be grounds for abandoning
it right away. If you need faster failover, you could change the
Path.Max.Retrans setting to limit the number of consecutive timeouts
that will trigger the failover. Rather than 1 + 2 + ... + 32 = 63
seconds, you could set PMR to 0 or 1 and fail over immediately or in
about 1 second if desired. I think the suggested change to the wording
in 6.4 is actually already covered by the ability to set the PMR.

---Ryan

---

From: "Armando L. Caro, Jr." <[email-masked]>
Date: Fri, 11 Feb 2005 15:20:34 -0500 (EST)

> of a sense of transparent network redundancy.

I agree. I have simulation results that show that failing over on a single
timeout is best. The assumption is that a heartbeat is sent immediately to
the primary when a failover occurs. This way, if the primary is still
reachable, then the failover will be cancelled as soon as the heartbeat is
acked. If not, then useful data transfer can occur on the alternate until
a heartbeat sent to the primary gets acked.

www.armandocaro.net/papers/2004.TR2005-04.acaro.pdf

>     possibly source transport address) to use.

I don't think the RFC needs to be changed at all for this. An
implementation can simply lower it's Path.Max.Retrans to 0.

~armando

0--                                              --0
| Armando L. Caro, Jr. |  Protocol Engineering Lab |
| www.armandocaro.net  |    University of Delaware |
0--                                              --0

---

From: "Armando L. Caro, Jr." <[email-masked]>
Date: Fri, 11 Feb 2005 15:24:33 -0500 (EST)

> notice the problem long before the RTO.

Hmm... I think that is getting too aggressive. This behavior could
cause reordering and induce false fast retransmits. Why would you do that?

~armando

0--                                              --0

---

From: "Anatoly Khusid" <[email-masked]>
Date: Fri, 11 Feb 2005 15:32:09 -0500

Hi Randall,

I was not completely right that during this 60 second interval my peer does
not receive any messages.
I ran the test again, and noticed that peer DOES receive messages BUT at a
very slow rate (just like you suspected that only retransmissions are sent
over the alternative interface).
So the transmission for each message must fail first, before LKSCTP tries an
alternative interface.
This causes significant delays for data transmission during this 60 seconds
interval(RTO.MAX).  I wonder if this could be optimized?

Thanks,

Anatoly

-----Original Message-----
From: Randall Stewart [mailto:[email-masked]]
Sent: Friday, February 11, 2005 1:08 PM
To: Anatoly Khusid
Cc: [email-masked]
Subject: Re: SCTP failover

Anatoly:

--msg-1--->
--msg---->
--msg-n-->
T.O (1sec)

etc..

until you get

TO's
1sec
2sec
4sec
8sec
16sec
32sec
---

The fact that you don't see any messages for 60 or so second indicates
to me that maybe the retransmit to alternate is not working in lk-sctp,
or maybe there is an option to turn it on??Hi

Hope that helps..

R

---

From: "Anatoly Khusid" <[email-masked]>
Date: Fri, 11 Feb 2005 15:35:08 -0500

>If you need faster failover, you could change the
>that will trigger the failover.

If you modify SCTP provisioning to be something other than the proposed
defaults, I thought this might cause significant performance impacts?

-----Original Message-----
From: Ryan W Bickhart [mailto:[email-masked]]
Sent: Friday, February 11, 2005 3:23 PM
To: David Lehmann
Cc: [email-masked]
Subject: Re: SCTP failover

---Ryan

---

From: Ryan W Bickhart <[email-masked]>
Date: Fri, 11 Feb 2005 15:53:22 -0500

> defaults, I thought this might cause significant performance impacts?

The defaults could reamin the same, but since the PMR setting is tunable
by the application, it can be adjusted according to the particular needs
and network situation where it is deployed. See:

7.1.13  Peer Address Parameters in
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-sctpsocket-09.txt

---Ryan

---

From: Randall Stewart <[email-masked]>
Date: Fri, 11 Feb 2005 16:04:21 -0500

IMO considering Armando's work and all.. we really need:

1) 2 levels of failure.
     a) When you hit a Retranmist limit (set to say 1) you
        switch to the alternate
     b) When you hit a Max retransmit you mark the destination
        as down.
     c) If the HB sent to the place the t-o happened is sucessful
        then you clear your count and the primary stays where it
        was... Note you should send a HB right after a RTO on the
        place where the RTO happened IMO

2) Always send T-O's to the alternate.. not FR's since Armandos
    work shows this is not a good idea... and besides you don't
    switch off due to one loss.. after all you still have an
    ACK clock or you would not be FR'ing.

3) CMT, which is what Jana is working on, may be even a better
    method assuming we can find shared bottlenecks.

R

---

From: "Armando L. Caro, Jr." <[email-masked]>
Date: Fri, 11 Feb 2005 16:16:27 -0500 (EST)

>         place where the RTO happened IMO

Yes, I agree. Although you are "failing over" to the alternate on a single
timeout, the primary shouldn't be considered failed just yet. However, if
the primary is eventually marked as failed (eg, 5-6 consecutive timeouts),
then the sender would want to avoid using it as a possible dest to send
new data or rtxs until it becomes reachable again.

~armando

0--                                              --0