Subject: [tcpm] Minimum RTO

From: Mark Lloyd <[email-masked]>
Date: Tue, 10 Apr 2012 19:52:52 +0000

Reading RFC 6298

http://tools.ietf.org/html/rfc6298

 (2.4) Whenever RTO is computed, if it is less than 1 second, then the
         RTO SHOULD be rounded up to 1 second.

While troubleshooting a problem that involves minimum RTO, I found that RFC 6298 suggests a 1 second value for minimum RTO that is not honored in the implementations I have tested, and even in these implementation TCP does not do a good job of handling packet loss in a common scenario I tested. 

I use two Linux hosts as my test. They are setup to imitate the kind of connection that would be used for a database or RPC transaction.

My server does a netcat command that loops back data as it is received:

touch /tmp/testfile
tail -f /tmp/testfile | sudo nc -l 1000 >>/tmp/testfile

The client sends data, then echos a response simulating a TCP conversation of small transactions.

touch  /tmp/testfile
echo hello >/tmp/testfile
tail -f /tmp/testfile | nc 10.10.30.121 1000 >>/tmp/testfile

I then setup a wan emulator to simulate a 1% loss rate back to the client. 

I'm showing frames leading up to the packet loss to show the what the traffic looks like up to the packet loss. 

Server side:

Frm    Time          
Num    Delta         
1941   0.0000830     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646100 - 1538646106, Ack=1257857589, 
1942   0.0005920     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857589 - 1257857595, Ack=1538646106
1943   0.0000920     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646106 - 1538646112, Ack=1257857595
1944   0.0005590     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857595 - 1257857601, Ack=1538646112
1945   0.0000730     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646112 - 1538646118, Ack=1257857601
1946   0.0005770     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857601 - 1257857607, Ack=1538646118
1947   0.0000930     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646118 - 1538646124, Ack=1257857607
1948   0.0006190     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857607 - 1257857613, Ack=1538646124
1949   0.0000790     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646124 - 1538646130, Ack=1257857613
1950   0.0005110     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619, Ack=1538646130
1951   0.0000850     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 - 1538646136, Ack=1257857619
                                         Packet 1951 will be lost in transit. Note 205ms gap.

1952   0.2053670     Client -> Server     TCP:[ReTransmit #1950]Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619
1953   0.0000420     Server -> Client     TCP:Flags=...A...., SrcPort=1001, DstPort=55552, PayloadLen=0, Seq=1538646136, Ack=1257857619
1954   0.0007630     Server -> Client     TCP:[ReTransmit #1951] Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 - 1538646136, Ack=1257857619
1955   0.0006640     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857619 - 1257857625, Ack=1538646136

Client side:

1928   0.0006400     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646094 - 1538646100, Ack=1257857583
1929   0.0000690     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857583 - 1257857589, Ack=1538646100
1930   0.0006200     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646100 - 1538646106, Ack=1257857589
1931   0.0000690     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857589 - 1257857595, Ack=1538646106
1932   0.0006300     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646106 - 1538646112, Ack=1257857595
1933   0.0000730     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857595 - 1257857601, Ack=1538646112
1934   0.0005230     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646112 - 1538646118, Ack=1257857601
1935   0.0000830     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857601 - 1257857607, Ack=1538646118
1936   0.0006140     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646118 - 1538646124, Ack=1257857607
1937   0.0000750     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857607 - 1257857613, Ack=1538646124
1938   0.0006230     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646124 - 1538646130, Ack=1257857613
1939   0.0000650     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619, Ack=1538646130
                                         Packet does not arrive from server. Client retransmits in 205ms.

1940   0.2053300     Client -> Server     TCP:[ReTransmit #1939] Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619, Ack=1538646130
1941   0.0006840     Server -> Client     TCP:Flags=...A...., SrcPort=1001, DstPort=55552, PayloadLen=0, Seq=1538646136, Ack=1257857619
1942   0.0006970     Server -> Client     TCP:Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 - 1538646136, Ack=1257857619,
1943   0.0001120     Client -> Server     TCP:Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857619 - 1257857625, Ack=15386461

Linux responds in ~200ms, not the suggested 1 second, but this is still not very helpful.  The RTO  of 200ms recovers better than the suggested minimum RTO, but 200ms is way too long as we have a sub-millisecond RTT. A similar test with a Windows host showed a minimum RTO of ~300ms. 

1. Why was the now quaint 1 second minimum RTO carried over into RFC 6298? 
2. Is there a way we can get the minimum RTO down below even 200ms in a low latency network without breaking other TCP features?

I can see some discussion on the internet about this but I don't see movement to come up reasonable modern value the most recent RFC 6298.

e.g. http://www.icir.org/mallman/papers/draft-allman-tcpm-rto-consider-00.txt

    Also, note, that in addition to the experiments discussed in [AP99],
    the Linux TCP implementation has been using various non-standard RTO
    mechanisms for many years seemingly without large scale problems
    (e.g., using different EWMA gains).  Also, a number of
    implementations use minimum RTOs that are less than the 1 second
    specified in [RFC2988].  While the precise implications of this may
    show more spurious retransmits (per [AP99]) we are aware of no large
    scale problems caused by this change to the minimum RTO

Mark Lloyd | Enterprise Network Engineer
F5 Networks

---

From: Jerry Chu <[email-masked]>
Date: Tue, 10 Apr 2012 13:57:04 -0700

Mark,

RFC6298 is mainly for reducing the initial RTO from 3secs to 1sec. We
did not touch the minimum RTO due to a lack of consensus and also
for fear of opening a prolonged debate. Besides, OSes have reduced
minRTO anyway as you noted.

Personally I don't think a somewhat arbitrary RTO lower bound is really
necessary, but delayed acks complicate the matter.

Jerry

On Tue, Apr 10, 2012 at 12:52 PM, Mark Lloyd <[email-masked]> wrote:

> While troubleshooting a problem that involves minimum RTO, I found that
> RFC 6298 suggests a 1 second value for minimum RTO that is not honored in
> the implementations I have tested, and even in these implementation TCP
> does not do a good job of handling packet loss in a common scenario I
> tested.
>
> I use two Linux hosts as my test. They are setup to imitate the kind of
> connection that would be used for a database or RPC transaction.
>
> I'm showing frames leading up to the packet loss to show the what the
> traffic looks like up to the packet loss.
>
> Server side:
>
> Frm    Time
> Num    Delta
> 1941   0.0000830     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646100 - 1538646106,
> Ack=1257857589,
> 1942   0.0005920     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857589 - 1257857595,
> Ack=1538646106
> 1943   0.0000920     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646106 - 1538646112,
> Ack=1257857595
> 1944   0.0005590     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857595 - 1257857601,
> Ack=1538646112
> 1945   0.0000730     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646112 - 1538646118,
> Ack=1257857601
> 1946   0.0005770     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857601 - 1257857607,
> Ack=1538646118
> 1947   0.0000930     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646118 - 1538646124,
> Ack=1257857607
> 1948   0.0006190     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857607 - 1257857613,
> Ack=1538646124
> 1949   0.0000790     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646124 - 1538646130,
> Ack=1257857613
> 1950   0.0005110     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619,
> Ack=1538646130
> 1951   0.0000850     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 - 1538646136,
> Ack=1257857619
> 1952   0.2053670     Client -> Server     TCP:[ReTransmit
> #1950]Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6,
> Seq=1257857613 - 1257857619
> 1953   0.0000420     Server -> Client     TCP:Flags=...A....,
> SrcPort=1001, DstPort=55552, PayloadLen=0, Seq=1538646136, Ack=1257857619
> 1954   0.0007630     Server -> Client     TCP:[ReTransmit #1951]
> Flags=...AP..., SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 -
> 1538646136, Ack=1257857619
> 1955   0.0006640     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857619 - 1257857625,
> Ack=1538646136
>
> Client side:
>
> 1928   0.0006400     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646094 - 1538646100,
> Ack=1257857583
> 1929   0.0000690     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857583 - 1257857589,
> Ack=1538646100
> 1930   0.0006200     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646100 - 1538646106,
> Ack=1257857589
> 1931   0.0000690     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857589 - 1257857595,
> Ack=1538646106
> 1932   0.0006300     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646106 - 1538646112,
> Ack=1257857595
> 1933   0.0000730     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857595 - 1257857601,
> Ack=1538646112
> 1934   0.0005230     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646112 - 1538646118,
> Ack=1257857601
> 1935   0.0000830     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857601 - 1257857607,
> Ack=1538646118
> 1936   0.0006140     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646118 - 1538646124,
> Ack=1257857607
> 1937   0.0000750     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857607 - 1257857613,
> Ack=1538646124
> 1938   0.0006230     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646124 - 1538646130,
> Ack=1257857613
> 1939   0.0000650     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 - 1257857619,
> Ack=1538646130
> 1940   0.2053300     Client -> Server     TCP:[ReTransmit #1939]
> Flags=...AP..., SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857613 -
> 1257857619, Ack=1538646130
> 1941   0.0006840     Server -> Client     TCP:Flags=...A....,
> SrcPort=1001, DstPort=55552, PayloadLen=0, Seq=1538646136, Ack=1257857619
> 1942   0.0006970     Server -> Client     TCP:Flags=...AP...,
> SrcPort=1001, DstPort=55552, PayloadLen=6, Seq=1538646130 - 1538646136,
> Ack=1257857619,
> 1943   0.0001120     Client -> Server     TCP:Flags=...AP...,
> SrcPort=55552, DstPort=1001, PayloadLen=6, Seq=1257857619 - 1257857625,
> Ack=15386461
>
> Linux responds in ~200ms, not the suggested 1 second, but this is still
> not very helpful.  The RTO  of 200ms recovers better than the suggested
> minimum RTO, but 200ms is way too long as we have a sub-millisecond RTT. A
> similar test with a Windows host showed a minimum RTO of ~300ms.
>
>
> 2. Is there a way we can get the minimum RTO down below even 200ms in a
> low latency network without breaking other TCP features?
>
> I can see some discussion on the internet about this but I don't see
> movement to come up reasonable modern value the most recent RFC 6298.
>
> e.g.
> http://www.icir.org/mallman/papers/draft-allman-tcpm-rto-consider-00.txt
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Mark Allman <[email-masked]>
Date: Wed, 11 Apr 2012 09:32:29 -0400

> 1. Why was the now quaint 1 second minimum RTO carried over into RFC

As Jerry noted, we have no idea what else to do.  There is no empirical
evidence that says we have a better general value.  That doesn't mean it
needs to be 1sec.  It just means we have no good basis for some other
arbitrary number.

And, we empirically know 1sec is fairly safe in terms of spurious RTOs.

One of the things I have found over many years of talking to many people
about this is that they do precisely what you showed in your email: they
observe that the min means you wait too long for needed retransmissions.
Well, for needed retransmits *anything* is too long.
I.e., we could wait 1msec and retransmit and *that* would delay us by
1msec over some ideal.  So, no matter what the min is, it is too
conservative.

However, that doesn't take into account the other side of the equation,
which is that aggressiveness leads to unnecessary retransmits and those
lead to collapsing the cwnd.  Now, that might not be an issue for tiny
db transactions.  And, we might be able to undo things with a bit of
effort and a bit of delay.  But, in general, this is the balancing act
one is trying to perform.  You want to wait just long enough to
unambiguously determine you need a rexmt, but no longer.  If you fall on
either side of that point then you're sacrificing performance in one way
or the other.

> 2. Is there a way we can get the minimum RTO down below even 200ms in
>    a low latency network without breaking other TCP features? 

I think a couple of things ...

  - In your own internal, low-latency network you can tune your TCP to
    use whatever min you want.  Who cares?  If you end up shooting
    yourself in the foot then, well, its your own damn foot.

  - My *hunch* is that a general setting of less than 200msec would
    probably mean a reduction in overall performance.  That doesn't mean
    it would do so in your particular environment.  It means that if
    some general purpose OS set it to something under 200msec then I
    would bet a single beer that would not be an overall helpful
    setting.

  - But, that said, I think in general we should get out of the
    parameter setting business where we can (per the quote from the
    expired I-D in your note).  In this case, the process fails safe.
    I.e., if you're too aggressive with the RTO estimator (because you
    tweaked the min or the gains or whatever) then it means you
    retransmit when you don't need to.  And, so (a) there is no
    congestion to react to, but you're reacting anyway and (b) even if
    there is some congestion you are reacting conservatively by using a
    cwnd of one and employing exponential backoff.  So, as long as a few
    big picture principles are used I think the details do not need to
    be uniformly applied.  (It doesn't hurt to write down RTO estimators
    as particular ways to do things---a la RFC 6298---but there is IMHO
    no reason everyone has to use the same RTO estimator.

> e.g. http://www.icir.org/mallman/papers/draft-allman-tcpm-rto-consider-00.txt

This has expired.  It didn't gain a lot of traction.  I'd be happy to do
a quick pass through it for any comments that never got rolled in and
re-issue it if folks think it is useful.  I personally think it is The
Way To Go.  But, I am not WG consensus.

allman

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Wed, 11 Apr 2012 08:21:52 -0700

Hi Mark(s),

FWIW, just as a data point.  Our main Os'es (some of them in the field
for more than 20+ years, although I suspect the RTO change would have
been there for at least 10+ years) have been using min RTO less than 1
sec. in the range (300ms - 600ms). In some cases we had to go back to
the 1 sec (esp in case of WAN optimization proxies which is over a WAN
link where spurious timeout's were interfering with the throughput.)
However for most of the other apps, sub-second RTO's have been working
ok.

In other words, most implementations (if I can say that) have been
relaxing the min RTO restriction. It is treated like a SHOULD (MAY?)
instead of a MUST. 

I support the "rto-reconsider" draft.

Thanks,
-Anantha

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Mark Allman
> Sent: Wednesday, April 11, 2012 6:32 AM
> To: Mark Lloyd
> Cc: [email-masked]
> Subject: Re: [tcpm] Minimum RTO
> 
> 
> evidence that says we have a better general value.  That doesn't mean
> it needs to be 1sec.  It just means we have no good basis for some
> other arbitrary number.
> 
> One of the things I have found over many years of talking to many
> people about this is that they do precisely what you showed in your
> email: they observe that the min means you wait too long for needed
> retransmissions.
> unambiguously determine you need a rexmt, but no longer.  If you fall
> on either side of that point then you're sacrificing performance in
one
> way or the other.
> > http://www.icir.org/mallman/papers/draft-allman-tcpm-rto-consider-
> 00.t
> > xt
> 
> This has expired.  It didn't gain a lot of traction.  I'd be happy to
> do a quick pass through it for any comments that never got rolled in
> and re-issue it if folks think it is useful.  I personally think it is
> The Way To Go.  But, I am not WG consensus.
> 
> allman
> 
>

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 11 Apr 2012 18:56:33 +0200

> low latency network without breaking other TCP features?

Maybe a helpful link:

http://utopia.duth.gr/~ipsaras/minrto-networking07-psaras.pdf

Hagen

PS: the IETF do not focus on low latency networks. The sum of all network
characteristics is important. (just as a reminder).

PPS: with Linux you can already modify the minimum RTO via a per route
option (RTAX_RTO_MIN via iproute)

---

From: Murari Sridharan <[email-masked]>
Date: Wed, 11 Apr 2012 17:04:16 +0000

We have tried to solve these problems by creating templates, so we have a Datacenter template, Internet template and so on. The defaults are appropriately chosen for each template. If the user knows better he can configure a template or we configure automatically based on latency characteristics. More often than not its enough to differentiate a super low latency datacenter network and Internet correctly. If the admin knows better they can configure one. This allowed us to satisfy the diverse needs. There is also a custom template for experimental stuff like larger initial CWnd etc. 

Thanks

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Mark Allman
Sent: Wednesday, April 11, 2012 6:32 AM
To: Mark Lloyd
Cc: [email-masked]
Subject: Re: [tcpm] Minimum RTO

> 1. Why was the now quaint 1 second minimum RTO carried over into RFC

As Jerry noted, we have no idea what else to do.  There is no empirical evidence that says we have a better general value.  That doesn't mean it needs to be 1sec.  It just means we have no good basis for some other arbitrary number.

One of the things I have found over many years of talking to many people about this is that they do precisely what you showed in your email: they observe that the min means you wait too long for needed retransmissions.
Well, for needed retransmits *anything* is too long.
I.e., we could wait 1msec and retransmit and *that* would delay us by 1msec over some ideal.  So, no matter what the min is, it is too conservative.

However, that doesn't take into account the other side of the equation, which is that aggressiveness leads to unnecessary retransmits and those lead to collapsing the cwnd.  Now, that might not be an issue for tiny db transactions.  And, we might be able to undo things with a bit of effort and a bit of delay.  But, in general, this is the balancing act one is trying to perform.  You want to wait just long enough to unambiguously determine you need a rexmt, but no longer.  If you fall on either side of that point then you're sacrificing performance in one way or the other.

> http://www.icir.org/mallman/papers/draft-allman-tcpm-rto-consider-00.t

This has expired.  It didn't gain a lot of traction.  I'd be happy to do a quick pass through it for any comments that never got rolled in and re-issue it if folks think it is useful.  I personally think it is The Way To Go.  But, I am not WG consensus.

allman