Subject: [tcpm] About the SYN/ACK-timer

From: Michael Welzl <[email-masked]>
Date: Sat, 29 Dec 2012 13:21:34 +0100

Dear all,

In my opinion, the web (and with it, probably many other applications too) exhibits a somewhat annoying problem that we still haven't fixed, but just gotten used to:
sometimes, when I click something, it doesn't immediately work. So I stop the transfer and click again. Then it works.

Someone once told me that, when this happens, it's probably because a SYN or SYN/ACK was lost. I believe this is correct. Several years ago, this notion has prompted me to do some measuring + ugly fixing:
http://heim.ifi.uio.no/~michawe/research/tools/syn-retransmit/index.html
…in doing so, we found that, indeed, the problem is significant (in terms of number of SYNs or SYN/ACKs lost). It's probably not hard to find more such data - I guess that Google has produced it when they first worked on Fast Open  (sorry for not checking right now).

When I approached TCPM with a suggestion to reduce the timers for SYN and SYN/ACK retransmission, I ended up with a dialogue with Joe Touch, in which he convinced me that reducing the SYN timer could be harmful, as it could cause busy servers to get hammered by even more SYNs when they just can't cope with them. I buy that; but IIRC he had no good reason against making the SYN/ACK-timer more aggressive. BTW sorry if I'm using a wrong name here, but I guess it's clear: I mean the timer used by the server to resend the SYN/ACK if it doesn't get a response for a while.

So, the thought occurs to me every once in a while (especially when I do surf the web, and something temporarily hangs), and just now I felt like sharing it again: why don't we really make the SYN/ACK timer more aggressive? This should for instance appeal to the Googlers on the list who are trying to improve TCP for their cause, no?

I think if it's about establishing that the problem is significant, that shouldn't be too hard. Measurements show that SYN/ACKs do get lost, and when they do, the user feels it. Sure, it's not the worst problem in the world, but I do believe it's worth fixing.

I'm just curious what everyone thinks… maybe I'm completely off track for one reason or another?!

Cheers,
Michael

---

From: Marco Mellia <[email-masked]>
Date: Sat, 29 Dec 2012 14:54:32 +0100

Hi Michael,

below you can find some numbers, derived from passive measurements of about 20k ADSL customers, during 1h @peak time. It seems that the 1st syn lost affects mostly P2P traffic... for HTTP, it's less than 0.7%. Heavy hitters do not suffer for that a lot. The analysis of the most affected http servers seems to point to server congestion too.
[We can provide more numbers in the context of the mPlane project (www.ict-mplane.eu) if interested.]

My 2c is that 3s of initial timeout is possibly conservative. However, lowering it to 1s will not change user experience a lot.
Below 1s will cause lot of false retransmissions (especially on wireless access scenarios where RTT can easily grow above 1s for the first packet...). More complicated solutions can be engineered (I did something on this line 5/6y ago...). For example, choosing the initial RTO based on RTT history, or access network could be an option. Still, I'm not sure that it is worth the complexity.

Nice 2013,
Ciao
M
----------
#Protocol    #connections  #multiSYN  %multiple  SYN
UNKNOWN     524112        29299      5.59022
HTTP        1834776       12154      0.662424
P2P-plain   766858        62290      8.12276
BT-tracker  31533         1682       5.33409
SMTP        1682          3          0.178359
POP3        16107         39         0.242131
SSL/TLS     422609        2704       0.639835
P2P-obf     270055        19401      7.18409
RTMP        2255          13         0.576497
BT-obf      266951        21866      8.19102

Among HTTP connections, these are numbers for the top 10 domains names...
#Domain name                    #connections  #multiSYN  %multiple  SYN
apps.facebook.com              10014         34         0.339525
pagead2.googlesyndication.com  10728         22         0.205071
ad.yieldmanager.com            10944         25         0.228436
googleads.g.doubleclick.net    11976         32         0.267201
external.ak.fbcdn.net          12285         20         0.1628
s-static.ak.facebook.com       13298         39         0.293277
s-static.ak.fbcdn.net          14642         39         0.266357
www.google.com                 14740         184        1.2483
www.google-analytics.com       15755         38         0.241193
fbcdn-photos-a.akamaihd.net    15932         34         0.213407
secure-it.imrworldwide.com     16229         56         0.345061
www.google.it                  17158         75         0.437114
static.ak.fbcdn.net            22751         62         0.272515
fbcdn-profile-a.akamaihd.net   22892         65         0.283942
profile.ak.fbcdn.net           36777         116        0.315415
www.facebook.com               94317         574        0.608586

-- 
Marco Mellia - Assistant Professor
Dipartimento di Elettronica e Telecomunicazioni
Politecnico di Torino
Corso Duca Degli Abruzzi 24
10129 - Torino - IT
Tel: +39-011-090-4173
Cel: +39-331-6714789
Skype: mgmellia
Home page: http://www.tlc-networks.polito.it/mellia

Il giorno 29/dic/2012, alle ore 13:21, Michael Welzl <[email-masked]> ha scritto:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Jakob Heitz <[email-masked]>
Date: Sat, 29 Dec 2012 18:19:46 +0000

There is no need to fix this in TCP. The browser could fix it.
The browser could start a new TCP connection every ~250mS.
Then use whichever TCP connection that completes first.
It could keep the remaining TCP sessions open, just in case it needs to get more objects or pages from the same web site.

Now, there is nothing stopping browsers from doing such workarounds...
...except a fix in TCP. Why is a SYN retransmission timer of 100mS so bad, even if it does put a few more packets into the network?

--
Jakob Heitz.

---

From: Michael Welzl <[email-masked]>
Date: Sat, 29 Dec 2012 19:35:00 +0100

Hi,

The percentage matches what we found in our measurements (around 0.5%). However I don't think that's so small? It's not "occurrences per user per server", it's "occurrences in total", i.e. across all connections that are opened to the server… some websites cause tons of connections to be opened, so you have to multiply with the number of such connections to get the chance of such a problem happening to you when you visit a website

I agree that more complicated solutions may not be worth the complexity. I wonder about smaller values, like 1s as you say. Is the increased number of potential unnecessary retransmits a big problem? Dunno…

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Sat, 29 Dec 2012 19:42:39 +0100

> ...except a fix in TCP. Why is a SYN retransmission timer of 100mS so bad, even if it does put a few more packets into the network?

Not sure if it really would be that bad, not sure at all.
Anyway, my main point was that there's probably even less harm in making the SYN/ACK timer more aggressive, so why not do at least that?

Cheers,
Michael

---

From: Marco Mellia <[email-masked]>
Date: Sun, 30 Dec 2012 10:51:10 +0100

Oh sure, you could (allow google to) do everything at the application
layer, so that your browser can open tons of connections, resolve all
hostnames, prefetech all links in a page, and download the same stuff
twice just in case a copy gets corrupted.
Who cares about fairness, efficiency, waste of resources, etc.etc. We
are living in an infinite resource world.
We already see youtube wasting 40% of the network capacity, so who cares?

Sorry for the harsh tone, but honestly if there is a (possible) problem
at the TCP layer, you should work to fix it there.
Any solution must also weight advantages and drawbacks. "few more
packets" in the network can be overkilling in some scenarios (just
thinking on slow and congested 3g network). Few more connections may
kill your NAT modem.
Lowering the initial RTO to 1s (or less, considering 95% of possible
scenarios) seems a simple and straight forward solution here.
Ciao
M

-- 
Ciao,                    /\/\/\rco

+-----------------------------------+
| Marco Mellia - Assistant Professor|
| Skypeid: mgmellia                 |
| Tel: +39-011-090-4173             |
| Cel: +39-331-6714789              |   /"\  .. . . . . . . . . . . . .
| Politecnico di Torino             |   \ /  . ASCII Ribbon Campaign  .
| Corso Duca degli Abruzzi 24       |    X   .- NO HTML/RTF in e-mail .
| Torino - 10129 - Italy            |   / \  .- NO Word docs in e-mail.
| http://www.telematica.polito.it   |        .. . . . . . . . . . . . .
+-----------------------------------+
The box said "Requires Windows 95 or Better." So I installed Linux.

---

From: Joe Touch <[email-masked]>
Date: Sun, 30 Dec 2012 08:45:32 -0800

On 12/30/2012 1:51 AM, Marco Mellia wrote:
...
> thinking on slow and congested 3g network). Few more connections may

Right now I hear the following from this discussion:

	- a *hypothesis* that connections are slow because either
	SYNs __or__ SYN/ACKs are getting lost

	- a proposed solution based on that hypothesis that
	makes connections less susceptible to timeout-based delays

What I have not seen:

	- any evidence that either SYN or SYN/ACK loss is the problem
		the former would be measured at a client;
		the latter needs to be measured at a loaded server

	- any analysis as to whether the proposed solution creates
	a problem
		e.g., if the SYN/ACKs aren't going out because of
		server overload, firing off more timers isn't going
		to help

	- any analysis of interactions between this and emerging
	uses of TCP
		e.g., one proposed SYN-loss solution is to open
		multiple connections and use the first one
		that responds; how would this interact with a
		server that also implemented lower SYN/ACK rexmits?

So, this all sounds like a very interesting area for someone to gather
data and analyze, but just because the *potential* solution is simple
doesn't mean it's warranted, safe, or appropriate at this time, IMO.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 30 Dec 2012 18:16:35 +0000

Consider:
A browser starts 8 TCP sessions
vs.
A TCP stack retransmits 8 SYN packets.

Which causes more stress on the Internet and the server?

Inaction at the TCP layer causes action at the application layer.
BTW, I think the SYN packet should include data. A simple change to the sockets API should do it.

--
Jakob Heitz.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Sun, 30 Dec 2012 19:22:52 +0100 (CET)

> sockets API should do it.

While I am all for it, I believe some security products will get upset by 
it:

<http://www.symantec.com/connect/articles/abnormal-ip-packets>

"A SYN only packet, which should only occur when a new connection is being 
initiated, should not contain any data."

So this needs to be investigated what the impact of this might be. I still 
remember security products dropping packets with ECN markings back when 
that was introduced > 10 years ago.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Joe Touch <[email-masked]>
Date: Sun, 30 Dec 2012 10:43:41 -0800

> Which causes more stress on the Internet and the server?

Answer:

	it depends on a lot of factors:
		- is the client overloaded or memory-constrained?
		- are the 8 sessions to different servers or the same?
		- are the paths lossy?
		- are the paths congested?
		- are the servers overloaded?

If the 8 connections are to different servers, it's a completely 
different answer than if they're to the same one.

> Inaction at the TCP layer causes action at the application layer.

By that argument, your stack should send 1000 copies of every packet, in 
the hopes that at least one gets through.

See: "tragedy of the commons", however.

> BTW, I think the SYN packet should include data. A simple change to the sockets API should do it.

Please review RFC793; you would need to change more than just the API. 
This "dog" has been dragged around the block a *lot*, and it's a lot 
more complex.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sun, 30 Dec 2012 10:45:46 -0800

> Please review RFC793; you would need to change more than just the API.

...if it is to have any real effect. You can send data in the SYN now, 
but the server can't deliver it to the app layer until the three-way 
handshake completes. And it also increases the amount of state for 
pending connections, increasing the impact of DOS attacks.

Joe

---

From: "Agarwal, Anil" <[email-masked]>
Date: Sun, 30 Dec 2012 19:11:11 +0000

Michael et al,

I tend to agree that reducing the TCP SYN-ACK timeout value would be
beneficial.

A typical Linux TCP implementation retransmits SYN-ACKs 5 times
at times shown below, assuming the first SYN-ACK is sent at time 0 -
    3, 9, 21, 45, 93 seconds
and finally give up at time 189 seconds.

If we start with an initial timer value of 1 second, then TCP would
Retransmit SYN-ACK segments at times -
    1, 3, 7, 15, 31 seconds
and finally give up at time 63 seconds.

At first blush, this looks better, for the vast variety of network conditions today.

Some expected objections:
1. This would cause unnecessary retransmissions in some cases, 
   when RTT is > 1 second.
   Firstly, these unnecessary retransmissions would not break anything;
   TCP knows how to handle duplicate SYNs and SYN-ACKs.
   If we are concerned with causing network congestion, then keep
   in mind this injects < 64 bytes in a network per new connection.
   Surely, most networks can afford to carry an additional < 512 bps for
   a new connection, given that established connections consume much more
   bandwidth than that.

2. This cause additional processing overhead at servers
   Not really.
   The server is going to retransmit up to 5 times for a new connection.
   Whether it does it in 93 seconds or 31 seconds should not make much
   difference.
   Quicker establishment and completion of connections is better for servers
   compared to keeping connections in SYN-Recieved state for extended
   periods of time.

Other benefits:
This enables servers to free up resources quicker, when connections fail
to complete or under SYN flood attacks.

Also, TCP should allow the SYN-ACK timer value and the number of
SYN Retransmissions value to be configurable and perhaps should allow 
history-based RTT learning, to handle cases where RTT is known to be 
greater than 1 second. RFC 5482 "TCP User Timeout Option" would also be useful.
We could recommend to implementers that reduced SYN-ACK timer value
always be supplemented by history-based RTT learning.

One could argue that the reduced initial timer value would be beneficial for
SYN packets as well.

I don't think we should encourage proposals that open multiple connections
to the same server and select the first one that succeeds. It consumes more
resources at the client, in the network and at the server. The technique is
probably a reaction to the "sluggishness" of TCP connection establishment
when SYN or SYN-ACK segments are lost.

Yes, we need to "collect" more data on current Internet traffic and
quantify the benefits of the proposed change, but I think it an idea worth
pursuing. It has similarities to the proposals to increase the initial TCP
congestion window size.

Cheers,
Anil Agarwal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Sunday, December 30, 2012 1:46 PM
To: Jakob Heitz
Cc: tcpm ([email-masked]); Marco Mellia; Michael Welzl
Subject: Re: [tcpm] About the SYN/ACK-timer

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 30 Dec 2012 19:23:05 +0000

On Dec 30, 2012, at 10:46 AM, "Joe Touch" <[email-masked]> wrote:
> 
> ...if it is to have any real effect. You can send data in the SYN now, but the server can't deliver it to the app layer until the three-way handshake completes. And it also increases the amount of state for pending connections, increasing the impact of DOS attacks.
> 
> Joe

It could, with a change to the sockets API. Of course, the app will need to participate in SYN flood protection, but that's not impossible given the right architecture.

--
Jakob Heitz.

---

From: Joe Touch <[email-masked]>
Date: Sun, 30 Dec 2012 13:17:04 -0800

On Dec 30, 2012, at 11:23 AM, Jakob Heitz <[email-masked]> wrote:

> It could, with a change to the sockets API.

No, it cannot without a change in CTO semantics, which affects the state diagram and its processing on both ends - which also changes the meaning of an ack and the concept of reliability. 

There is a lot required, and for the first connection to a site it won't work to allow the server to act on data before a connection has been established. 

This is well- trod ground. 

Joe

> Of course, the app will need to participate in SYN flood protection, but that's not impossible given the right architecture.
> 
> --
> Jakob Heitz.
>

---

From: Mark Allman <[email-masked]>
Date: Sun, 30 Dec 2012 22:50:46 -0500

Um.  So, I just skimmed this thread.  I see people suggesting a "SYN-ACK
timer" of 1sec---even though it seems that the original note is that it
be "reduced" without providing any concreteness to that.  But,
subsequently we see things like this ...

> and finally give up at time 63 seconds.

Well, um, err, AHEM ... this working group produced RFC 6298, which says
the initial RTO SHOULD be 1sec.

So, "typical Linux" should either be using the latter progression given
above or someone has decided there is a good reason to be more
conservative.

Or, is someone proposing we lower the initial RTO further?

allman

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 31 Dec 2012 10:23:18 +0100 (CET)

On Sun, 30 Dec 2012, Agarwal, Anil wrote:

> quantify the benefits of the proposed change, but I think it an idea 
> worth pursuing. It has similarities to the proposals to increase the 
> initial TCP congestion window size.

I agree. Whilst some users still are on 2G connections which might be very 
slow (12-14 kilobit/s), they still can handle both an initial larger 
congestion window (there is sufficient buffering) and a lower SYN+ACK 
retransmit timer. For most users around the world now and more going 
forward, RTT is going to be less than 1 second.

I wholeheartedly support any work on TCP that will make it more adaptable 
to user conditions. TCP was designed to handle everything from a low bit/s 
access connection to tens of gigabit/s, but these initial conditions 
before the connection is up and running with sufficiently sized congestion 
window for the high bandwidth access can still be improved. One help with 
this might be heuritics to understand what the general access latency is 
for the user connection, and how it behaves.

A FTTH connection is vastly different in behaviour from a UMTS/3G 
connection which is vastly different from a 64 kilobit/s fixed connection.

If we want TCP to behave more optimized for each access scenario, I 
believe TCP needs to know about it over time and behave differently 
depending on detected access type, and most likely also communicate the 
access it has to the other end of the sessions being initiated.

---

From: Michael Welzl <[email-masked]>
Date: Mon, 31 Dec 2012 10:52:43 +0100

Hi,

Not sure about everyone else, but the way I interpreted (and meant) the 1s bit was to say "a new value, such as 1s, as opposed to a complex adaptive mechanism". It wasn't about the exact value of 1s per se.

What would be a good new value? No idea… as Joe has pointed out, there is still some work to be done here, before proposing a change. I agree that my story of web surfing hiccups (where you have to press reload to get the whole thing that you're waiting for) is merely a hypothesis, and e.g. investigating lost SYNs / SYN-ACKs are really the root of the problem, and how severe that problem is, could be quite interesting.

A part of the problem as I described it involves the user pressing the browser's "refresh" button. The effect of that could be seen on the wire… HTTP requests would be sent again, at user-determined times… does it happen often? Does it coincide with lost SYNs or SYN-ACKs? Interesting stuff to look into, IMO.

About lowering the initial RTO further: I, for one, hadn't thought of that, and am not fully aware of all the pro's and con's. But yeah, it might be worth considering?

Cheers,
Michael

---

From: "Agarwal, Anil" <[email-masked]>
Date: Mon, 31 Dec 2012 14:11:50 +0000

Mark,

I stand corrected; forgot about RFC 6298.
RFC 6298 clearly states that until an RTT measurement is made, "the sender SHOULD set RTO <- 1 second". RFC 6298 contains rationale, supporting data and additional rules/guidelines for use of this shortened timer.

Looks like the initial timeout was reduced to 1 second in Linux 3.1, both for SYN and SYN-ACK segments. Did a quick test with Linux 3.2.14; the SYN retransmission times are -
     1, 3, 7, 15, 31 seconds
with a final give up time of ~63 seconds.

Anil

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Sunday, December 30, 2012 10:51 PM
To: Agarwal, Anil
Cc: Joe Touch; Jakob Heitz; tcpm ([email-masked]); Michael Welzl; Marco Mellia
Subject: Re: [tcpm] About the SYN/ACK-timer 

allman

---

From: Jakob Heitz <[email-masked]>
Date: Mon, 31 Dec 2012 19:24:28 +0000

I have an alternative to cookies.

Send data with the SYN.
When retransmission of the SYN is required, retransmit it without the data.
Retransmit the SYN quickly: 100 to 200 mS.

It works with Middleboxes.
SYN flood protection works as normal, just bias it against SYN with data.

Cheers,
Jakob.

On Dec 31, 2012, at 1:44 AM, "Michael Welzl" <[email-masked]> wrote:

> Hi,
> 
> About data-in-the-SYN: I agree with Joe that this has been debated a lot already - see https://tools.ietf.org/html/draft-ietf-tcpm-fastopen-02 and the discussions surrounding it. FWIW this draft is a (IMHO, good) proposal to do what you want, in the right way
> 
> Cheers,
> Michael
> 
> 
> On Dec 30, 2012, at 10:17 PM, Joe Touch <[email-masked]> wrote:
> 
>> 
>>