Subject: Re: [tcpm] New Version Notification for draft-hurtig-tcpm-rtorestart-02.txt

From: Per Hurtig <[email-masked]>
Date: Thu, 1 Nov 2012 21:18:27 +0100

Hi,

We've run experiments (using Linux TCP) to determine whether our restart
approach is more likely to cause spurious retransmissions in a burst
transmission scenario as suggested by Yuchung.

In the experiments 30 packets are transmitted in total. At first there is a
very low initial delay, but from packet 11 and onward the bandwidth is
restricted so that the spacing between packets become 200 ms. Given an
initial window of 10 and that Linux ACKs each packet in the initial window
the scenario creates a burst transmission of 20 packets as suggested in
Yuchung's example. In our experiments we see the "balloon effect" that Mark
predicted very clearly (the RTO becomes extremely inflated due to the
variation). There is no risk that the modified RTO approach causes spurious
retransmissions.

Furthermore, possible problems in burst scenarios are quite unlikely, as
spurious retransmissions would only affect the transmission if a new large
burst of data is to be sent after the RTO timer spuriously expires
(otherwise the spurious retransmission is just a wasted packet in the end
of a short flow), but before an RTO worth of time has elapsed (otherwise
cwnd is reset to the restart window).

Per

On Tue, Jul 10, 2012 at 5:13 PM, Mark Allman <[email-masked]> wrote:

>
> > I am concerned about this following hypothetical scenario:
> >
> > t0.0s: bursts packets 1-20, mount RTO to 2second
> > t1.8s: gets ack of packets 2,4,6,8,10,12,14,16, 18 every 200ms.
> >        Ack of 18 arrives at 1.8s
> >
> > This draft will mount an RTO of 200ms for packet 19 on ack18, which
> > is likely to be spurious. The problem is that T_earliest could be
> > starting from large send a while back even when current FlightSize
> > is small.
>
> I hear you.  But, I don't buy your framing.  You keep the RTO at a
> constant throughout the process.  But, with a variance like you describe
> above it certainly seems plausible that the RTO will balloon.  And, the
> way the document is framed when 18 arrives we restart the timer with
> RTO-T_earliest.  But, if RTO is > 2sec then the timeout will be >
> 200ms.
>
> So, I don't think it is as easy as you have portrayed it.
>
> allman
>
>
>
>

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 1 Nov 2012 21:58:56 +0100

On Nov 1, 2012, at 9:18 PM, Per Hurtig wrote:

> Hi,
> 
> We've run experiments (using Linux TCP) to determine whether our restart approach is more likely to cause spurious retransmissions in a burst transmission scenario as suggested by Yuchung.
> 
> In the experiments 30 packets are transmitted in total. At first there is a very low initial delay, but from packet 11 and onward the bandwidth is restricted so that the spacing between packets become 200 ms. Given an initial window of 10 and that Linux ACKs each packet in the initial window the scenario creates a burst transmission of 20 packets as suggested in Yuchung's 
Just curios: How does the receiver know what the sender uses as its initial window?

Best regards
Michael
> example. In our experiments we see the "balloon effect" that Mark predicted very clearly (the RTO becomes extremely inflated due to the variation). There is no risk that the modified RTO approach causes spurious retransmissions.
> 
> Furthermore, possible problems in burst scenarios are quite unlikely, as spurious retransmissions would only affect the transmission if a new large burst of data is to be sent after the RTO timer spuriously expires (otherwise the spurious retransmission is just a wasted packet in the end of a short flow), but before an RTO worth of time has elapsed (otherwise cwnd is reset to the restart window).
> 
> 
> 
> Per
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Michael Welzl <[email-masked]>
Date: Thu, 1 Nov 2012 22:05:51 +0100

On Nov 1, 2012, at 9:58 PM, Michael Tuexen wrote:

>> We've run experiments (using Linux TCP) to determine whether our  
>> restart approach is more likely to cause spurious retransmissions  
>> in a burst transmission scenario as suggested by Yuchung.
>>
>> In the experiments 30 packets are transmitted in total. At first  
>> there is a very low initial delay, but from packet 11 and onward  
>> the bandwidth is restricted so that the spacing between packets  
>> become 200 ms. Given an initial window of 10 and that Linux ACKs  
>> each packet in the initial window the scenario creates a burst  
>> transmission of 20 packets as suggested in Yuchung's

RTO Restart is a sender-side only change. Neither this draft nor the  
scenario that Yuchung described (his concern that we're trying to  
address here) requires the receiver to know the sender's IW.
If you're wondering where the bandwidth restriction from packet 11 and  
onward comes from: we made this, artificially, to recreate the  
scenario of Yuchung's concern.

Cheers,
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Thu, 1 Nov 2012 22:10:10 +0100

On Nov 1, 2012, at 10:05 PM, Michael Welzl wrote:

> RTO Restart is a sender-side only change. Neither this draft nor the scenario that Yuchung described (his concern that we're trying to address here) requires the receiver to know the sender's IW.
> If you're wondering where the bandwidth restriction from packet 11 and onward comes from: we made this, artificially, to recreate the scenario of Yuchung's concern.
I was wondering how the receiver can ack each packet in the senders initial window?
Does the receiver assume that the sender's initial window is the same is the initial
window the receiver would use?

Best regards
Michael

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 2 Nov 2012 10:31:01 +0200

> window the receiver would use?

I believe this refers to the quick ack mechanism Linux applies in the beginning of connection. If I recall it right, the number of packets to quick-ack was some fraction of the initial receiver window.

- Pasi

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 2 Nov 2012 10:22:59 +0100

> I believe this refers to the quick ack mechanism Linux applies in the beginning of connection. If I recall it right, the number of packets to quick-ack was some fraction of the initial receiver win…
Ahh, I see. Thanks for the clarification. This is something Linux specific, right?

Best regards
Michael

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 2 Nov 2012 12:18:05 +0200

On Nov 1, 2012, at 10:18 PM, Per Hurtig wrote:

> In the experiments 30 packets are transmitted in total. At first there is a very low initial delay, but from packet 11 and onward the bandwidth is restricted so that the spacing between packets become 200 ms. Given an initial window of 10 and that Linux ACKs each packet in the initial window the scenario creates a burst transmission of 20 packets as suggested in Yuchung's example. In our experiments we see the "balloon effect" that Mark predicted very clearly (the RTO becomes extremely inflated due to the variation). There is no risk that the modified RTO approach causes spurious retransmissions.

An additional complication is that not all RTO implementations are similar.

The risk of spurious RTO depends on minimum RTO and the development of the variance component in the RTO computation. With RFC 6298 - compliant implementation the minimum RTO is 1 second, so I wouldn't be so worried about spurious timeouts, because the RTTs are typically much smaller. On the other hand, the relative benefit of rtorestart might also be more modest then.

But there are implementations (such as Linux) with smaller minimum RTO and possibly some other tweaks on the RTO computation algorithm. It is harder to see how different scenarios play out in such cases, for example after a longer period of steady RTTs followed by a tiny delay spike.

Perhaps the draft should discuss the factors affecting the likelihood of spurious timeouts in different implementations. A strawman mitigation that comes to mind could be to ensure that the variance component also has a reasonable minimum bound, if a non-RFC 6298 compliant implementation with RTO less than one second is used together with rtorestart. (There is 'G' in RFC 6298, but clock granularity can also be quite fine in some systems)

- Pasi

---

From: Michael Welzl <[email-masked]>
Date: Fri, 2 Nov 2012 12:27:26 +0100

On Nov 2, 2012, at 11:18 AM, Pasi Sarolahti wrote:

>> transmission of 20 packets as suggested in Yuchung's example. In  
>> our experiments we see the "balloon effect" that Mark predicted  
>> very clearly (the RTO becomes extremely inflated due to the  
>> variation). There is no risk that the modified RTO approach causes  
>> spurious retransmissions.
>
> The risk of spurious RTO depends on minimum RTO and the development  
> of the variance component in the RTO computation. With RFC 6298 -  
> compliant implementation the minimum RTO is 1 second, so I wouldn't  
> be so worried about spurious timeouts, because the RTTs are  
> typically much smaller. On the other hand, the relative benefit of  
> rtorestart might also be more modest then.
>
> But there are implementations (such as Linux) with smaller minimum  
> RTO and possibly some other tweaks on the RTO computation algorithm.  
> It is harder to see how different scenarios play out in such cases,  
> for example after a longer period of steady RTTs followed by a tiny  
> delay spike.
>
> Perhaps the draft should discuss the factors affecting the  
> likelihood of spurious timeouts in different implementations. A  
> strawman mitigation that comes to mind could be to ensure that the  
> variance component also has a reasonable minimum bound, if a non-RFC  
> 6298 compliant implementation with RTO less than one second is used  
> together with rtorestart. (There is 'G' in RFC 6298, but clock  
> granularity can also be quite fine in some systems)

I see your point; that can be done.

Note however this paragraph from Per's email, which describes a  
further limitation of the possible effect of spurious timeouts:

**
Furthermore, possible problems in burst scenarios are quite unlikely,  
as spurious retransmissions would only affect the transmission if a  
new large burst of data is to be sent after the RTO timer spuriously  
expires (otherwise the spurious retransmission is just a wasted packet  
in the end of a short flow), but before an RTO worth of time has  
elapsed (otherwise cwnd is reset to the restart window).
**

To give an example:
The TCP sender transmits 10 packets in one go, transmitted at times  
t1, t2, t3... t10, and then stops because the application doesn't have  
any more data to send. As per RFC 5681, cwnd SHOULD be set to RW after  
t10+RTO (or even less if congestion is detected). To avoid that this  
happens, the application has to start sending again before t10+RTO.  
The earliest spurious RTO that could occur because of our draft is the  
RTO for packet 8 which was transmitted at t8. As a result, a spurious  
timeout from our draft would only have an effect if the application  
sends data again within the time window [t8+RTO, t10+RTO]

Scenarios can probably be constructed where this can play out worse,  
but it's an additional limitation worth noting, I think, on top of  
what we're discussing above.

Cheers,
Michael

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 2 Nov 2012 14:32:19 +0200

On Nov 2, 2012, at 1:27 PM, Michael Welzl wrote:

> To give an example:
> The TCP sender transmits 10 packets in one go, transmitted at times t1, t2, t3... t10, and then stops because the application doesn't have any more data to send. As per RFC 5681, cwnd SHOULD be set to RW after t10+RTO (or even less if congestion is detected). To avoid that this happens, the application has to start sending again before t10+RTO. The earliest spurious RTO that could occur because of our draft is the RTO for packet 8 which was transmitted at t8. As a result, a spurious timeout from our draft would only have an effect if the application sends data again within the time window [t8+RTO, t10+RTO]

I guess the hypothetical problem cases are such where the computed RTO is close to RTT, and for that reason the timer would be at some sort of risk of expiring spuriously even without RTO-restart. Then RTO-restart adds to this risk by some factor, that probably depends on the bandwidth and delay characteristics on the path.

I wonder if it was possible to give a rough guideline on when RTO-restart is at spurious timeout risk zone, without taking position on how likely or unlikely that might be. When RTO < 2*RTT, perhaps?

- Pasi

---

From: "Anna Brunström" <[email-masked]>
Date: Fri, 02 Nov 2012 16:40:45 +0100

-----Original Message-----
From: Pasi Sarolahti <[email-masked]>
To: Michael Welzl <[email-masked]>
Cc: tcpm <[email-masked]>, "[email-masked]" <[email-masked]>
Date: Fri, 2 Nov 2012 14:32:19 +0200
Subject: Re: [tcpm] New Version Notification for
draft-hurtig-tcpm-rtorestart-02.txt

> happens, the application has to start sending again before t10+RTO. The
> earliest spurious RTO that could occur because of our draft is the RTO
> for packet 8 which was transmitted at t8. As a result, a spurious
> I guess the hypothetical problem cases are such where the computed RTO
> is close to RTT, and for that reason the timer would be at some sort of
> risk of expiring spuriously even without RTO-restart. Then RTO-restart
> adds to this risk by some factor, that probably depends on the
> bandwidth and delay characteristics on the path.

The problem cases should indeed be very similar. If a new burst is
transmitted in Michael's example above, packet 11 will have the same RTO
as packet 8 (assuming the RTO itself has not changed).   

So in cases where spurious retransmissions cause problems, any mitigations
should be useful also when you do not use RTO-restart. 

BR,
Anna

---

From: Yuchung Cheng <[email-masked]>
Date: Sat, 3 Nov 2012 17:33:16 -0700

> variation). There is no risk that the modified RTO approach causes spurious
Thanks for the experiment. Per-packet RTT sampling is critical to "ballon" the
RTO on a window worth of burst.

I wonder if you can share the Linux patch with me. I am interested to
experiment more.
> (otherwise the spurious retransmission is just a wasted packet in the end of
> a short flow), but before an RTO worth of time has elapsed (otherwise cwnd
> is reset to the restart window).
I am puzzled by this part: if there is no risk to trigger spurious RTO
like you said why bother its effects anyways?

Off-topic but web servers don't restart cwnd after idle, at least I know
several ones that we probably all visit everyday.

---

From: Pasi Sarolahti <[email-masked]>
Date: Sun, 4 Nov 2012 14:29:18 +0100

> should be useful also when you do not use RTO-restart. 

I'm not so sure about the mitigations when we are at the end of the send window.

Different scenarios aside, I understand that we are only talking about few packets, so erring on one side or another may not be a big issue in practice, but as a design point:

It seems possible that under some circumstances RTO could get values close to SRTT.

It seems possible that under some conditions T_earliest could be almost one RTT away.

So in principle, isn't it possible that RTO - T_earliest could get values close to 0? Would it make sense to add some low bound to RTO restart, for example
max(RTO - T_earliest, SRTT)?

- Pasi

---

From: Michael Welzl <[email-masked]>
Date: Sun, 4 Nov 2012 21:36:50 +0100

On Nov 4, 2012, at 1:33 AM, Yuchung Cheng wrote:

>> restart
>> there is a
>> spurious

Sure! (stay tuned..)

> like you said why bother its effects anyways?

The point is not that there is no risk, but little risk.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Sun, 4 Nov 2012 21:58:45 +0100

Hi,

On Nov 4, 2012, at 2:29 PM, Pasi Sarolahti wrote:

> Different scenarios aside, I understand that we are only talking  
> about few packets, so erring on one side or another may not be a big  
> issue in practice, but as a design point:
>

I think we assume that T_earliest is always at least an RTT.

> So in principle, isn't it possible that RTO - T_earliest could get  
> values close to 0? Would it make sense to add some low bound to RTO  
> restart, for example

Yes, very valid point, the draft doesn't discuss the <= 0 case; but I  
think this should then be max(RTO - T_earliest, 0).

(It's about firing the timer (RTO-T_earliest) seconds *after* the ACK  
comes in)

Cheers,
Michael

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 4 Nov 2012 22:05:26 +0100

One random community member question inline ([MS])...

[MS] Maybe I am lost here. But in case of a spurious timeout, TCP uses the loss window, whereas after an idle timeout it would use the restart window, right? With IW10, this is one order of magnitude, and for standards-track, it is still factor 3. Does this not result in a somehow relevant risk of rto restart?

> several ones that we probably all visit everyday.

[MS] I would indeed be interested to see real data on the risk/benefit tradeoff of draft-hurtig-tcpm-rtorestart for HTTP traffic, before deciding whether to go down that road.

Michael

---

From: Michael Welzl <[email-masked]>
Date: Sun, 4 Nov 2012 22:26:03 +0100

On Nov 4, 2012, at 10:05 PM, Scharf, Michael (Michael) wrote:

> [MS] Maybe I am lost here. But in case of a spurious timeout, TCP  
> uses the loss window, whereas after an idle timeout it would use the  
> restart window, right? With IW10, this is one order of magnitude,  
> and for standards-track, it is still factor 3. Does this not result  
> in a somehow relevant risk of rto restart?

Yes. That diminishes the value of the problem case restriction  
described above.
(Note: I'm using this complicated phrase because all of this only  
applies if a spurious RTO is caused, the chance of which we still have  
to check some more. But so far, it doesn't seem to be very likely.

> [MS] I would indeed be interested to see real data on the risk/ 
> benefit tradeoff of draft-hurtig-tcpm-rtorestart for HTTP traffic,  
> before deciding whether to go down that road.

We have shown some data on the benefit side. Per has done a test,  
described on the list, to evaluate the risk, but yes, we can go  
further in risk evaluation.

Cheers,
Michael

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 4 Nov 2012 22:53:19 +0100

As far as I (chair hat off) understand the data presented in the last meeting, this is for thin stream gaming applications, right? I guess I understand the impact for that class of applications. But I am interested in the benefit/risk for typical Web traffic.

Technically, I also wonder about the difference to the "noRestart" approach. I think there was some valuable discussion on the pros and cons of that. Unfortunately, I don't recall the exact difference ;) Would it make sense to discuss this in the draft?

Thanks

Michael

________________________________________
Von: Michael Welzl [[email-masked]]
Gesendet: Sonntag, 4. November 2012 22:26
An: Scharf, Michael (Michael)
Cc: Yuchung Cheng; tcpm; [email-masked]
Betreff: Re: [tcpm] New Version Notification for draft-hurtig-tcpm-rtorestart-02.txt

Cheers,
Michael

---

From: Anna Brunstrom <[email-masked]>
Date: Sun, 04 Nov 2012 23:21:26 +0100

On 2012-11-04 21:58, Michael Welzl wrote:
>> Different scenarios aside, I understand that we are only talking about
>> few packets, so erring on one side or another may not be a big issue
>> in practice, but as a design point:
>>

I do not think this is true. As the application may send data at any 
time I do not think that there is any assumption on T_earliest. It could 
be close to zero or it could be almost an RTT away.

> (It's about firing the timer (RTO-T_earliest) seconds *after* the ACK

Maybe if the RTO is decreasing, the <= 0 case can happen. Having some 
check seems good.

BR,
Anna

---

From: Anna Brunstrom <[email-masked]>
Date: Mon, 05 Nov 2012 00:06:47 +0100

Hi

> several ones that we probably all visit everyday.

To continue off topic, just a thought on this scenario. Isn't the first 
packet in a new burst the packet to worry about the most with regards to 
a spurious retransmit? If the idle time is long I guess the likelihood 
of the path characteristics having changed could be larger than for 
packets sent in the same burst. As there is no data sent, the RTO will 
also not adapt to the changes.

Assuming the new burst is not to small (i.e > 4), increasing the RTO 
after idle may be beneficial as we want a fast retransmit anyway in case 
a packet is lost. Hence we do not run the same risk of delaying loss 
recovery as we do in the end of a burst.

BR,
Anna

---

From: Michael Welzl <[email-masked]>
Date: Mon, 5 Nov 2012 00:12:47 +0100

Hi,

On Nov 4, 2012, at 10:53 PM, Scharf, Michael (Michael) wrote:

> As far as I (chair hat off) understand the data presented in the  
> last meeting, this is for thin stream gaming applications, right? I  
> guess I understand the impact for that class of applications. But I  
> am interested in the benefit/risk for typical Web traffic.

Yes, it was. About web traffic, sure, I agree.

> Technically, I also wonder about the difference to the "noRestart"  
> approach. I think there was some valuable discussion on the pros and  
> cons of that. Unfortunately, I don't recall the exact difference ;)  
> Would it make sense to discuss this in the draft?

We have found that the mechanism doesn't work. It was a nice idea, but  
led to problems with thin streams (greater potential for spurious  
timeouts, with the benefit of a slightly simpler implementation). We  
can discuss why we didn't pick it, but personally I think it's not  
worth it (not helping the reader much). But no problem, can be done if  
you guys feel otherwise...

Cheers,
Michael