Subject: Re: [tcpm] Settling Time and RFC3517bis

From: Anthony Sabatini <[email-masked]>
Date: Thu, 2 Aug 2012 14:07:46 -0400

Joe -

Thank you for your insightful comments.

1) I am used to ITU terminology.  Settling time in ITU is used in optical, electrical and infrastructure standards.  It was in that last context that I used it.  I will change all references to "Reordering Time".

2) My point about fragmentation is that "Fragmentation Reassembly Timer" and "Reordering Time" are different names to describe the same issue.

3) STANDARD TCP can not recover in less than one RTT, SACK TCP as written certainly can and does currently recover in less that one RTT on at least long links.  With the changes I have been proposing nearly all errors can be recovered in less than one RTT.

4) Even terrestrial Google CA to Google NY is 45 ms minimum, add switching equipment without MPLS and that goes to 100 ms, with MPLS who knows.  One RTT is a huge amount of time, on a satellite or ocean link it is glacial.  TCPM has no business discussing forward error correction before exploring the mechanics of making the current protocol more efficient, lower latency and more reliable.

5) The 1.25 * Fragmentation Reassembly Timer was chosen as a minimum to include a bit more reordering time.  Regardless if a packet has not arrived in .25 * RTT can it even remotely be considered as "out of order" as opposed to lost?

6) Why is the IETF spending all of its time covering for greedy carriers who do not install enough fiber or switching equipment?  ALL of the changes being made in the congestion area result in a WORSE experience for the end user, the people we should really care about.

Tony

Sent from my ASUS Pad

Joe Touch <[email-masked]> wrote:

>
>
>On 8/1/2012 12:23 PM, Anthony Sabatini wrote:
>> Settling Time is the concept of how long we wait for an out of order packet.
>
>That's already called reordering time. Settling time is a term from 
>analog electronics/optics, and I don't see a good reason to import it here.
>
>> It has been an unnamed mainstay of IPv4 from the beginning,
>> masquerading under the name "Fragmentation Reassembly Timer" but at the
>> TCP level its physical existence has gone unacknowledged.
>
>There has been no masquerading, but there appears to be confusion ;-)
>
>Fragmentation reassembly is based on the time between the arrival of the 
>first fragment of a set (FWIW, I mean "first" as "first in time", not 
>"the packet with an offset of 0") and the last (again, in time, not the 
>fragment where MF=0), and is not affected at all by the order of the 
>arrival of the fragments.
>
>In TCP, the reordering is of segments, and is tolerated by the receive 
>window.
>
>> Settling Time is rather easy to calculate as it has a ceiling of
>> .25*RTT (any number over that and you are in retransmission
>> territory)
>
>TCP should never do anything RTT-based in less than a full RTT. It has 
>no business retransmitting in less than one RTT.
>
>> and a floor of Fragmentation Reassembly Time * 1.25 (need
>> to leave enough time to process reassembled packet).
>
>The additional time is whatever time is needed to:
>	- complete reassembly
>	- get the resulting IP datagram to TCP for processing
>
>This should be on the order of microseconds, vs. milliseconds for most 
>RTTs; 25% of the reassembly timer is thousands of time larger.
>
>However, again, the issue for TCP is the segment reordering time, which 
>does not necessarily have anything to do with the reassembly time. E.g., 
>even atomic (non-fragmented) segments can arrive out of order.
>
>> Obviously the
>> lower number is better and there are probably thousands of emails on
>> how  to set the Fragmentation Reassembly Timer properly.
>>
>> The point here is we are dancing around issues by saying we wait for
>> multiple ACKs before a SACK block is processed, you are badly
>> simulating reassembly delay.  Better to act on the first appearance
>> of a hole in the sequence but delay the actual call for
>> retransmission by Settling Time to see if the world has changed
>> before you issue the retransmission.  Also the time represented by
>> multiple ACKs is extremely variable,  short segments and random times
>> for WOW, long segments and continuous for file transfer.
>
>As the use of multiple concurrent paths becomes more common, 
>out-of-order segment receipt becomes the norm. Jumping the gun on 
>retransmitting holes could turn a mild reordering situation into a 
>congested system quickly.
>
>I see no reason to act on holes on timescales shorter than 1-2 RTTs.
>
>Joe
>
>
>> Note for the record a DELAY TIMER is not in and of itself an event,
>> it is delaying and existing event, therefore its expiration does not
>> have any implications of its own, the event already  happened. >
>> Tony
>>
>> Sent from my ASUS Pad
>> _______________________________________________
>> tcpm mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2012 11:32:55 -0700

Hi, Tony,

On 8/2/2012 11:07 AM, Anthony Sabatini wrote:
> 1) I am used to ITU terminology. Settling time in ITU is used in
> optical, electrical and infrastructure standards. It was in that last
> context that I used it. I will change all references to "Reordering Time".

AOK.

> 2) My point about fragmentation is that "Fragmentation Reassembly
> Timer" and "Reordering Time" are different names to describe the same issue.

But they are not. Fragmentation reassembly time is the time of the 
spread-out of a set of fragments of a single IP packet -- it does not 
matter at all whether the fragments are reordered or not. There is 
neither a performance nor a time delay difference when fragments come 
out of order vs. in-order. The only impact is when fragments of 
different packets are *interleaved* - whether they are reordered or not.

Reordering time would affect only things that care about order. TCP 
doesn't care strictly about reordering of the incoming segments, but as 
you note there is a performance impact of things coming out of order at 
the TCP level.

>> 3) STANDARD TCP can not recover in less than one RTT, SACK TCP as
> written certainly can and does currently recover in less that one RTT on
> at least long links. With the changes I have been proposing nearly all
> errors can be recovered in less than one RTT.

Please explain. In less than 1 RTT the source cannot know what packets 
have arrived at the sender, so it cannot know what to retransmit. In 
less than 1 RTT, the sender has no new information, and cannot do much 
of anything.

The *receiver* can react differently, which can impact the sender later, 
but the loop cannot be closed in less than 1 RTT.

> 4) Even terrestrial Google CA to Google NY is 45 ms minimum, add
> switching equipment without MPLS and that goes to 100 ms, with MPLS who
> knows. One RTT is a huge amount of time, on a satellite or ocean link it
> is glacial. TCPM has no business discussing forward error correction
> before exploring the mechanics of making the current protocol more
> efficient, lower latency and more reliable.

TCP does not use FEC, but many links - especially satellite and oceanic 
- do. That's at the link layer. The tradeoff between ARQ-based error 
correction and FEC error correction is a known issue, and TCP is not 
exempt from it.

> 5) The 1.25 * Fragmentation Reassembly Timer was chosen as a minimum
> to include a bit more reordering time. Regardless if a packet has not
> arrived in .25 * RTT can it even remotely be considered as "out of
> order" as opposed to lost?

Reordering typically has no relationship to the RTT, except that 
reordering is always less than the max RTT.

Neither reordering nor the RTT is related at all to the fragmentation 
reassembly timer, unless a stream includes some fragments and some 
non-fragment packets. In that case, you would need to wait at least 1 * 
reassembly time to account for differences between reassembled packets 
and non-reassembled ones.

> 6) Why is the IETF spending all of its time covering for greedy
> carriers who do not install enough fiber or switching equipment? ALL of
> the changes being made in the congestion area result in a WORSE
> experience for the end user, the people we should really care about.

The IETF is focused on making sure things work in the most possible 
cases - especially in the cases of IP and TCP. Optimization is nice, but 
can *always* be sacrificed when it compromises generality.

The purpose of congestion control is to improve everyone's experience as 
a group by making some user's experience worse, by definition. The IETF 
cares about the end users as a group, but not individual greedy end 
users whose behavior impacts others.

Joe

---

From: Rick Jones <[email-masked]>
Date: Thu, 02 Aug 2012 11:48:58 -0700

Joe -

Just so as I am sure I am thinking the same thing as you when you write 
"RTT" you mean that from the time a segment is first transmitted rather 
than from the time when some indication of a hole arrives at the sender yes?

rick jones

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Aug 2012 11:58:05 -0700

> than from the time when some indication of a hole arrives at the sender

RTT = round trip time.

It's the time delay between when a message is sent from the source to 
when any confirmation of receipt of that message could return back to 
that source.

Holes are determined by gaps of sequences of packets. They ALWAYS 
involve delays larger than 1 RTT because within 1 RTT either a packet 
arrives or it doesn't; when it doesn't, there's no indication of a gap 
until the *next* packet arrives. Packets that are reordered that cause 
that same effect are reordered because at least one packet (the earlier 
one) is delayed longer, which means the second packet arrives but the 
first didn't. The time is thus still longer than 1 RTT.

Joe

---

From: Mark Allman <[email-masked]>
Date: Fri, 03 Aug 2012 11:12:31 -0400

This is all pretty mis-guided.  Let me say just a few quick things.

> 3) STANDARD TCP can not recover in less than one RTT, 

Correct.  Nor would we want it to.  If you retransmit in less than one
RTT from the time a segment is originally sourced then you will always
be retransmitting blind.  I.e., you'll have no indication as to whether
that segment needs to be retransmitted or not.

>    SACK TCP as written certainly can and does currently recover in
>    less that one RTT on at least long links.  

Incorrect.  At least I am aware of no TCP SACK-based loss recovery
variant that retransmits a segment less than one RTT after it was
originally sent.  You reference 3517bis in the subject.  The algorithm
specified in RFC 3517 and 3517bis *does not* retransmit a segment in
less than one RTT.

>    With the changes I have been proposing nearly all errors can be
>    recovered in less than one RTT.

The only way to recover in less than one RTT is to retransmit blind.
I.e., retransmit segments without any indication as to whether they have
arrived at the receiver or not.  One could write such an algorithm.
And, probably show it fixes lost segments sooner.  But, the question
would immediately become "at what cost?".  I.e., how many needless
retransmits are being piled into the network to accomplish this task?
And, the answer nearly has to be proportional to the benefit you obtain.
I.e., I bet that as you speculatively retransmit more segments you will
also spurious retransmit more segments.  

And, this is needless retransmits that we know are being piled into a
congested network and so are using scarce network resources.  So, all
around a bad idea.

allman

---

From: Ethan Blanton <[email-masked]>
Date: Thu, 2 Aug 2012 17:57:11 -0400

> Timer" and "Reordering Time" are different names to describe the same

They are not.  This seems to reflect a fundamental misunderstanding of
the respective roles of IP and TCP.  There are two primary differences
between IP fragmentation and TCP segmentation for the purposes of this
specific discussion, namely:

 * IP fragmentation and delivery are best-effort, while TCP segments
   are reliable (at a higher layer in the stack than the IP datagrams
   they ride in)

 * IP fragments of a single datagram are always generated nearly
   simultaneously (within the performance limits of the IP stack and
   the local transport hardware), while TCP segments may be generated
   and emitted arbitrarily far apart in time

Each of these points has an important implication.  The first means
that, if you do not receive the fragments of an IP segment quite
rapidly (and a single RTT is a reasonable upper bound, though
certainly one could envision cases where it may take longer), you are
not going to receive them at all.  The IP datagram can be discarded
blindly, and the sending machine will retransmit its contents via
whatever appropriate means if its upper-layer protocols direct so.
The TCP segment, on the other hand, must have its suspected loss
somehow transmitted back to the sending TCP so that it can be
retransmitted.

The second point means that a TCP segment can arrive at any time and
may arrive a very long time (days, months, or years, in the extreme)
after the previous segment.  It is true that, under normal conditions,
a TCP segment should not arrive long after a *succeeding* segment, but
it can happen, and as TCP is a reliable transport, this should be
accounted for.  Since TCP cannot react to loss in less than one full
RTT anyway (another misconception in your logic, more below), the
receiving TCP might as well wait a full RTT or more for progress. If
the segment is truly lost, the connection will only begin to
(substantially) suffer performance-wise if the sending 3517-compliant
TCP runs out of new data to send.  Until that time, whether the
segment is retransmitted in the meantime or not, the TCP will proceed
essentially unaltered.  (The precise timing of the retransmission will
affect segment clocking somewhat, but in the long run, in the case of
a true loss, this should be a moderate effect.)

> written certainly can and does currently recover in less that one RTT
> on at least long links.  With the changes I have been proposing nearly
> all errors can be recovered in less than one RTT.

This is entirely untrue.  It takes one RTT at minimum for a sending
TCP to receive indication from the receiver that a segment was lost.
The sending TCP must send segments following the lost segment that the
receiving TCP SACKs.  Only when those SACK blocks arrive at the
sending TCP does it know that the segment was possibly lost.

> carriers who do not install enough fiber or switching equipment?  ALL
> of the changes being made in the congestion area result in a WORSE

I recommend that you read the following paper:

    Congestion Avoidance and Control.  Van Jacobson.  Proceedings of
    the SIGCOMM 1988 Symposium on Communications Architectures and
    Protocols.  pp. 314--329.

A version of this paper was published in the 1988 Computer
Communications Review, and a version co-authored by Michael Karels
also appeared in 1988 with additional information.

The takeaway from these papers is that, not only is congestion control
a good idea on paper, but its absence has been documented to cause
congestion collapse and complete non-responsiveness of the network to
useful traffic.  You are correct that provisioning can help this
circumstance *in some cases*, but it cannot, by definition, help in
all cases -- there can always exist a situation where the network has
more packets in flight than it can support, no matter how widely
provisioned, if the provisioning is finite.  A lesson learned from the
congestion collapses of the 80s is that this condition of over
over-subscription of valuable data does not even have to persist -- it
only has to last long enough for the retransmissions of that valuable
data to overwhelm the network.

Ethan