Subject: Re: [tcpm] Timestamps rejuvenated was: A new ID on TCP option space extension.

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 2 Apr 2012 20:40:55 +0000

Hi Andre,

nice to hear from you!

See inline

Richard Scheffenegger

>> The only thing that we really should verify is, if there are middleboxes 
>> that will actually terminate a TCP session, when all of a sudden the 
>> TS option appears... I hope this won't be the case, but it needs to be verified.
> 
> It's really difficult to change the behavior of a system that was very
> clearly spelled out 20 years ago.  It's too ingrained everywhere, too
> many (rightful) assumptions have been made and any change will just
> open a new can of worms.

I think Joe rightfully said some time back, that we shouldn't limit ourselves by trying to work around implementation bugs in some stacks. The issue about the TCP SYN option space getting really scarce really fast now won't go away when we don't try to address this.

>> For a 1323bis, I think we should limit this late negotiation to 
>> timestamps for now; other options are really beneficial from 
>> the start (SACK, MPTCP,MSS), while stil others would require 
>> some definition of new semantics if late negotiation would 
>> be used (WS).
> 
> I strongly recommend against changing RFC1323 timestamps in any way.
> Make it new options but leave the current behavior as it is.

Perhaps we need a new TCP option number and new semantics to address late negotiation. After all, the experiments conducted during MPTCP showed, that a majority of middleboxes won't fuss with unknown options once they are allowed through.  Perhaps we can get away by defining additional semantics to the current TCP TS option. 

Right now, I think it is too early to make a facts-based decision either way.

Therefore, I want to run some small-scale tests with TBIT against a bunch of popular OS and see how they react when TS show up in mid-connection...

>> * Window reduction during recovery bug - Matt
> 
> ?

>From my casual understanding (more knowledgeable people please correct my ignorance):

When the receiver's buffer cannot be delivered up to the application, and at the same instance you have a loss; eventually, the receiver buffer will be announced as ("0"<<WS) [which is still 0], thus potentially preventing the sender from retransmitting the lost packet (because now it is outside the receive window), even though the actual receive window on the receiver side is still  ("1"<<WS - 1) open...

Basically a rounding-error due to the WS scale shifting.

This suggested fix, as I understood it from Matt, is to either keep track of the largest receive window ever seen on the session, and use that for retransmissions, while the currently signaled receive window is used for newly injected packets; or use (Recv.Window + 1) << Recv.WSopt during retransmissions.

Currently, such a stall may lead to a terminated session (ie. if the receiver app waits for a certain amount of data, that is blocked due to the retransmission never coming through because of the zero window advertised).

> > * Enabling TS (PAWS) midstream - David, Matt, and others
> 
> Disagreed due to interop issues.  The current behavior is engraved in
> stone in 20 years worth of implementations of any kind.  We have enough
> trouble keeping TCP fully interoperable as it is.  Don't add yet another
> fuzzy failure case.  Even worse that connections may suddenly fail mid-
> transfer without anyone knowing why (because firefall gets tripped up).

See comments above. As long as a middlebox only stops forwarding packets, but doesn't remove state or actively terminates sessions, a graceful retry may work so that the session is not completely stalled (ie. if a RTO occurs while trying to do delayed TS negotiation, stop trying to use TS on this session in the future; so we may need to define the state machine for delayed negotiation in some detail to cover (most) failure modes...

> And don't forget that timestamps add some entropy to the already limited
> 32^2 sequence space.

Must have been 2^32. Also note that may people don't deploy TS because of the perceived reduction in net bandwidth (~0,82% "slower" when doing dumb bulk transfer tests over "nice" links). Using "delayed" TS could happen quite far into "bulk" transfers, so that more sessions can make use of Timestamps once they need it, but don't have to carry the burden as long as they don't require TS. All the small object transfers up to 1 GB may be eligible to run without TS, if the server doesn't want to use TS heuristics for fast TCP Port recycling [but strictly speaking, that part is beyond the specs anyway].
 
> > * References / interaction with SACK (TS check does not look if SACK is
> > present - to cover certain cased of lost ACKs that inflate RTO) [?]
> 
> The presence of SACK removes ambiguity from timestamps and also from
> the somewhat brittle window update logic (see my long post from some
> two years ago).

Well, I looked at one implementation of RTTM, and there was no check on the presence of SACKs there. Thus an ACK (containing a SACK block) after a lost ACK during forward loss will probably still inflate RTTvar and thus RTO; this is probably more of an issue with Karn's algorithm (as that is being referenced in RFC1323).

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Apr 2012 14:05:30 -0700

Hi, all,

Three observations:

1. Late negotiation is very difficult. TCP's state diagram only 
understands one ESTABLISHED state. Before that state, TCP's state 
mechanism ensures an ordering so that the SYN options can be coordinated.

After that, while ESTABLISHED, there's no way to ensure that options are 
transmitted reliably and in-order.

We can easily define a single SYN option that, in effect, says "if you 
agree to this option, you're no longer TCP". Options can have sequence 
numbers of their own, and be retransmitted asynchronously and 
acknowledged independently of the data.

That's a mess, IMO, and should be avoided. Yes, one example of that mess 
is called MPTCP.

2. Alternately, we can consolidate some of the options, as was done with 
TCP-CT. The current header requires:
    o  SACK permitted (2 bytes) [RFC2018][RFC3517]
    o  Timestamps (10 bytes) [RFC1323]
    o  Window scale (3 bytes) [RFC1323]

We can knock that 15 (typically allocated as 16) bytes down to half that 
- 7 (allocated as 8) easily:

	1 byte kind
	1 byte length
	1 byte scale value
	4 bytes TSval

The SYN-ACK would send TSSecr in the TSval location, and the sender 
would cache TSval (matched with the ISN) to recover it when receiving 
the SYN-ACK.

That helps reduce the current use of the SYN space by half, and similar 
approaches that aggregate other typical option sets to reduce the number 
of KIND/LENGTH bytes might be useful, but overall this plays in the 
margins at best.

3. What's really needed is a way to get beyond the 40-byte limit - 
primarily in SYNs, but even in subsequent segments -- and there's 
provably no way to do that which is backward compatible (for SYNs) and 
NAT-transparent (for data segments).

Do we really need to go around this block again?

Joe

---

From: Rick Jones <[email-masked]>
Date: Mon, 02 Apr 2012 14:12:06 -0700

> Must have been 2^32. Also note that may people don't deploy TS
> because of the perceived reduction in net bandwidth (~0,82% "slower"
> when doing dumb bulk transfer tests over "nice" links). Using
> "delayed" TS could happen quite far into "bulk" transfers, so that
> more sessions can make use of Timestamps once they need it, but don't
> have to carry the burden as long as they don't require TS. All the
> small object transfers up to 1 GB may be eligible to run without TS,
> if the server doesn't want to use TS heuristics for fast TCP Port
> recycling [but strictly speaking, that part is beyond the specs
> anyway].

"Far" is probably relative.  4GB is not all that "far" in time at 10 GbE 
speeds.  And at 10 GbE speeds, a human being probably won't perceive the 
difference that ~0.82% makes).  At << 10 GbE speeds, disabling 
timestamps to get a measly ~0.82% of throughput seems silly.  Even with 
acknowledgement that adding little to little results in a great pile...

To a human - at least this human - the smallest difference in transfer 
time that "matters" is one minute.

To save me one minute, I need a 100 minute transfer. (Handwaving math 
since it isn't 1%)  4GB-1 of data in 100 minutes is ~5.7 Mbit/s.  Any 
slower than that and I'm talking about a transfer time where, to a human 
at least ~0.82% is epsilon.  Any faster than that, and I save 
progressively less than a minute.

If I'm talking about a transfer that takes more than 100 minutes at that 
~5.7Mbit/s, then I'm talking about a transfer where timestamps will 
end-up being added-on, which makes my savings go to epsilon the longer 
the transfer.  And if it was an automated transfer, perhaps a video 
stream? we hope it wasn't cutting things so fine that the loss of that 
~0.82% 4GB in isn't going to screw with its timing.  Probably right when 
the movie got to the good part.

If people are desperate enough for ~0.82% more throughput, we'll "never" 
get them to light-up IPv6...   As ironic as it sounds, I'd think we 
would want timestamps on universally to make the hit from the IPv6 
header look less.  Unless, perhaps, one day, we can get the IEEE to 
accept larger frame sizes...

rick jones

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Mon, 2 Apr 2012 15:06:39 -0700

> Do we really need to go around this block again?

Answer : Yes, please see the motivation section of the document.

At least the reason for writing this document is to basically re-visit
this whole TCP option thing. List all the methods and see if there is
something that can be done. May be we would end up with (something -
caveats), assuming there is a consensus. Nothing is perfect, PMTU wasn't
perfect to begin with,  did we say that stop doing PMTU since there is
ICMP blackholing?  RFC 4281 was written to circumvent the issues... For
example TCP-AO would not work with NAT ALG's today, but didn't we write
an RFC? I am sure we can keep getting some "imperfect examples" if one
looks around.  Before writing this document I did chat to the chairs and
other folks and there seemed to a general interest in reviving this TCP
option debate. 

To me, the biggest hurdle is the middleboxes (NAT and firewall), that is
where we need some data to be collected and may be some lines that need
to be drawn. I would like to note that the draft mentions a paper
(Michio Honda et al) titled "is it still possible to extend TCP"? which
provides the list of experiments conducted to observe middlebox
behaviors.

I would think it would be prudent to continue the discussion in the
lines of "can we do something with the timestamp option" ? (which is
where the current thread started) can we compress options (TCP option
cookies OR have some different encoding/grouping etc.,?, you cited
TCP-CT or sending multiple duplicate segments?, or "happy-eyeballs" in
the lines done for v6 etc.,  Again, I am sure no solution wouldn't
satisfy all the corner cases, but you could always experiment "a chosen
one" and eventually the middleboxes would change. IPv6 was not supported
by the middleboxes a while ago, but no I don't see any middlebox
dropping a IPv6 packet today!

-Anantha

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Apr 2012 15:20:06 -0700

> Answer : Yes, please see the motivation section of the document.

I agree with your claim that there is a need.

I disagree with the claim that there's a solution. Although no solutions 
are perfect, there are some that are simply not possible - this is one 
of them.

The only possible solution is to extend the option for segments after 
the SYN using an option in the SYN
	1. at best that helps non-SYN segments

	2. it also has the easy potential to be
	corrupted by middleboxes

		corruption is both possible and currently likely

		corruption detection is impossible without adding
		(yet another) checksum, which slows things down
		at best

It's provable that there is no SYN option extension that is backward 
compatible. Non-backward compatible solutions are as trivial as the 
above, and as (not) useful.

Option compression reduces the current use of option space, but does 
nothing to extend it beyond 40.

I see no point in reviving this. It'd be better to nail it shut and call 
it dead.

Joe

PS- Regarding other imperfect solutions:
> .... For
> an RFC?

FWIW, as noted during the design, a NAT ALG is an attack that TCP-AO is 
intended to detect.

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Mon, 2 Apr 2012 16:07:21 -0700

Joe,

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Monday, April 02, 2012 3:20 PM
> To: Anantha Ramaiah (ananth)
> Cc: Scheffenegger, Richard; Andre Oppermann; David Borman;
> [email-masked]; [email-masked]
> Subject: Re: [tcpm] Timestamps rejuvenated was: A new ID on TCP option
> space extension.
> 
> 
> 
> I disagree with the claim that there's a solution. Although no
> solutions are perfect, there are some that are simply not possible -
> this is one of them.

"Simply not possible" - I am afraid that isn't the consensus. This is
one of the reasons for reviving the draft. We have not discussed the
other methods for instance, which were not debated (atleast I did not
see that in the list) For example you cited in one of your earlier post
that "after SYN it is not an issue, only getting it in SYN is
problematic". Assuming you really meant what you said, it means that if
we can squeeze the ones that needs to be negotiated in SYN and rest
after the 3WHS then is is ok. Then schemes like "sending multiple
SYN's", option encoding to save space etc., would come into play. In
your earlier post you also cited that" it is ok if someone wants to
write a draft for extending TCP option space after SYN's" So, this is
again not complete but should serve as a intermediate step. [I guess you
are saying the same thing below] 

> 		(yet another) checksum, which slows things down

Well, anything you do that alters the meaning of the TCP DO field is
susceptible for middlebox issues. The current draft gives examples and
classifies it based on the middlebox property (M1 to M6)

> above, and as (not) useful.

Please explain. If you read my draft I do list proposals that are
"backward compatible" w.r.t to end hosts. Middleboxes does prove to be
an issue, but I believe I have listed proposals where which probably has
better chances with middle boxes but maybe less efficient.

> I see no point in reviving this. It'd be better to nail it shut and
> call it dead.

Well, I disagree here since we have not debated every solution  and
things like trying to save some option space in SYN (eg timestamp
discussion) *does* count as "step in the right direction" that help the
TCP option space conservation. Another point which I wish to repeat is :
when the discussions happened on this topic earlier, there was not a
stronger use case for TCP option space, things have changed now
(changing more as there are new TCP options evolving) and that may
encourage different opinions on this topic. This is one of the other
motivating factors for writing the draft. Of course I can see that your
mileage varies here.

> > example TCP-AO would not work with NAT ALG's today, but didn't we
> > write an RFC?
> 

Ah, you call it a "design choice", but the point is that it does not
interoperate with a property of the middlebox. Period. We made a
decision to stick with that. I am not questioning the decision here, as
the use cases of TCP-AO may not be susceptible to such an issue. This is
what I call "drawing the line", I am afraid we haven't had that sort of
discussion given the fresh use cases, so it would be a bad idea to pull
down the curtains on this one, it would be pre-mature, IMO. Ok, I'll
happy to hear thoughts from other participants, if the consensus is to
shutdown this investigation (let alone document the issues which acts as
a reference point for future), needless to mention, then there is no
point wasting any energy in the draft.  

-Anantha

---

From: Joe Touch <[email-masked]>
Date: Mon, 02 Apr 2012 16:24:04 -0700

Amanth,

Cutting things down a bit...

On 4/2/2012 4:07 PM, Anantha Ramaiah (ananth) wrote:
...
> "Simply not possible" - I am afraid that isn't the consensus.

Consensus doesn't make it happen. A solution does, and no viable 
solution has been proposed.

...
> For example you cited in one of your earlier post
> after the 3WHS then is is ok.

Yes, but then any rewriting middlebox that doesn't understand the option 
will ignore it and the options will move around and be intermingled with 
the data. That's not a viable alternative.

> Then schemes like "sending multiple
> SYN's", option encoding to save space etc., would come into play.

Encoding to save space saves space; it doesn't create more space.

Sending multiple SYNs isn't a solution either, as it would interact 
badly with NATs.

 > In

I also point out that this is a meaningless step for two reasons:

1) the key place where space is limited is the SYNs

2) such an option won't survive anything that rewrites segments

> classifies it based on the middlebox property (M1 to M6)

We can classify it, but we can't get around it.

> "backward compatible" w.r.t to end hosts.

TCP is allowed - and supposed - to ignore options it doesn't understand.

An option in the SYN creates more option space, either by redefining the 
DO field (e.g., to have a scale beyond x4), or by specifically 
interpreting the SYN data as option.

An endpoint that doesn't understand these legitimately ignores the 
option, and then interprets the extended space as data.

Sending multiple SYNs has even more problems - how do you know how long 
to wait to get all the ones you need? What happens when you get a set of 
SYNs that conflict? And what do NATs do with all these SYNs?

> Middleboxes does prove to be

Your argument - and all its details - recurs in the IETF every 5-6 
years. The counterarguments that will kill this are:

	- the SYN is where space is really needed

	- saving space isn't sufficient; new space needs to be created

	- there is no backwards-compatible way to solve SYN extension

You can continue to try all possible combinations and hope there's a 
hole in that argument, but if that's your approach here, maybe you ought 
to let the WG know... that's not the same as a consensus solution.

> interoperate with a property of the middlebox. Period.

It does with the experimental extension I wrote. Period.

The only reason that's not in TCP-AO is the WG wasn't sure whether it 
was needed.

This was a design choice. It's not a protocol limitation.

Joe

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Mon, 2 Apr 2012 18:24:31 -0700

<snip>

> TCP is allowed - and supposed - to ignore options it doesn't

That's obvious. I believe I have listed the shortcomings and analyzed
multiple solutions w.r.t to each property in the draft.

> 
> An option in the SYN creates more option space, either by redefining
> the DO field (e.g., to have a scale beyond x4), or by specifically
> to wait to get all the ones you need? What happens when you get a set
> of SYNs that conflict? And what do NATs do with all these SYNs?

How long to wait, what happens if SYN's get dropped, arrive in different
order etc., has been discussed. The "happy eyeballs" scheme has some of
the issues. Not sure about the concern on NAT's since our NAT
implementations do handle duplicate SYN's.  

<snip>

> > discussion) *does* count as "step in the right direction" that help
> > the TCP option space conservation. Another point which I wish to
> repeat is :
> > motivating factors for writing the draft. Of course I can see that
> > your mileage varies here.
> 
> hole in that argument, but if that's your approach here, maybe you
> ought to let the WG know... that's not the same as a consensus
> solution.

Well, the abstract already covers the intentions of the draft, IMO.

> >> FWIW, as noted during the design, a NAT ALG is an attack that
TCP-AO
> > is

We are talking about NAT-ALG,  which looks at the TCP payload. In your
experimental extension, it simply gives provision to the basic NAT
functionality by not including the stuff which NAT munges around (IP
address and port) of the TCP/IP headers. (even that has practical
deployment issues since you never know where is a NAT unless you use
some neighbor discovery protocol)  It has no provision to circumvent the
NAT-ALG issues where you need to use the TCP DO offset to get to the
application payload and then fix the appropriate contents like FTP port
information etc., Repeat : this is the biggest headache for all TCP
option space proposals. Both TCP-AO and TCP option proposals *are*
susceptible to this issue for the sake of comparison. [ I may agree it
is not a fair comparison but in theory the issue is there] 

-Anantha

---

From: Mark Allman <[email-masked]>
Date: Mon, 02 Apr 2012 22:06:01 -0400

> Three observations:

I like powers of two (although, I had to stretch to get there!):

(0) Yes, there is only one ESTABLISHED, but WRT the timestamp option
    that is not germane.  Because of how timestamps work we can turn
    them on mid-stream pretty easily.  Borman sketched the scheme.  Turn
    them on in a data-carrying packet and watch for the ACK of that
    sequence number.  If it comes back with the expected timestamp we
    just verified the other end received it and echoed back the right
    thing and so is likely to play ball.  For some other option that
    doesn't have the same sort of echoing operation then I agree that
    reliability might be an issue.  But, for timestamps I think the path
    is pretty clear.

(1) Further, everything doesn't have to be negotiated in the SYN.  We
    have precedent for this.  E.g., DSACK is not negotiated in the SYN.
    If a host doesn't support DSACK then a DSACK SACK block will look
    pretty weird and nonsensical.  But, we evolved SACK without an
    explicit negotiation.  It seems at least plausible that we can
    change this one (**one***) thing about timestamps (when to start
    using them) in a similar fashion (*plausible*; see next point).

(2) We should know the failure cases.  As someone said, MPTCP stuff has
    shown that middleboxes don't much care about options.  Our work that
    I pointed to yesterday---which admittedly is a little old---shows
    that a TS in the middle of the stream without a TS in the SYN works
    just fine in the vast, vast majority of the cases.  If we go down
    this road someone should do the experiments to understand how often
    such a change will cause issues (connection crash, middlebox
    interaction, TS ignored, etc.).  But, I don't see any evidence that
    says we should take this off the table.

(4) More generally, I think if we decide we need more option space in
    the SYN it is fine to fire two SYNs (i.e., start two connections).
    I mean, its ugly and its a hack and it isn't ideal.  But, that
    pretty much describes the entirety of the world, right?  It would
    work.  Yeah, we'd have to decide some things about how long to wait
    and how to rip down the unneeded state and whatever.  But, it isn't
    like this is somehow science fiction.  So, we should stop saying
    this is just somehow impossible.

(5) You might not call it "TCP".  I might.  But, really, I don't much
    care.  I mean, its just arguing My Grandfather's Axe and so we might
    as well just get ourselves a couple of beers, belly up to the bar
    and dig into the finer points of axe heads, eh?  At the end of the
    day it doesn't matter if it's my Grandfather's axe or not as long as
    it chops the wood nobody gives a shit, right?

(6) I also agree with you that using the current space smarter is
    cleaner than trying to create new space.  But, also this has its
    limits.  I think timestamps in every packet on the *off chance* [*]
    that you'll need PAWS is just damn wasteful of the constrained
    resource.  We should be smarter.

    [*] Off chance: Sending at least 4GB **and** sending fast enough to
        cycle the sequence space fast enough for there to exist
        confusion about whether some sequence number is new or old.

(7) TCP timestamps as motivation for IPv6 ... man is IPv6 in some
    serious trouble! :-)

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 3 Apr 2012 08:34:37 +0000

Hi Joe,

I think you confused me around what your point really is.

You say, during a connection, option space is not limited, there is no problem.

You mention, there is no way of reliably signaling options during a connection, so one cann't use that to move some options from a SYN down to later in the connection.

You appear to be saying, saving space in the SYN option space is not a proper solution.

Finally, you assess that there is no foolproof method of extending the SYN option space at all...

This leaves me to speculate, that what you are saying is that TCP is frozen and we should tear down TCPM.

This sounds very depressing indeed.

However, if we can give precedent as to how to properly negotiate certain options later during a connection (and IMHO, TS are an ideal candidate here as the encoding doesn't need to change; some stacks may already implicitly support this even due to "incomplete" implementation of all the checks), people in need of more negotiated TCP options may have a framework they could use without inventing the wheel over and over again. And true, perhaps a separate state machine is needed to track late option negotiation - to make sure the session ends up in a defined state.

Also, bundling some typically used options together and saving SYN option space thereby should also not be ruled out. You actually give a good example, even though I would probably encode MSS in the high bits (nibble) of the WS byte for some typical MSS values, and allowing the current options to be present in addition, taking priority over the consolidated values (i.e. if your MSS is odd,  you can bundle the consolidated option with an explicit option).

To me at least, that approach using consolidation and late negotiation appears as the path forward, as it won't violate current specs. Only middleboxes too eager to lock down TCP in a frozen state may pose problems.

Richard Scheffenegger

> Sent: Dienstag, 03. April 2012 01:24
> Consensus doesn't make it happen. A solution does, and no viable solution
> has been proposed.
> 
> ...
> > For example you cited in one of your earlier post that "after SYN it
> > is not an issue, only getting it in SYN is problematic". Assuming you
> > really meant what you said, it means that if we can squeeze the ones
> > that needs to be negotiated in SYN and rest after the 3WHS then is is
> > ok.
> 
> will ignore it and the options will move around and be intermingled with the
> data. That's not a viable alternative.
> 
> Sending multiple SYNs isn't a solution either, as it would interact badly with
> NATs.
> 
>  > In
> > again not complete but should serve as a intermediate step. [I guess
> > you are saying the same thing below]
> 
> An option in the SYN creates more option space, either by redefining the DO
> field (e.g., to have a scale beyond x4), or by specifically interpreting the SYN
> data as option.
> 
> An endpoint that doesn't understand these legitimately ignores the option,
> and then interprets the extended space as data.
> 
> > an issue, but I believe I have listed proposals where which probably
> > has better chances with middle boxes but maybe less efficient.
> >
> > the TCP option space conservation. Another point which I wish to repeat is :
> Your argument - and all its details - recurs in the IETF every 5-6 years. The
> counterarguments that will kill this are:
> 
> You can continue to try all possible combinations and hope there's a hole in
> that argument, but if that's your approach here, maybe you ought to let the
> WG know... that's not the same as a consensus solution.
> 
> The only reason that's not in TCP-AO is the WG wasn't sure whether it was
> needed.
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 3 Apr 2012 08:04:18 -0700

> This leaves me to speculate, that what you are saying is that TCP is frozen and we should tear down TCPM.

Well, you could also interpret that as "we've reached a place in TCPM where there are some areas of TCP that cannot be expanded without compromising ubiquity, backward compatibility, etc.". That shouldn't be surprising.

This is TCP, and we're in TCPM, not UTM (a universal Turning Machine).

> However, if we can give precedent as to how to properly negotiate certain options later during a connection (and IMHO, TS are an ideal candidate here as the encoding doesn't need to change; some st…

We'd need to define a TWHS *inside* TCP to ensure that we handshake option negotiation after the initial TWHS. That's TCP on TCP. Yuk.

> Also, bundling some typically used options together and saving SYN option space thereby should also not be ruled out. You actually give a good example, even though I would probably encode MSS in th…

I didn't rule it out, I just said it had limited benefit. It won't result in more than 40 bytes of option space in a SYN.

FWIW, sending multiple SYNs will be incompatible with legacy endpoints unless every SYN includes the options needed for the legacy connection (since any SYN could get there first). What happens if the SYN options differ and you're talking to a legacy endpoint? How will you know which options are ackd, given the potential loss of SYN-ACKs? (the SYNs need to have the same ISNs or you risk having higher ISN SYNs reset the connection) Further, sending multiple SYNs kills the use of data (if you ACK the data, then the ISNs will need to increase, and then if they arrive in order they'll kill a legacy connection). Double-yuk.

> To me at least, that approach using consolidation and late negotiation appears as the path forward, as it won't violate current specs. Only middleboxes too eager to lock down TCP in a frozen state…

My definition of TCP, however, requires:

	- is compatible with legacy TCPs in all current ways, i.e.,
		- allows SYN data
		- allows any existing TCP option, space permitting
		- connects with legacy TCPs with the first SYN

If any of the above aren't true (at least), it isn't TCP anymore. 

Joe

---

From: Andre Oppermann <[email-masked]>
Date: Tue, 03 Apr 2012 18:24:56 +0200

On 02.04.2012 22:40, Scheffenegger, Richard wrote:
> Hi Andre,
>
> nice to hear from you!
>
> See inline
>
> Richard Scheffenegger
>
>>> The only thing that we really should verify is, if there are middleboxes that will actually
>>> terminate a TCP session, when all of a sudden the TS option appears... I hope this won't be
>>> the case, but it needs to be verified.
>>
>> It's really difficult to change the behavior of a system that was very clearly spelled out 20
>> years ago.  It's too ingrained everywhere, too many (rightful) assumptions have been made and
>> any change will just open a new can of worms.
>
> I think Joe rightfully said some time back, that we shouldn't limit ourselves by trying to work
> around implementation bugs in some stacks. The issue about the TCP SYN option space getting
> really scarce really fast now won't go away when we don't try to address this.

Unfortunately with this proposed change you end up with exactly
the opposite of what you're claiming here.  The timestamp option
is clearly defined as negotiated as SYN and ever after present
on all segments.  This has been implemented for 20 years.

Now you come around and want to change that in a way that probably
trips up enough compliant stacks and firewalls that it end with
a drama like the PMTU discovery saga on PPPoEoA DSL and filtered
ICMP needfrags.

Please don't.  The few bytes are not worth it.

>>> For a 1323bis, I think we should limit this late negotiation to timestamps for now; other
>>> options are really beneficial from the start (SACK, MPTCP,MSS), while stil others would
>>> require some definition of new semantics if late negotiation would be used (WS).
>>
>> I strongly recommend against changing RFC1323 timestamps in any way. Make it new options but
>> leave the current behavior as it is.
>
> Perhaps we need a new TCP option number and new semantics to address late negotiation. After all,
> the experiments conducted during MPTCP showed, that a majority of middleboxes won't fuss with
> unknown options once they are allowed through.  Perhaps we can get away by defining additional
> semantics to the current TCP TS option.

We even have trouble with the order of options on SYN segments.  What
makes you believe you can get away with anything?

> Therefore, I want to run some small-scale tests with TBIT against a bunch of popular OS and see
> how they react when TS show up in mid-connection...

Please do.  And then you test across a large deployed section of
middle-boxes doing NAT and firewalling and whatnot.

> When the receiver's buffer cannot be delivered up to the application, and at the same instance
> you have a loss; eventually, the receiver buffer will be announced as ("0"<<WS) [which is still
> 0], thus potentially preventing the sender from retransmitting the lost packet (because now it is
> outside the receive window), even though the actual receive window on the receiver side is still
> ("1"<<WS - 1) open...
>
> This suggested fix, as I understood it from Matt, is to either keep track of the largest receive
> window ever seen on the session, and use that for retransmissions, while the currently signaled
> receive window is used for newly injected packets; or use (Recv.Window + 1)<<  Recv.WSopt during
> retransmissions.
>
> Currently, such a stall may lead to a terminated session (ie. if the receiver app waits for a
> certain amount of data, that is blocked due to the retransmission never coming through because of
> the zero window advertised).

Agreed.

>> Disagreed due to interop issues.  The current behavior is engraved in stone in 20 years worth
>> of implementations of any kind.  We have enough trouble keeping TCP fully interoperable as it
>> is.  Don't add yet another fuzzy failure case.  Even worse that connections may suddenly fail
>> mid- transfer without anyone knowing why (because firefall gets tripped up).
>
> See comments above. As long as a middlebox only stops forwarding packets, but doesn't remove
> state or actively terminates sessions, a graceful retry may work so that the session is not
> completely stalled (ie. if a RTO occurs while trying to do delayed TS negotiation, stop trying to
> use TS on this session in the future; so we may need to define the state machine for delayed
> negotiation in some detail to cover (most) failure modes...

Too complex and brittle.  See comments above.

Finish up RFC1323bis to remove edge-cases and ambiguities from
RFC1323 and get it out of the door.  Then another experimental
RFC can be furbished exploring new behaviors.  There is no need
to bundle this together.  Actually it is harmful.

>> And don't forget that timestamps add some entropy to the already limited 32^2 sequence space.
>
> Must have been 2^32. Also note that may people don't deploy TS because of the perceived reduction
> in net bandwidth (~0,82% "slower" when doing dumb bulk transfer tests over "nice" links). Using
> "delayed" TS could happen quite far into "bulk" transfers, so that more sessions can make use of
> Timestamps once they need it, but don't have to carry the burden as long as they don't require
> TS. All the small object transfers up to 1 GB may be eligible to run without TS, if the server
> doesn't want to use TS heuristics for fast TCP Port recycling [but strictly speaking, that part
> is beyond the specs anyway].
>
>>> * References / interaction with SACK (TS check does not look if SACK is present - to cover
>>> certain cased of lost ACKs that inflate RTO) [?]
>>
>> The presence of SACK removes ambiguity from timestamps and also from the somewhat brittle
>> window update logic (see my long post from some two years ago).
>
> Well, I looked at one implementation of RTTM, and there was no check on the presence of SACKs
> there. Thus an ACK (containing a SACK block) after a lost ACK during forward loss will probably
> still inflate RTTvar and thus RTO; this is probably more of an issue with Karn's algorithm (as
> that is being referenced in RFC1323).

These things should be fleshed out in RFC1323bis.  Not new fancy
stuff of questionable value and deployability.

-- 
Andre

---

From: Andre Oppermann <[email-masked]>
Date: Tue, 03 Apr 2012 18:34:34 +0200

> Answer : Yes, please see the motivation section of the document.

Please decouple much needed progress on RFC1323bis to clarify
implementation issues and edge cases from the whole TCP options
space debate!  Different wars!

-- 
Andre

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Tue, 3 Apr 2012 10:01:41 -0700

> implementation issues and edge cases from the whole TCP options space
> debate!  Different wars!

Makes sense, these issues are not related.

-Anantha

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Apr 2012 10:32:06 -0700

On 4/2/2012 7:06 PM, Mark Allman wrote:
>      them on mid-stream pretty easily.

IMO, if timestamps are optional on a per-packet basis, this is fine. 
That's not changing "state" mid-stream, though.

>	... Borman sketched the scheme.  Turn
>      sequence number.

Hmm. That presumes:
	a) that midbox devices preserve sequence numbers in TCP headers
	(they don't necessarily when they rewrite the stream)

	b) the endpoints keep per-segment state (retain a list of
	sequence numbers with timestamps)

	c) TCP sends back ACKs with sequence numbers that match the
	packets sent (which they are NOT required to do at all).

These are all highly problematic, IMO.

...
> (1) Further, everything doesn't have to be negotiated in the SYN.
...

	Sure. You can then create a TWHS for anything later - on top
	of the ESTABLISHED state.

	This doesn't hold for options that are 'optional' at any time;
	for those, you don't need to negotiate when to enable them, and
	how to handle segments that come out of order that don't yet
	support them vs. later ones that might.

>      just fine in the vast, vast majority of the cases.
...

	Operational evidence that something works doesn't mean that it's
	OK; it means you haven't found an implementation that doesn't
	work. That's a dangerous way to extend TCP.

...

	If you intend this to work with legacy TCP, then the options
	you need to have for legacy connections need to be in BOTH SYNs.

	You also can't have data in either SYN, because if you do the
	sequence number will increase, the other SYN will be interpreted
	as out of window, and it will kill the connection.

	You also need to copy the ISNs (same reason as above).

	You also need to ensure that the old SYNs are out of the net
	before you start using the connection for data, or the old SYN
	will kill the connection.

	That's a partial list of the problems. So this interferes with
	legacy TCP way too much, IMO.

> (5) You might not call it "TCP".  I might. ...

	IMO, TCP options need to play nice with other TCP options.

	Any option that doesn't isn't TCP anymore. We saw a recent
	proposal for a protocol that uses TCP's protocol number
	and a few key fields, but isn't TCP.

	If that's where you're going, please take the work outside TCPM
	at least.

>      resource.  We should be smarter.

Timestamps are starting to be used for a lot of other things (time-wait 
issues, randomness in various seeds, etc.). Be careful before calling 
them optional all over the place.

> (7) TCP timestamps as motivation for IPv6 ... man is IPv6 in some

I never said this, but yes, and not just for that reason ;-)

Joe

---

From: Fernando Gont <[email-masked]>
Date: Tue, 03 Apr 2012 19:46:48 +0200

Hi, Joe,

A few comments...

>     (they don't necessarily when they rewrite the stream)

Does this matter? SEQ-rewrite is transparent to the endpoint (for
instance, that's why it can actually work). Am I missing something?

>     sequence numbers with timestamps)

Yes, this is what Borman said, IIRC.

>     packets sent (which they are NOT required to do at all).

Not necessarily. You simply stick the option to a sequence number, and
resend the option whenever that sequence number is resent -- and stop
sending the option when that SEQ has been ack'ed.

>     sequence number will increase, 

IIRC, some implementations can't handle data in the SYN, anyway. -- not
to mention that SYN-cookies don't support data in the SYN.

> the other SYN will be interpreted

No. It will be considered a stale segment, and an ACK will be sent. It's
*in window* SYNs that kill the connections.

Thanks,
-- 
Fernando Gont
SI6 Networks
e-mail: [email-masked]
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 3 Apr 2012 18:32:12 +0000

Agreed! 

>From what I have read in 1323bis, these are the things which need some polish:

Only a few nits and all the necessary boilerplate updates before a new version could be posted and discussed in the WG; (with or without the stuff mentioned below:)

The SACK interaction (ignoring TS whenever SACK is present during RTTM) is not yet described (even though presence of a SACK, except DSACK, is an indication of loss);

A few historic RFC references could potentially be removed (they are already in RFC1323). 

The Window Reduction Bug described may warrant a more specific wording for implementers than what is currently in 1323bis (reference to 1122...)

Updating some references ie 5681 instead 2581 for TCP congestion control.

The discussion around TS extensions really is something for a dedicated experimental draft later -> exclude in 1323bis, that needs to be proposed standard.

Richard Scheffenegger

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Andre Oppermann
> Sent: Dienstag, 03. April 2012 18:35
> Cc: David Borman; [email-masked]; [email-masked]; Joe Touch
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Andre Oppermann <[email-masked]>
Date: Tue, 03 Apr 2012 20:52:01 +0200

On 03.04.2012 19:32, Joe Touch wrote:
> Timestamps are starting to be used for a lot of other things (time-wait issues, randomness in
> various seeds, etc.). Be careful before calling them optional all over the place.

Full ACK.

-- 
Andre

---

From: Andre Oppermann <[email-masked]>
Date: Tue, 03 Apr 2012 21:07:35 +0200

>> various seeds, etc.). Be careful before calling them optional all over the place.

Bringing up another angle:

Every congestion control algorithm that wants to take delay changes
into consideration depends on RTTM for high precision and frequency
feedback.  That's why we should allow it to be updated in the case
of retransmits in the presence of SACK.

The RTO calculation doesn't need the high frequency measurements and
could do with once or a couple of time per RTT.

-- 
Andre

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Apr 2012 12:44:28 -0700

PS - why not compress timestamps?

Send only the low byte in both directions; treat the high bytes as 
either phantoms or sent only as needed / periodically (as we did with 
64-bit seqnos in TCP-AO).

That would save 6 of the 10 bytes for most of the packets (if not nearly 
all of them). That's most of the bang for the buck. Ultimately, if you 
compress whatever set of options you expect together as a single field, 
you'll knock off another 2 as well, so that TS maintenance costs only 2 
bytes per packet.

Joe

---

From: Andre Oppermann <[email-masked]>
Date: Tue, 03 Apr 2012 21:58:58 +0200

On 03.04.2012 20:32, Scheffenegger, Richard wrote:
> Only a few nits and all the necessary boilerplate updates before a new version could be posted
> and discussed in the WG; (with or without the stuff mentioned below:)
>
>
> The SACK interaction (ignoring TS whenever SACK is present during RTTM) is not yet described
> (even though presence of a SACK, except DSACK, is an indication of loss);
>
> The Window Reduction Bug described may warrant a more specific wording for implementers than what
> is currently in 1323bis (reference to 1122...)
>
> The discussion around TS extensions really is something for a dedicated experimental draft later
> ->  exclude in 1323bis, that needs to be proposed standard.

Full ACK to all of them.

-- 
Andre

> Richard Scheffenegger
>
>> -----Original Message----- From: [email-masked] [mailto:[email-masked]] On Behalf
>> Of Andre Oppermann Sent: Dienstag, 03. April 2012 18:35 To: Anantha Ramaiah (ananth) Cc: David
>> Borman; [email-masked]; [email-masked]; Joe Touch Subject: Re: [tcpm] Timestamps rejuvenated
>> was: A new ID on TCP option space extension.
>>
>> Please decouple much needed progress on RFC1323bis to clarify implementation issues and edge
>> cases from the whole TCP options space debate!  Different wars!
>>
>> -- Andre _______________________________________________ tcpm mailing list [email-masked]

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Tue, 3 Apr 2012 13:32:07 -0700

> Sent: Tuesday, April 03, 2012 12:44 PM
> To: Andre Oppermann
> Cc: [email-masked]; David Borman; [email-masked]; Anantha Ramaiah
> (ananth)

???
In one your earlier posts you said :-

<Begin Quote>

<End Quote>

I should be glad that at least you are now thinking in the right
direction :-) Since you won't be suggesting this if does not buy us
anything, after all we are talking about different ways to conserve TCP
option space!

-Anantha

> That would save 6 of the 10 bytes for most of the packets (if not
> nearly all of them). That's most of the bang for the buck. Ultimately,
> if you compress whatever set of options you expect together as a
single
> field, you'll knock off another 2 as well, so that TS maintenance
costs
> only 2 bytes per packet.
> >> various seeds, etc.). Be careful before calling them optional all
> over
> >> the place.
> >
> > Full ACK.
> >

---

From: Joe Touch <[email-masked]>
Date: Tue, 03 Apr 2012 13:34:44 -0700

> I see no point in reviving this. It'd be better to nail it shut and call

"This" means a way to *extend* the option space (the title of your doc, 
and it's focus).

There's nothing wrong with compressing it, but it won't solve the big 
picture - the 40-byte limit.

Joe

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Tue, 3 Apr 2012 13:41:35 -0700

> picture - the 40-byte limit.

"extension" can be implicit as well, i.e, if you make room for more
options to be packed, then you have indirectly extended it, you can pick
on the "English" if you want. (pl read sections 4.6 and 4.8 of the draft
which are relevant to the current discussion)

-Anantha

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 3 Apr 2012 20:51:46 +0000

Andre,

Actually, specific CC algos want even more fine-grained measurements, One-Way Delay Variation; LEDBAT has been finalized by the IETF (but within TCP, it currently can not get the necessary input signals properly).

Having an understanding on both ends about what the timestamps represent, would be very beneficial for these mechanisms! Current approach here is to have a limited set of probable timestamp clock rates, and trying to match the observed timestamp values into one of these bins. Or simply assuming the other end with run at a specific timestamp clock rate...

Finally, during reordering / loss, these mechanisms probably want a different semantic to obtain the most accurate OWDV measurements. That is something I intended to address with my timestamp-negotiation draft...

Richard Scheffenegger

> Sent: Dienstag, 03. April 2012 21:08
> To: Joe Touch
> Cc: David Borman; [email-masked]; Anantha Ramaiah (ananth);
> [email-masked]
> >> Timestamps are starting to be used for a lot of other things
> >> (time-wait issues, randomness in various seeds, etc.). Be careful before
> calling them optional all over the place.
> >
> > Full ACK.
> 
> Every congestion control algorithm that wants to take delay changes into
> consideration depends on RTTM for high precision and frequency feedback.
> That's why we should allow it to be updated in the case of retransmits in the
> presence of SACK.
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 3 Apr 2012 23:14:31 +0200

> Your argument - and all its details - recurs in the IETF 
> every 5-6 years. The counterarguments that will kill this are:
> 
> You can continue to try all possible combinations and hope 
> there's a hole in that argument, but if that's your approach 
> here, maybe you ought to let the WG know... that's not the 
> same as a consensus solution.

My observation is that we keep having this discussion on the TCPM list every 5-6 *months*.

If we invested only half of the energy of these threads into xml2rfc (or whatever other draft generation tool), we'd easily have some informational draft that at least explains the drawbacks and the costs of each "solution" or "non-solution" (whatever you prefer).

And while such a document might not be of any benefit to all of us that obviously know every past e-mail in the TCPM list archive, it might perhaps be useful at least for some naive PhD students (or, e. g., other IETF WGs) that continuously come up with fancy new SYN options...

Just a thought

Michael (obviously as individual)