Subject: Re: [quicwg/base-drafts] Required state for retaining unacked RETIRE_CONNECTION_ID frames is unbound (#3509)

From: Marten Seemann <[email-masked]>
Date: Wed, 01 Apr 2020 01:47:16 -0700

While increasing the limit will certainly help, this introduces a really unpleasant flakiness in the protocol. QUIC, so far, has the property that no amount or pattern of packet loss will lead to a protocol violation (it might lead to a connection timeout if all packets are blackholed, but that's a different kind of error). This means that when running QUIC in production, screening for (transport-level) errors will be very useful for finding bugs in implementations.

My fear regarding #3547 is that no matter how high we choose the limit, there will always be a loss pattern that will trigger a protocol violation.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607119264

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 01 Apr 2020 05:30:53 -0700

@marten-seemann 
> My fear regarding #3547 is that no matter how high we choose the limit, there will always be a loss pattern that will trigger a protocol violation.

Thank you for raising the concern, I think the recommended behavior when exceeding that limit is to stop sending RCID frames for some sequence numbers rather than triggering a protocol violation. That's actually what I had in mind hence called it "leaky."

The benefit of stopping retiring some sequence numbers is that the worst case becomes endpoints running out of CIDs - something that is expected to happen on ordinary connections too. Note also that in HTTP/3, reestablishing a connection in 0-RTT is always a viable option.

To summarize, I think that the concern can be resolved for HTTP/3 by changing the recommended behavior from resetting the connection to stop sending RCIF frames for some sequence numbers.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607220688

---

From: ianswett <[email-masked]>
Date: Wed, 01 Apr 2020 06:36:37 -0700

@kazuho I don't think not sending RCID frames is that easy.  Tracking something that needs to be sent consumes comparable state to tracking it in flight.  Or are you suggesting a different algorithm that indicates a RCID frame doesn't need to be sent?

If the peer has to wait for one Retire Prior To to take effect before it's increased again, that leaves only the peer migrating really quickly and lost ACKs as the unbounded case. As @kazuho said, I think it's more likely a well-behaved peer would run out of CIDs in this case than any other failure mode.  I previously suggested stopping sending out NCID frames when the RCID frame limit had been reached to force the issue, but that just changes who has to close the connection.

FWIW, we already have sanity checks where our implementation closes the connection when a datastructure becomes too large.  The limits are high enough that they're almost never hit unless there is a bug, but they've been helpful in finding bugs.  I would advocate everyone have these.  @marten-seemann You'll never get to 0 transport errors.  Some implementations will have a bug, and some peers or servers will have faulty or corrupted memory.  For example, in Google QUIC we found we received the ClientHello on a stream besides 1 surprisingly often.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607253576

---

From: Martin Thomson <[email-masked]>
Date: Wed, 01 Apr 2020 15:56:45 -0700

I think that the idea is to reduce the set of RCID frames needed to a range of sequence numbers.  You only have to remember the few exceptions for sequence numbers that are in use, sequence numbers for which you have outstanding frames, and a range of sequence numbers that require retirement.

As long as your discretionary use is sequential, this is bounded state.  As mentioned, the risk is that lack of acknowledgment leads to potentially large delays in getting new connection IDs, but if you ever run short, abandon the connection.  Either you have a pathological loss pattern or your peer is messing with you.  In either case, it is probably not worth continuing.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607529679

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 01 Apr 2020 16:38:42 -0700

@martinthomson 
> As long as your discretionary use is sequential, this is bounded state.

Yes, this is one method of limiting the state that is being used for retaining sequence numbers that need to be sent. And it works even when discretionary use is not sequential, as the number of "exceptions" (i.e. gaps) is bounded by max_connection_id_limit. The other assumption of using this method is that you would somehow bound the state that is used to track RCID frames being inflight. There are various ways of accomplishing that, e.g., have a limit on the number of RCID frames sent at once, track bunch of inflight RCID frames as one meta-frame.

And it is correct that the outcome would be a "lag" in this method.

The downside of this method is that it might be a disruptive change to existing implementations. It is essentially equivalent to what #3553 proposes, with the exception being that tracking RCID frames are difficult (as you track to send many).

For existing implementations, it would be easier to adopt a "leaky" method, i.e.:

* let endpoints have a counter counting on the number of sequence numbers that are retired but are yet to be acknowledged
* when a CID is being retired, if the value of the counter is below a certain threshold (i.e. 2 * max_connection_id_limit), the counter is incremented, and a RCID frame carrying the sequence number of that CID is added to the outstanding queue. If the counter was not below the threshold, no RCID frame will be sent (leak!)
* when an RCID frame is deemed lost, it will be resubmitted into the outstanding queue
* when an RCID frame is acknowledged before deemed lost, the counter is decremented
* when an RCID frame is late-acked, we do nothing

Assuming that the solution we prefer is the least-disruptive approach, I think we need to  recommend the latter. If we have distaste in the latter and prefer the former, I think we'd better switch to #3553, as it more cleanly reflects the design of the former.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607541624

---

From: Martin Thomson <[email-masked]>
Date: Wed, 01 Apr 2020 16:43:54 -0700

Yep, I think that is a good approach.  Obviously, the range encoding is superior in terms of how much state it can hold, but a simple list is fine.  And the effects of lag will be limited by the tolerance for retaining retired connection IDs, which seems proportional to me.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607543035

---

From: martinduke <[email-masked]>
Date: Wed, 01 Apr 2020 16:50:54 -0700

@kazuho I doubt that's less impactful on existing implementations, certainly from an interoperability standpoint. Today endpoints should hold on to incoming CIDs until they are formally retired. Making them leaky can lead to trapped state.

Sending them all, but with a delay, is no added state but solves this problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-607544837

---

From: Martin Thomson <[email-masked]>
Date: Tue, 07 Apr 2020 19:14:23 -0700

After a lot of back-and-forth, I think that we've all generally converged on #3547.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3509#issuecomment-610712858