Subject: [quicwg/base-drafts] Request to Retire Locally Issues CIDs (#2769)

From: Nick Banks <[email-masked]>
Date: Tue, 04 Jun 2019 12:48:30 -0700

Adds a new (optional) field, 'Retire Prior To' (placeholder name), to the NEW_CONNECTION_ID frame, which allows a sender to request that its peer immediately retire all connection IDs up to a certain sequence number. Once the peer acknowledges this packet, it has at most 3 PTO to follow through or risk the sender closing the connection.
You can view, comment on, or merge this pull request online at:

  https://github.com/quicwg/base-drafts/pull/2769

-- Commit Summary --

  * Retire Prior To

-- File Changes --

    M draft-ietf-quic-transport.md (50)

-- Patch Links --

https://github.com/quicwg/base-drafts/pull/2769.patch
https://github.com/quicwg/base-drafts/pull/2769.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769

---

From: MikkelFJ <[email-masked]>
Date: Tue, 04 Jun 2019 14:04:48 -0700

mikkelfj commented on this pull request.

> @@ -994,6 +994,14 @@ packets sent from only one local address.  An endpoint that migrates away from a
 local address SHOULD retire all connection IDs used on that address once it no
 longer plans to use that address.
 
+The endpoint can explicitly request its peer to retire connection IDs by sending
+a NEW_CONNECTION_ID frame that contains a Retire Prior To field.  The peer is
+required to retire these connection IDs in a timely manner.  On receipt of the
+acknowledgement for the packet that contained the NEW_CONNECTION_ID frame, the
+endpoint may start a 3 PTO timer, which on expiring may close the connection
+with a PROTOCOL_VIOLATION error if all the connection IDs were not retired as
+requested.
+

I think this is too aggressive. A single delayed packet could randomly take down a connection and attackers could speculate in that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-245702594

---

From: MikkelFJ <[email-masked]>
Date: Tue, 04 Jun 2019 14:32:35 -0700

> @@ -5018,10 +5034,16 @@ sequence number, or if a sequence number is used for different connection
 IDs, the endpoint MAY treat that receipt as a connection error of type
 PROTOCOL_VIOLATION.
 
+If the Retire Prior To field is present it is a request for the peer to
+immediately retire all connection IDs with a sequence number less than the
+specified value.  For more information see {{retiring-cids}}.  This field
+should only ever increase, though reordering and retransmission of older packets
+may make it seem to decrease.  So any decrease in this field should be ignored.
+

Retransmission of packets is not the right term here but I agree with the overall intent. To be consistent with other monotonic fields perhaps says something like the SHOULD not send a smaller value unless a previous frame is retransmitted verbatim. MUST ignore frames with a prior to value less than one previously received.

This means that the entire frame is to be ignored, not just the prior to field.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-245714760

---

From: MikkelFJ <[email-masked]>
Date: Tue, 04 Jun 2019 21:35:08 +0000 (UTC)

Or, I'm not sure: it is a burden for the receiver to check the prior to field. The sequence number ought to be the only value ruling validity. One could state that the receiver MAY issue a protocol violation if the a sequence number is received hat contradict monotonic increase in prior to values.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290506794

---

From: Nick Banks <[email-masked]>
Date: Tue, 04 Jun 2019 14:38:41 -0700

nibanks commented on this pull request.

How could a single delayed packet take the connection down? Best case, the same packet that acknowledges the NEW_CONNECTION_ID also contains all the RETIRE_CONNECTION_ID frames. Worst case, maybe you send them an RTT later. The timer here is 3 PTOs, not 3 RTT. If you haven't retired in that amount of time, how long of a time window should we allow?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290507916

---

From: Nick Banks <[email-masked]>
Date: Tue, 04 Jun 2019 21:40:47 +0000 (UTC)

We have the same problem with flow control values. We allow for implementations to naively retransmit packets that were lost, as is, so it's possible you have already updated the prior to field in a more recent packet, but then retransmit an older packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290508547

---

From: Nick Banks <[email-masked]>
Date: Tue, 04 Jun 2019 14:42:33 -0700

The other option here is not to kill the connection, but the sender just removes those CIDs from its lookup/routing tables. This means, if the peer continues to get them, they will get a stateless reset. I prefer the explicit behavior, because I feel like it is easier to diagnose when there is a problem; but I am willing to accept either solution.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290509109

---

From: MikkelFJ <[email-masked]>
Date: Tue, 04 Jun 2019 14:51:21 -0700

I know, I just suggest a wording more close like those (don't recall the details)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290511852

---

From: MikkelFJ <[email-masked]>
Date: Tue, 04 Jun 2019 14:56:29 -0700

Yes, I was thinking 3 RTT. 3 PTO may be enough, but an attacker could still delay so I prefer to drop such packets.

If you remove them from the routing table that is a deployment strategy - but most would likely keep a grace period before resetting or dropping. I'd imagine the middlebox load balancer is slow to update and only drops while backends are more aggressive because they are directly targeted. This is actually also true without explicit retirement - do we have any text on how to deal with that?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290513357

---

From: ianswett <[email-masked]>
Date: Tue, 04 Jun 2019 15:02:47 -0700

ianswett commented on this pull request.

I'd like to make it a protocol violation for the gap between the CID sequence number and the "Retire Prior To" value being more than the active_connection_id_limit specified in the transport params.  Previously people said implementing that was too difficult, but having both values in a single frame make it pretty easy to get right and detect when it's violated.

@marten-seemann 

I was going to suggest that the peer MUST change CIDs immediately or the issuer MAY close the connection.  Admittedly that could cause the sender of CIDs to burn through a CID per packet, but I'm not sure what its incentive to do that would be?

I think you can say that the field MUST not decrease if the CID sequence number increases?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-245723996

---

From: Martin Thomson <[email-masked]>
Date: Tue, 04 Jun 2019 19:54:44 -0700

martinthomson commented on this pull request.

I really don't like the fact that this is a field on NEW_CONNECTION_ID rather than a new frame type that just carries the proposed "Retire Prior To" field.  This change renumbers a bunch of frame types just to make this part of a frame that already exists.

The cost of a separate frame type is that you don't have an atomic assurance that a connection ID is available.  That's OK.  I mean, this PR doesn't even require that Sequence Number be equal to or greater than Retire Prior To, so clearly that isn't a concern that is top-of-mind.  I would prefer to instead reiterate that the recipient of a "RETIRE_ALL_CONNECTION_IDS_WITH_SEQUENCE_NUMBERS_BEFORE" frame can't retire connection IDs until it has a connection ID that it can use (which is basically what we say elsewhere anyway).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-245789447

---

From: Nick Banks <[email-masked]>
Date: Tue, 04 Jun 2019 20:32:56 -0700

First, I don't see the frame type rearrange as a very large change at all. Second, I do think the atomic assurance is important, and it's just an oversight that we aren't explicitly require Sequence Number to be greater. I didn't think of that scenario.

Without the atomic assurance, I worry about the scenario where an endpoint retires all the CIDs without replacing any. It gets a lot more complicated if you have separate frames. What conditions are considered protocol violations?

In order to replace all existing CIDs, the order of the frames in a single packet starts to make a difference. You have to make sure you have the NEW_CONNECTION_ID frame followed by the RETIRE_ALL_CONNECTIONS_IDS_WITH_SEQUENCE_NUMBER_BEFORE frame.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-498925956

---

From: Kazuho Oku <[email-masked]>
Date: Tue, 04 Jun 2019 21:10:57 -0700

I think I agree with @martinthomson that we should refrain from renumbering frames (again). One of the issues regarding the proposed renumbering is that the two CONNECTION_CLOSE frame types would not have the same MSB bits; it could be a concern for certain stacks.

OTOH, it is also my view that we do not need a new frame nor a new frame type.

How about just making the "Retire Prior To" a mandatory field of the NEW_CONNECTION_ID frame? The value can be set to zero when retirement is not necessary. We do not expect a lot of NEW_CONNECTION_ID to be inflight; spending one additional byte is IMO a non-issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-498931519

---

From: Martin Thomson <[email-masked]>
Date: Tue, 04 Jun 2019 23:13:17 -0700

MAY and MAY here

> @@ -4984,6 +4994,12 @@ Sequence Number:
 : The sequence number assigned to the connection ID by the sender.  See
   {{issue-cid}}.
 
+Retire Prior To:
+
+: A variable-length integer specifying the limit, for which all connection IDs
+  assigned to the receiver that have a sequence number less this value should be
+  retired.  See {{retiring-cids}}.  This field is only present for type=0x19.

This doesn't say anything about what to do with the connection ID that is negotiated during the handshake, and the one attached to a server preferred address.

There's lots of vague 2119-lite language here.  The language on flow control can be used here: 

> A receiver cannot renege on an advertisement however. That is, once a receiver advertises an offset, it MAY advertise a smaller offset, but this has no effect.
> [...]
> A sender MUST ignore any MAX_STREAM_DATA or MAX_DATA frames that do not increase flow control limits.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-245819379

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 07:23:05 -0700

@nibanks pushed 1 commit.

13f452a1556564d5ceffd9424525ed2d430b6de4  clearer text. 2119 language. retire prior to less than seq number

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/cf17a89edd1b238a5fd7df4e400b180cb675765a..13f452a1556564d5ceffd9424525ed2d430b6de4

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 14:25:20 +0000 (UTC)

091b4f5f81ae8dfb10155853ae48c1fad098ab57  Missed a type change

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/13f452a1556564d5ceffd9424525ed2d430b6de4..091b4f5f81ae8dfb10155853ae48c1fad098ab57

---

From: ianswett <[email-masked]>
Date: Wed, 05 Jun 2019 07:25:43 -0700

+1 to @kazuho suggestion of making it mandatory and not renumbering frame types.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499105867

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 07:27:18 -0700

I would also prefer a requirement that the peer MUST change CIDs immediately. I heard push back on that design. I would like to hear reasons why that wouldn't be a good idea, if anyone has some.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r290766869

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 07:34:39 -0700

Thanks @ianswett @martinthomson @kazuho for your feedback. I have made some changes, making the Retire Prior To field required, so we only have one frame again, added more 2119 language, cleaned up the text on ignoring reductions in the Retire Prior To field, and made it a protocol violation to have Retire Prior To greater than or equal to Sequence Number.

@ianswett on your request to require the difference between the Retire Prior To field and Sequence Number field to be no greater than active_connection_id_limit, that would mean the sender has to keep track of both what's the highest retired sequence number retired by the peer and what it wants locally, and the max of those two values in this field. I'm not sure the extra complexity for keep track of that is worth the cost. What benefit do you see in that requirement?

@martinthomson on the topic of the handshake and preferred address CIDs, I thought we already had text that implicitly assigned them sequence numbers. Is that not the case? What text beyond that would you like me to add?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499109697

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 05 Jun 2019 10:28:27 -0700

Thanks for writing this up! 
Specific to this text (see additional discussion on the issue): 

I don't think that we need any requirement about the distance between RetirePriorTo/SequenceNumber active_connection_id_limit. Setting a super high value would be a bit nonsensical, so some non-normative text explaining why doing that is a complete waste of everyone's time would be good, but it doesn't actually break anything. Saying "hey this is silly, consider that limit when you set this" is probably good enough just to make implementors aware of the foot gun. 

To the point about handshake and preferred address CIDs, I think it would be nice to have some text just as a reminder along the lines of "don't forget these have sequence numbers too" to help folks avoid confusion.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499179347

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 10:29:02 -0700

f80ffce179dabc3c0f72ef6ea8d0c199f0eb91e0  Change restriction from >= to >

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/091b4f5f81ae8dfb10155853ae48c1fad098ab57..f80ffce179dabc3c0f72ef6ea8d0c199f0eb91e0

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 10:37:19 -0700

75ed143f62141b429057c54827fefb660ced7367  Mention initial and preferred_address CIDs

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/f80ffce179dabc3c0f72ef6ea8d0c199f0eb91e0..75ed143f62141b429057c54827fefb660ced7367

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 10:45:19 -0700

db6980c75761ff46772d685a60d2a97e29baff0f  Yet another < vs <= fix

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/75ed143f62141b429057c54827fefb660ced7367..db6980c75761ff46772d685a60d2a97e29baff0f

---

From: Kazuho Oku <[email-masked]>
Date: Wed, 05 Jun 2019 16:42:32 -0700

kazuho commented on this pull request.

+Retire Prior To:
+

less "than" this value?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-246308811

---

From: Nick Banks <[email-masked]>
Date: Wed, 05 Jun 2019 18:36:03 -0700

2db529c95f40f1118cd04e371325764aca1bec78  Added a 'than'

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/db6980c75761ff46772d685a60d2a97e29baff0f..2db529c95f40f1118cd04e371325764aca1bec78

---

From: Marten Seemann <[email-masked]>
Date: Thu, 06 Jun 2019 00:03:29 -0700

I agree that atomicity is important here, if we want to avoid the situation where all connections IDs are retired at the same time. Adding the field to the NEW_CONNECTION_ID frame as @kazuho suggested seems to be the most straightforward way to do so.

I'm not a big fan of the 3 PTO requirement here. I don't see why it shouldn't be possible to retire the connections immediately when receiving the frame. There's a DoS risk here: if an attacker delays a packet with a connection ID that was retired afterwards, it can cause either a stateless reset or even a PROTOCOL_VIOLATION. The better way to enforce the retiring of the connection IDs is to look at packet numbers: The old connection IDs MUST NOT be used in packets with higher packet numbers than the packet that acknowledged the NEW_CONNECTION_ID frame.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499375012

---

From: Kazuho Oku <[email-masked]>
Date: Thu, 06 Jun 2019 00:12:29 -0700

@marten-seemann 
> The better way to enforce the retiring of the connection IDs is to look at packet numbers: The old connection IDs MUST NOT be used in packets with higher packet numbers than the packet that acknowledged the NEW_CONNECTION_ID frame.

Am I correct in understanding that this is the way key update works? Assuming that is the case, my +1 goes here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499377284

---

From: Marten Seemann <[email-masked]>
Date: Thu, 06 Jun 2019 01:17:15 -0700

> Am I correct in understanding that this is the way key update works? Assuming that is the case, my +1 goes here.

I think that's an implementation decision. You could certainly enforce this by requiring that `PN(packet sent with old connection IDs) < PN(first packet received with new connection ID)`. This doesn't stop a peer from acknowledging the NEW_CONNECTION_ID frame, but never acting on it though.

Another way to implement this would be to enforce that `PN(packet sent with old connection ID) < PN(ACK for NEW_CONNECTION_ID frame)`. This would be a somewhat stricter way, since it doesn't allow the peer to ignore the frame (without inducing artificial packet loss).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499396313

---

From: MikkelFJ <[email-masked]>
Date: Thu, 06 Jun 2019 04:24:26 -0700

> The better way to enforce the retiring of the connection IDs is to look at packet numbers: The old connection IDs MUST NOT be used in packets with higher packet numbers than the packet that acknowl…

I think this solves the problem since I really dislike that a delayed packet, 3PTO or otherwise, can take down a connection.

However, I also dislike that this is coupled strongly to ACK behaviour. I'm not sure how key updates work but assuming it does not use ACK, ACK handling can run asynchronously in a separate task, and indeed this is how I would architect a high performance implementation.

Then one could send a dedicated frame to acknowledge receiving the retire frame, but that is pointless since any packet using the new CID would have that effect.

So for the time being I don't have a good answer to this.

Altogether, if this facility opens up to DoS attacks and prevents ACK handling from running asynchronously, I think it does more harm than good and should at least be moved to v2 giving more time to consider the implications.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499455400

---

From: Mike Bishop <[email-masked]>
Date: Thu, 06 Jun 2019 13:19:09 -0700

MikeBishop commented on this pull request.

This is straddling the line between "request" (5037) and "require" (998).  Pick one.  Given that you're going to kill the connection if they don't comply, "request" doesn't seem like a strong enough statement here.  But the issuer only MAY kill the connection if the peer doesn't comply, so....  🤷‍♀️

If connection destruction is the intended result, go stronger.  You SHOULD set the timer and blow up the connection if it expires; you MUST retire the CIDs upon receipt of an increased RPT field.

Mostly because "change immediately" is a difficult requirement to police.  Couching it in more concrete terms -- error on use of this CID in a packet with a higher packet number than the one which carried the ACK -- makes it easier to police in theory, but highlights what different parts of the system have to be coordinated to do it in practice.

Frankly, I'd even be okay with no longer recognizing those CIDs 3xPTO after sending the frame.  Fewer moving parts, easy to understand, but a small risk of stateless reset if the packet is severely delayed.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-246789355

---

From: Mike Bishop <[email-masked]>
Date: Thu, 06 Jun 2019 13:24:55 -0700

@ianswett, the problem is that the sequence can be sparse.  The client isn't required to use the CIDs in order any more, IIRC; so you could have four outstanding CIDs, but your four are sequence numbers (2, 127, 324, 324).  Nick wants to expire everything prior to 300 and he's tired of waiting for you to use and discard the two early ones.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499653514

---

From: Nick Banks <[email-masked]>
Date: Thu, 06 Jun 2019 13:28:16 -0700

@MikeBishop I'm all for making it immediate and adding stronger language. That's exactly what I wanted to do in the first place. Initially, I was getting push back though, so I relaxed it a bit. I agree it's a bit too wishy-washy in its current form. Unless anyone has reasons why they can't or shouldn't update immediately I will go ahead and strengthen the text.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499654630

---

From: Mike Bishop <[email-masked]>
Date: Thu, 06 Jun 2019 13:53:37 -0700

I think the main reason not to be totally immediate is when you have some distributed processing generating packets.  You might not be able to guarantee that every thread/process generating packets can stop using those CIDs immediately.  However, we already went through that with the RETIRE_CID discussion -- sending RETIRE is supposed to be the signal that you've hit your sync point and *will not* use them any more.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499663518

---

From: Eric Kinnear <[email-masked]>
Date: Thu, 06 Jun 2019 13:58:28 -0700

Brainstorming times when it would be valuable to not change immediately: 
- Privacy, a client that wants to avoid linkability on different links shouldn't have to change all of its CIDs regardless of link immediately, since that would trivially link those different CIDs
- If we have a server behind a load balancer and it wants to rotate keys, having every client connected to that individual server change CIDs right away could betray a bunch of information

I don't know if we need to go as far as recommending that clients wait some amount of time, but we could potentially (in QUIC LB maybe?) recommend that servers rotate their clients not all at once. And clients might want to change on different interfaces at different times.

I do like the idea of having a packet number cutoff when the client retires the CID to simplify things on the other end.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499665095

---

From: Mike Bishop <[email-masked]>
Date: Thu, 06 Jun 2019 14:11:26 -0700

IIRC, the conclusion we reached on the CID design team was that:
- When the client sends RETIRE, that's a statement that it has purged that CID and SRT; the CID will never be used again, and the SRT won't be recognized.  The server can immediately declare that CID void upon receipt.
- If an earlier packet got delayed, it will trigger a SR, because the server doesn't recognize the CID any more.  This is also true if the client messed up and sends a later packet with the retired CID; the server will discard it.
- The client will receive the server's reset, but not recognize it as such because the SRT isn't one that it knows any more.

The important thing is that the client has already deleted the SRT before sending the RETIRE and the server can safely purge state immediately upon receipt.

In this PR, I think we have the right model by letting the client have a little time to make sure those requirements are true before sending the RETIRE.  You MUST retire the CIDs, but you're not obligated to do so instantly.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499669154

---

From: MikkelFJ <[email-masked]>
Date: Thu, 06 Jun 2019 21:58:37 -0700

In line of @MikeBishop's comments and what I have said earlier:

Implicit retirement of "Prior To" has the issues mentioned with distributed ACK processing. But the problem is in fact worse: For almost the same reason that the sender wants to retired CID's, the receiver may want to be either migrating internally to another host, or process different parts of a connection on different servers. It can be detrimental to such deployments to have a force expiration. Therefore I think Mike is right in tying this to RETIRE_CID.

Perhaps the right model is to use RETIRE_CID as the sync point regardless of whether this is done voluntarily or motivated by a "Prior To" flush. Mike suggests the sender should defer RETIRE_CID until safe, but I think the receiver should make that call because it might have a 10 minute periodic load balancer flush or something completely different while the sender can only estimate based on RTT and the sender also wants to forget the CID ASAP while the receiver can track CID's algorithmically.

The recommendation could be to invalidate CIDs from RETIRE_CID frames no earlier than 3PTO and possibly longer if it suits the deployment. Receiving a non-zero "Prior To" field SHOULD cause the CIDs to be flushed and retired with RETIRE_CID as soon as practical. The issuer of "Prior To" MAY close a connection with the error CONNECTION_ID_TIMEOUT but deployments SHOULd avoid doing that aggressively unless critical to infrastructure since clients may not be able or willing to upgrade fast.

Note that there can be race conditions where both endpoints want to retire CID's because they are internally migrating. This can lead to problems if a CID is tied to the peers CID indirectly via source IP. We chose to have asymmetrical path migration due the complexities of this. I start to think we could some nasty corner cases here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-499756719

---

From: Nick Banks <[email-masked]>
Date: Fri, 07 Jun 2019 10:06:27 -0700

4696d46774b50379d3d1823470d195ff975abcbc  Make it a requirement; SHOULD do immediately

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/2db529c95f40f1118cd04e371325764aca1bec78..4696d46774b50379d3d1823470d195ff975abcbc

---

From: Nick Banks <[email-masked]>
Date: Fri, 07 Jun 2019 10:11:26 -0700

ff007e11812de4775eaf91de75b33aa6dd789164  Clean up the other text too

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/4696d46774b50379d3d1823470d195ff975abcbc..ff007e11812de4775eaf91de75b33aa6dd789164

---

From: Mike Bishop <[email-masked]>
Date: Fri, 07 Jun 2019 15:28:38 -0700

MikeBishop approved this pull request.

I think "SHOULD ... immediately" is a workable balance.  There will be implementations which can't do it immediately, and the text describes how the issuer time-bounds them; for everyone else, immediately will be fine.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-247328689

---

From: David Schinazi <[email-masked]>
Date: Sat, 08 Jun 2019 01:28:05 -0700

I'm OK with "SHOULD immediately" but I still think having an endpoint close the connection with protocol violation if a packet is delayed is a security risk for the reasons I detailed in the issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500106799

---

From: Nick Banks <[email-masked]>
Date: Sat, 08 Jun 2019 08:08:17 -0700

Personally, I don't think there is a big difference between explicitly closing the connection VS just removing the CIDs from the routing lookup, which would cause a stateless reset to be sent if the peer continues to use it.

I like the explicit nature of the close, because it will be easier to debug if there are bugs; though I do concede that if the peer did locally retire the CIDs, but somehow the packets/frames just took an extreme amount of time to get there, the connection isn't penalized.

Perhaps a vote?
Thumbs up if you prefer connection close with protocol violation.
Thumbs down if you prefer the CIDs to be removed, resulting in a stateless reset instead.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500131371

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 08 Jun 2019 19:10:09 -0700

@nibanks 
> I don't think there is a big difference between explicitly closing the connection VS just removing the CIDs from the routing lookup, which would cause a stateless reset to be sent if the peer continues to use it.

As @DavidSchinazi has pointed out in https://github.com/quicwg/base-drafts/issues/2645#issuecomment-499470589, "having the use of a CID you've asked to retire trigger a protocol violation is a security risk. It allows an attacker to close the connection by delaying a valid packet."

If it is the case that we would have the same issue in sending a STATELESS_RESET (I assume that is what you are suggesting), I think we need to fix the issue, rather than choosing one between the two designs that both have the same security concern.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500178996

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 08 Jun 2019 19:24:43 -0700

PS. One way of resolving the security concern would be to require the use of Stateless Reset (instead of a CONNECTION_CLOSE), and also require that Stateless Reset Tokens for CID above "Retire Prior To" to be different from the tokens below or equal to "Retire Prior To."

Note: the latter requirement is necessary because [transport draft section 10.4.2](https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#rfc.section.10.4.2) states that "the same Stateless Reset Token MAY be used for multiple connection IDs on the same connection."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500179488

---

From: David Schinazi <[email-masked]>
Date: Mon, 10 Jun 2019 01:16:11 -0700

I agree with @kazuho. If we want to close the connection (which I still think we shouldn't mandate but I think we can allow) then we should use stateless resets, mandate each CID has a distinct reset token, and mandate that endpoints ignore reset tokens for CIDs they have retired. That'll prevent an attacker from closing the connection by delaying a packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500332142

---

From: MikkelFJ <[email-masked]>
Date: Mon, 10 Jun 2019 01:59:37 -0700

> I agree with @kazuho. If we want to close the connection (which I still think we shouldn't mandate but I think we can allow) then we should use stateless resets, mandate each CID has a distinct res…

That goes back to my suggestion of using epochs ("prior to" being the simplest such) because you might not want to store a reset token for every CID, only for every epoch.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500344318

---

From: Marten Seemann <[email-masked]>
Date: Mon, 10 Jun 2019 02:03:43 -0700

I agree with @DavidSchinazi and @kazuho.

@mikkelfj In general, each CID is associated with its own reset token. This allows for a straightforward stateless implementation of a stateless reset token generator. I don't think it would be a good idea to revisit that principle.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500345548

---

From: MikkelFJ <[email-masked]>
Date: Mon, 10 Jun 2019 02:11:55 -0700

@marten-seemann no you are probably right.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500348008

---

From: Nick Banks <[email-masked]>
Date: Mon, 10 Jun 2019 07:28:09 -0700

87b268bcd6c8c11c443a92c99486dfe9239e9603  Don't Close; Just Remove after 3 PTO

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/ff007e11812de4775eaf91de75b33aa6dd789164..87b268bcd6c8c11c443a92c99486dfe9239e9603

---

From: Nick Banks <[email-masked]>
Date: Mon, 10 Jun 2019 07:30:21 -0700

Thanks. All your arguments make sense. I have updated the text to say the connections IDs should just be removed after the 3 PTO, instead of closing the connection. I also added some text warning against reuse of the stateless reset tokens.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500437895

---

From: David Schinazi <[email-masked]>
Date: Mon, 10 Jun 2019 15:37:54 -0700

I'm not sure I know what "remove the connection ID" means on the receiver. I would remove any mention of 3 PTO as well. Simply state that if a sender keeps using a CID that its peer has asked to retire then packets can get misrouted or dropped.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500619879

---

From: Nick Banks <[email-masked]>
Date: Mon, 10 Jun 2019 16:34:38 -0700

@DavidSchinazi by "remove the connection ID" I meant that the connection IDs would be implicitly treated as retired. I'm not sure how best to word that section of the text. Suggestions are welcome. Additionally, I don't think many folks (though I'm not among them) would like the removal of the 3 PTO text. Several people have argued for allowing continued use of the CIDs for a short while.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500631097

---

From: MikkelFJ <[email-masked]>
Date: Mon, 10 Jun 2019 22:42:20 -0700

I think the text needs to be more clear on whether RETIRE CID must be send for those requested to be retired, or if that is implicit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500694964

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 06:09:56 -0700

@mikkelfj we have the following text already:

> The Retire Prior To field is a **requirement for the peer to retire** all connection
> IDs with a sequence number less than the specified value.  This includes the
> initial and preferred_address transport parameter connection IDs.  The **peer
> SHOULD immediately retire** all the connection IDs.

Is there something else you'd like added?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500833557

---

From: MikkelFJ <[email-masked]>
Date: Tue, 11 Jun 2019 06:16:59 -0700

"The peer SHOULD stop using all the connection IDs as soon as possible and then retire the connection IDs by stop sending a RETIRE_CONNECTION_ID frame for each connection ID."

"retire immediately" is a bit too strong as several have commented.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500836205

---

From: Mike Bishop <[email-masked]>
Date: Tue, 11 Jun 2019 10:34:10 -0700

This is true, but the text currently allows for the SRTs associated with various CIDs to be identical.  This becomes problematic if you're forcing retirement and want to stop recognizing some of the CIDs previously issued.  However, that's a problem we already have and cover in the text:

> The same Stateless Reset Token MAY be used for multiple connection IDs on the same connection. However, reuse of a Stateless Reset Token might expose an endpoint to denial of service if associated connection IDs are forgotten while the associated token is still active at a peer. An endpoint MUST ensure that packets with Destination Connection ID field values that correspond to a reused Stateless Reset Token are attributed to the same connection as long as the Stateless Reset Token is still usable, even when the connection ID has been retired. Otherwise, an attacker might be able to send a packet with a retired connection ID and cause the endpoint to produce a Stateless Reset that it can use to disrupt the connection, just as with the attacks in Section 21.8.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-500946427

---

From: Mike Bishop <[email-masked]>
Date: Tue, 11 Jun 2019 10:42:43 -0700

> @@ -5018,6 +5034,31 @@ sequence number, or if a sequence number is used for different connection
 
+The Retire Prior To field is a requirement for the peer to retire all connection
+IDs with a sequence number less than the specified value.  This includes the
+initial and preferred_address transport parameter connection IDs.  The peer
+SHOULD immediately retire all the connection IDs.
+
+The sender of the NEW_CONNECTION_ID frame MAY remove the connection IDs after 3
+PTO, even if the peer has not retired them yet.  The 3 PTO timer starts on

```suggestion
PTO, even if the peer has not retired them yet.  The 3-PTO timer starts on
```

> @@ -5018,6 +5034,31 @@ sequence number, or if a sequence number is used for different connection
 
+The Retire Prior To field is a requirement for the peer to retire all connection
+IDs with a sequence number less than the specified value.  This includes the
+initial and preferred_address transport parameter connection IDs.  The peer
+SHOULD immediately retire all the connection IDs.
+
+The sender of the NEW_CONNECTION_ID frame MAY remove the connection IDs after 3
+PTO, even if the peer has not retired them yet.  The 3 PTO timer starts on
+acknowledgement of the packet containing the NEW_CONNECTION_ID frame.  Continued
+use of the retired connection IDs after this point will likely result in a
+stateless reset being sent.
+
+In order to prevent a stateless reset, sent in response to a heavily delayed

```suggestion
In order to prevent a stateless reset sent in response to a heavily delayed
```

> @@ -5018,6 +5034,31 @@ sequence number, or if a sequence number is used for different connection
 
+The Retire Prior To field is a requirement for the peer to retire all connection
+IDs with a sequence number less than the specified value.  This includes the
+initial and preferred_address transport parameter connection IDs.  The peer
+SHOULD immediately retire all the connection IDs.
+
+The sender of the NEW_CONNECTION_ID frame MAY remove the connection IDs after 3
+PTO, even if the peer has not retired them yet.  The 3 PTO timer starts on
+acknowledgement of the packet containing the NEW_CONNECTION_ID frame.  Continued
+use of the retired connection IDs after this point will likely result in a
+stateless reset being sent.
+
+In order to prevent a stateless reset, sent in response to a heavily delayed
+packet using a retired connection ID, from closing the connection, the

```suggestion
packet using a retired connection ID from closing the connection, the
```
This parenthetical phrase is pretty critical and this sentence is pretty comma-heavy.  Consider promoting it.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-248334842

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 10:44:18 -0700

6a8d79bc7e743972250e4ed94008ed734935fb93  Mike's suggestion.

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/87b268bcd6c8c11c443a92c99486dfe9239e9603..6a8d79bc7e743972250e4ed94008ed734935fb93

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 10:44:38 -0700

5035bae0fc51c06048ea68798e8565c33b230025  Mike's suggestion.

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/6a8d79bc7e743972250e4ed94008ed734935fb93..5035bae0fc51c06048ea68798e8565c33b230025

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 10:46:23 -0700

6cc8ad2c9406a26b5c222f529d43458e40491bd6  Mike's suggestion.

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/5035bae0fc51c06048ea68798e8565c33b230025..6cc8ad2c9406a26b5c222f529d43458e40491bd6

---

From: ianswett <[email-masked]>
Date: Tue, 11 Jun 2019 11:35:15 -0700

Some small suggestions, but this LG overall.

+PTO, even if the peer has not retired them yet.  The 3-PTO timer starts on

I forgot about RETIRE_CONNECTION_ID frame the first time I read this.  Can you say something like: "not retired them with RETIRE_CONNECTION_ID yet."

+PTO, even if the peer has not retired them yet.  The 3-PTO timer starts on
+In order to prevent a stateless reset sent in response to a heavily delayed

Rewording suggestion: "In order to prevent a heavily delayed packet with a retired connection ID from eliciting a stateless reset that closes the connection, the stateless reset token...."

+PTO, even if the peer has not retired them yet.  The 3-PTO timer starts on

It may be worth adding a paragraph for the receiver of the NEW_CONNECTION_ID frame about stateless reset. ie:
"If a stateless reset is received in response to a retired connection ID, the reset SHOULD be ignored and the receiver should immediately switch to a connection ID that has not been retired if it has not done so already."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-248359629

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 12:38:54 -0700

8275d504bc118ac651b7d4071c73392179945853  Ian's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/6cc8ad2c9406a26b5c222f529d43458e40491bd6..8275d504bc118ac651b7d4071c73392179945853

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Jun 2019 15:14:02 -0700

+The endpoint can explicitly require its peer to retire connection IDs by sending
+a NEW_CONNECTION_ID frame with an increased Retire Prior To field.  The peer is
+required to retire these connection IDs in a timely manner.  If the peer does
+not retire the connection IDs within 3 PTO of acknowledging the packet with the
+NEW_CONNECTION_ID frame, the endpoint MAY remove the connection IDs anyways.  If

I don't think "anyways" is right.

```suggestion
NEW_CONNECTION_ID frame, the endpoint MAY remove the connection IDs.  If
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-247903956

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 15:24:47 -0700

65e576d03e3c985d3fbed778bec0bf17702ff375  David's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/8275d504bc118ac651b7d4071c73392179945853..65e576d03e3c985d3fbed778bec0bf17702ff375

---

From: David Schinazi <[email-masked]>
Date: Tue, 11 Jun 2019 15:29:49 -0700

DavidSchinazi commented on this pull request.

> @@ -5018,6 +5042,35 @@ sequence number, or if a sequence number is used for different connection
 
+The Retire Prior To field is a request for the peer to retire all connection IDs
+with a sequence number less than the specified value.  This includes the initial
+and preferred_address transport parameter connection IDs.  The peer SHOULD
+immediately retire all the connection IDs.

I would change this sentence to match the text before: `The peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frame in a timely manner.` and also delete the next two paragraphs as they are redundant and contradict the "Detecting a Stateless Reset" section.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-248469991

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 15:50:40 -0700

To be clear, you suggesting I remove these two paragraphs?

> If a stateless reset is received in response to a retired connection ID, the
> reset SHOULD be ignored and the receiver SHOULD immediately switch to a
> connection ID that has not been retired if it has not done so already.
> 
> In order to prevent a heavily delayed packet with a retired connection ID from
> eliciting a stateless reset that closes the connection, the stateless reset
> token of the retired connection IDs SHOULD NOT be reused.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r292689902

---

From: David Schinazi <[email-masked]>
Date: Tue, 11 Jun 2019 15:53:33 -0700

Yes

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r292690556

---

From: Nick Banks <[email-masked]>
Date: Tue, 11 Jun 2019 15:54:24 -0700

54c308a351d5293b66964a32e4bab00f23c35a27  More of David's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/65e576d03e3c985d3fbed778bec0bf17702ff375..54c308a351d5293b66964a32e4bab00f23c35a27

---

From: ianswett <[email-masked]>
Date: Wed, 12 Jun 2019 12:31:09 -0700

Some more small suggestions, but looking close.

> @@ -994,6 +994,22 @@ packets sent from only one local address.  An endpoint that migrates away from a
 
+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,

New_CONNECTION_ID frame(s)?

> @@ -994,6 +994,22 @@ packets sent from only one local address.  An endpoint that migrates away from a
 
+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,
+the peer SHOULD retire the corresponding connection IDs and send the
+corresponding RETIRE_CONNECTION_ID frame in a timely manner.  Failing to do so
+can cause packets to be delayed or lost and harm connection performance as the

I'm a bit vague on "delayed or lost and harm connection performance as the original endpoint might not route those connection IDs optimally after some delay."  Instead, how about "delayed, lost, or cause the original endpoint to send a stateless reset in response to a connection ID it can no longer route correctly."?

> @@ -994,6 +994,22 @@ packets sent from only one local address.  An endpoint that migrates away from a
 
+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,
+the peer SHOULD retire the corresponding connection IDs and send the
+corresponding RETIRE_CONNECTION_ID frame in a timely manner.  Failing to do so
+can cause packets to be delayed or lost and harm connection performance as the
+original endpoint might not route those connection IDs optimally after some
+delay.
+
+The sender of the Retire Prior To field MUST keep track of the connection IDs it

I still think this MUST is unenforceable and a bit odd.  If my load balancer restarts in 5 seconds and the peer hasn't retired the old CIDs by then, they're still getting retired.

> @@ -994,6 +994,22 @@ packets sent from only one local address.  An endpoint that migrates away from a
 
+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,
+the peer SHOULD retire the corresponding connection IDs and send the
+corresponding RETIRE_CONNECTION_ID frame in a timely manner.  Failing to do so
+can cause packets to be delayed or lost and harm connection performance as the
+original endpoint might not route those connection IDs optimally after some
+delay.
+
+The sender of the Retire Prior To field MUST keep track of the connection IDs it
+wishes to retire until it has received a corresponding RETIRE_CONNECTION_ID
+frame, with one exception: if the sender of the Retire Prior To field has used
+distinct stateless reset tokens for all of their issued connection IDs, and 3
+times the PTO has elapsed since it received an acknowledgment for its Retire
+Prior To field, then it MAY lose track of that connection ID, and respond to

```suggestion
Prior To field, then it MAY drop state for that connection ID, and respond to
```

> @@ -5018,6 +5042,28 @@ sequence number, or if a sequence number is used for different connection
 
+The Retire Prior To field is a request for the peer to retire all connection IDs
+with a sequence number less than the specified value.  This includes the initial
+and preferred_address transport parameter connection IDs.  The peer SHOULD
+retire the corresponding connection IDs and send the corresponding
+RETIRE_CONNECTION_ID frame in a timely manner

frame(s)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-248966981

---

From: Nick Banks <[email-masked]>
Date: Wed, 12 Jun 2019 13:18:57 -0700

4a446a068f57b98e60a3ca0439e408fa0f093829  Ian's suggestion

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/54c308a351d5293b66964a32e4bab00f23c35a27..4a446a068f57b98e60a3ca0439e408fa0f093829

---

From: Nick Banks <[email-masked]>
Date: Wed, 12 Jun 2019 13:23:22 -0700

Change it to a `SHOULD`?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293100894

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 12 Jun 2019 13:24:12 -0700

erickinnear requested changes on this pull request.

Thanks for writing this up! 
This seems like a pretty good start if we're adding a "Please Retire" field to a NEW_CONNECTION_ID. A few comments and some wording suggestions. I think we also need the requirement to replace all of the retired ones, otherwise we risk closing down to just a single CID every time we retire.

Editorial nit: Missing `.`
```suggestion
RETIRE_CONNECTION_ID frame in a timely manner.
```

For readability, I might make a few editorial changes to this paragraph: 
```
The issuer of a connection ID can request that its peer retire that 
connection ID by sending a NEW_CONNECTION_ID frame with a 
non-zero Retire Prior To field.
```

```
Upon receipt of such a frame, an endpoint SHOULD retire active connection IDs 
with sequence numbers lower than that specified in the Retire Prior To field 
by sending a RETIRE_CONNECTION_ID frame for each retired connection ID 
in a timely manner.
```

```
Failure to retire connection IDs in a timely manner when requested can 
harm connection performance by causing packets to be delayed or lost, 
since the issuer of the connection IDs might no longer have the ability 
to route packets using those connection IDs correctly.
```

Minor, but this also helps to align with the language here -- referring to "issuers" of connection IDs, and behavior specific to each endpoint, etc.

+The Retire Prior To field is a request for the peer to retire all connection IDs
+with a sequence number less than the specified value.  This includes the initial
+and preferred_address transport parameter connection IDs.  The peer SHOULD
+retire the corresponding connection IDs and send the corresponding
+RETIRE_CONNECTION_ID frame in a timely manner
+
+The sender of the NEW_CONNECTION_ID frame MAY remove the connection IDs after 3
+PTO, even if the peer has not retired them with RETIRE_CONNECTION_ID yet.  The
+3-PTO timer starts on acknowledgement of the packet containing the
+NEW_CONNECTION_ID frame.  Continued use of the retired connection IDs after this
+point will likely result in a stateless reset being sent.

This is slightly conflicting with the text above -- do we really want to have the same (but not quite) text in two places vs. just keeping it above and describing the field down here?
We could just remove this paragraph.

+
+The sender of the NEW_CONNECTION_ID frame MAY remove the connection IDs after 3
+PTO, even if the peer has not retired them with RETIRE_CONNECTION_ID yet.  The
+3-PTO timer starts on acknowledgement of the packet containing the
+NEW_CONNECTION_ID frame.  Continued use of the retired connection IDs after this
+point will likely result in a stateless reset being sent.
+
+The Retire Prior To field MUST be less than or equal to the Sequence Number
+field.  Values greater than the Sequence Number MUST be treated as a connection
+error of type PROTOCOL_VIOLATION.
+
+The sender cannot reduce the value of the Retire Prior To field in subsequent
+NEW_CONNECTION_ID frames.  That is, once a sender indicates a Retire Prior To
+value, it MAY indicate a smaller value, but it has no effect.  A receiver MUST
+ignore any Retire Prior To fields that do not increase.

Sounds good. A slight rewording perhaps: 
```
Once a sender indicates a Retire Prior To value, smaller values sent 
in subsequent NEW_CONNECTION_ID frames have no effect. A receiver 
MUST ignore any Retire Prior To fields that do not increase the largest 
received Retire Prior To value.
```

> @@ -4984,6 +5002,12 @@ Sequence Number:
 
+Retire Prior To:
+

Editorial nit
```suggestion
: A variable-length integer specifying the limit for which all connection IDs
```

+packets with that connection ID with the corresponding stateless reset token.

I would still prefer to avoid tracking ACKs of packets, but if we're going to do it, then can we condense this down into a sentence or two? 

Something along the lines of: 
```
An endpoint MAY discard a connection ID for which retirement has been requested 
once an interval of not less than 3 PTO has elapsed since an ack is received for 
the NEW_CONNECTION_ID frame requesting that retirement. Subsequent incoming 
packets using that connection ID will elicit a response with the 
corresponding stateless reset token.
```

This depends on preventing reuse of SRTs, but I think that's a good thing to do anyways.

+packets with that connection ID with the corresponding stateless reset token.

Also noting here that we still need some text about requiring that they be replaced.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-248485443

---

From: Nick Banks <[email-masked]>
Date: Wed, 12 Jun 2019 13:24:25 -0700

e8df2f6a6622d14d40810d0d93e4483ca412051a  Ian's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/4a446a068f57b98e60a3ca0439e408fa0f093829..e8df2f6a6622d14d40810d0d93e4483ca412051a

---

From: Nick Banks <[email-masked]>
Date: Wed, 12 Jun 2019 13:32:39 -0700

@erickinnear can you elaborate on this? We already have the requirement that `Retire Prior To` is <= than `Sequence Number` so it cannot retire all CIDs. I'm not sure if we can require replacement of ALL previously existing CIDs.

I'm fine with removing it if you feel having it above is enough.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293104150

---

From: Nick Banks <[email-masked]>
Date: Wed, 12 Jun 2019 13:38:06 -0700

933ac365d1d324e07bee34e69045ad674287b3fe  Eric's suggestions

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/e8df2f6a6622d14d40810d0d93e4483ca412051a..933ac365d1d324e07bee34e69045ad674287b3fe

---

From: ianswett <[email-masked]>
Date: Wed, 12 Jun 2019 13:39:00 -0700

There's no requirement to supply as many as the peer wants, so there can't be a requirement to replace all it wants or as many as are retired either.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293107210

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 12 Jun 2019 13:41:38 -0700

erickinnear commented on this pull request.

We had previously discussed introducing such a requirement, but I think we might already be covered here, since we're using the existing RETIRE_CID frame for the retirement, by the "SHOULD replace when retired" that's over in that part of the text.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293108241

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 12 Jun 2019 13:44:32 -0700

(So you have 3 that get retired, for each that you retire, you get back a NEW_CID with the same RetirePriorTo as the original one)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293109426

---

From: Mike Bishop <[email-masked]>
Date: Wed, 12 Jun 2019 13:56:00 -0700

It's currently a MUST to persist all the CIDs with the same SRT.  The way to be compliant would be to advance RPT enough to expire all the CIDs, even if there are some in the middle that could otherwise be kept.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r293113887

---

From: Eric Kinnear <[email-masked]>
Date: Wed, 12 Jun 2019 15:48:23 -0700

Looking good, I think we're getting there!

Yeah, at least personally it feels like there's guidance on how to do that up above already, I'd probably just skip it. :) 

> @@ -994,6 +994,19 @@ packets sent from only one local address.  An endpoint that migrates away from a
 
+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frames with an increased Retire Prior To field.  Upon receipt,

Sorry, typo: 
```suggestion
NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-249054249

---

From: Nick Banks <[email-masked]>
Date: Thu, 13 Jun 2019 06:29:45 -0700

10d19646b5774de97b1e594bdf71b3aed139fe76  Update draft-ietf-quic-transport.md

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/933ac365d1d324e07bee34e69045ad674287b3fe..10d19646b5774de97b1e594bdf71b3aed139fe76

---

From: ianswett <[email-masked]>
Date: Tue, 18 Jun 2019 15:15:26 -0700

ianswett approved this pull request.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-251390367

---

From: Eric Kinnear <[email-masked]>
Date: Tue, 18 Jun 2019 15:26:37 -0700

This is looking good (modulo the one extra paragraph at the end)!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#issuecomment-503334979

---

From: Marten Seemann <[email-masked]>
Date: Tue, 18 Jun 2019 18:16:00 -0700

marten-seemann commented on this pull request.

> @@ -4984,6 +4999,12 @@ Sequence Number:
 
+Retire Prior To:
+
+: A variable-length integer specifying the limit for which all connection IDs
+  assigned to the receiver that have a sequence number less than this value
+  should be retired.  See {{retiring-cids}}.

This sentence is kind of difficult to parse.

We should also specify that this value MUST be smaller or equal to the Sequence Number carried by this frame.

> @@ -5018,6 +5039,27 @@ sequence number, or if a sequence number is used for different connection
 
+RETIRE_CONNECTION_ID frames in a timely manner

This feels duplicate to the first diff of this PR. Maybe we can move the sentence about the initial SPA connection ID up there?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-251433130

---

From: Nick Banks <[email-masked]>
Date: Thu, 20 Jun 2019 09:08:18 -0700

62795f89f068f02159dca106ebd7a7fe94aae2f5  Remove paragraph; add period

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/10d19646b5774de97b1e594bdf71b3aed139fe76..62795f89f068f02159dca106ebd7a7fe94aae2f5

---

From: Nick Banks <[email-masked]>
Date: Thu, 20 Jun 2019 09:10:59 -0700

I prefer the text I have already.

+Retire Prior To:
+

The text below already species this must be <=. As improving the text, I agree it's a bit wordy. Do you have any specific suggestions for improving it?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r295883205

---

From: Mike Bishop <[email-masked]>
Date: Thu, 20 Jun 2019 10:40:56 -0700

+Retire Prior To:
+

Given that the specifics of the limit are explained elsewhere, keep the field description brief.  "A variable-length integer indicating which connection IDs should be retired."

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r295920232

---

From: Nick Banks <[email-masked]>
Date: Thu, 20 Jun 2019 11:16:58 -0700

8cbcebf6ccafee170693188161e402506f2d2ef7  Reword field desc

-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769/files/62795f89f068f02159dca106ebd7a7fe94aae2f5..8cbcebf6ccafee170693188161e402506f2d2ef7

---

From: Eric Kinnear <[email-masked]>
Date: Thu, 20 Jun 2019 14:48:42 -0700

erickinnear approved this pull request.

Latest changes are looking great, thanks for driving this @nibanks! 
Let's merge and we can update any further editorial nits when we next do a pass.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-252555900

---

From: Martin Thomson <[email-masked]>
Date: Sun, 30 Jun 2019 19:31:30 -0700

I had this comment sitting in a buffer.

> @@ -5018,6 +5038,21 @@ sequence number, or if a sequence number is used for different connection
 
+RETIRE_CONNECTION_ID frames in a timely manner.

```suggestion
field.  Receiving a value greater than the Sequence Number MUST be treated as a connection
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-254326658

---

From: Igor Lubashev <[email-masked]>
Date: Sun, 30 Jun 2019 23:13:06 -0700

igorlord commented on this pull request.

+An endpoint can request that its peer retire connection IDs by sending a
+NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt,
+the peer SHOULD retire the corresponding connection IDs and send the
+corresponding RETIRE_CONNECTION_ID frames in a timely manner.  Failing to do so
+can cause packets to be delayed, lost, or cause the original endpoint to send a
+stateless reset in response to a connection ID it can no longer route correctly.
+
+An endpoint MAY discard a connection ID for which retirement has been requested
+once an interval of no less than 3 PTO has elapsed since an acknowledgement is
+received for the NEW_CONNECTION_ID frame requesting that retirement.  Subsequent
+incoming packets using that connection ID will elicit a response with the

You want to say "Subsequent incoming packets using that connection ID **MAY** elicit a response with the corresponding stateless reset token."

Note that both behaviors are possible: MAY drop packets silently (misrouted, for example), and/or MAY elicit a stateless reset.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#pullrequestreview-256144731

---

From: Martin Thomson <[email-masked]>
Date: Sun, 30 Jun 2019 23:16:52 -0700

If by "MAY" you mean "could", that makes sense.  "MAY" in the sense of permitting something, is not really in question here.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/pull/2769#discussion_r298892610