Subject: Re: [quicwg/base-drafts] Stateless Reset Eternal Ping Pong (#1443)

From: Igor Lubashev <[email-masked]>
Date: Sun, 08 Jul 2018 12:54:07 -0700

If a typical sender of a SR is the server, and a typical thing that a client sends is just a ACK, would size-based squelching of SR prevent most SRs from being sent?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-403312437

---

From: MikkelFJ <[email-masked]>
Date: Sun, 08 Jul 2018 14:17:20 -0700

@igorlord not if the additional statistical rule is appied for small packets. But it would results in some some ACK's not triggering a reset. Since ACK's won't be retransmitted it could lead to a mexican standoff until timeout. The ACK's would be a response to data still in the pipeline. A server going down is probably not online again or failed-over before that pipeline has drained. So I doubt it will make much difference.

However, there are also cases where many small packets are sent such as play position updates. But then chances are higher that one of the packets trigger a reset.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-403317907

---

From: Martin Thomson <[email-masked]>
Date: Wed, 18 Jul 2018 08:27:51 -0700

Discussed in Montreal.  We concluded that a combination of a continually reducing packet size or a continually reducing TTL would suffice here.  There was less enthusiasm for a non-deterministic drop chance, particularly in light of the prevalence of packet duplication in some networks.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-405972124

---

From: MikkelFJ <[email-masked]>
Date: Wed, 18 Jul 2018 17:05:39 +0000 (UTC)

Can you explain how this would work for very small packet sizes? I'm not sure what TTL means here - how long state is remembered about an unknown endpoint?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406004701

---

From: Martin Thomson <[email-masked]>
Date: Wed, 18 Jul 2018 11:29:08 -0700

TTL means the TTL field in the IP header.  It is a number that routers decrement until it reaches 0, at which point the packet is dropped rather than forwarded.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406029851

---

From: Igor Lubashev <[email-masked]>
Date: Wed, 18 Jul 2018 18:48:03 +0000 (UTC)

Could someone explain the mechanics of the TTL proposal?

When a packet arrives, it has a residual TTL (originalTTL - #hops). So the proposal is to send a SR packet with the residual TTL and not reset the TTL to the server's default value?

This is somewhat brittle, since it depends on both peers to implement TTL logic correctly. If someone does not, pointing a broken endpoint at a victim would cause a forever-loop.  Even if everything is implemented properly, this could cause over 100 bounces per reset -- a powerful amplification attack.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406035578

---

From: ianswett <[email-masked]>
Date: Wed, 18 Jul 2018 11:55:09 -0700

Yes, it relies on both sides doing this.  If you're mostly worried about your own infrastructure ping-ponging with itself, this does solve the issue, though as you said, it could take a while.

Separating out client ports from servers ports is also a potential mitigation, as long as the client doesn't send these in a given use case.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406037677

---

From: MikkelFJ <[email-masked]>
Date: Wed, 18 Jul 2018 12:00:51 -0700

I was thinking about IP TTL when creating this issue originally, but each reset is new packet origin, so it won't affect TTL, at least not unless you specifically construct the IP header that way, and that isn't very practical for a general purpose implementation. The reset packet could have an explicit TTL - but that would expose it. Actually, doing it at IP level would also somewhat expose it.
I still don't understand why the reset needs to be secret.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406039413

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 18 Jul 2018 13:05:03 -0700

> not unless you specifically construct the IP header that way, and that isn't very practical for a general purpose implementation.

This is a good point.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406057122

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Jul 2018 05:30:33 -0700

> When a packet arrives, it has a residual TTL (originalTTL - #hops). So the proposal is to send a SR packet with the residual TTL and not reset the TTL to the server's default value?

Yes, though I would act as a router and decrement.

> that isn't very practical for a general purpose implementation.

If you can't set TTL, or reduce its size, the recommendation would be to not send the packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406259628

---

From: Marten Seemann <[email-masked]>
Date: Thu, 19 Jul 2018 06:42:55 -0700

> If you can't set TTL, or reduce its size, the recommendation would be to not send the packet.

I don't think this is a good solution. I don't have access to the TTL field in Go, but I wouldn't deploy QUIC with disabled Stateless Resets. I'd rather do something non-deterministic, or keep a bit of short-lived state around to prevent looping.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406280519

---

From: ianswett <[email-masked]>
Date: Thu, 19 Jul 2018 13:45:40 +0000 (UTC)

A short lived time wait list with exponential backoff is what I plan to deploy.  Of course, I already have that, so conveniently it fixes this case.

I do think setting TTL can avoid loops in some useful cases, but I agree that if everyone can't set it, it's potentially not a practical solution.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406281405

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Jul 2018 07:51:36 -0700

Maintaining state for 5-tuples on which a stateless reset was sent is probably the best solution, yes.  That would be a valid alternative to this.  Documenting that in the form of (maintain state || (reduce size || reduce TTL)) seems reasonable.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406304162

---

From: ianswett <[email-masked]>
Date: Thu, 19 Jul 2018 19:18:14 -0700

A simpler idea I just thought of was making the stateless reset < SOME_SIZE and not responding with a stateless reset unless the payload was > SOME_SIZE.  

Given ACK-only packets are relatively common, particularly in response to a large packet, possibly we could advise generating an ACK-only sized packet and sending that in response, with a constraint it must be less than whatever the magic size is.

I think this completely fixes the issue, assuming it's workable in practice.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406468870

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 20 Jul 2018 04:27:29 -0700

@ianswett Doesn't that go against the principle to "- to the extent possible - indistinguishable from a regular packet"?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406573225

---

From: Martin Thomson <[email-masked]>
Date: Fri, 20 Jul 2018 09:24:45 -0700

Also, ACK-only packets are generally too small.  In the extreme case, an ACK frame is only 4 octets, leading to a minimum-sized ACK-only packet that is a mere 22 octets.  Coincidentally, that's the minimum size of just the connection ID part of a stateless reset.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406652220

---

From: ianswett <[email-masked]>
Date: Fri, 20 Jul 2018 12:01:18 -0700

ACK-only packets are very common, typically the second most common packets after full-sized packets.

If we think 0 length server to client CIDs are the most common, maybe we should use those for stateless resets from server to client.  If we think 8 bytes are common from client to server, then 8 + typical ACK length?

This also has the perk that the stateless reset is smaller.  That being said, I'm happy with not fully stateless resets with exponential backoff, but I think this is also plausible.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406697836

---

From: MikkelFJ <[email-masked]>
Date: Fri, 20 Jul 2018 12:09:02 -0700

I was thinking through the ACK scenario earlier - there are two points:

1. ACKS eventually die because they are generated in response to traffic that does not arrive (unless routing is seriously messed up)
2. There can a backlog ACK's before they die out, and perhaps RESET need to act on these timely.

But there are also ping frames and small message updates like player status in games or IoT temperature measurements.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406699660

---

From: Martin Thomson <[email-masked]>
Date: Fri, 20 Jul 2018 13:49:21 -0700

An ACK- only packets seem like they are less critical to reset. The sender of an ACK- only packet is unlikely to send again shortly. That cuts two ways though. Either it would be a huge optimization to get the reset, because it would otherwise be a fair while until the next packet would be sent; or it would be wasteful, because the endpoint isn't planning to use the connection any time soon.

To me that suggests that ACK-only packets aren't worth paying special attention to in the design, even if they are super common. 

Either way, guessing the size of the peer's connection ID doesn't seem right to me. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406722859

---

From: ianswett <[email-masked]>
Date: Fri, 20 Jul 2018 13:59:04 -0700

To clarify, I'm suggesting the reset packet attempt to look like an ACK-only packet instead of attempting to look like a full-sized data packet.

I'm not trying to ensure a reset is sent in response to an ACK-only packet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406725294

---

From: MikkelFJ <[email-masked]>
Date: Fri, 20 Jul 2018 15:06:42 -0700

OK, I missed that. Back the < > SOME_SIZE, why would it be worse to statistically reset packets below SOME_SIZE than to ignore them entirely? That is what I would suggest. Above SOME_SIZE it is fine to gradually reduce size or to choose a camouflaged ACK rendition.

Some argue that gradual size reduction is not a problem but I think that unless he reduction is aggressive, it can lead to heavy amplification attack, similar to slow TTL reduction. In that light, simply sending an ACK-like reset would be safer because it aggressively drops the size down.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406738824

---

From: Martin Thomson <[email-masked]>
Date: Sun, 22 Jul 2018 19:28:50 -0700

> I'm not trying to ensure a reset is sent in response to an ACK-only packet.

A misunderstanding then...

> I'm suggesting the reset packet attempt to look like an ACK-only packet 

Yes, but I think that you're missing the point that an ACK-only packet for ANY connection is potentially much bigger than an ACK-only packet for THIS connection.

Let's say that your peer chose no connection ID (this would be common for a server that sends a stateless reset).  Then a minimal ACK-only packet is 4 octets of ACK, 2 octets of short header, and 16 octets of authentication tag.  That's 22 octets.  4 octets of ACK is unlikely, but it is a size you might expect to see early in a connection.

Now if you don't know how long the connection ID of your peer is and you want to ensure that the packet you send is plausible, you have to assume that they used the maximum size.  That's 18 octets.  Then add the short header, a PING frame (which is smaller than the ACK), and authentication tag.  That leads to 37 octets - the number in the draft.

While it might be valid to send less than 37 octets, you can't be sure that it always will be.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-406921495

---

From: ianswett <[email-masked]>
Date: Mon, 23 Jul 2018 05:44:34 -0700

I didn't miss that point about ANY connection, I'm just not sure that is the goal we want given the design we have.

We have two goals:
1) Look like a plausible non-reset packet to a passive observer. ie: "to the extent possible - indistinguishable from a regular packet"
2) Allow the peer to be reset via processing the agreed upon token.

I think number 2 only requires the packet arrive at the right IP/port, since we've already established that if the peer is relying on it's CID coming back, this design is impractical.

This makes 1 the more interesting one.  The server receives a packet it can't associate with a connection.  At that point, it may have knowledge of what typical client CIDs are for the given application(ie: 0, 8, etc) and what traffic it commonly sends, so it may be able to craft a more plausible reset than the min 37 octet suggestion.  That makes me think the current min should be relaxed, to the min non-empty packet size, though I admit by itself that makes this problem slightly worse, not better.

If we think requiring sending larger packets is extra burden on observers that want to reset connections, then a larger packet is certainly better, but I don't think we think that's a real form of protection?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407045187

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Jul 2018 17:36:50 -0700

What is typical doesn't matter.  What matters is what was used on that flow.  And we have to expect that (in the spirit of greasing at a minimum) some clients will use long connection IDs.

Unless you believe that we are OK in sending a signal to on-path entities, that is.  But we had that debate already.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407242519

---

From: ianswett <[email-masked]>
Date: Mon, 23 Jul 2018 18:49:55 -0700

For a given flow, one of two things is true:
1) The CID towards the client is 0-length:  In this case, almost any random-looking packet is plausible.
2) The CID towards the client is >0-length: In this case, any return packet will have a different CID from the correct/previous CID, so an observer would have to assume the server decided to change CIDs.  In which case, it's difficult to know what to assume, though changing CIDs by the server may be considered an "unlikely" event by observers and thus may be an indication of reset, since typically we expect the client to initiate CID changes.  I think this is a natural fallout of the asymmetric CID design that I didn't fully appreciate until recently.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407253960

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Jul 2018 18:57:15 -0700

I see, so if the client requested 18 octet connection IDs, then a change to 8 is perhaps acceptable because it just looks like it changed to a shorter value, which is possible?

What if no other connection from that client has ever used a shorter connection ID?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407255101

---

From: ianswett <[email-masked]>
Date: Mon, 23 Jul 2018 19:02:06 -0700

That's a good question.  At the moment for HTTP, I think it's reasonable to expect <=8 byte CIDs towards the client, with 0 being very common.   For WebRTC, I expect 0 bytes to also be very common, but there may be use cases when non-zero is necessary.  It's difficult for me to speculate about other applications.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407255829

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Jul 2018 19:06:53 -0700

It's possible that we could allow the 20 octets of randomness to be shortened, though observe the risks inherent in doing that.  Would that be acceptable?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407256553

---

From: ianswett <[email-masked]>
Date: Mon, 23 Jul 2018 19:10:47 -0700

That seems reasonable to me.  

I'm trying to understand what the constraints on the design space are, and I'd love to see a ping-pong solution that ended with a single reset sent, but I understand that may not always be practical.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/1443#issuecomment-407257174