Subject: [quicwg/base-drafts] QUIC lacks on-path exposure of packet loss (#3189)

From: Alexandre Ferrieux <[email-masked]>
Date: Mon, 04 Nov 2019 10:34:04 -0800

QUIC exposes the Spin Bit to the path in order to help operators debug their
networks. It allows them to locate segments with an abnormal contribution to
latency by dichotomy, since observing the Spin Bit at a given middle point
allows to compute separately the contribution to RTT of both sides.

This is good, but in many cases, latency is not the crucial element; packet loss
is the real killer. The reason for this lies on the foreseeability of the fault
location:

1. The location of significant latency buildup is most often known in advance: it is the last segment of the access network (capacity bottleneck with a large per-user buffer)
2. Packet loss, by contrast, happens in all segments of the end-to-end path, with low correlation with latency (high-speed interfaces of core routers with limited RAM and large amounts of cross-traffic). 

For a network operator, the key feature is thus the ability to do dichotomy on
packet loss. One might imagine that it is doable by simultaneous packet captures
on both ends of a suspected segment; this simply doesn't help further
investigations on other segments, since one doesn't know in which direction to
proceed (upstream or downstream). Dichotomy is impossible.

In this context, it is very frustrating to anybody in network operations to see
QUIC v1 ship with the less important of on-path measurements, the Spin Bit, and
without an effective way to identity and locate loss. This means several years
of near blindness on network issues and degraded network QoS for QUIC. Everybody
(including content providers and customers) will directly or indirectly suffer
from this.

Early in the discussions around the Spin Bit, two "reserved" bits have been
secured "for experimentation" in the first byte of short headers. These bits are
currently unused. We propose to use them for on-path exposure of packet loss in

https://datatracker.ietf.org/doc/draft-ferrieuxhamchaoui-quic-lossbits/

Note the efficiency of the method ("QL bits") has been verified in the field
through a full-size experiment in 4 countries conducted by Orange and Akamai, as
described at IETF105 in TSVWG and MAPRG

https://datatracker.ietf.org/meeting/105/materials/slides-105-tsvwg-sessb-32-loss-signaling-for-encrypted-protocols-03

https://datatracker.ietf.org/meeting/105/materials/slides-105-maprg-packet-loss-signaling-for-encrypted-protocols-01

The purpose of the current Issue is to raise awareness of the upcoming
difficulty to debug networks in the current state of the specification, and to
request the inclusion of the QL bits in the transport draft, with the same
status as the Spin Bit – “experimental” in v1 and with anti-ossification
requirement to only activate the QL bits on no more than 80% of connections. We
are hereby suggesting the constitution of a design team to conduct the same
analysis of privacy and ossification impacts as was done for the Spin Bit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 04 Nov 2019 11:06:29 -0800

I certainly support this.  The amount of changes required from existing implementation is probably 1 line (allow bits to be what they may).  The only real work that this Issue proposes is a design team to assess privacy & ossification concerns.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549500906

---

From: Nicolas Kuhn <[email-masked]>
Date: Mon, 04 Nov 2019 12:22:28 -0800

+1
We have experimented the proposed solution. It helped us identify losses occuring before a satellite gateway. Operating a network involves dealing with different equipment providers proposing specific management API and with 'old' equipment with specific management API. This solution is a real helper when it comes to identify the faulty segment in a network composed of various equipment. 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549530336

---

From: Martin Thomson <[email-masked]>
Date: Mon, 04 Nov 2019 16:02:07 -0800

This issue was discussed previously in #632.  Is there new information that might result in us reaching a different conclusion?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549602474

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 04 Nov 2019 16:12:01 -0800

The last comment from @martinthomson in #632 was:
> In the future, if there are more concrete proposals for what can be measured (and ideally how), then we will need a thorough analysis along the lines of that performed for the spin bit.

This issue references a concrete proposal as well as a measurement study of that proposal performed on actual user QUIC traffic.  It is asking for a consideration along the lines of that performed for the spin bit. Is that not so?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549604994

---

From: Martin Thomson <[email-masked]>
Date: Mon, 04 Nov 2019 16:30:52 -0800

Yes, that comment also says "this is not going to happen for QUIC version 1".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549609681

---

From: Jana Iyengar <[email-masked]>
Date: Mon, 04 Nov 2019 18:25:15 -0800

Specifically, we discussed this in Tokyo and decided to not pursue this for
QUICv1. From #632: "Tokyo conclusion: this requires something more concrete
and anything about this is not going to happen for QUIC version 1."

I'll echo Martin's comment -- what has changed?

On Mon, Nov 4, 2019 at 4:30 PM Martin Thomson <[email-masked]>
wrote:

> —
> Reply to this email directly, view it on GitHub
> <https://github.com/quicwg/base-drafts/issues/3189?email_source=notifications&email_token=ACUOBVH3JXVPAWQ6IEV5BG3QSC5DXA5CNFSM4JIXQLJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBGBUI#issuecomment-549609681>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACUOBVBTKSG7MWBIQKPAYGDQSC5DXANCNFSM4JIXQLJQ>
> .
>

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549634907

---

From: Igor Lubashev <[email-masked]>
Date: Mon, 04 Nov 2019 18:37:31 -0800

@janaiyengar See my reply to @martinthomson. What's changed is that we now have a concrete proposal and it has been validated to work on real world traffic. The lack of the above was the reason #632 was not pursued further, per the entirety of the quoted comment.

The WG may still decide not to pursue this, but it would have to be for a different reason.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549637589

---

From: MikkelFJ <[email-masked]>
Date: Tue, 05 Nov 2019 00:25:14 -0800

If it works, if it is tested, and if it is easy to implement, there are great benefits to having it in version 1 because then equipment can monitor all QUIC traffic without exceptions.

But if there is the slightest doubt that the design is absolutely right it definitely would be premature to introduce this now.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-549714684

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 05 Nov 2019 16:39:06 -0800

_Chair hat on_

You have brought what appears to be a concrete proposal, but it comes extremely late in the process; we have come to consensus (in #632) not only that we won't address this in QUICv1, but also that we've [raised the bar for new issues](https://mailarchive.ietf.org/arch/msg/quic/tsm7Dln_ssvHiBSZLp7LH9x44DU) on this draft. 

The question, then, is whether this proposal counts as new information (and note that the late-stage process consensus call lists the primary types of new information as "security and interoperability impact"). Given not only that higher bar but also that so many participants considered this a settled question, I want to consult with @larseggert about how we should triage this issue, and _if_ we accept it as a new design issue, how the discussion should commence.

Until then, there's no need for WG participants to weigh in on this; please use your time to work on other issues and your implementations.

P.S. You [mentioned this work](https://mailarchive.ietf.org/arch/msg/quic/FL_6Qxrz-RejKlBpGLxdnivPK5Q) in the IETF105 timeframe (and presented to other groups then), but did not make it clear that it was a proposal for QUIC -- @igorlord's words on list were "The draft is not targeted at the QUIC WG specifically". We took you at your word and did not give your draft time in the meeting then; that's a shame, because if you had been more forthcoming, we would have had more options to evaluate this work.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550088196

---

From: Igor Lubashev <[email-masked]>
Date: Tue, 05 Nov 2019 20:10:45 -0800

@mnot, @larseggert I am sure this is not an easy call, and it is late in the process. While this is not a straightforward "draft has an interop [between QUIC implementations] problem" it is a potential "draft has an interop with Internet ops problem" issue. This could be a stretch of what interoperability 
impact means, or not; please use your best Hat of Wisdom here.

P.S. In the IETF105 timeframe, while the draft I was presenting was, indeed, not targeting QUIC WG specifically, I did ask for the consideration of the topic by QUIC WG: "if the WG is interested, we are happy to have a quick QUIC-specific presentation on this". But you are correct that I could had been much more forceful in the request. In any case, we [got a rejection](https://mailarchive.ietf.org/arch/msg/quic/wsVao34--oHbH6Qtf9V0h8SVCSo) (and, as you said, we later [presented this in other WGs and held a side meeting](https://mailarchive.ietf.org/arch/msg/quic/UMcte0FnHGUSHs7lnVpZ0tGX_Vw)).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550134604

---

From: jesusmamartin <[email-masked]>
Date: Wed, 06 Nov 2019 01:44:54 -0800

@ferrieux. Good job, now supported by a concrete proposal and validated on real world traffic. I hope it can be taken into account by the group as soon as possible.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550230403

---

From: Alexandre Ferrieux <[email-masked]>
Date: Wed, 06 Nov 2019 05:51:42 -0800

@mnot :
While this indeed is not new (as in "heavily validated in the field"), we are stepping forth to provide the "last" element that you've been requesting: a manifestation of support from the industry. Here is a list of players who have expressed support so far:
 - Orange
 - Akamai
 - CNES(Satcom)
 - Telefonica
 - British Telecom
 - Microsoft
The list is growing daily. Shipping v1 without loss observation support would undermine daily network maintenance work in at least that part of the ecosystem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550317513

---

From: Brian Trammell <[email-masked]>
Date: Wed, 06 Nov 2019 12:06:48 -0800

(For the record, despite the length of the following comment, as the reporter of #632, let me reiterate that I accept the WG consensus on its closure. But this thread caught my attention because it is relevant to my interests.)

I'm supportive of the goal of this work - actionable measurability is a key property for any transport protocol traversing networks that might occasionally break - but it seems to me we're conflating three questions here:

- The requirement to determine and localize loss in a network through purely passive measurement (i.e., without resorting to encapsulation-based methods, for which e.g. the IOAM work in IPPM would be more appropriate; or just pinging the hell out of everything, which is certainly a common current practice if not a best one). 
- The requirement to support this loss measurement via bits in the transport protocol header. 
- The suitability of the particular mechanism defined in https://datatracker.ietf.org/doc/draft-ferrieuxhamchaoui-quic-lossbits/ to address these requirements. 

To the first question, I managed to convince myself a while ago that loss at the transport layer is overrated as a metric due to the vagaries of congestion control in the pre-ECN world we inhabit: while the lost packets are the actionable-for-the-operator cause of performance issues, it's the reaction of the transport that is interesting to the endpoint users and therefore likely to annoy the customers -- or in other words, if a packet is lost in the woods and there is nobody to NACK it because it was application-irrelevant, did it make a sound? I clearly haven't convinced anyone else, though, there is a lot of common understanding of what loss means as a metric, and the actions taken in response to that loss do correlate with better user experience, so okay, let's stick with the metric we think we know.

To the second question, as a proponent of the spin bit, I'm obviously a fan of this general class of approaches, given that it was not possible to come to consensus that the IETF should define a common wire image for transports carried over UDP, which IMO would have been a preferable architecture.

To the third question, I am delighted to see that there is a proof of concept showing that this approach works. Running code is great. :) 

I would repeat @DavidSchinazi's point, though, that the privacy implications of this particular mechanism would need to be clarified. The same approach as was followed with the spin bit privacy design team -- determining the space of metrics exposed by the signal, then showing that the marginal privacy risk of exposing those metrics compared to the information available in a version of the protocol without the signal was acceptable -- would work here, but I don't see that that analysis has been done. As one of the people driving the effort on the spin bit analysis, I can say it was a little more than a PAM paper's worth of work.

I'm also not convinced it's the best use of the two bits it wants to allocate. The spin bit, you may recall, was defined in the [IMC paper](https://mami-project.eu/wp-content/uploads/2018/09/spinbit.pdf) we published to explore its properties as a three bit signal, and the consensus of the working group was that the two bits in the VEC (valid edge counter) could be replaced with better heuristics for bad edge rejection at the measurement analysis point (and shifting the burden to the measurement point, as removing the VEC did, is just [good engineering](https://arxiv.org/pdf/1612.02902v1.pdf), considering the relative numbers of endpoints and measurement points in the Internet). I don't see any analysis done of how well the proposed signal would work, say, if one dropped Q and just used L (or vice versa), or the impact of alternate semantics for L. I haven't done the work (and sadly, my cycles for doing exploratory measurement design ain't what they used to be, so I probably won't do the work), but I suspect a slight redefinition of L to bring its semantics closer to CWR in TCP+ECN would provide comparable metric fidelity to the proposal here at the cost of slightly more measurement-point complexity and one fewer header bit.

I'm a little skeptical that we can come to convergence on the privacy-implications question and the alternate-signal-fitness question in a timeframe that is reasonable for the WG to ship version 1 of the protocol. From my own personal experience, these two questions took about a year of calendar time to answer satisfactorily for the spin bit.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550479029

---

From: Alexandre Ferrieux <[email-masked]>
Date: Wed, 06 Nov 2019 12:49:49 -0800

@britram:

Re "okay, let's stick with the metric we know": there is a bit more do be said. In real-world, heterogeneous/not-too-modern networks, we are essentially stuck in a pre-ECN world; there, even the smartest of congestion control algorithms must end up interpreting some loss patterns as congestion, and sometimes, wrongly.

Re privacy analysis: indeed this is what we are explicitly asking for. But using the Spin Bit as a reference to infer the amount of work needed is not really appropriate I think, as the very nature of the exposed signal is entirely different.

Re using the bits differently: you'll surely remember that, as an early admirer of the sheer beauty of the VEC, I certainly have no preconceptions against the principle. However:

1. As Marcus' research showed, it turns out that observer-side denoising does the job reasonably well, with a definition of "reasonably" that led to the current consensus, a one-bit Spin Bit.

2. Firsthand experience with debugging in the aforementioned real-life networks in several countries and varied multi-stakeholder scenarii tells that loss is *much* harder to locate than latency (see this Issue's introductory text for details). 

3. No, dropping one of the bits is not an option: the key feature of the method is *dichotomy*, meaning we need to somehow convey measurements about either side of the observer. And any ECN-based "trick" to emulate Q and carry only L is, again, predicated on the availability of ECN-capable layer 3 on the uninterrupted path.

As a result, I believe that the Spin+Q+L combination is, given a 3-bit budget, the best compromise to support network troubleshooting.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550494396

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 06 Nov 2019 13:55:43 -0800

I'm locking this issue until we determine it's in-scope for the WG.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-550518225

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 18 Nov 2019 22:01:42 -0800

Discussed in Singapore. The discussion there came to the conclusion that the best path forward was to not consider this in-scope for QUICv1, but to encourage the proponents to rework their draft as an extension proposal.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#issuecomment-555347498

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 18 Nov 2019 22:01:43 -0800

Closed #3189.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/quicwg/base-drafts/issues/3189#event-2810113212