Subject: [apps-discuss] draft-ietf-appsawg-http-forwarded-05 issue: multiple copies of Forwarded vs comma-separated list

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 14:33:11 -0400

In my AD review of draft-ietf-appsawg-http-forwarded-05 I found myself
significantly bothered by the fact that there are multiple ways to
represent the same thing, that proxies are giving the choice of how to
add their information (and delete older, inappropriate information),
and, therefore, that consumers of the information have to cope with it
in multiple ways.

Here's a re-edit of my comment to the document editors:

Last paragraph of Section 4:
>    Given that a proxy wishes to add a Forwarded header field to the
>    outgoing request, if the incoming request has no such header field,
>    the proxy simply adds the header field with the list of parameters
>    desired.  If, on the other hand, the incoming request has such a
>    header field, the proxy can either add a comma and the list of
>    parameters, or add a new instance of the header field.  A proxy MAY
>    remove all Forwarded header fields from a request.  It MUST, however,
>    ensure that the correct header field is updated in case of multiple
    Forwarded header fields.

I find this confusing, and I question the value of having more than
one way of representing things.  Why would a proxy want to add a comma
and a list, rather than adding a new instance?  If a proxy later wants
to selectively remove earlier information, how would it safely do it when
some had been added as list items into another header field?  Doesn't it
complicate things to have to support various combinations?  How would I
choose between these?:

    Forwarded: for=192.0.2.43,for=198.51.100.17

    Forwarded: for=192.0.2.43
    Forwarded: for=198.51.100.17

You either need to explain the pros and cons, or just provide one way
to do it.  I guess I'd want to see a good reason for not just having one
header field per proxy.  I realize that existing fields use this
mechanism, but we have a chance to be specific in a new specification.
Is there a good reason not to be?

It's also very awkward to have punctuation precedence that's backward
from how it works in English (and again, I know this is used elsewhere).
Here, commas are more powerful than semicolons, so you might have
something like this:

  Forwarded: for=192.0.2.43;proto=http,for=198.51.100.17;proto=https

As someone who reads English, I find the way that gets grouped to be
surprising and unsettling, because I expect the break between "43" and
"proto=http" to be *stronger* than the break after "proto=http", and
it's not.

I think this point is significant enough that I really need to bring it up on
the mailing list and get the working group's opinion.

Barry

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 3 Jul 2012 20:37:30 +0200

Hi Barry,

>     Forwarded: for=198.51.100.17

I agree with you on this point. Both forms are valid once received, and
parsers MUST be able to parse both forms. So it's certain that some senders
will do the later because it's easier and is valid HTTP-wise. For instance,
in haproxy, we never update the existing header, which requires complicated
memmove() and header re-indexation, we always append at the end.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 20:38:09 +0200

> Is there a good reason not to be?

Yes. Consistency in header parsing. No new special cases, please.

> the mailing list and get the working group's opinion.

Again, this is consistent with other HTTP header fields.

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 14:44:39 -0400

> Yes. Consistency in header parsing. No new special cases, please.

How is there any inconsistency?  Anyone receiving the fields already
needs to know how to parse both forms.  Why is there a specific
problem with requiring those who create the fields to create one field
per proxy?

Barry

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 20:51:49 +0200

> problem with requiring those who create the fields to create one field

If all recipients need to understand both forms (per HTTP/1.1), what's 
the point in requiring one of these for senders?

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 15:02:07 -0400

> If all recipients need to understand both forms (per HTTP/1.1), what's the
> point in requiring one of these for senders?

Consistency.  THAT is where the consistency comes in.

Barry

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 3 Jul 2012 21:04:01 +0200

> Consistency.  THAT is where the consistency comes in.

Barry, the consistency comes from the reuse of existing parsers and the
syntax already in use with other header fields. Or maybe we're all discussing
different things ?

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 21:06:18 +0200

On 2012-07-03 21:02, Barry Leiba wrote:
> ...

A big -1 on that.

We have a syntax that allows two forms. Disallowing or discouraging one 
of these for just *one* header field will either be (rightfully) 
ignored, or lead to completely unneeded special-casing in libraries.

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 15:08:40 -0400

> syntax already in use with other header fields. Or maybe we're all discussing

You and Julian appear to be arguing that because existing parsers can
handle both forms, there's no problem with allowing generators to
generate both forms.

I am arguing that simplicity says that we should tell generators to
only generate one form.  The fact that parsers can already handle this
is a good thing.

In addition, this spec suggests that proxies might want to selectively
*remove* stuff that had been added by earlier proxies.  Is that
already done for other header fields?  Isn't it simpler to explain the
removal process if there's only one way for the information to appear?

Barry

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 21:15:17 +0200

> handle both forms, there's no problem with allowing generators to

That's only part of the argument.

> only generate one form.  The fact that parsers can already handle this

Telling generators to produce only one form requires them to either 
special case the new header field (bad), or change their habits for all 
existing header fields as well (even worse).

Also, I think we have seen that there are reasons for both formats; 
sometimes it's easier to insert a new header field instance, sometimes 
it's easier to append to a field value.

There's really no reason to require a specific format.

> removal process if there's only one way for the information to appear?

Removing things is specific to this header field; so I don't have a 
"consistency" argument here. Yes, it probably would be easier to 
explain, but no, as the proxy as recipient needs to understand both 
forms anyway that's a complication the spec needs to deal with.

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 15:18:42 -0400

> Telling generators to produce only one form requires them to either special
> case the new header field (bad),

Ack.  This makes sense.

> or change their habits for all existing header fields as well (even worse).

Well, I hardly see how that's bad at all, much less worse.  Still...

> sometimes it's easier to insert a new header field instance, sometimes it's
> easier to append to a field value.

"Easier"?  Easier how?

> "consistency" argument here. Yes, it probably would be easier to explain,
> but no, as the proxy as recipient needs to understand both forms anyway
> that's a complication the spec needs to deal with.

I completely disagree here.  The proxy needs to understand how to
parse both types of fields, but does NOT already need to know how to
delete information from an existing field.  It's much easier both to
understand and to code the removal of a header field than to deal with
the selective deletion of information from a field.

Barry

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 3 Jul 2012 21:18:58 +0200

> existing header fields as well (even worse).

Exactly. And it does not tell what an intermediary which receives both
forms despite the spec should do with it ! We already have this issue
with the non-foldable Set-Cookie header which is handled as an exception
in every parser, please do not add a second one !

Regards,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Tue, 3 Jul 2012 21:22:01 +0200

> > easier to append to a field value.

Some do simply append a header field. Other ones would rather apply a regex
to an existing header. It all depends on the proxy's abilities after all.

> the selective deletion of information from a field.

Depends. I'm used to see people process headers with regex that are
declared in their configurations. You need not to stick to code, config
is a large part of header handling.

Regards,
Willy

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 15:23:54 -0400

> in every parser, please do not add a second one !

Ack.  I want to say that we can't design protocols based on the
expectation of faulty implementations, but I've been in the email
world too long to be that casual about it.  This makes sense.

Barry

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 21:28:19 +0200

> Well, I hardly see how that's bad at all, much less worse.  Still...

Do you believe people will change existing HTTP header field libraries 
just to accommodate a new requirement in this spec?

>> easier to append to a field value.

It depends on the implementation.

> the selective deletion of information from a field.

Conceptually, the proxy needs to parse the header field instances into 
an ordered list of forwarded-element, remove what it wants to remove, 
add what it wants to add, then reserialize. This is all the spec needs 
to say.

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 15:32:41 -0400

> Do you believe people will change existing HTTP header field libraries just
> to accommodate a new requirement in this spec?

That's not the point: you're saying it would be *bad* if they did; I
don't see how.

> It depends on the implementation.

That is not an answer.

> Conceptually, the proxy needs to parse the header field instances into an
> ordered list of forwarded-element, remove what it wants to remove, add what
> it wants to add, then reserialize. This is all the spec needs to say.

I understand how I would implement it.  But you can't really argue
that it's easier to code that (and test it and debug it) than it is to
simple remove an entire header field, can you?

Barry

---

From: Julian Reschke <[email-masked]>
Date: Tue, 03 Jul 2012 21:49:02 +0200

> That's not the point: you're saying it would be *bad* if they did; I

What if another header field definition prefers the other notation?

>> It depends on the implementation.

Sorry, I don't know how to explain it better. It simply depends on the 
way an implementation represents header data.

> simple remove an entire header field, can you?

If the recipient uses a proper header field parser, it doesn't matter, 
because it will execute exactly the same methods to remove an entry.

If it does not, it's *probably* easier to drop a header field instance.

That being said, a proxy is a recipient as well, so it can't *rely* on 
the format, so I'm not sure how this helps.

Manipulating HTTP messages *properly* is non-trivial; I don't believe 
that trying to add more special cases will do anything except making 
things worse.

Best regards, Julian

---

From: Barry Leiba <[email-masked]>
Date: Tue, 3 Jul 2012 16:10:07 -0400

> Manipulating HTTP messages *properly* is non-trivial; I don't believe that
> trying to add more special cases will do anything except making things
> worse.

Ack.

OK, I think Julian, Willy, and I have exhausted ourselves on this.
:-)  Any comments from others?

Barry

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 04 Jul 2012 10:07:48 +0200

>:-)  Any comments from others?

HTTP does not allow specifications for individual headers to re-define
the semantics of multiple instances of the same list-valued header in a
message; if X is a list-valued header, then

  X: 1,2

and

  X: 1
  X: 2

are equivalent; there is no option for the specification of X to say one
conforms to the X header specification while the other does not; they're
the same by definition for the purpose of finding the header value, see
http://tools.ietf.org/html/draft-ietf-httpbis-p1-messaging-19#section-3.2
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 4 Jul 2012 07:59:29 -0700

On Tue, Jul 3, 2012 at 1:10 PM, Barry Leiba <[email-masked]> wrote:

> that
Was this general two-format header thing done deliberately, or were their
competing implementations or specifications at some time in the past and
they were just merged?

I basically concur with Barry, that having two formats for something is
just bad form.  We should pick one and go with that.

But I think that idea becomes pretty hard to enforce when apparently, with
HTTP, that ship has already unfortunately sailed and parsers to handle both
formats are widely deployed.  Converging on one is bigger than this
document.

It would be nice of this got "fixed" at a lower level in httpbis, but I'll
shut up now as I feel them turning their guns on me.  :-)

-MSK

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 04 Jul 2012 16:06:49 +0100

On 04/07/2012 15:59, Murray S. Kucherawy wrote:
> On Tue, Jul 3, 2012 at 1:10 PM, Barry Leiba <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> Was this general two-format header thing done deliberately,
Yes!
> or were their competing implementations or specifications at some time 
> in the past and they were just merged?
>
> I basically concur with Barry, that having two formats for something 
> is just bad form.  We should pick one and go with that.
>
> But I think that idea becomes pretty hard to enforce when apparently, 
> with HTTP, that ship has already unfortunately sailed and parsers to 
> handle both formats are widely deployed.  Converging on one is bigger 
> than this document.
>
> It would be nice of this got "fixed" at a lower level in httpbis, but 
> I'll shut up now as I feel them turning their guns on me.  :-)

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 4 Jul 2012 17:48:07 +0200

> competing implementations or specifications at some time in the past and

I think it dates at least back to SMTP, where you very commonly see this with
the Cc and To headers. Also, keep in mind that the two formats have a real use,
to limit the max length of a given header field value without forcing everyone
to send multiple occurrences each on their own line.

Regards,
Willy

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 4 Jul 2012 21:23:57 -0700

> to send multiple occurrences each on their own line.
I'm not sure that's a valid comparison.  You can't have more than one To or
Cc field with mail.  The only option is to update the one that's there, or
add one if it's missing.

In this case we're talking about forwarding, and the path requests take.
That's analogous to trace fields in email, and you're never allowed to
modify those.  You always add new ones.

-MSK

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 5 Jul 2012 08:20:57 +0200

> Cc field with mail.  The only option is to update the one that's there, or

Well, then maybe it's not valid but my mail agent (mutt) perfectly
recognizes multiple headers and happily merges them. I've been used
to do this for sending e-mails for a long time, maybe I took a bad
habit for something non-standard.

> modify those.  You always add new ones.

Yeah, just like the Received: header. However, just like the HTTP
Set-Cookie header, it's non-foldable because it contains a date which
embeds a comma.

Regards,
Willy

---

From: Barry Leiba <[email-masked]>
Date: Fri, 6 Jul 2012 14:44:28 -0400

Roy Fielding posted the following response on the httpbis mailing
list, and I'm re-posting it here, so everyone sees it and so I can
respond to it in context.
-----------------------------------------------------

Regarding your comments on Forwarded:

> I find this confusing,

I find that confusing too.  It should just say that the new value
MUST either be appended to the existing field-value or appended
as a new field at the end of the header block.

> and I question the value of having more than
> and a list, rather than adding a new instance?

For performance and implementation reasons.  Some proxies will store
the field-value as a compounded hash of ordered list values until some
generic routine writes the value as a single field; some will have already
sent the received header fields downstream and thus have no other
choice but to append as a new fieled; others will choose to append
a separate Forwarded at the end in order to avoid buffer copies
or for easier debugging.

I don't think an HTTP header field needs to define one way to do it.
We have left it unspecific in order to maximize flexibility in
implementation.

> If a proxy later wants
> some had been added as list items into another header field?

I assume by looking at the information it wants to selectively remove.
It would be more common to simply delete the existing field and create
a new one on the way downstream.

>  Doesn't it
> complicate things to have to support various combinations?

Yes, but that is governed by HTTP/1.1 parsing rules.

> How would I

They mean the same thing in HTTP (as in RFC822, MIME, etc.).

> mechanism, but we have a chance to be specific in a new specification.

No, we don't.  HTTP header fields cannot redefine HTTP parsing.
All this field is doing is restating (unnecessarily) the facts
implied by the list rule in HTTP field parsing.

> "proto=http" to be *stronger* than the break after "proto=http", and

People don't read HTTP header fields.  This precedence is a leftover
from MIME field parsing and it makes far less sense to create
specialized parsers for each new field defined.  The only header fields
that got it wrong are Cookie and Set-Cookie, which resulted in more
interoperability problems than all of the rest of HTTP combined.

....Roy

---

From: Barry Leiba <[email-masked]>
Date: Fri, 6 Jul 2012 14:53:10 -0400

> as a new field at the end of the header block.

OK... we need to clean up the text a bit; see below.

> We have left it unspecific in order to maximize flexibility in

I continue to wish that HTTP had not defined the multiple-field-values
situation the way it did, but through the course of this conversation
with Julian, Willy, and now Roy, I have been convinced that it is
defined how it is, and this is not the place to try to deal with that
at all.

My request now turns into one that asks for tighter text, making the
instructions for adding and removing these fields as simple and
clearly explained as possible.  Andreas, I think you have a bunch of
input for this.  Please address my other comments along with this one
and the result of your wordsmithing with Stephen, and post an -06
version for us to review and -- I hope -- proceed with.

If you need help getting the words right for a clearer version of what
we're discussing here, post to this thread and I'm sure help will soon
arrive.
:-)

Thanks, everyone, for the input, and for convincing me that I was
fighting the wrong battle.

Barry