Subject: [apps-discuss] Feedback about "Update to MIME regarding Charset Parameter Handling in Textual Media Types"

From: Henri Sivonen <[email-masked]>
Date: Wed, 22 Feb 2012 15:25:02 +0200

In reference to
https://svn.tools.ietf.org/svn/wg/appsawg/draft-ietf-appsawg-mime-default-charset/latest/draft-ietf-appsawg-mime-default-charset.html

First of all, thank you for finally taking on the much-needed update
to RFC 2046 rules.

Unfortunately, the draft doesn't address the problem the right way in
my opinion. Quotes from the draft.

> Each subtype of the "text" media type which uses the "charset" parameter can define its own default value for the "charset" parameter, including absence of any default.

Additionally, media types should be able to define circumstances where
in-band indicators override the charset parameter even if the charset
parameter is present.

In particular, media types should be allowed to override the charset
parameter if the first two or three bytes of the payload look like an
UTF-16 or UTF-8 BOM.

See:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=15359
https://bugzilla.mozilla.org/show_bug.cgi?id=716579
https://bugzilla.mozilla.org/show_bug.cgi?id=687859

> In order to improve interoperability with deployed agents, "text/*" media type definitions SHOULD either a) specify that the "charset" parameter is not used for the defined subtype, because the charset information is transported inside the payload (as in "text/xml")

This seems wrong. If the charset parameter is present, it has an
effect for text/xml.

> or b) require explicit unconditional inclusion of the "charset" parameter eliminating the need for a default value.

This seems naïve. Formats need to specify what happens when a charset
parameter is missing, since no matter how much the format says it's
"required", the party sending data can omit the charset parameter.

> In accordance with option (a), above, "text/*" media types that can transport charset information inside the corresponding payloads, specifically including "text/html" and "text/xml", SHOULD NOT specify the use of a "charset" parameter, nor any default value, in order to avoid conflicting interpretations should the charset parameter value and the value specified in the payload disagree.

For backwards compatibility, pretty much every existing text/* type
will have to violate this "SHOULD NOT".

> New subtypes of the "text" media type, thus, SHOULD NOT define a default "charset" value. If there is a strong reason to do so despite this advice, they SHOULD use the "UTF-8" [RFC3629] charset as the default.

Seems reasonable.

> Specifications of how to specify the "charset" parameter, and what default value, if any, is used, are subtype-specific, NOT protocol-specific.

Seems reasonable.

> Protocols that use MIME, therefore, MUST NOT override default charset values for "text/*" media types to be different for their specific protocol. The protocol definitions MUST leave that to the subtype definitions.

Seems reasonable.

> The default charset parameter value for text/plain is unchanged from [RFC2046] and remains as "US-ASCII".

This is incompatible with reality. Web browsers, for instance, assume
a configuration-dependent default (which correlates with browser
localization) and may also (depending on configuration which, again,
correlates with localization by default) perform a heuristic analysis
on the payload.

I suggest specifying the following instead of sections 3 and 4 of the draft:

3. New rules for determining the character encoding for text/* media types

Each text/* media type MUST specify an algorithm for establishing the
character encoding of the entity body from the entity body (or
preferably the first N bytes thereof, preferably with N = 1024), the
charset parameter and Other Information. Other Information MAY include
configuration, an encoding label supplied by the referrer, the
previous encoding of an entity body retrieved from the same location
or the encoding of the referrer. New text/* media types MUST not use
Other Information in the algorithms they specify. New text/* media
types SHOULD use the following algorithm:

The character encoding is UTF-8. Terminate these steps.

4. Determining the character encoding for text/plain

If the first 2 octets of the entity body are 0xFE followed by 0xFF,
the character encoding is big-endian UTF-16. Terminate these steps.

If the first 2 octets of the entity body are 0xFF followed by 0xFE,
the character encoding is little-endian UTF-16. Terminate these steps.

If the first 3 octets of the entity body are 0xEF followed by 0xBB
followed by 0xBF, the character encoding is UTF-8. Terminate these
steps.

If the value of the charset parameter is a ASCII case-insensitive[1]
match for a label[2] of a supported encoding, the character encoding
is the encoding whose label was matched. Terminate these steps.

If the entity is being navigate to in a browsing context[3] and the
previous document had the same origin[4] as the text/plain entity, the
character encoding is the encoding of the referring document.
Terminate these steps. (Disclaimer: I'm not 100% sure that this step
is in the right order relative to the others.)

Optional: If a heuristic detector recognizes the octets of the entity
body as being encoding according to an encoding, the character
encoding is that encoding. Terminate these steps. This step SHOULD NOT
be implemented for locales where it has not been implemented
traditionally.

If the entity is being loaded into a nested browsing context that has
the same origin as the parent browsing context, the encoding is the
encoding of the document loaded in the parent browsing context.
Terminate these steps.

If the entity is being loaded into a browsing context and is being
fetched from a location from which an entity has been loaded before
and the previous character encoding has been cached, the character
encoding is the cached encoding. Terminate these steps.

If the entity is being loaded via a non-browsing context mechanism
(such as XMLHttpRequest) that defines a fallback encoding, use that
encoding. Terminate these steps.

Otherwise, the character encoding is a configuration-dependent
encoding. The default configuration SHOULD depend on the locale of the
user agent according to the table given in step 8 in [5]. Terminate
these steps.

[1] http://www.whatwg.org/specs/web-apps/current-work/#ascii-case-insensitive
[2] http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html#concept-encoding-label
[3] http://www.whatwg.org/specs/web-apps/current-work/#browsing-context
[4] http://tools.ietf.org/html/rfc6454
[5] http://www.whatwg.org/specs/web-apps/current-work/#determining-the-character-encoding

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 23 Feb 2012 14:53:22 +0900

Hello Henri, Alex,

Some comments to Henri and some to Alex inline.

>> Each subtype of the "text" media type which uses the "charset" parameter can define its own default value for the "charset" parameter, including absence of any default.

I'm not opposed to this in principle, but I'm worried that as written, 
it will lead to more and more unnecessarily different 'algorithms' and 
heuristics. The spec should make clear that where possible, established 
patters should be followed to reduce variety.

> This seems wrong. If the charset parameter is present, it has an

Yes indeed.

> will have to violate this "SHOULD NOT".

Yes. I think it's inappropriate to give such advice on existing formats. 
If we agree that changes are needed (which I think in the cases at hand 
would be somewhat wishful thinking), they should be specified directly, 
either in this document or preferably in an update to the spec of the 
format itself.

> 4. Determining the character encoding for text/plain

This looks like a very long algorithm. It may work pretty well in a Web 
context (definitely better than an US-ASCII default), but what about 
other contexts (e.g. mail)?

[Part of the algorithm cut out to save electrons.]

> user agent according to the table given in step 8 in [5]. Terminate

What is missing here is that there is sometimes a need for user 
intervention if all else fails. While it may be possible to look at this 
as something outside of this algorithm, or as something subsumed under 
"configuration" or whatever, I'm afraid that the current text will give 
many implementers the impression that user overrides are not allowed. So 
I suggest adding something like:

    In an interactive context, the user may occasionally want to
    override the character encoding determined by this algorithm.

Regards,    Martin.

---

From: Ned Freed <[email-masked]>
Date: Wed, 22 Feb 2012 19:59:01 -0800 (PST)

> in-band indicators override the charset parameter even if the charset

That's a terrible way to do it - if the type is self-identifying in terms of
charset, a charset parameter should simply not be defined for the type -
exactly what the current specification says to do. The result is effectively
the same as what you're proposing since undefined parameters are supposed to be
ignored. The difference is your approach is effectively sanctioning active
mislabelling, and that's a road we've already explored way too much in an
informal fashion, especially in the CJK arena. Formalizing it is the last
thing we should be doing.

> parameter if the first two or three bytes of the payload look like an

There are quite a few charsets in existence where it is perfectly permissible
for the first few bytes to match a BOM, except that it means something entirely
different.

> > In order to improve interoperability with deployed agents, "text/*" media
> type definitions SHOULD either a) specify that the "charset" parameter is not
> used for the defined subtype, because the charset information is transported
> inside the payload (as in "text/xml")

> This seems wrong. If the charset parameter is present, it has an

That's only because the definition of test/xml did it incorrectly. I'll grant
you, however, that the example is a little wierd. What it should say is that
XML-based formats can and should self-identify the format. But this has no
technical effect on the specification.

> > or b) require explicit unconditional inclusion of the "charset" parameter
> eliminating the need for a default value.

> "required", the party sending data can omit the charset parameter.

Yep. They can also misspell the parameter name, misspell the charset name, use
the wrong type/subtype, misspell the header field name, omit the field
entirely, or any of a million other mistakes.

And when (not if) this sort of thing happens, the receiver can elect to pursue
whatever course of action it deems appropriate for invalid material. It can
reject it, refuse to show it, use a default, sniff it and make a guess, prompt
the user and ask what to do, whatever. Not allowing specification of a default
in the type registration has no impact at all on the options an implementation
has available. In fact it's the other way around - allowing specifiction of a
default *limits* implementation choices.

We learned long ago through multiple painful experiences that attempting to
specify the handling of invalid stuff is an enormous rathole that is best given
wide berth. And even if the cases where it is successful in producing some sort
of advice, or worse making what should be illegal legal, that advice or
approach is usually bound to a place and time and become the wrong advice or
approach at some point in the future. So, far from being naive, it is your
approach that has been shown, over and over, to the the naive one.

> will have to violate this "SHOULD NOT".

Yep. That's the main reason why it needs to be a SHOULD.

> > [RFC2046] and remains as "US-ASCII".

Note that this has the effect of rendering any content that contains 8bit
as being invalid. Implementation are then free to choose how to handle
that, as above. Of course this leaves 7bit charsets in the lurch, notably
the iso-2022-* group, which is the main reason why I'm not happy with
the US-ASCII default, as I mention below.

> correlates with localization by default) perform a heuristic analysis

Well, on this one you'll have to argue with someone else. I don't especially
like the approach in the draft, but I have been unable to come up with any
reasonable alternative. Your proposed alternative below, is totally unworkable.

> Other Information in the algorithms they specify.
> New text/* media types SHOULD use the following algorithm:

> The character encoding is UTF-8. Terminate these steps.

This approach is ridiculous when you consider how general the usage of some
text types are and  the close similarity of many charsets in common use. It is
not at all common for there to only be one or two characters that in a large
document that display incorrectly when the wrong charset is selected.

> the character encoding is little-endian UTF-16. Terminate these steps.

See my note above about BOMs.

> followed by 0xBF, the character encoding is UTF-8. Terminate these

I've checked and I have yet to receive a single message in utf-8 with a leading
BOM. But this sequence can show up in material in other charsets.

> is the encoding whose label was matched. Terminate these steps.

So the label is secondary to sniffing. Totally unacceptable. It is also
unacceptable to ignore a valid label just because you don't support 
the encoding.

> user agent according to the table given in step 8 in [5]. Terminate

Finally, I'd say the odds of any implementor following such a complex
set of rules is remote in the extreme.

> [5] http://www.whatwg.org/specs/web-apps/current-work/#determining-the-character-encoding

				Ned

---

From: Henri Sivonen <[email-masked]>
Date: Thu, 23 Feb 2012 11:51:35 +0200

On Thu, Feb 23, 2012 at 7:53 AM, "Martin J. Dürst"
<[email-masked]> wrote:
> I'm not opposed to this in principle, but I'm worried that as written, it
> will lead to more and more unnecessarily different 'algorithms' and

I suggesting that new stuff be UTF-8 only.

For old stuff, everything is a bit different. Off the top of my head:
text/javascript uses HTTP charset or the BOM or declaration from the
referrer (the charset="" attribute on <script>) or the encoding of the
referrer
text/css uses what text/javascript uses or an in-band declaration
text/plain uses what I described in my message (well, strictly what I
said was a synthesis of what Gecko, IE and WebKit do; Gecko will do it
when I get around to changing the precedence of the BOM to work as in
WebKit and IE)
text/html uses what text/plain uses or an in-band declaration
text/xml uses HTTP charset or the BOM or an in-band declaration

text/cache-manifest is sane an always uses UTF-8 (HTTP charset is ignored).

For new stuff, I'd much rather recommend the text/cache-manifest
pattern than the BOM/HTTP charset/in-band commonality of text/css,
text/html and text/xml.

> context (definitely better than an US-ASCII default), but what about other
> contexts (e.g. mail)?

I don't see why mail would need to be different. There are just some
steps in the algorithm (same-origin navigation and XMLHttpRequest or
similar) that never apply to mail.

> What is missing here is that there is sometimes a need for user intervention
> if all else fails. While it may be possible to look at this as something
> outside of this algorithm, or as something subsumed under "configuration" or
> whatever, I'm afraid that the current text will give many implementers the
> impression that user overrides are not allowed. So I suggest adding
> something like:
>

Right. User override should come right after the BOM steps.

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Henri Sivonen <[email-masked]>
Date: Thu, 23 Feb 2012 12:11:56 +0200

On Thu, Feb 23, 2012 at 5:59 AM, Ned Freed <[email-masked]> wrote:
> That's a terrible way to do it

It's reality in WebKit and IE for text/html. So far, evidence suggests
that Gecko would serve users better if it changed to treat the BOM as
having higher precedence than the HTTP-level charset parameter for
text/html and text/javascript. (text/plain is loaded using the HTML
parser per the HTML spec and per reality, so it makes sense to do the
same for text/plain even though I don't have evidence to show about
compatibility issues either way.)

> - if the type is self-identifying in terms of
> exactly what the current specification says to do.

Logically, yes, but that's not how text/html, text/xml and text/css work.

> for the first few bytes to match a BOM, except that it means something entirely

Seems to work for IE and WebKit in practice.

> That's only because the definition of test/xml did it incorrectly.

There's text/xml legacy by now, so chances are it's too late to make
HTTP charset have no effect for text/xml.

>> This seems naïve.

It's awesome that the IETF's own Web interface to the list archive
mangles the ï here. Yay dogfood.

>> Formats need to specify what happens when a charset
> whatever course of action it deems appropriate for invalid material.

No. To achieve interoperability, the behavior in error cases needs to
be well-defined. Broken content will depend on the behavior of
incumbent consumers. If that behavior isn't specified, new entrants to
the market face a barrier to competition when they need to reverse
engineer incumbents instead of just reading specs to see what needs to
be done.

> as being invalid.

I'm ok with defining the absence of the charset parameter for
text/plain invalid if the BOM is also absent.

> reasonable alternative. Your proposed alternative below, is totally unworkable.

Why is it unworkable. It's what Gecko does with the modification that
the precedence of the BOM is as in WebKit and IE.

> text types are and  the close similarity of many charsets in common use.

The UTF-8-only recommendation is for new text/* types which (by
definition of "new") don't have any charsets in common use previously.

> It is

You mean like "naÃve" in the archive copy of your email? ;-)

> It is also

That's how browsers work.

> set of rules is remote in the extreme.

As an implementor intending to implement the steps, I disagree. Gecko
already implements those steps with the BOM steps in a different order
relative to the charset parameter. I expect to patch Gecko to make the
BOM take precedence over the charset parameter in due course to match
IE and WebKit in text/html handling (and text/plain reuses the
text/html code path).

-- 
Henri Sivonen
[email-masked]
http://hsivonen.iki.fi/

---

From: Ned Freed <[email-masked]>
Date: Thu, 23 Feb 2012 08:42:33 -0800 (PST)

> It's reality in WebKit and IE for text/html.

You continue to confuse what implementations are allowed to do with what
registrations are allowed to contain. This is the fundamental error you make
repeatedly.

Again, if the labelling is wrong, the object is incompliant and
*implementations* are free to handle that case however they wish. But this
doesn't mean we should codify specific implementation practice in our
registratration process. In fact, as I said previously, this has the effect of
*limiting* implemetaitons options, so when the world changes, as it always
does, we end up with inflexible standards incapable of handling that new
reality.

If you want a specific example of why this sort of thing is a bad idea, you
need look no further than the original HTTP specifications. Those
specifications changed the default charset for text/plain from us-ascii to
iso-8859-1 for HTTP. It may have been a good idea at the time (or not), but how
about now?

> So far, evidence suggests
> text/html and text/javascript.

You're missing a critical qualifier here: *current* evidence. It may be true
now. What happens if when practices change and we end up stuck with a bunch of
advice that doesn't match whatever practices show up in the future?

> (text/plain is loaded using the HTML

> Seems to work for IE and WebKit in practice.

It doesn't work well at all in my experience, but even if you're generally
right at present, what guarantees can you offer for tomorrow?

Anyway, I see no point in continuing to argue this further. I've registered my
strong objections to all but one of your proposed changes, and that's
sufficient. I'm done.

				Ned

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Feb 2012 23:41:34 +0100

> https://svn.tools.ietf.org/svn/wg/appsawg/draft-ietf-appsawg-mime-default-charset/latest/draft-ietf-appsawg-mime-default-charset.html

Henri,

thanks a lot for the feedback.

I believe that one thing that makes the current spec a bit confusing is 
that part of it sound like we're changing the defaults for existing 
types that specify their own defaults. We are not; *but* we are changing 
the framework so that new types can be defined in a saner way, and also 
so that (maybe) the definitions for text/xml and text/html can be tuned 
in the future. (Not sure about text/plain)

So, maybe add this as a clause to the abstract; and also clarify the 
prose where we mention text/xml and text/html?

Also, I see that you would like us to sanction whatever algorithm the 
HTML5 spec has come up with for handling text/html; potentially 
including content inspection and out-of-band information.

I think out-of-band information is a non-starter; it just doesn't work 
for many environments, and breaks basic assumptions about message 
semantics; if you need to do that for text/html so be it, and label it a 
"willful violation", but by all means don't let this hack leak into new 
media type definitions.

Another change I want to do is to add to:

"New subtypes of the "text" media type, thus, SHOULD NOT define a 
default "charset" value. If there is a strong reason to do so despite 
this advice, they SHOULD use the "UTF-8" [RFC3629] charset as the default."

a sentence that if it's not UTF-8, it MUST be an ASCII-compatible encoding.

Feedback appreciated,

Julian

---

From: Ned Freed <[email-masked]>
Date: Tue, 28 Feb 2012 18:19:13 -0800 (PST)

> "willful violation", but by all means don't let this hack leak into new

Agreed.

> a sentence that if it's not UTF-8, it MUST be an ASCII-compatible encoding.

I think that's a very good addition to make.

				Ned

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 29 Feb 2012 11:34:56 +0900

> I think that's a very good addition to make.

Do we really, in this day and age, expect a new type to define an ASCII 
compatible encoding that's not UTF-8 as a default?

If the MUST is some basic Mime requirement, then I don't see any reason 
to repeat it here. If it's not, then I don't see why it couldn't be 
UTF-16(LE|BE) or some such. In any case, I don't see why we need to be 
that specific, we are already outside a SHOULD NOT and then outside a 
SHOULD, and that isn't supposed to happen very often.

Regards,   Martin.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Feb 2012 11:02:55 +0100

> compatible encoding that's not UTF-8 as a default?

I hope not. If we were 100% sure, the "SHOULD be UTF-8" would be a "MUST 
be UTF-8".

> SHOULD, and that isn't supposed to happen very often.

It could be considered a requirement to not break software too much that 
has been written with the previous ASCII default in mind. As such, I 
really believe that using something like UTF-16 without labeling it as 
such would be a bad bad idea.

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Wed, 29 Feb 2012 18:11:13 -0800 (PST)

> compatible encoding that's not UTF-8 as a default?

I have no idea. There's lots of legacy charset usage out there, some extremely
entrenched. Better safe than sorry.

> to repeat it here.

It's a requirement for MIME, yes. But speaking as the types reviewer, expecting
people to read even the primary registration documents has proved to be a bit
problematic. Expecting them to dig a detail out of a not-obviously-related
document? Possible, but very unlikely.

It's not like the MIME requirement is going to change, either, so there's no
cost to duplicating it. The benefit is that of possibly saving someone a
reviewer round trip.

> If it's not, then I don't see why it couldn't be

Hopefully not. But it's hard to predict how this stuff will play out.

				Ned