Subject: Re: [apps-discuss] Last Call: <draft-ietf-appsawg-mime-default-charset-03.txt> (Update to MIME regarding Charset Parameter Handling in Textual Media Types) to Proposed Standard

From: Barry Leiba <[email-masked]>
Date: Tue, 8 May 2012 10:06:30 -0400

> Abstract
>   This document changes RFC 2046 rules regarding default charset
>   parameter values for text/* media types to better align with common
>   usage by existing clients and servers.
...
> The file can be obtained via
> http://datatracker.ietf.org/doc/draft-ietf-appsawg-mime-default-charset/
>
> IESG discussion can be tracked via
> http://datatracker.ietf.org/doc/draft-ietf-appsawg-mime-default-charset/ballot/

This document sailed through IETF last call with no comments, but
something has come up in IESG evaluation -- Robert chatted with me
about this, and I suggested that he put a DISCUSS ballot in until we
resolve it.

The document makes it very clear that "existing registrations" are not
affected (and therefore retain their RFC 2046 default of US-ASCII for
charset).  But how does someone TELL that a subtype is "existing"?
Chasing documentation pointers and checking dates is not a reasonable
way.  Five years from now, when 60 more text subtypes have been added
to the 60 that are there, how will anyone know *which* of those 120
are affected by this spec and which are not?

Further, both (a) and (b) in section 3 are things that SHOULD be done;
what if a new registration does neither, violating both SHOULDs?  How
will someone know what its default is?

I think the right way to fix this is to put new text in near the end
of section 3, just to make things absolutely clear:
--------
OLD
  New subtypes of the "text" media type, thus, SHOULD NOT define a
  default "charset" value.  If there is a strong reason to do so
  despite this advice, they SHOULD use the "UTF-8" [RFC3629] charset as
  the default.

NEW
  New subtypes of the "text" media type, thus, SHOULD NOT define a
  default "charset" value.  If there is a strong reason to do so
  despite this advice, they SHOULD use the "UTF-8" [RFC3629] charset as
  the default.

  To maintain compatibility with existing registrations, this fallback rule
  applies: any subtype of the "text" media type that does not comply with
  the rules above retains US-ASCII as its default, as originally specified
  in RFC 2046.
--------

The document editors are OK with this text, but we want to pass it by
the working group for comment.  Does anyone object to this suggestion?
 Does anyone think the issue should be addressed differently?

Barry, Applications AD

---

From: Ned Freed <[email-masked]>
Date: Tue, 08 May 2012 14:09:32 -0700 (PDT)

> charset).  But how does someone TELL that a subtype is "existing"?

This was brought up and resolved during last call comments.

The current rules allow a text/* media type to say anything about charsets,
and if it allows a optional charset parameter, there's no need to state what
the default is.

The new rules say that a media type either

(1) Specifies that no charset parameter is used and that the charset is
    determined from inspection of the content, or

(2) Requires inclusion of a charset parameter specifying what the charset
    is, or

(3) Explicitly states what the default charset is. (Either with or without
    allowing an optional charset parameter as a means of overriding the
    default.)

The last option is dicouraged (SHOULD NOT).

No other options are given, so it follows that a media type that fails to do
one of these things is supposed to be rejected by the reviewer.

As such, there is no possibility of confusion. An old text/* media type can get
away without specifying charset information and relying on the language in RFC
2045; a new one cannot.

> are affected by this spec and which are not?

A nonissue. See above.

> will someone know what its default is?

And this is precisely why it is so important to be able to use compliance
language in registration RFCs. SHOULD means "do it unless you have a good
reason not to". In this case this isn't a judgement call made by an
implementor; the question is whether or not you can convince the reviewer you
have sufficient cause to violate the SHOULD.

>   the rules above retains US-ASCII as its default, as originally specified

That's a really bad idea for all sorts of reasons, including but not limited to
it makes the document self-contradictory. You shouldn't remove a rule then say
it's OK to fall back on it.

Now, if you want to add something that says:

  Regardless of the approach chosen, all text/* registrations MUST clearly
  specify how the charset of the content is determined and MUST NOT rely
  on the RFC 2045 rule.

I think this falls out of the existing text, but if you want to make it crystal
clear I don't have a problem.

> the working group for comment.  Does anyone object to this suggestion?

I strongly object.

>  Does anyone think the issue should be addressed differently?

See above.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 08 May 2012 18:02:13 -0700 (PDT)

> The current rules allow a text/* media type to say anything about charsets,

Correction:

"anything" -> "nothing"

> 				Ned
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

---

From: Barry Leiba <[email-masked]>
Date: Tue, 8 May 2012 21:18:20 -0400

>    allowing an optional charset parameter as a means of overriding the

The document makes all of those SHOULD.  From section 3:

   In order to improve interoperability with deployed agents, "text/*"
   media type registrations SHOULD either

   a.  specify that the "charset" parameter is not used for the defined
       subtype, because the charset information is transported inside
       the payload (such as in "text/xml"), or

   b.  require explicit unconditional inclusion of the "charset"
       parameter eliminating the need for a default value.

There's nothing that I read in this document that says that
registrations MUST do anything with regard to charset parameters.
Now, you may say that the designated expert would never accept a
registration that didn't (and well you might say that), but someone
looking at the registrations and their associated documentation won't
know that.

> away without specifying charset information and relying on the language in RFC

There is absolutely a possibility of confusion.  Someone who doesn't
know you and the process well may see something registered and be
completely uncertain whether it's old (and therefore gets US-ASCII as
its default) or whether it's new and doesn't comply with the SHOULDs
here (and therefore gets &deity only knows what as its default).

> the question is whether or not you can convince the reviewer you

And if you do so convince the reviewer, what does the default value
wind up being?  And, again, how's an arbitrary person to know?

> it's OK to fall back on it.

Well, I disagree with that.  No matter, see below.

>  specify how the charset of the content is determined and MUST NOT rely

I'd change that to "all new text/* registrations", and I still want to
add a sentence that says that existing text/* registrations that don't
specify this retain the 2045 rule.

> clear I don't have a problem.

I think it does not fall out of the existing text at all.

Barry

---

From: Ned Freed <[email-masked]>
Date: Tue, 08 May 2012 19:33:43 -0700 (PDT)

> registrations MUST do anything with regard to charset parameters.

I never said it did. The requirement falls out of the options this document
provides coupled with something I thought was obvious: The requirement that
registrations not be obviously ambiguous.

> looking at the registrations and their associated documentation won't

So what you're saying is that someone could look at this document and conclude
that it allows inherently ambiguous registrations?

I guess I give the people who'd actually bother to read these specifications at
this level of detail a bit more credit than that, but fair enough.

> here (and therefore gets &deity only knows what as its default).

This has nothing to do with me. I would expect any reviewer to perform this
level of checking. In fact one of the reasons expert review was instituted was
that IANA internal review wasn't familiar enough with the technical details and
thus was incapable of performing these sorts of checks.

> wind up being?  And, again, how's an arbitrary person to know?

Well, as it happens a similar case has come up in the past, so I don't have to 
guess. There was a text media type that was intended for use with video
subtitles, where the format of the text was controlled by bilateral agreement
between the sender and the receiver. So the security considerations could not
be specified because they were entirely dependent on the format arranged by
that agreement (could contain dangerous active content, might be nothing but
ASCII, or anything in between). But the closest thing we allow for is to say
"security considerations have not be assessed", which doesn't really fit.

In that case I allowed the registration to say "... the security considerations
of this type cannot be assessed because the format is determined by bilateral
agreement ...". But what would never be allowed is to do something similar with
a charset definition without saying *why* it was being done.

> specify this retain the 2045 rule.

It's still a bad idea to continue to rely on the old rule in any way. The other
problem with the RFC 2045 rule is that it fell apart the minute HTTP overrode
it and said that text/html without a charset parameter defaults to iso-8859-1.
So right there you have what is likely the most commonly used subtype of text
at this point breaking the rule you're now saying is what old types fall back
to. Talk about confusing! (Really, in regards to that rule, all this document
is doing is formalizing something that's been true for almost two decades.)

So how about:

   Rgardless of the approach chosen, all new text/* registrations MUST
   clearly specify how the charset of the content is determined; relying
   on the RFC 2045 is no longer permitted. However, existing text/*
   registrations that fail to specify how the charset is determined still
   default to US-ASCII.

Note that this is different than the old rule, which says that if no charset
parameter is present the charset must default to us-ascii.

And we probably want to update the main registration document with something
similar.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Wed, 9 May 2012 06:54:19 -0400

>   registrations that fail to specify how the charset is determined still

That works for me.

Barry

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 May 2012 13:04:11 +0200

> this level of detail a bit more credit than that, but fair enough.

+1

> it and said that text/html without a charset parameter defaults to iso-8859-1.

Actually, text/*. But we have fixed this in httpbis 
(<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/20>).

>     registrations that fail to specify how the charset is determined still

Sounds good to me.

> And we probably want to update the main registration document with something

I consider this document to be an RFC-ized erratum. If we're going to 
update RFC 2045, it should include the changes we're doing here, and 
obsolete this document (-> historic), right?

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 May 2012 05:52:05 -0700 (PDT)

> (<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/20>).

A *very* good thing IMO.

> obsolete this document (-> historic), right?

That's a good way to look at it. And yes, if RFC 2045 gets updated, we'll
need to replace what's there with some version of this.

				Ned

---

From: t.petch <[email-masked]>
Date: Fri, 11 May 2012 14:59:56 +0200

What charset should be used to inspect the content?  Should this be defined in
this I-D?

The new rules say that
"charset" parameter is not used for the defined
       subtype, because the charset information is transported inside
       the payload (such as in "text/xml")

 But a charset is needed in order to read the payload and determine that
charset.  Catch-22?

Tom Petch

----- Original Message -----
From: "Ned Freed" <[email-masked]>
To: "Barry Leiba" <[email-masked]>
Cc: "Ned Freed" <[email-masked]>; <[email-masked]>
Sent: Wednesday, May 09, 2012 4:33 AM

---

From: Ned Freed <[email-masked]>
Date: Fri, 11 May 2012 07:42:21 -0700 (PDT)

> What charset should be used to inspect the content?

That depends on the syntax of the content. For every syntax that supports
extraction of charset information, there are clearly defined rules for
how this is done.

> Should this be defined in this I-D?

It can't be. The rules can be syntax-specific.

>  But a charset is needed in order to read the payload and determine that

Nope, nothing of the sort. For, say, an XML-based type, only certain sorts of
charsets are allowed. Given these constraints, you look at the first few bytes;
these determins what general sort of charset you're  dealing with (one versus
two octet, say). Then you have enough information to extract the charset label.

				Ned

---

From: Julian Reschke <[email-masked]>
Date: Fri, 11 May 2012 16:49:11 +0200

>   But a charset is needed in order to read the payload and determine that

No. Just not trivial.

See for instance <http://www.w3.org/TR/REC-xml/#sec-guessing>; HTML5 has 
similar (but surely more complex) rules.

Best regards, Julian

---

From: t.petch <[email-masked]>
Date: Fri, 11 May 2012 16:02:12 +0200

---- Original Message -----
From: "Julian Reschke" <[email-masked]>
To: "t.petch" <[email-masked]>
Cc: "Barry Leiba" <[email-masked]>; "Ned Freed"
<[email-masked]>; <[email-masked]>
Sent: Friday, May 11, 2012 4:49 PM

> similar (but surely more complex) rules.

Julian

Right, thanks for the reference.
"In the general case, this is a hopeless situation. It is not entirely hopeless
in XML, however, because XML limits the general case in two ways"

But ... that is saying that XML recognises the need to be able to auto-detect
the character encoding and is designed to do so.  This change to mime default
charset imposes the need to auto-detect on all future text/* where the encoding
is transported inside the payload, which the old rules did not.  Perhaps a
corner case not worth mentioning.

Tom Petch

---

From: Julian Reschke <[email-masked]>
Date: Fri, 11 May 2012 17:12:04 +0200

> corner case not worth mentioning.

a) The spec has been approved yesterday, so changing it would be tricky.

b) That being said: when a format is designed to carry the character set 
information in-line, then yes, that's should be used to detect the 
character set. If it's not possible to do so reliably, then the format 
doesn't really fall into that category, no?

Best regards, Julian

---

From: Ned Freed <[email-masked]>
Date: Fri, 11 May 2012 08:59:30 -0700 (PDT)

> is transported inside the payload, which the old rules did not.

It does nothing of the sort. The preferential use of inline information is a
SHOULD, not a MUST. If a given syntax carries such information inline but
doesn't define a reliable standalone way to extract it (it depends on
additional external information, for example), that would be more than
sufficient reason not to use inline information.

> Perhaps a corner case not worth mentioning.

It's not even that.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Fri, 11 May 2012 09:46:30 -0700 (PDT)

Thinking about it more, the far more likely scenario is that a slot for
inline information is available but existing applications don't use it and
cannot be upgraded to use it.

				Ned