Subject: Re: AddDefaultCharset considered harmful (Re: Requesting a revision of RFC3023)
From: Tim Bray <[email-masked]>
To: Martin Duerst <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Date: Thu, 25 Sep 2003 13:45:42 -0700
List-Id: <ietf-xml-mime.imc.org>

Martin Duerst wrote:

> I have filed a bug with Apache bugzilla, asking that the
> AddDefaultCharset setting in the httpd.conf file when shipped
> be removed/commented out, and the comment fixed, at
> http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23421.

+1, if I can help by chiming in anywhere let me know.  BTW, I actually 
use this directive at 'ongoing' (I set it to UTF-8), it works perfectly 
for a website which is a dictatorship and everything is under the steely 
control of one person and one publishing program, but I'm sure that is a 
minority situation.  -Tim

Date: Thu, 25 Sep 2003 16:22:20 -0400
To: Tim Bray <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: AddDefaultCharset considered harmful (Re: Requesting a revision of RFC3023)
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

I have filed a bug with Apache bugzilla, asking that the
AddDefaultCharset setting in the httpd.conf file when shipped
be removed/commented out, and the comment fixed, at
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23421.

I have commented on a related bug that I found, at
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=14513.

I'm copying this list because what we have discussed here is
related. In particular, the comment in httpd.conf file
seems to say some quite strange things:

 >>>>>>>>
#
# Specify a default charset for all pages sent out. This is
# always a good idea and opens the door for future internationalisation
# of your web site, should you ever want it. Specifying it as
# a default does little harm; as the standard dictates that a page
# is in iso-8859-1 (latin1) unless specified otherwise i.e. you
# are merely stating the obvious. There are also some security
# reasons in browsers, related to javascript and URL parsing
# which encourage you to always set a default char set.
#
AddDefaultCharset ISO-8859-1
 >>>>>>>

Regards,    Martin.

Reply-To: <[email-masked]>
From: "Addison Phillips [wM]" <[email-masked]>
To: "MURATA Makoto" <[email-masked]>
Cc: "WWW-Tag" <[email-masked]>, <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Wed, 24 Sep 2003 13:20:38 -0700
Importance: Normal
Sender: [email-masked]

Hi Makoto,

Thanks for your response! I'm happy to contribute as best I can.
>
> This issue was discussed when RFC 2376 was developed.  I recall
> that my then-co-author (E. Whitehead) proposed exactly the same
> thing, but
> that proposal was not agreed.

I think it will continue to keep coming back until the policy is changed.
Too many XML implementators are frustrated by it.
>
> In my understanding, MIME people in IETF would like to keep the charset
> parameter of text/* authoritative, since a number of mail
> programs rely only on
> the charset parameter for text/*.

I don't think it is a problem for the charset parameter *when it is present*
to be authoritative. I think the real problem here is that it is also
considered to be authoritative when it is NOT present--overriding other
potential means for identifying content encoding that are available. The
lack of information should not take precedence over positive information in
the content when the implementation can take advantage of said information.

That doesn't mean that there can't be an outlet for implementations to
choose not to interpret from the content, only that implementations that can
choose to interpret the content be allowed to do so.
>
> However, as people correctly pointed out, omission of the charset
> parameter of text/xml is typically caused by the fact that authors
> cannot change the configuration of WWW servers.  For this reason, W3C
> recommendations for CSS and HTML say something similar to your
> suggestion,
> but the IETF RFC for CSS does not say anything about the default.

Then each standard or RFC should be modified over time to include such
information, either directly or by reference to parent standards. For
example, SOAP does not include any statement whatsoever about the charset to
use in a SOAP message. XML 1.0 and the RFC for the media type contain that
information (and in the case of the media type, IIRC that is by reference to
RFC 3023). But certainly Ur-standards, like CSS, XML, HTML, and so on should
contain a default (and not leave unannounced values open to interpretation).

> The IETF RFC 2854 for HTML says the default is US-ASCII (MIME) or
> 8859-1 (HTTP), but also says that "the actual default is either a
> corporate character encoding or character encodings widely deployed in a
> certain national or regional community"

That last statement is quite sloppy, in my opinion. It basically says: the
default is ASCII/8859, except when it isn't (e.g. you cannot rely on there
being a default). This is basically akin to just saying that the content
should be introspected by the receiver or user agent as best as possible.

I think it is reasonable (if occasionally a problem) to preserve the
authority of the charset tag when it is present in the media type.

I think that when it is not present, it should be permissable to detect the
encoding from the content. This suggests that implementations which cannot
determine the charset when transmitting content should be discouraged from
emitting the charset parameter ("guessing").

Finally, I think that the various default encodings should be modified and
harmonized in a reasonable way.

Personally I think that references to US-ASCII as a default should be
replaced with UTF-8 wherever possible (which is a superset of US-ASCII).
Since UTF-8 is highly patterned, it is also highly detectable. It could be
followed by a reasonable fallback (possibly US-ASCII or ISO8859-1: at least
from 8859-1 you can reconstruct the bytes). The existing RFC3023 contains
the counter argument to this, the logic of which I don't dispute. I'm merely
suggesting that a change in policy here might be a better "move towards the
future".

So to reiterate, I think that the RFC should be modified to try and avoid
the deliberate loss of data (as when using US-ASCII with data that contains
non-US-ASCII sequences) where it is possible for the implementation to cope.
This move won't generate any new incompatibilities for existing
implementations that do not grope the content. I understand that MIME
applications in particular cannot always avoid this kind of data degradation
and that's okay with me, so long as it isn't "wrong" to detect *missing*
charset values.

I'll probably come to regret that paragraph ;-).

Best Regards,

Addison

Addison P. Phillips
Director, Globalization Architecture
webMethods | Delivering Global Business Visibility

432 Lakeside Drive, Sunnyvale, CA, USA
+1 408.962.5487 (office)  +1 408.210.3569 (mobile)
mailto:[email-masked]

Chair, W3C-I18N-WG, Web Services Task Force
http://www.w3.org/International/ws

Internationalization is an architecture.
It is not a feature.

Date: Wed, 24 Sep 2003 22:59:23 +0900
From: MURATA Makoto <[email-masked]>
To: "Addison Phillips [wM]" <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Sender: [email-masked]

Addison,

Thanks for your contribution to this discussion.

> The problem with changing rfc3023 is that there are a number of
> implementations out there that adhere to the exact letter of the involved
> RFCs (3023/2045/2046/etc.). I seem to recall that there are implementations
> that require the charset parameter or which forceably filter the data to
> ASCII (converting all 8th-bit bytes to the '?' character) and thus there are
> many implementations that, to get the right results with these, forceably
> emit charset parameters.
> 
> Therefore, unless absolutely forbidden, implementations would still have to
> support the use of charset with both media types. And I don't see how we can
> forbid the use of the charset parameter given the need for need for
> interoperability with extant sensitive systems.

I think that this is a good reason to keep the charset parameter.

> <snip>
>        Conformant with [RFC2046], if a text/xml entity is received with
>        the charset parameter omitted, MIME processors and XML processors
>        MUST use the default charset value of "us-ascii"[ASCII].  In cases
>        where the XML MIME entity is transmitted via HTTP, the default
>        charset value is still "us-ascii".
> </snip>

This issue was discussed when RFC 2376 was developed.  I recall 
that my then-co-author (E. Whitehead) proposed exactly the same thing, but 
that proposal was not agreed.

In my understanding, MIME people in IETF would like to keep the charset 
parameter of text/* authoritative, since a number of mail programs rely only on 
the charset parameter for text/*.

However, as people correctly pointed out, omission of the charset 
parameter of text/xml is typically caused by the fact that authors 
cannot change the configuration of WWW servers.  For this reason, W3C 
recommendations for CSS and HTML say something similar to your suggestion, 
but the IETF RFC for CSS does not say anything about the default.  The 
IETF RFC 2854 for HTML says the default is US-ASCII (MIME) or 
8859-1 (HTTP), but also says that "the actual default is either a
corporate character encoding or character encodings widely deployed in a
certain national or regional community"

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Mon, 22 Sep 2003 16:50:02 -0400
To: "WWW-Tag" <[email-masked]>, [email-masked]
From: "Addison Phillips [wM]" <[email-masked]>(by way of Martin Duerst <[email-masked]>)
Subject: Re: Requesting a revision of RFC3023
Cc: "Addison Phillips" <[email-masked]>
Sender: [email-masked]

Hi Martin, et al,

Having tracked the thread down and having read it I feel like I can
contribute something to it. This is a common and not very fun problem that
our implementations encounter frequently with XML documents transmitted to
our system by other products.

SOAP 1.2 recommends the use of application/soap+xml as the media type
(although it is not required, see section 7.1.4 of [SOAP12-PART2], it is
pretty close to a requirement for HTTP). Noah is correct that charset is
optional. In the absence of charset, the application/*xml types default to
the encoding embedded in the XML document itself, which I think is generally
seen to be the desirable way to go.

Various SOAP implementations less than 1.2 use various media types,
including text/xml, depending on the transport, etc.

The problem with changing rfc3023 is that there are a number of
implementations out there that adhere to the exact letter of the involved
RFCs (3023/2045/2046/etc.). I seem to recall that there are implementations
that require the charset parameter or which forceably filter the data to
ASCII (converting all 8th-bit bytes to the '?' character) and thus there are
many implementations that, to get the right results with these, forceably
emit charset parameters.

Therefore, unless absolutely forbidden, implementations would still have to
support the use of charset with both media types. And I don't see how we can
forbid the use of the charset parameter given the need for need for
interoperability with extant sensitive systems.

It would be nice if text/xml could be modified, since it is quite common to
get un-charset-labeled content that really is NOT US-ASCII. Since one can
always detect that a data stream is not US-ASCII, it has always seemed a bit
odd to me that the RFCs require the data to be destroyed when there is clear
evidence that one is losing something. I understand the reasoning, but I
think there is a difference between saying that omission of a charset
parameter invites data corruption (e.g. the MIME or XML processor is not
required to look at the XML content and thus MAY use US-ASCII to interpret
the data) and one that insists on it (e.g. the MIME or XML processor is
required to interpret the data using US-ASCII-7).

Perhaps we should focus on the semantics of charset not being present,
instead of focusing on forbidding/requiring charset itself. Consider this
paragraph of RFC3023:

<snip>
       Conformant with [RFC2046], if a text/xml entity is received with
       the charset parameter omitted, MIME processors and XML processors
       MUST use the default charset value of "us-ascii"[ASCII].  In cases
       where the XML MIME entity is transmitted via HTTP, the default
       charset value is still "us-ascii".
</snip>

This could be changed to something more friendly, like:

<snip>
       If a text/xml entity is received with
       the charset parameter omitted, MIME processors and XML processors
       MAY attempt to detect the charset from the XML content itself. Such
       detection MUST follow the requirements of section 4.3.3 of [XML].
       MIME and XML processors that do not attempt or are unable to detect
       the charset using this requirement must use US-ASCII (or UTF-8????)...
etc. and so forth...
</snip>

This allows receivers the leeway to detect errant senders (while leaving
errant senders of text/xml as non-conforming). This seems like a reasonable
compromise to me.

[SOAP12-PART2] http://www.w3.org/TR/2003/PR-soap12-part2-20030507

Just my two cents. Best Regards,

Addison

Addison P. Phillips
Director, Globalization Architecture
webMethods | Delivering Global Business Visibility

432 Lakeside Drive, Sunnyvale, CA, USA
+1 408.962.5487 (office)  +1 408.210.3569 (mobile)
mailto:[email-masked]

Chair, W3C-I18N-WG, Web Services Task Force
http://www.w3.org/International/ws

Internationalization is an architecture.
It is not a feature.

 > -----Original Message-----
 > From: Martin Duerst [mailto:[email-masked]]
 > Sent: Friday, September 19, 2003 11:05 AM
 > To: Addison Phillips
 > Subject: Fwd: Re: Requesting a revision of RFC3023
 >
 >
 > Hello Addison,
 >
 > This is from two lists ([email-masked], WWW-Tag <[email-masked]>).
 > Re SOAP, I guess you might have some answer. If yes, can you send it to
 > those lists or to me for forwarding?
 >
 > Regards,    Martin.
 >
 >
 > >To: MURATA Makoto <[email-masked]>
 > >Cc: [email-masked], WWW-Tag <[email-masked]>
 > >Subject: Re: Requesting a revision of RFC3023
 > >From: [email-masked]
 > >Date: Fri, 19 Sep 2003 11:04:11 -0400
 > >Sender: [email-masked]
 > >List-Archive: <http://www.imc.org/ietf-xml-mime/mail-archive/>
 > >List-ID: <ietf-xml-mime.imc.org>
 > >List-Unsubscribe: <mailto:[email-masked]?body=unsubscribe>
 >
 > >Murata Makoto writes:
 > >
 > > >> I believe that SOAP implementations use the
 > > >> charset parameter.  If we remove the charset
 > > >> parameter, we will make them non-conformant.
 > >
 > >This is not my area of expertise, but I note that the HTTP binding [1]
 > >provided by SOAP 1.2 Recommendation uses an application/soap+xml media
 > >type, a definition of which is at [2] (I believe it is working its way
 > >through the formal registration process.)  My reading is that the
 > >definition lists charset as optional, and makes clear that its proper use
 > >is to be found in RFC 3023.
 > >
 > >I am not aware of what typical implementations of SOAP 1.1 or
 > SOAP 1.2 are
 > >doing, but the 1.2 spec at least seems to list it as optional.
 > Again, I'm
 > >not expert in this stuff and am not offering an opinion, but I
 > thought the
 > >links might be helpful.
 > >
 > >[1]http://www.w3.org/TR/soap12-part2/#soapinhttp
 > >[2] http://www.w3.org/TR/soap12-part2/#ietf-draft
 > >
 > >------------------------------------------------------------------
 > >Noah Mendelsohn                              Voice: 1-617-693-4036
 > >IBM Corporation                                Fax: 1-617-693-8676
 > >One Rogers Street
 > >Cambridge, MA 02142
 > >------------------------------------------------------------------
 > >
 > >
 > >
 > >
 > >
 > >
 > >
 > >MURATA Makoto <[email-masked]>
 > >Sent by: [email-masked]
 > >09/19/03 08:10 AM
 > >
 > >
 > >         To:     [email-masked], WWW-Tag <[email-masked]>
 > >         cc:     (bcc: Noah Mendelsohn/Cambridge/IBM)
 > >         Subject:        Re: Requesting a revision of RFC3023
 > >
 > >
 > >
 > >
 > >On Fri, 19 Sep 2003 03:50:11 +0200
 > >Bjoern Hoehrmann <[email-masked]> wrote:
 > >
 > > > You want to change something that has been STRONGLY
 > RECOMMENDED for over
 > > > five years to (ideally) MUST NOT just because it could cause trouble
 > > > when used improperly or with broken implementations. Today I am good
 > > > with web standards if I use the charset parameter, tommorow I am bad
 > > > with web standards if I do. What's next on #W3C? Use tables for layout
 > > > because people could get CSS wrong and old browsers get some
 > CSS wrong?
 > > > I don't think this leads anywhere.
 > >
 > >I believe that SOAP implementations use the charset parameter.  If we
 > >remove the charset parameter, we will make them non-conformant.
 > >
 > >Cheers,
 > >
 > >--
 > >MURATA Makoto <[email-masked]>
 > >
 > >

Date: Mon, 22 Sep 2003 10:45:38 -0400
To: MURATA Makoto <[email-masked]>, "Aaron Swartz" <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Sender: [email-masked]

At 20:56 03/09/22 +0900, MURATA Makoto wrote:

>On 21 Sep 2003 22:06:30 -0500
>"Aaron Swartz" <[email-masked]> wrote:
>
> > > I think that persuading users is more difficult than
> > > persuading programmers.  I have encouraged use of Unicode
> > > for XML in Japan, but nothing has happened.
> >
> > The users shouldn't have to know what a character encoding is! Their
> > software should just use UTF-8 automatically.
>
>Will users discard existing software, which supports legacy encogins, and
>existing data, which are in legacy encodings?  I am not saying Unicode
>everywhere is bad.  To the contrary, I think Unicode everywhere is better.
>But it is extremely unlikely.

New versions of software in many cases supports Unicode. As users
upgrade, they will be more and more able to use Unicode. XML made
a big step forward by defining UTF-8 and UTF-16 as default encodings
required to be understood by the parser. This is helping getting
software upgraded.

At the next step, some formats may decide to rely even more on Unicode
(such as N3). So we are moving in the right direction, slower than
we would like, but moving nonetheless. Data will move less quickly
than software, that has always been the case. In the short time,
many things seem extremely unlikely. But in the long run, we will
get closer.

Regards,  Martin.

Date: Mon, 22 Sep 2003 20:56:02 +0900
From: MURATA Makoto <[email-masked]>
To: "Aaron Swartz" <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Sender: [email-masked]

On 21 Sep 2003 22:06:30 -0500
"Aaron Swartz" <[email-masked]> wrote:

> > I think that persuading users is more difficult than
> > persuading programmers.  I have encouraged use of Unicode
> > for XML in Japan, but nothing has happened.
> 
> The users shouldn't have to know what a character encoding is! Their 
> software should just use UTF-8 automatically.

Will users discard existing software, which supports legacy encogins, and 
existing data, which are in legacy encodings?  I am not saying Unicode 
everywhere is bad.  To the contrary, I think Unicode everywhere is better.  
But it is extremely unlikely.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: 21 Sep 2003 22:06:30 -0500
From: "Aaron Swartz" <[email-masked]>
To: "MURATA Makoto" <[email-masked]>
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

[re UTF-8 being the only long-term solution]
> I think that persuading users is more difficult than
> persuading programmers.  I have encouraged use of Unicode
> for XML in Japan, but nothing has happened.

The users shouldn't have to know what a character encoding is! Their 
software should just use UTF-8 automatically.

-- 
Aaron Swartz: http://www.aaronsw.com/

Date: Sun, 21 Sep 2003 20:25:45 -0400
From: "Simon St.Laurent" <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
To: WWW-Tag <[email-masked]>
cc: [email-masked]
Sender: [email-masked]

[email-masked] (MURATA Makoto) writes:
>> > I'm just saying UTF-8 everywhere is even  more unrealistic than any 
>> > other options at hand.
>> 
>> Too bad, because it's the only option that's remotely practical in
>> the long term. Do you really think every programmer who wants to
>> mung text is going to include code that supports not only the
>> hundreds of extant character encodings but also the seventeen kinds
>> of in-band and out-of-band declarations of them?
>
>I think that persuading users is more difficult than 
>persuading programmers.  I have encouraged use of Unicode 
>for XML in Japan, but nothing has happened.

Even in the US, when I talk to users about Unicode I get blank stares.
An awful lot of people have simply been told "UTF-8 is the same as
ASCII" and write XML in 100% ASCII, using character references or entity
references for anything outside of ASCII.  Heck, they don't even specify
encoding a lot of the time, just trusting blindly in ASCII magic.

The ASCII-derived inertia in this country may favor UTF-8, but I don't
think that inertia is sufficient cause to drive the whole world toward
UTF-8.  UTF-16 I could see as plausible, as more and more tools cope
with it, but that's a real change.  

I thought XML was wise to demand support for at least two encodings
(UTF-8 and UTF-16) and leave the door open for others.  In any case, I
don't think a revision of RFC 3023 is the place to attempt to make the
whole world use UTF-8, whether or not it's a good idea.

Simon St.Laurent
http://simonstl.com
http://monasticxml.org

Date: Mon, 22 Sep 2003 09:08:11 +0900
From: MURATA Makoto <[email-masked]>
To: "WWW-Tag" <[email-masked]>, [email-masked]
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

> > I'm just saying UTF-8 everywhere is even  more unrealistic than any 
> > other options at hand.
> 
> Too bad, because it's the only option that's remotely practical in the 
> long term. Do you really think every programmer who wants to mung text 
> is going to include code that supports not only the hundreds of extant 
> character encodings but also the seventeen kinds of in-band and 
> out-of-band declarations of them?

I think that persuading users is more difficult than 
persuading programmers.  I have encouraged use of Unicode 
for XML in Japan, but nothing has happened.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: 21 Sep 2003 10:52:33 -0500
From: "Aaron Swartz" <[email-masked]>
To: "MURATA Makoto" <[email-masked]>
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

> I'm just saying UTF-8 everywhere is even  more unrealistic than any 
> other options at hand.

Too bad, because it's the only option that's remotely practical in the 
long term. Do you really think every programmer who wants to mung text 
is going to include code that supports not only the hundreds of extant 
character encodings but also the seventeen kinds of in-band and 
out-of-band declarations of them?

Even if there was a easily-usable well-supported library to do all 
this, and you managed to get everyone to use it, the minor differences 
between encodings hidden by its abstractions would lead to subtle bugs 
that would continue to plague international users and force them to be 
second-class citizens forever.

If you want international users to be on the same level as those who 
just use plain ASCII, a drop-in solution is the only way to go, and 
UTF-8 is the obvious drop-in solution.

-- 
Aaron Swartz: http://www.aaronsw.com/

(For the purposes of this email, "international users" are users who 
need characters other than those in plain ASCII.)

Date: Mon, 22 Sep 2003 00:46:49 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked]
Subject: Re: Requesting a revision of RFC3023
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

On Fri, 19 Sep 2003 11:04:11 -0400
[email-masked] wrote:

> I am not aware of what typical implementations of SOAP 1.1 or SOAP 1.2 are 
> doing, but the 1.2 spec at least seems to list it as optional.  Again, I'm 
> not expert in this stuff and am not offering an opinion, but I thought the 
> links might be helpful.

I googled three words "charset",  "soap", and "User-Agent" and found so many 
WWW pages.  Apparently, some implementations (including Apache Axis) use the charset 
parameter.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: 21 Sep 2003 10:42:22 -0500
From: "Aaron Swartz" <[email-masked]>
To: "Francois Yergeau" <[email-masked]>
Cc: "WWW-Tag" <[email-masked]>, [email-masked]
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

>> Programming languages are broken as designed?
> In this respect, yes.  All programming languages should provide for 
> charset
> identification of their source files.  Alas, none do, AFAIK.

Python does; it uses an emacs convention:

     To define a source code encoding, a magic comment must
     be placed into the source files either as first or second
     line in the file:

           #!/usr/bin/python
           # -*- coding: <encoding name> -*-

http://www.python.org/peps/pep-0263.html

-- 
Aaron Swartz: http://www.aaronsw.com/

Date: Mon, 22 Sep 2003 00:37:18 +0900
From: MURATA Makoto <[email-masked]>
To: Elliotte Rusty Harold <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

> By Unicode signature, I'm guessing you mean the BOM? That problem 
> seems to have been easily dealt with by simply deciding to allow it 
> in UTF-8. It doesn't appear to have caused any problems in practice 
> today.

In the case of XML, I think you are right.  In general, however, see

http://www.ietf.org/internet-drafts/draft-yergeau-rfc2279bis-05.txt

> I don't know what you problems you refer to with "representation of 
> non-BMP characters". UTF-8 precisely specifies how these characters 
> are represented. There's no issue here. Did you mean something else?

Quite a few implementations use 6 bytes (rather than 4 bytes) to represent 
non-BMP characters.  See

http://www.unicode.org/reports/tr26/

-- 
MURATA Makoto <[email-masked]>

Date: Sun, 21 Sep 2003 10:04:36 -0400
To: MURATA Makoto <[email-masked]>, [email-masked]
From: Elliotte Rusty Harold <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

At 9:08 PM +0900 9/21/03, MURATA Makoto wrote:

>UTF-8 has its own technical problems (the Unicode signature, representation
>of non-BMP characters, etc.).

By Unicode signature, I'm guessing you mean the BOM? That problem 
seems to have been easily dealt with by simply deciding to allow it 
in UTF-8. It doesn't appear to have caused any problems in practice 
today.

I don't know what you problems you refer to with "representation of 
non-BMP characters". UTF-8 precisely specifies how these characters 
are represented. There's no issue here. Did you mean something else?

-- 

   Elliotte Rusty Harold
   [email-masked]
   Processing XML with Java (Addison-Wesley, 2002)
   http://www.cafeconleche.org/books/xmljava
   http://www.amazon.com/exec/obidos/ISBN%3D0201771861/cafeaulaitA

Date: Sun, 21 Sep 2003 21:08:12 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked]
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

On Fri, 19 Sep 2003 20:55:46 +0200
Bjoern Hoehrmann <[email-masked]> wrote:

> UTF-8 everywhere is a reasonable principle and much simpler to
> understand and implement than any means to specify use of legacy
> encoding schemes. 

UTF-8 has its own technical problems (the Unicode signature, representation 
of non-BMP characters, etc.).  Moreover, people do not throw away legacy 
encodings but stick to them.  For example, although I think that 
Unicode is better than Shift-JIS and I do have Unicode-aware text editors, I 
still use Shift-JIS, which is so convenient at present.

I am not saying UTF-8 is bad.  I'm just saying UTF-8 everywhere is even 
more unrealistic than any other options at hand.

>For inbound encoding declarations, generic syntax
> does not work. Whatever syntax you choose, it will look odd in many
> formats and many authors won't use it anyway.

Please see my mail to Martin.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Sat, 20 Sep 2003 13:08:50 -0700
From: Tim Bray <[email-masked]>
To: Bert Bos <[email-masked]>
Cc: [email-masked]
Subject: Re: transcoding nearly certainly wrong?
Sender: [email-masked]

Bert Bos wrote:

> But if it doesn't work, you can look more closely at the MIME type and
> use your knowledge about foo. If foo is css or html, you can convert
> from any encoding to any encoding (and I do so often). If foo is
> something+xml, you can still do do it, as long as all element names
> and attribute names are in ASCII.

... and if you check the first few bytes and if they say, for example,

<?xml version='1.0' encoding='ISO-8859-1' ?>

then you also fix them.  -Tim

Date: Sat, 20 Sep 2003 21:57:42 +0200
From: Bert Bos <[email-masked]>
To: [email-masked]
Subject: Re: transcoding nearly certainly wrong?
Sender: [email-masked]

Larry Masinter writes:
> 
> Why is transcoding nearly certain to be wrong with XML?

Can't be *that* wrong. I do it all the time...

> 
> Or, to put it another way, why not limit the use
> of text/xml to XML instances for which transcoding
> is certain not to be wrong, and for which US-ASCII
> is acceptable (because the XML uses numeric character
> references or character entities, is only used to
> code a limited schema with numeric data, etc.?)
> 
> Limiting the scope is less radical than deprecating.

I agree.

Every text/foo format can be transcoded from any encoding to UTF-8.
Something I do often. Going the other way doesn't always work, of
course.

But if it doesn't work, you can look more closely at the MIME type and
use your knowledge about foo. If foo is css or html, you can convert
from any encoding to any encoding (and I do so often). If foo is
something+xml, you can still do do it, as long as all element names
and attribute names are in ASCII. Which happens to be the case with
all XML-based formats created by W3C and all formats that I work with,
fortunately. And thus my conversion program doesn't even check, but
blindly converts every non-ASCII character to &#nnn;. Very useful,
since I create non-English files by cut & paste and my Emacs only
understands Latin-1.

So, for many XML-based formats, including all W3C's formats, using
text/foo instead of application/foo makes a lot of sense. Reserve
application/something+xml for formats that cannot be transcoded.

Bert
-- 
  Bert Bos                                ( W 3 C ) http://www.w3.org/
  http://www.w3.org/people/bos/                              W3C/ERCIM
  [email-masked]                             2004 Rt des Lucioles / BP 93
  +33 (0)4 92 38 76 92            06902 Sophia Antipolis Cedex, France

Date: Sat, 20 Sep 2003 09:26:15 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

> Can you explain what in particular you like about the CSS
> mechanism? 

I like the CSS mechanism because it appears at the very beginning.

> Do you expect that all formats would use exactly
> the same character sequence, or only that they would use
> the same structure?

An optimistic answer to the first question is yes, although it may 
be more practical to provide a bit of flexibility (e.g.,   @
as well as #).  The structure and the semantics should be standardized.  
At present, everything is a hopeless mess.

Although I suggested this approach, I still feel uneasy about in-band 
encoding declarations.  I'm trying to generalize them only because 
I believe ad-hoc mechanisms are worse.

Cheers,

Makoto

From: Bjoern Hoehrmann <[email-masked]>
To: John Cowan <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 23:27:45 +0200
Sender: [email-masked]

* John Cowan wrote:
>> And by the way, some formats you mention are binary
>> and not text formats and could thus not make use of file system encoding
>> information.
>
>Which?  Programming languages, text markup languages, scripts, or
>mbox-type message archives?

Mbox is a good example, it is text-separated binary data, not text.

Date: Fri, 19 Sep 2003 17:02:15 -0400
From: John Cowan <[email-masked]>
To: Bjoern Hoehrmann <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Bjoern Hoehrmann scripsit:

> No. I said that I consider formats that leave implementations to guess
> how to process instances of the format broken as designed. I did not
> suggest to fix them.

Okay, since this judgment has no real-world consequences I can live with it.

> And by the way, some formats you mention are binary
> and not text formats and could thus not make use of file system encoding
> information.

Which?  Programming languages, text markup languages, scripts, or
mbox-type message archives?

-- 
I marvel at the creature: so secret and         John Cowan
so sly as he is, to come sporting in the pool   [email-masked]
before our very window.  Does he think that     http://www.reutershealth.com
Men sleep without watch all night?  --Faramir   http://www.ccil.org/~cowan

Date: Fri, 19 Sep 2003 16:29:55 -0400
To: Tim Bray <[email-masked]>, [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Hello Tim, others,

At 10:50 03/09/16 -0700, Tim Bray wrote:

>[Some of you will get this twice, sorry; Larry Masinter pointed out that 
>my initial choice of destinations was poor.  I slightly revised the note 
>to provide more context.]
>
>The W3C TAG (http://www.w3.org/2001/tag/) has an open issue about proper 
>handling of MIME headers, with a draft in progress "Client Handling of 
>MIME Headers" (http://www.w3.org/2001/tag/doc/mime-respect.html);

Some comments on this in a separate mail.

>the draft finds some fault with the contents of RFC3023.
>
>I took an action item to ask about the chances of revising what 3023 says 
>about the charset parameter; while I'm not sure, I suspect that there may 
>actually be some level of consensus about the desirable changes:
>
>1. Deprecate text/* for anything that's in XML.

Agreed, in general.

>That's because it forces the provider to provide a charset header, because 
>in its absence the receiver is required to assume either ASCII or 8859 
>depending on the context,

Do you mean that it is US-ASCII for email and iso-8859-1 (there are many
parts of iso 8859) for HTTP? My understanding is that RFC 3023 says
that the US-ASCII default applies for all protocols.

>which has a very high probability of being wrong, which is irritating 
>because if there were no charset header the client would be certain of 
>either getting it right or failing deterministically.

That was part of the design of making US-ASCII the default, too,
similar to the design of XML in general: Either the charset parameter
is set (correctly), or it is missing, and then the parser has a very
easy job to detect the problem if the data is not US-ASCII.

On the other hand, there is in some cases no other way than to ask
a linguistic expert to detect a mistaken iso-8859-1 (e.g. for a correct
iso-8859-2).

>And forcing the server to provide a charset= is wrong; see the next point.
>
>2. Deprecate the charset parameter for application/xml and 
>application/*+xml.  I think that Roy Fielding would like to go far as to 
>simply outlaw it; I'd be fine with that too.

I think moving from the current wording which is very close to required
to a more optional wording is fine. However, I'm not sure about deprecating
it (which basically means that it is a bad idea but still tolerated for some
time), and I don't think outlawing is a good idea.

>The reason is that the client is almost certain to get it right, and will 
>fail deterministically if it doesn't.  For the server, on the other hand, 
>this is easy to get wrong, particularly with the introduction of various 
>kinds of filters in modern web servers.

It's not about client or server. It's about author or server, or
any other producer such as some software. The client (hopefully)
never does anything else than follow the spec.

>And since the Web architecture and the XML spec both say that the server's 
>claim has to be taken as authoritative, this is really highly 
>dysfunctional.  At the very least, it should be made clear that nobody 
>sending a media-type should send a charset for an XML media-type unless it 
>REALLY REALLY KNOWS what it's sending, and in that case should consider 
>not sending it anyhow.

I don't disagree with this. But I wonder why this would have to be
stressed that much. It's a REALLY REALLY bad idea to create or send
non-wellformed XML, it's a REALLY REALLY bad idea to send documents
with a wrong mime type, and so on. Yet our specs don't contain
"REALLY REALLY bad" very often.

>Is there any chance we could do this?  It's going to be kind of 
>embarrassing for TAG findings and the Webarch doc to be saying "don't do 
>what this RFC says".

It would definitely be good for things to be in sync. And I'm sure
we can find an adequate compromise.

Regards,    Martin.

Date: Fri, 19 Sep 2003 16:16:45 -0400
To: WWW-Tag <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Comments on mime-respect
Cc: [email-masked]
Sender: [email-masked]

These are my comments on http://www.w3.org/2001/tag/doc/mime-respect.html,
various issues mixed a bit, sorry.
[I have cross-posted [email-masked] because some of them are relevant
to the recent discussion about the charset paramenter on Content-Type.]

- Headings: Is this a completed finding, or a draft finding?

- "HTTP/1.1 a a response": word duplication

- Overall, it seems difficult to identify what is general architecture,
   and what is the way it is just because it is the way it (mostly) is.

- My understanding is that one origin of the 'charset' parameter was
   that it was useful to invoke different applications for different
   values. That was definitely the case 10 years or so ago when MIME
   was designed. I remember reading my email that way. This has gone away.
   It may happen that in a somewhat similar way, a lot of what we now
   see as different XML types, in need of different applications, may
   go away in a few years.

- Section 4: "The Unicode encoding of a message body (XML document) is
   inconsistent with the value of the charset parameter in the message
   headers."
   - Please replace 'Unicode encoding' with 'character encoding'.
     It would be strange to e.g. call iso-8859-1 an 'Unicode encoding'.
   - Please remove, or reword "XML document", to not give the impression
     that message bodies are always XML documents.
   - I'm not clear why this is in section 4, entitled "Why user agent
     behavior that misrepresents the user is harmful". This is a
     server problem, the user is not in any way misrepresented.

- The big problem with wrong encoding information for XML and other
   documents is not in a server-user context (where the user has
   to be able to read the document, such problems are usually
   discovered very quickly), but with XML sent between machines.
   This probably should be noted.

- The structure of sections 3 and 4 should be improved. It is good
   style to have an introductory paragraph or two before subsection.
   It is confusing to have a few paragraphs in the first subsection
   of the section after a lot of text that is not in subsections.

- "For this reason, servers should only supply a character encoding
    header when there is complete certainty as to the encoding in use.
    Otherwise, an error will cause a perfectly usable representation
    to be rejected by an architecturally sound client."

    Why doesn't the document say e.g. that a mime type should only be
    supplied when there is complete certainty that this type is
    appropriate? Why does this text assume that the XML is 'perfectly
    usable'? It might not be valid, it might be the wrong mime type,
    or it might not have the right 'encoding' attribute.

- "Servers which generate representations MUST NOT generate the charset
    parameter unless there is certainty that the headers are correct.
    When correct, this information can be used by non-XML processors
    to determine authoritatively the character encoding of the XML MIME
    entity."

    How is a server ever going to know, or going to be able to check,
    what the right character encoding is? Making this a requirement
    on the server itself seems inadequate.

- Section 5: "For instance, the http-equiv attribute of the HTML meta
   element is intended for servers (not clients)."
   Please change 'is' to 'was'. In particular with respect to character
   encoding, current practice is that it's used on the client. If you
   think that this should change, you should say so.

- SMIL 2.0 is "outmoded": I would prefer a different word here.
   I strongly agree that what SMIL 2.0 is saying on content types
   is a very bad idea, and I have said so to the SMIL WG (and more
   recently the Voice browser WG, I think). But given the 2001
   date, I don't think 'outmoded' is the right word, because it was
   never in fashion in the first place.

- Section 6: There is advice to server managers and authors. But
   I think we need to go one more step back, to server implementers
   and the default settings when servers are shipped.
   For example, some servers have an easy way to explore configurations
   and check settings. Others don't. Some servers come with default
   configurations that may be suboptimal. For example (not picking on
   it, just because that's the one I know), Apache at
   http://httpd.apache.org/docs-2.0/en/mod/core.html#adddefaultcharset
   says: "AddDefaultCharset On enables Apache's internal default charset
   of iso-8859-1 as required by the directive."
   Also, the default configuration file contains this:
    #
    # Specify a default charset for all pages sent out. This is
    # always a good idea and opens the door for future internationalisation
    # of your web site, should you ever want it. Specifying it as
    # a default does little harm; as the standard dictates that a page
    # is in iso-8859-1 (latin1) unless specified otherwise i.e. you
    # are merely stating the obvious. There are also some security
    # reasons in browsers, related to javascript and URL parsing
    # which encourage you to always set a default char set.
    #
    AddDefaultCharset ISO-8859-1

   This seems to be 180 degrees opposite to what the TAG is saying.
   It is more about text/html,... than about application/...+xml, but
   there is considerable potential for harm here, too, in particular
   when combined with the default setting that Apache comes with that
   does not allow people managing a directory to override file info.

Regards,     Martin.

From: Bjoern Hoehrmann <[email-masked]>
To: John Cowan <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 21:56:14 +0200
Sender: [email-masked]

* John Cowan wrote:
>> [A]nd even if they did, this would cause interoperability
>> problems with file systems and protocols which do not provide such
>> means. If you transfer the document using FTP to your web server the
>> information is lost and the document will break.
>
>No worse than today's situation, and FTP could be enhanced or abandoned
>in favor of HTTP PUT.

RFC 1342 is eleven years old and I would still see replies containing
Bj<garbage>rn H<garbage>hrmann if I made use of its features.

>*You* are suggesting that every text file format that has ever existed --
>innumerable assembly languages, C, C++, Java, Fortran, Lisp, Scheme, Prolog,
>Perl, Python, Smalltalk, awk, sed, ... sh, csh, bash, zsh, ... mail archives,
>news archives, ... Tex, LaTex, nroff/troff, ... -- be revised to find someplace
>to stuff a charset indication, and then that every one of the billions of
>documents in each of those formats be changed to carry that information.

No. I said that I consider formats that leave implementations to guess
how to process instances of the format broken as designed. I did not
suggest to fix them. And by the way, some formats you mention are binary
and not text formats and could thus not make use of file system encoding
information.

Date: Fri, 19 Sep 2003 15:20:57 -0400
From: John Cowan <[email-masked]>
To: Bjoern Hoehrmann <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Bjoern Hoehrmann scripsit:

> Impractical. File systems commonly do not support encoding such
> information

In fact most file systems support extended attributes today.

> [A]nd even if they did, this would cause interoperability
> problems with file systems and protocols which do not provide such
> means. If you transfer the document using FTP to your web server the
> information is lost and the document will break.

No worse than today's situation, and FTP could be enhanced or abandoned
in favor of HTTP PUT.

> Further, file system
> information is typically almost invisible to authors and would thus
> have the same problem as the charset parameter. If I edit a document
> in an XML unaware text editor, change the encoding declaration and
> some text nodes and save the file, file system and encoding declaration
> are likely to contradict each other and the document would break.

No worse than today's situation.

> You are basically suggesting to change all file systems and software
> that interacts with it and expect everyone to upgrade the software and
> the file system information of all documents.

*You* are suggesting that every text file format that has ever existed --
innumerable assembly languages, C, C++, Java, Fortran, Lisp, Scheme, Prolog,
Perl, Python, Smalltalk, awk, sed, ... sh, csh, bash, zsh, ... mail archives,
news archives, ... Tex, LaTex, nroff/troff, ... -- be revised to find someplace
to stuff a charset indication, and then that every one of the billions of
documents in each of those formats be changed to carry that information.

> If an applicable solution
> may go this far, you should rather suggest to outlaw all non-Unicode
> encodings, much simpler, more consistent and more interoperable. This
> would also work if the text is not stored in the file system but rather
> generated by software, something your solution does not consider.

Indeed, which is why Plan 9 sensibly makes everything UTF-8 and Windows NT/2K/XP
makes most things UTF-16, at least under the covers.

> >Otherwise, generic text-processing tools become impossible,
> 
> They are impossible today.

The impossible does not happen, but I usefully use generic text processing
tools every hour of every working day.

> They are not trying to read the format, they are trying to read byte
> streams as character streams. If they are trying to read the format,
> they have to support that format anyway, including mechanisms to
> determine the character encoding.

Not so.  If I want to process a Fortran 77 program as text (to find the
identifiers which occur only once, e.g.) then I can use generic tools
(tr, sort, uniq) and supply the character encoding out of band.  This is
annoying, but it works.  If the tools had to understand where backpatched
Fortran 77 text hides its in-band character encoding declaration, the
results would be as I describe: huge amounts of useless hair.

> If you consider HTTP a file system, it
> already implements your solution; all text is identified using text/*
> types and either the file system provides encoding information (charset
> parameter) or text processors are required to treat the document as
> ISO-8859-1 encoded. Text processors would actually only get character
> streams from the HTTP implementation and would not have to worry about
> character encodings and stuff. Does it work? No. 

It does not work because HTTP is layered over file systems which don't bother
to support the notion of encoding declarations persistently.

-- 
"We are lost, lost.  No name, no business, no Precious, nothing.  Only empty.
Only hungry: yes, we are hungry.  A few little fishes, nassty bony little
fishes, for a poor creature, and they say death.  So wise they are; so just,
so very just."  --Gollum        [email-masked]  www.ccil.org/~cowan

Date: Fri, 19 Sep 2003 15:03:31 -0400
To: Tim Bray <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: John Cowan <[email-masked]>, Francois Yergeau <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

At 11:21 03/09/19 -0700, Tim Bray wrote:

>Martin Duerst wrote:
>
>>This will of course take some time. As one example, N3 just says
>>that it's in UTF-8. For other new formats, that may make sense,
>>too.
>
>In principle, I agree.  In practice, there's a problem, for example I 
>actually don't know how, in my favorite editor on Mac OS X,

What editor is that?

>to type in the u-umlaut in Martin's last name,

How to type it shouldn't depend on the editor, but should be
something the OS deals with. And for an average Mac application,
it actually does.

On the Mac, go to Control Pannel -> International -> Input Menu.
You then get a list of keyboards and other input methods.
One that is particularly useful for occasional input of characters
is the Character palette. As soon as you have selected more than
one keyboard/..., you get an additional menu item where you can
switch these.

The old resedit also had a way to inspect and change your keyboard
configurations, but I'm not sure this is still available.

Regards,    Martin.

>and how to force saving a file in UTF-8.  This isn't a problem for me 
>because I'm usually writing XML and so I just write D&uuml;rst.  So I 
>actually have trouble making N3 assertions about Martin.
>
>If we could make the whole world always use UTF-8 for everything life 
>would be so much simpler :)
>--
>Cheers, Tim Bray
>         (ongoing fragmented essay: http://www.tbray.org/ongoing/)
>

From: Bjoern Hoehrmann <[email-masked]>
To: MURATA Makoto <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 20:55:46 +0200
Sender: [email-masked]

* MURATA Makoto wrote:
>> Depends on the format. Formats should provide means to specify
>> the encoding, if they do not they are BAD, broken as designed.
>
>If we really promote in-band declarations, we should recommend 
>one mechanism for all textual formats.  I think that CSS style 
>declaration (@chaset) is the best I have seen.  If the TAG recommends 
>this mechanism for all formats except XML and HTML, that would 
>be one reasonable principle.

UTF-8 everywhere is a reasonable principle and much simpler to
understand and implement than any means to specify use of legacy
encoding schemes. For inbound encoding declarations, generic syntax
does not work. Whatever syntax you choose, it will look odd in many
formats and many authors won't use it anyway.

From: Bjoern Hoehrmann <[email-masked]>
To: John Cowan <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 20:55:39 +0200
Sender: [email-masked]

* John Cowan wrote:
>Rather than having thousands of ad hoc mechanisms for encoding declarations
>in each of the thousands of text formats now extant, file systems should have
>a convenient mechanism for recording the encoding of each file, and character
>processing libraries should have convenient reading and writing operations that
>do the necessary conversions.

Impractical. File systems commonly do not support encoding such
information and even if they did, this would cause interoperability
problems with file systems and protocols which do not provide such
means. If you transfer the document using FTP to your web server the
information is lost and the document will break. Further, file system
information is typically almost invisible to authors and would thus
have the same problem as the charset parameter. If I edit a document
in an XML unaware text editor, change the encoding declaration and
some text nodes and save the file, file system and encoding declaration
are likely to contradict each other and the document would break.

You are basically suggesting to change all file systems and software
that interacts with it and expect everyone to upgrade the software and
the file system information of all documents. If an applicable solution
may go this far, you should rather suggest to outlaw all non-Unicode
encodings, much simpler, more consistent and more interoperable. This
would also work if the text is not stored in the file system but rather
generated by software, something your solution does not consider.

>Otherwise, generic text-processing tools become impossible,

They are impossible today.

>because each tool has to have a vast library that understands the
>mechanics of the encoding declaration specific to the format it is trying to
>read.

They are not trying to read the format, they are trying to read byte
streams as character streams. If they are trying to read the format,
they have to support that format anyway, including mechanisms to
determine the character encoding. If you consider HTTP a file system, it
already implements your solution; all text is identified using text/*
types and either the file system provides encoding information (charset
parameter) or text processors are required to treat the document as
ISO-8859-1 encoded. Text processors would actually only get character
streams from the HTTP implementation and would not have to worry about
character encodings and stuff. Does it work? No. Especially not because
W3C publishes Recommendations that make it impossible to write
conforming HTTP implementations. That way madness lies.

Date: Fri, 19 Sep 2003 11:21:47 -0700
From: Tim Bray <[email-masked]>
To: Martin Duerst <[email-masked]>
Cc: John Cowan <[email-masked]>, Francois Yergeau <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Martin Duerst wrote:

> This will of course take some time. As one example, N3 just says
> that it's in UTF-8. For other new formats, that may make sense,
> too.

In principle, I agree.  In practice, there's a problem, for example I 
actually don't know how, in my favorite editor on Mac OS X, to type in 
the u-umlaut in Martin's last name, and how to force saving a file in 
UTF-8.  This isn't a problem for me because I'm usually writing XML and 
so I just write D&uuml;rst.  So I actually have trouble making N3 
assertions about Martin.

If we could make the whole world always use UTF-8 for everything life 
would be so much simpler :)
-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)

Date: Fri, 19 Sep 2003 11:17:36 -0700
From: Tim Bray <[email-masked]>
To: Martin Duerst <[email-masked]>
Cc: Bjoern Hoehrmann <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Martin Duerst wrote:

>> I can live with removing the STRONGLY RECOMMENDED status and an
>> informative note that you typically do not need to specifiy the
>> charset parameter but anything beyond that goes much too far.
> 
> I quite agree with this statement.

At the very least, you have to explain the harmful consequences of 
getting the charset wrong and the fact that in normal operational 
scenarios, if you leave it off you are less likely to have problems than 
if you provide it. -Tim

Date: Fri, 19 Sep 2003 13:40:35 -0400
To: Bjoern Hoehrmann <[email-masked]>, Tim Bray <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

At 03:50 03/09/19 +0200, Bjoern Hoehrmann wrote:

>You want to change something that has been STRONGLY RECOMMENDED for over
>five years to (ideally) MUST NOT just because it could cause trouble
>when used improperly or with broken implementations. Today I am good
>with web standards if I use the charset parameter, tommorow I am bad
>with web standards if I do. What's next on #W3C? Use tables for layout
>because people could get CSS wrong and old browsers get some CSS wrong?
>I don't think this leads anywhere.
>
>The charset parameter is useful if you cannot or do not want to use an
>encoding declaration,

Yes. One particular example that came up recently is the case of IE
going into quirks mode when seeing an XML declaration on an XHTML file.
I guess we can assume that those sites serving different content types
based on browser type can somehow set the charset parameter.

>for content negotiation, for view source
>functionality, if you perform protocol operations that change the
>encoding without changing the document or if you have to deal with
>legacy applications that could break your document if no charset
>parameter is present.

I'd also want to mention server technology that links the 'charset'
parameter with the actual encoding. For example, for Java servlets,
saying
     resource.setContentType ("type/foo;charset=encoding");
will not only produce the relevant header, it will also make
sure that the right conversion (from the internal UTF-16 to
the specified encoding) happens. It would be a bad idea
to disallow this because it works.

>I admit that there is probably no strong enough
>use case to introduce it, but we have the parameter already and it has
>been STRONGLY RECOMMENDED for ages across various W3C technologies.
>
>I can live with removing the STRONGLY RECOMMENDED status and an
>informative note that you typically do not need to specifiy the
>charset parameter but anything beyond that goes much too far.

I quite agree with this statement.

Regards,    Martin.

> >To put it another way, quoting Larry Wall: "An XML document knows what
> >encoding it's in."
>
><http://www.w3.org/People/Bos/DesignGuide/stability>:
>
>   ...
>   Having to re-learn how to do something is costly, creating new
>   programs to do the same thing in a different way is costly, and
>   converting existing documents and other resources to a different
>   format is also costly, so changes with little or no benefit should
>   be avoided.
>   ...

Date: Fri, 19 Sep 2003 13:19:46 -0400
To: MURATA Makoto <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

At 21:25 03/09/19 +0900, MURATA Makoto wrote:

>On Fri, 19 Sep 2003 03:50:11 +0200
>Bjoern Hoehrmann <[email-masked]> wrote:
>
> > Depends on the format. Formats should provide means to specify
> > the encoding, if they do not they are BAD, broken as designed.
>
>If we really promote in-band declarations, we should recommend
>one mechanism for all textual formats.  I think that CSS style
>declaration (@chaset) is the best I have seen.

Can you explain what in particular you like about the CSS
mechanism? Do you expect that all formats would use exactly
the same character sequence, or only that they would use
the same structure?

As an example, should C/C++ use '@charset', or should they
use a more C-like '#charset'?

Regards,    Martin.

>If the TAG recommends
>this mechanism for all formats except XML and HTML, that would
>be one reasonable principle.
>
>Cheers,
>
>--
>MURATA Makoto <[email-masked]>

Date: Fri, 19 Sep 2003 13:53:08 -0400
To: Tim Bray <[email-masked]>, Bjoern Hoehrmann <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

At 09:37 03/09/18 -0700, Tim Bray wrote:

>The argument is precisely is that it is not in the slightest useful. 
>Please read appendix F to the XML specification.  Then please suggest a 
>plausible scenario in which an XML instance unaccompanied by a charset 
>parameter can cause breakage.  You'll have to work hard.

No, I think it's quite easy. User buys an English XML book.
XML book has examples with and without XML declarations (and those
examples with an encoding parameter will use any of 'US-ASCII',
'iso-8859-1', and 'UTF-8'). The book probably somewhere has a note
somewhere explaining what the encoding parameter means,...,
but that might not be featured prominently, because it's not usually
that important for English.

User copies/adapts examples. Uses a text editor, adds text
(e.g. Polish, or Chinese, or whatever). The XML file is broken.

>To put it another way, quoting Larry Wall: "An XML document knows what 
>encoding it's in."

In the above example, it pretends to know, but it is wrong.

Regards,    Martin.

Date: Fri, 19 Sep 2003 10:55:02 -0700
From: Tim Bray <[email-masked]>
To: Larry Masinter <[email-masked]>
Cc: [email-masked]
Subject: Re: transcoding nearly certainly wrong?
Sender: [email-masked]

Larry Masinter wrote:

> Why is transcoding nearly certain to be wrong with XML?

First, transcoding XML is technically difficult, because you can't just 
do a dumb byte-level job, you have to be careful to also take care of 
the BOM & encoding declaration.  Second, given the wide range of 
encodings supported by popular deployed XML software, it is rarely the 
case that transcoding offers any benefit.  Something that is technically 
difficult (i.e. easy to get wrong) and offers little practical benefit 
is nearly certain to be wrong.

> Or, to put it another way, why not limit the use
> of text/xml to XML instances for which transcoding
> is certain not to be wrong, and for which US-ASCII
> is acceptable (because the XML uses numeric character
> references or character entities, is only used to
> code a limited schema with numeric data, etc.?)

This is plausible I guess, except for if you avoid text/*, you just 
don't need to worry whether they used entities the right way or what 
kind of data it is.  Essentially, application/* with no charset is 
highly robust.  Using text/* or adding a charset reduces that robustness 
and is just bad practice.

> Limiting the scope is less radical than deprecating.

True, but practices that cause problems while offering no practical 
benefits should be deprecated.
-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)

Date: Fri, 19 Sep 2003 12:50:00 -0400
To: John Cowan <[email-masked]>, Francois Yergeau <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

In the long term, I think the fundamental problem is with the large
number of encodings, not with the encoding identifications. The
example of US-ASCII very clearly shows the huge advantages of
having a single encoding.

This will of course take some time. As one example, N3 just says
that it's in UTF-8. For other new formats, that may make sense,
too.

Regards,     Martin.

At 12:10 03/09/19 -0400, John Cowan wrote:

>Francois Yergeau scripsit:
>
> > In this respect, yes.  All programming languages should provide for charset
> > identification of their source files.  Alas, none do, AFAIK.
>
>I almost, but not quite, entirely disagree with this position.
>
>Rather than having thousands of ad hoc mechanisms for encoding declarations
>in each of the thousands of text formats now extant, file systems should have
>a convenient mechanism for recording the encoding of each file, and character
>processing libraries should have convenient reading and writing operations 
>that
>do the necessary conversions.  Otherwise, generic text-processing tools become
>impossible, because each tool has to have a vast library that understands the
>mechanics of the encoding declaration specific to the format it is trying to
>read.  That way madness lies.

Date: Fri, 19 Sep 2003 12:10:17 -0400
From: John Cowan <[email-masked]>
To: Francois Yergeau <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Francois Yergeau scripsit:

> In this respect, yes.  All programming languages should provide for charset
> identification of their source files.  Alas, none do, AFAIK.

I almost, but not quite, entirely disagree with this position.

Rather than having thousands of ad hoc mechanisms for encoding declarations
in each of the thousands of text formats now extant, file systems should have
a convenient mechanism for recording the encoding of each file, and character
processing libraries should have convenient reading and writing operations that
do the necessary conversions.  Otherwise, generic text-processing tools become
impossible, because each tool has to have a vast library that understands the
mechanics of the encoding declaration specific to the format it is trying to
read.  That way madness lies.

-- 
As you read this, I don't want you to feel      John Cowan 
sorry for me, because, I believe everyone       [email-masked]
will die someday.    -- From a Nigerian-type    http://www.reutershealth.com
                        scam spam I got         http://www.ccil.org/~cowan

To: MURATA Makoto <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
From: [email-masked]
Date: Fri, 19 Sep 2003 11:04:11 -0400
Sender: [email-masked]

Murata Makoto writes:

>> I believe that SOAP implementations use the 
>> charset parameter.  If we remove the charset 
>> parameter, we will make them non-conformant.

This is not my area of expertise, but I note that the HTTP binding [1] 
provided by SOAP 1.2 Recommendation uses an application/soap+xml media 
type, a definition of which is at [2] (I believe it is working its way 
through the formal registration process.)  My reading is that the 
definition lists charset as optional, and makes clear that its proper use 
is to be found in RFC 3023. 

I am not aware of what typical implementations of SOAP 1.1 or SOAP 1.2 are 
doing, but the 1.2 spec at least seems to list it as optional.  Again, I'm 
not expert in this stuff and am not offering an opinion, but I thought the 
links might be helpful.

[1]http://www.w3.org/TR/soap12-part2/#soapinhttp
[2] http://www.w3.org/TR/soap12-part2/#ietf-draft

------------------------------------------------------------------
Noah Mendelsohn                              Voice: 1-617-693-4036
IBM Corporation                                Fax: 1-617-693-8676
One Rogers Street
Cambridge, MA 02142
------------------------------------------------------------------

MURATA Makoto <[email-masked]>
Sent by: [email-masked]
09/19/03 08:10 AM

        To:     [email-masked], WWW-Tag <[email-masked]>
        cc:     (bcc: Noah Mendelsohn/Cambridge/IBM)
        Subject:        Re: Requesting a revision of RFC3023

On Fri, 19 Sep 2003 03:50:11 +0200
Bjoern Hoehrmann <[email-masked]> wrote:

> You want to change something that has been STRONGLY RECOMMENDED for over
> five years to (ideally) MUST NOT just because it could cause trouble
> when used improperly or with broken implementations. Today I am good
> with web standards if I use the charset parameter, tommorow I am bad
> with web standards if I do. What's next on #W3C? Use tables for layout
> because people could get CSS wrong and old browsers get some CSS wrong?
> I don't think this leads anywhere.

I believe that SOAP implementations use the charset parameter.  If we 
remove the charset parameter, we will make them non-conformant.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Fri, 19 Sep 2003 10:02:05 -0400
To: Larry Masinter <[email-masked]>, Tim Bray <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: transcoding nearly certainly wrong?
Cc: [email-masked]
Sender: [email-masked]

Hello Larry,

At 23:36 03/09/18 -0700, Larry Masinter wrote:

>Why is transcoding nearly certain to be wrong with XML?

The argument is that people are not able to set up
their servers correctly.

>Or, to put it another way, why not limit the use
>of text/xml to XML instances for which transcoding
>is certain not to be wrong, and for which US-ASCII
>is acceptable (because the XML uses numeric character
>references or character entities, is only used to
>code a limited schema with numeric data, etc.?)
>
>Limiting the scope is less radical than deprecating.

I'm not sure this makes sense. For types with mostly
just numeric data, text/... may be the wrong type.

And defining types that are XML, but restricted to
US-ASCII would be a really bad idea for types that
actually can contain text.

Regards,    Martin.

From: Francois Yergeau <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 10:11:09 -0400
Sender: [email-masked]

MURATA Makoto wrote:
> If we really promote in-band declarations, we should recommend 
> one mechanism for all textual formats.  I think that CSS style 
> declaration (@chaset) is the best I have seen.  If the TAG recommends 
> this mechanism for all formats except XML and HTML, that would 
> be one reasonable principle.

The Character Model section 3.6.2 [1] addresses this question but stops
short of recommending a single mechanism.  I guess you think it should.

[1] http://www.w3.org/TR/charmod/#sec-EncodingIdent

-- 
François Yergeau

From: Francois Yergeau <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 10:08:42 -0400
Sender: [email-masked]

John Cowan wrote:
>Programming languages are broken as designed?

In this respect, yes.  All programming languages should provide for charset
identification of their source files.  Alas, none do, AFAIK.

-- 
François

From: Bjoern Hoehrmann <[email-masked]>
To: John Cowan <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 15:41:27 +0200
Sender: [email-masked]

* John Cowan wrote:
>Bjoern Hoehrmann scripsit:
>
>> Depends on the format. Formats should provide means to specify
>> the encoding, if they do not they are BAD, broken as designed.
>
>Plain text is broken as designed?  Programming languages are broken
>as designed?

If a processor is required to guess how to process instances of a
format, the format is clearly broken as designed, yes.

Date: Fri, 19 Sep 2003 21:25:06 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

On Fri, 19 Sep 2003 03:50:11 +0200
Bjoern Hoehrmann <[email-masked]> wrote:

> Depends on the format. Formats should provide means to specify
> the encoding, if they do not they are BAD, broken as designed.

If we really promote in-band declarations, we should recommend 
one mechanism for all textual formats.  I think that CSS style 
declaration (@chaset) is the best I have seen.  If the TAG recommends 
this mechanism for all formats except XML and HTML, that would 
be one reasonable principle.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Fri, 19 Sep 2003 21:10:52 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

On Fri, 19 Sep 2003 03:50:11 +0200
Bjoern Hoehrmann <[email-masked]> wrote:

> You want to change something that has been STRONGLY RECOMMENDED for over
> five years to (ideally) MUST NOT just because it could cause trouble
> when used improperly or with broken implementations. Today I am good
> with web standards if I use the charset parameter, tommorow I am bad
> with web standards if I do. What's next on #W3C? Use tables for layout
> because people could get CSS wrong and old browsers get some CSS wrong?
> I don't think this leads anywhere.

I believe that SOAP implementations use the charset parameter.  If we 
remove the charset parameter, we will make them non-conformant.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Fri, 19 Sep 2003 20:56:08 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked], "WWW-Tag" <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

> > [5] http://www.asahi-net.or.jp/~eb2m-mrt/charsetDetection.html
> 
> I was quite surprised you didn't address the role of byte order marks.
> BOMs are not limited to XML, neither do they depend on MIME or HTTP. Of
> course, they can only be used for UTF encodings.

Thanks for this comment.  I think that the use of the BOM or Unicode signature 
is an example of charset sniffing based on byte patterns.  However, I agree that 
my document was not clear enough.  I revised it.

Cheers,

-- 
MURATA Makoto <[email-masked]>

Date: Fri, 19 Sep 2003 20:54:01 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

> I happen to agree with that.  But please consider that saying 'XML is text'
> is not at all the same thing as saying 'XML is "text/"', as the latter is a
> MIME concept that carries a lot of additional (and unfortunate) baggage.

I agree with François because I have learned from Ned Freed that text/* is not 
always appropriate for textual data.

http://www.imc.org/ietf-xml-mime/mail-archive/msg00203.html

-- 
MURATA Makoto <[email-masked]>

Date: Thu, 18 Sep 2003 23:36:41 -0700
From: Larry Masinter <[email-masked]>
Subject: transcoding nearly certainly wrong?
To: Tim Bray <[email-masked]>
Cc: [email-masked]
Importance: Normal
Sender: [email-masked]

Why is transcoding nearly certain to be wrong with XML?

Or, to put it another way, why not limit the use
of text/xml to XML instances for which transcoding
is certain not to be wrong, and for which US-ASCII
is acceptable (because the XML uses numeric character
references or character entities, is only used to
code a limited schema with numeric data, etc.?)

Limiting the scope is less radical than deprecating.

Date: Fri, 19 Sep 2003 00:28:53 -0400
To: Bjoern Hoehrmann <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
From: John Cowan <[email-masked]>
Sender: [email-masked]

Bjoern Hoehrmann scripsit:

> Depends on the format. Formats should provide means to specify
> the encoding, if they do not they are BAD, broken as designed.

Plain text is broken as designed?  Programming languages are broken
as designed?

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.reutershealth.com
"In computer science, we stand on each other's feet."
        --Brian K. Reid

From: Bjoern Hoehrmann <[email-masked]>
To: Tim Bray <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Fri, 19 Sep 2003 03:50:11 +0200
Sender: [email-masked]

* Tim Bray wrote:
>Agreed, which is another of the advantages of XML, since it doesn't need 
>a charset parameter.  You are right about the shortcomings of the 
>charset parameter but for the moment it's the best tool we have.

Depends on the format. Formats should provide means to specify
the encoding, if they do not they are BAD, broken as designed.

>>>I agree, but for XML formats, I still think the charset parameter is 
>>>actively harmful and should be deprecated or even forbidden.
>> 
>> Deprecating something useful just because it could cause trouble when
>> used improperly does not make sense to me.
>
>The argument is precisely is that it is not in the slightest useful.

Which makes me wonder why there is such a parameter. I think W3C should
have raised this concern during IESG review of RFC 2376. Complaining
about it know seems a bit late.

>Please read appendix F to the XML specification.  Then please suggest a 
>plausible scenario in which an XML instance unaccompanied by a charset 
>parameter can cause breakage.  You'll have to work hard.  Then suggest a 
>dozen ways in which deployed software is known to get the charset wrong. 
>You'll have no trouble.

I will neither have trouble to suggest ways in which deployed software
is known to get something wrong when the encoding declaration or the
byte order mark are involved, especially if those are used improperly.
But my logical conclusion is not to forbid the byte order mark or the
encoding declaration.

You want to change something that has been STRONGLY RECOMMENDED for over
five years to (ideally) MUST NOT just because it could cause trouble
when used improperly or with broken implementations. Today I am good
with web standards if I use the charset parameter, tommorow I am bad
with web standards if I do. What's next on #W3C? Use tables for layout
because people could get CSS wrong and old browsers get some CSS wrong?
I don't think this leads anywhere.

The charset parameter is useful if you cannot or do not want to use an
encoding declaration, for content negotiation, for view source
functionality, if you perform protocol operations that change the
encoding without changing the document or if you have to deal with
legacy applications that could break your document if no charset
parameter is present. I admit that there is probably no strong enough
use case to introduce it, but we have the parameter already and it has
been STRONGLY RECOMMENDED for ages across various W3C technologies.

I can live with removing the STRONGLY RECOMMENDED status and an
informative note that you typically do not need to specifiy the
charset parameter but anything beyond that goes much too far.

>To put it another way, quoting Larry Wall: "An XML document knows what 
>encoding it's in."

<http://www.w3.org/People/Bos/DesignGuide/stability>:

  ...
  Having to re-learn how to do something is costly, creating new
  programs to do the same thing in a different way is costly, and
  converting existing documents and other resources to a different
  format is also costly, so changes with little or no benefit should
  be avoided.
  ...

Date: Thu, 18 Sep 2003 09:37:50 -0700
From: Tim Bray <[email-masked]>
To: Bjoern Hoehrmann <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

Bjoern Hoehrmann wrote:

> The charset parameter does not work as web servers or web server
> configurations often do not allow content providers to setup this
> parameter. In general, technical solutions should either work
> automagically behind the scene or be visible to those who need to
> be aware; HTTP headers are invisible to most content providers so
> I doubt that this is an architecturally-sound position.

Agreed, which is another of the advantages of XML, since it doesn't need 
a charset parameter.  You are right about the shortcomings of the 
charset parameter but for the moment it's the best tool we have.

>>I agree, but for XML formats, I still think the charset parameter is 
>>actively harmful and should be deprecated or even forbidden.
> 
> Deprecating something useful just because it could cause trouble when
> used improperly does not make sense to me.

The argument is precisely is that it is not in the slightest useful. 
Please read appendix F to the XML specification.  Then please suggest a 
plausible scenario in which an XML instance unaccompanied by a charset 
parameter can cause breakage.  You'll have to work hard.  Then suggest a 
dozen ways in which deployed software is known to get the charset wrong. 
  You'll have no trouble.  Given that Web Architecture says that the 
charset takes precedence *if provided*, and given that this can add no 
useful information and can easily cause breakage if (as is known to 
happen) it is wrong, there is normally no good reason to provide a 
charset parameter for a media type which is known to be XML.

Francois Yergeau managed to dream up a couple of scnenarios where you 
might want a charset with XML, but they were somewhere on the border 
between obscure and bad practice (as I'm sure Francois would agree).

To put it another way, quoting Larry Wall: "An XML document knows what 
encoding it's in."
-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)

Date: Thu, 18 Sep 2003 10:53:30 -0400
From: "Simon St.Laurent" <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
cc: [email-masked], WWW-Tag <[email-masked]>
Sender: [email-masked]

[Should we be cross-posting?  I suspect ietf-xml-mime is probably a
better forum for detailed discussion, but as TAG made the initial
request, maybe cross-posting is appropriate.  Someone please advise.]

[email-masked] (Francois Yergeau) writes:
>Tim Bray wrote:
>> I agree entirely with Michael and feel that the, er, 
>> textuality of XML is at the centre of everything.
>
>I happen to agree with that.  But please consider that saying 'XML is
>text' is not at all the same thing as saying 'XML is "text/"', as the
>latter is a MIME concept that carries a lot of additional (and
>unfortunate) baggage.

That's my position as well, and a nice way to say it.  While in more
general terms I feel very strongly that XML is text, I don't feel that
XML is text/.

It's unfortunate that text/ doesn't correspond cleanly with text, but
there are lots of good historical reasons for the baggage.  The cost of
that baggage for XML is such that I'm happy to move to application/.

From: Francois Yergeau <[email-masked]>
To: "'Tim Bray'" <[email-masked]>, "C. M. Sperberg-McQueen" <[email-masked]>
Cc: MURATA Makoto <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Thu, 18 Sep 2003 10:16:48 -0400
Sender: [email-masked]

Tim Bray wrote:
> I agree entirely with Michael and feel that the, er, 
> textuality of XML is at the centre of everything.

I happen to agree with that.  But please consider that saying 'XML is text'
is not at all the same thing as saying 'XML is "text/"', as the latter is a
MIME concept that carries a lot of additional (and unfortunate) baggage.

> Thus, I disagree with Francois and 
> Makoto in their contention that XML is not usefully 
> considered as text for humans to look at.

I'm afraid this is the case for most humans in most situations.  Sure, there
are people who like to view and edit XML as plain text (pleading guilty
here;-), but deprecating "text/*xml" won't stop me: save to disk and fire up
the editor.  I don't fell I'm going to lose anything of great pragmatic
value, although there is indeed a loss of conceptual value in not calling
XML what it is: text.

> Having said that, I feel that the use of media 
> types beginning with "text/" remains inappropriate, but primarily 
> because of the charset defaulting baggage that comes with those five 
> characters.

Yes.  Oh, and "text/" also implies the MIME canonical form with short lines
delimited by CR-LF, making UTF-16 and UTF-32 impossible, etc.  Broken.  To
be deprecated.

-- 
François

From: Christoph Schneegans <[email-masked]>
To: <[email-masked]>
Cc: "WWW-Tag" <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Thu, 18 Sep 2003 12:45:27 GMT
Lines: 12
Sender: [email-masked]

MURATA Makoto wrote:

> At present, different technologies have introduced different ad-hoc
> solutions  [5].
>
> [5] http://www.asahi-net.or.jp/~eb2m-mrt/charsetDetection.html

I was quite surprised you didn't address the role of byte order marks.
BOMs are not limited to XML, neither do they depend on MIME or HTTP. Of
course, they can only be used for UTF encodings.

-- 
<http://schneegans.de/>

Date: Thu, 18 Sep 2003 08:17:05 -0400
To: MURATA Makoto <[email-masked]>, [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

At 08:34 03/09/18 +0900, MURATA Makoto wrote:

> > Hmm, the TAG is pretty convinced that 3023 needs to change, so maybe Dan
> > or Chris or TimBL could take this up internally.  I disagree that this
> > should be frozen at the moment, since the TAG is quite likely to publish
> > a document saying "RFC 3023 is wrong".
>
>Since the TAG and the W3C team share some members, please have some
>internal discussion.  Our hands are tied and are also requested to
>take some action.  Please do not blame us.

Tim is currently in the UK. I'll check directly with him when he is
back next week. The idea is that we can pool various needed updates
rather than reissuing the RFC several times.

Regards,    Martin.

Date: Wed, 17 Sep 2003 22:44:04 -0700
From: Tim Bray <[email-masked]>
To: "C. M. Sperberg-McQueen" <[email-masked]>
Cc: MURATA Makoto <[email-masked]>, [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

C. M. Sperberg-McQueen wrote:

> I'm sorry to see two people whose opinions I value so highly
> agreeing with a position that so troubles me.  One of the 
> most important characteristics of XML, as compared with many,
> many competing formats for the storage and/or transmission of
> data is that it is textual

I agree entirely with Michael and feel that the, er, textuality of XML 
is at the centre of everything.  Thus, I disagree with Francois and 
Makoto in their contention that XML is not usefully considered as text 
for humans to look at.  Having said that, I feel that the use of media 
types beginning with "text/" remains inappropriate, but primarily 
because of the charset defaulting baggage that comes with those five 
characters.  And secondarily because of the fact that the rules allow 
transcoding, which is nearly certain to be wrong with XML.  The second 
exercises me less because I don't think it's actually happening, so the 
core issue is really the charset default stuff.
-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)

Subject: Re: Requesting a revision of RFC3023
From: "C. M. Sperberg-McQueen" <[email-masked]>
To: MURATA Makoto <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Organization: 
Date: 17 Sep 2003 21:54:07 -0600
Sender: [email-masked]

On Wed, 2003-09-17 at 17:34, MURATA Makoto wrote:
> On Wed, 17 Sep 2003 14:10:53 -0400
> Francois Yergeau <[email-masked]> wrote:
> 
> > But stating that "most XML is not text for casual users" says that there is
> > no loss in deprecating text/*xml (save perhaps transition issues), the text/
> > top-level buys nothing of value.
> 
> Agreed.

I'm sorry to see two people whose opinions I value so highly
agreeing with a position that so troubles me.  One of the 
most important characteristics of XML, as compared with many,
many competing formats for the storage and/or transmission of
data is that it is textual (in the sense of being conceptually
a sequence of characters, and represented on the wire -- at least
so far -- as such).  Since much of the XML which I care about
is also a digital representation of texts (in the sense of
being natural-language utterances with a certain degree of
intra-document linguistic and thematic cohesion), it troubles
me to think that labeling XML as text buys us nothing of
value.  On the contrary, I think: it stresses two important
facts.

I assume that both of you are thinking primarily of browser 
fallback behavior, and reflecting your view that users will
be able to make nothing of XML source if they are confronted
with it -- in that context, I understand, the proposition you
endorse is at least plausible.  Even there, though, I don't
find it compelling:  much XML is quite legible to naive humans --
as legible as anything displayed without much intelligent
formatting will ever be.  And for some humans, almost all XML
is legible without special tools.  (I recognize that the latter
group is a relatively small minority of the human population.)

-C. M. Sperberg-McQueen

From: Bjoern Hoehrmann <[email-masked]>
To: Tim Bray <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Date: Thu, 18 Sep 2003 02:42:53 +0200
Sender: [email-masked]

* Tim Bray wrote:
>I read [3] and while I agree with much of it, it's obviously far too 
>late to change the XML encoding declaration.  For the moment, I think 
>that the architecturally-sound position is, for Web data formats, either 
>(a) use XML, or (b) use the charset parameter.

The charset parameter does not work as web servers or web server
configurations often do not allow content providers to setup this
parameter. In general, technical solutions should either work
automagically behind the scene or be visible to those who need to
be aware; HTTP headers are invisible to most content providers so
I doubt that this is an architecturally-sound position.

>I agree, but for XML formats, I still think the charset parameter is 
>actively harmful and should be deprecated or even forbidden.

Deprecating something useful just because it could cause trouble when
used improperly does not make sense to me.

Date: Thu, 18 Sep 2003 08:34:30 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked]
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

On Wed, 17 Sep 2003 14:10:53 -0400
Francois Yergeau <[email-masked]> wrote:

> But stating that "most XML is not text for casual users" says that there is
> no loss in deprecating text/*xml (save perhaps transition issues), the text/
> top-level buys nothing of value.

Agreed.

-- 
MURATA Makoto <[email-masked]>

Date: Thu, 18 Sep 2003 08:34:21 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked]
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

> Hmm, the TAG is pretty convinced that 3023 needs to change, so maybe Dan 
> or Chris or TimBL could take this up internally.  I disagree that this 
> should be frozen at the moment, since the TAG is quite likely to publish 
> a document saying "RFC 3023 is wrong".

Since the TAG and the W3C team share some members, please have some 
internal discussion.  Our hands are tied and are also requested to 
take some action.  Please do not blame us.

> > As for the charset parameter, I am still uneasy to disallow or
> > deprecate it.  
...
> 
> I think I provided a detailed explanation of why the charset is in fact 
> actively harmful in the context of XML.  If you're not convinced it 
> would be helpful if you could address those points.  If you already 
> have, my apologies, perhaps you could give a pointer.

First, I agree on your request beginning "it should be made clear 
that nobody sending a media-type should send a charset for an XML 
media-type unless it REALLY REALLY KNOWS what it's sending".  Is 
this an acceptable position?

You explained that an ad-hoc mechanism (i.e., encoding dcls) almost perfectly work 
for XML, and I agree.  The above position implies that (1) the ad-hoc mechanism 
is allowed, (2) the generic mechanism is recommended but optional, and (3) if
specified, the generic mechanism takes precedence.  You provided a detailed 
explanation of (1), but your argument against (3) is not persuasive.

> I read [3] and while I agree with much of it, it's obviously far too 
> late to change the XML encoding declaration.  

True.  But what happens to upcoming non-XML formats?

>For the moment, I think 
> that the architecturally-sound position is, for Web data formats, either 
> (a) use XML, or (b) use the charset parameter.  

It would be great if the TAG finding document explicitly state this.  If 
we reach a consensus on this principle, my worries will be lessened.  
I am worried because I think the TAG is trying to deprecate the generic mechanism 
without establishing any principle.

> > 	(1) non-self-describing data formats should rely on the
> >             charset parameter, and
> > 	(2) self-describing data formats should introduce their own
> > 	    mechanism for specifying charsets.
> 
> I'll review the webarch doc, I suspect we haven't thought closely enough 
> about this.

Please think about this.  Frankly, as for this issue, I think that the 
TAG does not have an architecture but only has an ad-hoc solution.   Does 
the I18N WG agree with the position of the TAG?

> I agree, but for XML formats, I still think the charset parameter is 
> actively harmful and should be deprecated or even forbidden.  This is 
> orthogonal to the larger question you (correctly) raise, of charset 
> detection for non-XML formats.

I do not think that this is orthogonal.  This is the point.  Should we 
deprecate the generic mechanism in the particular case that an 
ad-hoc mechanism works better?  WWW programmers are already tired of 
ad-hoc mechanisms.

CHeers,

Makoto

From: Francois Yergeau <[email-masked]>
To: "'MURATA Makoto'" <[email-masked]>, [email-masked]
Cc: WWW-Tag <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Wed, 17 Sep 2003 14:10:53 -0400
Sender: [email-masked]

MURATA Makoto wrote:
> and to deprecate text/xml not
> because the charset parameter is harmful but because most XML is not
> text for casual users.

It's not the charset parameter that's harmful, it's its absence.  Or rather
the mandated policy when it is absent: MUST assume ASCII.

But stating that "most XML is not text for casual users" says that there is
no loss in deprecating text/*xml (save perhaps transition issues), the text/
top-level buys nothing of value.

Regards,

-- 
François Yergeau

From: Francois Yergeau <[email-masked]>
To: "'Tim Bray'" <[email-masked]>, [email-masked]
Cc: WWW-Tag <[email-masked]>
Subject: RE: Requesting a revision of RFC3023
Date: Wed, 17 Sep 2003 14:04:12 -0400
Sender: [email-masked]

Tim Bray wrote:
> I took an action item to ask about the chances of revising what 3023 
> says about the charset parameter; while I'm not sure, I suspect that 
> there may actually be some level of consensus about the 
> desirable changes:

Quite possible.

> 1. Deprecate text/* for anything that's in XML.

Yes, please!

> 2. Deprecate the charset parameter for application/xml and 
> application/*+xml.

This would hurt some legitimate (but probably rare, perhaps unimportant) use
cases where:

1) a server generates some XML without an encoding decl, knowing that it
will send an authoritative charset parameter, or

2) a proxy transcodes an XML entity without fixing the encoding decl
therein,  knowing that it will send an authoritative charset parameter.

I'm not saying this kills the proposal, just that it needs to be weighed in.
Perhaps it will make the difference between "deprecate", "outlaw" or various
strengths of "discourage".

> For the server, on the other hand, 
> this is easy to get wrong, particularly with the introduction 
> of various kinds of filters in modern web servers.

No need to invoke filters, older and simpler servers are known historically
to get it wrong most of the time.

> it should be made clear that nobody sending a media-type 
> should send a charset for an XML media-type unless it REALLY REALLY KNOWS
what it's 
> sending, and in that case should consider not sending it anyhow.

Yes.

-- 
François Yergeau

Date: Wed, 17 Sep 2003 13:55:41 -0400
From: John Cowan <[email-masked]>
To: MURATA Makoto <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

MURATA Makoto scripsit:

> This implies that XQuery should introduce its own mechanism (see [4])
> and that the compact syntax of RELAX NG should introduce another.

I don't see that either of these are self-describing in the relevant sense,
any more than a C program is.

> As far as I know, the charset parameter is the only generic mechanism.  I 
> know the charset parameter is not working well, but I do not see any other 
> generic mechanisms.

It's an acceptable mechanism when there is no internal declaration present.
It causes problems when internal declarations are or may be present.

-- 
John Cowan  [email-masked]  www.ccil.org/~cowan  www.reutershealth.com
"In computer science, we stand on each other's feet."
        --Brian K. Reid

Date: Wed, 17 Sep 2003 10:38:47 -0700
From: Tim Bray <[email-masked]>
To: MURATA Makoto <[email-masked]>
Cc: [email-masked], WWW-Tag <[email-masked]>
Subject: Re: Requesting a revision of RFC3023
Sender: [email-masked]

MURATA Makoto wrote:

> First, Simon and I were asked by the W3C team not to take any action
> on RFC 3023.  This is because the MIME type registration procedure was
> expected to change (see [1] and [2]).  So, Simon, Dan, and I can't do 
> anything right now.

Hmm, the TAG is pretty convinced that 3023 needs to change, so maybe Dan 
or Chris or TimBL could take this up internally.  I disagree that this 
should be frozen at the moment, since the TAG is quite likely to publish 
a document saying "RFC 3023 is wrong".

> As for the charset parameter, I am still uneasy to disallow or
> deprecate it.  But I agree to make "clear that nobody sending a
> media-type should send a charset for an XML media-type unless it
> REALLY REALLY KNOWS what it's sending," and to deprecate text/xml not
> because the charset parameter is harmful but because most XML is not
> text for casual users.

I think I provided a detailed explanation of why the charset is in fact 
actively harmful in the context of XML.  If you're not convinced it 
would be helpful if you could address those points.  If you already 
have, my apologies, perhaps you could give a pointer.

> I have repeatedly asked (e.g., [3]) what is the position of the TAG on
> charset detection for non-XML formats.  The latest version of the TAG
> finding document "Client handling of MIME headers" appears to
> recommend:

I read [3] and while I agree with much of it, it's obviously far too 
late to change the XML encoding declaration.  For the moment, I think 
that the architecturally-sound position is, for Web data formats, either 
(a) use XML, or (b) use the charset parameter.  I'm generally in favor 
of a general-purpose encoding-detection scheme such as you propose, but 
I'm pessimistic about getting it widely deployed for legacy formats.

> 	(1) non-self-describing data formats should rely on the
>             charset parameter, and
> 	(2) self-describing data formats should introduce their own
> 	    mechanism for specifying charsets.

I'll review the webarch doc, I suspect we haven't thought closely enough 
about this.

> As far as I know, the charset parameter is the only generic mechanism.  I 
> know the charset parameter is not working well, but I do not see any other 
> generic mechanisms.

I agree, but for XML formats, I still think the charset parameter is 
actively harmful and should be deprecated or even forbidden.  This is 
orthogonal to the larger question you (correctly) raise, of charset 
detection for non-XML formats.

-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)

Date: Thu, 18 Sep 2003 01:30:54 +0900
From: MURATA Makoto <[email-masked]>
To: [email-masked]
Subject: Re: Requesting a revision of RFC3023
Cc: WWW-Tag <[email-masked]>
Sender: [email-masked]

First, Simon and I were asked by the W3C team not to take any action
on RFC 3023.  This is because the MIME type registration procedure was
expected to change (see [1] and [2]).  So, Simon, Dan, and I can't do 
anything right now.

As for the charset parameter, I am still uneasy to disallow or
deprecate it.  But I agree to make "clear that nobody sending a
media-type should send a charset for an XML media-type unless it
REALLY REALLY KNOWS what it's sending," and to deprecate text/xml not
because the charset parameter is harmful but because most XML is not
text for casual users.

I have repeatedly asked (e.g., [3]) what is the position of the TAG on
charset detection for non-XML formats.  The latest version of the TAG
finding document "Client handling of MIME headers" appears to
recommend:

	(1) non-self-describing data formats should rely on the
            charset parameter, and

	(2) self-describing data formats should introduce their own
	    mechanism for specifying charsets.

This implies that XQuery should introduce its own mechanism (see [4])
and that the compact syntax of RELAX NG should introduce another.
CSS, HTML, and XML already have different mechanisms.  I personally
think that this approach will make the current situation even worse.

Many textual data on the WWW requires charset detection.  For example:

1) plain text, XML, HTML, CSS, XQuery, VBScript, Javascript, JSP, perl, 
   RNG compact schemas,  etc. on the server side,

2) textual data generated by CGI programs, Servlets,  Applets, XSLT 
   stylesheets, etc. on the server side,

3) text typed in forms of HTML and sent as multipart/form-data via 
   HTTP also require charset information.

At present, different technologies have introduced different ad-hoc 
solutions  [5].  As a result, it is VERY HARD  to create well-internationalized 
WWW applications.  You have to use several mechanisms correctly.  Nobody
is trying to provide a generalized solution.

As far as I know, the charset parameter is the only generic mechanism.  I 
know the charset parameter is not working well, but I do not see any other 
generic mechanisms.

[1] http://lists.w3.org/Archives/Public/public-ietf-w3c/2003Jul/0000.html
[2] http://www.ietf.org/internet-drafts/draft-freed-mime-p4-02.txt
[3] http://lists.w3.org/Archives/Public/www-tag/2003Apr/0104.html
[4] http://www.w3.org/TR/xquery/#xquery-encoding
[5] http://www.asahi-net.or.jp/~eb2m-mrt/charsetDetection.html

Cheers,

Makoto
-- 
MURATA Makoto <[email-masked]>

Date: Tue, 16 Sep 2003 10:50:11 -0700
From: Tim Bray <[email-masked]>
To: [email-masked]
Subject: Requesting a revision of RFC3023
Sender: [email-masked]

[Some of you will get this twice, sorry; Larry Masinter pointed out that 
my initial choice of destinations was poor.  I slightly revised the note 
to provide more context.]

The W3C TAG (http://www.w3.org/2001/tag/) has an open issue about proper 
handling of MIME headers, with a draft in progress "Client Handling of 
MIME Headers" (http://www.w3.org/2001/tag/doc/mime-respect.html); the 
draft finds some fault with the contents of RFC3023.

I took an action item to ask about the chances of revising what 3023 
says about the charset parameter; while I'm not sure, I suspect that 
there may actually be some level of consensus about the desirable changes:

1. Deprecate text/* for anything that's in XML.  That's because it 
forces the provider to provide a charset header, because in its absence 
the receiver is required to assume either ASCII or 8859 depending on the 
context, which has a very high probability of being wrong, which is 
irritating because if there were no charset header the client would be 
certain of either getting it right or failing deterministically.  And 
forcing the server to provide a charset= is wrong; see the next point.

2. Deprecate the charset parameter for application/xml and 
application/*+xml.  I think that Roy Fielding would like to go far as to 
simply outlaw it; I'd be fine with that too.  The reason is that the 
client is almost certain to get it right, and will fail 
deterministically if it doesn't.  For the server, on the other hand, 
this is easy to get wrong, particularly with the introduction of various 
kinds of filters in modern web servers.  And since the Web architecture 
and the XML spec both say that the server's claim has to be taken as 
authoritative, this is really highly dysfunctional.  At the very least, 
it should be made clear that nobody sending a media-type should send a 
charset for an XML media-type unless it REALLY REALLY KNOWS what it's 
sending, and in that case should consider not sending it anyhow.

Is there any chance we could do this?  It's going to be kind of 
embarrassing for TAG findings and the Webarch doc to be saying "don't do 
what this RFC says".
-- 
Cheers, Tim Bray
         (ongoing fragmented essay: http://www.tbray.org/ongoing/)