Subject: [websec] #22: content-type sniffing should include charset sniffing

From: "websec issue tracker" <[email-masked]>
Date: Sun, 23 Oct 2011 23:55:03 -0000

#22: content-type sniffing should include charset sniffing

 the HTML5 spec contains some algorithms for sniffing charset, overriding
 labeled charset, etc.

 MIME parameters like charset are as much a part of the content-type as the
 base internet media type, and any sniffing of parameters and other
 metadata (overriding content-type or guessing where it is not supplied or
 wrong) should be included in this document, since the sniffing will happen
 at the same time.

-- 
------------------------+--------------------------------------------
 Reporter:  masinter@…  |      Owner:  draft-ietf-websec-mime-sniff@…
     Type:  defect      |     Status:  new
 Priority:  major       |  Milestone:
Component:  mime-sniff  |    Version:
 Severity:  -           |   Keywords:
------------------------+--------------------------------------------

Ticket URL: <http://trac.tools.ietf.org/wg/websec/trac/ticket/22>
websec <http://tools.ietf.org/websec/>

---

From: Tobias Gondrom <[email-masked]>
Date: Mon, 24 Oct 2011 04:24:23 +0100

<hat="individual">
I tend not to agree with that.

The fact that charset sniffing might happen at the same time as 
mime-sniffing does not seem like a strong argument to include this in 
the draft.

Furthermore I would rather have these issues separate:
First you determine the content-type and then after that you may want to 
determine the charset used within that content-type (if you really have 
to sniff the charset). I can also imagine that charset sniffing 
algorithm might be depending on the application identified by the 
sniffed mime-type, which again would speak against throwing it in 
together with mime-sniffing....

Kind regards, Tobias

---

From: Adam Barth <[email-masked]>
Date: Sun, 23 Oct 2011 20:28:28 -0700

The charset sniffing is also complicated by the fact that sometimes
user agents need to parse some of the HTML to find a <meta> element.
In some situations, user agents need to restart the parsing algorithm,
which is quite delicate and better to describe in the same document as
HTML parsing (at least for use by HTML processing engines).

Adam

On Sun, Oct 23, 2011 at 8:24 PM, Tobias Gondrom
<[email-masked]> wrote:
> mime-sniffing does not seem like a strong argument to include this in the
> draft.
>
> determine the charset used within that content-type (if you really have to
> sniff the charset). I can also imagine that charset sniffing algorithm might
> be depending on the application identified by the sniffed mime-type, which
> again would speak against throwing it in together with mime-sniffing....
>
> Kind regards, Tobias
>
>
>
> _______________________________________________
> websec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/websec
>

---

From: Larry Masinter <[email-masked]>
Date: Sun, 23 Oct 2011 20:29:57 -0700

> First you determine the content-type and then after that you may want to determine the charset used within that content-type

That's wishful thinking that doesn't match what has to happen ... the mime-sniffing document ALREADY is looking at the charset, by looking for byte-order-mark signatures to decide whether the content is text or binary.
So we're already doing charset detection, just not calling it that or completely specifying it.

---

From: Adam Barth <[email-masked]>
Date: Sun, 23 Oct 2011 20:31:55 -0700

> So we're already doing charset detection, just not calling it that or completely specifying it.

I mean, the full charset sniffing algorithm is significantly more
complicated that that.  For example, it also takes into account other
frames being rendering by the user agent.

Adam

---

From: Larry Masinter <[email-masked]>
Date: Sun, 23 Oct 2011 20:32:51 -0700

I know it's complicated, but scanning text is necessarily part of determining which application/something+xml  you have.  I think (but should really check before saying this) that XML media type registrations describe what the DOCTYPE or XML namespace or root element are, and that, to properly "sniff" them, you'd have to scan text. But before you scan text, you have to determine charset.

So if we're going to support sniffing of media types in general, I don't see how we can do that without also specifying charset determination.

Larry
]

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Adam Barth
Sent: Sunday, October 23, 2011 8:28 PM
To: Tobias Gondrom
Cc: [email-masked]
Subject: Re: [websec] #22: content-type sniffing should include charset sniffing

The charset sniffing is also complicated by the fact that sometimes user agents need to parse some of the HTML to find a <meta> element.
In some situations, user agents need to restart the parsing algorithm, which is quite delicate and better to describe in the same document as HTML parsing (at least for use by HTML processing engines).

Adam

On Sun, Oct 23, 2011 at 8:24 PM, Tobias Gondrom <[email-masked]> wrote:
> First you determine the content-type and then after that you may want 
> to determine the charset used within that content-type (if you really 
> have to sniff the charset). I can also imagine that charset sniffing 
> sniffed mime-type, which again would speak against throwing it in together with mime-sniffing....
>
> Kind regards, Tobias
>
>
>

---

From: Adam Barth <[email-masked]>
Date: Sun, 23 Oct 2011 20:37:27 -0700

I mean, that's how the code works, so it must be possible.  :)

Adam

---

From: Larry Masinter <[email-masked]>
Date: Sun, 23 Oct 2011 21:14:21 -0700

I was talking about the necessary dependency of the specifications -- that you couldn't specify media type sniffing completely without making at least a normative reference to charset sniffing. 

The fact that the code works that way is evidence, of course, but we're not talking about possibility of implementation (where a single implementation is evidence) but rather orthogonality of interfaces (where the question is whether ALL implementations must follow this pattern.)

Larry

-----Original Message-----
From: Adam Barth [mailto:[email-masked]] 
Sent: Sunday, October 23, 2011 8:37 PM
To: Larry Masinter
Cc: Tobias Gondrom; [email-masked]
Subject: Re: [websec] #22: content-type sniffing should include charset sniffing

Adam

---

From: Adam Barth <[email-masked]>
Date: Sun, 23 Oct 2011 21:37:54 -0700

One way to look at the situation is that the sniffing algorithm
operates on octets, not on characters, which is actually what the
draft says.  In that view, it's perfectly well-defined without
reference to character sets.  The fact that the octets happen to
correspond to certain ASCII characters is somewhat beside the point.

Adam

---

From: "Martin J. Dürst" <[email-masked]>
Date: Mon, 24 Oct 2011 15:37:15 +0900

I agree with Adam and Tobias that we should not pull all of charset 
sniffing into this document. Many charset details depend on the mime 
type in the first place, and are carefully described in the respective 
specs. For some transfer protocols, the question of charset may be 
irrelevant (e.g. for text over Websocket, which prescribes and checks 
for UTF-8).

Larry is right that in some cases, some preliminary charset sniffing is 
necessary to get at some information at the start of the document, but I 
think we should strictly limit this draft to these cases.

Regards,    Martin.

On 2011/10/24 13:14, Larry Masinter wrote:
> On Sun, Oct 23, 2011 at 8:32 PM, Larry Masinter<[email-masked]>  wrote:
>> The charset sniffing is also complicated by the fact that sometimes user agents need to parse some of the HTML to find a<meta>  element.
>> On Sun, Oct 23, 2011 at 8:24 PM, Tobias Gondrom<[email-masked]>  wrote:
>>> <hat="individual">

---

From: Larry Masinter <[email-masked]>
Date: Sun, 23 Oct 2011 23:47:46 -0700

The charset sniffing documentation in the HTML5 document isn't all that complicated, anyway. 

And it has to be somewhere. 

What's the point of standardizing sniffing of the internet media type without also standardizing the sniffing of all of the relevant parameters.... the goal is to sniff the content-type, the media type by itself isn't what's used.
It's just for text and xml types, the 'charset' parameter is already there.

Also, the algorithm in the document currently is incomplete and inappropriate if you're going to sniff XML-based media types, so the fact that the current algorithm can get away with hiding "charset guessing" as if it were just on octets and not the characters -- well, that's just a superficial work-around.

Larry

-----Original Message-----
From: "Martin J. Dürst" [mailto:[email-masked]] 
Sent: Sunday, October 23, 2011 11:37 PM
To: Larry Masinter
Cc: Adam Barth; [email-masked]
Subject: Re: [websec] #22: content-type sniffing should include charset sniffing

I agree with Adam and Tobias that we should not pull all of charset sniffing into this document. Many charset details depend on the mime type in the first place, and are carefully described in the respective specs. For some transfer protocols, the question of charset may be irrelevant (e.g. for text over Websocket, which prescribes and checks for UTF-8).

Larry is right that in some cases, some preliminary charset sniffing is necessary to get at some information at the start of the document, but I think we should strictly limit this draft to these cases.

Regards,    Martin.

On 2011/10/24 13:14, Larry Masinter wrote:
> The fact that the code works that way is evidence, of course, but 
> we're not talking about possibility of implementation (where a single 
> implementation is evidence) but rather orthogonality of interfaces 
> (where the question is whether ALL implementations must follow this 
> pattern.)
>
> Larry
>
>
>
>
>>> mime-sniffing does not seem like a strong argument to include this 
>>> in the draft.
>>>
>>> First you determine the content-type and then after that you may 
>>> want to determine the charset used within that content-type (if you 
>>> really have to sniff the charset). I can also imagine that charset 
>>> sniffing algorithm might be depending on the application identified 
>>> by the sniffed mime-type, which again would speak against throwing it in together with mime-sniffing....
>>>
>>> Kind regards, Tobias
>>>
>>>
>>>

---

From: Tobias Gondrom <[email-masked]>
Date: Mon, 24 Oct 2011 08:07:06 +0100

<hat="individual">
Well, I don't feel that looking at the octets aka magic numbers is a 
"superficial work-around", rather find it a reasonable argument.
Personally, I don't think we need to go into charset-sniffing in the 
draft, but if you can make it an uncomplicated sub-set of some of the 
content-types, it probably won't harm.

However, I have some doubts that we can limit the extend of 
charset-sniffing to a small area in the draft as discussed and may not 
end up with a whole other set of additional problems.... - in which case 
I would rather like to see the content-type sniffed and talk about 
charset-sniffing in another draft - if we want to go there...

just my 5cents, Tobias

---

From: "Anne van Kesteren" <[email-masked]>
Date: Mon, 24 Oct 2011 19:28:01 +0900

> The charset sniffing documentation in the HTML5 document isn't all that  

You have to run the HTML parser for it. It is orders of magnitude more  
complicated than MIME type sniffing.

Sniffing for an encoding always happens after you determine what the MIME  
type is though. So it depends on the outcome of the MIME type sniffing  
algorithm what kind of encoding sniffing you might want to do. E.g. you  
are not going to run XML encoding sniffing on a text/html resource.

It is probably confusing to refer to MIME type sniffing as Content-Type  
sniffing, because that is not really what it does.

-- 
Anne van Kesteren
http://annevankesteren.nl/