Subject: [websec] Issue 17: Registry for magic numbers

From: Adam Barth <[email-masked]>
Date: Mon, 24 Oct 2011 19:21:35 -0700

http://trac.tools.ietf.org/wg/websec/trac/ticket/17 refers to an IANA
registry with magic numbers for various media types.  I wanted to
compare them to what's in the draft, but I couldn't find it.  I found
the media type registry, e.g., for images:

http://www.iana.org/assignments/media-types/image/index.html

but I don't see any magic numbers.  Would someone be willing to point
me in the right direction?

Thanks,
Adam

---

From: "Anne van Kesteren" <[email-masked]>
Date: Tue, 25 Oct 2011 11:32:20 +0900

> me in the right direction?

I don't think using a registry is a good idea. When a new MIME type comes  
along it needs to be determined at that point whether or not we want to  
sniff for it. E.g. for image/svg+xml, a new image MIME type, we decided we  
would not sniff for it.

I suppose we could somehow encode all that information in a registry, but  
I do not see it making things any better for implementors.

-- 
Anne van Kesteren
http://annevankesteren.nl/

---

From: Adam Barth <[email-masked]>
Date: Mon, 24 Oct 2011 19:34:18 -0700

On Mon, Oct 24, 2011 at 7:32 PM, Anne van Kesteren <[email-masked]> wrote:
> I suppose we could somehow encode all that information in a registry, but I
> do not see it making things any better for implementors.

Yeah, I don't think a registry is a good idea either.  Constructing
these signatures is too subtle, but I wanted to give the idea a fair
shake.  Looking at the existing registry will give us a sense for its
quality.

Adam

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 25 Oct 2011 13:07:40 +0900

Hello Adam,

> me in the right direction?

They are in the templates. To get the template for a registration, start 
at the overview page 
(http://www.iana.org/assignments/media-types/index.html).

Then go to the page that lists all the registration for a give top 
level, e.g. http://www.iana.org/assignments/media-types/image/index.html 
for images.

Then look at each registration template (click on the link in the left 
column, or in the right column if the left one doesn't have a link and 
the right one is to an RFC). You may then find a magic number in the 
registration template. As an example, for image/jp2, the template is at 
http://www.iana.org/assignments/media-types/image/jp2.

But it looks like earlier templates didn't have a field for a magic 
number, and this and the reasons Anne gave make this information helpful 
for cross-checking, but not much more.

Regards,   Martin.

---

From: Tobias Gondrom <[email-masked]>
Date: Tue, 25 Oct 2011 05:10:11 +0100

On 25/10/11 03:34, Adam Barth wrote:
> On Mon, Oct 24, 2011 at 7:32 PM, Anne van Kesteren<[email-masked]>  wrote:
>> On Tue, 25 Oct 2011 11:21:35 +0900, Adam Barth<[email-masked]>  wrote:
> _______________________________________________
> websec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/websec

The existing registry is here:
http://www.iana.org/assignments/media-types/index.html
And if you want to see how things look like for one Mime-type:
e.g. for html: http://events.linkedin.com/Ietf-82/pub/803707
(as you can see it is very short and easy to register a mime-type...)

On a technical note:
There might be a good reason for the registry over only by RFC: The RFC 
will remain static (though you can update it with another draft, this 
should not necessarily be the main intention from the get-go doing on a 
regular basis).
A registry is dynamic, so you can add information easily later (by RFC 
or expert review, ...) - adding mime-types is easy and we could enrich 
the registration of mime-types by the information you need to decide on 
whether to sniff and how....

Kind regards, Tobias

---

From: Adam Barth <[email-masked]>
Date: Mon, 24 Oct 2011 21:23:59 -0700

On Mon, Oct 24, 2011 at 9:07 PM, "Martin J. Dürst"
<[email-masked]> wrote:
> They are in the templates. To get the template for a registration, start at
> the overview page (http://www.iana.org/assignments/media-types/index.html).
>
> Then go to the page that lists all the registration for a give top level,
> e.g. http://www.iana.org/assignments/media-types/image/index.html for
> images.
>
> column, or in the right column if the left one doesn't have a link and the
> right one is to an RFC). You may then find a magic number in the
> But it looks like earlier templates didn't have a field for a magic number,
> and this and the reasons Anne gave make this information helpful for
> cross-checking, but not much more.

== Images ==

PNG has a registration template
<http://www.iana.org/assignments/media-types/image/png>, but lacks a
signature.
JPEG doesn't have a template.
GIF doesn't have a template.
BMP isn't even registered.
WEBP isn't even registered.
ICO has a registration template
<http://www.iana.org/assignments/media-types/image/vnd.microsoft.icon>
and has the correct signature.  Yay!

== Text ==

HTML lacks a registration template.

== Application ==

PDF doesn't have a template.
Postscript doesn't have a template.
OGG doesn't have a template.
RAR isn't even registered.
ZIP has a registration template
<http://www.iana.org/assignments/media-types/application/zip>, but
lacks a signature.
GZIP isn't even registered.
RSS isn't even registered.
Atom lacks a registration template.

== Audio ==

WAV isn't even registered.

== Video ==

MP4 lacks a registration template.
WebM isn't even registered.

This does not look like a promising approach.  Note: I haven't even
looked through all the registrations to see how many have signatures
that we shouldn't be using.

Adam

---

From: Tobias Gondrom <[email-masked]>
Date: Wed, 26 Oct 2011 04:52:23 +0100

<hat="individual">
For me the point is, currently we have a table in the document, which 
inside an RFC is rather static and hard to extend.
So it looks like a good case for a registry to allow for extendibility 
for new mime-types. (e.g. we keep the table in the document, create an 
IANA registry, copy the values to the registry and allow for future 
entries by expert review)
That can either be added to the current Mime-type registry, or we create 
a new one (e.g. within the websec namespace) with only these elements.

Just my 5cents.

Tobias

---

From: Adam Barth <[email-masked]>
Date: Tue, 25 Oct 2011 20:59:44 -0700

Yeah, I think we're much better off creating a new registry rather
than using the MIME registry.  The truth is that most MIME types that
get registered don't need sniffing rules.  The only ones that need it
are the legacy ones and the ones browser vendor cause to need it
because of the prisoner's dilemma in the browser market.

Adam

On Tue, Oct 25, 2011 at 8:52 PM, Tobias Gondrom
<[email-masked]> wrote:
> <hat="individual">
> For me the point is, currently we have a table in the document, which inside
> an RFC is rather static and hard to extend.
> So it looks like a good case for a registry to allow for extendibility for
> new mime-types. (e.g. we keep the table in the document, create an IANA
> registry, copy the values to the registry and allow for future entries by
> expert review)
> That can either be added to the current Mime-type registry, or we create a
> new one (e.g. within the websec namespace) with only these elements.
>
> Just my 5cents.
>
> Tobias
>
>
>
>>> at
>>> the overview page
>>> the
>>> number,

---

From: Larry Masinter <[email-masked]>
Date: Tue, 25 Oct 2011 23:31:54 -0700

This gets back to the question of the scope of the document. Does it, or does it not, handle sniffing of arbitrary blobs of data that come in without any content-type, blobs of data labeled application/octet-stream, and data coming via ftp (through ftp URIs) or thumb drives or mounted NFS file systems or whatever? Does it, or does it not, handle sniffing rules inside ZIP packaged web applications?   

If it does, then sniffing should cover everything that is sniffable, including almost all MIME types  -- you say "most MIME types that get registered don't need sniffing rules", I don't know what the percentage is, but after all, don't you want to be able to discover file types??  

Of course, maybe that broad applicability of sniffing isn't appropriate, but then ... where's are the boundaries? Which situations are in scope vs. not? And don't some of the "in-scope" situations need almost all MIME types to be sniffable?

Larry

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Adam Barth
Sent: Tuesday, October 25, 2011 9:00 PM
To: Tobias Gondrom
Cc: [email-masked]
Subject: Re: [websec] Issue 17: Registry for magic numbers

Yeah, I think we're much better off creating a new registry rather than using the MIME registry.  The truth is that most MIME types that get registered don't need sniffing rules.  The only ones that need it are the legacy ones and the ones browser vendor cause to need it because of the prisoner's dilemma in the browser market.

Adam

On Tue, Oct 25, 2011 at 8:52 PM, Tobias Gondrom <[email-masked]> wrote:
> entries by expert review) That can either be added to the current 
> Mime-type registry, or we create a new one (e.g. within the websec 
> namespace) with only these elements.
>
> Just my 5cents.
>
> Tobias
>
>
>
>>>> http://trac.tools.ietf.org/wg/websec/trac/ticket/17 refers to an 
>>>> IANA registry with magic numbers for various media types.  I wanted 
>>>> to compare them to what's in the draft, but I couldn't find it.  I 
>>>> found the media type registry, e.g., for images:
>>>>
>>>> but I don't see any magic numbers.  Would someone be willing to 
>>>> point me in the right direction?
>>>
>>> They are in the templates. To get the template for a registration, 
>>> start at the overview page 
>>> level, e.g. 
>>> http://www.iana.org/assignments/media-types/image/index.html for images.
>>>
>>> Then look at each registration template (click on the link in the 
>>> left column, or in the right column if the left one doesn't have a 
>>> link and the right one is to an RFC). You may then find a magic 
>>> number in the registration template. As an example, for image/jp2, 
>>> the template is at 
>>> number, and this and the reasons Anne gave make this information 
>>> helpful for cross-checking, but not much more.
>>
>> == Images ==
>>
>> <http://www.iana.org/assignments/media-types/image/vnd.microsoft.icon
>> >

---

From: Adam Barth <[email-masked]>
Date: Tue, 25 Oct 2011 23:45:07 -0700

You've posed a large number of questions.  I'll do my best to answer them.

On Tue, Oct 25, 2011 at 11:31 PM, Larry Masinter <[email-masked]> wrote:
> This gets back to the question of the scope of the document. Does it, or does it not, handle sniffing of arbitrary blobs of data that come in without any content-type,

User agents that implement sniffing are expected to sniff HTTP
responses that lack a Content-Type header, yes.

> blobs of data labeled application/octet-stream,

User agents that implement sniffing are not expected to sniff HTTP
responses that contain Content-Type header with the value
application/octet-stream.

> and data coming via ftp (through ftp URIs)

Yes.

> or thumb drives

Yes.

>or mounted NFS file systems or whatever?

Yes.

You can answer these questions by reading the document.  For example,
the document explicitly states the set of Content-Type header values
that trigger sniffing.  The document also explicitly calls out FTP as
an example.

> Does it, or does it not, handle sniffing rules inside ZIP packaged web applications?

Not as described by this document.  However, I've been told that
another document has re-used the algorithm for that purpose.

> If it does, then sniffing should cover everything that is sniffable, including almost all MIME types

Why is that?  The document describes what is essentially the minimal
amount of sniffing a user agent needs to perform in order to be
competitive in the browser market.  I don't think we should be
encouraging sniffing beyond that.

> -- you say "most MIME types that get registered don't need sniffing rules", I don't know what the percentage is,

With the possible exception of fonts, I believe the document describes
all the sniffing rules that are necessary today.  You can compare with
list of MIME types in the document with the list of registered MIME
types if you wish to get a sense of what I mean when I say that "most"
don't need sniffing rules.

> but after all, don't you want to be able to discover file types??

I'm not sure what you mean by "discover file types".  There's no
discovery going on here.

> Of course, maybe that broad applicability of sniffing isn't appropriate, but then ... where's are the boundaries?

The boundaries are exactly what's described in the document.  There's
been a great deal of research and implementation experience poured
into the document to determine precisely where to draw the boundaries.
 As far as I can tell, the document describes the optimal point.  If
you have data that shows otherwise, I'd like to see it.

> Which situations are in scope vs. not?

The criteria I would use is the following one:

"Given a diverse market of browser vendors, is this a sniffing
algorithm that all browser vendors are mutually interested in
converging upon."

If the answer is "yes", then you've identified the correct scope and
rules.  If "no", then the spec needs to be improved.  If there is no
such set of rules, then this endeavor is a waste of time and any spec
we create will be dead letter.

> And don't some of the "in-scope" situations need almost all MIME types to be sniffable?

No.

Adam

---

From: Larry Masinter <[email-masked]>
Date: Wed, 26 Oct 2011 01:38:00 -0700

A standards specification should meet the requirements of the use cases that are in scope for the specification. 

If you only evaluate adequacy against  a narrow set of requirements, then the scope should be limited to those situations where those requirements are adequate.

If you're evaluating against what "a  user agent needs to perform in order to be competitive in the browser market"  then the only use cases you're validating against are "popular web browsers in 2012", which is a very narrow scope.

If, on the other hand, you expect the standard to have value over the long term, you need a longer-term and broader set of requirements and use cases, which will add additional complexity to meet requirements.

Larry

-----Original Message-----
From: Adam Barth [mailto:[email-masked]] 
Sent: Tuesday, October 25, 2011 11:45 PM
To: Larry Masinter
Cc: Tobias Gondrom; [email-masked]
Subject: Re: [websec] Issue 17: Registry for magic numbers

On Tue, Oct 25, 2011 at 11:31 PM, Larry Masinter <[email-masked]> wrote:
> This gets back to the question of the scope of the document. Does it, 
> or does it not, handle sniffing of arbitrary blobs of data that come 
> in without any content-type,

User agents that implement sniffing are expected to sniff HTTP responses that lack a Content-Type header, yes.

> blobs of data labeled application/octet-stream,

User agents that implement sniffing are not expected to sniff HTTP responses that contain Content-Type header with the value application/octet-stream.

Yes.

Yes.

Yes.

You can answer these questions by reading the document.  For example, the document explicitly states the set of Content-Type header values that trigger sniffing.  The document also explicitly calls out FTP as an example.

> Does it, or does it not, handle sniffing rules inside ZIP packaged web applications?

Not as described by this document.  However, I've been told that another document has re-used the algorithm for that purpose.

> including almost all MIME types

Why is that?  The document describes what is essentially the minimal amount of sniffing a user agent needs to perform in order to be competitive in the browser market.  I don't think we should be encouraging sniffing beyond that.

> -- you say "most MIME types that get registered don't need sniffing 
> rules", I don't know what the percentage is,

With the possible exception of fonts, I believe the document describes all the sniffing rules that are necessary today.  You can compare with list of MIME types in the document with the list of registered MIME types if you wish to get a sense of what I mean when I say that "most"
don't need sniffing rules.

> but after all, don't you want to be able to discover file types??

I'm not sure what you mean by "discover file types".  There's no discovery going on here.

> Of course, maybe that broad applicability of sniffing isn't appropriate, but then ... where's are the boundaries?

The boundaries are exactly what's described in the document.  There's been a great deal of research and implementation experience poured into the document to determine precisely where to draw the boundaries.
 As far as I can tell, the document describes the optimal point.  If you have data that shows otherwise, I'd like to see it.

"Given a diverse market of browser vendors, is this a sniffing algorithm that all browser vendors are mutually interested in converging upon."

If the answer is "yes", then you've identified the correct scope and rules.  If "no", then the spec needs to be improved.  If there is no such set of rules, then this endeavor is a waste of time and any spec we create will be dead letter.

No.

Adam

>> So it looks like a good case for a registry to allow for 
>> extendibility for new mime-types. (e.g. we keep the table in the 
>> document, create an IANA registry, copy the values to the registry 
>> and allow for future entries by expert review) That can either be 
>> added to the current Mime-type registry, or we create a new one (e.g. 
>> within the websec
>>>>> IANA registry with magic numbers for various media types.  I 
>>>>> wanted to compare them to what's in the draft, but I couldn't find 
>>>>> it.  I found the media type registry, e.g., for images:
>>>>>

---

From: Adam Barth <[email-masked]>
Date: Wed, 26 Oct 2011 01:56:01 -0700

It might be narrow in your view, but it's the use case I'm interested
in addressing.

Adam