Subject: [dispatch] I18ndir early review of draft-ietf-dispatch-javascript-mjs-07

From: John Levine via Datatracker <[email-masked]>
Date: Fri, 08 May 2020 13:17:25 -0700

Reviewer: John Levine
Review result: Ready with Issues

This is my take on issues with this document mostly from my personal
review but also after some discussion we've had on the i18ndir list.

Some parts of this draft are quite hard to follow, so I'm giving my
understanding of the parts I'm commenting on in case I got them wrong.
I realize that a lot of this is unchanged from 4329, which we should
have reviewed more carefully 15 years ago.

Section 4 on Encoding: I believe it says that the preferred encoding
for all javascript is UTF-8, but some sources use other encodings and
sometimes mislabel them.  So for anything that you don't know is a
module, you have to sniff the contents to see if starts with a BOM,
and if so, use the BOM's encoding and delete the BOM.  If the BOM uses
an encoding the consumer doesn't support, fail.  If there's no BOM,
use the declared character set, or if it's one the consumer doesn't
understand, treat it as UTF-8 anyway.

Step 1 says "The longest matching octet sequence determines the encoding." 
which I don't understand, since none of the encodings overlap.  Does that 
mean it should interpret a partial BOM, e.g., EF BB 20 for UTF-8? Also, 
why is the BOM deleted?  ECMAscript says a BOM is a space so it should be 
harmless.

While I understand that there is a lot of history here, I'm wondering if 
the range mislabeling is really as extreme as this implies.  Is there, 
say, text labelled Shift-JIS which is really UTF-8 or UTF-16? If the 
mislabelled stuff is consistently mislabelled as one of UTF-8/16/16BE/16LE
perhaps it could say to try the BOM trick on those encodings and fail otherwise.

I don't understand step 3, "The character encoding scheme is
determined to be UTF-8."  How can it be determined to be UTF-8 other
than by steps 1 and 2?  Or is it saying that if the declared charset
is one the consumer doesn't understand such as KOI8-U, assume it's
UTF-8 anyway.

I'd suggest rewriting the section to make it clearer that if it's not
a module, you look for a BOM, use its encoding if you find one, and (I
think) otherwise use the declared encoding.

Section 4.3 on error handling: I think it says that if there's a byte
sequence that isn't a valid code point in the current encoding, it can
fail or it can turn the bytes into Unicode replacement characters, but
MUST NOT try anything else.  I agree with this advice but again it
could be clearer.

Section 3 on Modules: I believe it says that JS scripts and modules have 
different syntax but you can't easily tell them apart by inspection.  
(The term "goal" is familiar since I used to write books about compiler 
tools, and I realize it's what the ECMAscript spec uses, but it's 
confusing if you're not a programming language expert.  How about just 
saying that scripts and modules have different syntax?)

Hence some software uses a .mjs filename as a hint that something is a
module.  Again I realize that there is a bunch of existing code but
this is not great MIME practice.  If the difference matters, it's
worth providing a new MIME type such as text/jsmodule, which could
have consistently accurate content encodings.  It would coexist with
all of the other old MIME types and the .mjs hints. Since this draft
deprecates a bunch of existing types and de-deprecates another, this
seems as good a time as any to do it.

I also wonder whether it's worth making a distinction in MIME
processing between modules and scripts.  Would there be any harm in
saying to sniff everything for a BOM?  If a .mjs file turns out to
have a UTF-16 BOM, it's wrong, but is it likely to be anything other
than a javascript module in UTF-16?

---

From: Barry Leiba <[email-masked]>
Date: Fri, 8 May 2020 19:18:48 -0400

Thanks, John, for taking the time on this.  It’s a really helpful review,
and appreciated.

Barry

---

From: Myles Borins <[email-masked]>
Date: Fri, 8 May 2020 19:24:46 -0400

Regarding the mime type "text/javascript"

The HTML Specification is rather explicit
<https://html.spec.whatwg.org/multipage/scripting.html#scriptingLanguages>

Servers should use text/javascript for JavaScript resources. Servers should
not use other JavaScript MIME types for JavaScript resources, and must not
use non-JavaScript MIME types.

Browsers validate that source text being loaded is delivered with the
"text/javascript" mime type and fails if it is any other mime is used.
There is no in-band way to determine the goal of a Module, and Node.js
needed a signal to be able to determine how to interpret source text. This
was the motivation behind creating .mjs.

The problem this created was that webservers, not knowing the extension,
were not serving the test/javascript mimetype and browsers were then in
turn failing to load them as modules. This was a pretty awful developer
experience, especially for folks experimenting with a newer technology they
didn't entirely understand to begin with.

With this context in mind I don't think it is really worth going and making
a distinct mimetype for modules, at least not one that would be
strictly enforced, at the very least this probably shouldn't be
pursued without broader socialization and buy-in from browser vendors.

My gut is that we are best to use this proposal to document the existing
status quo... what people are using and is working, rather than introduce
new constraints that do not have any existing implementation or adoption.

I'll let folks more familiar with your other notes field those questions.

On Fri, May 8, 2020 at 7:19 PM Barry Leiba <[email-masked]> wrote:

> On Fri, May 8, 2020 at 4:17 PM John Levine via Datatracker <
> [email-masked]> wrote:
>
>> Reviewer: John Levine

---

From: "John R Levine" <[email-masked]>
Date: 8 May 2020 19:31:15 -0400

> new constraints that do not have any existing implementation or adoption.

I completely agree that for something like this which is widely 
implemented, the spec should describe what actually works.  But I was 
wondering if you could narrow down the range of what you describe so 
future implementors don't waste time worrying about screwups that aren't 
an issue in practice.

I'm sorry to hear that people are using filename extensions to tell what's 
in a file.  I'd hoped that all of the badness from old versions of Windows 
doing that might have been a hint.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "Matthew A. Miller" <[email-masked]>
Date: Fri, 8 May 2020 18:00:00 -0600

Thank you very much for the review, John.  This is very helpful, and
will work on changes to resolve the concerns.

Regarding the encoding:

UTF-8 is strongly encouraged, and required for a source to be a module.
 Preferring BOM sniffing over a charset declaration is the result of
general purpose browsers having to deal with a myriad of misconfigured
servers; this being the defense mechanism that permitted the best
interoperability.

The part about "longest matching octet sequence" is a missed remnant
from the original document which included UTF-32BE/LE entries.  As
UTF-32BE/LE is no longer found in the wild*, those entries were removed
but I missed cleaning up that sentence.

Implementations today do ignore the BOM once the character encoding is
worked out, so this was portion from Section 4.2 was kept to reflect
that.  I'm not sure what changes to make here regarding that, although
the ending "to source text" is another missed remnant from my editing
that will be removed.

Section 4.2 still includes step #3 to deal with the (in practice quite
common) case of a missing BOM and the media type missing a charset
parameter.  There are also too many servers that set this to
"ISO-8859-1" without otherwise examining the sources being served.
We'll make it clearer this is a default/fallback case.

- m&m

Matthew A. Miller

* Web browsers haven't accepted UTF-32 encodings of JavaScript for quite
a long while
On 20/05/08 17:24, Myles Borins wrote:
>     Servers should use text/javascript for JavaScript resources. Servers
>     should not use other JavaScript MIME types for JavaScript resources,
>     and must not use non-JavaScript MIME types.
> 
> 
> needed a signal to be able to determine how to interpret source text.
> This was the motivation behind creating .mjs.
> 
> experience, especially for folks experimenting with a newer technology
> they didn't entirely understand to begin with.
> 
> With this context in mind I don't think it is really worth going and
> making a distinct mimetype for modules, at least not one that would be
> status quo... what people are using and is working, rather than
> introduce new constraints that do not have any existing implementation
> or adoption.
> 
> On Fri, May 8, 2020 at 7:19 PM Barry Leiba <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     Thanks, John, for taking the time on this.  It’s a really helpful
>     review, and appreciated.
> 
>     Barry
> 
>     On Fri, May 8, 2020 at 4:17 PM John Levine via Datatracker
>     <[email-masked] <mailto:[email-masked]>> wrote:
> 
>         Reviewer: John Levine

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 09 May 2020 08:03:43 +0200

On 9 May 2020, at 1:31, John R Levine wrote:

> I completely agree that for something like this which is widely implemented, the spec should describe what actually works.  But I was wondering if you could narrow down the range of what you describe so future implementors don't waste time worrying about screwups that aren't an issue in practice.

It is also the case that the IETF document can explain exactly this, that the use of filename extensions is discouraged (for a multitude of reasons), and still define a new MIME-type which is the recommended use case. Collaboration with W3C is of course essential.

> I'm sorry to hear that people are using filename extensions to tell what's in a file.  I'd hoped that all of the badness from old versions of Windows doing that might have been a hint.

This is something that just must go away, and I think a big note in non-friendly letters must be added in the security considerations section is needed that explain the situation and why the current practice is not really what we want in the wild.

I.e. yes, while still documenting "what works", that should not be the goal. It will lead to even more problems getting secure mechanisms in browsers deployed.

   Patrik

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 12:36:20 -0400

> I.e. yes, while still documenting "what works", that should not be the goal. It will lead to even more problems getting secure mechanisms in browsers deployed.

I agree with Patrik, this reinvents a security hole from the floppy disk 
era and is not something we should let pass without at least making it 
clear that we believe it is a significant mistake.

It occurs to me that http clients can send an Accept header, so how about 
saying that modules SHOULD be text/jsmodule, but MAY be text/javascript 
for backward compatibiltiy with clients that don't handle text/jsmodule?

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 13:19:39 -0400

--On Saturday, May 9, 2020 12:36 -0400 John R Levine
<[email-masked]> wrote:

>>> I'm sorry to hear that people are using filename extensions
>>> to tell what's in a file.  I'd hoped that all of the badness
>>> from old versions of Windows doing that might have been a
>>> hint.
>> 
>> This is something that just must go away, and I think a big
>> note in non-friendly letters must be added in the security
>> considerations section is needed that explain the situation
>> and why the current practice is not really what we want in
>> the wild.
>> 
>> I.e. yes, while still documenting "what works", that should
>> not be the goal. It will lead to even more problems getting
>> secure mechanisms in browsers deployed.
> 
> I agree with Patrik, this reinvents a security hole from the
> floppy disk era and is not something we should let pass
> without at least making it clear that we believe it is a
> significant mistake.
> 
> It occurs to me that http clients can send an Accept header,
> so how about saying that modules SHOULD be text/jsmodule, but
> MAY be text/javascript for backward compatibiltiy with clients
> that don't handle text/jsmodule?

And these kinds of suggestions (both of them) along with some
caveats about heuristics that will work well most of the time
but maybe not always, and some text that should be much more
obvious to someone reading quickly to figure out what to do
(even if they are clear to a very careful reader) identify, to
me, the main deficiencies in the current I-D.

   john

---

From: Barry Leiba <[email-masked]>
Date: Sat, 9 May 2020 14:36:34 -0400

> for backward compatibiltiy with clients that don't handle text/jsmodule?

I'm very much not fond of the SHOULD... but MAY construction, because
I think the MAY contradicts the SHOULD.

I think the right way to say this is more along the line of "SHOULD
use text/jsmodule.  If that is not practical because backward
compatibility with clients that do not support text/jsmodule is
necessary, then the alternative MUST be text/javascript."

Barry

---

From: "John R Levine" <[email-masked]>
Date: 9 May 2020 15:14:41 -0400

> necessary, then the alternative MUST be text/javascript."

Fine by me.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Sat, 09 May 2020 16:28:03 -0400

+1, fwiw.
   john

--On Saturday, May 9, 2020 14:36 -0400 Barry Leiba
<[email-masked]> wrote:

>> MAY be text/javascript for backward compatibiltiy with
>> clients that don't handle text/jsmodule?
> 
> I'm very much not fond of the SHOULD... but MAY construction,
> because I think the MAY contradicts the SHOULD.
> 
> I think the right way to say this is more along the line of
> "SHOULD use text/jsmodule.  If that is not practical because
> backward compatibility with clients that do not support
> text/jsmodule is necessary, then the alternative MUST be
> text/javascript."
> 
> Barry

---

From: "Patrik Fältström" <[email-masked]>
Date: Sun, 10 May 2020 06:49:02 +0200

> +1, fwiw.

Agree

---

From: Mathias Bynens <[email-masked]>
Date: Mon, 11 May 2020 09:22:36 +0200

On Sat, May 9, 2020 at 6:36 PM John R Levine <[email-masked]> wrote:

> >> I'm sorry to hear that people are using filename extensions to tell
> what's in a file.  I'd hoped that all of the badness from old versions of
> Windows doing that might have been a hint.
> >
> > This is something that just must go away, and I think a big note in
> non-friendly letters must be added in the security considerations section
> is needed that explain the situation and why the current practice is not
> really what we want in the wild.
> >
> > I.e. yes, while still documenting "what works", that should not be the
> goal. It will lead to even more problems getting secure mechanisms in
> browsers deployed.

> for backward compatibiltiy with clients that don't handle text/jsmodule?

I agree it would have been nice if the HTML spec + browsers had restricted
modules to a new MIME type, but this proposal was discussed and rejected
<https://github.com/whatwg/html/issues/558>. In 2017, all major browsers
supported JavaScript modules using the recommended MIME type of
text/javascript (Myles already posted the details), or any other JS MIME
type <https://mimesniff.spec.whatwg.org/#javascript-mime-type> as fallback.

At this point, I’m afraid the ship has sailed. Introducing yet another MIME
type *now* is unlikely to gain adoption in the HTML Standard, browsers, or
on the web (because it wouldn’t be backwards compatible). Let’s not publish
specs that intentionally conflict with both the HTML Standard and web
reality.

Also, why is the BOM deleted?  ECMAscript says a BOM is a space so it
> should be harmless.

It’s not entirely harmless. The following JS program behaves differently if
it’s preceded by a BOM:

#!/usr/bin/env bash

Without BOM, this is a hashbang comment, and no exception is thrown. With
BOM, this is invalid syntax, resulting in a SyntaxError exception.
Stripping away the BOM is something hosts (e.g. browsers) do before
processing the source as JS.

---

From: "John R Levine" <[email-masked]>
Date: 11 May 2020 12:55:26 -0400

> specs that intentionally conflict with both the HTML Standard and web

It actually would be backward compatible because of the advice to use 
Accept media types to tell when to send which media type, but if browsers 
aren't going to do it, I guess we put a warning in the security section 
and give up.

> It’s not entirely harmless. The following JS program behaves differently if

I really don't want to know when self-running shell scripts became part of 
Ecmascript.

Regards,

---

From: John C Klensin <[email-masked]>
Date: Mon, 11 May 2020 15:16:18 -0400

--On Monday, May 11, 2020 12:55 -0400 John R Levine
<[email-masked]> wrote:

>> At this point, I'm afraid the ship has sailed. Introducing
>> yet another MIME type *now* is unlikely to gain adoption in
>> the HTML Standard, browsers, or on the web (because it
>> wouldn't be backwards compatible). Let's not publish
>> specs that intentionally conflict with both the HTML Standard
>> and web reality.
> 
> It actually would be backward compatible because of the advice
> to use Accept media types to tell when to send which media
> type, but if browsers aren't going to do it, I guess we put a
> warning in the security section and give up.

Well, just a minute.  Despite approval by DISPATCH for further
discussion and processing, this is basically an individual
submission, not a draft developed by a WG.  With the
understanding that what I'm about to say is not I18n-specific
and therefore should perhaps go on the agenda of the next
DISPATCH meeting or wait for IETF Last Call, the IETF has, as
Patrik has pointed out, both a collection of standards-track and
BCP specifications in these areas and experience with what
happens when other approaches (like reliance on name suffixes)
are used instead.  Recommending something else is not just a
matter of Security Considerations and, fwiw, I note that at
least one document has been stuck after IETF Last Call completed
in September because the IESG had DISCUSS-level concerns about
telling implementers what they should do if they chose to not
conform to the IETF's normative recommendations.   If this is
really just a description of what browser vendors are doing or
will do now and in the future, then we have a long history of
publishing "for the information of the community, this is what
some vendor or cluster of vendors are doing" descriptive
Informational documents.  They are not standards track, they do
not use normative language (2119/8174 terminology included),
they just describe what is being done and are quite explicit
about that.  

So, if the decision criteria for what belongs in this I-D is
"whatever the browser vendors are going to do whether the IETF
thinks it is a good idea or not", then let's see it recast as a
"what is being done in practice" description, with the normative
language removed or transformed into "if you want to do what the
vendors are doing, then this is what you need to do" (or even
"should" do - noting lower case).  If that means updating the
registration for text/javascript (which, I note, is in the
registry at
https://www.iana.org/assignments/media-types/media-types.xhtml#text
as "OBSOLETED in favor of application/javascript" -- exactly the
opposite of which this I-D proposes) and application/javascript
to treat them as vendor registrations (applying the exception
for non-faceted names allowed by Appendix A of RFC 6838) then so
be it... because that is, apparently, the truth of the matter.

Sorry, but it seems to me that this document is trying to have
it both (or all three) ways: to be a normative document without
meeting the higher bar the IETF usually applies for standards
track documents, to be a vendor specification without making
that clear in the text (and noting that vendor specifications
are still expected to have accurate Security Considerations
sections), and to make requirements that are at variances with
significant community experience about bad results when similar
things have been done in only slightly different areas and still
expect IETF consensus for the document.  I don't think that
works.  YMMD.

And...

>> It's not entirely harmless. The following JS program
>> behaves differently if it's preceded by a BOM:
>> 
>> # !/usr/bin/env bash
> 
> I really don't want to know when self-running shell scripts
> became part of Ecmascript.

Presumably, by an extension of the comments that caused the rant
above, whatever "part of ECMAscript" means, they are a
consideration because some or all browser vendors decided to
treat them as if they were.

    john

---

From: Bradley Farias <[email-masked]>
Date: Mon, 11 May 2020 17:21:27 -0500

Forgive me, but the tone appears to be rather harsh against people who have
spent quite a few years waiting on this document to even get to a
visibility that we can discuss things. Per Adam Roach's recommendation we
did not go through a WG when I made this document years ago. The past few
emails seem rather derisive against such decisions, but I am still new to
IETF workings and would love help through the process; there seems to be
issues that cause me some personal confusion on the intent of a review.

Is there a way we can follow or be aided such that we can reach the "higher
bar" that we are unaware we were missing here such that we can go for a
normative change? I'm unclear but it seems like there is a desire to design
some solution for concerns about ECMA262 in host environments but I'm
unclear on this being the venue to do so and would be appreciative of
guidance on how to proceed.

Additionally, I have concerns with shipping things like MIMEs that are not
supported in any environment and are not discussed in the host venues they
intend to be used in and without any buy in from a host. This document is
trying to be normative to match the environment scenarios of not just
browsers but other environments that run ECMAScript such as server side
environments like Node.js are using text/javascript as the preferred
non-obsoleted form in order to coalesce around a standard that matches the
browser recommendations. The reversal of the application/javascript as the
preferred MIME was intentionally discussed in those environments. I'm
unclear on how creation of a new MIME would help here given we cannot
remove support for the status quo; creation of a new MIME seems like it
would require implementer interest to be a viable replacement.

Per things like hash bang comments and concerns about the language being
different from when older RFCs existed; I would view this document as being
a way to catch up to at least the status quo; creation of
unrelated evolution of standards paths that are not intended to match the
status quo likely won't be useful without other standards groups buy in. In
the future I'm sure people with such varied interests in the evolution of
ECMAScript certainly could participate in the evolution of the language if
they want to, but that was not the intent of this document. It seems like
an additional RFC might be desirable for other issues we are seeing brought
up here? Once again, I'm less familiar with IETF process and am unclear if
all possible design decisions need to be made in this document.

---

From: Barry Leiba <[email-masked]>
Date: Mon, 11 May 2020 18:57:16 -0400

Thanks for this, Bradley.  A couple of clarifications, to make sure we’re
together on this:

The IETF crowd can tend to be blunt — sorry for that.  If it helps, keep in
mind that the purpose in that bluntness is not to insult, but to aim for
the highest quality documents we can get.

As to the higher bar for non-WG documents, it’s not that there’s a higher
bar for the document content — we’re looking, as I say, for high quality in
general — but a higher bar for getting reviews at this stage.  Working
group documents have time in working group development and review, and the
IETF last call process is bringing the document to the community beyond the
working group for final review.  Non-WG documents lack the work and review
in the working group, and, frequently, the first significant review is
during last call.  There is, consequently, a stronger-than-usual desire to
ensure thorough review at this stage.

I, as the sponsoring AD, am aware of the work that’s been done on the
document already, and am not concerned, in general, about the quality of
that.    That said, this particular review is from experts in
internationalization issues, and that’s a quite specialized skill set held
by a relatively few participants.  It’s really important to get this part
right, and I very much appreciate their review and discussion.  I hope we
can put all this together to make the document even better.

Barry, ART AD

On Mon, May 11, 2020 at 6:21 PM Bradley Farias <[email-masked]>
wrote:

> Forgive me, but the tone appears to be rather harsh against people who
> have spent quite a few years waiting on this document to even get to a
> Is there a way we can follow or be aided such that we can reach the
> "higher bar" that we are unaware we were missing here such that we can go
> for a normative change? I'm unclear but it seems like there is a desire to
> design some solution for concerns about ECMA262 in host environments
> but I'm unclear on this being the venue to do so and would be appreciative
> of guidance on how to proceed.
>