Subject: [Tools-discuss] BOMs and rfcdiff

From: Brian E Carpenter <[email-masked]>
Date: Wed, 20 Sep 2017 08:48:40 +1200

rfcdiff can't quite deal with BOMs apparently - see screenshot attached.
The file on the left is the original RFC; on the right is a BOM-free version.

I'm sending it here because there may be a number of tools with a similar problem.

Regards
   Brian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Sep 2017 23:10:26 +0200

On Sep 19, 2017, at 23:03, Carsten Bormann <[email-masked]> wrote:

(Which was too big.)

What I wanted to say:

> The file on the left is the original RFC; on the right is a BOM-free version.

This is maybe less of a BOM problem than a problem with UTF-8 support.

Below is 8187 with the pound sign (U+00A3) changed into a currency sign (U+00A4) (and the comment changed nonsensically for more fun).

The differ finds that the first byte hasn’t changed, but the second byte has, so it inserts a color marker between the two bytes of the character…

See http://www.tzi.org/~cabo/rfcdiff-utf-8-issue.png

Grüße, Carsten

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 20 Sep 2017 11:47:09 +1200

Just a note that https://tools.ietf.org/html/rfc8187 seems
to be fine, iff your browser is set to assume UTF-8. But it
does include the BOM.

If your browser is set to assume what Firefox calls "Western",
the document starts with ï»¿ (which is the IS8859 interpretation
of a UTF-8 BOM, expressed in UTF-8, if that doesn't make your head
hurt and if it survives the email system). Naturally, the £, € and
ü do not display correctly.

The BOM is in the generated HTML immediately after the <pre>.
However, with Firefox it performs no useful function; all that matters
is the View/Text Encoding setting. Exactly the same applies to Internet
Explorer.

Thus, at least for these two browsers, including the BOM in the HTML
file seems to be pointless. I'd vote for removing it.

Regards
   Brian

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 19 Sep 2017 21:30:47 -0400

Brian E Carpenter <[email-masked]> writes:
> the document starts with [...] (which is the IS8859 interpretation
> hurt and if it survives the email system). Naturally, the [pound], [euro] and
> [u-umlaut] do not display correctly.
>

The situation is nastier than that.  Assuming that the server delivers
the same bytes to every requester, what I see just after "<pre>SP" is
0xEF 0xBB 0xBF, which is the UTF-8 encoding of the (non-existent) code
point 0xFEFF.  That is, something generated a file starting with BOM
(which is fine), but some process read that file and tried to convert
the characters into UTF-8, but mistook BOM as a character, rather than
an encoding signature.  It then took the UTF-8 and wrapped it in HTML
structures, leaving the BOM inside the <pre>, where it does no good.

It looks like the Unicode Consortium isn't as rigid as it used to be
regarding BOM, and it is now a sort-of character that can appear in text
formats other than UTF-16.  But it's still bad practice to put one in
the middle of a file.  (http://www.unicode.org/faq/utf_bom.html#BOM)

Dale

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 20 Sep 2017 13:52:14 +1200

On 20/09/2017 13:30, Dale R. Worley wrote:
> point 0xFEFF.  

Yes, that's what I meant by "a UTF-8 BOM". And those three bytes
when interpreted as IS8859-1 give you the glyphs in the attached PNG file.

> That is, something generated a file starting with BOM

I agree, fwiw.

    Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 20 Sep 2017 06:46:51 +0200

> ü do not display correctly.

If you set your browser to ignore the server-supplied encoding 
information, it will. This is your problem, not a problem with the 
server or the payload.

See

   https://redbot.org/?uri=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc8187

>     HTTP/1.1 200 OK
>     Date: Wed, 20 Sep 2017 04:42:56 GMT
>     Server: Apache/2.2.22 (Debian)
>     Content-Location: rfc8187.html
>     Vary: negotiate,Accept-Encoding
>     TCN: choice
>     Last-Modified: Thu, 14 Sep 2017 21:58:22 GMT
>     ETag: "22cd1ad-c36c-5592d6002c6b7;559978fb430ee"
>     Accept-Ranges: bytes
>     Content-Length: 50028
>     Cache-Control: max-age=604800
>     Expires: Wed, 27 Sep 2017 04:42:56 GMT
>     Strict-Transport-Security: max-age=3600
>     X-Frame-Options: SAMEORIGIN
>     X-Xss-Protection: 1; mode=block
>     X-Content-Type-Options: nosniff
>     X-Clacks-Overhead: GNU Terry Pratchett
>     Keep-Alive: timeout=5, max=100
>     Connection: Keep-Alive
>     Content-Type: text/html; charset=UTF-8

(where "X-Clacks-Overhead" looks funky).

> ... > The BOM is in the generated HTML immediately after the <pre>.
> ...

It's pointless and harmless. But yes, it should be removed.

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 20 Sep 2017 20:34:55 +0200

Hi Brian,

> file seems to be pointless. I'd vote for removing it.

I've updated rfcmarkup to strip out BOMs; the link above should now
give you a document without a BOM after the <pre>.

Will look at rfcdiff next.

Best regards,

	Henrik

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 21 Sep 2017 08:15:07 +1200

> give you a document without a BOM after the <pre>.

Thanks much. It's fine, after remembering to refresh my browser cache ;-)

> Will look at rfcdiff next.

I imagine that will be a bit more work.

Regards
    Brian

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 20 Sep 2017 23:29:31 +0200

Hi Brian,

> I'm sending it here because there may be a number of tools with a similar problem.

For a while I was baffled -- as to why, see the entry under release
1.44 in the changelog here:

  https://zin.tools.ietf.org/tools/rfcdiff/changelog

But then I looked more closely at your screendump, and found that the
rfcdiff version on www.ietf.org is at 1.41, so hasn't been updated
for a while.

I'll fix that.

Best regards,

	Henrik

---

From: Adam Roach <[email-masked]>
Date: Wed, 20 Sep 2017 16:44:37 -0500

> give you a document without a BOM after the <pre>.

Wait. WAIT! No. This is the wrong answer. If you are in a tool that 
makes the BOM visible at all, the problem isn't the presence of the BOM 
(which was well-defined rendering semantics in UTF8 that basically say 
"this shouldn't be visible in any way at all"); the problem is that you 
are loading a UTF-8 document and treating it as _some_ _other_ _encoding_.

It does no good to paper over the presence of a BOM, since that just 
defers the problem to elsewhere (namely, any other non-ASCII codepoints 
in the document).

The proper behavior here, and especially for rfcdiff, is to make sure 
everything is using UTF-8. Leaving the BOM in place serves as a useful 
canary in this coal mine, and I would suggest *not* stripping it, at 
least until we're sure the tools *otherwise* handle UTF-8 correctly. 
That is, the BOM provides a good litmus test: if you can see it, you're 
doing it wrong.

/a

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 21 Sep 2017 00:08:37 +0200

Hi Adam,

> are loading a UTF-8 document and treating it as _some_ _other_ _encoding_.

I'm sorry, but I disagree (at least in one sense).  Earlier, rfcmarkup
left a BOM in the middle of the html, which I believe is wrong.  You
could say that this is the result of not reading a UTF-8 file with BOM
correctly, and I would agree.  So now I'm reading the content as UTF-8,
but not keeping the BOM, and not placing it in the middle of a html file
(which is served with Content-Type: text/html; charset=UTF-8).  I think
that's right.

> defers the problem to elsewhere (namely, any other non-ASCII codepoints 

Nope.  I've been handling UTF-8 correctly in rfcmarkup for several years;
it has behaved correctly for draft-ietf-httpbis-rfc5987bis for every
version which has had non-ascii characters.

> That is, the BOM provides a good litmus test: if you can see it, you're 

I think my experience is that rfcmarkup has bean dealing just fine with
utf-8 in general, but the BOM (which I think is nonsense in UTF-8, but
that's a different discussion) has been visible, which I've now fixed.

Best regards,

	Henrik

---

From: Adam Roach <[email-masked]>
Date: Wed, 20 Sep 2017 17:11:51 -0500

> that's a different discussion) has been visible, which I've now fixed.

The BOM is formally a "ZERO WIDTH NO-BREAK SPACE."  How would it be 
visible if you're handling UTF-8 correctly?

/a

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 21 Sep 2017 01:15:22 +0200

> On 21 Sep 2017, at 00:11, Adam Roach <[email-masked]> wrote:
> 
> The BOM is formally a "ZERO WIDTH NO-BREAK SPACE."  How would it be visible if you're handling UTF-8 correctly?

Sorry. Bad wording. In rfcmarkup it was _present_ (it could not be visually identified because it did not display).

    Henrik

> 
> /a
> 
> -- 
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at http://tools.ietf.org/tools/issues or
> send email to [email-masked]

---

From: Adam Roach <[email-masked]>
Date: Wed, 20 Sep 2017 18:17:58 -0500

> Sorry. Bad wording. In rfcmarkup it was _present_ (it could not be visually identified because it did not display).

Ah! Okay. In that case, I have no issues with stripping it out.

/a