Subject: [Tools-discuss] xml2rfc HTML renderer breaks at Fig. 100?

From: Yaron Sheffer <[email-masked]>
Date: Fri, 21 Nov 2014 21:32:25 +0200

Hi,

I am reviewing this draft-ietf-jose-cookbook-06. This is a very long
document. The text version is fine, and so is the "HTMLized"
version. But the nicely formatted HTML (http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html),
my favorite format by far, is garbled exactly at Figure 100. I can
reproduce the problem when running xml2rfc locally.

Thanks,

Yaron

---

From: Tony Hansen <[email-masked]>
Date: Fri, 21 Nov 2014 14:45:13 -0500

On 11/21/14, 2:32 PM, Yaron Sheffer wrote:
> document. The text version is fine, and so is the "HTMLized" version. 
> But the nicely formatted HTML 
> (http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html), my 
> favorite format by far, is garbled exactly at Figure 100. I can 

Yaron, can you be more specific? Did you actually mean 
http://tools.ietf.org/html/draft-ietf-jose-cookbook-06.html instead of 
http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html? (The url you 
specified does not exist.)

What I see in 
http://tools.ietf.org/html/draft-ietf-jose-cookbook-06.html looks like this?

    produces the following encrypted key:

    d3qNhUWfqheyPp4H8sjOWsDYajoej4c5Je6rlUtFPWdgtURtmeDV1g

                Figure 100: Encrypted Key, base64url-encoded

If not the above, what did you expect? What am I missing?

Note: the HTML file displayed in 
http://tools.ietf.org/html/draft-ietf-jose-cookbook-06.html was not 
generated by xml2rfc.

     Tony Hansen

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 21 Nov 2014 22:35:38 +0200

Hi Tony,

There's something strange going on. When you go to: 
http://tools.ietf.org/html/draft-ietf-jose-cookbook-06

(the HTMLized version of the draft, not generated by XML2RFC), and press 
the "html" link at the top of the document, you are redirected to 
another version, this time one that's generated by "xml2rfc --html". 
However this link 
(http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html) sometimes 
works for me and sometimes returns a 404.

And when it does work, it is garbled.

There must be some CDN jinx here, because the "xml" link gave the the 
actual XML output as a download just a few hours ago, and now it gives a 
beautiful CSS-formatted rendering of the draft.

In my day job I have seen weird caching behavior caused by CloudFlare, 
so I'm not too surprised.

Thanks,
	Yaron

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Nov 2014 22:18:04 +0100

> And when it does work, it is garbled.

It's a bug in xml2rfc (the Python code). Easily reproducable by running 
the XML source through it directly.

> beautiful CSS-formatted rendering of the draft.

That view is generated by rfc2629.xslt (my code), which, dare I say, of 
course works well.

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Fri, 21 Nov 2014 16:33:10 -0500

Yes, something strange IS going on, in two different ways.

1) I had gotten the 404 when I had gone to that URL before, but this 
time I do get the file. Your comments about CloudFlare might be correct. 
I also tried running it from validator.w3.org/check and it too got a 
404. Sigh. So I stuck a copy of it at 
http://zinfandel.tools.ietf.org/~tonyh/draft-ietf-jose-cookbook-06.html.

2) And yup, there does appear to be a problem with xml2rfc -html. when I 
view it under FireFox on either my Mac or PC. However,

     *) Opera, Chrome, Safari look just fine on my Mac.

     *) It looks fine under IE8 on my PC.

     *) Passing another copy of the file to validator.w3.org/check shows 
some HTML issues in the table of contents, but nothing later on in the file.

I'm going to mark THIS one down as a bug in the FireFox renderer, and 
NOT as an XML2RFC bug.

     Tony Hansen

On 11/21/14, 3:35 PM, Yaron Sheffer wrote:
> (the HTMLized version of the draft, not generated by XML2RFC), and 
> press the "html" link at the top of the document, you are redirected 
> to another version, this time one that's generated by "xml2rfc 
> --html". However this link 
> actual XML output as a download just a few hours ago, and now it gives 
> a beautiful CSS-formatted rendering of the draft.
>

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 21 Nov 2014 23:33:34 +0200

> That view is generated by rfc2629.xslt (my code), which, dare I say, of

Speaking of which: this URL bombs: 
http://tools.ietf.org/id/draft-ietf-jose-json-web-encryption-37.xml (the 
XSLT complains about an XML entity).

Thanks,
	Yaron

---

From: Yaron Sheffer <[email-masked]>
Date: Fri, 21 Nov 2014 23:44:40 +0200

Yes, Firefox chokes on the HTML, while Chrome renders it correctly (both 
on Ubuntu 14.04).

Thanks,
	Yaron

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Nov 2014 23:48:52 +0100

> XSLT complains about an XML entity).

Actually, it's the browser's XML parser that complains (browsers do not 
read DTDs, thus the entity isn't defined).

Best regards, Julian

---

From: Yaron Sheffer <[email-masked]>
Date: Sat, 22 Nov 2014 02:12:43 +0200

> read DTDs, thus the entity isn't defined).
Could we think of a workaround, such as adding the line

<!ENTITY nbsp "&#0160;">

into the DOCTYPE?

People expect "nbsp" to work, and it actually works well with xml2rfc.

Thanks,
	Yaron

---

From: Tony Hansen <[email-masked]>
Date: Fri, 21 Nov 2014 20:26:39 -0500

> People expect "nbsp" to work, and it actually works well with xml2rfc.

Yes, xml2rfc does an automatic inclusion of rfc2629-other.ent and 
rfc2629-xhtml.ent. A workaround is for you to explicitly force those to 
be included yourself.

     Tony Hansen

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Nov 2014 10:37:02 +0100

> People expect "nbsp" to work, and it actually works well with xml2rfc.

Sure. 
<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#examples.internalsubset>

Best regards, Julian

---

From: Yaron Sheffer <[email-masked]>
Date: Sat, 22 Nov 2014 12:30:55 +0200

> <http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#examples.internalsubset>

My question/request was to have the server add this entity definition to 
the XML document, just like it adds a reference to the XSLT file.

Thanks,
	Yaron

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Nov 2014 13:10:58 +0100

> the XML document, just like it adds a reference to the XSLT file.

It doesn't add the reference to the XSLT file, AFAIK.

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Sat, 22 Nov 2014 10:49:40 -0500

On 11/22/14, 4:54 AM, Julian Reschke wrote:
> On 2014-11-21 22:33, Tony Hansen wrote:
> The version I tried yesterday in Firefox indeed contains:
>
>> <p class="figure">Figure 99: Salt, base64url-encoded</p>
>> <p id="rfc.section.5.3.3.p.3">Performing the key encryption operation 
>> over the CEK (Figure 97)) with the following:</p>
>> <p>
>>
>> </p><ul>
>>   Password (Figure 96;
>>   <li>Salt (Figure 99), encoded as an octet string; and</li>
>>   <li>Iteration count (8192)</li>
>> </ul>
>> <p id="rfc.section.5.3.3.p.5">produces the following encrypted key:</p>
>> <div id="rfc.figure.100">
>>
>> 5.3.4. Encrypting the Content
>> The following are generated before encrypting the content:
>> ...
>
> So an open <div> after fig 100 that never gets closed.

The source HTML file contains:

<p id="rfc.section.5.3.3.p.5">produces the following encrypted key:</p>
<div id="rfc.figure.100"/>
<div id="jwe-pbes2-encdkey"/>
<pre>
d3qNhUWfqheyPp4H8sjOWsDYajoej4c5Je6rlUtFPWdgtURtmeDV1g
</pre>
<p class="figure">Figure 100: Encrypted Key, base64url-encoded</p>
<h1 id="rfc.section.5.3.4"><a href="#rfc.section.5.3.4">5.3.4.</a> <a 
href="#jwe-pbes2-contentenc" id="jwe-pbes2-contentenc">Encrypting the 
Content</a></h1>
<p id="rfc.section.5.3.4.p.1">The following are generated before 
encrypting the content:</p>
<p/>

<ul>
   <li>JWE Protected Header; this example uses the header from <a 
href="#jwe-pbes2-protected">Figure 101</a>, encoded using <a 
href="#RFC4648">[RFC4648]</a> base64url to produce <a 
href="#jwe-pbes2-protected_b64u">Figure 102</a>.</li>
</ul>
<div id="rfc.figure.101"/>

I don't see any unclosed <div>s.

     Tony

---

From: Julian Reschke <[email-masked]>
Date: Sat, 22 Nov 2014 17:30:38 +0100

> I don't see any unclosed <div>s.

I do, you don't. We got different replies from the server.

Is there an error condition that would cause xml2rfv to stop processing 
and just dump the remainder of the input as plain text?

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Sat, 22 Nov 2014 17:43:20 -0500

> I do, you don't. We got different replies from the server.

I'm trying to figure out how this could have happened. I've tried 
multiple methods of retrieving the html file, including regenerating it 
from the xml using xml2rfc, and

*) I ALWAYS see <div id="rfc.figure.100"/> when I view the source
*) I ALWAYS see mangled text from that point forward in the FireFox 
view, but not other browsers' views.

Also, I still do sometimes get a 404 when trying to retrieve the file 
from 
<http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html#rfc.section.5.3.3>, 
but not always. It appears that one of the tools.ietf.org servers does 
not have the file.

> Is there an error condition that would cause xml2rfv to stop 
> processing and just dump the remainder of the input as plain text?

I'm not aware of any.

The frustrating thing is that we're seeing different source forms. 
That's the worse kind of bug to track down.

     Tony Hansen

---

From: Julian Reschke <[email-masked]>
Date: Sun, 23 Nov 2014 13:00:13 +0100

> That's the worse kind of bug to track down.

I just tried and swapped figure 99 with figure 100, and the error still 
happens at #100. So it's not that particular figure.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Nov 2014 14:18:29 +0100

> I don't see any unclosed <div>s.

Maybe I’m confused, but I see *only* unclosed divs in the vicinity of the error.
HTML does not know anything about the XML syntax of interpreting <gi/> as an empty element.
So it seems to be there is a very deep nesting of div elements here.
That might explain the reason Firefox gives up exactly when the 100th level of such nesting is reached; the other browsers just seem to allow deeper nesting.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Nov 2014 14:27:20 +0100

> Maybe I’m confused, but I see *only* unclosed divs in the vicinity of the error.

(The only div that is actually being closed in the entire document is the <div class="avoidbreak”> around the author’s address information.)

The parsed HTML ends as shown below (476 automatically closed divs, 2 per each of the 238 figures).

(And I also do get the annoying 404s now and then:
Connecting to tools.ietf.org (tools.ietf.org)|64.170.98.42|:80... connected.
HTTP request sent, awaiting response... 200 OK
Connecting to tools.ietf.org (tools.ietf.org)|209.208.19.222|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
)

Grüße, Carsten

  </address>
</div>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Nov 2014 14:36:02 +0100

On 23 Nov 2014, at 14:18, Carsten Bormann <[email-masked]> wrote:
> confused

The HTML also lies about its encoding:

  <meta http-equiv="Content-Type" content="text/html; charset=us-ascii" />

which at first confused my tools a bit.  It is actually UTF-8, as is correctly indicated by:

$ curl -I "http://tools.ietf.org/id/draft-ietf-jose-cookbook-06.html"
HTTP/1.1 200 OK
Date: Sun, 23 Nov 2014 13:31:15 GMT
Server: Apache/2.2.22 (Debian)
Last-Modified: Fri, 14 Nov 2014 14:51:24 GMT
ETag: "b862be-3f19e-507d2c343464e"
Accept-Ranges: bytes
Content-Length: 258462
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 23 Nov 2014 15:08:16 +0100

> Maybe I’m confused, but I see *only* unclosed divs in the vicinity of the error.

To make this actionable:

replacing
  <div\(.*?\)/>
by
  <div\1></div>

fixes the display in Firefox.

(Sorry for the many messages in sequence.)

Grüße, Carsten

---

From: Yaron Sheffer <[email-masked]>
Date: Sun, 23 Nov 2014 23:29:16 +0200

One more data point: when I save the document to a file named f.html, 
Firefox breaks. But when I rename the file to f.xhtml, Firefox renders 
it correctly (<div id="bla"/> is valid XHTML). In other words, it parses 
the file by its extension and ignores the DOCTYPE.

Thanks,
	Yaron

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Nov 2014 06:49:07 +0100

> On 23.11.2014, at 22:40, Julian Reschke <[email-masked]> wrote:
> 
>> On 2014-11-23 22:29, Yaron Sheffer wrote:
> I agree that something is wrong with Firefox.

Yes, there is a bug triggered in Firefox by the bug in the HTML converter.

The bug in the HTML converter is that it is generating valid XHTML.   This same byte string happens also to be valid HTML, but with a different meaning, which is not reflecting the intention of the source.  While most browsers will render that alternate meaning in a way that resembles the XHTML meaning, this is just an accident.

> 
> That being said: the web service really really shouldn't serve XHTML content as text/html.

That battle was lost more than a decade ago, together with the whole botched upgrade to XHTML.  Very sad, but bridge under the water.

So I was proposing to bugfix the HTML converter to generate HTML that means the same that the XHTML means.  Never generate <gi/> syntax for elements that are not empty elements in HTML.  Instead, generate the equivalent <gi></gi> syntax for those.  br, meta etc. of course should stay <gi/> as with all elements declared empty in HTML. 

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 24 Nov 2014 07:18:41 +0100

> The bug in the HTML converter is that it is generating valid XHTML.   This same byte string happens also to be valid HTML, but with a different meaning, which is not reflecting the intention of the…

Actually, it's not valid HTML.

> So I was proposing to bugfix the HTML converter to generate HTML that means the same that the XHTML means.  Never generate <gi/> syntax for elements that are not empty elements in HTML.  Instead, g…

Right.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 24 Nov 2014 13:14:45 +0100

Hi there,

so I asked Henri Sivonen, author of the HTML5 parser in Firefox.

He points out that the "HTML" contains empty div tags

   <div .../>

which are invalid in HTML, and will be processed as just opening tags; 
thus each of these causes another nesting level in the DOM. Browsers are 
allowed not to support arbitrary nesting levels, and that's exactly what 
happens in this case.

The fix is not to serve XHTML with an HTML media type, or to use 
open/close syntax instead of empty syntax, such as in:

   <div ...></div>

Best regards, Julian

---

From: Yaron Sheffer <[email-masked]>
Date: Mon, 24 Nov 2014 14:22:24 +0200

> open/close syntax instead of empty syntax, such as in:

Firefox does read the file correctly when it knows that it is XHTML. So 
I think it is simplest to serve it with the correct MIME type.

By the way, I would also suggest to modify the xml2rfc utility so that 
the --html option will generate a file with an "xhtml" extension.

Thanks,
	Yaron

---

From: Tony Hansen <[email-masked]>
Date: Mon, 24 Nov 2014 10:57:46 -0500

On 11/24/14, 7:14 AM, Julian Reschke wrote:
> thus each of these causes another nesting level in the DOM. Browsers 
> are allowed not to support arbitrary nesting levels, and that's 
> exactly what happens in this case.
>

I've created a trac MR against xml2rfc to make this change.

     Tony Hansen

---

From: Paul Hoffman <[email-masked]>
Date: Mon, 24 Nov 2014 07:59:22 -0800

On Nov 24, 2014, at 4:14 AM, Julian Reschke <[email-masked]> wrote:
> which are invalid in HTML, and will be processed as just opening tags

Did Henri also say he might fix that obvious bug? That is <anything/> is never just an opening tag.

This is not to say that we shouldn't fix our declaration, just that he might also fix Firefox.

--Paul Hoffman

---

From: Tony Hansen <[email-masked]>
Date: Mon, 24 Nov 2014 11:20:50 -0500

> This is not to say that we shouldn't fix our declaration, just that he might also fix Firefox.

There are evidently two parts to the bug, only one of which that Henri 
has control of.

     Tony

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 24 Nov 2014 17:29:37 +0100

> Did Henri also say he might fix that obvious bug? That is <anything/> is never just an opening tag.

Welcome to the wonderful world of HTML syntax.

The <gi/> syntax is not part of HTML (it is tolerated for HTML5-“void”, i.e. SGML-“empty", elements, because it doesn’t cause ambiguity there).

Changing Firefox to “fix this” would likely break tons of websites.

> This is not to say that we shouldn't fix our declaration, just that he might also fix Firefox.

The only bug in Firefox here is that the deep nesting resulting from interoperably interpreting the malformed tags confuses it (unlike Chrome or Safari, which just live with that deep nesting).

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 24 Nov 2014 19:25:05 +0100

> Did Henri also say he might fix that obvious bug? That is <anything/> is never just an opening tag.

It's not a bug. The parser is free to limit the amount of recursion it 
supports in the DOM.

Best regards, Julian