Subject: [Tools-discuss] non-ASCII, non-UTF-8 in rsync'ed html

From: Daniel Kahn Gillmor <[email-masked]>
Date: Mon, 19 Nov 2018 18:53:18 -0500

I ran the following to get all html files:

     rsync -az rsync.tools.ietf.org::tools.html/ html/

But i found that there were several files in that repo that contained
non-ASCII and non-UTF-8 octet sequences.

In particular, i ran:

    find html/ -iname '*.html' -print0 | xargs -0 file | grep -v 'ASCII text' | grep -v 'UTF-8 Unicode text'

and it produced this output:

html/draft-calabrese-requir-logprot-04.html:                     data
html/draft-dewinter-queue-start-00.html:                         data
html/draft-dewinter-queue-start-01.html:                         data
html/draft-duerst-iri-05.html:                                   data
html/draft-dusse-smime-msg-03.html:                              data
html/draft-felton-universal-language-00.html:                    data
html/draft-fujiwara-dnsop-resolver-update-00.html:               data
html/draft-iana-special-ipv4-03.html:                              data
html/draft-iana-special-ipv4-04.html:                              data
html/draft-ietf-2000-issue-00.html:                                data
html/draft-ietf-2000-issue-03.html:                                data
html/draft-ietf-2000-issue-04.html:                                data
html/draft-ietf-2000-issue-05.html:                                data
html/draft-ietf-2000-issue-06.html:                                data
html/draft-ietf-ccamp-mpls-tp-rsvpte-ext-associated-lsp-08.html: data
html/draft-ietf-dhc-mdhcp-00.html:                               data
html/draft-ietf-dhc-multopt-00.html:                             data
html/draft-ietf-disman-remops-mib-00.html:                       data
html/draft-ietf-disman-remops-mib-01.html:                       data
html/draft-ietf-dnsop-interim-signed-root-01.html:               data
html/draft-ietf-hubmib-mau-mib-03.html:                          data
html/draft-ietf-hubmib-mau-mib-04.html:                          data
html/draft-ietf-hubmib-repeater-dev-02.html:                     data
html/draft-ietf-hubmib-repeater-dev-03.html:                     data
html/draft-ietf-ipcdn-pktc-signaling-07.html:                   data
html/draft-ietf-ippm-ipdv-01.html:                              data
html/draft-ietf-isis-opexp-01.html:                             data
html/draft-ietf-ldapext-matchedval-06.html:                       data
html/draft-ietf-mpls-p2mp-loose-path-reopt-01.html:              data
html/draft-ietf-ospf-hmac-sha-01.html:                           data
html/draft-ietf-pkix-cmmf-02.html:                               data
html/draft-ietf-pkix-scvp-15.html:                               data
html/draft-ietf-rmonmib-pi-ipv6-01.html:                         data
html/draft-ietf-rtfm-new-traffic-flow-00.html:                   data
html/draft-ietf-smime-domsec-00.html:                                data
html/draft-ietf-spki-cert-req-00.html:                               data
html/draft-ietf-spki-cert-structure-00.html:                         data
html/draft-ihren-dnsop-interim-signed-root-01.html:                   data
html/draft-klensin-dns-role-01.html:                             data
html/draft-lear-ietf-rfc2026bis-00.html:                         data
html/draft-leung-sigtran-stream-sctp-00.html:                    data
html/draft-loughney-sctp-sig-prot-00.html:                        data
html/draft-murray-auth-ftp-ssl-03.html:                              data
html/draft-newnan-isomib-internet-00.html:                           data
html/draft-rabbat-ccamp-carrier-survey-01.html:                      data
html/draft-rfced-info-dudley-01.html:                                               data
html/draft-vasseur-mpls-backup-computation-02.html:               data
html/draft-vrancken-oauth-redelegation-01.html:                   data
html/draft-wenzel-cctld-bcp-00.html:                              data
html/draft-white-slapm-mib-00.html:                               data
html/draft-worster-mpls-in-ip-02.html:                            data
html/draft-zhu-rmcat-nada-02.html:                              data
html/draft-zhu-rmcat-nada-03.html:                              data
html/rfc1142.html:                                              data
html/rfc542.html:                                            data
html/rfc652.html:                                            data
html/rfc674.html:                                            data
html/rfc684.html:                                            data
html/rfc731.html:                                            data
html/rfc734.html:                                            data
html/rfc736.html:                                            data
html/rfc752.html:                                            data
html/rfc774.html:                                            data
html/rfc776.html:                                            data
html/rfc783.html:                                            data

Are these things that can be cleaned up by a future html re-rendering?
they're likely to break html parsers or other transformations that work
from the html source.

Regards,

    --dkg

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Nov 2018 14:45:28 +0100

Hi Daniel,

> non-ASCII and non-UTF-8 octet sequences.

Yes.  Many of these (possibly all, but I haven't inspected each file
individually) are due to control characters in the text documents.
SUB (^Z) seems to be the most common one, but there are BS, DC2, SI,
ACK, FS, and more.

Yes, I could have a go at filtering out those at the same time as I
generate the htmlized versions.  I'll do so for control characters,
then we can revisit and see if additional action is needed.

Best regards,

	Henrik

> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]
>

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 20 Nov 2018 11:16:26 -0500

On Tue 2018-11-20 14:45:28 +0100, Henrik Levkowetz wrote:
>> i found that there were several files in that repo that contained

Thanks!  Filtering those out for the HTML would be good.  But that's
clearly not the only thing remaining, since some files contain high
bytes.

I did a review of all of the documents that contain octets above 0x7f:

 * draft-ietf-2000-issue-05.html is the most troubling one -- it appears
   to contain fragments of an embedded pdf document, if i'm reading it
   correctly.

 * draft-felton-universal-language-00.html contains what appears to be
   attempted embedded unicode Chinese, but it has been garbled.

 * draft-ietf-dnsop-interim-signed-root-01.html and
   draft-ihren-dnsop-interim-signed-root-01.html contain more mangled
   unicode in the author names.

 * draft-ietf-pkix-cmmf-02.html contains some kind of mangled em dashes
   and smartquotes

 * draft-ietf-rmonmib-pi-ipv6-01.html has a lot of weirdness in the very
   end of the document

 * draft-ietf-smime-domsec-00.html appears to have mangled smartquotes

 * draft-klensin-dns-role-01.html has some breakage around "Montréal"
   and 'German glyph "ö"'

 * draft-loughney-sctp-sig-prot-00.html contains some mangled characters
   in the organization addresses

 * draft-rabbat-ccamp-carrier-survey-01.html contains a weird character
   in the page headers between "Expires" and "April 2005"

 * draft-vasseur-mpls-backup-computation-02.html mangles "España"

 * draft-worster-mpls-in-ip-02.html has a damaged Fax number for Rick
   Wilder

Regards,

        --dkg

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 20 Nov 2018 17:28:59 +0100

> I did a review of all of the documents that contain octets above 0x7f:

Are these all expired I-Ds?

I don’t think anything can really be done about those, as the archival value of keeping them around is based on them being unchanged.

Grüße, Carsten

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 20 Nov 2018 12:30:58 -0500

On Tue 2018-11-20 17:28:59 +0100, Carsten Bormann wrote:
> I don’t think anything can really be done about those, as the archival
> value of keeping them around is based on them being unchanged.

what is the value of unchanged invalid bytestreams?  we re-render other
old documents when the html rendering process is improved.  why not do
the same for these if we know there is cleanup that will make the
document valid?

I'm willing to entertain specific arguments, but it's not clear to me
that we've ever held out that "these HTML renderings must remain
byte-for-byte identical" -- have we?

         --dkg

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Nov 2018 18:40:19 +0100

> I did a review of all of the documents that contain octets above 0x7f:

I've looked through the equivalent .txt files, and I believe that common
for all except draft-ietf-2000-issue is that they use unknown code pages;
not latin-1 and certainly not utf-8.  An attempt to check some of them
against other less code pages failed to find a match.

Any fixup of these would have to be built into the htmlizer as custom
tweaks for each specific document (except for draft-ietf-2000-issue,
where some other solution would be needed).

All of these are old, I think the latest from 2005; two from 2004; and
the rest earlier.  I doubt it's meaningful to add custom fixes...

Best regards,

	Henrik

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 20 Nov 2018 13:11:00 -0500

> against other less code pages failed to find a match.

yep, i think there's been some sort of multi-trip conversion that
happened to them in the past, which makes them not quite right.

> the rest earlier.  I doubt it's meaningful to add custom fixes...

What does "i doubt it's meaningful" mean?  I'm working on trying to
import all of these drafts into a single index as a first step toward
providing nice/simple search and inference about these drafts.  It's
certainly meaningful to me to have syntactically valid documents for
doing this kind of work.

Of course, i could add custom fixes to the docs myself before i index
them, but that seems like the wrong way to fix things for the broader
community.

        --dkg

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 20 Nov 2018 10:41:51 -0800

> certainly meaningful to me to have syntactically valid documents for

Why are you doing this on the HTML instead of the source text files? The 
latter seems less likely to be borked.

--Paul

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Nov 2018 19:58:36 +0100

On 2018-11-20 19:11, Daniel Kahn Gillmor wrote:
> What does "i doubt it's meaningful" mean?

It means that I already work way too many hours per week and have a hard
time justifying putting hours into this, I'm afraid.  It is more a matter
of what I can justify at the moment, not an attempt to state what's
meaningful in a wider context.

> I'm working on trying to

Good background info; makes me understand better what's driving this.

> them, but that seems like the wrong way to fix things for the broader

Ack.

If you could provide it, I can easily insert python code on this pattern
into the htmlizer:

    if draftname == "draft-ietf-dnsop-interim-signed-root-01":
        text = text.replace(u"F\x84ltstr\xF7m", u"F\u00e4ltstr\u00f6")
        text = text.replace(u"Ihr\x89n", u"Ihr\u00e9n")
        # ...

Best,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Nov 2018 20:05:21 +0100

> latter seems less likely to be borked.

The source text files *are* broken.  That's the origin of the problem.

	Henrik

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 20 Nov 2018 14:26:33 -0500

> meaningful in a wider context.

gotcha, thanks!  I wasn't trying to imply that you should drop
everything and work on this :)

>         text = text.replace(u"Ihr\x89n", u"Ihr\u00e9n")

cool, great. any particular flavor of python that you prefer (when
dealing with strings and bytes, python2 vs. python3 code tends to be
pretty different)?

If you point me toward the preferred revision control system where this
work would go in, maybe i can send patches, rather than just raw python
snippets.

thanks,

        --dkg

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Nov 2018 22:08:09 +0100

> everything and work on this :)

Allright :-)

> dealing with strings and bytes, python2 vs. python3 code tends to be

These days I try to write code that works on 2.7, 3.4, 3.5, and 3.6.

In the code above, 'text' is unicode under 2.7 and str under 3.x; and
the code should work as written under both 2.7 and 3.4 and upwards.

> work would go in, maybe i can send patches, rather than just raw python

It will have to go into both the standalone htmlizer lib and the legacy
rfcmarkup scripts, so I'll have to adapt a bit.  Fwiw, the lib is
html2rfc at https://pypi.org/project/rfc2html/ , with SVN at
https://svn.tools.ietf.org/svn/src/rfc2html/.

The legacy script is in https://svn.tools.ietf.org/svn/src/rfcmarkup.
Please don't consider providing patches for both; I'm happy to do the
adaptations.

The legacy script has code to pull all the stuff in the 'header' above the
actual document from various places, and also insert CSS and dublin core
meta-information.  At some point I'll have to rewrite it to use the lib,
and pull information from the datatracker database instead of from misc.
files.

I'll have to add an optional 'name' or 'draftname' parameter to the function
call in the lib in order to have something to test the document name against,
but conceptually the code would look like what you see above.

This was converted using the rfc2html lib under python 3.6:
http://durif.tools.ietf.org/src/rfc2html/tmp/draft-ietf-dnsop-interim-signed-root-01.html

and this using the legacy script under 2.7:
http://durif.tools.ietf.org/src/rfcmarkup/tmp/draft-ietf-dnsop-interim-signed-root-01.html

Best regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 21 Nov 2018 07:05:17 +0100

Ah, this is about modern HTML generated from ancient borken expired I-Ds.

In that case, scrubbing the input indeed seems like the thing to do.

Sorry for misunderstanding the thread.

Grüße, Carsten

> On Nov 20, 2018, at 18:30, Daniel Kahn Gillmor <[email-masked]> wrote:
> 
> Signed PGP part