Subject: [xml2rfc] #338 (Version 2 cli): Python3 cannot read a utf-8 file w/o the BOF being present

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 05:26:27 -0000

#338: Python3 cannot read a utf-8 file w/o the BOF being present

 In python3 if there is no BOF marker at the start of the file, but the
 file is still UTF-8 and not ASCII then there is an error when the first
 UTF-8 character is reached.

-- 
------------------------------------+----------------------------------
 Reporter:  [email-masked]  |      Owner:  [email-masked]
     Type:  defect                  |     Status:  new
 Priority:  major                   |  Milestone:  1.37
Component:  Version 2 cli           |    Version:  2.5.x
 Keywords:                          |
------------------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 05:29:22 -0000

Comment (by [email-masked]):

 I have made a fix local to my system for this in parser.py about line 440

 OLD
             self.text = six.binary_type(open(self.source, "rU").read(),
 'utf8')

 NEW
             self.text = six.binary_type(open(self.source, "rU",
 encoding='utf8').read(), 'utf8')

 This will make the file be read as utf-8 even if there is not a BOM
 character present in the file.  It may be that this needs to be pushed
 back into the Python2 code as well, but it seems to work just fine based
 on testing several files.

-- 
-------------------------------------+----------------------------------
Resolution:                          |   Keywords:
-------------------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:1>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 06:03:36 -0000

Comment (by [email-masked]):

 What's relevant should be the XML declaration, and UTF-8 with or without
 BOM should work even without. See XML spec... A proper XML parser ought to
 deal with all these cases...

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:2>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 13:10:03 -0000

Comment (by [email-masked]):

 Yes, I suspect this is an incorrect bug report.  There's a test to read a
 unicode xml file in the xml2rfc test suite, and the test suite is run for
 python 2.7, 3.3, 3.4, and 3.5 using {{{tox}}} before releases (see
 {{{xml2rfc/trunk/cli/tox.ini}}}).

 I think the proposed fix will break xml2rfc under python3 if it is given
 files with an xml declaration that specifies for instance
 {{{encoding="latin-1"}}}; that is, any encoding different from ascii and
 utf-8.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:3>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 17:57:00 -0000

I have added the .md file which sourced the failing .xml file.  It has
 double quotes that are angled rather than straight.  The fix I gave makes
 this file run.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:4>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 19:10:15 -0000

Ok, but when running with these combinations:

   * xml2rfc 2.8.2, python 3.4.2, Linux 3.2.0
   * xml2rfc 2.8.2, python 3.4.6, OS-X 10.11.6 (El Capitan)

 the provided xml file, with unicode double-quotes, works as-is without any
 fix.

 With which xml2rfc / python / OS versions do you see this failing?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:5>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 19:56:32 -0000

I am running python 3.6.3 on windows.  I do not know what version is
 running on the Circle CI work that Martin is doing.
 https://circleci.com/gh/quicwg/base-drafts/3856?utm_campaign=build-
 failed&utm_medium=email&utm_source=notification

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:6>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 20:49:06 -0000

Ok.  xml2rfc version and windows version, please?

 It would also be good to have the exact failure output.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:7>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 02 Jan 2018 23:44:26 -0000

Comment (by [email-masked]):

 The docker image I used is here:
 https://hub.docker.com/r/martinthomson/i-d-template/builds/bfea63z4fkjwv6bkacjbfyk/

 As you can see, this is using python 3.5.2 on ubuntu 16.04 with xml2rfc
 2.8.2.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:8>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 03 Jan 2018 14:10:59 -0000

2.8.2.

 Ahh.  Splendid.  Together with additional info from Jim, this makes me
 believe the problem could be related to environmental settings, rather
 than OS or python version.  Could you try this patch, please:

 {{{
 Index: xml2rfc/parser.py
 ===================================================================
 --- xml2rfc/parser.py   (revision 2395)
 +++ xml2rfc/parser.py   (working copy)
 @@ -437,7 +437,7 @@
          if six.PY2:
              self.text = open(self.source, "rU").read()
          else:
 -            self.text = six.binary_type(open(self.source, "rU").read(),
 'utf8')
 +            self.text = open(self.source, "rUb").read()

          # Get an iterating parser object
          file = six.BytesIO(self.text)

 }}}

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:9>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 03 Jan 2018 23:57:41 -0000

That change worked for me.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:10>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 04 Jan 2018 13:29:33 -0000

Changes (by [email-masked]):

 * status:  new => closed
 * resolution:   => fixed

Comment:

 Excellent.  I've released 2.8.3 with this fix.

-- 
-------------------------------------+----------------------------------
  Reporter:  [email-masked]  |      Owner:  [email-masked]
      Type:  defect                  |     Status:  closed
  Priority:  major                   |  Milestone:  1.37
 Component:  Version 2 cli           |    Version:  2.5.x
Resolution:  fixed                   |   Keywords:
-------------------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/338#comment:11>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>