Subject: [xml2rfc] #401 (Version 2 cli): <u> for code point 128 (C1 control) causes exception

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 11 Apr 2019 07:02:10 -0000

#401: <u> for code point 128 (C1 control) causes exception

 Input:

 {{{
 <t><u format="lit-num-name">&#x80;</u></t>
 }}}

 Output:

 {{{
 Traceback (most recent call last):
   File "/bin/xml2rfc", line 10, in <module>
     sys.exit(main())
   File "/usr/lib/python2.7/site-packages/xml2rfc/run.py", line 549, in
 main
     writer.write(filename)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 228, in write
     lines = self.render(self.root, width=72, joiners=joiners)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 2834, in render_rfc
     lines = self.ljoin(lines, c, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 446, in ljoin
     res = mklines(self.render(e, width, **kwargs), e)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 2024, in render_middle
     lines = self.ljoin(lines, c, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 446, in ljoin
     res = mklines(self.render(e, width, **kwargs), e)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 3102, in render_section
     lines = self.ljoin(lines, c, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 446, in ljoin
     res = mklines(self.render(e, width, **kwargs), e)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 3102, in render_section
     lines = self.ljoin(lines, c, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 446, in ljoin
     res = mklines(self.render(e, width, **kwargs), e)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 3470, in render_t
     text = fill(self.inner_text_renderer(e), width=width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 504, in inner_text_renderer
     text += self.render(c, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 271, in render
     res = func(e, width, **kwargs)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/text.py", line
 4241, in render_u
     self.err(e, exception)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/base.py", line
 1735, in err
     msg = self.msg(e, 'Error:', text)
   File "/usr/lib/python2.7/site-packages/xml2rfc/writers/base.py", line
 1676, in msg
     msg = "%s(%s): %s %s" % (file or self.xmlrfc.source, lnum, label,
 text, )
 UnicodeEncodeError: 'ascii' codec can't encode character u'\x80' in
 position 30: ordinal not in range(128)

 }}}

-- 
-----------------------------------+----------------------------------
 Reporter:  [email-masked]  |      Owner:  [email-masked]
     Type:  defect                 |     Status:  new
 Priority:  medium                 |  Milestone:
Component:  Version 2 cli          |    Version:  2.22.*
 Keywords:                         |
-----------------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 11 Apr 2019 07:02:27 -0000

Changes (by [email-masked]):

 * component:  Version 2 cli => Version_3_cli_txt

-- 
------------------------------------+----------------------------------
  Reporter:  [email-masked]  |      Owner:  [email-masked]
      Type:  defect                 |     Status:  new
  Priority:  medium                 |  Milestone:
 Component:  Version_3_cli_txt      |    Version:  2.22.*
Resolution:                         |   Keywords:
------------------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401#comment:1>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 11 Apr 2019 11:26:15 -0000

Comment (by [email-masked]):

 An error may be better than an exception. But one or the other is
 appropriate, because &#x80; is just an undefined control codepoint, and
 shouldn't turn up in an RFC at all.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401#comment:2>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 11 Apr 2019 12:35:57 -0000

Comment (by [email-masked]):

 AFAICT, it's not an undefined control point in Unicode - otherwise why
 does it show up in the Unicode database?

 But yes, *if* we want to exclude C1 controls, we need to clarify that in
 the documentation.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401#comment:3>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Fri, 12 Apr 2019 05:16:47 -0000

With "undefined control codepoint", I meant that it's a control, but it's
 not defined what kind of control. Some other controls have explanations
 (e.g. 009D <control> = OPERATING SYSTEM COMMAND), see
 https://www.unicode.org/charts/PDF/U0080.pdf. U+0080 doesn't. On top of
 that, even for those controls that have explanations, this is just the
 ISO/IEC 6429:1992 interpretation, and Unicode doesn't force that.

 As for documentation, I hope there is already something about how to use
 non-ASCII characters carefully. Not using U+0080 can easily be seen as
 including that, there's no need for explicitly mentioning it.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401#comment:4>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Fri, 12 Apr 2019 05:36:06 -0000

Martin, the code fails in the same way for U+0092.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/401#comment:5>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Fri, 26 Apr 2019 13:58:31 -0000

Changes (by [email-masked]):

 * status:  new => closed
 * resolution:   => fixed

Comment:

 Fixed in [3114]:

 Added a default rendering (code point number) for code points without
 unicode code point names.  Fixes issues #401 and #402.

-- 
------------------------------------+----------------------------------
  Reporter:  [email-masked]  |      Owner:  [email-masked]
      Type:  defect                 |     Status:  closed
  Priority:  medium                 |  Milestone:
 Component:  Version_3_cli_txt      |    Version:  2.22.*
Resolution:  fixed                  |   Keywords:
------------------------------------+----------------------------------

Ticket URL: </ticket/401#comment:6>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>