Subject: [xml2rfc] Unable to run xml2rfc --help

From: Roger Price <[email-masked]>
Date: Fri, 25 Dec 2020 15:24:25 +0100 (CET)

I downloaded xml2rfc onto a Debian Stretch machine and ran the command

  xml2rfc --help

as described at 
https://tools.ietf.org/src/xml2rfc/trunk/cli/doc/xml2rfc3.html#format-switches

The result was

Traceback (most recent call last):
   File "/usr/bin/xml2rfc", line 7, in <module>
     from xml2rfc.run import main
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/__init__.py", line
     14, in <module>
     from xml2rfc.parser import  XmlRfcError, CachingResolver, XmlRfcParser, XmlRfc
...
   File "/usr/lib/python3.5/sre_parse.py", line 437, in _parse_sub
     itemsappend(_parse(source, state))
   File "/usr/lib/python3.5/sre_parse.py", line 575, in _parse
     raise source.error(msg, len(this) + 1 + len(that))
sre_constants.error: bad character range ”-“ at position 7

I am obviously doing something wrong, but I can't see what it is.  Any 
suggestion would be much appreciated.

Roger

---

From: Roger Price <[email-masked]>
Date: Mon, 28 Dec 2020 16:49:23 +0100 (CET)

> sre_constants.error: bad character range ”-“ at position 7

I went back to this problem.  xml2rfc --help still crashes, so I tried xml2rfc 
-h.  That worked, so I went back to xml2rfc --help which then worked correctly.

I then tried xml2rfc draft-davies-template-bare-07.xml : Python crashes with the 
same error message.  I tried xml2rfc --debug draft-davies-template-bare-07.xml 
and that worked. I tried xml2rfc --text draft-davies-template-bare-07.xml and 
Python crashed, so I tried xml2rfc draft-davies-template-bare-07.xml --text and 
that worked.

I repeated xml2rfc draft-davies-template-bare-07.xml --text and this time Python 
crashed with the usual error message.

It looks as if xml2rfc is relying on some work area which it assumes has some 
initial value, but in practice has a random value.

I get the same randomness with Python 3.5.3 and Python 3.8.1. Command uname -a 
reports Linux maria 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) 
x86_64 GNU/Linux

Roger

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 28 Dec 2020 19:14:13 +0100

Can only guess, but probably your locale isn’t en_US.UTF-8 which you may want to fix. 

Sent from mobile, sorry for terse

> On 28. Dec 2020, at 16:49, Roger Price <[email-masked]> wrote:
> 
> I went back to this problem.  xml2rfc --help still crashes, so I tried xml2rfc -h.  That worked, so I went back to xml2rfc --help which then worked correctly.
> 
> I then tried xml2rfc draft-davies-template-bare-07.xml : Python crashes with the same error message.  I tried xml2rfc --debug draft-davies-template-bare-07.xml and that worked. I tried xml2rfc --text draft-davies-template-bare-07.xml and Python crashed, so I tried xml2rfc draft-davies-template-bare-07.xml --text and that worked.
> 
> I repeated xml2rfc draft-davies-template-bare-07.xml --text and this time Python crashed with the usual error message.
> 
> It looks as if xml2rfc is relying on some work area which it assumes has some initial value, but in practice has a random value.
> 
> I get the same randomness with Python 3.5.3 and Python 3.8.1. Command uname -a reports Linux maria 4.9.0-4-amd64 #1 SMP Debian 4.9.65-3+deb9u1 (2017-12-23) x86_64 GNU/Linux
> 
> Roger_______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Roger Price <[email-masked]>
Date: Mon, 28 Dec 2020 21:35:02 +0100 (CET)

> Can only guess, but probably your locale isn’t en_US.UTF-8 which you may want to fix. 

The command locale reports:
  LANG=en_US.UTF-8
  LANGUAGE=en_US:en
  LC_CTYPE="en_US.UTF-8"
  LC_NUMERIC="en_US.UTF-8"
  LC_TIME="en_US.UTF-8"
  LC_COLLATE="en_US.UTF-8"
  LC_MONETARY="en_US.UTF-8"
  LC_MESSAGES="en_US.UTF-8"
  LC_PAPER="en_US.UTF-8"
  LC_NAME="en_US.UTF-8"
  LC_ADDRESS="en_US.UTF-8"
  LC_TELEPHONE="en_US.UTF-8"
  LC_MEASUREMENT="en_US.UTF-8"
  LC_IDENTIFICATION="en_US.UTF-8"
  LC_ALL=

I tried LANG=C xml2rfc -h but this still works/fails randomly.

The half of the time when xml2rfc works, it seems to work correctly and I have 
produced text output from the sample draft.  I can also get debug output on 
STDERR.

Roger

---

From: Roger Price <[email-masked]>
Date: Tue, 29 Dec 2020 11:21:45 +0100 (CET)

On Mon, 28 Dec 2020, Carsten Bormann wrote:

> On 2020-12-25, at 15:24, Roger Price <[email-masked]> wrote:
>> python3.5
> but this got my attention: Python 3.5 is end-of-life; did you try a newer 
> Python?

I changed the xml2rfc shebang to #!/usr/bin/python3.8 and ran Python 3.8.1 but 
it made no difference.

> I’m out of educated guesses (at least until you are showing more of the traceback),

The full Python error message is:

Traceback (most recent call last):
   File "/usr/bin/xml2rfc", line 7, in <module>
     from xml2rfc.run import main
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/__init__.py", line
     14, in <module>
     from xml2rfc.parser import  XmlRfcError, CachingResolver, XmlRfcParser, XmlRfc
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/parser.py",
     line 20, in <module> from xml2rfc.writers import base
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/__init__.py",
     line 2, in <module> from xml2rfc.writers.base import RfcWriterError
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/writers/base.py",
     line 30, in <module>
     from xml2rfc.util.unicode import ( punctuation, unicode_replacements,
     unicode_content_tags, bare_unicode_tags,
   File "/mnt/home/rprice/.local/lib/python3.5/site-packages/xml2rfc/util/unicode.py",
     line 260, in <module>
     punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))
   File "/usr/lib/python3.5/re.py", line 224, in compile
     return _compile(pattern, flags)
   File "/usr/lib/python3.5/re.py", line 293, in _compile
     p = sre_compile.compile(pattern, flags)
   File "/usr/lib/python3.5/sre_compile.py", line 536, in compile
     p = sre_parse.parse(p, flags)
   File "/usr/lib/python3.5/sre_parse.py", line 829, in parse
     p = _parse_sub(source, pattern, 0)
   File "/usr/lib/python3.5/sre_parse.py", line 437, in _parse_sub
     itemsappend(_parse(source, state))
   File "/usr/lib/python3.5/sre_parse.py", line 575, in _parse
     raise source.error(msg, len(this) + 1 + len(that))
sre_constants.error: bad character range …-‚ at position 3

In file unicode.py at line 260

  punctuation_re = re.compile(r'[%s]'%''.join(list(punctuation.keys())))

I added the two lines

  import sys
  print("unicode.py: list(punctuation.keys())) = {}".format(list(punctuation.keys())))

On runs when Python crashes I see outputs:

unicode.py: list(punctuation.keys()) = ['−', '‐', '′', '\u2002', '’', '´',
                  '\u2003', '‚', '–', '\u2009', '”', '-', '“', '‘', '…', '„', '—']
unicode.py: list(punctuation.keys()) = ['„', '”', '—', '–', '…', '-', '\u2009',
                  '‘', '’', '\u2003', '−', '´', '\u2002', '‚', '′', '‐', '“']

When the runs succeed I see outputs:

unicode.py: list(punctuation.keys()) = ['−', '\u2003', '\u2009', '‘', '“', '‚',
                  '-', '”', '—', '’', '′', '\u2002', '‐', '–', '…', '´', '„']
unicode.py: list(punctuation.keys()) = ['\u2002', '‐', '′', '\u2003', '−', '´',
                  '“', '”', '„', '–', '-', '‘', '’', '…', '‚', '\u2009', '—']

I do not understand why such a list of constants has to be so random. I'm not 
sure where to look next, but if you would like to see further traces of other 
functions, just ask. Roger

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 29 Dec 2020 16:44:54 +0100

That appears to be a rather funny bug.
Apparently, the ASCII neutral minus hyphen is in the list of keys. Depending on how the Python hashing random number generator is seeded, that hyphen is going to a position where it can’t be in the regexp or where it can. True Heisenbug. 
(The hyphen needs to be protected to go into a character class, even when the resulting character range seems to be syntactically acceptable.)

> On 29. Dec 2020, at 11:22, Roger Price <[email-masked]> wrote:
> 
>> but this got my attention: Python 3.5 is end-of-life; did you try a newer Python?
> 
> I changed the xml2rfc shebang to #!/usr/bin/python3.8 and ran Python 3.8.1 but it made no difference.
> 
> I do not understand why such a list of constants has to be so random. I'm not sure where to look next, but if you would like to see further traces of other functions, just ask. Roger_______________________________________________
> xml2rfc mailing list
> [email-masked]