Subject: [xml2rfc] WARNING: Illegal character replaced in string:

From: Niels ten Oever <[email-masked]>
Date: Fri, 13 May 2016 00:19:17 +0200

Hi all,

We made the attached ID in markdown, which we then made into xml using
kramdown-rfc2629, but at the moment xml2rfc is called, an infinitely
repeating 'WARNING: Illegal character replaced in string:' ensues. See
below:

[code]

$ make
xml2rfc draft-research.xml --text
Parsing file draft-research.xml
WARNING: Illegal character replaced in string: &#8232;
WARNING: Illegal character replaced in string: 	
WARNING: Illegal character replaced in string: 	
WARNING: Illegal character replaced in string:

<continue until CRTL-C>

^CTraceback (most recent call last):
  File "/usr/bin/xml2rfc", line 225, in <module>
    main()
  File "/usr/bin/xml2rfc", line 210, in main
    pagedwriter.write(filename)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/writers/base.py", line
1173, in write
    xml2rfc.utils.safeReplaceUnicode(self.r)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/utils.py", line 298, in
safeReplaceUnicode
    element.text = _replace_unicode_characters(element.text)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/utils.py", line 324, in
_replace_unicode_characters
    str = re.sub(match.group(1), entity, str)
  File "/usr/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
KeyboardInterrupt
Makefile:18: recipe for target 'draft-research.txt' failed
make: *** [draft-research.txt] Error 1

[/code]

Any suggestions on how I can solve this?

Thanks in advance,

Niels

-- 
Niels ten Oever
Head of Digital

Article 19
www.article19.org

PGP fingerprint    8D9F C567 BEE4 A431 56C4
                   678B 08B5 A0F2 636D 68E9

---

From: "Jim Schaad" <[email-masked]>
Date: Thu, 12 May 2016 16:30:01 -0700

My version of xml2rfc (which has the bug fix for this)  says you have a &#287; character someplace in your text.  The other possibility is that you have a tab character someplace in the middle of a string (i.e. not at the end or beginning of a line) and that gets it confused as well.

I would start by removing the tab characters and then see what happens.

Jim

-----Original Message-----
From: xml2rfc [mailto:[email-masked]] On Behalf Of Niels ten Oever
Sent: Thursday, May 12, 2016 3:19 PM
To: [email-masked]
Subject: [xml2rfc] WARNING: Illegal character replaced in string:

Hi all,

We made the attached ID in markdown, which we then made into xml using kramdown-rfc2629, but at the moment xml2rfc is called, an infinitely repeating 'WARNING: Illegal character replaced in string:' ensues. See
below:

[code]

$ make

<continue until CRTL-C>

^CTraceback (most recent call last):
  File "/usr/bin/xml2rfc", line 225, in <module>
    main()
  File "/usr/bin/xml2rfc", line 210, in main
    pagedwriter.write(filename)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/writers/base.py", line 1173, in write
    xml2rfc.utils.safeReplaceUnicode(self.r)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/utils.py", line 298, in safeReplaceUnicode
    element.text = _replace_unicode_characters(element.text)
  File "/usr/lib/python2.7/dist-packages/xml2rfc/utils.py", line 324, in _replace_unicode_characters
    str = re.sub(match.group(1), entity, str)
  File "/usr/lib/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count) KeyboardInterrupt
Makefile:18: recipe for target 'draft-research.txt' failed
make: *** [draft-research.txt] Error 1

[/code]

Thanks in advance,

Niels

--
Niels ten Oever
Head of Digital

Article 19
www.article19.org

---

From: Matthew Kerwin <[email-masked]>
Date: Fri, 13 May 2016 09:36:26 +1000

On 13 May 2016 at 08:19, Niels ten Oever <[email-masked]>
wrote:

> repeating 'WARNING: Illegal character replaced in string:' ensues.
> ​
>

<snip>
> Any suggestions on how I can solve this?

​There are a whole bunch of (presumably cut-and-pasted?) non-ASCII
codepoints in the document; most of them are dashes, bullets, and quotation
marks. However just before line ~955 there's a U+2028 LINE SEPARATOR
character in there. That's the one that xml2rfc is complaining about in
your error message.

There are also a whole lot of tabs in strange places.

By fixing a lot of those issues, I made a copy of the markdown that renders
down to XML, that in turn passes through xml2rfc with only a warning about
overlong lines in some artwork.

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Niels ten Oever <[email-masked]>
Date: Fri, 13 May 2016 09:38:28 +0200

Dear Mathew, Alice and Jim,

Thanks so much for your very quick and great responses. I should have
thought of the possibility of the Debian repositories having an old
version of xml2rfc.

Again thanks a lot for your support! Writing with a lot of different
authors in a doc and pulling in changes via Git can make the build
process quite adventurous.

Cheers,

Niels

On 05/13/2016 01:36 AM, Matthew Kerwin wrote:
> On 13 May 2016 at 08:19, Niels ten Oever <[email-masked]
> <mailto:[email-masked]>>wrote:
> 
>     Hi all,
> 
> codepoints in the document; most of them are dashes, bullets, and
> quotation marks. However just before line ~955 there's a U+2028 LINE
> SEPARATOR character in there. That's the one that xml2rfc is complaining
> about in your error message.
> 
> By fixing a lot of those issues, I made a copy of the markdown that
> renders down to XML, that in turn passes through xml2rfc with only a
> warning about overlong lines in some artwork.
> 
> -- 
>   Matthew Kerwin

-- 
Niels ten Oever
Head of Digital

Article 19
www.article19.org

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Fri, 13 May 2016 12:50:29 +0000

You might consider adding something to your conversion process from markdown to xml2rfc that 

*) looks for various common non-ascii characters and replaces them with ascii equivalents (e.g. ',",-)
*) warns about other non-ascii characters

	Tony Hansen

On 5/13/16, 3:38 AM, "xml2rfc on behalf of Niels ten Oever" <[email-masked] on behalf of [email-masked]> wrote:

>_______________________________________________
>xml2rfc mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/xml2rfc