Subject: [xml2rfc] The bibxml entry of RFC 8516 creates an error with xml2rfc

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 13:35:20 +0100

Hello list,

The bibxml entry for RFC 8516
(https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.8516.xml)

  contains umlauts, namely in this line:
<author initials="A." surname="Keränen" fullname="A. Keränen">

This in turn makes xml2rfc on a draft including this reference go like so:

WARNING: Illegal character replaced in string: &#258;

and the reference looks like this in the txt version:

   [RFC8516]  Ker&#258;[currency units]nen, A., ""Too Many Requests"
               Response Code for the Constrained Application Protocol",
               RFC 8516, DOI 10.17487/RFC8516, January 2019,
               <https://www.rfc-editor.org/info/rfc8516>.

Can someone kindly fix this? Ari Keränen really doesn't need all those 
pesky umlauts ;-)

/Ludwig

-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 31 Jan 2019 14:13:52 +0100

> On Jan 31, 2019, at 13:35, Ludwig Seitz <[email-masked]> wrote:
> 
> Hello list,
> 
> <author initials="A." surname="Keränen" fullname="A. Keränen”>

Nice to have real names in the bibxml now.

> WARNING: Illegal character replaced in string: &#258;

Hmm, it doesn’t for me.

   [RFC8516]  Keraenen, A., ""Too Many Requests" Response Code for the
              Constrained Application Protocol", RFC 8516,
              DOI 10.17487/RFC8516, January 2019,
              <https://www.rfc-editor.org/info/rfc8516>.

Now we can discuss whether Germanizing that last name is such a good idea, but that’s how xml2rfc’s ASCII transliteration works at the moment.

>              <https://www.rfc-editor.org/info/rfc8516>.

Something appears to be really broken on your side.  How does “ä” (which is &#228;) turn into &#258;[currency units]?

Can you look into your cache file whether it arrived intact?

~/.cache/xml2rfc/reference.RFC.8516.xml

> Ari Keränen really doesn’t need all those pesky umlauts ;-)

I’ll let Ari decide that :-)

Grüße, Carsten

---

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 14:53:57 +0100

> Nice to have real names in the bibxml now.
I agree in principle.

> Something appears to be really broken on your side.  How does “ä” (which is &#228;) turn into &#258;[currency units]?
That was the question I was asking myself.

> ~/.cache/xml2rfc/reference.RFC.8516.xml

There I find:

<author initials='A.' surname='KerĂ¤nen' fullname='A. 
KerĂ¤nen'><organization /></author>

/Ludwig

-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 15:05:19 +0100

> Something appears to be really broken on your side.  How does “ä” (which is &#228;) turn into &#258;[currency units]?

More data:

I'm also using xml2rfc version 2.5.1 on a Ubuntu 18.04 machine. I can 
reproduce the error (even after manually deleting the cache file).

/Ludwig

-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 31 Jan 2019 15:07:47 +0100

What is your locale setting?

Sent from mobile

> On 31. Jan 2019, at 15:05, Ludwig Seitz <[email-masked]> wrote:
> 
> I'm also using xml2rfc version 2.5.1 on a Ubuntu 18.04 machine. I can reproduce the error (even after manually deleting the cache file).
> 
> /Ludwig
> 
> -- 
> Ludwig Seitz, PhD
> Security Lab, RISE
> Phone +46(0)70-349 92 51
>

---

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 15:21:42 +0100

> What is your locale setting?
$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=en_GB.UTF-8
LC_TIME=en_GB.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=en_GB.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=en_GB.UTF-8
LC_NAME=en_GB.UTF-8
LC_ADDRESS=en_GB.UTF-8
LC_TELEPHONE=en_GB.UTF-8
LC_MEASUREMENT=en_GB.UTF-8
LC_IDENTIFICATION=en_GB.UTF-8
LC_ALL=

-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Julian Reschke <[email-masked]>
Date: Thu, 31 Jan 2019 15:29:59 +0100

On 2019-01-31 14:13, Carsten Bormann wrote:
> ...

Not so sure about that, unless the tool chain (xml2rfc, idnits etc 
actually handle them properly.

Also, I would expect the bib entry to contain the name the same way it 
was spelled on the document, and that's not the case here:

> Internet Engineering Task Force (IETF)                        A. Keranen
> Request for Comments: 8516                                      Ericsson
> Category: Standards Track                                   January 2019
> ISSN: 2070-1721

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 31 Jan 2019 15:35:44 +0100

> Not so sure about that, unless the tool chain (xml2rfc, idnits etc actually handle them properly.

Right.  But this now gives us an opportunity to debug things that we would have to wait for for another year.

> Also, I would expect the bib entry to contain the name the same way it was spelled on the document, and that's not the case here:
> 

Right.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 31 Jan 2019 16:31:35 +0100

Hi Ludwig,

> reproduce the error (even after manually deleting the cache file).

Version 2.5.1 is quite old (released 19 Oct 2015) and a lot has happened
since then with respect to non-ascii support.  I'd suggest upgrading to
a newer version through 'pip install'.  If the 2.5.1 release had the --utf8
switch, you could try to do a run with that switch specified to get around
the immediate issue.  But even if that works, upgrading to a newer version
seems advisable.

Regards

	Henrik

---

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 16:46:47 +0100

> the immediate issue.  But even if that works, upgrading to a newer version

"pip install xml2rfc" gives me version 2.5.1 on Ubuntu 18.04.

/Ludwig

-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Ludwig Seitz <[email-masked]>
Date: Thu, 31 Jan 2019 16:55:32 +0100

On 31/01/2019 16:46, Ludwig Seitz wrote:
>>>> Something appears to be really broken on your side.  How does “ä” 
>>>> (which is &#228;) turn into &#258;[currency units]?
>>>>
>>>
>>> More data:
>>>

I tested manually editing the xml2rfc cache file to fix the umlauts, and 
that works like  a charm, the problem seems to be that the bibxml file 
is mangled in transfer
(https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.8516.xml 
looks ok to me in a browser).

/Ludwig
-- 
Ludwig Seitz, PhD
Security Lab, RISE
Phone +46(0)70-349 92 51

---

From: Julian Reschke <[email-masked]>
Date: Thu, 31 Jan 2019 16:57:54 +0100

> the immediate issue.  But even if that works, upgrading to a newer version

Henrik, do you know whether idnits would actually accept a draft with 
non-ASCII characters nowadays?

(If there was an announcement, I have missed it...)

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 31 Jan 2019 17:04:44 +0100

Hi Julian,

> (If there was an announcement, I have missed it...)

I haven't looked for an official announcement either, but this is what
I have, from the idnits release notes:

idnits (2.16.00)

  * Modified idnits to be more discerning in identifying possible FQDNs,
    in particular to not identify email localparts as FQDNs.

  * After guidance from the IESG, changed the error for ASCII characters
    in drafts to a warning.

 -- Henrik Levkowetz <[email-masked]>  07 Oct 2018 14:54:28 +0200

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 31 Jan 2019 17:07:07 +0100

>   -- Henrik Levkowetz <[email-masked]>  07 Oct 2018 14:54:28 +0200

Awesome.

I guess it's time to try that then :-)

Best regards, Julian

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 1 Feb 2019 01:48:19 +0000

On 2019/01/31 23:35, Carsten Bormann wrote:

 >On 2019/01/31 23:29, Julian Reschke wrote:

>>> Category: Standards Track                                   January 2019

Actually, I'd prefer that references use the real name where possible. 
To that effect, most of the RFCs I have been involved in contain 
something like

Authors' Addresses

    Martin Duerst  (Note: Please write "Duerst" with u-umlaut wherever
                   possible, for example as "D&#252;rst" in XML and
                   HTML.)

But that may be just me.

Regards,   Martin.

---

From: Martin J. Dürst <[email-masked]>
Date: Fri, 1 Feb 2019 01:53:05 +0000

> KerĂ¤nen'><organization /></author>

That looks like a double encoding. UTF-8 data was interpreted as Latin-1 
and converted again to UTF-8. Somewhat surprising with all your locale 
settings at UTF-8. Could have happened at many different points, 
including e.g. when copying from an editor that thought the file was 
Latin-1. You may need to use some hex dump to find the culprit.

Regards,   Martin.

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 1 Feb 2019 07:06:50 +0100

> On Feb 1, 2019, at 02:53, Martin J. Dürst <[email-masked]> wrote:
> 
> and converted again to UTF-8.

Look closely: This is a Ă (U+0102), not the Ã (U+00C3) I’d expect from false-interpreting UTF-8 U+00C4 as UCS-1 (“Latin-1”) 0xC3A4.  Which charset has a Ă (U+0102) in position 0xC3?

I confirmed from Ludwig that he didn’t use kramdown-rfc on his draft (if he did, that would have left a correct cache file), and Python 2.7 (not Python 3); my tentative diagnosis is that xml2rfc 2.5.1 somehow leaves a bad download in the cache.  Would be interesting to check that with a current xml2rfc.

And, yes, and somewhat sadly, I agree with the position that having the ä in the bibxml is somewhat premature, but let’s locate the problem while it is still there.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 1 Feb 2019 07:12:34 +0100

On Feb 1, 2019, at 07:06, Carsten Bormann <[email-masked]> wrote:
> 
> Which charset has a Ă (U+0102) in position 0xC3?

I just checked Latin-2, and yes that has the Ă (U+0102) in position 0xC3.
So the most likely hypothesis seems to be: something is mis-interpreting the downloaded UTF-8 as Latin-2 and then storing it as UTF-8.
Wot?

Grüße, Carsten

https://de.wikipedia.org/wiki/Ă

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 1 Feb 2019 07:32:15 +0100

> "pip install xml2rfc" gives me version 2.5.1 on Ubuntu 18.04.

pip3 install --upgrade xml2rfc

(Or pip, if you want to use the default Python, which may be old.)

Grüße, Carsten