Subject: [xml2rfc] Failed decoding the uploaded file

From: Robert Moskowitz <[email-masked]>
Date: Mon, 14 Dec 2020 19:49:01 -0500

But I am getting the follow error from the txt file:

Failed decoding the uploaded file: "'ascii' codec can't decode byte 0xc3 
in position 135051: ordinal not in range(128)"

How do I find what character is the problem?  Could it be:

R., Håstad

This is in a reference produced via a DOI reference.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Dec 2020 01:55:01 +0100

> But I am getting the follow error from the txt file:

Don’t send the txt file with a v3 source.

(Another thing suggested by the submission interface that causes problems.)

Grüße, Carsten

---

From: Robert Moskowitz <[email-masked]>
Date: Mon, 14 Dec 2020 22:58:56 -0500

> (Another thing suggested by the submission interface that causes problems.)

I did not include the txt, only the xml and I was right that the problem 
was the author's name in the reference.

locally I get
R., Håstad

the submit generates:

R., HÃ¥stad

Not good. can we get this fixed and I will have to rev the draft just 
for this one error?

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 05:04:12 +0100

> Not good. can we get this fixed and I will have to rev the draft just

You have a mismatch between the declared (or implicit) character
encoding of the XML, and the actual data you sent.

I recommend running the file through a local XML parser (such as a
browser) before submitting.

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Mon, 14 Dec 2020 23:18:50 -0500

> browser) before submitting.

The xml comes via an include:

      <?rfc 
include="https://xml2rfc.tools.ietf.org/public/rfc/bibxml7/reference.DOI.10.1007/978-3-540-28628-8_30.xml"?>

It should not be that this works locally, but not with the submission tool.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 05:26:23 +0100

> It should not be that this works locally, but not with the submission tool.

The include looks fine to me.

I'll repeat what I keep saying when problems with included references
(network, encoding, whatnot) come up: don't include, just download and
paste in. That way, the XML is under your control and self-contained.

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Tue, 15 Dec 2020 00:03:15 -0500

> The include looks fine to me.

It looks fine, but the online submission tool processes it wrong.

> paste in. That way, the XML is under your control and self-contained.

It sounds  reasonable, but:

How do you know there will be a problem until you figure out what caused 
the failure.  Like that IEEE DOI include I point to in another message.  
Took me a few hours to back out a lot of changes and reapply and run 
xml2rfc with each one until I hit on the change that actually generated 
the error.

Supposedly an include like this one is an outside source, so it would 
seem we want to keep the flexibility of dynamically grabbing this 
outside source at the last step possible.

Sigh.  LIfe was never meant to be easy.  Signing off for the night.

>
> Best regards, Julian
>
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Dec 2020 08:20:36 +0100

On 2020-12-15, at 05:18, Robert Moskowitz <[email-masked]> wrote:
>      <?rfc include="https://xml2rfc.tools.ietf.org/public/rfc/bibxml7/reference.DOI.10.1007/978-3-540-28628-8_30.xml"?>
> 
> 

The XML at draft-ietf-hip-dex-22.xml starts with:

<?xml version='1.0' encoding='utf-8’?>

Is that how you actually submitted it?

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 09:21:38 +0100

> Is that how you actually submitted it?

Why would that matter? (unless there's a bug in xml2rfc...)

FWIW, it *seems* to me that all is well, except that the text file

   https://www.ietf.org/archive/id/draft-ietf-hip-dex-22.txt

is served with incorrect HTTP header fields (as Martin correctly pointed
out), while

   https://www.ietf.org/id/draft-ietf-hip-dex-22.txt

is ok.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Dec 2020 09:29:12 +0100

> Why would that matter? (unless there's a bug in xml2rfc...)

I have no idea how <?rfc include… is handled, so this was a shot in the dark.
>  https://www.ietf.org/id/draft-ietf-hip-dex-22.txt

Yep.  Should have checked with my local copy, which is also OK (rsync doesn’t break UTF-8).  Since the incorrect HTTP header is entirely an ietf.org infrastructure problem; I have CCed tools-discuss.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 09:34:28 +0100

> Yep.  Should have checked with my local copy, which is also OK (rsync doesn’t break UTF-8).  Since the incorrect HTTP header is entirely an ietf.org infrastructure problem; I have CCed tools-discus…

There's already a ticket (should have repeated that here):
<https://trac.tools.ietf.org/tools/ietfdb/ticket/3151>

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Tue, 15 Dec 2020 06:28:18 -0500

> Is that how you actually submitted it?

I believe ALL my draft xml start with:

<?xml version='1.0' encoding='utf-8'?>

That is what is in the template I have here.

Am I behind the times?   :)

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 13:08:36 +0100

Am 15.12.2020 um 12:28 schrieb Robert Moskowitz:
> ...
> ...

No, that's fine (assuming that any non-ASCII characters in your document
are actually encoded in UTF-8).

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Tue, 15 Dec 2020 09:30:17 -0500

> are actually encoded in UTF-8).

And how am I to know that with all this masked thanks to 'working' gui 
tools.

I suppose I need something that shows the binary content of the xml and 
then knowledge on how to read it.  Anything as nice as dumpasn.1 in openSSL?

:)

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 15 Dec 2020 15:51:09 +0100

> Not good. can we get this fixed and I will have to rev the draft just 

I suspect the file on disk on the ietf server is correct, and the issue is
that the mime-type indicated by the server is incorrect.

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Dec 2020 15:59:54 +0100

On 2020-12-15, at 15:30, Robert Moskowitz <[email-masked]> wrote:
> 
> And how am I to know that with all this masked thanks to 'working' gui tools.

What you need to know is that no tool is trying to “help” you.

Today, everything is in UTF-8.
Our systems are still littered with little modules that try to convert things that no longer exist into UTF-8 (and possibly even back).

When I import a csv into MOff, it interprets the data as MacRoman.  
An encoding that has been obsolete for at least 22 years.
(Obviously, Libreoffice gets this right.)

HTTP defaults to ISO-8859-1.  Well, that probably was as far as things could go in the early 1990s.  But, today, server configs need to default to UTF-8 (i.e., always send content-type) to override this no longer useful defaults.

And that is the problem here: The data file in the I-D repo is correct, but the server doesn’t say it is UTF-8 and so the “helpful” default of HTTP takes over.
Only in /archive, because that is the part that doesn’t have the necessary “here, too” UTF-8 configuration.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2020 16:17:28 +0100

> What you need to know is that no tool is trying to “help” you.

Just use Notepad :-)

> HTTP defaults to ISO-8859-1.  Well, that probably was as far as things could go in the early 1990s.  But, today, server configs need to default to UTF-8 (i.e., always send content-type) to override…

That default has been removed six years ago.

The server is sending *no* content-type at all, plus
"x-content-type-options: nosniff", so the UA isn't told what it is, but
*is* told not to sniff. Sigh.

Best regards, Julian

---

From: Robert Moskowitz <[email-masked]>
Date: Tue, 15 Dec 2020 10:18:20 -0500

> that the mime-type indicated by the server is incorrect.

I only submitted the xml, not the txt that my local xml2rfc built. It 
was that local txt that the upload barfed on with the illegal 
character.  What you have only is the base xml and the txt built by the 
online submission tool.