Subject: [xml2rfc] Submission problem due to xmltrfc not putting asciiSurname into .txt output

From: "David R. Oran" <[email-masked]>
Date: Thu, 02 Apr 2020 10:10:04 -0400

I tried to submit a V3 draft today by uploading the .txt, and got:

Failed decoding the uploaded file: "'ascii' codec can't decode byte 0xc3 
in position 69240: ordinal not in range(128)"

this is in an <author> element which in my .xml file is:

<author surname="Augé" initials="J." fullname="Jordan Augé" 
asciiSurname="Auge" asciiFullname="Jordan Auge"/>

But the .txt output of xml2rfc generates:

    [Auge2018] Augé, J., Carofiglio, G., Grassi, G., Muscariello, L.,
               Pau, G., and X. Zeng, "MAP-Me: Managing Anchor-Less
               Producer Mobility in Content-Centric Networks, in IEEE
               Transactions on Network, Volume 15, Issue 2",
               DOI 10.1109/TNSM.2018.2796720, June 2018,
               <https://ieeexplore.ieee.org/document/8267132>.

Am I missing something obvious or is this a bug?

DaveO

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 2 Apr 2020 16:20:38 +0200

Hi David,

> Am I missing something obvious or is this a bug?

This is probably caused by the file being identified as having ascii
encoding by the uploading browser.  There are currently two workarounds:
Upload the XML and let the submission tool do the conversion, or use the
--bom switch to generate an initial BOM in the file, which will let the
browser identify the file as UTF-8, which should result in the right
handling in the submission tool.

Best regards,

	Henrik

---

From: "David R. Oran" <[email-masked]>
Date: Thu, 02 Apr 2020 10:35:58 -0400

On 2 Apr 2020, at 10:20, Henrik Levkowetz wrote:

> Upload the XML and let the submission tool do the conversion,
I can’t - it has included files and there seems to be no way to upload dependencies. Or am I missing something?

> or use the
Ah, that worked. Didn’t know about that option. Good to know for future reference.

DaveO

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 2 Apr 2020 16:57:35 +0200

On 2020-04-02 16:35, David R. Oran wrote:
> I can’t - it has included files and there seems to be no way to
> upload dependencies. Or am I missing something?

The intention is that you should be able to generate an uploadable file
by using the --expand switch; but while that processes XIncludes, it
does not pull in external artwork at the moment, and it should.

You can create an uploadable file using --prep, but that will also do
a lot of other things, and result in a file which is not suited for
later editing, due to the added verbiage.

> Ah, that worked. Didn’t know about that option. Good to know for

GOod :-)

Best regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 Apr 2020 17:03:05 +0200

> Ah, that worked. Didn’t know about that option. Good to know for future reference.

FYI: BOMs in UTF-8 are seriously deprecated.
We are actively trying to get rid of BOM infections, and I’d rather not see this being used as a workaround for anything.
In particular not in the IETF.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 Apr 2020 17:05:40 +0200

> later editing, due to the added verbiage.

This works well, and is in active use (kdrfc -p, in kramdown-rfc).

(There is no point in editing uploaded XML for I-Ds, unless we are talking about final versions going to the RFC editor, where special arrangements can easily be made.)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Thu, 2 Apr 2020 14:09:21 -0500

On Thu, Apr 02, 2020 at 05:03:05PM +0200, Carsten Bormann wrote:
> We are actively trying to get rid of BOM infections, and I’d rather
> not see this being used as a workaround for anything.  In particular
> not in the IETF.

+1.

The submission tool should require and assume UTF-8, or at most have a
checkbox for ASCII vs UTF-8.  In any case, there should be no codeset
detection in the submission tool.

Nico
--

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 2 Apr 2020 21:13:57 +0200

> detection in the submission tool.

There isn't.  The submission tool uses the charset provided by the browser
at upload time.

This fails if the browser says that an UTF-8 file is ASCII, which was the
case here.

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 2 Apr 2020 21:19:58 +0200

> On 2020-04-02, at 21:13, Henrik Levkowetz <[email-masked]> wrote:
> 
> 
> 

Why?
Browsers do get these things wrong — how would they even know?  

> This fails if the browser says that an UTF-8 file is ASCII, which was the

Why?
It is extremely easy to sniff whether a file is UTF-8 or not, and rather more reliable than what browsers say.  Now if it is not, the submission tool might want to translate koi-8r into UTF-8 if the browser says that is the encoding, but that sounds marginal already.

ASCII is irrelevant.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 2 Apr 2020 21:31:13 +0200

> Browsers do get these things wrong — how would they even know?

Below you say that it's extremely easy to sniff whether a file is UTF-8 or
not, and here you ask how would the browser even know.  I see a contradiction.

The upload comes with a mime type and charset -- why should the submission
tool _not_ use those?

> It is extremely easy to sniff whether a file is UTF-8 or not, and
> rather more reliable than what browsers say.

Why?  With browsers having a userbase of hundreds of millions, why should
we assume that our custom code will do better?

	Henrik

> Now if it is not, the
> submission tool might want to translate koi-8r into UTF-8 if the
> browser says that is the encoding, but that sounds marginal already.
> ASCII is irrelevant.
> 
> Grüße, Carsten
> 
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 07:06:21 +0200

> The upload comes with a mime type and charset -- why should the submission

Because they are mostly wrong.

This thread started with an observation that the charset provided by the browser was wrong.
Previously, we have had discussions about markdown files being submitted as application/octet-stream.

The way browsers are set up, they are trying to use OS hints for assigning a media type and, possibly, a charset.  However, these OS hints are not designed in a way that they can be relied upon.

So in legacy environments with multiple charsets, server-side libraries such as Microsoft's MLang are being used to sniff the uploads for their charset.  I’m not sure we are in such an environment any more in 2020, but it is still easy to check for UTF-8-ness and fall back to a supplied charset if that is not the case.

It is never productive to believe a browser’s supplied media type that an uploaded file is ASCII and then fail if it isn’t.

(XML files, of course, carry their charset in them, so no guessing is required, just checking.)

> we assume that our custom code will do better?

Because the browsers don’t try, and we could.

I apologize that my Python-fu is not very strong, but the code looks like these two lines (plus test harness):

#!/usr/bin/env ruby
data = ARGF.binmode.read
text = data.force_encoding(Encoding::UTF_8)    # Assume UTF-8
if text.valid_encoding?                        # and check
  warn "Valid UTF-8"
else
  # ... guess charset from uploaded media type, maybe
  # or simply scrub:
  text = text.scrub("*")
  warn "Text wasn't valid UTF-8, scrubbed"
end
puts text

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 3 Apr 2020 08:57:06 +0200

> This thread started with an observation that the charset provided by the browser was wrong.

Do we have any details/traces? Maybe it's worth filing browser issues.

 > ...

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 3 Apr 2020 13:08:53 +0200

> Because they are mostly wrong.

So far we've seen 2 cases (I think) of being wrong about the charset, out
of a large number of uploads.  I think you're overstating things when you
say 'mostly wrong'.

(Nevertheless, since I have already called the UTF-8 workaround for just
that , a workaround, I've already started thinking of how to do better.)

> This thread started with an observation that the charset provided by
> the browser was wrong. Previously, we have had discussions about
> markdown files being submitted as application/octet-stream.
> 
> The way browsers are set up, they are trying to use OS hints for
> assigning a media type and, possibly, a charset. However, these OS
> hints are not designed in a way that they can be relied upon.

I've found 'libmagic' highly reliable on Debian Linux and BSD, less
so on OpenSUSE Linux.

> So in legacy environments with multiple charsets, server-side
> libraries such as Microsoft's MLang are being used to sniff the
> uploads for their charset. I’m not sure we are in such an environment
> any more in 2020, but it is still easy to check for UTF-8-ness and
> fall back to a supplied charset if that is not the case.
> 
> It is never productive to believe a browser’s supplied media type
> that an uploaded file is ASCII and then fail if it isn’t.

Agreed; we should not fail (which is why the workaround is a workaround,
not a solution).

> Because the browsers don’t try, and we could.

Of course we could.  But the browsers also should.  If we start out writing
code with the assumption that everything we haven't coded ourselves is
faulty, we're bound to re-invent the wheel and never have time to build
a better carriage.  I disagree with your implied assertion that we should
not first try to use the tools and libs around us.

> I apologize that my Python-fu is not very strong, but the code looks
> like these two lines (plus test harness):

How to do something sensible isn't hard, and isn't really the point here;
we already have code to try to deal with similar things a few places, for
instance when reading files with unknown encoding off disk:

https://trac.tools.ietf.org/tools/ietfdb/browser/trunk/ietf/group/utils.py#L59

I think the point here is that we should not try to replace existing libs
and tools that already exist with the assumption that they are broken before
we even try to work with them.

Of course we should do better when we find places where things fall down;
but your underlying premise when you say "Why?", "Why?" seems to be that
we should have written more code from the start, rather than try to rely
on existing code and fix at failure.

	Henrik

---

From: David Oran <[email-masked]>
Date: Fri, 3 Apr 2020 07:48:38 -0400

Warning: naïf alert

Since it was my .txt file generated from xml2rfc on Mac OS that generated the original bug report, and HenriK says this is not a common problem for uploads (perhaps because very few people are doing V3 and with UTF-8 stated in the file yet), perhaps we could nip the problem in the bud so to speak.

Could we not just make sure that on MacOS, xml2rfc outputs a .txt file marked correctly as UTF-8 in the OS file system?

___________________________
iDevice - please excuse typos.

> On Apr 3, 2020, at 7:26 AM, Henrik Levkowetz <[email-masked]> wrote:
> 
> 
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 3 Apr 2020 14:18:47 +0200

Hi David,

On 2020-04-03 13:48, David Oran wrote:
> Warning: naïf alert
> 
> Since it was my .txt file generated from xml2rfc on Mac OS that
> generated the original bug report, and HenriK says this is not a
> common problem for uploads (perhaps because very few people are doing
> V3 and with UTF-8 stated in the file yet), perhaps we could nip the
> problem in the bud so to speak.
> 
> Could we not just make sure that on MacOS, xml2rfc outputs a .txt
> file marked correctly as UTF-8 in the OS file system?

Ah!  That's a thought.

We could, and I'll add that in the next release.  I think, however, that
we'll still need to do better on the server side.  I'll look more at that
once the work on improved interim support on the 'Upcoming Meetings' and
related pages is in place.

Best regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 14:46:04 +0200

> of a large number of uploads.  I think you're overstating things when you

Well, I’ve seen hundreds of cases before.
I do not think I am overstating this.
Of course, the occurrences of tend to get fewer, as systems get cleaner (i.e., UTF-8 only).

> I've found 'libmagic' highly reliable on Debian Linux and BSD, less

OK, that is *not* using OS hints (which in these cases are limited to filename “extensions” and locale information).

Libmagic on the server side would be mostly OK, with the occasional misattribution(*) to be expected.
On the client side: Which browsers use libmagic to derive content types (media types + charsets) for uploads?

> Agreed; we should not fail (which is why the workaround is a workaround,

This is the Internet, which is workaround stacked on workaround.
I am not aware of a solution.

> faulty,

That would be futile.
But on the other hand, believing something works because it “should work” is not productive either.
In particular in the presence of decades of experience to the contrary.

(And, yes, when people are moving towards cleaning up things, I’m all for following along.
I’m not aware of that happening in the space of media type sniffing.)

> on existing code and fix at failure.

My “why” was trying to find out if there is a design rationale I’m oblivious to.

Grüße, Carsten

(*) `file -I` on a random collection of "text files” (by filename) on my laptop:
   1 application/json; charset=us-ascii
   4 application/octet-stream; charset=binary
   2 message/rfc822; charset=us-ascii
   1 text/html; charset=iso-8859-1
   2 text/html; charset=utf-8
 145 text/plain; charset=us-ascii
  99 text/plain; charset=utf-8
   1 text/x-c; charset=us-ascii
   1 text/x-diff; charset=us-ascii
   1 text/xml; charset=us-ascii

Using `file --mime-encoding` to get the charset part only:

   4 binary
   1 iso-8859-1
 151 us-ascii
 101 utf-8

The file identified as iso-8859-1 is windows-1252 actually.

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 15:01:14 +0200

> Could we not just make sure that on MacOS, xml2rfc outputs a .txt file marked correctly as UTF-8 in the OS file system?

Hi Dave,

unfortunately, macOS no longer has metadata in the file system about the character encoding used for a text file.  ISTR macOS classic had this, but when OS X came out, Apple decided to follow the lead of UNIX here, where a file is just a string of bytes.  (Since xml2rfc is essentially using the UNIX API internally, it would not be clear now to exercise a metadata system even if macOS still had one.)

[It gets more interesting with filenames as opposed to file content, but going into that detail would be a whole paper on its own.]

Grüße, Carsten

---

From: "David R. Oran" <[email-masked]>
Date: Fri, 03 Apr 2020 09:04:38 -0400

On 3 Apr 2020, at 9:01, Carsten Bormann wrote:

> unfortunately, macOS no longer has metadata in the file system about 
> the character encoding used for a text file.
Are you sure?
Look at 
https://developer.apple.com/documentation/mobilecoreservices/uttype/uti_text_types

which has
let kUTTypeUTF8PlainText: CFString The type identifier for plain text in 
a UTF-8 encoding.

> ISTR macOS classic had this, but when OS X came out, Apple decided to 
> follow the lead of UNIX here, where a file is just a string of bytes.  
> (Since xml2rfc is essentially using the UNIX API internally, it would 
> not be clear now to exercise a metadata system even if macOS still had 
> one.)
>

> [It gets more interesting with filenames as opposed to file content, 
> but going into that detail would be a whole paper on its own.]
>
> Grüße, Carsten

DaveO

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 3 Apr 2020 15:11:54 +0200

> unfortunately, macOS no longer has metadata in the file system about the character encoding used for a text file.  ISTR macOS classic had this, but when OS X came out, Apple decided to follow the l…

Modern OS-X instead has extended attributes, and there are attributes defined
(and not exclusive to OS-X) for 'user.mime_type', 'user.charset', and more.

There's a Python lib 'xattr' that will let us set these, as appropriate.

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 3 Apr 2020 15:17:56 +0200

On 2020-04-03 14:46, Carsten Bormann wrote:
> OK, that is *not* using OS hints (which in these cases are limited to
> filename “extensions” and locale information).

So your concept of OS hints did not match my concept of OS hints.

> On the client side: Which browsers use libmagic to derive content types (media types + charsets) for uploads?

I have no idea.

> My “why” was trying to find out if there is a design rationale I’m oblivious to.

In this case, there's an attempt at sanity checking uploads, catching things
like binary and XML files uploaded by mistake or bad intent as if they were
text, and when text, trying to accomidate various upload encodings.  The
current code has worked well for a number of years.

Yes, we've now found a spot where we fall down.  Yes, we're going to fix it.

But there's just a few people (like ~1.5 plus sprint participants) trying to
maintain a huge codebase, with urgent requests coming in quite often, and
trying to make everything work better every day, while still adding features
and upgrading to new releases of libs and frameworks.

Please be gentle.

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 16:54:23 +0200

On 2020-04-03, at 15:04, David R. Oran <[email-masked]> wrote:
> 
>> unfortunately, macOS no longer has metadata in the file system about the character encoding used for a text file.
> Are you sure?
> Look at https://developer.apple.com/documentation/mobilecoreservices/uttype/uti_text_types
> 
> which has
> let kUTTypeUTF8PlainText: CFString The type identifier for plain text in a UTF-8 encoding.

“Uniform Type Identifiers” (UTIs) are mostly there to identify pasteboard content — clearly, you need to know the charset when copying and pasting data between applications, some of which may be stuck in legacy.  They are also used in a macOS structured metadata thing called “Bundle”.  Within a program, they may be useful as a summary of what you have learned from filename extensions and the OSType file system metadata that was deprecated in 2003 with OS X Panther.  (I’m not sure how much remains of OSType, but I’m not aware that ever contained charset information.)

Henrik seems to have dug out an ~2005 initiative from Lennart Poettering to use file system xattr for media types and charsets.  I’m not aware of any traction this has had, except for the mod-mime-xattr Apache module (the only recent information I can find about is from some bitrot this module has accreted since it last was actively worked on at the start of the previous decade).  The specification lives on in https://www.freedesktop.org/wiki/CommonExtendedAttributes/ which essentially refers to mod-mime-xattr and https://specifications.freedesktop.org/shared-mime-info-spec/latest/ar01s02.html#idm46096015332608 .  So I have to agree with Henrik that "there are attributes defined”, but it would be a bit of an overstatement to say that macOS “has” these attributes — yes, you can stash them there, but they will be ignored.

I am not a fan of the current situation.  It becomes bearable because these are all problems of the past; they simply no longer occur on modern UTF-8-clean systems.  The reason this took 20 years to clear up was because people were trying to be “helpful” towards deprecated stuff, and thus prolonged the problem for a near-eternity [sound familiar from other transition situations?].  Here, people have given up “being helpful” for a while now, and the situation is becoming better.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 3 Apr 2020 17:10:13 +0200

> Henrik seems to have dug out an ~2005 initiative from Lennart Poettering to use file system xattr for media types and charsets.  I’m not aware of any traction this has had, except for the mod-mime-…

This is a rather severe misrepresentation of extended file attributes under
OS X.  Extended file attributes are in active use by the operating system
and applications, and help OS X do things such as displaying directory
trees as documents where appropriate, and quarantining downloaded documents.

There's a bit more information here, although not a lot:

  https://en.wikipedia.org/wiki/Extended_file_attributes#macOS

This also gives a bit of information:

https://eclecticlight.co/2017/08/14/show-me-your-metadata-extended-attributes-in-macos-sierra/

> I am not a fan of the current situation. It becomes bearable because
> these are all problems of the past; they simply no longer occur on
> modern UTF-8-clean systems. The reason this took 20 years to clear up
> was because people were trying to be “helpful” towards deprecated
> stuff, and thus prolonged the problem for a near-eternity [sound
> familiar from other transition situations?]. Here, people have given
> up “being helpful” for a while now, and the situation is becoming
> better.

I think we're all happy that we're moving towards UTF-8 support becoming
more ubiquitous.

	Henrik

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 Apr 2020 10:22:23 -0500

> The upload comes with a mime type and charset -- why should the submission

We should require UTF-8.  Or we should require ASCII | UTF-8.

> we assume that our custom code will do better?

Requiring ASCII | UTF-8 would let us not care what the browser thinks.

Certainly you could reject submissions where the browser thinks the text
is in some codeset other than ASCII or UTF-8.

And you should reject submissions where the contents is clearly not
ASCII and not UTF-8.

We have no need to support any other codeset.  Not ISO-8859-*, not
SHIFT-JIS, nothing.

UTF-8 or bust.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 17:49:23 +0200

> https://eclecticlight.co/2017/08/14/show-me-your-metadata-extended-attributes-in-macos-sierra/

Certainly macOS uses xattr.

I was talking about “user.charset”, which essentially does not exist in macOS.

There also is a “com.apple.TextEncoding”, which is used by applications such as TextEdit and I think approximately two more.  But that is macOS-specific.  You can set that to "utf-8;134217984” [sic] and TextEdit and BBedit will know that the file is UTF-8.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 Apr 2020 14:56:22 -0500

> I was talking about “user.charset”, which essentially does not exist in macOS.

By which you mean "it's not universally used".

> There also is a “com.apple.TextEncoding”, which is used by
> applications such as TextEdit and I think approximately two more.  But
> that is macOS-specific.  You can set that to "utf-8;134217984” [sic]
> and TextEdit and BBedit will know that the file is UTF-8.

But the right thing is to just use UTF-8.  Use UTF-8 locales only.

Perform codeset conversions only as needed for interop, but default to
UTF-8.

No tasting of encoding.  Do not trust, just validate, and do not taste.

The I-D submission tool should loudly advertise that the .xml file MUST
be in UTF-8 and should reject submissions with non-UTF-8.  No tasting of
the contents' encoding should be perfomed.  If a user submits
valid-looking garbage, then they'll get garbage and reviewers will
notice.

Another thing that could be done is require no alternate format
submission when submitting XML: just apply xml2rfc on the server side.

Cheers,

Nico
--

---

From: "John Levine" <[email-masked]>
Date: 3 Apr 2020 15:57:36 -0400

>Do we have any details/traces? Maybe it's worth filing browser issues.

My guess is that they would say the take the file type from the
filename, that's the only clue they have and if we don't like it we
should tell our users to go edit some obscure MIME type file.

But here's a question: if we pretended that files that were tagged as
ASCII are actually UTF-8, what bad things could happen?  If they
really are ASCII, that should be OK.  If they're really UTF-8, that
should be OK.  I'd only expect a problem if they're some in other
character set but these days that doesn't seem particularly likely.

R's,
John

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 22:12:24 +0200

> By which you mean "it's not universally used".

I really meant “it’s universally not used”.

> But the right thing is to just use UTF-8.  Use UTF-8 locales only.

Yes!  Up to a few years you would have been met with “my IT people set the locale on my laptop to Armenian ArmSCII and locked that down so I can’t do UTF-8”.  This is no longer a problem (*).

> Perform codeset conversions only as needed for interop, but default to

I would be even more radical here.

> No tasting of encoding.  Do not trust, just validate, and do not taste.

Works for me.

> valid-looking garbage, then they'll get garbage and reviewers will

(One hopes :-)

> submission when submitting XML: just apply xml2rfc on the server side.

I think we are slowly getting to the place where this may be a valid option for most documents that go to RFC.  But it remains important to be able to type up an I-D in Wordperfect, and be able to submit it (as long as it is UTF-8, which it may be by way of being ASCII), so I’m not sure we can go the full way here.

Grüße, Carsten

(*) At least I believe so; I have not talked to all Armenian IETFers to verify this.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 4 Apr 2020 09:13:22 +1300

John,
> character set but these days that doesn't seem particularly likely.

If you mean, idnits should not barf on valid UTF-8 for the purpose
of I-D submission, I wouldn't see that as a disaster, but the problem
still needs to be solved before the RPC posts RFCnnnn.txt.

However, we would need to get used to things like UniversitÃ¤t Bremen.

   Brian

---

From: "John R Levine" <[email-masked]>
Date: 3 Apr 2020 16:27:27 -0400

> still needs to be solved before the RPC posts RFCnnnn.txt.

We allow all sorts of crud in I-D's that we fix before publication, of 
which rogue UTF-8 is a very minor example. We have policies about where 
and how non-ASCII text can appear in RFCs. That's what RFC 7997 is about.

We need to update 7997, but that doesn't matter for this question.

(See https://github.com/rfc-format/draft-iab-rfc-nonascii-bis if 
interested.)

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Apr 2020 22:28:18 +0200

Why?

Since everything is UTF-8, there won’t be this kind of Mojibake any more…
(Yeah, I know, language tags, RTL, …)

Grüße, Carsten

(Of course, I *am* used to that.  From Websites done by the, er, less competent or not updated since 1999.  The people doing the tools for IETF are way better than that.)

---

From: Nico Williams <[email-masked]>
Date: Fri, 3 Apr 2020 16:03:17 -0500

> I really meant “it’s universally not used”.

:)

> I would be even more radical here.

Well, I mean, if people have archived documents in not-Unicode that they
need converted to Unicode as needed -- that's OK.

And clearly UTF-16 -horrible as it is- has to be used in some contexts.
(But let us hope it dies eventually.)

> I think we are slowly getting to the place where this may be a valid
> option for most documents that go to RFC.  But it remains important to
> be able to type up an I-D in Wordperfect, and be able to submit it (as
> long as it is UTF-8, which it may be by way of being ASCII), so I’m
> not sure we can go the full way here.

Note that I wrote "no alernate format .. when submitting XML" -- that
does not exclude alternate formats when... not submitting XML!

If you have XML, that's all that should be used, and might as well have
that be all that is submitted too.

Nico
--

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 6 Apr 2020 10:07:36 +0900

> The way browsers are set up, they are trying to use OS hints for assigning a media type and, possibly, a charset.  However, these OS hints are not designed in a way that they can be relied upon.

Explaining things a bit differently, browsers have to send charset 
information based only on the information that they send it to *some* 
Web site. They also have to deal with historical issues such as "This 
browser used to send this charset information in this situation; the Web 
site is written with that assumption, so if the browser suddenly changes 
this (even if the new information is more often correct) the Web site 
may look bad, and the browser will be blamed."

We have a big advantage on the server side: We know that what we get 
should be UTF-8 (or ASCII which is UTF-8 anyway) text. The browser 
doesn't have this knowledge about our application. That's why it makes 
sense to deal with this (to some extent) on the server side, even if in 
an ideal world, we wouldn't have to.

Regards,   Martin.

---

From: Nico Williams <[email-masked]>
Date: Sun, 5 Apr 2020 21:01:39 -0500

On Mon, Apr 06, 2020 at 10:07:36AM +0900, Martin J. Dürst wrote:
> browser used to send this charset information in this situation; the
> Web site is written with that assumption, so if the browser suddenly
> changes this (even if the new information is more often correct) the
> Web site may look bad, and the browser will be blamed."
> 
> sense to deal with this (to some extent) on the server side, even if
> in an ideal world, we wouldn't have to.

+1.

For .xml files we should require UTF-8.  Ditto for .txt files, but if
.xml is submitted, then the .txt should be generated by the server.

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Mon, 6 Apr 2020 06:26:04 +0200

>> in an ideal world, we wouldn't have to.

Maybe.

> For .xml files we should require UTF-8.  Ditto for .txt files, but if

No. Why? What problem are you solving here?

> ..xml is submitted, then the .txt should be generated by the server.

Optionally. Right now there are command line options that affect the
output, and it would be necessary to support them in the UI. (Or better,
get a way to get rid of them).

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 Apr 2020 00:07:12 -0500

> No. Why? What problem are you solving here?

Oh sure, the XML can tell us its codeset.  If it simplifies things to
require UTF-8, I'd rather do that.  Certainly I don't want the codeset
tasted.  Who needs to edit I-D source XML in not-UTF-8?  Can --exp or a
new option transcode to UTF-8 for submission?

> get a way to get rid of them).

 * Don't use --raw for submission.  Unpaginated text is great for
   reviewing changes while editing so please don't remove it, but it
   shouldn't be accepted for submission).

 * -u should be required for submission if you're going to submit .txt.

 * Alternate DTDs... is there a use for that?  anyways, the xml tells us
   which DTD, so the server can still generate the right .txt.

 * Any options I missed?

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Mon, 6 Apr 2020 10:13:42 +0200

>     which DTD, so the server can still generate the right .txt.

Date formats...

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 Apr 2020 10:17:48 +0200

On 2020-04-06, at 10:13, Julian Reschke <[email-masked]> wrote:
> Date formats…

Let’s introduce a few PIs for controlling that!

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 Apr 2020 10:00:08 -0500

> On 06.04.2020 07:07, Nico Williams wrote:

    -D DATE, --date=DATE      run as if the date is DATE (format: yyyy-mm-dd)

That's an option to set a date.

.xml should be submitted in "expanded" form, with all relevant options
from the xml2rfc accounted for, and in UTF-8.  Why require UTF-8?
Simplicity, for one.  We'll be producing UTF-8 anyways, so why require
server-side codeset conversions?

Nico
--

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 6 Apr 2020 13:47:47 -0400

On 4/6/20 11:00 AM, Nico Williams wrote:
> ..xml should be submitted in "expanded" form, with all relevant options

If there are particular rules for the xml that is submitted, then please 
put a reference to them on <https://datatracker.ietf.org/submit/>.

Is it sufficient to use <https://xml2rfc.tools.ietf.org/> with the 
"expanded XML " output option selected?

And what do you mean by "submitted"? Do you mean with any draft 
submitted using <https://datatracker.ietf.org/submit/>? Or do you only 
mean when submitting a final pre-rfc version intended for the editor? Or 
what?

	Thanks,
	Paul

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 Apr 2020 12:52:34 -0500

On Mon, Apr 06, 2020 at 01:47:47PM -0400, Paul Kyzivat wrote:
> If there are particular rules for the xml that is submitted, then please put
> a reference to them on <https://datatracker.ietf.org/submit/>.

I'm advocating for new rules.  I don't get to set them.

> Is it sufficient to use <https://xml2rfc.tools.ietf.org/> with the "expanded
> XML " output option selected?

I'm saying that it should be.

> And what do you mean by "submitted"? Do you mean with any draft submitted
> using <https://datatracker.ietf.org/submit/>? Or do you only mean when
> submitting a final pre-rfc version intended for the editor? Or what?

The former.

Maybe xml2rfc could have a --submit option.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 Apr 2020 19:58:18 +0200

On 2020-04-06, at 19:47, Paul Kyzivat <[email-masked]> wrote:
> 
> And what do you mean by "submitted"? Do you mean with any draft submitted using <https://datatracker.ietf.org/submit/>? Or do you only mean when submitting a final pre-rfc version intended for the editor? Or what?

Datatracker doesn’t know that.  (It can’t, because approval happens after submission.)
If the RFC editor needs something special, they’ll make it or discuss it with the authors.

This discussion is *not* about RFCs, it is about XML2RFC’s dominating usage: Authoring I-Ds.
The name of the tool may obscure that, but it is really what we need to optimize for.

Grüße, Carsten

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 6 Apr 2020 14:39:02 -0400

> The name of the tool may obscure that, but it is really what we need to optimize for.

OK. I just wanted to be sure.

Is the need for expanded form to ensure that there are no dependencies 
on the local environment of the submitter? Or are there other reasons as 
well?

(If I'm successfully using xml2rfc on the tools web page then there 
clearly aren't any dependencies on my local environment.)

My preference with v2 was to submit the source xml along with the txt of 
each version, so that it is available to anybody else who might take up 
the pen on the document. Submitting the expanded form potentially 
divorces it from the *source*. Typically my use of includes in the xml 
has been limited to references, so dependencies were pretty stable.

Requiring the expanded form be submitted means that there is no reliable 
and consistent way for somebody to obtain the *source* form. (E.g., 
somebody who wants it as the starting point for a bis.)

I think we need a way to save both.

	Thanks,
	Paul

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 Apr 2020 20:56:55 +0200

> 
> Is the need for expanded form to ensure that there are no dependencies on the local environment of the submitter? Or are there other reasons as well?

The discussion of this started with svg files linked from the XML.
These obviously need to be included for a single-file XML submission.

> (If I'm successfully using xml2rfc on the tools web page then there clearly aren't any dependencies on my local environment.)

Right.  One of the problems we sometime have is that the velocity of the tools web page is divorced from the velocity on datatracker.

> My preference with v2 was to submit the source xml along with the txt of each version, so that it is available to anybody else who might take up the pen on the document.

I did that for a while, but then most documents nowadays live on github, which is much sourcier.

> Submitting the expanded form potentially divorces it from the *source*. Typically my use of includes in the xml has been limited to references, so dependencies were pretty stable.

In the current system, most authors cannot submit their source anyway, so one more or less processing step does not make a big difference.

> Requiring the expanded form be submitted means that there is no reliable and consistent way for somebody to obtain the *source* form. (E.g., somebody who wants it as the starting point for a bis.)

Point to the github…

(If you write a bis after 10 years, the source is probably already much less useful.
Henrik even has a tool that upconverts .txt to .xml…)

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 Apr 2020 14:44:36 -0500

On Mon, Apr 06, 2020 at 08:56:55PM +0200, Carsten Bormann wrote:
> > Requiring the expanded form be submitted means that there is no
> > reliable and consistent way for somebody to obtain the *source*
> > form. (E.g., somebody who wants it as the starting point for a bis.)

Not requiring the expanded form means possibly not being able to use the
submitted .xml.  Why even bother asking for the .xml then?  It has got
to be expanded.

> Point to the github…

Right, either you submit an expanded .xml, or a tarball with all the
includes, or a Git repo URI that can be cloned server side to find the
.xml and its dependencies.  But what if you're building the .xml from
other things?  This way madness lies.

E.g., I used to use LyX + lyx2rfc, so some of my canonical sources are
.lyx files, not .xml, but I wouldn't expect the submission server to
take a Git repo link and know what to do to get .xml.  I just check in
the .xml, and the .txt, and the .unpg, and the .html -- especially the
.xml and the .unpg, since that makes reviewing commits much easier.

No, the simplest thing to do is to require the expanded .xml, at which
point it might as well be in UTF-8.  And the server might as well use it
to generate .txt and other formats, thus giving the server a chance to
insist on expanded .xml.

The form could also stand to have a field for a Git repository URI --
that would be very nice documentation, actually.  And the datatracker
page could link it too.  Might as well also ask if issues and PRs are
accepted at the link (if it's not just a Git link, but actually richer
and supports issues and PRs).

> (If you write a bis after 10 years, the source is probably already
> much less useful.  [...]

Yes, but if the archived .xml is complete (expanded) then you can work
from that, much better than working from the .txt, even if:

>            [...].  Henrik even has a tool that upconverts .txt to
> .xml…)

Really!  But that can't possibly be as good as working from the expanded
.xml.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 6 Apr 2020 21:59:02 +0200

>>> form. (E.g., somebody who wants it as the starting point for a bis.)

(Actually, I didn’t.)

And I should maybe point out that kramdown-rfc cheats: it hides the source (.md) in the generated XML, which survives prepping (I currently use prepping as a slightly more reliable substitute for expansion).  So you get the markdown source, but not the source of the scripts that generated and validated the examples etc.  You need the github repo for that.

I like the idea of being able to submit via a repo link.  Martin Thomson’s template has the ghpages convention that could be used to find the XML.  And the repo could be cloned/fetched for future perusal.

Grüße, Carsten

---

From: Paul Kyzivat <[email-masked]>
Date: Mon, 6 Apr 2020 16:04:31 -0400

> and supports issues and PRs).

"Source" is a very slippery term, and can include input to some very 
unstable home-brew tools. Yet, if it is to be possible for somebody new 
to take over authorship then it is very difficult without the source and 
the tools that expand it.

Having a reference to the Git repository would be go quite a ways to 
achieving that. If that is the direction, then it indeed should be 
available in the datatracker. And there should be some assurance (how?) 
that the parts of the repository that this reference depends upon are 
stable and reliable.

I've also long been bothered by the submission took accepting multiple 
forms where some are derived from others (e.g., an xml and a txt) 
because there is no guarantee that those forms are consistent. It is 
only an implied *assertion* by the submitter. It would be much better if 
derived forms were always created by the submission tool. And this could 
extend all the way to cases where the "source" is in a Git repository. 
But it only works if the tooling has access to, and understanding of, 
the tooling used for the derivation.

	Thanks,
	Paul

---

From: Nico Williams <[email-masked]>
Date: Mon, 6 Apr 2020 15:47:06 -0500

On Mon, Apr 06, 2020 at 04:04:31PM -0400, Paul Kyzivat wrote:
> unstable home-brew tools. Yet, if it is to be possible for somebody
> new to take over authorship then it is very difficult without the
> source and the tools that expand it.

If the goal is to enable someone else to take over the editor baton,
then a Git repo link should suffice.

If the goal is to have the submission server be able to build the
Internet-Draft, then you really need to submit expanded .xml.

Of course, we could do both.

> only an implied *assertion* by the submitter. It would be much better
> if derived forms were always created by the submission tool. And this

With the caveat that some editors don't do XML and we insist on
supporting that, so it has to be possible to submit .txt-but-not-.xml.

But yes, that's my point: if you're submitting .xml, it should be
expanded .xml and the .txt should either be left out or checked for
exact match with the .txt produced by the server from the expanded .xml.

> could extend all the way to cases where the "source" is in a Git
> repository. But it only works if the tooling has access to, and
> understanding of, the tooling used for the derivation.

And that's hard.  That's why the .xml needs to be expanded.  That's just
so much easier on everyone.

And if it's expanded?  It might as well be UTF-8.

The .txt should absolutely be required to be UTF-8 when .xml is not
submitted.

Nico
--