Subject: [xml2rfc] Silly unicode question

From: "David R. Oran" <[email-masked]>
Date: Wed, 18 Mar 2020 11:26:03 -0400

I would have thought unicode characters with diereses would be rendered 
directly in v3, e.g. for the word “naïvely”.

Instead, if I type it directly, I get “na&#239;vely” in HTML and 
PDF; if I enclose it as
“na<u>ï</u>vely” I get “na"ï" (LATIN SMALL LETTER I WITH 
DIAERESIS, U+00EF)vely”

Cluebat?

DaveO

---

From: Julian Reschke <[email-masked]>
Date: Wed, 18 Mar 2020 17:38:39 +0100

On 18.03.2020 16:26, David R. Oran wrote:
> Instead, if I type it directly, I get “na&#239;vely” in HTML and PDF; if
> I enclose it as
> “na<u>ï</u>vely” I get “na"ï" (LATIN SMALL LETTER I WITH DIAERESIS,
> U+00EF)vely”
>
> Cluebat?
>
> DaveO

Yes.

IMHO the intention was to allow that in RFC 7991, but it hasn't been
implemented like that.

A similar example are non-ASCII characters in names, for which a new
element was added (which we wouldn't have needed if we had stuck to RFC
7991).

Best regards, Julian

---

From: "John Levine" <[email-masked]>
Date: 18 Mar 2020 16:46:33 -0400

In article <[email-masked]> you write:
>element was added (which we wouldn't have needed if we had stuck to RFC 7991).

That seems like a bug to me.  We assume that even the unpaginated text will be
rendered on something with a UTF-8 character set, right?

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 18 Mar 2020 21:52:57 +0100

Hi John,

> rendered on something with a UTF-8 character set, right?

What's currently implemented aims to follow RFC 7997, which is the relevant
RFC in this case, I believe.

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Wed, 18 Mar 2020 22:16:19 +0100

> RFC in this case, I believe.

FWIW, RFC 7997 explains when using non-ASCII characters is ok. That does
not necessarily imply that the vocabulary needs to enforce that
(otherwise, RFC 7997 would be in conflict with RFC 7991, which IMHO is
not the case).

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 18 Mar 2020 17:33:19 -0400

> (otherwise, RFC 7997 would be in conflict with RFC 7991, which IMHO is

It seems to me that a fair amount of 7997 became obsolete once we started 
publishing in v3.  I'll run it past the RSOC and see what they think.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Mar 2020 12:35:47 +0100

Hi John,

> publishing in v3.  I'll run it past the RSOC and see what they think.

That's not the line Heather took.  She talked about doing a 7997bis after
experience with v3 schema *and* RFC 7997.

It's also not backed up by the RFCs as published: since RFC 7997 was
published as part of the v3 RFC 7991-7998 group of RFCs, I don't think
you can argue that the using the schema from that group (RFC 7991)
renders another RFC from that cluster (RFC 7997) obsolete.

This would be a major shift in policy.  I'm not arguing for or against
the actual change in saying this; but I am saying that it's a major change,
and I don't think it's appropriate to do it without wider discussion and a
7997bis document.

(On the actual change, I think it is quite reasonable to permit the full
Latin script range in running text.  I'm more doubtful about moving
directly to permitting the full unicode range without more experience with
v3 publication.  There are too many pitfalls there.  Just handling full-width
and half-width CJK glyphs and right-to-left scripts in the limited locations
we permit them now has been a challenge, and I don't think we've had many
actual instances of publishing documents using such yet.)

Regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Mar 2020 12:37:23 +0100

On 18.03.2020 22:33, John R Levine wrote:
> It seems to me that a fair amount of 7997 became obsolete once we
> started publishing in v3.  I'll run it past the RSOC and see what they
> think.

Trying to summarize...

When RFC 7991 and 7997 were written, the idea was that use of non-ASCII
characters would *not* be controlled by the vocabulary, but by the
Production Center. This is why RFC 7991 is silent on this, except for
the cases where *alternatives* for things with non-ASCII characters are
needed (such as in contact information).

Later on, <u> was proposed, so that authors needing non-ASCII character
in protocol descriptions would have a tool to automatically insert code
points, long names, etc. That's a good thing. However, <u>, as
implemented, *also* removes the ability to use non-ASCII character in
prose except when inside <u>. I think that went too far; in any case it
needs to be discussed.

With that of course, legit uses such as contact names in
Acknowledgements became impossible, which lead to the introduction of
<contact> as inline element.

I think <u> is a good thing to have, although I'd prefer it to be just a
tool to make things easier to write correct text, not a requirement.

If we keep it, there are at least two things to be done.

1) Bug: (see
<https://mailarchive.ietf.org/arch/msg/xml2rfc/dT_nSNngdaJkfs2PunTkcrdiFss/>)
- the output of <u> changes when it becomes the target of a reference.
This doesn't make any sense; this *absolutely* needs to be fixed.

2) Task: <u> inserts automatically generated text based on external data
(the Unicode database), as such, we need to preserve the expanded form
using the preptool step; otherwise we loose information in the archive
format (yes, the Unicode database changes).

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Mar 2020 12:41:38 +0100

On 2020-03-19, at 12:37, Julian Reschke <[email-masked]> wrote:
> Production Center. 

I think this is an important point:  If we want to automate some of the checks that RPC would be doing, the place to automate this is idnits or an idnits-like tool, not xml2rfc (maybe outside of warnings).  Specifically, xml2rfc is also used for authoring I-Ds, where none of these restrictions make sense.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Mar 2020 13:19:12 +0100

> renders another RFC from that cluster (RFC 7997) obsolete.

Absolutely; these were *meant* to be consistent.

Speaking of which: once we start actually revising those, we need to
include the RFC Style Guide into this. It doesn't make sense to have
Style requirements that can not be met by the tools; on the other hand,
what we discover when writing/testing the tools needs to inform
decisions on the Style Guide.

> Latin script range in running text.  I'm more doubtful about moving

+1.

> actual instances of publishing documents using such yet.)

True.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Mar 2020 13:31:07 +0100

>> actual instances of publishing documents using such yet.)

There is something in between Latin and full Unicode: math symbols etc.

Adding characters is much easier if they are compatible with the Latin writing system; we should not put math symbols into the same bin as RTL and scripts with complicated combining characters (the latter are only needed in those RFCs that have them as subject matter, and <u> is already geared to those).

Grüße, Carsten

---

From: "John R Levine" <[email-masked]>
Date: 19 Mar 2020 10:19:27 -0400

Thanks for all of the thoughtful comments.

> Adding characters is much easier if they are compatible with the Latin writing system; we should not put math symbols into the same bin as RTL and scripts with complicated combining characters (the…

Could we say something like single width LTR characters?  I wouldn't think 
it would be hard to render, say, Cyrillic in line.

Regards,

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Mar 2020 15:24:24 +0100

Hi Carsten,

> There is something in between Latin and full Unicode: math symbols

When math symbols were first brought up, I looked into permitting them
inline, and found to my frustration that there's not any single script or
unicode block that is suitable for inclusion; the best I could come up
with is captured in Section 5.1 of my implementation notes:

https://tools.ietf.org/html/draft-levkowetz-xml2rfc-v3-implementation-notes-10#section-5.1

> Adding characters is much easier if they are compatible with the
> Latin writing system; we should not put math symbols into the same
> bin as RTL and scripts with complicated combining characters (the
> latter are only needed in those RFCs that have them as subject
> matter, and <u> is already geared to those).

Well, yes; but on the other hand simply saying 'Latin script' is more
problematic than I made it sound above; they include half-width, full-width
and spacing modified characters that complicate matters:
https://en.wikipedia.org/wiki/Latin_script_in_Unicode

The easiest approach that would still be fairly inclusive might be to say
'Latin script below U+2000'.

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Mar 2020 15:31:19 +0100

Hi John,

> it would be hard to render, say, Cyrillic in line.

It sounds as if you're seriously considering obsoleting RFC 7997 outside
the usual process, based only on discussion on this list.  I think this
needs much wider discussion than what we get here.

What I've written on this issue in my recent messages tried to address what
might make sense or not given a revision of RFC 7997.  If we are now going
to land on saying anything particular that overrides 7997 without a -bis,
I'm not comfortable with the direction.

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Thu, 19 Mar 2020 09:33:56 -0500

> https://tools.ietf.org/html/draft-levkowetz-xml2rfc-v3-implementation-notes-10#section-5.1
We also need to be sensitive to what's available in the fonts we've 
decided are ok to use.
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: "John R Levine" <[email-masked]>
Date: 19 Mar 2020 10:50:57 -0400

> needs much wider discussion than what we get here.

Remember, I don't get to decide anything.  But I'm trying to figure out 
what to proposed to RSOC et al.

Regards,

---

From: Nico Williams <[email-masked]>
Date: Thu, 19 Mar 2020 10:49:12 -0500

On Thu, Mar 19, 2020 at 12:41:38PM +0100, Carsten Bormann wrote:
> I think this is an important point:  If we want to automate some of
> the checks that RPC would be doing, the place to automate this is
> idnits or an idnits-like tool, not xml2rfc (maybe outside of
> warnings).  Specifically, xml2rfc is also used for authoring I-Ds,
> where none of these restrictions make sense.

+1.

xml2rfc is a typesetting tool.  It should not complain about non-ASCII
character usage (perhaps it could warn, but that's it).

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Mar 2020 16:49:49 +0100

> On 2020-03-19, at 15:24, Henrik Levkowetz <[email-masked]> wrote:
> 
> Hi Carsten,
> 
> unicode block that is suitable for inclusion;

So just include everything in the tool and let RPC sort it out.

> the best I could come up

That leaves out some of the most crucial elements of the Latin writing system: punctuation.

It also leaves out the math symbols, arrows, Latin extended C, ...

Whatever we come up with here is going to be bizarre.

Again, there is little gain from the formatting tool enforcing a specific snapshot of what was considered useful at some time by a small group that of course didn’t encompass the author that is trying to do something that is obvious to his peer group.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Mar 2020 17:23:07 +0100

Hi Carsten,

> That leaves out some of the most crucial elements of the Latin writing system: punctuation.

Umm.  Which Latin-script code points are you referring to here? 

> It also leaves out the math symbols, arrows, 

And here?

> Latin extended C, ...

Yes, agreed.  

> Again, there is little gain from the formatting tool enforcing a
> specific snapshot of what was considered useful at some time by a
> small group that of course didn’t encompass the author that is trying
> to do something that is obvious to his peer group.

Which is why I'm advocating taking this to a wider audience and doing a
7997bis if we're going to do changes from what 7997 requires.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Mar 2020 17:36:33 +0100

> Umm.  Which Latin-script code points are you referring to here? 

The whole range U+2000 to U+206F.
Includes dashes, quote marks, thin spaces, …
Absolutely nondispensable.
(Did I mention the Euro sign?)

>> It also leaves out the math symbols, arrows, 

U+2190 to U+21FF for the arrows (U+27F0…, U+2900…, U+1F800…).
U+2200 to U+22FF for math (and U+27C0..., U+2980..., U+2A00...).

> 7997bis if we're going to do changes from what 7997 requires.

RFC 7997 does not require any specific tool to try to figure out what it meant and implement draconic measures.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Thu, 19 Mar 2020 17:42:33 +0100

On 19.03.2020 15:50, John R Levine wrote:
> ...

So is the *RSOC* supposed to decide?

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 19 Mar 2020 13:03:59 -0400

> So is the *RSOC* supposed to decide?

As I understand it, the policy authority I don't have went back to the 
RSOC until they appoint another real RSE.  Not clear how this will work in 
practice.

Regards,

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Mar 2020 18:20:23 +0100

Hi Carsten,

> Absolutely nondispensable.

Ok. Not part of the Unicode Latin script, though.

> U+2200 to U+22FF for math (and U+27C0..., U+2980..., U+2A00...).

Ok, but these aren't part of the Unicode Latin-script, either.

> meant and implement draconic measures.

I think this is now becoming a discussion of the statement of work for the
xml2rfc implementation of 7991-7998.

None of these RFCs require any specific tool to do anything, but that
doesn't mean that they don't apply.

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 19 Mar 2020 18:28:47 +0100

> Ok. Not part of the Unicode Latin script, though.

No, because much punctuation is shared between a few scripts including the Latin one.
That doesn’t make it less essential.  It is just a coincidence that a small part of the punctuation we need already was in ASCII and there are workarounds known in the CS community to get by with that if necessary.

> Ok, but these aren't part of the Unicode Latin-script, either.

... which was what I was trying to point out: These do need to be part of the set that we consider normal in RFCs.

> xml2rfc implementation of 7991-7998.

I never read those and apparently should have.

> doesn't mean that they don't apply.

Indeed, no.  
But then, RFC 3552 applies, too, and xml2rfc is not going to check compliance to that, either.

Grüße, Carsten

---

From: Jonathan Lennox <[email-masked]>
Date: Thu, 19 Mar 2020 14:11:32 -0400

> 'Latin script below U+2000'.

Not to speak to whether this is a good idea or not, but in the interest of
not re-doing work other people have already done, I'd say that the right
approach here would be to use one of the Multilingual European Subsets of
Unicode <http://www.open-std.org/CEN/TC304/p10_1998_05_30.pdf> -- probably
MES-3, the broadest.

-- 
Jonathan Lennox
[email-masked]