Subject: [Tools-discuss] [Cellar] non-ascii characters (fwd) Dave Rice: [Cellar] non-ascii characters

From: Michael Richardson <[email-masked]>
Date: Tue, 27 Aug 2019 14:29:12 -0400

reposting to tools-discuss for comments.
I suspect that the idnits tool may be in error in this case.

We are also using xml-v3 for ffv1, since it is full of math which gets
rendered to SVG.  

From: Dave Rice <[email-masked]>
Date: Tue, 27 Aug 2019 10:24:41 -0400
To: Codec Encoding for LossLess Archiving and Realtime transmission
 <[email-masked]>

Hi cellar,

I’m doing an idnits review on the latest ffv1 draft and am confused about the
IETF rules for non-ascii characters in txt rfc drafts. The update at
https://tools.ietf.org/html/rfc7997 <https://tools.ietf.org/html/rfc7997>
seems to allow the RFC Editor to use some discretion in using non-ascii
characters, but idnits <https://tools.ietf.org/tools/idnits/> still complains
about that. One instance is using a Unicode minus sign to be more
semantically clear than an ascii hyphen. Also since the Unicode minus sign
uses more bytes than the ascii hyphen, idnits will say the line is too long,
even though in appearance the line displays within the correct width. 

In the RFC for ffv1, we’ve used unicode symbols for floor and ceiling
functions, such as ⌊a⌋ and ⌈a⌉, but rfc2xml transforms these into their
hexadecimal expressions such as &#8970;a&#8971; and &#8968;a&#8969;. Any
advice, should we listen to idnits and avoid non-ascii characters or listen
to rfc7997 and use them when it makes the content more clear? 

Kind Regards,
Dave Rice

-- 
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 27 Aug 2019 20:46:37 +0200

Hi Michael,

Here's my take on this:

A number of people has highlighted the desirability of using math symbols
in text.  However, this is not currently supported by xml2rfc v3, and will
make idnits unhappy.  Putting math in SVG is currently the supported
approach.

Several approaches to permitting math other than by providing SVG artwork
has been raised and discussed, and there may be ways forward for this once
the current focus on the v3 format transition is complete, but until we
reach completion on the transition effort, I don't think it's going to be
fruitful to push for this.

(Personally, I've felt sufficiently strongly about inline and display math
support to propose a number of possible approaches to the RFC Editor, but I
don't have time over to do any prototyping of any of the most promising
approaches until well after the xml2rfc v3 formatters are seen to be solid.)

Best regards,

	Henrik

On 2019-08-27 20:29, Michael Richardson wrote:
> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]
>

---

From: Michael Richardson <[email-masked]>
Date: Tue, 27 Aug 2019 15:35:10 -0400

> A number of people has highlighted the desirability of using math
    > symbols in text.  However, this is not currently supported by xml2rfc
    > v3, and will make idnits unhappy.  Putting math in SVG is currently the
    > supported approach.

Yes, and we are using SVG. But we can't do it with XML yet (now Sept. 16).
So we format a txt version for submission to post to the datatracker,
and it has some UTF-8 in it.

Are you agreeing that idnits is in error?

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 27 Aug 2019 22:10:39 +0200

Hi Michael,

> Yes, and we are using SVG. But we can't do it with XML yet (now Sept. 16).

That's when the RFC-Editor intends to start publishing xml, I believe?

But you should already be able to post drafts using v3 XML -- any reason not
to do so?  (It doesn't affect your primary question, but I'm curious.)

> Are you agreeing that idnits is in error?

For publication purposes, the final arbiter isn't idnits, and the idnits
message for non-ascii characters has been downgraded from error to warning,
since it's not sufficiently sophisticated to clearly tell what's acceptable
and what's not, based on RFC 7997.  I think it's right to emit a warning,
in order to have both authors, shepherd and ADs consider the case, but
can't make any further judgement.

	Henrik

---

From: Tom Pusateri <[email-masked]>
Date: Tue, 27 Aug 2019 17:06:10 -0400

> On Aug 27, 2019, at 3:35 PM, Michael Richardson <[email-masked]> wrote:
> 
> 

RFC 7997 is pretty clear when UTF-8 is permissible and math symbols are not in the list.

I also wanted to use math symbols recently and the TXT version and the HTML version looked fine but the PDF version did not and so during IESG review, I was asked to revert the math symbols to their ASCII equivalent.

I think you’ll probably run into the same problem if you try to use them at this point other than for the particular use cases in RFC 7997 (names and titles).

Tom

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 29 Aug 2019 21:11:58 -0400

Michael Richardson <[email-masked]> writes:
> [...] One instance is using a Unicode minus sign to be more
> semantically clear than an ascii hyphen. [...]

Given that using non-ASCII characters is still considered doubtful, and
given that using hyphen to mean "minus" is a long-established tradition,
I would say to use hyphen.

The place where I'd expect the Editor to show flexiblity is where ASCII
makes it difficult to express the intended meaning.  Do you have such a
situation?

Dale

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Aug 2019 06:20:38 +0200

> RFC 7997 is pretty clear when UTF-8 is permissible and math symbols are not in the list.

I believe, with a 2019 perspective on the world, this is no longer the right decision.
We are incurring the costs of supporting Unicode but not reaping its benefits.
Providing HTML as a main viewing format and not even being able to solve the hyphen-dash-minus problem leads to weird-looking documents.
How is requiring SVG display support for understanding a document better than simply requiring Unicode support?

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 30 Aug 2019 06:25:40 +0200

> I believe, with a 2019 perspective on the world, this is no longer the right decision.

Did the perspective change significantly within the last 4 years??

> How is requiring SVG display support for understanding a document better than simply requiring Unicode support?

I do prefer proper hyphens, but I also note that the proper choice can
lead to bikeshedding.

My recollection is that we started with the use cases that seemed most
important back then. I'll also note that Unicode math characters won't
be sufficient in all cases, so MathML definitively is something to
consider as well.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Aug 2019 07:16:26 +0200

> Did the perspective change significantly within the last 4 years??

I think so.  Of course, tipping points are always a perception issue, but I think it would not be hard to obtain consensus today that we are past the UTF-8 tipping point.  Of course, there will be people who remember that they argued otherwise a decade ago and will try to maintain that position, but outside of vintage computer museums, the reality is pretty well-defined now.

> I do prefer proper hyphens, but I also note that the proper choice can

I’m not aware of an opportunity for that (well, maybe except for the choice between en-dashes and em-dashes for the punctuation mark in the form of a horizontal line used to indicate a pause, to delimit an inserted sentence or phrase, or to indicate a deliberately omitted word).    But given that this is the IETF, I’m sure we’ll find one :-)
More seriously, this is one place where copy-editing can simply solve the problem.

> be sufficient in all cases, so MathML definitively is something to

Having math characters available is not a valid replacement for something like MathML.
Being able to throw in some casual in-line math (that an authoring tool would convert to the right characters) is useful, though.  As in ⌈a⌉ and friends…  (Nobody would argue that I can’t use a*b in a spec to talk about the product of a and b, so why not a∉B or ∀x.x²≥0.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 30 Aug 2019 09:39:18 +0200

On 30.08.2019 07:16, Carsten Bormann wrote:
> ...

AFAIR, there are also different opinions about whether there should be
whitespace around it.

> More seriously, this is one place where copy-editing can simply solve the problem.

That is true.

> Being able to throw in some casual in-line math (that an authoring tool would convert to the right characters) is useful, though.  As in ⌈a⌉ and friends…  (Nobody would argue that I can’t use a*b i…

Ack. Just explaining how we got there.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 30 Aug 2019 10:00:09 +0200

> AFAIR, there are also different opinions about whether there should be

Right.

Traditional German: em-dash with whitespace
“Modern” German: en-dash with whitespace
Traditional US English: em-dash without whitespace, or sometimes with (AP style, i.e., news)
Traditional UK English: it’s complicated
Today’s English: hodgepodge
Laypeople: essentially, use dashes only where Microsoft Word turns a hyphen into a dash automatically, and that involves typing spaces, or so

(And, of course, there are range and versus dashes as well, which are always en-dashes, but might incomprehensibly have whitespace in English.)

I don’t have my CMOS with me, or I would cite chapter and verse.
But the important observation is that this is a well-known editorial style issue to which a copy-editor should have an answer.

More generally speaking, the fact that Unicode has more characters does not mean we need to spend a lot of time deciding which character to use.  I think we are also old enough to no longer fall into Unicode traps (e.g., Ω vs. Ω).

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Fri, 30 Aug 2019 12:29:18 -0400

> Did the perspective change significantly within the last 4 years??

While it was published Dec. 2016, which is three years ago, DT says it was
started Feb. 2014, do that's 5.5 years ago, based upon discussion that was
probably older.

I'm one of the people who has been difficult to move to UTF-8.
(I still run xterm and vtwm, not gnome-terminal and sawfish/metacity)
But, I'm UTF-8 capable, even if I can't always figure out how to type the
accents in Malisa's Montenegrin name.  I usually have to copy&paste them :-)

---

From: Michael Richardson <[email-masked]>
Date: Fri, 30 Aug 2019 12:33:45 -0400

If we accept the rules of RFC7997 (nice palindrone) suggests UTF-8 for proper
names only,  is it acceptable to use other symbols in the XML for use when
translating to to HTML?

And if so, is there a way to tell xml2rfc (v3) what to put when producing the ASCII^Wtxt
copy?

---

From: Julian Reschke <[email-masked]>
Date: Fri, 30 Aug 2019 18:44:26 +0200

> names only,  is it acceptable to use other symbols in the XML for use when

I don't quite understand. Example maybe?

> And if so, is there a way to tell xml2rfc (v3) what to put when producing the ASCII^Wtxt

You mean plain text? That can contain non-ASCII just fine...

Best regards, Julian

---

From: Dave Rice <[email-masked]>
Date: Sat, 31 Aug 2019 19:02:08 -0400

Hi all,

> On Aug 30, 2019, at 4:00 AM, Carsten Bormann <[email-masked]> wrote:
> 

Thanks very much for this discussion, based on this and also running idnits, I’ve created this pull request for the FFV1 draft, https://github.com/FFmpeg/FFV1/pull/158/files <https://github.com/FFmpeg/FFV1/pull/158/files>. Related to this discussion the pull request changes the unicode minus sign to an ascii hyphen and uses a "ceil(a)" and "floor(a)" function rather than using "⌈a⌉" and "⌊a⌋”. After this PR, it passes idnits with no errors, warnings, or comments.

Thanks much,
Dave Rice