Subject: [xml2rfc] Vexing issue with artwork, twice too many lines

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Feb 2018 11:39:51 -0800

In several drafts, I observed that "artwork" was not rendered properly.
It inserts an empty line after each line of input. The attached file
shows a simple example. My input is:

<figure align="center"><artwork align="center">
  Client                       Server
  ClientHello
  + early_data
  + key_share*
  + psk_key_exchange_modes
  + pre_shared_key
  + SNI = fronting
                    --------&gt;
                                                      ServerHello
                                                +  pre_shared_key
                                                     + key_share*
                                            {EncryptedExtensions}
                                            {CertificateRequest*}
                                                   {Certificate*}
                                             {CertificateVerify*}
                                                       {Finished}
                         &lt;--------------------
</artwork></figure>

I would expect that to be rendered on 17 lines. But instead,
https://xml2rfc.tools.ietf.org/cgi-bin/xml2rfc.cgi produces 34 lines:

       Client                       Server

       ClientHello

       + early_data

       + key_share*

       + psk_key_exchange_modes

       + pre_shared_key

       + SNI = fronting

                         -------->

                                                           ServerHello

                                                     +  pre_shared_key

                                                          + key_share*

                                                 {EncryptedExtensions}

                                                 {CertificateRequest*}

                                                        {Certificate*}

                                                  {CertificateVerify*}

                                                            {Finished}

                              <--------------------

I suppose I may be doing something wrong, but I have no idea. Can someone help me?

Thanks

-- Christian Huitema

---

From: Stewart Bryant <[email-masked]>
Date: Mon, 19 Feb 2018 19:57:38 +0000

Modulo getting it exactly as you intended, Kramdown generates the 
following xml for that figure:

<figure><artwork><![CDATA[
Client                                      Server
ClientHello
+ early_data
+ key_share*
+ psk_key_exchange_modes
+ pre_shared_key
+ SNI = fronting
                      -------->
                                          ServerHello
                                    +  pre_shared_key
                                         + key_share*
                                {EncryptedExtensions}
                                {CertificateRequest*}
                                       {Certificate*}
                                 {CertificateVerify*}
                                           {Finished}
                <--------------------
]]></artwork></figure>

which renders to

Client                                      Server
    ClientHello
    + early_data
    + key_share*
    + psk_key_exchange_modes
    + pre_shared_key
    + SNI = fronting
                         -------->
                                             ServerHello
                                       +  pre_shared_key
                                            + key_share*
                                   {EncryptedExtensions}
                                   {CertificateRequest*}
                                          {Certificate*}
                                    {CertificateVerify*}
                                              {Finished}
                   <--------------------

- Stewart

On 19/02/2018 19:39, Christian Huitema wrote:
> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Feb 2018 12:00:47 -0800

> https://xml2rfc.tools.ietf.org/cgi-bin/xml2rfc.cgi produces 34 lines:

OK, I know the answer now. I am working on Windows, and the text lines
are separated by "CRLF" instead of just "LF". If I strip the line
terminations to just "LF", I get the figures that I expect. But that's a
bit painful. Surely there should be a way to work around that!

-- Christian Huitema

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 19 Feb 2018 22:51:54 +0100

Hi Christian,

> bit painful. Surely there should be a way to work around that!

xml2rfc is coded to handle all of the line terminations LF (unix),
CRLF (windows) and CR (traditional mac) correctly.  I think the
appearance of line termination conversion helping is because it also
happens to convert another aspect of the xml file you enclosed:

If I ask the command 'file' on unix to report on your xml file, this
is what I get:

    XML 1.0 document, Little-endian UTF-16 Unicode text, with CRLF, CR line
    terminators

So your file actually has mixed line terminators, but the big issue here is
that while the xml header declares that the content is UTF-8, the file
encoding is actually Little-endian UTF-16.

I bet that if you save the file as UTF-8, xml2rfc will deal fine with the
line terminators without any need to convert or strip them.

Best regards,

	Henrik

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Feb 2018 14:19:33 -0800

OK, that explains it. Now I have to understand why MMark produced the  Unicode output...

-- Christian Huitema 

> On Feb 19, 2018, at 1:51 PM, Henrik Levkowetz <[email-masked]> wrote:
> 
> Hi Christian,
>

---

From: Christian Huitema <[email-masked]>
Date: Mon, 19 Feb 2018 16:12:10 -0800

I have verified that if the output of mmark.exe is saved as UTF8 instead
of Unicode, then the art work looks fine. Thanks Henrik for the advice.

-- Christian Huitema

---

From: Julian Reschke <[email-masked]>
Date: Tue, 20 Feb 2018 07:13:38 +0100

On 2018-02-19 22:51, Henrik Levkowetz wrote:
> ...

If that's the case, any XML parser should reject it. Am I missing something?

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Feb 2018 14:29:56 +0100

Hi Julian,

> If that's the case, any XML parser should reject it. Am I missing something?

xml2rfc doesn't hand the file name directly to the parser, it reads the
file text first, using different code paths under python2 vs. python3,
and hands the resulting text blob to the parser.  It looks as if the
python2 file open and read recognizes the UTF-16 format and deals well
enough with it to produce the 8-bit stream the xml parser wants, but
not well enough to avoid problems Christian experienced with <artwork>.
I'm actually surprised that it deals with the UTF-16 file at all, but
there it is.

Next time I revisit the parser, I'll see about rewriting the file read to
produce a warning, and also bring the py2 and py3 code closer.

Best regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 20 Feb 2018 14:35:24 +0100

> produce a warning, and also bring the py2 and py3 code closer.

Thanks for the explanation.

Is there a good reason for the extra step? This really sounds like a 
recipe for interoperability problems with other code that reject 
anything not accepted by XML parsers...

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 20 Feb 2018 15:37:14 +0100

Hi Julian,

> anything not accepted by XML parsers...

Agreed.  There is a reason; to dig it out I'd have to dive into the commit
logs.  Without that I don't recall enough to say it's a good reason.  FWIW
I agree with your reasoning, but there could still be reasons things didn't
work with that solution.

Best regards,

	Henrik