Subject: Re: [xml2rfc] #367 (Version_3_cli_txt): CODE BEGINS in text output

From: "xml2rfc issue tracker" <[email-masked]>
Date: Mon, 17 Sep 2018 21:39:29 -0000

#367: CODE BEGINS in text output

Comment (by [email-masked]):

 This is generated if you use <sourcecode>.  If you don't want those, use
 <artwork> instead.

-- 
--------------------------------+-------------------
  Reporter:  [email-masked]       |      Owner:
      Type:  defect             |     Status:  new
  Priority:  medium             |  Milestone:
 Component:  Version_3_cli_txt  |    Version:  2.4.x
Resolution:                     |   Keywords:
--------------------------------+-------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:1>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Tue, 18 Sep 2018 04:53:59 -0000

Comment (by [email-masked]):

 Why does it include it? Is there anything in the specs that says it
 should?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:2>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 14:40:22 -0000

Not in RFC 7991.  The background is that <CODE BEGINS> and <CODE ENDS>
 informally started to be used to mark code in drafts a number of years
 ago.  That was followed by this statement from the IETF Trust:

   https://trustee.ietf.org/license-info/IETF-TLP-1.htm

 and later codified more for YANG documents in

   https://tools.ietf.org/html/rfc6087#section-3.1

 It seems to make sense to support this in 7991 <sourcecode>, to encourage
 consistency.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:3>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 14:53:50 -0000

That would need to be discussed.

 My understanding is that the markers are only to be used if the language
 is not on a white list (see https://trustee.ietf.org/license-info/Code-
 Components-List-4-23-09.txt)

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:4>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 15:37:08 -0000

> That would need to be discussed.

 Ack.

 Umm.  That doesn't make sense to me, given that one of the reasons I've
 seen for standardizing this is to be able to write tools that consistently
 extract code components from drafts and RFCs.  Several YANG tools uses
 this already in order to extract YANG modules from documents.

 > See also related discussion in https://github.com/rfc-format/draft-iab-
 xml2rfc-v3-bis/issues/8 (raised 20 months ago).

 Ack.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:5>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 15:48:02 -0000

The reason for the code tags wasn't extraction, but copyright/licensing.

 If you want to automate extraction, the answer is to process the XML, not
 the TXT output.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:6>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 16:05:39 -0000

> The reason for the code tags wasn't extraction, but copyright/licensing.

 No, that's simply not correct.  I was part of the discussions that lead to
 the TLP wording,
 and pointed at the already established usage of <CODE BEGINS> for
 extraction.

 > If you want to automate extraction, the answer is to process the XML,
 not the TXT output.

 Good answer if xml is available.  Not so good otherwise.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:7>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 19 Sep 2018 16:10:18 -0000

Comment (by [email-masked]):

 > No, that's simply not correct. I was part of the discussions that lead
 to the TLP wording, and pointed at the already established usage of <CODE
 BEGINS> for extraction.

 Interesting - I wasn't aware of that.

 > Good answer if xml is available. Not so good otherwise.

 I thought the implementation of the v3 processor was part of the move to
 XML as canonical format? Also, what's the point to implement this in this
 processor, because, by definition, it runs on XML input?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:8>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 10:44:08 -0000

As an example, let's look at the YANG toolchain.  Both
 https://yangvalidator.org/ and https://yangcatalog.org/ routinely take in
 drafts in development, extract YANG modules, and process them (validation,
 meta-data extraction, dependency tree computation, and more).  In the end,
 documents may end up as RFCs, but the bulk of the validation etc. is done
 during the draft development.  The draft text, using the specific
 formatting defined in RFC 6087, is the common basis of this, no matter if
 the draft was created using xml2rfc or through some other means.  Given
 that we have a clearly defined way of delimiting code chunks in drafts and
 RFCs, what reason could we possibly have for *not* letting the xml2rfc
 formatter produce correct delimitation and chunk filename according to RFC
 6087?  Forcing people to add the <CODE BEGINS> text manually for text
 output doesn't make sense to me.

 Now, I'm open to doing things differently for formats that can carry
 metainformation.  In RFC 7992, the rendering of <sourcecode> specifies
 that <pre class="sourcecode"> should be used -- this would be sufficient
 for tools to reliably extract code from HTML rendered documents.

 If xml is available, (as it will consistently be for RFCs when the RFC-
 Editor has transitioned to XML as the archival format) the extraction is a
 no-brainer, exactly as you describe.

 But I don't see why we should use that as a reason not to make life easy
 for people writing drafts where the code might be extracted from the text
 document for validation or other processing.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:9>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 11:48:09 -0000

I have no problem with an *option* to include the code markers. Inserting
 them however when they aren't needed doesn't make sense to me.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:10>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 12:02:58 -0000

Ok, so what about this: give {{{<sourcecode>}}} an attribute {{{code-
 markers}}} with possible values {{{ "default" | "true" | "false" }}} where
 {{{default}}} would insert {{{<CODE BEGINS>}}} etc. in formats without any
 meta-information carrying capability, such as the plaintext format, but
 not otherwise?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:11>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 12:14:07 -0000

I think an attribute is what we need indeed.

 I'm unhappy with new features that will lead to text content and HTML
 content to differ in anything other than presentation.

 The default behavior really should be not to insert the code markers. Let
 the author override it when they really really need it.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:12>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 13:13:41 -0000

I'm ok with consistently inserting markers in all formats, or not,
 depending on attribute settings.

 I think your view of what should be the default is coloured by what you
 would personally use, but I also believe what makes life easiest for most
 draft authors wanting to insert code in drafts is to have as default to
 insert the markers.

 If this was a general markup language, I would not suggest this.  But as
 it's geared towards IETF documents where we *want* people to insert code
 markers, we should make that happen by default.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:13>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 13:35:22 -0000

Actually, we only want people to insert code markers when they are needed.

 As far as I can tell, they are only needed for code fragments not covered
 by the TLP, and for YANG modules. Unless I'm missing something that's an
 exception, not the rule, right?

 (In any case, this discussion probably should take place in
 https://github.com/rfc-format/draft-iab-xml2rfc-v3-bis/issues/8)

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:14>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 13:56:02 -0000

> Actually, we only want people to insert code markers when they are
 needed.

 I don't agree with this statement.  On what is it based?

 > As far as I can tell, they are only needed for code fragments not
 covered by the TLP, and for YANG modules. Unless I'm missing something
 that's an exception, not the rule, right?

 The IESG and others have expressed a desire to add validation for other
 code fragments than Yang in the datatracker, basically any formal
 languages we routinely use, such as ABNF; not only YANG.

 I guess, yes.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:15>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Paul Kyzivat <[email-masked]>
Date: Thu, 20 Sep 2018 10:41:38 -0400

>   languages we routinely use, such as ABNF; not only YANG.

++

IMO putting code markers around ABNF would be very objectionable. Quite 
often very small snippets of ABNF are run into text, where the code 
markers would be very distracting. (Less of a problem when a large ABNF 
grammar stands as a single block.)

I care less whether they are optional or default, so long as they can be 
disabled.

	Thanks,
	Paul

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 20:27:02 -0000

> I don't agree with this statement. On what is it based?

 On the theory that bloat that affects the readability should be avoided.

 We're talking about the future, right? I just don't see how extraction
 from plain text is of relevance, when, by definition, XML is available (If
 it wasn't, you wouldn't have anything to run the formatter on...)

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:16>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Thu, 20 Sep 2018 20:43:10 -0000

Comment (by [email-masked]):

 I very much agree that adding this in for text is generally not what I
 want.  I would question even more if this behavior is desired for the HTML
 format as I would expect that this would be highlighted by color and not
 by text.  If there is going to be a default behavior, then it should
 perhaps depend on the language involved rather than being universal across
 all languages.  The RFC Editor in consultation w/ the powers that be can
 then set a default for different languages.

 The idea of moving things from sourcecode to artwork would only work if
 you were to move the 'type' attribute as well.  If you don't do that then
 extractors based on XML would no longer be successful.  However the whole
 concept was to separate artwork from code so that does not make any sense.

 From the RFCs that I have normally looked at, the frequency of using the
 text tagging at the start and end has been zero.  I realize that there are
 cases where this is very common but it is not by any means universal.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:17>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 20 Sep 2018 22:54:24 +0200

Interesting discussion.

Of course, the author likely actually writes markdown, so what you are discussing at the XML level somehow needs to be mapped to markdown.

I have noticed that I often have two kinds of code in my drafts:

— actual source code that I expect people to use (so extraction is important)

— examples and other snippets

In the second group, there are also invalid or incomplete examples which should not trip up a CI process.

E.g., for CDDL, I’m right now (with RFCXML v2) using

~~~ CDDL
…
~~~

if I have actual source code, and

~~~ CDDLx
…
~~~

for those snippets (not currently distinguishing between valid and invalid ones).

So then I can do a

	xpath "//artwork[@type='CDDL']/text()”

to extract all the (valid, non-snippet) CDDL.

It might make sense to mirror such a convention at the XML level (and maybe generalize it so it can be used for ABNf etc. as well).

Grüße, Carsten

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Fri, 21 Sep 2018 16:26:17 -0000

I think you miss part of the point here:

 1. Not all drafts are being submitted with XML
 2. The datatracker and toolschains to validate code need to work on all
 drafts
 3. Toolchains have been built and are being built to extract from all
 drafts using code markers
 4. To make it easy for people to produce compliant drafts with _this_
 tool, we should make it easy to provide the code markers.

 In an ideal future where all drafts were produced from xml2rfc v3, the
 extraction toolchains would not need to work on text documents.  Till
 then, building one toolchain that works on all drafts wins over building
 two, especially when there's not enough hours to do anywhere close to all
 the work items queued for the datatracker.

 And by the way, not nearly all drafts produced from xml source is
 submitted with xml source.  Out of 11914 drafts from the last 365 days,
 there were 7137 .xml files.  I believe the percentage actually produced
 from .xml sources is much higher than 60% (although I'm gratified to find
 that we've reached 60% .xml availability).

 Anyway, I think we agree on having an attribute to control rendering of
 code markers; the only question is what would be the default setting.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/367#comment:18>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>