Subject: [Tools-discuss] bibxml3 database reload frequency

From: Kent Watsen <[email-masked]>
Date: Thu, 20 Aug 2020 16:05:42 +0000

Hi Tools Team.

How often is https://xml2rfc.tools.ietf.org/public/rfc/bibxml3 rebuilt?  Is there a way to have it rebuild more often and/or for draft authors to externally prompt it to rebuild?

See below for background info.

Thanks,
Kent

Background:

I have one draft depending on another.  The new <relref> element is used, but the resulting hyperlinks point to the previous version of the referenced draft (before it was updated ~1 hour ago).

Looking at https://xml2rfc.tools.ietf.org/public/rfc <https://xml2rfc.tools.ietf.org/public/rfc>, I see that “bibxml3” hasn’t been updated for roughly one day:

	bibxml3/	19-Aug-2020 00:49

Note that the timestamp for “bibxml” (without the ‘3’) is roughly 24 hours newer:

	bibxml/	20-Aug-2020 00:45

In any case, the hyperlinks produced by the <relref> elements are all resolving to the previous version of the referenced draft.   I tried `xml2rfc —clear-cache`, but I think the source issue is with https://xml2rfc.tools.ietf.org/public/rfc/bibxml3 not having updated…

How often is https://xml2rfc.tools.ietf.org/public/rfc/bibxml3 built?  Is there a way to prompt it to reload?

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 20 Aug 2020 18:29:50 +0200

> On 2020-08-20, at 18:05, Kent Watsen <[email-masked]> wrote:
> 
> 
> Hi Tools Team.
> 
> How often is https://xml2rfc.tools.ietf.org/public/rfc/bibxml3 rebuilt?  

I’m not the tools team, and I cannot answer the question in general.
(I seem to remember the answer used to be: hourly.)
Right now, the last additions are dated 2020-08-17 07:00Z, e.g.:

reference.I-D.corcoran-cnsa-ipsec-profile.xml

The most recent I-D I can find is: draft-ietf-bier-tether-00.txt

Datatracker does have the libxml:
https://datatracker.ietf.org/doc/bibxml3/draft-ietf-bier-tether/xml

Protip:

curl https://datatracker.ietf.org/doc/bibxml3/draft-ietf-bier-tether/xml -o ~/.cache/xml2rfc/reference.I-D.ietf-bier-tether.xml

(And set:
export KRAMDOWN_REFCACHEDIR=$HOME/.cache/xml2rfc
in your ~/.profile to get that same cache in kramdown-rfc as well.)

Grüße, Carsten

> Is there a way to have it rebuild more often and/or for draft authors to externally prompt it to rebuild?
> 
> Looking at https://xml2rfc.tools.ietf.org/public/rfc, I see that “bibxml3” hasn’t been updated for roughly one day:
> 
> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 20 Aug 2020 18:36:54 +0200

Hi Kent,

On 2020-08-20 18:05, Kent Watsen wrote:
> 
> Hi Tools Team.
> 
> How often is https://xml2rfc.tools.ietf.org/public/rfc/bibxml3
> rebuilt? Is there a way to have it rebuild more often and/or for
> draft authors to externally prompt it to rebuild?

THe short answers, which won't help you are: Once per day, and sorry, no.

The answer that I believe will help you is this:

Please use the draft bibxml references under datatracker.ietf.org/doc/bibxml3/
instead.  Those are generated on-the-fly and will reflect a new submission
at once.

You can use the old form

  https://datatracker.ietf.org/doc/bibxml3/reference.I-D.draft-ietf-netconf-crypto-types.xml

but also a simplified form that omits the 'reference.I-D.' part:

  https://datatracker.ietf.org/doc/bibxml3/draft-ietf-netconf-crypto-types.xml

Best regards,

	Henrik

---

From: Kent Watsen <[email-masked]>
Date: Thu, 20 Aug 2020 22:24:26 +0000

Hi Henrik and Carsten - thanks for the responses!

It worked, and then it didn’t.  Too late for me now, but you should know, I think datatracker's cache may need to be cleared befor each run...

Below for details.

K

My adventure:

I first updated my XML, e.g.: 

	<xi:include href="https://datatracker.ietf.org/doc/bibxml3/reference.I-D.ietf-netconf-crypto-types.xml”/>

But my locally compiled drafts continued to load the *previous* version (-17) of the crypto-types draft (though I submitted -18 a few hours ago).

So I ran `xml2rfc —clear-cache`, after which the current (-18) revision showed.  Yeah!

Thinking that I conquered it, I submitted the dependent drafts, but looking at the uploaded documents, I see the previous version again.  For instance, the references section of the just-uploaded -13 of the “trust-anchors” draft shows -17 for the “crypto-types” draft: https://tools.ietf.org/html/draft-ietf-netconf-trust-anchors-13#ref-I-D.ietf-netconf-crypto-types <https://tools.ietf.org/html/draft-ietf-netconf-trust-anchors-13#ref-I-D.ietf-netconf-crypto-types>

Oh, well, can’t win them all.

K.

> On Aug 20, 2020, at 12:36 PM, Henrik Levkowetz <[email-masked]> wrote:
> 
> Hi Kent,
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Aug 2020 07:48:30 +0200

On 2020-08-21, at 00:24, Kent Watsen <[email-masked]> wrote:
> 
> It worked, and then it didn’t. 

Yes.  One of the features and bugs of uploading XML is that you rely on the server-side XML2RFC to do your processing for you.  Here it seems, that still had -17 in its cache.

The kramdown-rfc process has a setting, “stand_alone: true”, which asks kramdown-rfc to resolve any external nreferences.  This is what I recommend people to use.  (It is not the default only because kramdown-rfc tries to support the long document development times typical for the IETF by being insanely backwards-compatible.)

To get the same level of assurance out of XML authoring, you need to prep the XML before uploading.

	xml2rfc --preptool foo.xml

You then have a foo.prepped.xml that you can upload.
(This step is also needed with any local references to SVG, source-code etc.)

Note that prepping means the uploaded XML is less useful for the RFC editor.
So, ironically, despite the switch to XML uploading, you will want to provide an unprepped XML, as well as all the locally referenced files, to the RFC editor -- even though the approved, but prepped, XML is in the I-D repository.
But this is irrelevant during the document development process up to approval.

Ultimately, I hope that xml2rfc gains a reference resolution function that does not involve all the preptool processing.  For now, prepping it is.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Aug 2020 09:43:24 +0200

Am 21.08.2020 um 00:24 schrieb Kent Watsen:
> ...

FWIW, another way to avoid this is to reference the ID *with* version
number. That's a static, reliable reference.

And yes, that means that the source document needs to be updated when
the ID is replaced. I consider that a feature, at least in those cases
where the concrete version of the ID matters (such as when referencing
specific details that might change or move to other sections).

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Aug 2020 10:59:29 +0200

> number. That's a static, reliable reference.

Using kramdown-rfc’s indirection methods that actually is less painful than it sounds.  Still requires going through the references once (which idnits can do, with a few missed and a few false positives).  Certainly better than fighting with an old cache entry.

> specific details that might change or move to other sections).

That is indeed sometimes a useful editorial device.  It is too bad that the standard bibxml files use the same anchor for specific versions and for the generic “most-recent" reference, so you cannot have both references (or references to multiple versions) in one document; again, fixed by kramdown-rfc’s anchor replacement mechanism.

My recommendation would be:
When you need to reference a specific version, reference that, and use anchor replacement to get the version number into the label.
When you just want to reference the draft, use the “most-recent” reference, unless you have just changed that — in which case, use the reference to that specific version.  Occasionally, you will get an old version.  Kramdown-rfc’s cache lifetime is 1 h, but that doesn’t help with the longer processing intervals on the server.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 21 Aug 2020 13:13:10 +0200

Hi Kent,

On 2020-08-21 00:24, Kent Watsen wrote:
> It worked, and then it didn’t. Too late for me now, but you should
> know, I think datatracker's cache may need to be cleared befor each
> run...

Huh, right (or rather, the xml2rfc reference cache on datatracker.ietf.org).

Good point.  Will have to think about how to best deal with this.

Best regards,
	
	Henrik

> But my locally compiled drafts continued to load the *previous*
> version (-17) of the crypto-types draft (though I submitted -18 a few
> hours ago).
> 

Yes, xml2rfc keeps a local cache for 24 hours, and that may need clearing.
> 
> Thinking that I conquered it, I submitted the dependent drafts, but
> looking at the uploaded documents, I see the previous version again.
> For instance, the references section of the just-uploaded -13 of the
> “trust-anchors” draft shows -17 for the “crypto-types” draft:
> https://tools.ietf.org/html/draft-ietf-netconf-trust-anchors-13#ref-I-D.ietf-netconf-crypto-types
> <https://tools.ietf.org/html/draft-ietf-netconf-trust-anchors-13#ref-I-D.ietf-netconf-crypto-types>

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 21 Aug 2020 13:25:53 +0200

> 	xml2rfc --preptool foo.xml

Since release 2.44.0 in April this year, --expand is a much better option,
as it leaves a file that can still be worked with (--preptool expands all
attribute default values, adds ToC xml, and a lot of other things that you
don't really want in a document you might want to edit later).

> Note that prepping means the uploaded XML is less useful for the RFC
> editor. So, ironically, despite the switch to XML uploading, you will
> want to provide an unprepped XML, as well as all the locally
> referenced files, to the RFC editor -- even though the approved, but
> prepped, XML is in the I-D repository. But this is irrelevant during
> the document development process up to approval.

The --expand option should avoid these drawbacks.

> Ultimately, I hope that xml2rfc gains a reference resolution function
> that does not involve all the preptool processing. For now, prepping
> it is.

Maybe try --expand?

Regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Aug 2020 13:36:19 +0200

Hi Henrik,

> The --expand option should avoid these drawbacks.

That doesn’t do the v2v3 conversion, though, which is needed to make sure that the xml2rfc formatter behind the submission system is run in RFCXMLv3 mode.

Maybe it would be useful to have an option that does all that is needed for trouble-free I-D submission, like --preptool does now (except that it does too much for the RFC editor, which is however mostly not the target for an I-D submission).

Once that is in place, the legend:

> For v3 .xml sources which include external files, this requires them to be run through xml2rfc with the --prep option, to produce a completely expanded (standalone) .xml source file for upload.

on https://datatracker.ietf.org/submit/ could be updated to point to that option.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Fri, 21 Aug 2020 14:05:56 +0200

>    https://datatracker.ietf.org/doc/bibxml3/draft-ietf-netconf-crypto-types.xml

+1 on tuning the name, but *this* might be confusing
("draft-ietf-netconf-crypto-types.xml" not being the xml source for
"draft-ietf-netconf-crypto-types" violates the principle of least surprise).

I'd prefer "reference.draft-ietf-netconf-crypto-types.xml" or
"draft-ietf-netconf-crypto-types.reference.xml" or even
"reference.I-D.draft-ietf-netconf-crypto-types.reference".

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 21 Aug 2020 14:06:14 +0200

Hi Carsten,

> That doesn’t do the v2v3 conversion, though, which is needed to make sure that the xml2rfc formatter behind the submission system is run in RFCXMLv3 mode.

Indeed, to force v2v3 conversion (if you have v2 documents), you should use
--v3 --expand.  (The --expand switch is valid for both v2 and v3 documents,
while --prep is valid only for v3, so it implies --v3.)

> Maybe it would be useful to have an option that does all that is
> needed for trouble-free I-D submission, like --preptool does now
> (except that it does too much for the RFC editor, which is however
> mostly not the target for an I-D submission).

That should be --v3 --expand, then.

>> For v3 .xml sources which include external files, this requires
>> them to be run through xml2rfc with the --prep option, to produce a
>> completely expanded (standalone) .xml source file for upload.

Right; will update.

Regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Aug 2020 14:13:59 +0200

> while --prep is valid only for v3, so it implies --v3.)

Now I’m confused, as this doesn’t seem to do the same thing as --v2v3 first and then --expand.  (I don’t understand what the differences are about, maybe it is just the order of elements, but I see a missing target for instance.)

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 21 Aug 2020 14:26:54 +0200

Hi Carsten,

On 2020-08-21 14:13, Carsten Bormann wrote:
> Now I’m confused, as this doesn’t seem to do the same thing as --v2v3
> first and then --expand. (I don’t understand what the differences are
> about, maybe it is just the order of elements, but I see a missing
> target for instance.)

Mmm.  Right.  --v3 is appropriate when invoking a v3 formatter, but not in
this case, when no formatter is involved.  Mea culpa.  I think that should
have been --v2v3 --expand.  Does that work for you in your use case?

(FWIW, I don't have a lot of test cases for v2 input with v3 functionality,
and in particular not --v2v3 in combination with other v3 processing, only
standalone and implied, i.e. with --v3 forcing a v3 formatter.)

Regards,

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 21 Aug 2020 14:44:45 +0200

> I think that should

That seems to create two output files, one expanded, one converted.

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 21 Aug 2020 14:47:50 +0200

> That seems to create two output files, one expanded, one converted.

Ugh.  Right.

Since --expand is a valid option for both v2 and v3 files, that doesn't
leave a lot of room for a one-step conversion and expansion, unless I
define a new switch to do that specifically.

Regards,

	Henrik