Subject: [Tools-discuss] Looking for a Python-based parser for rfc-index.xml

From: Paul Hoffman <[email-masked]>
Date: Wed, 13 Jul 2011 17:19:15 -0700

Greetings again. I would like to extend the "ietf command-line tool" functionality to include knowing when an RFC has been obsoleted. I could write a parser for rfc-index.xml, but I would guess at least two or three of you have already done so. Free code would be appreciated.

--Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Jul 2011 09:50:16 +0200

> Greetings again. I would like to extend the "ietf command-line tool" functionality to include knowing when an RFC has been obsoleted. I could write a parser for rfc-index.xml, but I would guess at…

If you can call an XML parser plus either an XPath or XSLT 
implementation it should be dead simple...

Best regards, Julian

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 14 Jul 2011 09:58:22 -0700

On Jul 14, 2011, at 12:50 AM, Julian Reschke wrote:

> If you can call an XML parser plus either an XPath or XSLT implementation it should be dead simple...

For should value of "should", yes. I am hoping to not waste time discovering problems with either of them if others have already faced that pain.

--Paul Hoffman

---

From: Alice Hagens <[email-masked]>
Date: Thu, 14 Jul 2011 13:15:50 -0400

Although it's not what you're asking about, just fyi, there exists a Perl script rfc-what-i-mean by Joe Touch (available from http://www.isi.edu/touch/tools/), which uses rfc-index.txt to output a list of RFCs that are mentioned in the input file and that have been updated or obsoleted.

# output = RFCs you really mean in two sets:
# 	first set: one line each, your RFC and its 
#	appropriate successor(s), including updates:
#	a single line "RFCS TO REPLACE:"
#		rfc2255 -> rfc4516
#		rfc2798 -> rfc3698, rfc4517
#		rfc3698 -> rfc3698, rfc4517
#	a single line of "RFCS YOU MEAN:"
#	second set: one line each, of the sorted, 
#	unique set of RFCs you want:
#		rfc3698
#		rfc4516
#		rfc4517

Thanks,
Alice
RFC Production Center

On Jul 13, 2011, at 8:19 PM, Paul Hoffman wrote:

> _______________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
>

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 14 Jul 2011 10:25:59 -0700

> Although it's not what you're asking about, just fyi, there exists a Perl script rfc-what-i-mean by Joe Touch (available from http://www.isi.edu/touch/tools/), which uses rfc-index.txt to output a…

Thanks, but it is not what I am interested in. :-) I want Python (even though I wrote _Perl For Dummies_), and I want to go directly from the source database, rfc-index.xml.

FWIW, I will be producing output like Joe's, but in addition be acting on some of the results in the IETF cli tool.

--Paul Hoffman

---

From: Nico Williams <[email-masked]>
Date: Thu, 14 Jul 2011 14:27:33 -0500

> FWIW, I will be producing output like Joe's, but in addition be acting on some of the results in the IETF cli tool.

from xml.dom.minidom import parse, parseString
d = parse('rfc-index.xml')
docids = d.getElementsByTagName("doc-id")
for docid in docids:
    print docid.childNodes[0].data

That should get you a list of RFC/BCP/FYI/STD numbers.  With a bit
more work you can get exactly what you're after.

Nico
--

---

From: Paul Hoffman <[email-masked]>
Date: Thu, 14 Jul 2011 18:58:21 -0700

I ended up just powering through it myself, with a dollop of help from Dive Into Python 3.

In case this is of use to folks:

from xml.etree import ElementTree as ElTree
. . .
RFCZerosPat = re.compile(r'^0+(.*)')
. . .
TagBase = "{http://www.rfc-editor.org/rfc-index}"
try:
	ParsedRFCDB = ElTree.parse("rfc-index.xml")
except:
	exit("Weird: could not find '" + "rfc-index.xml" + "' when building the status index. Exiting.")
TreeRoot = ParsedRFCDB.getroot()
RFCStatus = {}
def StripLeadingZeros(InStr):
	return(sub(RFCZerosPat, "\\1", InStr))
for ThisTopNode in TreeRoot:
	if ThisTopNode.tag == TagBase + "rfc-entry":
		ThisRFCNum = StripLeadingZeros(ThisTopNode.find(TagBase + "doc-id").text.replace("RFC", ""))
		RFCStatus[ThisRFCNum] = {}
		for ThisLookedFor in ("obsoletes", "obsoleted-by", "updates", "updated-by"):
			if ThisTopNode.findall(TagBase + ThisLookedFor):
				RFCStatus[ThisRFCNum][ThisLookedFor] = []
				for ThisFoundItem in ThisTopNode.findall(TagBase + ThisLookedFor):
					RFCStatus[ThisRFCNum][ThisLookedFor].append(StripLeadingZeros(ThisFoundItem.find(TagBase + "doc-id").text.replace("RFC", "")))
		if ThisTopNode.findall(TagBase + "draft"):
			RFCStatus[ThisRFCNum]["draft"] = ThisTopNode.find(TagBase + "draft").text
		if ThisTopNode.findall(TagBase + "errata-url"):
			RFCStatus[ThisRFCNum]["errata"] = True

I then pickle RFCStatus for later use because thawing it is much, much faster than parsing the XML. (List comprehensions might shorten the code a bit, but I don't mind verbosity.)

--Paul Hoffman