Subject: [Tools-discuss] xml2rfc failing against lxml 3.3.1

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 25 Feb 2014 12:15:55 -0500

hi IETF tools and LXML folks --

The IETF's xml2rfc tool (tested with versions 2.4.5 and 2.4.3) is
failing for me when used with python's lxml module version 3.3.1.
for example:

======================================================================
ERROR: test_header_footer (__main__.WriterDraftTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 264, in setUp
    self.parse('tests/input/draft_root.xml')
  File "./test.py", line 222, in parse
    self.xmlrfc = self.parser.parse()
  File "/home/dkg/src/xml2rfc/xml2rfc/xml2rfc/parser.py", line 408, in parse
    context.resolvers.add(caching_resolver)
AttributeError: 'lxml.etree.iterparse' object has no attribute 'resolvers'

This was not a problem when i used lxml 3.2.0.

I haven't dived deep into lxml yet, but looking in the upstream
changelog at https://github.com/lxml/lxml/blob/master/CHANGES.txt, i
see:

  * The externally useless class ``lxml.etree._BaseParser`` was removed
    from the module dict.

and lxml.etree._BaseParser does have a "resolvers" attribute.

So if i'm interpreting this right, either lxml upstream was mistaken
about the idea that _BaseParser is externally useless (because it's
being used by xml2rfc), or xml2rfc needs to find another way to set up
its caching resolvers.

What should be done here?

(i've only just now tried to subscribe to the lxml mailing list and
their subscription confirmation hasn't come through;, so i'm not sure
this message will make it to their list.  But i'm happy to try to follow
up more with the lxml elsewhere if there are suggestions for a better
way to get in touch, and if we determine that this isn't something we
can fix cleanly in xml2rfc)

Regards,

     --dkg

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 25 Feb 2014 18:35:36 +0100

Hi Daniel,

> failing for me when used with python's lxml module version 3.3.1.

Correct.  This is a known bug, and setup.py for xml2rfc specifies a lower
version of lxml than the one that fails.

Unfortunately, at least some of the installation tools which read setup.py
fail to process this properly; in particular pip versions lower than 1.4
fail to stop or downgrade lxml if a version higher than the required one
has been installed and is present in pip's build cache.

For the next version, I plan to see if I can work around this, so xml2rfc
again works with recent lxml versions, but till then I can't suggest anything
but upgrading pip and then downgrading lxml to a release lower than 3.3.x.

Best regards,

	Henrik

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Tue, 25 Feb 2014 14:19:13 -0500

I just got a response (forwarded here) from Stefan Behnel on the lxml
mailing list suggesting that this API breakage wasn't intentional for
3.3.x, and it will be fixed in 3.3.2.

sounds like the long-term plan is to move away from this setup, though,
and lxml upstream understands the desire for a stable API, so maybe we
can figure out a way to do a smoother transition within xml2rfc in
coordination with the lxml folks.

Regards,

   --dkg

Subject: Re: [lxml] xml2rfc failing against lxml 3.3.1
From: Stefan Behnel <[email-masked]>
To: Daniel Kahn Gillmor <[email-masked]>, python LXML <[email-masked]>
Date: Tue, 25 Feb 2014 20:12:22 +0100

Hi,

thanks for bringing this up.

> failing for me when used with python's lxml module version 3.3.1.

The relevant changelog entry is actually this:

* ``iterparse()`` was rewritten to use the new ``*PullParser``
  classes internally instead of being a parser itself.

Meaning: it no longer inherits from _BaseParser (which really was
externally useless, because it couldn't be instantiated anyway).
Essentially, iterparse() being a parser was a design bug from the very
beginning.

> What should be done here?

The fact that the "resolvers" property got lost was mainly due to an
incomplete refactoring during a long, political fight regarding the future
ET/lxml compatibility of the parsers. While I'm planning to deprecate and
clean up the current overly packed interface of iterparse() at some point,
breaking backwards compatibility wasn't intended for 3.x.

Due to the unclear situation in ElementTree, the newly designed pull parser
API still isn't in a state that can completely replace the old API, so what
I did now is to restore the missing properties and methods on iterparse(),
so that they'll be available in lxml 3.3.2 again (which I was already
planning to release soonish).

https://github.com/lxml/lxml/commit/976c463249d41953c91ec3d07db7ecd9c56fc3fa

Sorry for the inconvenience.

Stefan

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 28 Feb 2014 09:32:07 +0000

> 3.3.x, and it will be fixed in 3.3.2.

3.3.2 is out not, and xml2rfc seems to work fine with it.

Lars