Subject: [xml2rfc] #393 (Version 2 cli): Core dump on Travis

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 10:12:46 -0000

#393: Core dump on Travis

 https://travis-ci.org/cbor-wg/cddl/builds

 Build 66 to 70 ended in core dumps.

 {{{
 xml2rfc -q draft-ietf-cbor-cddl.xml -o draft-ietf-cbor-cddl.htmltmp --html
 make: *** [draft-ietf-cbor-cddl.htmltmp] Segmentation fault (core dumped)
 }}}

-- 
---------------------------+----------------------------------
 Reporter:  [email-masked]   |      Owner:  [email-masked]
     Type:  defect         |     Status:  new
 Priority:  blocker        |  Milestone:
Component:  Version 2 cli  |    Version:  2.17.*
 Keywords:                 |
---------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 10:27:50 -0000

Comment (by [email-masked]):

 Good to have the report, but it's doubtful whether it's possible to get
 further without having the dump.

 FWIW, every time I've come across a python program leading to a core dump,
 and been able to trace it down, it's been a problem in a directly or
 indirectly called binary lib, and the remedy has been to avoid the lib
 version that has the problem.  If you're able to get lists of libs on the
 CI system that changed version between the last successful run and the
 failed run, it might tell us something.

-- 
----------------------------+----------------------------------
Resolution:                 |   Keywords:
----------------------------+----------------------------------

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:1>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 10:41:41 -0000

Comment (by [email-masked]):

 I have a reproducible crash on Cygwin (quick draft repo):
 {{{
 cat draft-ietf-quic-qpack.md | sed -e 's/{DATE}/2019-02-13/g' | sed -e 's
 /draft-ietf-quic-http-latest/draft-ietf-quic-http/g; s/draft-ietf-quic-
 invariants-latest/draft-ietf-quic-invariants/g; s/draft-ietf-quic-
 recovery-latest/draft-ietf-quic-recovery/g; s/draft-ietf-quic-spin-exp-
 latest/draft-ietf-quic-spin-exp/g; s/draft-ietf-quic-tls-latest/draft-
 ietf-quic-tls/g; s/draft-ietf-quic-transport-latest/draft-ietf-quic-
 transport/g;' | kramdown-rfc2629 > draft-ietf-quic-qpack.xml
 xml2rfc -q draft-ietf-quic-qpack.xml -o draft-ietf-quic-qpack.htmltmp
 --html
 make: *** [lib/main.mk:80: draft-ietf-quic-qpack.htmltmp] Segmentation
 fault (Speicherauszug erstellt)
 rm draft-ietf-quic-qpack.xml
 }}}

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:2>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 10:52:38 -0000

Still, it's a start.  If it's a full dump in a format that gdb can read,
 it should be helpful, and it might be possible to pinpoint the offending
 binary.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:3>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Feb 2019 12:20:26 +0100

> further without having the dump.

Right.  I was wondering how to get the I-D makefile/template to save the core…
Maybe we’ll need to construct a leaner test case.

> version that has the problem.  

Right.  The libs are visible in the build log:

{{{
Collecting xml2rfc
  Downloading https://files.pythonhosted.org/packages/94/11/991783a57d55c1c462ff82a59143b940b3dbde3f9a44a940f5f93edea880/xml2rfc-2.18.0.tar.gz (3.6MB)
Collecting google-i18n-address>=2.3.2 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/3a/a8/ab3b57025039480a739e348b088b89e897c08babf3644bc31497dbc49103/google_i18n_address-2.3.4-py2.py3-none-any.whl (772kB)
Collecting html5lib>=1.0.1 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
Collecting intervaltree!=3.0.0,>=2.1.0 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/e8/f9/76237755b2020cd74549e98667210b2dd54d3fb17c6f4a62631e61d31225/intervaltree-3.0.2.tar.gz
Collecting lxml>=2.2.8 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/79/85/121bd54a83d0183c7ccbb8f827f5e96fb38c0066f2440782f638367db4b8/lxml-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)
Collecting pycountry>=1.8 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/f8/39/04d8215fddd5b263beb36cc324e5656714a19081cacb2fc7c8e6afc2817d/pycountry-18.12.8-py2.py3-none-any.whl (10.5MB)
Collecting pyflakes>=0.8.1 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/16/3b/b6a508ad148ce1ef50bd7a9a783afbb8d775616fc4ae5e3007c8815a3c85/pyflakes-2.1.0-py2.py3-none-any.whl (62kB)
Collecting requests>=2.5.0 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
Requirement already satisfied: setuptools>=18.0 in /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages (from xml2rfc)
Requirement already satisfied: six>=1.4.1 in /home/travis/virtualenv/python2.7.14/lib/python2.7/site-packages (from xml2rfc)
Collecting webencodings (from html5lib>=1.0.1->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting sortedcontainers<3.0,>=2.0 (from intervaltree!=3.0.0,>=2.1.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/13/f3/cf85f7c3a2dbd1a515d51e1f1676d971abe41bba6f4ab5443240d9a78e5b/sortedcontainers-2.1.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting idna<2.9,>=2.5 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Building wheels for collected packages: xml2rfc, intervaltree
  Running setup.py bdist_wheel for xml2rfc: started
  Running setup.py bdist_wheel for xml2rfc: finished with status 'done'
  Stored in directory: /home/travis/.cache/pip/wheels/c5/86/11/481dd67b334f4398cfd672c588091cda561677f91f09e2238a
  Running setup.py bdist_wheel for intervaltree: started
  Running setup.py bdist_wheel for intervaltree: finished with status 'done'
  Stored in directory: /home/travis/.cache/pip/wheels/08/99/c0/5a5942f5b9567c59c14aac76f95a70bf11dccc71240b91ebf5
Successfully built xml2rfc intervaltree
Installing collected packages: certifi, chardet, idna, urllib3, requests, google-i18n-address, webencodings, html5lib, sortedcontainers, intervaltree, lxml, pycountry, pyflakes, xml2rfc
Successfully installed certifi-2018.11.29 chardet-3.0.4 google-i18n-address-2.3.4 html5lib-1.0.1 idna-2.8 intervaltree-3.0.2 lxml-4.3.1 pycountry-18.12.8 pyflakes-2.1.0 requests-2.21.0 sortedcontainers-2.1.0 urllib3-1.24.1 webencodings-0.5.1 xml2rfc-2.18.0
}}}

> If you're able to get lists of libs on the

I don’t have a good set of specimens, but e.g. https://travis-ci.org/core-wg/senml-etch/builds/492386161 went through just 14 hours ago and used this set:

{{{
Collecting xml2rfc
  Downloading https://files.pythonhosted.org/packages/94/11/991783a57d55c1c462ff82a59143b940b3dbde3f9a44a940f5f93edea880/xml2rfc-2.18.0.tar.gz (3.6MB)
Collecting google-i18n-address>=2.3.2 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/3a/a8/ab3b57025039480a739e348b088b89e897c08babf3644bc31497dbc49103/google_i18n_address-2.3.4-py2.py3-none-any.whl (772kB)
Collecting html5lib>=1.0.1 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl (117kB)
Collecting intervaltree!=3.0.0,>=2.1.0 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/e8/f9/76237755b2020cd74549e98667210b2dd54d3fb17c6f4a62631e61d31225/intervaltree-3.0.2.tar.gz
Collecting lxml>=2.2.8 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/79/85/121bd54a83d0183c7ccbb8f827f5e96fb38c0066f2440782f638367db4b8/lxml-4.3.1-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)
Collecting pycountry>=1.8 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/f8/39/04d8215fddd5b263beb36cc324e5656714a19081cacb2fc7c8e6afc2817d/pycountry-18.12.8-py2.py3-none-any.whl (10.5MB)
Collecting pyflakes>=0.8.1 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/16/3b/b6a508ad148ce1ef50bd7a9a783afbb8d775616fc4ae5e3007c8815a3c85/pyflakes-2.1.0-py2.py3-none-any.whl (62kB)
Collecting requests>=2.5.0 (from xml2rfc)
  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
Requirement already satisfied: setuptools>=18.0 in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from xml2rfc) (40.6.3)
Requirement already satisfied: six>=1.4.1 in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from xml2rfc) (1.11.0)
Collecting webencodings (from html5lib>=1.0.1->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
Collecting sortedcontainers<3.0,>=2.0 (from intervaltree!=3.0.0,>=2.1.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/13/f3/cf85f7c3a2dbd1a515d51e1f1676d971abe41bba6f4ab5443240d9a78e5b/sortedcontainers-2.1.0-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting idna<2.9,>=2.5 (from requests>=2.5.0->xml2rfc)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
Requirement already satisfied: certifi>=2017.4.17 in /home/travis/virtualenv/python2.7.15/lib/python2.7/site-packages (from requests>=2.5.0->xml2rfc) (2018.10.15)
Building wheels for collected packages: xml2rfc, intervaltree
  Running setup.py bdist_wheel for xml2rfc: started
  Running setup.py bdist_wheel for xml2rfc: finished with status 'done'
  Stored in directory: /home/travis/.cache/pip/wheels/c5/86/11/481dd67b334f4398cfd672c588091cda561677f91f09e2238a
  Running setup.py bdist_wheel for intervaltree: started
  Running setup.py bdist_wheel for intervaltree: finished with status 'done'
  Stored in directory: /home/travis/.cache/pip/wheels/08/99/c0/5a5942f5b9567c59c14aac76f95a70bf11dccc71240b91ebf5
Successfully built xml2rfc intervaltree
Installing collected packages: urllib3, chardet, idna, requests, google-i18n-address, webencodings, html5lib, sortedcontainers, intervaltree, lxml, pycountry, pyflakes, xml2rfc
Successfully installed chardet-3.0.4 google-i18n-address-2.3.4 html5lib-1.0.1 idna-2.8 intervaltree-3.0.2 lxml-4.3.1 pycountry-18.12.8 pyflakes-2.1.0 requests-2.21.0 sortedcontainers-2.1.0 urllib3-1.24.1 webencodings-0.5.1 xml2rfc-2.18.0
}}}

The main difference I see is python 2.7.15 in the one that worked and 2.7.14 in the one that crashed, and the unbundling of certifi.

Grüße, Carsten

> _______________________________________________
> xml2rfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Feb 2019 12:25:23 +0100

Hmm, Ari (whose build works) is using xenial (Ubuntu 16) in his .travis.yml, while I’m still using trusty (Ubuntu 14).  Upgrading .travis.yml now…

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Feb 2019 12:31:34 +0100

> Hmm, Ari (whose build works) is using xenial (Ubuntu 16) in his .travis.yml, while I’m still using trusty (Ubuntu 14). Upgrading .travis.yml now…

That was not it.

https://travis-ci.org/cbor-wg/cddl/builds/492649664

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 13 Feb 2019 12:44:43 +0100

On 2019-02-13 12:20, Carsten Bormann wrote:
> The main difference I see is python 2.7.15 in the one that worked and
> 2.7.14 in the one that crashed, and the unbundling of certifi.

Both those seem like possibles, although the level of testing a Python
release goes through make me suspect certifi more.

I'll upgrade python here and see if that lets me reproduce this.

	Henrik

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 12:14:15 -0000

I don't see anything except the dump I inlined above.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:4>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 13 Feb 2019 13:19:22 +0100

Ok, thanks.

	Henrik

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 13:30:28 -0000

Thanks.  Using that with python 2.7.13 works fine.

 I've just built Python 2.7.15 from source, and using that I get a
 segmentation fault here.

 Can you tell the CI framework to avoid Python 2.7.15?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:5>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 13:35:32 -0000

FWIW, I'm on 2.7.14.

Ticket URL: <https://tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:6>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Feb 2019 14:38:57 +0100

FWIW, 2.7.15 works for me on my Mac (10.13.6).
(I’m normally using Python 3.)

Grüße, Carsten

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 13:40:02 -0000

I've just generated a core dump from the segmentation fault, and it turns
 out that although it's triggered under Python 2.7.15, the segmentation
 fault is in the lxml 'etree.so' lib; so not the Python executable itself.
 Avoiding Python 2.7.14 and 2.7.15 for now if possible should still let you
 get past this.

 I'll spend some time later this week to see if this is triggered only for
 particular versions of lxml (I'm on lxml 4.3.1 here when this is
 triggered).

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:7>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 13:41:42 -0000

That reminds me that I just recently updated lxml, as xml2rfc politely
 asked me to do so (WRT XPATH locations in validation errors). So this is
 probably the reason why I started to see these.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:8>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 13 Feb 2019 14:58:35 +0100

> Avoiding Python 2.7.14 and 2.7.15 for now if possible should still let you

Just tried Python 3.7.1.  Core dump.

https://travis-ci.org/cbor-wg/cddl/builds/492709063

Grüße, Carsten

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 13 Feb 2019 15:09:52 +0100

Ugh. Then it's a problem with a recent lxml. Ok, I'll work backwards towards a functioning lxml now that I can reproduce this.

Best regards,

        Henrik

> On 13 Feb 2019, at 14:58, Carsten Bormann <[email-masked]> wrote:
>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 14:16:38 -0000

Wild guess:
 https://github.com/lxml/lxml/commit/201b712edf0478e6a94ace984c1e8435bf3bc3c3

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:9>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 15:10:05 -0000

Right.  Downgrading lxml from 4.3.1 to 4.3.0 here removes the issue.  I'll
 disallow 4.3.1 in the next release.

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:10>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 15:13:23 -0000

Changes (by [email-masked]):

 * status:  new => closed
 * resolution:   => fixed

Comment:

 Fixed in [2994]:

 Disallow lxml 4.3.1, as it can cause segfaults with some Python versions.
 Fixes issue #393.

-- 
----------------------------+----------------------------------
  Reporter:  [email-masked]   |      Owner:  [email-masked]
      Type:  defect         |     Status:  closed
  Priority:  blocker        |  Milestone:
 Component:  Version 2 cli  |    Version:  2.17.*
Resolution:  fixed          |   Keywords:
----------------------------+----------------------------------

Ticket URL: </ticket/393#comment:11>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 13 Feb 2019 16:14:39 +0100

Yes, the issue seems to be lxml==4.3.1.  Downgrading to 4.3.0 avoids the
segfault here.

	Henrik

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 15:16:19 -0000

Actually, it seems you need to disallow 4.3.0. 4.3.1 fixes the issue, no?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:12>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>

---

From: "xml2rfc issue tracker" <[email-masked]>
Date: Wed, 13 Feb 2019 15:25:02 -0000

Not so in my tests.  With lxml 4.3.1 under Python 2.7.15 I get segfaults,
 with 4.3.0 I don't.  Are you sure that the lxml commit you pointed at
 (reasonable guess) was included in 4.3.1?  It could be slated for the next
 release, maybe?

Ticket URL: <https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/393#comment:13>
xml2rfc <http://tools.ietf.org/tools/xml2rfc/>