Subject: [Tools-discuss] Introducing reaex: a Regular Expression-Based ABNF Extractor

From: Sean Leonard <[email-masked]>
Date: Wed, 28 Sep 2016 00:16:04 -0700

Hi abnf-discuss and tools-discuss:

While working on ABNF, I felt it desirable to extract ABNF from RFCs and other documents (Internet-Drafts). Unfortunately, existing tools were not getting the job done.

Introducing reaex: Sean’s SUPER Regular Expression-Based ABNF Extractor!

This tool is a replacement for Bill Fenner’s “aex” tool. It uses regular expressions derived from ABNF itself (specifically RFC 5234 and RFC 7405) to identify the ABNF. Therefore, the tool is non-heuristic: if it is ABNF (syntactically), then it matches; if not, it will not match. This formulation leads to very consistent and predictable results. One consequence is that if the syntax of ABNF is extended then the regular expressions in the tool need to be updated; however, this is a straightforward exercise.

There is also code to output line (location) and caption comments: one could describe that code as heuristic. Emitting those comments is controlled by --linecomments (-l) and --captioncomments (-c), respectively. The program supports plain text (ASCII, UTF-8) and xml2rfc (v2, v3) input.

The tool now lives publicly at:
https://github.com/seantek/reaex

and will be released under an open source license (likely a BSD License) after ironing out any bugs. Comments and tests welcome.

Regards,

Sean

PS To test the code I wrote a shell script, extract-abnf-from-rfcs.sh, that also lives in that repository. You can use it to see how well it works for yourself. On my MacBook Pro it takes about 11 minutes to go through the entire RFC series. In total it identifies 533 RFCs as containing conforming ABNF (up through RFC 7979).

PPS The # list-extension syntax is not included by design. It turns out that # support is not really necessary for modern RFCs because RFC 7230 etc. provide the collected grammar in standard RFC 5234 notation.

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 28 Sep 2016 10:43:08 -0400

Sean Leonard <[email-masked]> writes:
> Introducing reaex: Sean's SUPER Regular Expression-Based ABNF Extractor!

The idea is certainly cool.  I assume that if there are just a few
syntax errors in otherwise correct block of ABNF, that the tool finds
most of the ABNF and the user can look back at the document to see why
the rest wasn't extracted.  I guess I'm assuming here that the tool can
tell the user *where* it got the various bits of ABNF that it extracts.

Dale

---

From: Sean Leonard <[email-masked]>
Date: Wed, 28 Sep 2016 15:34:09 -0700

> tell the user *where* it got the various bits of ABNF that it extracts.

Correct on all points.

With -l, you get lines, and optionally pages (plain text) or @anchor (XML):

   ; lines 733-786 [Pages 10-11]

   rule = foo

   ; fig-id-abnf at lines 733-786

   rule = foo

With -c, you get captions, determined using the following algorithm:
if caption, use caption. (caption = center-aligned "Figure X: YYYYY" on 
immediately succeeding lines, or similar)

else if one-line introducer, use one-line introducer. (one-line 
introducer = "   The following foo is the ABNF foo:" or similar, on 
immediately preceding line. In XML, it only matches if the preceding <t> 
is 69 characters or less, trimmed for whitespace.)

else if immediately preceding section header is found, use immediately 
preceding section header. (section header = "Appendix B.2.3.333.  
Collected ABNF for Mumble Protocol", or similar)

Here is an example:

   ; Figure 33: ABNF for the frobber parameter

   frobber-param = "hello" / "goodbye"

If both -lc are specified, you get both separated by a colon:

   ; lines 733-786 [Pages 10-11]: Figure 33: ABNF for the frobber parameter

-Sean