Subject: [ietf-calsify] ABNF intent in rfc2445bis-09

From: Ken Shoemake <[email-masked]>
Date: Fri, 14 Nov 2008 03:30:19 -0500

With regard to syntax, first some context:

Note that rfc4234 on ABNF says (section 3.1)
   ----------
   LINEAR WHITE SPACE: Concatenation is at the core of the ABNF parsing
   model.  A string of contiguous characters (values) is parsed
   according to the rules defined in ABNF.  For Internet specifications,
   there is some history of permitting linear white space (space and
   horizontal tab) to be freely and implicitly interspersed around major
   constructs, such as delimiting special characters or atomic strings.

   NOTE:

      This specification for ABNF does not provide for implicit
      specification of linear white space.

   Any grammar that wishes to permit linear white space around
   delimiters or string segments must specify it explicitly.
   ----------
One way to convey that implicit white space is not allowed is to state 
that the ABNF uses this newer rfc4234 instead of rfc2234; that's obscure 
and error prone compared to explicitly stating the intent.

So now the question:

In rfc2445, line folding is specified, but beyond that the explicit ABNF 
syntax apparently does not permit white space between tokens. For example,
  BEGIN:VEVENT
would be correct but
  BEGIN: VEVENT
would not. A robust parser will accept the latter anyway, but the spec 
might best spell out the intent: is such white space definitely allowed 
or definitely disallowed?

Also:
The IETF-Tools page (<http://tools.ietf.org/>) provides two tools for 
automatically combing through the ABNF in a specification (here, 
'draft-ietf-calsify-rfc2445bis-09'). The first extracts lines of syntax; 
it is Bill Fenner's 'aex' (<http://tools.ietf.org/abnf/>). 
Unfortunately, the current draft causes it to stumble in a few places. 
The second tool parses ABNF for errors; it is Bill's ABNF Parser, 'bap' 
(<http://tools.ietf.org/tools/bap/abnf.cgi>). It finds issues with the 
ABNF in the spec even after the obvious problems left by aex are cleaned 
up. Nevertheless, its strict read of the grammar seems to imply that 
white space between tokens is not formally allowed at present.

Is there intent to make automatic extraction and parsing of the ABNF 
work? Is there intent to clarify the permissibility of white space 
between tokens?

-- Ken Shoemake
_______________________________________________
ietf-calsify mailing list
[email-masked]
http://lists.osafoundation.org/mailman/listinfo/ietf-calsify

---

From: Bernard Desruisseaux <[email-masked]>
Date: Fri, 14 Nov 2008 09:39:02 -0500

Hi Ken,

See answer inline.

> Note that rfc4234 on ABNF says (section 3.1)

The draft already makes reference to RFC5234 (which obsoleted RFC4234):

http://tools.ietf.org/html/draft-ietf-calsify-rfc2445bis-09#section-1

  > The memo also includes a formal grammar for the content type based on
  > the Internet ABNF defined in [RFC5234].

http://tools.ietf.org/html/draft-ietf-calsify-rfc2445bis-09#section-2

  > The notation used in this memo is the ABNF notation of [RFC5234].

> up.

Please clarify which issues are reported. Bill's ABNF Parser currently 
reports "No errors during parsing". That being said, it is reporting 
multiple differences between the original and Bill's "canonical" form. 
For instance, the draft currently specifies:

   "BEGIN" ":" "VCALENDAR"

instead of:

   "BEGIN:VCALENDAR"

Is this what you're making reference to?

 > Nevertheless, its strict read of the grammar seems to imply that
> work?

I've been using Julian Reschke's "extract-artwork.xslt" transformation 
against the XML version of the draft to extract ABNF from the draft. See:

http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#extract-artwork

I was not aware of the existence of the 'aex' tool. How do you recommend 
to modify the draft such that 'aex' won't pick up the following lines by 
mistake:

   > c=US???(cn=Jim%20Dolittle)":mailto:[email-masked]
   >
   > TZID=America/New_York:19980119T020000
   >
   > FREQ=MONTHLY;BYDAY=MO,TU,WE,TH,FR;BYSETPOS=-1
   >
   > BYMINUTE=30
   >
   > FREQ=DAILY;COUNT=10;INTERVAL=2
   >
   > TZID=America/New_York:083000
   >
   > decimal = degrees + minutes/60 + seconds/3600.
   >
   > c=US???(cn=Jim%20Dolittle)":Jim Dolittle\, ABC Industries\,
   >
   > BYMONTH=1;BYDAY=SU,MO,TU,WE,TH,FR,SA
   > BYDAY=MO,WE,FR
   >
   > BYMONTHDAY=2,3,4,5,6,7,8

> Is there intent to clarify the permissibility of white space 
> between tokens?

Given that we are already making reference to RFC5234, and that our 
intent is NOT to permit linear white space around delimiters or string 
segments, what change or clarifying text would you like to see added in 
the draft?

Thanks,
Bernard

---

From: Ken Shoemake <[email-masked]>
Date: Sat, 15 Nov 2008 03:32:01 -0500

Trying reply again.

intent is NOT to permit linear white space around delimiters or string 
segments, what change or clarifying text would you like to see added in 
the draft?
In section 2, Basic Grammar and Conventions, I would revise the text to 
include a new note, and move the note on notes, as follows.
   ----------
    This memo makes use of both a descriptive prose and a more formal
    notation for defining the calendaring and scheduling format.

       Note: All indented editorial notes, such as this one, are intended
       to provide the reader with additional information.  The
       information is not essential to the building of an implementation
       conformant with this memo.  The information is provided to
       highlight a particular feature or characteristic of the memo.

    The notation used in this memo is the ABNF notation of [RFC5234].
    Readers intending on implementing the format defined in this memo
    should be familiar with this notation in order to properly interpret
    the specifications of this memo.

       Note: Spaces and horizontal tabs (formally, linear white space)
       are only allowed where explicitly stated in the notation, never
       implicitly; see section 3.1 of [RFC5234].

    All numeric values used in this memo are given in decimal notation.

    All names of properties, property parameters, enumerated property
    values and property parameter values are case-insensitive.  However,
    all other property values are case-sensitive, unless otherwise
    stated.
   ----------
    This memo makes use of both a descriptive prose and a more formal
    notation for defining the calendaring and scheduling format.

    The notation used in this memo is the ABNF notation of [RFC5234].
    Readers intending on implementing the format defined in this memo
    should be familiar with this notation in order to properly interpret
    the specifications of this memo.

    All numeric values used in this memo are given in decimal notation.

    All names of properties, property parameters, enumerated property
    values and property parameter values are case-insensitive.  However,
    all other property values are case-sensitive, unless otherwise
    stated.

       Note: All indented editorial notes, such as this one, are intended
       to provide the reader with additional information.  The
       information is not essential to the building of an implementation
       conformant with this memo.  The information is provided to
       highlight a particular feature or characteristic of the memo.
   ----------

The single new sentence provides two benefits.
  1. It states the intent here, rather than forcing the reader to sift 
through and interpret a large secondary document.
  2. It shields this document from historical interpretations of ABNF so 
there can be no question of accidental intent.

Although files will be read by programs more than humans, text formats 
frequently admit flexible use of linear white space. When going against 
convention it is best to alert readers explicitly.

  -- Ken Shoemake

---

From: Ken Shoemake <[email-masked]>
Date: Sat, 15 Nov 2008 03:33:06 -0500

Trying this reply again also.

Bernard Desruisseaux wrote:
 > I've been using Julian Reschke's "extract-artwork.xslt" 
transformation against the XML version of the draft to extract ABNF from 
the draft. See:

Will the XML version be available when the standard is approved? If so, 
then extraction from that would indeed seem easier and more reliable 
than using a text-based tool like aex. Unfortunately, I cannot seem to 
find a copy of the draft in XML format in the obvious place,
   <http://www.ietf.org/internet-drafts/>
Where should I be looking?

  -- Ken

---

From: Ken Shoemake <[email-masked]>
Date: Tue, 18 Nov 2008 05:38:39 -0500

Nigel Swinson wrote:
> But then immediately proceeding that you've included two 20+ line 
> sections supposedly where one is different to the other.  Sifting 
> through what you have changed is extremely difficult.

Before I gave the change I stated that it consisted of moving the note 
on notes and including a new note. Perhaps it was confusing because I 
gave the new text first, and did not label which was old and which new.

Still, I assumed that Bernard, who had asked "[W]hat change or 
clarifying text would you like to see added in the draft?", would have 
little difficulty with my reply.

An ancient but still useful standard for changes is much as you suggest, 
however in this case it would obscure the fact that a paragraph (the 
note on notes) was moved.

Moving that paragraph makes logical sense for two reasons:
  1. It should precede the use of the indented note convention for my 
new remark.
  2. It seems natural in the flow of exposition for it to follow the 
paragraph on formal and informal notation.

Again, the only substantive change proposed is to add one new sentence.

I am satisfied that such a note will suffice, and there is precedent. 
Among the ABNF given in rfc2445bis-09 is this:
   eventc     = "BEGIN" ":" "VEVENT" CRLF
                eventprop *alarmc
                "END" ":" "VEVENT" CRLF
Section 2.3 of [RFC5234] says that "ABNF strings are case insensitive", 
so implicitly we know that "begin:vevent" or "Begin:vEvent" are to be 
accepted along with "BEGIN:VEVENT". Yet rfc2445bis-09 finds it prudent 
to explicitly say "All names of properties, property parameters, 
enumerated property values and property parameter values are 
case-insensitive."

Adding the comment about white space is also prudent, as I hope we agree.

  -- Ken Shoemake

---

From: Bernard Desruisseaux <[email-masked]>
Date: Wed, 19 Nov 2008 20:15:13 -0500

Hi Ken,

I finally submitted the XML version of the draft to the IETF.
It is now available here:

http://tools.ietf.org/wg/calsify/draft-ietf-calsify-rfc2445bis/draft-ietf-calsify-rfc2445bis-09.xml

It is also accessible from the "Calsify Status Pages" here:

http://tools.ietf.org/wg/calsify/draft-ietf-calsify-rfc2445bis/

Cheers,
Bernard