
From tap-bounces@ietf.org  Sun Feb  1 00:58:16 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 79DB33A67B0; Sun,  1 Feb 2009 00:58:16 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id CF1133A6994 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 00:58:15 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.311
X-Spam-Level: 
X-Spam-Status: No, score=-2.311 tagged_above=-999 required=5 tests=[AWL=0.288,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tpE-1oAmhmgt for <tap@core3.amsl.com>; Sun,  1 Feb 2009 00:58:14 -0800 (PST)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id 67E713A67B0 for <tap@ietf.org>; Sun,  1 Feb 2009 00:58:13 -0800 (PST)
Received: (qmail invoked by alias); 01 Feb 2009 08:57:53 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp009) with SMTP; 01 Feb 2009 09:57:53 +0100
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX18dGOoG5GMd9A00ZVvcca1gX9w/OJQll+njdDz3ff ogygZdgXcAb6Ll
Date: Sun, 1 Feb 2009 09:57:41 +0100
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20090201085741.GA18834@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <20090131213720.GB30031@klangraum.plasmasturm.org> <4984D8F4.1080806@pobox.com>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <4984D8F4.1080806@pobox.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Y-GMX-Trusted: 0
X-FuHaFi: 0.71
Subject: Re: [tap] [offlist] Re: Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

KiBNaWNoYWVsIEcgU2Nod2VybiA8c2Nod2VybkBwb2JveC5jb20+IFsyMDA5LTAyLTAxIDAwOjA1
XToKPiBBcmlzdG90bGUgUGFnYWx0emlzIHdyb3RlOgo+ID4gQmFzaWNhbGx5LiBJIHRoaW5rIHdl
IHNob3VsZCBkaWZmZXJlbnRpYXRlIGJldHdlZW4KPiA+IHN5bnRhY3RpY2FsIGFuZCBzZW1hbnRp
YyBlcnJvcnMuCj4KPiBBc3N1bWluZyBhIHNlbWFudGljIGVycm9yIGlzIHRyZWF0ZWQgYXMgYSBm
YWlsdXJlLCBJJ2xsIHRha2UKPiB0aGF0IGFzIGEgeWVzLgoKWWVzLiA6LSkKCkkganVzdCBkaWRu
4oCZdCBxdWl0ZSBsaWtlIHRoZSDigJxmYWlsdXJlLXdpdGgtd2FybmluZ+KAnSB0ZXJtOiBpdApm
ZWVscyBhIGJpdCB0b28gcHJlc2NyaXB0aXZlLiBJbiBwcmFjdGljZSBtb3N0IHByb2Nlc3NvcnMg
d2lsbAp0cmVhdCBzZW1hbnRpYyBlcnJvcnMgdGhhdCB3YXksIGJ1dCBJ4oCZZCBwcmVmZXIgdG8g
ZG8gdGhlIHNhbWUgYXMKdGhlIEF0b20gV0csIHdoZXJlIHdlIHNhaWQgd2UganVzdCB0ZWxsIHlv
dSAqZXhhY3RseSogd2hhdCB0aGUKYml0cyBtZWFuIGJ1dCB3aGF0IHRvIGRvIHdpdGggdGhlbSBp
cyB1cCB0byB5b3UuCgpBbHRob3VnaCBJIGd1ZXNzIOKAnGZhaWx1cmXigJ0gY2FuIGJlIHNlZW4g
YXMgbWVyZWx5IGEgbWVhbmluZyBmb3IKdGhlIHN0cmVhbSBhcyBhIHdob2xlLCBpbiB3aGljaCBj
YXNlIOKAnGZhaWx1cmUgd2l0aCB3YXJuaW5n4oCdIHdvdWxkCmJlIGFub3RoZXIsIGluIHdoaWNo
IGNhc2UgdGhpcyBhbGwgd291bGQgYmUgbWVyZSBuaXRwaWNraW5nLiBJCmRvbuKAmXQgY2FyZSBh
Ym91dCB0aGUgcHJlY2lzZSB0ZXJtaW5vbG9neSB0aG91Z2gsIGFzIGxvbmcgYXMgdGhlCnByaW5j
aXBsZSBpcyBrZXB0IGluIG1pbmQ6IGRvbuKAmXQgcHJlc2NyaWJlIHByb2Nlc3NvciBiZWhhdmlv
dXIsCmp1c3QgZGVzY3JpYmUgZG9jdW1lbnQgbWVhbmluZy4KClJlZ2FyZHMsCi0tIApBcmlzdG90
bGUgUGFnYWx0emlzIC8vIDxodHRwOi8vcGxhc21hc3R1cm0ub3JnLz4KX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KdGFwIG1haWxpbmcgbGlzdAp0YXBAaWV0
Zi5vcmcKaHR0cHM6Ly93d3cuaWV0Zi5vcmcvbWFpbG1hbi9saXN0aW5mby90YXAK

From tap-bounces@ietf.org  Sun Feb  1 05:37:06 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 20E9A3A68C4; Sun,  1 Feb 2009 05:37:06 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6741F3A68F0 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 05:37:05 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sMJvP54Y+cP4 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 05:37:04 -0800 (PST)
Received: from web65707.mail.ac4.yahoo.com (web65707.mail.ac4.yahoo.com [76.13.9.99]) by core3.amsl.com (Postfix) with SMTP id 76FCC3A687A for <tap@ietf.org>; Sun,  1 Feb 2009 05:37:04 -0800 (PST)
Received: (qmail 21726 invoked by uid 60001); 1 Feb 2009 13:36:45 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; b=c4fEVn9VJ49zC2/7dPs86ap77dqx6+l7J9ynmDtHaaZ/RYeDY+yp7yYdX6gycHAkw8zD4OhOEE22bw2mSv8guLVXDYrh6qO2q0+7X9ZHPX6Yue3heNkBOj+JXypxmrj9kmFNHnUy0Aunk08+GNT1ptlLrSeYpalMxV/s7xvkk/0=;
X-YMail-OSG: b1BS61oVM1lDBIYu1.FB9r7V99ZKyVjxhs2bo5PEvxmsBgcnCgOFXlWDySiTZe7tkkajNX5zRsnSiTr7eLKvfe.poMvIotdP9AO4oMOkoTIaaqxRsPB4o2iAQjIQ6tgKxA1xTXch9r0fwep8D5sNvYZxaiBNNWEOuPYoSBoq4HQpz83VWNU2LhVowcX4unAjh5rVpTogROa_4PDCTQ--
Received: from [82.153.209.174] by web65707.mail.ac4.yahoo.com via HTTP; Sun, 01 Feb 2009 05:36:45 PST
X-Mailer: YahooMailRC/1155.45 YahooMailWebService/0.7.260.1
References: <4984B200.6060907@pobox.com>
Date: Sun, 1 Feb 2009 05:36:45 -0800 (PST)
From: Ovid <publiustemp-tapx@yahoo.com>
To: Michael G Schwern <schwern@pobox.com>, tap@ietf.org
MIME-Version: 1.0
Message-ID: <439925.21633.qm@web65707.mail.ac4.yahoo.com>
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

----- Original Message ----

> From: Michael G Schwern <schwern@pobox.com>

> In Oslo, when we came across an edge case that was considered nonsense, the
> tendency was to say that its a parse error.  For example, "not ok 4 # SKIP",
> because why would you fail a test that didn't run?  Also "1..0" without a
> skip, because why would you not run any tests and yet not skip the test?

Your arguments made sense, but I have a question:  what does 'not ok 4 # SKIP' even mean?  To my mind, that's a parse error because something has gone horribly wrong or the person writing the TAP producer is terribly confused.  There's no way the parser can understand what the intent is, so making it a parse error makes sense to me because if *I* can't parse it, how can a TAP consumer?

I know I'm the odd man out here and I don't feel strongly enough about this to fight it :)

 
Cheers,
Ovid
--
Buy the book         - http://www.oreilly.com/catalog/perlhks/
Tech blog            - http://use.perl.org/~Ovid/journal/
Twitter              - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
_______________________________________________
tap mailing list
tap@ietf.org
https://www.ietf.org/mailman/listinfo/tap

From tap-bounces@ietf.org  Sun Feb  1 12:28:57 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 43FE33A6407; Sun,  1 Feb 2009 12:28:57 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5B3B63A67BD for <tap@core3.amsl.com>; Sun,  1 Feb 2009 12:28:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nZbFQ+l7Rk+L for <tap@core3.amsl.com>; Sun,  1 Feb 2009 12:28:55 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id 5516E3A6407 for <tap@ietf.org>; Sun,  1 Feb 2009 12:28:54 -0800 (PST)
Received: from ss5 by h132974.serverkompetenz.net with local (Exim 4.63) (envelope-from <ss5@renormalist.net>) id 1LTivg-0000Md-DY for tap@ietf.org; Sun, 01 Feb 2009 21:28:32 +0100
To: tap@ietf.org
References: <497A77D0.8070500@pobox.com> <335014.20536.qm@web65715.mail.ac4.yahoo.com> <20090124115427.GC9114@pjcj.net> <87vds4podb.fsf@renormalist.net> <497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net> <4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net> <49850F8C.4020904@pobox.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Sun, 01 Feb 2009 21:28:32 +0100
In-Reply-To: <49850F8C.4020904@pobox.com> (Michael G. Schwern's message of "Sat, 31 Jan 2009 18:57:16 -0800")
Message-ID: <87eiyhvr4f.fsf@renormalist.net>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.19 (linux)
MIME-Version: 1.0
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

Michael G Schwern <schwern@pobox.com> writes:
> Steffen Schwigon wrote:
>> So it already flip-flop'ed in the short history
>> 
>>  TH2early (not stripped),
>>  TH2late  (stripped),
>>  TH3/TP   (not stripped),
>> 
>> ?
>
> Yep, that's about right.  Nothing before TH2 did anything at all
> with the description.

(I'm dreaming about TAP... my thoughts drip out incrementally...)

I'm getting unsure again because: It could also mean that not
stripping was ok and the stripping was an experiment that was reverted
with some reason with TH3. I don't know. Who doeas? $Ovid? @Andy? 
$You?


>> I agree then that stripping the dash from description might not be
>> so end-of-world as I initially thought.
>> 
>> I'm maybe one of the very few who currently depend on the exact
>> descriptions but I can probably even change my evaluation tools.
>
> Fortunately, it will be self consistent within one tool.

Hm, not neccessarily as I provide an API to evaluate the archived TAP
by descriptions. It would just work as long as I control the
evaluation environment, which I still do at the moment.


> What are you doing with them?

Archiving to track history of testruns and recognize result
changes. Probably similar to Smolder, I think, maybe more
finegrained. Therefore I care for single tests and their descriptions.

Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>
_______________________________________________
tap mailing list
tap@ietf.org
https://www.ietf.org/mailman/listinfo/tap

From tap-bounces@ietf.org  Sun Feb  1 12:42:01 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id EEBCC3A6947; Sun,  1 Feb 2009 12:42:00 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 206173A6947 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 12:41:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jjwTyX+DY8rl for <tap@core3.amsl.com>; Sun,  1 Feb 2009 12:41:58 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id A8D143A6882 for <tap@ietf.org>; Sun,  1 Feb 2009 12:41:57 -0800 (PST)
Received: from ss5 by h132974.serverkompetenz.net with local (Exim 4.63) (envelope-from <ss5@renormalist.net>) id 1LTj8J-0000NX-Lf for tap@ietf.org; Sun, 01 Feb 2009 21:41:35 +0100
To: tap@ietf.org
References: <4984B200.6060907@pobox.com> <439925.21633.qm@web65707.mail.ac4.yahoo.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Sun, 01 Feb 2009 21:41:35 +0100
In-Reply-To: <439925.21633.qm@web65707.mail.ac4.yahoo.com> (Ovid's message of "Sun, 1 Feb 2009 05:36:45 -0800 (PST)")
Message-ID: <87wsc9uby8.fsf@renormalist.net>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.19 (linux)
MIME-Version: 1.0
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

Ovid <publiustemp-tapx@yahoo.com> writes:
> ----- Original Message ----
>
>> From: Michael G Schwern <schwern@pobox.com>
>
>> In Oslo, when we came across an edge case that was considered
>> nonsense, the tendency was to say that its a parse error.  For
>> example, "not ok 4 # SKIP", because why would you fail a test that
>> didn't run?  Also "1..0" without a skip, because why would you not
>> run any tests and yet not skip the test?
>
> Your arguments made sense, but I have a question: what does 'not ok
> 4 # SKIP' even mean?  To my mind, that's a parse error because
> something has gone horribly wrong or the person writing the TAP
> producer is terribly confused.  There's no way the parser can
> understand what the intent is, so making it a parse error makes
> sense to me because if *I* can't parse it, how can a TAP consumer?

Example story:

I, for instance, mark failing results *dynamically* as "# TODO" under
cirumstances when I generically convert results from foreign test
suites into TAP. This way I keep them "ok enough" but can still
recognize the failure.

I once had the idea to use "# SKIP" to model slightly different
situations from those foreign suites.

I quickly reverted due to some amount of stupidity in it. But it's
anyway a nice niche for expressing test results.

I therefore wouldn't strongly forbid them, particularly not early in
the grammar.


Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>
_______________________________________________
tap mailing list
tap@ietf.org
https://www.ietf.org/mailman/listinfo/tap

From tap-bounces@ietf.org  Sun Feb  1 21:21:04 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 6057F3A6822; Sun,  1 Feb 2009 21:21:04 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 5DB643A6924 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 21:21:02 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.333
X-Spam-Level: 
X-Spam-Status: No, score=-2.333 tagged_above=-999 required=5 tests=[AWL=0.266,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sw5U2HrCAUZ3 for <tap@core3.amsl.com>; Sun,  1 Feb 2009 21:21:01 -0800 (PST)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id F2C1E3A6822 for <tap@ietf.org>; Sun,  1 Feb 2009 21:21:00 -0800 (PST)
Received: (qmail invoked by alias); 02 Feb 2009 05:20:40 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp026) with SMTP; 02 Feb 2009 06:20:40 +0100
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX18Zw1MAp+gZN+o2wkwOqD5OXWelYTItGfvlEkaP2k dWbsMa7GWDdqEA
Date: Mon, 2 Feb 2009 06:20:27 +0100
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20090202052027.GA12868@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <4984B200.6060907@pobox.com> <439925.21633.qm@web65707.mail.ac4.yahoo.com>
MIME-Version: 1.0
Content-Disposition: inline
In-Reply-To: <439925.21633.qm@web65707.mail.ac4.yahoo.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Y-GMX-Trusted: 0
X-FuHaFi: 0.71
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

KiBPdmlkIDxwdWJsaXVzdGVtcC10YXB4QHlhaG9vLmNvbT4gWzIwMDktMDItMDEgMTQ6NDBdOgo+
IFlvdXIgYXJndW1lbnRzIG1hZGUgc2Vuc2UsIGJ1dCBJIGhhdmUgYSBxdWVzdGlvbjogd2hhdCBk
b2VzCj4gJ25vdCBvayA0ICMgU0tJUCcgZXZlbiBtZWFuPyBUbyBteSBtaW5kLCB0aGF0J3MgYSBw
YXJzZSBlcnJvcgo+IGJlY2F1c2Ugc29tZXRoaW5nIGhhcyBnb25lIGhvcnJpYmx5IHdyb25nIG9y
IHRoZSBwZXJzb24gd3JpdGluZwo+IHRoZSBUQVAgcHJvZHVjZXIgaXMgdGVycmlibHkgY29uZnVz
ZWQuCgpZb3Ugc2VlbSB0byBoYXZlIHBhcnNlZCBpdCBzdWNjZXNzZnVsbHkgZmFyIGVub3VnaCB0
byBkaXNjb3ZlcgphIGNvbnRyYWRpY3Rpb24gaW4gaXRzIG1lYW5pbmcuIEluIHRoaXMgd2F5IGl0
IHNlZW1zIHRvIG1lIHRoYXQKYG5vdCBvayA0ICMgU0tJUGAgaXMgZGlmZmVyZW50IGZyb20gYG9z
OWFzbzJrazIxMDAjYXN6YCwgd2hpY2gKZG9lc27igJl0IG1lYW4gYW55dGhpbmcgYXQgYWxsLgoK
UmVnYXJkcywKLS0gCkFyaXN0b3RsZSBQYWdhbHR6aXMgLy8gPGh0dHA6Ly9wbGFzbWFzdHVybS5v
cmcvPgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwp0YXAg
bWFpbGluZyBsaXN0CnRhcEBpZXRmLm9yZwpodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xp
c3RpbmZvL3RhcAo=

From tap-bounces@ietf.org  Mon Feb  2 04:15:36 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 99F5F28C12C; Mon,  2 Feb 2009 04:15:36 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 342D828C13A for <tap@core3.amsl.com>; Mon,  2 Feb 2009 04:15:36 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XHgxaVGAoSWF for <tap@core3.amsl.com>; Mon,  2 Feb 2009 04:15:35 -0800 (PST)
Received: from post.hexten.net (post.hexten.net [75.127.65.154]) by core3.amsl.com (Postfix) with ESMTP id 8798328C12C for <tap@ietf.org>; Mon,  2 Feb 2009 04:15:32 -0800 (PST)
Received: from voodoo.ripley (ca93nt.hexten.net [82.152.157.85]) by post.hexten.net (Postfix) with ESMTP id E9CFE6CBC4C; Mon,  2 Feb 2009 12:15:01 +0000 (GMT)
Message-Id: <9AC77404-FBC1-4292-AC47-5B6E5D9B628E@hexten.net>
From: Andy Armstrong <andy@hexten.net>
To: Michael G Schwern <schwern@pobox.com>
In-Reply-To: <4984B200.6060907@pobox.com>
Mime-Version: 1.0 (Apple Message framework v930.3)
Date: Mon, 2 Feb 2009 12:15:00 +0000
References: <4984B200.6060907@pobox.com>
X-Mailer: Apple Mail (2.930.3)
Cc: tap@ietf.org
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"; DelSp="yes"
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

On 31 Jan 2009, at 20:18, Michael G Schwern wrote:
> Consensus?  Generally prefer failure-with-warning over parse error  
> for valid
> but sort of nonsense output?


Yeah, they're semantic rather than syntactic errors.

-- 
Andy Armstrong, Hexten



_______________________________________________
tap mailing list
tap@ietf.org
https://www.ietf.org/mailman/listinfo/tap

From tap-bounces@ietf.org  Mon Feb  2 04:15:55 2009
Return-Path: <tap-bounces@ietf.org>
X-Original-To: tap-archive@ietf.org
Delivered-To: ietfarch-tap-archive@core3.amsl.com
Received: from [127.0.0.1] (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id A97F628C118; Mon,  2 Feb 2009 04:15:55 -0800 (PST)
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4AE5128C13A for <tap@core3.amsl.com>; Mon,  2 Feb 2009 04:15:55 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9+tm-ajZJhS9 for <tap@core3.amsl.com>; Mon,  2 Feb 2009 04:15:54 -0800 (PST)
Received: from post.hexten.net (post.hexten.net [75.127.65.154]) by core3.amsl.com (Postfix) with ESMTP id 9820C28C118 for <tap@ietf.org>; Mon,  2 Feb 2009 04:15:54 -0800 (PST)
Received: from voodoo.ripley (ca93nt.hexten.net [82.152.157.85]) by post.hexten.net (Postfix) with ESMTP id 6623F6CBC3E; Mon,  2 Feb 2009 12:15:35 +0000 (GMT)
Message-Id: <F265F639-7B5E-43B6-9C0E-B3C0FE23C2B0@hexten.net>
From: Andy Armstrong <andy@hexten.net>
To: Aristotle Pagaltzis <pagaltzis@gmx.de>
In-Reply-To: <20090131213720.GB30031@klangraum.plasmasturm.org>
Mime-Version: 1.0 (Apple Message framework v930.3)
Date: Mon, 2 Feb 2009 12:15:35 +0000
References: <4984B200.6060907@pobox.com> <20090131213720.GB30031@klangraum.plasmasturm.org>
X-Mailer: Apple Mail (2.930.3)
Cc: tap@ietf.org
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/pipermail/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Sender: tap-bounces@ietf.org
Errors-To: tap-bounces@ietf.org

On 31 Jan 2009, at 21:37, Aristotle Pagaltzis wrote:
> * Michael G Schwern <schwern@pobox.com> [2009-01-31 21:20]:
>> Generally prefer failure-with-warning over parse error for
>> valid but sort of nonsense output?
>
> Basically. I think we should differentiate between syntactical
> and semantic errors.


Oops. Note to self: read all of your mail backlog before replying :)

-- 
Andy Armstrong, Hexten



_______________________________________________
tap mailing list
tap@ietf.org
https://www.ietf.org/mailman/listinfo/tap

From schwern@pobox.com  Tue Feb  3 22:13:58 2009
Return-Path: <schwern@pobox.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 3C6283A67E1 for <tap@core3.amsl.com>; Tue,  3 Feb 2009 22:13:58 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.484
X-Spam-Level: 
X-Spam-Status: No, score=-2.484 tagged_above=-999 required=5 tests=[AWL=0.115,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BpmpkdJKG+DQ for <tap@core3.amsl.com>; Tue,  3 Feb 2009 22:13:57 -0800 (PST)
Received: from sasl.smtp.pobox.com (a-sasl-fastnet.sasl.smtp.pobox.com [207.106.133.19]) by core3.amsl.com (Postfix) with ESMTP id DD80D3A6774 for <tap@ietf.org>; Tue,  3 Feb 2009 22:13:55 -0800 (PST)
Received: from localhost.localdomain (unknown [127.0.0.1]) by a-sasl-fastnet.sasl.smtp.pobox.com (Postfix) with ESMTP id 6C48F96369; Wed,  4 Feb 2009 01:13:34 -0500 (EST)
Received: from [10.23.42.2] (unknown [69.64.236.3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-sasl-fastnet.sasl.smtp.pobox.com (Postfix) with ESMTPSA id 4C1B096364; Wed, 4 Feb 2009 01:13:31 -0500 (EST)
Message-ID: <49893207.30406@pobox.com>
Date: Tue, 03 Feb 2009 22:13:27 -0800
From: Michael G Schwern <schwern@pobox.com>
User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209)
MIME-Version: 1.0
To: Steffen Schwigon <ss5@renormalist.net>
References: <497A77D0.8070500@pobox.com> <335014.20536.qm@web65715.mail.ac4.yahoo.com> <20090124115427.GC9114@pjcj.net> <87vds4podb.fsf@renormalist.net> <497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net> <4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net> <49850F8C.4020904@pobox.com> <87eiyhvr4f.fsf@renormalist.net>
In-Reply-To: <87eiyhvr4f.fsf@renormalist.net>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Pobox-Relay-ID: F3FE8C68-F282-11DD-8A48-8B21C92D7133-02258300!a-sasl-fastnet.pobox.com
Cc: tap@ietf.org
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 06:13:58 -0000

Steffen Schwigon wrote:
>>>  TH2early (not stripped),
>>>  TH2late  (stripped),
>>>  TH3/TP   (not stripped),
>>>
>>> ?
>> Yep, that's about right.  Nothing before TH2 did anything at all
>> with the description.
> 
> (I'm dreaming about TAP... my thoughts drip out incrementally...)
> 
> I'm getting unsure again because: It could also mean that not
> stripping was ok and the stripping was an experiment that was reverted
> with some reason with TH3. I don't know. Who doeas? $Ovid? @Andy? 
> $You?

An intersection of the three.

The REALLY IMPORTANT thing to realize was that so few people do anything with
descriptions except maybe print them out that very little thought has been put
into their exact parsing.  So don't read too much into what the
implementations did/do.

TH2early didn't strip the description because it was the first TAP parser that
allowed you to do anything with the parsed results so it just wasn't considered.

TH2late did strip the description because it was conforming to what Test::More
did.

TH3 removed the stripping because it wasn't considered part of the fledging
protocol which we were just then starting to write down.

But in all that we never considered the simple possibility that the dash could
be an optional element.


>>> I agree then that stripping the dash from description might not be
>>> so end-of-world as I initially thought.
>>>
>>> I'm maybe one of the very few who currently depend on the exact
>>> descriptions but I can probably even change my evaluation tools.
>> Fortunately, it will be self consistent within one tool.
> 
> Hm, not neccessarily as I provide an API to evaluate the archived TAP
> by descriptions. It would just work as long as I control the
> evaluation environment, which I still do at the moment.

Are you going to be passing interpretations of the descriptions back and forth
between different implementations?  That is, when does the interpretation of
the description matter to anything but a single implementation?


>> What are you doing with them?
> 
> Archiving to track history of testruns and recognize result
> changes. Probably similar to Smolder, I think, maybe more
> finegrained. Therefore I care for single tests and their descriptions.

Looking at it another way, the WORST thing that happens with mixing a legacy
producer which isn't aware of the dash with a post-spec, dash stripping parser
is that the description might get interpreted wrong and you can't associate a
description with it's historical results (though that would indicate that the
test started using a different producer, which stretching this scenario even
further).  Since using the description as a unique test ID is already a
heuristic (something that was discussed in some depth at Oslo) your system
must already be resistant to this problem.

Finally, if the parser always strips the optional dash it should always come
out the same whether the producer uses a leading dash or not EXCEPT for one
contrived case.  When a description deliberately is supposed to have a leading
" -".  I can only think of a fairly contrived example, but if you wanted to
output, say, Morse code "- ... -.-", and your producer doesn't know about the
optional dash, it would say:

ok 1 - ... -.-

Whereas a compliant producer would say:

ok 1 - - ... -.-

A compliant parser would parse the former as "... -.-" and the latter as "-
... -.-".  Same intended meaning, different interpreted descriptions.

Is this edge case important?  It seems one simple way to allay your fears is
to simply go through your historical TAP archives and look for instances where
a description starts with "- ".  If all the descriptions in a stream start
with "- " assume it's Test::Builder output and ignore it.  If only some of
them do, then we've found our exception.

I'm interested to hear what the results are.


-- 
94. Crucifixes do not ward off officers, and I should not test that.
    -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
           http://skippyslist.com/list/

From schwern@pobox.com  Tue Feb  3 22:58:06 2009
Return-Path: <schwern@pobox.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 9E34D3A6A47 for <tap@core3.amsl.com>; Tue,  3 Feb 2009 22:58:06 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.194
X-Spam-Level: 
X-Spam-Status: No, score=-2.194 tagged_above=-999 required=5 tests=[AWL=-0.195, BAYES_00=-2.599, J_CHICKENPOX_31=0.6]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nsIFnfoAYZtY for <tap@core3.amsl.com>; Tue,  3 Feb 2009 22:58:05 -0800 (PST)
Received: from sasl.smtp.pobox.com (a-sasl-fastnet.sasl.smtp.pobox.com [207.106.133.19]) by core3.amsl.com (Postfix) with ESMTP id 8D3903A694C for <tap@ietf.org>; Tue,  3 Feb 2009 22:58:04 -0800 (PST)
Received: from localhost.localdomain (unknown [127.0.0.1]) by a-sasl-fastnet.sasl.smtp.pobox.com (Postfix) with ESMTP id 1C81396632; Wed,  4 Feb 2009 01:57:45 -0500 (EST)
Received: from [10.23.42.2] (unknown [69.64.236.3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by a-sasl-fastnet.sasl.smtp.pobox.com (Postfix) with ESMTPSA id D649496631; Wed, 4 Feb 2009 01:57:42 -0500 (EST)
Message-ID: <49893C64.1050604@pobox.com>
Date: Tue, 03 Feb 2009 22:57:40 -0800
From: Michael G Schwern <schwern@pobox.com>
User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209)
MIME-Version: 1.0
To: Ovid <publiustemp-tapx@yahoo.com>
References: <4984B200.6060907@pobox.com> <439925.21633.qm@web65707.mail.ac4.yahoo.com>
In-Reply-To: <439925.21633.qm@web65707.mail.ac4.yahoo.com>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Pobox-Relay-ID: 1FEB290C-F289-11DD-B53A-8B21C92D7133-02258300!a-sasl-fastnet.pobox.com
Cc: tap@ietf.org
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 06:58:06 -0000

Ovid wrote:
> ----- Original Message ----
>
>> From: Michael G Schwern <schwern@pobox.com>
>
>> In Oslo, when we came across an edge case that was considered nonsense, the
>> tendency was to say that its a parse error.  For example, "not ok 4 # SKIP",
>> because why would you fail a test that didn't run?  Also "1..0" without a
>> skip, because why would you not run any tests and yet not skip the test?
>
> Your arguments made sense, but I have a question: what does 'not ok
> 4 # SKIP' even mean?  To my mind, that's a parse error because
> something has gone horribly wrong or the person writing the TAP
> producer is terribly confused.  There's no way the parser can
> understand what the intent is, so making it a parse error makes
> sense to me because if *I* can't parse it, how can a TAP consumer?
>
> I know I'm the odd man out here and I don't feel strongly enough
> about this to fight it :)

I think the important thing here is the difference between a syntax
error and a semantic error.  Let me illustrate:

"The color eight died" is syntactically correct.  There's the noun
"eight", modified by the adjective "color" and the verb "died" which
"the color eight" did.  There is meaning there to be parsed.  You can
draw a sentence diagram for it.  It is English.

"The color eight died" is a semantic error.  It is nonsense.  Eight is
not a color and numbers don't die.

You can have something which is syntactically correct but semantic
nonsense, and there is still meaning in semantic nonsense.

"not ok 4 # SKIP" is syntactically correct.  It applies to test #4.
That test failed.  The code was not run.

"not ok 4 # SKIP" is semantic nonsense.  You don't fail a test you
never ran.

Pragmatically, something has gone terribly wrong and the consumer
should flag that line.  So the desired result is still there for a
syntax error or a semantic error.  I hope that's clear, it still
results in an error.

This is, to a certain extent, splitting hairs, but that's part of what
formalizing TAP is about.  However, there are practical implementation
advantages to making it a semantic error.  Consider a grammar-based TAP
parser, rather than using ad-hoc regexes.

* Simpler grammar.

A test result line can be described as:

Test-Result [SP Test-Number] [SP Description] [SP # SP Directive [SP Reason]]

Allowing all combinations of Test-Results with Directives.  Meaning is applied
after they are parsed.

Whereas if "not ok 4 # SKIP" is made a syntax error that exception has
to be written into the grammar so we need to spell out all the combinations:

Test-Result-Line = Simple-OK / Simple-Fail / Passing-TODO /
                   Failing-TODO / Passing-SKIP
Simple-OK       = "ok" [SP Test-Number] [SP Description]        ; pass
Simple-Fail     = "not ok" [SP Test-Number] [SP Description]    ; fail
Passing-TODO    = Simple-OK   SP TODO-Directive                 ; pass
Failing-TODO    = Simple-Fail SP TODO-Directive                 ; pass
Passing-SKIP    = Simple-OK   SP SKIP-Directive                 ; pass

Which you might want to do to grammatically determine passing or failing, but
it does leave us with no basic definition for a test result line.


* All syntax errors are ignored.

Since we ignore all syntax errors, a failing skip test must be detected and
treated specially.  That means it must be valid syntax!

Failing-SKIP    = Simple-Fail   SP SKIP-Directive               ; error


* Better error reporting.

This is the most pragmatic reason.  Since "not ok 4 # SKIP" is an
actual error, as opposed to an ignored line, you're going to
have to detect it and parse it anyway to inform the user.

Consider if you see:

TAP Version 14
1..3
ok 1
not ok 2 - no database # SKIP
  ---
  file: foo.t
  line: 42
  ...
ok 3

Now, what would you rather tell the user?  If it's just a syntax
error, then all you get is:

  Syntax error at TAP stream line 4.
  Diagnostics without test at line 5.

But if you parse and understand it, you can use that information to say:

  Nonsense failing skip test "no database" at foo.t line 42.

pointing the user at the problem.  Isn't that better?


-- 
Robrt:   People can't win
Schwern: No, but they can riot after the game.

From h.m.brand@xs4all.nl  Wed Feb  4 00:07:43 2009
Return-Path: <h.m.brand@xs4all.nl>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 53E983A6B65 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 00:07:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.504
X-Spam-Level: 
X-Spam-Status: No, score=-0.504 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, HELO_EQ_NL=0.55, HOST_EQ_NL=1.545]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L7PBNG4XHG5Y for <tap@core3.amsl.com>; Wed,  4 Feb 2009 00:07:42 -0800 (PST)
Received: from smtp-vbr4.xs4all.nl (smtp-vbr4.xs4all.nl [194.109.24.24]) by core3.amsl.com (Postfix) with ESMTP id D2C4F3A6991 for <tap@ietf.org>; Wed,  4 Feb 2009 00:07:41 -0800 (PST)
Received: from pc09.procura.nl (procura.xs4all.nl [82.95.216.29]) (authenticated bits=0) by smtp-vbr4.xs4all.nl (8.13.8/8.13.8) with ESMTP id n1487F7q057105 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 4 Feb 2009 09:07:16 +0100 (CET) (envelope-from h.m.brand@xs4all.nl)
Date: Wed, 4 Feb 2009 09:07:14 +0100
From: "H.Merijn Brand" <h.m.brand@xs4all.nl>
To: Michael G Schwern <schwern@pobox.com>
Message-ID: <20090204090714.18bdc1a6@pc09.procura.nl>
In-Reply-To: <49893207.30406@pobox.com>
References: <497A77D0.8070500@pobox.com> <335014.20536.qm@web65715.mail.ac4.yahoo.com> <20090124115427.GC9114@pjcj.net> <87vds4podb.fsf@renormalist.net> <497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net> <4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net> <49850F8C.4020904@pobox.com> <87eiyhvr4f.fsf@renormalist.net> <49893207.30406@pobox.com>
X-Mailer: Claws Mail 3.7.0 (GTK+ 2.12.0; x86_64-unknown-linux-gnu)
Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwEAIAAACI8LKTAAAACXBIWXMAAABIAAAASABGyWs+AAAC JElEQVRo3u2aMY4CMQxFczZ6RItEzRm4DBINDbRUSPRInIRbsNK6+dJfezN4kokn48IaCSjysL8d e9Knoj2fr9f9/gllqQ6U9/vxWK3EdwdIEGjRIVCu18NhuxUfK46SH81+fzrdbuKPx/P5ctHQdAdI TKAgpvV6s9ntBEfXEYSGgMQzIHnuFBBjkshCNJ2KtJZ04hHNAugP8bZr3NIHhbcF0AKoK0CoaHXU LUWBIs1n+jV+Fl8CVqOApEXAwyMO/DSR4XVntoAYDR7eBjQupuYAYTMph8Rj21D4m7MChN02tpqs NSnb/KqU2oHCXu5xDCgflj/RAgBiKBIXnICzAsSjWBsTz5K4/HeXYvb8yK5lY3VGEwPi2aONKT+5 AlcxrTPOwcTiraGRChgMEKJh0bVVifGVTq6qgBiNVl8QE29EsK6VE+YJAOG2wz5AvsqUS6uqgHCA n4NGvBYpnJ64Jgg27sCtxtBk1CJIA4S/GhdWKh07QxUB48jWGhZ4jKamRRr/T8/M0AaEyctry6YB 4dTGj9iWZNs3DahES5kPCJOu0RQbF/fQOBprsB9gaO9JtPDzII9U5ySXX7AnuIt91y54AAW7rPpT LCe5gt3F+CLqr2UarGB3MXvMylWGq4+9RCx3TW1oJq1t3HPQlFs6N1fFNEB4s8dn7Ne7ACSm7TPQ I5quAWmw6qBpulHM33B0Csge4Nd8JTTYG2b1XyRe3lH8x34ABJ6aePuQ2N4AAAAASUVORK5CYII=
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by XS4ALL Virus Scanner
Cc: tap@ietf.org
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 08:07:43 -0000

On Tue, 03 Feb 2009 22:13:27 -0800, Michael G Schwern
<schwern@pobox.com> wrote:

> >> What are you doing with them?
> > 
> > Archiving to track history of testruns and recognize result
> > changes. Probably similar to Smolder, I think, maybe more
> > finegrained. Therefore I care for single tests and their descriptions.
> 
> Looking at it another way, the WORST thing that happens with mixing a legacy
> producer which isn't aware of the dash with a post-spec, dash stripping parser
> is that the description might get interpreted wrong and you can't associate a
> description with it's historical results (though that would indicate that the
> test started using a different producer, which stretching this scenario even
> further).  Since using the description as a unique test ID is already a
> heuristic (something that was discussed in some depth at Oslo) your system
> must already be resistant to this problem.
> 
> Finally, if the parser always strips the optional dash it should always come
> out the same whether the producer uses a leading dash or not EXCEPT for one
> contrived case.  When a description deliberately is supposed to have a leading
> " -".  I can only think of a fairly contrived example, but if you wanted to
> output, say, Morse code "- ... -.-", and your producer doesn't know about the
> optional dash, it would say:

It would be nice to take this example in the docs, but then use

	- .- .--.

:)

-- 
H.Merijn Brand  http://tux.nl      Perl Monger  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.3, and 11.0, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

From mpeters@plusthree.com  Wed Feb  4 06:17:57 2009
Return-Path: <mpeters@plusthree.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 99DA13A6937 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 06:17:57 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[AWL=0.000,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X3LZa0OH5ai6 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 06:17:56 -0800 (PST)
Received: from mail.plusthree.com (mail.plusthree.com [70.42.42.197]) by core3.amsl.com (Postfix) with ESMTP id B8BD23A6802 for <tap@ietf.org>; Wed,  4 Feb 2009 06:17:56 -0800 (PST)
Received: from localhost.localdomain (dsl093-241-113.ral1.dsl.speakeasy.net [66.93.241.113]) (authenticated bits=0) by mail.plusthree.com (8.14.2/8.14.2) with ESMTP id n14E8bXR010784 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 4 Feb 2009 09:08:38 -0500
Message-ID: <4989A2F7.1010304@plusthree.com>
Date: Wed, 04 Feb 2009 09:15:19 -0500
From: Michael Peters <mpeters@plusthree.com>
User-Agent: Thunderbird 2.0.0.19 (X11/20090105)
MIME-Version: 1.0
To: Michael G Schwern <schwern@pobox.com>
References: <497A77D0.8070500@pobox.com>	<335014.20536.qm@web65715.mail.ac4.yahoo.com>	<20090124115427.GC9114@pjcj.net> <87vds4podb.fsf@renormalist.net>	<497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net>	<4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net>	<49850F8C.4020904@pobox.com> <87eiyhvr4f.fsf@renormalist.net> <49893207.30406@pobox.com>
In-Reply-To: <49893207.30406@pobox.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Cc: tap@ietf.org
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 14:17:57 -0000

Michael G Schwern wrote:

> Finally, if the parser always strips the optional dash it should always come
> out the same whether the producer uses a leading dash or not EXCEPT for one
> contrived case.  When a description deliberately is supposed to have a leading
> " -".  I can only think of a fairly contrived example, but if you wanted to
> output, say, Morse code "- ... -.-", and your producer doesn't know about the
> optional dash, it would say:

What about a description that begins with a negative number? Like say:

   -1 < 0 ?

-- 
Michael Peters
Plus Three, LP


From fawaka@gmail.com  Wed Feb  4 06:36:34 2009
Return-Path: <fawaka@gmail.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 40D1C3A6BFE for <tap@core3.amsl.com>; Wed,  4 Feb 2009 06:36:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.855
X-Spam-Level: 
X-Spam-Status: No, score=-1.855 tagged_above=-999 required=5 tests=[AWL=0.745,  BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SeO0auvHEvRx for <tap@core3.amsl.com>; Wed,  4 Feb 2009 06:36:33 -0800 (PST)
Received: from fk-out-0910.google.com (fk-out-0910.google.com [209.85.128.189]) by core3.amsl.com (Postfix) with ESMTP id 4C06D3A6802 for <tap@ietf.org>; Wed,  4 Feb 2009 06:36:33 -0800 (PST)
Received: by fk-out-0910.google.com with SMTP id f33so2106217fkf.5 for <tap@ietf.org>; Wed, 04 Feb 2009 06:36:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=xn+gC1SVBMvSFFPJ+ZU3WCzhH3lNLe0C8dsxvd4NyOM=; b=XdUpUs6Pv3hiinB3cu6guTTBl5RHhe0GjF2Dea5CH8rHc3asVbTQrU/o3+Cy52cnah dw0XVmCiTDAjPrjn6qBK3BYjRS/Ryfdp2pB38EO6iJjiDpAddavVDPR2pAswd286YShK ibyso0sx7l7JQ05rax4YXKQmWU6zzpsC+GuPU=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=aKLOsJLSf4snIV8tNovgAPVvNaqG0VE2R7nHMzYX8QJDJNLOoznzjrJRvr/HguESre fbdPCwfN4jpTlV40w3+BLhfjqULn8TwNvD3uBLF2XNdQ5Oj1ObEFNMrDDbGGTZKmb06D GiXfcHKGibt+j6jfD3X9rbpXeMsVMfBrt/oMA=
MIME-Version: 1.0
Received: by 10.223.104.140 with SMTP id p12mr527835fao.7.1233758172209; Wed,  04 Feb 2009 06:36:12 -0800 (PST)
In-Reply-To: <4989A2F7.1010304@plusthree.com>
References: <497A77D0.8070500@pobox.com> <87vds4podb.fsf@renormalist.net> <497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net> <4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net> <49850F8C.4020904@pobox.com> <87eiyhvr4f.fsf@renormalist.net> <49893207.30406@pobox.com> <4989A2F7.1010304@plusthree.com>
Date: Wed, 4 Feb 2009 15:36:12 +0100
Message-ID: <a51e610c0902040636r2a77c337uff3b0254af1b527c@mail.gmail.com>
From: Leon Timmermans <fawaka@gmail.com>
To: tap@ietf.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 14:36:34 -0000

On Wed, Feb 4, 2009 at 3:15 PM, Michael Peters <mpeters@plusthree.com> wrote:
>
> What about a description that begins with a negative number? Like say:
>
>  -1 < 0 ?
>

As long as there is no space between the minus and the number, it
won't get chopped off.

Leon

From ss5@renormalist.net  Wed Feb  4 14:16:23 2009
Return-Path: <ss5@renormalist.net>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 690273A6934 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 14:16:23 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MYpA-xV3xwI4 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 14:16:22 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id BB8E23A685A for <tap@ietf.org>; Wed,  4 Feb 2009 14:16:21 -0800 (PST)
Received: from ss5 by h132974.serverkompetenz.net with local (Exim 4.63) (envelope-from <ss5@renormalist.net>) id 1LUq2J-0005rH-95 for tap@ietf.org; Wed, 04 Feb 2009 23:15:59 +0100
To: tap@ietf.org
References: <497A77D0.8070500@pobox.com> <335014.20536.qm@web65715.mail.ac4.yahoo.com> <20090124115427.GC9114@pjcj.net> <87vds4podb.fsf@renormalist.net> <497B8530.4080903@pobox.com> <873aezxgen.fsf@renormalist.net> <4984E18D.7000606@pobox.com> <87tz7fvv0h.fsf@renormalist.net> <49850F8C.4020904@pobox.com> <87eiyhvr4f.fsf@renormalist.net> <49893207.30406@pobox.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Wed, 04 Feb 2009 23:15:59 +0100
In-Reply-To: <49893207.30406@pobox.com> (Michael G. Schwern's message of "Tue, 03 Feb 2009 22:13:27 -0800")
Message-ID: <87k585c0gw.fsf@renormalist.net>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.19 (linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Subject: Re: [tap] "ok 1 description" vs "ok 1 - description"
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 22:16:23 -0000

Hi!

[Don't worry. Just explanations follow, I will agree you in the end.]

[And I write some words bold, just for better readability, not
shouting.]


Michael G Schwern <schwern@pobox.com> writes:
> Steffen Schwigon wrote:
>> Hm, not neccessarily as I provide an API to evaluate the archived
>> TAP by descriptions. It would just work as long as I control the
>> evaluation environment, which I still do at the moment.
>
> Are you going to be passing interpretations of the descriptions back
> and forth between different implementations?

No, not back and forth, but through 3 independent layers:

 1. producer,
 2. abstract API (using TAP::Parser),
 3. query tools (using the API).

I am level 2, level 1 and 3 just use me.


> That is, when does the interpretation of the description matter to
> anything but a single implementation?

It's about beeing *the description is the interface*.

I do *not* provide TAP::Parser directly but provide an abstract API
which can search for a description and let others write third-party
tools which depend on what they themself pushed in the *beginning* of
the process queue and let them also query results at the *end* of the
queue about their own descriptions.

Then they depend on my middle layer (TAP::Parser) beeing *constant*
in these descriptions throughout the years.

Because if today they query results for "- DESCRIPTION" (because they
created it with dash) and TAP::Parser changes later then they will not
find the results because it's now just "DESCRIPTION".

Changing this behaviour was what I was worrying about in the beginning
of the discussion.


>>> What are you doing with them?
>> 
>> Archiving to track history of testruns and recognize result
>> changes. Probably similar to Smolder, I think, maybe more
>> finegrained. Therefore I care for single tests and their
>> descriptions.
>
> Looking at it another way, the WORST thing that happens with mixing
> a legacy producer which isn't aware of the dash with a post-spec,
> dash stripping parser is that the description might get interpreted
> wrong and you can't associate a description with it's historical
> results

Near the truth. Indeed. It's about matching "historical results", yes.

My users own the "entry" and the "exit" of the queue. My task is to
provide the stable "middle layer".


> (though that would indicate that the test started using a different
> producer, which stretching this scenario even further).

Hm, not exactly. My problem is about starting to use a different
*middle layer* in an environment with different producers at the same
time.


> Since using the description as a unique test ID is already a
> heuristic (something that was discussed in some depth at Oslo) your
> system must already be resistant to this problem.

Fully correct. But *that* (i.e., create useful descriptions they can
later query again) is *their* responsibility, because they own "entry"
and "exit".

My layer is the middle one, therefore *my* task is about the stability
of the description.


> Finally, if the parser always strips the optional dash it should always come
> out the same whether the producer uses a leading dash or not

*EXACTLY*. That's why in the mean time I *FULLY AGREE* with you.

(I hope you made it through here. :-)

Because that would make my middle layer independent from the different
provider styles by "stabilising" or "harmonizing" the descriptions in
a defined way.

And all my reports from different producers would always be queryable
in the same way, independent from their "dash style".


> EXCEPT for one contrived case.  When a description deliberately is
> supposed to have a leading " -".  I can only think of a fairly
> contrived example, but if you wanted to output, say, Morse code

I *really* had to lough here. :-)


> Is this edge case important?

Of course. Supporting Morse descriptions is a strong MUST HAVE. :-)
No, not really.


> It seems one simple way to allay your fears is to simply go through
> your historical TAP archives and look for instances where a
> description starts with "- ".

I don't need to. For the above "stabilizing" effect I'm willing to fix
the third party tools around me to just query without dashes, once
that dash is optional. And in the mean time I need to be able to fuzzy
search anyway somehow.

Am I strange? Or is it my approach? I should probably print out this
mail and present it on the next yapc::europe. :-)


Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>

From ss5@renormalist.net  Wed Feb  4 14:29:40 2009
Return-Path: <ss5@renormalist.net>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 0D6263A68F7 for <tap@core3.amsl.com>; Wed,  4 Feb 2009 14:29:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.299
X-Spam-Level: 
X-Spam-Status: No, score=-2.299 tagged_above=-999 required=5 tests=[AWL=-0.300, BAYES_00=-2.599, J_CHICKENPOX_31=0.6]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vGViDzhr6+yb for <tap@core3.amsl.com>; Wed,  4 Feb 2009 14:29:38 -0800 (PST)
Received: from h132974.serverkompetenz.net (renormalist.net [81.169.141.46]) by core3.amsl.com (Postfix) with ESMTP id 903C83A685A for <tap@ietf.org>; Wed,  4 Feb 2009 14:29:38 -0800 (PST)
Received: from ss5 by h132974.serverkompetenz.net with local (Exim 4.63) (envelope-from <ss5@renormalist.net>) id 1LUqFB-0005wZ-HB for tap@ietf.org; Wed, 04 Feb 2009 23:29:17 +0100
To: tap@ietf.org
References: <4984B200.6060907@pobox.com> <439925.21633.qm@web65707.mail.ac4.yahoo.com> <49893C64.1050604@pobox.com>
From: Steffen Schwigon <ss5@renormalist.net>
Date: Wed, 04 Feb 2009 23:29:17 +0100
In-Reply-To: <49893C64.1050604@pobox.com> (Michael G. Schwern's message of "Tue, 03 Feb 2009 22:57:40 -0800")
Message-ID: <87ab91bzuq.fsf@renormalist.net>
User-Agent: Gnus/5.1008 (Gnus v5.10.8) XEmacs/21.4.19 (linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Subject: Re: [tap] Parse error vs failure
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Feb 2009 22:29:40 -0000

Michael G Schwern <schwern@pobox.com> writes:
> "The color eight died" is a semantic error.  It is nonsense.  Eight
> is not a color and numbers don't die.
>
> You can have something which is syntactically correct but semantic
> nonsense, and there is still meaning in semantic nonsense.

The English comparison is a good idea.

The sentence might still be called lyrics or poetry. No one would try
to disallow this in the English language just for semantic reasons.

I wonder why we cannot model the SKIP semantics similar to #TODO. With
results like "is_actual_ok" to access the raw "ok/not ok" but
interpret the line as "ok" for the stats.


> TAP Version 14
> 1..3
> ok 1
> not ok 2 - no database # SKIP
>   ---
>   file: foo.t
>   line: 42
>   ...
> ok 3
>
> Now, what would you rather tell the user?  If it's just a syntax
> error, then all you get is:
>
>   Syntax error at TAP stream line 4.
>   Diagnostics without test at line 5.

That's why we all already agreed that we do not want it a syntax
error.


> But if you parse and understand it, you can use that information to say:
>
>   Nonsense failing skip test "no database" at foo.t line 42.
>
> pointing the user at the problem.  Isn't that better?

As I said, why not just "fix" the stats, similar to what #TODO does
and provide details via "is_actual_ok" and friends?


Kind regards,
Steffen 
-- 
Steffen Schwigon <ss5@renormalist.net>
Dresden Perl Mongers <http://dresden-pm.org/>
German Perl-Workshop 2009 <http://www.perl-workshop.de/en/2009>

From schwern@pobox.com  Thu Feb 19 15:57:04 2009
Return-Path: <schwern@pobox.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id D5C4C28C232 for <tap@core3.amsl.com>; Thu, 19 Feb 2009 15:57:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NPqIB21iOL5Q for <tap@core3.amsl.com>; Thu, 19 Feb 2009 15:57:03 -0800 (PST)
Received: from sasl.smtp.pobox.com (a-sasl-quonix.sasl.smtp.pobox.com [208.72.237.25]) by core3.amsl.com (Postfix) with ESMTP id C783D28C21C for <tap@ietf.org>; Thu, 19 Feb 2009 15:57:03 -0800 (PST)
Received: from localhost.localdomain (unknown [127.0.0.1]) by b-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTP id 3A82B2B810 for <tap@ietf.org>; Thu, 19 Feb 2009 18:57:16 -0500 (EST)
Received: from windhund.local (unknown [75.150.41.177]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by b-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTPSA id B2C0B2B80F for <tap@ietf.org>; Thu, 19 Feb 2009 18:57:15 -0500 (EST)
Message-ID: <499DF1D6.9020009@pobox.com>
Date: Thu, 19 Feb 2009 15:57:10 -0800
From: Michael G Schwern <schwern@pobox.com>
User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209)
MIME-Version: 1.0
To: tap@ietf.org
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Pobox-Relay-ID: 08F33168-FEE1-11DD-A55B-6F7C8D1D4FD0-02258300!a-sasl-quonix.pobox.com
Subject: [tap] Rethinking the TAP version
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Feb 2009 23:57:04 -0000

After I was the one who pushed for a simple integer TAP version, I now want to
argue for an X.Y format.  Why?

In Oslo we went nuts trying to figure out a syntax for sub-plans that was
elegant and backwards compatible.  By "backwards compatible" I mean that the
suite passes and fails in the same way.  Diagnostic information may be lost,
and even individual test results may be lost, but you still get that critical
pass/fail across all versions.

I see this becoming more and more of a problem as time goes on.  At some
point, we're either going to want to break compatibility or the protocol will
stagnate and die.

I propose that the version number change to X.Y where X indicates the
compatibility series and Y indicates an incremental, but still compatible,
revision.  Everything that can read protocol 2.0 can read 2.1 and 2.13 and
2.99 but if a 2.x parser sees 3.1 it knows it can't handle it and can act
appropriately.  Similarly, a 2.x parser is not guaranteed to be able to read 1.x.

This gives us a lot more latitude for future change.  It enables a sane
deprecation cycle.  It allows readers to explicitly know what it can and can
not handle.  It lets us correct some of the mistakes of the past and build on
our experience for the future.

Can I get some +1s?


PS  My objection to X.Y versions has always been that what X and Y mean
changes from person to person [1], so you can't reliably convey any meaning,
and humans aren't going to check to see what it means for YOUR particular
project.  But we can define what X and Y mean in the protocol, and we have a
program to interpret it.  No ugly flesh bags need be consulted.


[1] http://use.perl.org/~schwern/journal/35127


-- 
I do have a cause though. It's obscenity. I'm for it.
    - Tom Lehrer


From pagaltzis@gmx.de  Fri Feb 20 03:50:04 2009
Return-Path: <pagaltzis@gmx.de>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 2A4B43A6B2B for <tap@core3.amsl.com>; Fri, 20 Feb 2009 03:50:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.792
X-Spam-Level: 
X-Spam-Status: No, score=-0.792 tagged_above=-999 required=5 tests=[AWL=-0.607, BAYES_40=-0.185]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tXvVYA0Zg+Kb for <tap@core3.amsl.com>; Fri, 20 Feb 2009 03:50:03 -0800 (PST)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id C52CE3A6AF3 for <tap@ietf.org>; Fri, 20 Feb 2009 03:50:02 -0800 (PST)
Received: (qmail invoked by alias); 20 Feb 2009 11:50:16 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp049) with SMTP; 20 Feb 2009 12:50:16 +0100
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX1+UeSsvN2+U3SFvE8Z7KG0laJ5ID6o7VDvS4g4Yzy IAVJzbceCm6Y4M
Date: Fri, 20 Feb 2009 12:49:47 +0100
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20090220114947.GB16059@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <499DF1D6.9020009@pobox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <499DF1D6.9020009@pobox.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Y-GMX-Trusted: 0
X-FuHaFi: 0.65
Subject: Re: [tap] Rethinking the TAP version
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Feb 2009 11:50:04 -0000

* Michael G Schwern <schwern@pobox.com> [2009-02-20 01:00]:
> I propose that the version number change to X.Y where X
> indicates the compatibility series and Y indicates an
> incremental, but still compatible, revision. Everything that
> can read protocol 2.0 can read 2.1 and 2.13 and 2.99 but if a
> 2.x parser sees 3.1 it knows it can't handle it and can act
> appropriately. Similarly, a 2.x parser is not guaranteed to be
> able to read 1.x.

It sounds good in theory, but doesn’t work out very well in
practice. It’s only a win if you want to break backcompat often,
which is not a smart idea. But if you revv that major version
only once in every decade, you effectively create a new protocol
each time, not a revision to the old one. Going from `TAP version
1.13` to `TAP version 2.0` is effectively indistinguishable from
going to `TAP2 version 1`, and you don’t need any version check
code to make consumers fail to parse future TAP versions.

(That’s how we handled it in the Atom WG too. There’s no format
version in Atom; if backwards-incompatible changes are ever
necessary, the XML namespace will change, so old consumers will
fail to process the document at all.)

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>

From schwern@pobox.com  Fri Feb 20 15:24:10 2009
Return-Path: <schwern@pobox.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 1C1DD3A6B38 for <tap@core3.amsl.com>; Fri, 20 Feb 2009 15:24:10 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ke4iDdxOVX3z for <tap@core3.amsl.com>; Fri, 20 Feb 2009 15:24:09 -0800 (PST)
Received: from sasl.smtp.pobox.com (a-sasl-quonix.sasl.smtp.pobox.com [208.72.237.25]) by core3.amsl.com (Postfix) with ESMTP id 3A98A3A6AE9 for <tap@ietf.org>; Fri, 20 Feb 2009 15:24:07 -0800 (PST)
Received: from localhost.localdomain (unknown [127.0.0.1]) by b-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTP id 269402B936 for <tap@ietf.org>; Fri, 20 Feb 2009 18:24:22 -0500 (EST)
Received: from windhund.local (unknown [97.115.125.54]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by b-sasl-quonix.sasl.smtp.pobox.com (Postfix) with ESMTPSA id 6E8C82B935 for <tap@ietf.org>; Fri, 20 Feb 2009 18:24:21 -0500 (EST)
Message-ID: <499F3BA3.5010604@pobox.com>
Date: Fri, 20 Feb 2009 15:24:19 -0800
From: Michael G Schwern <schwern@pobox.com>
User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209)
MIME-Version: 1.0
To: tap@ietf.org
References: <499DF1D6.9020009@pobox.com> <20090220114947.GB16059@klangraum.plasmasturm.org>
In-Reply-To: <20090220114947.GB16059@klangraum.plasmasturm.org>
X-Enigmail-Version: 0.95.7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Pobox-Relay-ID: 9AB8047E-FFA5-11DD-A957-6F7C8D1D4FD0-02258300!a-sasl-quonix.pobox.com
Subject: Re: [tap] Rethinking the TAP version
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Feb 2009 23:24:10 -0000

Aristotle Pagaltzis wrote:
> * Michael G Schwern <schwern@pobox.com> [2009-02-20 01:00]:
>> I propose that the version number change to X.Y where X
>> indicates the compatibility series and Y indicates an
>> incremental, but still compatible, revision. Everything that
>> can read protocol 2.0 can read 2.1 and 2.13 and 2.99 but if a
>> 2.x parser sees 3.1 it knows it can't handle it and can act
>> appropriately. Similarly, a 2.x parser is not guaranteed to be
>> able to read 1.x.
> 
> It sounds good in theory, but doesn’t work out very well in
> practice. It’s only a win if you want to break backcompat often,
> which is not a smart idea. But if you revv that major version
> only once in every decade, you effectively create a new protocol
> each time, not a revision to the old one. Going from `TAP version
> 1.13` to `TAP version 2.0` is effectively indistinguishable from
> going to `TAP2 version 1`, and you don’t need any version check
> code to make consumers fail to parse future TAP versions.
>
> (That’s how we handled it in the Atom WG too. There’s no format
> version in Atom; if backwards-incompatible changes are ever
> necessary, the XML namespace will change, so old consumers will
> fail to process the document at all.)

I get what you're saying.  The problem is we have no namespace.  Technically,
because we ignore anything we can't parse, everything is a TAP document.
There's currently no way for a TAP parser to say "I can not parse this".

TAP parsers will ignore any line they don't recognize.  Upon failing to see a
TAP version they will assume it's TAP version 12.  At best, they'll see a
suite with no tests and fail.  At worst they'll only see the passing tests,
miss new-style failures, and pass what should have failed.

So we need some way to indicate a backwards incompatible change.


-- 
<Schwern> What we learned was if you get confused, grab someone and swing
          them around a few times
        -- Life's lessons from square dancing

From scratchcomputing@gmail.com  Fri Feb 20 22:45:09 2009
Return-Path: <scratchcomputing@gmail.com>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 544CA3A6B5C for <tap@core3.amsl.com>; Fri, 20 Feb 2009 22:45:09 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.599
X-Spam-Level: 
X-Spam-Status: No, score=-2.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 67VexqVT0WfB for <tap@core3.amsl.com>; Fri, 20 Feb 2009 22:45:08 -0800 (PST)
Received: from wf-out-1314.google.com (wf-out-1314.google.com [209.85.200.168]) by core3.amsl.com (Postfix) with ESMTP id 88DB23A6883 for <tap@ietf.org>; Fri, 20 Feb 2009 22:45:08 -0800 (PST)
Received: by wf-out-1314.google.com with SMTP id 27so1542949wfd.31 for <tap@ietf.org>; Fri, 20 Feb 2009 22:45:23 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:subject:date :user-agent:references:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:message-id; bh=ScNxbEYNHDsZzIbdklq6ERHfmv04H/BunMNMN1DzkvQ=; b=uQZRylI5plmwN/eLtjzjxIUXJAC1GujrXbU7X8PcnQDK4yiu5HP0QdTNA1hmZXbzMj 7YICHRtGmjOGigGdZ64uM9GP7Pg/qu2XGF4kch1ux6kIYtEpli8wbmX5bowhxrGR4fbf RGMdNf/XrKkmUX2RyvruhQDV7qxy2AFeqoHlM=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:subject:date:user-agent:references:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :message-id; b=SCMXcvNhCGta+x1fawS7e+9PtvQNJATB7p0i1yU/bjC/QCAznr44JDtb8WYnk1UiE+ eRAVpagK7jFe542RSq/x0osSF/c34XgJHvwctFDbhwQuirvCMqEpYLYv/66z6R4mXg9Z GDzIXYHVQal8C9j7np6Ig86dvznViKM6JwQuE=
Received: by 10.142.230.11 with SMTP id c11mr798533wfh.246.1235198723200; Fri, 20 Feb 2009 22:45:23 -0800 (PST)
Received: from gungnir.scratchcomputing.com (pool-71-111-78-159.ptldor.dsl-w.verizon.net [71.111.78.159]) by mx.google.com with ESMTPS id 30sm8723329wfc.44.2009.02.20.22.45.21 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 20 Feb 2009 22:45:22 -0800 (PST)
From: Eric Wilhelm <scratchcomputing@gmail.com>
To: tap@ietf.org
Date: Fri, 20 Feb 2009 22:45:19 -0800
User-Agent: KMail/1.9.5
References: <499DF1D6.9020009@pobox.com> <20090220114947.GB16059@klangraum.plasmasturm.org> <499F3BA3.5010604@pobox.com>
In-Reply-To: <499F3BA3.5010604@pobox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Message-Id: <200902202245.20010.scratchcomputing@gmail.com>
Subject: Re: [tap] Rethinking the TAP version
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 21 Feb 2009 06:45:09 -0000

# from Michael G Schwern
# on Friday 20 February 2009 15:24:

>I get what you're saying. =C2=A0The problem is we have no namespace.

That's exactly what I was thinking.  Yes a major version bump is=20
basically a new protocol, but you still need some way to indicate it.

I'm in favor of a $major.$minor version scheme.  You could choose to=20
name the major (incompatible) bumps after cartoon characters or types=20
of fish, but numbers have a nice sequence.

=2D-Eric
=2D-=20
"It is impossible to make anything foolproof because fools are so=20
ingenious."
=2D-Murphy's Second Corollary
=2D--------------------------------------------------
    http://scratchcomputing.com
=2D--------------------------------------------------

From pagaltzis@gmx.de  Mon Feb 23 01:30:59 2009
Return-Path: <pagaltzis@gmx.de>
X-Original-To: tap@core3.amsl.com
Delivered-To: tap@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id 4B9EB3A6954 for <tap@core3.amsl.com>; Mon, 23 Feb 2009 01:30:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.766
X-Spam-Level: 
X-Spam-Status: No, score=-0.766 tagged_above=-999 required=5 tests=[AWL=-0.026, BAYES_20=-0.74]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 02SR5QRaN9YY for <tap@core3.amsl.com>; Mon, 23 Feb 2009 01:30:58 -0800 (PST)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by core3.amsl.com (Postfix) with SMTP id E72933A68AF for <tap@ietf.org>; Mon, 23 Feb 2009 01:30:54 -0800 (PST)
Received: (qmail invoked by alias); 23 Feb 2009 09:31:10 -0000
Received: from static-87-79-236-202.netcologne.de (EHLO klangraum) [87.79.236.202] by mail.gmx.net (mp023) with SMTP; 23 Feb 2009 10:31:10 +0100
X-Authenticated: #163624
X-Provags-ID: V01U2FsdGVkX18004UQ1Sg0J9RPjcf/xR9aD+gkwhdgZghtCTgOND X0ZChg1eCRCPBd
Date: Mon, 23 Feb 2009 10:30:35 +0100
From: Aristotle Pagaltzis <pagaltzis@gmx.de>
To: tap@ietf.org
Message-ID: <20090223093035.GA11875@klangraum.plasmasturm.org>
Mail-Followup-To: tap@ietf.org
References: <499DF1D6.9020009@pobox.com> <20090220114947.GB16059@klangraum.plasmasturm.org> <499F3BA3.5010604@pobox.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <499F3BA3.5010604@pobox.com>
User-Agent: Mutt/1.5.18 (2008-05-17)
X-Y-GMX-Trusted: 0
X-FuHaFi: 0.66
Subject: Re: [tap] Rethinking the TAP version
X-BeenThere: tap@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Test Anything Protocol WG discussions <tap.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tap>
List-Post: <mailto:tap@ietf.org>
List-Help: <mailto:tap-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tap>, <mailto:tap-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Feb 2009 09:30:59 -0000

* Michael G Schwern <schwern@pobox.com> [2009-02-21 00:25]:
> I get what you're saying. The problem is we have no namespace.
> Technically, because we ignore anything we can't parse,
> everything is a TAP document. There's currently no way for a
> TAP parser to say "I can not parse this".

Ugh. Yeah.

A corollary is that pre-v13 consumers are a problem no matter
what, as well.

> So we need some way to indicate a backwards incompatible
> change.

How about `TAP 1 version 12` then? Because the reason I brought
this up in the first place is that I’d like the notation to
convey that we expect the basic protocol to almost never change.
Using a $major.$minor versioning scheme seems to suggest that we
expect the major version to change relatively frequently. The
notation should be designed for psychological friction against
the idea of willy-nilly change and over-generalisation in
consumers.

Regards,
-- 
Aristotle Pagaltzis // <http://plasmasturm.org/>
