Subject: Re: [Tools-team] New tool, that might be useful to put on the tools.ietf.org

From: Bill Fenner <[email-masked]>
Date: Wed, 12 Jul 2006 18:47:48 -0400

Tero,

  I have a lot of dependencies data at
http://rtg.ietf.org/~fenner/ietf/deps/index.cgi
(including I-Ds).  It does do the downref check for
I-Ds that have the right intended state in the tracker,
but the idea of asking the question about a document
and *supplyiing* the intended state is a good one.

  Bill

_______________________________________________
Tools-team mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tools-team

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 13 Jul 2006 02:06:46 +0300

> and *supplyiing* the intended state is a good one.

I remember that I had seen something like that earlier, but couldn't
find it from the http://tools.ietf.org/inventory/all-tools (closest
one was the Draft Dependency Vizualisation)...

What kind of method did you use to find normative vs non-normative
references? I used very ad-hoc method tryign to find "normative
references" vs "informative references" either in the beginning of the
line or number followed those terms.

That works mostly, but there are some problematic documents. 
-- 
[email-masked]

---

From: Bill Fenner <[email-masked]>
Date: Thu, 13 Jul 2006 07:46:24 -0400

>one was the Draft Dependency Vizualisation)...

Same database, different interface.

>line or number followed those terms.

I did similar -

        if (/^\s*(Appendix\s+)?(([A-Z]\.)?[0-9.]*\s+)?((\S+)\s+)?references:?\s*$/i) {
                $type = $5 || "Reference";
        } elsif (/^\s*(([A-Z]\.)?[0-9.]*\s+)?(\S+ormative)\s*$/i) {
                $type = $3;
        } elsif (/^\s*References \((\S+ormative)\)\s*$/i) {
                $type = $1;
	}

  Bill

---

From: Tero Kivinen <[email-masked]>
Date: Thu, 13 Jul 2006 22:51:23 +0300

> Same database, different interface.

Guessed so. 

>         } elsif (/^\s*References \((\S+ormative)\)\s*$/i) {

Hmm... bit different but about same. I noticed that this generates
quite often normative and informative references to the rfc itself, as
if there is page break inside the reference list it will have rfc xxxx
in it. I assume you also search for (BCP|FYI|STD|RFC)\s*(\d+) regexp
for references.

Hmmm.. you do not seem to map BCP/STD/FYI numbers to actual RFC
numbers. I did that so I can do the recursive search....

You also do not have the obsoleted / updated by information. My script
takes the rfc-index.xml and gets that information from there. The
recursive searchs also automatically follows obsoleted by references,
i.e. if there is reference in RFC 4306 to RFC 2463 (PS), and that RFC
2463 is obsoleted by RFC 4443 (DS) then my draft will show 4306 having
reference to 4443 i.e. reference to DS instead of PS.

BTW, my database looks like this:

CREATE TABLE rfc (
docid                 VARCHAR( 8 ) NOT NULL ,
title                 TEXT,
status                VARCHAR( 4 ) NOT NULL ,
PRIMARY KEY ( docid ) ,
INDEX ( status )
) TYPE = MYISAM ;

CREATE TABLE obsoletes (
docid                  VARCHAR( 8 ) NOT NULL ,
obsoletes              VARCHAR( 8 ) NOT NULL ,
INDEX ( docid ),
INDEX ( obsoletes )
) TYPE = MYISAM ;

CREATE TABLE updates (
docid                  VARCHAR( 8 ) NOT NULL ,
updates                VARCHAR( 8 ) NOT NULL ,
INDEX ( docid ),
INDEX ( updates )
) TYPE = MYISAM ;

CREATE TABLE refs (
docid                  VARCHAR( 8 ) NOT NULL ,
reference              VARCHAR( 8 ) NOT NULL ,
type                   VARCHAR( 4 ) NOT NULL ,
INDEX ( docid ),
INDEX ( reference ),
INDEX ( type )
) TYPE = MYISAM ;

and searches are done like this:

SELECT b.reference, c.status, c.title
FROM rfc c INNER JOIN rfc a ON c.docid = b.reference
        INNER JOIN refs b ON a.docid = b.docid
WHERE b.type = 'n' AND
      c.status IN ('PS') AND
      a.docid = 'RFC4302';
-- 
[email-masked]

---

From: Bill Fenner <[email-masked]>
Date: Thu, 13 Jul 2006 14:48:17 -0700

>in it. I assume you also search for (BCP|FYI|STD|RFC)\s*(\d+) regexp

It's kind of horrible,

        if (s/(draft-[-\w]*|rfc)\s*$/$1/i) {
                my $newline = <>;
                $newline =~ s/^\s*//;
                $_ .= $newline;
        }
        s/(draft-(?:[-\w]+(?=-\d\d)|[-\w]+))(-\d\d)?/$& . ".txt" ne $ARGV && handleref($1, $type); ""/ieg;
        s/(rfc|std|bcp|fyi)[- ]?(\d+)/handleref(lc($1) . $2,$type); ""/ieg;

>numbers. I did that so I can do the recursive search....

Yeah, for your purposes my search is under-done - my purposes
were to find what *I-Ds* refer to and specifically if a *given*
draft had a downref.  The info is surely in my databases somewhere
;-)

>You also do not have the obsoleted / updated by information.

s/have/display/ ;-)

>reference to 4443 i.e. reference to DS instead of PS.

Excellent.

Visit https://compost.research.att.com/dbpeek/ and log in with
username ietf, password ietf to see my data.  My data was created
for more general purposes so doesn't lend itself to an easy join
(especially since draft info and RFC info are in different databases).

  Bill

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 14 Jul 2006 01:27:54 +0300

> draft had a downref.  The info is surely in my databases somewhere

Couldn't find it from the tables you had...

> >You also do not have the obsoleted / updated by information.

Same for that....

> (especially since draft info and RFC info are in different databases).

Them being in the different databases do not make joins impossible,
you can simply refer them as draftdep.draftdep.foo and ietf.bar in
select...

One thing they asked when I talked to those doing reviews, was to get
easy access to the document author email and working group chair's
email. I.e. when they make the review of the document they need to
send email to the review team, ad's and then authors and WG chairs.
Getting those last 2 sometimes requires extra searching from the ietf
web page, so if that information could be received directly by the
tool I make it would make their work a bit easier.

I noticed you have list of WG's in the database, but no email address
of the chairs. Also ID database does not have email addresses of the
authors. The trackerdata seems to have some notify emails, don't know
what they are intended to be used.
-- 
[email-masked]

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 13 Jul 2006 18:46:13 -0400

> what they are intended to be used.

I maintain aliases for all wg chairs and ADs, of the form
<wg>[email-masked] and <wg>[email-masked].

There is no reliable source of author's email addresses, but
I'm considering extracting them from the drafts using Jari's
script, and push them into the draft status information I'm
building up to support the notification engine.  In that case
I could also implement an idea from Bill of creating email
aliases like [email-masked] which would
go to authors, chairs and ads.

	Henrik

---

From: Bill Fenner <[email-masked]>
Date: Thu, 13 Jul 2006 16:13:54 -0700

>you can simply refer them as draftdep.draftdep.foo and ietf.bar in

No, but I can't say "join with ietf.rfcindex if it's an
rfc, otherwise join with ietf.allid and trackerdata.docs".

>tool I make it would make their work a bit easier.

I sometimes use Henrik's data for WG chairs, but I have no source
for authors.

>what they are intended to be used.

That's for datatracker state change notifications.  It's automatically
populated with the WG chairs when the document is added, and can be edited
by the AD.

  Bill