Subject: [Tools-discuss] all_id2.txt not getting updated

From: Eric Rescorla <[email-masked]>
Date: Thu, 19 Jul 2018 08:06:47 -0700

On a recent rsync, all_id2.txt shows draft-10 of a draft, but -11 is in the
repo.

08:05:11|~/dev/ietf-phab (master)$ grep ietf-ipsecme-split-dns
ids/all_id2.txt
draft-ietf-ipsecme-split-dns-10    -1    Active    AD Evaluation::AD
Followup            2018-07-18    ipsecme    sec    Eric Rescorla
Proposed Standard        .txt,.xml    Split DNS Configuration for IKEv2
Tommy Pauly <[email-masked]>, Paul Wouters <[email-masked]>    David
Waltermire <[email-masked]>    Eric Rescorla <[email-masked]>
draft-pauly-ipsecme-split-dns-02    -1    Replaced
draft-ietf-ipsecme-split-dns    2016-09-21    ipsecme    sec
    Split DNS Configuration for IKEv2    Tommy Pauly <[email-masked]>,
Paul Wouters <[email-masked]>
08:05:23|~/dev/ietf-phab (master)$ ls
ids/draft-ietf-ipsecme-split-dns-11.txt
ids/draft-ietf-ipsecme-split-dns-11.txt

---

From: Glen <[email-masked]>
Date: Thu, 19 Jul 2018 08:17:22 -0700

> 08:05:11|~/dev/ietf-phab (master)$ grep ietf-ipsecme-split-dns

On the IETF production server, I have:

ietfa:~ietf-ftp/internet-drafts # l all_id2.txt
-rw-rw-rw- 2 wwwrun www 7483796 Jul 19 08:07 all_id2.txt

ietfa:~ietf-ftp/internet-drafts # l *ietf-ipsecme-split-dns*
-rw-rw-rw- 2 wwwrun www 29620 Jul 19 07:46 draft-ietf-ipsecme-split-dns-11.txt
-rw-rw-rw- 2 wwwrun www 29310 Jul 19 07:46 draft-ietf-ipsecme-split-dns-11.xml

ietfa:~iietf-ftp/internet-drafts # grep ietf-ipsecme-split-dns all_id2.txt
draft-ietf-ipsecme-split-dns-11 -1      Active  AD Evaluation::AD
Followup               2018-07-19       ipsecme sec     Eric Rescorla
 Proposed Standard               .txt,.xmlSplit DNS Configuration for
IKEv2        Tommy Pauly <[email-masked]>, Paul Wouters
<[email-masked]>        David Waltermire
<[email-masked]>    Eric Rescorla <[email-masked]>
draft-pauly-ipsecme-split-dns-02        -1      Replaced
         draft-ietf-ipsecme-split-dns      2016-09-21      ipsecme sec
                                    Split DNS Configuration for IKEv2
Tommy Pauly <[email-masked]>, Paul Wouters <[email-masked]>

Glen

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 19 Jul 2018 08:22:05 -0700

> -rw-rw-rw- 2 wwwrun www 7483796 Jul 19 08:07 all_id2.txt

Yes, it is updated now. However, it seems that it's possible for it to get
temporarily out of sync, which is suboptimal.

-Ekr

---

From: Glen <[email-masked]>
Date: Thu, 19 Jul 2018 08:32:07 -0700

> temporarily out of sync, which is suboptimal.

Agreed.  IIRC all_id2.txt is generated on a periodic cron job, so it
may at any time be momentarily out of sync... up to, I suppose, 60
minutes.

I don't think regenerating a large file like that on every draft
upload is... workable?  I must leave it to others to make that call,
and/or to determine if there is a better way to get the data you need
in real time (API Datatracker call?)

Glen

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Jul 2018 11:37:38 -0400

> in real time (API Datatracker call?)

I would greatly appreciate having an API.  I have recently found a
need to retrieve draft status.  Separately, we've had requests for an
API to get working group metadata (chair identities, for instance).

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Jul 2018 17:42:40 +0200

> API to get working group metadata (chair identities, for instance).

  https://datatracker.ietf.org/api/
  https://datatracker.ietf.org/api/v1/
  https://datatracker.ietf.org/api/v1/group/role/?group__acronym=httpbis
  https://datatracker.ietf.org/api/v1/doc/document/?name=draft-ietf-ipsecme-split-dns

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Jul 2018 17:47:16 +0200

Oh, and you can force json by adding a 'format=json' query argument.
Otherwise will choose between returning xml or json depending your
http accept header field.

	Henrik

On 2018-07-19 17:42, Henrik Levkowetz wrote:
> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]
>

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 19 Jul 2018 08:51:29 -0700

On Thu, Jul 19, 2018 at 8:32 AM, Glen <[email-masked]> wrote:

> upload is... workable?

Really? It's order 30K lines. This seems like it ought to be very
straightforward.
For reference, the time for me to retrieve it and parse it in my script is
<2s.

  I must leave it to others to make that call,
> in real time (API Datatracker call?)

Perhaps. What I need is "here are the new drafts since the following time",
so i do that by taking the entire draft DB and just looking for new stuff. I
imagine we could invent some kind of timestamp/checkpoint protocol,
but that seems pretty complicated and brittle.

-Ekr

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Jul 2018 11:52:35 -0400

Oh, that's nice.  Thanks.

I apologize for having opened tickets for these.

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 19 Jul 2018 17:56:13 +0200

Hi Martin,

> I apologize for having opened tickets for these.

No worries.

	Henrik

---

From: Chris Morrow <[email-masked]>
Date: Thu, 19 Jul 2018 13:37:22 -0400

On Thu, 19 Jul 2018 11:51:29 -0400,
Eric Rescorla <[email-masked]> wrote:
> 
> [1  <multipart/alternative (7bit)>]
> [1.1  <text/plain; UTF-8 (7bit)>]

it MIGHT be that you have to walk the directory tree to build the
content for the file, right? and depending upon filesystem with ~30k
files that could be an unhappy operation.

-chris

---

From: Glen <[email-masked]>
Date: Thu, 19 Jul 2018 10:48:43 -0700

On Thu, Jul 19, 2018 at 10:37 AM, Chris Morrow <[email-masked]> wrote:
> On Thu, 19 Jul 2018 11:51:29 -0400, Eric Rescorla <[email-masked]> wrote:

Correct on every count.  The size of the file is not a valid indicator
of processing burden.

And just to be clear, although I'm sure many people know this...  my
role at AMS for the IETF is to handle operations (keep the servers
running, clean, updated, and answer what questions I can) but the
actual engineering of things (such as the IETF's Datatracker software,
and related things developed by the IETF) is the TMC's area, and is
handled by Henrik and others.

So I was trying to provide a snapshot of what I saw to help out here,
but I don't know the methods used to generate that file, and so could
not even comment reliably on it.   (Saying this just so you know I'm
not ignoring - I simply don't have the answer.)

Glen

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 19 Jul 2018 11:21:03 -0700

> Correct on every count.  The size of the file is not a valid indicator

Well, I suppose that's true as a general matter (e.g., if it contains a
giant RSA key which
I'm expected to factor), but this kind of text processing task is typically
pretty linear.

I don't know how the data is stored on the server, but it's usually pretty
straightforward
to build a cache that lets you generate this kind of data quickly. With
that said, scraping
the filesystem should also be pretty fast. I just measured this quickly on
my laptop
with an rsynced copy and building up the list of drafts took less than a
second. It
might take a little longer if you had to extract the metadata, but again,
this kind of problem
can be addressed with caching.

> not ignoring - I simply don't have the answer.)

Sure. Thanks for your response.

-Ekr

---

From: Robert Sparks <[email-masked]>
Date: Thu, 19 Jul 2018 14:37:01 -0400

Given what you're doing, would a 'new-revision' document feed be useful?

On 7/19/18 2:21 PM, Eric Rescorla wrote:
>
>
> On Thu, Jul 19, 2018 at 10:48 AM, Glen <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     On Thu, Jul 19, 2018 at 10:37 AM, Chris Morrow
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     > On Thu, 19 Jul 2018 11:51:29 -0400, Eric Rescorla <[email-masked]
>     <mailto:[email-masked]>> wrote:
> Well, I suppose that's true as a general matter (e.g., if it contains 
> a giant RSA key which
> I'm expected to factor), but this kind of text processing task is 
> typically pretty linear.
>
> I don't know how the data is stored on the server, but it's usually 
> pretty straightforward
> to build a cache that lets you generate this kind of data quickly. 
> With that said, scraping
> the filesystem should also be pretty fast. I just measured this 
> quickly on my laptop
> with an rsynced copy and building up the list of drafts took less than 
> a second. It
> might take a little longer if you had to extract the metadata, but 
> again, this kind of problem

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 19 Jul 2018 11:39:14 -0700

> Given what you're doing, would a 'new-revision' document feed be useful?
Probably would have, but I ended up just using the directory listing.

-Ekr

> Tools-discuss mailing [email-masked]://www.ietf.org/mailman/listinfo/tools-discuss
>
> Please report tools.ietf.org bugs athttp://tools.ietf.org/tools/issues

---

From: Chris Morrow <[email-masked]>
Date: Thu, 19 Jul 2018 17:42:56 -0400

On Thu, 19 Jul 2018 14:21:03 -0400,

> can be addressed with caching.

'depending on the file system'

with, for example, ext3 30k files can be very expensive.

(I don't know what the server is, but tools == debian and datatracker
appears to be fbsd, so you can make some guesses about filesystem
type)

-chris

---

From: Glen <[email-masked]>
Date: Thu, 19 Jul 2018 15:03:51 -0700

> appears to be fbsd, so you can make some guesses about filesystem

The production servers run OpenSuse Leap 42.3, the filesystem is
currently ext4, which was the newest filesystem when we took these
over years ago.  I know better filesystems options exist now, and we
may consider a migration when we do a future server upgrade.

But to clarify - I did not mean to imply that we could not cope with a
given frequency of transaction - the servers are quite powerful and
could probably "cope" with a bit more than they currently do (although
they do quite a bit already.)  I was clearly imprecise, and my
imprecision caused a follow-on discussion, for which I apologize.
(Even though we all love such discussions anyway. :-)

Rather, I think I meant to say something like:

At present, the regeneration of all_id2.txt (and similar files) is
done by the Datatracker, in a job scheduled by cron to run once an
hour.  I do not know how feasible it would be to - for example -
change it to run the job once per draft submission instead of once per
hour.  I have no idea whether it would be helpful or off point,
whether it would be right or wrong, better or worse, but, absent such
a change, the original assertion that the file can be out of immediate
sync at any given instant seems to me to be correct. These are all
questions for Henrik and Engineering volunteers.

And while I'm correcting myself, I was reminded that another statement
I made was also imprecise.  Instead of TMC, I should have said "Tools
Team".  They are the ones who handle the engineering and new code
development, with Henrik being a primary point of contact for topics
like this.

Hope this is a bit more clear.

Glen

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 21 Jul 2018 07:33:22 -0700

On Thu, Jul 19, 2018 at 3:03 PM, Glen <[email-masked]> wrote:

> hour.

Do you happen to know the name of the script? That would save me some time
trawling through the tracker code.

-Ekr

  I do not know how feasible it would be to - for example -

---

From: Chris Morrow <[email-masked]>
Date: Sat, 21 Jul 2018 13:49:35 -0400

On Sat, 21 Jul 2018 10:33:22 -0400,
> trawling through the tracker code.

https://github.com/mcr/ietfdb/blob/210f761f678600cd81d8824ade2be649451aeda7/ietf/idindex/index.py

may be relevant to your interests. though this appears to pull all ids from 'some data store' convert to txt and update the id_blah file when done.

> [1.2  <text/html; UTF-8 (quoted-printable)>]
> [2  <text/plain; us-ascii (7bit)>]

---

From: Glen <[email-masked]>
Date: Sat, 21 Jul 2018 10:59:11 -0700

Hi Eric (and thanks Chris for your email also!)

> trawling through the tracker code.

Absolutely.  Sort of.  :-)  In the old days we inherited just a bunch
of random scripts.  I tried consolidating them over the years, but
then Henrik took a much greater step in building them right into the
datatracker.   So we have this crontab that runs:

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DTDIR=/a/www/ietf-datatracker/web
@monthly          wwwrun  test -x $DTDIR/bin/monthly  && $DTDIR/bin/monthly
@weekly           wwwrun  test -x $DTDIR/bin/weekly  && $DTDIR/bin/weekly
05 00   *  *  *   wwwrun  test -x $DTDIR/bin/daily  && $DTDIR/bin/daily
05 1-23 *  *  *   wwwrun  test -x $DTDIR/bin/hourly && $DTDIR/bin/hourly
05 *    *  *  *   mailman test -x $DTDIR/bin/mm_hourly && $DTDIR/bin/mm_hourly

And I'm guessing it's /bin/hourly that you want.

There are still some legacy scripts that *also* run at 5 past the hour
- I don't *think* they're relevant but I'll mention them for
completeness...

PYTHONPATH=/a/www/ietf-datatracker/web export PYTHONPATH
/a/www/ietf-datatracker/web/ietf/bin/rfc-editor-index-updates #
(replaces mirror_rfc_index)
/a/www/ietf-datatracker/web/ietf/bin/rfc-editor-queue-updates #
(replaces mirror_rfc_queue)
/a/www/ietf-datatracker/web/ietf/bin/generate-draft-aliases && \
        ( cd /a/postfix; /usr/sbin/postalias -o draft-aliases; ) && \
        ( cd /a/postfix; /usr/sbin/postmap -o draft-virtual; )
/a/www/ietf-datatracker/web/ietf/bin/generate-wg-aliases && \
        ( cd /a/postfix; /usr/sbin/postalias -o group-aliases; ) && \
        ( cd /a/postfix; /usr/sbin/postmap -o group-virtual; )

Hope this helps!

Glen

---

From: Eric Rescorla <[email-masked]>
Date: Sat, 21 Jul 2018 11:17:15 -0700

Thanks!

---

From: Michael Richardson <[email-masked]>
Date: Sat, 21 Jul 2018 18:06:09 -0400

> https://github.com/mcr/ietfdb/blob/210f761f678600cd81d8824ade2be649451aeda7/ietf/idindex/index.py

That tree is 8 years old.
I shall remove it.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        | network architect  [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Robert Sparks <[email-masked]>
Date: Sun, 22 Jul 2018 21:58:52 -0500

The essential bit you're looking for is here:

https://trac.tools.ietf.org/tools/ietfdb/browser/trunk/ietf/idindex/index.py#L102

Further up the file, you'll find what builds all_id.txt as well.

Cron calls this thing periodically:

https://trac.tools.ietf.org/tools/ietfdb/browser/trunk/ietf/idindex/generate_all_id2_txt.py

Which is a thin wrapper around the first thing I point to above.

(If you really need the glue from cron, this file is run hourly:

https://trac.tools.ietf.org/tools/ietfdb/browser/trunk/bin/hourly

)

RjS

On 7/21/18 1:17 PM, Eric Rescorla wrote:
> Thanks!
>
> On Sat, Jul 21, 2018 at 10:59 AM, Glen <[email-masked] 
>     On Sat, Jul 21, 2018 at 7:33 AM, Eric Rescorla <[email-masked]