Subject: [Tools-discuss] Performance problems retrieving drafts

From: Eric Rescorla <[email-masked]>
Date: Sun, 15 Nov 2015 08:04:53 -0800

Hi folks,

tools.ietf.org has a major performance problem whenever you fetch a
draft without a version number. So, for instance, dereference:

  https://tools.ietf.org/html/draft-ietf-acme-acme

This does a 302 redirect to:
  https://tools.ietf.org/html/draft-ietf-acme-acme-01

The second fetch is not too terrible (250ms or so), but the first fetch
spends
6 seconds waiting for the site to do the redirect. A bit of digging shows
that
essentially all this time is spent after the request is sent and waiting
for the
site. Based on inferences from other behaviors, I suspect that this is some
sort of database query to see what the latest draft version is. Given the
trivial size of this data set, that seems like it should be easy to fix,
though...

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 17:36:08 +0100

Hi Eric,

Actually, it's a very simple-minded throttling, in defence of aggressive
robots: sleep for 5 seconds.

I've rearranged the code a bit, to reduce the wait if the html file doesn't
need to be re-generated.

(When you see an issue with anything on tools.ietf.org, please email me, rather
than the anonymous 'folks'; they can't do anything about it.  If I happen to
notice messages on the list, I fix things; but I don't always see list messages
in a timely manner.)

Regards,

	Henrik

---

From: Richard Barnes <[email-masked]>
Date: Sun, 15 Nov 2015 11:42:19 -0500

Very simple-minded indeed. And it would be totally unnecessary if we were
to just put this site behind a CDN.

On Sun, Nov 15, 2015 at 11:36 AM, Henrik Levkowetz <[email-masked]>
wrote:

> --
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
>
> Please report datatracker.ietf.org bugs at
> http://tools.ietf.org/tools/ietfdb
> Please report tools.ietf.org bugs at http://tools.ietf.org/tools/issues or
> send email to [email-masked]
>
>

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 15 Nov 2015 08:43:32 -0800

> robots: sleep for 5 seconds.

That seems... suboptimal.

> need to be re-generated.

Hmm... Doesn't seem to be fixed yet.

(When you see an issue with anything on tools.ietf.org, please email me,
> rather

OK.

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 18:19:53 +0100

Hi Richard,

> to just put this site behind a CDN.

Sure.

It's been a shoestring operation from the start, with most of the equipment
donated by me and others; no monies for CDN.

Of course we could do CDN for tools now, but I'm mostly focused on moving
functionality over to the datatracker these days, and hope to move the
htmlized documents there too, in the not too far future.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 18:21:42 +0100

Hi Eric,

> Hmm... Doesn't seem to be fixed yet.

It needs to propagate to all the servers; I normally only spend time fixing
it on one, and let it propagate in its own good time.  If you wish, you
can hit https://zinfandel.tools.ietf.org/html/draft-ietf-acme-acme directly.

	Henrik

---

From: Richard Barnes <[email-masked]>
Date: Sun, 15 Nov 2015 12:24:26 -0500

> donated by me and others; no monies for CDN.

Fortunately, there are free CDNs nowadays, e.g.:

https://www.cloudflare.com/plans/

Cloudflare is also what fronts for www.ietf.org.  We could probably get
tools.ietf.org added for minimal cost.

--Richard

> > On Sun, Nov 15, 2015 at 11:36 AM, Henrik Levkowetz <[email-masked]
> >
> > wrote:
> >
> >> Hi Eric,
> >>
> >> than the anonymous 'folks'; they can't do anything about it.  If I
> happen
> >> to

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 18:26:00 +0100

> tools.ietf.org added for minimal cost.

Yes.  It's probably worth doing, now that we've got the main IETF domain
on CDN.

	Henrik

---

From: Richard Barnes <[email-masked]>
Date: Sun, 15 Nov 2015 12:30:56 -0500

On Sun, Nov 15, 2015 at 12:26 PM, Henrik Levkowetz <[email-masked]>
wrote:

> > On Sun, Nov 15, 2015 at 12:19 PM, Henrik Levkowetz <[email-masked]
> >
> > wrote:
> >
> >> Hi Richard,
> >>

Great.  I'll get in touch with AMS to see what needs to happen here.

In the meantime, maybe make a TODO to remove the sleep(5) once CF is set up?

--Richard

> >> >> notice messages on the list, I fix things; but I don't always see
> list
> >> >> messages
> >> >> in a timely manner.)
> >> >>
> >> >> Regards,
> >> >>
> >> >>         Henrik
> >> >>
> >> >> > site. Based on inferences from other behaviors, I suspect that
> this is
> >> >> some
> >> >> > sort of database query to see what the latest draft version is.
> Given
> >> the
> >> >> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> >> or

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 18:31:58 +0100

> In the meantime, maybe make a TODO to remove the sleep(5) once CF is set up?

Sure.  It's currently sleep(1) unless you hit the htmlization, which is costly.

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 15 Nov 2015 09:34:15 -0800

Is there a reason not to pre-HTMLize, thus removing the need for the sleep
entirely?

-Ekr

On Sun, Nov 15, 2015 at 9:31 AM, Henrik Levkowetz <[email-masked]>
wrote:

> > On Sun, Nov 15, 2015 at 12:26 PM, Henrik Levkowetz <[email-masked]
> >
> > wrote:
> >
> >>
> >>
> >> >> > Very simple-minded indeed. And it would be totally unnecessary if
> we
> >> were
> >> >> >> I've rearranged the code a bit, to reduce the wait if the html
> file
> >> >> doesn't
> >> >> >> need to be re-generated.
> >> >> >>
> >> >> >> (When you see an issue with anything on tools.ietf.org, please
> email
> >> >> me,
> >> >> >> rather
> >> >> >> > tools.ietf.org has a major performance problem whenever you
> fetch
> >> a
> >> >> >> > The second fetch is not too terrible (250ms or so), but the
> first
> >> >> fetch
> >> >> >> > spends
> >> >> >> > 6 seconds waiting for the site to do the redirect. A bit of
> digging
> >> >> shows
> >> >> >> > that
> >> >> >> > trivial size of this data set, that seems like it should be
> easy to
> >> >> fix,
> >> >> >> > though...
> >> >> >> >
> >> >> >> > -Ekr
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >> --

---

From: Chris Morrow <[email-masked]>
Date: Sun, 15 Nov 2015 12:37:44 -0500

At Sun, 15 Nov 2015 09:34:15 -0800,
> Is there a reason not to pre-HTMLize, thus removing the need for the sleep
(peanut gallery comment)

it does sound like a cron that runs every ~5 mins or something to find
all xml withouth html and 'make the html' would be useful... churn
can't be THAT high on this, right?

also, perhaps that would mean changes to the url handling...

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 15 Nov 2015 18:55:23 +0100

Hi Eric,

> Is there a reason not to pre-HTMLize, thus removing the need for the sleep

I do pre-HTMLize, and keep for 3 months unless a new draft version comes out,
or status changes in some manner which should cause re-generation.  I also
flush cached documents after 3 months to catch minor fixes to the htmlization
script.  Older versions need to be re-generated to list newer versions, and
there are also other things, like for example the IPR link, Errata links on
RFCs, etc., which necessitate regeneration.

Best regards,

	Henrik

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 16 Nov 2015 07:23:13 +1300

> donated by me and others; no monies for CDN.

And to set this thread in context: thank you very very much.

   Brian

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 16 Nov 2015 11:41:47 +0100

> And to set this thread in context: thank you very very much.

Thanks, Brian :-)

	Henrik