Subject: [Tools-discuss] Datatracker is being very slow

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 14:30:28 -0700

I am seeing delays of seconds to 10s of seconds for simple tasks

Any idea what's up?

---

From: Ben Campbell <[email-masked]>
Date: Mon, 2 Apr 2018 16:34:10 -0500

It's improved considerably from the 60 secondsand up that I was seeing last week. :-)

Ben.

> On Apr 2, 2018, at 4:30 PM, Eric Rescorla <[email-masked]> wrote:
> 
> ___________________________________________________________
> Tools-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-discuss
> 
> Please report datatracker.ietf.org and mailarchive.ietf.org
> bugs at http://tools.ietf.org/tools/ietfdb
> or send email to [email-masked]
> 
> Please report tools.ietf.org bugs at
> http://tools.ietf.org/tools/issues
> or send email to [email-masked]

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 2 Apr 2018 23:49:09 +0200

> I am seeing delays of seconds to 10s of seconds for simple tasks

I had to do a number of reloads about an hour ago, and noticed that
re-loading the login page directly after each apache reload was
very slow.  It seems reasonably fast now, though (~1s here).  Are
you still seeing slowness?  For which URLs?

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 14:53:25 -0700

5.5s wait time for:
https://datatracker.ietf.org/iesg/agenda/documents/

---

From: Adam Roach <[email-masked]>
Date: Mon, 2 Apr 2018 17:07:15 -0500

I can corroborate that I'm seeing 6s between request and response for 
this URL also.

/a

On 4/2/18 4:53 PM, Eric Rescorla wrote:
> On Mon, Apr 2, 2018 at 2:49 PM, Henrik Levkowetz <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 00:07:29 +0200

> https://datatracker.ietf.org/iesg/agenda/documents/

Heh. Ok. That's probably the most complex page in the whole datatracker,
and the rendering time is roughly proportional to the number of documents
on the page.  With more than 50 documents on the page, it's probably
heavier than usual.

Ben Campbell pointed out at the end of last week that incremental changes
had brought it up to a response time of about 50s, so I worked during the
weekend to bring it down.  Starting from 50s on Friday, I consider 5.5s
today a good result.

I'll look at caching it, but since parts of the content is individually
rendered for each AD, too simple a cache won't work.

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 15:13:59 -0700

> on the page.  With more than 50 documents on the page, it's probably

It's not just this page. Searching for documents with "ice" takes 19+s and
"trickle" takes 5+s.

> rendered for each AD, too simple a cache won't work.

FWIW, the customary approach would be to have the server serve a JSON file
consisting of the telechat status (this could be cached) and then have the
client retrieve it with XHR and customize it there.

-Ekr

---

From: Adam Roach <[email-masked]>
Date: Mon, 2 Apr 2018 17:22:42 -0500

> on the page.  With more than 50 documents on the page, it's probably

The ">100ms per document" seems in line with the fact that this very 
simple endpoint that just returns 20 documents is running about 2.5s for me:

https://datatracker.ietf.org/api/v1/doc/document/

100ms per document seems like a surprising amount of processing time. Do 
you know where in the technology stack this is coming from? Like, would 
selecting 20 documents out of the database from the commandline take 
this long? Or is it being introduced by Django? Apache?

I'm hesitant to rely on caching too much; as you say, there's 
significant per-AD information here.

/a

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 00:51:59 +0200

Hi Adam,

> this long? Or is it being introduced by Django? Apache?

No, simply returning 20 documents from the database on the command line takes
less than 10ms (the mysql command line says '20 rows in set (0.00 sec)' ).
I haven't looked at where the response time for the tastypie API comes from.

However, for the IESG agenda documents page, there are a number of individual
lookups for each document, including things such as 'is this document being
tracked by this user (flag in left-hand margin), 'get the latest document
event opening a ballot that hasn't already been closed', 'get the AD positions
for this ballot', 'is this document on the agenda for any upcoming meeting
(.ics link in the left-hand margin)', 'get historic events in order to 
calculate time in IESG evaluation', etc., etc., ...

I'm sure this can be produced much more efficiently than it is now; we have
collectively been adding information bit after bit to this view for some
time, without a lot of consideration for lookup and rendering time.  But
having spent a bunch of hours inside the code of exactly this page during
the weekend, I can say that currently it's doing a surprising amount of
necessary work in order to collect all the bits needed to put together the
page.  A complete refactoring is probably the only way to reduce the
rendering time substantially from what it is now.

> significant per-AD information here.

Yes.  If I can do caching, each AD will still get the full 5+s hit on the
first visit of the day, and maybe multiple times, depending on the cache
time.  But I'll at least investigate.  Do you have any suggestion regarding
caching time if caching seems feasible?

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 01:00:20 +0200

Hi Ekr,

> It's not just this page. Searching for documents with "ice" takes 19+s and

19+s for "ice" documents definitely seems over the top.  I'll have a look.

> client retrieve it with XHR and customize it there.

The JSON file would have to comprise much more information than is sent
to each client today, in order to be able to do the customization client-
side.  I'm not sure if that refactoring would be any easier than the
refactoring necessary to do the lookups and server-side rendering of this
page more efficiently.  I'm sure both approaches will work an be able to
reduce overall page delivery time, but neither is trivial.

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 16:00:39 -0700

> rendering time substantially from what it is now.

Can this work be parallelized, e.g., by rendering the shell of the page to
the
user and then letting the browser independently request these values.

> caching time if caching seems feasible?

Depends which cache you mean.

-Ekr

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 16:02:33 -0700

> reduce overall page delivery time, but neither is trivial.

Sure, but it's typically the back-and-forth to the database that's
expensive in this
kind of app, not the bandwidth to send the joined records for each draft.

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 01:08:24 +0200

> user and then letting the browser independently request these values.

Given that the biggest problem right now is the number of lookups generated
individually per document, instead of being retrieved in bulk by fewer more
complex queries, the approach doesn't seem promising.

> Depends which cache you mean.

Server-side per-user cache (memcached).

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 16:12:27 -0700

On Mon, Apr 2, 2018 at 4:08 PM, Henrik Levkowetz <[email-masked]>
wrote:

> >> > simple endpoint that just returns 20 documents is running about 2.5s
> for
> >> me:
> >> >

I wonder if we're talking past each other. My point here is that many of
these values
aren't necessary to view the page. For instance, I'd be willing to have the
page render
but not be able to see the ballots or colors or whatever. Based on the
description
of the problem, it seems like you ought to be able to do that and then
fetch those in
the background, no?

> Server-side per-user cache (memcached).

Why not indefinitely? You don't have a way to know when the cache ought to
be invalidated?

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 01:20:28 +0200

> Why not indefinitely? You don't have a way to know when the cache ought to

Not trivially.  But it can probably be built based on the time of the latest
document event for the documents on the agenda.

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 01:27:42 +0200

On 2018-04-03 01:12, Eric Rescorla wrote:
>> Given that the biggest problem right now is the number of lookups
>> generated individually per document, instead of being retrieved in
>> bulk by fewer more complex queries, the approach doesn't seem
>> promising.
>> 
> I wonder if we're talking past each other. My point here is that many
> of these values aren't necessary to view the page. For instance, I'd
> be willing to have the page render but not be able to see the ballots
> or colors or whatever. Based on the description of the problem, it
> seems like you ought to be able to do that and then fetch those in 
> the background, no?

Yes.  But I don't see that refactoring being any easier than the other
possible approaches.  And I don't see it being inherently superior,
given that the raw database accesses currently take up only about 1/6
of the the total time, when measured on my dev server.  It's the
accumulated cost of fetching individual pieces from many different
places in the code and template that results in the long rendering time.

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 2 Apr 2018 16:32:12 -0700

> places in the code and template that results in the long rendering time.

Yes, but the problem is that you aren't displaying *anything* until you
have done all of them.

-Ekr

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 3 Apr 2018 01:49:15 +0200

> Yes, but the problem is that you aren't displaying *anything* until you

Ack.  True :-}

	Henrik

---

From: Eric Rescorla <[email-masked]>
Date: Mon, 16 Apr 2018 12:51:09 -0700

Update: ballotting via the API is now glacially slow.