Subject: [codesprints] Who knows about web browser caching?

From: Bill Fenner <[email-masked]>
Date: Sun, 7 Nov 2010 15:17:48 +0800

Hey everyone,

  Does anyone know how to encourage a web browser (I'm testing with
Firefox) to validate the etag when clicking on a page?

  I've set USE_ETAGS=True in settings.py, which causes django to
create an etag for each page - and this works fine - here is Firefox
prefetching the history tab for a document:

GET /doc/draft-ietf-ipsecme-ikev2-redirect/history/ HTTP/1.1
Host: merlot.tools.ietf.org:31415
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://merlot.tools.ietf.org:31415/doc/draft-ietf-ipsecme-ikev2-redirect/
X-Moz: prefetch
Cookie: sessionid=6d82210d7761d52315fd2319be54a5e7

HTTP/1.0 200 OK
Server: WSGIServer/0.1 Python/2.6.6
Content-Length: 12210
Content-Encoding: gzip
Vary: Accept-Encoding, Cookie
Etag: "b33787a30797c25bd3c18f8d1d863ba4"
Date: Sun, 07 Nov 2010 06:36:59 GMT
Content-Type: text/html; charset=utf-8

However, I clicked on a direct link to the same URL, and Firefox
requested it again, without using If-None-Match:

GET /doc/draft-ietf-ipsecme-ikev2-redirect/history/ HTTP/1.1
Host: merlot.tools.ietf.org:31415
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://merlot.tools.ietf.org:31415/doc/draft-ietf-ipsecme-ikev2-redirect/
Cookie: sessionid=6d82210d7761d52315fd2319be54a5e7

HTTP/1.0 200 OK
Server: WSGIServer/0.1 Python/2.6.6
Content-Length: 12210
Content-Encoding: gzip
Vary: Accept-Encoding, Cookie
Etag: "ba02420d1da3a92ad8f80ce148edb72f"
Date: Sun, 07 Nov 2010 06:37:23 GMT
Content-Type: text/html; charset=utf-8

I also tried adding "Cache-control: private" headers, to try to
encourage Firefox to cache it, but that didn't help.  It does work for
the static files:

GET /css/base2.css HTTP/1.1
Host: merlot.tools.ietf.org:31415
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
Accept: text/css,*/*;q=0.1
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://merlot.tools.ietf.org:31415/doc/draft-ietf-ipsecme-ikev2-redirect/history/
Cookie: sessionid=6d82210d7761d52315fd2319be54a5e7
If-None-Match: "b73424ca331305280f9937208d73f862"

HTTP/1.0 304 NOT MODIFIED
Date: Sun, 07 Nov 2010 06:37:24 GMT
Server: WSGIServer/0.1 Python/2.6.6
Content-Length: 0
Content-Type: text/html; charset=utf-8

I realize that the fact that the Etag varies here is going to be a
problem (it's because the Etag is calculated after the gzip), but this
is a separate problem.

Anyone have any ideas?

Thanks,
  Bill

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Nov 2010 09:47:51 +0100

Bill,

you may want to test with redbot, see

<http://redbot.org/?uri=http://merlot.tools.ietf.org:31415/doc/draft-ietf-ipsecme-ikev2-redirect/history/>

Also...
> problem (it's because the Etag is calculated after the gzip), but this

...the etag is *supposed* to be different across varying 
content-encodings...

Best regards, Julian

---

From: Bill Fenner <[email-masked]>
Date: Sun, 7 Nov 2010 18:17:08 +0800

On Nov 7, 2010, at 4:47 PM, Julian Reschke <[email-masked]> wrote:

> ...the etag is *supposed* to be different across varying content-encodings...

Yes, of course, but in this case it varies with the same encoding (because we are just using an md5 of the returned data, and the metadata in the gzip header is varying, even though the page contents are the same).

Thanks for the redbot pointer!

  Bill

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Nov 2010 11:40:23 +0100

On 07.11.2010 11:17, Bill Fenner wrote:
> On Nov 7, 2010, at 4:47 PM, Julian Reschke<[email-masked]>  wrote:
>
>> Bill,
>>

Um, if the payload changes the (strong) etag needs to change as well, right?

So *why* is the metadata in the gzip header changing?

Best regards, Julian

(greetings to Bejing!)

---

From: Bill Fenner <[email-masked]>
Date: Sun, 7 Nov 2010 19:35:51 +0800

On Sun, Nov 7, 2010 at 6:40 PM, Julian Reschke <[email-masked]> wrote:
>> Yes, of course, but in this case it varies with the same encoding (because
>> we are just using an md5 of the returned data, and the metadata in the gzip
>> header is varying, even though the page contents are the same).
>>

The data that you get as a result of decompressing the payload does
not change.  The bits in the stream that are different are thrown away
while decompressing.

> So *why* is the metadata in the gzip header changing?

The gzip encoder follows the "If the compressed data did not come from
a file..." part of the spec:

         MTIME (Modification TIME)
            This gives the most recent modification time of the original
            file being compressed.  The time is in Unix format, i.e.,
            seconds since 00:00:00 GMT, Jan.  1, 1970.  (Note that this
            may cause problems for MS-DOS and other systems that use
            local rather than Universal time.)  If the compressed data
            did not come from a file, MTIME is set to the time at which
            compression started.  MTIME = 0 means no time stamp is
            available.

so each time you get a gzip stream you get a different MTIME.

> (greetings to Bejing!)

(greetings from Beijing!)

  Bill

---

From: Julian Reschke <[email-masked]>
Date: Sun, 07 Nov 2010 12:42:41 +0100

On 07.11.2010 12:35, Bill Fenner wrote:
> ...
> ...

But it's still payload (from HTTP's point of view, so a strong etag 
needs to change).

> so each time you get a gzip stream you get a different MTIME.

It appears to me that you really need to force the encoder not to change 
the date.

I wonder this is a common problem? Does mod_gzip do this as well?

Best regards, Julian

---

From: Bill Fenner <[email-masked]>
Date: Sun, 7 Nov 2010 20:05:19 +0800

On Sun, Nov 7, 2010 at 8:00 PM, Mark Nottingham <[email-masked]> wrote:
>
> On 07/11/2010, at 10:42 PM, Julian Reschke wrote:
>
>> It appears to me that you really need to force the encoder not to change the date.
>>
> Somewhat; most implementations IME will work around it, but people rolling their own usually don't.

Unfortunately, django uses the python gzip library, which
unconditionally uses the current time in the header:

        self.fileobj.write(chr(flags))
        write32u(self.fileobj, long(time.time()))
        self.fileobj.write('\002')

so we need to create our own gzip and patch django to use it.

I don't have any experience with mod_gzip and dynamic content, so
can't comment on what it does.

  Bill

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 7 Nov 2010 23:00:41 +1100

--
Mark Nottingham   http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 7 Nov 2010 23:36:20 +1100

This might be useful, although it's been a while since I've used it:

def _gzip_body(i):
    "an ugly hack to make sure gzip file doesn't have time embedded"
    tmptime = time.time
    time.time = lambda a = None:0
    sb = cStringIO.StringIO()
    gb = gzip.GzipFile(mode='wb', fileobj=sb, compresslevel=2)
    gb.write(i)
    gb.close()
    time.time = tmptime
    sb.seek(0)
    o = sb.read()
    sb.close()
    return o