Subject: [Tools-discuss] mirroring IDs & RFCs

From: Lars Eggert <[email-masked]>
Date: Mon, 29 Jan 2007 15:14:47 +0200

Hi,

I live behind a more restrictive firewall now that doesn't let me  
mirror the ID and RFC databases via rsync or FTP anymore.

Anyone have a handy script that does so over HTTP and only transmits  
deltas?

Thanks,
Lars

_______________________________________________
Tools-discuss mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tools-discuss

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 29 Jan 2007 15:14:01 +0100

Hi Lars,

> Anyone have a handy script that does so over HTTP and only transmits  

I'd suggest wget with the -N switch to avoid downloading unchanged
files.

	Henrik

---

From: Lars Eggert <[email-masked]>
Date: Mon, 29 Jan 2007 16:33:09 +0200

> I'd suggest wget with the -N switch to avoid downloading unchanged

I played with wget, but there is a ton of stuff under http:// 
www.ietf.org/rfc/ other than RFCs, and wget seems to download-then- 
delete items on the "exclude list", which is pretty braindead.

Or maybe I need to be clued in on how wget works?

Lars

---

From: Lars Eggert <[email-masked]>
Date: Mon, 29 Jan 2007 16:36:13 +0200

On 2007-1-29, at 16:33, ext Lars Eggert wrote:
> http://www.ietf.org/rfc/

Also: do we have a similar, http-accessible place for IDs? http:// 
www.ietf.org/ID.html just talks about FTP and rsync.

(Man, how I miss rsync.)

Lars

---

From: Rob Austein <[email-masked]>
Date: Mon, 29 Jan 2007 12:05:00 -0500

> Anyone have a handy script that does so over HTTP and only transmits  

You might try something like:

#!/bin/sh -

wget -q -O - http://www.ietf.org/internet-drafts/all_id.txt |
awk '/^draft-/ {print "http://www.ietf.org/internet-drafts/" $1 ".txt"}' |
wget -nv -nd -N -P ${1-drafts} -i -

This is an even worse kludge than the usual screen scraping of Apache
directory indices, but you sound desperate.

Parsing http://www.ietf.org/rfc/rfc-index.xml to produce a list of
URLs left as an exercise for the reader.

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 29 Jan 2007 18:40:09 +0100

Hi Lars,

> Or maybe I need to be clued in on how wget works?

Mmm.  I haven't checked how much cruft there is under ietf.org/rfc/,
but if you get RFCs from rfc-editor.org, and limit it to one level,
there isn't much cruft.  I'd try this:

wget -N -r -l 1 -nd -nH http://rfc-editor.org/rfc/

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 29 Jan 2007 18:42:17 +0100

Hi Lars,

> www.ietf.org/ID.html just talks about FTP and rsync.

http://www.ietf.org/internet-drafts/

> (Man, how I miss rsync.)

I feel with you.  It makes a big difference in some cases.

	Henrik

---

From: Lars Eggert <[email-masked]>
Date: Tue, 30 Jan 2007 09:28:54 +0200

> wget -nv -nd -N -P ${1-drafts} -i -

Thanks for the tip! I've played with this a bit and came up with the  
scripts attached below. They first try to mirror from tools.ietf.org,  
which is an order of magnitude faster that the secretariat server,  
which they fall back to for anything they can't find on the tools  
server. This lets me survive for now.

Henrik: There are a lot of older RFCs (<1000) that are under http:// 
www.ietf.org/rfc/ but not http://tools.ietf.org/rfc/ - mirroring issue?

Lars

---

#! /bin/sh

wget -nv -O- http://www.ietf.org/internet-drafts/all_id.txt |
         grep -Ev ".+[[:space:]]+.+[[:space:]]+(RFC|Expired)" |
         awk '/^draft-/ {print "http://tools.ietf.org/internet- 
drafts/" $1 ".txt"}' |
         wget -nv -nd -N -i- 2>&1 |
         awk '{print > "/dev/fd/2"}; /^http:/ { sub(/tools/, "www",  
$0); print substr($0, 1, length - 1)}' |
         wget -nv -nd -N -i-

wget -nv -O- http://www.ietf.org/rfc/rfc-index.txt |
         awk '/^[0-9]+ / {print "http://tools.ietf.org/rfc/rfc" $1  
".txt"}' |
         wget -nv -nd -N -i- 2>&1 |
         awk '{print > "/dev/fd/2"}; /^http:/ { sub(/tools/, "www",  
$0); print substr($0, 1, length - 1)}' |
         wget -nv -nd -N -i-

---

From: Rob Austein <[email-masked]>
Date: Tue, 30 Jan 2007 03:23:50 -0500

You want some real fun, try using "xsltproc --html" to parse href
attributes out of <a/> elements in your typical Apache directory index
and formating that up as a list of URLs to "wget -i -".  Whee!

But seriously, if watching all these scripts run is like watching
paint dry, you might want to try "mirror" and FTP.  Yes, I said FTP.
See, what's really hurting your download speed is all the round trips
to check dates.  mirror doesn't have that problem because it parses
the directory listing and zooms right along.  Yes, the parsing is
totally ad hoc and an offense to Diety, Human, and Small Furry
Creatures from Alpha Centauri, but hey it works.

mirror is what I used before I switched to rsync.  Eg:

#!/bin/sh -

mirror -N /dev/stdin <<'EOF'
    package=drafts
	site=ftp.ietf.org
	remote_dir=/internet-drafts/
	local_dir=ietf/draft
	get_patt=^(all_|draft|[0-9])[---0-9a-zA-Z_.]+\.txt$
	passive_ftp=1
	recursive=false
	save_dir=ietf/expired
	save_deletes=true
	max_delete_files=100%
    package=rfcs
	site=ftp.rfc-editor.org
	remote_dir=/in-notes/
	local_dir=ietf/rfc
	get_patt=^rfc(-index\.txt|-index\.xml|[0-9]+\.txt)$
	passive_ftp=1
	recursive=false
EOF

gzip -9rNfq ietf/expired

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 30 Jan 2007 11:20:45 +0100

Hi Lars,

> www.ietf.org/rfc/ but not http://tools.ietf.org/rfc/ - mirroring issue?

No, just different labelling.  I mirror from rfc-editor.org, where 
RFC 123 is stored as rfc123.txt, while on www.ietf.org it's stored
as rfc0123.txt.

I thought I'd set up aliases to handle that, but apparently not.
Will do so.

(To be honest, I find it rather silly to choose storage names
such as rfc0012.txt and rfc0123.txt -- when we pass beyond RFC 9999,
are these going to be re-named to rfc00012.txt and rfc00123.txt,
and rfc9999.txt become rfc09999.txt, just to have the same number
length as rfc10000.txt and later??  Bah.  But I definitely should
be able to present the right RFC no matter what label is used, from
tools.ietf.org.  As mentioned, I'll fix an alias pattern for this.)

Regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 30 Jan 2007 11:59:24 +0100

Hi again

> I thought I'd set up aliases to handle that, but apparently not.

Duplicates with zero-filled numbers were missing on one of the
servers.  Fixed now.  Sorry 'bout that.

	Henrik