Subject: [Tools-discuss] Looking for a tool for grepping mail archives

From: Fred Baker <[email-masked]>
Date: Tue, 17 Oct 2017 10:30:29 -0700

Speaking as a working group chair, I find myself from time to time wanting to assess the mail history of a draft. I usually do that by searching the mail archive on my laptop. I'm not crazy about the tools for that; rather than use the helpful things provided by my laptop OS, I usually wind up with grep. I'm not happy with it.

Is there a tool that will search an online mail archive and cough up the URLs of each individual email in that archive that contains a specified character string?

---

From: Chris Morrow <[email-masked]>
Date: Tue, 17 Oct 2017 13:54:25 -0400

On Tue, 17 Oct 2017 13:30:29 -0400,
Fred Baker <[email-masked]> wrote:
> 
> [1 <multipart/signed (7bit)>] [1.1 <text/plain; us-ascii
> (quoted-printable)>] Speaking as a working group chair, I find
> myself from time to time wanting to assess the mail history of a
> draft. I usually do that by searching the mail archive on my
> laptop. I'm not crazy about the tools for that; rather than use the
> helpful things provided by my laptop OS, I usually wind up with
> grep. I'm not happy with it.
>

grepmail .. could do what you want I suspect as well.

---

From: Fred Baker <[email-masked]>
Date: Tue, 17 Oct 2017 11:42:24 -0700

> On Oct 17, 2017, at 10:54 AM, Chris Morrow <[email-masked]> wrote:
> 

Two issues, as it turns out. A Mac doesn't use traditional unix box format, so grepmail as specified doesn't work. Maybe I need to hack grepmail. Also, I was hoping for something that would use the IETF archive as its source rather than what I have on my laptop.

But thanks, that's at least a start.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 17 Oct 2017 15:21:27 -0400

Fred Baker <[email-masked]> wrote:
    > Speaking as a working group chair, I find myself from time to time
    > wanting to assess the mail history of a draft. I usually do that by
    > searching the mail archive on my laptop. I'm not crazy about the tools
    > for that; rather than use the helpful things provided by my laptop OS,
    > I usually wind up with grep. I'm not happy with it.

    > Is there a tool that will search an online mail archive and cough up
    > the URLs of each individual email in that archive that contains a
    > specified character string?

I use thunderbird configured to talk to the IETF IMAP server.

There is a search feature that seems to operate on the downloaded/cached
message headers.   It may not do as good a search as grep though.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Chris Morrow <[email-masked]>
Date: Tue, 17 Oct 2017 16:07:53 -0400

At Tue, 17 Oct 2017 15:21:27 -0400,
Michael Richardson <[email-masked]> wrote:
> 
> [1  <multipart/signed (7bit)>]
> [1.1  <text/plain (7bit)>]
> 

odd, most times I thought the SEARCH keyword was done on the server...

There's also:
$ rsync rsync.ietf.org::mailman-archive

which appears to have:
drwxrwsr-x          4,096 2017/05/09 21:10:43 venue-selection
drwxrwsr-x          4,096 2017/09/29 05:57:15 video-codec
drwxrwsr-x          4,096 2009/01/01 10:01:17 videomgmt
drwxrwsr-x          4,096 2014/01/24 18:13:30 vipr
drwxrwsr-x          4,096 2017/10/02 23:54:17 vmeet
drwxrwsr-x          4,096 2016/06/22 01:20:43 vnfpool
drwxrwsr-x          4,096 2012/02/08 19:40:59 vnrg
drwxrwsr-x          4,096 2017/09/14 00:57:04 vot

lots of wg archives... like:
$ rsync rsync.ietf.org::mailman-archive/vot/
drwxrwsr-x          4,096 2017/09/14 00:57:04 .
-rw-rw-r--         18,085 2014/09/18 07:09:26 2014-09.mail
-rw-rw-r--      1,121,038 2014/10/30 00:37:46 2014-10.mail
-rw-rw-r--        587,679 2014/11/17 20:49:02 2014-11.mail

then you're probably into the joys of grepmail :) I imagine a little
rsync command-line work you can even limit to the N groups you really
care about alone.

for the curious:
sent 1,802 bytes  received 1,292,141 bytes  66,356.05 bytes/sec
total size is 24,110,876,592  speedup is 18,633.65

so, 24gB or so of mail... if you download it all.

-chris

---

From: Fred Baker <[email-masked]>
Date: Tue, 17 Oct 2017 13:09:14 -0700

> On Oct 17, 2017, at 1:07 PM, Chris Morrow <[email-masked]> wrote:
> 

Daily, of course :-)

Thanks.

---

From: Chris Morrow <[email-masked]>
Date: Tue, 17 Oct 2017 16:14:14 -0400

Well, of course the updates are smaller. Rsync should do smart updates.

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 17 Oct 2017 23:27:28 +0200

Hi Fred,

> Is there a tool that will search an online mail archive and cough up the URLs of each individual email in that archive that contains a specified character string?

These days, I would do this:

Go to https://mailarchive.ietf.org/ and paste the draft name into the search
field, or go directly to for instance

  https://mailarchive.ietf.org/arch/search/?q=draft-ietf-v6ops-rfc6555bis

There I'd select "URL's" from the 'Export' drop-down at the top of the page,
right, or alternatively go directly to for instance

  https://mailarchive.ietf.org/arch/export/url/?q=draft-ietf-v6ops-rfc6555bis

Does that do what you want, and if not, how could it be improved?

Best regards,

	Henrik

---

From: Michael Richardson <[email-masked]>
Date: Wed, 18 Oct 2017 13:11:31 -0400

Henrik Levkowetz <[email-masked]> wrote:
    > There I'd select "URL's" from the 'Export' drop-down at the top of the
    > page, right, or alternatively go directly to for instance

    > Does that do what you want, and if not, how could it be improved?

Cool... could the result be returned as imap:// urls?
(Can Thunderbird or some other program accept such things? I've no idea.)