Subject: [Tools-discuss] nice mail archive web interface

From: Aaron Falk <[email-masked]>
Date: Thu, 17 Mar 2005 14:36:14 -0500

Hi-

I ran into this today digging up something on mutt, my MUA.  It's a
very nice interface for digging through email archives.  In
particular, check out the thread visualization.  The search function
is good, too.

http://lurker.sourceforge.net/

--aaron

_______________________________________________
Tools-discuss mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tools-discuss

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 18 Mar 2005 16:51:00 +0100

Hi Aaron,

> http://lurker.sourceforge.net/

Yes, I've run into this before, and tried it out for a short time
for archiving, but I got rather quickly tired of the much larger
number of clicks needed to get to the messages I wanted to see, and
the screen real-estate taken up by cute non-essentials.  To me it was
beguiling at first, but didn't have staying power :-)

	Henrik

---

From: Thomas Narten <[email-masked]>
Date: Fri, 18 Mar 2005 11:31:48 -0500

> I wish there was a nice tool for cleaning spam entries from list
> archives.  Some of the IRTF RG list archives at ietf.org have been
> horribly corrupted.  See, e.g.,
> http://www1.ietf.org/mail-archive/web/asrg-announce/current/ 

What I've done in the past is something like the following:

Convert the raw archive to unix mbox format, if necessary. (I think
you have to strip off leading ">" characters in some cases, though I
didn't below.)

Then, run through procmail via formail using something like:

formail -s procmail tmp/procmail.dhc < /tmp/current

where procmail.dhc looks something like:

DEFAULT=/tmp/dhcwg-spam
::
^List-Id:.*dhcwg.ietf.org
/tmp/dhcwg-incoming

Doing the above on the "current" contents of the dhcwg archive, I get
78 messages dumped into spam, vs. 67 kept as legit.

This (with some additional tuning) would probably be a straightforward
way to clean out spam. e.g., legit "From" lines should all start with
something like:

>From [email-masked]  Tue Mar  1 01:56:01 2005

and so forth.

Thomas

---

From: Aaron Falk <[email-masked]>
Date: Fri, 18 Mar 2005 11:40:40 -0500

> beguiling at first, but didn't have staying power :-)

Perhaps if the cute stuff could be disabled...  

--aaron

---

From: Aaron Falk <[email-masked]>
Date: Fri, 18 Mar 2005 11:47:20 -0500

Thomas Narten wrote:
> Convert the raw archive to unix mbox format, if necessary. 

yes, this would work for raw archives.  But what about the web
archives?  The link I sent is basically useless.

--aaron

---

From: Thomas Narten <[email-masked]>
Date: Fri, 18 Mar 2005 11:55:35 -0500

> archives?  The link I sent is basically useless.

Raw archives need to be fixed, and then an appropriate web interface
run on top of the cleaned-up data.

IMO, the current web interface is unusable for many reasons, with spam
not even being the biggest one. E.g., try finding a thread that you
know took place (say) last summer.

Thomas

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 18 Mar 2005 18:14:15 +0100

> you have to strip off leading ">" characters in some cases, though I

Umm, I think it's best to leave the added ">" (in front of
email body lines starting with "From ") to make sure formail
doesn't mistake them for the envelope "From ".  If needed they
can be stripped after going through formail and procmail, though.

> ^List-Id:.*dhcwg.ietf.org

Looks right, simple and economical to me :-)

>>From [email-masked]  Tue Mar  1 01:56:01 2005

Right.  

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 18 Mar 2005 18:20:02 +0100

> archives?  The link I sent is basically useless.

Oh, once the incoming hole is plugged, I would expect the mail
archive maintainers to be able to do that kind of laundering process on
the raw archive data, and have the web-pages re-generated based on
the laundered archive.  

I don't know for certain that MHonArch has the capability to re-generate
the web-pages, but even if it hasn't got that out of the box, it should
be fairly easy to tweak it to run the same program on all of the archives
as it normally runs on the most recent archive to update the web-pages.

	Henrik

---

From: Pekka Savola <[email-masked]>
Date: Fri, 18 Mar 2005 20:26:09 +0200 (EET)

> as it normally runs on the most recent archive to update the web-pages.

Ummm.?  Doesn't this have a big problem?  AFAIK, if the raw archive 
(available from ftp.ietf.org) is cleaned up, and the web archive 
regenerated, I think the html mail generator changes the URLs for 
existing the messages?

For example, the message:
http://www1.ietf.org/mail-archive/web/v6tc/current/msg00161.html
..becomes a completely different message, because the numbers are 
assigned sequentially.

As people refer to messages by the above URLs, we can't just remove 
them.  Maybe just mark them as spam/cleaned-up, and not show them in 
the date/thread index.

I wonder how doable this would be.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 18 Mar 2005 21:24:28 +0100

Hi Pekka,

> On Fri, 18 Mar 2005, Henrik Levkowetz wrote:
[...]
> regenerated, I think the html mail generator changes the URLs for 

Ahh... You're quite right.  We can't re-generate, we'll have to edit
the existing html files.

> I wonder how doable this would be.

Oh, it would be doable all right, but one would have to edit the html
files instead of regenerating them.  A bit more work, but not really
a major undertaking.  I'd estimate up to a week's worth of work for a
capable contractor to understand the situation, build the tool, run
it on all the archives, verify them, and wrap up.

	Henrik

---

From: Rob Austein <[email-masked]>
Date: Thu, 24 Mar 2005 00:20:41 -0500

if one wants to hack the raw archives, the easiest tool to use would
probably be nmh:

$ inc -file rawarchive.txt +tainted
$ pick --list-id mylist.ietf.org | xargs refile +cleaned

"inc" breaks the archive into message files in a mailbox ("+tainted"),
pick selects the ones with the desired list-id: header, refile moves
the selected messages to a different mailbox ("+cleaned").

<rant>

the main problem with this approach is that, decades of tradition
notwithstanding, uucp format makes a really bad mail archive.  no
internal sanity checks, crazy matching rules that routinely require an
mda to mange text just to crock around the shortcomings of the mailbox
format, and very widely implemented, badly.  so in a typical archive,
inc (or procmail's formail command or pretty much anything else) will
fail to recognize some of the messages boundaries.

basicly, all known mailbox formats that depend on magic delimiters
between messages are broken and have been known to be broken for
decades, but people keep writing tools that use them.  uucp format is
a pathologically silly example.  mmdf and babyl formats are somewhat
better (at least their magic delimiters don't routinely appear in
ordinary text), but a sane format would use separate files (mh,
maildir), counted length (mm), or something that was designed to be
used as an archive format (tar, zip).

but i digress.

</rant>