Subject: [Tools-discuss] What's happening to the e-mail archives?

From: [email-masked] (Dale R. Worley)
Date: Thu, 10 Dec 2015 11:25:16 -0500

What's happening to the e-mail archives?  The links to the good archives
(http://www.ietf.org/mail-archive/web/WGNAME/current/threads.html) seem
to be being replaced with links to the gackiferous archives
(https://mailarchive.ietf.org/arch/search/?email_list=WGNAME).

Dale

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 11 Dec 2015 08:16:35 +1300

I believe it's intentional.

I agree with you that the old format is much better for many purposes.
Whatever we promote as the searchable, IMAPpable solution, it's essential
to keep the old format around.

   Brian

---

From: Robert Sparks <[email-masked]>
Date: Thu, 10 Dec 2015 14:39:45 -0600

If you haven't seen it already, please review and comment on 
https://datatracker.ietf.org/doc/draft-sparks-genarea-mailarch-improvements/

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 10 Dec 2015 17:10:07 -0500

> (https://mailarchive.ietf.org/arch/search/?email_list=WGNAME).

Or to phrase that better, I find the old archive web pages much
preferable to the new ones.  No doubt, many people have the reverse
opinion or the new format wouldn't exist.  But I'd like to ensure that
the old format (or something functionally equivalent) does not
disappear.

I'll go over draft-sparks-genarea-mailarch-improvements and see if I can
make some concrete suggestions.

Dale

---

From: "Eggert, Lars" <[email-masked]>
Date: Fri, 11 Dec 2015 07:14:37 +0000

Hi,

On 2015-12-10, at 23:10, Dale R. Worley <[email-masked]> wrote:
> preferable to the new ones.
(...)
> But I'd like to ensure that

I wanted to point out that while this may sound like a reasonable suggestion, it in fact doubles the maintenance load, because now both the old and new need to be maintained in perpetuity.

In a single instance, this may not be a huge issue, but every time the tools team changes something, there are some who liked the old way better and want it to be also maintained.

This doesn't scale. If we have rough consensus for a change, that should by default mean that the old way disappears (after a transition period if not immediately.)

Yep, this sucks for those of us on the rough side of the consensus, but that's the IETF for you.

Lars

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 11 Dec 2015 11:30:20 +0100

"Eggert, Lars" <[email-masked]> writes:

> I wanted to point out that while this may sound like a reasonable
> suggestion, it in fact doubles the maintenance load, because now both
> the old and new need to be maintained in perpetuity.
>
> In a single instance, this may not be a huge issue, but every time the
> tools team changes something, there are some who liked the old way
> better and want it to be also maintained.
>
> This doesn't scale. If we have rough consensus for a change, that
> should by default mean that the old way disappears (after a transition
> period if not immediately.)
>

Are the raw mailbox files available for all lists?  Then if people
really like another format can easily import the raw mailbox files and
create other pages.

Whatever is done here, I believe it is critical that access to the raw
data here is provided, so that people don't have to screen scrape it
From the new webpages.

/Simon

---

From: Dave Cridland <[email-masked]>
Date: Fri, 11 Dec 2015 10:42:34 +0000

> data here is provided, so that people don't have to screen scrape it

I understood that all the archives are available over IMAP, so that would
presumably satisfy that requirement very well indeed.

Dave.

---

From: Simon Josefsson <[email-masked]>
Date: Fri, 11 Dec 2015 11:47:15 +0100

> presumably satisfy that requirement very well indeed.

Right -- assuming the data is updated immediately.  How long does it
take after an email has been sent out to the participants of an IETF
mailing list before it is available 1) on the official web archive?, 2)
on the official IMAP server?, and 3) in the official mbox files?  I'm
not looking for an absolute time, just to identify if there is anywhere
in the process a daily/hourly cron-job that needs to run, of it happens
"as soon as possible".  Having a daily/hourly cron job to export emails
to IMAP/mbox is not good enough, IMHO.

/Simon

---

From: Dave Cridland <[email-masked]>
Date: Fri, 11 Dec 2015 10:49:44 +0000

> to IMAP/mbox is not good enough, IMHO.
Oh, hmm. Good point, I'd assumed that the IMAP server was fed the emails
directly from mailman, and that the new web archive just talked to IMAP
directly. That'd be the sane option, though, so it might not be what
happens.

---

From: Robert Sparks <[email-masked]>
Date: Fri, 11 Dec 2015 08:11:15 -0600

Yes, the messages make it into the archives as they are sent to the 
lists - there is no additional delay to anything like a cron heartbeat.

RjS

On 12/11/15 4:49 AM, Dave Cridland wrote:
>
>
> On 11 December 2015 at 10:47, Simon Josefsson <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Dave Cridland <[email-masked] <mailto:[email-masked]>> writes:
>
>     > On 11 December 2015 at 10:30, Simon Josefsson
>     <[email-masked] <mailto:[email-masked]>> wrote:
>     >
> Oh, hmm. Good point, I'd assumed that the IMAP server was fed the 
> emails directly from mailman, and that the new web archive just talked 
> to IMAP directly. That'd be the sane option, though, so it might not 
> be what happens.
>
>     /Simon
>
>
>
>

---

From: Robert Sparks <[email-masked]>
Date: Fri, 11 Dec 2015 08:14:44 -0600

<snip/>
> to IMAP directly. That'd be the sane option, though, so it might not 
Both the web archive and the imap server are backed by exactly the same 
bits - there's only one copy of each message on disk, and both use it.

As Lars pointed out, this _dramatically_ simplifies management of the 
archives.

RJS

---

From: Michael Richardson <[email-masked]>
Date: Fri, 11 Dec 2015 13:39:10 -0500

I like, Dave Worley, prefer the old mailman (pipermail) archives to the new
interface.  While I can't search by string, I get much better threading and
context, and if I I want to refer someone (by URL) to the thread, the old
archive systems wins hands down.

If I'm trying to understand some discussion that has occured in the past,
I would now attempt to use the IMAP interface via Thunderbird (which I
installed and configured just for IMAP).   I suggest trying this.

I note that many of our lists are also on gmane.org, and they have a variety of
ways to access things, but this is relying on the kindness of gmane volunteer work.

And the URLs that result are simpler, and the files are resting on disk,
and I could wget --mirror the whole thing if I wanted to.

Having said this, I appreciate maintaining two things is harder.
I know that there is work to make the threading interface to the new archives
better, and perhaps the resulting URL *for the thread* will be easier to
obtain.  I very very much like that the emails going out have their
mailarchive URL in them already --- that's really cool.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 12 Dec 2015 07:57:09 +1300

> presumably satisfy that requirement very well indeed.

You are joking, I trust. We're talking about massive quantities of data, which
should stay where they belong, somewhere at ietf.org.

    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 12 Dec 2015 08:01:06 +1300

> I wanted to point out that while this may sound like a reasonable suggestion, it in fact doubles the maintenance load, because now both the old and new need to be maintained in perpetuity.

But that's what will be needed, unless the new system provides sufficient
functionality, which at the moment it doesn't. Of course, Robert is correct,
those of us who are complaining need to read his draft.

    Brian

---

From: Dave Cridland <[email-masked]>
Date: Fri, 11 Dec 2015 20:17:11 +0000

On 11 December 2015 at 18:57, Brian E Carpenter <[email-masked]
> wrote:

> should stay where they belong, somewhere at ietf.org.
No, I'm not joking. If people want access to the raw message data, then
IMAP is well-suited to this (and more so, I think, than mbox files). I'm
not claiming mirroring everything is anything I particularly want to do, or
would see the point of, but back-filling my local archive of a WG list I
came late to would seem sensible - I assume that Simon was suggesting
something along those lines.

Dave.

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 12 Dec 2015 11:13:33 +1300

> something along those lines.

Fair enough if you are selective.

   Brian

---

From: James Cloos <[email-masked]>
Date: Sat, 12 Dec 2015 18:47:49 -0500

>>>>> "EL" == Eggert, Lars <[email-masked]> writes:

EL> I wanted to point out that while this may sound like a reasonable
EL> suggestion, it in fact doubles the maintenance load, because now
EL> both the old and new need to be maintained in perpetuity.

Will the mbox files in rsync.ietf.org::everything-ftp/ietf-mail-archive/
continue to be updated?  One mbox per calendar month?

I gave up on the web interface and use my rsync copy of everything-ftp
whenever I need to look at ietf archives.

-JimC
-- 
James Cloos <[email-masked]>         OpenPGP: 0x997A9F17ED7DAEA6

---

From: [email-masked] (Dale R. Worley)
Date: Mon, 14 Dec 2015 15:25:45 -0500

> the old and new need to be maintained in perpetuity.

My hope was that the old format was generated by something that was part
of Mailman, and so didn't require much maintenance.

> If we have rough consensus for a change, that should by default mean
> that the old way disappears (after a transition period if not
> immediately.)

True...  I'd like to know if there was a rough consensus for change, in
the sense "Do the users prefer the new system?", rather than the sense
"Do the people who talk about the tools want something different?"
After all, there's a serious loss of upward-compatibility here!

Dale

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 15 Dec 2015 15:47:51 +1300

> https://datatracker.ietf.org/doc/draft-sparks-genarea-mailarch-improvements/

Firstly, I think the new archive has already been improved since I first
looked at it. It's much closer now to something I would consider using as
a default choice.

Secondly, no direct comments on the draft, but...

Feature request: I think the little menu for FILTER BY TIME
really lacks a Start/End option.  To find that, you have to go through
Modify Search/Advanced Search/Time/Custom/, by which point the tool
has forgotten which list you want.

Feature request: When you finally find that Start/End option, you have
to type in the dates. I'd expect to find one of those little pop-up
calendars.

Question:
Despite that, I managed to find the same (randomly chosen) message
in the two archives:
http://www.ietf.org/mail-archive/web/ietf/current/msg17113.html
https://mailarchive.ietf.org/arch/msg/ietf/cAzJlZ2cUsSMIVEgbvwc2dt9exI
So my question is, why does the new archive wrap so aggressively
in this view? (The old archive doesn't wrap plain text at all, as
far as I know, which is also a pain.)

We have a mixture of wrapped and flowed text in IETF mail.
Wrapping everything at 72 produces ugliness, as you can see
in my example.

Wrapping isn't mentioned in RFC 6778, so I'm not sure where
the decision to wrap at 72 came from.

Feature request: don't wrap text that is already wrapped (pragmatically,
let's say, if it's wrapped somewhere before 100, don't wrap it at 72).

Note - this is not an issue for the message display below the search panel,
where wrapping seems to be done intelligently.

Thanks
    Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 16 Dec 2015 16:30:39 +1300

I have just discovered that the wrapping problem is browser-dependent
(ugly on Firefox 42.0, OK on IE 11). Maybe a CSS issue?

So maybe this is a bug fix.

    Brian

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 16 Dec 2015 17:55:59 -0500

> Wrapping everything at 72 produces ugliness, as you can see

I don't see any ugliness, but that could be a browser difference.
Certainly, I'd like to be able to get wrapping near column 80, as I'm a
long-time Emacs user.  My understanding is that the quasi-standard
default for "fill text" commands is 72, left over from the old
typewriter days.

Dale

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 17 Dec 2015 13:16:34 +1300

> default for "fill text" commands is 72, left over from the old

Yes, it is a browser issue, or perhaps a browser/CSS interaction issue.
So it's a bug, not a feature request.

Sure, as far as there's a "standard" it's 72, although I believe the
true origin is that there were 80 columns on a standard punch card, of
which the last 8 were reserved for the card number, so there were 72
useable columns. That bled over into printers and later into VDUs.

    Brian

---

From: Robert Sparks <[email-masked]>
Date: Thu, 17 Dec 2015 09:45:07 -0600

Yes - this is a pretty hard problem, and the current tool doesn't make 
the best tradeoffs yet.
There are some things we've tried on the datatracker that should work 
better here than what's currently deployed.

RjS

---

From: Adam Roach <[email-masked]>
Date: Thu, 17 Dec 2015 10:12:59 -0600

> So maybe this is a bug fix.

Can I get you to send a bug to <https://webcompat.com/>? In general, 
when you see behavior on the web that varies between browsers, it's an 
indication that someone has a bug. For the sake of the web as a whole, 
it's better if we fix these bugs rather than work around them.

For background:
https://hacks.mozilla.org/2014/06/introducing-webcompat-com/

/a

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 18 Dec 2015 17:13:08 -0500

Here are some suggestions that I think would improve the usability of
the new IETF mailing list archive web pages.  This list is not carefully
studied, so it should be considered a basis for discussion rather than
my fixed opinions.

(In all of these, "window" means the browser window, the space in which
the browser displays a page.)

* Usable in "profile"

The pages should be usable if the proportions of the window are
"profile", which has height about 1.5 times width.  The new page looks
... odd when it is put into profile size.  On my screen, the Subject
listing is folded at every word.

* It's a web page, not an app

All features should be accessible without Java, Javascript, Flash, or
HTML 5.

* Vertical space is precious

It is important that information be provided in context, which
generally requires plenty of screen space above and below the datum
that the user is looking at.  Thus, it is important to maximize the
amount of vertical space available for displaying things.

There should be no non-scrollable elements at the top or bottom of the
window.

Large information items, those that can't be displayed in their
entirety, should be displayed as separate pages rather than in little
"viewport" windows in other pages.

If it's necessary to display, e.g., two large items, such as a list of
message identifications along with a single message, the window should
be partitioned as frames, which in most browsers allows the user to
re-allocate space between the two scrolling subwindows.  (It looks
like the new archive page does that, but at least on my Firefox,
doesn't have any little graphic dingus on the dividing line between
them to show that the user can move it, so I didn't discover that
functionality until I was reviewing this item.)

In the current archive page, the sequence of items on the page is:

- "IETF ... Sign In"
- "Modify search ... Ungroup ... Search ... Export"
- "Date ... Subject ... From ... List"
- the list-of-messages
- "nnnn Messages"
- a single message
- "Email archive vx.x.x".

There's also lots of whitespace between the items in the list of
messages.  In comparison, the maximum number of messages listable in
the new archive page is 13 (on my system), but in the old archive
page, it is typically 20 or more (using a substantially larger font).

* Don't use too much whitespace

Whitespace makes a page *look nicer* but it reduces the amount of
context that can be shown.  The new design is particularly unfortunate
in the amount of whitespace on the left and right sides, which forces
the subject lines in the message list to be folded more often than is
necessary.

* Font choice is important

The current archive page favors tiny sans-serif fonts, some of them
having thin stroke-widths and light colors.  Why, I don't know, as no
commercial web site seems to do that.  Serif fonts are a lot easier to
read and using a light color gratuitously reduces readability.

* Things should be links

That way the user's reflexes work correctly.  Also, the browser's
copy-link-URL function works correctly.  In particular, it helps if an
item in a list is a link that will display that item (in another
window).

In the new archive, screen space is taken at the bottom of the message
display to give a link to the page that displays the message.  But it
would be much preferable if the item in the list-of-messages was that
link, so middle-click could easily create a new tab with the message.
(In particular, I could go through a thread doing middle-click on each
message, rather than repeating left-click, mouse down to bottom,
middle-click, mouse up to list, ...)  Also, that would free up one
more line for the message display.

There would probably have to be some Javascript so that left-click in
the list would display the message in the message window (rather than
fetching it into the whole window), but presumably that can be done
while allowing middle-click to do the normal fetch-in-another-window.

* Thread display

The new archive page has a thread display option, but it doesn't show
what messages are in response to what messages.  The details of that
information is useful to some degree, but the big benefit is that the
structure makes each thread group visually distinct, as only
thread-start messages are at the left margin.  In the new page, the
subject lines are so tightly folded one has to nearly read them,
rather than just scanning for lines that don't look like the lines
above them, to find the thread starts.

* Getting a list of the last year's messages

It's not clear how one can get a text list of the last year's messaged
(e.g.).  I can click on "FILTER BY TIME ... Past year", but that
doesn't put the past year's messages in the list-of-messages; the
list-of-messages remains one of those annoying auto-extending lists
that you have to pull down repeatedly in order to force it to load
everything that should be in it.

This item is a derivative of my strategy, "If you want to do serious
searching on a list, get it into an Emacs buffer."  Emacs (and the Un*x
stuff that can be accessed through it) have been under development for
~40 years, and no web project is going to replicate its functionality
(at any reasonable cost).

* Truncating Subjects with "..." is not desirable

Generally, I don't like truncating long but important data items with
"...", because often it is the end of the string that is important to
see if I care about that item.  For instance, the last message to
Dispatch is displayed as:

    [dispatch] Draft new version: draft-holmberg-dispatch-rfc7315-updat...

What you can't see is the part you can't predict from context, namely
"updates-03".  And if I copy the message list with ctrl-A, ctrl-C, I
still don't get the full subject line, making it hard to search the
list with other tools.

* Sender's names

The old archive pages give the "real" sender's names, whereas the new
archive pages give their e-mail addresses.  The real sender's names are
more pleasant and easier to read.

* "References" and "referenced by" links

The old display of a message listed 

    Follow-Ups:
    References:
    Prev by Date: 
    Next by Date: 
    Previous by thread: 
    Next by thread: 
    Index(es):
        Date
        Thread

Most of links show the author (useful) and Subject (no so useful) of the
message linked to.

The new display of a message lists

    View by Date
    View by Thread

I've frequently used the Follow-Ups and References and it would be
helpful if they were continued.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Sun, 20 Dec 2015 22:37:17 -0500

Let me add this:

* Accessible to search engines

The structure of links in the archive shall make all messages visible
(as individual pages) to the spiders of search engines.

At least one URL for each message shall contain the mailing list name
(or a substitute for it) so that a "site:..." restriction given to a
search engine will confine the search to a single mailing list.  E.g.,
"site:www.ietf.org/mail-archive/web/dispatch
draft-holmberg-dispatch-rfc7315-updates".

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Sun, 20 Dec 2015 22:52:16 -0500

Now that I've written up my issues, it's becoming clearer to me that the
underlying problem is one of the paradigm or philosophy of the interface.

One possible paradigm is that the archive pages are an *interface to a
search tool*.  Once you think of it this way, it's clear that is what
RFC 6778 is driving at.  The table of contents starts with:

   1. Introduction ....................................................2
   2. List Search and Archive Requirements ............................3
      2.1. Search and Browsing ........................................3

and the first sentences of the abstract are:

   The IETF makes heavy use of email lists to conduct its work.
   Participants frequently need to search and browse the archives of
   these lists and have asked for improved search capabilities.

and the first paragraph of the Introduction is:

   The IETF makes heavy use of email lists to conduct its work.
   Participants frequently need to search the archives of these lists
   and have asked for improved search capabilities, particularly when
   the search needs to cover a large period of time or cross several
   lists.  For instance, document editors, shepherds, working group
   chairs, and area directors may need to review all discussion of a
   particular draft.  That discussion may be spread across the working
   group list, one or more directorate lists, and the IETF general list.
   Occasionally, work impacts multiple groups, possibly in different
   areas, and the search must cover additional working group lists.

What this paradigm envisions is:

- The user formulates a query that will isolate the messages he wants to
  see.
- He clicks the button.
- The archive returns a list of those messages.
- He reads the messages.

In this paradigm, exactly how the interface displays the messages is
unimportant, as the user will go through them all one at a time.

The paradigm that I use is that the archive *displays the messages in
the archive* and that I then use one of several external tools to
identify and examine the messages I'm interested in, viz.:

- Scanning the list of messages by eye, particularly the
  list-in-thread-order page.  (For practical purposes, the list is
  usually restricted to a time-based segment of the archive, e.g., six
  months.)
- Searching the list (incrementally!) using the ctrl-F capability of my
  browser.
- Copying the index page text, inserting it into an Emacs buffer, and
  searching the Emacs buffer (using any of the Emacs search facilities,
  or a Un*x command applied to the buffer).
- Querying the list archives using a search engine.  (People have
  observed that Google searchs most business's web sites better than the
  "search" feature of the web sites.)

This paradigm is a lot more demanding of how the archive contents are
displayed.  The first search method requires that the index be organized
in a way that the critical features are immediately accessible to the
feature-detectors of the human visual system (as opposed to requiring
the user to read the text).  The remaining methods require that the text
content of the index page fully contain the information that is relevant
to the search (generally, Subject, From, and Date).

Note that from this point of view, RFC 6778 appears to be the
specification of a search feature to be added to the archive, rather
than a specification that "everything not listed here will be removed".
This was what led me to a sense of being badly done by in the new
system -- the support of my method of using the archives was removed,
apparently without warning.

Dale