Subject: [Tools-discuss] For tomorrows call: Email handling in the datatracker

From: Robert Sparks <[email-masked]>
Date: Mon, 09 Mar 2015 16:52:20 -0500

All -

As I hope you've seen on [email-masked], there is some pain around how 
email is currently being sent through the aliases and from the tracker.
Here is some background for discussion tomorrow. Please read through 
this before we get to the meeting.

Henrik and I have both been studying this, and have several steps we are 
going to take right away. I've also been discussing how draft ballot 
email is handled with Pete to prepare a proposal for the IESG to agree on.

I will send a separate message with more detail than you want (but you 
need it) about where the tracker is sending mail right now.

In short, we have these factors primarily contributing to the pain:

1) In November, we started adding the working group list to the .notify 
alias by default at the request of the IESG. This was driven by the IESG 
wanting the group to see ballot positions, and make it likely that 
responses to those positions would be in-thread.

2) Shortly after, we started migrating to @ietf.org aliases instead of 
tools.ietf.org alias.

3) 10-11 days ago (depending on when you read this message), we started 
forwarding the aliases @tools.ietf.org to use the aliases at @ietf.org 
instead of expanding them on the tools server.

Now, there were several things related to those 3 changes that have 
rough edges:

a) Adding the working group list to .notify had an unintended 
consequence. What we didn't account for is that the contents of .notify 
get expanded into the .all alias, so people sending mail to .all for 
some draft end up sending to the list.

b) When tools.ietf.org was expanding aliases, it added Resent-To and 
Resent-From headers. The expansion at ietf.org does not currently do 
that, and we are working to fix it. Some people were filtering on these 
headers. It's also likely (we haven't confirmed this yet) that Mailman 
was using the headers to bypass putting mail to .all into moderation for 
having an implicit destination.

c) The aliases at ietf.org are _NOT THE SAME_ as the aliases at 
tools.ietf.org! The datatracker allows addresses to be marked active or 
inactive. If an address that's in a draft is marked inactive, then the 
ietf.org alias will not use it - it will find the most recently changed 
active address for that person instead and use that. The tools.ietf.org 
aliases are not aware of these bits, and used the address in the draft 
always. Further, when addresses are learned from drafts by the submit 
tool, with the currently deployed code, they are marked inactive (the 
idea at the time was that we didn't want mail to start going to a 
different address for a person each time they submitted a draft using a 
different email address - several years ago we had a big problem with 
people being surprised by such automatic changes. Instead, they could 
activate each address they wanted the tracker to use.)

So, we are making the following changes right away:

Now - i) We are teaching the tracker about a "preferred" address to use 
whenever context doesn't indicate a better choice (often context will 
say to use the address attached to a chair role for instance). The 
address the tracker is preferring now (specifically the last active 
address that's been touched) will be marked "preferred".

Now - ii) We are marking the addresses that are in drafts that are 
current as active. If someone with an account has an addresses active 
state changed by this, we will send mail to the affected address and all 
other active addresses for them letting them know it happened (in case 
they really _wanted_ that address to be inactive). As new drafts go 
through the submit tool, any learned addresses will be marked active 
(but note that the "preferred" address will not be automatically changed).

After that release, most of the addressing that has been surprising 
people in the last 10 days will be resolved. The ietf.org aliases will 
much more closely match what had been being generated at tools.ietf.org.

We (primarily Glen and Matt) are also continuing to work on getting 
Resent-To and Resent-From into mail forwarded through the aliases at 
ietf.org. We should set the priority for this on tomorrows call (I 
suggest it that it's priority should be high).

Now, early discussions with Pete and Henrik lead to this rough proposal 
that will be sent to the IESG. This is a draft idea, and we won't 
execute on it until the IESG says to (Pete, note this is slightly 
different than what we discussed last). I expect the discussion with the 
IESG to change this some.

Soon - i) The "Save and Send Email" page in the ballot editing workflow 
currently offers the content of the .notify field with a checkbox 
(selected by default). We plan to add another row, with the group's list 
address, also with a checkbox selected by default. If the AD sending the 
message doesn't want the list copied, they unselect the box next to the 
list address. (We might, instead, have to turn this into a bit that 
lives on the document that the sponsoring AD manages, making it their 
choice rather than the balloting ADs choice).

Soon - ii) We will scrub all the documents and remove the list address 
from the .notify field. We'll send notes to the document personnel of 
each affected document so they can verify that the result is what they want.

Soon - iii) We'll add a .group alias for people to use to copy the WG 
list for a draft without having to look it up. The contents of this 
alias would not be included when sending to .all

Sorry for the long message, but there's a lot involved here. Hopefully 
I've teased it apart into easier to follow chunks.

Talk to you all tomorrow,

RjS

---

From: Robert Sparks <[email-masked]>
Date: Mon, 09 Mar 2015 17:20:34 -0500

Pete pointed out that I missed one thing in the contributing factors 
that has already been changed. See inline below.

On 3/9/15 4:52 PM, Robert Sparks wrote:
> Henrik and I have both been studying this, and have several steps we 
> are going to take right away. I've also been discussing how draft 
> ballot email is handled with Pete to prepare a proposal for the IESG 
> to agree on.
>
> 1) In November, we started adding the working group list to the 
> .notify alias by default at the request of the IESG. This was driven 
> by the IESG wanting the group to see ballot positions, and make it 
> likely that responses to those positions would be in-thread.
In November, when we made that change, we also included the .all alias 
in the default for .notify.
That was a lazy way of saying "Get the chairs and the shepherd, etc.", 
and it introduced an expansion loop (since .all has all of .notify in 
its extension)
In a release in early February (IIRC) the default changed to using the 
constituent components instead, and there was work in the alias handling 
system to not break when loops happened. But we have a few drafts out 
there that still have .all in their .notify field.
> 3) 10-11 days ago (depending on when you read this message), we 
> started forwarding the aliases @tools.ietf.org to use the aliases at 
> @ietf.org instead of expanding them on the tools server.
>
> consequence. What we didn't account for is that the contents of 
> .notify get expanded into the .all alias, so people sending mail to 
> .all for some draft end up sending to the list.
>
> that, and we are working to fix it. Some people were filtering on 
> these headers. It's also likely (we haven't confirmed this yet) that 
> Mailman was using the headers to bypass putting mail to .all into 
> moderation for having an implicit destination.
>
> tools.ietf.org! The datatracker allows addresses to be marked active 
> or inactive. If an address that's in a draft is marked inactive, then 
> the ietf.org alias will not use it - it will find the most recently 
> changed active address for that person instead and use that. The 
> tools.ietf.org aliases are not aware of these bits, and used the 
> address in the draft always. Further, when addresses are learned from 
> drafts by the submit tool, with the currently deployed code, they are 
> marked inactive (the idea at the time was that we didn't want mail to 
> start going to a different address for a person each time they 
> submitted a draft using a different email address - several years ago 
> we had a big problem with people being surprised by such automatic 
> changes. Instead, they could activate each address they wanted the 
> tracker to use.)
>
> Now - i) We are teaching the tracker about a "preferred" address to 
> use whenever context doesn't indicate a better choice (often context 
> will say to use the address attached to a chair role for instance). 
> The address the tracker is preferring now (specifically the last 
> active address that's been touched) will be marked "preferred".
>
> state changed by this, we will send mail to the affected address and 
> all other active addresses for them letting them know it happened (in 
> case they really _wanted_ that address to be inactive). As new drafts 
> go through the submit tool, any learned addresses will be marked 
> active (but note that the "preferred" address will not be 
> automatically changed).
>
> Now, early discussions with Pete and Henrik lead to this rough 
> proposal that will be sent to the IESG. This is a draft idea, and we 
> won't execute on it until the IESG says to (Pete, note this is 
> slightly different than what we discussed last). I expect the 
> discussion with the IESG to change this some.
>
> Soon - i) The "Save and Send Email" page in the ballot editing 
> workflow currently offers the content of the .notify field with a 
> checkbox (selected by default). We plan to add another row, with the 
> group's list address, also with a checkbox selected by default. If the 
> AD sending the message doesn't want the list copied, they unselect the 
> box next to the list address. (We might, instead, have to turn this 
> into a bit that lives on the document that the sponsoring AD manages, 
> making it their choice rather than the balloting ADs choice).
>