Subject: [Jmap] Submission: options for github issue

From: Bron Gondwana <[email-masked]>
Date: Mon, 24 Apr 2017 11:35:37 +1000

Hi All,

I've been following the discussion with interest, and am trying to distill it down into a github issue for the authors to work from.  As I see it there are roughtly the following options being thrown around.

0) degenerate case: don't support submission at all via JMAP
1) current proposal: extract all envelope information in-band from headers in RFC5322 message content
2) new proposal: store structure envelope information separate from RFC5322 message content

There are some sub-categories as well:

1a) extract envelope data out of From/To/CC/BCC headers only.
1b) add new headers to RFC5322 message content which are kept in the "Sent" copy, but stripped from the copy sent via SMTP.

I'd like to note here that regardless of what method is used, we need to generate two distinct copies of the message, one that goes down the wire without the BCC field, and one that is stored locally and has BCC included.  Sites may have their own magic headers as well, for example FastMail keeps an X-ME-Personality-Id header in drafts which is used to identify which set of signatures to apply to a message, and also whether the email needs to be sent with SMTP-Auth to a smarthost rather than placed into our own outbound queues.

Then we look at the new proposal, from which there are a few options:

2a) add support for arbitrary key/value annotations to all messages, allowing envelope data to be stored and edited along with a draft and retained as it moves to Outbox.
2b) add envelope data as special properties which can only be set when moving a message to Outbox, and must be set when moving to Outbox.
2c) add envelope data as special properties which can only be set when moving a message to Outbox (like 2b), but if not set are inferred from the content of the message (like 1a).
2d) do away with the Outbox concept entirely, either losing delayed send capability from the protocol, or having another way to manage delayed messages - and submit messages using a different technique than "move to Outbox".

Do you think I have adequately captured all the possible ways to do submission here?

Bron.

P.S. There is another question being raised by this discussion, which is whether Draft messages should be editable while retaining the same ID.  One of the best bits of IMAP is that messages are immutable, but it does create frustrations around Draft message handling.  I'll start a separate email thread for that.

-- 
  Bron Gondwana
  [email-masked]

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 24 Apr 2017 11:49:12 +1000

> Do you think I have adequately captured all the possible ways to do submission here?

Noting also that regardless of what method we use, it could be possible to derive default envelope information from a create which didn't explicitly override it, based on server settings.  A very contrived example:

(pseudo-jmap-code)

["setMessages", { "create" : {
 "newid" : {
    "to" : "[email-masked]",
    "bcc" : "[email-masked]",
    "from" : "[email-masked]",
    "subject" : "test",
    "inMailboxes" : [ "draftfolderid" ],
    "isDraft" : true,
    "textBody" : "some text"
  },
}, "#1"]

["messagesSet", {
   created: {
    "newid" : {
      "id" : "84130d20-9b6f-4059-8895-a15c638a5bfa",
      "envelopeFrom" :  ["[email-masked]", "BODY=8BITMIME"],
      "envelopeTo" : {
        "[email-masked]": ["NOTIFY=SUCCESS"],
        "[email-masked]": ["NOTIFY=FAILURE"],
      },
    },
  },
}, "#1"]

Obviously the ability to edit those fields later depends how immutable we make the Draft message, and whether it's being stored in the RFC5322 or as annotations.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 02:38:53 +0000

Hi Bron

I'm in favour of keeping the envelope separate from the message content.

I have less concern about exactly how the envelope is treated.  in the 
end, it's meta-data associated with the message.

There could be some value in keeping the envelope after a message is 
submitted.

There are some pathological cases relating to sending messages to 
multiple recipients that could perhaps be simplified if the envelope 
were kept.

for example per-recipient status.  It may be possible to avoid having to 
include the BCC field and have 2 copies of the message as well if the 
BCCed addresses are kept in the envelope instead of the message.

Adrien

------ Original Message ------
From: "Bron Gondwana" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 24/04/2017 1:35:37 PM
Subject: [Jmap] Submission: options for github issue

>Hi All,
>
>I've been following the discussion with interest, and am trying to 
>distill it down into a github issue for the authors to work from.  As I 
>see it there are roughtly the following options being thrown around.
>
>1b) add new headers to RFC5322 message content which are kept in the 
>"Sent" copy, but stripped from the copy sent via SMTP.
>
>I'd like to note here that regardless of what method is used, we need 
>to generate two distinct copies of the message, one that goes down the 
>wire without the BCC field, and one that is stored locally and has BCC 
>included.  Sites may have their own magic headers as well, for example 
>FastMail keeps an X-ME-Personality-Id header in drafts which is used to 
>identify which set of signatures to apply to a message, and also 
>whether the email needs to be sent with SMTP-Auth to a smarthost rather 
>than placed into our own outbound queues.
>
>2a) add support for arbitrary key/value annotations to all messages, 
>allowing envelope data to be stored and edited along with a draft and 
>retained as it moves to Outbox.
>2b) add envelope data as special properties which can only be set when 
>moving a message to Outbox, and must be set when moving to Outbox.
>2c) add envelope data as special properties which can only be set when 
>moving a message to Outbox (like 2b), but if not set are inferred from 
>the content of the message (like 1a).
>2d) do away with the Outbox concept entirely, either losing delayed 
>send capability from the protocol, or having another way to manage 
>delayed messages - and submit messages using a different technique than 
>"move to Outbox".
>
>P.S. There is another question being raised by this discussion, which 
>is whether Draft messages should be editable while retaining the same 
>ID.  One of the best bits of IMAP is that messages are immutable, but 
>it does create frustrations around Draft message handling.  I'll start 
>a separate email thread for that.
>
>
>--
>   Bron Gondwana
>   [email-masked]
>
>_______________________________________________
>Jmap mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/jmap

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 24 Apr 2017 09:52:05 -0400

Hi Adrien,

> There could be some value in keeping the envelope after a message is

One key value: the ability to re-send the message in case of a failure that 
can easily be corrected. Also, users expect to see Bcc information in their 
sent-mail copies, so if that detail was only in the envelope then it would 
need to be preserved. However, clients have traditionally used the Bcc 
header for maintaining that information, so we may need to special Bcc 
headers even if the envelope is preserved - but then it because a little 
awkward because of the duplication of information in two places.

-- 
Cyrus Daboo

---

From: Ted Lemon <[email-masked]>
Date: Mon, 24 Apr 2017 10:12:56 -0400

On Apr 24, 2017, at 9:52 AM, Cyrus Daboo <[email-masked]> wrote:
> we may need to special Bcc headers even if the envelope is preserved - but then it because a little awkward because of the duplication of information in two places.

Except that the bcc header shouldn't be in the message in the first place, for privacy reasons.   So either the message in the JMAP store is different than what was actually sent, or the Bcc info should _only_ be on the envelope.

Of course, this requires the MUA to do some creative gymnastics to make it all look good, but such is life.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 24 Apr 2017 10:26:20 -0400

Hi Ted,

--On April 24, 2017 at 10:12:56 AM -0400 Ted Lemon <[email-masked]> wrote:

>> we may need to special Bcc headers even if the envelope is preserved -
>> but then it because a little awkward because of the duplication of
>> information in two places.
>
> Except that the bcc header shouldn't be in the message in the first
> place, for privacy reasons.   So either the message in the JMAP store is
> different than what was actually sent, or the Bcc info should _only_ be
> on the envelope.
>

Most MUAs today store Bcc information in the sent-mail copy of a message - 
of course they never include the Bcc header in the message that was 
actually sent

However it is worth noting that RFC 5322 
(<https://tools.ietf.org/html/rfc5322#section-3.6.3>) does define 3 
different ways that the Bcc header can be used and in two of those the Bcc 
header is included in some fashion in a message being sent. I don't think 
those two modes are common enough to warrant support in JMAP, but I could 
be wrong.

-- 
Cyrus Daboo

---

From: Gren Elliot <[email-masked]>
Date: Mon, 24 Apr 2017 07:43:01 -0700

I have worked on a server that did BCC splitting - i.e. A separate message
was sent to each BCC user including their address in the BCC header and a
message sent to all the rest excluding the BCC header.

Regards,
Gren Elliot

From: Cyrus Daboo <[email-masked]> <[email-masked]>
Reply: Cyrus Daboo <[email-masked]> <[email-masked]>
Date: 24 April 2017 at 15:26:35
To: Ted Lemon <[email-masked]> <[email-masked]>
Cc: Adrien de Croy <[email-masked]> <[email-masked]>, [email-masked]
<[email-masked]> <[email-masked]>, Bron Gondwana <[email-masked]>
<[email-masked]>
Subject:  Re: [Jmap] Submission: options for github issue

Hi Ted,

-- 
Cyrus Daboo

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 24 Apr 2017 11:24:02 -0400

Hi Bron,

> Do you think I have adequately captured all the possible ways to do

Well for completeness there is the BURL(RFC4468)/URLAUTH(RFC4467) extension 
that works today in conjunction with IMAP. Whilst I am not proposing JMAP 
support that, it is another mechanism that could be used.

-- 
Cyrus Daboo

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2017 15:46:37 -0000

>2c) add envelope data as special properties which can only be set when moving a message to Outbox (like 2b), but if not set are inferred from the content of the message (like 1a).

Most submission servers clean up the message on the way out.  RFC 4409
describes some of the things they do, making all addresses FQDNs
rather than local abbreviations (joe@sales -> [email-masked]),
They can also invent missing To:, From: and Sender: lines from the
envelope, add Date: and Message-ID: if they're missing or invalid, fix
common address syntax errors like no comma between addresses, and
delete Bcc.  Modern MUAs tend to send mail with fewer defects than the
old unix Mail program, but my MSA still does this stuff, and I've
found that when I've misconfigured my MSA to skip the cleanup step, I
can tell.

This means that if the MUA directly stores copies of the message in
the Sent folder, the stored copies are not the same as what really got
sent.  This is not new, and it's probably why a lot of MUAs give you
the option of adding an implicit BCC so you can send copies to yourself
or perhaps to an address filtered into your Sent folder.

So anyway, I have no objection to inferring the envelope so long as
it's all or nothing, if you specify any of the envelope you specify
all of it, but I wouldn't worry too much about whether the message
then gets cleaned up by BCC deletion since that's nothing new.

R's,
John

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Mon, 24 Apr 2017 16:31:08 +0000

That has certainly always been my preferred way of seeing BCCs handled.

                Tony Hansen

From: Jmap <[email-masked]> on behalf of Gren Elliot <[email-masked]>
Date: Monday, April 24, 2017 at 10:43 AM
To: Ted Lemon <[email-masked]>, Cyrus Daboo <[email-masked]>, "[email-masked]" <[email-masked]>
Cc: Adrien de Croy <[email-masked]>, Bron Gondwana <[email-masked]>
Subject: Re: [Jmap] Submission: options for github issue

I have worked on a server that did BCC splitting - i.e. A separate message was sent to each BCC user including their address in the BCC header and a message sent to all the rest excluding the BCC header.

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 24 Apr 2017 12:59:01 -0400

Hi TONY,

> That has certainly always been my preferred way of seeing BCCs handled.

Actually what we ended up doing in Mulberry is add an option where a 
separate message could be sent to just the bcc recipients, but with 
additional text added to the top of the message. The text could be 
customized, but the default was a warning message:

    IMPORTANT! This message has been blind-carbon-copied to you.
    Please do not reply-to-all or forward this without permission of the
    author.

This was deemed to reduce the chance of accidental reply-all by bcc 
recipients (in the absence of those recipients' email clients providing 
their own warning mechanism).

-- 
Cyrus Daboo

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 21:38:09 +0000

yes, the MUA could feasibly fairly easily fake up the BCC field for the 
user interface based on envelope information.

It could even do better, such as show status per recipient if it had 
that information.

------ Original Message ------
From: "Ted Lemon" <[email-masked]>
To: "Cyrus Daboo" <[email-masked]>
Cc: "Adrien de Croy" <[email-masked]>; "[email-masked]" <[email-masked]>; 
"Bron Gondwana" <[email-masked]>
Sent: 25/04/2017 2:12:56 AM
Subject: Re: [Jmap] Submission: options for github issue

>place, for privacy reasons.   So either the message in the JMAP store 
>is different than what was actually sent, or the Bcc info should _only_ 
>be on the envelope.
>

---

From: Mads Hjorth <[email-masked]>
Date: Tue, 25 Apr 2017 07:15:28 +0000

Hi all,

First of all, thank you for the great work you are doing on figuring out how to crack this nut!

I dont know too much about the specific implementation of the issues being discussed, but I will like to throw in a few observations.

- Self-contained messages (not keeping metadata separate) makes archiving, transformation and a lot of other handling eaiser. Remember that mails have a whole life outside the context of webmail. It also avoid the never ending discussion about what is data and what is meta data. Maybe it is more a question about what parts of a message that are visible/encrypted.

- Sending to named groups and Bcc. The little awkward-ness is central to some of the issues I am currently looking add. I am trying to figure out the best way to allow recipients to see why they receive a group message (beside managed distribution lists). The whole semantics and practice around this is rather unclear to me. Currently my best bet is an empty group address in To: and addresses of the members in Bcc:

The intention of simplifying the eco-system by including submission in JMAP is great, but I have a feeling that if it was possible/feasible/easy it would have been done by IMAP all ready.

As a future user of this specification I would rather see full RFC compliance (e.g. groups and comments in address), than focusing on futures that are reasonably well supported by other protocols.

Thanks again for some great and important work!

           †
        [[ o ]]
         *****
DIGITALISERINGSSTYRELSEN

Mads Hjorth
Chefkonsulent

tel:+4541782351
mailto:[email-masked]
http://www.digst.dk

"I would have got rid of the slash slash
 after the colon. You don't really need it.
 It just seemed like a good idea at the time."

- Tim Berners-Lee

Den 24. apr. 2017 kl. 23.38 skrev Adrien de Croy <[email-masked]<mailto:[email-masked]>>:

yes, the MUA could feasibly fairly easily fake up the BCC field for the user interface based on envelope information.

It could even do better, such as show status per recipient if it had that information.

------ Original Message ------
From: "Ted Lemon" <[email-masked]<mailto:[email-masked]>>
To: "Cyrus Daboo" <[email-masked]<mailto:[email-masked]>>
Cc: "Adrien de Croy" <[email-masked]<mailto:[email-masked]>>; "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>; "Bron Gondwana" <[email-masked]<mailto:[email-masked]>>
Sent: 25/04/2017 2:12:56 AM
Subject: Re: [Jmap] Submission: options for github issue

On Apr 24, 2017, at 9:52 AM, Cyrus Daboo <[email-masked]<mailto:[email-masked]>> wrote:
we may need to special Bcc headers even if the envelope is preserved - but then it because a little awkward because of the duplication of information in two places.

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Wed, 26 Apr 2017 16:15:39 +0000

Cyrus, that customizable text is a nice option.

	Tony

On 4/24/17, 12:59 PM, "Cyrus Daboo" <[email-masked]> wrote:

    Hi TONY,

    

    
    -- 
    Cyrus Daboo

---

From: Ned Freed <[email-masked]>
Date: Wed, 26 Apr 2017 10:24:36 -0700 (PDT)

> Cyrus, that customizable text is a nice option.

You're think so, but I've lost track of the number of times I've received
replies where that sort text was included, quoted, in the response. And this is
when I was a recipient of the original message. Not the author.

The problem is you cannot count on people reading the contents of the
message they're responding to. This has always been true, but in the age
of Twitter it's gotten much, much worse.

FWIW, the approach I prefer, and which my own client uses, is to check
bcc information at reply time, prompt and require signoff before allowing
reply-all.

But regardless, we're now talking about client quality issues that seem,
at least to me, to be entirely independent of access and submission protocol.

More generally, there are some problems that can be solved by switching to  a
different access protocols. And there are lots of problems that cannot. We
need to focus on the problems that can be solved and which fit into this
group's charter.

				Ned

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Apr 2017 15:45:43 -0700

Gmail does bcc splitting, has for several years now.  About once a year we
get someone complaining that it exposes private data (I didn't want people
to know they were on the losers@ mailing list) or confused that they think
this means everyone got the bcc header.

Apparently it's subtle.

Brandon