Subject: [Jmap] Submission

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 19 Apr 2017 06:40:08 +0000

Hi

I just looked through the specs for mail submission and couldn't find 
where it's covered.

In the "Message Submission" section in the jmap.io site it talks about 
submission being merely a copy (move?) of a message (presumably from 
drafts) to an outbox.  Presumably after that it gets moved or copied to 
the sent folder?

Does this mean the server must parse the message to get SMTP envelope 
information?  There are all kinds of problems with that.

As for the justifying discussion on the site about this also then 
allowing support for "delayed sending" by way of the client setting the 
date in the message (bad idea IMO, server should set message date, 
client often has wrong date, or mis-matched to server date) in the 
future which the client can then cancel, I think this is also several 
bad ideas.

* It gives the server the job of maintaining when to send the message
* It requires the client and server clocks to be in synch
* it delays the message (who ever wants to do this?)
* opening a window to cancel the message before it's sent seems a bit 
nonsensical / contrived to me sorry.  A UA using this is just mainly 
going to make itself useless for mail conversations by unnecessarily 
delaying mail.

So I think using "delayed send" as a justification for "copy to outbox" 
submission falls over under scrutiny.

I think submission should be a separate command so that the client can 
specify needed envelope information independently of the message 
content.  Even if this were an additional option.  E.g. just move it to 
outbox if you want the server to try to obtain an envelope from the 
message content, or specifically provide that data from the client.

Otherwise list processors won't be able to use this, server will need to 
strip BCC etc etc.

Adrien

---

From: Neil Jenkins <[email-masked]>
Date: Wed, 19 Apr 2017 17:11:16 +1000

Yes, the intention is the server will build the RFC5322 message, add and
strip headers as appropriate and set the SMTP envelope information.
After sending, the server moves the message from the Outbox to Sent, or
deletes it and writes a new one if it adds in extra information (since
the message is immutable with the same id). This covers everything
normal email clients need to do. There are other niche use cases, but
JMAP doesn't have to deal with everything. SMTP Submission will still
exist, but this simplifies things in the overwhelmingly common case.
> * It requires the client and server clocks to be in synch

Almost every device these days uses NTP or equivalent to set their
system clock. We have not observed significant clock skew issues with
our client for years.
> * it delays the message (who ever wants to do this?)

You're joking, right? The ability to "undo send" is *enormously*
popular. Being able to do this in a standardised way for desktop clients
would be a huge win over the current situation. Normally you would use a
window of something like 30 seconds in the future. Scheduled send is
also moderately popular in a certain niche.
>   going to make itself useless for mail conversations by unnecessarily
The UA does not *have* to delay the send. It can omit setting an
explicit date on the message, in which case the server will set the date
to now(), and immediately send the message. But the ability to do so for
clients that want to make use of this is very important.
> content.  Even if this were an additional option.
The trouble is options increase complexity and testing requirements. The
current spec does not *require* servers to support delayed send. The
canDelaySend capability property lets clients know if it does. Servers
that don't support it could even not have a real "outbox" folder but
just a synthetic one as anything moved to it is sent immediately (or the
move is rejected if the message is invalid).
Neil.

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 19 Apr 2017 08:31:27 +0000

OK on delayed send.

NTP I wouldn't rely on.  I still often see clock divergence.

but frankly the server parsing the envelope out of the message seems 
like a hack, which also limits options to a subset of what a sender can 
do with SMTP.

Surely it's simple to have a submit command which takes envelope 
attributes.  That way is deterministic.

I agree about option hell, that's one thing I'd like to leave behind 
with IMAP.

But parsing messages, scraping an envelope, and stripping unwanted 
headers seems fragile.  A server failing to strip BCC headers leaks 
private information.  It's also a lot more work for the server, and no 
real difference either way for the client.

Adrien

------ Original Message ------
From: "Neil Jenkins" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Sent: 19/04/2017 7:11:16 PM
Subject: Re: [Jmap] Submission

>Yes, the intention is the server will build the RFC5322 message, add 
>and strip headers as appropriate and set the SMTP envelope information. 
>You're joking, right? The ability to "undo send" is enormously popular. 
>Being able to do this in a standardised way for desktop clients would 
>be a huge win over the current situation. Normally you would use a 
>The UA does not have to delay the send. It can omit setting an explicit 
>date on the message, in which case the server will set the date to 
>now(), and immediately send the message. But the ability to do so for 
>The trouble is options increase complexity and testing requirements. 
>The current spec does not require servers to support delayed send. The 
>just a synthetic one as anything moved to it is sent immediately (or 
>the move is rejected if the message is invalid).
>
>Neil.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Apr 2017 10:11:42 +0100

Hi,

> On 19 Apr 2017, at 09:31, Adrien de Croy <[email-masked]> wrote:
> 
> 
> OK on delayed send.
> 
> but frankly the server parsing the envelope out of the message seems like a hack, which also limits options to a subset of what a sender can do with SMTP.
> 
> Surely it's simple to have a submit command which takes envelope attributes.  That way is deterministic.

Personally I don't care whether it is a separate command versa extra (possibly optional) parameters for envelope on an existing one.

I think having parameters for envelope would be a good idea (and it would be consistent with earlier discussions on Submit in IMAP.) But I am wondering if we can make common cases simple for clients by not requiring the parameters.

> I agree about option hell, that's one thing I'd like to leave behind with IMAP.
> 
> But parsing messages, scraping an envelope, and stripping unwanted headers seems fragile.  A server failing to strip BCC headers leaks private information.  It's also a lot more work for the server, and no real difference either way for the client.
> 
> Adrien
> 
> 
> 
> 
>> Yes, the intention is the server will build the RFC5322 message, add and strip headers as appropriate and set the SMTP envelope information. After sending, the server moves the message from the Outbox to Sent, or deletes it and writes a new one if it adds in extra information (since the message is immutable with the same id). This covers everything normal email clients need to do. There are other niche use cases, but JMAP doesn't have to deal with everything. SMTP Submission will still exist, but this simplifies things in the overwhelmingly common case.
>> 
>> Almost every device these days uses NTP or equivalent to set their system clock. We have not observed significant clock skew issues with our client for years.
>> 
>> You're joking, right? The ability to "undo send" is enormously popular. Being able to do this in a standardised way for desktop clients would be a huge win over the current situation. Normally you would use a window of something like 30 seconds in the future. Scheduled send is also moderately popular in a certain niche.
>> 
>>> * opening a window to cancel the message before it's sent seems a bit nonsensical / contrived to me sorry.  A UA using this is just mainly going to make itself useless for mail conversations by unnecessarily delaying mail.
>> 
>> The UA does not have to delay the send. It can omit setting an explicit date on the message, in which case the server will set the date to now(), and immediately send the message. But the ability to do so for clients that want to make use of this is very important.
>> 
>>> I think submission should be a separate command so that the client can specify needed envelope information independently of the message content.  Even if this were an additional option.
>> 
>> The trouble is options increase complexity and testing requirements. The current spec does not require servers to support delayed send. The canDelaySend capability property lets clients know if it does. Servers that don't support it could even not have a real "outbox" folder but just a synthetic one as anything moved to it is sent immediately (or the move is rejected if the message is invalid).
>> 
>> Neil.
> _______________________________________________
> Jmap mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/jmap

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 19 Apr 2017 12:14:20 +0100

Neil Jenkins writes:
> Scheduled send is also moderately popular in a certain 
> niche.

Oh? That's very interesting. What niche?

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 19 Apr 2017 12:28:31 +0100

> NTP I wouldn't rely on.  I still often see clock divergence.

A lot of phones get their time from the telco, whose base stations are very 
well synchronised with each other but not always with that place in 
Greenwich.

But perhaps it's still reasonable to rely on correctness. If E-Plus is off 
(IIRC it was off by ~90 seconds when I was a customer) its customers either 
cannot undo send and/or can undo for 120 seconds rather than 30. Having 
that happen to a few people may be an acceptable price to pay for API 
simplicity. There are fewer than 1000 network operators, most of those 
<1000 clocks are correct to within a second or two, and the consequences of 
the errors seem minor.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 19 Apr 2017 12:38:41 +0100

I'd add a sentence saying that servers and clients are assumed to have 
correct clocks within ±3s, and that the authors have not paid attention to 
what might fail when a clock is off by more than that.

Arnt

---

From: Ricardo Signes <[email-masked]>
Date: Wed, 19 Apr 2017 08:28:46 -0400

* Alexey Melnikov <[email-masked]> [2017-04-19T05:11:42]
> I think having parameters for envelope would be a good idea (and it would be
> consistent with earlier discussions on Submit in IMAP.) But I am wondering if
> we can make common cases simple for clients by not requiring the parameters.

We discussed envelope parameters at IETF 98, and I hope that we do continue
forward by making them both possible and optional.

-- 
rjbs

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Wed, 19 Apr 2017 09:49:54 -0400

> forward by making them both possible and optional.

I somehow missed that discussion at IETF 98, but i want to say that i
think that envelope parameters during submission is an important
feature.  MUAs that want their RFC5322 message to go out exactly as they
composed it, but using an arbitrary SMTP envelope should be able to
exercise that level of control via JMAP, without worrying that the
server side is going to parse, interpret, rearrange, or mangle the
standard in-message headers.

Regarding one common mismatch between SMTP envelope and RFC5322 mail
headers:

I think that encouraging an MUA to include an explicit, in-band
Bcc: header in a message that it hands off to a server for delivery is
asking for trouble.

Making the SMTP envelope parameters "optional" needs more specification.
For who are they optional?  how?  optional for JMAP servers to
implement?  optional for JMAP servers to accept?  optional for JMAP
clients to include?  optional for JMAP clients to implement?

My preference (for simplicity, omitting the explicit mechanism and just
calling it "submission"):

 * JMAP servers MUST accept SMTP envelope parameters via the submission
   mechanism

 * JMAP clients MAY decline to provide SMTP envelope parameters when
   invoking the submission mechanism, in which case the JMAP server will
   synthesize SMTP envelope parameters from the submitted message.

 * synthesis of SMTP envelope headers at submission time by the JMAP
   server should leave the message otherwise unchanged.

 * delayed send should be implemented as an explicit envelope parameter,
   not as a parsing of the in-band Date: header.  There are a range of
   reasons why an MUA might want to submit a message with a in-band
   Date: that the server finds objectionable (including server-side
   clock failure).

Regards,

        --dkg

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Apr 2017 16:39:02 +0100

Hi,

> clients to include?  optional for JMAP clients to implement?
I meant the last two.
>     Date: that the server finds objectionable (including server-side
+1.

---

From: "John Levine" <[email-masked]>
Date: 19 Apr 2017 16:34:29 -0000

>Making the SMTP envelope parameters "optional" needs more specification.

Urrgh.  There is a long history of doing submission by moving messages
to a special folder, and one thing they have in common is that they
all failed to get much use.

I think there are two reasons they failed: a lack of feature
negotiation, and the overoptimistic belief that you can construct the
envelope from the message headers.

Feature negotiation should not be a big problem.  In RFC 6409
submission, you get a list of features back from EHLO.  Some of them
can affect the way a client formats an outgoing message, e.g.
8BITMIME or BURL or SMTPUTF8.  Some like DELIVERBY and DSN let the
client put annotations on the message or individual addresses.  Hence
we need a way to get the submission features from the server, and a
way to put submission annotations on the envelope addresses when
sending a message.

This leads to the second issue, where do the envelope addresses come
from.  While it is certainly true that in many simple cases you can
guess the envelope addresses from the message headers, guessing
robustly is hard, and in many cases it is not evident to me what the
correct guess would be.  With a message possibly having Sender, From,
To, Cc, Bcc, and Resent-* versions of all of those, there's over a
thousand possible header combinations to screw up and test.  Beyond
that, what do you do with a From: with two addresses or a group
address with no actual addresses?  If there's a Resent-To but no
Resent-From (a 5322 violation but not a surprising one) do you reject
it or do you invent an envelope address and if so, what address do you
use?

My strong preference would be that if you do submission, there should
be no envelope defaults and the entire envelope must be specified by
the client.  (This is separate from whether the submission agent
cleans up the submitted message on the way out.)  This shouldn't be a
burden on client writers, since the client presumably knows who it's
sending the mail to, and it provides a straightforward way to add
annotations to the envelope addresses, e.g., by sending a list of 
[ address, annotation, ... ] in place of a plain address.

Also, if the envelope is explicit, this lets people use jmap for some
interesting other applications, e.g., make a service like mailman a
daemon talking to a jmap server rather than having to configure it in
system specific ways. 

R's,
John

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Apr 2017 12:50:23 -0400

Of course you could do the envelope as attached attributes to the message that's stuffed in the special folder.   I'm not saying it has to happen that way, but it can be made to work.

---

From: Daniel Kahn Gillmor <[email-masked]>
Date: Wed, 19 Apr 2017 12:46:08 -0400

> [ address, annotation, ... ] in place of a plain address.

I think you've convinced me that while it is surely easy to write a
server-side envelope synthesizer, it looks hard to write a high-quality
and robust server-side envelope sythesizer.

You've also convinced me that even if submitting envelope parameters
ends up being nominally optional, any client i work on would always
explicitly send them anyway because i'd be afraid of whatever bugs lurk
inside whatever server-side synthesizer i happen to be talking to.

So from an implementer's perspective, i don't really care about making
the submission envelope parameters optional, and would happy if JMAP
ends up requiring the client to provide them during submission.

        --dkg

---

From: "John R Levine" <[email-masked]>
Date: 19 Apr 2017 13:33:09 -0400

> Of course you could do the envelope as attached attributes to the message that's stuffed in the special folder.   I'm not saying it has to happen that way, but it can be made to work.

Sure.  I don't see this as being hard to implement.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "John R Levine" <[email-masked]>
Date: 19 Apr 2017 14:04:40 -0400

> and robust server-side envelope sythesizer.

Right, and even then, if a client can't put parameters on the envelope, 
it's not going to be very satisfactory.  On the other hand, if you do have 
an explicit envelope with parameters, a lot of other stuff falls out 
naturally.

For example, the marinate* option that people were arguing about is the 
FUTURERELEASE extension defined in RFC 4865. with the HOLDFOR and 
HOLDUNTIL parameters.  No need to invent anything, it's already there.

I hope Ned Freed weighs in here, since he's done more hand to hand combat 
with this stuff than anyone.

R's,
John

* - as in, allow a message to marinate for a while before sending it

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 19 Apr 2017 12:29:29 -0700

The original IMAP designer always opposed adding submission to IMAP as a 
matter of functional separation. While I strongly agree with the need 
for functional specification, it is possible to design a protocol in 
such a way that multiple separable functions are provided in the same 
protocol. As a result I support adding submission to JMAP in a way that 
preserves functional separation.

When deploying a large mail service it's important to keep the 
submission mail queues (which have management/monitoring issues) 
functionally separate from the mail store (which has very different 
management/monitoring issues). If the JMAP design requires a mail queue 
to be present on the JMAP server or present in the mail store, that is 
an unacceptable protocol design error as it hinders the manageability of 
large deployments.

Furthermore, SMTP submission is a full standard IETF protocol that 
continues to be extended. JMAP should leverage the semantics of the 
existing standard in such a way that it's not necessary to change the 
JMAP specification when a new submission extension that's useful to JMAP 
is added. In addition to giving JMAP clients access to all the useful 
submission extensions like FUTURERELASE, UTF8SMTP, DELIVERBY, etc. This 
will ultimately make the JMAP specification simpler (as it won't have to 
deal with submission extensions directly) and more flexible to deploy.

Finally, if JMAP supports submission then I expect a client author will 
want to use the JMAP submission mechanism for all submissions. If JMAP 
has a rigid design where each submission extension requires an 
additional JMAP extension then a high-end client will eventually be 
forced to implement SMTP submission in addition to JMAP submission. And 
it will be stuck doing both (the former for functionality, the latter 
for ease-of-setup). I don't want JMAP clients to be forced down that 
path.

What this means is that for JMAP to support submission correctly, it 
must be able to function as an SMTP submission proxy from a semantic 
viewpoint. This creates a number of requirements. The basic requirements 
are:

1. It must be possible to query the JMAP server for the set of 
Submission ESMTP extensions semantically equivalent to the SMTP 
submission EHLO response. The JMAP server needs to omit the extensions 
that aren't relevant (e.g., AUTH, STARTTLS, PIPELINING).
2. It must be possible to provide an SMTP envelope for submission that 
is semantically identical to the submission protocol. This needs to 
support MAIL FROM, one or more RCPT TO, and ESMTP parameters to those 
envelope components. There needs to be an algorithmic way to convert 
from JMAP envelope syntax to SMTP Submission envelope syntax.
3. It must be possible to get submission errors that are semantically 
identical to those provided by a submission server. Including at least 
3-digit SMTP code, optionally enhanced SMTP code and human readable text 
of error and potentially one response for mail from, one for each rcpt 
to and one for data/bdat final response.

So that's the submission functionality JMAP needs to provide if it 
provides submission at all. If you want to also provide a simpler model 
for submission in JMAP, I won't object as long as the full model is 
mandatory.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Apr 2017 15:36:59 -0400

On Apr 19, 2017, at 3:29 PM, Chris Newman <[email-masked]> wrote:
> When deploying a large mail service it's important to keep the submission mail queues (which have management/monitoring issues) functionally separate from the mail store (which has very different management/monitoring issues). If the JMAP design requires a mail queue to be present on the JMAP server or present in the mail store, that is an unacceptable protocol design error as it hinders the manageability of large deployments.

Isn't this just an implementation detail?   IOW, how is it different to have a different protocol for queuing messages, versus having different handling for a mailbox with a specific name?

Your three requirements, which make sense to me, could be added as attributes to the message once it's been sent.

I'm not convinced that this is the right way to handle it—it means that the MUA has to be specifically watching for changes in the submit mailbox, or else that there has to be a submit and a sent mailbox, and the MUA has to watch the sent mailbox.   But ISTM that we have to provide ways of doing all these things anyway, so having an additional protocol may not actually be worth the complexity.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Apr 2017 21:38:47 +0100

> On 19 Apr 2017, at 20:36, Ted Lemon <[email-masked]> wrote:
> 

It is not clear to me why a queue can't be also a mailbox. Interfaces seem to be pretty similar for both.

But anyway, Outbox doesn't really have to be a part of the mailstore, it can be just exposed as a mailbox in JMAP.

> Your three requirements, which make sense to me, could be added as attributes to the message once it's been sent.

Right (or they can be a part of the message when it is created. But this is not important right now.)
> 
> I'm not convinced that this is the right way to handle it—it means that the MUA has to be specifically watching for changes in the submit mailbox, or else that there has to be a submit and a sent mailbox,

There are both.

> and the MUA has to watch the sent mailbox.

Can you elaborate on why you think MUA has to watch the Sent mailbox? In order to report to the user that a message was sent? I suspect we can use JMAP notifications for that.

>   But ISTM that we have to provide ways of doing all these things anyway, so having an additional protocol may not actually be worth the complexity.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Apr 2017 16:49:26 -0400

> Can you elaborate on why you think MUA has to watch the Sent mailbox? In order to report to the user that a message was sent? I suspect we can use JMAP notifications for that.

Possibly, but reliable delivery of notifications would potentially be an issue, for cases where the message is queued rather than being delivered as soon as it lands, or even in that case if the MUA disconnects immediately after submitting.   But my mental model of how notifications work is really thin, so if that's not an issue, then sure, notifications would work.   I would have assumed that watching the sent mailbox would be how the notification would be generated, no?

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 19 Apr 2017 18:28:39 -0700

On 19 Apr 2017, at 12:36, Ted Lemon wrote:
>> management/monitoring issues). If the JMAP design requires a mail 
>> queue to be present on the JMAP server or present in the mail store, 
>> that is an unacceptable protocol design error as it hinders the 
>> manageability of large deployments.
>
> Isn't this just an implementation detail?   IOW, how is it different 
> to have a different protocol for queuing messages, versus having 
> different handling for a mailbox with a specific name?

Not at all. Protocols can either be architecture neutral or impose an 
architecture on the system. An example where IMAP imposed an 
architecture on the system is the \Deleted flag and EXPUNGE command. 
This requires a server mail store to have a concept of deleted messages 
within a folder; something that doesn't fit well if the mail store had a 
"trash folder" model. And while it's possible to implement a virtual 
trash folder in a mail client using the \Deleted+EXPUNGE model so this 
wasn't a UI-constraining protocol design error, it turns out that 
clients didn't do that and decided to ignore the IMAP base spec 
semantics and impose the trash folder model on IMAP servers. That 
interoperates badly without correct implementation of RFC 6154 on all 
clients. So that's an example of a protocol design error imposing 
architectural constraints that resulted in real world problems.

To clarify my point, I think having an "outbox" with combined mail store 
and mail queue semantics as a mandatory architectural element of JMAP is 
a protocol design error because it breaks functional separation. And 
that violation of functional separation is particularly problematic in 
larger deployments where mail stores and mail queues each add management 
and monitoring complexity and require different designs and storage 
models to scale effectively.

JMAP needs to have a submission action that has semantics matching the 
IETF Submission standard. It's fine for that command to pull data from 
the mail store as part of the submission action (as RFC 4468 does). But 
the JMAP protocol needs to allow implementations where mail queue and 
mail store semantics are separate.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Apr 2017 21:55:39 -0400

On Apr 19, 2017, at 9:28 PM, Chris Newman <[email-masked]> wrote:
> To clarify my point, I think having an "outbox" with combined mail store and mail queue semantics as a mandatory architectural element of JMAP is a protocol design error because it breaks functional separation. And that violation of functional separation is particularly problematic in larger deployments where mail stores and mail queues each add management and monitoring complexity and require different designs and storage models to scale effectively.
> 
> JMAP needs to have a submission action that has semantics matching the IETF Submission standard. It's fine for that command to pull data from the mail store as part of the submission action (as RFC 4468 does). But the JMAP protocol needs to allow implementations where mail queue and mail store semantics are separate.

Perhaps so.   However, the semantics of submission protocols aren't really right either.   What I think we want is to tell the MUA that delivery failed if it did, and to allow the MUA to know the current status of delivery at least at the first hop, if delivery is not yet complete.   But the SMTP submit protocol doesn't do this.   It's fire and forget: all you get is confirmation that the message has been queued.

Granting that the point of this working group is not to fix SMTP, the semantics of the JMAP submit protocol ought to allow for delivery of failure status without generating a needless bounce message just as a place to put information from the site MTA's delivery attempt.   If the failure happens farther down the line in a forwarding chain, we can't do anything about it, but from a usability perspective bounce messages are worse than useless, so anything that can reduce the number of such messages that users see is a net win.

The point that I am arguing is that sent messages really do have a place in something that is notionally a mail store, and those messages serve as a place where delivery status can be found.   Message submission should not be a transaction: it should be a process.   Notification should work even if the MUA isn't present at the time of failure, and should be available in all MUAs, not just the sending MUA. It should be presentable to the user in a way that makes sense.   A "sent" folder makes sense.   A separate queue construct does not.   Forcing the MUA to fake this isn't going to produce a good user experience.

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 19 Apr 2017 19:05:51 -0700

> seem to be pretty similar for both.

The architectural requirements of a mail store and mail queue are very 
different. For comparison:

Mail store
  * data is "owned" by the end-user
  * long-lived small & large objects
  * write-once, read-multiple, important dynamic metadata (flags & 
annotations)
  * quota-based administrative limits
  * over quota is flagged for statistics but not an operational problem
  * for security, read can be restricted to only work when the end-user 
authenticates
  * final delivery agent can be limited to append-only access
  * data for different users must be kept separate with ACL restrictions
  * random access is important; message metadata must be cached for 
efficiency

Mail queue
  * data is "owned" by the delivery system
  * short-lived small & large objects
  * write-once, normally read+delete once, largely static metadata 
(envelope)
  * generally no administrative limits (or limits are imposed at 
submission time)
  * large queue indicates operational problems, needs to flag a 
monitoring system alert
  * for security, all read/write access can be restricted to delivery 
agent; no need for ACL complexity
  * data for different users can be combined for efficient queuing.
  * sending messages for different users over the same SMTP relay 
channel improves privacy by making traffic analysis harder
  * access follows queue pattern; many need to cache envelope metadata 
for efficiency but message metadata is not cached

It possible to implement something that provides the union of the 
required semantics. But combining the semantics creates security and 
privacy constraints, adds monitoring complexity to the deployment, and 
removes a bunch of potential optimizations.

So I object to JMAP if it requires me to implement something combining 
mail store and mail queue semantics. The protocol design needs to allow 
implementations with clean functional separation between mail queue and 
mail store.

		- Chris

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 19 Apr 2017 19:30:12 -0700

On 19 Apr 2017, at 18:55, Ted Lemon wrote:
>> To clarify my point, I think having an "outbox" with combined mail 
>> store and mail queue semantics as a mandatory architectural element 
>> of JMAP is a protocol design error because it breaks functional 
>> separation. And that violation of functional separation is 
>> particularly problematic in larger deployments where mail stores and 
>> mail queues each add management and monitoring complexity and require 
>> different designs and storage models to scale effectively.
>>
>> JMAP needs to have a submission action that has semantics matching 
>> the IETF Submission standard. It's fine for that command to pull data 
>> from the mail store as part of the submission action (as RFC 4468 
>> does). But the JMAP protocol needs to allow implementations where 
>> mail queue and mail store semantics are separate.
>
> Perhaps so.   However, the semantics of submission protocols aren't 
> really right either.   What I think we want is to tell the MUA that 
> delivery failed if it did, and to allow the MUA to know the current 
> status of delivery at least at the first hop, if delivery is not yet 
> complete.   But the SMTP submit protocol doesn't do this.   It's fire 
> and forget: all you get is confirmation that the message has been 
> queued.

There's RFC 3887 (Message Tracking Query Protocol). The product I work 
on has implemented that.

> Granting that the point of this working group is not to fix SMTP, the 
> semantics of the JMAP submit protocol ought to allow for delivery of 
> failure status without generating a needless bounce message just as a 
> place to put information from the site MTA's delivery attempt.

Agreed.

> If the failure happens farther down the line in a forwarding chain, we 
> can't do anything about it, but from a usability perspective bounce 
> messages are worse than useless, so anything that can reduce the 
> number of such messages that users see is a net win.

Bounce messages that are standards compliant are useful (and are 
frequently used to implement automatic un-subscription for mailing 
lists). But what I would say is that a mail client UI that shows bounce 
messages as "just another message in the INBOX" is not a good UI.

> The point that I am arguing is that sent messages really do have a 
> place in something that is notionally a mail store,

Draft Messages and Sent Messages belong in a mail store; the \Sent and 
\Draft RFC 6154 folders are well-within scope for JMAP.

> and those messages serve as a place where delivery status can be 
> found.

I'm generally supportive of this goal. However, I want the JMAP base 
spec to deploy quickly so I'd like to keep the amount of additional 
mandatory system complexity JMAP requires over-and-above Submission+IMAP 
to a minimum. I'm supportive of some additional complexity in exchange 
for ease-of-client development. But I don't think improved bounce 
handling is the right place to make that tradeoff in the base JMAP spec. 
Improved bounce handling could make a good JMAP extension.

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Wed, 19 Apr 2017 23:31:07 -0400

> Improved bounce handling could make a good JMAP extension.

That is the rathole IMAP went down.   The reason I'm advocating for doing this in the base spec is that then we don't have to tolerate clients that don't do the extension.   But the complexities that you mention are worth consideration—I'm saying this not to entirely disagree with you, but to advocate that we at least consider doing this in the base spec.

---

From: "John Levine" <[email-masked]>
Date: 20 Apr 2017 04:14:12 -0000

In article <[email-masked]> you write:
>-=-=-=-=-=-
>-=-=-=-=-=-
>That is the rathole IMAP went down.   The reason I'm advocating for doing this in the base spec is that then we don't have to tolerate clients that
>don't do the extension.   But the complexities that you mention are worth consideration—I'm saying this not to entirely disagree with you, but to
>advocate that we at least consider doing this in the base spec.

Having written some bounce processors for mailing list software, I can
assure you that improved bounce handling is a rathole all of its own.
You can handle all of the RFC compliant DSNs and such, and you're
about 1/3 of the way there.

It has been my impression that we want jmap to do what imap and submit
do, but not that we want to try and fix SMTP's problems.

R's,
John

---

From: Brandon Long <[email-masked]>
Date: Wed, 19 Apr 2017 21:36:58 -0700

On Wed, Apr 19, 2017 at 7:05 PM, Chris Newman <[email-masked]>
wrote:

>>>> management/monitoring issues). If the JMAP design requires a mail queue to
>>>> be present on the JMAP server or present in the mail store, that is an
>>>> unacceptable protocol design error as it hinders the manageability of large
>>>> deployments.
>>>>
>>>
>>> Isn't this just an implementation detail?   IOW, how is it different to
>>> have a different protocol for queuing messages, versus having different
>>> handling for a mailbox with a specific name?
>>>
>>
>  * generally no administrative limits (or limits are imposed at submission
> time)
>  * large queue indicates operational problems, needs to flag a monitoring
> system alert
>  * sending messages for different users over the same SMTP relay channel
> improves privacy by making traffic analysis harder
>  * access follows queue pattern; many need to cache envelope metadata for
> efficiency but message metadata is not cached
>
> It possible to implement something that provides the union of the required
> semantics. But combining the semantics creates security and privacy
> constraints, adds monitoring complexity to the deployment, and removes a
> bunch of potential optimizations.
>

My understanding of an "outbox" is typically "the message is here when I
want it submitted to the mail queue, and removed when that submission has
been accomplished".

Ie, on disconnected clients, this is often where messages are stored
locally before they can be submitted via MSA.

So, the outbox / jmap wouldn't require mail queue semantics, it's not the
mail queue, it's the precursor to the mail queue.

A more advanced solution might be for the message to be in the outbox for
the entire time it's in the mail queue.  I wouldn't implement them as one
and the same, but move a message from the outbox to the sent folder (on
success) or back to drafts or something (on failure) when the message has
exited the mail queue.  Ie, you could use DSN from the mall queue back to
the mail store to cause the messages to move folders.  I don't think anyone
is suggesting this is required for jmap, though.

I often considered something like that for Gmail, as it eliminates the
confusion people often have that a message may be pending in our mail
queues for days.  It does require a fairly integrated system, however.

Brandon

---

From: Brandon Long <[email-masked]>
Date: Wed, 19 Apr 2017 21:46:28 -0700

On Wed, Apr 19, 2017 at 12:29 PM, Chris Newman <[email-masked]>
wrote:

> matter of functional separation. While I strongly agree with the need for
> functional specification, it is possible to design a protocol in such a way
> that multiple separable functions are provided in the same protocol. As a
> result I support adding submission to JMAP in a way that preserves
> functional separation.
>
> When deploying a large mail service it's important to keep the submission
> mail queues (which have management/monitoring issues) functionally separate
> from the mail store (which has very different management/monitoring
> issues). If the JMAP design requires a mail queue to be present on the JMAP
> server or present in the mail store, that is an unacceptable protocol
> design error as it hinders the manageability of large deployments.
>
> existing standard in such a way that it's not necessary to change the JMAP
> specification when a new submission extension that's useful to JMAP is
> added. In addition to giving JMAP clients access to all the useful

SMTPUTF8 I presume.

> want to use the JMAP submission mechanism for all submissions. If JMAP has
> a rigid design where each submission extension requires an additional JMAP
> extension then a high-end client will eventually be forced to implement
> SMTP submission in addition to JMAP submission. And it will be stuck doing
> both (the former for functionality, the latter for ease-of-setup). I don't
> want JMAP clients to be forced down that path.
>

Is there any proof that the majority of clients would bother?  I realize
that's the argument that's always been made about including submission in
IMAP, but is there proof that the majority of clients used by people
implement these?  I mean, SMTPUTF8, sure, should be a requirement for jmap.

What this means is that for JMAP to support submission correctly, it must
> be able to function as an SMTP submission proxy from a semantic viewpoint.
> This creates a number of requirements. The basic requirements are:
>
> 1. It must be possible to query the JMAP server for the set of Submission
> ESMTP extensions semantically equivalent to the SMTP submission EHLO
> response. The JMAP server needs to omit the extensions that aren't relevant
> (e.g., AUTH, STARTTLS, PIPELINING).

2. It must be possible to provide an SMTP envelope for submission that is
> semantically identical to the submission protocol. This needs to support
> MAIL FROM, one or more RCPT TO, and ESMTP parameters to those envelope
> components. There needs to be an algorithmic way to convert from JMAP
> envelope syntax to SMTP Submission envelope syntax.
> 3-digit SMTP code, optionally enhanced SMTP code and human readable text of
> error and potentially one response for mail from, one for each rcpt to and
> one for data/bdat final response.
>

Why?

> provides submission at all. If you want to also provide a simpler model for
> submission in JMAP, I won't object as long as the full model is mandatory.

Maybe that's the simplest way to avoid the argument.

Brandon

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 20 Apr 2017 08:24:52 +0000

------ Original Message ------
From: "Ted Lemon" <[email-masked]>
To: "Chris Newman" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 20/04/2017 1:55:39 PM
Subject: Re: [Jmap] Submission

>>store and mail queue semantics as a mandatory architectural element of 
>>JMAP is a protocol design error because it breaks functional 
>>the mail store as part of the submission action (as RFC 4468 does). 
>>But the JMAP protocol needs to allow implementations where mail queue 
>>and mail store semantics are separate.
>
>place to put information from the site MTA's delivery attempt.   If the 
>failure happens farther down the line in a forwarding chain, we can't 
>do anything about it, but from a usability perspective bounce messages 
>are worse than useless, so anything that can reduce the number of such 
>messages that users see is a net win.
I agree

I'd love to see a mail client that does at least initial validation on 
destination email addresses as they are added to the message, so that 
errors in the addresses which can be resolved at that stage (e.g. 
NXDOMAIN) can be raised with the email author before the mail is even 
submitted.

Due to DNS issues, this may need to be handled on the server, but could 
be something as simple as an MX lookup, or something.  Late bounce 
messages relating to bad domain parts of an email address are 
ridiculously user-unfriendly and we could do so much better.

If we're doing a new mail protocol, we should be targeting the user 
experience we want, rather than just whatever it takes to replicate what 
we have maybe with fewer configuration points.

It should be possible in many cases to even give realtime updates about 
the delivery status of the message whilst it's sitting in some folder 
(whether that's outbox or sent) in the case where the JMAP server is 
also the SMTP client for delivery upstream.

I can imagine sitting at my desk after sending a mail, watching the 
status icon on the message go from submitted to connecting to delivered 
to destination mailbox.  Why not consider something like some way to 
give status updates back to the client for this kind of thing.

People are moving away from SMTP to things like Facebook messaging due 
to these sorts of reasons.  They focused on user experience.  If we 
don't then we are wasting our time.

Adrien

>place in something that is notionally a mail store, and those messages 
>serve as a place where delivery status can be found.   Message 
>submission should not be a transaction: it should be a process.   
>Notification should work even if the MUA isn't present at the time of 
>failure, and should be available in all MUAs, not just the sending MUA. 
>It should be presentable to the user in a way that makes sense.   A 
>"sent" folder makes sense.   A separate queue construct does not.   
>Forcing the MUA to fake this isn't going to produce a good user 
>experience.
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 20 Apr 2017 10:18:10 -0400

On Apr 20, 2017, at 12:36 AM, Brandon Long <[email-masked]> wrote:
> I often considered something like that for Gmail, as it eliminates the confusion people often have that a message may be pending in our mail queues for days.  It does require a fairly integrated system, however.

That seems like a good thing.   In any case, if the facility is present in the spec and is the standard way that MUAs communicate about messages, then whether or not this integration happens really comes down to quality of implementation.   If integration is possible, the UX will be better; if it is not, it will still be perfectly adequate.

---

From: "Chris Newman" <[email-masked]>
Date: Thu, 20 Apr 2017 13:56:48 -0700

On 19 Apr 2017, at 21:46, Brandon Long wrote:
> implement these?

Of course there is no proof related to future actions of third parties. 
But semantic equivalent guarantees the problem won't happen.

> I mean, SMTPUTF8, sure, should be a requirement for jmap.

I'm unsure we should require all JMAP servers to support SMTPUTF8. I 
presently don't have strong feelings one way or the other, but I think 
there are a number of important tradeoffs that should be understood to 
inform the decision.

>> must
>> Submission

These three requirements would give JMAP semantic equivalence to 
standard IETF Submission, which preserves functional separation 
(improving scalability and security/privacy models), and simplifies 
deployment in sites with existing submit servers (which is presently all 
sites). I don't think 1 or 2 should be controversial. Some debate about 
item 3 may be appropriate.

> Maybe that's the simplest way to avoid the argument.

Debating the technical tradeoffs between different proposals is an 
important part of getting a good standard. If you think I'm wrong, it'd 
be good to explain the alternate viewpoint and technical reasoning 
behind that viewpoint. As I work on server software products, I tend to 
have some server bias in designs but I recognize there are tradeoffs 
between ease-of-development/ease-of-deployment for server vs. client 
software. As a long time IETF participant, I also view 
IMAP+Submission+standard-extensions as the starting point for JMAP 
design. That's a different viewpoint from people who view the problem 
entirely from the client viewpoint and ignore the constraints of 
presently deployed infrastructure. A good JMAP standard should probably 
end up somewhere in the middle.

		- Chris

---

From: "Adrien de Croy" <[email-masked]>
Date: Thu, 20 Apr 2017 21:55:56 +0000

My first impression is that if we considered the entire gamut of ESMTP 
submission extensions, most would not be relevant for the client, and 
therefore most would not need to be reflected in JMAP.

The client should not care if the payload is sent chunked, whether the 
next hop server supports pipelining.  The only extensions would relate 
to the things the client needs to know in order to emit a properly 
formed message and envelope.  If we make delivery status updates a core 
(and mandatory) part of JMAP, whether the upstream SMTP infrastructure 
can support it or not is not particularly important.  The JMAP server 
can provide something regardless, and in a tightly-integrated system 
(e.g. one JMAP client sending mail to another on the same server) then 
the user experience can be excellent.  Even things like 8BITMIME should 
be the default, mandatory in the client<->JMAP server and if the 
upstream SMTP system needs to redo the message then that's what it needs 
to do.  It's ridiculous we are still bound to a 7-bit-compatible mail 
transport where the only machines now that are 7 bit are in museums.

I think it would be a shame to get hung up on non-necessary SMTP 
extensions and use that as a justification to not progress on 
improvements.  We need to be sure that we have a realistic but useful 
set of minimum requirements which safeguard an excellent user 
experience, because we need to do better than what is currently there or 
this will not be adopted.

Adrien

------ Original Message ------
From: "Chris Newman" <[email-masked]>
To: "Brandon Long" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 21/04/2017 8:56:48 AM
Subject: Re: [Jmap] Submission

>deployment in sites with existing submit servers (which is presently 
>all sites). I don't think 1 or 2 should be controversial. Some debate 
>about item 3 may be appropriate.
>

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 00:13:29 -0000

>therefore most would not need to be reflected in JMAP.

I haven't heard anyone demand that JMAP support every SMTP extension.
That would be ridiculous -- no SMTP or submission server supports them
all, either.

But what we do need is an architecture that can pass through whatever
extensions the server wants to support, without needing to revise JMAP
each time there's a new extension.

> Even things like 8BITMIME should be the default, mandatory in the
> client<->JMAP server and if the upstream SMTP system needs to redo
> the message then that's what it needs to do.

Those of us who apply DKIM signatures would be very unhappy with that
misdesign.  Sure, an MUA on a phone isn't likely to do that, but
consider my example of list management software running through JMAP,
they do it all the time.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 00:17:02 -0000

>NXDOMAIN) can be raised with the email author before the mail is even 

This is quite the can of worms.  It's easy enough to look up a domain
and see if it has an A or an MX, but whatever you were planning to do
to validate mailboxes will get you blacklisted as a listwashing
spammer.

Once again, I really do not think this is the time or place to try and
fix all of SMTP's faults.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Thu, 20 Apr 2017 20:06:57 -0700 (PDT)

> with this stuff than anyone.

I'm following the discussion, but I really have had nothing substantive to add
to what you and Chris Newman have already said.

In regards to envelope synthesizers, I guess I should note that we find we have
to tweak ours fairly regularly to keep up with interesting new ways various
pieces of software manage to implement someting other than RFC 5322. So not
only is writing a good one difficult, it's something you never really finish.

And since you've brought up the FUTURERELEASE SUBMIT extension, it's worth
pointing out the fundamental incompatibility between, on the one hand, people
saying that the "marinate until", aka FUTURERELEASE, capability is highly
desireable and widely used and other people saying that there are no SUBMIT
extensions worth considering.

More generally, the only difference between a JMAP capability tied to, say,
a magic header and a SUBMIT extension is syntax. But if you specify things
in a way that makes it difficult to map from one to the other, you'll pay
the price for that as long as either protocols continue to evolve - and that's
going to be a long time.

				Ned

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 06:16:42 +0000

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 21/04/2017 12:17:02 PM
Subject: Re: [Jmap] Submission

>to validate mailboxes will get you blacklisted as a listwashing
sure, I think there are fundamental issues with trying to go further 
than an A/MX for now, but if we built in a mechanism to verify 
addresses, it could be extended later, rather than being blocked off for 
all of eternity.

Even A/MX would have some value.  Who knows what kind of trusted mail 
delivery infrastructure may come out in the future, at some stage we 
have to fix SMTP.  Forever is a long time.

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 06:25:48 +0000

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 21/04/2017 12:13:29 PM
Subject: Re: [Jmap] Submission

>each time there's a new extension.
which server?  The upstream SMTP one?

I think trying to pass through SMTP extensions when there may not even 
be an upstream SMTP server is going to be problematic let alone ugly.

Which extensions do we think are even in scope for this?

>consider my example of list management software running through JMAP,
So in 100 years we will still be using the 7-bit limbo pole everyone has 
to go under

We need to move past the 1960s.

When will we abandon 7-bit?

Sounds like DKIM didn't think through the need to transform mail. What 
does a server do right now when an SMTP server needs to send 8bit mail 
to a server that doesn't advertise 8BITMIME?  Bounce it? There should be 
a lossless transform that DKIM could cope with.

I suspect many MTAs will happily accept binary mail, but just don't 
advertise 8BITMIME because all the other requirements were far too 
onerous.

Adrien

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Apr 2017 09:43:56 -0400

> the price for that as long as either protocols continue to evolve - and that's

I think the distinction comes from whether you actually think that the SMTP submission protocol is always going to be what's behind a message submitted via JMAP.   If you think that, then it's going to look really weird to not just replicate that exact paradigm with the JMAP submission process.

However, it's not at all clear to me that that makes any general sense.   I'm sure it makes sense in some implementations, but if we were to go that route, we would essentially be nailing JMAP to SMTP submission.   There's some value to doing this in that it avoids reinventing the wheel, but there is also damage, in that the SMTP submission paradigm really isn't particularly congruent to the general paradigm of JMAP.

The point is that whichever choice we make, we are going to pay a price for it.   It's worth asking what that price is, and also asking what the advantages are to the choices we have, and then deciding on that basis.   The fact that we would pay a price for not replicating SMTP submission can definitely be taken as a given.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 21 Apr 2017 15:34:44 +0100

Ted Lemon writes:
> I think the distinction comes from whether you actually think 
> that the SMTP submission protocol is always going to be what's 
> behind a message submitted via JMAP.   If you think that, then 
> it's going to look really weird to not just replicate that exact 
> paradigm with the JMAP submission process.

Hm?

Won't the nexthop generally be the thing that the port-587 server forwards 
to? After all, JMAP already handles authentication, and it's not clear to 
me that a typical JMAP server is easily able to authenticate on behalf of 
the end user in a way that a port-587 server accepts.

Assuming that JMAP forwards to the filter or smarthost, then requiring that 
JMAP servers be able to forward SMTP extensions looks really odd. A 
submission server on port 587 cannot forward SMTP extensions from its 
nexthop, why should one on port 443 have to?

The usual answer is "so as to not need a corresponding JMAP RFC whenever 
there's a new SMTP RFC". That strikes me as poor. A new extension requires 
an RFC and three kinds of implementations (SMTP client, submission server, 
filter server). The RFC publication bit isn't usually the bottleneck among 
those four.

> The point is that whichever choice we make, we are going to pay 
> a price for it.   It's worth asking what that price is, and also 
> asking what the advantages are to the choices we have, and then 
> deciding on that basis.   The fact that we would pay a price for 
> not replicating SMTP submission can definitely be taken as a 
> given.

+1

Arnt

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Apr 2017 11:02:24 -0400

On Apr 21, 2017, at 10:34 AM, Arnt Gulbrandsen <[email-masked]> wrote:
> Won't the nexthop generally be the thing that the port-587 server forwards to? After all, JMAP already handles authentication, and it's not clear to me that a typical JMAP server is easily able to authenticate on behalf of the end user in a way that a port-587 server accepts.

This would have been my assumption, but we seem to be hearing different things from some folks who are operating large installations, so I think it's worth discussing.

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 15:16:36 -0000

In article <[email-masked]> you write:
>I think the distinction comes from whether you actually think that the SMTP submission protocol is always going to be what's behind a message
>submitted via JMAP.   If you think that, then it's going to look really weird to not just replicate that exact paradigm with the JMAP submission process.

Um, this is the IETF.  If you want to invent a new, different, and
incompatible mail system, JMAP is definitely not the place to do so. 

I am scratching my head about why people are having such trouble with
the submission model.  It's not complicated.  The message goes into a
queue, the envelope and message are separate, the envelope addresses
can have annotations, and the server tells the client what optional
features it supports.  That's it.  The options are, you know,
optional, but many of them are useful.

Before someone suggests that we should use some better scheme, I would
point out that a lot of mail transport designs have come and gone over
the past 40 years, and all of the others are dead.  You're going to
need a really good story about why this time is different.

There's also a lot of issues that are really quality of
implementation.  For example, I agree that a server that doesn't
support 8BITMIME would be pretty lame, and I would not blame a client
that refused to work with it (and unsurprised to find that the MUA on
my phone already does that.)  But the place to deprecate not-8BITMIME
is in 5321bis, not here.

Finally, if people want to try whizbang ways to show animations of
mail zipping through the aether, that sounds like a swell optional
add-on to try and implement and test and perhaps bring back later as
an extension, but it'd be nuts to try to invent it now.

R's,
John

---

From: Ned Freed <[email-masked]>
Date: Fri, 21 Apr 2017 08:02:32 -0700 (PDT)

> I think the distinction comes from whether you actually think that the SMTP
> submission protocol is always going to be what's behind a message submitted via
> JMAP. If you think that, then it's going to look really weird to not just
> replicate that exact paradigm with the JMAP submission process.

I don't think that at all and I doubt anyone else does either, so this 
distinction seems completely irrelevant to me.

The issue is semantics, not syntax. If your design locks you in to a
particular set of semantics that cannot be extended - which is what any
design that fails to allow for capability discovery and attachment of
arbitrary parameters - you've put yourself in a position where you need
to continually revise things in order to keep them synchronized as the
capabilities of the transport infrasture continue to evolve.

None of this is in any way specific to SUBMIT or SMTP.

> However, it's not at all clear to me that that makes any general sense.   I'm
> sure it makes sense in some implementations, but if we were to go that route,
> we would essentially be nailing JMAP to SMTP submission.   There's some value
> to doing this in that it avoids reinventing the wheel, but there is also
> damage, in that the SMTP submission paradigm really isn't particularly
> congruent to the general paradigm of JMAP.

The message transport process is precisely defined as a series of steps that
begin with submission and end with delivery. Nothing in that process says
that SUBMIT has to be used for submission - and it frequently isn't in
practice - but in order to be compatible with the current transport
infrastructure submission does have to occur and where it occurs has to be
well defined.

> The point is that whichever choice we make, we are going to pay a price for
> it.   It's worth asking what that price is, and also asking what the advantages
> are to the choices we have, and then deciding on that basis.   The fact that we
> would pay a price for not replicating SMTP submission can definitely be taken
> as a given.

I'm afraid that's really not the point at all. There are always costs on
both sides of a decision; my point is that the costs on one side of this
decision are significant and ongong, whereas on the other, not so much.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Apr 2017 11:27:57 -0400

> capabilities of the transport infrasture continue to evolve.

Yes, I hear this concern.   I'm saying that it's to be weighed against the benefits of making the JMAP submission protocol primary, rather than nailing it to SMTP submission.   If you nail JMAP submission to SMTP submission, you are excluding any functionality that can't be provided by SMTP submission.   

And as a practical matter, as Arnt said, if SMTP submission continues to be the place where innovation occurs, then this means implementations will have to follow that.   I don't think it's fair to assume that preserving the semantics of SMTP submission in JMAP submission is going to get you to a place of win.

Any JMAP implementation that doesn't use SMTP submission as a backend  is still going to have to track SMTP submission RFCs, or else there will be interop problems.   For such implementations, the need for a JMAP RFC to track relevant new functionality added to SMTP submit is good, not bad.

For the use case you are concerned about, I think it doesn't matter; the point I am making here is just that that is not the only use case we ought to consider.

---

From: Ned Freed <[email-masked]>
Date: Fri, 21 Apr 2017 08:18:37 -0700 (PDT)

> optional, but many of them are useful.

Exactly right. And the envelope format can and should be dead easy to generate
- dare I suggest using JSON? - making this quite easy for clients to do.

> need a really good story about why this time is different.

Having spent considerable time implementing most of the other designs that have
come and gone over the years, one of the conclusions I've reached is that full
envelope separation is a key design principle along the same lines as the
end-to-end principle. It's a shame it isn't recognized as such.

				Ned

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 15:32:46 -0000

>be an upstream SMTP server is going to be problematic let alone ugly.

It seems very likely that there will be a submission server, if there
is any hope of sending mail to the billion people who use SMTP mail
now.

>Which extensions do we think are even in scope for this?

All of them.  A more interesting question is which ones are likely to
be useful in this context.  A quick look suggests 8BITMIME, SUBMITTER,
DSN, DELIVERBY, FUTURERELEASE, SMTPUTF8, CONPERM, MT-PRIORITY, RRVS,
and maybe BURL.

R's,
John

PS:

>Sounds like DKIM didn't think through the need to transform mail. ...

How about if you read the dkim list archives and get back to us on that?

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 16:00:47 -0000

>nexthop, why should one on port 443 have to?

Nobody's asking for that.  What we need are the extensions that the
submission server supports.

Perhaps some people here (not you) are unaware that submission and
SMTP are different protocols but share the same set of extensions.

R's,
John

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 22:22:39 +0000

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 3:32:46 AM
Subject: Re: [Jmap] Submission

>is any hope of sending mail to the billion people who use SMTP mail

sure, but conversely there very often will not be.  Intra-organizational 
mail for example is often not forwarded with SMTP.

Also if we don't prevent it, inter-organizational mail could use this as 
well.  The key is to not make that impossible or worthless.

SMTP is just not being fixed.  Until the IETF decides enough is enough, 
and agents acting on behalf of humans must no longer do things 
anonymously (and this means using certificates at any point mail is 
injected into the system so that these can be revoked on abuse), we will 
continue to be inundated with spam, with all the down-stream impact that 
has on the usefulness of electronic mail.

Just because there is a cost to doing something, doesn't necessarily 
mean that cost would not be well-spent.  There's an ongoing cost of 
leaving things the same as well.

All costs are very difficult to quantify, and therefore it's very 
difficult to determine which path we should take on a purely cost-basis.

But we could also consider maybe that if we actually came up with a 
design which was compelling enough, then adoption could make it 
worth-while.  People are sick of the problems with SMTP, and sick of the 
lack of progress addressing them.  We should be learning from things 
like Facebook messaging.

Good anti-spam is masking the issue, which is still growing, but the 
situation is actually insane.  How many legitimate mails are you missing 
out on because of the constant deluge of spam?

>How about if you read the dkim list archives and get back to us on 

I'm sure it was contentious.  Just saying it seems a bit silly to define 
a system that requires mail to not be transformed, and then layer it 
over a transport that may need to transform mail.  Just sayin'

Adrien

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 22:34:13 +0000

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 3:16:36 AM
Subject: Re: [Jmap] Submission is not hard
>an extension, but it'd be nuts to try to invent it now.

IME, if we don't design it now, it will never happen.  Surely it's 
important enough to do the work up front?

ll I'm suggesting here is that in the new MUA to JMAP protocol, we allow 
for notifications about delivery, and not use the MDNs for this (between 
client and JMAP server).

When a JMAP server needs to forward a message via SMTP, then it can use 
existing SMTP extensions for this if supported, or tell the client that 
it won't be getting any more delivery status updates for that message.

For internal mail that doesn't go anywhere near SMTP, it could be a very 
rich experience.

Eventually when organisations using this build enough web of trust to 
use it for inter-organizational messaging, the rich experience will come 
for free.  This is one way to provide a compelling reason to make a 
change.

Because sorry, but bounce messages are out of the dark ages.  We need an 
attitude change here, we're competing against systems that aren't bound 
by the need for consensus, and are just able to provide service and 
function which is enticing users away from our systems.  This is making 
existing systems weaker and gradually obsolete.

Cheers

Adrien

---

From: "John R Levine" <[email-masked]>
Date: 21 Apr 2017 18:46:36 -0400

> IME, if we don't design it now, it will never happen.  Surely it's important 
> enough to do the work up front?

If our goal is to drag JMAP down in an endless swamp of vaporware design, 
I'd entirely agree.  Otherwise, no.

R's,
John

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 22:52:30 +0000

It's only vapourware if nobody implements it.

MUA designers tend to be interested in things like usability.

Adrien

------ Original Message ------
From: "John R Levine" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 10:46:36 AM
Subject: Re: [Jmap] Submission is not hard

>If our goal is to drag JMAP down in an endless swamp of vaporware 
>design, I'd entirely agree.  Otherwise, no.
>
>R's,
>John
>

---

From: "John R Levine" <[email-masked]>
Date: 21 Apr 2017 18:54:40 -0400

> well.  The key is to not make that impossible or worthless.

Hmmn.  I have written submission servers.  They are not particularly 
complicated or difficult, particularly if they don't have to handle the 
crypto goop in STARTTLS and AUTH, both of which JMAP does other ways.

Assuming you mean intra-organizational mail, it would be a big help if you 
could explain concretely what about the standard submission model would 
make such mail impossible or worthless.  Please be specific.

R's,
John

---

From: "John R Levine" <[email-masked]>
Date: 21 Apr 2017 18:55:18 -0400

> MUA designers tend to be interested in things like usability.

Well, sure.  We eargerly await your prototypes.

Regards,

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 23:09:25 +0000

I did mean inter-organizational.  E.g. external messaging.

------ Original Message ------
From: "John R Levine" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 10:54:40 AM
Subject: Re: [Jmap] Submission

>>Also if we don't prevent it, inter-organizational mail could use this 
>>as well.  The key is to not make that impossible or worthless.
>
>Assuming you mean intra-organizational mail, it would be a big help if 
>you could explain concretely what about the standard submission model 
>would make such mail impossible or worthless.  Please be specific.
>
>R's,
>John
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 23:14:23 +0000

If I had a mail client I would do it already.  I can only commit to 
doing it for the server side.

In general, I think any system which involves humans, where things 
happen which cause the system to require the human to wait for something 
should be designed in a way that allows for status updates to be given 
to the human in a timely fashion.

This is why I wrote the draft for status updates in http.

This is why I suggest we allow for status notifications in mail.

because when you send a message, you have to wait to find out what 
happened.  And sometimes it's important.

When people are forced to wait for things for too long without any 
feedback, they start doing things (like getting on the phone to see if 
the message got through).  Often these things are unnecessary and 
wasteful of resources.  More wasteful than a system that provided 
adequate feedback in the first place.

Designing protocols with this in mind allows implementors of these 
systems to treat the humans (their customers) politely.  Leaving 
feedback / notifications out of the protocols puts the implementers in a 
position where they are forced to treat their users badly / rudely.

Adrien

---

From: "John Levine" <[email-masked]>
Date: 21 Apr 2017 23:19:22 -0000

>I did mean inter-organizational.  E.g. external messaging.

In that case, I don't understand your point at all.  Existing mail
software uses the IETF submission protocols to send billions of
inter-organizational messages every day, frequently from rather
limited clients like a 2G flip phone or the laser printer on my desk
or the NAS in the server room.

Given the multi-decade track record here, what makes this so daunting
if you wrap some JSON around it?

R's,
John

---

From: "Adrien de Croy" <[email-masked]>
Date: Fri, 21 Apr 2017 23:30:46 +0000

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 11:19:22 AM
Subject: Re: [Jmap] Submission

>if you wrap some JSON around it?

because in that case we're not looking to improve anything.

My point is that (wrt submision) if we design a system for MUA->"the 
system" which allows for a rich experience for the MUA, then if we don't 
make it impossible for this protocol to be used to extend the system 
outside of the organisation, then we can get the rich experience for all 
mail, not just mail to your colleagues on the same server.

I'm imagining a system where the JMAP server uses JMAP to send it the 
next hop as well if required, and so on, thereby avoiding the 
limitations of SMTP.

---

From: "John R Levine" <[email-masked]>
Date: 21 Apr 2017 20:09:47 -0400

> I'm imagining a system where the JMAP server uses JMAP to send it the next 
> hop as well if required, and so on, thereby avoiding the limitations of SMTP.

Well, then, have fun in your walled garden.

Regards,

---

From: "Adrien de Croy" <[email-masked]>
Date: Sat, 22 Apr 2017 00:28:14 +0000

I'm not suggesting a walled garden.

I'm suggesting that eventually SMTP can become the true relic it is.

We've had decades to fix SMTP.  It's not happening.  Especially when 
protocol designers don't seem to care about user experience beyond basic 
interop.

we can add extensions to SMTP until the cows come home.  We could build 
all these things into SMTP (and maybe we should attempt to).  But the 
legacy issues will continue to hamper it, until we decide to make a 
clean break from them (e.g. like deprecating non binary transport in 
SMTP).  We're moving too slowly, and the users are already voting with 
their feet.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Apr 2017 21:00:30 -0400

On Apr 21, 2017, at 7:30 PM, Adrien de Croy <[email-masked]> wrote:
> I'm imagining a system where the JMAP server uses JMAP to send it the next hop as well if required, and so on, thereby avoiding the limitations of SMTP.

I really don't think this is likely.   However, that doesn't mean that fixing the first hop is a bad idea.   I suspect that what will happen is that when we are done with this work, if there is demand to improve the experience for subsequent hops, the pressure will probably arrive the same way the pressure for JMAP did: from large sites, where large wins can be had by implementing something like this, even if it is not universal.

IOW, we should do the JMAP part right, in hopes that things will improve in the rest of the system, not tune the JMAP solution to the lowest common denominator.

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 21 Apr 2017 18:02:55 -0700

On 20 Apr 2017, at 23:16, Adrien de Croy wrote:
>> In article <emd3533c0d-dc08-45c3-801c-07972858ad64@bodybag> someone 
>> write:
> addresses, it could be extended later, rather than being blocked off 
> for all of eternity.
>

The present model for MUA mail address validation is to authenticate to 
an IETF standard submission server, issue a MAIL FROM + RCPT TO + RSET. 
Anyway, I support adding a feature in JMAP that does the equivalent of 
performing this sequence of commands on the Submission server and 
returns a 3-digit SMTP status code, optionally SMTP enhanced status 
codes (see RFC 3463 section 3.2) and text string.

This is quite extensible and I believe this specific functionality is 
in-scope of the JMAP charter as Submission (RFC 6409) is explicitly 
in-scope. Doing address validation this way is already quite powerful. 
For example, if the submission server chooses to implement the SMTP 
251/551 response codes for local users, the mail client gains the 
ability to prompt the user to correct the address book entry for that 
recipient.

How much validation the submission server does then becomes a 
quality-of-implementation issue (and there are, of course, 
security/privacy issues). Future work would be to define additional 
SMTP/Submit enhanced status codes for additional address validation, but 
that's presently out-of-scope for the JMAP charter.

The JMAP charter does not permit development of a new address validation 
mechanism and I object to development of a new incompatible validation 
mechanism on the grounds that having two mechanisms to perform the same 
function is poor design, when a full-standard mechanism to do address 
validation already exists. I support mapping the existing Submission 
validation mechanism to JMAP in a way that preserves all the semantics 
of the existing mechanism.

		- Chris

---

From: "Adrien de Croy" <[email-masked]>
Date: Sat, 22 Apr 2017 01:34:04 +0000

"two mechanisms to perform the same function is poor design"

Sorry I missed this one on the Mt Sinai tablets.

We are now and have throughout history been completely inundated with 
multiple designs and implementations of things for the same function. 
History has proven this to be a vital part of evolution.

Stating this is therefore by definition somehow "poor design" seems like 
an argument for never improving anything.

It could just as well be argued that if the first incarnation were not 
poor design there would be no need for the existence of (let alone 
proposals for) the second and subsequent.  So maybe with that 
interpretation you're right.

In any case validation upon submission of only local addresses is a long 
way short of validation prior to submission (during composition) of 
local and remote addresses, or at least even just allowing for the 
possibility of this.  It's too little too late.

Adrien

------ Original Message ------
From: "Chris Newman" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 1:02:55 PM
Subject: [Jmap] Address Validation (was Re: Submission)

>SMTP/Submit enhanced status codes for additional address validation, 
>but that's presently out-of-scope for the JMAP charter.
>
>The JMAP charter does not permit development of a new address 
>validation mechanism and I object to development of a new incompatible 
>validation mechanism on the grounds that having two mechanisms to 
>perform the same function is poor design, when a full-standard 
>mechanism to do address validation already exists. I support mapping 
>the existing Submission validation mechanism to JMAP in a way that 
>preserves all the semantics of the existing mechanism.
>
>   - Chris
>

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 21 Apr 2017 18:48:09 -0700

On 21 Apr 2017, at 7:34, Arnt Gulbrandsen wrote:
> Ted Lemon writes:
>> I think the distinction comes from whether you actually think that 
>> the SMTP submission protocol is always going to be what's behind a 
>> message submitted via JMAP.   If you think that, then it's going to 
>> look really weird to not just replicate that exact paradigm with the 
>> JMAP submission process.
>
> Hm?
>
> Won't the nexthop generally be the thing that the port-587 server 
> forwards to? After all, JMAP already handles authentication, and it's 
> not clear to me that a typical JMAP server is easily able to 
> authenticate on behalf of the end user in a way that a port-587 server 
> accepts.

There are lots of standard ways for a proxy to authenticate on behalf of 
another user to another server. SASL PLAIN over TLS is the most obvious, 
but a TLS client certificate plus SASL EXTERNAL also works. The OAuth 
SASL mechanism could also be useful if it was designed correctly (sorry, 
I haven't reviewed it in detail with respect to that use case).

My product's existing JMAP-like proxy is a JSON/HTTP to submission/IMAP 
proxy. One of the design errors that was made (back in the 1990s) was to 
not support submission extensions. This is one of the largest remaining 
design errors in our proxy and has caused significant problems. I do not 
want the JMAP standard to repeat the mistake we made.

> Assuming that JMAP forwards to the filter or smarthost, then requiring 
> that JMAP servers be able to forward SMTP extensions looks really odd. 
> A submission server on port 587 cannot forward SMTP extensions from 
> its nexthop, why should one on port 443 have to?

Submission servers do forward hop-to-hop extensions to the next hop. 
It's how they have to work. SMTP extensions like DELIVERBY, RRVS, 
SMTPUTF8, MTRK require hop-to-hop forwarding of SMTP parameters.

> The usual answer is "so as to not need a corresponding JMAP RFC 
> whenever there's a new SMTP RFC". That strikes me as poor. A new 
> extension requires an RFC and three kinds of implementations (SMTP 
> client, submission server, filter server). The RFC publication bit 
> isn't usually the bottleneck among those four.

We're going to need submission servers until all useful software is 
updated to use JMAP submission. I'd guess that at many sites that will 
never happen. So if we must have SMTP submission, then it's desirable to 
keep JMAP submission as close in semantics to SMTP submission as 
possible in order to minimize the attack surface and code complexity of 
the system. Yes, there are some specific tradeoffs where there might be 
a good reason to deviate from full submission semantics; for example, 
I'm unsure per-recipient status is useful to most clients so an option 
where any failed recipient results in failed submission would make sense 
to me in JMAP. And, of course, JMAP submission should be one round-trip 
and may have helpful side-effects like moving a message from the \Drafts 
folder to \Sent folder on success.

>> The point is that whichever choice we make, we are going to pay a 
>> price for it.   It's worth asking what that price is, and also asking 
>> what the advantages are to the choices we have, and then deciding on 
>> that basis.   The fact that we would pay a price for not replicating 
>> SMTP submission can definitely be taken as a given.
>
> +1
>
> Arnt

		- Chris

---

From: Ted Lemon <[email-masked]>
Date: Fri, 21 Apr 2017 22:16:49 -0400

On Apr 21, 2017, at 9:48 PM, Chris Newman <[email-masked]> wrote:
> My product's existing JMAP-like proxy is a JSON/HTTP to submission/IMAP proxy. One of the design errors that was made (back in the 1990s) was to not support submission extensions. This is one of the largest remaining design errors in our proxy and has caused significant problems. I do not want the JMAP standard to repeat the mistake we made.

Can you give an example of a failure mode that resulted from this omission?  (Not doubting you, just curious what the specific problem was).

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 21 Apr 2017 19:26:36 -0700

> important enough to do the work up front?

I believe that client-convenient message tracking status is a desirable 
feature of a mail system. But the real world shows that tracking status 
and automated DSN/MDN handling is not sufficiently important for clients 
to have used existing facilities to provide that functionality; even 
though it's completely possible to do so. So no, I do not think this 
feature should be included in the base JMAP specification and believe 
the client developer community has spoken through their actions that it 
is not high priority. Furthermore, Message Tracking functionality is 
presently out of scope for the JMAP charter. The WG Chair can shut down 
any significant discussion on the topic to keep us on track to deliver 
the base JMAP specification.

What I'd consider appropriate to discuss on this list is if JMAP is 
extensible in such a way that we can add client-convenient tracking 
status to JMAP in a future extension. Extensibility is critical for a 
base specification to get right.

Here's an example client-convenient tracking model: have a 
server-mutable object attached to messages in the \Sent folder that can 
convey tracking status along the lines of RFC 3887 + DSN + MDN. I 
believe that's a completely workable model for message tracking that can 
be added in the future as a JMAP extension. Such an extension could go 
further and allow the JMAP view of the mailstore to convert DSN and MDN 
to status on the message in the \Sent folder (if such a message can be 
found) for clients that want the convenient delivery/tracking/MDN-status 
feature.

So the question that's in-scope for the working group is if there's 
anything in the present design of JMAP that would prevent an extension 
along those lines?

		- Chris

---

From: "Adrien de Croy" <[email-masked]>
Date: Sat, 22 Apr 2017 02:34:06 +0000

yes, also curious about this.  Real world experience is always good to 
learn.

------ Original Message ------
From: "Ted Lemon" <[email-masked]>
To: "Chris Newman" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "Arnt Gulbrandsen" 
<[email-masked]>
Sent: 22/04/2017 2:16:49 PM
Subject: Re: [Jmap] Submission

>>My product's existing JMAP-like proxy is a JSON/HTTP to 
>>submission/IMAP proxy. One of the design errors that was made (back in 
>>the 1990s) was to not support submission extensions. This is one of 
>>the largest remaining design errors in our proxy and has caused 
>>significant problems. I do not want the JMAP standard to repeat the 
>>mistake we made.
>
>Can you give an example of a failure mode that resulted from this 
>omission?  (Not doubting you, just curious what the specific problem 
>was).
>

---

From: "Adrien de Croy" <[email-masked]>
Date: Sat, 22 Apr 2017 02:42:31 +0000

I think this is just another facet of the problem around implementation 
uptake of optional features.

There's still enough uncertainty about whether the entire end-to-end 
path of a message will support the envelope options to cast doubt onto 
any decision to implement it.  A lot of the specs for these options have 
quite onerous requirements for dealing with non-compliant systems (those 
that do not advertise the extension).

So I think this becomes a circular argument.  We don't make new things 
because history has shown new things don't get implemented.  New things 
don't get implemented because other existing things don't get upgraded 
to add them.

So I believe that's an argument to put things in from the start, rather 
than deferring things.

So much pain goes into trying to figure out how to deal with legacy 
non-compliant systems.

So I agree with Ted, we should do JMAP right from the start, and that 
can then provide some real benefit on the first hop, and real incentive 
for subsequent hops rather than a lowest-common-denominator approach.  
So IMO the starting point should be what can be the best possible first 
hop, in terms of best possible user experience.  Because for 
intra-server messaging, we can get (and then demonstrate/prove the value 
of) the benefit of this from the outset.

Adrien

------ Original Message ------
From: "Chris Newman" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 22/04/2017 2:26:36 PM
Subject: [Jmap] Message Tracking and JMAP extensibility (was Re: 
Submission is not hard)

>and automated DSN/MDN handling is not sufficiently important for 
>clients to have used existing facilities to provide that functionality; 
>even though it's completely possible to do so. So no, I do not think 
>this feature should be included in the base JMAP specification and 
>believe the client developer community has spoken through their actions 
>that it is not high priority. Furthermore, Message Tracking 
>functionality is presently out of scope for the JMAP charter. The WG 
>Chair can shut down any significant discussion on the topic to keep us 
>on track to deliver the base JMAP specification.
>
>believe that's a completely workable model for message tracking that 
>can be added in the future as a JMAP extension. Such an extension could 
>go further and allow the JMAP view of the mailstore to convert DSN and 
>MDN to status on the message in the \Sent folder (if such a message can 
>be found) for clients that want the convenient 
>delivery/tracking/MDN-status feature.
>

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 21 Apr 2017 20:08:23 -0700

On 21 Apr 2017, at 19:16, Ted Lemon wrote:
>> submission/IMAP proxy. One of the design errors that was made (back 
>> in the 1990s) was to not support submission extensions. This is one 
>> of the largest remaining design errors in our proxy and has caused 

A rigid proxy design means complexity-increasing changes have to be made 
to three pieces of software instead of just two pieces of software. That 
means the entire system is more complex and less agile. With a proxy 
design that's well-aligned to the submission protocol, most submission 
extensions can be supported with just a configuration change (to pass 
through the new extension). So an extension increases system complexity 
in only two components with the better design.

For our product, we have different developers on each component 
(Submission vs. HTTP proxy vs. Client UI) and some of the developers 
have a significant timezone shift. I leave the problems that causes to 
the imagination of the reader.

		- Chris

---

From: "Chris Newman" <[email-masked]>
Date: Fri, 21 Apr 2017 20:20:18 -0700

On 21 Apr 2017, at 19:42, Adrien de Croy wrote:
> I think this is just another facet of the problem around 
> implementation uptake of optional features.
>
> any decision to implement it.  A lot of the specs for these options 
> have quite onerous requirements for dealing with non-compliant systems 
> (those that do not advertise the extension).
>
> because history has shown new things don't get implemented.  New 
> things don't get implemented because other existing things don't get 
> upgraded to add them.

If JMAP is well aligned with submission, that's one less barrier to 
deployment of extensions. If it's not well aligned, then we have to 
change another spec and another piece of software to get something new 
deployed. I agree we should not make the extension uptake problem worse 
by having JMAP out of semantic alignment with Submission for extensions.

> So I believe that's an argument to put things in from the start, 
> rather than deferring things.

I made this exact argument many times while developing the ACAP base 
specification. It resulted in a 44-page specification that was unwieldy 
to implement and thus failed to deploy. I am opposed to changing the 
JMAP working charter to allow us to go down that path for message 
tracking/status functionality. I take this position because I want JMAP 
to deploy and I try to learn from my past mistakes.

At this point, I don't think I have anything more to add on the message 
tracking/status issue.

		- Chris

> can then provide some real benefit on the first hop, and real 
> incentive for subsequent hops rather than a lowest-common-denominator 
> approach.  So IMO the starting point should be what can be the best 
> possible first hop, in terms of best possible user experience.  
> Because for intra-server messaging, we can get (and then 
> demonstrate/prove the value of) the benefit of this from the outset.
>
> Adrien
>
>
>> I believe that client-convenient message tracking status is a 
>> desirable feature of a mail system. But the real world shows that 
>> tracking status and automated DSN/MDN handling is not sufficiently 
>> important for clients to have used existing facilities to provide 
>> that functionality; even though it's completely possible to do so. So 
>> no, I do not think this feature should be included in the base JMAP 
>> specification and believe the client developer community has spoken 
>> through their actions that it is not high priority. Furthermore, 
>> Message Tracking functionality is presently out of scope for the JMAP 
>> charter. The WG Chair can shut down any significant discussion on the 
>> topic to keep us on track to deliver the base JMAP specification.
>>
>> server-mutable object attached to messages in the \Sent folder that 
>> can convey tracking status along the lines of RFC 3887 + DSN + MDN. I 
>> can be added in the future as a JMAP extension. Such an extension 
>> could go further and allow the JMAP view of the mailstore to convert 
>> DSN and MDN to status on the message in the \Sent folder (if such a 
>> message can be found) for clients that want the convenient 
>> anything in the present design of JMAP that would prevent an 
>> extension along those lines?
>>
>>   - Chris
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_jmap&d=DwIFaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=QBZgPENFbjFadxqU4HJ3ZDpRz3X1JlDY-keqMt52FFo&m=3-MZtEL_JnGA64bIHNq8ZGszP3FJT07irtG2_lYIYrk&s=yXtYh6_dIHPe2bohL8G5YBE_9y5bxVWBqwJgKr_Pvzs&e=

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Apr 2017 16:13:36 -0700 (PDT)

> Yes, I hear this concern. I'm saying that it's to be weighed against the
> benefits of making the JMAP submission protocol primary, rather than nailing it
> to SMTP submission.

This is a entirely false dichotomy of choices you have going here. Nobody is
proposing "nailing it to SUBMIT". All anyone is saying is that you need to
have a means of specifying the envelope separate from the message and its
headers, and that the envelope needs to be extensible.

Aside from the fact that SUBMIT happens to be a protocol that implements these
semantics (actually with some limitations that probably should be removed in
JMAP), this has nothing to do with using SUBMIT.

If, given this framework or a superset of it, you want you implement mechanisms
that cannot be implemented in SUBMIT, that's an entirely separate discussion,
and I look forward to the specification describing such a mechanism.

The long and short of it is I really don't understand why you persist in
worrying about SUBMIT, but since neither of these fundamental concepts we're
concerned with having in JMAP originated in SUBMIT or even SMTP, maybe it would
help to name them after where they did (AFAIK) originate.

So if I say "IFIP style envelope/message separation" and "X.400-1988
style extensible envelope parameters", does it help?

> And as a practical matter, as Arnt said, if SMTP submission continues to be
> the place where innovation occurs, then this means implementations will have to
> follow that.

Indeed it does. And without having the right semantics they will do so slowly
and painfully.

> I don't think it's fair to assume that preserving the semantics of SMTP
> submission in JMAP submission is going to get you to a place of win.

Again with characterizing this as all about SUBMIT. It isn't.

> Any JMAP implementation that doesn't use SMTP submission as a backend  is
> still going to have to track SMTP submission RFCs, or else there will be
> interop problems.

Actually, no, it won't. We've been down this path before with nonextensible
submission APIs of various sorts. It has not gone well.

> For such implementations, the need for a JMAP RFC to track relevant new
> functionality added to SMTP submit is good, not bad.

So rather than having a clean extensible architecture with minimal costs to
use, you're arguing that we should start with a much more limited architecture 
that requires unnnecessary specifications to extend?

This strikes me as fundamentally wrongminded.

> For the use case you are concerned about, I think it doesn't matter; the
> point I am making here is just that that is not the only use case we ought to
> consider.

If you can name a possible use case that is hindered by having an extensible
separate envelope, I'd love to hear what it is and why this is the case.

				Ned

---

From: Ted Lemon <[email-masked]>
Date: Sun, 23 Apr 2017 19:49:15 -0400

> headers, and that the envelope needs to be extensible.

If that's the case, then I don't think there's anything to argue about.   But I've been getting the impression that that's not the case.   I am fully in agreement with the assertion, with which I don't recall anyone disagreeing, that the envelope has to be separate from the message and has to be extensible.

My concern is with avoiding a requirement that JMAP submit have the same _flow_ as SMTP submit.   So I want it to be the case that status can be delivered asynchronously, essentially.   I don't mind if it's also possible to deliver it synchronously, since that's just a degenerate case of asynchronous.

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Apr 2017 16:37:29 -0700 (PDT)

> to?

That's highly unlikely. There's quite a lot of activity associated with
message submission: Access control, auditing, spam/virus checks, content
filtering, etc. JMAP doesn't make any of that magically vanish.

Of course this doesn't mean that messages are going to be submitted using
the SUBMIT protocol. But there has to be some kind of agent that performs
these functions, irrespective of how you talk to it.

> After all, JMAP already handles authentication, and it's not clear to

Proxy authentication is widely if not universally supported using any
number of techniques. And again, you don't get rid of the need to
perform any number of actions based on the submitters identity just because
the user already authenticated to JMAP.

> JMAP servers be able to forward SMTP extensions looks really odd.

I have no idea what this "filter or smarthost" thing is that you're talking
about, but I can assure you that if it isn't directly integrated into the
SUBMIT server so it has direct access to credentials and can return immediate
failures rather than delayed DSNs, you have a serious implementation problem.

> A submission server on port 587 cannot forward SMTP extensions from its

Of course it can. Ours does this routinely.

> filter server). The RFC publication bit isn't usually the bottleneck among

I guess you missed the recent discussion about the incredibly high cost of
RFCs. (Not that I agree, but one of the reasons I disagree is that we've worked
hard to eliminate the need to generate gratuitous material from the process.)

				Ned

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 00:29:40 +0000

------ Original Message ------
From: "Ned Freed" <[email-masked]>
>the user already authenticated to JMAP.
I can think of one major platform (Windows) where this approach will be 
highly problematic.

Are we talking about a JMAP proxy (e.g. the client talks real-time via 
the JMAP server to the SUBMIT server), or are we talking about a system 
where the JMAP server accepts the message, and MAYBE forwards it via 
SMTP or SUBMIT depending on the destination.

If the latter, I would suggest that building a system that requires 
individual user credentials to be presented to the next hop will be 
highly problematic.

Adrien

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Apr 2017 16:50:54 -0700 (PDT)

> My concern is with avoiding a requirement that JMAP submit have the same
> _flow_ as SMTP submit.

Huh? Nobody has said that. AFAIK nobody has even hinted at that.

> So I want it to be the case that status can be delivered asynchronously,
> essentially.   I don't mind if it's also possible to deliver it synchronously,
> since that's just a degenerate case of asynchronous.

That's a completely different matter, and in that regard, I'll just make
a few observations:

(0) Synchronous delivery of status is, with the exception of a handful of
    egregiously broken clients, always going to provide the better user
    experience. This has everything to do with how people are wired and
    nothing to do with the technology.

(1) SUBMIT and SMTP both have the ability to deliver status both synchronously
    and asynchronously. We call the latter "DSNs". They have not worked out
    well.

(2) Asynchronous delivery of status information is NOT a degenerate case
    of synchronous delivery. For example, it's entirely possible for a SUBMIT
    server to always return success status codes and issue a DSN later. (In
    fact our SUBMIT server can be set to do exactly this, in order to
    deal with the aforementioned egregiously broken clients.)

(3) The argument that DSNs being in-band rather than an out-of-band mechanism
    is the root cause of all the problems in this area doesn't fly. Out-of-band 
    mechanisms for message status reporting have been tried - the obvious
    example is X.400 - and if anything they've worked less well than
    the in-band approach taken in SMTP. So nobody should assume that the
    definition of such a mechanism will magically lead to glory.

(4) There seems to be an assumption that an alternative asynchronous
    notification mechanism is something  that SUBMIT could not do. If so,
    that's incorrect. In fact I can easily imagine a SUBMIT extension that
    would specify an alternative mechanism for reporting delivery
    success/failures, which the server would then employ instead of
    generating one or more DSNs.

    In fact this would be so trivially easy to implement modulo whatever
    the backwards-pointing notification mechanism is I bet I could do it
    in only and hour or two of coding.

(5) Implementing proper error handling is hard, getting everyone to implement
    it correctly is probably impossible.

(6) There are a host of problems associated with the Outbox/Sent magic folder 
    proposal that have nothing to do with anything being discussed here.

In fact I believe I'm going to coin a new term at this point - the FUSMEHP. I
bet you can geuss what it means.

				Ned

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 00:54:36 +0000

Just to clarify about what I was thinking when I raised the spectre of 
notifications.

Quite often the person would like to see more than one.  The key 
attribute about notifications intended for humans is timeliness.  We 
have a system where you may get something that looks like a bounce 
message 4hrs after sending your message.  You may get a bounce message 
10 minutes later (or worse) if you entered the destination address 
incorrectly.  You may have already gone home, and now missed the 
deadline to submit your tender for the project.  There are real 
consequences to the delays that are created by mail systems.

As for DSNs the person generally doesn't want to see them in their inbox 
as messages.

Whether underneath these are multilpe DSNs or whatever I don't think 
many people really care, although we have questions about what should a 
client do to a server-stored message (e.g. as per IMAP) when it's a 
control message like this (should it be processed and then deleted?).

The problem with DSNs also is that they are not protocol.  they are just 
other messages that a MUA has to retrieve and process, based on whatever 
retrieval schedule.  This works against timeliness.

In any case, there will still be a lot of mail that is not delivered by 
SMTP or SUBMIT - and that's mail where the destination is to another 
user of the JMAP server.  This is a considerable amount of mail.  In 
this case also, the notifications could be in the protocol, since the 
time overall is probably very short.  All we need for mail that takes 
longer to change state (e.g. because it's going off-site via 
SMTP/SUBMIT) is a way to send a status message back.  This could be an 
email, or it could be some other kind of message.  The JMAP server 
submitting mail via SMTP/SUBMIT could then choose to request DSNs on 
behalf of the client, and the client will always get some kind of 
notification, whether that is

* Your message was delivered to another local mailbox
* your message was submitted for delivery and there will be further 
notifications
      - incoming DSN from SMTP etc
* your message was submitted for delivery, but there will be no more 
status notifications

This should help provide incentive for MTA vendors to implement DSNs as 
well.

As for other envelope extensions, I agree the SMTP/SUBMIT system needs 
to know certain envelope-related things, such as whether the content is 
binary or 7-bit safe or not.  This is where the discussion about 
shackling JMAP to SUBMIT/SMTP comes in.  It's a shame that we still have 
to support a transport that needs to care about the payload.  Seems like 
a basic layering violation to me.

If we aren't going to do anything about that, like set a minimum level 
of required support for the systems the JMAP will interface to, we won't 
improve this, nor provide any incentive for it to be improved, and so it 
won't be.  We will still be Base64 or quoted-printable encoding 
everything.

also we may find there is more interest in this protocol if we DO look 
at fixing something (anything) about SMTP.

Adrien

------ Original Message ------
From: "Ned Freed" <[email-masked]>
To: "Ted Lemon" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "Ned Freed" <[email-masked]>
Sent: 24/04/2017 11:50:54 AM
Subject: Re: [Jmap] Submission

>>  If that's the case, then I don't think there's anything to argue 
>>about.   But I've been getting the impression that that's not the 
>>case.   I am fully in agreement with the assertion, with which I don't 
>>recall anyone disagreeing, that the envelope has to be separate from 
>>the message and has to be extensible.
>

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Apr 2017 17:42:24 -0700 (PDT)

> I can think of one major platform (Windows) where this approach will be

We used to have a Windows implementation. Curiously, I don't recall it having
any problems with proxy authentication.

> SMTP or SUBMIT depending on the destination.

In answer to your question, we are not talking about any particular
architecture. That's a point I've been trying to make, I guess unsuccessfully,
since I first starting posting in this thread.

> individual user credentials to be presented to the next hop will be

Then I guess I should be thankful I never suggested it do such a thing.

				Ned

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 24 Apr 2017 11:21:55 +1000

> Sorry I missed this one on the Mt Sinai tablets.

There's no need to be nasty.  Chill it please.  We're a professional working group.

> ability to prompt the user to correct the address book entry for that 

Without getting bogged down in exact details just yet, we should address this question:

Do we want JMAP-Mail to include a facility to check whether an address would be accepted for mail delivery by the server - presumably the result is a tristate of [definite YES, Unknown, definite NO] via whatever response method is used.

Cheers,

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 01:32:24 +0000

------ Original Message ------
From: "Ned Freed" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "Ned Freed" <[email-masked]>; "Arnt Gulbrandsen" 
<[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 24/04/2017 12:42:24 PM
Subject: Re: [Jmap] Submission

>any problems with proxy authentication.
Maybe we have different things in mind when we think about proxy 
authentication, or even proxy.

In JMAPs case, I would suggest it's probably not a proxy, not in the way 
an HTTP proxy is.

>since I first starting posting in this thread.
Ok, except when working with existing systems such as challenge response 
auth protocols that take opaque blobs, then the architecture can be 
impinged upon.

>Then I guess I should be thankful I never suggested it do such a thing.

OK, does this mean you intended that the auth was of the JMAP server to 
the SUBMIT server not on behalf of (using creds of) a JMAP user?

In that case, it's a fairly trivial issue.

Adrien

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2017 01:49:57 -0000

In article <[email-masked]> you write:
>Do we want JMAP-Mail to include a facility to check whether an address would be accepted for mail delivery by the server - presumably the result is a
>tristate of [definite YES, Unknown, definite NO] via whatever response method is used.

I'm trying to think of a plausible use case for separate checks rather
than at submission time, and also trying to think of a plausible way
to do useful checks on non-local addresses.

The use cases I think of are pretty contrived, like someone just
uploaded an address book and wants to prune out the bogus addresses.

For the latter, since nobody supports VRFY and EXPN any more (for good
reasons,) and RCPT TO probes tend to get you blacklisted, the answer
for remote addreses is always going to be "maybe" unless you mistype
a nonexistent domain.

So I wouldn't bother.  The addresses will get checked when people send
the message, just like now.

R's,
John

---

From: Bron Gondwana <[email-masked]>
Date: Mon, 24 Apr 2017 12:03:58 +1000

> uploaded an address book and wants to prune out the bogus addresses.

I'm inclined to agree with you here that this is something best addressed as part of Addressbook handling (jmap-addressbook, when we get to that) allowing you to pre-fill known user addresses without typing them by hand!

Obviously the answer for non-local addresses would be "Unknown".  The interesting bit is having the information available to the client about whether an address is local, and hence they can get a clear indication of correctness before trying to send.  There's definitely value in that from a usability perspective, particularly in a large company or university or something where valid address lookup is available.

Bron.

-- 
  Bron Gondwana
  [email-masked]

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 02:07:31 +0000

when I made the suggestion, I was imagining that although you can't do 
VRFY or EXPN or probe emails and yes this is for good reason, the number 
of times I've had bounce messages a long time after submission for 
problems such as NXDOMAIN, means the current system isn't very good at 
even this.

Also if we put a system into JMAP to allow it, it can then be extended.  
If we don't, we are stuck with the current functionality.

I believe there is certainly some value in pre-checking an address.

The problem tends to happen in more cases than you'd think (e.g. not 
just importing / checking an address book).  I've seen many messages 
stuck in retry queues which are just replies to an email from someone 
who made a mistake setting their own email address in their mail client. 
  Presumably these people have a frustrating life wondering why they 
never get replies until they sort it out.  Also if you take someone's 
address down from some other medium (phone) and get it wrong.  People 
often write what they want to hear rather than what is spoken to them.

So you can't even rely on people to get their own email correct, and you 
can't rely on an address being correctly transcribed.  It's on first use 
that you find out whether the address is any good.  Also it's fairly 
common for addresses to get cancelled, even domains expire.

Checking (whatever form of checking is possible) early is better than 
later, as it can provide more options.  If you hit send and bolt for the 
door, even a submission bounce will be too late.

At the moment yes I believe all the checking that's really possible is 
just at the domain level.  that's not to say that's all the checking 
there will ever be, once (if) it's proven to have sufficient value.

Adrien

------ Original Message ------
From: "John Levine" <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 24/04/2017 1:49:57 PM
Subject: Re: [Jmap] Address Validation (was Re: Submission)

>In article 
><[email-masked]> you 
>write:
>>Do we want JMAP-Mail to include a facility to check whether an address 
>>would be accepted for mail delivery by the server - presumably the 
>>result is a

---

From: Ned Freed <[email-masked]>
Date: Sun, 23 Apr 2017 19:09:01 -0700 (PDT)

> the SUBMIT server not on behalf of (using creds of) a JMAP user?

On behalf of, but not using the user's credentials.

That's the functionality proxy authentication provides. Of course there are
many ways to do it, including but not limited to good old SASL PLAIN's authzid
parameter.

				Ned

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 24 Apr 2017 10:44:11 +0100

> a basic layering violation to me.

Specifically on the 8-bit-cleanliness issue:
http://cr.yp.to/smtp/8bitmime.html

-- 
Cheers,
  Jeremy

---

From: "John R Levine" <[email-masked]>
Date: 24 Apr 2017 09:43:23 -0400

> when I made the suggestion, I was imagining that although you can't do VRFY 
> or EXPN or probe emails and yes this is for good reason, the number of times 
> I've had bounce messages a long time after submission for problems such as 
> NXDOMAIN, means the current system isn't very good at even this.

Once again, that's a change to SMTP and in this case a fairly large and 
incompatible one.

On all the mail systems I know, if you try to send mail at a domain that 
doesn't exist, it fails instantly.  The problem is when you send mail to a 
domain that has an A record but no mail server.  There's no way for a 
client system to tell a nonexistent mail server from one that crashed and 
will be back shortly.  Maybe the retry timeout should be less than the 4-5 
days that RFC 5321 suggests, but that's a quality of implementation issue 
that has nothing whatsoever to do with JMAP.

Regards,

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 24 Apr 2017 15:16:13 +0100

John R Levine writes:
> Once again, that's a change to SMTP and in this case a fairly 
> large and incompatible one.

3885/7, isn't it? Would be nice if etc., but twelve years of 
nonimplementation is a strong sign that it isn't going to be implemented. 
IMO JMAP draft editors ought to pretend that 3885-8 never were written and 
nothing of the sort will be deployed.

Arnt

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2017 14:47:48 -0000

In article <[email-masked]> you write:
> There's definitely value in that from a
>usability perspective, particularly in a large company or university or something where valid address lookup is available.

If it's just address validation, I still don't see a lot of value in
doing that separately before you send the message.  On the other hand,
if the address is local there's likely to be an LDAP server that can
populate other parts of the address book than just the address.

R's,
John

---

From: "John Levine" <[email-masked]>
Date: 24 Apr 2017 14:51:16 -0000

>nonimplementation is a strong sign that it isn't going to be implemented. 

That, and the relief we all felt when default configs of sendmail
stopped sending those messages saying I've been trying for N hours and
will keep trying for a long, long, time.

R's,
John

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 24 Apr 2017 16:00:20 +0100

> will keep trying for a long, long, time.

I shudder at the memory ;) An excellent example of good intention ruined by 
annoying execution.

Arnt

---

From: Cyrus Daboo <[email-masked]>
Date: Mon, 24 Apr 2017 11:25:36 -0400

Hi Arnt,

--On April 24, 2017 at 3:16:13 PM +0100 Arnt Gulbrandsen 
<[email-masked]> wrote:

> IMO JMAP draft editors ought to pretend that 3885-8 never were written
> and nothing of the sort will be deployed.

At least one person has commented on this list that they have implemented 
the extension. Just because an extension is not implemented by every SMTP 
server, it doesn't mean that the extension is not useful and being used in 
some smaller market. Clearly we don't need to build message tracking into 
JMAP from the outset, but equally we should not design JMAP in a way that 
would prevent it from easily being added as an extension for those that 
want it.

-- 
Cyrus Daboo

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 24 Apr 2017 17:14:01 +0100

Cyrus Daboo writes:
> At least one person has commented on this list that they have 
> implemented the extension.

Yes.

> Just because an extension is not 
> implemented by every SMTP server, it doesn't mean that the 
> extension is not useful and being used in some smaller market. 
> Clearly we don't need to build message tracking into JMAP from 
> the outset, but equally we should not design JMAP in a way that 
> would prevent it from easily being added as an extension for 
> those that want it.

I agree that 3885 (et al) have some utility even if only a few people 
implement them. However, the question that I understood was asked upthread 
is roughly "to which recipients has x been delivered already?", and for 
that particular question an unbroken implementation chain is needed from 
sender to recipient.

Arnt

---

From: Ted Lemon <[email-masked]>
Date: Mon, 24 Apr 2017 15:25:40 -0400

On Apr 24, 2017, at 12:14 PM, Arnt Gulbrandsen <[email-masked]> wrote:
> However, the question that I understood was asked upthread is roughly "to which recipients has x been delivered already?", and for that particular question an unbroken implementation chain is needed from sender to recipient.

I think even knowing that the MTA for the admin domain of the JMAP server has successfully forwarded the message would be sufficient to be equivalent in most cases.   If someone does more, that's great, but best is the enemy of good enough.

---

From: "Adrien de Croy" <[email-masked]>
Date: Mon, 24 Apr 2017 21:37:01 +0000

------ Original Message ------
From: "John R Levine" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 25/04/2017 1:43:23 AM
Subject: Re: [Jmap] Address Validation (was Re: Submission)

>>VRFY or EXPN or probe emails and yes this is for good reason, the 
>>number of times I've had bounce messages a long time after submission 
>>for problems such as NXDOMAIN, means the current system isn't very 
>>good at even this.
>
actually I didn't envisage this one going anywhere near SMTP in the 
first instance.  The JMAP server can easily do domain-part validation, 
or any other validation available to it (address book lookup, security 
policy whatever) and deny the recipient validation.

>
>
>On all the mail systems I know, if you try to send mail at a domain 
>that doesn't exist, it fails instantly.
I've seen a lot delayed, especially when the initial submission server 
forwards to another pre-configured "smarthost" which then delays it 
further.

Adrien

>The problem is when you send mail to a domain that has an A record but 
>no mail server.  There's no way for a client system to tell a 
>nonexistent mail server from one that crashed and will be back shortly. 
>  Maybe the retry timeout should be less than the 4-5 days that RFC 5321 
>suggests, but that's a quality of implementation issue that has nothing 
>whatsoever to do with JMAP.
>
>Regards,

---

From: Bron Gondwana <[email-masked]>
Date: Tue, 25 Apr 2017 15:37:23 +1000

On Tue, 25 Apr 2017, at 05:25, Ted Lemon wrote:
> On Apr 24, 2017, at 12:14 PM, Arnt Gulbrandsen
> <[email-masked]> wrote:>> However, the question that I understood was asked upthread is roughly
>> "to which recipients has x been delivered already?", and for that
>> particular question an unbroken implementation chain is needed from
>> sender to recipient.> 
> I think even knowing that the MTA for the admin domain of the JMAP
> server has successfully forwarded the message would be sufficient to
> be equivalent in most cases.   If someone does more, that's great, but
> best is the enemy of good enough.
Our eventual plan for FastMail is to annotate every sent message with
the delivery log lines that were generated for each recipient at the
edge of our infrastructure as the messages get sent.  Self service
facilities are nice to have.
Bron.

--
  Bron Gondwana
  [email-masked]

---

From: Mads Hjorth <[email-masked]>
Date: Tue, 25 Apr 2017 07:36:21 +0000

I agree in the ambition to make a better mail experience :-) and in the principle of keeping protocols open and modular. 

Some of the main issues with the current mail experience is its lack of evidence of senders ID and delivery. 

A lot of organizations are looking into this. 

European Telecommunications Standards Institute has published a standard on Registered Electronic Mail (ETSI TS 102 640-1). 

Maybe this framework could be used to clarify the future role of JMAP e.g. does it cover the requirement of both a Sender-Message Submission Interface and Sender-Message Store Retrieval Interface. 

So… maybe prepare JMAP to help organizations go from email to registered email?

/madsh

> Den 21. apr. 2017 kl. 08.16 skrev Adrien de Croy <[email-masked]>:
> 
> 
> 
> sure, I think there are fundamental issues with trying to go further than an A/MX for now, but if we built in a mechanism to verify addresses, it could be extended later, rather than being blocked off for all of eternity.
> 
> Even A/MX would have some value.  Who knows what kind of trusted mail delivery infrastructure may come out in the future, at some stage we have to fix SMTP.  Forever is a long time.
> 
> Adrien
> 
> 
>> 
>>

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 09:26:46 +0100

> forwards to another pre-configured "smarthost" which then delays it 

I've seen delays as well. Often related to badly behaved DNS servers 
which do strange things with non-existent domains (eg not returning 
anything, rather than returning an NX error). The mail server then 
should play safe and retry for a while.

Remember that a lot of mail servers are deployed on small networks with 
badly behaving infrastructure. Eg - how is a mail server supposed to 
indicate whether a recipient address is valid if the Internet connection 
is down and the DNS server has crashed? I know it seems unpopular here, 
but there are good reasons for a lot (not all) of the ways things happen 
with SMTP/IMAP4.

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 09:33:24 +0100

On 24/04/2017 20:25, Ted Lemon wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
>> However, the question that I understood was asked upthread is roughly 

I may be wrong, but I believe you can already do that with DSN/MDNs. 
However, I have yet to see an MUA which does it even though it doesn't 
seem like it'd be that hard to do with the MUA requesting success DSNs 
and parsing the responses.

Maybe the JMAP server could handle the responses and somehow link the 
status to the sent message?

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 25 Apr 2017 09:26:20 +0000

I think the problem with lack of support for MDN / DSN in clients may be 
due to lack of implementation in the network of support for it.

If there's still a significant number of deployed systems that don't 
support it, then clients have increased doubts about its worth.  Chicken 
and egg issue again.

I wouldn't presume that the reason that it doesn't appear so much in the 
wild in clients is due to lack of interest.  Maybe some MUA authors 
could be approached for comment.

We looked several times at supporting DSN etc in our mail system, and 
balked at some of the more onerous requirements.

A lot of the extension RFCs have onerous requirements for interfacing 
with non-supporting systems.  8BITMIME springs to mind here.  So we are 
binary safe (even embedded NULLs), but don't advertise 8BITMIME due to 
not wanting to take on the responsibility of what to do when forwarding 
to another server that doesn't advertise it.  Maybe that just makes us 
lame.  Probably should revisit both these.

But we probably aren't the only ones either.

Adrien

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 10:50:43 +0100

On 25/04/2017 10:26, Adrien de Croy wrote:
>
> I think the problem with lack of support for MDN / DSN in clients may 
> be due to lack of implementation in the network of support for it.
>
> support it, then clients have increased doubts about its worth. 
>  Chicken and egg issue again.
>
> I wouldn't presume that the reason that it doesn't appear so much in 
> the wild in clients is due to lack of interest.  Maybe some MUA 
> authors could be approached for comment.
>

DSN wasn't that hard when we implemented it (many years ago so I've 
forgotten details). It wasn't trivial either (we had to change the way 
envelope data was stored), but there are some things which have been far 
more onerous (it's a 'separate' thing, so potential breaking effects in 
other code are limited - unlike full 8 bit support which affects pretty 
much everything).

But - if DSN is hard to implement, then any 'delivery status' type thing 
could be similarly hard so may affect JMAP implementation if delivery 
status is a mandatory part of that standard.

> with non-supporting systems.  8BITMIME springs to mind here.  So we 
> are binary safe (even embedded NULLs), but don't advertise 8BITMIME 
> due to not wanting to take on the responsibility of what to do when 
> forwarding to another server that doesn't advertise it.  Maybe that 
> just makes us lame.  Probably should revisit both these.

We support 8 bit, but not embedded NULLs. Changing that would be a big 
thing (probably not to implement in the message store, but to just to 
test (and write new tests for) EVERYTHING that looks at the data). We've 
thought about it, but since we probably wouldn't advertise 8BITMIME 
anyway, for the reasons you say, it's easier just not to potentially 
break things by supporting NULL characters ;-)

To me, having to rewrite messages is a really bad idea (DKIM etc), so 
8BITMIME just isn't worth doing now. If we were designing SMTP nowadays, 
obviously full 8 bit support would be mandatory, but we're not.

---

From: "John R Levine" <[email-masked]>
Date: 25 Apr 2017 07:51:29 -0400

> So… maybe prepare JMAP to help organizations go from email to registered email?

In the US there are patents on registered electronic mail and at least one 
patent holder with a history of aggressive litigation.  I would suggest 
staying away from it.

Regards,

---

From: Ted Lemon <[email-masked]>
Date: Tue, 25 Apr 2017 09:41:08 -0400

On Apr 25, 2017, at 4:33 AM, Paul Smith <[email-masked]> wrote:
> I may be wrong, but I believe you can already do that with DSN/MDNs. However, I have yet to see an MUA which does it even though it doesn't seem like it'd be that hard to do with the MUA requesting success DSNs and parsing the responses.

One of the stated goals of the JMAP work is to remove one of the major pain points in writing MUAs: IMAP (specifically the lack of consistent implementation, which causes interop issues).   So perhaps we will see more MUA innovation.

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 25 Apr 2017 09:53:33 -0400

Hi Adrien,

--On April 25, 2017 at 9:26:20 AM +0000 Adrien de Croy <[email-masked]> 
wrote:

> wild in clients is due to lack of interest.  Maybe some MUA authors could
> be approached for comment.

Yup, I implemented DSN and MDM in Mulberry - but as you say, due to the 
fact that there is no guarantee that status will be returned, it is not 
very useful in the general case. I can't remember the last time I used DSN 
- but I'll try it out on this message just to see what happens!

However, there some environments where it may be useful - e.g. an 
enterprise which has a required policy to enforce MDM read receipt or DSN 
success (something that could be controlled via device management 
restrictions). I doubt it is something that would ever be useful for a 
non-enterprise system.

-- 
Cyrus Daboo

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 15:01:31 +0100

On 25/04/2017 14:53, Cyrus Daboo wrote:
>
> Yup, I implemented DSN and MDM in Mulberry - but as you say, due to 
> the fact that there is no guarantee that status will be returned, it 
> is not very useful in the general case. I can't remember the last time 
> I used DSN - but I'll try it out on this message just to see what 
> happens!
>
> enterprise which has a required policy to enforce MDM read receipt or 
> DSN success (something that could be controlled via device management 

The original statement was something like "it would be useful to know 
that the submission server has sent the message out". It seems to me 
that DSN would meet that requirement, especially if the MUA or JMAP 
server parsed the MDN and flagged the message appropriately.

Of course, if the server doesn't support DSN, then it won't help - but 
you'd need some support from the server to handle delivery notifications 
of any type, so why not DSN?

(BTW, did you receive the "delivered" MDN from our server from the DSN 
request? It looks like it left here OK)

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 25 Apr 2017 10:12:30 -0400

Hi Paul,

> request? It looks like it left here OK)

Your server gave me a "delivered" status. For Adrien and ietf.org I only 
got a "relayed" status from my own server. I think the later is not very 
useful as I expect my own server to be reliable, and if not expect to know 
about problems pretty quickly. Note, I only used DSN on the last message - 
I'll try an MDM on this one.

-- 
Cyrus Daboo

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 25 Apr 2017 14:37:50 +0000

I'd expect a lot of servers don't honour SUCCESS DSNs for a number of 
reasons (privacy, spam avalanche etc).

Cyrus: Relayed is what I'd expect when forwarding to a server that 
doesn't support DSN (us), so that's expected.

Adrien

------ Original Message ------
From: "Paul Smith" <[email-masked]>
To: "Cyrus Daboo" <[email-masked]>; "Adrien de Croy" 
<[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 26/04/2017 2:01:31 AM
Subject: Re: [Jmap] Address Validation (was Re: Submission)

>you'd need some support from the server to handle delivery 
>notifications of any type, so why not DSN?
>

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 15:57:10 +0100

> reasons (privacy, spam avalanche etc).

I'm not sure of that - is it any worse than accepting a message with a 
2xx response? 'Delivered' means it has been dumped in a user's mailbox, 
not that it has been read or anything so it doesn't give any more 
information other than the final SMTP server accepting the message.

If it does cause problems, those would probably happen for any scalable 
delivery notification system dreamt up for JMAP as well.

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 15:46:34 +0100

On 25/04/2017 15:12, Cyrus Daboo wrote:
> Your server gave me a "delivered" status. 

Good :-) Does Mulberry indicate this in the 'sent items' folder somehow? 
I think that's what users would like (?)

(I haven't used Mulberry for a while - IIRC, it disappeared for a while 
some time ago, and I hadn't realised it had come back...)

> I think the later is not very useful as I expect my own server to be 
> reliable, and if not expect to know about problems pretty quickly.

Agreed, but that's what the earlier person was saying would be good. For 
the average user it tells them that THEIR mail server has sent the 
message out, which is the best you can expect if the onward server 
doesn't support any form of notification.

If the 'sent items' folder indicates somehow when messages have been 
sent out by the MSA (or delivered to the recipient if possible) then I 
can see that would help usability.

---

From: "Adrien de Croy" <[email-masked]>
Date: Tue, 25 Apr 2017 16:06:27 +0000

------ Original Message ------
From: "Paul Smith" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>; "Cyrus Daboo" 
<[email-masked]>; "[email-masked]" <[email-masked]>
Sent: 26/04/2017 2:57:10 AM
Subject: Re: [Jmap] Address Validation (was Re: Submission)

>2xx response?

it sure is, especially if the return path is invalid / spoofed / victim.

It's a whole new email, rather than a protocol response code.

>'Delivered' means it has been dumped in a user's mailbox, not that it 
>has been read or anything so it doesn't give any more information other 
>than the final SMTP server accepting the message.
>

If it uses emails for messages yes.  Which it probably would.

Adrien

---

From: Paul Smith <[email-masked]>
Date: Tue, 25 Apr 2017 17:18:07 +0100

On 25/04/2017 17:06, Adrien de Croy wrote:
>>> I'd expect a lot of servers don't honour SUCCESS DSNs for a number 
>>> of reasons (privacy, spam avalanche etc).
>>
>> I'm not sure of that - is it any worse than accepting a message with 
>> a 2xx response?
>

Well, you'd get the same effect with RELAYED/FAILED DSNs... I'm not sure 
the fact that it's a SUCCESS DSN would make any difference as far as 
that goes.

(BATV FTW?)

> If it uses emails for messages yes.  Which it probably would. 

It'd pretty much have to use messages if it was to be scalable.

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 25 Apr 2017 18:41:04 +0100

> On 25 Apr 2017, at 14:41, Ted Lemon <[email-masked]> wrote:
> 

I've done this in a MUA that talks IMAP. While further integration in JMAP would be nice, I don't think this is either hard or that it is the first order of priorities for this group.

---

From: Brandon Long <[email-masked]>
Date: Wed, 26 Apr 2017 15:08:40 -0700

This is the second time you've asked for the SMTP status codes and enhanced
status codes, I'm very unclear what the utility of those would be, and why
we would need
to expose them to JMAP.

We very rarely care about the difference between the codes beyond
ok/temp/perm, and find that folks are fairly inconsistent about the
enhanced codes, or that the enhanced codes don't actually translate into
useful information from a user perspective.  At best they provide a
fallback for what is otherwise a long list of regexes for the error
messages themselves.

It also ties the implementation to SMTP when SMTP may not even be used, ie
local delivery on a service using JMAP may never use SMTP.  It also ties it
to SMTP when we know that we can't implement it with SMTP (due to address
scraping concerns).

One could make a better case for tying it to LDAP, for example.

Brandon

On Fri, Apr 21, 2017 at 6:02 PM, Chris Newman <[email-masked]>
wrote:

>> sure, I think there are fundamental issues with trying to go further than
>> an A/MX for now, but if we built in a mechanism to verify addresses, it
>> could be extended later, rather than being blocked off for all of eternity.
>>
>> delivery infrastructure may come out in the future, at some stage we have
>> to fix SMTP.  Forever is a long time.
>>
>
> The present model for MUA mail address validation is to authenticate to an
> IETF standard submission server, issue a MAIL FROM + RCPT TO + RSET.
> performing this sequence of commands on the Submission server and returns a
> 3-digit SMTP status code, optionally SMTP enhanced status codes (see RFC
> 3463 section 3.2) and text string.
>
> in-scope. Doing address validation this way is already quite powerful. For
> example, if the submission server chooses to implement the SMTP 251/551
> response codes for local users, the mail client gains the ability to prompt
> the user to correct the address book entry for that recipient.
>
> quality-of-implementation issue (and there are, of course, security/privacy
> issues). Future work would be to define additional SMTP/Submit enhanced
> status codes for additional address validation, but that's presently
> out-of-scope for the JMAP charter.
>
> validation mechanism to JMAP in a way that preserves all the semantics of
> the existing mechanism.
>
>                 - Chris
>

---

From: "Chris Newman" <[email-masked]>
Date: Wed, 26 Apr 2017 18:48:18 -0700

On 26 Apr 2017, at 15:08, Brandon Long wrote:
> we would need to expose them to JMAP.

Good protocol design requires both machine readable and human readable 
status information, and IETF experience suggests that status codes need 
to be extensible in a way that is lighter-weight than publishing an RFC 
(a first-come-first-served or expert review IANA registry seems 
reasonable to me). As JMAP submission will need status codes for the 
different things that can go right or wrong at submission time, we can 
either use the already defined status codes and registry, or we can 
create a new set of status codes. If we create a JMAP submission status 
codes registry, the registry should include a column listing SMTP 
Submission status codes that map to the JMAP code. Using the existing 
codes is less work in the spec but I'm fine with either approach.

> ok/temp/perm,

I won't argue this point. But I'll observe that rarely is not the same 
as never.

> and find that folks are fairly inconsistent about the

That argument may support creation of a new JMAP submission status codes 
registry.

> local delivery on a service using JMAP may never use SMTP.

I don't see how status codes ties JMAP to SMTP. On the flip side, the 
JMAP charter already requires us to tie JMAP to Submission (RFC 6409). 
That's a good thing since all non-local messages will travel over SMTP 
for a very long time.

> It also ties it

Submission (RFC 6409, port 587/465) is not the same as SMTP (RFC 5321 
port 25). I'm not aware of any address scraping concerns in 
authenticated submission that wouldn't also apply to JMAP. Either the 
deployment allows authenticated users to validate addresses or it 
doesn't -- that's a policy choice of the deployment and nothing we say 
in any spec will change that.

> One could make a better case for tying it to LDAP, for example.

For curiosity's sake, I'd like to hear you make that argument but I 
doubt it's relevant to this list; maybe a fun bar BOF topic at an IETF 
;-). I don't think JMAP should be tied to LDAP but I wrote the pooling 
LDAP client library from scratch that is deeply integrated in our server 
so I'm quite familiar with that protocol.

		- Chris

>>> than
>> returns a
>> security/privacy
>> validation
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_jmap&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=QBZgPENFbjFadxqU4HJ3ZDpRz3X1JlDY-keqMt52FFo&m=1vWqS7msmD233b9CaI98kh4rSO2gJ-Vly0EVUxduLcE&s=kkW8-WUbouatu9xo78ad7vt4KcIqOQ5maMWA4RMew_M&e=
>>