Subject: [Tools-discuss] double new lines in drafts sent via e-mail

From: "Alex Rousskov" <[email-masked]>
Date: Mon, 25 Oct 2004 09:24:56 -0600

I am not sure why extra line breaks appear in the case quoted below. Using  
Pine, I have never had that problem with attachments. Perhaps others know  
what is going on here. This is something we may want to check for the  
email interface of the ID submission tool...

Alex.

------- Forwarded message -------
From: "Pekka Savola" <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Subject: RE: [newtrk] Re: I-D ACTION:draft-savola-ipr-lastcall-04.txt (fwd)
Date: Mon, 25 Oct 2004 12:44:14 +0300 (EEST)

On Mon, 25 Oct 2004 [email-masked] wrote:
> 1) Fix the double blank line problem.

This is actually caused by by the IETF secretariat, I think, due to
the way they process email attachments.  I send them a draft with
single line breaks, but the output always has two.

I guess I should try to send them a link rather than an attachment..

...

_______________________________________________
Tools-discuss mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tools-discuss

---

From: stanislav shalunov <[email-masked]>
Date: 25 Oct 2004 16:00:39 -0400

"Alex Rousskov" <[email-masked]> writes:

> I am not sure why extra line breaks appear in the case quoted
> below. Using  Pine, I have never had that problem with
> attachments. Perhaps others know  what is going on here. This is
> something we may want to check for the  email interface of the ID
> submission tool...

Messages from [email-masked] seem to be generated with
Microsoft Exchange V6.0.6603.0.  If they are stored in DOS format
locally, the DOS newlines are probably encoded by the client *if* the
draft is sent in base64 rather than quoted-printable (or with no
protection).

At this point, draft-savola-ipr-lastcall-04.txt is a deletion notice,
so it's difficult to see what was going on.  I don't have a message
with an attachment from John Loughney, either.

I think making the submission tool convert any newlines (\n, \r\n, \r,
and \n\r) into \n would not be a bad idea.  This applies equally to
web and email submissions.  (In fact, it might even be more important
for web submissions, as unprotected email takes care of canonicalizing
new lines -- except when sent through base64.)

Conceptually,

#include <stdio.h>

enum state_enum {NORM, N, R, RN};
typedef enum state_enum state_t;

#define NORMAL	do { 				\
			state = NORM;		\
			putchar('\n');		\
		} while (0)

int
main()
{
	int c;			/* Current character. */
	state_t state;

	state = NORM;
	while ((c = getchar()) != EOF)
		switch (state) {
		case NORM:
			if (c == '\n') state = N;
			else if (c == '\r') state = R;
			else putchar(c);
			break;
		case N:
			NORMAL;
			if (c != '\r') putchar(c);
			break;
		case R:
			NORMAL;
			if (c != '\n') putchar(c);
			break;
		}
	if (state != NORM) NORMAL;
	exit(0);
}

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

Just my 0.086g of Ag.

---

From: Alex Rousskov <[email-masked]>
Date: Mon, 25 Oct 2004 20:21:25 -0600 (MDT)

On Mon, 25 Oct 2004, stanislav shalunov wrote:

> I think making the submission tool convert any newlines (\n, \r\n, 
> \r, and \n\r) into \n would not be a bad idea.  This applies equally 
> to web and email submissions.  (In fact, it might even be more 
> important for web submissions, as unprotected email takes care of 
> canonicalizing new lines -- except when sent through base64.)

IMO, "fixes" and "corrections" of that kind are the wrong approach in 
general. The Toolset should show how the submitted draft looks like 
and should warn if it finds strange end-of-line characters. If it 
looks bad or there are warnings, the submitter should be responsible 
for applying the right encoding, transformations, etc. to get the 
results they want.

I cannot prove or even quantify it, of course, but I believe that 
complain-but-dont-fix-it approach results in a more compact, less 
buggy, and cheaper tools compared to a lets-try-to-fix-what-we-can 
approach.

There are always exceptions to general rules, of course. However, I 
think we tend to bloat the Toolset with features that may look like a 
10-line hack now, but have a tendency to grow beyond belief later.

Apologies for the rant.

Alex.

---

From: Pekka Savola <[email-masked]>
Date: Tue, 26 Oct 2004 07:29:27 +0300 (EEST)

This is probably getting a bit off-list, but I think a clarification 
is deserved..

> with an attachment from John Loughney, either.

The quoting was probably a bit off; the problem was with my 
attachments.  I'm sending them off from a Linux system, using pine, as 
Alex is using.  The versions which I put on my local web page are just 
fine, but the result that the secretariat posts is mangled.  So 
something is happening between when I attach the draft and when the 
secretariat posts them.

Pine sends the attachments in base64.  I've looked at the actual 
attachment sent, and undecoded it manually, and it has ^M sequence at 
the end of lines.  The originals are available at 
http://www.netcore.fi/pekkas/ietf/.  I've also temporarily zipped up 
one of them to preserve the control characters.

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Larry Masinter <[email-masked]>
Date: Mon, 25 Oct 2004 21:54:22 -0700

> general. 

If you're designing systems to interoperate with other systems,
then "be tolerant in what you accept" can lead to encouraging
future interoperability problems.

However, if you're designing a system to be usable by people -- and
mainly people who use the system infrequently, at most once every few
months -- then building the system so that it tolerates common user
errors is the only reasonable way to go.

It's very common that systems use different EOL conventions, and that
files sent as attachments rather than mail bodies aren't canonicalized.

So I disagree with your rant. Tools should be designed
so that the typical IETF user isn't confronted with meaningless and
useless error messages in the typical cases we are expected to encounter.

Larry

---

From: Rob Austein <[email-masked]>
Date: Tue, 26 Oct 2004 01:10:18 -0400

as far as i can tell from the mime specs, the canonical form of any
text/whatever body part -should- have crlf line termination, but the
specs hint that support for this has historically been somewhat lame,
particularly with base64 transfer encoding.  so whatever happened to
the draft in question was probably the result of wandering through a
twisty little maze of bugs that usually cancel each other out but
which, for some reason, failed to do so in this particular case.

i would be pleasantly surprised if a web-based submission tool did not
run into similar issues, so while i sympathize with alex's comment
about ten line hacks turning into maintenance nightmares, it is
probably not acceptable to create a submission tool that can't handle
whatever shows up when one submits what passes for ascii text on one
of the major operating systems using the commonly used web browsers on
those same operating systems.  so i suspect that something along the
lines of what stanislav suggested will be necessary.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Oct 2004 09:16:01 +0200

Pekka Savola wrote:
> http://www.netcore.fi/pekkas/ietf/.  I've also temporarily zipped up one 
> of them to preserve the control characters.

I've seen the same problem with drafts sent from Thunderbird on W2K. The 
problem doesn't seem to be reproducable, though,

Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Simon Leinen <[email-masked]>
Date: Tue, 26 Oct 2004 16:21:26 +0200

Pekka Savola writes:
> attachment sent, and undecoded it manually, and it has ^M sequence
> at the end of lines.

Hm, do you mean it had CR CR LF as the line separator?

Or did it use CR LF as the line separator, and your local system uses
LF as the line separator?

In the first case the encoding was incorrect, in the second case you
should have converted from the "canonical" to your local line-break
encoding.  See RFC 2046 section 4.1.1.

> The originals are available at http://www.netcore.fi/pekkas/ietf/.
> I've also temporarily zipped up one of them to preserve the control
> characters.

Yes, those files use the Unix convention (LF as line break).
Personally I prefer that one too, but such files will look funny on
machines with other line-break representations.  I don't know how HTTP
handles this issue, but mail (MIME) and FTP (text mode) fortunately
have well-defined line-preserving ways to transfer text :-)
-- 
Simon.

---

From: "Alex Rousskov" <[email-masked]>
Date: Tue, 26 Oct 2004 09:51:53 -0600

On Mon, 2004/10/25 (MDT), <[email-masked]> wrote:

> Tools should be designed

On Mon, 2004/10/25 (MDT), <[email-masked]> wrote:

> while i sympathize with alex's comment
> those same operating systems.

Just to clarify: I was ranting about the general approach/trend that I am  
worried about, not a specific ten-line hack (or it would not be a rant).  
In other words, we should auto-fix only when there is a single correct  
fix; and we should evaluate a feature based on its overall importance and  
usefulness rather than just on the number of code lines it takes to hack  
its alpha version in. Not everything that takes 10 lines of Perl should be  
accepted and not everything that takes 10K LOC should be rejected. Hack  
length is a [minor] factor, but there are other factors.

When I saw two ten-line hack examples (in a couple of days) possibly used  
as arguments for feature inclusion, I got worried and posted my rant.

Alex.

---

From: stanislav shalunov <[email-masked]>
Date: 26 Oct 2004 12:48:44 -0400

Alex Rousskov <[email-masked]> writes:

> IMO, "fixes" and "corrections" of that kind are the wrong approach
> in general.

I subscribe to this sentiment in general, but don't think that
applying it in this case (end of lines) is reasonable, given that
different local end of line conventions are common and users would
have considerable and unnecessary difficulty with the tool if it
simply gave errors (pages too long, presumably) for DOS/network
conventions.

We already have a real-life example of a MUA that converts text files
to non-'\n' ends of line, even though the MUA runs on Linux.

The situation is analogous to SMTP MTAs: in general, SMTP is not
supposed to fix up messages, reformat paragraphs, etc.  In particular,
however, MTAs treat ends of line differently from other characters,
converting them to network standard before putting on the wire and to
a local convention before delivery.

This message is designed to be viewed at 600dpi.

---

From: stanislav shalunov <[email-masked]>
Date: 26 Oct 2004 13:08:23 -0400

> LF as the line separator?

I'd assume it's this rather than CR CR LF.

> encoding.  See RFC 2046 section 4.1.1.

By ``you'' you mean the Secretariat here, right?  Because apparently
the conversion to network ends of line was done by Pekka's Pine
correctly.

Seems like RFC 2046 section 4.1.1 tells us to cause the draft
submission tool to do the conversion, if it's not already done by the
underlying MIME parsing routine.

The routine I posted is idempotent, so applying it to a text already
treated for line breaks would do no harm.

> Yes, those files use the Unix convention (LF as line break).

The files in the repositories (both RFCs and I-Ds) now follow the LF
convention.  We should, of course, follow it.

> machines with other line-break representations.

Well, they view the I-Ds and RFCs now, somehow.

This message is designed to be viewed at an angle of 45 degrees.

---

From: stanislav shalunov <[email-masked]>
Date: 26 Oct 2004 13:13:11 -0400

Pekka Savola <[email-masked]> writes:

> attachments.

Ah, OK.

> I'm sending them off from a Linux system, using pine, as Alex is
> using.  The versions which I put on my local web page are just fine,

I just checked and they seem to have LF line breaks.

> but the result that the secretariat posts is mangled.  So something
> is happening between when I attach the draft and when the
> secretariat posts them.
> 
> the end of lines.

So, your MUA (quite correctly, according to RFC 2046 section 4.1.1)
encoded line breaks in network format, and that caused the Secretariat
to post them with mangled line breaks.  Others say it happened before
and isn't reproducible.  I think it behooves us to make sure that the
draft submission tool does not suffer from this problem.  It should
also continue to use LF for line breaks locally, as is currently done.

This message is designed to be viewed in boustrophedon.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Oct 2004 19:44:57 +0200

> also continue to use LF for line breaks locally, as is currently done.

Another thing I've seen is that TXT versions of drafts were published 
concatenated with their XML version. -- so the submitter probably sent 
two attachments, and for some reasons the receiving mail program didn't 
detect those properly. Maybe it's time to ask the Secretariat about what 
software they actually use?

Best regards, Julian

---

From: stanislav shalunov <[email-masked]>
Date: 26 Oct 2004 13:54:24 -0400

Julian Reschke <[email-masked]> writes:

> Another thing I've seen is that TXT versions of drafts were
> published concatenated with their XML version. -- so the submitter
> probably sent two attachments, and for some reasons the receiving
> mail program didn't detect those properly.

Interesting.  That's a failure mode I would not expect in a typical
MIME parser.  If a MIME parser does that, it very clearly has a bug
and needs to be fixed, so no need to put that into the draft
submission tool requirements document.

> Maybe it's time to ask the Secretariat about what software they
> actually use?

I'd say that's largely immaterial for the purposes of the present
discussion.  The draft submission tool should not end up in the same
pitfalls as the current method.  For this reason, knowledge of current
pitfalls is important.  The exact way of falling into one is not so
important.

This message is designed to be viewed with 0.06479891g of NaCl.

---

From: stanislav shalunov <[email-masked]>
Date: 26 Oct 2004 14:23:50 -0400

"Alex Rousskov" <[email-masked]> writes:

> [...] we should auto-fix only when there is a single correct fix [...]

I think that the situation with line breaks is just such.  In fact,
the MIME specs seem to require similar treatment in the MIME parser
itself.  Apparently, there exist MIME parsers that don't do it.

> When I saw two ten-line hack examples (in a couple of days) possibly
> used as arguments for feature inclusion, I got worried and posted my rant.

It just seemed easiest to me to express what I wanted to say about
line breaks in a formal language such as C.  The fact that the program
to canonicalize line breaks is small (or large, depending on
perspective) is not important.

Had I wanted a hack, I'd've posted the functionally equivalent Perl
golfer's code:

#!/usr/bin/perl -p0777
s-\n\r|\r\n|\r-\n-g

Antibiotics comprise 2% of cow feed in U.S. feedlots, by weight.

---

From: "Alex Rousskov" <[email-masked]>
Date: Wed, 27 Oct 2004 10:12:10 -0600

> pitfalls is important.  The exact way of falling into one is not so

Agreed.

Alex.