Subject: Re: [openpgp] A way to securely define cleartext signature charset

From: Neil Hunsperger <[email-masked]>
Date: Mon, 10 Sep 2018 18:23:35 +0000

> today I struggled for several hours with "charset guessing" code, that handles
> cleartext signatures in outlook and I thought that maybe this situation could
> be improved a bit in the future?

I'll add a data point. Some years back, the PGP Desktop product added an unsigned "Charset" header to its ASCII armor. The result looked like this:

-----BEGIN PGP SIGNATURE-----
Version: PGP SDK 4.2.1
Charset: iso-8859-1

It solved a real-world problem of intermediate software re-writing character sets using lossless conversions. It didn't solve the security issue in your link to DKG's post. In practice it also didn't avoid 2-pass signature verification.

Cheers,
-Neil

---

From: Jon Callas <[email-masked]>
Date: Mon, 10 Sep 2018 16:53:03 -0700

> On Sep 10, 2018, at 11:23 AM, Neil Hunsperger <Neil_Hunsperger=[email-masked]> wrote:
> 

And for what it’s worth, section 6.2 of RFC 4880 says:

     - "Charset", a description of the character set that the plaintext
       is in.  Please note that OpenPGP defines text to be in UTF-8.  An
       implementation will get best results by translating into and out
       of UTF-8.  However, there are many instances where this is easier
       said than done.  Also, there are communities of users who have no
       need for UTF-8 because they are all happy with a character set
       like ISO Latin-5 or a Japanese character set.  In such instances,
       an implementation MAY override the UTF-8 default by using this
       header key.  An implementation MAY implement this key and any
       translations it cares to; an implementation MAY ignore it and
       assume all text is UTF-8.

All those MAYs are there because of the real world considerations. People still use JIS all over the place, for example, and this allows them to mark their text and have it work correctly. (That’s why we put it in both the standard and software. The examples of Latin-5 and JIS were real.) On the other hand, there was a completely reasonable objection that there are not only silly character sets that one could make up (nods to the computer language “Whitespace”), and real-world issues of what happens when the diehard Latin-5 people start sending messages to the diehard JIS people, and the resulting N^2 testing matrix.

Thus, this section lets an implementation throw its hands up in the air and scream wherever and whenever it wants, while giving a decent way to clearsign Japanese text.

	Jon

---

From: Andre Heinecke <[email-masked]>
Date: Tue, 11 Sep 2018 10:36:32 +0200

Hi Jon, Neil,

thanks for your comments!

On Monday, September 10, 2018 4:53:03 PM CEST Jon Callas wrote:
> > On Sep 10, 2018, at 11:23 AM, Neil Hunsperger 
<Neil_Hunsperger=[email-masked]> wrote:
> > I'll add a data point. Some years back, the PGP Desktop product added an 
unsigned "Charset" header to its ASCII armor. The result looked like this:

That would also be an option. I don't prefer it because it would be unsigned 
but it would already help with usability issues.

>       assume all text is UTF-8.

There is indeed very little definition in this section,...

>  However, there are many instances where this is easier
>  said than done. 

And that is the problem. E.g. a webmailer in which you paste UTF-8 Text, then 
the webmailer sees that it can encode that message as latin 1 and sends it as 
latin 1. Now on the receiving side you have a content-type saying "latin 1" 
but the message was actually signed in UTF-8. And so you have to "try" 
multiple charsets if you whish to verify the message (as it could also be 
signed as latin1).
 
> All those MAYs are there because of the real world considerations. 

Well, my proposed change would be optional. Without the "Charset" in the 
message an implementation would fallback to the "do what you want" guessing 
game with all these MAYs :-)

> People still use JIS all over the place, for example, and this allows them
> to mark their text and have it work correctly. (That’s why we put it in both
> the standard and software. The examples of Latin-5 and JIS were real.) On 
> the other hand, there was a completely reasonable objection that there are
> not only silly character sets that one could make up (nods to the computer
> language “Whitespace”), and real-world issues of what happens when the
> diehard Latin-5 people start sending messages to the diehard JIS people, and
> the resulting N^2 testing matrix.

I'm not sure if you say that we should not add standardized way to define the 
charset for cleartext signatures or that we should?

I don't really see the problem of either silly character sets or Latin-5 / JIS 
messages. As long as It can be converted to the display charset / for passage 
through the openpgp engine it should be ok.

> Thus, this section lets an implementation throw its hands up in the air and
> scream wherever and whenever it wants, while giving a decent way to
> clearsign Japanese text.

Yeah, but from a usability standpoint I do not like guessign, screaming and 
failing if it can be avoided at all :-).

Best Regards,
Andre

-- 
Andre Heinecke |  ++49-541-335083-262  | http://www.intevation.de/
Intevation GmbH, Neuer Graben 17, 49074 Osnabrück | AG Osnabrück, HR B 18998
Geschäftsführer: Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

---

From: Jon Callas <[email-masked]>
Date: Fri, 14 Sep 2018 23:47:34 -0700

> On Sep 11, 2018, at 1:36 AM, Andre Heinecke <[email-masked]> wrote:
> 
> Hi Jon, Neil,
> 

[…]

> multiple charsets if you whish to verify the message (as it could also be 

I think you’re over-thinking this, Andre.

OpenPGP has many things in it from its past, and its past includes the computing environment of 1991, namely DOS, and as a system to post messages on BBSes. There are many things that we’d do differently, if we were starting over. There are many things that are basically layering violations, as well.

For example, the compression stuff there is so that people wouldn’t have to edit something up, then run Zip, and then encrypt that. Another thing, and what is relevant here is the distinction between “Text” and “Binary,” which inherits in some backwards way from FTP.

With RFC 2440, we made a (then) bold step of effectively saying that OpenPGP says that Text is Unicode, coded in UTF-8. By the time of the early 2000s, just about everyone had moved to Unicode but with holdouts, particularly in Japan, where they still liked (and like) Shift-JIS. The amusingly named “ISO Latin-5” is actually Cyrillic, which is not Latin at all, but it was an eight-bit character set for them. I’m sure I am oversimplifying and there are other holdouts as well, because long tails are like that. Nonetheless, every day that tail becomes more tail-like, because Unicode is what just about everyone uses.

> charset for cleartext signatures or that we should?

I’m saying we *have* a standardized way. The charset is Unicode encoded in UTF-8.

And then we have this bag on the side so that the long-tail people can put hints in. Back to Neil’s comments, we found out in the mid-2000s that Unicode wasn’t getting traction in Japan and they were doing all sorts of contortions to deal with crossed character sets. The final compromise within the working group was to put this in so that we could both recognize that the official stance (text is Unicode) and let the dissenters have a better experience.

It’s actually good that the header isn’t signed, because if it were, it would end up causing more issues — is it part of the message or not, to start with. That header is metadata, and it’s metadata that can’t be definitely stated because of layering. It’s good that a component can slide that hint in without having to be integrated with the whole of the crypto system.

> through the openpgp engine it should be ok.

I concur; the OpenPGP engine should not have to worry about character sets. It’s got too much to worry about as it is.

> failing if it can be avoided at all :-).

The only time you have to guess is if someone is not using Unicode. That’s the whole point.

	Jon