Subject: [saag] ASN.1 vs. DER Encoding

From: "Dr. Pala" <[email-masked]>
Date: Tue, 26 Mar 2019 17:24:38 +0100

Hi SAAG,

I just wanted to provide some feedback based on the contents of some 
presentations I have seen in the security area. In particular, I noticed 
that some authors seem to confuse the definition of information objects 
(ASN.1) and their encoding (e.g., DER). I noticed that, sometimes, when 
ASN.1 was mentioned, what was really the topic of discussion was 
actually related to DER encoding.

Since I have seen this happening multiple times, I am starting to wonder 
if I am the one who is wrong. In particular, my question is: do people 
in the security area support the statement that ASN.1 is equivalent to 
DER encoding ?

I ask this because ASN.1 is "used for the definition of data types, 
values, and constraints on data types." independently from the how the 
data is actually encoded (BER, PER, XER, DER, etc.) - it just happens 
that in X.509 PKIs, we use DER as the preferred encoding (and PEM for 
7-bit transport mode). Therefore when we talk about certificate parsing, 
for example, we do parse DER/PEM, not ASN.1. For example, for the 
proposal around CBOR-encoded certificates (not endorsing the idea, just 
using this as an example), defining the CBOR Encoding Rules (CER ?) 
would provide a path to provide CBOR encoding for all ASN.1 definitions 
we use in PKIX.

Maybe this distinction is not important for people that already have a 
good understanding of the information model, however there might be 
newcomers (new IETF-ers or just new to the security area) that might 
think the two are the same when they are not, in my opinion.

Therefore, my recommendation is to keep this distinction in mind when 
talking about encoding and parsing of, for example, certificates. I hope 
this helps.

Just my 2 cents...

Cheers,
Max

-- 
Best Regards,
Massimiliano Pala, Ph.D.
OpenCA Labs Director
OpenCA Logo

---

From: Yoav Nir <[email-masked]>
Date: Tue, 26 Mar 2019 17:36:17 +0100

> On 26 Mar 2019, at 17:24, Dr. Pala <[email-masked]> wrote:
> 
> Hi SAAG,
> 
> I just wanted to provide some feedback based on the contents of some presentations I have seen in the security area. In particular, I noticed that some authors seem to confuse the definition of information objects (ASN.1) and their encoding (e.g., DER). I noticed that, sometimes, when ASN.1 was mentioned, what was really the topic of discussion was actually related to DER encoding.
> 
> Since I have seen this happening multiple times, I am starting to wonder if I am the one who is wrong. In particular, my question is: do people in the security area support the statement that ASN.1 is equivalent to DER encoding ?
> 
> I ask this because ASN.1 is "used for the definition of data types, values, and constraints on data types." independently from the how the data is actually encoded (BER, PER, XER, DER, etc.) - it just happens that in X.509 PKIs, we use DER as the preferred encoding (and PEM for 7-bit transport mode). Therefore when we talk about certificate parsing, for example, we do parse DER/PEM, not ASN.1. For example, for the proposal around CBOR-encoded certificates (not endorsing the idea, just using this as an example), defining the CBOR Encoding Rules (CER ?) would provide a path to provide CBOR encoding for all ASN.1 definitions we use in PKIX.
> 
> Maybe this distinction is not important for people that already have a good understanding of the information model, however there might be newcomers (new IETF-ers or just new to the security area) that might think the two are the same when they are not, in my opinion.
> 
> Therefore, my recommendation is to keep this distinction in mind when talking about encoding and parsing of, for example, certificates. I hope this helps.
> 

Hi, Max.

I have noticed this as well. The thing is, although there are several sets of encoding rules (like GSER <https://tools.ietf.org/html/rfc3641>), in practice all of our certificates (and CRLs, certificate requests, and any other structure defined in ASN.1) are encoded in DER or BER. So it has become common to refer to “things whose structure is defined with ASN.1” and “things whose encoding is DER” as the same sets. Because they are. So a lot of us are referring to DER encoding as ASN.1 even though this is a mistake.

Maybe if some other encoding rule (like a CBOR encoding rule) got some traction, people would be more motivated to use the terms correctly.

Yoav

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 11:49:51 -0500

On Tue, Mar 26, 2019 at 05:24:38PM +0100, Dr. Pala wrote:
> ASN.1 was mentioned, what was really the topic of discussion was actually
> related to DER encoding.

And then they (rightly!) hate BER/DER/CER, so they propose inventing
something new, often badly.  In this field, there is nothing new.  I'm
sure even flatbuffers isn't new.

I say "rightly" because TLV encodings are just terrible.  We really do
need non-TLV encodings (see below).

To save everyone else the bother of reading the rest of this: I agree
with you and we should have a non-TLV encoding for ASN.1 that makes
IETFers happy, and CBOR is a good a basis for that.

> Since I have seen this happening multiple times, I am starting to wonder if
> I am the one who is wrong. In particular, my question is: do people in the
> security area support the statement that ASN.1 is equivalent to DER encoding
> ?

I sure do not.

However, it is true that BER/DER are pretty much the only encoding rules
used in IETF documents, and that is mostly a result of lack of tooling.

Mind you, XDR is basically a PER- or OER-like encoding with 4-byte
alignment, for a subset of ASN.1.

Seen this way we could could easily produce an ASN.1-compatible ER that
nonetheless has no direct, normative link to ASN.1, and which has its
own syntax, just to please those who can't bother to find or built the
tools they need if they are based on a 40 year old technology.

> I ask this because ASN.1 is "used for the definition of data types, values,
> and constraints on data types." independently from the how the data is
> actually encoded (BER, PER, XER, DER, etc.) - it just happens that in X.509
> PKIs, we use DER as the preferred encoding (and PEM for 7-bit transport
> mode). Therefore when we talk about certificate parsing, for example, we do
> parse DER/PEM, not ASN.1. For example, for the proposal around CBOR-encoded
> certificates (not endorsing the idea, just using this as an example),
> defining the CBOR Encoding Rules (CER ?) would provide a path to provide
> CBOR encoding for all ASN.1 definitions we use in PKIX.

<sarcasm>
  Yes, but everybody knows you can't parse ASN.1 with a LALR(1)
  parser[*]!
</sarcasm>

* That's something I heard as a truism eons ago, but here's a
  bison-based ASN.1 compiler:

  https://github.com/heimdal/heimdal/tree/master/lib/asn1

I would be quite happy with a CBORER.  CER is already taken, as a flavor
of BER that is canonical, but in different ways than DER is.

BER/DER/CER and all TLV encoding rules are just awful, and we should
really have an alternative to them for Internet protocols.

I suspect proponents of new certificate encodings aren't interested only
in new encodings, but new schemas as well.  ASN.1 is too complicated and
all that.

> Maybe this distinction is not important for people that already have a good
> understanding of the information model, however there might be newcomers
> (new IETF-ers or just new to the security area) that might think the two are
> the same when they are not, in my opinion.

We are doomed to reinvent our wheels.  Often badly.

It's always easier to create new legacy than to deal with old legacy,
even though new legacy very soon becomes old legacy, and so new legacy
is invariably worse than old legacy, just not right away.  Few
understand this.

> talking about encoding and parsing of, for example, certificates. I hope

It probably won't.  You're inviting a flame war, and I'm probably
helping you.

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Mar 2019 18:20:57 +0100

On Mar 26, 2019, at 17:49, Nico Williams <[email-masked]> wrote:
> CBORER

Made my day :-)
(Probably works for the Boring Company?)

But seriously, we actually have a replacement for the ASN.1 level of the picture, as well (the picture where CBOR is a replacement for DER):  

	CDDL.

	https://tools.ietf.org/html/draft-ietf-cbor-cddl-08

In RFC editor queue since 2019-03-25 (a.k.a. yesterday).

https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl/referencedby/
(RFCs/drafts referencing previous versions: https://datatracker.ietf.org/doc/draft-greevenbosch-appsawg-cbor-cddl/referencedby/ — if your draft still does this, please update.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Mar 2019 18:25:45 +0100

> is invariably worse than old legacy, just not right away.  Few

Oh, it’s legacy when you already have typed it in.

But I’d still rather program in C++20 legacy than in FORTRAN II legacy.

Grüße, Carsten

---

From: Volker Birk <[email-masked]>
Date: Tue, 26 Mar 2019 18:25:50 +0100

> security area support the statement that ASN.1 is equivalent to DER encoding

For sure not. And you're not mistaken AFAICS.

> talking about encoding and parsing of, for example, certificates. 

I'm supporting this.

Yours,
VB.
-- 
Volker Birk, p≡p project
mailto:[email-masked]
https://pep.software

---

From: Volker Birk <[email-masked]>
Date: Tue, 26 Mar 2019 18:29:04 +0100

> used in IETF documents, and that is mostly a result of lack of tooling.

p≡p is using PER and XER. We're using this Free Software tool for it:

http://lionet.info/asn1c/compiler.html

Yours,
VB.

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 12:31:54 -0500

> http://lionet.info/asn1c/compiler.html

Cool!

The lack of tooling in question is in the past.  And, of course, if
you're writing in Go, Rust, node, etc., then the tooling has to support
your choice of programming language.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 12:33:33 -0500

> Oh, it’s legacy when you already have typed it in.

Yes.

> But I’d still rather program in C++20 legacy than in FORTRAN II legacy.

And I'd rather program in Rust.  But a lot of legacy is in C, so I've to
deal with C as well.  Thankfully no Fortran.  But someone will still
have to maintain the legacy you and I would rather not touch.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 12:34:40 -0500

> (RFCs/drafts referencing previous versions: https://datatracker.ietf.org/doc/draft-greevenbosch-appsawg-cbor-cddl/referencedby/ — if your draft still does this, please update.)

There had better be something remarkably new and interesting in that DL,
otherwise I don't see the point of abandoning ASN.1.  At the very least
it'd better have nicer syntax and have a great deal of functionality.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 26 Mar 2019 20:38:58 +0100

Dr. Pala <[email-masked]> wrote:
    > presentations I have seen in the security area. In particular, I
    > noticed that some authors seem to confuse the definition of information
    > objects (ASN.1) and their encoding (e.g., DER). I noticed that,
    > sometimes, when ASN.1 was mentioned, what was really the topic of
    > discussion was actually related to DER encoding.

    > Since I have seen this happening multiple times, I am starting to
    > wonder if I am the one who is wrong. In particular, my question is: do
    > people in the security area support the statement that ASN.1 is
    > equivalent to DER encoding ?

ASN.1 is the goo the causes DER encoding to be present on the wire.
While it is true that ASN.1 could be used with different encoding rules,
nobody really cares about that.   Almost nobody uses encoders/decoders that
generate code from ASN.1 today, it's all hand-coded...

This is why we hate DER, and we like CDDL/CBOR.

    > think the two are the same when they are not, in my opinion.

They are, from a practical point of view, the same.
That wasn't true back in 1986, but many of the people who would make the
mistake were not even alive then.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 26 Mar 2019 16:01:03 -0400

On Tue, Mar 26, 2019 at 08:38:58PM +0100, Michael Richardson wrote:

> Almost nobody uses encoders/decoders that

The exceptions are notable and praiseworthy.  Heimdal's (Kerberos
and X.509) ASN.1 codecs are generated by an ASN.1 compiler from the
ASN.1 module definitions.

> This is why we hate DER, and we like CDDL/CBOR.

Whatever the abstract syntax, the real solution is probably more
widely available/used compilers.  Hand-rolled codecs will have more
(frequent and subtle) bugs.

-- 
	Viktor.

---

From: Carl Wallace <[email-masked]>
Date: Tue, 26 Mar 2019 16:08:58 -0400

>(frequent and subtle) bugs.

Fully agree with this. The first time I had to consume CBOR/COSE the
encoded structures were every bit as buggy as folks complain about DER/CMS
structures. 
>
>-- 
>	Viktor.
>
>_______________________________________________
>saag mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/saag

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 26 Mar 2019 15:20:24 -0500

Hi Max,

> actually related to DER encoding.

Thanks for raising the topic.

> in the security area support the statement that ASN.1 is equivalent to 

I don't, though the comment downthread that in modern practice they appear
together is also t rue.

> talking about encoding and parsing of, for example, certificates. I hope 

Well, we should generally aim for precision in language, to the extent
reasonable.  Even occasionally being precise in this particular matter is
probably enough to keep us honest, whether we use sloppy shorthand the rest
of the time or not.

-Ben

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 16:48:18 -0500

> need non-TLV encodings (see below).

Now to back up that assertion:

1) TLV encodings are bloated by nature due to being highly redundant.

2) That redundancy is a source of errors when manually coding a codec.

   Now, this is not that big a deal because we should all be using code
   generators, and none should be manually writing a codec.  Yet there
   is so much hand-rolled BER/DER codec code out there...

   Also, all encodings will have lengths buried in structures whose
   lengths are also written elsewhere -- this redundancy is not entirely
   avoidable, but TLVs add more of it than is absolutely necessary.

3) DER is a canonical variant of BER, using a) definite length
   encodings of structures (SEQUENCEs) and other things, and b)
   minimal-length variable-length encodings of lengths and values.

   This has a few negative side-effects:

   a) it's not possible to know a length until the value it is the
      length of has been encoded, which means one must encode "from the
      right",

   b) there is no possibility of on-line encoding.

   On the other hand, CER uses indefinite length encoding, which avoids
   those two isses, but then nobody uses CER, not in Internet protocols
   anyways.  IIRC some other choice made in CER's specification turns
   out to be suboptimal, thus the choice of DER or CER is always
   dissatisfying in some sense.

   To be fair, we shouldn't need canonical encodings at all.  And yet
   isn't there a canonical JSON effort?  We never seem to fully stop
   having to re-encode structures...

   Also, it's fair to note that while DER has no possibility of on-line
   encoding, CER's use of indefinite length encodings means that
   decoding is necessarily online, with the recipient not able to know
   the total size of a message as it reads it.

   On the whole, indefinite length encodings are better.

4) BER (and DER and CER) is supposed to be self-describing, which means
   "you don't need to know the schema in order to parse the message",
   but this is only half true, as type information is lost when using
   IMPLICIT tags (you get to know if a value is of structured or scalar
   type, but not the actual type).

   Using EXPLICIT tags, on the other hand, makes the encoding a TLTLV
   encoding, thus adding even more bloat!  We use EXPLICIT tags in
   Kerberos, FYI.

5) If you're doing anything other than dump a structure, you don't need
   it to be self-describing -- you'll know the schema not least because
   we publish them.  As long as there's an indicator of top-level type
   on the outside, you can decode the inside by reference to the schema
   and encoding rules without needing TLV encoding rules.

   Thus there is almost zero benefit to self-describing encodings.
   
   Self-describing encodings are merely a crutch.  Perhaps they had
   their utility before compilers became the norm, but they have provide
   no real benefit now.

I think I could make more arguments in this vein.  I'll stop here.

Nico
--

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 26 Mar 2019 16:56:26 -0500

> 1) TLV encodings are bloated by nature due to being highly redundant.

Before I dig in too far to the rest, just to check: you're limiting to TLV
encodings that are nested and still do TLV at every level of the hierarchy?
I can't tell if you don't like things that are more like "a flat array of
things, each of which has tag, length, and (usually not nested) value.

-Ben

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 17:05:13 -0500

> ASN.1 is the goo the causes DER encoding to be present on the wire.

ASN.1 has a number of encoding rules options.  One can always add more.
Indeed, the IETF already has!  (Search for GSER.)

It would be more accurate to say that "ASN.1 is the goo that generally
causes us to use DER encoding", but to be truly accurate I'd rephrase it
as "often, because we've chosen DER in the past, we tend to or have to
stick to DER in the future".

E.g., if you're working on an extension to PKIX/CMS/Kerberos, it's
pretty natural to use DER for it because if you have a PKIX or Kerberos
implementation, chances are you have an ASN.1/DER implementation You
could use a different encoding, but sticking to what you already have
tooling for is the simplest thing to do.

If you're making changes to PKIX/CMS/Kerberos that don't even fit in BIT
STRING / OCTET STRING typed holes, then you must use DER.

> This is why we hate DER, and we like CDDL/CBOR.

It would be easier for me to adapt Heimdal's ASN.1 compiler to support
alternate encoding rules than to start from scratch or add a new syntax
to it.

The syntax is just a syntax, but by ditching ASN.1 gratouitously you
force me to spend a lot more effort on my tooling, thus creating more
new legacy, and also more old legacy if I do start from scratch (because
now I have TWO implementations of different but similar things to
maintain).

That is huge cost that CDDL is going to force on the rest of us, which
is why, unless there is some compelling reason for it, or unless there
is a straightforward mapping between the two syntaxes (which would make
clear that the one is not necessary, but at least not too problematic)
I'm opposed.

If there's a simple mapping between CDDL and ASN.1, then I'll grudgingly
accept CDDL.

> mistake were not even alive then.

It's the reverse.  In 1984 BER/DER/CER were pretty much the only ERs,
and PER was a thing the ITU-T was building in response to the Internet
community's dislike of TLV encodings.  Now there's a plethora of
encoding rules.

You might say that perception trumps things, but if it's your perception
and you're not willing to consider that it's wrong, then I'd say that's
a problem because it causes me to do more work to deal with the fallout
of your incorrect perception.

CBOR is fine, and we do need better alternatives to BER/DER in the IETF,
but we don't need a new syntax.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 17:10:55 -0500

> things, each of which has tag, length, and (usually not nested) value.

The former.  Specifically the BER family of encoding rules.

I indicated further down that it's not possible or easy to entirely
avoid nested lengths, but that TLV ERs necessarily have many more of
them than is absolutely necessary.

Each unnecessary length nesting is one more place where a careless or
inexperienced programmer can make a serious mistake in a hand-rolled
codec.

Sure, we should all be using Rust, and none should be hand-coding a
codec, which, if it were the case, would limit the TLV damage to bloat,
and possibly precluding on-line encoding (see further below in my post).

Bloat, however, is still a problem.

If you have tooling, then there's no use whatsoever to TLV ERs for new
protocols, and then you don't need to suffer even just the bloat.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 17:27:41 -0500

> encoded structures were every bit as buggy as folks complain about DER/CMS

I recently had to review a hand-rolled codec by a seasoned developer.
It had a number of fatal security bugs, exactly as one might have
expected (as I did, going in).  Hand-coding codecs is simply very
difficult.  It's easier when you have "bits on the wire" specifications,
like TCP/IP, or when you have XDR specifications.

No one uses a compiler to generate TCP/IP packet parsers, though this
is first and foremost a result of not having anything like a
machine-readable formal language specification for TCP/IP packet
headers.

XDR is so darned obvious and simple that in Heimdal we have just library
functions for it and simply don't have or need a compiler.  The
traditional XDR compiler, rpcgen, is also very limited, so generally XDR
codecs are hand-coded using well-maintained and well-tested libraries
that make it easy.

Something similar can be said of SSHv2's encoding rules.

The ASN.1 packed-like encoding rules are XDR-like, though with 1-octet
instead of 4-octet alignment -- these too are easier to hand-code, but
because there's often an escape to BER (e.g., for encoding of lengths of
fields that cannot have fixed lengths, or encoding OIDs), it's still
tricky by comparison to XDR.

But it is a very strong truism that, whatever the encoding rules, having
a compiler for an abstract syntax that generates codecs with usable APIs
and characteristics, is inordinately easier, safer, and less time-
consuming than hand-rolling a codec for any encoding rules.

It is also a strong truism that simple syntaxes with simple encodings
admit safer hand-coding of codecs, but also that, by being informal,
also typically preclude compilation.

I would and do accept informal syntaxes and encoding rules for protocols
like SSHv2, or TLS, because a) it's already done, b) the messages are
simple enough, c) it's easy enough to build libraries that make mistakes
unlikely, d) often the implementors do not believe they have access to
proper tooling, e) having access to ASN.1 tooling doesn't help if the
only ERs you get are TLV ones.

CDDL is not an informal syntax.  For new formal syntaxes I have a higher
bar just because it is very difficult to come up with something that
ASN.1 does not already have -- unless there's a trivial mapping between
the two.

BTW, who can forget FastInfoSet?  That's... XML with transliteration to
ASN.1 and application of PER as a compression of XML.  While XER is
essentially the reverse: transliteration of ASN.1 to XML, with XML as
the encoding rules.  These, for me, prove the point that syntaxes are
likely interchangeable (so why build a new one?).

Nico
--

---

From: "Dr. Pala" <[email-masked]>
Date: Wed, 27 Mar 2019 00:36:06 +0100

Hi Nico, all,

thanks to everybody for your messages - I think the conversation is very 
interesting. A complier for TCP/IP packet parser is really an 
interesting idea :D But only if we (just a joke, do not kill me!) use 
ASN.1 and define a TIED (TCP/IP Encoding Rules) :D

Back to be serious... At the end of this thread, you ask "/why build a 
new one/" ?

I think, but there might be other reasons I am not aware of, that the 
main argument was to save few hundreds bytes per certificate (but the 
first step was to change your certificates' profile as an interim 
solution - which in many environments you can not really do for many 
reasons.

... and this consideration reminds me about an observation that I have 
not brought up during the meeting, but I should have: the current trend 
in cryptography seems to be oriented towards the need to use larger 
structures (signatures and keys) to cryptographically authenticate data 
(e.g., certificates, etc.). I think that ECC was a pleasant aberration 
from this point of view that made everybody think we discovered gold - 
same or increased security at a fraction of the price! That is great - 
and we are all very happy we can now use a quite efficient 
crypto-systems like ECC. However, I think, things are going to change: 
most of the quantum-resistant algorithms come with increased signature 
and/or key sizes (up to 40K for signatures in some cases) or with new 
usage paradigms (stateful keys) that are not trivial to handle.

Besides the fact that I am all for being more efficient - the added 
efficiency (that frees more resources that can be dedicated to other 
tasks) can be leveraged to increase security and/or to provide more 
services without incurring in additional costs. However, my fear is that 
people will start to feel "comfortable" with the idea that the size of 
authentication data is very small and will start designing systems that 
do not take in consideration the envisioned evolution of crypto 
algorithms... leading to some inevitable disaster down the road.

In this view, maybe pushing for such an herculean effort to save few 
hundred bytes per certificate might come with costs so high (in terms of 
the needed changes to support the new format(s)) that would not justify 
its deployment, IMHO.

Last consideration about "This is why we hate DER, and we like 
CDDL/CBOR." statement - I think that each format has its own merits and 
downfalls (even the amount of code that exists out there that supports a 
format can be seen as a merit) and it is up to us to provide good 
indications about when and where the different formats make sense (i.e., 
it depends on the environment). For example, in most of my application 
the CDDL/CBOR approach would not be secure enough because of the 
"relaxed" nature of the parsers - but that is just one environment, 
other environments might have different reasons to deploy/use this 
format based on their requirements.

Bottom line I would say rephrase your statement to "In our 
environment/ecosystem DER is not the right choice because of these 
reasons ... the pair CDDL/CBOR, on the other hand, satisfies these 
requirements because .... "

Thanks again to everybody for the interesting discussion :D

Cheers,
Max

-- 
Best Regards,
Massimiliano Pala, Ph.D.
OpenCA Labs Director
OpenCA Logo

---

From: Nico Williams <[email-masked]>
Date: Tue, 26 Mar 2019 21:49:12 -0500

On Wed, Mar 27, 2019 at 12:36:06AM +0100, Dr. Pala wrote:
> Back to be serious... At the end of this thread, you ask "/why build a new
> one/" ?

I agree we need new encoding rules.  I'm skeptical that we need new
syntax.

> I think, but there might be other reasons I am not aware of, that the main
> argument was to save few hundreds bytes per certificate [...]

Yes, that's one reason to need new encoding rules.

(Though OER should fit the bill, but I understand that's not likely to
happen, so I won't mention it again.)

> .... and this consideration reminds me about an observation that I have not
> brought up during the meeting, but I should have: the current trend in
> cryptography seems to be oriented towards the need to use larger structures
> (signatures and keys) to cryptographically authenticate data (e.g.,
> certificates, etc.). I think that ECC was a pleasant aberration from this
> point of view that made everybody think we discovered gold - same or
> increased security at a fraction of the price! That is great - and we are
> all very happy we can now use a quite efficient crypto-systems like ECC.
> However, I think, things are going to change: most of the quantum-resistant
> algorithms come with increased signature and/or key sizes (up to 40K for
> signatures in some cases) or with new usage paradigms (stateful keys) that
> are not trivial to handle.

Sure.  Encoding efficiency is not a great motivator for new encoding
rules for PKIX.  It is for Kerberos though.

>                                 [...]. However, my fear is that people will
> start to feel "comfortable" with the idea that the size of authentication
> data is very small and will start designing systems that do not take in
> consideration the envisioned evolution of crypto algorithms... leading to
> some inevitable disaster down the road.

We can always push the PK to edges of the system and use symmetrically-
keyed tokens elsewhere.  Needham-Schroeder is post-quantum by dint of
being symmetrically keyed, but that also keeps it from scaling to
Internet scale.  An automatic, PK-based federation system can always be
added, and then use PQ PK to get a fully PQ system that scales up and
also amortizes the transmission and compute cost of the PQ PK.

(Otherwise I'm actually not fond of Kerberos, or Needham-Schroeder.  The
one nice thing about Kerberos is not needing CRLs/OCSP, and that's
easily accomplished in PKIX with online CAs that vend short-lived, fresh
certificates.)

> In this view, maybe pushing for such an herculean effort to save few hundred
> bytes per certificate might come with costs so high (in terms of the needed
> changes to support the new format(s)) that would not justify its deployment,
> IMHO.

Yes.

Nico
--

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Mar 2019 09:45:16 +0000

Nico Williams <[email-masked]> writes:

I realise this is the standard ASN.1-vs-everything-else debate that comes up
every year or so, without wanting to dig up the standard responses from every

>2) That redundancy is a source of errors when manually coding a codec.

It's actually the opposite, with a proper TLV encoding you can create a
simple, universal recursive-descent parser that will take any arbitrary
encoded blob and report "valid" or "not valid".  It's with the non-redundant,
or at least non-self-describing, encodings where you need to hand-roll a
parser each time any field anywhere is updated, and which can't be statically
checked like ASN.1 (meaning BER/DER) can.

>Thus there is almost zero benefit to self-describing encodings.

... apart from the fact that they can be statically analysed to check whether
they're well-formed or not, unlike the encodings in PGP, TLS, IPsec, SSH, ...

Peter.

---

From: Peter Gutmann <[email-masked]>
Date: Wed, 27 Mar 2019 09:49:09 +0000

>Something similar can be said of SSHv2's encoding rules.

The ones that arbitrarily mix binary data and comma-delimited text strings?
Of all the encodings I've had to write parsers for, SSH's one scares me the
most.

It's also the one where I've found the most crashes in the other system while
developing the code and getting various bits slightly wrong, e.g. sending
"foo," or ",," as a value, or getting a binary length field slightly wrong in
combination with text-string data.

Peter.

---

From: Nico Williams <[email-masked]>
Date: Wed, 27 Mar 2019 10:15:46 -0500

> previous time it's occurred I'll try and make just one comment:

It does.

> checked like ASN.1 (meaning BER/DER) can.

I covered that.  I understand that one can built BER/DER/CER dumpers
that know nothing of the schema, with some lossage of type information
when using IMPLICIT tagging (none otherwise).

I don't think that's important because as long as you know the outermost
type, you can always parse PER/OER/XDR/whatever by reference to the
actual schema.  The price to pay for self-describing data is bloat and
more chances for hand-rolled codecs to have security vulnerabilities.

> they're well-formed or not, unlike the encodings in PGP, TLS, IPsec, SSH, ...

The protocols you list don't use a formal syntax, which instantly makes
validity checking harder (can't generate the code!).  But if they had
used XDR, or ASN.1 with PER/OER/..., you could in fact automatically
check the validity of the encoding of a message.

A protocol using ASN.1 BER/DER/CER with IMPLICT tagging will lose some
type information, so while you can check a lot of an encoded message's
validity without reference to its schema, you cannot check all of it.
Whereas if you have reference to its schema, then you can check all of
it regardless of whether the encoding rules are TLV or not.

Nico
--

---

From: Sean Leonard <[email-masked]>
Date: Wed, 27 Mar 2019 17:14:11 +0100

> On Mar 27, 2019, at 4:15 PM, Nico Williams <[email-masked]> wrote:
> 

Back to the original topic of this thread, which is “distinguish between ASN.1 and BER/CER/DER”:

I agree that more care ought to be taken in distinguishing between the abstract syntax (ASN.1) and the encodings. If not sure which of BER/CER/DER to use, you can and should say “X.690 encoding”.

My suggestion is to say DER when that is required, and X.690 when it is not. This is because “BER” in people’s minds reads as “not DER”, which is not true. DER-encoded PDUs are BER-encoded, but not vice-versa.

There is nothing good to say about CER since it did not adequately solve any problems, and only created more. :-)

Usually when people say “this is ASN.1 encoded”, they mean “this is BER encoded; it could be CER or DER too, but this is not a guarantee of the protocol.” They almost never mean XER, PER, OER, or any other such encodings. Hence, s/ASN\.1 encod/X.690 encode/ and you are done.

> when using IMPLICIT tagging (none otherwise).

Dealing with tagging issues is supposed to be fixed with AUTOMATIC TAGS but I have yet to see an IETF spec that uses that ASN.1 feature.

> On Mar 26, 2019, at 5:24 PM, Dr. Pala <[email-masked]> wrote:
> in X.509 PKIs, we use DER as the preferred encoding (and PEM for 7-bit transport mode). Therefore when we talk about certificate parsing, for example, we do parse DER/PEM, not ASN.1.
> 
While we are arguing about nomenclature, it is “textual encoding”, not PEM. RFC 7468, thank you. :-)

https://tools.ietf.org/html/rfc7468

Sean

---

From: Nico Williams <[email-masked]>
Date: Wed, 27 Mar 2019 12:00:09 -0500

On Wed, Mar 27, 2019 at 05:14:11PM +0100, Sean Leonard wrote:
> I agree that more care ought to be taken in distinguishing between the
> abstract syntax (ASN.1) and the encodings. If not sure which of
> BER/CER/DER to use, you can and should say “X.690 encoding”.

+1

> My suggestion is to say DER when that is required, and X.690 when it
> is not. This is because “BER” in people’s minds reads as “not DER”,
> which is not true. DER-encoded PDUs are BER-encoded, but not
> vice-versa.
> 
> There is nothing good to say about CER since it did not adequately
> solve any problems, and only created more. :-)

There is little good to say about DER either...  Both made suboptimal
choices of constraints on BER.

> Dealing with tagging issues is supposed to be fixed with AUTOMATIC
> TAGS but I have yet to see an IETF spec that uses that ASN.1 feature.

Yes, sadly we don't use that.

Even with AUTOMATIC TAGS you can have some loss of type information in
the encoding due to automatically-added context tags being implicit.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 27 Mar 2019 12:08:47 -0500

> The ones that arbitrarily mix binary data and comma-delimited text strings?

You can do that in ASN.1 too if you want...  That was just lack of
discipline in using the features of the encoding.  I agree that it's
awful.

> Of all the encodings I've had to write parsers for, SSH's one scares me the

I've seen terrible DER hand-rolled codecs...  It's hard to say which is
worse for hand-coding.  But ad-hoc, informal designs like TLS' or SSH's
preclude code generation, so in that sense they are easily the worst.

XDR is one that's in the middle: you get a formal language, and the
encoding is trivial enough that you can hand-roll it with very good
chances of getting it right *if* you have suitable library.

(In Heimdal we have the krb5_{storage,store,ret}_*() functions, a subset
of which gives you XDR, and which make it easy to write safe codecs.
These should really be renamed heim_*() and moved from libkrb5 to
libheimbase, where they can do more good outside krb5 context.)

> It's also the one where I've found the most crashes in the other
> system while developing the code and getting various bits slightly
> wrong, e.g. sending "foo," or ",," as a value, or getting a binary
> length field slightly wrong in combination with text-string data.

I believe that.

---

From: Peter Gutmann <[email-masked]>
Date: Thu, 28 Mar 2019 01:38:59 +0000

Nico Williams <[email-masked]> writes:

>A protocol using ASN.1 BER/DER/CER with IMPLICT tagging will lose some type
>information, so while you can check a lot of an encoded message's validity
>without reference to its schema, you cannot check all of it.

You can check it, you just need to use heurstics.  That's what dumpasn1 does,
it's aware of the situations in which there's encapsulation and can dig down
into both implicit-tagged data and hole encodings.

(Don't look at the code, it's full of hacks to handle broken encodings and
display things in twenty different custom ways, so it's not representative of
what a decoder should look like).

>Whereas if you have reference to its schema, then you can check all of it
>regardless of whether the encoding rules are TLV or not.

And that's the problem, you need to have the schema for the latest version of
every possible protocol you're likely to examine, and update the code every
time anything anywhere changes.  With a self-describing data type, you only
need to write the format firewall once and it'll work indefinitely.  I've been
using the same ASN.1 firewall code for over fifteen years...

Peter.

---

From: Nico Williams <[email-masked]>
Date: Wed, 27 Mar 2019 23:42:51 -0500

> into both implicit-tagged data and hole encodings.

Yup.

> using the same ASN.1 firewall code for over fifteen years...

Consider extensibility markers: extensions your dumper doesn't know will
be skippable.  Or consider typed-holes: if a hole contains something
that isn't BER, dumpasn1 won't be able to understand it.

Non-TLV encodings exist and get used.  NFSv4 uses XDR -- not TLV.  CBOR
isn't TLV and you can see it's likely going to get used for important
applications.

At some point a generic dumper needs to know the various schemas, and
hopefully it can figure out contextually the type of the outer-most
value and then use schema to drive the rest.  Once you accept this your
dumper no longer needs heuristics.  Keeping up to date is no different
than keeping any TLS/whatever library up to date, but it's easier
because there are no compatibility requirements (it being just a
dumper).

Code bloat is not really an issue.  The Heimdal ASN.1 compiler has an
option for something it calls "templates", but which is really a bit
more like a bytecode compiler and interpreter.  That reduces code size
(because the bytecode is much smaller than the alternative generated C)
and improves performance (for the same reason, which reduces pressure on
the instruction cache).  With byte-compiled modules you could afford to
have all of them built-in to the dumper, and you could even make new
modules downloadable.

Nico
--

---

From: Michael Richardson <[email-masked]>
Date: Thu, 28 Mar 2019 16:04:46 +0100

Dr. Pala <[email-masked]> wrote:
    > thanks to everybody for your messages - I think the conversation is
    > very interesting. A complier for TCP/IP packet parser is really an

Did this back in 2000.
It was intended to enable generation of code/rules/tables for NPUs.
  http://www.sandelman.ca/SSW/ietf/pax-pdl/pax-pdl-00.txt

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 30 Mar 2019 10:31:02 -0500

> check the validity of the encoding of a message.

N.b. that the protocol descriptions in RFC 8446 were run through an
automated syntax checker (IIRC, by Kazuho, though I'm not confident of that
and only bought 20MB of data for this  plane flight).  So I'm not entirely
convinced that this claim applies to TLS 1.3.

-Ben

---

From: Nico Williams <[email-masked]>
Date: Sat, 30 Mar 2019 17:25:35 -0500

> convinced that this claim applies to TLS 1.3.

ISTR hearing about that, but the RFC does say:

  3.  Presentation Language

     This document deals with the formatting of data in an external
     representation.  The following very basic and somewhat casually
     defined presentation syntax will be used.

which is basically saying it's not a formal syntax.

Now, perhaps it can be be formalized.  If so, was that done for the
checking you mentioned, and was it lost?  That would be unfortunate.

Are there implementations that generate codecs for TLS?

Nico
--

---

From: Christian Huitema <[email-masked]>
Date: Sat, 30 Mar 2019 11:43:22 -0700

> On Mar 30, 2019, at 8:31 AM, Benjamin Kaduk <[email-masked]> wrote:
> 

The TLS messages are defined with a formal syntax. It may be an ad hoc syntax, but the descriptions are readable and verifiable by compilation or verification tools.

The TLS syntax appears specifically designed to avoid many of the pitfalls TLV encodings. For example, the syntax defines the fixed encoding length of all integer and length fields, and uses intermediate octet array encodings for extensions. It is certainly much easier to get right than BER or DER.

-- Christian Huitema

---

From: Viktor Dukhovni <[email-masked]>
Date: Sun, 31 Mar 2019 02:21:45 -0400

> On Mar 30, 2019, at 2:43 PM, Christian Huitema <[email-masked]> wrote:
> 

Yes, for bespoke codecs, written by hand.  Are there TLS implementations
that use the structures from the RFC in a machine-readable form, to produce
automatically generated codecs?

-- 
	Viktor.

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 31 Mar 2019 09:39:16 +0200

On Mar 30, 2019, at 19:43, Christian Huitema <[email-masked]> wrote:
> 
> The TLS syntax appears specifically designed to avoid many of the pitfalls TLV encodings. For example, the syntax defines the fixed encoding length of all integer and length fields, 

… which causes the result to be larger than it needs to be.
Compare https://tools.ietf.org/html/draft-rescorla-tls-ctls-01 or other proposals in this field.

> and uses intermediate octet array encodings for extensions. It is certainly much easier to get right

Yeah, as we all saw in heartbleed. :-)
[Don’t forget to celebrate its fifth discovery birthday tomorrow.]

> than BER or DER.

Oh, that I can concede.

There is no free lunch in this space.

The TLS encoding works by opting for simplicity, but it requires bespoke en-/decoders, which may or may not (more often) be compiled from the TLS “presentation language”.  These are built from a small set of primitives that are reasonably easy to get right even when manually coding.

TLS can do that because it delegates its more complex parts to other formats such as ASN.1 DER.

Since the mid-1990s, there has been a trend in the industry towards encodings that enable generic en-/decoders, handling the lexical level of (de-)serialization.  This, of course, was not new (RFC 713 had it in 1976, and ASN.1 BER decoding is generic on the lowest level, too).  The good thing is that these generic codecs can be hardened and used in a variety of applications; the bad thing is that the hardening does not always happen and gets more complicated with the complexity of the generic codec.  While the focus has been on text codecs (XML, JSON) for a while, binary codecs also exist (CBOR being an example from this decade, patterned on the earlier msgpack).

Binary generic codecs can opt to have redundant bytewise length information (such as BER does), or opt to count items instead of bytes except at the lowest level (as CBOR does).  Apart from causing pain when serially encoding, the ability to exploit this redundancy with inconsistent data is an attacker’s play field.  It also is useful when it is necessary to skip entire subtrees in one go; CBOR requires visiting all nodes on the subtree being skipped (unless the “wrap in byte string” design pattern is employed).

Generic codecs can be intricately tied to an arcane data model that calls for a data description language (as in ASN.1, where the tie is perceived so tight that it triggered this entire exchange, or in XML, which at least had an evolution of data description languages over its lifetime), or try to map to a generic data model that a programmer might want to use directly.  JSON, msgpack, and CBOR share most of one such generic data model and mostly differ on coverage and extensibility at this level.  Data description languages can then help map from that generic data model to the application data model, also possibly validating input data in the process, but are not required (“schemaless” decoding); CDDL is one such data description language.

Grüße, Carsten

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 31 Mar 2019 12:36:12 +0100

> There is no free lunch in this space.

Oddly, there sort-of is... if one ignores the ~every-5-years
debate that the new-way (asn.1/der,asn.1/<foo>,xml+dtd,xml+
schema,json,cbor...) is obviously the right answer to everything,
and just go eat your lunch... then that's nearly a free lunch:-)

I never understood how so many people get so excited by this
topic myself. Seems to me programmers will always find a way
to do as well or badly as ever despite the differences in this
kind of tooling. (Training, experience and other kinds of
tools like more strict compiler options can make a difference.)

S.

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 31 Mar 2019 15:38:53 +0000

>    which is basically saying it's not a formal syntax.

I tried to do that some time ago, and found a bunch of errors or ambiguities in the spec.  Yacc-based tool, and notes, here: https://github.com/richsalz/tlsparser

---

From: Nico Williams <[email-masked]>
Date: Sun, 31 Mar 2019 12:46:55 -0500

On Sun, Mar 31, 2019 at 03:38:53PM +0000, Salz, Rich wrote:
> I tried to do that some time ago, and found a bunch of errors or
> ambiguities in the spec.  Yacc-based tool, and notes, here:
> https://github.com/richsalz/tlsparser

ISTR that as well.  Thanks.  Can the ambiguities/errors be addressed?

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 31 Mar 2019 20:30:46 +0000

>    ISTR that as well.  Thanks.  Can the ambiguities/errors be addressed?

Yes.  The WG had no interest in doing so.