Subject: Re: [saag] Algorithms/modes requested by users/customers

From: Paul Hoffman <[email-masked]>
Date: Sun, 17 Feb 2008 09:47:53 -0800

At 3:42 PM -0800 2/16/08, Randall Atkinson wrote:
>1) NIST FIPS-140 conforming algorithms/modes.
>and
>2) Suite-B conforming algorithms/modes.

This seems reasonable, and similar to what I hear from VPNC members. 
(Sadly, these same members ship new equipment defaulting to 
algorithms not in that list, but that's a different thread...)

>This seems to be driven externally by insurance firms that tell
>their customers to only use equipment whose cryptographic
>subsystems/modules have been (or are going to be) evaluated
>formally under FIPS-140.

That is unreasonable. The ratio of "number of systems that use 
crypto" to "number of systems that have been or are going to be 
evaluated formally under FIPS-140" is incredibly high. The FIPS 140 
evaluation process is horribly broken on a number of axes, and all 
the vendors in the field know it and complain privately about it.

If you had instead said "have been (or at least could be) evaluated", 
I would agree with you. Buyers want to know that, if pressed hard, 
they can say "well, it is like that other system over there that got 
certified", and to do that, the uncertified system has to at least 
support all the right crypto. There is a noticeable but 
non-quantifiable preference for systems with a FIPS 140 certificate, 
but (outside the USGovt, of course) nearly no hard demand.

>And I'll note that this are not really driven particularly by US firms.
>European, Asia/Pacific, and Latin American firms are making the
>exact same requests for FIPS-140 of their equipment vendors.

I have heard that as well from VPNC members. But note the use of the 
term "requests".

It would be a mistake for the IETF to look like we support the FIPS 
140 certification process, even though it makes good sense for us to 
use the algorithms in that process as the basis for a suggested 
cross-IETF algorithm suite.

--Paul Hoffman, Director
--VPN Consortium

---

From: Randall Atkinson <[email-masked]>
Date: Sun, 17 Feb 2008 17:06:44 -0800

Earlier Paul Hoffman said:
% If you had instead said "have been (or at least could be) evaluated",
% I would agree with you. Buyers want to know that, if pressed hard,
% they can say "well, it is like that other system over there that got
% certified", and to do that, the uncertified system has to at least
% support all the right crypto. There is a noticeable but
% non-quantifiable preference for systems with a FIPS 140 certificate,
% but (outside the USGovt, of course) nearly no hard demand.

It seems that what you are hearing and what I am hearing
differ somewhat in this area.

As an example, I've had multiple large non-US commercial firms
in the City of London tell me that they require FIPS-140 approvals
for cryptographic equipment they will purchase -- and that this
was originally the idea of their insurers.   These same firms have
very substantial networking deployments at present and purchase
equipment in large quantities.

Separately, several governments other than the USA are requiring
FIPS-140 approvals in equipment that they purchase.  These
other countries include places in Europe and Asia/Pacific and
other continents/regions.  Again, the volume of such systems is
both non-trivial and quantifiable.

% It would be a mistake for the IETF to look like we support
% the FIPS 140 certification process,

So far as I am aware, your comments above are the first suggestion
of that.  Certainly I did NOT suggest that and don't suggest it now.

% even though it makes good sense for us to use the algorithms in
% that process as the basis for a suggested cross-IETF algorithm suite.

That is farther than I went.  I simply suggested that there ought to
be open specifications for how to use FIPS-conformant algorithms
and modes with Internet protocols.  An "open specification" could be
an Informational status RFC, for example.  That way users who
do care about FIPS approvals would have a reasonable shot at
being able to  purchase interoperable equipment from multiple vendors
-- and implementers would have a clear specification to work with.

I did not (and do not now) advocate that the IETF standards track
ought to do any particular thing with respect to cryptographic
algorithms/modes.

Yours,

Ran Atkinson
[email-masked]

---

From: Stephen Kent <[email-masked]>
Date: Tue, 19 Feb 2008 10:15:42 -0500

At 9:47 AM -0800 2/17/08, Paul Hoffman wrote:
>...

Paul,

Can you share the reasons cited by vendors to support the notion that 
the FIPS 140 process is broken.  Compared to other security 
evaluation criteria, e.g. CC or the old Organge Book, most security 
folks I know view the FIPS 140 evaluation program as well managed and 
not very onerous.

Also, if buyers believe that a device that "could be evaluated" is 
good enough, they are being rather naive.  Experience with the FIPS 
140 program has shown that a significant fraction of of products 
submitted for evaluation fail the process. Presumably a vendor 
submitting a product for evaluation believes that the product is 
ready, and will pass, otherwise the vendor is wasting money on the 
evaluation effort. The fact that many products fail evaluation (for 
good security reasons), tells me that a vendor's claim that a product 
"could be evaluated" is not much of an assurance.

Steve

---

From: Randall Atkinson <[email-masked]>
Date: Tue, 19 Feb 2008 07:35:08 -0800

Earlier, Steve Kent wrote:
% Also, if buyers believe that a device that "could be evaluated" is
% good enough, they are being rather naive.  Experience with the
% FIPS 140 program has shown that a significant fraction of of
% products submitted for evaluation fail the process.

The quoted text above is fully consistent with my information
(and my very limited personal experience) on this topic.

% Presumably a vendor submitting a product for evaluation
% believes that the product is ready, and will pass, otherwise the
% vendor is wasting money on the evaluation effort.

Some purchasers, including some US DoD components, do not
require that a product actually be FIPS-140 certified in order
for the purchaser to buy the product.  Instead, some purchasers,
including some US DoD components, only require that a product
be "in evaluation" on the NIST web site in order to bid/sell that
product.

This can incent some vendors to submit products for FIPS-140
evaluation in order to bid/sell the product -- even if the vendor
is unsure whether the product will pass that evaluation at that
point in time.  This last is a business decision based on various
business factors, and is not necessarily a technical decision.

Separately, there is substantial documentation required to pass a
FIPS-140 evaluation, and it is not crystal clear upfront what level
of detail on which topics is required to pass the evaluation.  So,
separate from the question of whether the product itself is ready,
there is usually substantial documentation work required of the
submitter after the formal FIPS-140 evaluation commences.

The one area where I have heard concerns expressed by many
implementers is that it can be difficult/time-consuming/expensive
to "RAMP" an approved product so that a later version is
approved formally under FIPS-140.  I don't know if this is an
actual process issue or merely a perceived issue.  At least one
implementer has commented to me that their latest FIPS-approved
firmware has known-bugs fixed in subsequent non-FIPS-approved
firmware because of (real or perceived) "RAMP" process issues.

% The fact that  many products fail evaluation (for good security
% reasons), tells me that a vendor's claim that a product
% "could be evaluated" is not much of an assurance.

Agreed.  And this point has been made to me by several different
commercial/financial firms.  It is not a coincidence that the
encryptor I use at my home has an actual FIPS-140 certificate,
nor that I use that system in FIPS mode.

(And at this point, I seem a bit far from the charter of the SAAG
list, but I'm not sure where followups should go.   My apologies
to any who viewed this as excessively off-topic.)

Yours,

Ran
[email-masked]

DISCLAIMER:  I am employed by Extreme Networks, but I am never
authorised to speak for my employer.  The text from me above
is my own personal views/understanding, and must not be taken
as a position held by my employer.

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2008 08:18:24 -0800

At 10:15 AM -0500 2/19/08, Stephen Kent wrote:
>Can you share the reasons cited by vendors to support the notion 
>that the FIPS 140 process is broken.

Sure.

- It takes way too long from submission of system to validation, even 
if no problems are found.

- If problems are found during evaluation, the restart time is too long.

- Some of the tests are fairly subjective, and it becomes a game of 
fixing code to please the testing service, not to make the product 
more secure.

- An evaluated product cannot have its core firmware updated without 
losing the validation. For example, if a customer asks for a new 
feature that might touch the crypto, the vendor has to choose between 
losing the validation (and paying for a new one, and waiting for the 
results) or keeping the customer happy.

- The validation doesn't even check for on-the-wire interoperability, 
which is what the customers care about most.

- The test process is too expensive for many low-end devices that 
would be very useful in USGovt offices.

- The system introduces silly modes that make the systems more complicated.

These are the top complaints I hear from both large and small 
vendors. There are certainly others. "Oh, don't get me started" is 
commonly heard when discussing FIPS 140 testing.

>   Compared to other security evaluation criteria, e.g. CC or the old 
>Organge Book, most security folks I know view the FIPS 140 
>evaluation program as well managed and not very onerous.

"Security folks" are not good evaluators on how some processes affect 
the market.

>good enough, they are being rather naive.

Maybe. If a buyer cares about "does this box support the needed 
crypto in an interoperable fashion", then the perception is fine. If 
they care about all the details that FIPS 140 testing covers, then of 
course it is naive.

>Experience with the FIPS 140 program has shown that a significant 
>fraction of of products submitted for evaluation fail the process.

This statement is usually bandied about without any quantification of 
what failures were found, as it is here. As someone who creates test 
suites, I can assure you that I can make a few picky rules for 
systems that would cause some of those systems to fail, but most end 
users who understood those few rules would say that those rules were 
silly. Of course, different end users have different requirements. It 
is generally felt by vendors that some/many of the rules for FIPS 140 
are silly and unneeded; on the other hand, it is likely that at least 
a few USGovt customers would really want some of those rules for 
particular reasons.

>Presumably a vendor submitting a product for evaluation believes 
>that the product is ready, and will pass, otherwise the vendor is 
>wasting money on the evaluation effort.

Wrong, very wrong. You cannot tell what the testing agency might try 
to knock you down on ahead of time. Of course, you cover what you 
can, but you also know that there is a good chance they'll just whack 
you a bit because it makes them look good.

>The fact that many products fail evaluation (for good security 
>reasons), tells me that a vendor's claim that a product "could be 
>evaluated" is not much of an assurance.

If you believe in the testing regime, that's fine. Others don't, and 
I believe for very good reasons in their own use scenarios.

--Paul Hoffman, Director
--VPN Consortium

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2008 10:04:12 -0800

At 11:49 AM -0500 2/19/08, Santosh Chokhani wrote:
>On the issue of restart, that is misunderstanding on Paul's part.  There
>is no restart.

Ahem. I am reporting what the members of VPNC report to me. They are 
the vendors in the testing, not me. I have heard this from multiple 
vendors. If I was not clear that I was reporting what others said, I 
apologize.

>Generally, problem is fixed and you pick up that point.

Not to be too picky, but how can you say "generally" without seeing 
every test that was stopped? Maybe you have not had this problem, but 
others say they have.

>Subjective tests seems to be misunderstanding on Paul's part.  Having
>been in the Orange Book, CC and FIPS process, I as validator, tester,
>and certifier as well as vendor generally rue about rigidness of the
>standard and lack of subjective judgment on the part of the tester or
>lack of subjective latitude to vendors.

If you, as a tester, feel that there are no subjective parts of the 
test process, that's fine. Some/many people who are being tested 
disagree with you.

>FIPS 140 has specific guidelines on how to deal with minor incremental
>changes that helps reduce cost and calendar time.

Great! It does not seem to have gotten through to the vendors 
themselves as "inexpensive enough" or "fast enough". As a tester, you 
may have a different view.

>The standard is NOT a protocol standard.  It does verify the algorithm
>implementations and thus, FIPS validated algorithms can interoperate.

Quite true.

>In terms of low end devices, 20-30K for level 1 testing amortized over
>devices does not seem too onerous.

...to you. Others disagree, both on the financial cost, and 
particularly on the cost of elapsed time before customers can use the 
latest release from a vendor.

>I do not understand what Paul refers to as silly modes.  The module
>being FIPS validated must use FIPS validated or recognized algorithms
>for a given crypto service.  That seems like a good thing to me.

That is a good thing; it is also not what I was talking about. There 
is disagreement about what needs to be in a "FIPS mode", and whether 
the shipped product needs to allow "FIPS mode" to be enabled, and if 
so, how.

Again: I am reporting what I hear from many VPNC members over many 
years. You as a single vendor and/or tester might feel differently; 
your feeling does not invalidate the views of others.

--Paul Hoffman, Director
--VPN Consortium

---

From: Jack Lloyd <[email-masked]>
Date: Tue, 19 Feb 2008 13:09:24 -0500

> fixing code to please the testing service, not to make the product 
[...]
> - The system introduces silly modes that make the systems more complicated.

I have no experience with the purchasing side, but in my experience
doing FIPS 140 validations, we often had to ask vendors to include
hooks for testing that, from any objective standpoint, made the system
less secure. And because the tests must be made on the same
firmware/software as the as-shipped one (not in a special test/debug
mode), that increased the attack surface of some of these devices
greatly. I will fondly remember the validation where I found several
exploitable buffer overflows in an HSM that had already passed two
previous validations - all the holes were found in the hooks used for
FIPS-140 testing.

The tests that require the RNG be able to seeded with fixed data
always seeemed particularly troublesome/dangerous to me.

Obvious disclaimer: Anecdotes are not data - I just thought a
relatively concrete example might be relevant to the discussion at
hand.

-Jack

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 19 Feb 2008 10:51:46 -0800

At 1:13 PM -0500 2/19/08, Santosh Chokhani wrote:
>My general observation is that vendors do not assign their engineers to
>these efforts and there is a dearth of qualified testers, resulting in
>blind leading the blind.

That is a fairly damning criticism of the process, of course.

It would not be bad if this were a voluntary logo program like VPNC 
or ICSA Labs has; when it prevents a large customer (many large 
customers, according to Ran) from buying up-to-date equipment that 
meets their security needs, it has serious implications for the 
security of the purchasers.

--Paul Hoffman, Director
--VPN Consortium

---

From: Stephen Kent <[email-masked]>
Date: Tue, 19 Feb 2008 17:35:18 -0500

At 8:18 AM -0800 2/19/08, Paul Hoffman wrote:
>- It takes way too long from submission of system to validation, 
>even if no problems are found.

I have gone through a level 3 eval and compared to ALL other security 
criteria evaluations, the time to perform the eval is much, much 
better. So, my guess is that anyone complaining about this has no 
experience with evals for things like CC.

>- If problems are found during evaluation, the restart time is too long.

ibid.

>fixing code to please the testing service, not to make the product 

It is true that some test criteria are not completely cut and dried. 
That is a problem with every one of these secruity eval criteria. My 
experience is that FIPS 40 is better i this regard than all the 
others.

>feature that might touch the crypto, the vendor has to choose 
>between losing the validation (and paying for a new one, and waiting 
>for the results) or keeping the customer happy.

Absolutely right, and it should be that way! To expect otherwise is 
to fail to understand what a security eval is al about. However, the 
representation above is silly. The old product with the old feature 
set is still validated. It's the new version that cannot yet be sold 
as evaluated. When we had a level 3 device, we also sold a version 
that was not yet evaluated, but had nifty new features, and explained 
the situation to clients, some of who opted for the old version until 
the new version was evaluated.

>- The validation doesn't even check for on-the-wire 
>interoperability, which is what the customers care about most.

It's a crypto security eval criteria, not a protocol compliance eval 
criteria. The labs are not testers of IPsec or TLS; they are testers 
of crypto module functions, period. a buyer who doesn't understand 
that is misinformed.

>- The system introduces silly modes that make the systems more complicated.

e.g., ?

>commonly heard when discussing FIPS 140 testing.

Whiners :-).

>>   Compared to other security evaluation criteria, e.g. CC or the 
>>old Organge Book, most security folks I know view the FIPS 140 
>"Security folks" are not good evaluators on how some processes 
>affect the market.

But we are experienced with various security eval criteria, and one 
ought to judge such criteria in context. In that context FIPS 140 is 
viewed as having a great track record.

>they care about all the details that FIPS 140 testing covers, then 
>of course it is naive.

Your statement seems to conflate two very separate issues.  if a 
buyer is concerned primarily with interoperability or (non-crypto) 
standards compliance, then FIPS 140 is irrelevant.  if they care 
about whether a product that purports to use crypto for security does 
so securely, then FIPS 140 eval is a solid prerequisite.  I think a 
client who cares more about interoperability of a crypto-based 
security product than the security of the product is naive.

>This statement is usually bandied about without any quantification 
>of what failures were found, as it is here. As someone who creates 
>test suites, I can assure you that I can make a few picky rules for 
>silly. Of course, different end users have different requirements. 
>It is generally felt by vendors that some/many of the rules for FIPS 
>140 are silly and unneeded; on the other hand, it is likely that at 
>least a few USGovt customers would really want some of those rules 
>for particular reasons.

I have served several terms on an committee operated by the National 
Academy of Sciences that provides an external review for the NIST 
Security Lab. In our review meetings I have seen the figures for the 
pas/fail rate, and have seen descriptions of the reasons for 
failures, some of which are hilariously bad. I is not only US 
Givernment clients who care about most of the "picky rules."  The 
banking community has mandated FIPS evaluation for years. However, if 
a client is more concerned about the performance of a VPN appliance 
than the security of the device, FIPS 140 is not a good criteria for 
that client.

>can, but you also know that there is a good chance they'll just 
>whack you a bit because it makes them look good.

Nonsense, utter nonsense (says the guy who went trough this process, 
as opposed to the guy who is relating what he has been told by 
others). The criteria are complemented by guidelines for eval labs, 
and these guidelines are available to vendors. So, a smart vendor 
would study the criteria and the guidelines before designing a 
product that will be evaluated. A smart vendor will discuss the 
criteria with the lad it chooses, to minimize uncertainty.

>I believe for very good reasons in their own use scenarios.

What I said is true, irrespective of whether one believes in the 
criteria or the testing methodology. The reality is the products 
fail, and many  (not all) of the failures are demonstrably due to 
serious security problems. So, given this experience, one really 
ought not believe a vendor (whose goal is to sell products) who says 
"oh yeah, our product would pass evaluation, but we don't think it's 
worth the investment." A client who believes this argument should buy 
a used car without a warranty or an inspection by a mechanic. The 
likelihood of a good outcome is similar.

Steve

---

From: Jon Callas <[email-masked]>
Date: Tue, 19 Feb 2008 16:10:06 -0800

> these efforts and there is a dearth of qualified testers, resulting in

I want to agree with Paul Hoffman that FIPS 140 is unnecessarily  
painful. I think I will also agree with Stephen Kent and say that FIPS  
is to CC as laparoscopic surgery is to open heart.

Santosh also gets a big +1 from me, and I'll tell how even this dark  
cloud has a silver lining.

When PGP first went through FIPS 140, we assigned a dedicated engineer  
to the process. Shepherding software through FIPS 140 was so painful,  
so mind-numbing, so annoying that he quit the company, quit  
cryptography, and quit computer security altogether. He took a job  
with a company that produced MP3 music software. That company was  
bought out by Apple, and the software turned into what we now know as  
iTunes. He is at Apple to this day as the lead of iTunes.

So the next time you listen to an iPod, think about FIPS 140, and  
thank the horrible process.

	Jon

---

From: [email-masked] (Peter Gutmann)
Date: Thu, 21 Feb 2008 00:15:56 +1300

Paul Hoffman <[email-masked]> writes:

Some quick notes on the consequences of three of these which may not be
immediately obvious:

>- It takes way too long from submission of system to validation, even if no
>problems are found.

Corollary: Users have the option of running a current, up-to-date non-
evaluated version or an obsolete evaluated version.  This becomes particularly
critical if there's a security advisory issued for the old version, because
you can't close the hole without losing the eval.  So you get the choice of
running an insecure FIPS 140-blessed version or a secure (well, as far as
anyone knows) non-FIPS 140-blessed version.  For some reason this doesn't seem
to go down well with customers.

>- Some of the tests are fairly subjective, and it becomes a game of fixing
>code to please the testing service, not to make the product more secure.

Corollary: You can get completely different results depending on how you
submit your product, and who to, and when.  I've heard of vendors doing the
equivalent of jury-shopping in order to ease the evaluation process.

>- The validation doesn't even check for on-the-wire interoperability, which
>is what the customers care about most.

Corollary: An ability to perform a TLS connect to Amazon (if the product is in
this case a TLS crypto system) tells the customer more than the FIPS 140 eval
ever can, since much of the TLS functionality is orthogonal to what FIPS 140
tests.

>"Oh, don't get me started" is commonly heard when discussing FIPS 140
>testing.

That's why I've tried to keep out of this so far :-).  Ask enough people and
you can get a very long list.  This and CC do make for good conversation
starters among security geeks though.

Peter.

---

From: "Santosh Chokhani" <[email-masked]>
Date: Tue, 19 Feb 2008 11:49:53 -0500

On the issue of it takes too long, the FIPS process is the envy of other
schemes.  One could always be faster.

On the issue of restart, that is misunderstanding on Paul's part.  There
is no restart.  Generally, problem is fixed and you pick up that point.
This is generally a vendor problem.  Depending on the nature of
problem/failure, other aspects of the testing can continue without loss
of calendar time.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Hoffman
Sent: Tuesday, February 19, 2008 11:18 AM
To: Stephen Kent
Cc: [email-masked]; Randall Atkinson
Subject: Re: [saag] Algorithms/modes requested by users/customers

Sure.

- The system introduces silly modes that make the systems more
complicated.

--Paul Hoffman, Director
--VPN Consortium
_______________________________________________
saag mailing list
[email-masked]
http://mailman.mit.edu/mailman/listinfo/saag

---

From: "Santosh Chokhani" <[email-masked]>
Date: Tue, 19 Feb 2008 13:13:06 -0500

Paul,

These statements are not based on "feelings".  They are based on
experience of doing third party testing such as TPEP (TCSEC), TTAP
(TCSEC and CC), CC testing and FIPS since 1987 and being involved in the
FIPS program since its pre-formative days.

If there are specific concerns, I would be happy to look into those.

If you get a good tester, he/she will force the vendor to provide a good
engineering answer and many of the general issues you are mentioning
will get tackled right.

I would recommend that VPNC members/vendors have good engineers making
sound arguments.  Most of the times, that works.

-----Original Message-----
From: Paul Hoffman [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2008 1:04 PM
To: Santosh Chokhani; Stephen Kent
Cc: [email-masked]; Randall Atkinson
Subject: RE: [saag] Algorithms/modes requested by users/customers

Quite true.

--Paul Hoffman, Director
--VPN Consortium

---

From: "Vishwas Manral" <[email-masked]>
Date: Thu, 21 Feb 2008 16:59:00 -0800

Hi Jon,

>  thank the horrible process.
Mind you, iTunes has its own DRM mechanism and that requires
cryptography. I have in the past worked on DTCP-IP which is now the
content protection mechanism in home devices (after DLNA adopted it)
used for content. It uses mechanisms similar to IKE (called AKE) and
DTCP (like IPsec).

So may be it is not as far away from cryptography as you might assume.

Thanks,
Vishwas

>         Jon
>
>
>  -----BEGIN PGP SIGNATURE-----
>  Version: PGP Universal 2.6.3
>  Charset: US-ASCII
>
>  wj8DBQFHu2/hsTedWZOD3gYRAuZbAJ9IFEWuafL6fAB+2MxJvwIEOmLJiACgkJrs
>  eRur6xWa+w6FdH022GobtDg=
>  =ZTOd
>  -----END PGP SIGNATURE-----
>
>