Subject: Re: [secdir] SECDIR review of draft-ietf-sfc-nsh-18

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 7 Sep 2017 16:06:00 +0000

Christian,

Thanks for taking time to review this document. The net-result of addressing them will be an improved specification.

We took some additional time before responding, because some of your comments are very likely addressed (or partially addressed at least) in the current revision -20. 

You raise some very general broadly-scoped questions, such as “Why is the IETF even working on such specifications?” Those are probably beyond the scope of a SecDir review, but nonetheless, important LC comments. As those are beyond the technology specifics of draft-ietf-sfc-nsh, I can (and I do) share my perspective, but I will let others (responsible AD, sfc chairs, doc shepherd) add on more authoritatively.

Please find some responses and observations inline.

> On Aug 11, 2017, at 6:20 PM, Christian Huitema <[email-masked]> wrote:
> 
> I have reviewed this document as part of the security directorate's
> ongoing effort to review all IETF documents being processed by the 
> IESG.  These comments were written primarily for the benefit of the 
> security area directors.  Document editors and WG chairs should treat 
> these comments just like any other last call comments.
> 
> The summary of this review is a puzzled reviewer. Why was I asked to 
> review this? Why is the IETF even working on such specifications?

The direct answer to that question is because there is an IETF WG chartered to do so.

There is a contract (borrowing BCP 25’s definition of “Charter”) between the Service Function Chaining (sfc) working group and the IETF to work on such specs.

https://datatracker.ietf.org/wg/sfc/about/

From that contract, deliverables #1 and #2 are fulfilled as RFC 7498 and RFC 7665, respectively. The spec draft-ietf-sfc-nsh is the SFC WG target for deliverable #3, “Generic SFC Encapsulation”.

But, if you knew all this, I am not sure what is the question behind your question… 

I can offer that there’s market demand for interoperable implementations of service function chaining solutions...

> What
> does this have to do with the Internet?

I’ll just share that, just like RFC 7665, this specification narrowscopes itself to a single provider’s operational domain. 

Relevant text from revision -20, that was not there on the revision -18 you reviewed, includes:

   The intended scope of the NSH is for use within a single provider's
   operational domain.  This deployment scope is deliberately
   constrained, as explained also in [RFC7665], and limited to a single
   network administrative domain.  In this context, a "domain" is a set
   of network entities within a single administration.  For example, a
   network administrative domain can include a single data center, a
   campus physical network, or an overlay domain using virtual
   connections and tunnels.  A corollary is that a network
   administrative domain has a well defined perimeter.

> And why such disregard
> for privacy and security? In any case, this document has significant
> issues. It does not explaining where exactly the proposed Network
> Service Header would be inserted (MPLS or IPv6?).

This is another good comment, we believe is already addressed in rev -20 with the additions to the Introduction and Figure 1, among others.

> The security
> provisions boil down to lip-service mentions of IPSEC, and that's way
> insufficient given the nature of the protocol.
> 

More below.

> Sometimes you read a document and it gives you a glimpse of an alternate
> universe in which the Internet has been replaced by a mad pile-up of
> middle boxes. Of course, they will not be called that. Instead, we will
> speak of "service functions" providing such essential features as NAT
> and deep packet inspection. I suppose that censorship and surveillance
> are not desirable key words in marketing brochures, and that metadata
> collection and the insertion of adverts are better described in yet to
> be published extensions.
> 

Do these comments target the sfc charter, RFC 7498, or RFC 7665?

> As far as I can tell, the so called "Service Function Chaining
> Architecture" tries to standardize this pile-up of middleboxes, so that
> service providers can procure them from multiple vendors. The draft that
> I am reviewing, draft-ietf-sfc-nsh-18, defines a clear-text "network
> service header" (NSH) that would inserted between the "transport" used
> in the domain and the IPv4 or IPv6 packet itself. Boxes could look at
> this header, add metadata to it, and forward it to other boxes along a
> "service path". I mentioned IPv4 or IPv6, but there are also provisions
> for passing pure NSH information, probably for maintaining the "service
> path", or passing Ethernet, MPLS or experimental frames, because why not.
> 
> The specification itself is appropriately baroque, with the header
> composed of three components and multiple options, but the basic idea is
> to allow different boxes to see the packet so they can read, update and
> process the metadata. As far as I can tell, there are no serious attempt
> to provide any kind of security or privacy protection except maybe some
> basic ingress filtering, which means that the metadata can be observed
> by any entity able to tap the path, or can be spoofed by any system
> inserted on the path, such as for example a virus infected box. The only
> security reference is a short note that deployments "could use IPSEC",
> through a token reference to RFC 6071.

I agree with the fact that draft-ietf-sfc-nsh-18 left much assumed in the Security Considerations section.

One such assumptions is that the security considerations of RFC 7665 apply to this document. In particular, the “SFC Encapsulation” heading of https://tools.ietf.org/html/rfc7665#section-6, which was worked with the Security ADs.

Revision -20 is a first take at correcting that assumption and corresponding omissions:
https://www.ietf.org/rfcdiff?url1=draft-ietf-sfc-nsh-18&url2=draft-ietf-sfc-nsh-20#diff0125

> 
> When first reading the document, I was under the impression that the
> "transport" used in the domain would be some kind of close layer 2
> system, MPLS or maybe Ethernet. That would give some assurance that the
> collection of middle boxes would be part of a somewhat controlled
> domain. It was only the reference to IPSEC that enlightened me. If IPSEC
> is plausible, it probably means that the packet would be composed of an
> IP header, the SFH, and the original IP payload.

It is constrained to a single admin domain not because of the underlying technology of choice (IP vs. MPLS vs. Ethernet), but rather by deployment and applicability considerations.

> So the proposed usage
> implies carrying metadata like subscriber information and identification
> of content in clear text  over long distance paths. What could possibly
> go wrong? And if that was not enough, imagine what an adversary could
> achieve by spoofing the service packets.

No. Although that was carrying over from RFC 7665, we are making it clear and explicit in the document. MPLS or Ethernet or something “close layer 2 system” would not give those assurances. It is an applicability.

> 
> Of course, the same threats would still apply if the protocol was
> strictly carried over layer 2, as several adversaries are quite capable
> of snooping and spoofing layer 2 traffic.
> 
> Coming at the end of this review, I hesitate between two options. One
> would be to just express my disgust, wash my hands, and hope that the
> effort will fail under its own weight, maybe after a couple of
> catastrophic security failures. The other option is to provide some
> advice on how to solve the most blatant issues.

I believe one source of confusion here is the extrapolation of carrying privacy-sensitive metadata across the Internet. That is not the case.

The so-called "RFC 7665" defined the applicability scope, but we the NSH specification has failed to make that clear. This is a good point, and one that needs text changes. We took initial steps in -20 to correct that and be clear and explicit.

> In a spirit of
> cooperation, I choose the latter, and here is the two steps advice:
> 
> The first step to better security and privacy there would be to present
> the practical deployment conditions. I could only make guesses, and
> that's silly. There should be some kind of diagram explaining how the
> SFH is inserted in packets, and where it sits between Ethernet, MPLS and
> IP.
> 
> The next step is to develop a protection model. Given that the goal of
> the architecture is to decorate the packets with sensitive metadata,
> there need to be some thinking about who should be able to see it. How
> would path encryption like IPSEC be deployed? Should all elements on
> path really be able to access all metadata, or should some of it be
> further protected?
> 
> Please fix that before the document is published.
> 

A lot of this conversation is déjà vu of the RFC 7665 WG, LC, and ballot discussions. Those discussions resulted in significant work with the Security ADs, security-focused SFC experts, and SFC-focused security experts; those in turn are codified in the text of RFC 7665’s security considerations.

> -- 
> Christian Huitema

Best,

—
Carlos Pignataro, [email-masked]

“Sometimes I use big words that I do not fully understand, to make myself sound more photosynthesis."

---

From: Christian Huitema <[email-masked]>
Date: Sat, 9 Sep 2017 16:42:02 -0700

> Thanks for taking time to review this document. The net-result of addressing them will be an improved specification.
I am looking at the changes between draft-18 and draft-20, and draft-20
is indeed improved.
> ...

I have no doubt that there are multiple products developed and sold, and
that interoperable specifications are preferable to competing
proprietary specifications. My initial reaction was largely motivated by
the lack of scoping in the document. Without that scoping, it looks like
a general architecture for adding metadata to packets on the Internet,
and that's a big red flag, big enough to state that no, the IETF should
not endorse that at all, interop be damned.

The scoping mitigates some of the concerns. I am still troubled by some
of the specification, such as the example giving a scope as "a campus
physical network". Environments like that are only partially controlled.
Adding metadata to packets in such wide locations could enable many
attacks.

>    administrative domain has a well defined perimeter.

Yes, I saw that. That's good. But I also only see minimal mechanisms in
the draft for enforcing the removal of the metadata before a packet
leaves the specified domain. Section 2 states that "the last SFF in the
service chain removes the NSH". That's fine, but that's not a fail-safe
mechanism. The draft mentions using IPv4 or IPv6 as transport. It seems
that in that case there should be some ingress/egress filtering, as in
"packets originating outside the service domain must be dropped if they
contain an NSH," and similarly must be drop on domain exit if they
contain an NSH.

> This is another good comment, we believe is already addressed in rev -20 with the additions to the Introduction and Figure 1, among others.

Figure 1 does help, thank you. I assume that the intent is to complete
the NSH draft with specifications of the actual encapsulations over
MPLS, Ethernet, IPv4 and IPv6. But it is pretty hard to review the
security of the system without a description of these encapsulations.

>> ...
>> ...
Yes, the new introduction is much better.
> A lot of this conversation is déjà vu of the RFC 7665 WG, LC, and ballot discussions. Those discussions resulted in significant work with the Security ADs, security-focused SFC experts, and SFC-foc…
The new security section does provide a number of recommendations, such
as the obfuscation of metadata. That's definitely an improvement. But I
believe there are still issues.

The first issue is that "Metadata privacy and security considerations
are a matter for the documents that define metadata format." That does
not give me a warm and fuzzy feeling at all. I understand that the
formats will be only registered "after IETF review", but these future
reviews would be much easier if the NSH mechanism defined at least a
baseline security posture, and maybe some generic mechanisms for
obfuscation or encryption.

The second issue is that the security section provide recommendations
about solutions, but does not analyze the threats. In particular, one of
the threats that I find worrisome is, what happens if a specific
function in a service chain gets subverted? In the current version, it
seems that subverting just one element in the chain will provide access
to all the metadata.

I may be paranoid, but there is already an history of adversaries
attacking complex systems like data centers, network control systems or
corporate networks, not to mention campus networks. These adversaries
typically proceed by lateral movement after an initial penetration until
they get closer to their actual target inside the domain. I can see an
adversary trying to penetrate one of these domains in order to access
the metadata. In our case, it would try to find a weak link in the
service function chain. It maybe that one of the functions is deemed
benign, and thus was less secured than the others. But if all functions
see the metadata, then the adversaries will achieve their goal by
targeting that weak link. Some application of the "least privilege"
principle would be useful there.

-- Christian Huitema

-- 
Christian Huitema

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 14 Sep 2017 11:22:21 +0000

Christian,

Thank you for the dialogue — please see inline.

> On Sep 9, 2017, at 7:42 PM, Christian Huitema <[email-masked]> wrote:
> 
> 
> 

This is great to hear. Thank you for your comments that, along with others', resulted in incorporating needed clarifications and a more precisely-scoped specification.

> the lack of scoping in the document.

it’s not uncommon to forget how broadly understood assumptions and design criteria for a document are, when being knee-deep in work. This is probably the case here, in which scope should have been more explicit.

For completeness, the document is the document plus its context. A peek at the normatively-cited RFC 7665, SFC architecture, (or at the charter as part of a Directorate review) would have proactively reduced your concerns, leaving the editorial need to make it very clear and explicit in “the document” (as opposed to your initial reaction).

That said, we arrived at the same place after just one round-trip, and rev -19 plus rev -20 successively make that more clear and less ambiguous.

The doc is improved, making those assumptions explicit in the text.

> Without that scoping, it looks like

OK, we can qualify the campus network with “controlled”, or frankly remove it or choose a different example. It is, after all, an example.

> mechanism.

The I-D mentions in several places beyond Section 2 the requirement to remove the NSH on SFP exit. The Security Considerations currently includes:

 Means to prevent
 leaking privacy-related information outside an administrative domain
 are natively supported by the NSH given that the last SFF of a
 service path will systematically remove the NSH encapsulation before
 forwarding a packet exiting the service path.

How would you suggest this can be strengthened? We do add some relevant text based on the next comment.

> The draft mentions using IPv4 or IPv6 as transport. It seems

This is a good suggestion. We can add some clarifying text after the first paragraph of the Security Considerations.
This can take care of your previous comment as well.

> security of the system without a description of these encapsulations.

The specifics of those are expected in separate documents, as needed (for MPLS, or UDP, or VXLAN-GPE, etc).

> Yes, the new introduction is much better.

Good to see. Thank you.

> obfuscation or encryption.

I do not know if future reviews might or might not be easier, since there would be a need for a reviewer to follow and read normative references, which practice shows not always happens... But, that aside, ease of future review for reviewers is not a design principle for NSH :-)

That said, I agree that there is an opportunity to, without specifying, provide forward-looking guidance or references to potential work. We will add that in.

> function in a service chain gets subverted?

If a firewall or a router gets subverted, we likely have bigger problems. More below.

> In the current version, it

Not really. NSH replies on a existing encryption solutions for both confidentiality and integrity protection.  As per the draft, if an operator deems additional protection needed, then they utilize time proven, standard protocols in conjunction with NSH.  Not only is this consistent layering-wise, but also ensure that the crypto solution is one that is well vetted and understood.

If you are directly talking about metadata-level encryption, I’ll let Joel (as shepherd and chair) comment since that was a topic discussed in the WG.

That said, again, there is likely an opportunity to polish a bit. Will try.

> principle would be useful there.

See above. Not all functions see the metadata, if so desired. For example, all SFFs do not see any metadata if the transport uses existing proven encryption techniques, IPsec, TLS, etc.

Thanks again for the review and comments.

Best,

—

“Sometimes I use big words that I do not fully understand, to make myself sound more photosynthesis.”

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Thu, 14 Sep 2017 11:22:20 +0000

Christian,

We worked on a revision -21 attempting to address your follow-ups and remaining concerns.

In an effort to improve the review efficiency, we are sending the Diffs (HTML side-by-side and wdiffs) of that -21 candidate to a reduced audience.

We would absolutely appreciate all feedback, particularly concrete (read: actual textual recommendations) input whenever possible.

And of course we will get back to the larger lists after converging and posting a new revision.

Let us know — Thanks!

---

From: Christian Huitema <[email-masked]>
Date: Sat, 16 Sep 2017 21:21:12 -0700

> For completeness, the document is the document plus its context. A peek at the normatively-cited RFC 7665, SFC architecture, (or at the charter as part of a Directorate review) would have proactive…
I did read RFC 7665. Before writing the initial review, and again now as
a refresher. Yes, RFC 7665 mentions an "administrative domain". But it
is also a very abstract document, leaving open many possibilities. And
it does contain the text about sharing metadata along a function path
that gave me pause: "For example, an external repository might provide
user/subscriber information to a service chain classifier. This
classifier could in turn impose that information in the SFC
encapsulation for delivery to the requisite SFs. The SFs could in turn
utilize the user/subscriber information for local policy decisions.
Metadata can also share SF output along the SFP." The draft translates
the architecture into practical protocol specification. It is good to
include at that stage concrete protections.
> OK, we can qualify the campus network with “controlled”, or frankly remove it or choose a different example. It is, after all, an example.
I saw the "controlled" addition in your text, but I don't quite
understand what a "controlled campus network" is. I understand that one
of the goals is to place a variety of functions in separate boxes
independently of topology, but if I was managing a campus network I
would much rather place these functions in controlled spaces like data
centers rather than, say, student dorms. Not all places on the campus
will be tightly controlled. It might be simpler to just not mention
these campus networks.
...
> This can take care of your previous comment as well.
I saw the text added in the latest draft. That's fine.

> That said, I agree that there is an opportunity to, without specifying, provide forward-looking guidance or references to potential work. We will add that in.

I see that you added references to "proof of transit" in draft-21. That,
and the reference to obfuscating subscriber info, certainly helps. It
seems that the security protection is based on three broad principles:

1) Encrypt the data in transit, using IPSEC or similar;

2) Obfuscate by default critical metadata such as subscriber info;

3) Encrypt some of the metadata.

That's not a bad posture, but I wish that you were explicit about the
threats. I am somewhat concerned that the "administrative domain"
approach leads to complacency, as in "my domain is secure, I am only
concerned with external leakage". I think it would be good to point at
explicit threats. Encrypt in transit addresses one type of threats,
adversaries tapping conduits to observe the data. But how about hacking
into an SFF? Or providing a "free" function that pays for itself by
collecting the meta-data? These all go into the idea that in a complex
system, it is good to compartment who routinely sees what information.
That's the rational motivating obfuscation and other such techniques,
and it would be nice to be explicit about it.

> If a firewall or a router gets subverted, we likely have bigger problems. More below.
Maybe. Maybe not. What if an intrusion detection system gets subverted?
What if an accounting system gets subverted? You hope that the intrusion
will be detected shortly, but principles like "least privilege" are a
good way to provide defense in depth and minimize the damage during the
early stages of the intrusion.

> See above. Not all functions see the metadata, if so desired. For example, all SFFs do not see any metadata if the transport uses existing proven encryption techniques, IPsec, TLS, etc.

Yes. Somehow stating that and explaining why it matters would be nice.

-- Christian Huitema

---

From: Randy Bush <[email-masked]>
Date: Sun, 17 Sep 2017 16:50:57 +0900

> 1) Encrypt the data in transit, using IPSEC or similar;

like this is gonna happen.  this is the secdir pacifier.

> concerned with external leakage".

there is common talk about pushing some services in a chain to the
cloud, aka other people's computers.

randy

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 17 Sep 2017 08:04:13 -0400

IF removing the two references to Campus networks would help, we can do 
that.  deployment in a campus (sa distinct from the DC within campus) is 
indeed more complex.

With regard to threats, a lot of the reason I have trouble with 
providing good answers is that a service function that stores the 
metadata for later analysis is perfectly legitimate.  If the operator 
contract with the SF provider is that the provider gets the metadata the 
service function sees, that the operators chaice.  Given that SFs have 
to be able to get at whatever metadata they need, the protocol can not 
inherently prevent that. People have suggested in Internet-Drafts 
mechanisms whereby the SFF trim and restore the metadata sets.  that 
does not actually rpevent this, it merely adds the complication that the 
operator has to configure what is allowed where.  If their contract for 
a given SF says that it gets all the metadata, it will get all the metadata.
The one piece we can do is that personally identifying metadata can be 
obfuscated, preferably by indirection.  We should however recognize that 
the indirection will be known to the operator, and if they choose to 
reveal it by other means we can not prevent that.

Yours,
Joel