Subject: Re: [spring] Benjamin Kaduk's Discuss on draft-ietf-spring-srv6-network-programming-20: (with DISCUSS and COMMENT)

From: Benjamin Kaduk <[email-masked]>
Date: Wed, 9 Dec 2020 09:04:13 -0800

Hi Pablo,

Thanks for the updates through the -26; I'll be posting an updated ballot
position shortly (with one late-breaking but hopefully trivial to resolve
new discuss point, plus more comments).

A few notes inline as well...

On Thu, Nov 26, 2020 at 04:50:36PM +0000, Pablo Camarillo (pcamaril) wrote:
> Hi Ben,
> 
> Please see inline with PC3.
> Note that we've posted rev26: https://www.ietf.org/rfcdiff?url2=draft-ietf-spring-srv6-network-programming-26
> 
> Many thanks for your time,
> Pablo.
> 
> -----Original Message-----
> From: Benjamin Kaduk <[email-masked]> 
> Sent: jueves, 26 de noviembre de 2020 1:12
> To: Pablo Camarillo (pcamaril) <[email-masked]>
> Cc: The IESG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; Bruno Decraene <[email-masked]>; Joel Halpern <[email-masked]>
> Subject: Re: Benjamin Kaduk's Discuss on draft-ietf-spring-srv6-network-programming-20: (with DISCUSS and COMMENT)
> 
> Hi Pablo,
> 
> Thanks for the updates; the changes in the -25 all look good to me.
> 
> More inline...
> 
> On Wed, Nov 25, 2020 at 09:42:42PM +0000, Pablo Camarillo (pcamaril) wrote:
> > Hi Ben,
> > 
> > Thanks for your careful review. Replies to your comments inline with PC2.
> > Note that I’ve posted rev25 of the draft.
> > 
> > Thanks,
> > Pablo.
> > 
> > -----Original Message-----
> > From: Benjamin Kaduk <[email-masked]>
> > Sent: miércoles, 25 de noviembre de 2020 2:13
> > To: Pablo Camarillo (pcamaril) <[email-masked]>
> > Cc: The IESG <[email-masked]>; 
> > [email-masked]; 
> > [email-masked]; [email-masked]; Bruno Decraene 
> > <[email-masked]>; Joel Halpern <[email-masked]>
> > Subject: Re: Benjamin Kaduk's Discuss on 
> > draft-ietf-spring-srv6-network-programming-20: (with DISCUSS and 
> > COMMENT)
> > 
> > Hi Pablo,
> > 
> > I must apologize twice over: first for the delay in responding, and second for the number of points in my discuss ballot that are not actual problems.
> > I recently had the time to go over the document again, with benefit of your responses here, and now have a better understanding of some aspects that were confusing to me during my first reading.  I still have a few things I want to take another look at, but I can respond to your comments now -- the good news is that these discuss points all look to be resolved.
> > 
> > More inline...
> > 
> > On Fri, Sep 25, 2020 at 06:31:52PM +0000, Pablo Camarillo (pcamaril) wrote:
> > > Hi Benjamin,
> > >
> > > Thank you for your time and review. Please see inline with [PC].
> > >
> > > Regards,
> > > Pablo.
> > >
> > > -----Original Message-----
> > > From: Benjamin Kaduk via Datatracker <[email-masked]>
> > > Sent: jueves, 24 de septiembre de 2020 9:52
> > > To: The IESG <[email-masked]>
> > > Cc: [email-masked];
> > > [email-masked]; [email-masked]; Bruno Decraene 
> > > <[email-masked]>; Joel Halpern <[email-masked]>; 
> > > [email-masked]
> > > Subject: Benjamin Kaduk's Discuss on
> > > draft-ietf-spring-srv6-network-programming-20: (with DISCUSS and
> > > COMMENT)
> > >
> > > Benjamin Kaduk has entered the following ballot position for
> > > draft-ietf-spring-srv6-network-programming-20: Discuss
> > >
> > > When responding, please keep the subject line intact and reply to 
> > > all email addresses included in the To and CC lines. (Feel free to 
> > > cut this introductory paragraph, however.)
> > >
> > >
> > > Please refer to
> > > https://www.ietf.org/iesg/statement/discuss-criteria.html
> > > for more information about IESG DISCUSS and COMMENT positions.
> > >
> > >
> > > The document, along with other ballot positions, can be found here:
> > > https://datatracker.ietf.org/doc/draft-ietf-spring-srv6-network-prog
> > > ra
> > > mming/
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > --
> > > DISCUSS:
> > > --------------------------------------------------------------------
> > > --
> > >
> > > [edited to remove nonsensical point that had crept in about the SRH 
> > > being a new extension header; it's "just" a routing header]
> > >
> > > The current requirement that IANA-assigned SRv6 Endpoint Behavior codepoints are used, with no range reserved for local assignment, seems to be inviting codepoint squatting from the nominally reserved range.
> > > Why is there an absolute requirement for registration (even FCFS) 
> > > without ranges for local or experimental use?  (What are the various reserved ranges reserved for?) [PC] I agree with you that a range for local use would come useful. Hence, we’ve carved 2k entries of the registry for Private Use.
> > > The Reserved range (half of the 65k registry) is for future allocation by IETF. Future RFCs might decide how to use that.
> > 
> > Sounds good; thanks!
> > 
> > > The (normative) pseudocode does not seem to handle the case when the SRH is omitted for the degenerate case where there is only a single segment, or for the PSP flavor.
> > > [PC] Each one of the pseudocodes provides the SRH processing and the Upper-Layer processing. If there is no SRH, then the SRH processing is not executed, but the Upper-layer Header processing is. This is the same as in RFC8754.
> > 
> > To expand a bit more on how I was confused here, I was looking particularly at the End, End.X, and End.T cases (though I guess it also applies to the Encaps cases).  In the normal flavor, upper-layer processing gets shunted over to section 4.1.1 that doesn't really do much -- in particular, there is no interconnect for End.X and no table lookup for End.T!  But this really is the right behavior, since the SRH is only absent when this SID is the final destination of the packet, and in that case we really must act as an endpoint and not a router.  If we did send the packet onward it would most likely just get looped back to us until the hop limit expires (or get some other error), so generating a local ICMP error is the right thing to do in the general case, unless there is local configuration for that upper-layer protocol.
> > 
> > [PC2] Ack
> > 
> > > The pseudocode for the PSP and USP procedures seem incorrect -- Hdr Ext Len is measured in units of 8 octets, and does not include the first 8 octets of the extension header, but Payload Length is measured in octets.  Literally decreasing the Payload Length by the Hdr Ext Len value will produce a malformed IPv6 packet.
> > > [PC] Indeed. Fixed.
> > >
> > > If "PSP operation is deterministically controlled by the SR Source Node", why do we need to define behavior codepoints that (for example) use both PSP and USP?  I don't see how there is full determinism in this case while being different from the "PSP only" flavor.
> > > [PC] In order for the PSP behavior to be executed there must happen two things. First, the SR Source Node must instruct it wants to use the PSP behavior. This is done by using an SRv6 SID associated with a behavior that support the PSP flavor (either End with PSP alone, or End with PSP in combination of other). Then, the second thing is that the received packet at the node it must have a SL value of 1, that during processing is decremented to 0. (In other words, it must be the penultimate segment). If both conditions are given, then lines S14.2-S14.4 of the pseudocode are executed.
> > 
> > I understand now; thanks for explaining it again.  (To reiterate: any 
> > given SID can appear "anywhere" in the segment list, so USP is 
> > relevant when it is last and PSP is relevant when it appears 
> > second-to-last.  The USP/PSP flavor doesn't need to be the same for 
> > all SIDs in the path, so marking a single SID with both can be 
> > relevant for different paths.)
> > 
> > [PC2] Correct
> > 
> > > There are numerous factual errors and un/under-specified protocol behavior (see COMMENT), including: how to set the outer Hop Limit (multiple instances), the order of segments in the SRH, specification of headend behavior by reference to informal example, L2 frame en/decapsulation procedures, and the "Opaque" note for endpoint behavior 65535.
> > > [PC] I’ll comment on each point in the comment.
> > >
> > >
> > > --------------------------------------------------------------------
> > > --
> > > COMMENT:
> > > --------------------------------------------------------------------
> > > --
> > >
> > > I note that this document references 
> > > draft-filsfils-spring-srv6-net-pgm-illustration, which uses the IPv4 CIDR 20/8 in examples instead of addresses from the RFC 5737 range.  It would be disappointing to have a PS refer to a draft that squats on the IPv4 namespace in such a manner.
> > > [PC] Ack. I’ve updated draft-filsfils-spring-srv6-net-pgm-illustration to correct that.
> > 
> > Thank you!
> > 
> > > Abstract, Introduction
> > >
> > >    This document defines the SRv6 Network Programming concept and
> > >    specifies the base set of SRv6 behaviors that enables the creation of
> > >    interoperable overlays with underlay optimization (Service Level
> > >    Agreements).
> > >
> > > I don't understand what the parenthetical is intending to convey.
> > > [PC] I’ve removed the parenthesis. Its unneeded.
> > >
> > > Section 2
> > >
> > >    The following terms used within this document are defined in
> > >    [RFC8402]: Segment Routing, SR Domain, Segment ID (SID), SRv6, SRv6
> > >    SID, SR Policy, Prefix SID, and Adjacency SID.
> > >
> > > [RFC 8402 spells it Prefix-SID (with hyphen) and Adj-SID.] [PC] 
> > > Fixed in rev21.
> > >
> > > Section 3
> > >
> > > The text allegedly reproduced from RFC 8754 §4.3 differs in case and hyphenation.
> > > [PC] Fixed in rev21.
> > >
> > > Section 3.1
> > >
> > >    This document defines an SRv6 SID as consisting of LOC:FUNCT:ARG,
> > >    where a locator (LOC) is encoded in the L most significant bits of
> > >    the SID, followed by F bits of function (FUNCT) and A bits of
> > >    arguments (ARG).  L, the locator length, is flexible, and an operator
> > >    is free to use the locator length of their choice.  F and A may be
> > >    any value as long as L+F+A <= 128.  When L+F+A is less than 128 then
> > >    the remaining bits of the SID MUST be zero.
> > >
> > > Why does A have to be a fixed value; can't it safely be a function of F (even if we exclude things like huffman coding for F)?
> > > [PC] Can you please re-state your point? The text above does not suggest that A is a fixed value, hence I’m not sure I fully understand what you mean.
> > 
> > It's true that there is no specific statement "A is a constant across 
> > <some-particular-scope>".  However, there is also not a specific 
> > statement "the value of A can depend on <some-particular-scope>", so 
> > the reader has to make an interpretation.  We do say that L is 
> > flexible and the operator can choose that length; I interpreted this 
> > to mean that the operator makes the choice once, and that L is fixed 
> > for the entire SR domain.  With that in my head, it was also natural 
> > to infer that F and A are also fixed for the entire SR domain.  (Is L 
> > allowed to vary even within the domain?)
> > 
> > [PC2]  Note that in RFC8754 we have “When an SRv6-capable node receives an IPv6 packet, it performs longest-prefix-match lookup on the packet’s destination address.”. Hence  things work even if two nodes use a different L  for their locators or different sizes of F and A for their SIDs.
> 
> I guess so.  (In my head the ability to use longest-prefix was useful for when the SID and non-SID addresses could be consolidated into a single prefix applicable to that node, but I have no particular reason to have been thinking that.)
> 
> If I might make a suggestion, writing "free to use the locator length of their choice for any given SID" would have forestalled my confusion in this regard, and seems like a minor change to make.  But it is your decision as to whether or not to make any changes here; I am satisfied just with the explanation above.
> 
> [PC3] Ack 
> 
> > If we do want to allow F and A to vary at a (e.g.) per-LOC scope, we should probably make some statement about being able to decode them properly on receipt; this could be done in many ways, including a strict lookup table and a self-deliniating length-encoding scheme, but there cannot be ambiguity about how to partition a given IP address into LOC:FUNCT:ARG.
> > 
> > [PC2] Please refer to previous comments and also the text in Sec 3. The SR Endpoint Node performs LPM lookup to match the FIB entry corresponding to SIDs that it has instantiated in order to execute the instructions specific to its behavior. The packet with the SID in the DA is routed through transit nodes based on LPM lookup that happens on the locator parts that are advertised via routing/control plane as described in Sec 3.3. The transit nodes do not have to decode or program SIDs that have been instantiated by other SR Endpoint nodes in the domain.
> 
> I agree that the transit nodes don't need to know anything about this.
> The node that instantiates the SID, however, does.  As a totally contrived example, suppose that I have a prefix of abcd:abcd:: as my LOC and I want to use FUNCT of 0x1234 to indicate behavior codepoint 1, FUNCT of
> 0x12351235 to indicate behavior codepoint 2, and FUNCT of 0x12351236 to indicate behavior codepoint 3.  I basically have to store that in a table, and can't write some logic that looks at the 16 bits after the LOC.  If I instead used 0x1234, 0x22351235, and 0x22351236, then the first four bits after LOC could be the length of FUNCT encoding in 2-byte increments, and I could have some "simple" code to unpack the self-delineating encoding.
> 
> [PC3] Lets remind that RFC8754 explicitly states "it performs a longest-prefix-match lookup". Anything after that is a local implementation decision by the segment endpoint that instantiates the SID and is outside the scope of this document. 
> 
> In short: this topic only considers how a given node assigns its own LOC:FUNCT:ARG values, and may be self-evident to most readers, but it can still be helpful to note as guidance on how to make those allocations.
> 
> > >    An SRv6 Segment Endpoint Behavior may require additional information
> > >    for its processing (e.g. related to the flow or service).  This
> > >    information may be encoded in the ARG bits of the SID.
> > >
> > > (Assuming that it can be sufficiently compactly encoded?) [PC] 
> > > Correct
> > >
> > >    The ARG value of a routed SID SHOULD remain constant among packets in
> > >    a given flow.  Varying ARG values among packets in a flow may 
> > > result
> > >
> > > (Thus limiting the type of information that can be represented in 
> > > the
> > > ARG?)
> > > [PC] This text provides a SHOULD, and a reason for it. If other behaviors defined in future documents would like to have ARG value varying for packets belonging to the same flow, it is important that they take this into consideration.
> > >
> > > Section 3.2
> > >
> > >    SIDs.  The provider historically deployed IPv6 and assigned
> > >    infrastructure addresses from a portion of the fc00::/7 prefix.  They
> > >    further subdivided the prefix into three /48 prefixes (Country X,
> > >    Country Y, Country Z) to support their SRv6 infrastructure.  From
> > >    those /48 prefixes each router is assigned a /64 prefix from which
> > >    all SIDs of that router are allocated.
> > >
> > > This is not the language I would have expected for use of the ULA address range.  It looks like this may be related to Erik Kline's Discuss point...
> > > [PC] Indeed related. Erik Kline has suggested text. Ill use Erik’s text as provided.
> > >
> > >    IPv6 address consumption in both these examples is minimal,
> > >    representing one billionth and one millionth of the assigned address
> > >    space, respectively.
> > >
> > > I'm not sure I understand the calculation that produced these values.
> > > [PC] If the operator got assigned a /20 by the RIR, then they have 2^28 /48 prefixes available. Thus, if they are only using a few /48 prefixes, then they are using less than a millionth of the available /48 prefixes.
> > 
> > Apparently I didn't take notes on my initial calculations that left me confused.  Assuming I have it right this time...
> > 
> > # Use 3 /48s in the whole ULA /40
> > >>> 3./2**40
> > 2.7284841053187847e-12
> > # Use "a few" 3 /48s from the assigned /20
> > >>> 3./2**28
> > 1.1175870895385742e-08
> > 
> > This seems to be well less than a billionth and a millionth, respectively, so I'm not sure if I am doing the math as intended.
> > 
> > [PC2] I believe your maths are right.
> > If an operator is allocated a /20, they can allocate 2**28  /48 prefixes.
> > 2**28 = 268435456.
> > Thus, If SRv6 requires 3 of such /48 prefixes, then we are using only 1.12e-8 of that space, which is less than a millionth (e-6).
> > No?
> > In other words: if we do a million of deployments of SRv6, there would 
> > be space to fit them all within that /20. (and actually there is 
> > slightly more)
> 
> I would suggest adding "less than" in the text, then.
> 
> [PC3] Got it. Updated.
>  
> > >    An SR Source Node cannot infer the behavior by examination of the
> > >    FUNCT value of a SID.
> > >
> > >    Therefore, the SRv6 Endpoint Behavior codepoint is advertised along
> > >    with the SID in the control plane.
> > >
> > > These two sentences feel a little out of place in this spot, occurring after previous discussion that the endpoint behavior codepoint is advertised in the control plane.
> > > [PC] Other ADs have expressed the value of precisely those two sentences.
> > 
> > I agree that these sentences are valuable.  I was proposing to move them two paragraphs earlier, so we transition directly from "advertises the SID ... in the control plane" to (paraphrasing) "cannot infer behavior by examining FUNCT" and "behavior codepoint is advertised along with SID".
> > 
> > [PC2] Misunderstood your original point. I moved those two up as you suggest.
> > 
> > >    o  At Router 3, within the locator 2001:db8:bbbb:3::/64, network
> > >       operator or the router performs dynamic assignment for:
> > >
> > > nit: missing article ("the network operator").
> > > [PC] Ack. Fixed in rev21.
> > >
> > >       *  Function 100 associated with the behavior End.X (Endpoint with
> > >          cross-connect) between router 3 and its connected neighbor
> > >          router, for example Router 4.  This function is encoded as
> > >          16-bit value and has no arguments.
> > >
> > > Please clarify that "function 100" is a hex literal.
> > > [PC] Ack. Fixed in rev21.
> > >
> > > I also suggest noting clearly at the start of the example that F is 16 and A is 0.
> > > [PC] Ack. Added for each SID in rev21.
> > >
> > >       *  Function 101 associated with the behavior End.X (Endpoint with
> > >          cross-connect) between router 3 and its connected neighbor
> > >          router, for example Router 2.  This function is encoded as
> > >          16-bit value and has no arguments.
> > >
> > > F is a constant for the SR domain; saying it at each SID assignment is misleading.
> > > [PC] Function is the bits in the SID instantiated by the router. Can you please explain what text gives the impression that F is a constant?
> > 
> > FUNCT is instantiated by the router, yes, and it is F bits long.  This ties in to my previous comment about L, F, and A (maybe) being constants across the SR domain, or clarifying at which scope they can vary.
> > 
> > [PC2] Indeed, see previous answer.
> > 
> > > [same note about hex for 101]
> > > [PC] Ack. Added for each SID in rev21.
> > >
> > > Section 4
> > >
> > >    The list is not exhaustive.  In practice, any function can be
> > >    attached to a local SID: e.g. a node N can bind a SID to a local VM
> > >    or container which can apply any complex processing on the packet.
> > >
> > > Yes, but it can't advertise the behavior corresponding to that function without a codepoint.
> > > [PC] This is intended to be an introductory paragraph to the behaviors. But you are absolutely right: for any new behavior you need to write a new I-D (as its currently happening in other SPRING WG docs), and then -once RFCed- get an SRv6 Endpoint Behavior codepoint for it.
> > 
> > Maybe we could add at the end ", provided there is a behavior 
> > codepoint allocated for that processing" or some similar edit?  (We 
> > don't have to; the decision is up to you.)
> > 
> > [PC2] I'm fine with it. Added as suggested.
> > 
> > > Section 4.1.1
> > >
> > >    When processing the Upper-layer Header of a packet matching a FIB
> > >    entry locally instantiated as an SRv6 End SID, if Upper-layer Header
> > >    processing is allowed by local configuration (e.g.  ICMPv6), then
> > >
> > > nit: the parenthetical is placed so as to apply to "local configuration", which doesn't really make sense; I presume the intent is to say that upper-layer header processing is allowed *for that upper-layer protocol*.  Also, comma after "e.g.".
> > > [PC] Agreed, but see next point.
> > >
> > >    problem message to the Source Address and discard the packet.  Error
> > >    code 4 (SR Upper-layer Header Error) and Pointer set to the offset of
> > >    the upper-layer header.
> > >
> > > nit: this is not a complete sentence.
> > > [PC] We have been asked to change the entire 4.1.1 to a pseudocode. Please review the pseudocode in revision 21 and let me know your feedback.
> > 
> > Yes, this pseudocode looks much better -- thank you!
> > 
> > That said, I am not sure that I understand why it is only RECOMMENDED 
> > (not
> > REQUIRED) to only allow upper-layer processing when that does not result in the packet being forwarded.  Are there cases when this would be useful and not result in a packet loop?
> > 
> > [PC2] We believe that RECOMMENDATION is strong enough for this scenario. I am not sure that we can say that packets forwarded after local processing will always result in a loop e.g. if there is an inner encapsulated IP packet that needs to get forwarded further.
> 
> Okay.
> 
> > > Section 4.2
> > >
> > >    It is the SRv6 instantiation of an Adjacency-SID [RFC8402] and it is
> > >    required to express any traffic-engineering policy.
> > >
> > > This text reads as if End.X specifically is required for TE, which does not seem correct to me.
> > > [PC] Fixed.
> > > OLD: and it is required to express any traffic-engineering policy.
> > > NEW: and its main use is for traffic-engineering policies.
> > >
> > > Section 4.4, 4.5
> > >
> > >                                         This is equivalent to the per-CE
> > >    VPN label in MPLS [RFC4364].
> > >
> > > The phrase "per-CE VPN" does not appear in RFC 4364.
> > > [PC] Added to the terminology section.
> > >
> > > Section 4.5
> > >
> > >    S05.  If the End.DX4 SID is bound to an array of L3 adjacencies, then
> > >    one entry of the array is selected based on the hash of the packet's
> > >    header Section 7.
> > >
> > > nit: I suggest "(see Section 7)" to match section 4.4.
> > > [PC] Indeed. Fixed.
> > >
> > > Section 4.6, 4.7
> > >
> > >    is required.  This is equivalent to the per-VRF VPN label in MPLS
> > >    [RFC4364].
> > >
> > > The phrase "per-VRF VPN" does not appear in RFC 4364.
> > > (A similar comment applies to Section 4.8 as well.) [PC] Added to 
> > > the terminology section of the document.
> > >
> > >    Note that an End.DT6 may be defined for the main IPv6 table in which
> > >    case and End.DT6 supports the equivalent of an IPv6inIPv6
> > >
> > > nit: s/and/an/
> > > [PC] Thanks. Fixed.
> > >
> > > Section 4.7
> > >
> > >    The "Endpoint with decapsulation and specific IPv4 table lookup"
> > >    behavior (End.DT4 for short) is a variant of the End behavior.
> > >
> > > nit: variant of End.T, just like End.DT6, no?
> > > [PC] Correct, fixed.
> > >
> > > Section 4.9
> > >
> > >    S04.  An End.DX2 behavior could be customized to expect a specific
> > >    IEEE header (e.g.  VLAN tag) and rewrite the egress IEEE header
> > >    before forwarding on the outgoing interface.
> > >
> > > Would this customized behavior require a new codepoint?
> > > Similarly for End.DX2V (Section 4.10) [PC] No, that customization is 
> > > local config to the node.
> > 
> > That is surprising to me (it seems like an important part of the behavior that should be signalled explicitly), but I am not familiar enough with this domain in order to argue about it any further.
> > 
> > [PC2] In MPLS this is local config. While someone might think about signaling it in the future -intuitively makes sense-, in today’s use-case it isn’t.
> 
> Thanks for the pointer about MPLS, that helps.
> And I agree that it seems straightforward to add signaling in the future if needed.
> 
> > > Section 4.10
> > >
> > >    S04. Remove the outer IPv6 Header with all its extension headers,
> > >            lookup the exposed VLANs in L2 table T, and forward
> > >            via the matched table entry.
> > >
> > > Just to check my understanding: the "exposed VLANs" (plural?!) are in the Ethernet header of the packet that we are left with after removing the outer IPv6 header, right?
> > > [PC] Correct. And multiple tags in case of Q-in-Q.
> > >
> > >    S01.  IANA has allocated the Internet Protocol number 143 to Ethernet
> > >    (see Section 10.1).
> > >
> > > This seems like a copy/paste leftover from End.DX2 -- we don't 
> > > mention
> > > 143 in this section.
> > > [PC] Indeed. I added by mistake on rev20 to address an AD comment and I have just removed in rev21 again.
> > >
> > > Section 4.12
> > >
> > >    Two of the applications of the End.DT2M behavior are the EVPN
> > >    Bridging of broadcast, unknown and multicast (BUM) traffic with
> > >    Ethernet Segment Identifier (ESI) filtering and the EVPN ETREE use-
> > >    cases.
> > >
> > > Are there references for either/both of these?
> > > [PC] Added in rev21.
> > 
> > Thanks!
> > 
> > >    S04. Remove the IPv6 header and all its extension headers
> > >
> > > nit: the rest of the document uses "Remove the outer IPv6 header with all its extension headers".
> > > [PC] Fixed. Thanks.
> > >
> > >    S06. Forward via all L2 OIFs excluding the one specified in 
> > > Arg.FE2
> > >
> > > nit: s/one/ones/
> > > [PC] Fixed. Thanks.
> > >
> > >    Arg.FE2 is encoded in the SID as an (k*x)-bit value.  These bits
> > >    represent a list of up to k OIFs, each identified with an x-bit
> > >    value.  Values k and x are defined on a per End.DT2M SID basis.  The
> > >    interface identifier 0 indicates an empty entry in the interface
> > >    list.
> > >
> > > (editorial) rather than have to come up with the concept of an "empty entry" in the interface list, I would probably just say that the identifier 0 is reserved and is ignored when doing the exclusion of OIFs.
> > >
> > > I also think we need to say that the order and identifier assignment 
> > > of elements of the interface list (that is, what the x-bit values 
> > > index
> > > into) is under the local control of N, but has to remain stable as 
> > > long as SIDs are advertised that list members of the list in ARG.  (There is otherwise a great deal of freedom in assignment since the values are only consumed by the node that advertises them.) [PC] This text is updated as part of Alvaro’s comment. Can you please check the current text in rev21?
> > 
> > I think the new text is doing the right thing -- the structure of the value is entirely local to the SR Endpoint advertising it.  I will have an editorial comment in my updated ballot position about the "signaling of the argument to other nodes..." clause, though.
> > 
> > > Section 4.13
> > >
> > > It's probably worth calling out the MTU considerations of pushing 
> > > the new IPv6 header+SRH and tunneling the packets.  (I know we 
> > > reference
> > > 2473 already, but the MTU is pretty important.) [PC] MTU 
> > > considerations are covered in RFC8754.
> > >
> > >    S17.  The Payload Length, Traffic Class and Next-Header fields are
> > >    set as per [RFC2473].  The Flow Label is computed as per [RFC6437].
> > >
> > > How is the Hop Limit set for the outer header?
> > > (I assume §6.3 of RFC 2473, but given that you provide a reference 
> > > for all the other fields, it seems like an omission to not mention Hop Limit as well.) [PC] Sure, added.
> > 
> > [Thanks. I'll note some similar reference nits regarding §5.1 in my 
> > updated comments]
> > 
> > > Section 4.14
> > >
> > >    The SRH MAY be omitted when the SRv6 Policy only contains one segment
> > >    and there is no need to use any flag, tag or TLV.
> > >
> > > If this is the case does it matter if I use End.B6.Encaps or End.B6.Encaps.Red?
> > > [PC] If you only have a single segment, then it does not matter. However when you have more than one segment the change is significant.
> > >
> > > Section 4.16
> > >
> > > I don't think I understand what it would mean for a specific SID to support the multiple flavors "in combinations".
> > > [PC] Convoluted sentence indeed.
> > > <OLD>
> > > For each of these
> > >    behaviors these flavors MAY be supported for a SID either
> > >    individually or in combinations.
> > > </OLD>
> > > <NEW>
> > > The End, End.X or End.T behaviors could support these flavors either individually or in combination.
> > > </NEW>
> > 
> > Much better; thanks!  It may be less relevant now that we are 
> > assigning behavior codepoints for each flavor combination, but IMO the 
> > USD flavor fits more naturally into being "part of the main behavior".  
> > E.g., we have
> > End.DT46 already, and I'm not sure how that (behavior codepoint 20) differs from End.T with USD (behavior codepoint 36).  I think it's generally a bad idea to have multiple ways of expressing the same behavior.
> > 
> > [PC2] End.T with USD and End.DT46 are the same only when they are the last SID of the SID list. Note that when they are used as intermediate segment they do different things. If End.DT46 is used as an intermediate segment in the sid list it will result in an ICMP Parameter Problem. If End.T is used as an intermediate segment the packet is forwarded.
> 
> I seem to still be having a hard time getting the hang of this stuff; thank you again for your patience with explanations!
> 
> So, a node would have to advertise both End.DT46 and End.T and have the route calculation use the respective one in order to get the same effect as End.T + USD.  That is different enough that I will not complain too loudly about it, though in a greenfield ecosystem one could conceivably go with either approach...
> 
> > > Section 4.16.1.2
> > >
> > >    A penultimate SR Segment Endpoint Node is one that, as part of the
> > >    SID processing, copies the last SID from the SRH into the IPv6
> > >    Destination Address and decrements Segments Left value from one to
> > >    zero.
> > >
> > > I think technically it copies the first SID from the SRH (SRH.Segment List[0]), which is the last SID from the SR Policy.
> > > [PC] RFC8754 uses the term “last SID” to refer to SRH.SegmentList[0]. Please refer RFC8754 Sec 6.1 for details.
> > > Also, nit: "the" for "the Segments Left value".
> > > [PC] Ack. Corrected.
> > >
> > >  S14.2.      Update the Next Header field in the preceding header to the
> > >                 Next Header value of the SRH
> > >
> > > nit: personally, I would write "from the SRH", since "next header value of the SRH" is very close to "next header value of SRH", i.e., 43.
> > > [PC] Reads better indeed. Corrected.
> > >
> > >    As a reminder, [RFC8754] defines in section 5 the SR Deployment Model
> > >    within the SR Domain [RFC8402].  Within this framework, the
> > >    Authentication Header (AH) is not used to secure the SRH as described
> > >    in Section 7.5 of [RFC8754].
> > >
> > > I strongly recommend adding another sentence clarifying why this is relevant to discussion of removing an extension header from the packet.
> > > [PC] I suggest that is not relevant to the discussion of PSP since AH is not defined for the SRH.
> > 
> > We should explain why we are saying anything at all about AH.  Right now it sounds like an irrelevant piece of information that should have been edited out of the document, but it actually is quite important in attempting to justify the design decision of allowing PSP!
> > 
> > [PC2] Actually, it was a requested clarification to be added to convey that AH not being specified and used for securing SRH results in there not being any issue with AH when doing PSP.
> 
> Yes, and I support that request.  This text seems to only do half of that, though -- it says that AH is not used for securing SRH, but skips the part about there not being any issue with AH when doing PSP.
> 
> [PC3] We are not considering the use of AH to protect the SR Domain as defined in Section 7.5 of RFC8754. Future documents may define the use of AH in the SR Domain.

Yes, I agree.  I still think that, from a rhetorical perspective, this
statement lacks a purpose as written.  (IMO) we should either say why a
reminder is needed or not say anything.

> > > Section 5
> > >
> > >    This list can be expanded in case any new functionality requires it.
> > >
> > > How?  By an RFC that Updates: this one?
> > > [PC] New functionality would have to be introduced via a new document but it does not have to update this one since it is not modifying what is in this document but adding something new.
> > > We have updated the text as s/This list can be expanded in case any new functionality requires it/This list is not exhaustive and future documents may define additional behaviors.
> > 
> > Thanks.
> > 
> > > Section 5.x
> > >
> > > I agree with the directorate reviewer that the protocol behavior should be written out explicitly, without reference to the handling of example packet(s).
> > > [PC] There is already pseudocode and descriptions in these sections that describes the behavior. Examples are to help understanding. Can you clarify what more is required?
> > 
> > The pseudocode hardcodes that the segment list is (S3, S2, S1; SL=2), which is just an example of what the SRH could be.  It's not truly generic pseudocode.
> > 
> > [PC2] Agreed. I’ve removed the “(S3, S2, S1; SL=2) from the pseudocode. Its specific to the example, and it does not bring any value (the encoding of the SRH is already defined in RFC8754).
> 
> Ah, that looks to be a good resolution; thank you!
> We might want to do a minor follow-up and say "outer IPv6 DA as the first SID in the segment list" (since 'S1' is no longer defined as part of the pseudocode).
> 
> [PC3] Changed.
>  
> > Also, I think it would be great if 5.2, 5.3, and 5.4 could do the sort of thing we did for the endpoitn behaviors where it replaces the definition of one or more of the numbered pseudocode steps.  (I don't think we can, right now, since the numbered pseudocode doesn't cover all the relevant parts that are changing, and I didn't think about how hard it would be to change the pseudocode to make that the case.
> > 
> > > Section 5.1
> > >
> > >    S03.   Set outer payload length, traffic class and flow label
> > >
> > > What about the outer Hop Limit?
> > > [PC] Added.
> > >
> > >    S03: As described in [RFC6437] (IPv6 Flow Label Specification)
> > >
> > > Why is this all the way at the bottom of the section instead of included with the corresponding content?
> > > [PC] Ack. Brought up.
> > >
> > > Section 5.3
> > >
> > >    The encapsulating node MUST remove the preamble or frame check
> > >    sequence (FCS) from the Ethernet frame upon encapsulation and the
> > >    decapsulating node MUST regenerate the preamble or FCS before
> > >    forwarding Ethernet frame.
> > >
> > > "or"?  I get to choose one or the other?  It's okay to only put back a preamble but no FCS (or vice versa)?
> > > [PC] Good catch. s/or/and; also preamble removed only if present
> > >
> > > Section 6
> > >
> > >    traffic that matched that SID and was processed correctly.  The
> > >    retrieval of these counters via MIB, NETCONF/YANG or any other means
> > >    is outside the scope of this document.
> > >
> > > (editorial) a MIB is an abstract data structure (likewise a YANG module); information might be retrieved *from* it but not *via* it.  For *via*, that would be SNMP, NETCONF, and/or RESTCONF.
> > > [PC] Fixed.
> > >
> > > Section 8.1
> > >
> > > Does it make sense to reference RFCs 8491, 8476, 8841, etc. for advertising MSD via IGP?
> > > [PC] I would leave that up to the srv6 control plane drafts; but no strong preference.
> > >
> > >    In particular, the SR source (e.g., H.Encaps), intermediate endpoint
> > >    (e.g., End, End.X) and final endpoint (e.g., End.DX4, End.DT6)
> > >    behaviors.  [...]
> > >
> > > This is not a complete sentence, and I really have no idea what verb was intended.
> > > [PC] Indeed, fixed in rev21.
> > >
> > > Section 8.2, 8.3
> > >
> > > Can we get references for the BGP variants here and their usage for signaling SRv6 capabilities and (SID,behavior codepoint) pairs?
> > > [PC] The references to the BGP features have already been provided in the respective sections where these behaviors are specified. There are WG drafts in progress for SRv6 extensions to routing protocols (not just BGP) that normatively reference this document. There used to be informative references the other way around from this document which were removed during WGLC based on feedback. We can add them back if that makes sense for the IESG.
> > 
> > I think that those informative references would be helpful, but I also don't know how strong the WG consensus was to remove them (or what reasoning was offered for doing so).
> > 
> > [PC2] I’ll leave it up to Martin (responsible AD).
> > On a separate note, on 8.3 I’ve also added the note on the Opaque as discussed.
> > <NEW>
> > In some scenarios, an egress PE advertising a VPN route might wish to abstract the specific behavior bound to the SID from the ingress PE and other routers in the network. In such case, the SID may be advertised using the Opaque SRv6 Endpoint Behavior codepoint defined in [Section 10.2.1]. The details of such control plane signaling mechanisms are out of the scope of this document.
> > </NEW>
> 
> Thanks again for this clarification; this behavior feels perfectly natural once explained, but really confused me on first reading.
> 
> > > Section 9
> > >
> > > I would really like to have a stronger reference to Section 5 of RFC
> > > 8754 where the conceptual model of a SR Domain as a single system, where all packets in the domain are encapsulated using IPv6 headers [with SIDs as SA/DA].  The part in brackets is not spelled out very clearly and could well be expounded upon in this document.  I can write up some (very ad hoc) ideas for what I had in mind, which you are welcome to adopt but expected to wordsmith/edit appropriately:
> > >
> > > % Section 5 of [RFC8754] describes the SR deployment model and the % baseline requirements for securing the SR Domain.  In many ways the SR % Domain is analogous to an MPLS domain -- it's a tightly controlled % system where packet processing is controlled by a list (or stack) of % opaque identifiers that, by and large, only have defined semantics in % the context of the node that receives and processes the packet when % that identifier is "topmost" or "current".  Experience in securing and % operating MPLS domains should be readily transferrable to SRv6 % domains, in ensuring that all traffic within the network is tightly % controlled, with all external inputs being properly encapsulated (for % SRv6, by an IPv6 header usually with SRH; for MPLS, by the label % stack).  The dedicated format for the MPLS label stack and label % identifiers makes a clear mechanical separation between "internal" and % "external" identifiers. In contrast, SRv6 uses SIDs for the internal % identifiers, which have format and operation of IPv6 addresses.  While % this lets SRv6 benefit from IP longest-prefix routing and thus the % structure of SID values, it also means that there is not a crisp % mechanical separation between "internal" and "external" identifiers.
> > > % As such, care and rigor in annotating which IPv6 addresses/headers are % internal vs external is required in order to maintain the integrity % and security of the SRv6 domain.
> > >
> > > [PC] Thanks for your suggestions. Indeed we should update the draft with something along those lines.
> > > The comparison or equivalence to MPLS is likely going to be confusing and hence we avoid going down that path. We were thinking on something like the following.
> > > Please let me know of what you think. (I have not pushed this into the draft yet).
> > >
> > > <Proposal>
> > 
> > [re-wrapped for ease of inline commenting]
> > 
> > That said, I am just offering editorial suggestions; I really like what you came up with -- thank you!
> > 
> > >   The security considerations for Segment Routing are discussed in
> > >   [RFC8402]. Section 5 of [RFC8754] describes the SR Deployment Model and
> > >   the requirements for securing the SR Domain. The security
> > >   considerations of [RFC8754] also cover aspects such as attack 
> > > vectors
> > 
> > nit: I'd consider s/aspects/topics/ but it's a bit of a judgment call 
> > [PC2] Changed.
> > 
> > >   and their mitigation mechanisms that also apply to the behaviors
> > >   introduced in this document. Together, they describe the required
> > >   security mechanisms that allow establishment of an SR domain of 
> > > trust
> > 
> > I would end the sentence here, after "SR domain of trust".  Then we could continue on as "Having such a well-defined trust boundary is necessary in order to operate [...]"
> > [PC2] Changed. Thanks.
> > 
> > >   to operate SRv6-based services for internal traffic while preventing
> > >   any   external traffic from accessing or exploiting the SRv6-based
> > >   services.  Care and rigor in IPv6 address allocation for use for SRv6
> > >   SID allocations and network infrastructure addresses to be separate
> > >   from IPv6 addresses allocated for end-users/systems (as illustrated in
> > >   Section 5.1 of [RFC8754]) help differentiate internal vs external
> > >   address space that is required to maintain the integrity and security
> > >   of the SRv6 Domain. Additionally, [RFC8754] defines an HMAC TLV
> > 
> > This sentence got a bit convoluted and maybe a bit long.  I propose (only helping with the "convoluted" but not the "long" part):
> > 
> > % Care and rigor in IPv6 address allocation for use for SRv6 SID % allocations and network infrastructure addresses, as distinct from IPv6 % addresses allocated for end-users/systems (as illustrated in Section 5.1 % of [RFC8754]), can provide the clear distinction between internal and % external address space that is required to maintain the integrity and % security of the SRv6 Domain.
> > [PC2] Changed. Thanks.
> > 
> > >   permitting SR Endpoint Nodes in the SR domain to verify that the SRH
> > >   applied to a packet was selected by an authorized party and to ensure
> > >   that the segment list is not modified after generation, regardless of
> > >   the number of segments in the segment list.  When enabled by local
> > >   configuration, HMAC processing occurs at the beginning of SRH
> > >   processing as defined in [RFC8754] Section 2.1.2.1.
> > >
> > > This document introduces SRv6 Endpoint and SR Policy Headend   behaviors
> > > for implementation on SRv6 capable nodes in the network.   As such, this
> > > document does not introduce any new security   considerations.
> > > </Proposal>
> > 
> > I'm less sure that we want to keep the "no new security considerations"
> > text.  I have a few notes staged in my updated ballot comments, but one noteworthy thing that comes to mind is that PSP prevents the egress node from validating the HMAC TLV.
> > 
> > [PC2] If you would like to use the HMAC TLV then why would the source choose to use a PSP-flavored SID? The SR Policy computation is aware of this and takes it into account upon doing path computation. Note that in rev24 we added this text as well:
> 
> It would be a pretty bizzare choice, but (even with the following
> paragraph) we don't seem to completely prohibit it.
> 
> > <quote>
> >    The headend policy definition should be consistent with the specific
> >    behavior used and any local configuration (as specified in
> >    Section 4.1.1).
> > </quote>
> 
> I think I had failed to internalize what this was saying, so thank you for pointing it out specifically.  It helps a lot, but I still think that "no new security considerations" is contingent on the headend policy heeding that advice.  We typically do not make such assumptions when writing security considerations sections, and describe the risks that can occur when a deployment diverges from the recommended configuration.
> 
> [PC3] The very essence of SR is about source routing of packets using a list of segments. The very basis of the source routing model is that the SR Source Node’s selection of these segments is based on some sound logic and awareness of their characteristics. This is data plane independent. So, I fail to understand what is it that you seek to add to the text and how it helps. I don’t think this document has anything further to add in that matter besides what is discussed in RFC8402.

In short, this is an additional consideration that the source node has to
take into account when deciding on the SR policy.  We are introducing the
ability to do PSP/etc., and the potential presence of them is a factor in
when and how HMAC TLV can be used, etc.

> > > There might also be room for discussion (in the main body of the
> > > document) of how the various decapsulation procedures occur precisely when the packet is leaving the SR Domain.
> > > [PC] The sections 4.4 through 4.12 do describe these procedures for the decapsulation behaviors specified in this document.
> > 
> > Yes, the decapsulation procedures themselves are covered well.  I am suggesting to mention that the act of decapsulation (generally) signals that the packet is leaving the SR domain, where we describe the procedures themselves.
> > 
> > [PC2] Decapsulation may primarily occur at the edge of an SR Domain, and RFC8754 describes this scenario in section 5.
> 
> I agree.  I'm not sure whether you are rejecting my suggestion (which, to be clear, is a perfectly fine thing to do) or it just wasn't coming across as I intended.  For clarity, I was proposing to add something like "a decapsulating behavior such as End.DX6 is typically (but not exclusively) used at the egress of the SR Domain in accordance with Section 5 of [RFC8754]" as a note at the end of (e.g.) §4.4, and similarly for the other decapsulating behaviors.
> 
> [PC3] The current text explicitly states that it is used at an egress PE for an L3VPN use-case. 
> e.g. for End.DX4:
>    One of the applications of the End.DX4 behavior is the L3VPNv4 use-
>    case where a FIB lookup in a specific tenant table at the egress
>    Provider Edge (PE) is not required.  This is equivalent to the per-CE
>    VPN label in MPLS [RFC4364].
>  
> > > Having the ARG structure specified as part of the behavior codepoint registration means that a node that can apply that SID can also spoof the ARG to force a different (e.g.) outgoing interface list.
> > > Specifically, it gives the attacker a degree of freedom greater than just the SID behaviors that the node advertises.  The HMAC TLV would mitigate this to the extent that it limits what SIDs the node in question is allowed to apply and what changes can be made on-path.
> > > [PC] The processing of ARG is specific to a behavior defined and not generic. Please refer to the updated text in Sec 4.12 for End.DT2M (which is currently the only specified behavior that accepts ARGs). Since the ARG bits are part of the SID there are no other considerations for them apart from the ones that apply to the SID as a whole.
> > 
> > Even though the ARG bits are specific to a behavior (a function, 
> > really, IIUC), if there is a need to have ARG bits then there is going 
> > to be more than one valid value for the ARG bits.  Now that we have 
> > removed a description for the structure of the ARG bits, it becomes a 
> > little harder to spoof them, but only a little bit.  Given the 
> > L+F+A<128 constraint, A may very well be something small, on the order 
> > of 10; an exhaustive search over the 2**10 possible values for ARG to 
> > see which are valid and what they do would be fairly easy to do.  An 
> > attacker in a position to do that exhaustive search (admittedly, one 
> > would be violating the "trusted SR domain" assumption to get there) 
> > could be able to observe the different behaviors provided by the 
> > different ARG values and select what behavior they wanted by spoofing 
> > ARG.  If A was larger, like 64 or 128, this search and guessing would 
> > be much less practical, but we don't really have that many bits lying 
> > around.  So, I think this is a new security consideration introduced 
> > by the presence of the ARG bits in the SID, and we should mention it.  
> > (We would also mention that the risk is largely mitigated by the 
> > assumption that the SR domain is trusted, of course.)
> > 
> > [PC2] This service theft consideration exists for every SID, regardless of the use of arguments.  It is discussed in RFC8754 section 7.2 and referenced from section 9 of this draft.
> 
> Well, yes and no.  I am imagining some risk about service theft for an SID that is not explicitly advertised anywhere, if the implementation does not specifically check for the exact SID values that are advertised.
> 
> Consider the case where a single node has SIDs that use multiple distinct LOCs, call them LOCs A and B.  If a SID (call it Q) is advertised with A and behavior code X with ARG1, and a different SID (call it O) is advertised with B and behavior code X with ARG2, then this implementation is likely to also process an SID with LOC A, the FUNCT value from Q, and ARG2, even though such an SID is not actually advertised.  So, the spoofing allows to create new behaviors in addition to just service theft of expected behaviors.
> 
> [PC3] Arguments do not create behaviors. So, the handling of unexpected arguments is entirely dependent on the behavior and its pseudocode. So in effect it is not different if someone were to spoof the entire SID as we were discussing earlier.

Arguments do not create "behavior"s in the sense that we use "behavior" as
a technical term for something that is tied to a function, yes.  They do
create different observable behaviors in the normal english usage of
"behavior", though -- the specific actions that the SR node takes will
depend on the argument.  When SIDs have no internal structure, the ability
to spoof (or not) an SID is a "one-shot" thing that is limited in scope to
being able to spoof a SID value that was advertised.  Now that there is
internal structure within SIDs, in certain implementation models, it is
possible to spoof even SID values that were not advertised, to cause the
observable operational behavior of the SR node to include actions that were
never advertised.  IMO, this is a new risk/new consideration due to the
functionality defined in this document, and should be documented as such.
(If the implementation chooses to ignore the internal structure of the SID
and retain a ~lookup table with the specific SIDs advertised, then there is
no new risk that I can see.)

Thanks again,

Ben

> 
> > >    services.  Additionally, [RFC8754] defines an HMAC TLV permitting SR
> > >    Endpoint Nodes in the SR domain to verify that the SRH applied to a
> > >    packet was selected by an authorized party and to ensure that the
> > >    segment list is not modified after generation, regardless of the
> > >    number of segments in the segment list.  When enabled by local
> > >
> > > I suggest adding a sentence similar to "(This does, however, require that the segment list, and thus, the SRH is present in the packet; the optional ability to elide the SRH when there is only a single segment and no flags, tags, or TLVs needed inherently excludes the protection of the HMAC TLV.)"
> > > [PC] All the SR Policy Headend (i.e. the SR Source Node) behaviors defined in Sec 5 state:
> > > >  The push of the SRH MAY be omitted when the SRv6 Policy only contains   one segment and there is no need to use any flag, tag or TLV.
> > > So, it follows that if HMAC protection is required, then SRH has to be used even with a single segment.
> > 
> > Ah, I see your point, yes.  Sorry for missing that.
> > (But I think my earlier comment about PSP still applies.)
> > 
> > > Section 10.1
> > >
> > >    definition: The value 143 in the Next Header field of an IPv6 header
> > >    or any extension header indicates that the payload is an Ethernet
> > >    [IEEE.802.3_2018].
> > >
> > > nit: is there a missing word here ("frame"?)?
> > > [PC] Ack, fixed.
> > >
> > > Many thanks for your time.
> > 
> > Thank you for yours as well, and the updates already present and queued.
> > 
> > Sorry again for taking so long to get back to you.
> > 
> > -Ben