Subject: [RTG-DIR] (resend) Re: RtgDir review: draft-ietf-rtgwg-cl-requirement-13

From: Curtis Villamizar <[email-masked]>
Date: Mon, 06 Jan 2014 12:00:00 -0500

Lou et al,

Resending.  I'm not sure if the original (sent Tue, 17 Dec 2013) went
through to all recipients. There may have been a problem (due to a
self inflicted DNS issue).

I did not get a response so I suspect Lou didn't get this email.

Curtis

In message <[email-masked]>
Lou Berger writes:
>
> Hello,
>  
> I have been selected as the Routing Directorate reviewer for this draft.
> The Routing Directorate seeks to review all routing or routing-related
> drafts as they pass through IETF last call and IESG review. The purpose
> of the review is to provide assistance to the Routing ADs. For more
> information about the Routing Directorate, please see
> http://www.ietf.org/iesg/directorate/routing.html
>  
> Although these comments are primarily for the use of the Routing ADs, it
> would be helpful if you could consider them along with any other IETF
> Last Call comments that you receive, and strive to resolve them through
> discussion or by updating the draft.
>  
> Document: draft-ietf-rtgwg-cl-requirement-13.txt
> Reviewer: Lou Berger
> Review Date: 2013-12-11
> IETF LC End Date: 2013-12-09
> Intended Status: Informational
>  
> Summary:
>  
>         I have some minor concerns about this document that I think
> should be resolved before publication.
>  
> Comments:
>  
>     I think the document is greatly improved since the previous last
> call.  I have some comments and reservations on the document that are
> described below.  As discussed on the list, I remain concerned about the
> value of defining requirements in terms of nebulous "Performance
> Objectives", but as this is acceptable to the WG I will not elaborate on
> this concern further.
>  
> Major Issues:
>  
>    No major issues found.
>  
> Minor Issues:
>  
> 1. Terminology & consistency: the document coins the term "Advanced
> Multipath":
>        Advanced Multipath is a formalization of multipath techniques
>  
> Do you see this term as a new noun, like LSP, or as an adjective?  I
> expected the latter, i.e. that it be used like "multipath" is used in
> the above sentence, but it seems to be used as a new noun.  I'm not sure
> coining a new noun really makes sense, but if this the intent then the
> last paragraph of section 2 needs to be revised as "Advanced Multipath"
> will now have a specific definition as opposed to a generic term. Also I
> suggest always capitalizing it (or even using the abbreviation "AM")
> will clarify this distinction.

I see multipath as a noun and therefore advanced multipath as a noun.
Advanced is an adjective, multipath as a noun.  Advanced Multipath is
the set of techniques that form a solution.  An advanced multipath is
a collection of componet links.  The former is capitalized, the latter
is not.  I'll check to see if I got that right in all occurances.  If
you would like I can also add this to the definition making it:

 OLD

   Advanced Multipath
       Advanced Multipath meets the requirements defined in this
       document.  A key capability of advanced multipath is the support
       of non-homogeneous component links.

 NEW

   Advanced Multipath
       Advanced Multipath is a formalization of multipath techniques
       that meets the requirements defined in this document.  A key
       capability of Advanced Multipath is the support of
       non-homogeneous component links.  An "advanced multipath" is a
       collection of component links.  In this document the former is
       capitalized and the latter is not.

If we agree on this I'll just search for "advanced" and make
capitalization consistent with the above statements.

I don't think the phrase "advanced multipath technique" makes it an
adjective any more than a statement such as make-before-break is an
MPLS technique" makes MPLS an adjective in normal use.  The phrase "an
X technique" is simply a shorthand for indicating a "technique
applicable to X".  Even if that makes it an adjective the phrase "MPLS
technique" is common and so that should not be a problem.  The
paragraph in question might be the following.  It seems benign to me.

   The term Advanced Multipath is intended to be used within the context
   of this document and the related documents,
   [I-D.ietf-rtgwg-cl-use-cases] and [I-D.ietf-rtgwg-cl-framework] and
   any other related document.  Other advanced multipath techniques may
   in the future arise.  If the capabilities defined in this document
   become commonplace, they would no longer be considered "advanced".
   Use of the term "advanced multipath" outside this document, if
   referring to the term as defined here, should indicate Advanced
   Multipath as defined by this document, citing the current document
   name.  If using another definition of "advanced multipath", documents
   may optionally clarify that they are not using the term "advanced
   multipath" as defined by this document if clarification is deemed
   helpful.

I see no problem with this.  Please be more specific if you think
there are places where advanced multipath is used as an adjective or
where its use is confusing.

> 2. Editorial: server and client layer vs upper and lower layer.
>  
> The document uses server and client layer networks and LSPs and,
> sometimes interchangeably or redundantly, upper and lower layer networks
> and LSPs.  I think this issue can be resolved by avoiding the term
> client/server when referring to network layers and just using the
> lower/upper terminology.  The one exception to this is in the definition
> Client LSP which should simply be defined in the context of a multipath,
> i.e.:
> OLD
> A client LSP is an LSP which has been set up over a server layer.
> NEW
> A client LSP is an LSP which has been set up over Advanced Multipath.

A client LSP can be set up over a server layer MPLS-TP LSP or any
server layer MPLS LSP or over a link layer or over a pseudowire ... or
over an advanced multipath.

> I think this also means that usage of the term "Client" is limited to
> "Client LSP".

I searched for all occurances of the word client.  All occurances are
"client LSP" excexpt the phrase "client of a client LSP" and that is
used only in the following paragraph.

   The ingress and egress of a multipath may be midpoint LSRs with
   respect to a given client LSP.  A midpoint LSR does not participate
   in the signaling of any clients of the client LSP.  Therefore, in
   general, multipath endpoints cannot determine requirements of clients
   of a client LSP through participation in the signaling of the clients
   of the client LSP.

THe point is that "A midpoint LSR does not participate in the
signaling of any clients of the client LSP" and that non-participation
in client (or higher layer) signaling applies to any "client of a
client LSP", not just other LSP running over it.

I then searched for all occurances of the words upper and lower and
higher.  There are no occurances of "upper".

There were a few occurances of "lower latency" and "higher latency".
Other than that, all occurances of lower and higher except one are in
the follwoing text:

 3.2.  Component Links Provided by Lower Layer Networks

   A component link may be supported by a lower layer network.  For
   example, the lower layer may be a circuit switched network or another
   MPLS network (e.g., MPLS-TP)).  The lower layer network may change
   the latency (and/or other performance parameters) seen by the client
   layer.  Currently, there is no protocol for the lower layer network
   to inform the higher layer network of a change in a performance
   parameter.  Communication of the latency performance parameter is a
   very important requirement.  Communication of other performance
   parameters (e.g., delay variation) is desirable.

   FR#8  The solution SHALL specify a protocol means to allow a lower
       layer server network to communicate latency to the higher layer
       client network.

The exception is this sentence:

     FR#22 The solution SHOULD support the use case where an advanced
       multipath itself is a component link for a higher order advanced
       multipath.  For example, an advanced multipath comprised of MPLS-
       TP bi-directional tunnels viewed as logical links could then be
       used as a component link in yet another advanced multipath that
       connects MPLS routers.

The terms lower layer and higher layer go all the way back to the ISO
seven layer model of ancient times (1970s?, 1980s?) and maybe further
back but that is before even my time.

I don't think this is unclear but I could add the following in
definitions:

   Higher Layers
      A client layer is the one immediately above a server layer.  The
      client layer and all layers above that layer are higher layers.

   Lower Layers
      A server layer is the later immediately below a client laer.
      The server layer and all layers below are lower layers.

Do I really need to put this in the definitions section?  If yoy think
it is necessary, I will add it.

> 3. Technical: The document should make it clear that LSPs can provide
> paths from an Advanced Multipath.  I suggest adding something like the
> the following at the end of page 3
>    d. a lower layer LSP.

Case "b." is intended to include LSP but is not specifically limited
to LSP.  Multipath (not Advanced Multipath) includes IGP ECMP,
Ethernet Link Aggregation, and all sorts of things that don't involve
MPLS at all.  For example, a path in Ethernet Link Aggregation Group
(LAG) could be supported by a pseudowires or ODU2e or GFP-F, though
this would be unknown to the LAG, or in the simplest case each path in
the set of LAG members could be supported by a plain Ethernet cable.

> and at the end of the Component Link definition:
>    A component link may be provided via an LSP.

I think this is covered in "3.2.  Component Links Provided by Lower
Layer Networks".

 3.2.  Component Links Provided by Lower Layer Networks

   A component link may be supported by a lower layer network.  For
   example, the lower layer may be a circuit switched network or another
   MPLS network (e.g., MPLS-TP)).

This sort of statement could be made earlier.  For example in the
discussion following the definitions (on page 5) the document covers
one case (where a component link is not an LSP) but it is not until
later that the other case is mentioned.

I could add a paragraph in the discussion following the definitions
that mentions that a component link can also be another LSP.

 OLD:

   A Component Link may be a point-to-point physical link (where a
   "physical link" includes one or more link layer plus a physical
   layer) or a logical link that preserves ordering in the steady state.
   A component link may have transient out of order events, but such
   events must not exceed the network's Performance Objectives.  For
   example, a component link may be comprised of any supportable
   combination of link layers over a physical layer or over logical sub-
   layers, including those providing physical layer emulation.

 NEW:

   A Component Link may be a point-to-point physical link (where a
   "physical link" includes one or more link layer plus a physical
   layer) or a logical link that preserves ordering in the steady state.
   A component link may have transient out of order events, but such
   events must not exceed the network's Performance Objectives.  For
   example, a component link may be comprised of any supportable
   combination of link layers over a physical layer or over logical sub-
   layers, including those providing physical layer emulation, or over
    MPLS server layer LSP.

These are identical except the phrase ", or over MPLS server layer
LSP" added to the end of the last sentence.

> 4. Editorial: Unless I'm mistaken, the requirements in this document
> apply to the Advanced Multipath solution.  In most cases the document
> states requirements in the form of "The solution", but in a few cases it
> just says "advanced multipath".  I think these cases should be changed
> to apply to an "Advanced Multipath solution".  I think this comment
> applies to FR1, FR7, FR18, FR23, GR3-5, and MR1-9.

That is not correct.  For example:

   FR#1  An advanced multipath MAY be announced in conjunction with
       detailed parameters about its component links, such as bandwidth
       and latency.  The advanced multipath SHALL behave as a single IGP
       adjacency.

In this requirement the "advanced multipath" is a specific collection
of component links that is announced by way of an MPLS Forwarding
Adjaccency (FA) in the IGP.  The same is true for the other
requirements cited.  In all of these cases an advanced multipath is a
specific collection of component links.  There is a typo in MR#1.

 OLD

   MR#1  Management Plane MUST support polling of the status and
       configuration of an advanced multipath and its individual
       advanced multipath and support notification of status change.

 NEW

   MR#1  Management Plane MUST support polling of the status and
       configuration of an advanced multipath and its individual
       component links and support notification of status change.

In the last line
s/advanced multipath/component links/

> 5. Technical: use of "indicate" in FR10-13, FR14, FR15:
> In these cases it is unclear if the requirements apply to
> (a) a client's ability to indicate a desired service/LSP constraint or
> (b) a selected component link's attribute that will be used by a client
> LSP,
> (c) both, or
> (d) something completely different
> The requirements should be clear to which entity the requirement applies.

This "indication" can be through signaling or the management plane and
both must be supported in later framework and protocol definitions and
we can argue then whether both must be supported or if one or the
other is a "should" in RFC 2119 speak.

There was concern that the word "signal" would exclude providing the
same informantion through the management plane and therefore the word
"indicate" is used.

Rather than spell this out many times we hoped this would be clear.  I
could add clarifying text up front if you think this is necessary.  For
example, I could add the following before FR#1.

   FR#0  In requirements that follow in this document the word
      "indicate" is used where information may be provided by either
      the combination of link state IGP advertisement and MPLS LSP
      signaling or via management plane protocols.  In later documents
      providing framework and protocol definitions both signaling and
      management plane mechanisms MUST be defined.

It would be trivial to make this FR#1 and renumber the rest.

Should I add it and renumber?

> 6. The last paragraph in section 3.3. strikes me as both odd and out of
> place.  There are so many possible decisions that must be considered by
> network operators relative to disruption and optimization.  Why mention
> just "power reduction"?  Is there something special about the
> interactions of multipath and power reduction?  What value / information
> does this paragraph add?

The last paragraph in section 3.3 is:

   Allowing multipath to contain component links with different
   characteristics can improve the overall load balance and can be
   accomplished while still accommodating the more strict requirements
   of a subset of client LSP.

I'm not sure if this is the "odd and out of place" paragraph you refer
to.  It summarizes the purpose of the requirements in this section.  I
think you mean section 3.5 and not 3.3.

The remainder of your paragraph above discusses power reducetion and
this is only mentioned in section 3.5 so I think you mean this
paragraph.

   As with any load balancing change, a change initiated for the purpose
   of power reduction may be minimally disruptive.  Typically the
   disruption is limited to a change in delay characteristics and the
   potential for a very brief period with traffic reordering.  The
   network operator when configuring a network for power reduction
   should weigh the benefit of power reduction against the disadvantage
   of a minimal disruption.

The paragraphs following a set of requirements provide discussion
related to that set of requirements.  I don't see this as out of
place.  The prior two paragraphs discuss delay discontinuity.  This
paragraph is a reminder that "As with any load balancing change, a
change initiated for the purpose of power reduction may be minimally
disruptive."  I don't see this as out of place.

For example, to compensate for a less than perfect load balance a
subset of LSPs may need to be moved and that subset can be chosen from
those which will be least affected (and there are plenty of
requirements that compell the fussy LSP to tell the network abour
their specifial needs).  To power down a component link, *all* LSP on
that component link need to be moved, even the very fussy ones.
Therefore I think mentioning that disruption should be considered is
not out of place in the discussion following these requirements.  We
like to keep the discussion brief so this level of detail was not
included.

> 7. Terminology: In GR4&5: what is a "MPLS network topology"? Do you mean AS?

At the very least it means something similar to same administrative
domain.  An AS can be subdivided within an organization.  Do you have
a better way to define intra-domain and inter-domain without going
down the rat hole of defining the term "domain"?  No on in the WG had
an issue with this so far but I'm open for a suggestion for better
wording.

> Nits:
>  
> - References should be provided in all cases when referring to
>   "existing ... techniques"

References are not allowed in the abstract.  In section 3.3 is the sentence:

   Refer to the Appendices in [I-D.ietf-rtgwg-cl-use-cases] for a
   description of existing techniques and a set of references.

Much of that document is dedicated to describing existing techniques
and it contains quite a few references.

I can move this sentence.  Where in the document would you like to see
it moved to?

> Using line numbers from
> http://tools.ietf.org/idnits?url=http://tools.ietf.org/id/draft-ietf-rtgwg-cl-requirement-13.txt

Using line number is a text editor counts the page headers.  What tool
are you using?  I didn't see anything on tools.ietf.org.

So I'll guess.  Looks like add 10 lines per page.

> Line 112
> s/SHOULD/should

s/keywords, SHOULD be interpreted/keywords SHOULD be interpreted/

Also remove stray comma.

> Line 118
> s/MAY/may

s/deployment MAY choose to/deployment may choose to/

OK.  Could go either way but s/MAY chose to/MAY/ would work too.

> Line 149, match intro:
> s/Advanced Multipath meets/Advanced Multipath is a formalization of
> multipath techniques that meet

OK.

> Lines 251-254, beginning with "The" through the end of the paragraph.
> This should be an FR.

Looks like you mean this:

   The transient period between some service disrupting
   event and the convergence of the routing and/or signaling protocols
   MUST occur within a time frame specified by Performance Objective
   values.

OK.  I'll add this as FR#6++.

> That's it,
> Lou

There are a few suggestions for changes to the text based on your
comments and a few questions for you.  Based on your response to this
email I will edit the document.

Thanks for the thorough review.

Curtis

- --rBI3apOu079413.1387337811/maildrop2.v6ds.occnc.com--

------- End of Forwarded Message

---

From: Lou Berger <[email-masked]>
Date: Mon, 06 Jan 2014 18:11:53 -0500

Hi Curtis,
	Thanks for the response.  See below

On 1/6/2014 12:00 PM, Curtis Villamizar wrote:
>        non-homogeneous component links. 

Okay.  Then I suggest using "AM" will help delineate this usage.

>  An "advanced multipath" is a

the distinction based on capitalization seems likely to introduce
confusion in this document and in future uses, so I think it is a really
bad idea.

> where its use is confusing.

I believe my confusion was not understanding the differences between
"Advanced Multipath" (with words capitalized) and "advanced multipath"
(with words lower cased).  While I understand this is your intent, I see
it as just leading to confusion. Ultimately, it's the author's / WG's
call, i.e., not mine, to make.

> over an advanced multipath.

Would you accept s/server layer/lower layer/g?

> it is necessary, I will add it.
Perhaps we've talked (okay written) past each other.  I was suggesting
using/keeping the "higher and lower layer" terminology, not dropping it.
And to use it (consistently) in place of "client and server Layer".

> the set of LAG members could be supported by a plain Ethernet cable.

okay, how about:
OLD
   b.  may be specific paths across a network to a destination
       node, or
NEW
   b.  specific paths across a network to a destination
       node (e.g., via an LSP), or

> LSP" added to the end of the last sentence.

Sure.

> specific collection of component links.  

Hereto, the distinction between "advanced multipath" and "Advanced
Multipath" was lost on me.

> There is a typo in MR#1.

So is this the IGP/signaling that runs between a client an an AM "LER",
between an AM "LSR" and its server network, or between AM LSRs?  If I
understand AM properly, there may be *up to* there separate instances.
Am I missing something?

> It would be trivial to make this FR#1 and renumber the rest.

I don't think the FR helped clarify the above question.

> think you mean section 3.5 and not 3.3.

Correct.

> this is only mentioned in section 3.5 so I think you mean this
Right again.

> like to keep the discussion brief so this level of detail was not

I guess we see the importance of this topic differently.  So will this
be the first RFC that uses the term "power reduction"?

> down the rat hole of defining the term "domain"?  

I'd use something like "multiple ASes or routing domains".  If you
really want to avoid domains you could say areas or levels.

> No on in the WG had

You're forgetting my "real" name;-)

> I can move this sentence.  Where in the document would you like to see
(Lines numbers from URL below.)
Lines 200/201 and  278 have this issue.  Lines 297 and 342 do not.

> are you using?  I didn't see anything on tools.ietf.org.
huh.  enter the above URL in a browser and you should see the line numbers.

> s/keywords, SHOULD be interpreted/keywords SHOULD be interpreted/
I don't think use of 2119 language being applied to the reader is
appropriate, but whatever...

> OK.  Could go either way but s/MAY chose to/MAY/ would work too.
No.  The issue is that this document doesn't place conformance
requirements on a service provider so RFC2119 language isn't appropriate
here.

>    MUST occur within a time frame specified by Performance Objective
Right.

> Thanks for the thorough review.

Thanks for the responses and the resend!

Lou

---

From: Curtis Villamizar <[email-masked]>
Date: Tue, 07 Jan 2014 21:56:29 -0500

> 	Thanks for the response.  See below

You pointed out a  a few things you thought should be changed but left
it open for me to propose how to change things, so we'll need at least
one more iteration after this.

> > I did not get a response so I suspect Lou didn't get this email.

FYI - most recent issue I found was an error in IN TXT v=spf1 line.
It might be a bad time to be experimenting with IPv6 transition
technologies and their effects of email.  :-(  Go back to plain old
dual stack and burn another IPv4 global?  Might be the right move.

Hopefully mail will get through to everyone now without a resend.  If
I have to resend I'll put the IPv6 mail server back into the DS
subnet before resending.

Curtis

> Okay.  Then I suggest using "AM" will help delineate this usage.

Later you indicate you prefer we don't use the same term.  Suggestion
follows that.

> confusion in this document and in future uses, so I think it is a really

Clarity is good.  If it lacks clarity we'll have to fix it.

> call, i.e., not mine, to make.

OK so here is a suggestion to improve this.

 OLD

   Advanced Multipath

 OLD NEW

   Advanced Multipath

 NEWER

   Advanced Multipath

   Advanced Multipath Group (AMG)

      An Advanced Multipath Group (AMG) is group of component links
      where Advanced Multipath techniques are applied.

This along the lines of "Link Aggregation" and LAG.  The first is
802.1ax (or whatever, see the references).  The latter is a group of
component links.

Not to be confused with Absorbed Glass Mat lead-acid batteries...
(which is AGM not AMG).  But if you see me write AGM, you'll know why
(habit, typo).

I think if we use these definitions and make if this terminology
change is made everywhere in the document we will acheive the improved
clarity that you were looking for.

Is this OK?

> Would you accept s/server layer/lower layer/g?

The definition of server layer is not the same as the definition of
lower layer.  See below.

> And to use it (consistently) in place of "client and server Layer".

I guess you missed the distinction in the definition above:

  For layer X layer Y is:

    client layer   ===   Y = X+1
    server layer   ===   Y = X-1
    higher layer   ===   Y > X
    lower layer    ===   Y < X

So the definition client layer is not the same as the definition of
higher layer.  The definition of server layer is not the same as the
definition of lower layer.

In some discussions we really do mean "the server layer" and not any
layer at any arbitrary depth below this one.

>        node (e.g., via an LSP), or

OK.  But I like to spell out "for example" and put it after the
definition.  That would give us:

       4.  The paths may be:

           a.  parallel links between two nodes, or

           c.  may be links or paths to an intermediate node used to
               reach a common destination.

       The paths need not have equal capacity.  The paths may or may not
       have equal cost in a routing protocol.
+      Paths across a network which serve as a component link in a AMG
+      may be provided by other LSP acting as server LSP for the
+      client AMG.  Component links of a given AMG need not all use the
+      same type of server layer technology.  For example, one may be a
+      physical Ethernet, another a Packet Over Sonet (POS) link,
+      another a pseudowire, and another an LSP with individual hops
+      using a mix of lower layer technologies.

Is that an acceptable clarification?

btw - in b. and c. s/may be// since "may be" precedes the list.

> Multipath" was lost on me.

OK.  "Advanced Multipath" vs AMG might improve clarity.

> understand AM properly, there may be *up to* there separate instances.

Please don't abbreviate Advanced Multipath as AM.  Link Aggregation is
not LA.

The word "indicate" is independent of the method of getting the
information there.  But yes in the example given the IGP advertisement
and MPLS LSP signaling that might be one such mechanism does refer to
the IGP and RSVP-TE MPLS/GMPLS that we all are familiar with.

If client and server LSP are PSC-3 and PSC-4 in MPLS hierarchy there
is only one IGP flooding information for both client and server layer,
hence the seemingly endless (and mostly pointless) discussion of
sub-layer vs layer from certain ITU people in the MPLS WG mailing list
who viewed MPLS as broken for not enforcing a strict layering in this
regard.  But we need not go into that level of detail in this example
of how independent of mechanism the "indicate" is and down that years
old MPLS WG terminology rat hole again.

> I don't think the FR helped clarify the above question.

The choice of the word "indicate" in FR10-13, FR14, FR15 does not
imply usage by a specific layer.  That is a direct answer to your
question.

In all of these specific cases, the the client layer is passing a
requirement to the server layer so in the IGP + RSVP-TE world that
would be a function of RSVP-TE.  In the absense of control plane, it
would have to be done through management plane interaction where the
client indicates requirements of an LSP and the server layer is free
to figure out how to meet those requirements.

Do you want me to add the clarification on the use of the word
"indicate" or not?

> be the first RFC that uses the term "power reduction"?

Perhaps.  A co-author originally wanted it added.  The set of authors
kicked this around and reworded the original.  The wording is brief
and vague and the use of MAY in the actual requirement means we will
not disallow this.  So far no one in the WG has objected to this
wording and its been there a long time.

Would you like me to remove the requirement and discussion of its
implications after the WG concensus so far was to keep it?

> really want to avoid domains you could say areas or levels.

OK.
s/MPLS network topology/routing domain/

> You're forgetting my "real" name;-)

You plan on living up to it I see.  :-)

> appropriate, but whatever...

You want me to downcase "SHOULD be interpreted".  I missed a M-l in
the s/// edit (emacs downcase next word) and got only the stray comma.

> requirements on a service provider so RFC2119 language isn't appropriate

No IETF document truly places conformance requirements on either the
service provider or implementor, but we put RFC2119 language in
telling them what to do all the time.  No one in IETF signed a legal
document forcing them to adhere to every RFC2119 keyword in each
document.

The requirements here are 1) implementor SHOULD provide a per feature
knob to turn individual features off, 2) provider MAY use those knobs,
including knobs that another implementor made available.

This is just trying to avoid unnecessarily inflexible specifications
or unnecessarily inflexible implementations where some feature doesn't
work when an unrelated feature is disabled.  This is in here because
this inflexibility happens.

The framework can outline any feature dependencies.

> Thanks for the responses and the resend!

Sorry to mess up my own DNS and require a resend and a delay.

Curtis

---

From: Lou Berger <[email-masked]>
Date: Wed, 08 Jan 2014 13:47:52 -0500

Hi Curtis,

See below.

> one more iteration after this.
Right, you're the author....

> I have to resend I'll put the IPv6 mail server back into the DS
no problem at all on this one, so I think you're up and working!

>>> In message <[email-masked]>
...

>       where Advanced Multipath techniques are applied.

Ahh.  I think this is much better/clearer!

> clarity that you were looking for.

Yes!

> layer at any arbitrary depth below this one.

I read your usage of client/server layer to be synonymous with
higher/lower layer, i.e. -+ 1.  Otherwise I'm not sure how to make sense
of FR#8 (I think you really mean client layer not lower layer.)

Given the current document usage, I still think it makes sense to
eliminate the two instances of "client layer":  How about the following:
OLD
   Client LSP
       A client LSP is an LSP which has been set up over a server layer.
       In the context of this discussion, a client LSP is a LSP which
       has been set up over a multipath as opposed to an LSP
       representing the multipath itself or any LSP supporting a
       component links of that multipath.
NEW

   Client LSP
       A client LSP is a LSP which
       has been set up over Advanced Multipath as opposed to an LSP
       representing the Advanced Multipath itself or any LSP that is
       part of an Advanced Multipath Group.

and
OLD
   The above set of requirements apply to component links with different
   characteristics regardless as to whether those component links are
   provided by parallel physical links between nodes or provided by sets
   of paths across a network provided by server layer LSP.
NEW
   The above set of requirements apply to component links with different
   characteristics regardless as to whether those component links are
   provided by parallel physical links between nodes or provided by
   LSPs that are part of an Advanced Multipath Group.

> Is that an acceptable clarification?

Given the the definition of AMG I no longer see a need for the change or
proposed text (and would prefer not having to discuss the proposed text.)
...

> Please don't abbreviate Advanced Multipath as AM.  Link Aggregation is

Perhaps you should a requirement on this ;-)

> old MPLS WG terminology rat hole again.
My comment is that you need to be clear to which layer a requirement
applies.  Is it the server layer, the client layer  or the Advanced
Multipath layer?

> Do you want me to add the clarification on the use of the word

I think you need to be clear that the requirement applies to all three
layers.

> implications after the WG concensus so far was to keep it?

I think the text opens the door to the objection/question as to why just
this "application" and not others are called out.  Now that I've asked
the question, we can move on.  The AD can always revisit if he so chooses.

...

Right.

> including knobs that another implementor made available.

I'd prefer the first to be a MUST, but you've documented wg consensus.
I don't think #2 makes any sense, but we can move on.

> Sorry to mess up my own DNS and require a resend and a delay.
NP!

Lou

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 08 Jan 2014 23:33:55 -0500

Converging.  But maybe one more round trip needed.

> Ahh.  I think this is much better/clearer!

OK.  This is easy enough since it is just a terminology change.

> > clarity that you were looking for.

Only 42 occurances in the XML file to look at.  Certainly doable.

> of FR#8 (I think you really mean client layer not lower layer.)

FR#8 and FR#9 are about the server layer telling the client layer
about the delays that can be expected.  There is a little redundancy
here so s/ower layer server network/server network/ and
s/higher layer client network/client network/.  No plans to skip
layers.

>        part of an Advanced Multipath Group.

Nope.  In general, a client LSP can be set up over a plain old
Ethernet on a given hop, therefore the second definition is too
narrow with this change.

>    LSPs that are part of an Advanced Multipath Group.

What about PW?  Those are paths too by the definition of paths, so
again, this change makes the definition too narrow.

> proposed text (and would prefer not having to discuss the proposed text.)

OK.  We won't make this addition.

> Perhaps you should a requirement on this ;-)

Obvious acronym collision here, just like LA.

> applies.  Is it the server layer, the client layer  or the Advanced

In each of the requirements you cited it is very clear that the client
later is communicating a requirement to the server layer, but if you'd
like I can reword to make that even more clear by rewording of the
form "SHALL provide a means for the client layer to indicate the
requirements of a client LSP [regarding X]", where X is minimize
latency (FR#10), bound latency (FR#11), bound jitter (FR#12), specific
component link (FR#13), bidirection co-routed (FR#14), no reordering
(FR#15), bounded frequency of rebalance (FR#20).

> I think you need to be clear that the requirement applies to all three

There are no distinct three layers.  You are imagining a model that is
not used in this document so please don't impose model on our document.

In your sentence above I have no idea what "the requirement" refers to.

In summary:

We were discussing why the word "indicate" was used to avoid requiring
specific mechanisms for information passing and I asked if I should
add the following.

Information is two way.  The requirements you cited were all worded in
the form "The solution SHALL provide a means to indicate that a client
LSP will ...".  So far I have offerred to reword this to the form "The
solution SHALL provide a means for the client layer to indicate a
requirement that a client LSP will ..." (ie: get minimum latency, get
bound latency, etc).

Is adding FR#0 OK?

Do you want this sort of rewording to make it more explicit that the
client layer is communication a requirement for a specific client LSP?

> the question, we can move on.  The AD can always revisit if he so chooses.

OK

Curtis

---

From: Lou Berger <[email-masked]>
Date: Fri, 10 Jan 2014 12:35:35 -0500

Curtis,

I think we have one disconnect (and corresponding related text) that we
need to resolve before we can be close the discussion. See below for
details..

...

>>>/>> "client LSP" excexpt the phrase "client of a client LSP" and that is

I think we need to hit the points below before this one...

> Ethernet on a given hop, therefore the second definition is too

humm, you didn't have that in your OLD text.  In fact, the old text
reads to me that an Advanced Multipath (solution?) logically sits
between a client LSP and an underlying server layer in all cases.

So the model I see defined by the text has the following layers

   Client LSP (layer)
   ----------
   Advanced Multipath (layer/solution/construct)
   ----------
   Server Layer (composed of AMGs which are LSPs and/or links)

Is this aligned with your intent?  If not, can you explain the
relationship you see for Advanced Multipath Client LSPs and Advanced
Multipath AMGs?

> again, this change makes the definition too narrow.

PWs weren't mentioned in your OLD text, so I didn't add them.  I also
have no objections to adding them.

The only change I made was
s/sets of paths across a network provided by server layer LSP./
 LSPs that are part of an Advanced Multipath Group.

...

> (FR#15), bounded frequency of rebalance (FR#20).

Okay, I think this will/may help.  My comment goes back to the model I
was asking about above.  And to which part of the puzzle the requirement
applies.

> not used in this document so please don't impose model on our document.

Well your old definition of Client LSP said that it was a client of
"multipath" and else where you say that "multipath" operates over a
server layer.  As mentioned above, this sounds like three layers to me.
 If this is not your intent, I think you need to make it clear (through
revised text) what model you do intend.

> client layer is communication a requirement for a specific client LSP?

I think we need to come back to this once we resolve the "model" topic.

...

Thanks,
Lou

---

From: Curtis Villamizar <[email-masked]>
Date: Wed, 15 Jan 2014 19:00:06 -0500

OK.

> between a client LSP and an underlying server layer in all cases.

The "OLD" was a quote from you and it was the original text in -13.  I
said "Nope. ..." to your suggested change to it in this way.

Earlier
(http://www.ietf.org/mail-archive/web/rtgwg/current/msg04274.html) I
had suggested the following change:

   Higher Layers

   Lower Layers

You will find this in the quoted text above.

The definition of client LSP does not imply that Advanced Multipath is
a layer.  This also used lower case "multipath" which we can replace
with AMG.  There is no LAG layer in a network in with MPLS runs over
Ethernet and some of the Ethernets use Link Aggregation.

> So the model I see defined by the text has the following layers

You imaginged this model and are trying to constrain the text to fit
into it..

Consider these examples of client layer and server layer.

  plain old rfc-4206

    LSP-1 (PSC-0) = client of LSP-2
    LSP-2 (PSC-1) = server of LSP-1

  LSP over Ethernet over PW over MPLS

    LSP-1 = client of Ethernet
    Ethernet = server of LSP-1, client of PW
    PW = server of Ethernet, client of LSP-2
    LSP-2 = server of PW

There is then the question of whether Advanced Multipath is a layer or
a set of techniques that can be applied at a given layer.  Example
potential config language:

  mple {
    tunnel x1 {
      type multipath {
        component tunnel x2;
	component tunnel x3;
	[...]
	component intf i1;
	component intf i2;
	[...]
      }
      [...]
    }
    [...]
  }

No distinct layer above.

An alternate:

  mple {
    tunnel x1 {
      [...]
    }
    [...]
  }

  interface amg1;
      type multipath {
        component tunnel x2;
	component tunnel x3;
	[...]
	component intf i1;
	component intf i2;
	[...]
      }
      [...]
    }
    [...]
  }

In the former example, the tunnel x1 is forced to use a specific set
of component links and apply multipath techniques to it.

In the latter example, if tunnel x1 happen to find that interface amg1
was the lowest cost or otherwise preferred way to get to its
destination it makes use of amg1 and if so inherits the use of the
multipath techniques.

There is no distinct multipath encapsulation at the date layer so some
might argue that even in the second case there is no distinct layer,
just a configuration convenience.

Claiming that Advanced Multipath is of itself a layer may get us into
a bigger can of worms.

For example in today's ECMP, the next hop consists of multiple
interfaces.  ECMP is not considered a layer between IP and those
interfaces.  Link bundle is also not considered a layer.

In the document we have not claimed that Advanced Multipath is a type
of layer and we have not claimed that Advanced Multipath is not a type
of layer.  Either way we would get someone arguing that the opposite
was true.  [Some individuals it seems would pick the opposite of
whatever we picked just because they like to argue about layering.]

Need I remind you of the painfully long and mostly pointless arguments
in MPLS during the early MPLS-TP work about whether an MPLS LSP
carried within another hierarchical MPLS LSP was a layer or a
sub-layer.  We don't want to repeat that and the best way to do so is
to not make any statement about whether Advanced Multipath is a layer
or a sub-layer or a layering NOOP.

Which of the above is the "right" way to do things may be at most a
framework issue but may not come up until management plane entities
are defined.  It is certainly not a topic for a requirements document
because it is deep into the "how it gets done".

> have no objections to adding them.

Actually the text you wrote is incorrect.  "The above requirement
applies to component links" can include physical links and server
layer LSP but adding that are part of an Advanced Multipath Group is
incorrect.

I'm not sure but I think this instance of "server layer" was added to
that LSP wording because you were confused on last review about client
LSP vs LSP over which Advanced Multipath was applied so I went around
changing everything to "client LSP" or "server layer LSP" to make it
more clear to you.  You now seem to be stuck on the wording that is
essentially saying "any type of component link" including "server
layer LSP".

>  LSPs that are part of an Advanced Multipath Group.

And that change was wrong.  A more correct change would be

s/provided by server layer LSP/including paths provided by server
layer LSP/

That would include any sort of path across the network.  PW could be
considered an emulated physcial link or another usable type of path
across the network.

The "above set of requirements" in this case have to do with passing
down requirements for min latency, bounded latency, and bounded
jitter.  The original paragraph is:

   The above set of requirements apply to component links with
   different characteristics regardless as to whether those component
   links are provided by parallel physical links between nodes or
   provided by sets of paths across a network provided by server layer
   LSP.

The intent is "The above set of requirements apply to component links"
followed by "regardless of what type of component links they are"
where we had disccssed two types being an interface or emulated
interface (or virtual interface in some major vendor terms) or an LSP
(which is also a virtual interface in some major vendor terms).

I don't understand how you can get so hung up on the wording of what
is intended to convey "regardless of what type of component links they
are".  This has always been clear to the WG.

> was asking about above.  And to which part of the puzzle the requirement

In some of the above requirements client layer communicated to the
server layer, whatever the server layer is.  Means of communication
presumed to be RSVP-TE but we can't say that until the framework and
same capability needs to be available to management plane.

In some of the above requirements server layer communicates to the
client layer.  Means of communication is presumed to be IGP
extensions, again with same capability available to management plane.

There is no distinct Advanced Multipath layer in control plane or data
plane.  And if there was a distinct control plane layer it would be
defined in a framework, not a requirements document.

> revised text) what model you do intend.

This is the current text in -13:

   Client LSP

The text is trying to clarify what a "client LSP" is in the first
sentence and then trying to give two cases of what a client LSP is
not.  This clarification was specifically added *for you* in the last
iteration.

There was no intention to imply that Advanced Multipath is or is not
in of itself a type of layer.  If you are getting that notion from
this text, then we need to change it.

  OLD:

   Client LSP

  NEW:

   Client LSP

       A client LSP is an LSP which has been set up over a set of one
       or more lower layers.  In the context of this discussion, one
       type of client LSP is a LSP which has been set up over an AMG.

We could also add a clarification to the definitions section that
states:

   This document makes no statement on whether Advanced Multipath is
   itself a layer or whether an instance of AMG is itself a layer.
   This is to avoid engaging in long and pointless discussions about
   what consistitutes a proper layer.

I would *really* like to add the above statement.

> I think we need to come back to this once we resolve the "model" topic.

I hope it is resolved.

Curtis

---

From: Lou Berger <[email-masked]>
Date: Thu, 16 Jan 2014 21:03:35 -0500

Curtis,
	I'm not sure if we're going around in circles or not, but in either
case case I think we're getting caught up in the weeds/details.

>From a high level perspective, my comments have been motivated by trying
to ensure the requirements are unambiguous -- as documented.  I think
the text changes that have been agreed to so far (notably the use of the
capitalized term and introduction of AMG) will help a lot. I think there
is still one more area that is unclear, but it's also possible that
we're arguing about text that you are planning to change.

Do you have a version that captures all the changes to date that you can
distribute (or submit, as you choose)?  Perhaps looking at this version
we'll find that the ambiguity is resolved or is sufficiently narrowed to
not be significant.  Either way, the discussion can then continue from
the most current text.

Thanks,
Lou

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 17 Jan 2014 12:41:06 -0500

> not be significant.  Either way, the discussion can then continue from

Lou,

I was going to suggest the same (if I understand your suggestion) -
that is that a new draft be submitted and you (and others) can review
and see if clarity has been sufficiently improved or if there are
still issues with clarity.

If that is OK with you and compatible with this process I will make
two draft submissions back to back.  One with just the OpsDir review
comments addressed and a second with your comments so far addressed.
IMHO it would be easier to discuss the clarity (or lack of) of the
wording once changes have been made particularly since there are a few
s/old/new/g terminology suggestions in the email thread.

Curtis

---

From: Lou Berger <[email-masked]>
Date: Fri, 17 Jan 2014 13:01:33 -0500

sounds like a plan!

Lou

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 25 Jan 2014 14:20:55 -0500

Lou,

I am in the process of submitting a -14 version followed with only the
changes for the OpsDir review followed by a -15 version with the
additional changes from your RtgDir review.  Please wait for the -15
version if you see the -14 and not the -15.

Curtis

---

From: Lou Berger <[email-masked]>
Date: Tue, 28 Jan 2014 18:06:16 -0500

Curtis,
	The changes in the current rev address most of my comments and the
document is significantly less ambiguous.

I'm still not a big fan of how client and layers are used in the
document, but I think this is in the weeds and we should move on and not
engage, as the draft says, "in long and pointless discussions".

Lou