Subject: Re: [spring] [Tsv-art] TSV-ART review of draft-ietf-spring-segment-routing-msdc-08

From: Gaurav Dawra <[email-masked]>
Date: Mon, 15 Oct 2018 22:45:46 -0700

Hi MIrja,

I have posted a new version. Please do let me know if we need to discuss
further. We can do it over the phone.

Cheers,

Gaurav

On Mon, Sep 3, 2018 at 4:29 AM Gaurav Dawra <[email-masked]> wrote:

> Hi Mirja,
>
> Please see inline...[Gaurav2]
>
> On Thu, Aug 9, 2018 at 8:30 AM Mirja Kühlewind <[email-masked]>
> wrote:
>
>> Hi Gaurav,
>>
>> please see inline.
>>
>> On 03.08.2018 07:20, Gaurav Dawra wrote:
>>
>> Hey Mirja,
>>
>> Sorry for the long delay. I was traveling constantly since IETF and bit
>> lost in my mailbox and discussion with Authors. Please see my response
>> inline[Gaurav]
>>
>>
>> I think with your changes you addressed explicit problems Martin called
>> out, however, I still have high level concerns about these sections as they
>> are mostly giving speculative recommendation which are not clear to me to
>> work in practice.
>>
>> Regarding section 7.1, you say
>> "A flowlet is defined as a burst of packets from the same flow followed
>> by an idle interval."
>> but then you say
>> "...then the application can break the elephant flow F into flowlets F1,
>> F2, F3, F4..."
>>
>> This sounds like you would just divide an elephant flow mostly randomly
>> into flowlets which can interact badly with the congestion control. If you
>> actually have chunks of data that are transmitted with large enough idle
>> interval in between (as you define flowlets in the first sentence), that is
>> not an elephant flow anymore and it will not help you to "spread the load
>> of the elephant flow through all the ECMP paths". In summary I actually
>> don't see how the concept of flowlets can be helpful to address the problem
>> you have at all, and I still advise you to remove section 7.1 entirely.
>>
>> [Gaurav] Hi Mirja, Thanks for the review. The proposal here is no
>> different that current ECMP hashing at flowlet level. The only difference
>> which is being pointed out here is that if we use SR, we could leverage on
>> the ability of be aware of multiple disjoint paths rather than the hashing.
>> It’s pins the flowlets to particular paths which is basic SR operations.
>>
>>
>> Yes the problem is that we usually don't recommend ECMP hashing on
>> flowlet level because it can interact badly with the end-end mechanisms of
>> that flow. As I said, I don't see how the notion of flowlets help you
>> problem and therefore I still recommend to remove that paragraph.
>> [Gaurav2] OK. It took sometime to get to consensus with authors. Will
>> update the text to use 5-tuple flows instead of flowlets. Would that
>> suffice? I will update the text shortly.
>>
>>
>> Regarding section 7.2, I also still skeptical about any benefits that can
>> be achieved. Given you are in a data center, the controller should already
>> know about static parameters such as the maximum bandwidth per link.
>>
>> For dynamic parameters, e.g. like loss rate, measuring them on a per-flow
>> bases is the wrong thing to do. What I mean is you can ask a router about
>> the average loss rate that it observes and that might give you some
>> valuable, however, if you ask a TCP flow about the average loss rate the
>> answer will mainly depend on the congestion controller used and the
>> currently available bandwidth, which is a very dynamic property and not a
>> link characteristic. So this information is not usable for performance
>> aware routing. A flow could give you information about the observed RTT
>> (min/max) on a certain path, or the maximum available bandwidth on a path,
>> but as I said, given you are in a data center environment these are
>> information that the controller already should have anyway.
>>
>> [Gaurav] They are two separate mechanisms. Most DCs have some sort of
>> data-plane/ECMP aware tracing mechanism to detect the loss/delays and can
>> be combined with Application back-off to detect issue. All this section is
>> suggesting is that SR can be used to pin the path to particular set of ECMP
>> paths instead of relying on ECMP hashing.
>>
>>
>> This is not quite what the text says. If that is the statement you want
>> to make, that is fine but then you don't need to talk about performance
>> aware routing at all.
>>
> [Gaurav2] I will update the text here with statement i mentioned above.
> IMHO, it's performance aware routing wrt to end-host traffic.
>
>>
>>
>> Your example with detecting a faulty path due to losses does not work
>> with TCP as you never know if these loses are due to a problem on the path,
>> self-induced or by a competing flow. And even if you don't use TCP and e.g.
>> send constant bit rate traffic, there may be a large number of competing
>> TCP flows that can induce the loses. Try to steer traffic "away" on a
>> time-scale that is slower than TCP dynamics or even your flow dynamic (when
>> flows start or end) in case you have a lot of very short flow, in the best
>> case doesn't work and in the worst case leads to oscillation.
>>
>> [Gaurav] As I said above, there are other mechanisms to detect loss and
>> trace the path on which loss is seen. This is a common mechanism used in
>> MSDCs.
>>
>> I think this is beyond the scope of the document.
>>
> [Gaurav2] Will update the text.
>
>>
>>
>>
>> I am happy to discuss further over the phone to try to explain the
>> thought process. I will also do check again with Authors to update the text
>> or something else based on our conversation.
>>
>>
>> Maybe see if some update can be made to the text first and then we can
>> have another discussion if needed.
>> [Gaurav2] Sounds good. Will update the text shortly and then we can
>> discuss if needed.
>>
>
> Cheers,
>
> Gaurav
>
>>
>>
>> Cheers,
>>
>>
>>
>> Gaurav
>>
>> If you want to make TCP use for handover situation where one path might
>> go away or become unusable, it's best to use Multipath TCP (with coupled
>> congestion control) instead because that works on the right time scale.
>> Again, I don't think this is a use case for SR and I would recommend to
>> remove the section entirely.
>>
>> Mirja
>>
>>
>>
>>
>> On Mon, Jul 16, 2018 at 11:08 PM, Gaurav Dawra <[email-masked]>
>> wrote:
>>
>>> Hi Mirja,
>>>
>>> Ack. Let me work with authors to close ASAP.
>>>
>>> Cheers,
>>>
>>> Gaurav
>>>
>>> Sent from my iPhone
>>>
>>> On Jul 5, 2018, at 10:06 AM, Mirja Kühlewind <[email-masked]>
>>> wrote:
>>>
>>> Hi Gaurav,
>>>
>>> sorry for my very long delay but this got somehow a bit lost in my
>>> mailbox.
>>>
>>> I think with your changes you addressed explicit problems Martin called
>>> out, however, I still have high level concerns about these sections as they
>>> are mostly giving speculative recommendation which are not clear to me to
>>> work in practice.
>>>
>>> Regarding section 7.1, you say
>>> "A flowlet is defined as a burst of packets from the same flow followed
>>> by an idle interval."
>>> but then you say
>>> "...then the application can break the elephant flow F into flowlets F1,
>>> F2, F3, F4..."
>>>
>>> This sounds like you would just divide an elephant flow mostly randomly
>>> into flowlets which can interact badly with the congestion control. If you
>>> actually have chunks of data that are transmitted with large enough idle
>>> interval in between (as you define flowlets in the first sentence), that is
>>> not an elephant flow anymore and it will not help you to "spread the load
>>> of the elephant flow through all the ECMP paths". In summary I actually
>>> don't see how the concept of flowlets can be helpful to address the problem
>>> you have at all, and I still advise you to remove section 7.1 entirely.
>>>
>>> Regarding section 7.2, I also still skeptical about any benefits that
>>> can be achieved. Given you are in a data center, the controller should
>>> already know about static parameters such as the maximum bandwidth per
>>> link. For dynamic parameters, e.g. like loss rate, measuring them on a
>>> per-flow bases is the wrong thing to do. What I mean is you can ask a
>>> router about the average loss rate that it observes and that might give you
>>> some valuable, however, if you ask a TCP flow about the average loss rate
>>> the answer will mainly depend on the congestion controller used and the
>>> currently available bandwidth, which is a very dynamic property and not a
>>> link characteristic. So this information is not usable for performance
>>> aware routing. A flow could give you information about the observed RTT
>>> (min/max) on a certain path, or the maximum available bandwidth on a path,
>>> but as I said, given you are in a data center environment these are
>>> information that the controller already should have anyway.
>>>
>>> Your example with detecting a faulty path due to losses does not work
>>> with TCP as you never know if these loses are due to a problem on the path,
>>> self-induced or by a competing flow. And even if you don't use TCP and e.g.
>>> send constant bit rate traffic, there may be a large number of competing
>>> TCP flows that can induce the loses. Try to steer traffic "away" on a
>>> time-scale that is slower than TCP dynamics or even your flow dynamic (when
>>> flows start or end) in case you have a lot of very short flow, in the best
>>> case doesn't work and in the worst case leads to oscillation.
>>>
>>> If you want to make TCP use for handover situation where one path might
>>> go away or become unusable, it's best to use Multipath TCP (with coupled
>>> congestion control) instead because that works on the right time scale.
>>> Again, I don't think this is a use case for SR and I would recommend to
>>> remove the section entirely.
>>>
>>> Mirja
>>>
>>>
>>> On 05.07.2018 04:08, Gaurav Dawra wrote:
>>>
>>> Hey Alvaro, Mirja,
>>>
>>> Friendly reminder to further progress this document.
>>>
>>> Cheers,
>>>
>>> Gaurav
>>>
>>> On Mon, Jun 18, 2018 at 5:13 PM, Gaurav Dawra <[email-masked]>
>>> wrote:
>>>
>>>> Hi Alvaro, Mirja
>>>>
>>>> Any feedback or next steps to close this?
>>>>
>>>> Cheers,
>>>>
>>>> Gaurav
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 12, 2018, at 7:06 AM, Gaurav Dawra <[email-masked]>
>>>> wrote:
>>>>
>>>> Hi Mirja,
>>>>
>>>> Friendly Reminder...Could you please also advice if there is anything
>>>> further to DISCUSS on this document which was also related to TCP updates
>>>> below?
>>>>
>>>> Cheers,
>>>>
>>>> Gaurav
>>>>
>>>> On Thu, Jun 7, 2018 at 9:02 AM, Alvaro Retana <[email-masked]>
>>>> wrote:
>>>>
>>>>> Thanks Martin!
>>>>>
>>>>> On June 6, 2018 at 3:14:45 PM, Martin Stiemerling ([email-masked])
>>>>> wrote:
>>>>>
>>>>> Hi Alvaro, all,
>>>>>
>>>>> Thanks for addressing my concerns.
>>>>>
>>>>> This version is good to go from my side.
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> ;Martin
>>>>>
>>>>> Am 30.05.18 um 21:55 schrieb Alvaro Retana:
>>>>> > Martin:
>>>>> > br/>> Hi!!  How are you?
>>>>> > br/>> Gaurav just posted a revision.  Please takke a look and let us
>>>>> know if br/>> the changes address your concerrns or not.
>>>>> > br/>>
>>>>> https://www.ietf.org/rfcdiff??url2=draft-ietf-spring-segment-routing-msdc-09
>>>>> > br/>> Thanks!!!
>>>>> > br/>> Alvaro. <
>>>>> > br/>> On May 25, 2018 at 12:08:46 PM, Gaurav Dawra ((
>>>>> [email-masked] br/>> <mailto:gdawra.ietf@@gmail.com>) wrote:
>>>>> > br/>>> Hi Martin, <
>>>>> >>
>>>>> >> Thanks for review. I will post the new version. Hopefully, it will
>>>>> br/>>> address all your comments and we can close thhis review.
>>>>> >>
>>>>> >> Any updates on below response?
>>>>> >>
>>>>> >> Cheers,
>>>>> >>
>>>>> >> Gaurav
>>>>> >>
>>>>> >> Sent from my iPhone
>>>>> >>
>>>>> >> On May 23, 2018, at 4:17 AM, Gaurav Dawra <[email-masked]
>>>>> br/>>> <mailto:gdawra.ietf@@gmail.com>> wrote:
>>>>> >>
>>>>> >>> Hi Martin,
>>>>> >>>
>>>>> >>> Thanks for the review. Any further comments here ? I will post the
>>>>> br/>>>> new version soon. <
>>>>> >>>
>>>>> >>> Cheers,
>>>>> >>>
>>>>> >>> Gaurav
>>>>> >>>
>>>>> >>> Sent from my iPhone
>>>>> >>>
>>>>> >>> On May 16, 2018, at 7:44 PM, Gaurav Dawra <[email-masked]
>>>>> br/>>>> <mailto:gdawra.ietf@@gmail..com <http://gmail.com>>> wrote:
>>>>> >>>
>>>>> >>>> Hi Martin,
>>>>> >>>>
>>>>> >>>> Apologies from my end we had few internal authors conversations
>>>>> on br/>>>>> the points you have raised. <
>>>>> >>>>
>>>>> >>>> Please find below my response. I will be happy to discuss
>>>>> further, br/>>>>> if needed. <
>>>>> >>>>
>>>>> >>>> <Gaurav> inline...
>>>>> >>>>
>>>>> >>>>> On Apr 9, 2018, at 7:58 AM, Martin Stiemerling <
>>>>> [email-masked] br/>>>>>> <mailto:[email-masked]>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi Gaurav,
>>>>> >>>>>
>>>>> >>>>> This got lost on my end, sorry for this. The filter just moved
>>>>> br/>>>>>> these messages out of my sight... :-/
>>>>> >>>>>
>>>>> >>>>> Am 15.02.18 um 05:47 schrieb Gaurav Dawra:
>>>>> >>>>>> Hey Martin,
>>>>> >>>>>> Sorry for late reply. Please see some comments inline[Gaurav]
>>>>> >>>>>>> On Jan 9, 2018, at 2:25 PM, Martin Stiemerling br/>>>>>>>>
>>>>> <mls.ietf@@gmail.com <mailto:[email-masked]> br/>>>>>>>>; <mailto:
>>>>> [email-masked]>> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Hi all,
>>>>> >>>>>>>
>>>>> >>>>>>> I've reviewed this document as part of the transport area
>>>>> review br/>>>>>>>> team's ongoing effort to review key IETF documents.
>>>>> These br/>>>&gtt;>>>> comments were written primarily for the transport
>>>>> area directors, br/>>>>>>>> but are copied to the doocument's authors for
>>>>> their information br/>>>>>>>&> and to allow them to address any issues
>>>>> raised. When done at the
>>>>> >>>>>>> time of IETF Last Call, the authors should consider this
>>>>> review br/>>>>>>>> together with any other last-call comments they receive.
>>>>> Please br/>>>&>>>>> always CC tsv-art@… if you reply to or forward
>>>>> this review.
>>>>> >>>>>>>
>>>>> >>>>>>> Summary:
>>>>> >>>>>>> This draft has serious issues in Section 7..1, 7.2 and in one
>>>>> part br/>>>>>>>> of Secction3, described in the review, and needs to be
>>>>> rethought. br/>>&>>>>>> The other sections are good AFAIK.
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> Technicals:
>>>>> >>>>>>> The overall draft looks ok, but the three points below look
>>>>> br/>>>>>>>> strange and need a fix before publication IMHO:
>>>>> >>>>>>>
>>>>> >>>>>>> Both Sections, 7.1. and 7.2., are describing ideas, but not
>>>>> well br/>>>>>>>> proven funcationality and not even safe to use
>>>>> functionality. br/>>>&>>>>> Both are some sort discussing that different
>>>>> paths in the network br/>>>>>>>> could be used by the eend host traffic.
>>>>> This sounds pretty much br/>>>>>>&gtt;> like the Path Aware Networking
>>>>> Proposed Research Group br/>&gtt;>>>>>> (https://irtf.org/panrg) and
>>>>> hints to the fact that there is no br/>>>>>>>> commonly understannd and
>>>>> accepted engineering solution in this space.
>>>>> >>>>>>>
>>>>> >>>>>>> Section 7.1:
>>>>> >>>>>>> [KANDULA04] is a really old reference that hasn't been
>>>>> followed br/>>>>>>>> up iin recent times and even worse there is no
>>>>> evidence that this br/>&gtt;>>>>>> is going to work good enough or stable
>>>>> enough under real Internet br/>>>>>>>> traffic. Additioonally, it is more
>>>>> than unclear how any modern TCP br/>>>>&ggt;>>> implementation will react
>>>>> to this
>>>>> >>>>>> [Gaurav] Will get back on this.
>>>>> >>>>>
>>>>> >>>>> I will reply to the other email dicussing this.
>>>>> >>>> <Gaurav> I have replied to other thread.
>>>>> >>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> Section 7.2:
>>>>> >>>>>>> This section describes an idea without detailing too much
>>>>> about br/>>>>>>>> any furtther aspects. Further it changes the commonly
>>>>> accepted br/>>>;>>>>> notion of what an end host can do with the network.
>>>>> At best this br/>>>>>>>> would require a good ddefinition of what an end
>>>>> host in your br/>>>>>>>&ggt; setting is, e.g., a highly modified piece of
>>>>> (at least) software
>>>>> >>>>>>> that usually not found in OS availble on the market (yet?)
>>>>> >>>>>>> Further communicating instantaneous path characteristics to a
>>>>> br/>>>>>>>> central point is potentially a bad idea, as the data is already
>>>>> br/>>>;>>>>> outdated when reported by any node.
>>>>> >>>>>> [Gaurav] I believe Authors are trying to highlight that Host
>>>>> which br/>>>>>>> is defineed in br/>>>>>>> (
>>>>> https://tools.ietf.org/html/draftt-ietf-spring-segment-routing-15)
>>>>> br/>>>>>>> can innfluence the traffic based on the Calculations locally or
>>>>> br/>>&gtt;>>>> jointly with the controller. Implementations can decide how
>>>>> much br/>>>>>>> Centralized -vs- localized coontrol is allowed at Host
>>>>> based on br/>>>>>>> perfoormance data collection.
>>>>> >>>>>
>>>>> >>>>> Performance data collection (monitoring?) isn't crucial when it
>>>>> br/>>>>>> comes to timely (actuaally real-time) reaction. However, this
>>>>> br/>>>>>> secttion isn't just about performance data collection as it is
>>>>> about br/>>>>>>> "Performance-aware routing" this seems to try to interact
>>>>> in br/>>>>>> real-time with the network behhavior of TCP. This isn't even
>>>>> close br/>>>>>> to acceeptable.
>>>>> >>>>>
>>>>> >>>>> I would be ok to say that it is useful to collect performance
>>>>> data br/>>>>>> for offline analysis and improvement of the data network.
>>>>> However, br/>>>>>&ggt; this is at completely different magnitues of time
>>>>> scales.
>>>>> >>>>>
>>>>> >>>>> I would recommend to remove the TCP part from this section
>>>>> entirely.
>>>>> >>>> <Gaurav>Ack, will update in next rev:
>>>>> >>>>
>>>>> >>>> Section will read like this:
>>>>> >>>>
>>>>> >>>> ;
>>>>> >>>> /Knowing the path associated with flows/packets, the end host
>>>>> may/
>>>>> >>>> /deduce certain characteristics of the path on its own, and/
>>>>> >>>> /additionally use the information supplied with path information/
>>>>> >>>> /pushed from the controller or received via pull request. The
>>>>> host/
>>>>> >>>> /may further share its path observations with the centralized
>>>>> agent,/
>>>>> >>>> /so that the latter may keep up-to-date network health map to
>>>>> assist/
>>>>> >>>> /other hosts with this information./
>>>>> >>>> //
>>>>> >>>> /For example, an application A.1 at HostA may pin a flow
>>>>> destined/
>>>>> >>>> /to HostZ via Spine node Node5 using label stack {16005, 16011}.
>>>>> The/
>>>>> >>>> /application A.1 may collect information on packet loss, deduced
>>>>> from/
>>>>> >>>> /Other offline mechanisms. [There are some pingMesh mechanisms
>>>>> which /
>>>>> >>>> /Can be used here]/
>>>>> >>>> /Through these mechanisms information to a centralized agent,
>>>>> e.g./
>>>>> >>>> /after a flow completes, or periodically for longer lived flows./
>>>>> >>>> /Next, using both local and/or global performance data,
>>>>> application/
>>>>> >>>> /A.1 as well as other applications sharing the same resources in
>>>>> the/
>>>>> >>>> /DC fabric may pick up the best path for the new flow, or update
>>>>> an/
>>>>> >>>> /existing path (e.g.: when informed of congestion on an existing/
>>>>> >>>> /path)./
>>>>> >>>> ;
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> Section 3, 3rd bullet point:
>>>>> >>>>>>> It is the foundation of TCP that the network is regarded as a
>>>>> br/>>>>>>>> black box and that you infer from the transmission of packets
>>>>> br/>>>>;>>>> what the current state of the network path is. Inferring
>>>>> network br/>>>>>>>> path metrics (you mention SRTT, MSS, CWND ) is a bad
>>>>> idea, as br/>>>>>>>>; this would required that all paths exhibit this and
>>>>> if not what br//>>>>>>>> is going to happen?
>>>>> >>>>>>> It could be an interesting research field to change many
>>>>> points br/>>>>>>>> in TCP'ss behavior, but this once again points to the
>>>>> fact that br/>>>&>>>>> this not the IETF works but IRTF or elsewhere.
>>>>> >>>>>> [Gaurav] Martin, Authors are trying to suggest that TCP is
>>>>> rightly br/>>>>>>> treating Network as Black Box. Authors are implying per
>>>>> path br/>>>>;>>> performance metrics as not cached. Is there some change in
>>>>> text br/>>>>>>> you are suggesting??
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> I would recommend to remove the 3rd bullet point completey. TCP
>>>>> br/>>>>>> isn't the place to rememmber "good" or "bad" paths. This is
>>>>> br/>>>>>> something the network could decide, e.g., rerouting TCP flows
>>>>> br/>&ggt;>>>> within the network or changing the forwarding path in the
>>>>> network br/>>>>>> for particular flows (if it is not routed).
>>>>> >>>>
>>>>> >>>> <Gaurav> Ack, after discussion, we will remove the Section 3 -
>>>>> 3rd br/>>>>> bullet point. Willl update in next rev - coming shortly.
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> Kind regards,
>>>>> >>>>>
>>>>> >>>>>  Martin
>>>>> >>>>>
>>>>> >>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Tsv-art mailing [email-masked]://www.ietf.org/mailman/listinfo/tsv-art
>>>
>>>
>>>
>>
>>

---

From: Mirja Kühlewind (IETF) <[email-masked]>
Date: Thu, 18 Oct 2018 12:25:33 +0200

Hi Gaurav,

thanks for your edits but unfortunately they don't address my concerns.

Saying that a flowlet is per 5-tuple is just a clarification of what a 
flowlet is but doeen't change anything about what the text says.

Please consider removing section 7 entirely because this is really 
beyond the doc of this document and would need further discussion in one 
or multiple separate documents. Why is that not an option for the 
authors? Why do you think that section 7 is important for this document?

Mirja

On 16.10.18 07:45, Gaurav Dawra wrote:
> Hi MIrja,
>
> I have posted a new version. Please do let me know if we need to 
> discuss further. We can do it over the phone.
>
> Cheers,
>
> Gaurav
>
>
> On Mon, Sep 3, 2018 at 4:29 AM Gaurav Dawra <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Mirja,
>
>     On Thu, Aug 9, 2018 at 8:30 AM Mirja Kühlewind
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         Hi Gaurav,
>
>         please see inline.
>
>>         Sorry for the long delay. I was traveling constantly since
>>         IETF and bit lost in my mailbox and discussion with Authors.
>>         Please see my response inline[Gaurav]
>>
>>         _
>>         I think with your changes you addressed explicit problems
>>         Martin called out, however, I still have high level concerns
>>         about these sections as they are mostly giving speculative
>>         recommendation which are not clear to me to work in practice.
>>
>>         Regarding section 7.1, you say _
>>         "A flowlet is defined as a burst of packets from the same
>>         flow followed by an idle interval."
>>         _but then you say _
>>         "...then the application can break the elephant flow F into
>>         flowlets F1, F2, F3, F4..."
>>
>>         This sounds like you would just divide an elephant flow
>>         mostly randomly into flowlets which can interact badly with
>>         the congestion control. If you actually have chunks of data
>>         that are transmitted with large enough idle interval in
>>         between (as you define flowlets in the first sentence), that
>>         is not an elephant flow anymore and it will not help you to
>>         "spread the load of the elephant flow through all the ECMP
>>         paths". In summary I actually don't see how the concept of
>>         flowlets can be helpful to address the problem you have at
>>         all, and I still advise you to remove section 7.1 entirely.
>>
>>         _[Gaurav] Hi Mirja, Thanks for the review. _The proposal here
>>         is no different that current ECMP hashing at flowlet level.
>>         The only difference which is being pointed out here is that
>>         if we use SR, we could leverage on the ability of be aware of
>>         multiple disjoint paths rather than the hashing. It’s pins
>>         the flowlets to particular paths which is basic SR operations.
>>
>
>         Yes the problem is that we usually don't recommend ECMP
>         hashing on flowlet level because it can interact badly with
>         the end-end mechanisms of that flow. As I said, I don't see
>         how the notion of flowlets help you problem and therefore I
>         still recommend to remove that paragraph.
>         [Gaurav2] OK. It took sometime to get to consensus with
>         authors. Will update the text to use 5-tuple flows instead of
>         flowlets. Would that suffice? I will update the text shortly.
>>
>>         _
>>         Regarding section 7.2, I also still skeptical about any
>>         benefits that can be achieved. Given you are in a data
>>         center, the controller should already know about static
>>         parameters such as the maximum bandwidth per link.
>>
>>         _For dynamic parameters, e.g. like loss rate, measuring them
>>         on a per-flow bases is the wrong thing to do. What I mean is
>>         you can ask a router about the average loss rate that it
>>         observes and that might give you some valuable, however, if
>>         you ask a TCP flow about the average loss rate the answer
>>         will mainly depend on the congestion controller used and the
>>         currently available bandwidth, which is a very dynamic
>>         property and not a link characteristic. So this information
>>         is not usable for performance aware routing. A flow could
>>         give you information about the observed RTT (min/max) on a
>>         certain path, or the maximum available bandwidth on a path,
>>         but as I said, given you are in a data center environment
>>         these are information that the controller already should have
>>         anyway.
>>
>>         _[Gaurav] They are two separate mechanisms. _Most DCs have
>>         some sort of data-plane/ECMP aware tracing mechanism to
>>         detect the loss/delays and can be combined with Application
>>         back-off to detect issue. All this section is suggesting is
>>         that SR can be used to pin the path to particular set of ECMP
>         _This is not quite what the text says. If that is the
>         statement you want to make, that is fine but then you don't
>         need to talk about performance aware routing at all.
>
>     [Gaurav2] I will update the text here with statement i mentioned
>     above. IMHO, it's performance aware routing wrt to end-host traffic.
>
>
>>         _
>>         Your example with detecting a faulty path due to losses does
>>         not work with TCP as you never know if these loses are due to
>>         a problem on the path, self-induced or by a competing flow.
>>         And even if you don't use TCP and e.g. send constant bit rate
>>         traffic, there may be a large number of competing TCP flows
>>         that can induce the loses. Try to steer traffic "away" on a
>>         time-scale that is slower than TCP dynamics or even your flow
>>         dynamic (when flows start or end) in case you have a lot of
>>         very short flow, in the best case doesn't work and in the
>>         worst case leads to oscillation.
>>
>>         _[Gaurav] As _I said above, there are other mechanisms to
>>         detect loss and trace the path on which loss is seen. This is
>>         a common mechanism used in MSDCs.
>>
>         _I think this is beyond the scope of the document.
>
>>         _
>>         _
>>         _I am happy to discuss further over the phone to try to
>>         explain the thought process. I will also do check again with
>>         Authors to update the text or something else based on our
>>         conversation.
>>
>
>         Maybe see if some update can be made to the text first and
>         then we can have another discussion if needed.
>         [Gaurav2] Sounds good. Will update the text shortly and then
>         we can discuss if needed.
>
>
>     Cheers,
>
>     Gaurav
>
>>         _
>>
>>         _
>>
>>         _Cheers,
>>
>>         _
>>
>>         _Gaurav
>>
>>         If you want to make TCP use for handover situation where one
>>         path might go away or become unusable, it's best to use
>>         Multipath TCP (with coupled congestion control) instead
>>         because that works on the right time scale. Again, I don't
>>         think this is a use case for SR and I would recommend to
>>         Mirja _
>>         _
>>         On Mon, Jul 16, 2018 at 11:08 PM, Gaurav Dawra
>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>             Hi Mirja,
>>
>>             On Jul 5, 2018, at 10:06 AM, Mirja Kühlewind
>>             <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>>             Hi Gaurav,
>>>
>>>             sorry for my very long delay but this got somehow a bit
>>>             lost in my mailbox.
>>>
>>>             I think with your changes you addressed explicit
>>>             problems Martin called out, however, I still have high
>>>             level concerns about these sections as they are mostly
>>>             giving speculative recommendation which are not clear to
>>>             me to work in practice.
>>>
>>>             "A flowlet is defined as a burst of packets from the
>>>             same flow followed by an idle interval."
>>>             but then you say
>>>             "...then the application can break the elephant flow F
>>>             into flowlets F1, F2, F3, F4..."
>>>
>>>             This sounds like you would just divide an elephant flow
>>>             mostly randomly into flowlets which can interact badly
>>>             with the congestion control. If you actually have chunks
>>>             of data that are transmitted with large enough idle
>>>             interval in between (as you define flowlets in the first
>>>             sentence), that is not an elephant flow anymore and it
>>>             will not help you to "spread the load of the elephant
>>>             flow through all the ECMP paths". In summary I actually
>>>             don't see how the concept of flowlets can be helpful to
>>>             address the problem you have at all, and I still advise
>>>             you to remove section 7.1 entirely.
>>>
>>>             Regarding section 7.2, I also still skeptical about any
>>>             benefits that can be achieved. Given you are in a data
>>>             center, the controller should already know about static
>>>             parameters such as the maximum bandwidth per link. For
>>>             dynamic parameters, e.g. like loss rate, measuring them
>>>             on a per-flow bases is the wrong thing to do. What I
>>>             mean is you can ask a router about the average loss rate
>>>             that it observes and that might give you some valuable,
>>>             however, if you ask a TCP flow about the average loss
>>>             rate the answer will mainly depend on the congestion
>>>             controller used and the currently available bandwidth,
>>>             which is a very dynamic property and not a link
>>>             characteristic. So this information is not usable for
>>>             performance aware routing. A flow could give you
>>>             information about the observed RTT (min/max) on a
>>>             certain path, or the maximum available bandwidth on a
>>>             path, but as I said, given you are in a data center
>>>             environment these are information that the controller
>>>             already should have anyway.
>>>
>>>             Your example with detecting a faulty path due to losses
>>>             does not work with TCP as you never know if these loses
>>>             are due to a problem on the path, self-induced or by a
>>>             competing flow. And even if you don't use TCP and e.g.
>>>             send constant bit rate traffic, there may be a large
>>>             number of competing TCP flows that can induce the loses.
>>>             Try to steer traffic "away" on a time-scale that is
>>>             slower than TCP dynamics or even your flow dynamic (when
>>>             flows start or end) in case you have a lot of very short
>>>             flow, in the best case doesn't work and in the worst
>>>             case leads to oscillation.
>>>
>>>             If you want to make TCP use for handover situation where
>>>             one path might go away or become unusable, it's best to
>>>             use Multipath TCP (with coupled congestion control)
>>>             instead because that works on the right time scale.
>>>             Again, I don't think this is a use case for SR and I
>>>             would recommend to remove the section entirely.
>>>
>>>             Mirja
>>>
>>>
>>>>             On Mon, Jun 18, 2018 at 5:13 PM, Gaurav Dawra
>>>>             <[email-masked] <mailto:[email-masked]>>
>>>>             wrote:
>>>>
>>>>                 Hi Alvaro, Mirja
>>>>
>>>>                 On Jun 12, 2018, at 7:06 AM, Gaurav Dawra
>>>>                 <[email-masked]
>>>>                 <mailto:[email-masked]>> wrote:
>>>>
>>>>>                 Hi Mirja,
>>>>>
>>>>>                 Friendly Reminder...Could you please also advice
>>>>>                 if there is anything further to DISCUSS on this
>>>>>                 document which was also related to TCP updates below?
>>>>>
>>>>>                 Cheers,
>>>>>
>>>>>                 Gaurav
>>>>>
>>>>>                 On Thu, Jun 7, 2018 at 9:02 AM, Alvaro Retana
>>>>>                 <[email-masked]
>>>>>                 <mailto:[email-masked]>> wrote:
>>>>>
>>>>>                     Thanks Martin!
>>>>>
>>>>>
>>>>>                     On June 6, 2018 at 3:14:45 PM, Martin
>>>>>                     Stiemerling ([email-masked]
>>>>>                     <mailto:[email-masked]>) wrote:
>>>>>
>>>>>>                     Hi Alvaro, all,
>>>>>>
>>>>>>                     > br/>> Gaurav just posted a revision. Please
>>>>>>                     takke a look and let us know if br/>> the
>>>>>>                     changes address your concerrns or not.
>>>>>>                     > br/>>
>>>>>>                     > br/>> On May 25, 2018 at 12:08:46 PM,
>>>>>>                     Gaurav Dawra (([email-masked]
>>>>>>                     <mailto:[email-masked]> br/>>
>>>>>>                     <mailto:gdawra.ietf@
>>>>>>                     <mailto:gdawra.ietf@>@gmail.com
>>>>>>                     <http://gmail.com>>) wrote:
>>>>>>                     > br/>>> Hi Martin, <
>>>>>>                     >>
>>>>>>                     >> Thanks for review. I will post the new
>>>>>>                     version. Hopefully, it will br/>>> address
>>>>>>                     all your comments and we can close thhis review.
>>>>>>                     >>
>>>>>>                     <mailto:[email-masked]> br/>>>
>>>>>>                     <mailto:gdawra.ietf@
>>>>>>                     <mailto:gdawra.ietf@>@gmail.com
>>>>>>                     <http://gmail.com>>> wrote:
>>>>>>                     >>
>>>>>>                     >>> Hi Martin,
>>>>>>                     >>>
>>>>>>                     >>> Thanks for the review. Any further comments
>>>>>>                     here ? I will post the br/>>>> new version
>>>>>>                     soon. <
>>>>>>                     >>>
>>>>>>                     >>> Cheers,
>>>>>>                     >>>
>>>>>>                     >>> Gaurav
>>>>>>                     >>>
>>>>>>                     >>> Sent from my iPhone
>>>>>>                     >>>
>>>>>>                     <mailto:[email-masked]> br/>>>>
>>>>>>                     <mailto:gdawra.ietf@
>>>>>>                     <mailto:gdawra.ietf@>@gmail..com
>>>>>>                     <http://gmail.com>>> wrote:
>>>>>>                     >>>
>>>>>>                     >>>> Hi Martin, 
>>>>>>                     >>>> 
>>>>>>                     >>>> Apologies from my end we had few internal
>>>>>>                     authors conversations on br/>>>>> the points
>>>>>>                     you have raised. <
>>>>>>                     >>>> 
>>>>>>                     >>>> Please find below my response. I will be happy to discuss further, br/>>>>> if needed. <
>>>>>>                     >>>> 
>>>>>>                     >>>> <Gaurav> inline... 
>>>>>>                     >>>>
>>>>>>                     >>>>> On Apr 9, 2018, at 7:58 AM, Martin Stiemerling <[email-masked]
>>>>>>                     <mailto:[email-masked]> br/>>>>>>
>>>>>>                     <mailto:[email-masked]
>>>>>>                     <mailto:[email-masked]>>> wrote:
>>>>>>                     >>>>> 
>>>>>>                     >>>>> Hi Gaurav, 
>>>>>>                     >>>>> 
>>>>>>                     >>>>> This got lost on my end, sorry for this. The
>>>>>>                     filter just moved br/>>>>>> these messages
>>>>>>                     out of my sight... :-/
>>>>>>                     >>>>>
>>>>>>                     >>>>>>> On Jan 9, 2018, at 2:25 PM, Martin
>>>>>>                     Stiemerling br/>>>>>>>> <mls.ietf@@gmail.com
>>>>>>                     <http://gmail.com> <mailto:[email-masked]
>>>>>>                     <mailto:[email-masked]>> br/>>>>>>>>;
>>>>>>                     <mailto:[email-masked]
>>>>>>                     <mailto:[email-masked]>>> wrote:
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> Hi all, 
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> I've reviewed this document as part of the
>>>>>>                     transport area review br/>>>>>>>> team's
>>>>>>                     ongoing effort to review key IETF documents.
>>>>>>                     These br/>>>&gtt;>>>> comments were written
>>>>>>                     primarily for the transport area directors,
>>>>>>                     br/>>>>>>>> but are copied to the doocument's
>>>>>>                     authors for their information br/>>>>>>>&>
>>>>>>                     and to allow them to address any issues
>>>>>>                     raised. When done at the
>>>>>>                     >>>>>>> time of IETF Last Call, the authors should consider this review br/>>>>>>>> together
>>>>>>                     with any other last-call comments they
>>>>>>                     receive. Please br/>>>&>>>>> always CC
>>>>>>                     tsv-art@… if you reply to or forward this
>>>>>>                     review.
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> Summary: 
>>>>>>                     >>>>>>> This draft has serious issues in Section 7..1, 7.2 and in one part br/>>>>>>>> of
>>>>>>                     Secction3, described in the review, and needs
>>>>>>                     to be rethought. br/>>&>>>>>> The other
>>>>>>                     sections are good AFAIK.
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> Technicals: 
>>>>>>                     >>>>>>> The overall draft looks ok, but the three points below look br/>>>>>>>> strange and
>>>>>>                     need a fix before publication IMHO:
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> Both Sections, 7.1. and 7.2., are describing ideas, but not well br/>>>>>>>>
>>>>>>                     proven funcationality and not even safe to
>>>>>>                     use functionality. br/>>>&>>>>> Both are some
>>>>>>                     sort discussing that different paths in the
>>>>>>                     network br/>>>>>>>> could be used by the eend
>>>>>>                     host traffic. This sounds pretty much
>>>>>>                     br/>>>>>>&gtt;> like the Path Aware
>>>>>>                     Networking Proposed Research Group
>>>>>>                     br/>&gtt;>>>>>> (https://irtf.org/panrg) and
>>>>>>                     hints to the fact that there is no
>>>>>>                     br/>>>>>>>> commonly understannd and accepted
>>>>>>                     engineering solution in this space.
>>>>>>                     >>>>>>> 
>>>>>>                     >>>>>>> Section 7.1: 
>>>>>>                     >>>>>>> [KANDULA04] is a really old reference that hasn't been followed br/>>>>>>>> up iin
>>>>>>                     recent times and even worse there is no
>>>>>>                     evidence that this br/>&gtt;>>>>>> is going
>>>>>>                     to work good enough or stable enough under
>>>>>>                     real Internet br/>>>>>>>> traffic.
>>>>>>                     Additioonally, it is more than unclear how
>>>>>>                     any modern TCP br/>>>>&ggt;>>> implementation
>>>>>>                     will react to this
>>>>>>                     >>>>>>> This section describes an idea without
>>>>>>                     detailing too much about br/>>>>>>>> any
>>>>>>                     furtther aspects. Further it changes the
>>>>>>                     commonly accepted br/>>>;>>>>> notion of what
>>>>>>                     an end host can do with the network. At best
>>>>>>                     this br/>>>>>>>> would require a good
>>>>>>                     ddefinition of what an end host in your
>>>>>>                     br/>>>>>>>&ggt; setting is, e.g., a highly
>>>>>>                     modified piece of (at least) software
>>>>>>                     >>>>>>> Further communicating instantaneous path
>>>>>>                     characteristics to a br/>>>>>>>> central
>>>>>>                     point is potentially a bad idea, as the data
>>>>>>                     is already br/>>>;>>>>> outdated when
>>>>>>                     reported by any node.
>>>>>>                     >>>>>> [Gaurav] I believe Authors are trying to highlight that Host which br/>>>>>>> is
>>>>>>                     defineed in br/>>>>>>>
>>>>>>                     (https://tools.ietf.org/html/draftt-ietf-spring-segment-routing-15)
>>>>>>                     br/>>>>>>> can innfluence the traffic based
>>>>>>                     on the Calculations locally or br/>>&gtt;>>>>
>>>>>>                     jointly with the controller. Implementations
>>>>>>                     can decide how much br/>>>>>>> Centralized
>>>>>>                     -vs- localized coontrol is allowed at Host
>>>>>>                     >>>>> Performance data collection (monitoring?) isn't crucial when it br/>>>>>> comes to timely
>>>>>>                     (actuaally real-time) reaction. However, this
>>>>>>                     br/>>>>>> secttion isn't just about
>>>>>>                     performance data collection as it is about
>>>>>>                     br/>>>>>>> "Performance-aware routing" this
>>>>>>                     seems to try to interact in br/>>>>>>
>>>>>>                     real-time with the network behhavior of TCP.
>>>>>>                     This isn't even close br/>>>>>> to acceeptable.
>>>>>>                     >>>>> 
>>>>>>                     >>>>> I would be ok to say that it is useful to collect performance data br/>>>>>> for offline
>>>>>>                     analysis and improvement of the data network.
>>>>>>                     However, br/>>>>>&ggt; this is at completely
>>>>>>                     different magnitues of time scales.
>>>>>>                     >>>>>
>>>>>>                     >>>>> I would recommend to remove the TCP part from this section entirely.
>>>>>>                     >>>> /Knowing the path associated with flows/packets, the end host may/
>>>>>>                     >>>> /pushed from the controller or received via pull request. The host/
>>>>>>                     >>>> /may further share its path observations with the centralized agent,/
>>>>>>                     >>>> /so that the latter may keep up-to-date network health map to assist/
>>>>>>                     >>>> /For example, an application A.1 at HostA may pin a flow destined/
>>>>>>                     >>>> /to HostZ via Spine node Node5 using label stack {16005, 16011}. The/
>>>>>>                     >>>> /application A.1 may collect information on packet loss, deduced from/
>>>>>>                     >>>> /Other offline mechanisms. [There are some pingMesh mechanisms which /
>>>>>>                     >>>> /Can be used here]/ 
>>>>>>                     >>>> /Through these mechanisms information to a centralized agent, e.g./
>>>>>>                     >>>> /Next, using both local and/or global performance data, application/
>>>>>>                     >>>> /A.1 as well as other applications sharing the same resources in the/
>>>>>>                     >>>> /DC fabric may pick up the best path for the new flow, or update an/
>>>>>>                     >>>>>>> It is the foundation of TCP that the network
>>>>>>                     is regarded as a br/>>>>>>>> black box and
>>>>>>                     that you infer from the transmission of
>>>>>>                     packets br/>>>>;>>>> what the current state
>>>>>>                     of the network path is. Inferring network
>>>>>>                     br/>>>>>>>> path metrics (you mention SRTT,
>>>>>>                     MSS, CWND ) is a bad idea, as br/>>>>>>>>;
>>>>>>                     this would required that all paths exhibit
>>>>>>                     this and if not what br//>>>>>>>> is going to
>>>>>>                     happen?
>>>>>>                     >>>>>>> It could be an interesting research field to change many points br/>>>>>>>> in TCP'ss
>>>>>>                     behavior, but this once again points to the
>>>>>>                     >>>>>> [Gaurav] Martin, Authors are trying to suggest that TCP is rightly br/>>>>>>> treating
>>>>>>                     Network as Black Box. Authors are implying
>>>>>>                     per path br/>>>>;>>> performance metrics as
>>>>>>                     not cached. Is there some change in text
>>>>>>                     br/>>>>>>> you are suggesting??
>>>>>>                     >>>>> 
>>>>>>                     >>>>> 
>>>>>>                     >>>>> I would recommend to remove the 3rd bullet point completey. TCP br/>>>>>> isn't the place to
>>>>>>                     rememmber "good" or "bad" paths. This is
>>>>>>                     br/>>>>>> something the network could decide,
>>>>>>                     e.g., rerouting TCP flows br/>&ggt;>>>>
>>>>>>                     within the network or changing the forwarding
>>>>>>                     path in the network br/>>>>>> for particular
>>>>>>                     flows (if it is not routed).
>>>>>>                     >>>> 
>>>>>>                     >>>> <Gaurav> Ack, after discussion, we will remove the Section 3 - 3rd br/>>>>> bullet
>>>>>>                     point. Willl update in next rev - coming
>>>>>>                     shortly.
>>>>>>                     >>>> 
>>>>>>                     >>>>> 
>>>>>>                     >>>>> Kind regards, 
>>>>>>                     >>>>> 
>>>>>>                     >>>>>  Martin 
>>>>>>                     >>>>> 
>>>>>>                     >>>> 
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>             Tsv-art mailing list
>>>>             [email-masked]  <mailto:[email-masked]>
>>>>             https://www.ietf.org/mailman/listinfo/tsv-art
>>>
>>
>
>
> Tsv-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tsv-art

---

From: Gaurav Dawra <[email-masked]>
Date: Mon, 29 Oct 2018 08:12:01 -0700

Hi Mirja,

Thanks again for working with us on these comments.

Since we have done few revisions to address these comments. I am pasting section 7.1 revised text. Please see if this makes sense. I will make the necessary edits.

“Large, long-lived "elephant" flows may affect each other’s performance or that of smaller, short-lived "mouse" flows when left to normal per-flow ECMP load-sharing. The host which is originating an “elephant” flow may share its application observations with a centralized agent by indicating its bandwidth requirements and the destination for the flow, that enables the latter to keep up-to-date network bandwidth demand maps for such flows correlated with the actual utilization of the paths in the network. The end host may receive updated information from the centralized agent, published via external mechanisms, of specific paths with their bandwidth availability on which to steer its flow.
 
For example, an application A.1 is informed about an explicit paths to Z {16006, 16011} which has bandwidth availability such as not to degrade other flows. The centralized agent may similarly pin “elephant” flows on other disjoint explicit paths. Over a period of time, or once the “elephant” flow is gone (as reported by the application), then the centralized agent updates the hosts to revert back to their normal per-flow ECMP based hashing for load-sharing. This allows for solving the "elephant flow" problem in the data-center and avoiding link imbalances.”

Gaurav

Sent from my iPhone

> On Oct 18, 2018, at 3:25 AM, Mirja Kühlewind (IETF) <[email-masked]> wrote:
> 
> Hi Gaurav,
> 
> Saying that a flowlet is per 5-tuple is just a clarification of what a flowlet is but doeen't change anything about what the text says.
> 
> Please consider removing section 7 entirely because this is really beyond the doc of this document and would need further discussion in one or multiple separate documents. Why is that not an option for the authors? Why do you think that section 7 is important for this document?
> 
> Mirja
> 
> 
>> I have posted a new version. Please do let me know if we need to discuss further. We can do it over the phone.
>> 
>> Cheers,
>> 
>> Gaurav
>> 
>> 
>>>> On Thu, Aug 9, 2018 at 8:30 AM Mirja Kühlewind <[email-masked]> wrote:
>>>> Hi Gaurav,
>>>> 
>>>> please see inline.
>>>> 
>>>>> Sorry for the long delay. I was traveling constantly since IETF and bit lost in my mailbox and discussion with Authors. Please see my response inline[Gaurav]
>>>>> 
>>>>> I think with your changes you addressed explicit problems Martin called out, however, I still have high level concerns about these sections as they are mostly giving speculative recommendation which are not clear to me to work in practice.
>>>>> 
>>>>> "A flowlet is defined as a burst of packets from the same flow followed by an idle interval."
>>>>> but then you say
>>>>> "...then the application can break the elephant flow F into flowlets F1, F2, F3, F4..."
>>>>> 
>>>>> This sounds like you would just divide an elephant flow mostly randomly into flowlets which can interact badly with the congestion control. If you actually have chunks of data that are transmitted with large enough idle interval in between (as you define flowlets in the first sentence), that is not an elephant flow anymore and it will not help you to "spread the load of the elephant flow through all the ECMP paths". In summary I actually don't see how the concept of flowlets can be helpful to address the problem you have at all, and I still advise you to remove section 7.1 entirely.
>>>>> 
>>>>> [Gaurav] Hi Mirja, Thanks for the review. The proposal here is no different that current ECMP hashing at flowlet level. The only difference which is being pointed out here is that if we use SR, we could leverage on the ability of be aware of multiple disjoint paths rather than the hashing. It’s pins the flowlets to particular paths which is basic SR operations.
>>>>> 
>>>> 
>>>> Yes the problem is that we usually don't recommend ECMP hashing on flowlet level because it can interact badly with the end-end mechanisms of that flow. As I said, I don't see how the notion of flowlets help you problem and therefore I still recommend to remove that paragraph.
>>>> [Gaurav2] OK. It took sometime to get to consensus with authors. Will update the text to use 5-tuple flows instead of flowlets. Would that suffice? I will update the text shortly.
>>>>> 
>>>>> Regarding section 7.2, I also still skeptical about any benefits that can be achieved. Given you are in a data center, the controller should already know about static parameters such as the maximum bandwidth per link.
>>>>> 
>>>>> For dynamic parameters, e.g. like loss rate, measuring them on a per-flow bases is the wrong thing to do. What I mean is you can ask a router about the average loss rate that it observes and that might give you some valuable, however, if you ask a TCP flow about the average loss rate the answer will mainly depend on the congestion controller used and the currently available bandwidth, which is a very dynamic property and not a link characteristic. So this information is not usable for performance aware routing. A flow could give you information about the observed RTT (min/max) on a certain path, or the maximum available bandwidth on a path, but as I said, given you are in a data center environment these are information that the controller already should have anyway.
>>>>> 
>>>>> [Gaurav] They are two separate mechanisms. Most DCs have some sort of data-plane/ECMP aware tracing mechanism to detect the loss/delays and can be combined with Application back-off to detect issue. All this section is suggesting is that SR can be used to pin the path to particular set of ECMP paths instead of relying on ECMP hashing.
>>>>> 
>>>> 
>>>> This is not quite what the text says. If that is the statement you want to make, that is fine but then you don't need to talk about performance aware routing at all.
>>> [Gaurav2] I will update the text here with statement i mentioned above. IMHO, it's performance aware routing wrt to end-host traffic.   
>>>> 
>>>>> 
>>>>> Your example with detecting a faulty path due to losses does not work with TCP as you never know if these loses are due to a problem on the path, self-induced or by a competing flow. And even if you don't use TCP and e.g. send constant bit rate traffic, there may be a large number of competing TCP flows that can induce the loses. Try to steer traffic "away" on a time-scale that is slower than TCP dynamics or even your flow dynamic (when flows start or end) in case you have a lot of very short flow, in the best case doesn't work and in the worst case leads to oscillation.
>>>>> 
>>>>> [Gaurav] As I said above, there are other mechanisms to detect loss and trace the path on which loss is seen. This is a common mechanism                               used in MSDCs.
>>>>> 
>>>>> I am happy to discuss further over the phone to try to explain the thought process. I will also do check again with Authors to update the text or something else based on                               our conversation.
>>>>> 
>>>> 
>>>> Maybe see if some update can be made to the text first and then we can have another discussion if needed.
>>>> [Gaurav2] Sounds good. Will update the text shortly and then we can discuss if needed.
>>> 
>>> Cheers,
>>> 
>>> Gaurav 
>>>>>  
>>>>> 
>>>>> Cheers,
>>>>> 
>>>>>  
>>>>> 
>>>>> Gaurav
>>>>> 
>>>>> If you want to make TCP use for handover situation where one path might go away or become unusable, it's best to use Multipath TCP (with coupled congestion control) instead because that works on the right time scale. Again, I don't think this is a use case for SR and I would recommend to remove the section entirely.
>>>>> 
>>>>> Mirja
>>>>> 
>>>>>  
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Mon, Jul 16, 2018 at 11:08 PM, Gaurav Dawra <[email-masked]> wrote:
>>>>>> Hi Mirja,
>>>>>> 
>>>>>> On Jul 5, 2018, at 10:06 AM, Mirja Kühlewind <[email-masked]> wrote:
>>>>>> 
>>>>>>> Hi Gaurav,
>>>>>>> 
>>>>>>> sorry for my very long delay but this got somehow a bit lost in my mailbox.
>>>>>>> 
>>>>>>> I think with your changes you addressed explicit problems Martin called out, however, I still have high level concerns about these sections as they are mostly giving speculative recommendation which are not clear to me to work in practice.
>>>>>>> 
>>>>>>> "A flowlet is defined as a burst of packets from the same flow followed by an idle interval."
>>>>>>> but then you say
>>>>>>> "...then the application can break the elephant flow F into flowlets F1, F2, F3, F4..."
>>>>>>> 
>>>>>>> This sounds like you would just divide an elephant flow mostly randomly into flowlets which can interact badly with the congestion control. If you actually have chunks of data that are transmitted with large enough idle interval in between (as you define flowlets in the first sentence), that is not an elephant flow anymore and it will not help you to "spread the load of the elephant flow through all the ECMP paths". In summary I actually don't see how the concept of flowlets can be helpful to address the problem you have at all, and I still advise you to remove section 7.1 entirely.
>>>>>>> 
>>>>>>> Regarding section 7.2, I also still skeptical about any benefits that can be achieved. Given you are in a data center, the controller should already know about static parameters such as the maximum bandwidth per link. For dynamic parameters, e.g. like loss rate,                                         measuring them on a per-flow bases is the wrong thing to do. What I mean is you can ask a router about the average loss rate that it observes and that might give you some valuable, however, if you ask a TCP flow about the average loss rate the answer will mainly depend on the congestion controller used and the currently available bandwidth, which is a very dynamic property and not a link characteristic. So this information is not usable for performance aware routing. A flow could give you information about the observed RTT (min/max) on a certain path, or the maximum available bandwidth on a path, but as I said, given you are in a data center environment these are information that the controller already should have anyway.
>>>>>>> 
>>>>>>> Your example with detecting a faulty path due to losses does not work with TCP as you never know if these loses are due to a problem on the path, self-induced or by a competing flow. And even if you don't use TCP and e.g. send constant bit rate traffic, there may be a large number of competing TCP flows that can induce the loses. Try to steer traffic "away" on a time-scale that is slower than TCP dynamics or even your flow dynamic (when flows start or end) in case you have a lot of very short flow, in the best case doesn't work and in the worst case leads to oscillation.
>>>>>>> 
>>>>>>> If you want to make TCP use for handover situation where one path might go away or become unusable, it's best to use Multipath TCP (with coupled congestion control) instead because that works on the right time scale. Again, I don't think this is a use case for SR and I would recommend to remove the section entirely.
>>>>>>> 
>>>>>>> Mirja
>>>>>>> 
>>>>>>> 
>>>>>>>>> On Mon, Jun 18, 2018 at 5:13 PM, Gaurav Dawra <[email-masked]> wrote:
>>>>>>>>> Hi Alvaro, Mirja 
>>>>>>>>> 
>>>>>>>>> On Jun 12, 2018, at 7:06 AM, Gaurav Dawra <[email-masked]> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Mirja,
>>>>>>>>>> 
>>>>>>>>>> Friendly Reminder...Could you please also advice if there is anything further to DISCUSS on this document which was also related to TCP updates below?
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> 
>>>>>>>>>> Gaurav
>>>>>>>>>> 
>>>>>>>>>>> On Thu, Jun 7, 2018 at 9:02 AM, Alvaro Retana <[email-masked]> wrote:
>>>>>>>>>>> Thanks Martin!
>>>>>>>>>>> 
>>>>>>>>>>>> On June 6, 2018 at 3:14:45 PM, Martin Stiemerling ([email-masked]) wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Alvaro, all, 
>>>>>>>>>>>> 
>>>>>>>>>>>> > br/>> Gaurav just posted a revision.  Please takke a look and let us know if br/>> the changes address your concerrns or not. 
>>>>>>>>>>>> > br/>> https://www.ietf.org/rfcdiff??url2=draft-ietf-spring-segment-routing-msdc-09 
>>>>>>>>>>>> > br/>> Thanks!!! 
>>>>>>>>>>>> > br/>> Alvaro. < 
>>>>>>>>>>>> > br/>> On May 25, 2018 at 12:08:46 PM, Gaurav Dawra (([email-masked] br/>> <mailto:gdawra.ietf@@gmail.com>) wrote: 
>>>>>>>>>>>> > br/>>> Hi Martin, < 
>>>>>>>>>>>> >> 
>>>>>>>>>>>> >> Thanks for review. I will post the new version. Hopefully, it will br/>>> address all your comments and we can close thhis review. 
>>>>>>>>>>>> >> 
>>>>>>>>>>>> >> On May 23, 2018, at 4:17 AM, Gaurav Dawra <[email-masked] br/>>> <mailto:gdawra.ietf@@gmail.com>> wrote: 
>>>>>>>>>>>> >> 
>>>>>>>>>>>> >>> Hi Martin, 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>> Thanks for the review. Any further comments here ? I will post the br/>>>> new version soon. < 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>> Cheers, 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>> Gaurav 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>> Sent from my iPhone 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>> On May 16, 2018, at 7:44 PM, Gaurav Dawra <[email-masked] br/>>>> <mailto:gdawra.ietf@@gmail..com>> wrote: 
>>>>>>>>>>>> >>> 
>>>>>>>>>>>> >>>> Hi Martin, 
>>>>>>>>>>>> >>>> 
>>>>>>>>>>>> >>>> Apologies from my end we had few internal authors conversations on br/>>>>> the points you have raised. < 
>>>>>>>>>>>> >>>> 
>>>>>>>>>>>> >>>>> On Apr 9, 2018, at 7:58 AM, Martin Stiemerling <[email-masked] br/>>>>>> <mailto:[email-masked]>> wrote: 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> Hi Gaurav, 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> This got lost on my end, sorry for this. The filter just moved br/>>>>>> these messages out of my sight... :-/ 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>>>> On Jan 9, 2018, at 2:25 PM, Martin Stiemerling br/>>>>>>>> <mls.ietf@@gmail.com <mailto:[email-masked]> br/>>>>>>>>; <mailto:[email-masked]>> wrote: 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> Hi all, 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> I've reviewed this document as part of the transport area review br/>>>>>>>> team's ongoing effort to review key IETF documents. These br/>>>&gtt;>>>> comments were written primarily for the transport area directors, br/>>>>>>>> but are copied to the doocument's authors for their information br/>>>>>>>&> and to allow them to address any issues raised. When done at the 
>>>>>>>>>>>> >>>>>>> time of IETF Last Call, the authors should consider this review br/>>>>>>>> together with any other last-call comments they receive. Please br/>>>&>>>>> always CC tsv-art@… if you reply to or forward this review. 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> Summary: 
>>>>>>>>>>>> >>>>>>> This draft has serious issues in Section 7..1, 7.2 and in one part br/>>>>>>>> of Secction3, described in the review, and needs to be rethought. br/>>&>>>>>> The other sections are good AFAIK. 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> Technicals: 
>>>>>>>>>>>> >>>>>>> The overall draft looks ok, but the three points below look br/>>>>>>>> strange and need a fix before publication IMHO: 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> Both Sections, 7.1. and 7.2., are describing ideas, but not well br/>>>>>>>> proven funcationality and not even safe to use functionality. br/>>>&>>>>> Both are some sort discussing that different paths in the network br/>>>>>>>> could be used by the eend host traffic. This sounds pretty much br/>>>>>>&gtt;> like the Path Aware Networking Proposed Research Group br/>&gtt;>>>>>> (https://irtf.org/panrg) and hints to the fact that there is no br/>>>>>>>> commonly understannd and accepted engineering solution in this space. 
>>>>>>>>>>>> >>>>>>> 
>>>>>>>>>>>> >>>>>>> Section 7.1: 
>>>>>>>>>>>> >>>>>>> [KANDULA04] is a really old reference that hasn't been followed br/>>>>>>>> up iin recent times and even worse there is no evidence that this br/>&gtt;>>>>>> is going to work good enough or stable enough under real Internet br/>>>>>>>> traffic. Additioonally, it is more than unclear how any modern TCP br/>>>>&ggt;>>> implementation will react to this 
>>>>>>>>>>>> >>>>>>> This section describes an idea without detailing too much about br/>>>>>>>> any furtther aspects. Further it changes the commonly accepted br/>>>;>>>>>                                                           notion of what an end host can do with the network. At best this br/>>>>>>>> would require a good ddefinition of what an end host in your br/>>>>>>>&ggt; setting is, e.g., a highly modified piece of (at least) software 
>>>>>>>>>>>> >>>>>>> Further communicating instantaneous path characteristics to a br/>>>>>>>> central point is potentially a bad idea, as the data is already br/>>>;>>>>> outdated when reported by any node. 
>>>>>>>>>>>> >>>>>> [Gaurav] I believe Authors are trying to highlight that Host which br/>>>>>>> is defineed in br/>>>>>>> (https://tools.ietf.org/html/draftt-ietf-spring-segment-routing-15) br/>>>>>>> can innfluence the traffic based on the Calculations locally or br/>>&gtt;>>>> jointly with the controller. Implementations can decide how much br/>>>>>>> Centralized -vs- localized coontrol is allowed at Host based on br/>>>>>>> perfoormance data collection. 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> Performance data collection (monitoring?) isn't crucial when it br/>>>>>> comes to timely (actuaally real-time) reaction. However, this br/>>>>>> secttion isn't just about performance data collection as it is about br/>>>>>>> "Performance-aware routing" this seems to try to interact in br/>>>>>> real-time with the network behhavior of TCP. This isn't even close br/>>>>>> to acceeptable. 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> I would be ok to say that it is useful to collect performance data br/>>>>>> for offline analysis and improvement of the data network. However, br/>>>>>&ggt; this is at completely different magnitues of time scales. 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>>>> It is the foundation of TCP that the network is regarded as a br/>>>>>>>> black box and that you infer from the transmission of packets br/>>>>;>>>> what the current state of the network path is. Inferring network br/>>>>>>>> path metrics (you mention SRTT, MSS, CWND ) is a bad idea, as br/>>>>>>>>; this would required that all paths exhibit this and if not what br//>>>>>>>> is going to happen? 
>>>>>>>>>>>> >>>>>>> It could be an interesting research field to change many points br/>>>>>>>> in TCP'ss behavior, but this once again points                                                           to the fact that br/>>>&>>>>> this not the IETF works but IRTF or elsewhere. 
>>>>>>>>>>>> >>>>>> [Gaurav] Martin, Authors are trying to suggest that TCP is rightly br/>>>>>>> treating Network as Black Box. Authors are implying per path br/>>>>;>>> performance metrics as not cached. Is there some change in text br/>>>>>>> you are suggesting?? 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> I would recommend to remove the 3rd bullet point completey. TCP br/>>>>>> isn't the place to rememmber "good" or "bad" paths. This is br/>>>>>> something the network could decide, e.g., rerouting TCP flows br/>&ggt;>>>> within the network or changing the forwarding path in the network br/>>>>>> for particular flows (if it is not routed). 
>>>>>>>>>>>> >>>> 
>>>>>>>>>>>> >>>> <Gaurav> Ack, after discussion, we will remove the Section 3 - 3rd br/>>>>> bullet point. Willl update in next rev - coming shortly. 
>>>>>>>>>>>> >>>> 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>> Kind regards, 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>>>  Martin 
>>>>>>>>>>>> >>>>> 
>>>>>>>>>>>> >>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Mon, 29 Oct 2018 16:34:09 +0100

Hi Gaurav,

this still sounds very much like inventing a new mechanism which seem a bit out of scope for this document. However, after all bandwidth requirements might not be known or are very dynamic because of congestion control or adaption mechanism in the application (e.g. adaptive video traffic), and therefore there it is still the same problem that it is no reasonable to make decision based on this very dynamic metric. 

The text below sounds like you are rather interested to a) distinguish elephant from mice flows and b) understand if the elephant flow has a maximum bandwidth cap (because it's application-limited). These are different information and might be more useful for your case. However, I still think having this discussion in this level of details goes beyond the scope of the document.

What’s about just saying something like, a central host can collect per-flow information, either from the host directly or measurement on the path, and use this information to impact routing. I would, however, also like to see a note/warning in this context that metrics that are changing very dynamically should not be used as input for routing decisions.

Mirja

> Am 29.10.2018 um 16:12 schrieb Gaurav Dawra <[email-masked]>:
> 
> Hi Mirja,
>

---

From: Alvaro Retana <[email-masked]>
Date: Mon, 29 Oct 2018 15:41:35 -0400

On October 29, 2018 at 11:34:13 AM, Mirja Kuehlewind (IETF) (
[email-masked]) wrote:

Hi!

FWIW, I agree with Mirja and her proposal below.  Not only does it sound
like this Informational document is talking about items that should be out
of scope, but the first paragraph in §7 says that it talks about "how the
problems described above (in section 3) could be solved using the segment
routing concept.”  To me, these are examples and (as the text also
mentions) "only parts of the solution”.

Let’s please wrap this document up!

Thanks!

Alvaro.

this still sounds very much like inventing a new mechanism which seem a bit
out of scope for this document. However, after all bandwidth requirements
might not be known or are very dynamic because of congestion control or
adaption mechanism in the application (e.g. adaptive video traffic), and
therefore there it is still the same problem that it is no reasonable to
make decision based on this very dynamic metric.

The text below sounds like you are rather interested to a) distinguish
elephant from mice flows and b) understand if the elephant flow has a
maximum bandwidth cap (because it's application-limited). These are
different information and might be more useful for your case. However, I
still think having this discussion in this level of details goes beyond the
scope of the document.

What’s about just saying something like, a central host can collect
per-flow information, either from the host directly or measurement on the
path, and use this information to impact routing. I would, however, also
like to see a note/warning in this context that metrics that are changing
very dynamically should not be used as input for routing decisions.