Subject: [lisp] draft-kouvelas-lisp-map-server-reliable-transport working group adoption

From: [email-masked]
Date: Tue, 5 Dec 2017 18:52:03 +0200

As discussed in Singapore, the authors of draft-kouvelas-lisp-map-server-reliable-transport are looking for working-group adoption of the draft. The proposed solution replaces the UDP based control plane LISP communication between xTRs and Map-Servers with a reliable transport session. This is done to eliminate the periodic messaging and improve scaling by leveraging the reliability and flow-controlled communication over the reliable transport.

The draft was first presented to the WG at IETF 91, and has been fairly stable since then. We would like to receive any comments you may have.

thanks
Isidor

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 5 Dec 2017 10:12:36 -0800

> draft-kouvelas-lisp-map-server-reliable-transport are looking for working-group adoption of the draft. The proposed solution replaces the UDP based control

Augments. We cannot remove UDP since there is too much deployment. And TCP is not always needed. And there is startup latency with the 3 way TCP handshake. This will slow mobile handoffs. 

I also brought up at the WG that we should consider other transport layers that are more secure than TCP by itself. Since your draft doesn’t spec how to use TLS with TCP, QUIC may be a more lightweight, efficient and secure alternative. 

> plane LISP communication between xTRs and Map-Servers with a reliable transport session. This is done to eliminate the periodic messaging and improve scaling by

You can eliminate the periodic overhead by sending Map-Registers less often and solicited Map-Notifies as acks. And you can test for map-server reachability with RLOC-probes. 

The only reason I bring this up is to disclose that we have existing mechanisms to provide reliability. 

And rather than have a completely new message parsing format, we could make this proposal simpler by using existing messages that are sent over ANY transport.  I have made this comment before. 

Note we did this in the PIM WG where existing messages are sent over TCP or SCTP. 

Dino

---

From: Fabio Maino <[email-masked]>
Date: Tue, 5 Dec 2017 10:44:12 -0800

> Augments. We cannot remove UDP since there is too much deployment. And TCP is not always needed. And there is startup latency with the 3 way TCP handshake. This will slow mobile handoffs.

Right, augment rather than replace. I believe there's no intention to 
replace UDP. The draft actually relies on UDP registration to bootstrap 
the TCP channel, and to ensure backward compatibility.

> I also brought up at the WG that we should consider other transport layers that are more secure than TCP by itself. Since your draft doesn’t spec how to use TLS with TCP, QUIC may be a more lightwe…

QUIC would be an interesting option as well, probably worth of another 
draft and possibly not limited to registration messages only.

I see this as a point optimization to the registration protocol, that 
might be orthogonal to other transport-related mechanisms. In my 
experience this has proved to be very effective in scalability of large 
LISP deployments, especially with the increased volume of registration data.

Fabio

> _______________________________________________
> lisp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/lisp

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 5 Dec 2017 10:52:05 -0800

>  registration protocol, that might be orthogonal to other transport-related mechanisms. In my experience this has proved to be very effective in scalability of large LISP deployments, especially with the increased volume of registration data.

I agree it’s a point solution for registration. Then why did you need to have a general format. 

I could support this draft if it was simplified to spec how to use Map-Registers in TCP and nothing more. 

The only thing I would add is how to use TLS so encryption is supported. More and more requirements are coming up for protecting the privacy of location information. And since Map-Registers carry RLOCs (and potential Geo-Coordnates) that information needs to be protected. 

Dino

---

From: "Johnson Leong (joleong)" <[email-masked]>
Date: Tue, 5 Dec 2017 20:06:17 +0000

Hi Dino,

A large portion of this draft discusses the state machine required for TCP and how to ensure the MS and xTR are in sync.  We literally reuse the entire UDP map-register code, we just wrap that message around the LISP TCP header so there's a lot of code reuse.  Finally, this draft is not meant to replace UDP register but in some of our use cases TCP would scale better to avoid the periodic registration.

-Johnson

> On Dec 5, 2017, at 10:52 AM, Dino Farinacci <[email-masked]> wrote:
>

---

From: "Marc Portoles Comeras (mportole)" <[email-masked]>
Date: Wed, 6 Dec 2017 00:25:27 +0000

Dino,

In addition to the previous arguments there are particular use-cases where the use of reliable transport simplified the deployment of LISP.
 
As an example, the moment we started scaling datacenters to support 10s of thousands of hosts, the use of a reliable transport helped a lot the management of scale: 
On one side it reduces the amount of signaling when nothing changes, since we use TCP state as an indication that xTRs and the MS are in sync and there is no need to deal with the optimization of the refresh logic (periodic or paced). 
On the other side, with reliable transport we offload the reliable delivery of information (and congestion control) from LISP to another process (TCP) that is entirely devoted and designed for this. For example, supporting events like mass VM moves relying purely on LISP based ACks became very challenging, as we ended up having to deal with congestion events related to the signaling load generated. The use of the reliable transport largely simplified the problem.

Marc

On 12/5/17, 12:06 PM, "lisp on behalf of Johnson Leong (joleong)" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Dino,

    -Johnson

---

From: Dino Farinacci <[email-masked]>
Date: Tue, 5 Dec 2017 17:56:46 -0800

> In addition to the previous arguments there are particular use-cases where the use of reliable transport simplified the deployment of LISP.

I understand its advantages. I am examining its costs.

> On one side it reduces the amount of signaling when nothing changes, since we use TCP state as an indication that xTRs and the MS are in sync and there is no need to deal with the optimization of t…

LISP (the application), does not know that itself, the xTR, is in sync with the map-server. The packets can be in flight or being retransmitted due to loss. But if a Map-Register is sent with a nonce and no Map-Notify is returned the xTR knows for sure the two are in sync.

I’d argue you may it worse. TCP does provide reliability but so does LISP itself. And the only reason the messages are periodic is because the spec said to send every 1 minute and timeout every 3 minutes. You can make it 1000 minutes and timeout every 3000 minutes. 

So let’s keep periiodic overhead, reliability, and staying in sync as separate issues.

> On the other side, with reliable transport we offload the reliable delivery of information (and congestion control) 

I understand that. But you can’t say TCP is keeping you in sync, because you have removed detail from the applicationis.

> from LISP to another process (TCP) that is entirely devoted and designed for this. For example, supporting events like mass VM moves relying purely on LISP based ACks became very challenging, as we ended up having to deal with congestion events related to the signaling load generated. The use of the reliable transport largely simplified the problem.

LISP can pack all those EID-records in a Map-Register just like TCP does. And if you want per nonce acks, you pack them in IP packets <= 65535 bytes. TCP will have to do that as well.

And guess what. What if there is an RLOC-change and you already gave the last one to TCP and can’t pull it back. If you were waiting for an ack and a new RLOC-change came in (during a lossy case), you wouldn’t have to retransmit the old information wastily. So keep the “retransmission queue” in LISP has its advantages.

Dino

---

From: "Marc Portoles Comeras (mportole)" <[email-masked]>
Date: Wed, 6 Dec 2017 16:36:42 +0000

Dino,

  >    LISP (the application), does not know that itself, the xTR, is in sync with the map-server. The packets can be in flight or being retransmitted due to loss. But if a Map-Register is sent with…

An application (and LISP in this case) should always be able to know the state of a (TCP) socket that it has opened with a server. I’m not entirely sure why we would not want to use this information. 
Besides, the reliable transport session does not invalidate the use of nonces and Map-Notifies as an indication that the MS has completely received the information, the just rely on the TCP state to know that nothing has changed.

  >  I’d argue you may it worse. TCP does provide reliability but so does LISP itself. And the only reason the messages are periodic is because the spec said to send every 1 minute and timeout every…

And this sounds as we would make the protocol very complicated to use (or code), as this would lead us to have to code/configure a specific registration pattern/logic for every use case that we want to support. Not saying we cannot, but it sounds like re-implementing TCP ;)

  >  LISP can pack all those EID-records in a Map-Register just like TCP does. And if you want per nonce acks, you pack them in IP packets <= 65535 bytes. TCP will have to do that as well.

This is exactly the point, while LISP signaling allows it we don’t need to re-implement every TCP feature in LISP, as TCP can already provide it.

  >  And guess what. What if there is an RLOC-change and you already gave the last one to TCP and can’t pull it back. If you were waiting for an ack and a new RLOC-change came in (during a lossy case…
    
I’m not sure this is so easy. UDP, just like TCP uses the RLOC (IP) as part of the “session identifier” and the nonce is per-packet, not per-session. The moment the RLOC changes on the xTR, the MS does not know that the xTR is the same so we’d need a retransmission process.

Marc

    

    

    

Dino

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 6 Dec 2017 11:17:20 -0800

> An application (and LISP in this case) should always be able to know the state of a (TCP) socket that it has opened with a server. I’m not entirely sure why we would not want to use this informatio…

All an app knows is the socket it has opened and any port it has bind()’ed to. It doesn’t know the connection state in terms of what packets have been sent and what has been ack’ed. There IS NOT enough information to do anything useful.

> Besides, the reliable transport session does not invalidate the use of nonces and Map-Notifies as an indication that the MS has completely received the information, the just rely on the TCP state t…

Right, so you have duplicate functionality. That isn’t efficient.

> And this sounds as we would make the protocol very complicated to use (or code), as this would lead us to have to code/configure a specific registration pattern/logic for every use case that we wan…

You already have that code or you aren’t implementing Map-Register acks corrrectly. And changing a timer is really a constant change in the code. And what is spec’ed today in LISP proper is much simpler than TCP, it is not reimplementing it.

But let's not drift from the point. The spec is written in a general way to solve only sending Map-Registers reliably. I suggest the text not mislead the reader to think this is a general new packet format for anything.

When people judge which protocols they want to deploy, the people that do the due diligence will look at the protocol specs and see how much mechanism is designed in. And they make judgement decisions about using the protocol.

I know of at least 2 vendors that said “I implemented pages 47-50 of the EVPN spec”.  ;-)

In LISP. we want to make sure what we spec is what is used and not ignored or considered optional. So please document the specific use-case you want to implement. And I’d suggest making the draft name draft-*-lisp-reliable-registers.

>> LISP can pack all those EID-records in a Map-Register just like TCP does. And if you want per nonce acks, you pack them in IP packets <= 65535 bytes. TCP will have to o that as well.
> 

You don’t need to. 

> I’m not sure this is so easy. UDP, just like TCP uses the RLOC (IP) as part of the “session identifier” and the nonce is per-packet, not per-session. The moment the RLOC changes on the xTR, the MS…

I’m not talking about when the xTR changes, I’m talking about an address on an interface on the same xTR changes since it was DHCP’ed to the xTR or behind a NAT. But for the LISP-MN case, the xTR is moving and its RLOCs are changing. You couple this with pubsub and the extra Map-Registers with old RLOCs has a ripple effect where then the Map-Server sends Map-Notify messages to all the subscribers, then followed by another set of Map-Notifies with the new RLOC-set. That is a lot of (unnecessary) messaging and processing.

We have to think about the implications of any one draft on the ENTIRE LISP architecture. It must work efficiently as one holistic distributed system.

Dino

---

From: "Johnson Leong (joleong)" <[email-masked]>
Date: Wed, 6 Dec 2017 19:44:13 +0000

Dino,

What we're trying to convey is that by using TCP it allows us to achieve reliability which significantly simplifies the code and error handling.  One can achieve the same with UDP with ack and retry but it would be reimplementing what TCP already offers and this added complexity typically makes the code error prone.

Another benefit with TCP is congestion control, with UDP if we send a map-register and we don't get an ack then we retry.  What if the MS is fully subscribed then this can lead to constant retry.  You can impose backoff mechanism but ultimately this would affect convergence.

-Johnson

> On Dec 6, 2017, at 11:17 AM, Dino Farinacci <[email-masked]> wrote:
> 
>> Dino,
>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 6 Dec 2017 15:17:52 -0500

<speaking without hats.>
Johnson, I think your example may cause more problems.
I am not sure what you mean by "over-subscribed".
But if the xTR sends a registration over TCP, and gets no answer, it is 
going to asume that it is properly registerd.  If the registration has 
not been received due to the receiver not reading his TCP queue (because 
he is over-subscribed) that is a bad result.

Yours,
Joel

---

From: "Johnson Leong (joleong)" <[email-masked]>
Date: Wed, 6 Dec 2017 20:25:57 +0000

Joel,

If there's flow control between the xTR and MS then the xTR wouldn't attempt to send the register if the MS cannot keep up.  Depending on your implementation the xTR can stop building registration message until flow control is off with the MS.

-Johnson

> On Dec 6, 2017, at 12:17 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> <speaking without hats.>
> But if the xTR sends a registration over TCP, and gets no answer, it is going to asume that it is properly registerd.  If the registration has not been received due to the receiver not reading his TCP queue (because he is over-subscribed) that is a bad result.
> 
> Yours,
> Joel
>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 6 Dec 2017 15:53:07 -0500

Sure, if the TCP queue backs up far enough that an effort to write by 
the application fails, the xTR can stop then.  But until that point, it 
has no way of knowing whether the registration got through or not.  TCP 
reliability != application delivery knowledge.

There may well be good reasons for wanting to use TCP for this exchange. 
  There often are.  But the example you gave in this case is arguably a 
counter-example.

Yours,
Joel

---

From: "Johnson Leong (joleong)" <[email-masked]>
Date: Wed, 6 Dec 2017 21:56:17 +0000

Joel,

I agree TCP reliability != application delivery knowledge but TCP reliability does ensure the message will *eventually* get delivered to the application (unless there's an application bug) without additional application reliability logic.  I am sure there are applications that require/expect the ACK to be sent from the application layer to avoid the time delta between transport reception and application reception seen in TCP but I don't think LISP needs that level of differentiation.

My previous example simply states that with congestion control the ETR would not end up keep re-tx if the MS is congested.  In a scale scenario re-tx can potentially cause the system to stuck in this state indefinitely.

-Johnson

> On Dec 6, 2017, at 12:53 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> Sure, if the TCP queue backs up far enough that an effort to write by the application fails, the xTR can stop then.  But until that point, it has no way of knowing whether the registration got through or not.  TCP reliability != application delivery knowledge.
> 
> There may well be good reasons for wanting to use TCP for this exchange.  There often are.  But the example you gave in this case is arguably a counter-example.
> 
> Yours,
> Joel
>

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 6 Dec 2017 17:59:35 -0800

> What we're trying to convey is that by using TCP it allows us to achieve reliability which significantly simplifies the code and error handling.  One can achieve the same with UDP with ack and retr…

I know what you are trying to do and your assertion is simply not true.

> Another benefit with TCP is congestion control, with UDP if we send a map-register and we don't get an ack then we retry.  What if the MS is fully subscribed then this can lead to constant retry.…

And you think in this condition that TCP won’t be retrying. THe arguments are weak. In fact, if TCP is retrying there is a greater window of time where old RLOC-set data is in the pipeline.

Dino

---

From: "Marc Portoles Comeras (mportole)" <[email-masked]>
Date: Thu, 7 Dec 2017 05:09:43 +0000

> And you think in this condition that TCP won’t be retrying. The arguments are weak. In fact, if TCP is retrying there is a greater window of time where old RLOC-set data is in the pipeline.

From a general perspective if TCP is adapting to network impairments or congestion in a given setting, any other solution will have to adapt too, so the experienced delay/loss problems should be there too. But the advantage in this case would be not having to re-implement an adaptive solution within the LISP application itself, but taking advantage of a solution that already provides this.

However, note that as Fabio and Johnson mentioned at the beginning of the thread, both LISP encoding of messages and UDP based signaling are preserved with the reliable transport implementation. xTRs that can take advantage of establishing the session and offloading certain tasks to the TCP socket, can do so. The rest of nodes (xTRs hosting a low number of EIDs or LISP-MN nodes) can (and will) continue using UDP based registrations normally. 

    

    
    Dino

---

From: Dino Farinacci <[email-masked]>
Date: Thu, 7 Dec 2017 13:23:38 -0800

> From a general perspective if TCP is adapting to network impairments or congestion in a given setting, any other solution will have to adapt too, so the experienced delay/loss problems should be th…

Let me make it perfectly clear. I am NOT against using TCP. We did it for PIM and thought it was a good idea. I have cc’ed Stig who has intimate knowledge and experience with PIM-over-TCP. Maybe he can comment. And we also have a lot of history with BGP-over-TCP.

I am rebuttaling some of your reasons why. That’s all.

> However, note that as Fabio and Johnson mentioned at the beginning of the thread, both LISP encoding of messages and UDP based signaling are preserved with the reliable transport implementation. xTRs that can take advantage of 

Right, that should be made more clear in the next revision to the draft. And looks like Fabio is already thinking of that (my reply is pending).

Dino