Subject: [core] CoAP for high throughput applications

From: "Fossati, Thomas (Nokia - GB)" <[email-masked]>
Date: Mon, 4 Apr 2016 22:11:47 +0000

Hi folks,

A slightly off-topic question -- though not too much, hopefully.

One of the LURK [0] proposals is draft-cairns-tls-session-key-interface [1].

In Section 7.1.2 [2] the authors propose to transport the LURK payloads over CBOR/CoAP (over DTLS/UDP, I guess).

Now, a single LURK box could have to handle lots of these requests, potentially in thousands per second, whereas CoAP's default congestion control algorithm parameters [3] are, by design, way too conservative to be suitable for high-throughput use cases.

Is there anyone that has played with CoAP for high-throughput applications who'd be willing to share his/her experience with the group and the wider IETF community?

Cheers, thanks very much,
t

[0] https://www.ietf.org/mail-archive/web/lurk/current/msg00083.html
[1] https://datatracker.ietf.org/doc/draft-cairns-tls-session-key-interface
[2] https://tools.ietf.org/html/draft-cairns-tls-session-key-interface-01#section-7.1.2
[3] https://tools.ietf.org/html/rfc7252#section-4.8

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 04 Apr 2016 19:22:54 -0300

> over CBOR/CoAP (over DTLS/UDP, I guess).

Interesting.

> Now, a single LURK box could have to handle lots of these requests,
> potentially in thousands per second, whereas CoAP's default congestion
> control algorithm parameters [3] are, by design, way too conservative to
> be suitable for high-throughput use cases.

That would be an interesting application for CoCoA...

> Is there anyone that has played with CoAP for high-throughput
> applications who'd be willing to share his/her experience with the group
> and the wider IETF community?

I think Matthias Kovatsch' Californium still is the record holder for
responses per second from a Web Server (a term that I take to include
both servers for CoAP and for one of the HTTPs)...

The 16-bit message ID is only good for ~ 10^3 requests per second, so a
client that needs a higher rate from a single server would need to use
multiple port numbers.

Grüße, Carsten

---

From: Zach Shelby <[email-masked]>
Date: Mon, 4 Apr 2016 22:45:05 +0000

Indeed, Matthias and I have done considerable work on the scalability of a single server talking to very large numbers of constrained endpoints [1], however each of those endpoints was NSTART=1. I have not seen any work done on high throughput between two endpoints, although my knee-jerk reaction would be that CoAP/TCP or some other TCP based protocol is more suitable.

[1] https://www.vs.inf.ethz.ch/publ/papers/mkovatsc-2014-iot-californium.pdf

Zach

On 04 Apr 2016, at 15:11, Fossati, Thomas (Nokia - GB) <[email-masked]<mailto:[email-masked]>> wrote:

Hi folks,

_______________________________________________
core mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/core

Zach Shelby
Vice President, Marketing
ARM Internet of Things BU
www.arm.com<http://www.arm.com>
US: +1 (408) 203-9434
Finland: +358 407796297
Skype: zdshelby
LinkedIn: fi.linkedin.com/in/zachshelby/<http://fi.linkedin.com/in/zachshelby/>

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

---

From: "Fossati, Thomas (Nokia - GB)" <[email-masked]>
Date: Mon, 4 Apr 2016 22:46:15 +0000

Hi Carsten, thanks for the quick reply.

>That would be an interesting application for CoCoA...

Is CoCoA available if I'd want to test it?

>client that needs a higher rate from a single server would need to use

I haven't re-done the maths, but the CoAP spec has a slightly lower number:

  "[...] The Message ID
   is compact; its 16-bit size enables up to about 250 messages per
   second from one endpoint to another with default protocol
   parameters.)"

Cheers, t

---

From: "Fossati, Thomas (Nokia - GB)" <[email-masked]>
Date: Mon, 4 Apr 2016 22:52:39 +0000

Hi Zach, thanks for the pointer.

The problem with TCP is that you don't want one lost packet to block the train of (totally independent) LURK transactions — and as a consequence the bunch of pending TLS handshakes that are waiting on them for completion.

A UDP based transport would be better suited for the LURK use case.  One that minimise the wire overhead (e.g. by using CBOR/CoAP) would be even better.

Cheers, t

From: core <[email-masked]<mailto:[email-masked]>> on behalf of Zach Shelby <[email-masked]<mailto:[email-masked]>>
Date: Monday, 4 April 2016 19:45
To: Thomas Fossati <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [core] CoAP for high throughput applications

Zach

Hi folks,

Zach Shelby

---

From: "Kovatsch Matthias" <[email-masked]>
Date: Tue, 5 Apr 2016 16:38:04 +0000

> Is CoCoA available if I'd want to test it?

Californium has expirmental support for CoCoA (contributed by August Betzler). The focus, however, was actual congestion in constrained networks, and not high-throughput. Hence, the CoCoA implementation isn't for that...

>    second from one endpoint to another with default protocol

This only limits the number of messages between two endpoints. A server can receive this rate from an arbitrary number of clients. I think this was the scenario for this LURK box, no?

In case you really need to send a higher rate of requests from a client, it could use multiple ports.

Ciao
Matthias

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 05 Apr 2016 13:57:47 -0300

> In case you really need to send a higher rate of requests from a client, it could use multiple ports.

... and it is worth noting that distributing requests over a large
number of outgoing ports is something that DNS servers and resolvers
know to do well.  It seems to me the need to change anything about the
message-ID here is very low even for high-rate applications.

Grüße, Carsten

---

From: "Fossati, Thomas (Nokia - GB)" <[email-masked]>
Date: Tue, 5 Apr 2016 20:26:58 +0000

On 05/04/2016 13:38, "EXT Kovatsch  Matthias" <[email-masked]> wrote:
>This only limits the number of messages between two endpoints. A server
>can receive this rate from an arbitrary number of clients. I think this
>was the scenario for this LURK box, no?

My target is for a *single* edge server to send around 20K transactions
per second to the LURK box.

>it could use multiple ports.

So, assuming we can't do anything at the CoAP layer, the number of
ephemeral ports that are available on the client is the limiting factor
here.  IIRC, on Linux that number is roughly 30K, which means that a
dedicated box could do >7M transactions per second?  That's quite a lot.

---

From: "Carles Gomez Montenegro" <[email-masked]>
Date: Wed, 6 Apr 2016 11:58:38 +0200

Hi Thomas,

> Is CoCoA available if I'd want to test it?

Here you may find the Californium implementation including CoCoA:
https://github.com/eclipse/californium

As Matthias said, the implementation including CoCoA is not actually
optimized for dealing with thousands of requests. However, CoCoA is
adaptive and, if network conditions allow it, the algorithm may behave
more aggressively than default CoAP's congestion control (while still
being safe).

In the experiments we have done so far, we have dealt with up to hundreds
of requests per second (outperforming default CoAP and also
state-of-the-art alternative approaches designed for TCP), since the
limitation in our scenarios was the network/channel capacity (e.g. [1,
2]).

It would be very interesting to see what happens in a scenario such as the
one you describe, where the limitation might not be the network
throughput. By the way, feedback on CoCoA would be very much welcome!

Thanks,

Carles

[1] A. Betzler, C. Gomez, I. Demirkol, J. Paradells, "CoAP congestion
control for the Internet of Things", IEEE Communications Magazine         
(In press).
https://www.researchgate.net/publication/297989374_CoAP_Congestion_Control_for_the_Internet_of_Things

[2] A. Betzler, C. Gomez, I. Demirkol, J. Paradells, "CoCoA+: an advanced
congestion control mechanism for CoAP", Ad-hoc Networks journal, 2015.
http://www.sciencedirect.com/science/article/pii/S1570870515000888

---

From: "Fossati, Thomas (Nokia - GB)" <[email-masked]>
Date: Wed, 6 Apr 2016 18:09:05 +0000

Hi Carles,

Very interesting stuff, and thanks for the pointers.

I'll try to put together Californium+CoCoA and Oscar's keyserver
(https://github.com/mami-project/KeyServer) to see what happens :-)

Cheers, t