Subject: Next draft of ECM WG congestion manager document

From: Hari Balakrishnan <[email-masked]>
Date: Sun, 15 Oct 2000 21:59:07 -0400

On Thu, 03 Aug 2000 00:33:20 PDT, Vern Paxson wrote:

> The chair then made the following proposal for moving forward with ecm-cm-01:
> 
> 	1. Add opaque (scheduler) data to the API when creating a macroflow.
> 	2. Add specification of how to compute the RTT variation.
> 	3. Change CM_PERSISTENT to CM_LOST_FEEDBACK, etc.
> 	4. Add a comment to the document that some key experience we don't
> 	   yet have is with scheduling APIs, and that this may lead to
> 	   possible changes in the CM API.
> 	5. Resolve the issue Joe raised regarding sending multiple packets.
> 
> After addressing these, there would be one more WG Last Call.  The chair
> then asked for a show of WG consensus for this plan, with it being understood
> that the chair would interpret consensus for the plan followed by a
> successful last call as consensus for the document.  A show of hands
> revealed good consensus and no opposition.

At long last, there's now a new draft of the IETF ECM WG Congestion Manager 
draft.  Many apologies for the delay.  It is called draft-ietf-ecm-cm-02.txt 
and is attached to the end of this message for comments (it is also available 
from http://nms.lcs.mit.edu/papers/draft-ietf-ecm-cm-02.txt).  It hasn't been 
submitted to the IETF yet; the intention is to get some discussion going and 
obtain comments well in advance of the deadline of the next IETF.

There were 6 pending issues, 5 of which are mentioned above.  Of the ones 
above, items 2, 3, and 4 have been incorporated into the document.  A sixth 
item, brought up by Tim Shepard, concerning NATs and the behavior of the CM in 
that context has been added to section 4.5 (sharing granularity).

We have thought a great deal about issues 1 and 5, and the rest of this note 
concerns them.  We would appreciate any thoughts you might have on this.

> 	1. Add opaque (scheduler) data to the API when creating a macroflow.

While we do agree that a sophisticated scheduler can enable a lot of new 
functionality, the CM itself (as described in this document) does not require 
anything more than a trivial round-robin scheduler to be valuable.  That's 
because even such a trivial scheduler provides the machinery to do proper 
congestion control and provides an API for adaptation.

Throughout the document, we have strived to maintain the modularity of the 
different CM components, in particular the congestion controller, the 
scheduler, and the API.  Thus, rather than add opaque scheduler data to the 
cm_open() call, which many (simple) schedulers may not even care to look at, we 
strongly suggest that the right approach (to achieve what Joe wants) would be 
to:
	1. call cm_open(stream_info) and get back a streamid.
	2. call a scheduler API function (which would in general be dependent on the 
particular scheduler), passing the streamid (and perhaps the macroflowid if 
required), in order for the appropriate scheduler parameters to be initialized.

Note that although this will require an additional system call, the additional 
call overhead is observed only once, at stream setup time.  In addition, the 
separated versions of the two calls (i.e., the non-cm_open version to invoke 
the scheduler) must be supported, since an application may want to change the 
scheduler parameters during the middle of the stream.  In fact, we believe that 
this is a common mode; a Web server might decide on the priority/bandwidth of a 
stream only after it sees the full GET request.

The specific details of the scheduler contents are obviously 
scheduler-dependent.  In fact, it may not be a good idea to standardize too 
much of this, since the details of a good scheduler may allow vendors to 
differentiate themselves.

> 	5. Resolve the issue Joe raised regarding sending multiple packets.

The issue here is to make the cm_request(id) call, which currently takes no 
other arguments, richer.  In particular, the suggestion is:
	cm_request(id, npkts, size)
where npkts is the number of packets to be sent and size the size of each 
packet.  The apparent reason for this is the smaller number of cm_request() 
calls.

I tried to articulate (at the last IETF) why this may not be a good idea.  
Here's the reasoning:

1. The current API is not as inefficient as one might believe.  It may be 
tempting to think that the current API causes a user-kernel crossing for each 
packet.  That's not necessarily true.  A user-level library that implements the 
API can package multiple such calls into one system call.  For in-kernel apps 
like TCP, cm_request(id) is only a function call.  And if you believe that a 
function call is expensive, inlining it (and inlining the in-kernel function 
callback) is pretty straightforward.

2. If we add a "MAY" clause for cm_request(id, npkts, size), it MUST require 
that applications negotiate the specific supported mode in the API.  And each 
application would have to be written to handle both modes, or face the risk of 
not running on some platform.  I am not sure at all that complicating the API 
to accommodate this, in the absence of significant experience that dictates 
that it's necessary, is a good idea.  Furthermore, as Vern has pointed out and 
we as fully also agree with, this is not cast in stone.  As we gain more 
experience with CM implementations, changes will both be made and be welcome.  
Adding this option to the cm_request() call may well be a prime candidate, when 
such experience is obtained.

Note that several non-ALF applications may not care to receive a callback for 
each transmission.  They can, and will be expected to, use the 
congestion-controlled UDP sockets option (desribed in Section 6.1.2).  TCP 
applications don't have a problem because it is possible to write a TCP-over-CM 
that performs virtually identically to a vanilla TCP despite using these 
per-function (in-kernel) callbacks.

Please send your comments and observations to the group, so that we may discuss 
these before the next IETF.

Thanks!

Hari & Srini

---

From: Joe Touch <[email-masked]>
Date: Tue, 17 Oct 2000 07:10:26 -0700

> callback) is pretty straightforward.

It may be difficult to inline functions in loadable kernel modules.

Further, beyond the function call overhead, there is the per-call
processing overhead. Taking this overhead (or any overhead) on 
a per-packet basis is asking for substantial performance penalty.

> that applications negotiate the specific supported mode in the API.

The proposed extension to the API contains two parameters:

	size:	this is consistent with the byte-counting proposal
		for TCP.

	npkts:	this allows per-API-call processing to occur on
		timescales coarser than per-packet. The issue is 
		being able to amortize the processing on the API;
		inlining isn't sufficient to provide amortization.

An application that desires to use the current interface can use
size=MTU and npkts=1.

> I am not sure at all that complicating the API
> that it's necessary, is a good idea. 

That experience is in abundance; per-packet overhead should be avoided,
especially where not otherwise necessary.

If you want a particular system not to support it (i.e., to require
npkts to be 1), that can be decided on a per-system basis. Assuming how
many bytes are in a packet or how many packets should be sent between 
API calls is the assumption to be avoided.

> each transmission.

ALF applications are more likely to require the kind of performance that
cannot take additional per-packet processing hits.

Joe