Importance: Normal
MIME-Version: 1.0
Sensitivity: 
To: Caitlin Bestler <cait@asomi.com>
Cc: ips@ece.cmu.edu
Subject: Re: iSCSI/iWARP drafts and flow control
Message-ID: <OFB8543DEB.DFE9E2A9-ON87256D70.0066CD15@us.ibm.com>
From: Mike Ko <mako@almaden.ibm.com>
Date: Sun, 27 Jul 2003 12:21:31 -0700
Content-Type: text/plain; charset="us-ascii"
Sender: owner-ips@ece.cmu.edu
Precedence: bulk

From sec.3.5.2.1 of the iSCSI spec, "Asychronoous Messages are used to 
carry SCSI asyncrhonous events (AEN) and iSCSI asynchronous messages." 
Sec.10.9.1 lists the codes for Asyncrhonous Messages which include 0) SCSI 
Asynchronous Event, 1) target requests Logout, 2) target indicates it will 
drop the connection, 3) target indicates it will drop all connections of 
this session, and 4) target requests parameter negotiation.  Since the 
iSER layer can estimate the number of Asynchronous Messages needed in a 
typical situation for the AsyncEvent codes listed in the iSCSI spec, it 
can determine how much to overprovision when a dedicated receive queue is 
used.  Additionally, if one is concerned about the atypical case, the 
optional features of graceful handling and the shared receive queue as you 
have pointed out can be used.  So it doesn't seem to me that flow control 
for Asynchronous Messages are needed.  Can you provide an example where an 
unbounded number of Asynchronous Messages would need to be sent?

Mike Ko
IBM Almaden Research
San Jose, CA 95120

To:     Mike Ko <mako@almaden.ibm.com>
cc:     ips@ece.cmu.edu 
Subject:        Re: iSCSI/iWARP drafts and flow control




On Saturday, July 26, 2003, at 03:57 PM, Mike Ko wrote:

> In iSER, we expect the flow control to be regulated by the Command
> Numbering mechanism in iSCSI.  In other words, since the queuing
> capacity
> of the receiving iSCSI layer is MaxCmdSN - ExpCmdSN + 1, the receiving
> iSER layer can use this information to determine the minimum number of
> untagged buffers.  In addition, it needs to provision a sufficient
> number
> of untagged buffers to allow enough time for the iSER layer to respond
> to
> incoming immediate commands, asynchronous messages, etc., and replenish
> the buffers.  The use of a buffer pool shared across multiple
> connections
> will allow the iSER layer to replenish the buffers on a statistical
> basis.
>

iSCSI does indeed provide control for untagged messages that
have CmdSNs. The issue relates to Asynchronous Messages.

Is there any protocol restraint on the number of Asynchronous
Messages that can be sent? The only one that I could find is in
section 3.2.2.2 where it is limited to an "implementation defined
constant that MUST NOT exceed 2**31-1".

You cannot know an implementation defined constant of your peer,
unless the protocol explicitly communicates it. So the only ULP
constraint on the number of in-flight Asynchronous messages is
2**31-1. That constraint already existed in iWARP (the Message
Sequence Number has the same need to determine which of two 32
bit numbers that wrap-around is "later").

This is an important change from iSCSI over a conventional
transport protocol. The asynchronous messages would be flow
controlled by the transport itself. The Data Sink is expected
to advertise the amount of in-flight data that it is willing
to accept.

It is important to separate buffering from flow control. Flow
control deals with what is legal to send. Buffering is strictly
a local issue. Even with conventional transports, there is no
requirement that every buffer byte advertised on a TCP connection
or SCTP association be pre-allocated and reserved in advance.
How a host manages its receive buffers is a local issue. The
protocol issue is what it tells its peer that the peer is
authorized to send.

Aside from the theoretical issues (such as definitions of "reliable
transport" and "flow control") there is a very nitty gritty issue
that buffer pools are an *optional* feature even under the RDMAC
proposed verbs. You also cannot "share" resources across connections
if there is only a single connection to deal with.

To achieve the goal of allowing iSER to work on a generic RDMA
card it must work with a *simple* Receive Queue.  That requires
that the Data Sink be able to predict the maximum number of
untagged messages that can be in-flight from a peer that is
following the protocol.

This actually would not be that hard to deal with. Several of
the Asynchronous Messages are inherently unique. There is no
reason to send two messages announcing the intent to terminate
a connection over a reliable connection.

If you simply add a constraint on the number of Asynchronous
Messages that may be sent without confirmation that they have
been processed there is no problem. iSER becomes a reliable
protocol. Peers are not required to estimate how fast the
other side can process messages. (And, to hit the theoretical
point, if you have to *estimate* how fast your peer operates
then you are no longer using a *reliable* protocol).

There are already "credits" for CmdSN-bearing untagged message.
The methods for establishing and modifying these credits already
exist in iSCSI so there is no need for iSER to do anything
other than be aware of the value.

iSER then has to factor in additional credits for Asychs.
This could be a ULP specified constant, or it could be
negotiated just as is proposed for RDMA Read credits.
Credits would be drained by sending. All that is required
is the addition of of a rule on how credits are restored.
I would suggest using reply to a *later* CmdSN and/or
a command that explicitly waits for all prior Asynch
Messages to have been processed.