Return-Path: <owner-ips-outgoing@ece.cmu.edu>
Received: from newbox.ece.cmu.edu (localhost [127.0.0.1])
	by newbox.ece.cmu.edu (Cyrus v2.1.12) with LMTP; Thu, 31 Jul 2003 15:42:48 -0400
X-Sieve: CMU Sieve 2.2
Received: from osgood.ece.cmu.edu (OSGOOD.ECE.CMU.EDU [128.2.129.25])
	by newbox.ece.cmu.edu (Postfix) with ESMTP id 2529E32FF
	for <ipsml@ece.cmu.edu>; Thu, 31 Jul 2003 15:42:48 -0400 (EDT)
Received: by osgood.ece.cmu.edu (Postfix, from userid 953)
	id 3F7F7A6; Thu, 31 Jul 2003 15:42:47 -0400 (EDT)
Received: from sos.ece.cmu.edu (SOS.ECE.CMU.EDU [128.2.129.27])
	by osgood.ece.cmu.edu (Postfix) with ESMTP
	id 1AE1D87; Thu, 31 Jul 2003 15:42:31 -0400 (EDT)
Received: by sos.ece.cmu.edu (Postfix)
	id 7ED278A13; Thu, 31 Jul 2003 15:42:18 -0400 (EDT)
Delivered-To: ips-outgoing@sos.ece.cmu.edu
Received: from hazard.ece.cmu.edu (HAZARD.ECE.CMU.EDU [128.2.129.24])
	by sos.ece.cmu.edu (Postfix) with ESMTP id 6EC9F8A11
	for <ips-outgoing@sos.ece.cmu.edu>; Thu, 31 Jul 2003 15:42:18 -0400 (EDT)
Received: by hazard.ece.cmu.edu (Postfix)
	id 3D167E3; Thu, 31 Jul 2003 15:42:18 -0400 (EDT)
Delivered-To: ips-outgoing@ece.cmu.edu
Received: by hazard.ece.cmu.edu (Postfix, from userid 953)
	id 0EB1FEB; Thu, 31 Jul 2003 15:42:18 -0400 (EDT)
Received: from sos.ece.cmu.edu (SOS.ECE.CMU.EDU [128.2.129.27])
	by hazard.ece.cmu.edu (Postfix) with ESMTP id 1C1D3E9
	for <ips-outgoing@ece.cmu.edu>; Thu, 31 Jul 2003 15:42:14 -0400 (EDT)
Received: by sos.ece.cmu.edu (Postfix, from userid 363)
	id 0977B8A16; Thu, 31 Jul 2003 15:42:14 -0400 (EDT)
X-Original-To: ips@sos.ece.cmu.edu
Delivered-To: ips@sos.ece.cmu.edu
Received: from bache.ece.cmu.edu (BACHE.ECE.CMU.EDU [128.2.129.23])
	by sos.ece.cmu.edu (Postfix) with ESMTP id B155A8A11
	for <ips@sos.ece.cmu.edu>; Thu, 31 Jul 2003 15:42:12 -0400 (EDT)
Received: by bache.ece.cmu.edu (Postfix)
	id 8D5F098; Thu, 31 Jul 2003 15:42:12 -0400 (EDT)
Delivered-To: ips@ece.cmu.edu
Received: by bache.ece.cmu.edu (Postfix, from userid 953)
	id 5749873; Thu, 31 Jul 2003 15:42:12 -0400 (EDT)
Received: from msgbas1x.cos.agilent.com (msgbas1x.cos.agilent.com [192.25.240.36])
	by bache.ece.cmu.edu (Postfix) with ESMTP id 9EFFF9E
	for <ips@ece.cmu.edu>; Thu, 31 Jul 2003 15:42:04 -0400 (EDT)
Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237])
	by msgbas1x.cos.agilent.com (Postfix) with ESMTP
	id B36B71C11D; Thu, 31 Jul 2003 13:42:01 -0600 (MDT)
Received: from wcosbh22.cos.agilent.com (wcosbh22.cos.agilent.com [130.29.152.178])
	by relcos2.cos.agilent.com (Postfix) with ESMTP
	id 026151F; Thu, 31 Jul 2003 13:42:04 -0600 (MDT)
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0
Subject: RE: iSCSI/iWARP drafts and flow control
Date: Thu, 31 Jul 2003 13:42:00 -0600
Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0465@axcs04.cos.agilent.com>
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
Thread-Topic: iSCSI/iWARP drafts and flow control
Thread-Index: AcNXjWQfPYoXpsOAEdesjwCQJ6pbUAAA9FVw
From: <pat_thaler@agilent.com>
To: <cait@asomi.com>, <pat_thaler@agilent.com>
Cc: <cbm@rose.hp.com>, <ips@ece.cmu.edu>, <rddp@ietf.org>
Sender: owner-ips@ece.cmu.edu
Precedence: bulk
X-Spam-Status: No, hits=-5.6 required=5.0
	tests=BAYES_01,NO_REAL_NAME
	version=2.50
X-Spam-Level: 
X-Spam-Checker-Version: SpamAssassin 2.50 (1.173-2003-02-20-exp)

All, sorry for the empty reply - I'm not sure how that happened.

Caitlin,

You asked some questions about how the other messages are flow =
controlled in iSCSI over TCP. The answer is that they aren't flow =
controlled. If iSCSI gets a PDU it cannot handle, it drops it and there =
are provisions to trigger it to be resent depending on the kind of =
recovery level supported. The only control for PDUs to the target is on =
non-immediate commands (both SCSI Command and Task Management Function =
Requeset PDUs). Note that when unsolicited non-immediate data is =
permitted, iSCSI allows the command to generate a command PDU plus an =
unknown number of SCSI Data-out PDUs to carry the unsolicted data. For =
iSER, we require that the unsolicted SCSI Data-out PDUs be full when =
there is enough unsolicted data to fill them (and we created a key to =
negotiate that size). Therefore, when operating over iSER the target =
does know the maximum number of PDUs that the initiator might send per =
SCSI command.

There is no deadlock in existing iSCSI because there is no flow control =
on NOP-In and the target can always send a NOP-In to advance MaxCmdSN.

To summarize, in current iSCSI, each opening in CmdSN window allows from =
1 to ? PDUs while in iSCSI over iSER, each opening in CmdSN window =
allows from 1 to n PDUs where n is the amount of unsolicited data =
divided by data per PDU (rounded up of course).

Note also that the CmdSN window is across a session. If you have =
connections in a session that are running over separate RNICs and are =
using CmdSN for flow control, each RNIC will have to have access to =
enough buffers for the whole window to land on it.=20

Between these two factors, CmdSN flow control will require over =
provisioning buffers much of the time. Perhaps memory is cheap enough =
that for an RNIC with a small number of connections this is acceptable =
in exchange for using an existing mechanism. On the other hand, we will =
have to create a mechanism to handle immediate commands and other PDUs =
that aren't covered by CmdSN so it isn't clear to me whether this is the =
right answer. The downside is overprovisioning buffers because of =
sessions spanning adapters and because each command might be a write =
with unsolicited data but many commands are reads. The upside is that =
CmdSN window can be managed to respond to changes in load while one has =
a less responsive simple mechanism to deal with the rest of the traffic.

From target to initiator, the initiator knows that each command will =
generate a response PDU so it can provision that before it sends a =
command.=20

Login and text negotiation PDUs both ways (other than perhaps the first =
PDU to open text negotiation) also have a form of iSCSI flow control. =
For these, having sent a PDU, one can't send another until one has =
gotten the response.

R2T PDUs are replaced by RDMA Reads in iSER so they are coverd by the =
RDMA Read flow control. SCSI Data-out for solicited and SCSI Data-in =
PDUs are replaced by RDMA operations so we don't have to cover them.


What isn't flow controlled by iSCSI:
initiator to target:
immediate command PDUs - existing iSCSI allows for the target to drop =
these if it gets more than it can handle and the initiator can only =
count on buffering for two, but the initiator can send more than that =
and hope the target has buffering. One can't count on how many of these =
there might be.

the first Text Request PDU - there can only be one

SNACK request - in theory, one shouldn't need to send this when =
operating over iSER, but an initiator might send one if a timeout =
occurs.

NOP-Out - one doesn't expect a lot of them, but there is no limit placed =
on them.=20

target to initiator:
Asynchronous Message

NOP-In - one doesn't expect a lot of them, but there is no limit placed =
on them.

While there usually won't be a lot of these PDUs, one has no way to put =
a maximum number on how many there will be.

There may be times when these PDUs are being sent and there is little or =
no command traffic. Also the replenishment of buffers for these has =
little relationship to the command processing. So I don't think one =
should link the replenishment to command related activities, e.g. =
advances in MaxCmdSN or reception of a command response.
=20
That is the problem space.

Does one put a flow control into iSER that looks at opcode and, for =
commands, at the I bit to control just the PDUs that aren't limited by =
MaxCmdSN or does one put in a mechanism that covers all PDUs going =
through iSER?

If one does the former, what happens when the transmitting iSER gets a =
PDU it have to stop because of the flow control and there are MaxCmdSN =
controlled PDUs after it?=20
Is iSCSI suppose to be flow control aware and manage the PDUs so that =
doesn't happen?=20
Does the whole connection transmit stop until more credit arrives?
Are the MaxCmdSN controlled PDUs allowed to pass the flow controlled PDU =
and be transmitted?

Is there a mechanism to disable flow control when the receiver doesn't =
require it, e.g. large shared buffer pool with statistical provisioning?

Pat
