Subject: [Ips] iSCSI: intent of MaxCmdSN

From: "Eddy Quicksall" <[email-masked]>
Date: Mon, 9 Feb 2004 13:23:25 -0500

Regarding MaxCmdSN, is the intent to regulate the flow of commands to the SCSI layer or to regulate the flow to iSCSI for multiple connections?

It seems that it must be for multiple connections to reduce the number of commands that are waiting to be ordered to SCSI. And that the SCSI layer should be responsible for its own queue depth.

But, I just wanted to see if I'm missing something.

Eddy

---

From: [email-masked]
Date: Mon, 9 Feb 2004 10:56:38 -0800 (PST)

On Mon, 9 Feb 2004, Eddy Quicksall wrote:

> Regarding MaxCmdSN, is the intent to regulate the flow of commands to
> the SCSI layer or to regulate the flow to iSCSI for multiple
> connections?

MaxCmdSN doesn't regulate flow for multiple connections, it regulates how
many new commands can be in-flight at once. It comes into play even in
single-connection sessions.

> It seems that it must be for multiple connections to reduce the number
> of commands that are waiting to be ordered to SCSI. And that the SCSI
> layer should be responsible for its own queue depth.

I don't think so. It represents how many commands the target iSCSI layer
has promised it can receive. This does impact multi-connection re-ordering
prevention, but it can also be used to apply back-pressure if the SCSI
layer is running low on resources.

The latter is important as the target's promising it can receive that many
commands. It should be careful about promising what it can't deliver.

Take care,

Bill

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: "Eddy Quicksall" <[email-masked]>
Date: Mon, 9 Feb 2004 14:21:19 -0500

It seems that this is actually an implementation issue and I was wondering
what the intent was.

Does SAM-2 mention transport flow control? It seems that the transport
should be unaware of the SCSI resources. Regarding SCSI, I thought it it
would return the TASK SET FULL status (or maybe BUSY status in some cases).
MaxCmdSN could be used for that purpose but I wonder if that was really the
intent.

Regarding commands in-flight, doesn't that fit into the re-ordering
category? i.e., they are in-flight to the SCSI layer. I suppose it could
also mean on-the-wire depending on implementation.

Am I missing something?

Eddy

---

From: [email-masked]
Date: Mon, 9 Feb 2004 14:07:04 -0800 (PST)

> MaxCmdSN could be used for that purpose but I wonder if that was really the

Well, while SCSI can return TASK SET FULL, the iSCSI layer certainly could
use it for backpressure.

> also mean on-the-wire depending on implementation.

Yes. How long it takes for a packet to go to and from the target.
Including processing delays.

Consider a session with only one connection. If ExpCmdSN == MaxCmdSN, then
the initiator has to receive acknowledgement of a command (receive a PDU)
before it can send the next one. Now say the initiator wants to send a
stream of writes. The natural thing for the target to do is send a PDU
when the write is over. Now say that the writes are all the same size as
FirstBurstLength (default of 64k and an OS that performs 64k writes) and
InitialR2T=NO. All of the Data-Out PDUs will be sent right after the
command request. Unless something happens, the initiator won't be able to
start another command until the transmitted one is completed.

With MaxCmdSN > ExpCmdSN, the initiator can start a second (or 3rd or 4th
or ...) write while the first one is being processed.

Yes, there are tricks that could be played in the ExpCmdSN == MaxCmdSN
case (like say a NOP-OUT with echo right after the SCSI command), but it's
simpler, at least to me, to just let MaxCmdSN == ExpCmdSN + some small
constant.

Take care,

Bill

---

From: "Eddy Quicksall" <[email-masked]>
Date: Mon, 9 Feb 2004 17:24:18 -0500

I agree on all points. That is why it is a bit of a dilemma for me.

If it is used to throttle the commands on the wire (including the commands
out of order) then it can cause extra traffic to advertise the new MaxCmdSN
when there are no responses ready.

But if it is used to throttle SCSI and the diff is large enough for SCSI
then it can mean queuing of incoming out-of-order commands and it is very
expensive to hit memory.

That is why I was wondering what the original intent was.

Eddy

---

From: [email-masked]
Date: Mon, 9 Feb 2004 14:39:58 -0800 (PST)

> when there are no responses ready.

I think it _can_ be used to throttle commands on the wire. That doesn't
mean it has to be. I'd expect a target to pick a value, like say 4 or 8,
and stick with that value in all but the most dire of circumstances.

If changing the value is hard and you can handle over-commit in other
ways, do that.

> then it can mean queuing of incoming out-of-order commands and it is very

Sounds like an implementation issue. Since you get to pick the number,
pick something that makes your life easy.

Take care,

Bill

---

From: Julian Satran <[email-masked]>
Date: Tue, 10 Feb 2004 10:21:58 -0800

Bassically to limit what you are keeping for reordering but obviously 
there is nothing to prevent an implementation to use it as a mechanism to 
limit the SCSI queue depth.

Julo

"Eddy Quicksall" <[email-masked]> 
Sent by: [email-masked]
09/02/2004 10:23

To
<[email-masked]>
cc

Subject
[Ips] iSCSI: intent of MaxCmdSN

Regarding MaxCmdSN, is the intent to regulate the flow of commands to the 
SCSI layer or to regulate the flow to iSCSI for multiple connections?
 
It seems that it must be for multiple connections to reduce the number of 
commands that are waiting to be ordered to SCSI. And that the SCSI layer 
should be responsible for its own queue depth.

Eddy