Subject: Re: iSCSI: target resource "leak"/logout command

From: [email-masked]
Date: Fri, 12 Jan 2001 07:54:35 +0200

Pierre,

You are correct in bot the issue and the solution. Will fix in 04 (it came
too late for 03).

Thanks,
Julo

Pierre Labat <[email-masked]> on 09/01/2001 12:15:59

Please respond to Pierre Labat <[email-masked]>

To:   [email-masked]
cc:
Subject:  iSCSI: target resource "leak"/logout command

Julian,

There is a corner case where
- a [group of] command completion[s]  will never be acked (ExpStatRN)
- the corresponding task[s] are not "retried" because they are perfectly

   completed from the initiator point of view and
- the session continues to work fine.
Thus the resource for the task[s] may never
be released on the target (the target waits for ExpStatRN acking
the completion before releasing task resources) because
on the target side the ack of the completion is not received
and on the initiator will not retry the command.

A resource (memory) "leak" happens on the target.

First i described the scenario, then the solution.

Scenario
=======
Session with 2 TCP connections 1 and 2

1)  A command completion (StatRN=10) is received by
      the initiator (over cx 1) for the command 5.

2) The initiator sends back ExpStatRN=11 with a command over  TCP cx 1

3) The TCP cx 1 drops and the command 5 (so ExpStatRN=11) will never
   make it to the target.

4) The command 5 is not retried on the connection 2 because for the
initiator
      it is perfectly completed.

5) The target never gets the completion acked and doesn't know when
    to release resources.

Solution
=======

- Add the field ExpStatRN in the logout command.
- The initiator puts in this field the last value of ExpStatRN of the
failed
    cx. (Not the ExpStatRN value of the connection used to transport the
logout)
- The target when receiving the logout, reads ExpStatRN and is able to
free up
   the resources.

Regards,

Pierre

---

From: Santosh Rao <[email-masked]>
Date: Sun, 14 Jan 2001 23:41:46 -0800 (PST)

Julian,

Another reason for enforcing "connection allegiance" of Abort Task and its
associated task :

In the case of multi-connection sessions, each connection can be using a
different NIC. The SCSI command state information is maintained in SCSI
hardware assists (SCSI Exchange State Tables) on the adapter and such
structures need to be invalidated while aborting an I/O.

Handling the Abort Task and the task on different connections implies
having to then share state information across NIC instances, something
that iSCSI has been trying to avoid.

Regards,
Santosh

> 
> 
> 
> Pierre,
> 
> I am not sure that I follow completely (and accurately) your line of
> thought  but if you have a multiple connection session the command
> numbering is meant to offer a total ordering.  Any of the scenarios you
> mention can't happen unless the target decides to deliver commands to the
> "SCSI layer" out of order.
> And even if it does deliver command out of order (there might be good
> reasons for it) it should build a "sync barrier" for those commands that
> mandate ordering.  This type of problem exist also in FCP (on a single
> connection!) and I assume that the thinking there is to use the per LU
> command ordering to avoid task management PDUs  (like clear a queue)
> arriving before the commands they are referring to.
> 
> I hope this clarifies the issue.
> 
> Julo
> 
> Pierre Labat <[email-masked]> on 09/01/2001 11:39:53
> 
> Subject:  iSCSI: Abort task
> 
> 
> 
> 
> Julian,
> 
> 
> It seems that we need to specify more the "Abort task" to avoid
> some problems. Below are 3 points we should add in the spec:
> 
> 
> 1) An "Abort task" must be sent on the TCP connection that was used
>    to send the command. This does not apply if the connection used
>    for the command has been "logged out". In this case of course, the
>    "Abort task" can be send on another  connection and it will not hurt.
> 
>   It is to avoid that a command that was delayed in the network, finally
> make it
>   after the "Abort task" as been processed.
> 
> 2) When the initiator receives the "Abort task" response, if the
>   ExpCmdRN returned in the "Abort task" response is
>    <= CmdRN of the aborted task, the initiator must re-use this
>    CmdRN for another task (any).
> 
> 3) The target when receiving an "Abort task"  for a task whose the
>      CmdRN (retreived from the initiator task tag) is >= ExpCmdRN
>      must not bridge this CmdRN. I mean by bridge: allow ExpCmdRN to
>      pass the CmdRN value of the aborted task. It will be up to the
> initiator
>      to send another command with the same CmdRN in order to allow
>      the target to move ExpCmdRN over the CmdRN value of the
>      aborted task.
> 
> 
> 
> "3)"  Is to prevent to following scenario:
> 
> Two TCP connections 1 and 2 in the session
> ExpCmdRN = 2
> 
> a) the command with CmdRN = 3 is sent over  connection1
> 
> b) the command with CmdRN = 4 is sent over connection 2
> 
> c) the command CmdRN = 4 reaches the target
> 
> d) the initiator aborts the command 4 and bridges CmdRN=4
> 
> e) the target receives the command 3, and as CmdRN=4 is bridged,
> advances ExpCmdRN to 5
> 
> f) the initiator sends another command (not related with the aborted
> one) with CmdRN=4
> 
> g) the target drops this command because CmdRN is less than ExpCmdRN.
> 
> 
> 
> "2)" needs to exist because if the initiator doesn't re-use the CmdRN
> the target
> will deadlock because ExpCmdRN will be stuck behind the CmdRN of
> the aborted command.
> 
> 
> 
> An alternative to the points 2) and 3) would be the initiator not
> re-using
> the CmdRN of the aborted commands. But this requires the target to
> bridge
> all the CmdRNs of the Aborted tasks (need extra states) and anyway there
> 
> is a scenario where it doesn't work. Hence this is a bad alternative.
> 
> Scenario where it doesn't work:
> 
> Session with 2 connections 1 and 2.
> ExpCmdRN=1
> 
> a) Command with CmdRN=4 is sent over connection1
> 
> b) Connection 1 drops and the command 4 will never make it to the
> target.
> 
> c) A logout for the connection 1 is sent over connection 2
> 
> d) The initiator sends an "Abort task" (for the task CmdRN=4) over
>     the connection 2
> 
> e) The target, from the referenced initiator task tag cannot find the
>      CmdRN of the aborted task because the initiator task tag
> corresponds
>      to nothing because the command has never been received.
>     Hence the target is not able to bridge CmdRN=4
> 
> f) As the initiator does not re-use CmdRN=4 the target will deadlock
>     when ExpCmdRN will be 4.
> 
> 
> Regards,
> 
> Pierre
> 
> 
> 
> 
> 

-- 
#################################
Santosh Rao
Software Design Engineer,
HP, Cupertino.
email : [email-masked]
Phone : 408-447-3751
#################################