Subject: [Ips] Immediate Commands in iSCSI

From: <[email-masked]>
Date: Sun, 2 Dec 2007 21:20:37 -0600

RFC 3720 states:

The number of commands used for immediate delivery is not limited and
their delivery for execution is not acknowledged through the numbering
scheme. Immediate commands MAY be rejected by the iSCSI target layer due
to a lack of resources. An iSCSI target MUST be able to handle at least
one immediate task management command and one immediate
non-task-management iSCSI command per connection at any time.

With storage systems that have very large number of logical units, we
run into an issue when we use LU Reset or Abort Task for reset recovery.
I couldn't find a way to tell an initiator how many immediate commands a
target can accept. The issue with this is that if we have to issue
multiple task management commands and it is more than what the target
can accept, it will reject the task management request. Since we don't
know why it was rejected, we will have to escalate the reset. This can
be very disruptive with nodes that have very large number of logical
units. There may be two ways to resolve this:

1)      Define a new Text negotiation key for number of outstanding
immediate commands supported. Unlike non-immediate (data) commands where
we can send TASK SET FULL, there is no way for a target to report in the
Task Management Function Response PDU that is unable to process the
current request before it is out of resources.  The default vaule for
this new key can be 1, so if this key is not negotiated initiators can
assume one per connection.

2)      Define a new value for the Response field in the Task Management
Function Response PDU for reporting that the target cannot process the
request currently.

Any suggestions on how to resolve this?

Thanks,

Jacob

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: William Stouder-Studenmund <[email-masked]>
Date: Sun, 2 Dec 2007 21:54:11 -0800

On Dec 2, 2007, at 7:20 PM, [email-masked] wrote:

> RFC 3720 states:
> The number of commands used for immediate delivery is not limited  
> and their delivery for execution is not acknowledged through the  
> numbering scheme. Immediate commands MAY be rejected by the iSCSI  
> target layer due to a lack of resources. An iSCSI target MUST be  
> able to handle at least one immediate task management command and  
> one immediate non-task-management iSCSI command per connection at  
> any time.
>
> With storage systems that have very large number of logical units,  
> we run into an issue when we use LU Reset or Abort Task for reset  
> recovery. I couldn’t find a way to tell an initiator how many  
> immediate commands a target can accept. The issue with this is that  
> if we have to issue multiple task management commands and it is more  
> than what the target can accept, it will reject the task management  
> request. Since we don’t know why it was rejected, we will have to  
> escalate the reset. This can be very disruptive with nodes that have  
> very large number of logical units. There may be two ways to resolve  
> this:
>
> immediate commands supported. Unlike non-immediate (data) commands  
> where we can send TASK SET FULL, there is no way for a target to  
> report in the Task Management Function Response PDU that is unable  
> to process the current request before it is out of resources.  The  
> default vaule for this new key can be 1, so if this key is not  
> negotiated initiators can assume one per connection.

If we were to add this key, I think there should be no default. No  
negotiation result should leave both parties with the current "try it  
and see what happens" behavior.

> 2)      Define a new value for the Response field in the Task  
> Management Function Response PDU for reporting that the target  
> cannot process the request currently.
>

Since we're talking about changing the protocol, why not just fix the  
problem? The problem is a lack of clarity about why an operation  
failed, so let's add an extra response code to indicate a transient  
issue. Something to the effect of, "I'm rejecting/failing this, but I  
think the reasons was transient and may not hold true if you try the  
command again."

I'm not sure if anything other than TMR needs this treatment.

Take care,

Bill

---

From: Julian Satran <[email-masked]>
Date: Mon, 3 Dec 2007 09:23:24 +0200

It looks to  me that the Reject response with reason 0x06 should be enough 
for the initiator to handle the situation you describe.
If the TM are the cause then each of those has a response indicating 
termination. After receiving a reject an initiator should reissue new TMs 
only after getting back one or more responses.  The initiator may even 
accurately guess the number of commands the targets is capable of handling 
per connection and per LU. The only case in which this handling may be 
awkward is over a WAN but if you are operating over a WAN and TM 
performance is your only worry you are in good shape :-)

Julo

[Ips] Immediate Commands in iSCSI

Jacob_Cherian 
to:
ips
12/03/07 05:23 AM

Cc:
Nam_Nguyen

RFC 3720 states:

With storage systems that have very large number of logical units, we run 
into an issue when we use LU Reset or Abort Task for reset recovery. I 
couldn?t find a way to tell an initiator how many immediate commands a 
target can accept. The issue with this is that if we have to issue 
multiple task management commands and it is more than what the target can 
accept, it will reject the task management request. Since we don?t know 
why it was rejected, we will have to escalate the reset. This can be very 
disruptive with nodes that have very large number of logical units. There 
may be two ways to resolve this:

1)      Define a new Text negotiation key for number of outstanding 
immediate commands supported. Unlike non-immediate (data) commands where 
we can send TASK SET FULL, there is no way for a target to report in the 
Task Management Function Response PDU that is unable to process the 
current request before it is out of resources.  The default vaule for this 
new key can be 1, so if this key is not negotiated initiators can assume 
one per connection.

Thanks,

Jacob

---

From: <[email-masked]>
Date: Mon, 3 Dec 2007 08:05:28 -0600

The goal of the new status or key is to help the initiator to know the
correct number of immediate commands that the target supports. The issue
with overloading a status that does clearly define the error condition
is that it does help in achieving interoperability between the myriad of
initiators and targets that are out there. I was looking to see if there
was another way that I had not considered.

Jacob

From: Julian Satran [mailto:[email-masked]] 
Sent: Monday, December 03, 2007 1:23 AM
To: Cherian, Jacob
Cc: [email-masked]; Nguyen, Nam
Subject: Re: [Ips] Immediate Commands in iSCSI

It looks to  me that the Reject response with reason 0x06 should be
enough for the initiator to handle the situation you describe. 
If the TM are the cause then each of those has a response indicating
termination. After receiving a reject an initiator should reissue new
TMs only after getting back one or more responses.  The initiator may
even accurately guess the number of commands the targets is capable of
handling per connection and per LU. The only case in which this handling
may be awkward is over a WAN but if you are operating over a WAN and TM
performance is your only worry you are in good shape :-) 

Julo 

Jacob_Cherian 

to: 

ips 

12/03/07 05:23 AM

Cc: 

Nam_Nguyen

________________________________

RFC 3720 states: 

Thanks, 

Jacob 
_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips
<https://www1.ietf.org/mailman/listinfo/ips>

---

From: Julian Satran <[email-masked]>
Date: Mon, 3 Dec 2007 16:42:29 +0200

Jacob,

I do not want to appear offensive - but did you consider the "larger 
picture"? Assuming that the negotiation key or status is there and 
assuming that the target talks to 7 (seven) different initiators what 
would it tell each of them (it has 7 different sessions)? Should he 
reserve a quota for each of them - so that it can meet what it declared? 
What should a target do with commands that exceed the number? 

In iSCSI as a rule we avoided setting or negotiating parameters that we 
don't need to set/negotiate and I think the number of accepted immediate 
commands is one of them.

Julo

RE: [Ips] Immediate Commands in iSCSI

Jacob_Cherian 
to:
Julian Satran
12/03/07 04:06 PM

Cc:
ips, Nam_Nguyen

The goal of the new status or key is to help the initiator to know the 
correct number of immediate commands that the target supports. The issue 
with overloading a status that does clearly define the error condition is 
that it does help in achieving interoperability between the myriad of 
initiators and targets that are out there. I was looking to see if there 
was another way that I had not considered.
 
Jacob

Julo 

Jacob_Cherian 
to: 
ips 
12/03/07 05:23 AM

Cc: 
Nam_Nguyen

RFC 3720 states: 

Thanks, 

Jacob

---

From: <[email-masked]>
Date: Mon, 3 Dec 2007 14:15:19 -0600

Julian,

    No offense taken. The scenario that you brought up was considered
since most configurations with iSCSI are with multiple hosts. However,
the allocation of immediate commands to connection/sessions/hosts
depends on the actual implementation. RFC 3720 already requires support
for at least one per connection. Even we cannot agree on a key, having a
more granular status for a rejected TMF that says the target is
currently busy processing a TMF and cannot process another at this time
(but will be able to sometime in the future), helps initiators to wait
and retry, instead of assuming a failure and escalating the reset. 

 Jacob

From: Julian Satran [mailto:[email-masked]] 
Sent: Monday, December 03, 2007 8:42 AM
To: Cherian, Jacob
Cc: [email-masked]; Nguyen, Nam
Subject: RE: [Ips] Immediate Commands in iSCSI

Jacob, 

Julo 

Jacob_Cherian 

to: 

Julian Satran 

12/03/07 04:06 PM

Cc: 

ips, Nam_Nguyen

  
Jacob 

Julo 

  

Jacob_Cherian 

to: 

ips 

12/03/07 05:23 AM

Cc: 

Nam_Nguyen

RFC 3720 states: 

Thanks, 

Jacob

---

From: Ken Sandars <[email-masked]>
Date: Wed, 5 Dec 2007 12:08:00 +1100

Hi Jacob,

A Reject PDU which indicates its reason as "Immediate Command Reject - too many immediate commands" seems pretty granular to me. :-) That's what the reject code 0x06 means (as Julo had recommended). The initiator may resubmit the PDU whenever it deems appropriate. Does that make sense?

IMHO, a target implementation constrains the number of non-TMF commands and might throttle repeated TMF commands when they are in progress (and the clearing effects are taking a long time to resolve). Even with a large number of LU's it would be unusual to constrain the number of concurrent LU Reset operations. Have you got a trace showing this behaviour?

HTH,
Ken

Subject: RE: [Ips] Immediate Commands in iSCSI
Date: Mon, 3 Dec 2007 14:15:19 -0600
From: [email-masked]
To: [email-masked]
CC: [email-masked]; [email-masked]

Julian,

    No offense taken. The scenario that you
brought up was considered since most configurations with iSCSI are with
multiple hosts. However, the allocation of immediate commands to connection/sessions/hosts
depends on the actual implementation. RFC 3720 already requires support for at
least one per connection. Even we cannot agree on a key, having a more granular
status for a rejected TMF that says the target is currently busy processing a
TMF and cannot process another at this time (but will be able to sometime in
the future), helps initiators to wait and retry, instead of assuming a failure
and escalating the reset. 

 Jacob

From: Julian Satran
[mailto:[email-masked]] 

To: Cherian, Jacob

 

Jacob, 

I do not want
to appear offensive - but did you consider the "larger picture"?
Assuming that the negotiation key or status is there and assuming that the
target talks to 7 (seven) different initiators what would it tell each of them
(it has 7 different sessions)? Should he reserve a quota for each of them - so
that it can meet what it declared? What should a target do with commands that
exceed the number? 

In iSCSI as a rule
we avoided setting or negotiating parameters that we don't need to
set/negotiate and I think the number of accepted immediate commands is one of
them. 

Julo 

  

    
    RE:

   
    
    Jacob_Cherian 

    to: 

    Julian
    Satran 

    12/03/07 04:06 PM

   

      
      Cc: 

      ips,
      Nam_Nguyen

   

 

The
goal of the new status or key is to help the initiator to know the correct
number of immediate commands that the target supports. The issue with
overloading a status that does clearly define the error condition is that it
does help in achieving interoperability between the myriad of initiators and
targets that are out there. I was looking to see if there was another way that
I had not considered. 

Jacob

 

From: Julian Satran
[mailto:[email-masked]] 

To: Cherian, Jacob

If the TM are the cause then each of those has a response indicating
termination. After receiving a reject an initiator should reissue new TMs only
after getting back one or more responses.  The initiator may even
accurately guess the number of commands the targets is capable of handling per
connection and per LU. The only case in which this handling may be awkward is
over a WAN but if you are operating over a WAN and TM performance is your only
worry you are in good shape :-) 

Julo 

  

    [Ips]
    Immediate Commands in iSCSI

    

    Jacob_Cherian 

    to: 

    ips

    12/03/07 05:23 AM

    

    

      Cc: 

      Nam_Nguyen

   

  

RFC 3720 states: 

The number of commands used for immediate delivery is not limited and their
delivery for execution is not acknowledged through the numbering scheme.
Immediate commands MAY be rejected by the iSCSI target layer due to a lack of
resources. An iSCSI target MUST be able to handle at least one immediate task
management command and one immediate non-task-management iSCSI command per
connection at any time. 

With storage systems that have very large number of logical units, we run into
an issue when we use LU Reset or Abort Task for reset recovery. I
couldn’t find a way to tell an initiator how many immediate commands a
target can accept. The issue with this is that if we have to issue multiple
task management commands and it is more than what the target can accept, it
will reject the task management request. Since we don’t know why it was
rejected, we will have to escalate the reset. This can be very disruptive with
nodes that have very large number of logical units. There may be two ways to
resolve this: 

1)      Define a new Text negotiation key for number of
outstanding immediate commands supported. Unlike non-immediate (data) commands
where we can send TASK SET FULL, there is no way for a target to report in the
Task Management Function Response PDU that is unable to process the current
request before it is out of resources.  The default vaule for this new key
can be 1, so if this key is not negotiated initiators can assume one per
connection. 

2)      Define a new value for the Response field in the Task
Management Function Response PDU for reporting that the target cannot process
the request currently. 

 

Thanks, 

Jacob 

Ips mailing list

[email-masked]

_________________________________________________________________
It's simple! Sell your car for just $30 at CarPoint.com.au
http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fsecure%2Dau%2Eimrworldwide%2Ecom%2Fcgi%2Dbin%2Fa%2Fci%5F450304%2Fet%5F2%2Fcg%5F801459%2Fpi%5F1004813%2Fai%5F859641&_t=762955845&_r=tig_OCT07&_m=EXT

---

From: "Jijo Chacko" <[email-masked]>
Date: Thu, 6 Dec 2007 14:40:40 +0530

A related question - from the above discussions, i assume that there is no,
application-level, end-to-end flow control from the initiator-to-target, as
it relates to queued commands. As TCP flow control layers already allowed
the command-bytes in to the host,  we have the memory buffers at the TCP
layer, and later commands are just ignored, with ICR return code 0x06. And
initiator re-submits, target re-rejects,  and this repeats this is not good
for internet, as it relates to congestion. When tagged queueing/no of
outstanding commands support added to SCSI, iSCSI was not in anybody's
mind(IMHO).. please dis-agree with me here.

And strongly feel that, we need to have some way of throttling the initiator
to prevent from swamping the target, or  target implementation code should
have the logic to re-submit the un-honored commands from the TCP buffers(or
local buffers), before returning an error code to the initiator. or there
can be an iSCSI, pending-coomands cache at the target stack, to solve this
problem.

In essence, IMHO, we should do some fixes for target iSCSI implementation,
to provide some staging buffers for already tcp-received commands. We should
not be returning wrong error code to the initiator, if we already received
the command at the TCP layer, and the target has the ability to apply it. We
should be returning the error code only if target failed to apply it.

IN the SCSI world, as the number of initiators are typically just 1, and the
data is not travelling across WAN, target can  return "Too many commands
pending" errors.. as it has only local effect, having few meters.

Please correct me, if i am wrong...

thanks,
Jijo Chacko
Bangalore.

On Dec 5, 2007 6:38 AM, Ken Sandars <[email-masked]> wrote:

>  Hi Jacob,
>
> A Reject PDU which indicates its reason as "Immediate Command Reject - too
> many immediate commands" seems pretty granular to me. :-) That's what the
> reject code 0x06 means (as Julo had recommended). The initiator may resubmit
> the PDU whenever it deems appropriate. Does that make sense?
>
> IMHO, a target implementation constrains the number of non-TMF commands
> and might throttle repeated TMF commands when they are in progress (and the
> clearing effects are taking a long time to resolve). Even with a large
> number of LU's it would be unusual to constrain the number of concurrent LU
> Reset operations. Have you got a trace showing this behaviour?
>
> HTH,
> Ken
>
> ------------------------------
> since most configurations with iSCSI are with multiple hosts. However, the
> allocation of immediate commands to connection/sessions/hosts depends on the
> actual implementation. RFC 3720 already requires support for at least one
> per connection. Even we cannot agree on a key, having a more granular status
> for a rejected TMF that says the target is currently busy processing a TMF
> and cannot process another at this time (but will be able to sometime in the
> future), helps initiators to wait and retry, instead of assuming a failure
>  Jacob**
> *From:* Julian Satran [mailto:[email-masked]]
> *Sent:* Monday, December 03, 2007 8:42 AM
> *To:* Cherian, Jacob
> *Cc:* [email-masked]; Nguyen, Nam
> *Subject:* RE: [Ips] Immediate Commands in iSCSI
>
>
>
>
> Jacob,
>
> picture"? Assuming that the negotiation key or status is there and assuming
> that the target talks to 7 (seven) different initiators what would it tell
> each of them (it has 7 different sessions)? Should he reserve a quota for
> each of them - so that it can meet what it declared? What should a target do
> with commands that exceed the number?
>
>      *RE: [Ips] Immediate Commands in iSCSI*
>
>
>
> *Jacob_Cherian *
>
> to:
>
> Julian Satran
>
> 12/03/07 04:06 PM
>
>
>
> Cc:
>
> ips, Nam_Nguyen
>
>
>  ------------------------------
>
>
>
>
> initiators and targets that are out there. I was looking to see if there was
> another way that I had not considered.
>
> Jacob
>
>
> *From:* Julian Satran [mailto:[email-masked]] *
> Sent:* Monday, December 03, 2007 1:23 AM*
> To:* Cherian, Jacob*
> Cc:* [email-masked]; Nguyen, Nam*
> Subject:* Re: [Ips] Immediate Commands in iSCSI
>
>
> awkward is over a WAN but if you are operating over a WAN and TM performance
> is your only worry you are in good shape :-)
>
> Julo
>
>
>
> *[Ips] Immediate Commands in iSCSI*
>
>
>
>
>
> *Jacob_Cherian *
>
> to:
>
> ips
>
> 12/03/07 05:23 AM
>
>
>
>
> Cc:
>
> Nam_Nguyen
>
>
>
>
>  ------------------------------
>
>
>
>
>
>
> RFC 3720 states:
>
> scheme. Immediate commands MAY be rejected by the iSCSI target layer due to
> a lack of resources. An iSCSI target MUST be able to handle at least one
> immediate task management command and one immediate non-task-management
> iSCSI command per connection at any time.
>
>
>
> couldn't find a way to tell an initiator how many immediate commands a
> will reject the task management request. Since we don't know why it was
> rejected, we will have to escalate the reset. This can be very disruptive
> with nodes that have very large number of logical units. There may be two
> ways to resolve this:
>
>
>
> immediate commands supported. Unlike non-immediate (data) commands where we
> can send TASK SET FULL, there is no way for a target to report in the Task
> Management Function Response PDU that is unable to process the current
> [email-masked]*
> *https://www1.ietf.org/mailman/listinfo/ips
>
>
> ------------------------------
> Sell your car for just $30 at CarPoint.com.au. It's simple!<http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fsecure%2Dau%2Eimrworldwide%2Ecom%2Fcgi%2Dbin%2Fa%2Fci%5F450304%2Fet%5F2%2Fcg%5F801459%2Fpi%5F1004813%2Fai%5F859641&_t=762955845&_r=tig_OCT07&_m=EXT>
>

---

From: [email-masked]
Date: Thu, 6 Dec 2007 09:46:20 -0500

Jijo,

> A related question - from the above discussions, i assume that
> there is no, application-level, end-to-end flow control from
> the initiator-to-target, as it relates to queued commands. As
> TCP flow control layers already allowed the command-bytes in
> to the host,  we have the memory buffers at the TCP layer,
> and later commands are just ignored, with ICR return code 0x06.
> And initiator re-submits, target re-rejects,  and this repeats
> this is not good for internet, as it relates to congestion.
> When tagged queueing/no of outstanding commands support added
> to SCSI, iSCSI was not in anybody's mind(IMHO).. please
> dis-agree with me here.

Happy to do so, as that last statement is incorrect, because
the initial assumption is wrong.  Tagged command queuing is in
SAM-2, which is normatively referenced by iSCSI (RFC 3720).  The
mechanism that you're looking for is TASK SET FULL status (28h),
which is widely supported.  The recent discussion on this list
was specific to iSCSI commands, which are intended for exceptional
cases where the outstanding queue of undelivered commands
in iSCSI needs to be bypassed (see Section 3.2.2.1 of RFC 3720).

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: [email-masked]
Date: Thu, 6 Dec 2007 09:53:13 -0500

Crucial editing mistake - see below.  Sorry, --David

> was specific to iSCSI commands, which are intended for exceptional

"iSCSI commands" --> "immediate iSCSI commands"