Subject: [Ips] Immediate Commands in iSCSI

From: <[email-masked]>
Date: Thu, 6 Dec 2007 15:39:34 -0600

Ken/Julo,
  We may have been talking past each other here :-)  I agree that
defining a status code for the Response field of the TMF Response PDU is
the simplest option:

0x07 - Immediate Command Reject - Too many immediate commands

0x06 is already defined as "Function authorization failed." Or were you
recommending changing the description for the 0x06 response code?

Thanks,
Jacob

RE: Immediate Commands in iSCSI  <javascript: void Nabble.setFlag();>  
<<Picture (Device Independent Bitmap)>>  <<Picture (Device Independent
Bitmap)>>  <<Picture (Device Independent Bitmap)>>  <<Picture (Device
Independent Bitmap)>> 
by Ken Sandars <http://www.nabble.com/user/UserProfile.jtp?user=989990>
Dec 04, 2007; 07:08pm ::
Reply <http://www.nabble.com/forum/Reply.jtp?post=14163229>  | Reply to
Author <http://www.nabble.com/user/SendEmail.jtp?type=pm&post=14163229>
| Show Only this Message
<http://www.nabble.com/RE%3A-Immediate-Commands-in-iSCSI-p14163229.html>

Hi Jacob,

A Reject PDU which indicates its reason as "Immediate Command Reject -
too many immediate commands" seems pretty granular to me. :-) That's
what the reject code 0x06 means (as Julo had recommended). The initiator
may resubmit the PDU whenever it deems appropriate. Does that make
sense?

IMHO, a target implementation constrains the number of non-TMF commands
and might throttle repeated TMF commands when they are in progress (and
the clearing effects are taking a long time to resolve). Even with a
large number of LU's it would be unusual to constrain the number of
concurrent LU Reset operations. Have you got a trace showing this
behaviour?

HTH,
Ken

Subject: RE: [Ips] Immediate Commands in iSCSI
Date: Mon, 3 Dec 2007 14:15:19 -0600
From: Jacob_Cherian@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=0> 
To: Julian_Satran@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=1> 
CC: ips@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=2> ;
Nam_Nguyen@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=3> 
Julian,
    No offense taken. The scenario that you brought up was considered
since most configurations with iSCSI are with multiple hosts. However,
the allocation of immediate commands to connection/sessions/hosts
depends on the actual implementation. RFC 3720 already requires support
for at least one per connection. Even we cannot agree on a key, having a
more granular status for a rejected TMF that says the target is
currently busy processing a TMF and cannot process another at this time
(but will be able to sometime in the future), helps initiators to wait
and retry, instead of assuming a failure and escalating the reset. 
 
 Jacob
 
From: Julian Satran [mailto:Julian_Satran@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=4> ]

Sent: Monday, December 03, 2007 8:42 AM
To: Cherian, Jacob
Cc: ips@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=5> ;
Nguyen, Nam
Subject: RE: [Ips] Immediate Commands in iSCSI

Jacob, 

I do not want to appear offensive - but did you consider the "larger
picture"? Assuming that the negotiation key or status is there and
assuming that the target talks to 7 (seven) different initiators what
would it tell each of them (it has 7 different sessions)? Should he
reserve a quota for each of them - so that it can meet what it declared?
What should a target do with commands that exceed the number? 

In iSCSI as a rule we avoided setting or negotiating parameters that we
don't need to set/negotiate and I think the number of accepted immediate
commands is one of them. 

Julo 
	RE: [Ips] Immediate Commands in iSCSI

Jacob_Cherian 
to: 
Julian Satran 
12/03/07 04:06 PM

Cc: 
ips, Nam_Nguyen

The goal of the new status or key is to help the initiator to know the
correct number of immediate commands that the target supports. The issue
with overloading a status that does clearly define the error condition
is that it does help in achieving interoperability between the myriad of
initiators and targets that are out there. I was looking to see if there
was another way that I had not considered. 
  
Jacob 

From: Julian Satran [mailto:Julian_Satran@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=6> ]

Sent: Monday, December 03, 2007 1:23 AM
To: Cherian, Jacob
Cc: ips@...
<http://www.nabble.com/user/SendEmail.jtp?type=post&post=14163229&i=7> ;
Nguyen, Nam
Subject: Re: [Ips] Immediate Commands in iSCSI 

It looks to  me that the Reject response with reason 0x06 should be
enough for the initiator to handle the situation you describe. 
If the TM are the cause then each of those has a response indicating
termination. After receiving a reject an initiator should reissue new
TMs only after getting back one or more responses.  The initiator may
even accurately guess the number of commands the targets is capable of
handling per connection and per LU. The only case in which this handling
may be awkward is over a WAN but if you are operating over a WAN and TM
performance is your only worry you are in good shape :-) 

Julo 
	 
[Ips] Immediate Commands in iSCSI

 
Jacob_Cherian 
to: 
ips 
12/03/07 05:23 AM

 
Cc: 
Nam_Nguyen

 

RFC 3720 states: 

The number of commands used for immediate delivery is not limited and
their delivery for execution is not acknowledged through the numbering
scheme. Immediate commands MAY be rejected by the iSCSI target layer due
to a lack of resources. An iSCSI target MUST be able to handle at least
one immediate task management command and one immediate
non-task-management iSCSI command per connection at any time. 

With storage systems that have very large number of logical units, we
run into an issue when we use LU Reset or Abort Task for reset recovery.
I couldn't find a way to tell an initiator how many immediate commands a
target can accept. The issue with this is that if we have to issue
multiple task management commands and it is more than what the target
can accept, it will reject the task management request. Since we don't
know why it was rejected, we will have to escalate the reset. This can
be very disruptive with nodes that have very large number of logical
units. There may be two ways to resolve this: 

1)      Define a new Text negotiation key for number of outstanding
immediate commands supported. Unlike non-immediate (data) commands where
we can send TASK SET FULL, there is no way for a target to report in the
Task Management Function Response PDU that is unable to process the
current request before it is out of resources.  The default vaule for
this new key can be 1, so if this key is not negotiated initiators can
assume one per connection. 

2)      Define a new value for the Response field in the Task Management
Function Response PDU for reporting that the target cannot process the
request currently. 

Any suggestions on how to resolve this? 

Thanks, 

Jacob 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Andy Spry <[email-masked]>
Date: Thu, 6 Dec 2007 15:54:36 -0600

Jacob,

I think that 0x06 code refers to the Reason code for an iSCSI Reject  
PDU.

The sequence would be

I->  Sends TMF Request PDU as immediate command, exceeding the  
capacity of the target for immediate commands.

T-> Responds with a Reject PDU with Reason code 0x06 (too many  
immediate commands).

The TMF Request is blocked at the door and never makes it to the TMF  
Request handler to generate a TMF Response.

The payload of the Reject PDU is the header of the TMF Request PDU, so  
the entire TMF request is returned to the initiator where it could be  
re-issued.  The hard part is determining when the load on the target  
has been reduced so the command can be resent.

Dr. Andrew J. Spry
Storage Architect
LSI Corporation

[email-masked]

On Dec 6, 2007, at 3:39 PM, <[email-masked]> wrote:

> Ken/Julo,
>
>   We may have been talking past each other here J  I agree that  
> defining a status code for the Response field of the TMF Response  
> PDU is the simplest option:
>
>
> 0x06 is already defined as “Function authorization failed.” Or were  
> you recommending changing the description for the 0x06 response code?
>
> Thanks,
>
> Jacob
>
>
> RE: Immediate Commands in iSCSI
>
> by Ken Sandars Dec 04, 2007; 07:08pm ::
>
> Reply | Reply to Author | Show Only this Message
>
> Hi Jacob,
>
> A Reject PDU which indicates its reason as "Immediate Command Reject  
> - too many immediate commands" seems pretty granular to me. :-)  
> That's what the reject code 0x06 means (as Julo had recommended).  
> The initiator may resubmit the PDU whenever it deems appropriate.  
> Does that make sense?
>
> IMHO, a target implementation constrains the number of non-TMF  
> commands and might throttle repeated TMF commands when they are in  
> progress (and the clearing effects are taking a long time to  
> resolve). Even with a large number of LU's it would be unusual to  
> constrain the number of concurrent LU Reset operations. Have you got  
> a trace showing this behaviour?
>
>
> HTH,
> Ken
>
>
> CC: ips@...; Nam_Nguyen@...
>
> Julian,
>
>     No offense taken. The scenario that you brought up was  
> considered since most configurations with iSCSI are with multiple  
> hosts. However, the allocation of immediate commands to connection/ 
> sessions/hosts depends on the actual implementation. RFC 3720  
> already requires support for at least one per connection. Even we  
> cannot agree on a key, having a more granular status for a rejected  
> TMF that says the target is currently busy processing a TMF and  
> cannot process another at this time (but will be able to sometime in  
> the future), helps initiators to wait and retry, instead of assuming  
> a failure and escalating the reset.
>
>
>  Jacob
>
>
> From: Julian Satran [mailto:Julian_Satran@...]
> Cc: ips@...; Nguyen, Nam
> assuming that the target talks to 7 (seven) different initiators  
> what would it tell each of them (it has 7 different sessions)?  
> Should he reserve a quota for each of them - so that it can meet  
> what it declared? What should a target do with commands that exceed  
> the number?
>
> In iSCSI as a rule we avoided setting or negotiating parameters that  
> we don't need to set/negotiate and I think the number of accepted  
> immediate commands is one of them.
>
> Julo
>
> The goal of the new status or key is to help the initiator to know  
> the correct number of immediate commands that the target supports.  
> The issue with overloading a status that does clearly define the  
> error condition is that it does help in achieving interoperability  
> between the myriad of initiators and targets that are out there. I  
> was looking to see if there was another way that I had not considered.
>
> Jacob
>
>
> From: Julian Satran [mailto:Julian_Satran@...]
> Cc: ips@...; Nguyen, Nam
> termination. After receiving a reject an initiator should reissue  
> new TMs only after getting back one or more responses.  The  
> initiator may even accurately guess the number of commands the  
> targets is capable of handling per connection and per LU. The only  
> case in which this handling may be awkward is over a WAN but if you  
> are operating over a WAN and TM performance is your only worry you  
> are in good shape :-)
>
> Julo
>
>
> The number of commands used for immediate delivery is not limited  
> and their delivery for execution is not acknowledged through the  
> numbering scheme. Immediate commands MAY be rejected by the iSCSI  
> target layer due to a lack of resources. An iSCSI target MUST be  
> able to handle at least one immediate task management command and  
> one immediate non-task-management iSCSI command per connection at  
> any time.
>
>
>
> With storage systems that have very large number of logical units,  
> we run into an issue when we use LU Reset or Abort Task for reset  
> recovery. I couldn’t find a way to tell an initiator how many  
> immediate commands a target can accept. The issue with this is that  
> if we have to issue multiple task management commands and it is more  
> than what the target can accept, it will reject the task management  
> request. Since we don’t know why it was rejected, we will have to  
> escalate the reset. This can be very disruptive with nodes that have  
> very large number of logical units. There may be two ways to resolve  
> this:
>
>
>
> immediate commands supported. Unlike non-immediate (data) commands  
> where we can send TASK SET FULL, there is no way for a target to  
> report in the Task Management Function Response PDU that is unable  
> to process the current request before it is out of resources.  The  
> default vaule for this new key can be 1, so if this key is not  
> negotiated initiators can assume one per connection.
>
> 2)      Define a new value for the Response field in the Task  
> Management Function Response PDU for reporting that the target  
> cannot process the request currently.
>
>
>