Subject: iSCSI: Use of the A bit

From: Eddy Quicksall <[email-masked]>
Date: Wed, 13 Mar 2002 10:12:17 -0500

Here is a case that I want to go over and if there is not already a
solution, I think a refinement to the A bit could solve it.
 
The problem is that a target (perhaps an iSCSI disk drive) does not have
enough memory to transfer the full READ request so it must read from the
medium as much as it can, transmit that, when that transmission is known to
be good, read the next bunch, transmit that and so on.
 
The problem we have is that the target must keep the buffer around until the
transfer has been "ack'd" via ExpStatSN. But that status can't be sent
because all of the requested data has not been sent. So the target would
have to refuse to do the command.
 
I was going to use the A bit for this thinking it would force the initiator
to give an "ack" but our current wording does not make this a sure fire
thing:
 
1) The initiator may not want to run at ErrorRecoveryLevel 1.
2) The initiator may ignore the A bit if it deems that the bit is being set
aggressively.
3) The target may set the A bit no more frequently than MaxBurstSize.
 
Comments?
 
mailto:[email-masked] <mailto:[email-masked]>

---

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Wed, 13 Mar 2002 18:39:27 -0000

Eddy,
 
The target is quite within its rights to use the A bit when at recovery
level 0.  If the session is re-established due to recovery 7.11.4 then the
relevant command is aborted anyway and so there is no reason to keep hold of
the data any way: With recovery level 0 there is no recovery mechanism that
requires the target to keep the data.  Therefore the A bit is redundant when
the recovery level is 0.
 
The spec says that the initiator MUST issue a SNACK if the A bit is set.
However, the MaxBurstSize restriction is there to prevent the initiator from
having to send a SNACK on every PDU in the case where a target inadvertently
sets the A bit in (for example) every data in PDU. The target may set the A
bit more often than the MaxBurstSize but it should not expect a SNACK more
often than this.
 
Matthew Burbridge

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

 

 
Comments?

---

From: Sukha Ghosh <[email-masked]>
Date: Wed, 13 Mar 2002 16:01:50 -0500

Is it possible that A bit is defined to be valid when F bit is '1' in
Data-In PDU so that the target sets the A bit (if it requires DataACK SNACK)
only on the last Data-In PDU of a sequence. This may reduce some
aggressiveness and still satisfies the DataACK requirement from Initiator.

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:03 PM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit

Then my only concern is that the initiator may ignore the A bit if it deems
that the bit is being set aggressively.
 
If it ignored it, then the target would be stalled waiting for he ACK.
 
Eddy

-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 1:39 PM
To: 'Eddy Quicksall'; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit
Importance: High

Eddy,

 
Matthew Burbridge

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

 

 
Comments?

---

From: "Julian Satran" <[email-masked]>
Date: Thu, 14 Mar 2002 03:17:12 +0200

Eddy,

You need the target to support SNACK so that you need some recovery
capabilities.
And yes the consensus of the group was that the scenario you describe is
not of paramount importance.
So you may ask for ACK and keep the data for a while hoping to get it :-)

Julo

                      Eddy Quicksall                                                         
                      <Eddy_Quicksall@i        To:       "BURBRIDGE,MATTHEW                  
                      vivity.com>               (HP-UnitedKingdom,ex2)"                      
                      Sent by:                  <[email-masked]>, "ips@ece. cmu.   
                      [email-masked]         edu (E-mail)" <[email-masked]>              
                      .edu                     cc:                                           
                                               Subject:  RE: iSCSI: Use of the A bit         
                                                                                             
                      13-03-02 22:02                                                         
                      Please respond to                                                      
                      Eddy Quicksall                                                         

Eddy
      -----Original Message-----
      From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
      [mailto:[email-masked]]
      Sent: Wednesday, March 13, 2002 1:39 PM
      To: 'Eddy Quicksall'; ips@ece. cmu. edu (E-mail)
      Subject: RE: iSCSI: Use of the A bit
      Importance: High

      Eddy,

      The target is quite within its rights to use the A bit when at
      recovery level 0.  If the session is re-established due to recovery
      7.11.4 then the relevant command is aborted anyway and so there is no
      reason to keep hold of the data any way: With recovery level 0 there
      is no recovery mechanism that requires the target to keep the data.
      Therefore the A bit is redundant when the recovery level is 0.

      The spec says that the initiator MUST issue a SNACK if the A bit is
      set.  However, the MaxBurstSize restriction is there to prevent the
      initiator from having to send a SNACK on every PDU in the case where
      a target inadvertently sets the A bit in (for example) every data in
      PDU. The target may set the A bit more often than the MaxBurstSize
      but it should not expect a SNACK more often than this.

      Matthew Burbridge
            -----Original Message-----

            Here is a case that I want to go over and if there is not
            already a solution, I think a refinement to the A bit could
            solve it.

            The problem is that a target (perhaps an iSCSI disk drive) does
            not have enough memory to transfer the full READ request so it
            must read from the medium as much as it can, transmit that,
            when that transmission is known to be good, read the next
            bunch, transmit that and so on.

            The problem we have is that the target must keep the buffer
            around until the transfer has been "ack'd" via ExpStatSN. But
            that status can't be sent because all of the requested data has
            not been sent. So the target would have to refuse to do the
            command.

            I was going to use the A bit for this thinking it would force
            the initiator to give an "ack" but our current wording does not
            make this a sure fire thing:

            1) The initiator may not want to run at ErrorRecoveryLevel 1.
            2) The initiator may ignore the A bit if it deems that the bit
            is being set aggressively.
            3) The target may set the A bit no more frequently than
            MaxBurstSize.

            Comments?

            mailto:[email-masked]

---

From: "John Hufferd" <[email-masked]>
Date: Wed, 13 Mar 2002 18:59:34 -0800

If it hurts, don't do that.  Keep it within MaxBurstSize constraints.  And
set MaxBurstSize correctly for the device.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

Eddy Quicksall <[email-masked]>@ece.cmu.edu on 03/13/2002
12:02:38 PM

Sent by:    [email-masked]

To:    "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)"
       <[email-masked]>, "ips@ece. cmu. edu (E-mail)"
       <[email-masked]>
cc:
Subject:    RE: iSCSI: Use of the A bit

Then  my only concern is that the initiator may ignore the A bit  if it
deems that the bit is being set  aggressively.

Eddy
-----Original Message-----

Eddy,

The target is quite within its rights to use the A  bit when at recovery
level 0.  If the session is re-established due to  recovery 7.11.4 then the
relevant command is aborted anyway and so there is no  reason to keep hold
of the data any way: With recovery level 0 there is  no recovery mechanism
that requires the target to keep the data.   Therefore the A bit is
redundant when the recovery level is  0.

The spec says that the initiator MUST issue a SNACK  if the A bit is set.
However, the MaxBurstSize restriction is there to  prevent the initiator
from having to send a SNACK on every PDU in the case  where a target
inadvertently sets the A bit in (for example) every data in  PDU. The
target may set the A bit more often than the MaxBurstSize but it  should
not expect a SNACK more often than this.

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall  [mailto:[email-masked]]
Sent: Wednesday, March 13,  2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject:  iSCSI: Use of the A bit

The problem is  that a target (perhaps an iSCSI disk drive) does not have
enough memory to  transfer the full READ request so it must read from the
medium as much  as it can, transmit that, when that transmission is known
to be good, read  the next bunch, transmit that and so on.

The problem we  have is that the target must keep the buffer around until
the transfer has  been "ack'd" via ExpStatSN. But that status can't be sent
because all of the  requested data has not been sent. So the target would
have to refuse to do  the command.

I was going to  use the A bit for this thinking it would force the
initiator to give an  "ack" but our current wording does not make this a
sure fire  thing:

1) The initiator  may not want to run at ErrorRecoveryLevel 1.
2) The initiator  may ignore the A bit if it deems that the bit is being
set  aggressively.
3) The target  may set the A bit no more frequently than MaxBurstSize.

Comments?

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 14 Mar 2002 00:54:10 -0800

I am not sure that was the problem.  The point was if they have such a
device that has such a shortage of memory that it can never complete the
I/O without running out of buffer.  It wants to free up the buffer so that
it can send some more data so that it can fulfill the Read request.

If the Initiator supports the A-bit SNACK (which is optional), then some
buffer control would be possible.  But as was pointed out by Matthew, with
ErrorRecoveryLevel=0, there is no need to even use the A-bit, since there
is no error recovery within command, you can just send the data, then
refill the buffer.  If there is an error, the session has to be torn down.

However, if you have that same type of device but support
ErrorRecoveryLevel=1, then SNACK with A-bit support seems like an strong
preference for a corresponding initiator.  If the device detects that the
A-bit is not supported at the initiator (because A-bit was set, but no
response was received at the approprate after a period of time), then it
probably should "promote" the recovery to level=0, and start refilling the
buffers as mentioned above.

.
.
.
John L. Hufferd

Santosh Rao <[email-masked]>@ece.cmu.edu on 03/13/2002 07:25:21
PM

To:    [email-masked] (ips)
cc:
Subject:    Re: iSCSI: Use of the A bit

>
> Actually it does not actually say that.  It says "This is limited to
> sessions that support error recovery and is implemented through the A bit
> ..."
>
> I know what I am about to say, is nit picking, but ....
> ErrorRecoveryLevel=0 is an error recovery technique.
>
> Having said that, we probably should just change that statement to "This
is
> implemented through the A bit ..."

John,

Initiators that use "ErrorRecoveryLevel=0" must not be required to respond
to
an A bit setting with a DataACK. "ErrorRecoveryLevel=0" implies the
initiator
need not support/use SNACK pdu.

A target can recycle its data buffers once it has completed sending the
data,
since it knows that the initiator will not issue Data SNACK while operating
as
"ErrorRecoveryLevel=0".

- Santosh

--
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: "BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)" <[email-masked]>
Date: Thu, 14 Mar 2002 12:22:18 -0000

I think that what Santosh and others have stated is the correct think to do
for initiators.  If the revovery level is 0 and the A bit is set then the
initiator is quite with in its rights to ignore the A bit (If the target is
at fault then penalise the target and not the initiator).

IMHO changing the spec to make it more specific is not the right thing and
it could lead to inter operability problems.  However, I do think something
needs to be added so I propose adding the statement:

"If ErrorRecoveryLevel=0 then the initiator MAY ignore the A bit (i.e.
assume A=0)".

One could argue for the MAY to be a MUST but if there is a kind initiator
out there then let it do its stuff - it can only help the rogue target.

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

	Actually 9.7.2 says almost exactly that ...

	"For sessions with ErrorRecoveryLevel 1 or higher, the target sets
this bit to 1 to indicate that it requests a positive acknowledgement
from the initiator for the data received."

	We've interpreted that to mean our initiator should ignore the A bit
if ErrorRecoveryLevel is 0.

	Having said that it is very easy to support just the positive form
of
SNACK, so I would have no objection to removing the A bits current
link to ErrorRecoveryLevel.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
John Hufferd
Sent: Thursday, March 14, 2002 3:12 AM
To: Rod Harrison
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit

Actually it does not actually say that.  It says "This is limited to
sessions that support error recovery and is implemented through the A
bit
..."

Having said that, we probably should just change that statement to
"This is
implemented through the A bit ..."

.
.
.
John L. Hufferd

"Rod Harrison" <[email-masked]>@ece.cmu.edu on 03/13/2002
01:05:51 PM

To:    "Eddy Quicksall" <[email-masked]>,
"BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <[email-masked]>, "ips@ece.
cmu.
       edu (E-mail)" <[email-masked]>
cc:
Subject:    RE: iSCSI: Use of the A bit

Eddy,

 You were right first time, in draft 11 section 2.5.1.5 paragraph 5
and 9.7.2 state that the A-bit is only supported if ErrorRecoveryLevel
is 1 or higher.

 - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Eddy Quicksall
Sent: Wednesday, March 13, 2002 8:03 PM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece. cmu. edu
(E-mail)
Subject: RE: iSCSI: Use of the A bit

Eddy
-----Original Message-----

Eddy,

The spec says that the initiator MUST issue a SNACK if the A bit is
set.  However, the MaxBurstSize restriction is there to prevent the
initiator from having to send a SNACK on every PDU in the case where a
target inadvertently sets the A bit in (for example) every data in
PDU. The target may set the A bit more often than the MaxBurstSize but
it should not expect a SNACK more often than this.

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

The problem is that a target (perhaps an iSCSI disk drive) does not
have enough memory to transfer the full READ request so it must read
from the medium as much as it can, transmit that, when that
transmission is known to be good, read the next bunch, transmit that
and so on.

The problem we have is that the target must keep the buffer around
until the transfer has been "ack'd" via ExpStatSN. But that status
can't be sent because all of the requested data has not been sent. So
the target would have to refuse to do the command.

I was going to use the A bit for this thinking it would force the
initiator to give an "ack" but our current wording does not make this
a sure fire thing:

1) The initiator may not want to run at ErrorRecoveryLevel 1.
2) The initiator may ignore the A bit if it deems that the bit is
being set aggressively.
3) The target may set the A bit no more frequently than MaxBurstSize.

Comments?

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 14 Mar 2002 08:51:17 -0500

I see a lot of responses talking about ErrorRecoveryLevel = 0. That was not
my original point. ErrorRecoveryLevel = 0 already has this solved. I am
talking about ErrorRecoveryLevel > 0 (I made an error in my first EMAIL and
the point may have been lost).

If the initiator has the option to ignore the A bit, the target will get
stuck. What is wrong with requiring the initiator to respond? Can we just
get that answered first?

I suspect that there is a concern that this may cause too much traffic. My
answer would be "don't buy a cheep target ... their lack of memory may cause
degraded performance" (not intended for the spec, of course :-) ).

I also do not see the rational in not allowing the A bit more often than
MaxBurstSize. The ULP layers may not know anything about the transport
layers. So their resource issues are not known by the transport(s). The
MaxBurstSize is an iSCSI issue and may not apply to future protocols.

So, why not just say something like:

1) The initiator MUST send a DataACK SNACK if it sees the A bit set (and if
ErrorRecoveryLevel > 0)
2) The A bit may be set only when the F bit is set.

And leave it go at that?

Eddy

-----Original Message-----
From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
[mailto:[email-masked]]
Sent: Thursday, March 14, 2002 7:22 AM
To: 'Rod Harrison'; John Hufferd
Cc: Eddy Quicksall; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

	- Rod

-----Original Message-----

.
.
.
John L. Hufferd

Eddy,

 - Rod

-----Original Message-----

Eddy
-----Original Message-----

Eddy,

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

Comments?

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 14 Mar 2002 08:55:32 -0500

Why is it not of paramount importance? 

How would the target throttle its Data-In's if DataACK is optional when
ErrorRecoveryLevel != 0?

I don't know a lot about FC but parallel SCSI has this capability.

Eddy

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 8:17 PM
To: Eddy Quicksall
Cc: ips@ece. cmu. edu (E-mail); BURBRIDGE,MATTHEW
(HP-UnitedKingdom,ex2); [email-masked]
Subject: RE: iSCSI: Use of the A bit

Eddy,

Julo

                      Eddy Quicksall

                      Sent by:                  <[email-masked]>,
"ips@ece. cmu.   
                      [email-masked]         edu (E-mail)"
<[email-masked]>              
                      .edu                     cc:

                                               Subject:  RE: iSCSI: Use of
the A bit         

                      13-03-02 22:02

                      Please respond to

                      Eddy Quicksall

Eddy
      -----Original Message-----
      From: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2)
      [mailto:[email-masked]]
      Sent: Wednesday, March 13, 2002 1:39 PM
      To: 'Eddy Quicksall'; ips@ece. cmu. edu (E-mail)
      Subject: RE: iSCSI: Use of the A bit
      Importance: High

      Eddy,

      Matthew Burbridge
            -----Original Message-----

            Comments?

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 14 Mar 2002 09:14:35 -0500

Regarding my concern about the A bit ... when I read "An initiator MAY
ignore the A bit if it deems that the bit is being set aggressively by the
target (i.e.,before the MaxBurstSize limit is reached)." I missed the "i.e."
part.

Aside from the fact that I was "stupid" for not noticing that, I think the
sentence should be written so it does not need the "i.e.".

Eddy

-----Original Message-----

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

	- Rod

-----Original Message-----

.
.
.
John L. Hufferd

Eddy,

 - Rod

-----Original Message-----

Eddy
-----Original Message-----

Eddy,

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

Comments?

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 14 Mar 2002 11:36:18 -0800

We got to this position, since so many folks did not want to support the
positive ack.  And this approach was put in as a compromise.  I would
suggest, that the thing we are chasing is a small corner case that may not
be smart enough to operate at anything other then ErrorRecoverylevel=0.  In
any event I do not see a problem with limiting the response to MaxBurstSize
intervals.  If the interval is too big, why did the unit negotiate it to
that size?

.
.
.
John L. Hufferd

"LEMAY,KEVIN (A-Roseville,ex1)" <[email-masked]>@ece.cmu.edu on
03/14/2002 10:43:31 AM

To:    "'Eddy Quicksall'" <[email-masked]>, "BURBRIDGE,MATTHEW
       (HP-UnitedKingdom,ex2)" <[email-masked]>, "'Rod Harrison'"
       <[email-masked]>, John Hufferd/San Jose/IBM@IBMUS
cc:    "ips@ece. cmu. edu (E-mail)" <[email-masked]>
Subject:    RE: iSCSI: Use of the A bit

I also do not like the comment about having to wait until maxBurstsize byte
to set the A-bit.

I wanted to use it as Eddie suggested. To force the initiator to
acknowledge
the IO so that I can release resources on the target.

This will be especially useful for hardware accelerated IOs where the chip
cannot declare the IO complete until it has been Ack'd.

Could we just change the wording to:
----
For sessions with ErrorRecoveryLevel 1 or higher the target sets this bit
to
1 to indicate that it requests from the initiator a positive
acknowledgement
for the data received. The target should use the A bit moderately; the
A-bit
MAY only be set to 1 at the end of a data sequence.

On receiving a Data-In PDU with the A bit set to 1 the initiator MUST
issued
a SNACK of type DataACK. If the initiator has detected holes in the input
sequence, it MUST postpone issuing the SNACK of type ACKN until the holes
are filled.
---

This way I can always set the A bit at the end of an IO.

Kevin Lemay

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 5:51 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod Harrison'; John
Hufferd
Cc: ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit

I suspect that there is a concern that this may cause too much traffic. My
answer would be "don't buy a cheep target ... their lack of memory may
cause
degraded performance" (not intended for the spec, of course :-) ).

And leave it go at that?

Eddy

-----Original Message-----

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

 - Rod

-----Original Message-----

.
.
.
John L. Hufferd

Eddy,

 - Rod

-----Original Message-----

Eddy
-----Original Message-----

Eddy,

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

Comments?

---

From: [email-masked]
Date: Thu, 14 Mar 2002 12:49:07 -0700

I cannot negotiate the maxBurstSize to a "nice" size because regardless of
what size is chosen, the Ios that I am processing will not line up nicely on
maxBurstSize boundaries.

Consider:

Assume the MaxBurstSize = 128K, This is the only IO on the connection.

1. The initiator sends a 64K read request to the target.
2. The target sends 64K to the initiator. Now it wants to know that all of
the data packets have been received so that resources for the IO can be
released and the IO complete sent to the OS. I cannot set the A-Bit on the
final data PDU because that would violate the clause that says:

"it MAY set the A-bit to 1 once at most every
MaxBurstSize bytes, and MUST NOT do so any more frequently than that."

But since I have the entire IO completed, I could send a Nop-Ping. That
would force the initiator to send the new StatSN which would imply all data
packets have been received. 

But there is also a more complicated case. Our chip allows the higher level
software to break IOs into its own data phases. If more than one phase is
used, then a response is not the last PDU sent and there is no work around
to get the data Ack'd.

Furthermore, negotiating the burstsize low so that I can set the a_bit when
I want has the undesired side effect of allowing only that much data per R2T
request. A definite performance squelcher.

Kevin Lemay

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 11:36 AM
To: LEMAY,KEVIN (A-Roseville,ex1)
Cc: 'Eddy Quicksall'; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod
Harrison'; ips@ece. cmu. edu (E-mail)
Subject: RE: iSCSI: Use of the A bit

.
.
.
John L. Hufferd

Kevin Lemay

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 5:51 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod Harrison'; John
Hufferd

And leave it go at that?

Eddy

-----Original Message-----

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

 - Rod

-----Original Message-----

.
.
.
John L. Hufferd

Eddy,

 - Rod

-----Original Message-----

Eddy
-----Original Message-----

Eddy,

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

Comments?

---

From: Eddy Quicksall <[email-masked]>
Date: Thu, 14 Mar 2002 15:02:36 -0500

> why did the unit negotiate it to that size?

Well, if things are layered properly, a ULP would be able to use several
transports and at the same time. And the ULP could have dynamic needs.

The ULP should be separate enough that it only issues a "send" request to
the transport and the transport does everything it can to send it. Then, it
informs the ULP that it is finished. The ULP then re-uses the buffer for the
next "send".

The ULP should not understand the transports and especially it should not
have different code for all the different transports.

So, negotiation should depend on the transports needs, not the ULP's.

> We got to this position, since so many folks did not want to support the
positive ack.

You can do this by negotiating for ErrorRecoveryLevel = 0. Also, what is so
difficult about the positive ack when ErrorRecoveryLevel > 0 (especially
when you have all that complex code to do error recovery anyway)?

BTW, parallel SCSI can do this. Why shouldn't iSCSI? What happens when you
want to use the same ULP but upgrade to iSCSI?

Eddy

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 2:36 PM
To: LEMAY,KEVIN (A-Roseville,ex1)
Cc: 'Eddy Quicksall'; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod

.
.
.
John L. Hufferd

Kevin Lemay

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 5:51 AM
To: BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); 'Rod Harrison'; John
Hufferd

And leave it go at that?

Eddy

-----Original Message-----

Matthew

-----Original Message-----
From: Rod Harrison [mailto:[email-masked]]
Sent: Thursday, March 14, 2002 9:27 AM
To: John Hufferd
Cc: Eddy Quicksall; BURBRIDGE,MATTHEW (HP-UnitedKingdom,ex2); ips@ece.
cmu. edu (E-mail)

John,

 - Rod

-----Original Message-----

.
.
.
John L. Hufferd

Eddy,

 - Rod

-----Original Message-----

Eddy
-----Original Message-----

Eddy,

Matthew Burbridge
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Wednesday, March 13, 2002 3:12 PM
To: ips@ece. cmu. edu (E-mail)
Subject: iSCSI: Use of the A bit

Comments?

---

From: Michael Schoberg <[email-masked]>
Date: Thu, 14 Mar 2002 17:14:20 -0600

9.4.3 I have a  more general SNACK question.  When would one use use the
iSCSI, SCSI Response message with sense (reason -> SNACK rejected)?  The
reason for using this hasn't jumped out at me yet.  Does it have to do with
connection recovery?

9.16 States that SNACK support is optional.  Since "ErrorRecoveryLevel"
controls SNACK support, can something in the SNACK message description
reference how this feature is negotiated?

Also a cleanup is needed for SNACK- "RunLength".

   2.5.3.4   SNACK Request

           With the SNACK Request, the initiator requests retransmission of
num-
           bered-responses or data from the target. A single SNACK request
covers
           a contiguous set of missing items called a run of a given type of
           items (the type is indicate in a type field in the PDU header).
The
           run is composed of an initial item (StatSN, DataSN, R2TSN) and
the
           number of additional missed Status, Data, or R2T PDUs (0 means
only
           the initial). For long data-in sequences, the target may request
(at
           predefined minimum intervals) a positive acknowledgement for the
data
           sent. A SNACK request with a type field that indicates ACK and
the
           number of Data-In PDUs acknowledged conveys this positive
acknowl-
           edgement.

9.16
           The SNACK request is used to request the retransmission of
numbered-
           responses, data, or R2T PDUs from the target.  The SNACK request
indi-
           cates the missed numbered-response or data "run" to the target,
where
           the run starts with the first missed StatSN, DataSN, or R2TSN and
           indicates also the number of missed Status, Data, or R2T PDUs (0
has
           the special meaning of "all after the initial").

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 14 Mar 2002 21:54:50 -0800

A lot of this thinking on this thread reflects the "send and Ack response"
type design.  The work last year rejected that thinking and we overtly
designed against that approach with iSCSI.  This is because, at distance,
this can be a major performance issue.  (round trip delay)

Anyway, that is why people did not want to get into the positive Ack, mode,
since 99.9999999% of the time, TCP/IP will deliver everything just fine.
Then if you look at the probability that TCP/IP thinks it did its job, but
did not detect the problem and the CRC-32c detects it, the probability is
even smaller.

To base a design on requirements for Positive Ack, when the probability of
failure is so remote, is in many people's opinions the wrong way to build
product.  And since our products need to inter-operate, and inter-operate
at distance, what one does poorly effects everyone's product.

When you put a Positive Ack into place, you encourage the type of thinking
we have been seeing on this thread, and that hurts others!  That is the
reason that folks did not like the idea of a positive Ack.

We rolled over with this small "A" bit approach because of issues with Tape
Drives since even if Rare, the resultant problem with tape can be severe,
but we put the restrictions on it that you currently see.

To make things work at distance, yes you need some additional buffering,
somewhere.

I heard the arguments that I cant get my ULP to give me enough buffers to
work with, I do not buy that argument, and I think folks are reaching for
straws to try to prove their point.  I do not doubt there will be some
really dumb devices that do not have enough buffers to do a really good
job, but that was the reason that ErrorRecoveryLevel=0 is there.  It will
happen very very rarely, that the CRC-32c will find a problem that TCP/IP
does not find.  So if you do not have the buffers to handle this "once in a
bluemoon" occurrence, then restart the session when this rare occurrence
happens.  But there is no need to impact the other implementations with a
design that will show up poorly at distance.

.
.
.
John L. Hufferd

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 15 Mar 2002 13:43:40 +0200

Thanks for catching 2.5.3.4 conflict (it was not updated together with 9).
It reads now:

With the SNACK Request, the initiator requests retransmission of 
num-bered-responses or data from the target. A single SNACK request covers 
a contiguous set of missing items called a run of a given type of items 
(the type is indicate in a type field in the PDU header). The run is 
composed of an initial item (StatSN, DataSN, R2TSN) and the number of 
missed Status, Data, or R2T PDUs. For long data-in sequences, the target 
may request (at predefined minimum intervals) a positive acknowledgement 
for the data sent. A SNACK request with a type field that indicates ACK 
and the number of Data-In PDUs acknowl-edged conveys this positive 
acknowledgement. 

SNACKs can be rejected when the target does not have the data or status or 
has never sent it (error in error recovery :-)).

Julo

Michael Schoberg <[email-masked]>
Sent by: [email-masked]
15-03-02 01:14
Please respond to Michael Schoberg

        To:     "IPS Reflector (E-mail)" <[email-masked]>
        cc: 
        Subject:        RE: iSCSI: Use of the A bit

9.4.3 I have a  more general SNACK question.  When would one use use the 
iSCSI, SCSI Response message with sense (reason -> SNACK rejected)?  The 
reason for using this hasn't jumped out at me yet.  Does it have to do 
with connection recovery?
9.16 States that SNACK support is optional.  Since "ErrorRecoveryLevel" 
controls SNACK support, can something in the SNACK message description 
reference how this feature is negotiated?
Also a cleanup is needed for SNACK- "RunLength".

   2.5.3.4   SNACK Request

           With the SNACK Request, the initiator requests retransmission 
of num-
           bered-responses or data from the target. A single SNACK request 
covers
           a contiguous set of missing items called a run of a given type 
of
           items (the type is indicate in a type field in the PDU header). 
The
           run is composed of an initial item (StatSN, DataSN, R2TSN) and 
the
           number of additional missed Status, Data, or R2T PDUs (0 means only
           the initial). For long data-in sequences, the target may request (at
           predefined minimum intervals) a positive acknowledgement for 
the data
           sent. A SNACK request with a type field that indicates ACK and 
the
           number of Data-In PDUs acknowledged conveys this positive 
acknowl-
           edgement.

9.16
           The SNACK request is used to request the retransmission of 
numbered-
           responses, data, or R2T PDUs from the target.  The SNACK 
request indi-
           cates the missed numbered-response or data "run" to the target, 
where
           the run starts with the first missed StatSN, DataSN, or R2TSN 
and
           indicates also the number of missed Status, Data, or R2T PDUs (0 has
           the special meaning of "all after the initial").

---

From: "John Hufferd" <[email-masked]>
Date: Sat, 16 Mar 2002 00:57:49 -0800

To any specific user, performance is judged by their own performance not
the performance of the link or the storage controller over all.

When you read an at-distance disk, and its performance sucks, because of a
Send and Ack design, you will not have good things to say about iSCSI.  And
it is not easy to tell what the problem is.  And it is not really possible
for a customer to determine before purchase that he is buying a poorly
designed target HBA or chip.  In fact they may have tested it out in a
local environment and found it OK, and then when contacted by a remote
initiator, that remote unit is very disappointed.  This gives everyone a
bad name.  Even if the Links are well used and the overall efficiency of
the storage controller good, if the individual performance is bad, iSCSI
will be seen to be bad.

Send and Ack designs are bad designs, and in almost all cases are not
needed.  And it gives iSCSI a bad name in the environment we have been
claiming as our own, the at-distance environment.

.
.
.
John L. Hufferd

Martins Krikis <[email-masked]>@ece.cmu.edu on 03/15/2002 01:45:14 PM

To:    [email-masked]
cc:

Just a quick note. Some people have implied that the
use of A-bit necessarily means poor performance.

I would like to disagree. If the target is just
sitting there waiting for acknowledgment to come
back before sending the next Data-In sequence, then
yes, that's poor performance. But I can imagine
targets that keep sending data while waiting for
the outstanding acknowledgments. Yes, it requires
more buffer space, but the acknowledgments are
actually helpful in managing that bigger buffer space.
It's a very similar situation to multiple outstanding
R2T-s. Hopefully everybody agrees that there may be
benefits to those.

Or am I just missing something again?

  Martins Krikis, Intel Corp.

Disclaimer: These opinions are mine and may not
            reflect those of my employer.

__________________________________________________
Do You Yahoo!?
Yahoo! Sports - live college hoops coverage
http://sports.yahoo.com/

---

From: Eddy Quicksall <[email-masked]>
Date: Sat, 16 Mar 2002 09:25:56 -0500

Nobody in this thread has proposed a "send and ack" design.

Eddy

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Saturday, March 16, 2002 3:58 AM
To: Martins Krikis
Cc: [email-masked]
Subject: RE: iSCSI: Use of the A bit

.
.
.
John L. Hufferd

To:    [email-masked]
cc:

---

From: "John Hufferd" <[email-masked]>
Date: Sat, 16 Mar 2002 12:09:42 -0800

Yes, that is what is going on.

Last year positive Acks were Overtly Voted Out by the iSCSI group (we had a
consensus call).  It was felt that since TCP/IP already did the error
detection and recovery -- the additional checking of CRC-32c would seldom
find an error which was not caught by TCP/IP.  Therefore, they did not want
to add stuff to the protocol which would inspire or require folks to
perform Send and Ack types of flow in order to guarantee safe arrival.
Instead, they wanted designs to assume that almost always the data arrived
correctly, and just had a back door recovery whenever the rare situation
occurred that our Digests found an error.  In that way:
* things did not hang around and wait, for Acks,
* there was a reduction in line protocol over-head,
* and at-distance interactions would not have to slow down.

This was a clear consensus, no positive Acks!!!

Well finally the case was made that Tapes needed special handling.  That
is, they needed this type of support, because many of the implementations
simply could not perform a reasonable recovery without adding lots more
buffers then they ever needed before.  Tape buffers are not, as a rule,
pooled memory, they are devoted to specific LUs (Tape Drives).  So any
increase to these memory requirements was simply multiplied by the number
of R/W units in the Library.  So because of this, we added back in, over
much contention, the "A" bit as it is today.

It did not need the Ack on the last I/O, since the tape buffer would not be
reused by anyone else anyway until the next command arrived, at which point
the success of the last operation is determined, etc.  This was not a
problem and the consensus was that it was tolerable in the case of tapes.

Now we have, what I think are contrived situations, in which people are
stating that there is no communication between the SCSI Buffer pool and the
iSCSI implementation so we have to act like a tape.  This is, in my opinion
not a correct assumption,  Buffer management was appropriately adjusted, in
most cases,  between SCSI and FC when FC brought the concepts of Credits
into the picture.   And there needs to be corresponding flexibility in the
buffering when you appropriately work with iSCSI.  The concepts of
immediate data, and unsolicited data, make that need clear.  Further the
use of the iSCSI Command Window management, is different then credits.
This means that to make the best use of iSCSI especially at-distance, where
we can clearly stand out from the other protocols,  we should push back on
this idea of Send and Ack.  We should leave the current Positive SNACK to
the realm of Tape.  However, if it is used for disk, consider it additional
information, that can occur at very infrequent intervals (MaxBurstSize),
which might help buffer management, but is not needed!!!

This thread has tried to come up with designs in which the use of the 'A'
bit is required, for their implementation to work.  I think that is just
wrong design, and will cause all of the other implementations to look bad
when placed in the same network at-distance.

There is not a way for the customer to know that the product they are
buying will not work well at-distance, since that is, after all, one of
iSCSI's claim to fame.  Therefore, it will make all the Initiators, which
interact with such devices, look bad and give all of iSCSI a bad name.

I think we should respect the previous consensus call, and Not add back
into the protocol, positive Acks that are not needed for Tape.

.
.
.
John L. Hufferd

Eddy Quicksall <[email-masked]> on 03/16/2002 06:25:56 AM

To:    John Hufferd/San Jose/IBM@IBMUS, Martins Krikis <[email-masked]>
cc:    [email-masked]
Subject:    RE: iSCSI: Use of the A bit

Eddy

-----Original Message-----
From: John Hufferd [mailto:[email-masked]]
Sent: Saturday, March 16, 2002 3:58 AM
To: Martins Krikis
Cc: [email-masked]
Subject: RE: iSCSI: Use of the A bit

.
.
.
John L. Hufferd

To:    [email-masked]
cc:

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 17 Mar 2002 14:53:51 +0200

You are not missing anything but a consensus was reached and I am not 
willing to reopen the issue.
You may choose to support errorlevel=1.

Julo

Martins Krikis <[email-masked]>
Sent by: [email-masked]
15-03-02 23:45
Please respond to Martins Krikis

        To:     [email-masked]
        cc:

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 18 Mar 2002 03:52:46 +0200

Dear all,

The text for the A bit part (9.7.2) reads now:

   A (Acknowledge) bit

       For sessions with ErrorRecoveryLevel 1 or higher, the target sets
       this bit to 1 to indicate that it requests a positive
       acknowledgement from the initiator for the data received.  The
       target should use the A bit moderately; it MAY set the A bit to 1
       only once every MaxBurstSize bytes or on the last Data-In PDU that
       concludes the entire requested read data transfer for the task from
       the target's perspective, and MUST NOT do so more frequently than
       this.

       On receiving a Data-In PDU with the A bit set to 1, the initiator
       MUST issue a SNACK of type DataACK.  If the initiator has detected
       holes in the input sequence, it MUST postpone issuing the SNACK of
       type DataACK until the holes are filled. An initiator MUST ONLY
       acknowledge the status for command that produced Data-In PDUs after
       acknowledging the Data-In PDUs if Data-In PDU acknowledgment is
       requested by the target. A status acknowledgement for a command that
       generated Data-In PDUs is considered by the target as an implicit
       acknowledgement of the Data-In PDUs if such an acknowledgement was
       requested by the target.

The ITT mandating text in 9,16.1 will read:

       For Status SNACK and DataACK, the Initiator Task Tag is reserved. In
       all other cases, the Initiator Task Tag field MUST be set to the
       Ini-tiator Task Tag of the referenced command.

       Julo

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 18 Mar 2002 02:50:16 +0200

That is correct but the race is not there :-) Julo

                      "Rod Harrison"                                                                     
                      <rod.harrison@win        To:       "Eddy Quicksall" <[email-masked]>,  
                      driver.com>               "Mallikarjun C." <[email-masked]>, <[email-masked]>    
                                               cc:       Julian Satran/Haifa/IBM@IBMIL                   
                      16-03-02 21:10           Subject:  RE: iSCSI: Use of the A bit                     
                      Please respond to                                                                  
                      "Rod Harrison"                                                                     

  >  Sent: Saturday, March 16, 2002 4:15 PM
  >  To: Rod Harrison; Mallikarjun C.; [email-masked]
  >  Cc: [email-masked]
  >  I don't know the original reason why TTT was added but
  >  from my perspective,
  >  I never liked the search for ITT (or hash table). So I
  >  thought it was a good
  >  idea.
  >
  >  Now that it has been added we should be able to get rid
  >  of the ITT and that
  >  will solve the problem you have pointed out.
  >
  >  BTW, the problem you pointed out has nothing to do with
  >  the A bit problem
  >  this thread was addressing. But, I'm really glad you
  >  pointed that out! (It
  >  would have been yet another good reason for the TTT).
  >

Since the SCSI status ends the task and therefore the lifetime of the
initiator task tag the problem I mentioned was specifically caused by
allowing the A bit to be set with the F bit when there is piggyback
status. But you are right, the problem was actually always there since
if the SCSI transfer length was an exact multiple of MaxBurstSize the
A and F bits would be valid together. In fact I now see there are
several other ways this combination would be valid.

Anyway, removing the requirement for the initiator to send the ITT in
the data SNACK solves the problem.

  >  If there is debate on what Rod pointed out, we should
  >  probably start another
  >  thread.
  >
  >  Eddy
  >
  >  Sent: Saturday, March 16, 2002 11:00 AM
  >  To: Eddy Quicksall; Mallikarjun C.; [email-masked]
  >  Cc: [email-masked]
  >          I believe so, but I thought targets were intending to do some
  >  indexing on the ITT? This has always seemed odd to me,
  >  but I thought
  >  it was the case.
  >
  >          - Rod
  >
  >  Sent: Saturday, March 16, 2002 3:57 PM
  >  To: Rod Harrison; Mallikarjun C.; [email-masked]
  >  Cc: [email-masked]
  >  The ITT has no good use in a DataACK SNACK and is
  >  residual after we
  >  added
  >  the TTT. It should  probably be "reserved" for DataACK SNACK.
  >
  >  Would that take care of the problem?
  >
  >  Eddy
  >
  >  Sent: Saturday, March 16, 2002 10:43 AM
  >  To: Eddy Quicksall; Mallikarjun C.; [email-masked]
  >  Cc: [email-masked]
  >  Mallikarjun, Eddy and all,
  >
  >          I think this might be dangerous.
  >
  >          If the initiator receives a DATA-IN with F and A bits
  >  set and a GOOD
  >  SCSI status it would be required to send a data SNACK
  >  PDU containing
  >  an initiator task tag that has "expired" at the initiator.
  >
  >          Initiator task tags are of course reused and there is
  >  no requirement
  >  for the data SNACK to be sent immediately so it would be
  >  possible to
  >  have a single ITT in the network referring to more than 1 task. I
  >  think it is way too late in the game to start
  >  considering conservative
  >  reuse rules for initiator task tags. That could be a potentially
  >  invasive implementation change, especially when you consider that
  >  reusing things is generally a good thing to do from a caching
  >  perspective.
  >
  >          Even if the data SNACK were required to be sent
  >  immediately there is
  >  no guarantee that it would be received by the target
  >  before PDUs for
  >  the task with the reused ITT when there is more than one
  >  connection
  >  per session.
  >
  >          Since the SCSI status ends the task and therefore
  >  releases the ITT
  >  it
  >  seems the only safe way to do this would be to make
  >  piggyback status
  >  mutually exclusive with the A bit in a DATA-IN with the
  >  F bit set.
  >  This would require the target to wait for the data SNACK before
  >  issuing a SCSI status PDU. Clearly this is unpalatable
  >  for latency
  >  reasons.
  >
  >          - Rod
  >
  >  From: [email-masked]
  >  [mailto:[email-masked]]On Behalf Of
  >  Eddy Quicksall
  >  Sent: Saturday, March 16, 2002 12:41 PM
  >  To: Mallikarjun C.; [email-masked]
  >  Cc: [email-masked]
  >  Yes, I agree.
  >  From: Mallikarjun C. [mailto:[email-masked]]
  >  Sent: Friday, March 15, 2002 11:22 PM
  >  To: [email-masked]
  >  I agree with Julian.
  >  Seems to me that targets should be allowed to ask for an ack
  >  on the last Data-In PDU that concludes the entire
  >  transfer for the
  >  task - a follow-up NOP-ping is needless.  I propose that we
  >
  >  replace:
  >    "it MAY set the A bit to 1 only once every
  >  MaxBurstSize bytes and
  >  MUST NOT
  >  do so more frequently than this."
  >
  >  with:
  >    "it MAY set the A bit to 1 only once every
  >  MaxBurstSize bytes or on
  >  the
  >  last
  >     Data-In PDU that concludes the entire requested
  >  transfer from the
  >  target's
  >     perspective, and MUST NOT do so more frequently than this."
  >  --
  >  Mallikarjun
  >
  >  Mallikarjun Chadalapaka
  >  Networked Storage Architecture
  >  Network Storage Solutions Organization
  >  Hewlett-Packard MS 5668
  >  Roseville CA 95747
  >  [email-masked]
  >
  >  ----- Original Message -----
  >  From: "Julian Satran" <[email-masked]>
  >  To: "Paul Koning" <[email-masked]>
  >  Cc: <[email-masked]>; <[email-masked]>;
  >  <[email-masked]>; <[email-masked]>
  >  Sent: Friday, March 15, 2002 11:50 AM
  >  > That could be so but it would be overkill. Status ACK
  >  can implicitly
  >  > acknowledge the last transfer.
  >  > And Yes the fact that the last transfer is not mentioned is an
  >  oversight
  >  > that I will correct.
  >  > This does not mean that you HAVE TO raise the A flag
  >  or that you are
  >  > ENCOURAGED to do so :-)
  >  >
  >  > Julo
  >  >
  >  >
  >  >
  >  >
  >  > Paul Koning <[email-masked]>
  >  > 15-03-02 16:09
  >  > Please respond to Paul Koning
  >  >
  >  >
  >  >         To:     [email-masked]
  >  >         cc:     [email-masked], [email-masked]
  >  > >>>>> "Eddy" == Eddy Quicksall
  >  <[email-masked]> writes:
  >  >
  >  >  Eddy> I think we may need better explanation about
  >  why some folks
  >  >  Eddy> don't want to do the "positive ack".
  >  >
  >  >  >> We got to this position, since so many folks did
  >  not want to
  >  >  >> support the positive ack.
  >  >
  >  > Something doesn't compute here.
  >  >
  >  > I don't believe the discussion has anything to do with
  >  whether you
  >  > support positive ACK or not.  If you're doing error
  >  recovery level 1
  >  > or above, then you are required to support it, because
  >  the other end
  >  > is allowed to say A=1 and you're required to answer that.
  >  >
  >  > If you don't want to support positive ACK, the solution is to
  >  support
  >  > only error recovery level 0.
  >  >
  >  > The issue under discussion is whether the rule "you
  >  are allowed to
  >  set
  >  > A=1 only once per MaxBurstSize" is correct.  At this point it's
  >  clear
  >  > to me that it is not, because you need to be able to
  >  set A=1 at the
  >  > end of the transfer.  The current rule forbids that
  >  unless the total
  >  > transfer size is >= MaxBurstSize.
  >  >
  >  > Kevin's proposal is a simple fix to this problem.
  >  >
  >  >                   paul
  >  >
  >  >
  >  >
  >  >

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 18 Mar 2002 23:32:56 +0200

That is neither the intent nor good enough.

Status ack may be delayed due to some other hole or no outgoing PDUs. In 
which case the initiator MUST issue the ACK.
The text says that if the initiator decides to issue an ACK it must issue 
it before the status ACK.

The target MUST take a status ACK as an implicit data ACK.

I could rephrase the text as follows:

 For sessions with ErrorRecoveryLevel 1 or higher, the target sets this bit 
to 1 to indicate that it requests a positive acknowledgement from the 
initiator for the data received.  The target should use the A bit 
moderately; it MAY set the A bit to 1 only once every MaxBurstSize bytes 
or on the last Data-In PDU that concludes the entire requested read data 
transfer for the task from the target's perspective, and MUST NOT do so 
more frequently than this. 

On receiving a Data-In PDU with the A bit set to 1, the initiator MUST 
issue a SNACK of type DataACK unless it has received the status and it is 
able to acknowledge the status on some other outgoing PDU (through 
ExpStatSN) in which case sending the SNACK of type DataACK is not 
man-adatory.  If the initiator has detected holes in the input sequence, 
it MUST postpone issuing the SNACK of type DataACK until the holes are 
filled. An initiator MUST NOT send a SNACK of type DataACK for the last 
Data-In PDU for a command after acknowledging the status for com-mand. A 
status acknowledgement for a command that generated Data-In PDUs is 
considered by the target as an implicit acknowledgement of the Data-In 
PDUs if such an acknowledgement was requested by the target.

Julo

"Mallikarjun C." <[email-masked]>
18-03-02 19:58
Please respond to "Mallikarjun C."

        To:     Julian Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS
        cc:     <[email-masked]>
        Subject:        Re: iSCSI: Use of the A bit

John,

I think you're describing the intent of the wording, though the
paragraph you point out doesn't seem consistent.

Here's what I suggest with a few tweaks -

       On receiving a Data-In PDU with the A bit set to 1, if there are no 
holes 
       in the read data until that Data-In PDU, the initiator MUST issue a 
SNACK 
       of type DataACK , or alternatively MAY acknowledge the status for 
the task 
       immediately via ExpStatSN on other outbound PDUs if the status for 
the task
       is also received.  If the initiator has detected holes in the read 
data until that Data-In 
       PDU, it MUST postpone issuing the SNACK of type DataACK until the 
holes 
       are filled. An initiator also MUST NOT acknowledge the status for 
the task 
       before those holes are filled.  A status acknowledgement for a task 
that generated 
       the Data-In PDUs is considered by the target as an implicit 
acknowledgement 
      of the Data-In PDUs if an acknowledgement was requested by the 
target.

I hope this works better....
--
Mallikarjun

Mallikarjun Chadalapaka

----- Original Message ----- 
From: "John Hufferd" <[email-masked]>
To: "Julian Satran" <[email-masked]>
Cc: "Mallikarjun C." <[email-masked]>; <[email-masked]>; 
<[email-masked]>
Sent: Monday, March 18, 2002 9:17 AM
Subject: Re: iSCSI: Use of the A bit

> 
> Julian the following statement is confusing:
> 
>  "An initiator MUST ONLY acknowledge the status for command that 
produced
>  Data-In PDUs after acknowledging the Data-In PDUs if Data-In PDU
>  acknowledgment is requested by the target.  A status acknowledgement 
for a
>  command that generated Data-In PDUs is considered by the target as an
>  implicit acknowledgement of the Data-In PDUs if such an acknowledgement
>  was requested by the target."
> 
> What did you intend to say.  All commands MUST have t status 
acknowledge,
> via ExpStatSN, so I do not understand the first sentence.  And if the
> Data-In PDUs must be acknowledged (if the A bit is set), what is the
> meaning of the last statement.
> 
> Since I do not believe there is a need to ack the last Data-In PDU, I am
> hoping you agree.  But it is not clear.
> 
> If the Target sets the A bit on the last Data-In PDU, which also has 
good
> status set -- and the Initiator has a PDU ready to send to which it can
> attach the value of ExpStatSN, which will show that the Data-In PDU was
> received -- there should be no reason to send a extra SNACK for the 
Data-In
> PDU.
> 
> Please tell me that is what you intended to say in the above quoted 
text.
> "Julian Satran" <[email-masked]>@ece.cmu.edu on 03/17/2002
> 05:52:46 PM
> 
> To:    "Mallikarjun C." <[email-masked]>
> cc:    [email-masked], [email-masked]

---

From: "Julian Satran" <[email-masked]>
Date: Tue, 19 Mar 2002 00:39:40 +0200

yes - I think that the latest text does just that. Julo

                      "Rod Harrison"                                                                     
                      <rod.harrison@win        To:       Julian Satran/Haifa/IBM@IBMIL, John Hufferd/San 
                      driver.com>               Jose/IBM@IBMUS                                           
                                               cc:       "Mallikarjun C." <[email-masked]>,             
                      18-03-02 23:29            <[email-masked]>, <[email-masked]>               
                      Please respond to        Subject:  RE: iSCSI: Use of the A bit                     
                      "Rod Harrison"                                                                     

             Can we explicitly allow the status and data ACKs to be at the
same
time? Otherwise we prevent sending the status ACK (expStatSn) in the
data ACK SNACK PDU.

             - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Monday, March 18, 2002 8:23 PM
To: John Hufferd
Cc: Mallikarjun C.; [email-masked]; [email-masked]
Subject: Re: iSCSI: Use of the A bit

John,

comments in text - JUlo

John Hufferd/San Jose/IBM@IBMUS
18-03-02 19:17
Please respond to John Hufferd

        To:        Julian Satran/Haifa/IBM@IBMIL
        cc:        "Mallikarjun C." <[email-masked]>,
[email-masked], [email-masked]
        Subject:        Re: iSCSI: Use of the A bit

implicit acknowledgement of the Data-In PDUs if such an

+++ The first statement says that IF data-ack is requested it MUST be
sent by the initiator before it acknowledges status.
+++

Since I do not believe there is a need to ack the last Data-In PDU, I
am
hoping you agree.  But it is not clear.

+++
In fact the target may want a data ack as the status ack might be
delayed for various reasons.
The second statement says that if the target sees a status ack that is
also an ack for the data

The first statement says that the initiator must send ack if asked -
and send status ack only afterwards.

The second says that when status is received the data are acked and
target should wait no longer (i.e., no recovery action).
+++
If the Target sets the A bit on the last Data-In PDU, which also has
good
status set -- and the Initiator has a PDU ready to send to which it
can
attach the value of ExpStatSN, which will show that the Data-In PDU
was
received -- there should be no reason to send a extra SNACK for the
Data-In
PDU.

.
.
.
John L. Hufferd

Dear all,

  A (Acknowledge) bit

      For sessions with ErrorRecoveryLevel 1 or higher, the target
sets
      this bit to 1 to indicate that it requests a positive
      acknowledgement from the initiator for the data received.  The
      target should use the A bit moderately; it MAY set the A bit to
1
      only once every MaxBurstSize bytes or on the last Data-In PDU
that
      concludes the entire requested read data transfer for the task
from
      the target's perspective, and MUST NOT do so more frequently
than
      this.

      On receiving a Data-In PDU with the A bit set to 1, the
initiator
      MUST issue a SNACK of type DataACK.  If the initiator has
detected
      holes in the input sequence, it MUST postpone issuing the SNACK
of
      type DataACK until the holes are filled. An initiator MUST ONLY
      acknowledge the status for command that produced Data-In PDUs
after
      acknowledging the Data-In PDUs if Data-In PDU acknowledgment is
      requested by the target. A status acknowledgement for a command
that
      generated Data-In PDUs is considered by the target as an
implicit
      acknowledgement of the Data-In PDUs if such an acknowledgement
was
      requested by the target.

      For Status SNACK and DataACK, the Initiator Task Tag is
reserved. In
       all other cases, the Initiator Task Tag field MUST be set to
the
      Ini-tiator Task Tag of the referenced command.

      Julo

---

From: "Julian Satran" <[email-masked]>
Date: Tue, 19 Mar 2002 00:52:38 +0200

Mallikarjun has pointed out to me that the latest text still contains some 
conflicting statements so here is another attempt:

On receiving a Data-In PDU with the A bit set to 1, if there are no holes 
in the read data until that Data-In PDU, the initiator MUST issue a SNACK 
of type DataACK except when it is able to acknowledge the status for the 
task immediately via ExpStatSN on other outbound PDUs if the status for 
the task is also received; in this later case (acknowledgement through 
ExpStatSN) sending a SNACK of type DataACK in response to the A bit is not 
mandatory but if it is done it must not be sent after the status 
acknowledgement through ExpStatSN.  If the initiator has detected holes in 
the read data until that Data-In PDU, it MUST postpone issuing the SNACK 
of type DataACK until the holes are filled. An initiator also MUST NOT 
acknowledge the status for the task before those holes are filled.  A 
status acknowledgement for a task that generated the Data-In PDUs is 
considered by the target as an implicit acknowledgement of the Data-In 
PDUs if such an acknowledge-ment was requested by the target.

Julo

John,

Mallikarjun Chadalapaka

----- Original Message ----- 
From: "John Hufferd" <[email-masked]>
To: "Julian Satran" <[email-masked]>
Cc: "Mallikarjun C." <[email-masked]>; <[email-masked]>; 
<[email-masked]>

produced
for a
acknowledge,
good
Data-In
text.
that
from
detected
of
after
that
was