Subject: [Ips] iSCSI question about positive data acknowledgement

From: [email-masked]
Date: Tue, 15 Feb 2005 16:17:46 -0500

I've run into a small problem while considering what it would take to
implement error recovery level 1 in an iSCSI target. The iSCSI specification
states in 10.7.2: 

"The target should use the A bit moderately; it MAY only set the A bit to 1
once every MaxBurstLength bytes, or on the last Data-In PDU that concludes
the entire requested read data transfer for the task from the target's
perspective, and it MUST NOT do so more frequently." 

Question: Is it okay for a target to set the A bit multiple times during the
transmission of MaxBurstLength bytes as long as it does not have more than
one PDU outstanding with the A bit set? For example, lets say that
MaxBurstLength is 256k and a target sends 32k of read data to an initiator
with the A bit set. After receiving the SNACK-DataACK the target sends more
data with the A bit set again even though MaxBurstLength bytes haven't been
sent. I don't see how this behavior would be problematic for an initiator
implementation but it seems to be disallowed by the statement in 10.7.2. 

The reason it would be desirable for a target to request multiple data ACKs
is that often data on large reads gets pipelined. A small amount of read
data is first burst to the initiator followed by larger ones. Lets say a
target iSCSI driver first receives 16k, then 32k, then 128k, then multiple
256k bursts from its SCSI layer when responding to a large read. If the
target driver can't set the A bit on each burst of data (as long as the
burst doesn't exceed MaxBurstLength) then the target may run into annoying
complexities and deadlock situations. 

You might say that the target iSCSI driver could wait for multiple bursts
and aggregate them into one burst on the wire of MaxBurstLength. This seems
possible but may lead to a target deadlock. If for example the target driver
receives a burst of data from its SCSI layer and wont get another burst
until buffers are freed up that were used for the first burst. 

Alternatively it might be suggested to limit MaxBurstLength to the smallest
burst that the iSCSI target driver will ever get from its SCSI layer. This
would solve the problem but might necessitate setting MaxBurstLength to a
prohibitively small value. Thus this solution doesn't seem optimal. 

A final obvious response would be to modify the target SCSI layer so that it
gives larger bursts to the iSCSI layer or to actually make the SCSI layer
aware of the iSCSI burst length requirements. However this may not be
practical for many target implementations whose SCSI layers have been around
long before iSCSI existed. These SCSI layers might require extensive
non-trivial rework. 

Any input would be appreciated thanks, 

Bill 

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Julian Satran <[email-masked]>
Date: Tue, 15 Feb 2005 18:36:39 -0500

Bill,

Some answers in text.

Your are not supposed to set the A bit more than once for every 
MaxBurstLength
However you don't have to stop sending data to initiator if you have an A 
bit "outstanding".
The A bit mechanism was introduced to limit the amount of resources a 
target will commit to hold
buffers that the initiator may require him to retransmit.
 
> complexities and deadlock situations. 

If you want to  pipeline A bits you have to state a lower MaxBurstLength. 
I have trouble seeing how you can be deadlocked.

> until buffers are freed up that were used for the first burst. 

The target is not deadlocked - it just works slower because you decided to
expose MaxBurstLength as the total of your buffer space.

> prohibitively small value. Thus this solution doesn't seem optimal. 

MaxBurstLength should take into account you buffer space and the RTT.

---

From: "Eddy Quicksall" <[email-masked]>
Date: Wed, 23 Feb 2005 09:46:04 -0500

Regarding the limitation put on the use of the A bit, there is a case for a deadlock with regards to a session. Consider the following:

1) the application uses double buffering
2) the application buffers are 64K each
3) the application needs to send READ data in buffered bursts
4) the application has >128K to send
5) the iSCSI layer uses 256K MaxBurstLength

As each buffer is filled by the application, the application layer uses the transport to send that "buffer burst". As each burst is finished, the application layer then re-uses the completed buffer to request more data from its environment.

To support this, the iSCSI layer uses the A bit so it can inform the application layer when each send is complete.

In the example above, the iSCSI layer will not be able to set the A bit and hence a dead lock for the session.

Eddy
  ----- Original Message ----- 
  From: Julian Satran 
  To: [email-masked] 
  Cc: [email-masked] ; [email-masked] 
  Sent: Tuesday, February 15, 2005 6:36 PM
  Subject: Re: [Ips] iSCSI question about positive data acknowledgement

  Bill, 

  Some answers in text. 

  > implementation but it seems to be disallowed by the statement in 10.7.2. 

  Your are not supposed to set the A bit more than once for every MaxBurstLength 
  However you don't have to stop sending data to initiator if you have an A bit "outstanding". 
  The A bit mechanism was introduced to limit the amount of resources a target will commit to hold 
  buffers that the initiator may require him to retransmit. 
   
  > complexities and deadlock situations. 

  If you want to  pipeline A bits you have to state a lower MaxBurstLength. I have trouble seeing how you can be deadlocked. 

  > Any input would be appreciated thanks, 

------------------------------------------------------------------------------

---

From: "Eddy Quicksall" <[email-masked]>
Date: Wed, 23 Feb 2005 13:30:03 -0500

By "application", my example is a piece of code at a higher level. Perhaps a tape application that is streaming in large blocks ... or the application that Bill Sears refers to. In my example, the application is unaware of the transport. So it would have no way to communicate MaxBurstLength to the transport.

I really don't want to go into a debate on how applications vs. transports should be written and if an application should be independent of the transport or not. I was simply pointing out a case for the deadlock.

A point that can be derived from this thread is that the restriction put on the use of the A bit is not really necessary and leads to unnecessary code.

Eddy
  ----- Original Message ----- 
  From: Julian Satran 
  To: Eddy Quicksall 
  Cc: [email-masked] ; [email-masked] ; [email-masked] 
  Sent: Wednesday, February 23, 2005 11:56 AM
  Subject: Re: [Ips] iSCSI question about positive data acknowledgement

  Eddy, 

  I am bit unsure of what you say. I will assume that the "application" is the target function. 
  If that is the case why did the target accept a MaxBurstLength of 256K? ( it should have set it to at most 128k and optimally to 64k to allow for some pipelining). 

  Julo 

        "Eddy Quicksall" <[email-masked]> 
        23/02/2005 16:46 
       To <[email-masked]>, Julian Satran/Haifa/IBM@IBMIL  
              cc <[email-masked]>, <[email-masked]>  
              Subject Re: [Ips] iSCSI question about positive data acknowledgement 

    

    
    
  Eddy 
  ----- Original Message ----- 
  From: Julian Satran 
  To: [email-masked] 
  Cc: [email-masked] ; [email-masked] 
  Sent: Tuesday, February 15, 2005 6:36 PM 
  Subject: Re: [Ips] iSCSI question about positive data acknowledgement 

  Bill, 

  Some answers in text.

---

From: Julian Satran <[email-masked]>
Date: Wed, 23 Feb 2005 18:56:24 +0200

Eddy,

I am bit unsure of what you say. I will assume that the "application" is 
the target function.
If that is the case why did the target accept a MaxBurstLength of 256K? ( 
it should have set it to at most 128k and optimally to 64k to allow for 
some pipelining).

Julo

To
<[email-masked]>, Julian Satran/Haifa/IBM@IBMIL
cc
<[email-masked]>, <[email-masked]>
Subject
Re: [Ips] iSCSI question about positive data acknowledgement

Regarding the limitation put on the use of the A bit, there is a case for 
a deadlock with regards to a session. Consider the following:

As each buffer is filled by the application, the application layer uses 
the transport to send that "buffer burst". As each burst is finished, the 
application layer then re-uses the completed buffer to request more data 
from its environment.
 
To support this, the iSCSI layer uses the A bit so it can inform the 
application layer when each send is complete.
 
In the example above, the iSCSI layer will not be able to set the A bit 
and hence a dead lock for the session.
 
Eddy
----- Original Message ----- 
From: Julian Satran 
To: [email-masked] 
Cc: [email-masked] ; [email-masked] 
Sent: Tuesday, February 15, 2005 6:36 PM
Subject: Re: [Ips] iSCSI question about positive data acknowledgement

Bill, 

Some answers in text.

---

From: Julian Satran <[email-masked]>
Date: Thu, 24 Feb 2005 08:25:07 +0200

Eddy,

[email-masked] wrote on 23/02/2005 20:30:03:

> By "application", my example is a piece of code at a higher level. 
> Perhaps a tape application that is streaming in large blocks ... or 
> the application that Bill Sears refers to. In my example, the 
> application is unaware of the transport. So it would have no way to 
> communicate MaxBurstLength to the transport.
> 

That is your design choice. MaxBurstLength is there to be used as 
"external" indication of capabilities
without requiring an interaction.
 
> I really don't want to go into a debate on how applications vs. 
> transports should be written and if an application should be 
> independent of the transport or not. I was simply pointing out a 
> case for the deadlock.
> 

Again the deadlock is a result of a design choice protocol.

> A point that can be derived from this thread is that the restriction
> put on the use of the A bit is not really necessary and leads to 
> unnecessary code.
> 

The restriction was put in place to avoid having ACK requests being 
requested beyond an initially negotiate rate
MaxBurstLength has not many other uses :-)
 
> If that is the case why did the target accept a MaxBurstLength of 
> 256K? ( it should have set it to at most 128k and optimally to 64k 
> to allow for some pipelining). 
> 
> Julo 
> 
> 
> 

> Regarding the limitation put on the use of the A bit, there is a 
> case for a deadlock with regards to a session. Consider the following: 
> 
> As each buffer is filled by the application, the application layer 
> uses the transport to send that "buffer burst". As each burst is 
> finished, the application layer then re-uses the completed buffer to
> request more data from its environment. 
> 
> In the example above, the iSCSI layer will not be able to set the A 
> bit and hence a dead lock for the session. 
> 
> Eddy 
MaxBurstLength
> If you want to  pipeline A bits you have to state a lower 
> MaxBurstLength. I have trouble seeing how you can be deadlocked. 
>

---

From: Paul Koning <[email-masked]>
Date: Thu, 24 Feb 2005 10:08:41 -0500

>>>>> "Eddy" == Eddy Quicksall <[email-masked]> writes:

 Eddy> By "application", my example is a piece of code at a higher
 Eddy> level. Perhaps a tape application that is streaming in large
 Eddy> blocks ... or the application that Bill Sears refers to. In my
 Eddy> example, the application is unaware of the transport. So it
 Eddy> would have no way to communicate MaxBurstLength to the
 Eddy> transport.  I really don't want to go into a debate on how
 Eddy> applications vs. transports should be written and if an
 Eddy> application should be independent of the transport or not. I
 Eddy> was simply pointing out a case for the deadlock.

 Eddy> A point that can be derived from this thread is that the
 Eddy> restriction put on the use of the A bit is not really necessary
 Eddy> and leads to unnecessary code.

The restriction on the A bit serves to create a rate limit on that
particular bit of processing.  We could have chosen a different rate
limit, or we could have left it out and trusted the initiator to
behave reasonably, or trusted that A bit processing is sufficiently
cheap that it's not an issue.  That's a design question that could
have gone any number of different ways when it was made.  

But at this point it's made.  And yes, it means that you can get into
the problem you described.  

That's fine.  I look at it this way: if your ONLY buffering is at the
application layer, then the application layer and the transport must
agree on the flow control parameters, otherwise deadlock will happen
when the parameters are inconsistent with the buffering algorithm.
That's the underlying issue in your scenario.

A design like that can't be reliable.  You have to change it in one of
two ways:

1. Have the transport and the application communicate the flow control
   assumptions, so either the application can change its buffering, or
   the transport can negotiate a different burst size.

2. Put buffering in the transport, so each layer has both its own 
   buffering and its own flow control.

       paul