Subject: RE: [Ips] resending Data-in with S bit set

From: "Dean Scoville" <[email-masked]>
Date: Tue, 5 Oct 2004 15:49:12 -0700

I'd say the RFC is ambiguous, because it sounds like it is now being interpreted to say that 
numbered responses include only SCSI Response PDUs, even though I thought the
statements in sections 3.2 and 6.1.4.2 indicated that numbered responses include any PDUs
carrying status... including Data-In PDUs with the S-bit set. Does that mean the initiator
shouldn't be able to SNACK for a Reject PDU or a NOP-PDU, since they aren't SCSI Response
PDUs either?
 
Dean

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:44 PM
To: Dean Scoville; Julian Satran
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Yes, when I said "Status PDUs", I used shorthand and meant SCSI Response PDUs".
 
What you point out is exactly the reason I wanted to know the intent and if I understood the spec correctly. I did not mean to say I agree or disagree with the intent.
 
Since the spec is finished, I think it is too late to argue for a change. It looks like the initiator just has to be a little smart when it needs all PDUs that carry status.
 
Eddy

----- Original Message ----- 
From: Dean Scoville <mailto:[email-masked]>  
To: Eddy Quicksall <mailto:[email-masked]>  ; Julian Satran <mailto:[email-masked]>  
Cc: [email-masked] 
Sent: Tuesday, October 05, 2004 6:22 PM
Subject: RE: [Ips] resending Data-in with S bit set

When you say only "Status" PDUs, do you mean only "SCSI Response" PDUs,
and that the target should remain silent about any phase collapsed Data-In PDUs? 
That doesn't make sense to me. If the initiator has 100 commands outstanding and
issues a Status SNACK because a received StatSN indicates that some PDUs are
missing, it shouldn't have to issue Data SNACKs for all of the outstanding commands
to see if one of them perhaps was using phase collapse and happened to have the
missing StatusSN. Is that really what is intended?
 
The last sentence in Section 3.2 defines a numbered response as:
 
    "Also, unless otherwise specified, status, response, or numbered response have the same meaning."
 
And Section 6.1.4.2 states:
 
    "Lost iSCSI numbered Response. It is recognized by either identifying a data digest
     error on a Response PDU or a Data-In PDU carrying the status, or by receiving a
     Response PDU with a higher StatSN than expected."
 
The above two RFC statements would lead me to believe that numbered responses include 
Data-In PDUs, and that the numbered responses referred to in Section 10.16.1 in 
reference to Status SNACKs should also include Data-In retransmissions.
 
Dean

-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:09 PM
To: Dean Scoville; Julian Satran
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Maybe my question was misunderstood. To say it another way:
 
Is it true that a Status SNACK only solicits Status PDUs and that it does not solicit Data-In PDUs which have the S bit set?
 
I interpret Julian's answer to this as "yes, that is true". 
 
Eddy

----- Original Message ----- 
From: Dean Scoville <mailto:[email-masked]>  
To: Julian Satran <mailto:[email-masked]>  ; Eddy  <mailto:[email-masked]> Quicksall 
Cc: [email-masked] 
Sent: Tuesday, October 05, 2004 5:54 PM
Subject: RE: [Ips] resending Data-in with S bit set

Julian,
 
Section 10.16 states the following about PDUs transmitted as a result of a SNACK:
 
   "The numbered-response(s) or R2T(s), requested by a SNACK, MUST be
    delivered as exact replicas of the ones that the target transmitted originally
    except for the fields ExpCmdSN, MaxCmdSN, and ExpDataSN, which MUST
    carry the current values. R2T(s) requested by SNACK MUST  also carry the
    current values and except for resegmentation (see Section 10.16.3 Resegmentation)."
 
Section 10.16.1 describes Status and Data/R2T SNACK types as:
 
    0-Data/R2T SNACK - requesting retransmission of one or more Data-In or R2T PDUs.
    1-Status SNACK - requesting retransmission of one or more numbered responses.
 
It seems to me that if the target receives a Status SNACK for status sent on a Data-In 
PDU due to phase collapse, the target should send an exact replica of the Data-In PDU.

Similarly, if the target receives a Data SNACK, it should retransmit the Data-In PDU with
the S bit set, since it is a retransmission.
 
I may have misunderstood your recent posting, but I thought it said the target should 
generate a totally new SCSI Response PDU in response to the Status SNACK rather
than retransmitting the original Data-In PDU. And that it should send a Data-In PDU
without the S-bit set in the case of a Data SNACK. Both of these behaviors seem to 
violate what is stated in Sections 10.16 and 10.16.1 of the iSCSI RFC, as quoted above.
 
I would appreciate clarification on the target's expected response in these situations.
 
thanks,
 
Dean Scoville

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of Julian Satran
Sent: Monday, October 04, 2004 7:43 AM
To: Eddy Quicksall
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Eddy, 

This is correct. 
The data-in PDUs don't have to be sent. 
The initiator has to request the data-in PDUs explicitly and MUST do so before it acks the status (that implicitly acks all the data and deletes any record of the command at target). 

Julo 

[email-masked] wrote on 04/10/2004 15:20:56:

> Can you confirm that the intent of 10.16.6. is as follows? 
>   
> If the Initiator has requested for a Status SNACK with BegRun=0 and 
> RunLength=0 (i.e, all unacknowledged status pdus) and the target has
> some Data-In PDUs carrying *status* that have not been acknowledged,
> the target will *not* resend those Data-In PDUs carrying the status. 
>   
> The unacknowledged queue looks like 
>   
>    response(statSN=5) ---> response(statSN=6) -----> Data-In w/S 
> bit(statSN=7) ----> response(statSN=8) 
>   
> The target will resend pdus with statSN 5, 6 and 8. The Data-In with
> statSN 7 will not be resent even though it sends status. 
>   
> Eddy 
>  _______________________________________________
> Ips mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ips

  _____  

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Julian Satran <[email-masked]>
Date: Wed, 6 Oct 2004 10:31:45 +0200

Dean & all,

Sorry - the spec is not ambiguous and I was wrong in my previous answer (I 
just overlooked the StatSN field in Data-In).
The target should (following 10.16) re-issue the exact copy of the Data-In 
PDU that included the status.

Julo

"Dean Scoville" <[email-masked]> 
06/10/04 00:49

To
"Eddy Quicksall" <[email-masked]>, Julian 
Satran/Haifa/IBM@IBMIL
cc
<[email-masked]>
Subject
RE: [Ips] resending Data-in with S bit set

I'd say the RFC is ambiguous, because it sounds like it is now being 
interpreted to say that 
numbered responses include only SCSI Response PDUs, even though I thought 
the
statements in sections 3.2 and 6.1.4.2 indicated that numbered responses 
include any PDUs
carrying status... including Data-In PDUs with the S-bit set. Does that 
mean the initiator
shouldn't be able to SNACK for a Reject PDU or a NOP-PDU, since they 
aren't SCSI Response
PDUs either?
 
Dean
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:44 PM
To: Dean Scoville; Julian Satran
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Yes, when I said "Status PDUs", I used shorthand and meant SCSI Response 
PDUs".
 
What you point out is exactly the reason I wanted to know the intent and 
if I understood the spec correctly. I did not mean to say I agree or 
disagree with the intent.
 
Since the spec is finished, I think it is too late to argue for a change. 
It looks like the initiator just has to be a little smart when it needs 
all PDUs that carry status.
 
Eddy
----- Original Message ----- 
From: Dean Scoville 
To: Eddy Quicksall ; Julian Satran 
Cc: [email-masked] 
Sent: Tuesday, October 05, 2004 6:22 PM
Subject: RE: [Ips] resending Data-in with S bit set

When you say only "Status" PDUs, do you mean only "SCSI Response" PDUs,
and that the target should remain silent about any phase collapsed Data-In 
PDUs? 
That doesn't make sense to me. If the initiator has 100 commands 
outstanding and
issues a Status SNACK because a received StatSN indicates that some PDUs 
are
missing, it shouldn't have to issue Data SNACKs for all of the outstanding 
commands
to see if one of them perhaps was using phase collapse and happened to 
have the
missing StatusSN. Is that really what is intended?

    "Also, unless otherwise specified, status, response, or numbered 
response have the same meaning."

    "Lost iSCSI numbered Response. It is recognized by either identifying 
a data digest
     error on a Response PDU or a Data-In PDU carrying the status, or by 
receiving a
     Response PDU with a higher StatSN than expected."
 
The above two RFC statements would lead me to believe that numbered 
responses include 
Data-In PDUs, and that the numbered responses referred to in Section 
10.16.1 in 
reference to Status SNACKs should also include Data-In retransmissions.
 
Dean
-----Original Message-----
From: Eddy Quicksall [mailto:[email-masked]]
Sent: Tuesday, October 05, 2004 3:09 PM
To: Dean Scoville; Julian Satran
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Is it true that a Status SNACK only solicits Status PDUs and that it does 
not solicit Data-In PDUs which have the S bit set?

Eddy
----- Original Message ----- 
From: Dean Scoville 
To: Julian Satran ; Eddy Quicksall 
Cc: [email-masked] 
Sent: Tuesday, October 05, 2004 5:54 PM
Subject: RE: [Ips] resending Data-in with S bit set

Julian,
 
Section 10.16 states the following about PDUs transmitted as a result of a 
SNACK:
 
   "The numbered-response(s) or R2T(s), requested by a SNACK, MUST be
    delivered as exact replicas of the ones that the target transmitted 
originally
    except for the fields ExpCmdSN, MaxCmdSN, and ExpDataSN, which MUST
    carry the current values. R2T(s) requested by SNACK MUST  also carry 
the
    current values and except for resegmentation (see Section 10.16.3 
Resegmentation)."

    0-Data/R2T SNACK - requesting retransmission of one or more Data-In or 
R2T PDUs.
    1-Status SNACK - requesting retransmission of one or more numbered 
responses.
 
It seems to me that if the target receives a Status SNACK for status sent 
on a Data-In 
PDU due to phase collapse, the target should send an exact replica of the 
Data-In PDU.
Similarly, if the target receives a Data SNACK, it should retransmit the 
Data-In PDU with
the S bit set, since it is a retransmission.
 
I may have misunderstood your recent posting, but I thought it said the 
target should 
generate a totally new SCSI Response PDU in response to the Status SNACK 
rather
than retransmitting the original Data-In PDU. And that it should send a 
Data-In PDU
without the S-bit set in the case of a Data SNACK. Both of these behaviors 
seem to 
violate what is stated in Sections 10.16 and 10.16.1 of the iSCSI RFC, as 
quoted above.
 
I would appreciate clarification on the target's expected response in 
these situations.
 
thanks,
 
Dean Scoville
-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of 
Julian Satran
Sent: Monday, October 04, 2004 7:43 AM
To: Eddy Quicksall
Cc: [email-masked]
Subject: Re: [Ips] resending Data-in with S bit set

Eddy, 

This is correct. 
The data-in PDUs don't have to be sent. 
The initiator has to request the data-in PDUs explicitly and MUST do so 
before it acks the status (that implicitly acks all the data and deletes 
any record of the command at target). 

Julo

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 06 Oct 2004 18:26:56 -0700

I agree with what Julian says below.  The "exact replica" wording of 
10.16 is quite unambiguous.
-- 
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Julian Satran wrote:

> 
> Dean & all,
> 
> 
> Sorry - the spec is not ambiguous and I was wrong in my previous answer 
> (I just overlooked the StatSN field in Data-In).
> The target should (following 10.16) re-issue the exact copy of the 
> Data-In PDU that included the status.
> 
> Julo
> 
> 
> *"Dean Scoville" <[email-masked]>*
> 
> 06/10/04 00:49
> 
> 	
> To
> -----Original Message-----*
> From:* Eddy Quicksall [mailto:[email-masked]]*
> Sent:* Tuesday, October 05, 2004 3:44 PM*
> To:* Dean Scoville; Julian Satran*
> Cc:* [email-masked]*
> Subject:* Re: [Ips] resending Data-in with S bit set
> 
> Since the spec is finished, I think it is too late to argue for a 
> change. It looks like the initiator just has to be a little smart when 
> it needs all PDUs that carry status.
>  
> Eddy
> *From:* _Dean Scoville_ <mailto:[email-masked]>
> *To:* _Eddy Quicksall_ <mailto:[email-masked]> 
> ; _Julian Satran_ <mailto:[email-masked]>
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Sent:* Tuesday, October 05, 2004 6:22 PM
> *Subject:* RE: [Ips] resending Data-in with S bit set
> 
> -----Original Message-----*
> From:* Eddy Quicksall [mailto:[email-masked]]*
> Sent:* Tuesday, October 05, 2004 3:09 PM*
> To:* Dean Scoville; Julian Satran*
> Cc:* [email-masked]*
> Subject:* Re: [Ips] resending Data-in with S bit set
> 
> Is it true that a Status SNACK _only_ solicits Status PDUs and that it 
> does _not_ solicit Data-In PDUs which have the S bit set?
>  
> *From:* _Dean Scoville_ <mailto:[email-masked]>
> *To:* _Julian Satran_ <mailto:[email-masked]> ; _Eddy 
> Quicksall_ <mailto:[email-masked]>
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Sent:* Tuesday, October 05, 2004 5:54 PM
> *Subject:* RE: [Ips] resending Data-in with S bit set
> 
> Julian,
>  
> -----Original Message-----*
> From:* [email-masked] [mailto:[email-masked]]*On Behalf Of 
> *Julian Satran*
> Sent:* Monday, October 04, 2004 7:43 AM*
> To:* Eddy Quicksall*
> Cc:* [email-masked]*
> Subject:* Re: [Ips] resending Data-in with S bit set
> 
> 
> Eddy,
> 
> This is correct.
> ------------------------------------------------------------------------
> 
> ------------------------------------------------------------------------
>

---

From: William Studenmund <[email-masked]>
Date: Thu, 7 Oct 2004 11:09:30 -0400

> 10.16 is quite unambiguous.

Just to play devil's advocate for a bit, what happens if the 
MaxRecvDataSegmentLength the initiator now advertises is so small that 
the original PDU may not be sent? Do we resend it anyway? Resend part?

Note: the scenario is that the task completed, the initiator changed 
its advertised MaxRecvDataSegmentLength, then it sent a SNACK to 
retransmit status.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 07 Oct 2004 10:09:49 -0700

Bill,

AFAICT, you are describing the same scenario currently covered by the 
R-Data SNACK (10.16.3).  I'd like to know more details if that's not the 
case.

M

William Studenmund wrote:

> Note: the scenario is that the task completed, the initiator changed its 
> advertised MaxRecvDataSegmentLength, then it sent a SNACK to retransmit 
> status.
> 
> Take care,
> 
> Bill

---

From: William Studenmund <[email-masked]>
Date: Thu, 7 Oct 2004 22:14:55 -0400

On Oct 7, 2004, at 1:09 PM, Mallikarjun C. wrote:

> R-Data SNACK (10.16.3).  I'd like to know more details if that's not 
> the case.

I agree 10.16.3 covers it, now that I re-read it; I'd missed the status 
considerations in R-Data SNACK. However the case I had in mind is a 
slightly-different side of the scenario, where we have resegmentation 
but the initiator sends a Status SNACK.

For an R-Data SNACK, I get that the status now has to be transmitted in 
a SCSI Response PDU; no phase collapsed read here. And if multiple 
R-Data SNACKs are sent for a task, the SCSI Response has to be sent 
with the latest SNACK Tag.

What I don't see discussed is what you do if you (the target) get a 
Status SNACK after resegmentation (no R-Data SNACK). Do you send a SCSI 
Response for the task? I think that'd be the best thing. Then the 
initiator can discover that it's missing data and send an R-Data SNACK. 
Obviously since it lost the status, it didn't get the data, so it 
shouldn't acknowledge the status. :-)

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 08 Oct 2004 10:50:15 -0700

>what you do if you (the target) get a
 > Response for the task?

10.16.3 says the following in part:

"....if it has already sent a response containing another value
in the SNACK Tag field or had the status included in the last Data-In
PDU, it must send a new SCSI Response PDU.
....
If an
initiator attempts to recover a lost SCSI Response (with a
Status SNACK, see Section 10.16.1 Type) when more than one response
has been sent, the target will send the SCSI Response with the latest
content known to the target, including the last SNACK Tag for the
command."

Does that text address the scenario you've in mind?

Mallikarjun

William Studenmund wrote:
> R-Data SNACKs are sent for a task, the SCSI Response has to be sent with 
> the latest SNACK Tag.
>

---

From: William Studenmund <[email-masked]>
Date: Fri, 8 Oct 2004 14:17:03 -0400

> Does that text address the scenario you've in mind?

Almost. I'm thinking about the case where we have sent only one 
response (thus not "more than one"). We then get a Status SNACK, and we 
can't retransmit the status as it was phase-collapsed in with a Data-In 
that is too big to send now.

Since we haven't sent more than one status, I don't think the text 
speaks directly to this case. I think the spirit of the text, send a 
SCSI Command Response PDU, is the right thing to do. I'm just not sure 
if the text tells me that's what I should do.

One feeling I get is that the text in this section was written with the 
assumption that the first SNACK that will be sent for a task after 
resegmentation will be an R-Data SNACK. For that sequence, I think the 
text works well. This case is where the first SNACK for a task is a 
Status SNACK, as the initiator didn't know what task corresponded to a 
given StatSN value. i.e. the initiator noticed a hole in StatSN 
clearing, and went to inquire about the hole. This case isn't covered 
so well.

Thank you for continuing to help me with this question.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 08 Oct 2004 12:08:32 -0700

First off, I agree with the required target behavior you describe. 
That's the intended behavior.

Secondly, you say:
 >i.e. the initiator noticed a hole in StatSN
 > clearing, and went to inquire about the hole. This case isn't covered so
 > well.

I can agree with this too, in that the text is not terribly explicit 
about your scenario.

Finally, regarding the scenario you describe:

"We then get a Status SNACK, and we can't retransmit the status as it 
was phase-collapsed in with a Data-In that is too big to send now."

I continue to think that the first part of the quoted text addresses 
your scenario (this is only for the record, since you figured out the 
right way to handle the scenario anyway):

"....or had the status included in the last Data-In
PDU, it must send a new SCSI Response PDU."

The "must send a new SCSI Response PDU" requirement is not really for 
the reason you mention ("too big to send now"), but because of the 
following reasons:

10.4.4 says:
"After issuing a R-Data SNACK the initiator must discard any SCSI
status unless contained in an SCSI Response PDU carrying the same
SNACK Tag as the last issued R-Data SNACK for the SCSI command on the
current connection."

& a phase-collapsed SCSI Data-in PDU doesn't have room for a SNACK Tag.

Hope that helps.
-- 
Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:

> Almost. I'm thinking about the case where we have sent only one response 
> (thus not "more than one"). We then get a Status SNACK, and we can't 
> retransmit the status as it was phase-collapsed in with a Data-In that 
> is too big to send now.
> 
> clearing, and went to inquire about the hole. This case isn't covered so 
> well.
>

---

From: William Studenmund <[email-masked]>
Date: Fri, 8 Oct 2004 17:15:27 -0400

> & a phase-collapsed SCSI Data-in PDU doesn't have room for a SNACK Tag.

Oh, that makes a LOT of sense.

However I'm still a bit puzzled as this scenario is "before" an R-Data 
SNACK has been sent. Thus the text doesn't seem to directly apply. I 
mean I think the text is fine once the R-Data has gotten there, just 
not before.

I think it'd be fine to, after resegmentation but before an R-Data 
SNACK has been received, just resend a Data-In PDU with phase collapsed 
status iff it is still ok to send. Consider the case where the read was 
for say 2k of data and the resegmenting was from 8k to 4k. Just send 
the PDU again as it was.

If the Data-In PDU is too big to send (i.e. the initiator really should 
send an R-Data SNACK but hasn't realized it yet), I think the spec 
isn't clear, but that we have agreed that the target should send a SCSI 
Response. Since there has been no R-Data SNACK, there is no SNACK tag, 
and so 10.4.4 indicates that the SNACK tag field should be zero.

What I expect to then happen is that the initiator will now have an ITT 
for the StatSN, notice it has not received any or all of the data, then 
send an R-Data SNACK for that data. Then all of the text in 10.16.3 
kicks in.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 11 Oct 2004 15:07:15 -0700

I think I may be finally understanding your scenario, but let's confirm 
that....

A read SCSI task is in progress and the initiator changes 
MaxRecvDataSegementLength.  The task continues.  Subsequently, the 
initiator notices it lost a status.  Here's what I think should happen 
(I think the same as what you described, except for the ExpDataSN part).

- As always, initiator starts with a status SNACK.
- Regardless of the original transmission (phase collapse or not, old 
PDU still fits with the new size or not, single/multiple data PDUs), 
target retransmits the status in an explicit Response PDU with ExpDataSN 
set to something like 0xffffffff.
- Initiator checks ExpDataSN with its records.  They shouldn't match, 
leading to an R-Data SNACK.
- Rest of the semantics described in 10.16.3 apply.

My reasons are two:

- Target always does the same thing in this scenario (PDU size change 
followed by a status SNACK first) - i.e. doesn't phase collapse even if 
it still can.

- Initiator operates based on ExpDataSN as always, consistent with the 
spec intent to not have to do scoreboarding/counting.

OTOH, the RFC doesn't set aside 0xffffffff as a reserved value for 
ExpDataSN.  So I guess this is a hack for now.  Alternatively (or if 
target had already sent 2**32-1 PDUs for that task), what you described 
works as well (if the initiator is counting bytes).
-- 
Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:

> mean I think the text is fine once the R-Data has gotten there, just not 
> before.
> 
> I think it'd be fine to, after resegmentation but before an R-Data SNACK 
> has been received, just resend a Data-In PDU with phase collapsed status 
> iff it is still ok to send. Consider the case where the read was for say 
> 2k of data and the resegmenting was from 8k to 4k. Just send the PDU 
> again as it was.
> 
> send an R-Data SNACK but hasn't realized it yet), I think the spec isn't 
> clear, but that we have agreed that the target should send a SCSI

---

From: William Studenmund <[email-masked]>
Date: Mon, 11 Oct 2004 21:01:19 -0400

On Oct 11, 2004, at 6:07 PM, Mallikarjun C. wrote:

> I think I may be finally understanding your scenario, but let's 
> confirm that....

Ok!

> MaxRecvDataSegementLength.

Specifically, for what I'm thinking of, all of the data have already 
been sent before the MaxRecvDataSegmentLength

> The task continues.  Subsequently, the initiator notices it lost a 
> status.  Here's what I think should happen (I think the same as what 
> you described, except for the ExpDataSN part).
>
> target retransmits the status in an explicit Response PDU with 
> ExpDataSN set to something like 0xffffffff.
> followed by a status SNACK first) - i.e. doesn't phase collapse even 
> if it still can.

How about this idea. If the target can still phase collapse (the PDU 
would still fit), it MAY, but due to the fact that there was a 
resegmentation, the target MAY instead treat the PDU as if it could not 
fit. That is to say it is an implementation decision if the target just 
remembers "resegmented" or if it looks at the PDU.

> spec intent to not have to do scoreboarding/counting.

How does what I suggest break ExpDataSN?

I was about to say I like what you suggested, then I pondered a bit. I 
don't think we need the magic ExpDataSN value. If the PDU is too big to 
send, then the target can just generate a SCSI Response PDU with the 
correct ExpDataSN and a SNACK Tag of 0. Given that the initiator didn't 
notice the status value in the phase-collapsed read PDU, it should also 
have not noticed the data reception in it, so there should be a 
disconnect between ExpDataSN on the target (sent in the SCSI Response) 
and in the initiator. So a SCSI Response sent in place of the collapsed 
Data-In should trigger the initiator to send an R-Data SNACK. Shouldn't 
it?

> target had already sent 2**32-1 PDUs for that task), what you 
> described works as well (if the initiator is counting bytes).

I don't see how you have to count bytes. I'm not trying to sound 
belligerent, and I'm sorry if it comes across that way. But I think 
just sending the SCSI Response would work. I'm not seeing how the # of 
bytes'd be needed.

The one thing that would be impacted would be a broken initiator that 
got the data out of this phase-collapsed-read Data-In PDU but not the 
status. I think it deserves what it gets...

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Tue, 12 Oct 2004 16:00:36 -0700

Bill,

 >there should be a
 > and in the initiator.

Yes, that should work for the scenario you described (and in other cases 
that I can now think of).

In general, during the time we designed the iSCSI resegmentation 
approach, the operating assumption as I recall was to not rely on 
absolute sequence numbers from the previous epoch (i.e. before the PDU 
size change) after the change.  The R-Data SNACK simply causes a 
reversion to the last ack point which is consistent with not using 
absolute SNs from the past.

The thinking then was that between possibly multiple PDU size changes 
during the life of a task and and sharing of the DataSN/R2TSN space for 
bidirectional commands, there's enough room for implementations to get 
the absolute sequence numbers wrong in their SNACK requests.  Besides, 
we (at least I) didn't feel like we have exhaustively enumerated all 
potential scenarios.  As far as I recall, we have not defined a 
quiescing process to change the PDU size either.

I'd of course like to hear from other participants of those discussions 
wrt this recollection.

At least this is my rationale for suggesting a big hammer approach that 
doesn't rely on the ExpDataSN field in the status PDU (BTW, I don't 
recall a discussion on ExpDataSN in the past).  But I do realize that in 
my approach, under this scenario, the ExpDataSN field in the Response 
PDU is always suspect - so a SNACK is always required or the I/O would fail.

Thanks.
-- 
Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:

> Data-In should trigger the initiator to send an R-Data SNACK. Shouldn't it?
> 
> belligerent, and I'm sorry if it comes across that way. But I think just 
> sending the SCSI Response would work. I'm not seeing how the # of 
> bytes'd be needed.
>

---

From: "Eddy Quicksall" <[email-masked]>
Date: Wed, 13 Oct 2004 08:48:53 -0400

Is there any way the initiator can loose status other than from a CRC error? 
In which case if markers are not used, there is no way to recover other than 
reinstating the connection. Is that correct?

Eddy

----- Original Message ----- 
From: "Mallikarjun C." <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, October 12, 2004 7:00 PM
Subject: Re: [Ips] resending Data-in with S bit set

> In general, during the time we designed the iSCSI resegmentation approach, 
> the operating assumption as I recall was to not rely on absolute sequence 
> numbers from the previous epoch (i.e. before the PDU size change) after 
> the change.  The R-Data SNACK simply causes a reversion to the last ack 
> point which is consistent with not using absolute SNs from the past.
>
> bidirectional commands, there's enough room for implementations to get the 
> absolute sequence numbers wrong in their SNACK requests.  Besides, we (at 
> least I) didn't feel like we have exhaustively enumerated all potential 
> scenarios.  As far as I recall, we have not defined a quiescing process to 
> change the PDU size either.
>
> doesn't rely on the ExpDataSN field in the status PDU (BTW, I don't recall 
> a discussion on ExpDataSN in the past).  But I do realize that in my 
> approach, under this scenario, the ExpDataSN field in the Response PDU is 
> always suspect - so a SNACK is always required or the I/O would fail.
>
> Thanks.
> -- 
> Mallikarjun
>
> Mallikarjun Chadalapaka
>> Specifically, for what I'm thinking of, all of the data have already been 
>> sent before the MaxRecvDataSegmentLength
>>
>>> status.  Here's what I think should happen (I think the same as what you 
>>> described, except for the ExpDataSN part).
>>>
>> sending the SCSI Response would work. I'm not seeing how the # of bytes'd 
>> be needed.
>>
>> The one thing that would be impacted would be a broken initiator that got 
>> the data out of this phase-collapsed-read Data-In PDU but not the status. 
>> I think it deserves what it gets...
>>
>> Take care,
>>
>> Bill
>
>
>
>

---

From: William Studenmund <[email-masked]>
Date: Wed, 13 Oct 2004 10:00:25 -0400

On Oct 12, 2004, at 7:00 PM, Mallikarjun C. wrote:

> Yes, that should work for the scenario you described (and in other 
> cases that I can now think of).
>
> during the life of a task and and sharing of the DataSN/R2TSN space 
> for bidirectional commands, there's enough room for implementations to 
> get the absolute sequence numbers wrong in their SNACK requests.  
> Besides, we (at least I) didn't feel like we have exhaustively 
> enumerated all potential scenarios.  As far as I recall, we have not 
> defined a quiescing process to change the PDU size either.

I definitely agree that once there's been a resegmentation during the 
data sending, that something like R-Data SNACK is what we need. And as 
a target implementer, I think R-Data SNACK is the right thing to do; 
trying to figure out what was where & all just seems like a mess. Plus, 
it could be you can't resend any of the old PDUs....

> I'd of course like to hear from other participants of those 
> discussions wrt this recollection.
>
> At least this is my rationale for suggesting a big hammer approach 
> that doesn't rely on the ExpDataSN field in the status PDU (BTW, I 
> don't recall a discussion on ExpDataSN in the past).  But I do realize 
> that in my approach, under this scenario, the ExpDataSN field in the 
> Response PDU is always suspect - so a SNACK is always required or the 
> I/O would fail.

Here's another way to look at my suggestion. It is what you're 
suggesting, except that the magic DataSN number's different. Rather 
than 0xffffffff, I chose the DataSN we put in the unacknowledged 
Data-IN. My idea was that since the initiator didn't get the StatSN 
we're talking about, it didn't get the Data-IN PDU. So the _one_ DataSN 
number we're certain the initiator didn't get was the one in that 
packet. :-) And since the only thing the initiator will be able to do 
once it gets the SCSI Response with the DataSN it won't agree with is 
to send an R-Data SNACK, we will then have a SNACK tag and the sequence 
number spaces are safe.

Although to be honest, we could go with the 0xffffffff magic number you 
mentioned. We'd need to make some sort of supplemental RFC describing 
it, but we could. The reason I think it'd be fine is that 
MaxRecvDataSegmentLength can't go below 512. So we would need to go 
through around 512 error recovery actions for each original PDU in the 
transfer and have had a maximally-sized transfer to hit 0xffffffff for 
DataSN/R2T.

Take care,

Bill

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 14 Oct 2004 11:17:57 -0700

I think there are 3 cases here.

Status can be lost if it's phase-collapsed and there's a data digest 
failure on the last Data-in PDU.  A SNACK on the Data-in PDU should work 
here.

If there's no phase collapse, a data digest failure on the sense & 
response data segment of Response PDU would lead to loss of status.  A 
SNACK on the Response PDU should work here.

If there's no phase collapse, a header digest failure on the Response 
PDU would also lead to loss of status.  Markers would be generally 
required here, but the RFC doesn't mandate it other than saying PDU 
boundaries should be "reliably ascertained by other means" - other than 
based on length fields. (section 6.7).  Waiting for a sequence error 
should work here.
-- 
Mallikarjun

Mallikarjun Chadalapaka

Eddy Quicksall wrote:
> Is there any way the initiator can loose status other than from a CRC 
> error? In which case if markers are not used, there is no way to recover 
> other than reinstating the connection. Is that correct?
> 
> Eddy
> 
> ----- Original Message ----- From: "Mallikarjun C." <[email-masked]>
> To: <[email-masked]>
>>>> - Regardless of the original transmission (phase collapse or not, 
>>>> old PDU still fits with the new size or not, single/multiple data 
>>>> PDUs), target retransmits the status in an explicit Response PDU 
>>>> with ExpDataSN set to something like 0xffffffff.
>>>> - Initiator checks ExpDataSN with its records.  They shouldn't 
>>>> match, leading to an R-Data SNACK.
>>>> - Target always does the same thing in this scenario (PDU size 
>>>> change followed by a status SNACK first) - i.e. doesn't phase 
>>>> collapse even if it still can.
>>>
>>>
>>>
>>> resegmentation, the target MAY instead treat the PDU as if it could 
>>> not fit. That is to say it is an implementation decision if the 
>>> target just remembers "resegmented" or if it looks at the PDU.
>>>
>>>> - Initiator operates based on ExpDataSN as always, consistent with 
>>>> the spec intent to not have to do scoreboarding/counting.
>>>
>>>
>>>
>>> I was about to say I like what you suggested, then I pondered a bit. 
>>> I don't think we need the magic ExpDataSN value. If the PDU is too 
>>> big to send, then the target can just generate a SCSI Response PDU 
>>> with the correct ExpDataSN and a SNACK Tag of 0. Given that the 
>>> initiator didn't notice the status value in the phase-collapsed read 
>>> PDU, it should also have not noticed the data reception in it, so 
>>> there should be a disconnect between ExpDataSN on the target (sent in 
>>> the SCSI Response) and in the initiator. So a SCSI Response sent in 
>>> place of the collapsed Data-In should trigger the initiator to send 
>>> an R-Data SNACK. Shouldn't it?
>>>
>>> just sending the SCSI Response would work. I'm not seeing how the # 
>>> of bytes'd be needed.
>>>

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 14 Oct 2004 11:35:38 -0700

>I think R-Data SNACK is the right thing to do;
 > trying to figure out what was where & all just seems like a mess.

This is basically the approach taken in 10.16.3.

 > Here's another way to look at my suggestion. ...So the _one_ DataSN 
number we're
 > certain the initiator didn't get was the one in that packet.

As I already said, I don't see an obvious problem with your suggestion. 
  It's just the "certain" part (which assumes both sides getting the 
absolute DataSNs right) that we were not too sure of.
-- 
Mallikarjun

Mallikarjun Chadalapaka

William Studenmund wrote:

> data sending, that something like R-Data SNACK is what we need. And as a 
> target implementer, I think R-Data SNACK is the right thing to do; 
> suggesting, except that the magic DataSN number's different. Rather than 
> 0xffffffff, I chose the DataSN we put in the unacknowledged Data-IN. My 
> idea was that since the initiator didn't get the StatSN we're talking 
> about, it didn't get the Data-IN PDU. So the _one_ DataSN number we're 
> certain the initiator didn't get was the one in that packet. :-) And 
> since the only thing the initiator will be able to do once it gets the 
> SCSI Response with the DataSN it won't agree with is to send an R-Data 
> SNACK, we will then have a SNACK tag and the sequence number spaces are 
> safe.
>