Subject: [Ips] Handling error during packet transmission

From: [email-masked]
Date: Thu, 6 Nov 2003 16:21:41 -0800 (PST)

Ok, here's an error that I admit is a bit extreme.

What do you (a target or initiator) do if you have sent the header for a
PDU and then realize, say due to ECC memory error detection, that you
can't send the rest of the PDU?

Say the target is servicing a SCSI READ command. It sets up a PDU for
transmission. Assume we're on a link where we aren't sending jumbo grams
and the PDU will mean multiple ethernet frames. Now say that there is a
(detected) data corruption event during the transmission of the PDU. Like
a double bit error on ECC system memory for instance. The point is the PDU
header has gone out but we don't have all the data.

And assume that when we go to try and recover the data, we find out we
can't easily get it (it wasn't say a PCI transfer error that re-running
the transfer will recover). We decide we don't want to stall the TCP
connection getting it.

What do we do?

If we had data digests, one option would be to fill the rest of the
PDU with nonsense (zeros?), and deliberately send a bad digest.

If we didn't have digests, would it be reasonable to send garbage (zeros?)
to fill the PDU, then abort the command with a CHECK CONDITION and
indicate an underrun that didn't include the unsent PDU data?

As an example, we successfully send 8K of data, then we are sending a
DATA-IN PDU with 8k, and run into the error after 4k. We abort the
command, and indicate that only 12k of data were sent. ?

Yes, this shouldn't happen, but what do you do if it does?

Take care,

Bill

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Paul Koning <[email-masked]>
Date: Thu, 6 Nov 2003 21:19:00 -0500

>>>>> "wrstuden" == wrstuden  <[email-masked]> writes:

 wrstuden> Ok, here's an error that I admit is a bit extreme.  What do
 wrstuden> you (a target or initiator) do if you have sent the header
 wrstuden> for a PDU and then realize, say due to ECC memory error
 wrstuden> detection, that you can't send the rest of the PDU?
 wrstuden> ...
 wrstuden> What do we do?

 wrstuden> If we had data digests, one option would be to fill the
 wrstuden> rest of the PDU with nonsense (zeros?), and deliberately
 wrstuden> send a bad digest.

Agreed.

 wrstuden> If we didn't have digests, would it be reasonable to send
 wrstuden> garbage (zeros?)  to fill the PDU, then abort the command
 wrstuden> with a CHECK CONDITION and indicate an underrun that didn't
 wrstuden> include the unsent PDU data?

I don't like that; the garbage you sent might be delivered to user
buffers and mistaken for real data.

Better just to abort the connection.  This sort of thing should be
extremely rare; there is no need to make recovery from it efficient.

	  paul

---

From: Andre Hedrick <[email-masked]>
Date: Thu, 6 Nov 2003 18:39:16 -0800 (PST)

Paul and Bill,

I agree w/ Paul that loading the PDU with garbage is a bad idea.
One may not be able to tell the difference between a crypto fs on top.

If junk is going to be sent, logic dictates a defined pattern is to be
adopted.  Then the question comes, what is the granularity of the pattern?

Comments?

Andre Hedrick
LAD Storage Consulting Group

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 6 Nov 2003 20:56:00 -0600

> extremely rare; there is no need to make recovery from it efficient.

Why not simply send bad TCP frames? In other words, send a bad digest
down one layer. It's better than aborting the connection.

Admittedly, you are jumping a few layers. But if you can detect a
data error mid packet I would suspect that the implementation is
already just a little bit rototilled.

---

From: Julian Satran <[email-masked]>
Date: Fri, 7 Nov 2003 07:57:00 +0200

Bill,

You have now more advise that you would have cared for. Aborting TCP 
packets (since they are not replaced) won't get you very far.
We have decided early on that the underlying transport is reliable and (as 
TCP does) we did not provide for an abort.
For some streaming operations that might have been a mistake.

Regards,
Julo

Caitlin Bestler <[email-masked]> 
Sent by: [email-masked]
07/11/2003 04:56

To
[email-masked]
cc

Subject
Re: [Ips] Handling error during packet transmission

---

From: "Rod Harrison" <[email-masked]>
Date: Fri, 7 Nov 2003 10:47:14 -0000

I agree with Paul, this is rare enough to employ the big hammer.
Tear the connection down with an RST segment and let the
initiator recover. I wouldn't even bother adding logic to send a
bad digest if they are in use, doesn't happen often enough.

- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Friday, November 07, 2003 5:57 AM
To: Caitlin Bestler
Cc: [email-masked]; [email-masked]
Subject: Re: [Ips] Handling error during packet transmission

Bill,

You have now more advise that you would have cared for. Aborting
TCP packets (since they are not replaced) won't get you very far.
We have decided early on that the underlying transport is
reliable and (as TCP does) we did not provide for an abort.
For some streaming operations that might have been a mistake.

Regards,
Julo

Caitlin Bestler <[email-masked]>
Sent by: [email-masked]
07/11/2003 04:56 [email-masked]
cc
SubjectRe: [Ips] Handling error during packet transmission

> extremely rare; there is no need to make recovery from it
efficient.

Why not simply send bad TCP frames? In other words, send a bad
digest
down one layer. It's better than aborting the connection.

---

From: Paul Koning <[email-masked]>
Date: Fri, 7 Nov 2003 09:59:26 -0500

>>>>> "Caitlin" == Caitlin Bestler <[email-masked]> writes:

 Caitlin> On Thursday, November 6, 2003, at 08:19 PM, Paul Koning
 Caitlin> wrote:

 >> extremely rare; there is no need to make recovery from it

 Caitlin> Why not simply send bad TCP frames? In other words, send a
 Caitlin> bad digest down one layer. It's better than aborting the
 Caitlin> connection.

That will not work at all.  If you send a TCP packet with a bad TCP
checksum, it will be discarded by the receiver, which doesn't ACK it
and doesn't accept subsequent packets (because they are out of
order).  The sender would normally retransmit such dropped packets.
If the packet has a forced back checksum, then the connection cannot
make progress.  In other words, the effect is identical to dropping
the connection right away, except far more clumsily and less
reliably. 

	  paul

---

From: Jason Thorpe <[email-masked]>
Date: Thu, 6 Nov 2003 22:15:41 -0800

> down one layer. It's better than aborting the connection.

I don't see how that helps.  If you send a bad TCP packet, the iSCSI 
application layer won't see that.  The problem is that you have to 
somehow notify the peer iSCSI application.

I think Paul's suggestion of just shooting the connection in the head 
is a reasonable one.

> already just a little bit rototilled.

It's easier to detect an error mid-packet than you might think.  A trap 
occurs, the thread that caused the trap is identified, and a 
notification is sent to that thread that something bad happened.  Most 
operating systems I'm familiar with have had this capability for quite 
some time.

         -- Jason R. Thorpe <[email-masked]>

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 7 Nov 2003 10:01:56 -0600

> the connection right away, except far more clumsily and less

If you are cleanly layered, the entire iSCSI packet was probably
validated *before* you started transmitting.

If you are not cleanly layered, there is nothing to keep you from
refetching the content for the retransmit. Generating a bad TCP
checksum is just acknowledging that what was earlier in the packet
was a "bad copy". The retransmit should of course be of the correct
content.

If your layering allows you to generate a bad TCP checksum, the
odds are that it would also allow you to fix the buffer for the
retransmit.

Of course that assumes a very embedded implementation. But any
reaction that is taking place *during* the transmission of a packet
is likely to be in an embedded implementation.

---

From: Michael Krause <[email-masked]>
Date: Fri, 07 Nov 2003 08:07:18 -0800

>is likely to be in an embedded implementation.

Still seems simpler (especially as this may be a rare condition) to simply 
drop the connection and start anew which is just an application of KISS 
that will work across any implementation.

Mike

---

From: Paul Koning <[email-masked]>
Date: Fri, 7 Nov 2003 11:11:32 -0500

>>>>> "Caitlin" == Caitlin Bestler <[email-masked]> writes:

 Caitlin> On Friday, November 7, 2003, at 08:59 AM, Paul Koning wrote:

 >> That will not work at all.  If you send a TCP packet with a bad
 >> TCP checksum, it will be discarded by the receiver, which doesn't
 >> ACK it and doesn't accept subsequent packets (because they are out
 >> of order).  The sender would normally retransmit such dropped
 >> packets.  If the packet has a forced back checksum, then the
 >> connection cannot make progress.  In other words, the effect is
 >> identical to dropping the connection right away, except far more
 >> clumsily and less reliably.
 >> 
 >> paul

 Caitlin> If you are cleanly layered, the entire iSCSI packet was
 Caitlin> probably validated *before* you started transmitting.

 Caitlin> If you are not cleanly layered, there is nothing to keep you
 Caitlin> from refetching the content for the retransmit. Generating a
 Caitlin> bad TCP checksum is just acknowledging that what was earlier
 Caitlin> in the packet was a "bad copy". The retransmit should of
 Caitlin> course be of the correct content.

If you can re-fetch and get good content, you should just do that.  No
need to confuse the protocol with weird messages; recovery is
completely an internal matter.

I was assuming the question had to do with the case where the error
was NOT recoverable.  In that case you have to fail the I/O in
mid-transfer.  The way to do that, for cases where the problem is seen
in a place where the protocol doesn't provide for a way to abort
cleanly, is to reset the TCP connection.

	 paul

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 7 Nov 2003 11:43:16 -0600

> cleanly, is to reset the TCP connection.

Agreed, if the error cannot be recovered then the connection is
doomed anyway. If however, you could re-read the sector and
get a good copy, using TCP to give you the time to do that
re-read may be a better solution. Of course you already need
to be structured in a certain way to do this. I certainly would
not recommend removing layer separation *for* this purpose.

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 07 Nov 2003 10:47:11 -0800

Bill,

I think you have all the different possible solutions identified.
My take is that the choice is one of implementation/solution
considerations - such as how fault-tolerant you want to be,
how many other I/Os are likely active on the connection at
any time, how expensive it is to recover the other I/Os if connection
is dropped, how flexible is your code, can the target fence
the malfunctioning physical memory and continue on or the
session has to be killed anyway...etc.

Given that the SCSI initiator cannot make any assumptions
about the contents of a posted Data-in buffer unless it receives
a successful completion from the target, the only spec requirement
in the scenario you describe is that the target must not send
a successful SCSI status.

Mallikarjun