Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"

From: "Jon Hall" <[email-masked]>
Date: Mon, 02 Apr 2001 10:13:35 -0400

I agree with Somesh.  And would go farther -- the complexity
that results from retaining enough target-side state to respond
to a SACK/SNACK request is non-trivial and needs clear justification.
Intuitively, a CRC that discovers an error in an iSCSI pdu header
(that the TCP cksum missed) seems like it should be a rare event.

What is the frequency of this event?  IMO the answer to this
question should be written into the protocol spec -- assuming
that it substantiates the benefit of SACK/SNACK.  Otherwise, the
SACK/SNACK pdu should be removed.

-Jon

[email-masked] writes:
>
>Somesh,
>
>As I stated earlier - the DataSN was created to detect missing data PDUs.
>SNACK is needed to recover missing StatusSN and missing dataSN is only a
>bonus if the target wants to support it.  It is a trivial mechanism and I
>think it should stay.
>
>Julo
>
>"Somesh Gupta" <[email-masked]> on 31/03/2001 02:25:52
>
>Please respond to [email-masked]
>
>To:   Julian Satran/Haifa/IBM@IBMIL, [email-masked]
>cc:
>Subject:  RE: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>
>
>
>
>Sorry to have been missing for a while. Hope you will
>appreciate my being back in action :-). It was a fairly
>clear consensus in Orlando that applications broke up
>their transfers into reasonably small chunks i.e. they
>did not have very long running transfers.
>
>Therefore the consensus was that a command level recovery
>mechanism was sufficient instead of an ack/sack for each
>data PDU.
>
>The SACK mechanism was a post Orlando invention. Without
>an ack mechanism (for every data PDU), the SACK mechanism
>just imposes additional burden on either end of the session,
>without really much benefit.
>
>The benefit of having SACK is of saving bandwidth in case
>the data part of the data PDU failed an integrity check
>(but passed TCP checksum). This is a rare enough case that
>as a percentage, the bandwidth loss from retransmitting
>all the data associated with a read or write command is
>very very small.
>
>In addition, it avoids the complexity of restarting
>something from the middle, as compared to from the begining.
>
>To me it seems that there is significant simplicity (from
>implementation, reliability and recovery process) from
>having smaller data transfer per command.
>
>I would really like to get rid of the SACK command.
>
>Somesh
>
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]]On Behalf Of
>> [email-masked]
>> Sent: Wednesday, March 28, 2001 6:57 AM
>> To: [email-masked]
>> Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>>
>>
>>
>>
>> Mallikarjun,
>>
>> Last summer I thought that recovery within a connection should be left to
>> TCP. It is simple and could be made available through IPsec (if no new
>> option of any form can be added).
>>
>> Two things killed this:
>>
>>    The requirement to have a data encapsulation that can pass through
>>    application proxies (like a storage router)
>>    The "NO WAY" message we got from IESG-Security on a CRC only IPSec
>>    header
>>
>>
>> As for the ACK - I am very much in favor of it (it is a no brainer) and
>> implementations are in fact allowed to drop even unacked data.
>>
>> I am bound by the Orlando meeting decision to drop it. Except the regular
>> "oppose everything" crowd the two vocal opponents where Somesh Gupta and
>> Matt Wakeley.
>>
>> David may want or not to re-open the issue - I am not going to ask for
>it.
>>
>> Regards,
>> Julo
>>
>> "Mallikarjun C." <[email-masked]> on 28/03/2001 00:45:02
>>
>> Please respond to [email-masked]
>>
>> To:   [email-masked]
>> cc:   Julian Satran/Haifa/IBM@IBMIL, [email-masked], [email-masked],
>>       [email-masked], John Hufferd/San Jose/IBM@IBMUS,
>>       [email-masked], [email-masked]
>> Subject:  RE: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>>
>>
>>
>>
>> David and Julian,
>>
>> I appreciate both your views, and should I say that they're
>> along predicted lines :-)
>>
>> - David's right in saying that the situation is akin to FC's.
>>   However, I would like to point out that FC is an unreliable
>>   transport, and hence is forced to pick up a lot of the transport
>>   baggage (at least in FCP-2, as I understand), in addition
>>   to being a SCSI encapsulation layer.  Unfortunately, even with
>>   TCP being the "reliable" transport, iSCSI is going along the
>>   same lines - ie. transport baggage + SCSI encapsulation.  My
>>   point is - if this is indeed a necessary evil, why don't we
>>   complete iSCSI's transport functionality by data-ACKs?
>>
>> - If data SACK is introduced mostly to make up for TCP's shortcomings,
>>   we're making its usage (and implementation) drastically less appealing
>>   since the only way error recovery algorithms can *rely* on data SACK
>>   is when replay is supported (or, "ReplaySupport=yes"  in my proposal),
>>   which is extremely expensive.  IOW, we're defining data SACK in the
>>   draft and not providing any incentives to implement and use it!
>>
>> - I submit that since iSCSI is being hailed as the ideal SCSI Transport
>>   protocol in its definition so far (and I believe, rightly so -
>mandating
>>   command ordering, bi-di support, SCSI CRN support to name a few
>> examples),
>>   the perfectly SCSI-legal R/W interactions that break in other
>transports
>>   *do not* have to break in iSCSI.
>>
>> - A last idea (may seem radical at this point) in regards to iSCSI
>>   being a "full transport". This provides us an opportunity to "cast
>>   off" the transport baggage in future when we truly move to a "reliable"
>>   transport (perhaps TCP with CRCs/SCTP ?) - if we do a good job of
>>   keeping the encapsulation stuff separate from the transport stuff.
>>   (Julian, I heard from Randy that ideas similar to this were explored
>>   in your Haifa meeting.  And yes, he recalls they were given up since
>>   TCP was supposed to be reliable and granularity of recovery was deemed
>>   one I/O.)
>>
>> With that said, may I request David (with his co-chair hat on, :-))
>> to add some binding comments/observations on this discussion?
>>
>> If we decide to leave data SACKs as unattractive to implement, the draft
>> should in the least add a statement like - "Note that satisfying all
>> possible data SACK requests for a task with an unacknowledged status
>> implies implementing the I/O replay buffer on the part of targets."
>> --
>> Mallikarjun
>>
>>
>> Mallikarjun Chadalapaka
>> Networked Storage Architecture
>> Network Storage Solutions Organization
>> MS 5668   Hewlett-Packard, Roseville.
>> [email-masked]
>>
>>
>>
>>
>> >I think Julian's basically right -- I would point
>> >out that any case of write after read that breaks
>> >over iSCSI will also break over Fibre Channel.
>> >On FC, the scenario starts with a frame CRC failure
>> >on read data at the Initiator, so applications
>> >have to cope and typically do so by enforcing
>> >ordering at the app rather than using SCSI task
>> >ordering.
>> >
>> >While SCSI has clever tools like ACA and task
>> >ordering that appear to allow dependent operations
>> >to be sent to the target concurrently, in practice
>> >they don't work and/or aren't used (funny thing,
>> >those two reinforce each other ;-) ).  Hence
>> >a minimal approach to them is in order:
>> >- Make sure the result will interoperate.
>> >- Make sure T10 doesn't ding us for leaving something
>> >    completely out.
>> >- Don't specify anything not needed for the above.
>> >
>> >My 0.02,
>> >--David
>> >
>> >> -----Original Message-----
>> >> From:  [email-masked] [SMTP:[email-masked]]
>> >> Sent:  Tuesday, March 27, 2001 9:23 AM
>> >> To:    [email-masked]
>> >> Cc:    [email-masked]; [email-masked]; [email-masked];
>> >> [email-masked]; [email-masked]; Venkat Rangan;
>> >> [email-masked]
>> >> Subject:    Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>> >>
>> >>
>> >>
>> >> Mallikarjun,
>> >>
>> >> I commiserate with you at the lack of ack for data but the Orlando
>> meeting
>> >> stated - no.  Recall that I kept the number only as a mechanism to
>> detect
>> >> missing packets.
>> >>
>> >> You can achieve the effect you want by keeping around data for a while
>> >> (you
>> >> determine how long and then discard).
>> >>
>> >> If a SACK comes and you can recover - fine. If not you either reaccess
>> the
>> >> media (if you know how) or reject
>> >> and let the initiator retry.
>> >>
>> >> You should not worry about R/W conflicts as programs bound to have
>such
>> >> conflicts either:
>> >>
>> >> 1)can live with them or
>> >> 2)protect themselves through some locks and rely on
>> "operation-end-status"
>> >> to keep results deterministic.
>> >>
>> >> Regards,
>> >> Julo
>> >>
>> >>
>> >>
>> >> "Mallikarjun C." <[email-masked]> on 27/03/2001 03:34:16
>> >>
>> >> Please respond to [email-masked]
>> >>
>> >> To:   [email-masked], [email-masked], [email-masked],
>Julian
>> >>       Satran/Haifa/IBM@IBMIL, John Hufferd/San Jose/IBM@IBMUS
>> >> cc:   [email-masked]
>> >> Subject:  iSCSI ERT: data SACK/replay buffer/"semi-transport"
>> >>
>> >>
>> >>
>> >>
>> >> Hi Error Recovery Team,
>> >>
>> >> iSCSI can discard PDUs because of digest errors and request
>> >> retransmissions using the iSCSI data SACK.  To deal with such
>> >> an eventuality, targets that want to support data SACK have
>> >> the following options:
>> >>
>> >> (A) maintain a complete "replay" buffer for the entire I/O since
>> >>   a SACK could come anytime before the status is ack'ed by the
>> >>   initiator. [ simple, but extremely expensive in memory resources]
>> >>
>> >> (B) (re-introduce data-ACKs into the draft, and) implement data-ACKs.
>> >>   Thus enables keeping only those I/O buffers that haven't been ack'ed
>> >>   by the initiator. IOW, become a real full transport! [ everyone
>> disliked
>> >>   it earlier...]
>> >>
>> >> (C) re-access the medium for data retransmission requests.  Now there
>> >>   are 3 sub-cases in this to handle the changed data on the medium in
>a
>> >>   write-after-read scenario.  (SEE NOTE.1 at the bottom on how it is
>> >> legal.)
>> >>      (1) On seeing any write, stall till status is ack'ed for all the
>> >>             previous reads (basically drain the pipe). [simple, but
>> incurs
>> >>             an additional roundtrip delay for all writes].
>> >>      (2) A variation of the above, keep an eye only on the prior
>> >>             overlapping reads. [more BW efficient, but complicated to
>> >>             resolve the block dependencies in a stream of
>> reads followed
>> >>             by writes]
>> >>         (3) Document the caveat and leave it upto the applications
>> >>             to avoid this case since this leads to data integrity
>> issues.
>> >>             [pushing to apps since the transport can't get it right!]
>> >>
>> >> My first preference is (B), followed by (A), and I suggest we not go
>> >> to (C) at all with its inherent dangers.
>> >>
>> >> Doing (B) naturally completes the transport job that iSCSI has taken
>> >> on itself in view of TCP's claimed unreliable checksum.  That is the
>> >> right thing to do architecturally instead of being a "semi-transport"!
>> >>
>> >> Comments?
>> >> --
>> >> Mallikarjun
>> >>
>> >>
>> >> Mallikarjun Chadalapaka
>> >> Networked Storage Architecture
>> >> Network Storage Solutions Organization
>> >> MS 5668   Hewlett-Packard, Roseville.
>> >> [email-masked]
>> >>
>> >>
>>
>__________________________________________________________________________
>> >> Note.1: A Read followed by a Write (to the same blocks) is perfectly
>> legal
>> >>         if SCSI sets the ORDERED task attribute on both the
>> commands AND
>> >>         sets the NACA bit to one to indicate that Write shall be
>> executed
>> >>         only if the Read did not fail (result in a Check Condition).
>> >>
>> >>         In the current case, since Read completed just fine from
>SCSI's
>> >>         point of view, SCSI is moving on to execute Write.  Those read
>> >> buffers
>> >>         had been freed up since iSCSI received an ACK at the TCP
>level,
>> >> and
>> >>         since iSCSI has no other way to have the data ack'ed!

---

From: "Jon Hall" <[email-masked]>
Date: Mon, 02 Apr 2001 17:48:42 -0400

I thought (?) Stone and Partridge concluded somewhere in the
range between one packet in 16 million to one packet in 10
billion.

But, what percent of an iSCSI flow is comprised of iSCSI pdu
headers?  Is there a credible flow model for deciding the
probability of the one packet containing a StatSN?

-Jon

Pierre Labat writes:
>"Mallikarjun C." wrote:
>
>> To be fair to data SACK, one could think of an upper bound
>> on the unack'ed data - agreed on at the login time.  While not
>> requiring acks on every PDU, it gives targets the deterministic
>> maximum on the buffer size they have to keep around if they
>> choose to "reliably" support data SACK.  The current answer of
>> "replay buffer size/IO size", IMHO, is simply not attractive.
>> Also to be fair to data SACK, I believe FCP-2 allows sequence-level
>> error recovery in an I/O.
>>
>> However, I think that it's extremely useful to include a discussion
>> in the draft of  the TCP checksum "escape" statistics and the
>> device types for which this was considered an absolute requirement
>> to make forward progress at this error rates (like huge tape
>> backups?) - essentially the reasons that convinced Julian to define
>> this mechanism in. That gives credibility and acceptance to this,
>> or alternately may lead to the consensus that data SACK is not required.
>> --
>> Mallikarjun
>
>About  TCP checksum "escape" statistics what i saw is :
>
>A) Jonathan Stone,Craig Partridge, "When the CRC and TCP Checksum Disagree"
> http://www.acm.org/sigcomm/sigcomm2000/conf/abstract/9-1.htm
>
>1 escape in 200 millions
>
>B) V. Paxson "1999 End-to-End Internet Packet Dynamics."
>   http://www.aciri.org/vern/papers.html
>
> 1 escape in 300 millions
>
>C) J. Stone et. al "Performance of Checksums and CRC's over Real Data"
>  IEEE/ACM Transactions on Networking, Vol. 6, No. 5, October 1998
>
>http://dev.acm.org/pubs/articles/journals/ton/1998-6-5/p529-stone/p529-stone.pdf
>
>Less than 1 escape in 10e17 segments when taking into
>account the link layer AAL5 CRC. (see page 540 left column on top).
>
>
>
>
>Taking the worst is 1 in 200 millions.
>
>Regards,
>
>Pierre

---

From: Venkat Rangan <[email-masked]>
Date: Mon, 2 Apr 2001 14:48:19 -0700

Mallikarjun,

> error recovery in an I/O.  

While this is true, in practice, few implementations appear to actually
perform sequence-level error recovery. Passing up a CRC error as a 'Delivery
Subsystem Failure' is the way they seem to recover. Another point is that
with iSCSI (unlike FC), the TCP layer already has done the stream-level
recovery, based on link-level CRC mismatches dropping the frame (before TCP
sees it). So the coverage of data SACK is those times when link level CRC
errors are overcome by TCP recovery but TCP checksum is unable to detect the
error, but iSCSI data digest detects it.

Venkat Rangan
Rhapsody Networks Inc.
http://www.rhapsodynetworks.com

-----Original Message-----
From: Mallikarjun C. [mailto:[email-masked]]
Sent: Monday, April 02, 2001 10:34 AM
To: [email-masked]
Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport"

Mallikarjun Chadalapaka

it.
mandating
transports
such
Julian
a
SCSI's
level,
>_________________________________________________________
>Do You Yahoo!?
>Get your free @yahoo.com address at http://mail.yahoo.com
>
>

---

From: [email-masked]
Date: Wed, 4 Apr 2001 17:02:11 +0200

Sorry,

Sorry - I meant to say many commands with small data amounts.

Julo

"Somesh Gupta" <[email-masked]> on 04/04/2001 01:31:23

Julian,

Which part of my note were you raising a concern about?

Somesh

> Sent: Monday, April 02, 2001 6:25 PM
> To: [email-masked]
> That will certainly result in poor performance for important applications
> even with hardware implementations of iSCSI - mainly due to the large
SCSI
> command traffic and associated interrupts.
>
> Julo
>
> "Somesh Gupta" <[email-masked]> on 02/04/2001 22:23:25
>
> To:   [email-masked], [email-masked]
> cc:
> To beat a dead horse ..
> One has to really decide fundamentally whether
>
> 1. Commands are used to transfer very large amounts of
>    data (multiple data PDUs are needed)
> 2. Commands are used to transfer relatively small amounts
>    of data (few/about one data PDU) and multiple commands
>    are then used to do long transfers
>
> (Orlando consensus was #2)
>
> If we assume the first model, then we really should have
> a sequence # and acknowledgement of every PDU - not just
> data PDUs. In this case, it is important to fill holes
> in the iSCSI stream. We can have a "super-transport" as
> Mallikarjun suggested between the iSCSI protocol layer
> and the TCP layer that provides the various "transport"
> like features we seem to want.
>
> If we assume the second model, we assume that recovery at
> the command level is sufficient. In this case it is important
> to have whatever mechanisms are (including data seq #s) needed
> to detect that a command will not succeed without recovery
> at the command level. However, recovery is needed only
> at the command level.
>
> I would let the current application model decide the features
> in "version 1" of the iSCSI protocol.
>
> Somesh
>
> > Mallikarjun C.
> > >> David may want or not to re-open the issue - I am not going to
> > ask for it.
> > >>
> > >> Regards,
> > >> Julo
> > >>
> > >>   protocol in its definition so far (and I believe, rightly so
> > - mandating
> > >>   the perfectly SCSI-legal R/W interactions that break in
> > other transports
> > >> >> You should not worry about R/W conflicts as programs bound
> > to have such
> > >> >> conflicts either:
> > >> >>
> > >> >> 1)can live with them or
> > >> >> To:   [email-masked], [email-masked],
> > [email-masked], Julian
> > >> >>   are 3 sub-cases in this to handle the changed data on the
> > medium in a
> > >> >>         In the current case, since Read completed just fine
> > from SCSI's
> > >> >>         had been freed up since iSCSI received an ACK at
> > the TCP level,
> > >> >> and

---

From: [email-masked]
Date: Wed, 4 Apr 2001 20:55:48 +0200

What are the numbers you are looking at:

1 per 10 sec, 1/10h or 1 /10y?

Julo

"Somesh Gupta" <[email-masked]> on 04/04/2001 20:15:53

> Sent: Wednesday, April 04, 2001 7:32 AM
> To: [email-masked]
> SNACK is here for two reasons - Status retry (which is cheap) and Data
> retry as a side benefit.

  Unless there is clear benefit (i.e. the event is frequent enough
  to justify recovery at this level), the entire mechanism should be
  dropped - it is neither cheap nor free. If it is relatively
  infrequent, the recovery at the command level should be a sufficient
  mechanism

> CRC errors are not that rare (although we don't have real data the
> simulation with file systems seem to indicate that numbers could
> be as high
> a 0.0002%). A restart of link - is expensive (slow start) and even if
they
> are far lower for many applications a slow start is a painfull event.

  Intuitively, it seems that the combination of link level CRC, TCP
  checksum, and good hardware (ECC, parity etc) should lead to a
  much lower level of errors caught by the iSCSI CRC algorithm. We have
  to seperate error detection (i.e. what if I have bad hardware or
  some vendor makes bad/buggy intermediate system) from recovery
  mechanisms (not based on hardware being bad or buggy - market forces
  will wean out the vendor) which should not be based on assumptions
  of bugs in hardware/software of specific implementations.

>
> Removing them from the spec is not a path we should take lightly.

  I would phrase it the other way. We should not keep adding things
  unless there is very clear proof that the additional feature is
  beneficial and does not have negative side effects (and there is
  some consensus on adding it)
>
> Julo
>
> "Jon Hall" <[email-masked]> on 02/04/2001 16:13:35
>
> Please respond to "Jon Hall" <[email-masked]>
>
> To:   [email-masked]
> cc:
> medium in

---

From: "Somesh Gupta" <[email-masked]>
Date: Wed, 4 Apr 2001 14:02:06 -0700

Assuming that the packet corruption escape rate is 1 in 10billion,
we have (rough assuming 1K byte per packet), 1 escaped packet every
10 Trillion Bytes of data transfer. Seems to me that if I
had to transfer 1 MBytes for having to recover at the
command level rather than at a more granular level, that does
not pose much of an additional burden (1 MB out of 10 Trillion
bytes). Also assuming each i/o is 1 MByte in size, you would
have to do recovery for every 1 in 10 million transactions.

I don't know how realistic the 1 in 10 billion packet corruption
escape rate is but I am using the number from past discussions.

Somesh

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, April 04, 2001 11:56 AM
> To: [email-masked]
> Cc: [email-masked]
> > >> Last summer I thought that recovery within a connection
> should be left
> > to

---

From: Pierre Labat <[email-masked]>
Date: Wed, 04 Apr 2001 16:07:01 -0700

[email-masked] wrote:

> Yes - and I think that the effects can be observed even with random errors
> because of the weakness of the checksum.

Julian,

This paper shows that the TCP checksum is weaker in case of non uniform
data.

But we are interested by the overall performance Link layer CRC
+ TCP checksum.

And if i look at the conclusion of this paper
"VII. Observations and recommendations" there is:
page 540 upper left
"The ATM CRC will fail to detect a splice approximately at a rate of 1 in 2^32.
 Therefore the chance of the TCP checksum being called upon to detect a splice
 is much less than 1 in 10e-8 * 2e-32 or less than one chance in 10e17"

Hence i get an overall  escape rate of 1 in 10e17 or
in other words  0.000000000000001%

How do you get this number of 0.0002% ?

Regards,

Pierre

>
>
> Julo
>
> Pierre Labat <[email-masked]> on 04/04/2001 23:13:38
>
> Please respond to Pierre Labat <[email-masked]>
>
> To:   [email-masked]
> cc:
> [email-masked] wrote:
>
> > simulation with file systems seem to indicate that numbers could be as
> high
> > a 0.0002%).
>
> Julo,
>
> Could you explain how you get this number.
> Does it come from
>                      J. Stone et. al "Performance of Checksums and CRC's
> over
> Real Data"
> ???
> I don't see how you got this number.
> What i saw was:
>                     account the link layer AAL5 CRC. (see page 540 left
> column
> on top).
>
> Regards,
>
> Pierre
>
> > A restart of link - is expensive (slow start) and even if they

---

From: "Mallikarjun C." <[email-masked]>
Date: Wed, 04 Apr 2001 19:09:46 PDT

>
>Santosh,
>
>I can't find the place where this is stated. SNACK as a PDU type is
>mandated. But it can be rejected outright.

Sorry, you agreed that status SACK is mandatory in ERT forum last
week in response to my comments.  Has there been a change in your opinion?

Attached is the email in a long email thread (issue 3) where you agreed 
to make this explicit in rev06.
--
Mallikarjun 

Mallikarjun Chadalapaka

>1.2.2.2 show explicitely that SACK can be rejected. We can add a protocol
>specific parameter in the target Logical Unit Control Page (non-setable) by
>which the target will indicate support for SNACK.
>
>Julo

Santosh Rao <[email-masked]> on 04/04/2001 23:53:32

Please respond to Santosh Rao <[email-masked]>

To:   Julian Satran/Haifa/IBM@IBMIL
cc:   Jon Hall <[email-masked]>, [email-masked]
Subject:  Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"

[email-masked] wrote:
>
> Jon,
>
> Inexpensive implementation are always free to do away with recovery. That
> si true for targets too.

That's not the interpretation one gets from reading the spec and prior
discussions on this list. Per the spec, support for Status SACK is
mandatory while support for data SACK is optional.

IOW, targets MUST retains state information to satisfy a potential
status SACK request.

- Santosh

------------------------------------------------------------------------------

---

From: [email-masked]
Date: Thu, 5 Apr 2001 04:35:49 +0200

I will clarify. Julo

Santosh Rao <[email-masked]> on 05/04/2001 03:06:10

Julian,

"6.7.1.1 Recovery Within-connection

   At the initiator, the following cases lend themselves to
within-connection recovery: (1)Lost iSCSI numbered Response recognized
by either receiving it with a data digest error or receiving a Response
PDU with a higher StatSN than expected. The initiator MAY request the
missing responses through SACK, in which case the target MUST reissue
them. "

To me, the following statement :
"The initiator MAY request the missing responses through SACK, in which
case the target MUST reissue them."

implies that the target MUST be able to re-issue a SCSI Response PDU,
when a Status SACK is received, which implies the target MUST maintain
the I/O state information around [until StatSN is acknowledged.]

Section 1.2.2.2 of my version of iSCSI rev 05 (taken from
http://www.ietf.org/internet-drafts/draft-ietf-ips-iscsi-05.txt) reads
as follows :

"1.2.2.2 Response/Status Numbering and Acknowledging

   Responses in transit from the target to the initiator are numbered.
   The StatSN (Status Sequence Number) is used for this purpose. StatSN
   is a counter maintained per connection.  ExpStatSN is used by the
   initiator to acknowledge status.

   Status numbering starts after Login. During login, there is always
   only one outstanding command per connection and status numbering is
   not needed.
   The login response includes an initial value for status numbering.

Satran, J.       Standards-Track, Expire October 2001               12

                                iSCSI                   March 1, 2001

   To enable command recovery the target MAY maintain enough state
   information to enable data and status recovery after a connection
   failure.
   A target can discard all the state information maintained for
   recovery after the status delivery is acknowledged through ExpStatSN.
   A large difference between StatSN and ExpStatSN may indicate a failed
   connection.

   Initiators and Targets MUST support the response-numbering scheme. "

Can you please clarify where in the above section is there an example
that Status SACK can be rejected by targets ? Further, the Reject PDU
(Section 2.20.1) only allows a reject reason of Data SACK reject. There
is no reason code of "Status SACK Reject".

All this to me implies that Status SACK support is mandatory and targets
MUST maintain I/O state information until the StatSN is acknowledged.

Can you please clarify if this is not the case, and if so, what is the
intent of the above text ?

- Santosh

[email-masked] wrote:
> > But not specifying the mechanism for the more expensive one we make
them
> > non-interoperable.
 - santoshr.vcf

---

From: [email-masked]
Date: Thu, 5 Apr 2001 16:41:47 +0200

Correct - as to if  it happens when the Sun goes Nova I think it is far
more frequent and for critical applications (business or life) people might
be paying to avoid small glitches several times a day.

Julo

"Jon Hall" <[email-masked]> on 05/04/2001 15:55:11

To:   [email-masked]
cc:

Julian,

I don't understand.  Are you saying that an "expensive" target
will implement specific error recovery mechanisms for very rare
events?  Or are you saying that this case is not a rare event?

If the former, there is a problem of completeness (e.g., should
there be recovery procedures for when the sun goes nova :-).
If the latter, this would be very interesting and useful to
know about...

-Jon

[email-masked] writes:
>But not specifying the mechanism for the more expensive one we make them
>non-interoperable.
>
>Julo
>
>"Jon Hall" <[email-masked]> on 04/04/2001 22:55:40
>
>But CRC errors are not really the issue.  It is the
>singular case of a TCP cksum failing to detect what a
>CRC succeeds in detecting, and this occurring to a TCP
>segment containing an iSCSI hdr with a StatSN.
>
>Is there a reason to believe that iSCSI StatSNs will be
>lost at a higher rate than is currently documented for TCP
>cksum failure?  Or, is the problem a loss of one TCP segment
>in tens (possibly hundreds) of millions of segments.  Where
>the bad segment may contain a StatSN but probably doesn't
>because it is a data pdu.  If the latter, why does a SCSI-level
>timeout and retry (on the initiator) not suffice?  [Note,
>an initiator timeout/retry does not require a connection
>to be closed.]
>
>I realize that I am being annoyingly repetitious, but it is
>not an idle question.  For some targets, retained rsp status
>is not cheap (and retained rsp data is not tractable at all).
>
>IMO there appears to be no real need for SNACK.  And, more
>radically, there appears to be no need for StatSNs.
>
>Maybe, as Somesh said, this is a dead horse but why include
>something in the spec which suggests a need for target-side
>complexity, while not solving a clear and compelling
>requirement?
>
>-Jon
>
they

---

From: "Jon Hall" <[email-masked]>
Date: Thu, 05 Apr 2001 18:28:28 -0400

Mark,

Thanks for the info.  I don't claim to understand all of the
issues with stream commands, although the idempotent issue is
one that I had heard before.  Still, there seems to be no
supporting information that would suggest that this specific
error case is anything other than extremely rare.

What level of complexity should be attempted to recover from
rare errors -- complexity causes its own error cases (as has
been discussed on the list).  And, as you observe, streams
devices must support abort and retry for extreme errors in
any case.

Given that the error is extremely rare, IMO the protocol would
benefit from the removal of StatSN/SNACK/SACK.  If there is
information that shows that this error case is not very rare,
that would be interesting.  If there is no such information,
then I think the horse is well and truely dead :-).

-Jon

Mark Bakke writes:
>Jon-
>
>Part of the reason that the session recovery mechanism was added
>was to support "non-idempotent" devices, such as tape drives,
>media changers, printers (although there are better ways to get
>to printers over a network than iSCSI), and so on.  Disks can
>usually deal with error recovery at the SCSI level, since it's
>not so bad to (carefully) re-try reads and writes.  Multipath
>drivers, such as PowerPath, deal with these over multiple interfaces
>as well.  However, stream commands do not include the concept
>of a block offset, so sending a write twice will write two blocks
>on the tape, rather than just writing the same location twice as
>in a disk.  SCSI tape drivers do not attempt this sort of recovery;
>it has to either be handled by the application (most applications
>just abort the backup, restore, or whatever), or by the transport,
>which is what the use of StatSN was created to provide.
>
>DataSN and S(N)ACK was added later to provide a finer-grained
>recovery when a single command is transferring a large amount of
>data, but I agree that that part is overkill.  Just re-sending the
>whole command with all of its data is good enough.  Nevertheless,
>we still need to keep and acknowledge status using StatSN on at
>least the non-disk devices.
>
>So it's not just a matter of cheap/expensive devices, it's also
>a matter of device type.
>
>Regards,
>
>Mark
>
>Jon Hall wrote:
>> 
>> Julian,
>> 
>> >>a 0.0002%). A restart of link - is expensive (slow start) and even if they
>Mark A. Bakke
>Cisco Systems
>[email-masked]
>763.398.1054

---

From: Robert Snively <[email-masked]>
Date: Fri, 6 Apr 2001 13:17:13 -0700

Michael,

Let me explain why I feel that the iSCSI environment is
different than the college-student music-downloading
environment studied by Stone and Partridge.

All iSCSI targets are new designs.  They do not ride on
obsolete consumer TCP/IP stacks, but rather on proprietary
TCP/IP stacks, hardware assisted TCP/IP stacks, and 
robust embedded Unix TCP/IP stacks.

Most iSCSI initiators are new designs.  While some may ride
on present host TCP/IP stacks, it is likely that most
will be new designs or hosted by robust and up-to-date
TCP/IP stacks.

All storage applications are debugged in an end-to-end
manner impossible for most other applications.  The 
"write/read/compare" and performance testing required of 
storage eliminates almost all the bugs that were contributing
to the higher numbers found in the Stone and Partridge
environment.

That leaves only the residual error rate, which may allow
TCP/IP delivery of segments with undetected errors less
than one time in ten billion, not 1 in 16 million.

The most interesting problem is that most of the additional
verification and the packet CRC is calculated through the
same hardware stack that calculates the TCP/IP checksum and
is therefore susceptible to many of the same errors, but now
blessed by a valid CRC value and valid TCP/IP checksum.

The net I draw from this is that careful design is key to
success and that CRC or positionally 
dependent checksum on the iSCSI data packets is probably 
a good idea.  However, retry of iSCSI data packets may not
be necessary.

Bob 

>  From: Michael Krause [mailto:[email-masked]]
>  Sent: Thursday, April 05, 2001 10:10 AM
>  To: Stephen Bailey
>  Cc: [email-masked]
>  At 09:12 AM 4/3/2001 -0400, Stephen Bailey wrote:
>  > > The Stone and Partridge paper is mostly not applicable 
>  to an iSCSI
>  > > environment.  The principal failure mechanisms were 
>  major software
>  > > bugs in the driver stack of PC-oriented machines.
>  
>  People make mistakes in all implementations.  Examination of 
>  other similar 
>  packet processing technology for mistakes is applicable to 
>  any effort and 
>  one should perform a risk assessment as to the probability 
>  of the mistakes 
>  being repeated here.  The fact that the mistakes were in PC-oriented 
>  machines is basically irrelevant and storage is not immune 
>  from having 
>  similar mistakes (have seen storage implementations that 
>  were just as poor 
>  in terms of quality as any other segment of the industry).
>  
>  
>  >I'm in complete agreement with Bob.
>  >
>  >I haven't seen a good analysis of TCP checksum escapes 
>  which resulted
>  >from intermediary manipulation (I haven't read the papers, but
>  >hopefully soon), but my hunch is that it's incredibly rare.
>  >
>  >An endpoint precipiated TCP checksum `escape' also escape a 
>  CRC or any
>  >other similar integrity check.  That is why I think all this
>  >additional integrity checking (on iSCSI headers & data), is an
>  >incredible amount of extra work (not just in computing the CRCs, but
>  >also in designing the SACK mechanism and recovery for 
>  digest failures)
>  >for no real gain.
>  
>  I agree that some of the recovery is overkill but disagree 
>  that error 
>  detection is as well.  At a minimum, one needs to have a 
>  strong end-to-end 
>  error detection mechanism.  Many believe a 16-bit checksum 
>  is not adequate 
>  to protect their data and given the importance of this data to our 
>  customers, most feel the specification must define such a 
>  mechanism (with 
>  some having strong feelings that this mechanism should NOT be 
>  optional).  Now whether we need to have 2 CRCs, etc. is a 
>  separate debate 
>  but they need to be there and most of us will require that 
>  they be used in 
>  any product / solution delivered to the customer.
>  
>  >The real loss is that it's immensely slowing time-to-market 
>  for iSCSI 
>  >(both in the front end specification and the back end 
>  implementation).
>  
>  A fast TTM solution that is not the highest quality 
>  (prevents silent data 
>  corruption) will lead to customer distrust and a repeat of 
>  the FC adoption 
>  rate - only 10 years later has it really started to 
>  penetrate customer 
>  solutions.
>  
>  
>  >A straw-man proposal (very unpopular given where we are, I 
>  know) would
>  >be to specify iSCSI without additional integrity checks (other than
>  >what you can get through security mechanisms, which is probably not
>  >visible to iSCSI anyway), and if that `fails' (I'm sure it 
>  won't), we
>  >can put an integrity shim between iSCSI and the transport.
>  >
>  >One example of how to do this would be Julian's TAF.  
>  Another would be
>  >the WARP RDMA layer.
>  
>  If another layer is put in place that provides data 
>  integrity, then it is 
>  redundant to do this at the iSCSI layer as well and this is 
>  one place where 
>  an option can be used, i.e. one negotiates the underlying 
>  framing mechanism 
>  (e.g. WARP) and if it is present, then iSCSI does not 
>  activate the CRC 
>  services.  If it is not, then it does thereby insuring that 
>  there is always 
>  end-to-end data integrity present in the solution.
>  
>  
>  >We don't have to specify how to do this now
>  
>  If this is to be supported then it should be specified now 
>  (can be done 
>  rather opaquely by just setting a "transport services" 
>  attribute for strong 
>  end-to-end data integrity protection.
>  
>  >, and the point is that
>  >it's hard to do so, because we really don't know what problem we're
>  >solving with it.  We're OK as long as we have a way to address it in
>  >the future without completely chucking what already exists.
>  >
>  >The other point to remember is that iSCSI still has to make the
>  >ID->Proposed->Draft->Internet traversal, and anybody that 
>  thinks it's
>  >going to do that on the first try is kidding themselves.  It's more
>  >important to get SOMETHING out there that exposes the implementation
>  >holes than to design a cathedral on paper.
>  
>  Nothing is perfect the first time out but in the tightening 
>  economy and 
>  increasing customer quality demands from the get-go, the 
>  trade-off between 
>  quality / reliability and TTM is not something people should rush to 
>  make.  The market is not what it used to be where good 
>  enough was alright; 
>  customers expect more today and with good cause.
>  
>  Mike
>  
>

---

From: "John Hufferd" <[email-masked]>
Date: Fri, 6 Apr 2001 22:44:10 -0700

OK, if you go at it long enough you are punished with my two cents.

We of course need some real numbers on what the probability of the CRC
detected error, when TCP does not detect it.

Given the fact that we do not have that information,  I could only just use
some of the numbers that have been kicking around on this thread.

A Billion is NOT a large number, especially when we are talking about 10
Gigabit Links ( vendors sampling 10 Gigabit/sec HBAs, next year, some
shipping them in general availability (GA) in that year, and the rest in
2003.  And yes I also got information of a company that is currently
developing  100 Gigabit Links.)  So when I looked at some of the numbers, I
found that it meant that a link would see a failure about every twenty
minutes, some went for 200 minutes, etc.

I have one war story that might apply.  Years ago when we were first
thinking about putting the small disk drives in our large storage
controllers, we had folks calculate the Mean Time to Failure (MTF), of the
various Desktop HD.   Some individual MTF numbers sounded large, for any
given drive.  But then we computed the number of drives we would have in a
large installation,  it ended up that we would have a drive failure at
least every day.  (Thankfully, we had significantly better MTF numbers in
the drives that were actually used.)   So, the point is that sometimes
these large numbers come back to bite you in ways you had not considered at
first, when you think about it in a large installation.

OK, back to the thread.

Now I see sites all the time with 10s to 100s of Tape Units, and these
units.  In many cases this will mean that there will be a tape unit failure
that causes the critical  backup job to fail, somewhere on the computing
room floor, about every 2, 20, or 200 min.  This is a major impact on a
computing center that must process hundreds of backup each day.

Therefore, those of you that think you are talking about  very rare events,
should at least compute the 10 Gigabit/second Rates, and then the number of
paths, etc. that might be in an enterprise installation, and then state how
often a computing center will see such an event.  When many of these things
are done at night with unattended operations, these can be a significant
issue.  If it is probable that only one failure will occur per night, then
you are certain that when a disaster does occur,  they will not have a
valid backup over some amount of the data.

OK, I am not saying who's right or wrong here, but just that some of the
numbers I have heard, on this thread are not that impressive when looked at
with a 10 Gigabit/sec links and many paths.  (Let alone the future 100
Gigabit/sec links)  (Oh by the way, remember a 10 Gigabit link is really a
20 Gigabit link when you factor in full duplex.)

So it might be useful, for the Rare Event folks to do the calculations on
their numbers and tell us what they mean in terms of Minutes between
failures on 10 Gigabit links.  Then the rest of us can compute our own
picture on how many links we will probably have in our installation.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Internet address: [email-masked]

---

From: [email-masked]
Date: Tue, 10 Apr 2001 14:34:10 +0300

Steph,

You overlook several well documented papers that were referenced on the
list and the perenial issue of midleboxes.

Julo

P.S. - and my personal experience that is I get a corrupted file about 2-4
times a year and I am far from being a heavy user.

Stephen Bailey <[email-masked]> on 09/04/2001 21:57:19

Please respond to Stephen Bailey <[email-masked]>

To:   [email-masked]
cc:

> Exactly, I've worked in this context (though its been some years now).
> It was true (at one time) that tape had a tractability limit, e.g.,
> a tape backup of a terabyte was out of the question.  Has that changed?

I think this is precisely the point.  Existing, off-the-shelf SCSI
solutions DO NOT presently solve this problem.  Both ||SCSI an FCP
burp the operation on a expectable, O(days) failure rate.  The rate of
adoption for the FCP-2 command recovery feature is overwhelming to the
point that the tape guys have been talking about end-running the
problem with explicitly addressed commands.

What we have running iSCSI on TCP is such a drastic improvement in
what you can expect from your SCSI service that we can eventually
expect a disruptive change.  Trying to engineer it to the point where
its 2^100 times more disruptive, when we don't really know where it's
taking us in the first place is meaningless.

[Warning: repetition ahead]

TCP + link layer error detection is engineered precisely to ensure
reliable data delivery.  It's clear from an engineering stand point
that it is likely (not guaranteed, what is?) to do this quite well.
In spite of much research, it seems like nobody here has come up with
a strong indication that TCP + link layer error detection does NOT do
its job well.  I do not think this is because nobody has ever looked
at the problem.

The lack of concrete information to support the case that TCP + link
layer error detection is inadequate has us chasing our tails.

Given the layer iSCSI occupies in the protocol layer cake, if we don't
try to solve which is presently assigned to a lower layer, it seems
quite comfortable to shim additional checks or recovery, or even a
completely
different transport substrate underneath if we do discover TCP + link
layer error detection is not doing the trick, but it really seems like
folly to engineer based upon an assumption that nobody has done a good
job documenting.

Steph

---

From: Michael Krause <[email-masked]>
Date: Wed, 11 Apr 2001 11:59:33 -0700

>a good idea.  However, retry of iSCSI data packets may not

Agree with CRC being always present and that retry is not needed.

Mike

---

From: "John Hufferd" <[email-masked]>
Date: Thu, 19 Apr 2001 00:47:40 -0700

Jon,
You said:
...I've worked in this context (though its been some years now).
It was true (at one time) that tape had a tractability limit, e.g.,
a tape backup of a terabyte was out of the question.  Has that changed?

Not knowing how far back you go, or what technology you were using, Yes I
guess it has changed, terabytes are a common back-up size, even larger.
The current high-end tapes are really quite good, and their speed continues
to increase.  They are currently tracking Disk technology, maybe one half
to at most one generation behind in head technology, etc.

The disks are getting Larger and Larger, and the Tape technology has had to
track close to disk,  the new LTO tapes continue this track.

Nightly backups of the Terabytes of data used by a Computing Center's
Servers are the rule.  And by the way Tapes continue  increasing the speed.
Of course they are not increasing at 10x, but understand that they are
being written in parallel by backup application such as TSM (aka ADSM),
which write many tapes in parallel.  The Database of the backup
applications and the Tape Libraries have solved a lot of problems that we
use to have to deal with.

.
.
.
John L. Hufferd

"Jon Hall" <[email-masked]>@ece.cmu.edu on 04/09/2001 09:02:10 AM

Sent by:  [email-masked]

To:   [email-masked]
cc:

>OK, if you go at it long enough you are punished with my two cents.

:-)

>minutes, some went for 200 minutes, etc.

This is exactly why its necessary to understand the flow.  In a tape
context is it OK to assume that the data flowing from the target to
the initiator is responses to cmds, and that only a part of that is
iSCSI headers with StatSNs?  If that's right, then run the numbers
against the flow (don't use my numbers, they are riddled with guesses).

>picture on how many links we will probably have in our installation.

But why does the fact that we may someday run at 10 gig change the
question?  Is there some reason to believe that at 10 gig the nature
of a tape flow has changed?  You could certainly have more flows, but
the number of packets per flow will not increase.  The speed of tape
access won't change.  You could do more tapes simultaneously, but
you still have the tractability of handling large numbers of tapes.

As an aside, is a "Rare Event" person, like a flat-earth person?
(I want to get my role right :-).

-Jon