Subject: RE: iSCSI ERT: data SACK/replay buffer/"semi-transport"

From: Robert Snively <[email-masked]>
Date: Mon, 2 Apr 2001 16:56:10 -0700

The Stone and Partridge paper is mostly not applicable to an iSCSI
environment.  The principal failure mechanisms were major software
bugs in the driver stack of PC-oriented machines.  Such software
bugs would quickly be rung out in any iSCSI host driver and in
any iSCSI target and would certainly not exist in an enterprise
environment.  The 1 in 10 billion range is the more likely
environment.

Bob

>  -----Original Message-----
>  From: Jon Hall [mailto:[email-masked]]
>  Sent: Monday, April 02, 2001 2:49 PM
>  To: [email-masked]
>  Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>  
>  
>  
>  I thought (?) Stone and Partridge concluded somewhere in the
>  range between one packet in 16 million to one packet in 10
>  billion.
>  
>  But, what percent of an iSCSI flow is comprised of iSCSI pdu
>  headers?  Is there a credible flow model for deciding the
>  probability of the one packet containing a StatSN?
>  
>  -Jon
>  
>  Pierre Labat writes:
>  >"Mallikarjun C." wrote:
>  >
>  >> To be fair to data SACK, one could think of an upper bound
>  >> on the unack'ed data - agreed on at the login time.  While not
>  >> requiring acks on every PDU, it gives targets the deterministic
>  >> maximum on the buffer size they have to keep around if they
>  >> choose to "reliably" support data SACK.  The current answer of
>  >> "replay buffer size/IO size", IMHO, is simply not attractive.
>  >> Also to be fair to data SACK, I believe FCP-2 allows 
>  sequence-level
>  >> error recovery in an I/O.
>  >>
>  >> However, I think that it's extremely useful to include a 
>  discussion
>  >> in the draft of  the TCP checksum "escape" statistics and the
>  >> device types for which this was considered an absolute requirement
>  >> to make forward progress at this error rates (like huge tape
>  >> backups?) - essentially the reasons that convinced Julian 
>  to define
>  >> this mechanism in. That gives credibility and acceptance to this,
>  >> or alternately may lead to the consensus that data SACK 
>  is not required.
>  >> --
>  >> Mallikarjun
>  >
>  >About  TCP checksum "escape" statistics what i saw is :
>  >
>  >A) Jonathan Stone,Craig Partridge, "When the CRC and TCP 
>  Checksum Disagree"
>  > http://www.acm.org/sigcomm/sigcomm2000/conf/abstract/9-1.htm
>  >
>  >1 escape in 200 millions
>  >
>  >B) V. Paxson "1999 End-to-End Internet Packet Dynamics."
>  >   http://www.aciri.org/vern/papers.html
>  >
>  > 1 escape in 300 millions
>  >
>  >C) J. Stone et. al "Performance of Checksums and CRC's over 
>  Real Data"
>  >  IEEE/ACM Transactions on Networking, Vol. 6, No. 5, October 1998
>  >
>  >http://dev.acm.org/pubs/articles/journals/ton/1998-6-5/p529-
stone/p529-stone.pdf
>
>Less than 1 escape in 10e17 segments when taking into
>account the link layer AAL5 CRC. (see page 540 left column on top).
>
>
>
>
>Taking the worst is 1 in 200 millions.
>
>Regards,
>
>Pierre

---

From: Stephen Bailey <[email-masked]>
Date: Tue, 03 Apr 2001 09:12:35 -0400

> bugs in the driver stack of PC-oriented machines.

I'm in complete agreement with Bob.

I haven't seen a good analysis of TCP checksum escapes which resulted
from intermediary manipulation (I haven't read the papers, but
hopefully soon), but my hunch is that it's incredibly rare.

An endpoint precipiated TCP checksum `escape' also escape a CRC or any
other similar integrity check.  That is why I think all this
additional integrity checking (on iSCSI headers & data), is an
incredible amount of extra work (not just in computing the CRCs, but
also in designing the SACK mechanism and recovery for digest failures)
for no real gain.  The real loss is that it's immensely slowing
time-to-market for iSCSI (both in the front end specification and the
back end implementation).

A straw-man proposal (very unpopular given where we are, I know) would
be to specify iSCSI without additional integrity checks (other than
what you can get through security mechanisms, which is probably not
visible to iSCSI anyway), and if that `fails' (I'm sure it won't), we
can put an integrity shim between iSCSI and the transport.

One example of how to do this would be Julian's TAF.  Another would be
the WARP RDMA layer.

We don't have to specify how to do this now, and the point is that
it's hard to do so, because we really don't know what problem we're
solving with it.  We're OK as long as we have a way to address it in
the future without completely chucking what already exists.

The other point to remember is that iSCSI still has to make the
ID->Proposed->Draft->Internet traversal, and anybody that thinks it's
going to do that on the first try is kidding themselves.  It's more
important to get SOMETHING out there that exposes the implementation
holes than to design a cathedral on paper.

Steph

---

From: Michael Krause <[email-masked]>
Date: Thu, 05 Apr 2001 10:09:53 -0700

> > bugs in the driver stack of PC-oriented machines.

People make mistakes in all implementations.  Examination of other similar 
packet processing technology for mistakes is applicable to any effort and 
one should perform a risk assessment as to the probability of the mistakes 
being repeated here.  The fact that the mistakes were in PC-oriented 
machines is basically irrelevant and storage is not immune from having 
similar mistakes (have seen storage implementations that were just as poor 
in terms of quality as any other segment of the industry).

>for no real gain.

I agree that some of the recovery is overkill but disagree that error 
detection is as well.  At a minimum, one needs to have a strong end-to-end 
error detection mechanism.  Many believe a 16-bit checksum is not adequate 
to protect their data and given the importance of this data to our 
customers, most feel the specification must define such a mechanism (with 
some having strong feelings that this mechanism should NOT be 
optional).  Now whether we need to have 2 CRCs, etc. is a separate debate 
but they need to be there and most of us will require that they be used in 
any product / solution delivered to the customer.

>The real loss is that it's immensely slowing time-to-market for iSCSI 
>(both in the front end specification and the back end implementation).

A fast TTM solution that is not the highest quality (prevents silent data 
corruption) will lead to customer distrust and a repeat of the FC adoption 
rate - only 10 years later has it really started to penetrate customer 
solutions.

>One example of how to do this would be Julian's TAF.  Another would be

If another layer is put in place that provides data integrity, then it is 
redundant to do this at the iSCSI layer as well and this is one place where 
an option can be used, i.e. one negotiates the underlying framing mechanism 
(e.g. WARP) and if it is present, then iSCSI does not activate the CRC 
services.  If it is not, then it does thereby insuring that there is always 
end-to-end data integrity present in the solution.

>We don't have to specify how to do this now

If this is to be supported then it should be specified now (can be done 
rather opaquely by just setting a "transport services" attribute for strong 
end-to-end data integrity protection.

>, and the point is that

Nothing is perfect the first time out but in the tightening economy and 
increasing customer quality demands from the get-go, the trade-off between 
quality / reliability and TTM is not something people should rush to 
make.  The market is not what it used to be where good enough was alright; 
customers expect more today and with good cause.

Mike