Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"

From: "Jon Hall" <[email-masked]>
Date: Mon, 09 Apr 2001 12:02:10 -0400

"John Hufferd" writes:
>OK, if you go at it long enough you are punished with my two cents.

:-)

>We of course need some real numbers on what the probability of the CRC
>detected error, when TCP does not detect it.
>
>Given the fact that we do not have that information,  I could only just use
>some of the numbers that have been kicking around on this thread.
>
>A Billion is NOT a large number, especially when we are talking about 10
>Gigabit Links ( vendors sampling 10 Gigabit/sec HBAs, next year, some
>shipping them in general availability (GA) in that year, and the rest in
>2003.  And yes I also got information of a company that is currently
>developing  100 Gigabit Links.)  So when I looked at some of the numbers, I
>found that it meant that a link would see a failure about every twenty
>minutes, some went for 200 minutes, etc.

This is exactly why its necessary to understand the flow.  In a tape
context is it OK to assume that the data flowing from the target to
the initiator is responses to cmds, and that only a part of that is
iSCSI headers with StatSNs?  If that's right, then run the numbers
against the flow (don't use my numbers, they are riddled with guesses).

>I have one war story that might apply.  Years ago when we were first
>thinking about putting the small disk drives in our large storage
>controllers, we had folks calculate the Mean Time to Failure (MTF), of the
>various Desktop HD.   Some individual MTF numbers sounded large, for any
>given drive.  But then we computed the number of drives we would have in a
>large installation,  it ended up that we would have a drive failure at
>least every day.  (Thankfully, we had significantly better MTF numbers in
>the drives that were actually used.)   So, the point is that sometimes
>these large numbers come back to bite you in ways you had not considered at
>first, when you think about it in a large installation.
>
>OK, back to the thread.
>
>Now I see sites all the time with 10s to 100s of Tape Units, and these
>units.  In many cases this will mean that there will be a tape unit failure
>that causes the critical  backup job to fail, somewhere on the computing
>room floor, about every 2, 20, or 200 min.  This is a major impact on a
>computing center that must process hundreds of backup each day.

Exactly, I've worked in this context (though its been some years now).
It was true (at one time) that tape had a tractability limit, e.g.,
a tape backup of a terabyte was out of the question.  Has that changed?

>Therefore, those of you that think you are talking about  very rare events,
>should at least compute the 10 Gigabit/second Rates, and then the number of
>paths, etc. that might be in an enterprise installation, and then state how
>often a computing center will see such an event.  When many of these things
>are done at night with unattended operations, these can be a significant
>issue.  If it is probable that only one failure will occur per night, then
>you are certain that when a disaster does occur,  they will not have a
>valid backup over some amount of the data.
>
>OK, I am not saying who's right or wrong here, but just that some of the
>numbers I have heard, on this thread are not that impressive when looked at
>with a 10 Gigabit/sec links and many paths.  (Let alone the future 100
>Gigabit/sec links)  (Oh by the way, remember a 10 Gigabit link is really a
>20 Gigabit link when you factor in full duplex.)
>
>So it might be useful, for the Rare Event folks to do the calculations on
>their numbers and tell us what they mean in terms of Minutes between
>failures on 10 Gigabit links.  Then the rest of us can compute our own
>picture on how many links we will probably have in our installation.

But why does the fact that we may someday run at 10 gig change the
question?  Is there some reason to believe that at 10 gig the nature
of a tape flow has changed?  You could certainly have more flows, but
the number of packets per flow will not increase.  The speed of tape
access won't change.  You could do more tapes simultaneously, but
you still have the tractability of handling large numbers of tapes.

As an aside, is a "Rare Event" person, like a flat-earth person?
(I want to get my role right :-).

-Jon

---

From: Stephen Bailey <[email-masked]>
Date: Mon, 09 Apr 2001 14:57:19 -0400

> a tape backup of a terabyte was out of the question.  Has that changed?

I think this is precisely the point.  Existing, off-the-shelf SCSI
solutions DO NOT presently solve this problem.  Both ||SCSI an FCP
burp the operation on a expectable, O(days) failure rate.  The rate of
adoption for the FCP-2 command recovery feature is overwhelming to the
point that the tape guys have been talking about end-running the
problem with explicitly addressed commands.

What we have running iSCSI on TCP is such a drastic improvement in
what you can expect from your SCSI service that we can eventually
expect a disruptive change.  Trying to engineer it to the point where
its 2^100 times more disruptive, when we don't really know where it's
taking us in the first place is meaningless.

[Warning: repetition ahead]

TCP + link layer error detection is engineered precisely to ensure
reliable data delivery.  It's clear from an engineering stand point
that it is likely (not guaranteed, what is?) to do this quite well.
In spite of much research, it seems like nobody here has come up with
a strong indication that TCP + link layer error detection does NOT do
its job well.  I do not think this is because nobody has ever looked
at the problem.

The lack of concrete information to support the case that TCP + link
layer error detection is inadequate has us chasing our tails.

Given the layer iSCSI occupies in the protocol layer cake, if we don't
try to solve which is presently assigned to a lower layer, it seems
quite comfortable to shim additional checks or recovery, or even a completely
different transport substrate underneath if we do discover TCP + link
layer error detection is not doing the trick, but it really seems like
folly to engineer based upon an assumption that nobody has done a good
job documenting.

Steph

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 9 Apr 2001 14:50:37 -0700

Steph,

I am aware of telco equipment that does not have any error checking between
interfaces.  I would not place a limit on capacity as a means of justifying
full reliance on TCP checksums.

Doug

---

From: "Dave Peterson" <[email-masked]>
Date: Wed, 11 Apr 2001 18:06:34 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey
> Sent: Monday, April 09, 2001 1:57 PM
> To: [email-masked]
> Subject: Re: iSCSI ERT: data SACK/replay buffer/"semi-transport"
>
>

Not true. The reason for the explicit command set is a result of the
difficulties encountered when attempting to perform FC sequence-level
recovery in an out-of-order environment. The command set will also allow a
more robust tape command queuing implementation.

Dave