Subject: RE: iSCSI: Immediate Data

From: "Fischer, Michael" <[email-masked]>
Date: Mon, 1 Oct 2001 11:02:49 -0700

Well, here is my $.02 (overpriced as it may be).  I agree that the
performance of ImmediateData my be indisputable but as I have not seen any
metrics or performed any tests on it myself I cannot say.

I, however, do not think that it is as easy as just saying yes or no to the
ImmediateData command.  If you do not support the defaults, how can you
claim to be a iSCSI device?

Michael Fischer

---

From: "John Hufferd" <[email-masked]>
Date: Mon, 1 Oct 2001 12:51:48 -0700

Mike,
Defaults imply only that they are defaults, not that they are a required by
an iSCSI profile.  If you do not want to support a default, just say NO.

iSCSI conformance is determined by the MUST statements in the protocol
document.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136
Internet address: [email-masked]

"Fischer, Michael" <[email-masked]>@ece.cmu.edu on 10/01/2001
11:02:49 AM

Sent by:  [email-masked]

To:   [email-masked]
cc:
Subject:  RE: iSCSI: Immediate Data

Michael Fischer

---

From: Robert Snively <[email-masked]>
Date: Tue, 2 Oct 2001 10:06:04 -0700

John,

Some minor disagreements here:

> 1. Immediate Data permits the elimination of an additional network
> handshake/interaction.

Immediate Data is not necessary to eliminate the handshake.  Use
Unsolicited Data with InitialR2T = no, and (assuming you have a
decent initiator) you get the same result.

> 
> 2. This is important since some applications have a key sensitivity to
> Latency. The most important of these is Database.

Database is certainly sensitive to latency, but the real
latency is to the SCSI Response, which formalizes the promise
that the data will never, ever, be lost from the storage.
Even the use of InitialR2T = yes on long links is a second-order
performance effect on this if your storage is a disk drive.

> 3. Not only is Database sensitive to Latency, it generally 
> has small I/O
> writes.

This is application dependent.  Most database programs out of the
box tend to use small block transfers, but well-tuned databases
that are doing work other than OLTP tend to cluster writes into
much larger blocks when possible.

> 4. The hardware HBAs that many of you folks are building, 
> include a TCP/IP
> Offload Engines (TOEs).  This permits support of Gigabit Line 
> speed without
> Server Load.  However, even though many of you are attempting to
> parallelize as much of the TOE processing as possible, TCP/IP 
> processing
> will still add latency to each iSCSI interaction, as compared to Fibre
> Channel.

What can I say?  You have hit a basic truth.

....  snip   ....

> 
> 7. So it is the Immediate Data feature of iSCSI that will 
> make an important
> difference in the Key Response Time Sensitive Applications, 
> which by luck
> only transfer a small amounts of data at a time.

See 1-3.

> 8. When we attempt to let iSCSI shine on the "at-distance" storage
> environment, the value of Immediate Data, and Unsolicited 
> Data are even
> more valuable.  However, my concern at this time is for 
> Immediate Data.

I am absolutely in agreement with your assessment of unsolicited
data at distance, but immediate data is not a necessary 
part of that improvement.

As for simplicity:

> A. Creating a Buffer Manager that can allocate buffers that 
> are chained
> together, is kind of fundamental to the high performance 
> environment we are
> attempting to work with.  All the processes (whether iSCSI or 
> SCSI) will
> allocate buffers (from the Buffer Manager), place data in one 
> or more of
> these buffers, and pass the pointer list to the next process.  
> There will be
> no copying of data.

See my previous note considering the actual destination of
data, which is not in a memory address space.  It is in this
way that storage differs from networking.  You can't pass
pointers about, but must place the data into a storage
space that has the properties of non-volatility, redundancy,
coherence, and accessibility through the SCSI command set.

...   snip   ...

---

From: "John Hufferd" <[email-masked]>
Date: Tue, 2 Oct 2001 11:14:16 -0700

Rober, Comment in line.

.
.
.
John L. Hufferd

Robert Snively <[email-masked]> on 10/02/2001 10:06:04 AM

To:   John Hufferd/San Jose/IBM@IBMUS, [email-masked]
cc:
Subject:  RE: iSCSI: Immediate Data

John,

Immediate Data is not necessary to eliminate the handshake.  Use
Unsolicited Data with InitialR2T = no, and (assuming you have a
decent initiator) you get the same result.
[Huff]
The point is that Immediate Data is simpler then the above since it is with
the CDB
[/Huff]

[Huff]Now you are firmly on my turf.  Almost all Raid Controllers, have
Caches with Non Volatile Storage.  (Or at least all the ones to which
Database response is important have Non Volatile RAM for Write Caching.)
The Non Volatile Write to Cache is what takes the Disk Latency out of the
equation.  So with immediate data the response to small writes can be very
fast.
[/Huff]

This is application dependent.  Most database programs out of the
box tend to use small block transfers, but well-tuned databases
that are doing work other than OLTP tend to cluster writes into
much larger blocks when possible.
[Huff]
Most of the I/O operations are still small.  It is from these that we need
to remove as much latency as possible.
We do not need to do anything special for the Large I/O.
[/Huff]

....  snip   ....

> only transfer a small amounts of data at a time.

See 1-3.
[Huff] and my responses.
[/Huff]

[Huff] I can not understand why you dislike immediate data, and still seem
to like unsolicited data. You will have almost all the same iSCSI code path
in the node anyway if you handle Login.  Why you would not like a code path
that makes this important reduction, seems a bit strange.
[/Huff]

[Huff] I have no idea why you say this.  Our storage controllers and our
competition has been doing this for years.  The Non Volatile (RAM) Memory
is completely addressable.  And yes the Cache Manager looks a lot like all
buffer managers, and there are a lot of pointers.
[/Huff]

...   snip   ...