Subject: Re: iSCSI: Markers

From: Paul Koning <[email-masked]>
Date: Tue, 8 Jan 2002 12:13:47 -0500 (EST)

>>>>> "John" == John Hufferd <[email-masked]> writes:

 John> OK, Julian, I buy that argument.  I think that you have made
 John> the point that the value of Markers is Greater then I was
 John> giving credit for, in iSCSI/TOE integrated HBAs that are
 John> operating in CRC mode.  That is great, since I am afraid we
 John> will not see the Framing stuff for some time.

 John> So the questions are which form of markers is the best.

 John> My vote comes down on the side of Fixed Interval Markers
 John> (FIM). For the following reasons:

 John> On the SW sending side, FIM works with low overhead whether CRC
 John> is being used or not.  COWS, is a dramatic overhead unless the
 John> SW is performing CRC computation anyway.

 John> The COWS has a built in conflict with the Pipeline HW HBA, that
 John> has two different needs regarding the direction of the Pointers
 John> when talking to another partner HW HBA that uses a Pipeline
 John> approach.

I think you are making general assumptions about what is cheap in
software that may be valid for SOME software implementations but are
certainly not valid for others.

In particular, FIM is NOT necessarily low overhead.  Scatter/gather is
a reasonably cheap mechanis IF it is available for this job.  (Note
that it is not all that cheap: adding a bunch of extra descriptor
fetches to the I/O device's job is extra overhead, the scale of which
depends on the hardware.

The problem is that scatter/gather may not be available at all.  If it
is available in principle, it may have restrictions that clash with
what FIM needs.  (For example, a particular implementation may support
breaking buffers up in many pieces, but all breaks must be on cache
line boundaries.  For a cache line size of 32, clearly you cannot use
scatter/gather in that implementation to do FIM.)

So FIM may force a full buffer copy, which is a major hit.

As for COWS, it's clearly nasty.  Even if you're doing CRC, it adds a
significant increment to the processing cost.

In conclusion, I am opposed to mandating markers.  Adding such a large
increment in processing overhead doesn't make sense to me, given that
I don't see how it can be expected to save all that much memory in
implementations that use it.  (As far as I can see, the gain doesn't
come anywhere near a factor of two.  Is there any analysis available
that discusses this in any detail -- covering not just the specific
buffering needs of the reassembly side but also the other memory
requirements in a node?  It seems to me a node needs at least as much
memory in the other direction -- quite possibly more if it's the
storage end, given that a lot of the traffic is reads.)

	paul

---

From: Stephen Bailey <[email-masked]>
Date: Wed, 09 Jan 2002 08:34:04 -0500

> significant increment to the processing cost.

Unfortunately, all the choices are nasty.  Long time IETFers would
probably make the point that this is an indication we are trying to do
the wrong thing and should just use a message-based transport :^)

That said, one position that many people thought was the least nasty
was key/length-based TUF (with TCP sender segmentation support), or
nothing at all (== full reassembly).  A smaller number of `key'
participants felt that an intermediary framing solution for stock TCP
senders was necessary.

We proposed COWS-based TUF to try to bring harmony to the spheres, but
I'm not hearing the pleasant resonance (piano tuning is an emperical
process, right?), so perhaps we didn't get it quite right.

Absent any unification of sender segmentation-based and stock TCP
framing (perhaps 0-touch sends ARE too much to give up), I'm strongly
in the sender segmentation-based framing (TUF) or nothing at all camp.

Steph

---

From: "Mark Bradley" <[email-masked]>
Date: Wed, 9 Jan 2002 09:04:25 -0700

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey

> the wrong thing and should just use a message-based transport :^)
   Well, that was the early thinking (pre-pre-BOF and pre-IETF _at all_).
   The thinking at that time was that generic storage over IP was the 
   goal and that SCSI over TCP/IP would be just a proof of concept.
   But, that's just water under the bridge, as things change....  
   --  markb