Subject: Re: iSCSI: No Framing

From: "Julian Satran" <[email-masked]>
Date: Tue, 5 Feb 2002 03:09:39 +0200

Mallikarjun,

I doubt that FIM (or COWS) will fracture the market.
Hardware and software vendors will gain experience in what it takes to use
framing.
a specialized DDP and that could be useful later.  The first generation
although not imperiously needing any framing (I have proposed not less than
3 solutions!) will enable us to get a better second generation if we do
something in this area.

Julo

                    "Mallikarjun                                                             
                    C."                  To:     <[email-masked]>                           
                    <[email-masked]       cc:                                                 
                    om>                  Subject:     Re: iSCSI: No Framing                  
                    Sent by:                                                                 
                    owner-ips@ece.                                                           
                    cmu.edu                                                                  

                    04-02-02 23:34                                                           

I share the concern about iSCSI embracing a framing mechanism that is
not a MUST implement.  For all the reasons that Jim pointed out, OTOH,
I am not recommending a MUST framing either.  I suspect not many will
implement framing if it's a MAY, so it appears that we're talking about
a potential SHOULD.

Given that SHOULD is a fairly strong requirement, one "significant
justification"
for not doing framing could be (even while the NIC *may be* on the
expensive side) -
 - has less design complexity since no OOO placement.
 - can have quicker TTM since less design, testing, debugging etc.

I think ultimately it boils down to: how many vendors would use a
"significant
justfication" to not implement a SHOULD-requirement?

If that's a majority: let's say vendor X is implementing (whatever) framing
for
optimizing the memory requirements, it essentially means that X's product
will
perform poorly with (the majority) no-framing senders.  I don't think X
would
like that, nor the customer.  IOW, this situation doesn't seem to be
long-lasting.
OTOH, the situation of an almost equal number of  "framing" and
"no-framing"
products in the market (perhaps at different price points) could be
unfortunately
long-lasting....

To summarize, it is a troubling prospect that a framing technique (if
adopted as
SHOULD) has the potential to somewhat fracture the market and in effect
create "interoperability problems" (of performance sort) similar to that
affect
FC....
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
Hewlett-Packard MS 5668
Roseville CA 95747

>
> -----Original Message-----
> From: WENDT,JIM (HP-Roseville,ex1) [mailto:[email-masked]]
> Sent: Tuesday, January 29, 2002 10:47 PM
> To: [email-masked]
> Subject: iSCSI: No Framing
>
>
>
> So, it would be good to hear from several iSCSI
> NIC/chip implementors who:
> - have or plan to implement FIM or COWS (or some
> other framing mechanism) and take advantage of it to
> minimize demands on on-NIC buffer memory
> bandwidth/quantity.
> - believe that all-buffers-on-chip solutions are
> feasible and valid (wrt the points above, including
> #2)
> - can quantify the memory/pin/power/space cost
> savings for all-buffers-on-chip solutions
>
> Jim
>

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 5 Feb 2002 00:42:18 -0800

Julian,

Implementing a scheme for doing direct data placement by means of an
application specific, undocumented or proprietary layer below TCP will
fracture the market.  This will result in an unacceptable outcome having
hardware created for each application, limited by various vendors making as
yet unknown claims, if implementing the DDP feature using TCP.  In the
future, should this feature become desired, iSCSI can adopt a generic method
that is possible through the use of SCTP.  An eventual adoption of SCTP
would also simplify much of the existing complexity found with the
multi-connection TCP standard now proposed for iSCSI.  Use of SCTP would
allow common hardware for many protocols desiring Direct Data Placement
without modification of the SCTP transport as it can deliver objects
unordered.  A shim providing the DDP feature could ensure objects are
disclosed in the order sent if desired, independent of the reception at the
shim while not modifying SCTP or adding layers beneath this transport as
would be required for TCP.

Only when constrained to conventional memory allocation, does DDP become
beneficial.  The concept of placing a layer beneath TCP that structures an
array to hold each individual packet based on application specific
structures is fundamentally flawed.  This presumes the layer beneath TCP
knows before hand the content of the byte stream prior to TCP composing the
stream.  This lower layer will also need to create exceptions for handling
duplicates, for placement information not aligned with the TCP segment, and
for application informational errors.  SCTP can easily avoid these
exceptional cases making such development less disruptive.  In addition, an
SCTP stack that does not offer DDP in hardware need not change the API
behavior to the application nor modify any layer functionality.

The ability of SCTP to support many different higher level protocols comes
from the capability of SCTP to deliver objects unordered.  TCP's restriction
on ordered delivery mandates that any direct data placement is done prior to
TCP through the addition of a new and application unique layer between IP
and TCP.  The saving caveat has been that disclosure to TCP of these packets
are sequential but the application interface to TCP will change as a result
of this modification in some undefined way to associate the steering
information with the desired buffers.  This change impacts the interface
between this new layer, as well as the application and TCP.  The desire to
keep this DDP scheme for TCP proprietary will ensure a fractured market in
terms of hardware, OS services, and perhaps interchange not to mention that
each application then also needs unique hardware.  The security risks
involved in creating this new and highly complex layer below TCP is yet
unknown as details are lacking.  In essence, an attempt to implement DDP by
creating a new layer beneath TCP in a proprietary fashion is in direct
competition with the far better practice of using SCTP to implement DDP.

Doug