Subject: RE: iSCSI: No Framing

From: "John Hufferd" <[email-masked]>
Date: Tue, 5 Feb 2002 10:55:18 -0800

Somesh,
I usually give higher weight to the opinions of folks that take their
opinions and put their money on it, than the folks that do not.   So I am
not sure you can so easily dismiss the ideas of folks that are committed
with actual money.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Somesh Gupta" <[email-masked]> on 02/05/2002 10:25:45
AM

Please respond to <[email-masked]>

To:    "Jim Pinkerton" <[email-masked]>, John Hufferd/San
       Jose/IBM@IBMUS
cc:    "Mukund, Shridhar" <[email-masked]>, <[email-masked]>
Subject:    RE: iSCSI: No Framing

Jim,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Jim Pinkerton
> Sent: Tuesday, February 05, 2002 6:53 AM
> To: John Hufferd; [email-masked]
> Cc: Mukund, Shridhar; [email-masked]
> Subject: RE: iSCSI: No Framing
>
>
>
> In reviewing the mail alias, and after having many conversations with
> various folks, I would like to retract my prior opinion on ripping out
> markers. It appears to me as though _every_ NIC vendor is in support of
> markers.

  Every NIC vendor may implement markers (or as is obvious from
  interoperability results, may have the capability to implement
  markers). On whether every (person who works at a) NIC/Chip vendor
  supports it from a technical perspective (i.e. whether it solves
  anything, or is a useful solution, or whether the analysis people
  are using is correct) supports it, I think the count seems to be
  evenly split.
>
> To me the evidence in support of keeping in "Sync and Steering with
> Fixed Interval Markers" is pretty compelling (now that I've been
> re-educated). Without going into proprietary issues - how often do NIC
> vendors agree unanimously on something? Also, a software implementation
> of markers is tractable.
>
> I have not heard a single NIC vendor in support of COWS, and I
> personally don't support COWS either. Thus I would recommend that COWS
> be "ripped out".
>
>
> Jim
>

  Julian indicates in his message that we are not really sure what
  a good solution is and if we put something in, we are likely
  to get to a better one in the second draft. I also concur that
  the whole problem/solution space is experimental in nature, and
  ought to be treated as such.

  I think it is meaningless to assume that one person or another
  has a magic solution here. The problems are fairly well understood.
  One can talk about it from an algorithmic perspective (like
  error recovery) which will expose some of the assumptions/compromises
  behind this. Of course, whether that will pass muster with
  the community at large is another issue.
>
>
>
> -----Original Message-----
> From: John Hufferd [mailto:[email-masked]]
> Sent: Friday, February 01, 2002 11:31 PM
> To: [email-masked]
> Cc: Mukund, Shridhar; [email-masked]
> Subject: RE: iSCSI: No Framing
>
>
> Somesh,
> This is not fair, we asked for Chip Vendors that were planning on using
> it,
> to make themselves known and to tell us why.  Without exposing internal
> designs, he has tried to do that, likewise Trebia, and some others have
> said they want FIMs.  You may not want FIMs for your design, but you
> should
> respect these folks that have been following our spec and found it to be
> valuable in their designs.  They are putting their money on it not just
> talk, so we should at least give them the credit which their position
> deserves.  Yes, of course they are disappointed with the rhetoric around
> this issue, since they and others have been following the spec, and can
> not
> understand why others haven't, since they have determined it to be of
> significant value.
>
> His note does speak to the issues. (And clearly has some silly parts :-)
>
> In fact he has found it of enough value to use now, since he believes
> they
> can not wait for iWARP to happen before the issue is addressed (and he
> really wants iWARP).  It should be useful to listen to folks that are
> willing to put their money where their mouth is.
>
> OK, that said: don't you think we can find a way to address both your
> position and their position, since they both seem strong and heart felt.
> I
> would think that the "Should Implement" or at least "May Implement"
> would
> give both sides enough reasonable room to meet on an even business
> plane.
>
>
> .
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: [email-masked]
>
>
> "Somesh Gupta" <[email-masked]>@ece.cmu.edu on
> 02/01/2002 07:10:13 PM
>
> Please respond to <[email-masked]>
>
> Sent by:    [email-masked]
>
>
> To:    "Mukund, Shridhar" <[email-masked]>, "'WENDT,JIM
>        \(HP-Roseville,ex1\)'" <[email-masked]>, <[email-masked]>
> cc:
> Subject:    RE: iSCSI: No Framing
>
>
>
> Mukund,
>
> With all due respect, taking on the mantle of
> the noble engineer being denied by politically
> motivated people, belittles the technical expertise
> and strengths of the people who have expressed an
> opinion against having any of the proposed schemes
> at this time in the spec.
>
> I would be very happy to work alongside a number of
> the people on both sides of the issue (and that does
> include you).
>
> None of your arguments below refutes the technical
> and some non-technical opinions expressed against
> FIM.
>
> Somesh
>
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Mukund, Shridhar
> > Sent: Friday, February 01, 2002 5:24 PM
> > To: 'WENDT,JIM (HP-Roseville,ex1)'; [email-masked]
> > Subject: RE: iSCSI: No Framing
> >
> >
> > Dear Colleagues,
> >
> > PART 1 of 3 :
> >
> > The pertinent points that "I" see in Jim's email are:
> >
> > A. Need to hear from iSCSI chip implementers ...
> >    The issues that you raise for e.g. in #2&#4 are simply
> >    circumstantial( See PART 2 ). Answers to those questions
> >    unnecessarily call for data flows and implementation
> >    details that silicon vendors are not allowed to share
> >    in public. While I do not know of many silicon vendors
> >    in the multi gig space, clearly one of the competitions
> >    I respect, namely Trebia, point to FIM as well. Granted,
> >    no matter what, we are going to see several
> >    trebia-on-the-other-end and adaptec-on-the-other-end
> >    type of cost optimizations.
> >
> > B. The iWARP/TUF/SCTP under-current ...
> >    The "only" message of significance in Jim's email is #5.
> >    It seems like the iWARP/SCTP/TUF enthusiasts somehow feel
> >    threatened that FIM-iSCSI will dilute the perceived value
> >    proposition of iWARP :-) I for one am an enthusiast of
> >    iWARP ideals myself, barring the proposed mechanics. I
> >    would love to make a buck or two along with you in
> >    building iWARP NICs, "at the right time". In the
> >    meanwhile, iSCSI is the flag ship effort that has the
> >    unique opportunity to make a dent in enabling IP Storage
> >    visions. ( See PART 3 )
> >
> > My assertions are :
> >     i) TUF/SCTP/iWARP is ruled out in the near term. The
> >        mechanics are unclear as hell.
> >     ii) FIM is the least intrusive, TCP transparent, means for
> >        enabling low-cost(power,b/w,latency,memory,space,CPU)
> >        RDMA transport of bulk data.
> >     iii) I do not see significant technical reasons that
> >        merit major changes to the iSCSI draft at this late
> >        date.
> >     iii) Making it OPTIONAL to use, and SHOULD only on send
> >        provides a graceful and incremental deployment path
> >        for "real":-) providers and users to succeed.
> >
> > I have personally contributed nothing to the iSCSI effort
> > and do applaud the pains that several of you are taking to
> > pull it all together. In that very same spirit, I respect
> > David's decision w.r.t. consensus, whatever that turns out
> > to be.
> >
> > Thanks.
> >
> > -Shridhar Mukund + Todd Sperry, Siva Munnangi, Dev ...
> >
> > ------------------------------------------------------------
> > PART 2 of 3 : MUST delete, SHOULD read
> >
> > Dear Jim,
> >
> > Congratulations Jim! Seems like when you bowl, pins
> > roll, unintentional as they may be. You make a "seemingly"
> > well balanced set of arguments and "tactfully" tilt the
> > balance towards your chosen side. I would love to be on
> > your side ... maybe in another effort :-)
> >
> > I would like to introduce you to my respectable friend,
> > who "vehemently" contradicts you on all accounts. One of
> > his numerous quotes goes as follows:
> > http://groups.yahoo.com/group/rdma/message/486
> > "Much of today's usage of the Internet and IP networks
> > is for buffer-to-buffer data transfers, often in the
> > form of bulk data ... Gigabit and faster network transfers
> > incur 'heavy' system resource costs, including both CPU
> > use and system bus bandwidth, particularly on the
> > receiving side ... The costs incurred on hosts for protocol
> > processing and management has 'inhibited' the use of IP
> > in the high speed bulk data domain. ... Bulk data transfer
> > is dominated by the costs of copying and validating
> > incoming data in order to place it at its ultimate
> > destination."
> >
> > The good friend I quote above is none other than Jim
> > Wendt himself!!! I REST MY CASE.
> >
> > Thanks.
> >
> > -Shridhar Mukund
> >
> > ----------------------------------------------------------------
> > PART 3 of 3: MUST delete, OPTIONAL read
> >
> > On the lighter side ... since the opponents have "no" technical
> > arguments whatsoever against FIM and it is all turning out to
> > be a pure political gimmick, I will put on my rusting tin
> > politician hat :-)
> >
> > My dear fellow iSCSI country (wo)men: If our goals are anything
> > short of enabling the HP's, Cisco's, uSoft's, IBM's and EMC's of
> > the world in globalization of storage for McDonald's and Macy's
> > from Kabul to Somalia, via iSCSI, we have lost our identity!
> > ( \insert APPLAUSE for 13 seconds )We are no more protected by
> > the vast oceans between us and other Storage efforts. The
> > freedom of iSCSI America is threatened from within by elements
> > who will not blink twice when it comes to using the world's most
> > potent BOO-bombs ... and grinnn while we end up sifting through
> > the rubbles for iSCSI, youSCSI and theySCSI.
> > ( \insert APPLAUSE for 11 seconds ) Beware of the elements that
> > sleep with "our" very ideals in their privacy(burkha clad
> > though) and yet attempt to destabilize us, only to accomplish
> > their mutated agenda.( \insert APPLAUSE for 57 seconds )
> >
> > No offense folks. I respect each and every one of you and
> > especially Jim. I think that he was only attempting to
> > question, "are we sure ... should we commit ...".
> > I disagree with him anyway!
> >
> > - the running(actually limping) mate :-) :-) :-)
> >
> > -----Original Message-----
> > From: WENDT,JIM (HP-Roseville,ex1) [mailto:[email-masked]]
> > Sent: Tuesday, January 29, 2002 10:47 PM
> > To: [email-masked]
> > Subject: iSCSI: No Framing
> >
> >
> > Perhaps we should discuss the possibility of not
> > specifying any framing mechanism (FIM or COWS) in the
> > first version of iSCSI.
> >
> > The arguments for not specifying framing include
> > (there may be others as well):
> > 1) The brute force approach of putting TCP receiver
> > reassembly memory on the iSCSI NIC will work for both
> > 1Gbps and 10Gbps. Cost is incurred for memory chips,
> > ASIC pins, power, and board space. But, it is a
> > feasible approach.
> >
> > 2) I/O bus latency (or bandwidth limitations) could
> > mandate inbound buffering (as a 'smoothing' buffer)
> > from the iSCSI NIC to the host system. If this buffer
> > memory is large enough to have to be off-chip, then
> > it will require some minimum number of memory chips
> > to provide the necessary bandwidth, and the
> > memory/pins/power/space costs will be incurred
> > anyway.
> >
> > 3) Very large receive buffering on the NIC is only
> > required for high-bandwidth links traversing large
> > geographic distances (which may not be the
> > predominate case). These specialized implementations
> > can cost more (e.g. more memory/power/pins/etc) and
> > customers will likely be willing to pay accordingly.
> >
> > 4) Many NICs will likely aspire to be combo
> > iSCSI+TOE+Ethernet NICs allowing the host system to
> > use a single Ethernet port for all of its network
> > communications. The TOE function on this combo NIC
> > will already require TCP receive buffering and off-
> > chip buffer memory to support the existing sockets
> > interface receive model.  More importantly, to allow
> > senders to assume ownership of the buffer upon return
> > from a socket send call, the TOE NIC would need to
> > copy application send buffers into NIC memory as
> > well.
> >
> > 5) The framing/marker mechanism will be an iSCSI-only
> > solution. It is quite possible that the problem will
> > be solved via a different, and hopefully more
> > general, mechanism for other ULPs.
> >
> > 6) Both FIM and COWS are poor solutions for software
> > implementation. COWS requires, at a minimum, the
> > sender to read every word in the buffer. FIM either
> > imposes additional sender gather processing (iovecs)
> > or requires a data buffer copy (on systems that don't
> > support HW gather on sends).
> >
> > 7) Unless all senders thoroughly test framing/markers
> > now (i.e. unless the framing method is a MUST
> > implement), there is potential for future
> > interoperability problems as framing/marker receivers
> > are deployed in the future.
> >
> > 8) Neither FIM nor COWS is an elegant solution. Are
> > we comfortable with the legacy we are creating under
> > the umbrella of state-of-the-art IP networked
> > storage?
> >
> > 9) Is it essential to build in forward compatibility
> > now, or will there be a different solution in the
> > 10Gbps or 40Gbps timeframe - perhaps involving iSCSI-
> > 2?  Will it be reasonable to update or bridge initial
> > 1Gbps deployments?
> >
> >
> > So, it would be good to hear from several iSCSI
> > NIC/chip implementors who:
> > - have or plan to implement FIM or COWS (or some
> > other framing mechanism) and take advantage of it to
> > minimize demands on on-NIC buffer memory
> > bandwidth/quantity.
> > - believe that all-buffers-on-chip solutions are
> > feasible and valid (wrt the points above, including
> > #2)
> > - can quantify the memory/pin/power/space cost
> > savings for all-buffers-on-chip solutions
> >
> > Jim
> >
>
>
>
>

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 5 Feb 2002 11:20:34 -0800

John,

I have to point for the record, that I was the one
that proposed markers over a year ago (not that many
other did not also have this idea). I have also tried
to keep this debate at a technical level. There
are reasonable folks on both sides of the debate,
and we should just continue in that spirit.

Somesh

> Sent: Tuesday, February 05, 2002 10:55 AM
> Cc: Jim Pinkerton; Mukund, Shridhar; [email-masked]

---

From: "Joe Gervais" <[email-masked]>
Date: Tue, 5 Feb 2002 17:55:41 -0800

It seems that markers/framing is an issue open for debate for some time to
come. In my mind that indicates that this is still experimental in nature
and should be moved to an experimental RFC, outside of the scope of the
iSCSI RFC. It seems that this is trying to optimize the case of a high
bandwidth, long haul line. Certainly, iSCSI enables that, but also enables a
much broader market for IP Storage. As Jim Wendt originally pointed out at
the start of the thread, there are solutions today. Maybe these solutions
are not as elegant as one would like to architect, but they can be made to
work with Gigabit Ethernet and 10 Gigabit Ethernet.

I am in favor of moving framing out to an experimental draft, and finishing
this standard. Contrary to his opinion, Jim Pinkerton did not hear all
TOE/NIC vendors in favor of FIM.

Joe Gervais
Alacritech

-----Original Message-----
Somesh Gupta
John,

Somesh