Subject: Re: TCP Framing

From: "John Hufferd" <[email-masked]>
Date: Mon, 21 May 2001 01:07:09 -0700

I must agree with Matt.  And in addition to what Matt said, about the 1 Gig
Software,  we must remember that many Desktops and Laptops, will be able to
operate with CAT 5 Cable at 100 Mb/s, and some will be happy with that,
others using the same CAT 5 cable will be able to operate comfortably at
300+ Mb/s (there will not be many that will be comfortable at 1000 Mb/s,
but most will be very happy with 300 MB/S).  So what you will see across a
business campus is software implementations running on Desktops and Laptops
at an assortments of speeds, from 100 Mb/s, through 300+ Mb/s.  There of
course will be Hardware HBAs in some Desktops, running on the same Networks
(at Gigabit Speeds), but you should expect to see lots of SW
implementations running at these lesser speeds.  To be really successful
and fulfill the real promise of iSCSI, we want both the HW and the SW
implementations to function well.

With the 100 Mb/s to 300+ Mb/s many of these Desktops and Laptops will be
getting better performance from the Network then from their local Hard
Disk.  They will have more then enough CPU power to support 300+ Mb/s and
not be noticed in the individual's system.

As Matt was saying these different, systems will get multiplexed together
into the target device, and it is important that the target device be able
to support both HW and Software implementations with a minimum of cost, on
the targets 10 Gigabit iSCSI Portals.  Though I like the Warp support, I
think it is unlikely that the TCP/IP stacks of all the different Desktops
and Laptops, etc. will be updated to support that,  in any acceptable time
frame.

I think we must depend on Markers to insure that everything can operate at
top speed, and at the lowest cost.

That is why I think that the Markers should be "Must Implement", but
"Optional to use".

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
(408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Internet address: [email-masked]

"Matt Wakeley" <[email-masked]>@ece.cmu.edu on 05/20/2001
09:40:39 PM

Please respond to Matt Wakeley <[email-masked]>

Sent by:  [email-masked]

To:   [email-masked]
cc:
Subject:  Re: TCP Framing

"KRUEGER,MARJORIE (HP-Roseville,ex1)" wrote:

> Isn't it actually that a 10Gig implementation couldn't interoperate *at
> 10Gig speeds* without some sort of frame-sync method - they would be
> considerably "slowed-down" (or forced to have large TCP rcv buffers).
This
> doesn't mean 'no interoperation', just no 10 Gig thruput?

Marj,

The issue is, there will be lots of 1Gig applications out there - perhaps
one
on every desktop.  All these 1Gig applications will be aggregated through
switches to 10Gig servers.

Now, suppose you are the vendor of this server.  Are you going to say in
your
product specification that your product will only interoperate with
applications running new TCP stacks with this new framing solution?  I'll
bet
not.  So, unless there is a framing solution mandated by the iSCSI spec
(under
the control of the iSCSI spec), you will have to implement the full TCP
buffering solution, and will have lost all the cost and space benefits of a
framing solution.

To address your other comments:

> The reason people are working so hard on a TCP-level framing solution is:
>
> - marker scheme is so krufty and software soln's are penalized (they
> probably wouldn't be able to drive 10Gig speeds if 'markers' are
mandated)

10Gig solutions will not be implemented in software, markers or not.  But
due
to aggregation, the 10Gig solutions will require the 1Gig solutions to
implement framing (markers).

> - there are other TCP applications that benefit from a common frame-sync
> method.  This makes these changes more likely to be incorporated into
> software stacks (OS releases).

Yeah, but they will require further work, and will probably be adopted
after
the new frame-sync method.  And I still question how you can require a
optional enhancement to TCP to make your new protocol work.

> There's already grumbling in IETF that 'that IPS team' "seems to exhibit
a
> strong tendency to avoid
> reusing existing IETF standards that reasonably fit their needs".  It
would
> benefit us to support the common TCP framing solution and contribute to
it's
> early adoption.

They are only grumbling because we didn't embrace their new transport with
open arms.

> Even if the TCP framing RFC wasn't mandated by iSCSI, a vendor could
require
> it's implementation end-to-end to guarantee 10Gig speeds?

See my comments above. Do you want to be the vendor that comes out with a
solution that is more restrictive than your competition? If the framing
solution is part of the iSCSI standard, then everyone will have to
implement
it to claim iSCSI compliancy.

-Matt

---

From: Stephen Bailey <[email-masked]>
Date: Mon, 21 May 2001 09:50:41 -0400

John,

> top speed, and at the lowest cost.

A key question is whether markers actually ensure that everything
operates at `top speed, and at the lowest cost'.

Matt thinks so.  I (and, presumably those who wrote the framing
document) think not.

My issue is not even with `lowest cost'.  I don't believe markers will
allow you to run at top speed.  Specifically:
  1) I doubt the feasibility of implementing the control required for
     an eddy buffer (where you store data you can't place) at 10G.
     Admittedly, the validity of this claim can't really be assessed
     without actually working the implementation, so for 99% of the
     list participants (myself included) this is a `yes it is, no it
     isn't' point.
  2) an eddy buffer solution requires some substantial speed-up in
     both the NIC data path, and MOST IMPORTANTLY: the host bus.  In
     order to unload the eddy buffer while still handling incoming
     traffic at line rate, clearly the host bus bandwidth must be >
     line rate.  

I know of at least one general purpose framed solution operating at
10G which has been available for >3 years (SGI's GSN/ST/XIO NIC).  I'm
sure there are others.

I can't imagine there's any argument that a framed solution would be
voted `most likely to run fast and be cheap'.  Every storage network
and cluster interconnect has been designed that way since antiquity.

The key tradeoff involves the OS vendors, and I'm wondering why we're
speaking for them.  The question IS, how much more work is it to
introduce TCP framing over and above what is required to insert iSCSI
into their network framework.  My experience from writing NIC and
storage drivers for many commercial UNIX-family OSes is:
  1) it's an easy and well defined process to insert a new SCSI
     transport driver into the SCSI stack.
  2) it's hard and poorly defined process to insert ANYTHING into the
     network stack.

Networking has historically been a user-mode activity.  Architected
services are only provided to user mode programs.  Kernel clients have
been few and far between and so are handled on a case-by-case basis.
For example NFS.  Every OS has hacks to make NFS run fast, but they
are not stable interfaces for general purpose use.

Even Solaris' SysV-derived STREAMS stack, which is intended precisely
to provide flexible, crisp interfaces for kernel network clients, does
not document the relevant (IP stack) intermodule interfaces.

I know that there are more and more kernel network clients, but they
are coming either on fluid platforms (e.g. linux), in which case the
argument of `it'll take too long to get OS support' doesn't apply, or
they are vendor-supplied, in which case a performance iSCSI solution
in ANY form may take a while, and the choice of framing or markers
isn't going to make a difference.

I can't say squat about the architecture of Winsock, but the fact that
there is a Microsoft author of the framing proposal who seems very
serious about supporting framing and RDMA as quickly as possible
suggests that framing support should be available on Windows very
soon.

Steph

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 22 May 2001 11:34:40 -0700

I think if the framing solution finds favor with the TCP "gods",
it is a much better alternative than the markers (which I was
a proponent of - but only because I did not believe that
a framing mechanism would be approved).

Even if the framing mechanism is not approved, an alternate
mechanism which provides for alignment of iSCSI headers at
regular intervals (at negotiate progression in TCP sequence
#s e.g. 0x12345, 0x22345, 0x32345 and so on) in the TCP stream
is a much better solution. Unfortunately this would require
some change to the header format (for padding) which probably
no one would like to change at this time.

The assumption that people will implement some change as big
as iSCSI will be to the OS (in an integrated way - not as
a kludgy add-on for demos) without being able to make the
relatively minor change required to the API seems to be not
a very convincing argument.

Furthermore, at least in the Unix environment, iSCSI will be
implemented in the kernel, and the accomodation for framing
won't require an external API change. And as Steph says,
the framing proposal has Microsoft participation.

The insertion and deletion of markers seems to be painful
accomodation for software based iSCSI solutions. The Unix
folks might want to speak to this, but the markers will
probably require more of a kludge in the kernel (to make
it efficient) as compared to the changes required for framing.

Somesh

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey
> Sent: Monday, May 21, 2001 6:51 AM
> To: [email-masked]
> Subject: Re: TCP Framing (considered helpful?)
> 
> 
> John,
> 

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---

From: Mike Brown <[email-masked]>
Date: Tue, 22 May 2001 20:40:31 -0400 (EDT)

Hi all,

I am against using a tcp framing scheme for reasons previously stated by
Matt Wakely and John Hufferd.

I believe the assertion that iSCSI will be implmented in the kernel in the
UNIX environment is too simplistic a view.  I think robust implementations
will likely consist of some userspace/kernel hybrid.  (I am of the
opinion security associations using strong cryptography should not be
implemented in the kernel).  While this could also avoid an external API
change, requiring os vendors to change their existing TCP/IP stacks might
be too idealistic a view.

The insertion and deletion of markers from an implmentation prespective
would not be too painful an accomodation for software based solutions.
In fact, they would be simpler to implement since existing TCP/IP
stacks could be used.  John Hufferd's idea that "Markers should be 'Must
Implement', but 'Optional to use'" would enable software solutions to
ignore using them and still be efficient if their size is kept small
enough.

I am of the opinion that a new standard's probability of universal
acceptance is inversely proportional to the number of changes it requires
in other standards and existing implementations.  

>> suggests that framing support should be available on Windows very

-Michael F. Brown, UMass Lowell Computer Science

phone:  (978) 934-5354
email:  [email-masked]

"In theory, there is no difference between theory and practice,
 but in practice, there is."       - Jan L.A. van de Snepscheut