Subject: RE: iSCSI: Framing Steps

From: "Mukund, Shridhar" <[email-masked]>
Date: Mon, 28 Jan 2002 18:42:54 -0800

Dear colleagues,

1. Doug : David made the point, "it(FIM) does not modify 
   the TCP protocol", as lucid as it gets. It will be nice 
   to hear some opinions from the TCP gurus out there
   without mixing it up with SCTP and TUF.

2. It will also be nice if other iSCSI folks on the reflector 
   cast their votes per John's request.
   Putting on my politician hat :-), pl. vote for option 6.
   >>>> 6. FIM now, and some kind of Framing later

3. I do agree with David that the implication, "Senders that 
   do not insert markers on send should be willing to accept 
   up to RTT*BW drops" in my email was too strong as it goes 
   against a RFC1122-SHOULD. ( See reference below ). I
   stand corrected.

   The current iSCSI draft addresses my point anyway:
   "In certain environments, a sender not willing to supply 
   markers to a receiver willing to accept markers MAY 
   suffer from a considerable performance degradation."

   In my interpretation, the spirit behind the RFC1122-SHOULD 
   is to save IP networks from having to transport 
   same data all over again in the event of packet drop. 
   In fact, FIM directly addresses this very spirit.

   We may want to add a note in the iSCSI draft that 
   implementations should try to honor the RFC1122-SHOULD
   when the sender is not willing to supply markers.

Thanks.

-Shridhar Mukund

------------------------------------------------------------------------
Reference : David's response dtd 14 Jan. to my email:

>    b. I strongly recommend SHOULD implement FIM on the send side. 
>       Implication -> Senders that do not insert markers should be 
>       willing to accept up to RTT*BW data drops! Headers being 
>       "reasonably" out-of-order is OK. Of course, senders that do not 
>       insert markers but are willing to pay big $$$ to the SSP will 
>       get their buffer/BW allocation as usual and customary :-)

I think the Implication is too strong, as it goes against the following
SHOULD from RFC 1122 (which modifies RFC 793):

         4.2.2.20  Event Processing: RFC-793 Section 3.9
 
            While it is not strictly required, a TCP SHOULD be capable
            of queueing out-of-order TCP segments. 

A drop causes the segments up to the retransmission of the drop to 
be out-of-order.  This does not preclude "SHOULD implement FIM", but
the above Implication is overdone in my view as it appears to condone
drops of all out-of-order segments.

---

From: "Somesh Gupta" <[email-masked]>
Date: Mon, 28 Jan 2002 20:10:25 -0800

Sridhar,

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mukund, Shridhar
> Sent: Monday, January 28, 2002 6:43 PM
> To: 'Douglas Otis'; [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: iSCSI: Framing Steps
> 
> 
> 
> Dear colleagues,
> 

     IP networks so far have managed to handle packet drops
     (without complete retransmissions) without FIM. Again,
     I think the statement below is way overstating the case
     and you may want to add the significant number of
     qualifiers associated with it.

>    when the sender is not willing to supply markers.

     I think there is substantial disagreement as to whether
     any approach meets the objectives that people are
     attempting (of course there is probably no agreement
     on the requirements either). The debate so far, even
     though intelligent, has been at the surface and high
     level.

     I think to resolve this requires experimentation, or
     at least a very detailed hypothetical technical design
     and analysis (I would be more than happy to at least
     take pot-shots so the design and analysis would be
     more thorough).

     It will make visible at the compromises and assumptions
     being made - people can then decide whether we want
     iSCSI to make those assumptions and compromises.

     (or perhaps that will require people to share too much?)

---

From: Ajit Aryan <[email-masked]>
Date: Tue, 29 Jan 2002 11:08:42 +0530

>      qualifiers associated with it.

Its true that the current day IP networks are capable of handling packet
drops .... But it is high time to have a mechanism to handle this
more efficiently, keeping in view the importance of SANs and their
intended use. I think otion 6 with proper interpretation of *SHOULD* is 
a good option to opt for.

---

From: Stephen Bailey <[email-masked]>
Date: Tue, 29 Jan 2002 09:43:22 -0500

Sridhar,

> intended use.

The combination of IP+transport (TCP/IP) is precisely designed to
deliver data efficiently and scalably (as a function of network size).
It is the transport layer's role to maximize `goodput'.  Dropping
packets in the network as a result of congestion and having the
transport adjust to this is what results in IP's scalability as
compared to system area (flow controlled) network technologies.  In
other words, it's a feature, not a bug.  It's a large part of what
made IP the dominant networking technology, and it's never going away.

It is not the case that the designers of TCP said `heck, we've got
these miserable, lossy networks, I guess we'll just suck it up and do
the best we can, even though it'd work a lot better if we had some
other characteristics.'

TCP does have some warts in providing the best possible goodput, but
those are problems for TCP (or other transport) to fix.  Anyway, that
doesn't seem to be what you're referring to.

The RFC 1122 SHOULD says TCP endpoints should cling tightly to each
correctly received byte.  If an implementation is discarding portions
or entire flights of data because a packet is lost that's certainly
NOT TCP's problem, and not what FIM is intended to fix.  FIM's (or any
framing technique's) ONLY goal is to reduce the COST (which may equate
with `tractability' if the cost is otherwise prohibitive) of an
efficient implementation.  It does not improve the data transport
efficiency itself at all.

Steph

---

From: "Somesh Gupta" <[email-masked]>
Date: Tue, 29 Jan 2002 10:45:09 -0800

While I support a generic direct data placement model,
the following are additional points for consideration for
analysis of memory bandwidths and sizes.

1. A 10G link dropping packets will not do TCP at 10Gbps.
The rate drops as the packet loss increases. I don't recall
but Franco or Victor from Nortel had posted an equation once.

2. The amount of memory needed is of the order of delay-bandwidth
product and is reasonably independent of the number of connections.
Of course, statistically, worse situations are possible, but that
is not a design point. If lots of packets are being lost, things
are going significantly slower anyway (thus reducing the delay-
bandwidth product).

3. The analysis of 10Gbps, half way round the world
was initially used in this debate. ALthough interesting, the person
with the scenario above is not going to blink at the cost of
256MBytes of fastest memory considering what they are paying
for the link.

4. All assumptions are based on existing SCSI API models at
targets. Changes to target APIs/meta-data management would
let some vendors not have to do data copies anyway.

5. A solutions that is iSCSI specific is not likely to give
the savings (especially when it may not work) people expect.

As I mentioned, I think we should take the same approach as
we did for error recovery (if it is not spilling the beans
too much) and come up with a more detailed analysis.

Somesh

> Stephen Bailey
> Sent: Tuesday, January 29, 2002 6:43 AM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: iSCSI: Framing Steps
>
>
> Sridhar,
>

---

From: Paul Koning <[email-masked]>
Date: Tue, 29 Jan 2002 15:05:29 -0500

> but Franco or Victor from Nortel had posted an equation once.

A very old (40 years?) rule of thumb is that 1% loss costs you 50% in
throughput.  I expect that it gets a lot worse as links get faster.
> ...

Exactly.

One reason ATM failed as a LAN is that its design was burdened with
complexity based on that sort of scenario.  (In other words: "it has
to work at umpteen Gig, across the globe, and only use a tiny little
bit of memory because implementations can't handle a megabyte of
buffers".) 

A design optimized for the combination of very high bit rate and very
long latency will inevitably be way overpriced for the predominant
case, which is the LAN case.

      paul

---

From: Stephen Bailey <[email-masked]>
Date: Thu, 31 Jan 2002 09:54:07 -0500

> While I support a generic direct data placement model,

While you know that's the model I like, it isn't specifically what I
was talking about.

My only point was to enforce some separation of concerns between
iSCSI, which is basically an application protocol to carry the
RPC-type interactions defined by SAM, and how it is carried.

Agreed that TCP has `performance problems' at high speed (AI and MD

> the savings (especially when it may not work) people expect.

hits it on the head.  TCP (or whatever transport) needs to solve this
problem in a general way.  The reason Ethernet/TCP/IP is economical is
because its approach to data transport is scalable and general.  The
various dimensions of cheapness start to evaporate when you say `we
just need this special hook to get application X to work well'.

I'm am so down with not specifying ANY framing in iSCSI.

Steph