Subject: RE: iSCSI: No Framing

From: "THALER,PAT (A-Roseville,ex1)" <[email-masked]>
Date: Sun, 3 Feb 2002 16:01:12 -0700

Jim,

In answer to your questions:

Agilent is planning to implement framing. We view both FIM and 
COWS as having about the same utility so we would implement 
whichever one went into the standard. 

A smoothing buffer on the chip is feasible wrt your point 2. 
There are some configurations that would use external memory.

Also, one concern is that the very situation where one would
need large window size for efficiency - high bandwidth long
distance communication - is where out of order receipt becomes
increasingly likely so the amount of memory for this situation
could push up cost to an excessive extent.

Reducing the amount of traffic that has to be shunted to an 
external memory affects the bandwidth that needs to be provided
for that memory. If we can handle most PDUs internally then the
external memory doesn't have to be as wide and as fast. For 
instance, if a drop means that the iSCSI PDU with the drop 
is put into external memory then that takes less memory bandwidth
than if a drop means the whole data stream for that session will
be going into the buffer until the hole is filled.

This decision also effects latency after a drop and required 
backplane bandwidth.

Let's say one is on a 10 Gbit/s extreme long distance WAN 
connection and that there is a drop. If the round trip delay
for getting the whole filled is 100 ms, then without framing,
one could have about 1 Gbit stored in the local buffer memory 
by the time the hole is filled. One will have to process all 
of this through iSCSI and across the backplane before one can
deal with the new traffic coming in. If the backplane data rate
is closely sized to the external data rate, an extra 100 ms latency 
has just been inserted into the path until traffic slacks off.
(Congestion control might mitigate this to the extent that one
is talking to just one source, but with multiple sessions, one
will still have to keep up.) To get the buffer emptied to make
space for further drops and to get the latency back down, one
will have to size the backplane bandwidth wider and have the
iSCSI engine process at a higher data rate.

With FIM or COWS, one can end the incident with much less data
in the buffer as one processes PDUs after the hole while waiting
for it to be filled. Then, when the hole is filled one just has
to process a small amount of data to catch up.

Regards,
Pat Thaler

-----Original Message-----
From: WENDT,JIM (HP-Roseville,ex1) [mailto:[email-masked]]
Sent: Tuesday, January 29, 2002 10:47 PM
To: [email-masked]
Subject: iSCSI: No Framing

So, it would be good to hear from several iSCSI 
NIC/chip implementors who:
- have or plan to implement FIM or COWS (or some 
other framing mechanism) and take advantage of it to 
minimize demands on on-NIC buffer memory 
bandwidth/quantity.
- believe that all-buffers-on-chip solutions are 
feasible and valid (wrt the points above, including 
#2)
- can quantify the memory/pin/power/space cost 
savings for all-buffers-on-chip solutions

Jim

---

From: "Douglas Otis" <[email-masked]>
Date: Sun, 3 Feb 2002 17:42:41 -0800

Pat,

The macroscopic behavior of the TCP congestion avoidance algorithm by
Mathis, Semke, Mahdavi & Ott in Computer Communication Review, July 97,
provides a simple formula for an upper bound on the TCP transfer rate:

Rate <= (MSS/RTT)*(1/sqrt{p})

A .004% loss rate with 1460 MSS, and 100 ms RTT would be less than 2.3 MB/s
or 18 Mb/s.

In your senario, there would be a requirement to buffer 230 KB.  The system
will also see a reduction in send traffic accommodating receiver side
recovery following 157 packets after a loss seen about every 25,000 packets.
There should be little concern about dealing with a single connection across
great distances working at phenomenal data rates unless packet loss becomes
diminishingly small.  The cost for such a connection will also make related
memory concerns diminishingly small as well.  Reliability and security
should easily override benefits seen by placing the application below TCP as
advocated.

If you wish to implement framing, then may I suggest such a scheme is
possible using SCTP without placing the application beneath the transport.
SCTP offers may other benefits needed in any mission critical application.
You would save your customer grief by not introducing obsolete equipment as
there is already this emerging standard that provides many improvements over
such a kludge.

Doug

---

From: "Rod Harrison" <[email-masked]>
Date: Mon, 4 Feb 2002 16:35:46 -0000

Framing of any kind is not a panacea allowing for almost buffering
free target implementations, it can only move the buffering from the
transport layer up to the iSCSI layer. Remember that the target MUST
guarantee commands are delivered to SCSI in the command sequence
number order. This means if a dropped packet contains an iSCSI command
PDU you are going to have to buffer until the CmdSN hole is plugged.

	Consider the example of an initiator filling the command window,
perhaps sending CmdSNs 1 through 100, possibly along with immediate or
unsolicited data for some of the commands. If the packet dropped
contained the command PDU for CmdSN 50 then the target iSCSI layer
must buffer CmdSNs 51 through 100 and their associated DATA_OUT PDUs.
Since you can continue to process any DATA_OUT PDUs for CmdSNs 1
through 49 the buffering required is reduced, but not eliminated,

	Only if the packet dropped contains DATA_OUTs and no command PDUs can
you continue processing past the hole while waiting for it to be
filled. You can R2T for data associated with commands after the hole
but that serves only to potentially decrease recovery time, not reduce
buffer requirements.

	Don't get me wrong, I'm not saying framing isn't a good thing but it
is only one piece of the puzzle.

	- Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
THALER,PAT (A-Roseville,ex1)
Sent: Sunday, February 03, 2002 11:01 PM
To: WENDT,JIM (HP-Roseville,ex1); [email-masked]
Subject: RE: iSCSI: No Framing

Jim,

Regards,
Pat Thaler

-----Original Message-----
From: WENDT,JIM (HP-Roseville,ex1) [mailto:[email-masked]]
Sent: Tuesday, January 29, 2002 10:47 PM
To: [email-masked]
Subject: iSCSI: No Framing

Jim

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 4 Feb 2002 13:34:49 -0800

I share the concern about iSCSI embracing a framing mechanism that is
not a MUST implement.  For all the reasons that Jim pointed out, OTOH,
I am not recommending a MUST framing either.  I suspect not many will
implement framing if it's a MAY, so it appears that we're talking about
a potential SHOULD.

Given that SHOULD is a fairly strong requirement, one "significant justification"
for not doing framing could be (even while the NIC *may be* on the expensive side) -
 - has less design complexity since no OOO placement.
 - can have quicker TTM since less design, testing, debugging etc.

I think ultimately it boils down to: how many vendors would use a "significant
justfication" to not implement a SHOULD-requirement?

If that's a majority: let's say vendor X is implementing (whatever) framing for
optimizing the memory requirements, it essentially means that X's product will
perform poorly with (the majority) no-framing senders.  I don't think X would
like that, nor the customer.  IOW, this situation doesn't seem to be long-lasting.
OTOH, the situation of an almost equal number of  "framing" and "no-framing"
products in the market (perhaps at different price points) could be unfortunately
long-lasting....

To summarize, it is a troubling prospect that a framing technique (if adopted as
SHOULD) has the potential to somewhat fracture the market and in effect
create "interoperability problems" (of performance sort) similar to that affect
FC....
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
Hewlett-Packard MS 5668
Roseville CA 95747