Subject: RE: iSCSI: Framing Steps

From: "Shah, Hemal" <[email-masked]>
Date: Thu, 24 Jan 2002 07:29:52 -0800

David,

I agree with your conclusions. I would like to add something
related to framing. Framing by itself solves only part of the
problem from the perspective of sync and steering. I am not going
to summarize pro/cons of each framing proposal here as they have
been summarized in the previously posted emails.

I would add though that all different forms of framing proposals
put some sort of aritificial limitation on max. iSCSI PDU size
- TUF requires an iSCSI PDU to be contained within Framing PDU
- FIM & COWS require iSCSI PDU reassembly buffer in case of a
  packet loss (Thus, amount of buffering on NIC/HBA dictates max.
  iSCSI PDU size)

In addition to framing, you definitely need some sort of steering
mechanism that completely eliminates reassembly buffers on NIC
and does not put any artificial limitation on Max. iSCSI PDU size.
If this steering layer is generic (DDP/RDMA), then the NIC/HBA
now has wider applicability.

In my opinion, until sync and steering layers related issues are
resolved, framing/DDP/RDMA are premature at this point to be
included in iSCSI 1.0 draft.

Hemal

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Thursday, January 24, 2002 1:10 AM
To: [email-masked]
Subject: iSCSI: Framing Steps

I want to attempt to make some steps towards resolving
the framing issues.  In reviewing the recent discussions
on framing, I have a couple of conclusions:

(1) I do not believe that WG consensus (rough or otherwise)
	can be obtained for a "MUST implement" requirement
	for any form of framing.
(2) The COWS mechanism has a lot of potential, but
	its newness, the multiple versions that
	have been mentioned, and the desire for some
	sort of alignment with new work on DDP/RDMA
	suggest that COWS is premature to specify as
	part of iSCSI.

I suggest that these conclusions form the
basis for further ips WG consideration of framing.

Please think carefully before objecting to these
conclusions on the list (I'm happy to respond to
private questions/expressions of concern).  If the
framing issue cannot be driven to closure in
the next few weeks, I will be forced to conclude
that the entire topic is experimental, and hence
needs to be removed from the iSCSI specification
and handled in separate drafts intended to become
experimental RFCs.

Thanks,
--David

p.s. A desire to build NICs that never behave in
accordance with an important SHOULD in RFC 1122
(out-of-order segments SHOULD be queued) does not
strike me as a good reason for changing the first
conclusion above.

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
[email-masked]         Cell: +1 (978) 394-7754
---------------------------------------------------

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 28 Jan 2002 07:06:40 +0200

David and team,

I have to admit that I am not die-hard fan of Doug Otis's journalistic 
prose.
However by populating with his missives so many working groups he was able 
to carry some work from IPS to SCTP where CRC32C was selected (that's what 
bees do but in good faith).  And BTW the CRC32C was carried over without 
so much as a nod to this group or quoting the long analytic and 
experimental review that some of us (Dafna Sheinwald, Pat Thaler, Vince 
Cena and myself) worked on for more than a month.

On the other hand it sports a CRC implementation (instead of a concise 
mathematical definition) at least 5 years old and can somewhat mislead 
implementers, implementation that Doug tried to peddle to this group.

I find this behavior highly distasteful in a professional working group 
and somewhat peculiar as an engineering feat.

I will have to carefully consider when and how to use RFC2026 compliance 
statement in the future.

Julian Satran

Distinguished Engineer,
IBM Reaserch Lab. at Haifa

[email-masked]
Sent by: [email-masked]
26-01-02 10:31

        To:     [email-masked], [email-masked]
        cc:     [email-masked]
        Subject:        RE: iSCSI: Framing Steps

Attempting to kill a few red herrings ...

TCP is vulnerable to packet injection based on guessing or observing
the initial sequence number in the absence of fixed-interval-markers
(FIM).  The well-known connection hijack attack is an example.  An
attacker can inject arbitrary data into a connection without FIM and
cause TCP to misbehave as a result.  FIM does not make this any
easier or change the ways in which TCP would misbehave when attacked
in this fashion.

FIM does not entail any changes to TCP.  TCP does not care about
where incoming data is placed in memory - it cares about the order
in which data is processed by TCP and delivered to the application.
The intended usage of FIM affects only the placement of incoming
data in memory and hence does not change TCP.

Meanwhile, Doug appears to have resumed his smear campaign against
the IPS wg.  Here are three examples:

> FIM has been suggested by those within the IPS wg that implementers
consider
> implementing an otherwise obnoxious scheme requiring the injection of
> markers or suffer from discarded data otherwise retained using standard
> versions of TCP.

Indeed it is obnoxious, and Doug conveniently omitted the fact that the
original message in this thread contains a strong suggestion from this
IPS wg chair that such an implementation approach is a bad idea because
it violates an important SHOULD in RFC 1122 (SHOULD queue out of order
data).
Scroll to the bottom of this email - it's still there ...

> It is clear this group wishes extensive modifications to TCP and 
> does not wish to share these details openly.

Extensive is in the eye of the beholder, but ALL of the proposed framing
schemes have been openly described on the IPS and/or TSVWG lists and/or
in Internet-Drafts.

> One suggestion was to have each iSCSI PDU become framed by a TCP 
segment.
> Just the reduction in the average packet size within a high bandwidth
stream
> will stress standard versions of TCP as well as the intervening 
equipment.
> SCTP had the foresight to include bundling methods.  This is just one
aspect
> of these proposals to fail consideration within the IPS wg.

This refers to draft-ietf-tsvwg-tcp-ulp-frame-01.txt, a tsvwg that
proposes changing TCP.  Despite Doug's proclamations that the IPS wg
should not be discussing changes to TCP, he now criticizes the IPS
wg for failing to discuss an aspect of a change to TCP???  There's
just no pleasing some people ... especially as I don't recall this
specific issue being raised in the past.

Thanks,
--David

> From: Douglas Otis [mailto:[email-masked]]
> Sent: Friday, January 25, 2002 6:13 PM
> To: [email-masked]
> Cc: Tsvwg
> Subject: RE: iSCSI: Framing Steps
> 
> 
> David,
> 
> Thank you for the response.  I think a further point should have been
> strongly made.  Such changes to TCP are not actions of the IPS 
workgroup.
> These matters should be discussed within the tsvwg.  Even matters of 
Fixed
> Interval Marking brings concerns with respect to the impact even this
> seemingly innocuous iSCSI option may have.
> 
> Should the interval within the FIM be a power of 2, guessing an initial
> sequence value allows injection of packets within existing connections. 
A
> layer below TCP, as devised for iSCSI, is to place de-encapsulated iSCSI
> information from isolated packets that include marker information 
despite
> prior packet loss.  A general goal of all the framing approaches being
> sought by the IPS wg.
> 
> This would require extensive modification of TCP to facilitate this
> behavior.  In addition, without the specific details of this scheme,
largely
> due to a desire to claim no changes, it is difficult to evaluate related
> security risks.  One risk of this scheme is the requirement for
application
> specific code to be routinely placed below the transport.  There are no
> conventions for instituting the requisite signaling between this below 
the
> transport layer application process and the modified TCP.  Should an
attack
> successfully inject even null structures, it would not be clear how the
> transport would behave.
> 
> FIM has been suggested by those within the IPS wg that implementers
consider
> implementing an otherwise obnoxious scheme requiring the injection of
> markers or suffer from discarded data otherwise retained using standard
> versions of TCP.  Clearly just sending these markers are not in 
themselves
a
> risk to TCP.  It is their intended use that causes concern.  This FIM 
has
> also been indicated by several including the author of iSCSI that FIM is
> just an interim measure to be followed by a full framing scheme later. 
It
> is clear this group wishes extensive modifications to TCP and does not
wish
> to share these details openly.
> 
> One suggestion was to have each iSCSI PDU become framed by a TCP 
segment.
> Just the reduction in the average packet size within a high bandwidth
stream
> will stress standard versions of TCP as well as the intervening 
equipment.
> SCTP had the foresight to include bundling methods.  This is just one
aspect
> of these proposals to fail consideration within the IPS wg.  In general, 
I
> think the IETF made a wise choice in proposing SCTP as the means of
> incorporating framing within IP protocols and the IPS wg was informed of
> SCTP at the outset of their endeavors.  SCTP provides many advantages 
with
> respect to pressing concerns especially the less disruptive nature
> incorporation of framing using SCTP would cause over an attempt of 
framing
> within TCP.  It is clear there is consensus within the IPS wg that SCTP 
is
> not to be considered and that TCP framing is despite prohibitions within
the
> charter and the good judgement of the IETF.  The ongoing tacit approval 
of
> these discussions regarding modification of TCP within the IPS wg is
troubling.
> Doug
> > From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of
> > [email-masked] [mailto:[email-masked]]
> > Sent: Wednesday, January 23, 2002 11:10 PM
> > To: [email-masked]

---

From: "John Hufferd" <[email-masked]>
Date: Tue, 29 Jan 2002 01:17:26 -0800

Somesh,
Since the team convinced me to come off my proposal for "Must Implement on
Send", to "Should Implement on Send", I think it gives you the ability to
Not do it for all the reasons that are of major importance to you.  The
others, that can will Implement FIM.  You can then meet in the market
place.

Option 6 (FIM today and some type of Framing later) along with "Should
Implement" seem to be the right mix for you and the others.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

"Somesh Gupta" <[email-masked]>@ece.cmu.edu on
01/28/2002 08:10:25 PM

Please respond to <[email-masked]>

To:    <[email-masked]>
cc:

Sridhar,

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Mukund, Shridhar
> Sent: Monday, January 28, 2002 6:43 PM
> To: 'Douglas Otis'; [email-masked]; [email-masked]
> Cc: [email-masked]
> Dear colleagues,
> 1. Doug : David made the point, "it(FIM) does not modify
>    the TCP protocol", as lucid as it gets. It will be nice
>    to hear some opinions from the TCP gurus out there
>    without mixing it up with SCTP and TUF.
>
> 2. It will also be nice if other iSCSI folks on the reflector
>    cast their votes per John's request.
>    Putting on my politician hat :-), pl. vote for option 6.
>    >>>> 6. FIM now, and some kind of Framing later
>
> 3. I do agree with David that the implication, "Senders that
>    do not insert markers on send should be willing to accept
>    up to RTT*BW drops" in my email was too strong as it goes
>    against a RFC1122-SHOULD. ( See reference below ). I
>    stand corrected.
>
>    The current iSCSI draft addresses my point anyway:
>    "In certain environments, a sender not willing to supply
>    markers to a receiver willing to accept markers MAY
>    suffer from a considerable performance degradation."
>
>    In my interpretation, the spirit behind the RFC1122-SHOULD
>    is to save IP networks from having to transport
>    same data all over again in the event of packet drop.

     IP networks so far have managed to handle packet drops
     (without complete retransmissions) without FIM. Again,
     I think the statement below is way overstating the case
     and you may want to add the significant number of
     qualifiers associated with it.

>    In fact, FIM directly addresses this very spirit.
>
>    We may want to add a note in the iSCSI draft that
>    implementations should try to honor the RFC1122-SHOULD
>    when the sender is not willing to supply markers.
>
> Thanks.
>
> -Shridhar Mukund

     I think there is substantial disagreement as to whether
     any approach meets the objectives that people are
     attempting (of course there is probably no agreement
     on the requirements either). The debate so far, even
     though intelligent, has been at the surface and high
     level.

     I think to resolve this requires experimentation, or
     at least a very detailed hypothetical technical design
     and analysis (I would be more than happy to at least
     take pot-shots so the design and analysis would be
     more thorough).

     It will make visible at the compromises and assumptions
     being made - people can then decide whether we want
     iSCSI to make those assumptions and compromises.

     (or perhaps that will require people to share too much?)

>
>
> ------------------------------------------------------------------------
> Reference : David's response dtd 14 Jan. to my email:
>
> >    b. I strongly recommend SHOULD implement FIM on the send side.
> >       Implication -> Senders that do not insert markers should be
> >       willing to accept up to RTT*BW data drops! Headers being
> >       "reasonably" out-of-order is OK. Of course, senders that do not
> >       insert markers but are willing to pay big $$$ to the SSP will
> >       get their buffer/BW allocation as usual and customary :-)
>
> I think the Implication is too strong, as it goes against the following
> SHOULD from RFC 1122 (which modifies RFC 793):
>
>          4.2.2.20  Event Processing: RFC-793 Section 3.9
>
>             While it is not strictly required, a TCP SHOULD be capable
>             of queueing out-of-order TCP segments.
>
> A drop causes the segments up to the retransmission of the drop to
> be out-of-order.  This does not preclude "SHOULD implement FIM", but
> the above Implication is overdone in my view as it appears to condone
> drops of all out-of-order segments.
>
>
>

---

From: "ERICKSON,SHAWN (HP-Roseville,ex1)" <[email-masked]>
Date: Tue, 29 Jan 2002 15:36:54 -0500

> Excerpt of message (sent 29 January 2002) by Somesh Gupta:
> > While I support a generic direct data placement model,
> > the following are additional points for consideration for
> > analysis of memory bandwidths and sizes.
> > 
> > 1. A 10G link dropping packets will not do TCP at 10Gbps.
> > The rate drops as the packet loss increases. I don't recall
> > but Franco or Victor from Nortel had posted an equation once.
> 
> A very old (40 years?) rule of thumb is that 1% loss costs you 50% in
> throughput.  I expect that it gets a lot worse as links get faster.

I would expect is to stay the same, as in your example stay 50% regardless
of link speed. However 50% of a 1Gbps connection is ten times smaller then
50% of 10Gbps a connection. In other words 50% of 10Gbps hurts a lot more
then 50% 1 Gbps if you focus on the bit rate.

> > ...
> > 3. The analysis of 10Gbps, half way round the world
> > was initially used in this debate. ALthough interesting, the person
> > with the scenario above is not going to blink at the cost of
> > 256MBytes of fastest memory considering what they are paying
> > for the link.
> 
> Exactly.

Yup. I believe the concern about high-speed connections and longer round
trip times can be solved by upping the TCP window and correspondingly upping
the resource requirements (mostly additional RAM) to deal with it. This
could be avoided by not using TCP as it exists today but the issues with
that are many (already heavily discussed) and can IMHO outstrip the costs of
additional RAM. In general I think the costs of RAM are dropping far faster
then the costs of upgrading the network.

> One reason ATM failed as a LAN is that its design was burdened with
> complexity based on that sort of scenario.  (In other words: "it has
> to work at umpteen Gig, across the globe, and only use a tiny little
> bit of memory because implementations can't handle a megabyte of
> buffers".) 
> 
> A design optimized for the combination of very high bit rate and very
> long latency will inevitably be way overpriced for the predominant
> case, which is the LAN case.

I would tend to agree.

-Shawn

p.s. I am not speaking for Hewlett-Packard this is just my opinion.