Subject: RE: iSCSI: No Framing

From: [email-masked]
Date: Tue, 5 Feb 2002 19:44:10 -0500

Julian,

> Dough is continuing his rumblings - whether they are relevant to us or
not.
> I wonder if there is some administrative measure you can take to save us
> some bandwidth or we should install our own "Otis filters".

Doug is within bounds, but not by much.  He's basically correct that:

> Discussion regarding the concept of framing using FIM within iSCSI is a
> valid topic at this time.  [... snip ...]  With this topic to be decided
> shortly, now is my opportunity to speak.

On the other hand, Doug continues to engage in flawed reasoning that shows
a lack of respect for other members of the WG.  His very next sentence,
for example:

> A generalized scheme for implementing DDP is relevant as opposed to one
that
> is proprietary and application specific.  

The word "proprietary" is an unfortunately typical of Doug:
(1) assuming that standardization of certain components and/or interfaces 
	is/are necessary to the use of FIM,
(2) failing to acknowledge the existence of implementation approaches that
	may not require such components or interfaces, AND
(3) failing to consider the possibility that standardization of such
	components or interfaces may not be necessary to interoperability.
The result is that here and elsewhere when the IPS WG avoids Doug's
invitation to unnecessary standardization, Doug accuses the IPS WG
of working in secret on proprietary solutions.

Needless to say such accusations are not only groundless, but have also
become boring due to excessive repetition.  Doug might get more respect for
his ideas if he showed more respect for the ideas of others.  Nonetheless,
it is fair for Doug to advocate that SCTP is the right long-term solution
and that FIM is not a step in that direction:

> To prevent fracturing of the market and creating standards competition
> within IETF, FIM should be removed from the iSCSI draft.  Transition to
SCTP
> for this feature and do not add layers to TCP.

One's personal filters are ones own concern - I have a responsibility to
pay attention to the contributions of all, but it would help if Doug
would remove the words "proprietary" and "secret" from the vocabulary
he uses when posting to this list.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
[email-masked]         Cell: +1 (978) 394-7754
---------------------------------------------------

---

From: "Douglas Otis" <[email-masked]>
Date: Tue, 5 Feb 2002 22:00:22 -0800

David,

> > not. I wonder if there is some administrative measure you can take to
> > save us some bandwidth or we should install our own "Otis filters".
>
> > that is proprietary and application specific.
>

FIM makes absolutely no sense following TCP.  Advantages for FIM are
realized if done prior to placement within system memory as typified by the
general understanding this scheme is for NIC vendors.  Whether it is done
using specialized hardware, intelligent adapters, or software, the goal, as
often stated, is to reduce buffering required in the event of packet loss.
TCP is unable to deliver network traffic during such an event, and FIM is a
scheme to allow a layer prior to TCP to interpret "as an application" the
expected byte stream "out of sequence" as a means to steer encapsulated
payloads.  A generalization of this would be described as direct data
placement.

If done using a protocol employing the TCP transport, a layer must be added
ahead of TCP, which would be, without documentation, a "privately owned"
layer.  Privately owned as such details of this layer are not shared
publicly.  This layer may be hardware or an intelligent adapter, or it could
be a software solution within a memory starved CPU.  Without documentation
describing the information exchanged between the application and TCP,
together with the states involved in this process, any efforts to
extrapolate the details of this layer are likely to venture into proprietary
areas.  A means of avoiding this problem would be to document the basics and
thereby make such a process public.  It would also afford an ability for
review.

> 	may not require such components or interfaces, AND

Proprietary does not imply hardware although in this case it is a high
probability that hardware or an intelligent adapter would implement this
layer.  Unfortunately, the use of TCP will also require that actions of this
"synchronization and steering" layer will be unique for each application.
This is a significant problem that is completely overcome through the use of
SCTP.

> 	components or interfaces may not be necessary to interoperability.

This is difficult to determine due to the lack of details.  It is also
difficult to judge security risks for the same reason.  At this point, it is
a black box.  As this layer must anticipate the behavior of TCP, it would be
wise to expose techniques used for doing so and this should be done within
the tsvwg for a broader range of expertise.  There is also a requirement for
associations to be made between buffers and application specific structures
communicated in some fashion from the application to this layer.

> of working in secret on proprietary solutions.

The IPS wg has demonstrated a desire to institute a scheme for incorporating
direct data placement as a generalization of this feature allowed through
the use of FIM.  A noble goal.  This desire has manifested itself as drafts
designed to implement framing using Urgent Pointers, Fixed Interval Markers,
Byte Stuffing, to an all out mandated framing of TCP as detected by segment
length.  This "improvement" to TCP was initiated at the pre-BOF for IPS and
has not relented.  I have never invited change to TCP, but rather strongly
urged the wg to consider SCTP as an alternative solution for DDP versus
these "improvements" to TCP.  My continued opposition to these ideas have
not won me any popularity as I still view these concepts as a kludge and a
corruption of network layering.

My reasoning for pressing this issue is not to prevent framing, but to
suggest a better practice for achieving this feature.  Failing that, have
the iSCSI draft document the interface and states involved in this
"synchronization and steering" layer to assist efforts in the creation of a
generalized scheme for DDP using SCTP.  This would ensure migration from
this application specific layer if it were to become used in practice.  The
benefits of FIM would be marginal at this time as targets can use techniques
to avoid the need for relocating buffers without the use of framing.  This
should allow time for deployment of generalized schemes at the clients using
SCTP.  NIC vendors would reap a larger return on their investment when
offering a generalized feature as there would be a greater opportunity for
its use.  DDP could be employed in many protocols such as RPC as one
example, in addition to SCSI without introducing complex layers beneath the
transport.

> become boring due to excessive repetition.  Doug might get more respect
> for his ideas if he showed more respect for the ideas of others.
> Nonetheless, it is fair for Doug to advocate that SCTP is the right
> long-term solution and that FIM is not a step in that direction:
>
> > SCTP for this feature and do not add layers to TCP.
>

I should have started my response at the bottom and worked my way up.  By
"proprietary", I am referring to private.  If you see private, think
proprietary and vice versa.  I believe I am using this word correctly.  I
have not used the word "secret" except to comment about the NDT header
offered by Julian.  In that header is a "shared secret" field.  My comment
was that CRC does not mask the "shared secrets" or something to that effect.
I apologize if this is becoming boring, but do not think my concerns are
groundless.  It has been a long process, however.  Again, thank you for your
response.

Doug

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 6 Feb 2002 02:01:54 -0800

> David,
>
> On Tue, 05 Feb 2002, [email-masked] wrote:
>
> What's the matter with that?  I would say that 80-90% of
> those lurking on these working groups are doing exactly that:
> working in secret on proprietary solutions; forming trade
> secrets to embed in their implementations; and patenting
> whatever the can to encumber the competition.  That's just the
> way of the world.
>
> As with any other standard body, IETF standards represent a
> tenuous balance between interoperability and proprietary
> implementation.  Even so, the number of proprietary (and
> patented) protocols proliferate the RFC servers as
> "Informational" and some even as proposed standards with
> Intellectual Property claims.

By placing these concepts into the public forum, that marks the end in which
"invented here" can be claimed and those organizations participating are
expected to volunteer aspects they consider "proprietary."  You are right
that such is often the case and hence the general agreement for disclosure.

> I find it hard to believe that a tortuous approach that has
> only short-term viability is going to have such a deterious
> effect in an industry that holds the trade secret and
> intellectual property rights in so high an etseem.
>
> (I would invite Doug to re-read the copyright disclaimer
>  on all RFCs.)

I am aware of the disclaimer.  I am also aware of the need for review.
There are many other applications clamoring for this "useful" dodge.  It can
be done in a haphazard fashion, or this transition toward Direct Data
Placement can be done with little disruption to existing software and
equipment.  There is already an RDMA group wishing to form as well as other
clustering schemes that desire this ability, in addition to iSCSI.

> What will this market be fractured into?  Those with trade
> secrets and those without?  Or is this a Beta vs. VHS thing?

There will always be trade secrets or proprietary information regarding any
solution offered by a vendor.  Each has their investment to protect.  The
greatest danger regarding something as basic as a network interface comes
from the vendors.  Should a vendor lay claim to a means of accomplishing the
interface to this beneath TCP "synchronization and steering" layer, then
applications will be forced to modify their interface significantly between
systems.  In essence, it would be Beta vs VHS with respect to something that
should remain ubiquitous.

> At least the draft provides some basis for interoperability
> and addresses the extent to which those taking this short-term
> blind alley are willing to reveal their plans to each other.
>
> So how will it be fractured?--Into those up the blind alley
> and those in the traffic jam?

There are already many proprietary schemes in the market and the IETF has
not prevented any vendor from entering that arena whether they are blind
allies or not.  The IETF should take the long view and strive to prevent
unnecessary disruption to existing technology used within IP.  SCTP offers
the necessary features and protections to allow Direct Data Placement to be
deployed without disruption.  A clever state machines running beneath TCP
attempting to anticipate TCP will likely work in the general case, but then
fail completely when confronting a new and untried piece of equipment that
does something unexpected.  In that case, who would be at fault?  You would
have nothing upon which to judge the cause of the failure.  Was it due to
the application not properly monitoring this undefined status coming for the
"synchronization and steering" layer beneath TCP?  Was it the state machine
within the S&S layer?  There is nothing like complexity to allow a large
company to dominate however. : )

> Please Doug, if there is a technical (not political) basis
> on which this draft is somehow flawed, please express it.
> You seem so vehemenently opposed to this draft over the
> last number of months: surely there is some technical basis
> upon which your first aversions were based.

I think that the concept of network layering where functions are concisely
defined is a technical aspect and not a political one.  That does not mean,
there are not politics at play.  When you introduce a layer below the
transport that manipulates a partial data stream on behalf of the
application, you have thrown the concept of layering out the window.  This
goes well beyond techniques already on the fridge such as packet filters or
ALGs within NATs.  Here you would have continuous interchange between this
S&S layer and the application.  I must admit I am not surprised by the
opposition to a ready made solution that preserves network layering while
offering virtually every possible innovation.

The storage industry rightfully considers themselves an industry unto
themselves.  It is also not surprising the zeal in their attacking perceived
problems.  It has been my privilege to work in this industry and networking
for a quarter of a century.  Working within the tape industry, I have also
learned that it is difficult to make a difference in the direction the disk
industry takes.  The disk cadre and their barbs require a relatively thick
skin if you wish to oppose their consensus.  My resource in this effort is
an annoying persistence.  It may take a prolonged period for what may appear
as a minor detail.

> If it is the protocol layer boundary violations that concern
> you, what is the technical (rather than preceived market)
> disadvantages of such violation?

This new layering concept is not without its risks as it devises placement
of information prior to the protections offered by the transport.  It does
so in perhaps a dangerously predictable manner in that the location of the
marking scheme may easily be spoofed later in time.  This S&S layer through
the use of a highly predictable marker is intended to allow "out of
sequence" segments to be placed.  Depending on how a duplication problem is
handled within this S&S layer, evidence of this event may be obscured.
There is also opportunity for information to be diverted by means of the
communication sent to the S&S layer.  As this layer is below the transport,
this link may suffer weaknesses not present at the more robust API for the
transport itself.  If this communication is present at the transport API, it
should be documented to allow applications consistent essential constructs
when supporting this feature.

> It is only on the basis of technical or procedural error that
> an appeal can be honestly and successfully lodged under the
> IETF process anyway and you had certainly better air those
> technical objections within the WG first.

I see the function of the IETF in broad terms.  There is a general
architecture that is being created in addition to individual protocols.
This process should not opt for a proprietary solution (sorry David)
especially making such a Must, Should, or even Recommended within a draft
when the IETF has already provided a technically superior competing
solution.  SCTP is years ahead on this curve, and although the IPS wg is
good, there is no doubt which is the technically superior solution.
Although David has done an admirable job point out the lines I should not
cross, I too feel there have been some lines crossed.

> Could you provide a summarized list of your technical
> objections with proposed solutions so that others in the
> WG can comment?

I think that I have indicated my concerns and these areas are judgment calls
for those responsible perhaps should framing remain within the iSCSI draft.
More than a year ago my opposition to the use of the TCP Urgent Pointer for
framing caused a similar conflict between David and myself and perhaps set
the stage for this friendly give and take.  One might say that necessity is
the mother of invention, but in this case, it is not necessity as SCTP is a
ready made solution for this framing and layering problem.  I think that if
the IPS wg could see past creating new solutions for a problem already
solved, they would be months, if not years, ahead.  It is a minor point,
remove the FIM from the iSCSI draft and forego use of a layer beneath TCP
for the reasons stated.

Doug

> There might be several solutions that might
> lead to improvements even at this late a stage.  It is
> normally only a technical argument that will lead to concensus
> here.  (I doubt that the WG chair could ask for comments on
> a question concerning market fragmentation and trade secret
> plots.)
>
> --brian
>
> --
> Brian F. G. Bidulock
> [email-masked]
> http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 5 Feb 2002 22:21:53 -0600

David,

WG can comment?  There might be several solutions that might

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Lloyd Wood <[email-masked]>
Date: Wed, 6 Feb 2002 15:34:51 +0000 (GMT)

On Wed, 6 Feb 2002, Brian F. G. Bidulock wrote:

> What is lost by throwing layering out the window?

modularity, ease of understanding, and portability.

L.

<[email-masked]>PGP<http://www.ee.surrey.ac.uk/Personal/L.Wood/>

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 6 Feb 2002 10:12:28 -0600

Lloyd,

> modularity, ease of understanding, and portability.

Modularity can be acheived without layering.  The entire
field of OOAD proves this.

Ease of understanding is acheived through accurate
interface description.  Again OO shows this.

Portability can also be acheived through OO approaches.

The classical layering of communications functions
into functional groupings is not a necessary condition
for any of these benefits.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 6 Feb 2002 09:23:54 -0600

Douglas,

On Wed, 06 Feb 2002, Douglas Otis wrote:

[snip]
> offering virtually every possible innovation.
[snip]

What is lost by throwing layering out the window?  Layering is an
artificial categorization of functions at best.  What technical
advantages are lost?

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 6 Feb 2002 09:38:18 -0800

Brian,

> for any of these benefits.

Unlike a typical play dough program, this stuff is more like working with
concrete.  A portion of each application demanding a direct placement
feature would be included in hardware or intelligent adapters if using FIM
and TCP, or framing and TCP, for the most part.  SCTP allows layering for
this "clean modularity" needed if one is to design an adapter that need not
understand the complexities and structures of every application that desires
this function.  The adapter interface only needs to understand SCTP and not
the application.  By using the unordered delivery mode, a shim would be able
to implement generalized structures for including Direct Data Placement or
even a full implementation of RDMA.  It would be possible for more elaborate
interfaces to built upon this foundation, but at least this establishes the
modularity, ease of understanding, and portability desired, if one is going
to start working in concrete.

Unlike the case with FIM and TCP, this would introduce NIC devices that will
be forever sensitive to even minor changes to structures employed in these
applications and the desire for this feature does not even begin to end at
just one application.

Doug

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 6 Feb 2002 12:14:05 -0600

Douglas,

> to start working in concrete.

Not necessarily...  Nowadays entire OS's and applications are embedded on
the adapter.  The adtapter can have full knowledge of all applications
which it supports and provide only higher level application interfaces
to app blades.  This is the preferred architecture for scalability and
redundancy as well as proving a clean separation between applications
and the application engine.

If some want to agree on how this embedding is going to occur with some
vestige of interoperability, what does it matter?  Would not your time be
better spent furthering DDP/SCTP?  It is certainly not the first time that
IETF has had competing approaches: ultimately the market decides.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: "Douglas Otis" <[email-masked]>
Date: Wed, 6 Feb 2002 11:05:10 -0800

Brian,

> > Unlike a typical play dough program, this stuff is more like
> > working with concrete.  A portion of each application demanding
> > a direct placement feature would be included in hardware or
> > intelligent adapters if using FIM and TCP, or framing and TCP,
> > for the most part.  SCTP allows layering for this "clean
> > modularity" needed if one is to design an adapter that need not
> > understand the complexities and structures of every application
> > that desires this function.  The adapter interface only needs to
> > understand SCTP and not the application.  By using the unordered
> > delivery mode, a shim would be able to implement generalized
> > structures for including Direct Data Placement or even a full
> > implementation of RDMA.  It would be possible for more elaborate
> > interfaces to built upon this foundation, but at least this
> > establishes the modularity, ease of understanding, and portability
> > desired, if one is going to start working in concrete.
>

The rationale for FIM was to avoid the rather modest use of adapter buffer
space.  Now you are suggesting that the adapter will include the entire OS
and application.  It sounds as if you are describing clustering where I
think RDMA is an attempt to standardize this type of interface.

> IETF has had competing approaches: ultimately the market decides.

This change in architecture is profound with respect to impact on competing
approaches.  I would not wish to under estimate leverage found from storage
as it can be used to wrest control away from standards into an array of
vendor unique solutions.  This FIM/TCP scheme adds a layer beneath the
transport and either modifies the layer above or adds a new interface to
this added layer.  Either way, the definition of what is included within a
transport is greatly diminished.  You are asking me to turn my back to this
while I see it in my interest to keep this interface open and I see SCTP as
a viable solution for doing so.  Restricting iSCSI to TCP would not prevent
future use of SCTP for this application provided this interface remains
open.  If you are right about what can be included within an adapter, then
justification for FIM/TCP vanishes.

Doug

---

From: Lloyd Wood <[email-masked]>
Date: Wed, 6 Feb 2002 19:06:28 +0000 (GMT)

[much oo psychobabble]
> vestige of interoperability, what does it matter?

so why are you here?

[I simply can't be bothered to talk about how design choices that
 might make sense within some closed oo system become meaningless
 outside that system. Solipsistic ontologies have little intrinsic worth.
 Network layering provides basic axioms, and lacks solipsism.]

L.