Subject: Re: iSCSI Data Integrity - Digests

From: Michael Krause <[email-masked]>
Date: Thu, 01 Feb 2001 08:01:58 -0800

At 10:51 PM 1/29/2001 -0500, Sean Quinlan wrote:
>I would suggest that if iSCSI requires that a PDU does not span a
>TCP segment then iSCSI is indeed changing TCP.  A valid implementations
>of TCP has considerable freedom in deciding how to pack data into segments
>and this is not under the control of the application layer.  Such a
>requirement in iSCSI would mean it is not feasible to implement iSCSI on
>many existing TCP stacks.

The continual assertion that TCP implementations are sacrosanct and thus 
not cannot be modified to support iSCSI combined with a tendency to make 
everything an option should someone object quite frankly leads one to 
question whether iSCSI will suffer the same 10-year growing pains / 
interoperability problems FC suffered and thus seriously raises a question 
of its viability within the industry.

Yes, iSCSI could be implemented using SCTP which is, from an architecture 
point of view, more suitable in many ways than TCP.  However, SCTP isn't 
really deployed today, is not targeted by those few that have deployed it 
for a high-volume market, and thus unclear whether it can generate 
sufficient interest to make it a viable transport from a business 
perspective.  I don't want to debate SCTP vs. TCP - there has been enough 
of it already on this reflector and the only point was while it is 
interesting, the business issues are a major problem that is not going to 
be solved in the desired deployment time frames.

Summary: Either iSCSI starts getting realistic in terms of what is subject 
to change without violating / major modifications to existing protocols 
(not implementations) and starts reducing the number of options so 
interoperability has a chance of success or it will suffer the fate of FC's 
early years.

>If you change the requirements of for a valid TCP implementation I believe 
>you have effectively changed the protocol.

The protocol is defined as a set of wire formats and semantics.  The 
proposal did not violate any aspect of the RFC which does allow TCP a great 
deal of freedom in terms of how it packages segments.

>I agree with Douglas Otis that such a change may not be well received.
>
>What would you suggest for situations such a iSCSI over a protocol such as 
>TLS(SSL).
>TLS uses an internal framing mechanism that is not under the applications
>control and is also not related to the underlying TCP segments.

Put some intelligence into the solution's implementation and segment 
accordingly.

>It would seem to me a little unwise to mandate end-to-end data integrity 
>within iSCSI
>especially using a CRC algorithm.  Rather a lot of data goes over protocols
>such as HTTP, FTP, NFS and CIFS without any additional end-to-end data 
>integrity
>over what is provided by TCP and link level integrity - I think it is a little
>strong to say
>         strong end-to-end data integrity is not an option as customers will
>         not adopt solutions without such guarantees.

The original proposal stated what the difference here is: these protocols 
target buffers and associated information that is not transmitted over the 
wire thus reducing the probability of a DMA targeting an incorrect location 
in memory.  iSCSI communicates addressing information either direct VAs or 
a handle for the look-up.  In either case, there is a greater probability 
of a DMA targeting the wrong location as a result.  FC protects their data 
with a 32-bit CRC which provides customers with a lot greater confidence in 
the data integrity than what they get from TCP's checksum.

Also, as noted by numerous people there is a measured evidence of silent 
data corruption getting past TCP's checksum today in the Internet.  If 
people really believe iSCSI will flow across the Internet, then silent data 
corruption must be dealt with in a much stronger manner than what is being 
proposed and having it as an option is unacceptable for most vendors and 
customers.

>If TLS or IPsec are used in conjunction with iSCSI, then strong
>end-to-end integrity is already provided and duplicating this in iSCSI seem
>rather a waste.  It may be the case that some people are not satisfied with
>the integrity provided by TCP and yet do not want to use TLS or IPsec to 
>ensure
>integrity/authenticity, but I suspect that such people are not the 
>majority and
>thus mandating a data integrity mechanism within iSCSI seems like a mistake -
>perhaps as an option or perhaps leave it to other layers in the protocol 
>stack.

One cannot mandate IPSec /  TLS usage within the standard nor can one rely 
upon it for data integrity. To assume that the majority of people want it 
is incorrect as well since this is a function of the usage models being 
deployed.  Many people interested in iSCSI are not interested per se in 
block storage over the actual Internet but block storage over IP-based 
networks - there is a very big difference here and the requirements for 
security as a result.

>If data integrity is provided by iSCSI, CRC algorithms are not particularly
>ideal from a software implementation point of view - surely there
>are better alternatives than CRC that are easy to implement in hardware and
>can take advantage of 32bit wide data ops.

CRCs are well understood.  While software is an interesting implementation 
and certainly one that I do not want to see harmed, many vendors are 
developing hardware-based solutions (keeps the CPU consumption to be the 
same as FC which is important for customers and vendors).  The proposal put 
forth also provided a mechanism for some intelligence to be added to 
non-iSCSI hardware implementations similar to the way one does checksum 
off-load today allowing software to not necessarily be burdened with 
unnecessary overheads if so desired.

Mike

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 1 Feb 2001 11:44:17 -0800

Mike,

I agree with most of your points with the exception of TCP being a wire
protocol.  Use the SCTP structures and apply them to TCP where the SCTP port
locations become a pseudo frame word length and then allow padding to a
predefined interval and you then have the required framing and all the
benefits of SCTP while not harming TCP at the API.  As we get mired down in
amorphic iSCSI, a transport specification as complete as SCTP would
immediately provide productive solutions and a clean migration to that
uncharted land where SCTP dominates.

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 2 Feb 2001 07:22:25 -0800

> interoperability problems FC suffered and thus seriously raises a
> question of its viability within the industry.

Mike, if iSCSI is deployed through the use of an adapter which collapses all
three layers, SCSI, iSCSI, and TCP, into a single set of microcode, then,
you got a new implementation of TCP anyway.  All those sacrosanct talks that
forbid changes of TCP implementations do not apply to an iSCSI adapter which
at login time will exchange text parameters with each other to deploy new
TCP options.  Everything you wanted can be inside an iSCSI adapter.
However, the right place to talk about what you want from TCP is in the
end2end group, not here, as so many people repeatedly pointed out to me. :-)

Y.P.

---

From: David Robinson <[email-masked]>
Date: Fri, 02 Feb 2001 09:23:20 -0800

> end2end group, not here, as so many people repeatedly pointed out to me. :-)

YP,
While I understand and can appreciate all the tricks that can be done
with
a merged stack that avoids the implementation layer, I strongly object
to
this WG allowing standard options that change the standard TCP wire
protocol.
Even if both ends agree to do the "special" processing, other nodes and
protocols generally share these wires and "special" non-standard options
can have nasty indirect effects.  If your custom adapter does things
that
are not changes in the protocol, like always putting an iSCSI request in
a
single segment which allows the other end to fast-path processing, is
an acceptable implementation trick as either end not "knowing" about the
trick will function correctly, although maybe not optimally.

Any changes to the standard TCP must go through end2end, this WG MUST
NOT
allow options that circumvent that process.

	-David

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 2 Feb 2001 12:41:09 -0800

> with a merged stack that avoids the implementation layer, I strongly
> object to this WG allowing standard options that change the standard
> TCP wire protocol. Even if both ends agree to do the "special"
> processing, other nodes and protocols generally share these
> wires and "special" non-standard options can have nasty indirect
> effects.  If your custom adapter does things that are not changes
> in the protocol, like always putting an iSCSI request in a
> NOT allow options that circumvent that process.
> 	-David

David,

Each time when I touched this area, I always felt like walking around a
snake pit.  It is neither my intent to discuss changes of TCP wire protocol
in this WG nor my desire to make such changes.  I've been always carefully
in using the words "TCP implementation."  In fact, after reading a lots of
RFC's, I have come to the conclusion that there are ways within TCP that
would allow iSCSI work well in 10GB/100Ms Network.  The slow TCP transport
problem is in the existing implementations that do not have all the new TCP
options.

For example, you can have 16-bit TCP window-size or 32-bit.  Using 16-bit
window size, there in no way we can keep the 10Gb/100Ms pipe full.  But, the
use of 32-bit window-size does not change the TCP protocol.  Another
example, an iSCSI adapter can always send each iSCSI Command PDU in a
separate TCP segment.  In so doing, URG pointer is perfect for marking
message boundary. While the gurus of TCP can quickly point out that it is
impossible for URG pointer to mark the SCSI command boundary in the BSD
implementation, there is nothing preventing two iSCSI adapters from using
URG pointer when they send command PDUs to each other.  In fact, whether two
iSCSI adapters using URG pointer in their exchanges, IMHO, is even outside
the jurisdiction of this WG.  Because the use of URG pointer does not
violate TCP protocol.  It is just that old TCP implementations not able to
keep command/status PDUs in separate segments can not benefit from URG
pointer.

Therefore, while an iSCSI adapter can interoperate with a client running in
TCP of BSD 4.3, it should also be able to interoperate with another
state-of-art iSCSI adapter which is much more intelligent.  The difference
in their interoperation will be decided at login with text messages.  I
think we don't have to limit ourselves to stone age tools if electrical
drill is already invented.  Please note that I am not asking the invention
of new TCP options here.

Y.P.

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 2 Feb 2001 15:04:57 -0800

<snip>
> While the gurus of TCP can quickly point out that it is

I see a spark of life still lives in hopes of redefining the Urgent pointer.
A problem you will see is this pointer is only 16 bits.  As it is defined to
coalesce should new points become defined, it takes a small backlog in the
TCP stack to create a pointer that can not reach a point being defined at
the API.  To use the Urgent pointer as a message marker, this issue has been
decided and, in short, your conclusion about the ability to use the urgent
pointer in the manner you wish is re-inventing TCP and not to be done.  If
you wish such non-standard implementations of TCP, it clearly is outside the
benefits of this WG.

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Fri, 2 Feb 2001 16:04:06 -0800

> I see a spark of life still lives in hopes of redefining the
> Urgent pointer. A problem you will see is this pointer is only 16 bits.
> As it is defined to coalesce should new points become defined,
> it takes a small backlog in the TCP stack to create a pointer that
> can not reach a point being defined at the API. To use the Urgent
> pointer as a message marker, this issue has been
> you wish such non-standard implementations of TCP, it clearly is
> outside the benefits of this WG.
>
> Doug

Hi, Doug:

The Charters of this WG prohibits us discussing any change of TCP protocol
and I was not talking about changing any TCP protocol.  I am quite familiar
with the history why urgent pointer wouldn't work as so many TCP gurus were
too quickly to point out.  This WG does not recommend the use of urgent
pointer nor does it endorse the placement of a command/status PDU in a
separate TCP segment.

However, for microcode that running in my iSCSI interface chip with my TCP
acceleration implementation, I can place each command/status PDU in a
separate TCP segment without violating the TCP protocol.  Similarly, in the
manner I understood about TCP if I turn on the urgent pointer when I send
the PDU to another iSCSI chip of which I came to know at login, I don't
believe I have altered TCP.

Some people in this WG might not like what could be done by each iSCSI
interface chip designer.  However, this method of doing something special in
a SCSI device not available by others gives one a unique advantage and is a
common practice by the SCSI industry for many years.  I will not be
surprised that each iSCSI interface chip will have its own bag of tricks to
accelerate its performance and still totally confirm the iSCSI
specifications.  This is how the technology world is. :-)

Y.P.

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 5 Feb 2001 11:18:09 -0800

Y.P.

You are free to add non-standard TCP and perhaps both server and client may
implement the same level of alteration. Change aggressiveness of the TCP
congestion algorithm, force framing, implement non-standard definitions of
urgent pointers, add special flags, and seldom seen options.  View TCP as
important in name only or wire only.  An alternative to chaos generated by
multitudes of such clever creations would be adoption of SCTP to provide
options already found within this standard.

iSCSI is a good example why transports are not easily developed.  If to
allow fail-over with iSCSI, even StatSN should be unique across all
connections and a strict monotonic sequence should be mandated for both
CmdSN or StatSN within every PDU (rename to ClientSN and ServerSN.)  Better
yet, adopt the SCTP structures within TCP (replace the SCTP ports with a
pseudo-frame length and allow padding to fixed intervals for frame
recovery.)  I fail to see how either client or server knows the state of the
other and how to prevent a cascade, and what benefit is derived by the
complex process of rediscovery of discarded relationships.

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Mon, 5 Feb 2001 13:29:25 -0800

> options already found within this standard.

Doug,

May be I am mistaken, but I don't think what I am suggesting is doing
non-standard TCP.  What is a non-standard TCP if I follow all the RFCs
approved by IETF?  What I think you are referring to the non-standard
implementation, i.e., not working working with existing TCP stacks.

The urgent pointer, 32-bit TCP window, one segment per PDU, even with the
addition of SACK are all part of "standard" TCP, just not every TCP
implementations are supporting them.  This is why option negotiation comes
in.  I hope I am not mistaken.

Y.P.

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 5 Feb 2001 16:43:31 -0800

Y.P.

Forcing frame alignment does little to assist iSCSI if mid-tier equipment
removes such framing despite option negotiation.  The urgent pointer has a
defined behavior within RFC documentation such that to define it as a record
boundary is not in keeping with standards.  To implement and use an urgent
pointer as a record boundary pointer clearly modifies the TCP standard.
Requiring modification of all compliant TCP implementations is modifying the
standard.  There are already several TCP negotiated options such as SACK and
no one suggested disallowing use of existing standard options.  The urgent
pointer as a record boundary marker, as several people have indicated, is
not part of the standard, can not be done reliably with existing
implementations, and clearly places this concept into the non-standard
category.

PDU discovery can be done in a number of ways.  If the SCTP structures were
adopted, then a pseudo-frame could be declared to start at fixed intervals
within the TCP byte stream.  This could be done without modifying any TCP
implementations and as every device could support this mechanism, it could
be done in a uniform way.  The SCTP structures allow encapsulation of larger
structures and could comply with this interval and define handshakes and
retries independent of TCP.  This becomes required with added digest errors
that would need handling above TCP and hopefully below SCSI.  I understand
the desire to use reserved bits and tweak behavior but with such a sizeable
install base, such desires only appear deceivingly simple to realize.

The NET-SCSI standard should be independent of the underlying transport.
There should be a transport layer added to TCP to provide all needed
features demanded by SAN.  SCTP does a good job in defining these modern
transport requirements and an additional layer could be added to TCP to
provide a transport compliant to both SCTP using TCP.  In doing so, there
would be virtually no difference between the two transports and there would
be little doubt about eventual migration.

Doug

---

From: "Y P Cheng" <[email-masked]>
Date: Mon, 5 Feb 2001 17:16:04 -0800

> removes such framing despite option negotiation.

I am not aware of any mid-tier box being TCP-aware and repackaging TCP
segments.  If so, it must also involve in different TCP options including
SACK and 32-bit window, not to mention changing the TCP header and
recalculating checksum.

> The urgent pointer has a defined behavior within RFC documentation
> such that to define it as a record boundary is not in keeping
> with standards.  To implement and use an urgent pointer as a record
> boundary pointer clearly modifies the TCP standard.

Very interesting argument.  Urgent pointer is meant to be a inband signal
that can be used for any particular purpose a user has in mind.  Saying
urgent pointer as a marker for a record or a message being a modification of
TCP standard, IMHO, is definitely a stretch.  However, I yield the debate to
the TCP gurus.

---

From: "Douglas Otis" <[email-masked]>
Date: Mon, 5 Feb 2001 18:35:19 -0800

Y.P.

<snip>
> TCP standard, IMHO, is definitely a stretch.  However, I yield

The urgent pointer marks urgent data and must coalesce to include additional
urgent data.  To use the urgent pointer to mark boundaries in close
proximity, as close as the next segment as you described, then you will be
required to modify TCP to not coalesce.  Starving send will not be conducive
to accelerated use and not coalescing the pointer is a violation of the TCP
standard.  Both TCP send and receive coalesce.  Yes, the urgent pointer is
an inband signal but with these clearly defined limitations.  These
limitations make it impractical for use as a record marker and likely to
thwart TCP Offload Engines as it also requires unique data handling.  The
urgent pointer is not a general purpose marker.  Its devised use is to
expedite and only requires a single variable.  You are attempting to expand
use of the urgent pointer to encompass a greatly revised definition
requiring an attribute reserved for every message unit up to one per
segment.  It is definitely a stretch to consider this not a modification of
the TCP standard, IMO.

Doug