Subject: iscsi: 08 Comment, framing

From: Dave Sheehy <[email-masked]>
Date: Mon, 08 Oct 2001 23:54:22 PDT

Since the following is the consensus of the framing team:

> iSCSI, with minor modifications since the presentation at the IETF:
> 
> 
> 	The Sender:
> 		- MUST support no framing
> 		- MUST support at least one framing solution
> 		- MUST use the framing specified by the receiver, 
> 			if it supports that framing mode

and ...

> 
> First, a quick summary of the issues:
> 	- Deploying iSCSI sender can be done:
> 		- on top of an unmodified TCP stack with:
> 			o No framing
> 			o Marker based framing 
> 		- on top of a modified TCP stack can implement
> 			PDU-alignment.
> 	- The receiver trade-offs are:
> 		o No framing - large receive reassembly buffer 
> 			(higher cost solution)
> 		o Marker framing - receive reassembly buffer is reduced 
> 			to an eddy buffer, but requires modifications to
> 
> 			TCP receive stack (medium cost solution)
> 		o PDU-alignment - receive reassembly buffer can be
> reduced 
> 			even further, but requires modifications to TCP 
> 			receive stack (lowest cost solution and enables 
> 			eventual deployment of a viable chunking protocol).
> 	- The cost of implementing markers on unmodified TCP stacks
> 		o sender cost is acceptable, assuming a gather list is
> 			reasonably implemented.
> 		o receiver cost is unacceptable

> Initial implementations for initiator appear to be in two camps:
> 	- Unmodified TCP software stacks
> 	- Embedded TCP offload in the NIC (essentially TCP 
> 		is hidden from the host SCSI stack)
> 
> Initial implementations for the target appear to be in two camps:
> 	- Optimized NICs which will support framing - probably both 
> 		framing modes. Because both modes
> 		require modifications to the receive TCP stack and 
> 		PDU-alignment is viewed as straightforward, it is 
> 		assumed that most implementations will implement 
> 		both framing solutions to allow them to transfer 
> 		data optimally with either unmodified send TCP stacks 
> 		or PDU-alignment send TCP stacks.
> 	- Unmodified TCP software stacks
> 
> It is primarily a receiver cost issue that motivates the framing
> discussion. The primary sender issue is what, if any, optimizations can
> be done for unmodified TCP send stacks? Note however, that the proposed
> iSCSI requirements are not target or initiator oriented - they are
> sender and receiver oriented. The receiver cost issue applies to either
> a target or initiator - and they each have very different cost
> trade-offs. 
> 
> Thus the best solution is to allow the receive side to control their
> destiny - and require the sender to obey the receiver (within limits).
> If this approach were taken, the sender would have to implement all
> three framing options. Unfortunately, a highly desirable design goal is
> to allow the sender (either the target or the initiator) to run on an
> unmodified TCP stack. Thus the compromise in terms of sender
> functionality. 

The iSCSI 08 draft Appendix C MUST be amended,

from,

"The use of markers is negotiable. The initiator and target MAY
indicate their readiness to receive and/or send markers during login
separately for each connection. The default is NO. In certain
environments a sender not willing to supply markers to a receiver
willing to accept markers MAY suffer from a considerable performance
degradation."

to:

"The use of markers is negotiable. The initiator and target MUST
indicate their readiness to receive and/or send markers during login
separately for each connection. The default is NO. In certain
environments a sender not willing to supply markers to a receiver
willing to accept markers MAY suffer from a considerable performance
degradation."

and in the next paragraph:

"If a receiver indicates that it desires a marker, the
sender SHOULD agree (during negotiation) and provide the marker at
the desired interval."

MUST be changed to:

"If a receiver indicates that it desires a marker, the
sender MUST agree (during negotiation) and provide the marker at
the desired interval."

Dave Sheehy

---

From: Dave Sheehy <[email-masked]>
Date: Tue, 09 Oct 2001 10:01:05 PDT

Julo,

> We went through this several times.
> Once the Framing becomes an RFC (informational) we will adjust the 
> appendix.

I am not at all confident that framing will become an RFC prior to iSCSI
being approved. I'd like to get the iSCSI spec changed now to cover that
possibility.

> Untill then this appendix represents the WG consensus and there no reason 
> to change that.

So, you're saying that the framing team consensus is in conflict with the 
WG consensus?

Dave

> 
> Julo
> 
> 
> 
> 
> 
> Dave Sheehy <[email-masked]>
> Sent by: [email-masked]
> 09-10-01 08:54
> Please respond to Dave Sheehy
> 
>  
>         To:     [email-masked] (IETF IP SAN Reflector)
>         cc: 
>         Subject:        iscsi: 08 Comment, framing
> 
>  
> 
> --=_alternative 004F567E42256AE0_=
> Content-Type: text/html; charset="us-ascii"
> 
> 
> <br><font size=2 face="sans-serif">Dave,</font>
> <br>
> <br><font size=2 face="sans-serif">We went through this several times.</font>
> <br><font size=2 face="sans-serif">Once the Framing becomes an RFC (informational) we will adjust the appendix.</font>
> <br><font size=2 face="sans-serif">Untill then this appendix represents the WG consensus and there no reason to change that.</font>
> <br>
> <br><font size=2 face="sans-serif">Julo</font>
> <br>
> <br>
> <br>
> <br>
> <table width=100%>
> <tr valign=top>
> <td>
> <td><font size=1 face="sans-serif"><b>Dave Sheehy &lt;[email-masked]&gt;</b></font>
> <br><font size=1 face="sans-serif">Sent by: [email-masked]</font>
> <p><font size=1 face="sans-serif">09-10-01 08:54</font>
> <br><font size=1 face="sans-serif">Please respond to Dave Sheehy</font>
> <br>
> <td><font size=1 face="Arial">&nbsp; &nbsp; &nbsp; &nbsp; </font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; To: &nbsp; &nbsp; &nbsp; &nbsp;[email-masked] (IETF IP SAN Reflector)</font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; cc: &nbsp; &nbsp; &nbsp; &nbsp;</font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; Subject: &nbsp; &nbsp; &nbsp; &nbsp;iscsi: 08 Comment, framing</font>
> <br>
> <br><font size=1 face="Arial">&nbsp; &nbsp; &nbsp; &nbsp;</font></table>
> <br>
> <br><font size=2 face="Courier New">Since the following is the consensus of the framing team:<br>
> <br>
> &gt; iSCSI, with minor modifications since the presentation at the IETF:<br>
> &gt; <br>
> &gt; <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;The Sender:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - MUST support no framing<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - MUST support at least one framing solution<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - MUST use the framing specified by the receiver, <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;if it supports that framing mode<br>
> <br>
> and ...<br>
> <br>
> &gt; <br>
> &gt; First, a quick summary of the issues:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- Deploying iSCSI sender can be done:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - on top of an unmodified TCP stack with:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;o No framing<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;o Marker based framing <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; - on top of a modified TCP stack can implement<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;PDU-alignment.<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- The receiver trade-offs are:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; o No framing - large receive reassembly buffer <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;(higher cost solution)<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; o Marker framing - receive reassembly buffer is reduced <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;to an eddy buffer, but requires modifications to<br>
> &gt; <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TCP receive stack (medium cost solution)<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; o PDU-alignment - receive reassembly buffer can be<br>
> &gt; reduced <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;even further, but requires modifications to TCP <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;receive stack (lowest cost solution and enables <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;eventual deployment of a viable chunking protocol).<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- The cost of implementing markers on unmodified TCP stacks<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; o sender cost is acceptable, assuming a gather list is<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;reasonably implemented.<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; o receiver cost is unacceptable<br>
> <br>
> &gt; Initial implementations for initiator appear to be in two camps:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- Unmodified TCP software stacks<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- Embedded TCP offload in the NIC (essentially TCP <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; is hidden from the host SCSI stack)<br>
> &gt; <br>
> &gt; Initial implementations for the target appear to be in two camps:<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- Optimized NICs which will support framing - probably both <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; framing modes. Because both modes<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; require modifications to the receive TCP stack and <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; PDU-alignment is viewed as straightforward, it is <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; assumed that most implementations will implement <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; both framing solutions to allow them to transfer <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; data optimally with either unmodified send TCP stacks <br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; or PDU-alignment send TCP stacks.<br>
> &gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;- Unmodified TCP software stacks<br>
> &gt; <br>
> &gt; It is primarily a receiver cost issue that motivates the framing<br>
> &gt; discussion. The primary sender issue is what, if any, optimizations can<br>
> &gt; be done for unmodified TCP send stacks? Note however, that the proposed<br>
> &gt; iSCSI requirements are not target or initiator oriented - they are<br>
> &gt; sender and receiver oriented. The receiver cost issue applies to either<br>
> &gt; a target or initiator - and they each have very different cost<br>
> &gt; trade-offs. <br>
> &gt; <br>
> &gt; Thus the best solution is to allow the receive side to control their<br>
> &gt; destiny - and require the sender to obey the receiver (within limits).<br>
> &gt; If this approach were taken, the sender would have to implement all<br>
> &gt; three framing options. Unfortunately, a highly desirable design goal is<br>
> &gt; to allow the sender (either the target or the initiator) to run on an<br>
> &gt; unmodified TCP stack. Thus the compromise in terms of sender<br>
> &gt; functionality. <br>
> <br>
> The iSCSI 08 draft Appendix C MUST be amended,<br>
> <br>
> from,<br>
> <br>
> &quot;The use of markers is negotiable. The initiator and target MAY<br>
> indicate their readiness to receive and/or send markers during login<br>
> separately for each connection. The default is NO. In certain<br>
> environments a sender not willing to supply markers to a receiver<br>
> willing to accept markers MAY suffer from a considerable performance</font>
> <br><font size=2 face="Courier New">degradation.&quot;<br>
> <br>
> to:<br>
> <br>
> &quot;The use of markers is negotiable. The initiator and target MUST<br>
> indicate their readiness to receive and/or send markers during login<br>
> separately for each connection. The default is NO. In certain<br>
> environments a sender not willing to supply markers to a receiver<br>
> willing to accept markers MAY suffer from a considerable performance<br>
> degradation.&quot;<br>
> <br>
> and in the next paragraph:<br>
> <br>
> &quot;If a receiver indicates that it desires a marker, the<br>
> sender SHOULD agree (during negotiation) and provide the marker at<br>
> the desired interval.&quot;<br>
> <br>
> MUST be changed to:<br>
> <br>
> &quot;If a receiver indicates that it desires a marker, the<br>
> sender MUST agree (during negotiation) and provide the marker at<br>
> the desired interval.&quot;<br>
> <br>
> Dave Sheehy<br>
> <br>
> </font>
> <br>
> <br>
> --=_alternative 004F567E42256AE0_=--
>

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 10 Oct 2001 00:58:39 +0300

Dave,

This was the consensus of the framing team - not of the IPS list.
It is inappropriate for us to replace a thing that we have and is agreed 
to something that
that is not here yet. 

As I stated if you want to do it independent of the ULP progress you have 
to reopen the issue on this list.

Regards,
Julo

and ...

from,

to:

MUST be changed to:

Dave Sheehy

---

From: [email-masked]
Date: Tue, 9 Oct 2001 20:18:28 -0400

Julian is correct.  In London the framing team consensus did not
translate into WG rough consensus.  Until it does, I think modifications
to the spec are not appropriate.  I suspect Framing is going to be on
the Salt Lake City agenda :-(.  --David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 435-1000 x75140     FAX: +1 (508) 497-8500
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Tuesday, October 09, 2001 5:59 PM
To: [email-masked]
Subject: Re: iscsi: 08 Comment, framing

Dave, 

This was the consensus of the framing team - not of the IPS list. 
It is inappropriate for us to replace a thing that we have and is agreed to
something that 
that is not here yet.   

As I stated if you want to do it independent of the ULP progress you have to
reopen the issue on this list. 

Regards, 
Julo 

09-10-01 08:54 

and ...

from,

to:

MUST be changed to:

Dave Sheehy