Subject: RE: [Ips] [IPS]: Considerations for Asymmetric iSCSI Implementati ons Draft Interest?

From: [email-masked]
Date: Mon, 17 Nov 2003 12:17:02 -0700

I'm not clear on what you mean by "asymetric".

-----Original Message-----
From: Nicholas A. Bellinger [mailto:[email-masked]]
Sent: Monday, November 17, 2003 10:20 AM
To: [email-masked]; [email-masked]
Subject: [Ips] [IPS]: Considerations for Asymmetric iSCSI
Implementations Draft Interest?

Greetings All,
	
	I am inquiring about interest in seeing a draft and possibly RFC
regarding Considerations for Asymmetric iSCSI Implementations.  I recall
seeing some discussion on this topic back in 2000, and aside from a few
other mentions of the model in various design documents I have not seen
any followups.  I am curious if the model had all but been abandoned for
the symmetric model in traditional and iSER iSCSI, or if there is still
room in the protocol for asymmetric operations?

I have been thinking about this topic for a while and specifically its
affects on ErrorRecoveryLevel=2 implemenatations and wanted to get some
feedback from list members if it would be worthwhile for someone (mabye
even me!) to put the thoughts down for other interested parties to
consider.

Aside from the expected responses of complexity issues for simple
implementations,  I do not see a terrible amount of extra functionality
required for ErrorRecoveryLevel=1 and above implementations to operate
in an Asymmetric model, and literally no changes to RFC 3347.  As to the
benefits of running in a Asymmetric model I can only speculate, 
although I suppose most are related performance in multiple connection
ErrorRecoveryLevel=1 and above software implementations.

Any feedback is welcome!	

-- 
Nicholas A. Bellinger <[email-masked]>

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Andre Hedrick <[email-masked]>
Date: Mon, 17 Nov 2003 11:42:22 -0800 (PST)

Pat:

A simple method of spliting the command and data transports over their own
respective connections.  It is clear the command layer transactions will
arrive before the following data transactions.

There are obvious reason to split the layers, yet they generate a new
level of error recovery.

The principle reason to split the two is based on performance by allowing
the target to prebuild buffers to optimize memory management and io
scheduling events.

PyX will present such a model for the next version of the iSCSI RFC.
There are no patents involved, and this conforms to the goals of IETF.

Regards,

Andre

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 17 Nov 2003 12:01:57 -0800

Pat,

http://www.haifa.il.ibm.com/Workshops/Storage2002/papers/iscsiDesign.pdf

The following paper from Kalman Meth decribes them on page 12:

Asymmetric:
Single Control Channel / iSCSI Connection.

Multiple Data Channels / iSCSI Connection.

Control Channel used to transfer commands, status, and task management.

Symmetric:
All Channels / iSCSI Connections identical.

Send Data and Status over same Channel / iSCSI Connection as  
corresponding command.

I am not exactly sure why the single control channel restriction was
imposed in the asymmetric case, but imagine its no longer an issue to
have multiple control channels.  Perhaps this was before the advent of
session wide CmdSN ordering?

Some of the more interesting questions:

Connection recovery case, which connection an inflight command is
alligent to when the control and/or data connection fails.

Requirements for falling back when an session runs out of control and/or
data channels.  Or is it just easier to do session reinstatement into
an symmetric session.

How strict are the requirements for data on the control channel and
control primatives in the data channel?  Immediate DataOut comes to mind
in the former case, and Phase Collapse for DataIn in the latter.

Thanks!

---

From: Paul Koning <[email-masked]>
Date: Mon, 17 Nov 2003 16:41:51 -0500

>>>>> "Andre" == Andre Hedrick <[email-masked]> writes:

 Andre> Pat:

 Andre> A simple method of spliting the command and data transports
 Andre> over their own respective connections.  It is clear the
 Andre> command layer transactions will arrive before the following
 Andre> data transactions.

Is it?  If immediate or unsolicited data flows over data connections,
then this doesn't seem to be true.

 Andre> There are obvious reason to split the layers...

Do you have examples of such reasons?

   paul

---

From: Andre Hedrick <[email-masked]>
Date: Mon, 17 Nov 2003 14:02:25 -0800 (PST)

> then this doesn't seem to be true.

You missed the point of a possible addition of error recovery and in a
split path model, the immediate/unsolicited data are points which
conflict.  These may be excepts or a new behavior to be set into a new
FSM.

> Do you have examples of such reasons?

Simple rules of multi path or raid extentions requiring specific
sizes/break points of memory buffers for host side operations

There are many reasons on the storage side to have earily knowledge of the
expected incoming command.  Surely it is obvious the huge assist a target
would gain in managing transactions and commits to real/virtual platters.
Multi path operations gains are potentially huge.

Well maybe it is not time until everyone has deployed ERL 2.

Cheers,

Andre

---

From: "Mark S. Edwards" <[email-masked]>
Date: Mon, 17 Nov 2003 22:24:11 +0000

I can't access the doc you reference, but I believe that it is very old and 
part of the idea bouncing during the early stages of iSCSI.

I've been trying to remember the various discussions at the time as to the 
pro's and con's of the different models, but I do remember that I was quite 
opposed to this one largely due to the pain of what happens if one or other 
of the control or data plane fabrics breaks.  Remember, any IETF based 
solution MUST be scalable to the Internet.  Stuff that works when two hosts 
are side by side in a data centre has a habit of biting back when they are 
on opposite sides of the World.

The earliest reference I have in my mail store goes back April 2001 and the 
Last Call discussions on the requirements document.  There were a couple of 
emails asking Marjorie and team to remove the connection model discussions.

This was particularly apt from Bob Snively and also tells you where to 
trawl for further enlightenment on the original discussions, presumably 
there may be some IPS meeting notes on this in there, too.

>6)  ALTERNATE CONNECTION BINDING
>
>         The section in 2.4 discussing an alternate mechanism for
>         connection binding merely serves to weaken the stand
>         in favor of the selected binding relationship.  The following
>         text should be deleted:
>
>    "An alternate approach that was extensively discussed involved
>    sending all commands on a single connection and the associated data
>    and status on a different connection (asymetric approach). In this
>    scheme, the transport ensures the commands arrive in order. The
>    protocol on the data and status connections is simpler, perhaps
>    lending itself to a simpler realization in hardware.  One
>    disadvantage of this approach is that the recovery procedure is
>    different if a command connection fails vs. a data connection. Some
>    argued that this approach would require greater inter-processor
>    communication when connections are spread across processors.
>    The reader may reference the mail archives of the IPS mailing list
>    between June and September of 2000 for extensive discussions on
>    symmetric vs asymmetric connection models."

regards,

Mark.

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 17 Nov 2003 15:05:50 -0800

Mark,

> on opposite sides of the World.

What I am proposing is not a requirement for iSCSI, but an optional
extension for the high speed ErrorRecoveryLevel=1 and above software
implementations which would most likely be used in enviroment you
mention above.

> >    symmetric vs asymmetric connection models."

Also,  the single control channel approach was to ensure SCSI transport
ordering before the advent of session wide CmdSN ordering for non
immediate commands.  What I am proposing is multiple control channels
following Session wide CmdSN ordering rules to stay in line with T10.

In the connection recovery case it is almost certain that a command is
alligent to a control channel and not data channel.  Granted there are a
few issues to consider when one or the other fails for a inflight
command,  but nothing above and beyond what a ErrorRecoveryLevel=2
implementation has already had to consider in normal symmetric
operation.

Thanks!

---

From: Julian Satran <[email-masked]>
Date: Tue, 18 Nov 2003 10:57:31 +0200

I doubt that you will find enough interest in the (IETF) community to 
pursue this seriously.
We did try this path (not very long nor very deep) because it felt 
promising and with a single active control channel we could ahve avoided 
command numbering and eliminate the usual head-of-queue blocking by large 
data transfers usually associated with symmetric solutions.

There where a number of reasons why the "community" felt at the time that 
this is not worth pursuing - complexity, ability to operate (in case of 
hardware adapters) across card/chip boundaries, lack of compelling 
advantages (although many of us did not feel that way). We did even 
discuss the very attractive fact the control and data paths could be 
targeted to different units. After a while we felt that it is more 
important to make progress that to pursue both the asymmetric and the 
symmetric solutions - and the rest is history.

Julo

Andre Hedrick <[email-masked]> 
17/11/2003 21:42

To
[email-masked]
cc
[email-masked], [email-masked], Julian Satran/Haifa/IBM@IBMIL
Subject
RE: [Ips] [IPS]: Considerations for Asymmetric iSCSI Implementati ons 
Draft Interest?

Pat:

Regards,

Andre

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 19 Nov 2003 11:09:27 -0600

> On Mon, 17 Nov 2003 [email-masked] wrote:
Wouldn't either iSER (iSCSI over iWarp) or iSCSI over SCTP
provide the same benefits of splitting control and data traffic
without the complexities of dealing with failures on a subset
of the connections?

-- 
Caitlin Bestler - [email-masked] - http://asomi.com/
http://asomi.com/CaitlinBestlerPublicPgpKey.html