Subject: [Ips] [IPS]: Considerations for Asymmetric iSCSI Implementations Draft Interest?

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 17 Nov 2003 10:19:38 -0800

Greetings All,
	
	I am inquiring about interest in seeing a draft and possibly RFC
regarding Considerations for Asymmetric iSCSI Implementations.  I recall
seeing some discussion on this topic back in 2000, and aside from a few
other mentions of the model in various design documents I have not seen
any followups.  I am curious if the model had all but been abandoned for
the symmetric model in traditional and iSER iSCSI, or if there is still
room in the protocol for asymmetric operations?

I have been thinking about this topic for a while and specifically its
affects on ErrorRecoveryLevel=2 implemenatations and wanted to get some
feedback from list members if it would be worthwhile for someone (mabye
even me!) to put the thoughts down for other interested parties to
consider.

Aside from the expected responses of complexity issues for simple
implementations,  I do not see a terrible amount of extra functionality
required for ErrorRecoveryLevel=1 and above implementations to operate
in an Asymmetric model, and literally no changes to RFC 3347.  As to the
benefits of running in a Asymmetric model I can only speculate, 
although I suppose most are related performance in multiple connection
ErrorRecoveryLevel=1 and above software implementations.

Any feedback is welcome!	

-- 
Nicholas A. Bellinger <[email-masked]>

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: John Hufferd <[email-masked]>
Date: Tue, 18 Nov 2003 00:30:00 -0800

Well until I see some compelling numbers, we will probably just continue 
to disagree.  But I think compelling numbers is what is needed before we 
would think about opening this up.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Nicholas A. Bellinger" <[email-masked]>
Sent by: [email-masked]
11/17/2003 10:17 PM

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     [email-masked], [email-masked], Julian Satran 
<[email-masked]>, [email-masked]
        Subject:        RE: [Ips] : Considerations for Asymmetric iSCSI Implementations Draft 
Interest?

John,

On Mon, 2003-11-17 at 17:06, John Hufferd wrote:
> Yes, it is anyone's guess what the performance improvement (if any)
> there would be. But you are saying that with specialty engines, you
> could go faster.  On the other hand many of the implementations I have
> seen have speciality engines and speciality paths.  In your example,
> if I understood it correctly, you have 4 connections for the
> Asymmetric case.  But you don't seem to talk about that with your
> Symmetric case.
> 

Point taken in the symmetric case.  The goal was to show operations
based with multiple connections.   The same basic operation would occur
in the symmetric example with four connections.

> Also, you talk about the CmdSN window, but it seemed that you
> specified it as a connection based item.  However, since it is a
> Session Wide value, any of the example's four connections would be
> able to start transferring the very next command.  So there would be
> NO hold up waiting to send the command on another connection, as long
> as a connection was freed.  And since that is basically the same
> limitation for the Asymmetric case (a connection must be free to send
> data, or anything), I do not see the problem, let alone the fix.
> 

The CmdSN was used in context of each control connection or normal
connection TX thread checking the session wide CmdSN window for an
opening to send non immediate commands, sorry for the confusion.  The
delay I was trying to convey in the symmetric example is not a session
wide delay of other connections unable to send non immediate commands
when the CmdSN window opens, but other connections being busy
fullfilling interconnection requests such as Unsolicited DataOut.

> Remember you can have Multiple Connections per Session (MC/S), between
> physically different Portals, or you can have MC/S within a single
> Physical Portal via multiple logical connections.  So everything you
> want to do is doable, and without a clear reason why an Asymmetric
> case is better, I can not see us dealing with it.  (Some deep math is
> probably needed here.) 
> .
> .

I am not questioning any of these points,  the main concept being
addressed here with control and data connections is keeping each of them
as busy as possible doing their specifics jobs:

Control Connection: Checking CmdSN window openings for sending non
immediate commands. ie: a single session wide request queue.

Data Connection: Checking for pending Unsolicited or Solicited DataOUT.
ie: a single connection wide request queue.

Compared to:

Normal Connection: At absolute least two queues.  One checking for
pending Unsolicited or Solicited DataOUT requests (connection wide), and
one for checking openings in the CmdSN window for sending non immediate
commands (session wide).
(I believe the IBM iSCSI Initiator for Linux based on draft v8 had three
queues.  Immediate (connection wide), R2T (connection wide), and session
wide queue checked when the CmdSN window opened).

Back to the performance question,  I completely agree.  It is anyones
guess and will remain that until someone comes up with numbers for
various cases. (It could very well be difficult to beat symmetric
ImmediateData performance when the entire request size fits in
FirstBurstLength, as good implementations do today).  In any event I
still firmly believe that an asymmetric model will scale futher and have
a better cache hit ratio than its symmetric couterpart in the pure
software case.

Thanks!

> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/System Group, San Jose CA
> Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
> Alt Office: (408) 997-6136, Cell: (408) 499-9702
> Internet Address: [email-masked]
> 
> 
> 
> "Nicholas A. Bellinger"
> <[email-masked]>
> 
> 11/17/2003 02:42 PM
> 
>         To:        John
> Hufferd/San
> Jose/IBM@IBMUS
>         cc: 
> [email-masked],
> [email-masked],
> Julian Satran
> <[email-masked]>, [email-masked]
>         Subject: 
> RE: [Ips] :
> Considerations for
> Asymmetric iSCSI
> Implementations 
> Draft Interest?
> 
> 
> John,
> 
> I can think of a few reasons while trying not to go too deep into
> implementation specific details.  These have to do with my idea of
> achieving maximum parallelism,  which probably is not the same as
> other
> peoples. Again these are purely speculation. (speculation == more
> threads doing simpler operations will be faster as connection count
> increases)
> 
> Non Immedate Unsolicited DataOUT Case, assume FirstBurstLength >= SCSI
> Request size.
> 
> 
> Asymmetric:
> 
> 2 Control Channels and 2 Data Channels:
> CID 0 and 1: Control Channels 0 and 1
> CID 2 and 3: Data Channels 0 and 1
> 
> Only operation is checking CmdSN window to send commands: 
> 
> 0) TX threads for CID 0 and 1 checking CmdSN window to send non
> immediate commands.
> 1) CmdSN window opens for CID 0, iSCSI command becomes alligent to CID
> 0.
> 2) iSCSI Scsi Cmnd Opcode is built on CID 0 and unsolicited DataOUT
> request gets added to CID 2 from CID 0.
> 3) A and B can happen in any order:
> 3A) iSCSI Scsi Cmnd Opcode gets sent on CID 0.
> 3B) iSCSI Unsolicited DataOUT request gets queued and Unsolicited
> DataOUT is built on CID 2.
> 4) iSCSI Unsolicited DataOUT is sent on CID 2 after iSCSI Scsi Cmnd
> Opcode is sent on CID 0.
> 5) TX thread for CID 1 detects open CmdSN window and the process
> starts
>   over again with CID 1 and CID 3.
> 
> Problem: 
> Latency issue of iSCSI Unsolicited DataOUT arriving before iSCSI Scsi
> Cmnd Opcode.
> 
> 
> Symmetric: 
> 
> 2 iSCSI Connections:
> CID 0 and 1
> 
> 0) TX threads for CID 0 and 1 checking CmdSN window to send non
> immediate commands.
> 1) CmdSN window opens for CID 0, iSCSI command becomes alligent to CID
> 0.
> 2) iSCSI Scsi Cmnd Opcode is built on CID 0.
> 3) iSCSI Scsi Cmnd Opcode is sent on CID 0, and adds Unsolicited
> DataOUT
> request to CID 0 queue.
> 4) Either A or B may happen next (matter of implementation)
> 4A) CID 0 builds Unsolicited DataOUT.
> 4B) CID 0 detects CmdSN window is open and sends more non-immediate
> commands, only until after CmdSN window closes does it send
> Unsolicited
> DataOUT.
> 5) TX thread for CID 1 detects CmdSN window and the process starts
> over
> again with CID 1.
> 
> Problem:
> CID 0 having to build and send DataOUT while it could be sending
> another
> iSCSI Scsi Cmnd Opcode while the CmdSN window is open.
> 
> 
> This is just one scenario off the top of my head and could be written
> off as a matter of implementation,  but the asymmetric goal is to
> achieve maximum parallelism by A) threads doing smaller specialized
> jobs, and B) Control threads queuing up more iSCSI Scsi Cmnds faster
> to
> Data Threads checking a single queue.  As far as performance
> percentage
> improvement I cannot begin to dream up numbers until an asymmetric
> prototype is complete, and very well could only show improvements in
> specific configurations,  but it is anyones guess.
> 
> Thanks!
> 
> --
> On Mon, 2003-11-17 at 13:26, John Hufferd wrote:
> > I am not sure why we would ever want to do this.  How does this
> > improve the performance over normal Multiple Connections per
> Session?
> > And what is the projected improvement (%)?
> > .
> > .
> > John L. Hufferd
> > Senior Technical Staff Member (STSM)
> > IBM/System Group, San Jose CA
> > Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
> > Alt Office: (408) 997-6136, Cell: (408) 499-9702
> > Internet Address: [email-masked]
> > 
> > 
> > 
> > "Nicholas A. Bellinger"
> > <[email-masked]>
> > Sent by:
> > [email-masked]
> > 
> > 11/17/2003 12:01 PM
> > 
> >         To: 
> > [email-masked]
> >         cc: 
> > [email-masked], Julian
> > Satran
> > <[email-masked]>
> >         Subject: 
> > RE: [Ips] [IPS]:
> > Considerations for
> > Asymmetric iSCSI
> > Implementati        ons
> > Draft Interest?
> > 
> > 
> > Pat,
> > 
> >
> http://www.haifa.il.ibm.com/Workshops/Storage2002/papers/iscsiDesign.pdf
> > 
> > The following paper from Kalman Meth decribes them on page 12:
> > 
> > Asymmetric:
> > Single Control Channel / iSCSI Connection.
> > 
> > Multiple Data Channels / iSCSI Connection.
> > 
> > Control Channel used to transfer commands, status, and task
> > management.
> > 
> > Symmetric:
> > All Channels / iSCSI Connections identical.
> > 
> > Send Data and Status over same Channel / iSCSI Connection as 
> > corresponding command.
> > 
> > 
> > 
> > I am not exactly sure why the single control channel restriction was
> > imposed in the asymmetric case, but imagine its no longer an issue
> to
> > have multiple control channels.  Perhaps this was before the advent
> of
> > session wide CmdSN ordering?
> > 
> > Some of the more interesting questions:
> > 
> > Connection recovery case, which connection an inflight command is
> > alligent to when the control and/or data connection fails.
> > 
> > Requirements for falling back when an session runs out of control
> > and/or
> > data channels.  Or is it just easier to do session reinstatement
> into
> > an symmetric session.
> > 
> > How strict are the requirements for data on the control channel and
> > control primatives in the data channel?  Immediate DataOut comes to
> > mind
> > in the former case, and Phase Collapse for DataIn in the latter.
> > 
> > Thanks!
> > 
> > On Mon, 2003-11-17 at 11:17, [email-masked] wrote:
> > > I'm not clear on what you mean by "asymetric".
> > > 
> > > -----Original Message-----
> > > From: Nicholas A. Bellinger [mailto:[email-masked]]
> > > Sent: Monday, November 17, 2003 10:20 AM
> > > To: [email-masked]; [email-masked]
> > > Subject: [Ips] [IPS]: Considerations for Asymmetric iSCSI
> > > Implementations Draft Interest?
> > > 
> > > 
> > > Greetings All,
> > > 
> > >                  I am inquiring about interest in seeing a draft
> and
> > possibly RFC
> > > seeing some discussion on this topic back in 2000, and aside from
> a
> > few
> > > any followups.  I am curious if the model had all but been
> abandoned
> > for
> > > in an Asymmetric model, and literally no changes to RFC 3347.  As
> to
> > the