Subject: RE: [Ips] : Considerations for Asymmetric iSCSI Implementations Draft Interest?

From: John Hufferd <[email-masked]>
Date: Mon, 17 Nov 2003 17:06:24 -0800

Yes, it is anyone's guess what the performance improvement (if any) there 
would be. But you are saying that with specialty engines, you could go 
faster.  On the other hand many of the implementations I have seen have 
speciality engines and speciality paths.  In your example, if I understood 
it correctly, you have 4 connections for the Asymmetric case.  But you 
don't seem to talk about that with your Symmetric case.

Also, you talk about the CmdSN window, but it seemed that you specified it 
as a connection based item.  However, since it is a Session Wide value, 
any of the example's four connections would be able to start transferring 
the very next command.  So there would be NO hold up waiting to send the 
command on another connection, as long as a connection was freed.  And 
since that is basically the same limitation for the Asymmetric case (a 
connection must be free to send data, or anything), I do not see the 
problem, let alone the fix.

Remember you can have Multiple Connections per Session (MC/S), between 
physically different Portals, or you can have MC/S within a single 
Physical Portal via multiple logical connections.  So everything you want 
to do is doable, and without a clear reason why an Asymmetric case is 
better, I can not see us dealing with it.  (Some deep math is probably 
needed here.) 
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: [email-masked]

"Nicholas A. Bellinger" <[email-masked]>
11/17/2003 02:42 PM

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     [email-masked], [email-masked], Julian Satran 
<[email-masked]>, [email-masked]
        Subject:        RE: [Ips] : Considerations for Asymmetric iSCSI Implementations Draft 
Interest?

John,

I can think of a few reasons while trying not to go too deep into
implementation specific details.  These have to do with my idea of
achieving maximum parallelism,  which probably is not the same as other
peoples. Again these are purely speculation. (speculation == more
threads doing simpler operations will be faster as connection count
increases)

Non Immedate Unsolicited DataOUT Case, assume FirstBurstLength >= SCSI
Request size.

Asymmetric:

2 Control Channels and 2 Data Channels:
CID 0 and 1: Control Channels 0 and 1
CID 2 and 3: Data Channels 0 and 1

Only operation is checking CmdSN window to send commands: 

0) TX threads for CID 0 and 1 checking CmdSN window to send non
immediate commands.
1) CmdSN window opens for CID 0, iSCSI command becomes alligent to CID
0.
2) iSCSI Scsi Cmnd Opcode is built on CID 0 and unsolicited DataOUT
request gets added to CID 2 from CID 0.
3) A and B can happen in any order:
3A) iSCSI Scsi Cmnd Opcode gets sent on CID 0.
3B) iSCSI Unsolicited DataOUT request gets queued and Unsolicited
DataOUT is built on CID 2.
4) iSCSI Unsolicited DataOUT is sent on CID 2 after iSCSI Scsi Cmnd
Opcode is sent on CID 0.
5) TX thread for CID 1 detects open CmdSN window and the process starts
   over again with CID 1 and CID 3.

Problem: 
Latency issue of iSCSI Unsolicited DataOUT arriving before iSCSI Scsi
Cmnd Opcode.

Symmetric: 

2 iSCSI Connections:
CID 0 and 1

0) TX threads for CID 0 and 1 checking CmdSN window to send non
immediate commands.
1) CmdSN window opens for CID 0, iSCSI command becomes alligent to CID
0.
2) iSCSI Scsi Cmnd Opcode is built on CID 0.
3) iSCSI Scsi Cmnd Opcode is sent on CID 0, and adds Unsolicited DataOUT
request to CID 0 queue.
4) Either A or B may happen next (matter of implementation)
4A) CID 0 builds Unsolicited DataOUT.
4B) CID 0 detects CmdSN window is open and sends more non-immediate
commands, only until after CmdSN window closes does it send Unsolicited
DataOUT.
5) TX thread for CID 1 detects CmdSN window and the process starts over
again with CID 1.

Problem:
CID 0 having to build and send DataOUT while it could be sending another
iSCSI Scsi Cmnd Opcode while the CmdSN window is open.

This is just one scenario off the top of my head and could be written
off as a matter of implementation,  but the asymmetric goal is to
achieve maximum parallelism by A) threads doing smaller specialized
jobs, and B) Control threads queuing up more iSCSI Scsi Cmnds faster to
Data Threads checking a single queue.  As far as performance percentage
improvement I cannot begin to dream up numbers until an asymmetric
prototype is complete, and very well could only show improvements in
specific configurations,  but it is anyones guess.

Thanks!

--
Nicholas A. Bellinger <[email-masked]>

On Mon, 2003-11-17 at 13:26, John Hufferd wrote:
> I am not sure why we would ever want to do this.  How does this
> improve the performance over normal Multiple Connections per Session?
> And what is the projected improvement (%)?
> .
> .
> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/System Group, San Jose CA
> Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
> Alt Office: (408) 997-6136, Cell: (408) 499-9702
> Internet Address: [email-masked]
> 
> 
> 
> "Nicholas A. Bellinger"
> <[email-masked]>
> Sent by:
> [email-masked]
> 
> 11/17/2003 12:01 PM
> 
>         To: 
> [email-masked]
>         cc: 
> [email-masked], Julian
> Satran
> <[email-masked]>
>         Subject: 
> RE: [Ips] [IPS]:
> Considerations for
> Asymmetric iSCSI
> Implementati        ons
> Draft Interest?
> 
> 
> Pat,
> 
> http://www.haifa.il.ibm.com/Workshops/Storage2002/papers/iscsiDesign.pdf
> 
> The following paper from Kalman Meth decribes them on page 12:
> 
> Asymmetric:
> Single Control Channel / iSCSI Connection.
> 
> Multiple Data Channels / iSCSI Connection.
> 
> Control Channel used to transfer commands, status, and task
> management.
> 
> Symmetric:
> All Channels / iSCSI Connections identical.
> 
> Send Data and Status over same Channel / iSCSI Connection as 
> corresponding command.
> 
> 
> 
> I am not exactly sure why the single control channel restriction was
> imposed in the asymmetric case, but imagine its no longer an issue to
> have multiple control channels.  Perhaps this was before the advent of
> session wide CmdSN ordering?
> 
> Some of the more interesting questions:
> 
> Connection recovery case, which connection an inflight command is
> alligent to when the control and/or data connection fails.
> 
> Requirements for falling back when an session runs out of control
> and/or
> data channels.  Or is it just easier to do session reinstatement into
> an symmetric session.
> 
> How strict are the requirements for data on the control channel and
> control primatives in the data channel?  Immediate DataOut comes to
> mind
> in the former case, and Phase Collapse for DataIn in the latter.
> 
> Thanks!
> 
> On Mon, 2003-11-17 at 11:17, [email-masked] wrote:
> > I'm not clear on what you mean by "asymetric".
> > 
> > -----Original Message-----
> > From: Nicholas A. Bellinger [mailto:[email-masked]]
> > Sent: Monday, November 17, 2003 10:20 AM
> > To: [email-masked]; [email-masked]
> > Subject: [Ips] [IPS]: Considerations for Asymmetric iSCSI
> > Implementations Draft Interest?
> > 
> > 
> > Greetings All,
> > 
> >                  I am inquiring about interest in seeing a draft and
> possibly RFC
> > regarding Considerations for Asymmetric iSCSI Implementations.  I
> recall
> > seeing some discussion on this topic back in 2000, and aside from a
> few
> > other mentions of the model in various design documents I have not
> seen
> > any followups.  I am curious if the model had all but been abandoned
> for
> > the symmetric model in traditional and iSER iSCSI, or if there is
> still
> > room in the protocol for asymmetric operations?
> > 
> > I have been thinking about this topic for a while and specifically
> its
> > affects on ErrorRecoveryLevel=2 implemenatations and wanted to get
> some
> > feedback from list members if it would be worthwhile for someone
> (mabye
> > even me!) to put the thoughts down for other interested parties to
> > consider.
> > 
> > Aside from the expected responses of complexity issues for simple
> > implementations,  I do not see a terrible amount of extra
> functionality
> > required for ErrorRecoveryLevel=1 and above implementations to
> operate
> > in an Asymmetric model, and literally no changes to RFC 3347.  As to
> the
> > benefits of running in a Asymmetric model I can only speculate, 
> > although I suppose most are related performance in multiple
> connection
> > ErrorRecoveryLevel=1 and above software implementations.
> > 
> > Any feedback is welcome! 
> > 
> > -- 
> > Nicholas A. Bellinger <[email-masked]>
> > 
> > 
> > _______________________________________________
> > Ips mailing list
> > [email-masked]
> > https://www1.ietf.org/mailman/listinfo/ips
> -- 
> Nicholas A. Bellinger <[email-masked]>
> 
> 
> _______________________________________________
> Ips mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ips

---

From: "Nicholas A. Bellinger" <[email-masked]>
Date: 17 Nov 2003 22:17:01 -0800

John,

On Mon, 2003-11-17 at 17:06, John Hufferd wrote:
> Yes, it is anyone's guess what the performance improvement (if any)
> there would be. But you are saying that with specialty engines, you
> could go faster.  On the other hand many of the implementations I have
> seen have speciality engines and speciality paths.  In your example,
> if I understood it correctly, you have 4 connections for the
> Asymmetric case.  But you don't seem to talk about that with your
> Symmetric case.
> 

Point taken in the symmetric case.  The goal was to show operations
based with multiple connections.   The same basic operation would occur
in the symmetric example with four connections.

> Also, you talk about the CmdSN window, but it seemed that you
> specified it as a connection based item.  However, since it is a
> Session Wide value, any of the example's four connections would be
> able to start transferring the very next command.  So there would be
> NO hold up waiting to send the command on another connection, as long
> as a connection was freed.  And since that is basically the same
> limitation for the Asymmetric case (a connection must be free to send
> data, or anything), I do not see the problem, let alone the fix.
> 

The CmdSN was used in context of each control connection or normal
connection TX thread checking the session wide CmdSN window for an
opening to send non immediate commands, sorry for the confusion.  The
delay I was trying to convey in the symmetric example is not a session
wide delay of other connections unable to send non immediate commands
when the CmdSN window opens, but other connections being busy
fullfilling interconnection requests such as Unsolicited DataOut.

> Physical Portal via multiple logical connections.  So everything you
> want to do is doable, and without a clear reason why an Asymmetric
> case is better, I can not see us dealing with it.  (Some deep math is
> probably needed here.) 
> .
> .

I am not questioning any of these points,  the main concept being
addressed here with control and data connections is keeping each of them
as busy as possible doing their specifics jobs:

Control Connection: Checking CmdSN window openings for sending non
immediate commands. ie: a single session wide request queue.

Data Connection: Checking for pending Unsolicited or Solicited DataOUT.
ie: a single connection wide request queue.

Compared to:

Normal Connection: At absolute least two queues.  One checking for
pending Unsolicited or Solicited DataOUT requests (connection wide), and
one for checking openings in the CmdSN window for sending non immediate
commands (session wide).
(I believe the IBM iSCSI Initiator for Linux based on draft v8 had three
queues.  Immediate (connection wide), R2T (connection wide), and session
wide queue checked when the CmdSN window opened).

Back to the performance question,  I completely agree.  It is anyones
guess and will remain that until someone comes up with numbers for
various cases. (It could very well be difficult to beat symmetric
ImmediateData performance when the entire request size fits in
FirstBurstLength, as good implementations do today).  In any event I
still firmly believe that an asymmetric model will scale futher and have
a better cache hit ratio than its symmetric couterpart in the pure
software case.

Thanks!

>         To:        John
> Hufferd/San
> Jose/IBM@IBMUS
> [email-masked],
> [email-masked],
> Julian Satran
> RE: [Ips] :