Subject: RE: iscsi : DataPDULength can differ in each direction.

From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>
Date: Wed, 3 Oct 2001 20:56:10 +0200

All,

Yes, this is a much better solution unless the implementation has some
problem in sending certain sizes of dataPduLength :-), so if initiator has
problem in sending PDUs of 16K which is required dataPDULength of target
then even this procedure will be a failure and it would be better to do LIST
negotiation.

Sanjeev

-----Original Message-----
From: Robert Snively [mailto:[email-masked]]
Sent: Wednesday, October 03, 2001 8:15 PM
To: 'Santosh Rao'; Dev - Platys; Julian Satran
Cc: Sandeep Joshi; [email-masked]
Subject: RE: iscsi : DataPDULength can differ in each direction.

I would have thought that it was indicating the DataPDULength it
was capable of accepting, committing the other port to not
exceeding that value.

> -----Original Message-----
> From: Santosh Rao [mailto:[email-masked]]
> Sent: Wednesday, October 03, 2001 10:32 AM
> To: Dev - Platys; Julian Satran
> Cc: Sandeep Joshi; [email-masked]
> Subject: iscsi : DataPDULength can differ in each direction.
> 
> 
> Deva, Julian & All,
> 
> I think we have a more fundamental problem here, that spans beyond the
> issue of which negotiation model should be used for numerical & binary
> key negotiation. This problem needs to be addressed seperately.
> 
> The DataPDULength can and should be allowed to be different in each
> direction. I -> T direction PDUs should be allowed to use a different
> PDU Length than T -> I direction PDUs. 
> 
> Each side should be allowed to specify the DataPDULength it will be
> using and there should be no attempt to negotiate this value. 
>

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 3 Oct 2001 22:00:16 +0300

Santosh,

From an academics point of view I tend to agree with you and I played with 
the of having 2 different negotiated values.
On practical terms however those sizes are determined slightly by the 
connection characteristics and more by the buffering scheme used by most 
cache/buffer-managers. The later tend to use uniform sizes and a common 
pool.  Moreover framing mechanisms will align them to MTU.   It does not 
make too much sense to use 2 values.

Julo

Santosh Rao <[email-masked]>
Sent by: [email-masked]
03-10-01 19:31
Please respond to Santosh Rao

        To:     Dev - Platys <[email-masked]>, Julian Satran/Haifa/IBM@IBMIL
        cc:     Sandeep Joshi <[email-masked]>, [email-masked]
        Subject:        iscsi : DataPDULength can differ in each direction.

Deva, Julian & All,

Each side should be allowed to specify the DataPDULength it will be
using and there should be no attempt to negotiate this value. Rather,
the key is exchanged in either direction to inform the other side as to
what sized PDUs it should expect.

This is somwhat akin to a protocol's MTU definition or a max_frame_size
definition (ex : FC PLOGI Receive Data Field Size in its common service
parameters.). THese values are allowed to be different going in each
direction, allowing for implementation specific quirks that restrict the
use of certain values. (ex : some DMA bugs may require implementations
to have their DataPDULength to be a cache line size shorter, etc.)

I suggest the definition of the DataPDULength key be re-phrased taking
the above into account.

Regards,
Santosh

Dev - Platys wrote:
> 
> Santosh,
> 
> The point is, if the target is not capable of supporting 16K (it is just 
an
> example that is being discussed), then it indicates that is what it 
wants.
> Initiator can close the connection. (No reject is necessary though).
> 
> However, if the target knows it can support the initiator negotiated 
value,
> it could return it.
> 
> Again, Yes, implicitly it becomes the responder selects logic.
> 
> Deva
> Adaptec
> 
> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Sandeep Joshi
> Sent: Wednesday, October 03, 2001 7:29 AM
> To: [email-masked]
> Subject: Re: iscsi : numerical negotiation wording is ambiguous
> 
> The first & third example below seem illogical.
> 
> The responder should not be sending 24K if the initial
> sender has chosen a value of 16K, since there is no
> possibility of that 24K being accepted (unless we want
> three way exchanges for every negotiation??)
> 
> The problem of a Reject message does not arise since the
> responder should only be choosing something less than 16K.
> So I guess this puts me in the "responder selects logic"
> camp.
> 
> -Sandeep
> 
> Julian Satran wrote:
> >
> > An example:
> >  08.txt logic
> > i->t DataPDULength=16k
> > t->i DataPDULength= 24k
> >
> > Selected value is 16k
> >
> > i->t DataPDULength=16k
> > t->i DataPDULength= 8k
> >
> > Selected value is 8k
> >
> > t->i DataPDULength=16k
> > i->t DataPDULength= 24k
> >
> > Selected value is 16k
> >
> > t->i DataPDULength=16k
> > i->t DataPDULength= 8k
> >
> > Selected value is 8k
> >
> > -----
> >
> > Responder selects logic
> >
> > i->t DataPDULength=16k
> > t->i DataPDULength= 24k
> >
> > Error - Initiator Reject - Closes connection
> >
> > i->t DataPDULength=16k
> > t->i DataPDULength= 8k
> >
> > Selected value is 8k
> >
> > t->i DataPDULength=16k
> > i->t DataPDULength= 24k
> >
> > Error Target has to terminate with parameter error
> >
> > t->i DataPDULength=16k
> > i->t DataPDULength= 8k
> >
> > Selected value is 8k
> >
> >   "Dev - Platys"
> >   <[email-masked]>                To:        "Julian Satran"
> >                            <[email-masked]>,
> >   01-10-01 23:46           <[email-masked]>
> >   Please respond to "Dev -         cc:
> >   Platys"                          Subject:        RE: iscsi :
> >                            numerical negotiation wording is ambiguous
> >
> >
> >
> > Julian,
> >
> > I am not sure why a 'REJECT' is required.
> > Can you please explain why is this required?
> >
> > I am with Santosh in this.
> >
> > >There is NO need for any REJECT in the above >case. If the initiator
> > >is'nt satisfied with the value returned by the >originator and cannot
> > >function with the negotiated values, it can simply >close the TCP
> > >connection. There is no need to send any >REJECT.
> >
> > Thanks
> >
> > Deva
> > Adaptec
> >
> > From: [email-masked] [mailto:[email-masked]]On Behalf Of
> > Julian Satran
> > Sent: Monday, October 01, 2001 1:28 PM
> > To: [email-masked]
> > Subject: Re: iscsi : numerical negotiation wording is ambiguous
> >
> > Santosh,
> >
> > We are getting nowhere. Even if requester is doing it's stuff - the
> > requester will have to check and be prepared for a mistake. The coding
> > will require a reject.
> >
> > Julo
> >
> >   Santosh Rao
> >   <[email-masked]>        To:        [email-masked]
> >   Sent by:                     cc:        "Sanjeev Bhagat
> >   [email-masked]   (TRIPACE/Zoetermeer)" <[email-masked]>,
> >                         Julian Satran/Haifa/IBM@IBMIL
> >   01-10-01 20:37               Subject:        Re: iscsi : numerical
> >   Please respond to     negotiation wording is ambiguous
> >   Santosh Rao
> >
> >
> > Julian & Sanjeev,
> >
> > Responding to both your mails......
> >
> > Julian :
> >
> > I think you may have mis-interpreted my comments. I believe Sanjeev
> > has
> > clarified the intent of my suggestions.
> >
> > I am *not* suggesting that the responder send back its values and
> > these
> > be blindly imposed on the originator. On the contrary, my suggestion
> > is
> > that the computation of the result of the negotiation (higher or lower
> > of the 2 values) be only performed by the responder and sent back to
> > the
> > originator.
> >
> > The result of the negotiation is the same in both cases and there is
> > no
> > REJECT required in my case nor yours. The difference is the advantages
> > I've stated in my model.
> >
> > Sanjeev, in response to your comment :
> >
> > " [Sanjeev Bhagat (TRIPACE/Zoetermeer)] Although there is
> > >  no reject , but it can be a problem in future .
> > >  Consider your example of DataPDULength in your own
> > >  message. Suppose offering party sends a value of 16,384
> > >  (this is also lowest value it can send) and responding
> > >  party responds with 8,192. In this case the offering
> > >  party will have to reject the negotiation. So a reject
> > >  message is needed in this case also. "
> >
> > There is NO need for any REJECT in the above case. If the initiator
> > is'nt satisfied with the value returned by the originator and cannot
> > function with the negotiated values, it can simply close the TCP
> > connection. There is no need to send any REJECT.
> >
> > Thanks,
> > Santosh
> >
> > > "Sanjeev Bhagat (TRIPACE/Zoetermeer)" wrote:
> > >
> > > Thanks Julian, please find my further comments in the message
> > >
> > >      From: Julian Satran [mailto:[email-masked]]
> > >      Sent: Sunday, September 30, 2001 6:07 PM
> > >      To: [email-masked]
> > >      Subject: RE: iscsi : numerical negotiation wording is
> > >      ambiguous
> > >
> > >      Sanjeev,
> > >
> > >      I am not sure clear I made the tiny diference between what I
> > >      say and what Santosh said:
> > >
> > >      Santosh says:
> > >
> > >        1. a requester sends A=valuex
> > >        2. a responder sends B=valuey
> > >        3. the responder assumes that the value y is the correct
> > >           value and so does an external observer
> > >           [Sanjeev Bhagat (TRIPACE/Zoetermeer)] I would rather
> > >           say it this way the responder computes the value with
> > >           its own supported values and responds with a value y
> > >           which the responder thinks is correct and so does an
> > >           external observer
> > >        4. the requester checks that valuey is conformant (he will
> > >           not believe it) and will reject it if not conformant
> > >           else it will accept it
> > >           [Sanjeev Bhagat (TRIPACE/Zoetermeer)] Although correct,
> > >           but it is highly unlikely for the responder to reject
> > >           the final result because the rule states that (lower or
> > >           higher of two will be the result) and so the offering
> > >           party should be able to accept the lower or higher
> > >           range as defined by rule. In case the key is dependent
> > >           upon some range of fixed values then the negotiation
> > >           should be performed as list negotiation and not
> > >           numerical negotiation.
> > >
> > >           This might be what you "conventionally" do - I don't
> > >           and that shows that convention like morals are a matter
> > >           of geography :-)
> > >           [Sanjeev Bhagat (TRIPACE/Zoetermeer)] :-)
> > >
> > >           The advantage of this set of rules is that it allows an
> > >           external observer to know what was selected without
> > >           knowing the rules
> > >           [Sanjeev Bhagat (TRIPACE/Zoetermeer)] Even in this
> > >           case, I guess that the external observer has to know
> > >           the rules to double check the value is correct or not
> > >           The disadvantage is that messages have to be "built",
> > >           an additional reject states exists and MOST IMPORTANT
> > >           you need both messages
> > >
> > >           In what I said:
> > >
> > >           1. The requester sends A=valuex
> > >           2. The Responder has to send either nothing (if valuex
> > >           is enough on both sides to compute the result like in
> > >           the case in which the function is a Boolean AND and the
> > >           value is NO) or valuey
> > >           3. Both the requestor and responder have to compute the
> > >           value (they have to know the rules anyhow) and so does
> > >           the external observer
> > >
> > >           The disadvantage is that the external observer has to
> > >           know the rules
> > >           The advantage is that there is no reject, in binary
> > >           negotiations you can go away with shorter negotiations
> > >           and you can set strings at fixed values.
> > >           [Sanjeev Bhagat (TRIPACE/Zoetermeer)] Although there is
> > >           no reject , but it can be a problem in future .
> > >           Consider your example of DataPDULength in your own
> > >           message. Suppose offering party sends a value of 16,384
> > >           (this is also lowest value it can send) and responding
> > >           party responds with 8,192. In this case the offering
> > >           party will have to reject the negotiation. So a reject
> > >           message is needed in this case also.
> > >
> > >
> > >      Sanjeev
> > >       "Sanjeev Bhagat
> > >       (TRIPACE/Zoetermeer)"                    To:        "'Santosh
> > Rao'"
> > >       <[email-masked]>            <[email-masked]>, Julian
> > >       Sent by: [email-masked]   Satran/Haifa/IBM@IBMIL
> > >                                                cc:
> >  [email-masked]
> > >       30-09-01 16:32                           Subject:        RE:
> > iscsi :
> > >       Please respond to "Sanjeev       numerical negotiation wording
> > is
> > >       Bhagat (TRIPACE/Zoetermeer)"     ambiguous
> > >
> > >
> > >
> > >      All,
> > >
> > >      I agree with Santosh that the responding party must respond
> > >      the result of
> > >      the negotiation as compared with the value from offering
> > >      party. All
> > >      negotiations in SCSI like (sync, disconnect etc) are also
> > >      done the same way.
> > >      I also object to Julian's reason of a simple minded target
> > >      which wants to
> > >      send certain fixed values only. In case a simple minded
> > >      target identifies a
> > >      value which it cannot support or is less than the value a
> > >      target can
> > >      support, then there should be a method for target to reject
> > >      such a
> > >      negotiation and the default values be accepted on both side
> > >      as a result of
> > >      negotiation.
> > >
> > >      1 Because even if simple minded target sends its fixed value
> > >      which is
> > >      greater than the value sent by offering party then the final
> > >      result of
> > >      negotiation will be taken as ( lesser of the two) and in
> > >      this case target
> > >      which which cannot support the lower value will produce some
> > >      illegal
> > >      results.
> > >
> > >      2. if simple minded target wants to send its own value and
> > >      wants it to be
> > >      accpeted then the responding party is not negotiating but
> > >      forcing the result
> > >      on initiator(this method should not be allowed unless
> > >      explicitly mentioned).
> > >
> > >      however if there is another proposal of numeric negotiation
> > >      in which the
> > >      responding party can force its result to be over ridden by
> > >      the offering
> > >      party's result then it is acceptable for offering party and
> > >      responding party
> > >      to send there own supported key-value result and it can then
> > >      be computed at
> > >      offering party's end.
> > >
> > >      IMP: (May be I am missing something here) I do not see how a
> > >      numeric
> > >      negotiation can be rejected. Is it possible to reject such
> > >      kind of
> > >      negotiaion?
> > >
> > >      Sanjeev
> > >
> > >      Sent: Friday, September 28, 2001 11:15 PM
> > >      To: Julian Satran
> > >      Cc: [email-masked]
> > >      Subject: Re: iscsi : numerical negotiation wording is
> > >      ambiguous
> > >
> > >      Julian & All,
> > >
> > >      I request the group to take a close look at this negotiation
> > >      process
> > >      again and [re-]evaluate the alternative being proposed.
> > >
> > >      Further, it appears that the login stage negotiation  is
> > >      also following
> > >      the same algorithm as the login key negotiation, wherein
> > >      originator &
> > >      responder offer their respective values and both sides need
> > >      to determine
> > >      the result of the negotiation. i.e. both initiator and
> > >      target need to
> > >      compare their NSG with the other party's NSG and pick the
> > >      lower of the
> > >      2.
> > >
> > >      I suggest that both the login key negotiation and the login
> > >      stage
> > >      negotiation follow the policy that the originator offers a
> > >      value and the
> > >      responder picks the result of the negotiation based on (the
> > >      offered
> > >      value & its own value). The value picked by the responder is
> > >      sent back
> > >      to the originator.
> > >
> > >      This model has the following advantages :
> > >
> > >      1) Only one side picks the result of the negotiaton. Hence,
> > >      the 2 sides
> > >      cannot go out of sync on the value picked.
> > >
> > >      2) The originator knows the negotiated result at the the
> > >      responder since
> > >      the responder sends back the result of the negotiation.
> > >
> > >      3) This model is easier to debug because of (1) & (2).
> > >
> > >      4) Less code since only 1 party (responder) needs to perform
> > >      the
> > >      computation to pick the result of the negotiation.
> > >
> > >      5) This scheme leaves less room for interop problems and
> > >      mis-interpretation since it is the more familiar negotiation
> > >      technique
> > >      that is in use in most other mass storage protocols. (ex :
> > >      FC PLOGI, FC
> > >      PRLI, etc). From a first reading of the current negotiation
> > >      scheme, it
> > >      is'nt readily apparent that the currently defined model is
> > >      different
> > >      from the above and requires both sides to be picking the
> > >      result of the
> > >      negotiation, instead of just the responder.
> > >
> > >      Comments ?
> > >
> > >      Thanks,
> > >      Santosh
> > >
> >
> > --
> > ##################################
> > Santosh Rao
> > Software Design Engineer,
> > HP-UX iSCSI Driver Team,
> > Hewlett Packard, Cupertino.
> > email : [email-masked]
> > Phone : 408-447-3751
> > ##################################

-- 
##################################
Santosh Rao
Software Design Engineer,
HP-UX iSCSI Driver Team,
Hewlett Packard, Cupertino.
email : [email-masked]
Phone : 408-447-3751
##################################

---

From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>
Date: Thu, 4 Oct 2001 00:26:28 +0200

All,

I think making DataPDULength different on two side will bring complexity in
implementation. Calculation of CRC, handling unknown and very high value of
DataPDUlength, change in buffer management code etc etc and above all.. a
radical change so late in the specs chould be avoided.

In my view we rather can choose for either of the two ways
1. Either make DataPDULength as list negotiation, which makes sure that the
computing party choses a result which other party will definitley support

or 
2. keep it like numeric negotiation with result being computed by responding
party and double checked by the offering party. In case there is still any
error when the offering party gets the result, they may either go for reject
or re-nogotiation.

Sanjeev

-----Original Message-----
From: Santosh Rao [mailto:[email-masked]]
Sent: Wednesday, October 03, 2001 11:46 PM
To: Rod Harrison
Cc: Ayman Ghanem; [email-masked]
Subject: Re: iscsi : DataPDULength can differ in each direction.

Rod,

The issue came up as a result of Deva pointing out that an initiator may
not be able to function with the minimal of the 2 DataPDULengths
(initiator's & target's) in the case where the value chosen was not its
value.

In order to allow for such implementations that had issues with the
DataPDULength, I raised this question. 

The benefit of both sides being allowed different DataPDULengths is :

a) Each side can specify its max supported value which the other side
would not exceed. The sender & receiver would be able to function with
different DataPDULengths, with the guarantee that their advertised
receive pdu length will not be exceeded, thus, allowing more flexible
interoperability of implementations. (btw, the proposal should have read
: Each side is allowed to advertise its maximum receive pdu length.)

b) It may also allow 2 parties with differing PDU lengths to send larger
sized PDUs in one direction to the party that advertised a higher
receive pdu length.

However, I do admit there's a cost to making this change, given we are
so late in the draft rev and several implementations have already been
made. If this is not an issue of concern to the majority of
implementations, I am ok with the current definition, although it is
more limiting on implementations.

Thanks,
Santosh

Rod Harrison wrote:
> 
>         Can someone give a tangible benefit to this that can outweigh the
> spec and implementation churn at this late stage of the game?
> 
>         From my point of view the benefit of asymmetric PDU sizes would
have
> to be very large to make it worth the extra complexity in buffer
> management code alone.
> 
>         - Rod
> 
> Ayman Ghanem
> Sent: Wednesday, October 03, 2001 8:29 PM
> To: [email-masked]
> Well, the proposal says:
Rather,
>         the key is exchanged in either direction to inform the other side
as
> to
> So, I understand that as if the initiator sends
> DataPDULength=reallyBigNumber, then it is informing the target that
> this is
> what it will be using, but the target should not attempt to negotiate
> it.
> 
> -Ayman
> 
> > Buck Landry
> > Sent: Wednesday, October 03, 2001 2:13 PM
> > To: [email-masked]
> > It may add complexity, but, since DataPDULength is only a boundary
> on
> > the maximum number of bytes transmitted in a Data PDU (not the
> minimum),
> > "the first" *does* have a say about it.  Right?
> >
> > From: Ayman Ghanem [mailto:[email-masked]]
> > Sent: Wednesday, October 03, 2001 1:38 PM
> > To: [email-masked]
> > I think this will add unnecessary complexity, specially for data
> CRCs
> > having
> > to be calculated based on two different PDU sizes for Reads and
> Writes.
> > Also, one end may choose to do data digests in software. If the
> other
> > end
> > decides to use a really large PDU length (and the first doesn't have
> a
> > say
> > about it), this could be a problem.
> >
> > -Ayman
> >
> > <snip>
> >

---

From: Dave Sheehy <[email-masked]>
Date: Thu, 04 Oct 2001 16:13:46 PDT

Comments in text.

>   >  It would allow iSCSI HBAs to interact more efficiently
>   >  with SW iSCSI
>   >  implementations and vice versa.
>   >
> 
> I don't believe it would in practice. Consider the following. The max
> PDU size sent during login is more that just that, it is in fact the
> senders maximum supportable max PDU size. If one side sends 64k and
> the other side 8k although it is technically indicating it can't
> receive more than 8k in a single PDU, for all practical purposes it is
> also indicating it can't handle, and therefore can't send PDUs bigger
> than 8k.

I think that's a faulty interpretation of what's being proposed. As in both
Fibre Channel and TCP the offered PDU size is the size you're willing to
accept. It can and often is completely decoupled from the size you're capable
of sending. This is commonly implemented in Fibre Channel and it's not 
exactly rocket science.

> I believe if we go this route we'll simply see the side with the lower
> DataPDULength sending its "natural" size PDUs and never sending the
> larger size wanted by the other side. More on this below ...
> 
>   >  > 	From my point of view the benefit of asymmetric PDU
>   >  sizes would have
>   >  >From the vantage point of an iSCSI HBA it doesn't seem
>   >  all that hard.
>   >
> 
> Well, it seems to me faced with a peer with a different max PDU size
> there are relatively few ways to proceed.

Again, I believe it's fairly straightforward and there are real life 
examples that work exactly this way. The only unfortunate thing is that it's
been proposed at this late date. In hind sight, it's a feature with an
obvious benefit.

> If the peer has a lower max PDU size there are 2 choices. Use 2 buffer
> pools, one for receive set to the local Max PDU size, and one for send
> set to the peer Max PDU size. This is where the extra buffer
> management complexity comes in. Or, use one buffer pool and simply
> part fill the buffers for sending. This is the easy case.

I don't see what buffer size has to do with PDU size that's not implementation
dependent. The PDU content is going to be fragmented anyway (i.e. network
header, iSCSI header, data, digest, ...) so the problem you describe exists
regardless.

> If the peer has a larger max PDU size then either only send up to the
> local PDU size, as I mentioned above, or chain buffers together to
> build larger than the local max PDU size. Again, this is where the
> extra buffer management complexity comes in. Remember that by
> definition these chains will need to be bigger than the largest chain
> size the implementation can handle. Unless for some reason the
> DataPDULength sent was chosen at some arbitrary size smaller than the
> implementations maximum.

I think SW iSCSI stacks are going to want to use big PDUs and in general
iSCSI HBAs are going to use small ones. I think the SW stacks are going to
set up their buffer structures to use large PDUs. I don't think (although
it's certainly possible) that they are going to dynamically adjust their
buffer sizes for each iSCSI session. So, in the scenario where a SW iSCSI
stack is talking to an iSCSI HBA it's buffering is going to be used
inefficiently in both directions. If the Data PDU Length is negotiated
separately for each direction then the buffering on the SW side is being
used inefficiently in only one direction. Both sides win with this behavior.

Dave

---

From: "Sanjeev Bhagat (TRIPACE/Zoetermeer)" <[email-masked]>
Date: Fri, 5 Oct 2001 02:39:58 +0200

Pat,

Sorry for my previous response. I have tried to re-examine the things on my
own end. I think i am going to agree with you. I donot think buffer
complexity will be any issue as the two things can be very easily handled.
but i do have one doubt at this moment for buffer management: 

If the dataPDUSize as informed by the responder is very high and initiator
cannot supply that high PDU size because it does not have enough capacity to
make that huge amount of dataPDUSize , dont u think it will paralyze the
initator's end. ( i might be wrong, but i think there is a limit to sender's
buffer sending capacity as well. And if that is so then there has to be a
negotiation of the dataPDUSize.

Sanjeev

-----Original Message-----
From: THALER,PAT (A-Roseville,ex1) [mailto:[email-masked]]
Sent: Friday, October 05, 2001 2:26 AM
To: Sanjeev Bhagat (TRIPACE/Zoetermeer); 'Santosh Rao'; Rod Harrison
Cc: Ayman Ghanem; [email-masked]
Subject: RE: iscsi : DataPDULength can differ in each direction.

Sanjeev,

I don't see how allowing each side to independently specify the
DataPDULength it can accept complicates anything. If Device X can receive 10
Kbyte PDUs and Device Y can only receive 2 Kbyte PDUs, then the implication
of allowing them to be independent is

  Device X can only send 2 Kbyte PDUs to Device Y - it has to obey Device
Y's input limitations.

  Device Y may send 10 Kbyte PDUs to Device X if it has that capability or
it can send 2 Kbyte PDUs if its transmit side isn't able to send longer
ones. The transmitter isn't required to send the maximum PDU size that the
receiver can handle.

Furthermore, in many cases buffer pools will be shared among multiple
connections. Therefore a buffer management strategy that depends upon the
max PDU length would be non-ideal.

Pat

-----Original Message-----
From: Sanjeev Bhagat (TRIPACE/Zoetermeer) [mailto:[email-masked]]
Sent: Wednesday, October 03, 2001 3:26 PM
To: 'Santosh Rao'; Rod Harrison
Cc: Ayman Ghanem; [email-masked]
Subject: RE: iscsi : DataPDULength can differ in each direction.

All,

or 

Sanjeev

-----Original Message-----
From: Santosh Rao [mailto:[email-masked]]
Sent: Wednesday, October 03, 2001 11:46 PM
To: Rod Harrison
Cc: Ayman Ghanem; [email-masked]
Subject: Re: iscsi : DataPDULength can differ in each direction.

Rod,

Thanks,
Santosh

have
Rather,
as

---

From: "THALER,PAT (A-Roseville,ex1)" <[email-masked]>
Date: Thu, 4 Oct 2001 18:40:28 -0600

Ayman,

A few people have raised the issue of added complexity because of the CRC
calculation having to be done on different PDU sizes and I don't understand
the difficulty. No matter what maximum PDU size one supports one has to do
the CRC on the actual PDUs which are sent which will vary. One doesn't
always send the maximum - sometimes the write or read is for less data than
that. Anyway the calculation is done one byte/word/(other chunk size) at a
time until one gets to the end of the PDU regardles of the length.

I would appreciate if someone could clarify why different max PDU sizes
complicate CRC calculation.

Pat

-----Original Message-----
From: Ayman Ghanem [mailto:[email-masked]]
Sent: Wednesday, October 03, 2001 11:38 AM
To: [email-masked]
Subject: RE: iscsi : DataPDULength can differ in each direction.

I think this will add unnecessary complexity, specially for data CRCs having
to be calculated based on two different PDU sizes for Reads and Writes.
Also, one end may choose to do data digests in software. If the other end
decides to use a really large PDU length (and the first doesn't have a say
about it), this could be a problem.

-Ayman

> Sent: Wednesday, October 03, 2001 12:32 PM
> > The point is, if the target is not capable of supporting 16K
> (it is just an
> > example that is being discussed), then it indicates that is
> what it wants.
> > However, if the target knows it can support the initiator
> negotiated value,
> > it could return it.
> >

---

From: Dave Sheehy <[email-masked]>
Date: Fri, 05 Oct 2001 13:59:55 PDT

Santosh,

> Is this a feature that you believe is going to be useful ? 

Yes, I do. I already described the specific scenario in a prior email.

> I proposed
> this because I believed the way it is currently done is incorrect. The
> DataPDULength negotiation should ideally follow the model of the FC
> PLOGI common service params "receive buffer length" which an initiator &
> target exchange. In essence, each side tells the other what is the
> maximum length it can receive.

I agree. In retrospect it seems obvious.

> However, I'm also pragmatic and realize that several vendors have
> already gotten their implementations done and so, I did'nt push for this
> strongly. Do you believe this is something that is going to be useful
> and implementations will require specific tweaks to the PDU lengths they
> can use ?

Since it is a change in behavior it will require tweaks as you say but not
significant changes. I believe that the gains make it worthwhile.

Dave

---

From: Dave Sheehy <[email-masked]>
Date: Fri, 05 Oct 2001 13:44:25 PDT

> Having now accepted that DataPDULength is a "maximum receive pdu length"
> limitation that each end point is allowed to express, this key needs to
> be on a per connection basis, and not a per session basis.
> 
> The reason for this being that limitations on DataPDULength (more aptly,
> max receive pdu length) stem from implementation specific quirks. (One
> of the reasons for allowing this feature, in the first place.). 

Following this same line of reasoning the keys:

	MaxBurstSize
	FirstBurstSize
	MaxOutstandingR2T
	DataPDUInOrder
	DataSequenceInOrder
	ErrorRecoveryLevel

are all potentially implementation specific and should not be categorized
as LO.

Dave

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 7 Oct 2001 09:38:24 +0300

Mallikarjun,

We go offline on this. The dataoffset brings us to where we did not want 
ever to be - scoreboarding by initiator.
This is against SCSI basics - data transfer is managed by target.

Julo

"Mallikarjun C." <[email-masked]>
Sent by: [email-masked]
05-10-01 19:35
Please respond to cbm

        To:     ips <[email-masked]>
        cc: 
        Subject:        Re: iscsi : DataPDULength can differ in each direction.

Julian,

I don't really have a strong opinion on connection vs session
applicability of DataPDULength, but one comment.

If we specify the current ExpDataSN in Task Management function 
request (reassign) to be in stead "Expected Buffer Offset", this 
appears to enable per-connection DataPDULength.  Specifying a
buffer offset may also be helpful to targets in general, not 
requiring them to bring the DataSN->buffer_offset translation 
from a (potentially failed) different NIC (though perhaps it is
not all that serious in practice since implementations will
send DataPDULength-sized PDUs except for the last in an I/O).

Do you see any other issues that complicate a connection-specific
value?

Regards.
-- 
Mallikarjun 

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions Organization
MS 5668 Hewlett-Packard, Roseville.
[email-masked]

Julian Satran wrote:
> 
> I think that the set of arguments we have heard are solid enough to
> the DataPDUlength becoming a parameter that characterizes the receiver
> only.
> 
> However this parameter should be also connection specific so that the
> sender and receiver can align (some framing mechanisms may require
> adaptation to path MTU) and having a different DataPDUlength on
> different paths can affect recovery.
> 
> If we leave it session specific the logic of iSCSI can easily
> incorporate different PDU lengths and the draft changes are very
> limited.
> 
> Julo
> 
>  Dave Sheehy
>  <[email-masked]>                 To:
>  Sent by: [email-masked]           [email-masked] (IETF IP
>                                          SAN Reflector)
>  05-10-01 01:13                                  cc:
>  Please respond to Dave Sheehy                   Subject:        RE:
>                                          iscsi : DataPDULength can
>                                          differ in each direction.
> 
> 
> 
> Comments in text.
> 
> >   >  >                  Can someone give a tangible benefit to this
> that can
> >   >  outweigh the

---

From: "Julian Satran" <[email-masked]>
Date: Sun, 7 Oct 2001 12:05:02 +0300

Dave,

Implementation specific? I assume you wanted to say connection specific.

No the parameters you quote (except MaxOutstandingR2T) are not connection
specific.
They reflect multiplexing capability that the transport allows for SCSI.
Making them connection specific will
only complicate things with no added benefit.
MaxOutstandingR2T might be viewed as a way to compensate for RTT bu depend
also on handling capabilities
of the end-points.

Julo

                    Dave Sheehy                                                                      
                    <[email-masked]       To:     [email-masked] (IETF IP SAN Reflector)      
                    ilent.com>                  cc:                                                  
                    Sent by:                    Subject:     Re: iscsi : DataPDULength can differ in 
                    [email-masked]        each direction.                                     

                    05-10-01 22:44                                                                   
                    Please respond to                                                                
                    Dave Sheehy                                                                      

           MaxBurstSize
           FirstBurstSize
           MaxOutstandingR2T
           DataPDUInOrder
           DataSequenceInOrder
           ErrorRecoveryLevel

Dave

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 8 Oct 2001 14:55:28 +0300

Rod,

Actually you bring up a valid point.  The length different in the two 
directions is perhaps the less important feature in the floated proposal. 
However it stressed (once again) that we need a more flexible PDUlength 
scheme:

that should allow dynamic changes (some of the framing schemes proposed 
will require that)
that can be different on different connections

Having it different on the two directions is a slight side effect that can 
be beneficial and comes at zero price once you have done the former.

And the I think that it can be done with minimal disruption (very little 
code and this only if you implement ErrorLevel 1 and 2).

Julo

"Rod Harrison" <[email-masked]>
Sent by: [email-masked]
07-10-01 22:22
Please respond to "Rod Harrison"

        To:     Julian Satran/Haifa/IBM@IBMIL, <[email-masked]>
        cc: 
        Subject:        RE: iscsi : DataPDULength can differ in each direction.

Julian and all,

    Before you go ahead and spend time and energy on this I would just
like to re-iterate my concerns that in practice few, if any,
implementations will support asymmetric PDU sizes.

                 If, for example, a software initiator indicates a max PDU 
size of
128k and a hardware target indicates 8k, do we really think the target
will ever send a 128k PDU? I don't believe it will. The supporting
argument seems to be predicated on the fact that the buffer management
changes needed to make this happen are trivial. However, if an
implementation can chain buffers together on transmit to build PDUs
larger than the maximum size offered why can't it do so on receive?
And if it can do so on receive why would it ever offer a maximum lower
than the real maximum it can support?

                 As an aside, I think this change might create something 
of a headache
for sniffers since they now have to buffer the larger of the offered
sizes instead of the smaller.

                 I agree that asymmetric PDU sizes seem beneficial but I 
have a very
hard time believing we'll see any implementations that can take
advantage of this feature. I also think the benefit might be
relatively small and is probably not worth the level of spec change at
this late stage.

                 - Rod

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Saturday, October 06, 2001 8:04 PM
To: [email-masked]
Subject: RE: iscsi : DataPDULength can differ in each direction.

As I said this requested change has value and we are going to do our
best to incorporate it in 09.

Fortunately PDUlength has few dependencies in the protocol.

Unfortunately (as I said already) we would like PDUlength to a be a
per connection parameter as it is it closely related to the Path MTU.

This later assumption may require some changes in the recovery
mechanism (the current recovery mechanism assumes that any PDU can be
"replayed" as it was on every connection and this assumption won't
hold anymore).

Mallikarjun and myself will attempt to get a better understanding of
the things required and will keep you updated.

I think we have enough information to work on it and we will keep the
list updated on what we think can be done and how.

We have several alternatives and would appreciate some timeout on this
thread.

Julo

---

From: Dave Sheehy <[email-masked]>
Date: Mon, 08 Oct 2001 11:14:07 PDT

Julian,

> Implementation specific? I assume you wanted to say connection specific.

No, I said what I meant and I meant what I said. :-)

I can easily envision implementation restrictions (bounds if you will) with
regard to any of the parameters I listed. I think you are being far too
optimistic that implementors will enable all that the spec allows.

> Following this same line of reasoning the keys:

An implementation can easily be limited by the size of burst it can handle
due to buffering capacity or other data structure limitations of the HW.

>            MaxOutstandingR2T

This requires resources in the HW implementation to track the outstanding 
requests. Implementations will vary in the capacity of tracking said requests.

>            DataSequenceInOrder

Some HW implementations will limited in their capability to handle out of 
order data. Current Fibre Channel implementations illustrate this type of 
constraint well.

>            ErrorRecoveryLevel

Implementations may limit the types of error recovery available. I can
envision different implementation choices being made in the ability to 
restart IOs. I also think it may be problematic moving and restarting an
IO between implementations.

> only complicate things with no added benefit.

In which case you add the complication that the iSCSI layer will have to
query (or know apriori) the capabilities of all its interfaces prior to 
opening any leading connections. That way it won't negotiate beyond the
capabilities of any of its individual interfaces.

Dave

---

From: Dave Sheehy <[email-masked]>
Date: Mon, 08 Oct 2001 11:26:54 PDT

> 	If, for example, a software initiator indicates a max PDU size of
> will ever send a 128k PDU? 

You can count on it. Fibre Channel works that way today and if allowed 
iSCSI will as well.

> I don't believe it will. The supporting

Because the receive is doing direct placement and the transmit side is not.
Efficient direct placement works best with small PDUs. The transmit side
could care less.

> 	As an aside, I think this change might create something of a headache

I haven't heard the Fibre Channel analyzer folks complain about assymetric
frame sizes.

> 	I agree that asymmetric PDU sizes seem beneficial but I have a very

On the contrary, I think this is a big win for the SW implementations. 
Header parsing accounts for a good deal of the receive side processing.
If that can be reduced it will free up CPU bandwidth on the SW side.

Dave

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 8 Oct 2001 21:57:03 +0300

Dave,

Implementation is not a session parameter :-)
What do you mean by an implementation specific parameter?  Is it that 
every connection has a different implementation and they cooperate? Or do 
not cooperate?

Julo

Dave Sheehy <[email-masked]>
Sent by: [email-masked]
08-10-01 20:14
Please respond to Dave Sheehy

        To:     [email-masked] (IETF IP SAN Reflector)
        cc: 
        Subject:        Re: iscsi : DataPDULength can differ in each direction.

Julian,

I can easily envision implementation restrictions (bounds if you will) 
with
regard to any of the parameters I listed. I think you are being far too
optimistic that implementors will enable all that the spec allows.

This requires resources in the HW implementation to track the outstanding 
requests. Implementations will vary in the capacity of tracking said 
requests.

Dave

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 8 Oct 2001 23:58:30 +0300

If they cooperate they have to share goals. The parameters you quoted 
don't make sense per connection.
PDUlength makes a lot of sense.

Julo

08-10-01 22:43

Julian,

> Implementation is not a session parameter :-)

The implementation directly affects the values that a session acquire 
though. :-)

> every connection has a different implementation and they cooperate?

That is the scenario I'm addressing. There has been discussion of late 
which has indicated that initiators may have several different vendors 
products (i.e. implementations) installed at once and that it is expected 
that a session may span those multiple implementations. Each of those 
implementations will have its own constraints and limitations. So, you 
either have to adopt the greatest common denominator approach (i.e. limit 
connections to the intersection of all individual feature sets) or you 
allow 
each connection to conform to the implementation that it resides on.

Dave

---

From: Dave Sheehy <[email-masked]>
Date: Mon, 08 Oct 2001 13:43:22 PDT

Julian,

connections to the intersection of all individual feature sets) or you allow 

Dave

---

From: Dave Sheehy <[email-masked]>
Date: Mon, 08 Oct 2001 16:54:10 PDT

Julo,

> PDUlength makes a lot of sense.

Just so I understand what you're saying you mean that

> > you 
> > connections to the intersection of all individual feature sets) 

is the approach you are advocating and each operating system vendor is going
to have to develop that code if they expect to support a multi-vendor
configuration with spanning sessions?

Dave

> --=_alternative 007343E2C2256ADF_=
> Content-Type: text/html; charset="us-ascii"
> 
> 
> <br><font size=2 face="sans-serif">If they cooperate they have to share goals. The parameters you quoted don't make sense per connection.</font>
> <br><font size=2 face="sans-serif">PDUlength makes a lot of sense.</font>
> <br>
> <br><font size=2 face="sans-serif">Julo</font>
> <br>
> <br>
> <br>
> <table width=100%>
> <tr valign=top>
> <td>
> <td><font size=1 face="sans-serif"><b>Dave Sheehy &lt;[email-masked]&gt;</b></font>
> <br><font size=1 face="sans-serif">Sent by: [email-masked]</font>
> <p><font size=1 face="sans-serif">08-10-01 22:43</font>
> <br><font size=1 face="sans-serif">Please respond to Dave Sheehy</font>
> <br>
> <td><font size=1 face="Arial">&nbsp; &nbsp; &nbsp; &nbsp; </font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; To: &nbsp; &nbsp; &nbsp; &nbsp;[email-masked] (IETF IP SAN Reflector)</font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; cc: &nbsp; &nbsp; &nbsp; &nbsp;</font>
> <br><font size=1 face="sans-serif">&nbsp; &nbsp; &nbsp; &nbsp; Subject: &nbsp; &nbsp; &nbsp; &nbsp;Re: iscsi : DataPDULength can differ in each direction.</font>
> <br>
> <br><font size=1 face="Arial">&nbsp; &nbsp; &nbsp; &nbsp;</font></table>
> <br>
> <br><font size=2 face="Courier New">Julian,<br>
> <br>
> &gt; Implementation is not a session parameter :-)<br>
> <br>
> The implementation directly affects the values that a session acquire <br>
> though. :-)<br>
> <br>
> &gt; What do you mean by an implementation specific parameter? &nbsp;Is it that <br>
> &gt; every connection has a different implementation and they cooperate?<br>
> <br>
> That is the scenario I'm addressing. There has been discussion of late <br>
> which has indicated that initiators may have several different vendors <br>
> products (i.e. implementations) installed at once and that it is expected <br>
> that a session may span those multiple implementations. Each of those <br>
> implementations will have its own constraints and limitations. So, you <br>
> either have to adopt the greatest common denominator approach (i.e. limit <br>
> connections to the intersection of all individual feature sets) or you allow <br>
> each connection to conform to the implementation that it resides on.<br>
> <br>
> Dave<br>
> <br>
> </font>
> <br>
> <br>
> --=_alternative 007343E2C2256ADF_=--