Subject: [Ips] iSER parameter resources

From: "Robert D. Russell" <[email-masked]>
Date: Tue, 15 Mar 2005 16:53:48 -0500 (EST)

Mike:

In the process of working with iSER we hit the following problems.

The first has to do with the way parameters are passed to the operational
primitives.  The specifications in the iSER draft intentionally leave
many details to the implementation, but consider the following situation:

The Send_Control primitive is used by iSCSI to pass an iSCSI READ command
PDU to iSER.  The parameters are connection_handle, BHS, AHS (if any),
and DataDescriptorIn.

These parameters themselves are resources, and hence may be dynamically
allocated by iSCSI, but the problem then is: when can these be deallocated?
The answer may be that it is up to the implementation, but that is not
entirely satisfactory:

If the Send_Control either does not return until iSER has finished using
these parameters, or makes a copy of the BHS, AHS, and DataDescriptorIn,
then iSCSI can free these resources as soon as the Send_Control returns.
This is clean, but inefficient, since it involves extra copying and/or
delays.

If the Send_Control does not make a copy of the BHS, AHS, and
DataDescriptorIn, and returns without waiting for the operation to
complete, then iSCSI must not free these resources until it
knows that the iSER layer is finished using them.  The present
draft provides no notification mechanism for this -- the
Control-Notify operational primitives is used by iSER only to notify
iSCSI about the availability of new inbound iSCSI control-type PDUs.

Therefore, the only thing iSCSI can do is wait until the iSCSI Response
PDU for the command is received before it can free these resources.
For a command like READ such a delay does not tie up a lot of resources,
but for a WRITE with unsolicited Data-out PDUs more resources are involved,
because each unsolicited Data-out PDU requires its own Send_Control,
with its own BHS and DataDescriptorOut resources.  Unless iSER is known
to copy these resources, iSCSI cannot free any of them until it receives
the iSCSI Response PDU for the entire WRITE command.

One solution to this problem would be to add a new operational primitive
called Send_Completion_Notify (or something like that), which would be
used by iSER to notify the iSCSI layer that a Send_Control operation had
completed (and therefore iSCSI is able to free the associated resources).
The use of this notification would be enabled by a new Notify_Enable
parameter to the Send_Control primitive (completely analogous with the
Notify_Enable parameter to Put_Data).

Note that there is a similar problem that works in the other direction:
When iSER receives a new inbound iSCSI control-type PDU, it calls the
Control_Notify primitive, passing that newly received PDU as a parameter.
Is iSCSI expected to completely utilize that PDU before returning
from Control_Notify, so that iSER can always free or reuse the buffer
containing that PDU after the return?

If the answer is yes, then in many circumstances iSCSI will have to
make a copy of that PDU (or the relevant parts of it) before returning
from Control_Notify.  Again this could be inefficient.

If the answer is no, then how does iSCSI notify the iSER layer that
it has finished with the PDU, so that iSER can free or reuse the buffer?

Again, one solution would be to add another new operational primitive
called Release_Control (or something like that), which would be
used by iSCSI to notify the iSER layer that a buffer passed from iSER
to iSCSI in a previously completed Control_Notify could now be freed.
Since use of this extra primitive would not always be required, the
Control_Notify could be modified to return a result that indicated whether
or not iSER could free or reuse the PDU buffer after the return.
Presumably an indication not to free or reuse the buffer requires that
iSCSI would later call the Release_Control primitive to release the buffer.

As an editorial comment, I believe that in section 7.3.1, the explanation
of the DataDescriptorOut should be moved from where it is now,
buried as the second sentence of point a of the third star (*), to a more
prominent position in the general text just before the first star (*),
so that it unambiguously applies to all SCSI WRITE or bidirectional
commands.  This would make clear that the WRITE PDU is always accompanied
by a DataDescriptorOut for ALL the data to be transfered by the WRITE --
at present it could be read that this DataDescriptorOut is not needed
if all data is sent unsolicited.  If it does apply in all cases, it should
not be defined as part of just one case (especially the last case).

As a final point, the current iSER draft requires that when a Send_Control
primitive is called to send a Data-out PDU, a DataDescriptorOut be
specified that defines the buffer containing the unsolicited SCSI Write
data.  This is inefficient and may lead to the allocation of extra
resources to store that DataDescriptorOut, since it appears that this
DataDescriptorOut needs to uniquely define the data for each Data-out PDU.

In fact, iSER does not need this DataDescriptorOut at all -- it could
calculate the location of the unsolicited Write data from the Buffer
Offset and DataSegmentLength fields of the Data-out PDU, together with the
DataDescriptorOut supplied with the original WRITE PDU.

For efficiency in practice, this DataDescriptorOut should always be just
the buffer or scatter-gather list generated by SCSI and passed through to
iSER by iSCSI (without interpretation by iSCSI).  SCSI does not know
anything about Immediate or unsolicited or solicited data, and therefore
every DataDescriptorOut is most naturally a description of all the data
to be transfered by the entire WRITE command.

Requiring a unique DataDescriptorOut for each Data-out PDU seems to
introduce needless inefficiencies by potentially requiring iSCSI to
interpret the SCSI scatter-gather list, and in any case requiring iSCSI
to allocate new resources for the unique DataDescriptorOut.

One possible solution would be to simply eliminate the requirement for a
DataDescriptorOut qualifier when sending a SCSI Data-out PDU.
Another would be to generalize the wording in section 7.3.4 to leave
the use of this qualifier up to the implementation, so that it could be
eliminated or NULL or the same DataDescriptorOut as was passed with the
WRITE command.  The present wording in section 7.3.4 seems to imply that
this DataDescriptorOut is required and must define exactly and only the
unsolicited data.

Thanks for your consideration.

Bob Russell
InterOperability Lab
University of New Hampshire
[email-masked]
603-862-3774

_______________________________________________
Ips mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ips

---

From: Mike Ko <[email-masked]>
Date: Tue, 15 Mar 2005 17:08:02 -0800

Bob,

My comments are embedded.

Mike
To:     Mike Ko <[email-masked]>
cc:     [email-masked] 
Subject:        iSER parameter resources

Mike:

These parameters themselves are resources, and hence may be dynamically
allocated by iSCSI, but the problem then is: when can these be 
deallocated?
The answer may be that it is up to the implementation, but that is not
entirely satisfactory:

but for a WRITE with unsolicited Data-out PDUs more resources are 
involved,

[[mk
The Datamover Architecture as described in the DA draft, upon which the 
iSER/iSCSI interface is based, specify the required exchanges at the 
datamover interface to separate the role of the data mover (iSER) from the 
iSCSI layer.  Note that it does not preclude other implementation specific 
exchanges between the two layers to ensure correct and efficient 
operation.  Consistent with this philosophy, I don't think it is necessary 
to add a new operational primitive in this case for the temporal control 
of resources at the implementation level. 
mk]]

iSCSI would later call the Release_Control primitive to release the 
buffer.

[[mk
Again, how the iSCSI and the iSER layer handle the receive buffer is best 
left to implementation.
mk]]

[[mk
The list of PDU-specific qualifiers for the SCSI Command PDU include 
DataDescriptorOut.  However, DataDescriptorOut is not required if all the 
data is sent unsolicited.  How DataDescriptorOut should be used for the 
immediate data and solicited data case are specified under those bullets.
mk]]

[[mk
An implementation is free to choose how DataDescriptorOut is implemented. 
It can certainly take the form of the scatter-gather list with offsets as 
you have suggested.  We thought about reusing the DataDescriptorOut from 
the SCSI Command PDU but decided against it and instead, DataDescriptorOut 
is specified with the SCSI Data-out PDU.  The reason is that it frees the 
iSER layer of the responsibility of having to keep track of the 
DataDescriptorOut for every SCSI Write and also offers more flexibility 
for the iSCSI layer on how the buffer should be handled.
mk]]

Bob Russell
InterOperability Lab

---

From: "Robert D. Russell" <[email-masked]>
Date: Wed, 16 Mar 2005 09:40:23 -0500 (EST)

Mike:

Thanks for the quick reply.

Ok, so the philosophy is basically that everything up to the
implementation.  That's fine and I have no problem with that.

However, it seems that if that philosophy is followed,
some of the details in the current spec could be eliminated.
Consider the two cases cited in your reply:

[[mk

It may not be required for unsolicited data, but some implementations may
find it useful.  So, in the end, isn't it up to the implementation?
Why bother to put such a fine level of detail in the specs?

Furthermore, in point a. of the third star (*) in section 7.3.1, the
spec says the DataDescriptorOut describes the complete I/O buffer,
including immediate, unsolicited, and solicited data.  Isn't that
too fine a level of detail to have in the specs?  Isn't it up to
the implementation to decide whether or not the unsolicited data
is described by that DataDescriptorOut?

[[mk

Fine, but isn't that a decision that is best left up to the implementation?
Why make this decision in the specs?

In the first case above, the specs go out of their way to define a case when
the DataDescriptorOut is not needed.  In the second case, they don't even
mention that it might not be needed (in fact, the first sentence in section
7.3.4 seems to imply that the DataDescriptorOut MUST be supplied to define
the I/O Buffer containing the unsolicited Write data).

In both cases, shouldn't it be up to the implementation to decide what is
needed and what is not at this level of detail?
So in both cases, couldn't the spec be simplified by eliminating the
details that can be decided by the implementation?

Thanks,

Bob Russell
InterOperability Lab

---

From: Mike Ko <[email-masked]>
Date: Wed, 16 Mar 2005 16:44:10 -0800

Bob,

You might have a point here.  Most of the text dealing with PDU-specific 
qualifiers are provided as informational only.  But there are places where 
the directive "MUST" have been used.  This goes beyond what was stated in 
section 11.3 of the DA draft which states that "Some implementations may 
choose to deduce the qualifiers in other ways that are optimized for the 
implementation specifics".  One of the examples cited was "For SCSI 
Data-Out (section 11.3.5.1), deducing the DataDescriptorOut input 
qualifier from the associated SCSI Command invocation qualifiers (assuming 
such state is maintained) in conjunction with BHS fields of the SCSI 
Data-out PDU."  I can revisit the usage of the "MUST" directive as used in 
conjunction with the input qualifiers and remove them as necessary to 
adhere to the spirit of the DA draft and to allow for implementational 
flexibility as intended by the DA draft.

Mike
Sent by:        [email-masked]
To:     Mike Ko <[email-masked]>
cc:     [email-masked] 
Subject:        [Ips] Re: iSER parameter resources

Mike:

[[mk

[[mk

Fine, but isn't that a decision that is best left up to the 
implementation?
Why make this decision in the specs?

In the first case above, the specs go out of their way to define a case 
when
the DataDescriptorOut is not needed.  In the second case, they don't even
mention that it might not be needed (in fact, the first sentence in 
section
7.3.4 seems to imply that the DataDescriptorOut MUST be supplied to define
the I/O Buffer containing the unsolicited Write data).

Thanks,

Bob Russell
InterOperability Lab

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 17 Mar 2005 12:07:34 -0800

Hi Bob,

A couple of comments on this thread.

- I can agree with the metapoint you're making - that the spec
   currently may not be consistent in the level of MUST detail
   that it mandates.  However....

 > Why bother to put such a fine level of detail in the specs?

Well, I assume the "fine level of detail" you're referencing is
the 3rd_star-bullet_a text.  I think that it is necessary for 
correctness to include all that detail for the reason stated in the last 
sentence there - "This implies zero TO for this STag points to the 
beginning of this I/O Buffer." - for correct wire protocol 
interoperability.

-
 >(in fact, the first sentence in section
 > the I/O Buffer containing the unsolicited Write data)

The necessary wire protocol requirement for iSER is to unambiguously 
know that the data to be put on the wire via the Send Message must be 
what the Data-Out BHS is describing.  In other words, the header and the 
data must match so iSER can encapsulate.  There is thus a MUST 
requirement wrt the level of detail on what must be communicated from 
iSCSI to iSER.  This can be realized however via what's described in 
that text, or via what the DA draft hints at section 11.3, or via some 
other TBD implementation approach.  So I'd like to see the MUST 
functional requirement to be captured.

- Like Mike, I too think that the extensions you were suggesting
   are better left alone as implementation specifics.

- Are you suggesting to change the iSER spec to adopt a consistent
   approach wrt mandatory requirements, or because you disagree with
   the functional requirements implied by the current mandatory text?

Thanks.

Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm [at] rose.hp.com

Robert D. Russell wrote:
> DataDescriptorOut.  However, DataDescriptorOut is not required if all 
> the data is sent unsolicited.  How DataDescriptorOut should be used for 
> the immediate data and solicited data case are specified under those 
> bullets.
> mk]]
> 
> An implementation is free to choose how DataDescriptorOut is 
> implemented. It can certainly take the form of the scatter-gather list 
> with offsets as you have suggested.  We thought about reusing the 
> DataDescriptorOut from the SCSI Command PDU but decided against it and 
> instead, DataDescriptorOut is specified with the SCSI Data-out PDU.  The 
> reason is that it frees the iSER layer of the responsibility of having 
> to keep track of the DataDescriptorOut for every SCSI Write and also 
> offers more flexibility for the iSCSI layer on how the buffer should be 
> handled.
> mk]]
>

---

From: "Robert D. Russell" <[email-masked]>
Date: Fri, 18 Mar 2005 11:18:13 -0500 (EST)

Mallikarjun:

> the 3rd_star-bullet_a text.  I think that it is necessary for correctness to 
> include all that detail for the reason stated in the last sentence there - 
> "This implies zero TO for this STag points to the beginning of this I/O 
> Buffer." - for correct wire protocol interoperability.

I was actually refering to the whole detailed approach of this section,
which delimits where the DataDescriptorOut is needed (stars 1 and 3)
and where it is not (star 2).  Section 11.3.1 of the DA draft says simply
that for SCSI write or bidirectional commands, the DataDescriptorOUT
defines the I/O Buffer meant for Data-out for the entire command --
it (correctly) does not go into details about immediate, unsolicited, etc.
Therefore, section 7.3.1 in the iSER spec should just restate that general
rule before going into the star-bullets.  Currently bullets 1 and 3 state
variations of this general requirement, and bullet 2 says nothing about it
(because in SOME implementations it may not be needed in bullet 2).
This is inconsistent -- either the statement in the DA draft is wrong,
(I don't think so, and don't claim this) or the text in 7.3.1 is 
overspecifying some details that are really up to the implementation.
The point about zero TO for this STag follows from the general DA 
requirement that the DataDescriptorOut define the entire command.

> The necessary wire protocol requirement for iSER is to unambiguously know 
> that the data to be put on the wire via the Send Message must be what the 
> Data-Out BHS is describing.  In other words, the header and the data must 
> match so iSER can encapsulate.  There is thus a MUST requirement wrt the 
> level of detail on what must be communicated from iSCSI to iSER.  This can be 
> realized however via what's described in that text, or via what the DA draft 
> hints at section 11.3, or via some other TBD implementation approach.  So I'd 
> like to see the MUST functional requirement to be captured.

There is a need for MUST to describe the use of the Send_Control 
primitive, and to say which data from the Data-out buffer for
the entire command should be associated with the Send_Control.
There is NOT a MUST requirement that this information be described
by a DataDescriptorOut qualifier to the SCSI Data-out.  That MAY be the
way some implementations do it, but as section 11.3 of the DA Draft
says, some implementations MAY deduce this information and therefore
choose not to require this qualifier for this command.
We all agree on this -- it is only the way it is worded in the iSER
spec.  It seems to me that the MUST applies to too much and results
in overspecification of implementation details.

>  are better left alone as implementation specifics.

Fine.  One of the advantages of having items like this at least
mentioned in the spec is uniformity across implementations that
choose to implement them (clearly they are not required), and
they give an indication what issues may be involved and how they
may be addressed.  This is somewhat in the same spirit as the
"optimizations" (really hints) given in section 11.3 of the DA draft.

>  approach wrt mandatory requirements,
Yes.

> or because you disagree with
No.

>> DataDescriptorOut for every SCSI Write and also offers more flexibility for 
>> the iSCSI layer on how the buffer should be handled.
>> mk]]
>>

---

From: "Mallikarjun C." <[email-masked]>
Date: Thu, 24 Mar 2005 17:07:01 -0800

Bob and all,

In working on iSER changes, Mike Ko & I had an offline conversation on 
the question of what's mandatory vs. implementation-specific.  Here are 
our recommendations:

1. DA draft: add a "MUST implement" requirement wrt the Operational
    Primitives (NOT on the realization) on both iSCSI & Datamover
    layers.  That would be one MUST in the first setence of section 9,
    and one MUST in the first sentence of section 10.
2. DA draft: add text in section 7.4 to make it clear that "qualifiers"
    of any Operational Primitive invocation represent the mandatory
    set of information context required for the Operational Primitive
    to act on.  While all the information context is required, the
    *method* of realizing the qualifiers (passed inline or retrieved from
    task context, or retrieved from shared memory etc.) is upto the
    implementations.
3. iSER draft: add clarifying text that says the word "mapping" refers
    to the mandatory task state the iSER layer must maintain for correct
    wire operation, but was not meant to refer to a specific
    implementation approach.
4. iSER draft: Mike had already made some changes to keep the iSER
    MUST requirements away from the implementation space.

Longer explanation on 1&2:

(1) Operational Primitives define an abstract communication model and 
imply no implementation specifics.  There is a mandatory requirement on 
both parties for supporting the abstract communication model so we can 
correctly specify the mandatory functional requirements on iSCSI & 
Datamover layers.   The communication model happens to have some named 
Operational Primitives for assisting in the functional description.  So 
they are mandatory to implement on both layers.  This was always 
intended, but Mike pointed out that the DA draft doesn't call them out 
as MUST today.  This is proposed to be fixed.

(2) Section 11.3 already makes this clear that the *deduction* of 
qualifiers can be done in several ways - depending on the Datamover 
protocol specifics (e.g., iWARP requires something), and the 
implementation specifics (e.g., implementation foo is optimized to 
specify the qualifiers via shared memory).  11.3 also already states 
that the qualifiers are the "semantically required set" for an 
invocation.  The new proposed clarifying text in 7.4 would make it clear 
that "qualifier" isn't something passed inline as an argument of a 
function call - it will differentiate a qualifier from the method of 
deducing it.

Thanks.

Mallikarjun

Mallikarjun Chadalapaka

Robert D. Russell wrote:
>> correctness to include all that detail for the reason stated in the 
>> last sentence there - "This implies zero TO for this STag points to 
>> the beginning of this I/O Buffer." - for correct wire protocol 
>> interoperability.
> 
> 
>> what the Data-Out BHS is describing.  In other words, the header and 
>> the data must match so iSER can encapsulate.  There is thus a MUST 
>>> the data is sent unsolicited.  How DataDescriptorOut should be used 
>>> for the immediate data and solicited data case are specified under 
>>> those bullets.
>>> mk]]
>>>
>>> implemented. It can certainly take the form of the scatter-gather 
>>> list with offsets as you have suggested.  We thought about reusing 
>>> the DataDescriptorOut from the SCSI Command PDU but decided against 
>>> it and instead, DataDescriptorOut is specified with the SCSI Data-out 
>>> PDU.  The reason is that it frees the iSER layer of the 
>>> responsibility of having to keep track of the DataDescriptorOut for 
>>> every SCSI Write and also offers more flexibility for the iSCSI layer 
>>> on how the buffer should be handled.
>>> mk]]
>>>