Subject: RE: iSCSI: need for new data SNACK code?

From: [email-masked]
Date: Thu, 11 Jul 2002 23:42:04 -0400

Mallikarjun,

> > The new code
> > provides the initiator with a more robust way to detect resegmentation
> > by requiring the initiator to explicitly ask for it.  The initiator
> > can take a simple approach of always starting with the existing
> > Data SNACK code that does not resegment and only using the new code
> > when the non-resegmenting SNACK doesn't work.  
> 
> That's one approach.  So, let's note that you're expecting the target
> to maintain a "PDU-size-changed" flag for every active task, and 
> expecting it to fail the "regular" data SNACK if the flag is set.

The flag per task is not needed - I'd expect the Target to look
at the Data PDUs it would have to resend, check them against the max
Data PDU size for this connection and fail the regular SNACK if any
PDU is too large.  This could even be lazily evaluated at the time
each PDU is to be sent because the odds are that if resegmenting
is necessary, the first Data PDU to be resent is going to need it.
The initiator still has to time out the failure of all the Data
PDUs to arrive in order to deal with header corruption (unfortunately,
the F bit doesn't help) - when it times out the "regular" Data SNACK,
it issues a "resegmenting" one (this deals with resegmenting that
becomes necessary after the first Data PDU for a Data SNACK has
been sent).

> Some issues -
>     a) would it really cover the (impossible, IMHO) case you're attempting
>         to cover, in the face of multiple PDU size changes?

Should work - permission to resegment includes permission to
re-resegment or worse.  Independent of how many size changes
happen, the status SNACK at the end returns a new status with
an ExpDataSN that reflects the right number of Data PDUs sent.
 
>     b) assuming that there indeed is a disconnect b/n the two,
>         what should the target do when a resegmenting data SNACK
>         is received, but there's no PDU size change?  I hope you
>         aren't mandating the specific approach.

Resegmenting SNACK is "permission to resegment", and the target need
not use that permission.  If the permission is not used, the Initiator's
status SNACK is not needed but does no harm.

>     c) this approach costs one additional round-trip delay, where
>         none is necessary (as argued below).

I make no apologies for spending a round trip to remove a data
corruption risk.  This is a rare case with a possibly nasty failure
mode - I'm much more interested in this working right than fast.

>     d) seems like it would need new Reject code(s) to distinguish
>         a "regular" reject from that of the PDU size change ones.

Could be useful, but is not strictly necessary.

> >If the target makes
> > its own choice to resegment, and the initiator doesn't think the
> > target resegmented, 
> 
> Now this is beginning to feel more like the option A vs B vs C debate
> we had a while ago.  If the protocol works correctly, both sides would
> be *completely synchronized* on the fact of PDU size change.

As the complexity of a protocol increases, that synchronized
state machine assumption becomes more prone to failure.  The
whole discussion of default values for text keys and the resulting
"if in doubt, negotiate it" maxim was one example.  The alternative
of relying on every default key value to be what was expected was
significantly less robust.

> There are two options for initiators to deal with this - 
> 
> a) don't issue any data SNACKs while any text negotiation is 
>     in progress - wait till the text response is received successfully.

That strikes me as a productive direction that I could see enforcing
with some "MUST"s / "MUST NOT"s - the initiator is causing this mess
by changing the max Data PDU size on the connection.  This is not a
friendly thing to do, and for an initiator to expects to be able to do
this with uninterrupted high performance is unrealistic ... so imposing
costs on the initiator for making this disruptive size change makes sense.

Suppose we went back to the old approach where Data SNACKs *never*
resegment and required that:
- Initiators MUST NOT issue Data SNACKs that could require
	resegmentation?
- Targets MUST reject or ignore Data SNACKs that require
	resegmentation.
- If resegmentation becomes necessary during retransmission
	of Data PDUs for a Data SNACK, PDUs retransmission
	MUST cease for that Data SNACK.
An initiator that wants to be able to issue a Data SNACK for
some or all of its commands then has to ensure that no such
commands are outstanding when/while it changes (in particular
reduces) the max Data PDU size.  In the limit, the initiator
has to wait for all of its commands on the connection to
complete before changing the max Data PDU size, and not
start any new ones until the size change is complete.

This is simpler and more robust than any of the options under
discussion and has the right sort of incentives in discouraging
initiators from changing the max Data PDU size.

Can you accept this? I would expect widespread support for
the resulting removal of target resegmentation from iSCSI.

I will however answer one more question ...

> > This requires additional Initiator
> > state per command for something that almost never happens, and if it
> > gets one of these markings wrong, 
> 
> Sorry, how is it different from the target getting wrong one 
> of its aforementioned
> "PDU-size-changed" flags for tasks?

(1) The per-task flags aren't needed - see above.
(2) The failure is harmless - if the target fails to resegment and
	sends a PDU that is too large, the initiator discards it,
	and then decides what to do about the broken target.  There's
	no possibility of completing a READ command without all of its
	data.

Thanks,
--David

---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449            FAX: +1 (508) 497-8018
[email-masked]       Mobile: +1 (978) 394-7754
---------------------------------------------------

> -----Original Message-----
> From: Mallikarjun C. [mailto:[email-masked]]
> Sent: Thursday, July 11, 2002 7:46 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: Re: iSCSI: need for new data SNACK code?
> 
> 
> David, comments in text.
> 
> > I disagree with the "careful enough" characterization.  The new code
> > provides the initiator with a more robust way to detect resegmentation
> > by requiring the initiator to explicitly ask for it.  The initiator
> > can take a simple approach of always starting with the existing
> > Data SNACK code that does not resegment and only using the new code
> > when the non-resegmenting SNACK doesn't work.  
> 
> That's one approach.  So, let's note that you're expecting 
> the target to maintain 
> a "PDU-size-changed" flag for every active task, and 
> expecting it to fail the 
> "regular" data SNACK if the flag is set.
> 
> Some issues -
>     a) would it really cover the (impossible, IMHO) case 
> you're attempting
>         to cover, in the face of multiple PDU size changes? 
>     b) assuming that there indeed is a disconnect b/n the 
> two, what should the
>         target do when a resegmenting data SNACK is received, 
> but there's no
>         PDU size change?  I hope you aren't mandating the 
> specific approach.
>     c) this approach costs one additional round-trip delay, 
> where none is
>         necessary (as argued below).
>     d) seems like it would need new Reject code(s) to 
> distinguish a "regular" reject
>         from that of the PDU size change ones.
> 
> >If the target makes
> > its own choice to resegment, and the initiator doesn't think the
> > target resegmented, 
> 
> Now this is beginning to feel more like the option A vs B vs C debate
> we had a while ago.  If the protocol works correctly, both sides would
> be *completely synchronized* on the fact of PDU size change.  
> 
> There are two options for initiators to deal with this - 
> 
> a) don't issue any data SNACKs while any text negotiation is 
> in progress - 
>     wait till the text response is received successfully.
> 
> OR
> 
> b) issue a data SNACK regardless, and if the text response 
> (that indicates 
>     a PDU size change) arrives before the data burst 
> completes, discard the 
>     status PDU, and ask for its retransmission.
> 
> Option a is what I suggest, and b is for the adventurous sort.
> 
> >there are error scenarios that combine this with
> > corrupt Data PDU headers to cause the initiator to successfully
> > complete a SCSI command that has not delivered all its data
> > (the resegmented PDUs caused the Data PDU count to match 
> the ExpDataSN
> > value in the response that should have been discarded, but wasn't).
> 
> Which is precisely why I'm suggesting that we mandate discarding the 
> status PDU.  What am I missing?
> 
> > While these should be rare, their consequences can be catastrophic.
> > 
> > It is conveying the Initiator's instructions that resegmentation is
> > permitted.  I am not comfortable with the last sentence 
> above that assumes
> > that the Initiator and Target will always have identical 
> views of all of
> > the effects of a full feature phase PDU size change - (which is
> > a rare event to begin with, and hence likely to involve code that
> > isn't well exercised/tested).
> 
> Obviously, I cannot guarantee the lack of bugs in any implementation.
> But again, let's not attempt to address implementation bugs 
> by protocol
> means (that's why we picked option A in the A vs B vs C debate I 
> referred to above - see the "reusing ISID for recovery" 
> thread; it's for the 
> same reason we removed the X-bit for connection reinstatement -
> see the "X-bit in Login" thread).
> 
> > 
> > > The only two changes from the rev14 text that I propose 
> are that we add:
> > >
> > >    a) The first status PDU must always be dropped after a
> > > MaxRecvDataSegmentLength change, if ever a data SNACK is
> > > employed for the task.
> > 
> > When does this obligation to drop the first status PDU expire?  
> 
> As it says: when the first status PDU is dropped for the task 
> - for each 
> active task during a PDU size change, *and* for which a data SNACK 
> is/was issued.
> 
> >I think
> > the Initiator has to mark all commands that are outstanding 
> or become
> > outstanding between the time it starts the negotiation that changes
> > MaxRecvDataSegmentLength and the time that it gets the 
> final Text Response
> > of that negotiation from the target.  This requires 
> additional Initiator
> > state per command for something that almost never happens, and if it
> > gets one of these markings wrong, 
> 
> Sorry, how is it different from the target getting wrong one 
> of its aforementioned
> "PDU-size-changed" flags for tasks?
> 
> I believe that the onus should be on the initiator to do what 
> it takes to 
> do the right recovery - as is the general error recovery 
> philosophy everywhere.
> Target cannot predict if the initiator would be interested in 
> recovering a
> particular I/O (regardless of the operational ErrorRecoveryLevel).
> 
> >it's vulnerable to failing to deliver
> > all the data for a SCSI command in a compound error situation.  An
> > alternative with the new code could involve a single bit 
> per connection
> > that records whether the PDU size was ever changed (if so, retry any
> > failed Data SNACK as a resegmenting Data SNACK). 
> 
> Or, use just "the Data SNACK", if we define only one.  I 
> can't see why this
> optimization needs two data SNACK codes.
> 
> > 
> > > Initiator MUST issue a status SNACK to recover the
> > > status PDU (i.e. move the onus of retransmitting
> > > status from the target to the initiator).
> > >     b) A SNACK requesting an R2T, Data or Status PDU for 
> a task MUST be 
> > >           issued before the status for the task is acknowledged.
> > 
> > I have no problem with these two.
> > 
> > > I'll be glad to see any technical reasons that I am 
> > > overlooking, that require two codes.
> > 
> > See above.  This is somewhat analogous to the "if in doubt, 
> negotiate
> > it" principle for login - telling the other side *exactly* 
> what is wanted
> > is more robust than assuming that it will do what is wanted, and in
> > this resegmenting Data SNACK case, there are potentially nasty
> > consequences to an incorrect assumption.  Does this make any sense?
> 
> I see what you're trying to get at.  However, IMHO, there is 
> no "assuming"
> involved here.  If the protocol works right, it should do the 
> right thing.  Or else, 
> we are in serious trouble despite this change.
> 
> Regards.
> --
> Mallikarjun
> 
> Mallikarjun Chadalapaka
> Networked Storage Architecture
> Network Storage Solutions
> Hewlett-Packard MS 5668 
> Roseville CA 95747
> [email-masked]
> 
>

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 12 Jul 2002 12:20:10 -0700

David,

I see some convergence, but still disagreement on several aspects.  There's
an attempt to summarize the options at the bottom, and I'd prefer that others
or the process co-chair to comment, rather than continue with this thread with 
us two alone going back and forth.

> PDU is too large

I am afraid there's no free lunch here.  In this description, you're now 
expecting targets to maintain the PDU size of every PDU that it shipped
for each of the tasks, which causes a metadata explosion.  This was discussed
by Eddy Quicksall and myself in an earlier thread - "changing MaxPDUDataLength".

>If the permission is not used, the Initiator's

Well, the point is - shouldn't the target be detecting these obvious bugs, and attempting
recovery/fix for these errors (it's a clear disconnect b/n target and initiator state).  Seems like
additional complexity on either end - to cover implementation bugs wrt prior synchronization.

> state machine assumption becomes more prone to failure.

I think this is where the major disconnect is between us.  As I responded to Dave
Sheehy yesterday, the iSCSI protocol specification *mandates* that a target must 
ship "exact replicas" of the data PDUs barring certain header fields unless the
PDU size was changed by an intermediate successful text negotiation.   What you're
suggesting is: despite this mandate, target may resegment illegally, so let's define a
new data SNACK code with identical wire semantics.  

>and for an initiator to expects to be able to do
> this with uninterrupted high performance is unrealistic 
..
>right sort of incentives in discouraging

You're making an incorrect assumption here that it's just the initiators that
are likely to change the max PDU size.  Either party can do it - and was the
whole point behind the recent addition of "negotiation prompt" Async Message.
(Initiator, on the prompt, may respond with a blank Text Request PDU.)
It's for this precise reason that my option (a) limits *data SNACKs* while 
"any text negotiation" is in progress.

Now, on to your proposal...

> reduces) the max Data PDU size. 

I am afraid you may have misunderstood what I was suggesting.  I was *not*
suggesting that PDU size must not be renegotiated with outstanding tasks. 
In stead, I was suggesting that there be no *data SNACKs* while any text
negotiation is going on (meaning that SNACKs can be issued after the negotiation 
completes, so the initiator can definitively throw out the status PDU for all
tasks with data recovery needs).

There's one weird corner case in the simplified option you're suggesting here - 
when a target wants to initiate a max PDU size change, it cannot know when the 
initiator is likely to quiesce the I/Os, nor there's a way to tell the initiator to stop.  
One way to deal with this - target issues a "negotiation prompt", and the initiator 
responds with a Text Request PDU only *after* all active tasks are completed 
and their statuses acknowledged.  This of course has the obvious drawback that
negotiation/declaration attempts by the target for *any key* would be rebuffed by
the initiator until the connection is quiesced.

With that said, let me suvey the available options:

Option.A
         - Keep the rev13 text, plus add the two additional text segments I proposed
            on the beginning of this thread (initiators must drop status in one case, SNACK
            must be issued only before the status is ack'ed), *and* add "no data SNACKs 
            while any text negotiation is on".
         Pros:   - No need for a new data SNACK code with identical wire semantics
                    - Can allow the PDU size change to happen with no wait for quiescing
                       any long running writes/reads (and those operations too benefit from ULPDU
                       containment from this changed PDU size).
         Cons:   - Additional complexity (compared to the standard data SNACK) on the 
                       initiator to drop the status SNACK; and to mark all active tasks while the 
                       PDU change had happened, so their statuses can dropped if necessary.

Option.B
          - Same as Option.A, but add the new resegmenting-Data SNACK code per 
             David's Last Call comment.
          Pros:  - Precludes surprises due to implementation errors (also a con, see below)
                    - Can allow the PDU size change to happen with no wait for quiescing
                       any long running writes/reads (and those operations too benefit from ULPDU
                       containment from this changed PDU size).
         Cons:  - Attempt to address an implementation error by protocol means, could be a
                      slippery slope.
                   - Requires a new data SNACK code which both sides have to handle, and which
                      conveys completely redundant information about the changed PDU size.
                   - Additional complexity (compared to the standard data SNACK) on the 
                      initiator to drop the status SNACK; and to mark all active tasks while the 
                      PDU change had happened, so their statuses can be dropped if necessary.

Option.C
           - Completely disallow PDU size changes (initiated by either party) while any tasks
              are active.  Rev13 text should be stripped of the resegmenting discussion.  Any 
              data SNACK always gets exact replicas.
           Pros:  - Simpler approach, initiators don't need to drop status PDUs, nor mark the
                       active tasks.
           Cons:  - Active tasks cannot dynamically adapt to PMTU degradation, so ULPDU
                        containment isn't always guaranteed - particulary painful for long-running tasks
                        for either party.
                      - Desired changes in max PDU size would need to wait for all tasks to quiesce
                        and the statuses be acknowledged, forcing a pause in the I/O activity.
                      - Any text negotiation prompted by the target can't be carried on until all 
                        active I/Os are quiesced (even if the target intends to negotiate other keys).

I prefer Option.A, followed by Option.B.  Option.C's cons appear to outweigh its simplicity,
so wouldn't prefer that.

Regards.
--
Mallikarjun

Mallikarjun Chadalapaka

---

From: "Randy Jennings" <[email-masked]>
Date: Fri, 12 Jul 2002 17:06:54 -0700

> by Eddy Quicksall and myself in an earlier thread - "changing

I did not check the thread, but I remember your conversation with Eddy
Quicksall.

However, I do not remember what was said about knowing the length of the
PDU.  From this isolated discussion, though, it seems like you would at
least be able to recalculate the length of the PDU at all times or you could
never send it again.  (Unless you redid all the steps that got to that point
in the first place, and I do not think that is possible based off a PDU id
number...)

Am I missing something?

Sincerely,
Randy
Data Transit

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 15 Jul 2002 12:33:45 -0700

[ Catching up on my email after a weekend site power shutdown....]

Randy,

Please refer to my old email to Eddy:

  http://www.pdl.cmu.edu/mailinglists/ips/mail/msg10674.html

It required the following -
    a) target must always send MaxRecvDataSegmentLegth sized data-in 
        PDUs except possibly for the last PDU.
    b) *if* a PDU size change happens during the life of  a task, target
         notes the DataSN (let's say "size_changed_from") when that happens.
    c) connection (*not* the task) maintains the current and previous 
        MaxRecvDataSegmentLength (let's say "PDU_size_history", an array of size 2).
    d) if "n" PDU size changes are to be designed for, during the life of a task,
        make the "size_changed_from" into an array of size n, and make the
        "PDU_size_history" an array of size (n+1).

I believe one doesn't need to maintain exhaustive DataSN-to-PDU_size tuples
for all PDUs for all active tasks as David's revised description implied.  IOW,
what I am suggesting is that a target can translate a DataSN to a buffer offset
(even with PDU size changes) with *much* less metadata, *if* it always sends 
MaxRecvDataSegmentLegth sized data-in PDUs except possibly for the last PDU.
Any retransmission request would have to take the aforementioned size_changed_from
and PDU_size_history arrays into consideration, and calculate the buffer offset 
accordingly.

Agreed, this is additional CPU cycles where a simple (and memory-consuming)
tuple table could be faster - but it's exception processing, speed is not much of
a consideration here.

Hope that explains my earlier response.
--
Mallikarjun

Mallikarjun Chadalapaka

---

From: "Randy Jennings" <[email-masked]>
Date: Mon, 15 Jul 2002 14:26:22 -0700

Fair enough.  That is actually what I meant about being able to recalculate
it somehow; although, I did not have it developed quite so neatly.  However,
your argument applies to both the previous discussion, and also seems to
fall in David's favor, because the necessary state would need to be saved
anyway (for any retransmission to happen).

I guess what I was confused about is how this argues against having another
opcode for the SNACK and against the target being the one catching the error
condition.

Sincerely,
Randy Jennings
Data Transit

> and PDU_size_history arrays into consideration, and calculate the
> buffer offset accordingly.
>
> > least be able to recalculate the length of the PDU at all times
> > or you could never send it again.  (Unless you redid all the steps that
got
> > to that point in the first place, and I do not think that is possible
based
> > off a PDU id

---

From: "Mallikarjun C." <[email-masked]>
Date: Mon, 15 Jul 2002 15:30:49 -0700

> However,

Remember, I didn't argue that "necessary state" doesn't have to be saved -
that would be too brave on my part to do that.....   My metadata explosion 
comment was based on David's description that seemed to imply a tuple table.

> opcode for the SNACK and against the target being the one catching the error

Though not sure about what you mean by " target being the one catching the error
condition" (there's no error in redeclaration of max PDU size, except for the lost data 
itself - and the onus is on the initiator to "catch" that)....

This doesn't argue either way (and a needless subroutine call), I was pointing this
out when a comparable scheme on the initiator side was argued as flawed.
My point was and is that I see only implementation errors that can make the 
one-code scheme untenable, not any protocol issues.  Ironically, the same applies to 
the two-code scheme as well.
--
Mallikarjun

Mallikarjun Chadalapaka