Subject: Re: iSCSI: Markers

From: "Julian Satran" <[email-masked]>
Date: Mon, 7 Jan 2002 08:13:25 +0200

Some comments in text - Julo

John Hufferd/San Jose/IBM@IBMUS
Sent by: [email-masked]
06-01-02 23:57

        To:     [email-masked]
        cc: 
        Subject:        iSCSI: Markers

When we were in Salt Lake City, we decided to hold a discussion on the
Reflector regarding Markers.  I think it is probably time to hold that
discussion.  Therefore, I will start it out.

First I want to thank Julian for going to the trouble to document the
proposal for COWS.  This now gives us two Marker proposals to consider.
They are the Fixed Interval Markers (FIM) and the Constant Overhead Word
Stuffing (COWS) proposals.

Having said that, lets examine each one and determine their usefulness.

1. Markers (FIM or COWS) are only needed by Hardware HBAs that are
attempting to limit the amount of Reassembly Buffers they need on-board.
This will always be a probabilistic determination by the vendors, but the
purpose of Markers are to hold down the amount of total RAM needed,
especially when factored into a probabilistic determination.
2. The use of Markers are negotiated per connection, and per direction. It
is therefore possible for a software implementation to send markers, but
not have to accept them.
+++ that is specially important for software implementations ++
3. When CRC Digest are negotiated to be used, it does no good to send out
Markers since the hardware side will have to have Reassembly Buffers to
compute the CRC Digest.  Therefore, Markers do not help anything when CRCs
are used on a connection.
+++ That is not correct - Framing mechanisms are used to avoid an RTT 
related size for the receive buffer not avoinding  a one-PDU buffer. Both 
FIM and COWS will help reducing HBA ememory requirements even when using 
CRC.  When Using CRC a PDU reassembly buffer is unavoidale but it can be 
limited by the MaxRcvPDU. +++

4. FIM can be implemented easily in software, and with an out going
Scatter/Gather technique could send PDUs across a network to a Hardware 
HBA
destination without requiring additional moves or touching each byte.
5. Software iSCSI implementations should always negotiate NOT to receive
markers since Markers do not help the SW in any way.
6. Hardware HBAs can also implement FIM easily, and HW can do it easily in
both directions.  Therefore, if FIM is the approved Marker proposal, a HW
HBA should send FIM whenever it sends iSCSI Commands and/or Data to other
HW HBAs (assuming iWARP is not an available option), and the other HW HBAs
wants them.
7. I do not foresee many installations using CRC Digests with Laptops and
Desktops, since the overhead will probably be noticeable, and most
installations do not judge Desktop and Laptop data to be key corporate
assets (I am not saying that opinion is right, or that it is Universal,
just that is a prevalent opinion).
+++CRCs are not relevant +++
8. A single software node would not be fast enough to cause a significant
load or a significant  memory requirement for the Reassembly Buffers.
However, the combination of many (perhaps Desktops and Laptops spread
across a real or virtual campus) can bring on a large load and the need 
for
many Reassembly Buffers.  The resultant amount of small Reassembly Buffers
could make the Total requirement very High (thus raising the cost of the
HBA).
9. Given the need for Desktops and Laptops to avoid noticeable 
degradation,
and the probability that they will not be using CRC, means that the use of
FIM is a compatible option when working across a network from a SW Node to
an iSCSI HW HBA implemented Storage Controller.
10. COWS was also designed to be easily implemented in Software and in
Hardware.
11. COWS will require the iSCSI Software implementations to touch each
byte, as it looks for matches to its Framing Pattern within the PDUs.
+++ except when implemented within the CRC, checksum where no ADDITIONAL 
touch is involved +++
12. When CRC-32 digests are being computed, the Software will have to 
touch
ever byte anyway, so the additional overhead to look for matches to the
Framing Pattern is negligible.  On the other hand FIM also works well with
SW CRC-32 Digest computation.
13. But the premies I set above (which is clearly up to debate), is that
Desktops and Laptops will not usually compute the CRC Digest.
14. COWS has the ability to have its pointing Markers, which point to the
Framing Pattern Matches, point either forward or backwards.  When being
processed by software, the direction does not matter.  So since the 
Markers
should only be used on outgoing PDUs, the approprate direction of the
pointers is what ever the destination needs.
15. Many vendors, especially at the higher speeds, are implementing a pipe
line design.  The pipeline receive process, will most likely want the COWS
pointers to be forward pointing (pointing in the direction of the end of
the PDU).  Since the SW does not care, forward pointers should be fine.
16. If the HW Pipelining iSCSI HBA is to send COWS Markers to another HW
Pipelining HBA, there may be a problem.  The outgoing pipeline would 
prefer
to have the pointers pointing backwards, but the receiving pipeline would
prefer to have the pointers going forwards.  Therefore with COWS, either
the sending or the receiving HW Pipelining HBA will be unhappy and 
pipeline
design will get very much more complicated.
17. Everyone likes iWARP better, but it requires changes to the Host SW
TCP/IP stacks in the case of SW Initiators.  Since these SW installations
are probably going to be Desktops and Laptops, these systems will very
likely have some non current version of the OS, (such as windows 2k, ME,
etc.).  Hence it is unlikely that they will have iWARP in the majority of
the SW TCP/IP Stacks).
+++iWARP needs framing as well and I would guess the technique will be 
close to what COWS is.  Future convergence speaks for COWS+++
18. If, for some reason, a Server uses Software iSCSI, it will probably
have the latest and greatest OS software, and if that is true, and if 
iWARP
is approved by the IETF then it will probably have the TCP/IP extensions
that are required by iWARP.  It will therefore use iWARP instead of any
Marker solution.
19. If the Server uses Software iSCSI,  and if iWARP is not approved by 
the
IETF, then COWS or FIM are both  reasonable choices  for the Server, if 
the
Server also uses the CRC option.  If CRC is not used on the Server, FIM is
a better match with the Server SW iSCSI.
19. iWARP is probably going to stay in IETF "experimental mode" for some
time into the future.  And without iWARP being specified by iSCSI as -- at
least, "MAY Implement" and "MAY use", the HW to HW mode will still need a
Marker solution.
20. Therefore, we are left with Markers as the only technique we can count
on to aid and assist the iSCSI HW HBAs, keep their "on-board" RAM
requirements to a minimum.

Based on the above I come to the following conclusions:
1. Markers are needed.
2. FIM works best with the probable SW environment, and has the lowest
overhead (since they will probably not use CRC Digests).
+++CRCs are not relevant+++
3. Since if CRC is used, the HW needs its Reassembly Buffers anyway, so
there is no reason to use COWS, or any type of Marker.
+++ again the same confusion betwen the PDU buffer and the RTT related 
holding are in the absence of Markers+++
4. FIM can work well between HW HBAs without significant impact to 
pipeline
design.
+++ Somebody already said that FIMs are like democracy - imperfect, ugly 
but work+++
+++ However COWS are not that much heavier if carefully implemented and 
might get us faster to converge on a framing model+++

Therefore, I recommend that the Draft continue to define FIM.  Also, I
recommend that the FIM type of Markers be "MUST Implement" on the 
Out-Going
direction, and "MAY Implement" on the incoming direction".   Further, I
recommend the Draft say that FIM "MAY be used" in any direction, depending
on the negotiations.

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

---

From: "John Hufferd" <[email-masked]>
Date: Mon, 7 Jan 2002 02:07:05 -0800

Julian,
There are two different issues with regards to CRC.

One is on the sender side, and if Markers are done in Software, of course
it matters if CRC is done, since if the sender does not use CRC, the cost
of COWS is very high.

The second point is about the Receiving Side.  I did not understand your
point, so I would appreciate you expanding on your comment that the Target
Side would still get value out of Markers even if they use CRC.  I can
understand that there would be some additional value, especially in TCP
Segment errors.  By permitting some additional TCP Segments to arrive
during the retry, but most of the value is lost because the Full Reassembly
Buffer is needed for CRC.  So now that I confess, that I do not completely
understand your point, perhaps you could step me through your view of the
value in the presents of CRC on a HW HBA/Chip.   I think it is important
that we are able to quantify the value.

With regards to iWARP, I was using that name as an alias for the technique
explained in the framing Draft (at
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-01.txt).
But except for the last part of COWS, there is little resemblance between
that and COWS.  The framing does NOT use Word replacement.  They only have
a 8 byte header in common.

That brings me to another point.  If FIM needs to double its Markers (two
words instead of one), would not COWS also need to double its Header
Markers, for the same reason?

.
.
.
John L. Hufferd

Julian Satran/Haifa/IBM@[email-masked] on 01/06/2002 10:13:25 PM

To:    [email-masked]
cc:
Subject:    Re: iSCSI: Markers

                                                                           
    John Hufferd/San                                                       
    Jose/IBM@IBMUS                   To:        [email-masked]            
    Sent by:                         cc:                                   
    [email-masked].               Subject:        iSCSI: Markers        
    edu                                                                    
                                                                           
    06-01-02 23:57                                                         

Scatter/Gather technique could send PDUs across a network to a Hardware HBA
across a real or virtual campus) can bring on a large load and the need for
9. Given the need for Desktops and Laptops to avoid noticeable degradation,
12. When CRC-32 digests are being computed, the Software will have to touch
processed by software, the direction does not matter.  So since the Markers
Pipelining HBA, there may be a problem.  The outgoing pipeline would prefer
the sending or the receiving HW Pipelining HBA will be unhappy and pipeline
have the latest and greatest OS software, and if that is true, and if iWARP
19. If the Server uses Software iSCSI,  and if iWARP is not approved by the
IETF, then COWS or FIM are both  reasonable choices  for the Server, if the

4. FIM can work well between HW HBAs without significant impact to pipeline

Therefore, I recommend that the Draft continue to define FIM.  Also, I
recommend that the FIM type of Markers be "MUST Implement" on the Out-Going
direction, and "MAY Implement" on the incoming direction".   Further, I
recommend the Draft say that FIM "MAY be used" in any direction, depending
on the negotiations.

.
.
.
John L. Hufferd

---

From: "Julian Satran" <[email-masked]>
Date: Mon, 7 Jan 2002 15:34:13 +0200

John,

The markers are supposed to help you get fat to the next PDU and start 
assembling and placing it.
It does not alleviate the need for a PDU assembly buffer - that you need 
anyhow - it alleviates the need to keep
TCP packets along when you have lost an iSCSI PDU header.

Once you have an iSCSI PDU header you need only the (as short as you want) 
PDU reassembly.

If you do not have an iSCSI header you have potentially to keep around an 
RTT dependent amount of data.
Markers are here to reduce the later to an amount that is not a function 
of Bandwidth*RTT.

Regards,
Julo

John Hufferd@IBMUS
07-01-02 12:07

        To:     Julian Satran/Haifa/IBM@IBMIL
        cc:     [email-masked]
        From:   John Hufferd/San Jose/IBM@IBMUS
        Subject:        Re: iSCSI: Markers

Julian,

The second point is about the Receiving Side.  I did not understand your 
point, so I would appreciate you expanding on your comment that the Target 
Side would still get value out of Markers even if they use CRC.  I can 
understand that there would be some additional value, especially in TCP 
Segment errors.  By permitting some additional TCP Segments to arrive 
during the retry, but most of the value is lost because the Full 
Reassembly Buffer is needed for CRC.  So now that I confess, that I do not 
completely understand your point, perhaps you could step me through your 
view of the value in the presents of CRC on a HW HBA/Chip.   I think it is 
important that we are able to quantify the value.

With regards to iWARP, I was using that name as an alias for the technique 
explained in the framing Draft (at http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-01.txt).  But except for the last part of COWS, there is little resemblance 
between that and COWS.  The framing does NOT use Word replacement.  They 
only have a 8 byte header in common. 

.
.
.
John L. Hufferd

06-01-02 23:57 

        To:        [email-masked] 
        cc:         
        Subject:        iSCSI: Markers 

2. The use of Markers are negotiated per connection, and per direction. 
 It

.
.
.
John L. Hufferd

---

From: Paul Koning <[email-masked]>
Date: Tue, 8 Jan 2002 17:31:20 -0500 (EST)

>>>>> "John" == John Hufferd <[email-masked]> writes:

 John> Paul, I did not understand your statement ...

 John> "...It seems to me a node needs at least as much memory in the
 John> other direction -- quite possibly more if it's the storage end,
 John> given that a lot of the traffic is reads.)"

 John> Are you talking about all the memory in the Node or the RAM on
 John> the HBA?  The focus should be on the HBA, and generally, that
 John> memory is small in the outgoing direction.  So perhaps I
 John> misunderstood your point.

I meant the HBA.  Would you expect TCP level retransmit to fetch the
data from the host again?  Yes, I guess then you could save the memory
on the HBA.

 John> About the Mandating of markers.  The proposal was that it
 John> should be required to implement (not required to use) in
 John> outgoing, and clearly it should be optional to use.  In that
 John> manner, the approprate trade offs can be made at execution
 John> time.

Yes, I know.  I'm objecting to the "required to implement".  It's
problematic to have a feature that someone else can turn on (by asking
for it when you're required to say "yes") and which has the side
effect of destroying your performance.

 John> It seems to be a small impact to the code to just to support
 John> this on the outgoing flow.  I did understand you comments about
 John> how some HW operates, and that needs to be considered in the
 John> executions time trade off.  In most Desktops, laptops, that is
 John> probably not an important issue (with 1.5-2 GHz processors),
 John> but it could be a serious consideration for any servers that
 John> need to have a SW implementation.  (Of course, many folks would
 John> tell the servers to get an HBA, but that is a different issue.)

I'm talking about embedded systems, where throwing an outboard HBA
into the picture makes things much bigger and more expensive, not to
mention slower.  In that example, turning on markers changes the
transmit flow from a straightforward DMA of the data into a memory to
memory copy with insertion of the markers every N bytes.  That's a
large increment in overhead.

      paul

---

From: "WENDT,JIM (HP-Roseville,ex1)" <[email-masked]>
Date: Tue, 8 Jan 2002 14:42:21 -0800

> From: Jonathan Stone [mailto:[email-masked]]
> ...
> Even if a COWS data-touching loop is folded into another data-reading
> loop, COWS encoding/decoding still dirties all the cache lines being
> encoded or decoded, and will therefore has the same overhead as an
> additional copy.

Since inline COWS encoding only has to write back chain links into buffer
locations that contain the flag value (CFP), isn't the dirty-cache-line
overhead proportional to the frequency of occurrence of the flag in the
buffer? Given a 32-bit flag, it would seem that the probability of
encountering a flag in the buffer would normally be quite low, and few if
any cache lines would be dirty, hence the overhead would be just in the
reads from the buffer - which could be hidden in another data reading loop.

COWS does require writing an initial flag and chain link just in front of
the buffer, but either a) that cache line may already be dirty because other
PDU header fields have just been written, or b) part of building the PDU
header can include writing the flag value and a default link chain that
spans the whole PDU.

[A follow up discussion point is that the original COBS scheme also appended
a flag value to the end of the PDU, so that the initial chain link (CLCE) or
final link chain could point to something associated with the current
buffer.

> Performing COBS encoding/decoding in an outbouard accelerator also
> makes it impossible for those of us who don't trust our I/O busses to
> perform a `trust, but verify' integrity check: e.g., recomputing the
> TCP/IP checksum over the data stream as the HBA-cum-NIC deposits it
> in host memory.

Are you assuming that the HBA/NIC is depositing TCP segments into host
memory, as an iSCSI HBA/NIC would be transferring iSCSI PDUs.
What is your preferred model for integrity checking of NIC/HBA transfers
across I/O buses in the face of some offloading of transport or ULP
functionality?

> Indeed, the original COBS implementation(s) provided only framing;
> COBS framing was wrapped outside an end-to-end integrity check.

This is also true for iSCSI/COWS, since the flag separates iSCSI PDUs and
each iSCSI PDU contains (optional) header and data digests (CRCs).

> I beleive Stuart is still on vacation, but I can ask him to comment on
> these issues when he gets back, if that would be worthwhile.

Absolutely.

Jim

---

From: "Julian Satran" <[email-masked]>
Date: Wed, 9 Jan 2002 08:30:37 +0200

One comment in text - julo

Paul Koning <[email-masked]>
Sent by: [email-masked]
08-01-02 18:59

        To:     John Hufferd/San Jose/IBM@IBMUS
        cc:     Julian Satran/Haifa/IBM@IBMIL, [email-masked]
        Subject:        Re: iSCSI: Markers

>>>>> "John" == John Hufferd <[email-masked]> writes:

 John> OK, Julian, I buy that argument.  I think that you have made
 John> the point that the value of Markers is Greater then I was
 John> giving credit for, in iSCSI/TOE integrated HBAs that are
 John> operating in CRC mode.  That is great, since I am afraid we
 John> will not see the Framing stuff for some time.

 John> So the questions are which form of markers is the best.

 John> My vote comes down on the side of Fixed Interval Markers
 John> (FIM). For the following reasons:

 John> On the SW sending side, FIM works with low overhead whether CRC
 John> is being used or not.  COWS, is a dramatic overhead unless the
 John> SW is performing CRC computation anyway.

 John> The COWS has a built in conflict with the Pipeline HW HBA, that
 John> has two different needs regarding the direction of the Pointers
 John> when talking to another partner HW HBA that uses a Pipeline
 John> approach.

I think you are making general assumptions about what is cheap in
software that may be valid for SOME software implementations but are
certainly not valid for others.

In particular, FIM is NOT necessarily low overhead.  Scatter/gather is
a reasonably cheap mechanis IF it is available for this job.  (Note
that it is not all that cheap: adding a bunch of extra descriptor
fetches to the I/O device's job is extra overhead, the scale of which
depends on the hardware.

The problem is that scatter/gather may not be available at all.  If it
is available in principle, it may have restrictions that clash with
what FIM needs.  (For example, a particular implementation may support
breaking buffers up in many pieces, but all breaks must be on cache
line boundaries.  For a cache line size of 32, clearly you cannot use
scatter/gather in that implementation to do FIM.)

So FIM may force a full buffer copy, which is a major hit.

As for COWS, it's clearly nasty.  Even if you're doing CRC, it adds a
significant increment to the processing cost.

In conclusion, I am opposed to mandating markers.  Adding such a large
increment in processing overhead doesn't make sense to me, given that
I don't see how it can be expected to save all that much memory in
implementations that use it.  (As far as I can see, the gain doesn't
come anywhere near a factor of two.  Is there any analysis available
that discusses this in any detail -- covering not just the specific
buffering needs of the reassembly side but also the other memory
requirements in a node?  It seems to me a node needs at least as much
memory in the other direction -- quite possibly more if it's the
storage end, given that a lot of the traffic is reads.)

+++ the gain is not a factor of two - it is a not a constant - it is 
rather
proportional to the bandwidth delay product.
The rational is simple - once a header is lost without any framing 
mechanism 
the receiver has to accumulate data until the missing piece is recovered.
This and other lists had long discussion threads on the subject - and if 
you care about it you may want to go over the archives.
As David has said it is generaly agreed that this is not an issue at 1Gbs 
yet - but is a also agreed that this is a major issue at 10 Gb/s.
That is why some of us think that a stronger statement about some form of 
framing is required.

+++

                 paul

---

From: "[email-masked]" <[email-masked]>
Date: Fri, 11 Jan 2002 15:26:51 -0500

My two cents worth in response to John's comments.

Before making any proposals and/or vote on choices we should be able to understand two things. 

1) why do we need framing?
2) based on the application (iscsi hardware or software assist) which framing technique will make more sense?

- Rakesh

John Hufferd wrote:

> OK, Folks, I have now talked to Steph, who authored  TUF, which is
> currently on the road to Experimental Status,  He has authored another
> version of TUF also, which uses a form of COWS.  So that means that we have
> two different versions of TUF as well as 2 versions of COWS (which are
> independent of Framing), and then there is FIM.  So let me list them and be
> sure we name them so that we are not in the middle of more confusion.
> 
> 1. Fixed Interval Markers (FIM) Currently In the iSCSI Draft
> 2. Constant Overhead Word Stuffing (COWS) as drafted by Julian and sent in
> his note of 12/23/2001 Subject "iSCSI - Synch an Steering Appendix -
> Markers & COWS"
> 3. TCP Upper-layer-protocol Framing (TUF) as drafted by Stephen Bailey in
> http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-tcp-ulp-frame-01.txt
> 4. COWS Drafted By Stephen Bailey which can be used in both in stream and
> with Framing in
> http://www.cs.uchicago.edu/~steph/draft-bailey-tsvwg-cows-00.txt
> 
> Now lets call Julian's proposal COWS with 2 way pointers (COWS2WP)
> Now lets call Steph's COWS with 1 way pointers (COWS1WP)
> 
> When the type of COWS does not matter we can just call them COWS.
> 
> Both COWS can be used in Framing.  But to keep this discussion somewhat
> simpler lets call the Framing without any COWS as "Bare Framing", and Both
> of the other as "COWS Framing".  Only when we need to talk about which type
> of COWS should we say "COWS2WP Framing" or "COWS1WP Framing".  But for most
> conversation it should be just "COWS Framing".
> 
> So we have FIM, COWS1WP, COWS2WP, Bare Framing, & COWS Framing (made up of
> COWS1WP Framing and COWS2WP Framing).
> 
> Now we also need to understand that one of the main reasons expressed to
> make Framing go experimental, instead of Standards Track was that folks
> were worried that Bare Framing was based on probability, and that there was
> a very remote possibility that something could be done incorrectly.
> 
> As a result of that Steph was considering, as part of the experimental
> work, seeing what the impact of his previous COWS Draft would be on the
> experimental work that was going to be done.  He had no intention of
> bringing it up now, since he felt work/thought was still needed.
> 
> As you know COWS came up anyhow (and in a different form).
> 
> So what we have are statements from folks like me that had read Julian's
> Draft and the ietf-tsvwg version of Framing (Bare Framing), which did not
> see in those drafts the overlap.  Clearly there is an overlap in the minds
> of Julian for COWS2WP and Steph for COWS1WP and how they might impact
> Framing.
> 
> NET of Bare Framing vs COWS Framing:
> Bare Framing is based on probability and does not have to inspect each Word
> (SW or HW) COW requires Touching each Word,
> COWS Framing is guaranteed to always be correct.
> 
> So the choices are:
> 1. FIM now, and Bare Framing later
> 2. FIM now, and COWS Framing later
> 3. COWS now, and Bare Framing later
> 4. COWS now, and COWS Framing Later
> 5. Nothing now, and some kind of Framing Later
> 
> If we chose to do any of the "COWS now" options we would need to hold the
> debate on which form, but we should assume that which ever COWS we chose
> now is the COWS for later.
> 
> Value Statements
> 1. FIM and Bare Framing: Means we never have the overhead of touching every
> word
> 2. FIM and COWS Framing: Means that touching is postponed until Framing,
> and perhaps Faster Desktops/Laptops or support even support in normal NICs.
> 3.  COWS now and Bare Framing later: Has issues of toughing everything now,
> and then not useful later
> 4. COWS now and COWS Framing Later: Means always touch, but current
> approach is extensible into Framing
> 5.Nothing now, and some kind of Framing later: Means No current help, and
> no guarantee of help in the future, but some reasonable probability that
> some form of Framing will happen.
> 
> So it is 1-5 upon which  we should be taking a position.
> .
> .
> .
> John L. Hufferd

--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .

---

From: Stuart Cheshire <[email-masked]>
Date: Sun, 13 Jan 2002 12:05:06 -0800

>Stuart,
>
>Are there any IPR issues associated with COBS that you might
>want to share i.e. patents etc?
>
>Somesh

COBS was published first at SIGCOMM in September 1997.

My PhD dissertation, published 1998 and formally registered with the 
United States Library of Congress Copyright Register in January 1999, 
describes extensions including using word lengths other than eight bits, 
eliminating framing marker values other than zero, eliminating multiple 
framing marker values simultaneously, etc.

I have not filed any patent on any aspect of COBS/COWS, and my prior 
publications should ensure that no one else can file a defensible patent 
on any aspect of COBS/COWS.

Stuart Cheshire <[email-masked]>
 * Wizard Without Portfolio, Apple Computer
 * Chairman, IETF ZEROCONF
 * www.stuartcheshire.org