Subject: RE: iSCSI: Markers

From: "Williams, Jim" <[email-masked]>
Date: Tue, 8 Jan 2002 13:48:53 -0500

> -----Original Message-----
> From: John Hufferd [mailto:[email-masked]]
> Sent: Sunday, January 06, 2002 4:58 PM
> To: [email-masked]
> Subject: iSCSI: Markers
> 
> 
> [...]
> 16. If the HW Pipelining iSCSI HBA is to
> send COWS Markers to another HW
> Pipelining HBA, there may be a problem.
> The outgoing pipeline would prefer
> to have the pointers pointing backwards,
> but the receiving pipeline would
> prefer to have the pointers going forwards.
> Therefore with COWS, either
> the sending or the receiving HW Pipelining
> HBA will be unhappy and pipeline
> design will get very much more complicated.

With the disclaimer that I am skeptical of the
value of markers and definitely opposed to 
mandating them, I would add the following
comment regarding COWS.

I believe its more important to make the
transmit side "happy" than the receive side.
There are two reasons here.

1.  It is desired to have much wider deployment
    of transmit side COWS, since the receive
    side is an option that many (most) vendors
    will choose not to support.  Transmit side
    may want (need) to be supported anyway
    for the benefit of other implementations
    that require it.

2.  NIC designers can defer
    fetching transmit data until late in the
    pipeline.  All protocol processing, header
    formatting, and bookkeeping can be done 
    prior to fetching the data.  Therefore
    the transmit pipeline will have a small
    amount of time to fully implement the COWS
    algorithm.

    On the receive side pipeline, the COWS algorithm
    can be done in parallel with the protocol
    processing since that is done when all the data
    is present in the receive buffer.  The 
    receive COWS algorithm can be done in place in the
    receive buffer, so backward pointers are
    not a problem.

    An efficient transmit COWS algorithm requires that
    hardware check each word in the outgoing
    PDU, whereas an efficient receive algorithm requires
    checking only the pointer (other than exceptional 
    cases).

> John L. Hufferd
> Senior Technical Staff Member (STSM)
> IBM/SSG San Jose Ca
> Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
> Home Office (408) 997-6136, Cell: (408) 499-9702
> Internet address: [email-masked]
>

---

From: "Amir Shalit" <[email-masked]>
Date: Tue, 8 Jan 2002 12:58:02 -0800

I agree with what Jim and John are saying.

I think COWS would present a major challenge to most software and more also
hardware implementations.
Software implementation may touch the data at least once for computing
checksum but must change
to do pattern matching. It's not clear to me that "off the shelf" network
stacks can easily change (i.e. logistics, support, etc.).
It gets much worse with hardware implementations since an existing CRC
circuit can not change
to support additional functions. A new H/W design that combines CRC and COWS
may not make sense in
terms of pipeline architecture as well as logic design.

In addition, rearranging the data stream by swapping each appearance of the
pattern
with a link may be time consuming for the transmitter.

I vote for Markers.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Williams, Jim
Sent: Tuesday, January 08, 2002 10:49 AM
To: 'John Hufferd'; [email-masked]
Subject: RE: iSCSI: Markers

---

From: Jonathan Stone <[email-masked]>
Date: Tue, 08 Jan 2002 13:18:30 -0800

....This may be a good point to observe that the original
implementation of COBS was for a wireless radio link which struggled
to deliver 19.2kbps of goodput to any individual end-station.

I believe the first two (or three, depending on how one counts the
appletalk implementation) of COBS all found the change in buffer sizes
diffiult to integrate into other operations, and they all implemented
COBS encoding/decoding as a copy between encoded and decoded buffers,
not as an in-place operation.

Even if a COWS data-touching loop is folded into another data-reading
loop, COWS encoding/decoding still dirties all the cache lines being
encoded or decoded, and will therefore has the same overhead as an
additional copy.

Performing COBS encoding/decoding in an outbouard accelerator also
makes it impossible for those of us who don't trust our I/O busses to
perform a `trust, but verify' integrity check: e.g., recomputing the
TCP/IP checksum over the data stream as the HBA-cum-NIC deposits it
in host memory.

Indeed, the original COBS implementation(s) provided only framing;
COBS framing was wrapped outside an end-to-end integrity check.

I beleive Stuart is still on vacation, but I can ask him to comment on
these issues when he gets back, if that would be worthwhile.