Subject: iSCSI: Markers and Framing

From: "John Hufferd" <[email-masked]>
Date: Fri, 11 Jan 2002 13:33:50 -0800

In addition to the 5 options
1. FIM now, and Bare Framing later
2. FIM now, and COWS Framing later
3. COWS now, and Bare Framing later
4. COWS now, and COWS Framing Later
5. Nothing now, and some kind of Framing Later

I have been asked to add a number six. as follows

6. FIM now, and some kind of Framing Later

.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/SSG San Jose Ca
Main Office (408) 256-0403, Tie: 276-0403,  eFax: (408) 904-4688
Home Office (408) 997-6136, Cell: (408) 499-9702
Internet address: [email-masked]

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Fri, 11 Jan 2002 17:51:12 -0800

Dear Colleagues,

As an implementer of silicon solutions up to 10Gbps, my take is ...

  1. Election # 6. FIM now and some kind of framing later. 

     Comments:

    a. By framing I mean what ever method iWarp effort ends up with.
       (TUF as proposed today may not be it)
    b. I strongly recommend SHOULD implement FIM on the send side. 
       Implication -> Senders that do not insert markers should be 
       willing to accept up to RTT*BW data drops! Headers being 
       "reasonably" out-of-order is OK. Of course, senders that do not 
       insert markers but are willing to pay big $$$ to the SSP will 
       get their buffer/BW allocation as usual and customary :-)
    c. I think that the proponents and beholders of FIM had 
       good reasons. They still hold and are even more stronger. We
       have had FIM in the iSCSI spec since version 02. Major changes 
       to the iSCSI draft, at this late date, should have significant 
       technical reasons.
     
  2. COBS is a good solution for the problem that Stuart and Mary originally
     set out to solve at Stanford. It's(COWS) use in the context of iSCSI 
     is "misuse" at best, esp. given that the use of virgin TCP for
transport 
     is mandatory(for good reasons).

     Several of you have alluded to why COWS is nasty to implement ... I 
     prefer not to get there. Markers on the other hand do bring in some 
     "essential" complexity but they are reasonable to implement even 
     at 10Gbps. We sure could brute-force COWS, but the point is why the 
     additional "incidental" complexity. 

     Do not read further, unless you are open to ... :-)
      
-Shridhar Mukund
---------------------------------------------------------------------------
                   	  
    CAUTION: MANDATORY to delete and OPTIONAL to read 

  >> Wouldn't it be stupid if we had a proliferation of framing 
  >> alternatives just because the "world originally seemed flat"?
  (My apology to Stephen Bailey for taking the quote out of context)

  A packetized HDLC stream(for which COBS was designed) has one whole 
  space-time dimension lesser than a TCP stream. Relative to TCP 
  stream, packetized HDLC stream is a flat world!

  As I see it, COBS is an alternative coding technique that makes 
  delimiters explicit in an otherwise reference-less "sequence" or
  byte/word-stream. A reference-less sequence is assumed to have 
  no ends or gradations. Yet the encoded sequence exposes 'handles' 
  for synchronization. -> relatively synchronous after encoding.

  TCP stream on the other hand is a "time-sequence". It starts
  with a big-bang after which every byte in the sequence has an
  explicit time(sequence number) associated with it. -> It is
  absolutely synchronous even to begin with!

  The increase in entropy because of assuming a time-sequence to
  be a mere-sequence is probably :-) demonstrated by the following:

  In the Marker lingo : 
	My second son was born in 99 and my first son in 94. 
      Of course, 1900 AD is the marker here.
  In the COW's moo:
      My second son was born 20 blue moons after my first son. 
      My first son ... I have no clue?

  Usage of COWS over TCP transport would be like loosing a needle
  in the hay stack, on purpose, and then devising a clever scheme
  to retrieve it.

  If you read on further, you may be wasting your time ...

  Are you certain that the world is round?
      If you read Stephen Hawkings or an article from Stanford in 
      Scientific American 3 blue moons ago, there is scientific 
      evidence of higher dimensions beyond the 4 space-time 
      dimensions we perceive. In other words, round world might 
      just be an illusion or a convenient definition of what we 
      perceive! When I am using maps, flat world seems perfectly
      fine to me.

  Since a mere-sequence is a one dimensional space and a time-sequence
  like TCP stream is two-dimensional, COBS needs to work harder with 
  packetized HDLC sequences. If COBS looks simpler than FIM, it is 
  just an artifact your specific implementation. Q.E.D.

-Merlin's apprentice

---

From: "John Hufferd" <[email-masked]>
Date: Sat, 12 Jan 2002 04:00:36 -0800

Shridhar,
You input is both timely and amusing.  A number of folks have wanted input
from  ASIC Designers that were focused on 1 and especially 10 Gig and I
guess you clearly fall into that category.

You bring up a few points, that I found useful:
1. That you like the FIM approach now, and that it is useful in your design
until something REALLY better comes along.
2. That you would like "SHOULD Implement".  And you did not push for the
"MUST Implement".
3. That you, in effect, did not want to bet on the Final Framing since
there will be perhaps lots of issues and solutions still to come with iWARP
& RDMA, so don't worry about Framing now;  there is lots of experimentation
and debate that is going to occur, and who knows what will happen there.

I think I did some interpretation on point 3 so if that was wrong please
correct me.

I think I could accept the "SHOULD Implement" statement.  And perhaps that
is better then the "MUST Implement" that we have been talking about, and to
which a number of folks have objected.

There have been several folks that seem to have  reasons for not wanting
"MUST Implement".  This would give them an out if they felt that they have
a just and valid reason in their product for not doing it.  On the other
hand "SHOULD" gives the customers enough other vendors that will support
Markers, that if the customers care about it, they can easily obtain a
Multi vendor solution which exploits it.

 I can accept a "SHOULD Implement on Send", but "Optional to use", for FIM.

I also think the point you made about  iWARP, was also valid.  So I am
starting to also lean towards the new Option 6, with SHOULD, instead of
MUST.

Comments from others?

.
.
.
John L. Hufferd

"Mukund, Shridhar" <[email-masked]> on 01/11/2002 05:51:12 PM

To:    John Hufferd/San Jose/IBM@IBMUS, [email-masked]
cc:    "Sperry, Todd" <[email-masked]>, "Munnangi, Siva"
       <[email-masked]>
Subject:    RE: iSCSI: Markers and Framing

Dear Colleagues,

     Comments:

  2. COBS is a good solution for the problem that Stuart and Mary
     originally
     set out to solve at Stanford. It's(COWS) use in the context of iSCSI
     is "misuse" at best, esp. given that the use of virgin TCP for
transport
     is mandatory(for good reasons).

-Shridhar Mukund

  In the Marker lingo :

-Merlin's apprentice

---

From: [email-masked]
Date: Mon, 14 Jan 2002 10:37:57 -0500

>       get their buffer/BW allocation as usual and customary :-)

I think the Implication is too strong, as it goes against the following
SHOULD from RFC 1122 (which modifies RFC 793):

         4.2.2.20  Event Processing: RFC-793 Section 3.9
 
            While it is not strictly required, a TCP SHOULD be capable
            of queueing out-of-order TCP segments. 

A drop causes the segments up to the retransmission of the drop to 
be out-of-order.  This does not preclude "SHOULD implement FIM", but
the above Implication is overdone in my view as it appears to condone
drops of all out-of-order segments.

>       to the iSCSI draft, at this late date, should have significant 

I just want to remind everyone that there can be "significant technical
reasons" to change things at this juncture.  One example is the change
made to SCSI Task Management (e.g., Abort Task Set) in Salt Lake City.
The balance in the above text between "changes ... at this late date"
and "significant technical reasons" is a good way to think about this
sort of issue.

Thanks,
--David
---------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 42 South St., Hopkinton, MA  01748
+1 (508) 249-6449 *NEW*      FAX: +1 (508) 497-8500
[email-masked]         Cell: +1 (978) 394-7754
---------------------------------------------------