Subject: RE: iSCSI: Markers

From: "Glenn Dasmalchi" <[email-masked]>
Date: Wed, 9 Jan 2002 20:21:26 -0800

I agree with Steph's conclusion on this one. The PDU Containment
Property of TUF is a good fit for high-bandwidth direct placement
receive hardware. If COWS *with* TUF-style segmentation is deemed
impractical, then I think it's better to have PDU containment in TCP
segments (with random key/length headers) than fixed interval markers.

I understood David's earlier posting about not allowing normative
references to experimental drafts (i.e. TUF), but I don't understand how
this precludes using the technique as the iSCSI framing mechanism. If we
decided it was the right technique for iSCSI, couldn't we just lift the
content and put it in the iSCSI draft?

-Glenn

> -----Original Message-----
> From: Stephen Bailey [mailto:[email-masked]]
> Sent: Wednesday, January 09, 2002 5:34 AM
> To: [email-masked]
> Subject: Re: iSCSI: Markers 
> 
> 
> > As for COWS, it's clearly nasty.  Even if you're doing CRC, 
> it adds a
> > significant increment to the processing cost.
> 
> Unfortunately, all the choices are nasty.  Long time IETFers would
> probably make the point that this is an indication we are trying to do
> the wrong thing and should just use a message-based transport :^)
> 
> That said, one position that many people thought was the least nasty
> was key/length-based TUF (with TCP sender segmentation support), or
> nothing at all (== full reassembly).  A smaller number of `key'
> participants felt that an intermediary framing solution for stock TCP
> senders was necessary.
> 
> We proposed COWS-based TUF to try to bring harmony to the spheres, but
> I'm not hearing the pleasant resonance (piano tuning is an emperical
> process, right?), so perhaps we didn't get it quite right.
> 
> Absent any unification of sender segmentation-based and stock TCP
> framing (perhaps 0-touch sends ARE too much to give up), I'm strongly
> in the sender segmentation-based framing (TUF) or nothing at all camp.
> 
> Steph
>

---

From: Stephen Bailey <[email-masked]>
Date: Thu, 10 Jan 2002 11:23:49 -0500

> If we decided it was the right technique for iSCSI, couldn't we just
> lift the content and put it in the iSCSI draft?

From a standardization standpoint, it's like JohnH said.  iSCSI is not
chartered to do anything that modifies the transport.  Even if it were
(which it isn't) the `experimental stigma' of TUF would certainly
spread to iSCSI if iSCSI ingested TUF in toto.

The allure of COWS is that iSCSI CAN specify the in-stream format
without saying anything about TCP segmentation---COWS in-stream format
solves the same problem as markers.  If iSCSI does that, it's a tiny
step, for implementations that can take it, to add TUF-compliant TCP
segmentation & PDU containment processing.

What would be bad is if the `performance community' (RDMA, iSCSI &
whomever else) decided that COWS is the best general solution to this
problem, and iSCSI chose a variant of COWS that didn't work for other
applications (current & future).

Basically, the choices, as I see them are:

  o nothing, FIM, COWS for unmodified senders
  o key/length or COWS for TUF-compliant senders

The key questions to make the choices come down to:

  1) Is PDU containment `worth it'?
  2) Is resynching in the absence of PDU containment `worth it'?
  3) Is 0-touch send for software clients `worth it'? 

Most seem to strongly believe that PDU containment is worth it but
there are some experienced, noteworthy objectors (e.g. Jim Williams).
If you don't believe PDU containment is sufficiently beneficial, then
you simply don't bother with TUF-compliant senders.

I have no idea what `most' think for 2) & 3), but I can speak for
myself.

I don't think resynching in the absence of PDU containment is worth
it.  That implies that I'd take nothing over FIM.  It also implies
that I'd take nothing over COWS w/o a TUF sender, except that if the
TUF sender approach uses COWS, having COWS in-stream is `free'.

I'm undecided on whether 0-touch send for software clients is `worth
it'.  If yes, I'd take key/length for a TUF sender.  If no,
I'd take COWS.

So the only free variable in my position is whether 0-touch software
senders are worth it.  I don't have the data (I'm sure you're
wondering why that's stopping me on this point, and not the others :^)
If yes, it's `nothing + key/length'.  If no, it's `COWS + COWS'.

Steph

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Jan 2002 10:47:41 -0800

Jim,

What are the issues with "one PDU per TCP segment"? I
think this would really enable simple, high performance
Upper Layer protocol processing over TCP. Of course,
you always have the issue that no matter what method
is used, data can only be placed and not processed
out of sequence (which implies that there is additional
complexity if barrier lists and so on and having to do
control processing at a faster rate when you do perform
out of order processing).

Somesh

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Stephen Bailey
> Sent: Thursday, January 10, 2002 8:24 AM
> To: [email-masked]

---

From: "Somesh Gupta" <[email-masked]>
Date: Thu, 10 Jan 2002 12:36:55 -0800

> From: Jonathan Stone [mailto:[email-masked]]
> Sent: Thursday, January 10, 2002 12:11 PM
> To: [email-masked]
> Cc: Stephen Bailey; [email-masked]
> In message 
> <[email-masked]>,
> "Somesh Gupta" writes:
> 
> >Jim,
> >
> >What are the issues with "one PDU per TCP segment"? 
> 
> Mostly that what you're asking for is no longer TCP.
> TCP does not preserve record, or packet, or push boundaries.

  Let us not be so religious about it. Everything must evolve.
  TCP has always evolved. To create cost-effective multi-gigabit
  adapters, I do think changes are needed to "packetize" TCP.
  Others are proposing changes to initial window size, ECN
  etc etc.

> 
> TCP retransmissions can, and will, resend a maximal-MTU worth of
> data from the point where a DUPACK triggers fast retramsit. 
> 
> That maximal segment contains the tail (perhaps all) of one iSCSI PDU,
> plus as much as of the following PDU as fits inside the TCP segment.

  Ok. We do require change to the TCP implementation. I already gave
  you that.

> Heck, if iSCSI can enqueue another PDU before the first one has gone
> out the wire, then standard TCP semantics is to compact the second PDU
> into the first one.  In the reference BSD implemetnation, see
> sbappend().

  OK. Changes are needed to the implementation. Some OSes already avoid
  willy-nilly segmentation.

> 
> SACK makes the picture more complicated. Retransmissions in the face
> of PMTU and route/MTU changes make it even worse. But you get the idea.

  This is where the determinism of COWS addresses lets you handle
  such issues on the slow path.
> 
> If iSCSI PDUs are bigger than TCP segments, then dropping segment
> with a PDU boundary leaves the receiver unable to synchronize until it
> receives the TCP retransission. That takes at least an RTT,
> which takes us back to  BW*RTT worth of buffering.

  There are many alternatives to avoid this one.
> 
> Changing TCP, giving TCP senders the option of asking their sending
> TCP to not coalesce segments across sender-marked boundaries, might
> well be generally useful (c.f. the history behind PSH); but it's a
> change to TCP. That's outside the IPS charter.  It probably requires
> an extension to the API to TCP as well, to mark the boundaries.

  Again, no protocol changes and interoperability with existing
  implementations is the goal. Even with markers, the receive
  TCP has to change significantly.

---

From: Jonathan Stone <[email-masked]>
Date: Thu, 10 Jan 2002 12:11:04 -0800

In message <[email-masked]>,

---

From: Jonathan Stone <[email-masked]>
Date: Thu, 10 Jan 2002 13:10:17 -0800

In message <[email-masked]>,

>  Let us not be so religious about it. Everything must evolve.

Somesh,

Nobody is being religious.  You are asking for changes to RFC 1122,
section 4.2.  Perhaps TCP will need that in the fullness of...well,
need it *now*, I guess, but I thought IPS was forbidden to go there.
Or am I mistaken on that?

>  Ok. We do require change to the TCP implementation. I already gave

If you want iSCSI receivers to rely on non-conforming assumptions
about segmentation and upper-level boundaries, then that's a change to
TCP (send policies, retransmit policies, and receiver-to-application
policies about coalescing segments), which are not part of extant TCP
implementations. The necessary changes fly in the face of the `MUST'
requirements on RFC 1122, sec. 4.2, pp 82-83; and also require that
the SHOULD in the `DISCUSSION' of filling become a `MUST NOT'.

I dont see how anyone can claim that's not a change to TCP.

[...]

>  TCP has to change significantly.

It *is* a protocol change.  Not a change to the on-the-wire format,
true; its a change to the state machines which form user SEND requests
into valid TCP segments, and in how received segments are passed up
to the application.  That's still a change to the protocol.

I dont see any way for a software iSCSI implementation to conform with
what you suggest, in general, without changing the innards of TCP in a
way which (again in general) violates the Host Requirements RFC.

Again, I'm not being doctrinaire about this. I'd use it myself, if I could.

---

From: "Douglas Otis" <[email-masked]>
Date: Thu, 10 Jan 2002 13:26:40 -0800

Somesh,

One view of evolution regarding TCP would be that it has already evolved
into SCTP.  There are many problems beyond framing being solved as a result.

Doug