From mailnull@www1.ietf.org Thu Jan 2 15:21:27 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13297 for ; Thu, 2 Jan 2003 15:21:27 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h02KTsn18881 for rddp-archive@odin.ietf.org; Thu, 2 Jan 2003 15:29:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h02KTsJ18878 for ; Thu, 2 Jan 2003 15:29:54 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13290 for ; Thu, 2 Jan 2003 15:20:56 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h02KTDJ18847; Thu, 2 Jan 2003 15:29:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h02KSMJ18822 for ; Thu, 2 Jan 2003 15:28:22 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13269 for ; Thu, 2 Jan 2003 15:19:24 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 2 Jan 2003 15:22:36 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C6CA@corpmx14.us.dg.com> To: rddp@ietf.org Date: Thu, 2 Jan 2003 15:21:58 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Alignment and TCP modification Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , In response to my request for explanations of opposition to requiring sender alignment: So, let me ask those three people, plus anyone else who is opposed to a general approach of requiring sender alignment to send me a brief email off the list summarizing their viewpoint and the reasoning behind it. I received a total of six responses, five of which expressed varying degrees of opposition to a sender alignment requirement: - Somesh Gupta, Caitlin Bestler, David Borman, Randy Stewart and Shridhar Mukund and one that indicated that alignment could be an acceptable compromise if it could be shown to be of practical value: - Jim Williams I'll leave it to these people to explain their views in further detail if they think it's warranted (in some cases, what was sent to me was a summary of what had already been posted to the list). I'm struck by the fact that a single theme was in common among the five individuals who expressed opposition, namely: - The RDDP specifications need to allow implementations over unmodified TCP/IP stacks. The primary rationales for this are: (1) An unmodified stack will send unaligned in some circumstances that cannot be avoided unless one resorts to truly draconian and unacceptable measures such as a ULP-level stop-and-wait protocol (ULP ACK between sends drains TCP send queue ensuring that next ULP send is aligned by TCP). (2) Even for a hardware-accelerated ULP server implementation, software ULP clients will be common and important. Implementing the RDDP functionality in software over an unmodified stack for ULP clients in this scenario is important to reduce barriers to adoption/use. For example, this enables all of RDDP to be implemented in user mode over an existing stack. (3) The ability to run over an unmodified stack makes the RDDP protocols far more applicable and hence amenable to widespread adoption, including the above ULP client scenario. Beyond this, I saw a divergence of opinion on how much attention implementers should pay to aligned vs. unaligned traffic beyond agreement that it should not be treated as an error case. This divergence ranged from "most traffic will be aligned, but unaligned traffic should not run any slower than ordinary TCP traffic" to "unaligned traffic will be common enough that implementations need to optimize for it, even if most/all senders send aligned" (quoted text is my paraphrase in both cases). Fortunately, the WG does not need to settle this issue - implementers are always free to optimize whatever cases they believe are important. Some of the emails also raised other issues that I'll leave it to those involved to decide whether to discuss further on this list. So, I believe the primary issue before the WG is whether to accept: - The RDDP protocols must allow implementations over unmodified TCP/IP stacks. as a requirement. If this should be accepted as a requirement, then sender alignment cannot be required of all senders (this wouldn't rule out negotiating it in some fashion). Here are a few more implications: - Unmodified means 100% unmodified, not even "minor" changes. - This would imply that a sender using an unmodified stack could send RDDP payloads that are directly placeable by an RDDP receiver. - There don't appear to be any receiver implications, as although an optimized RDDP implementation will likely use a modified stack, I haven't seen any proposal that requires such modifications (e.g., disallows the brute force technique of having TCP buffer as usual and then running RDDP functionality on the in-order TCP bytestream). - The practical meaning of "allow" would be that an implementation is possible without violating any MUST, MUST NOT, SHOULD, or SHOULD NOT requirement. The latter bullet is based on my view that "I don't want to modify an existing stack" is not a sufficient reason by itself to disregard a SHOULD or SHOULD NOT requirement. All of this is open to discussion (including the immediately preceding SHOULD/SHOULD NOT implication). That should be sufficient to get further discussion going ;-). I would ask that those opposed to accepting the requirement start from the above numbered rationale and explain what they believe to be wrong/unconvincing/ missing/etc. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 **NEW** FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jan 4 13:33:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14316 for ; Sat, 4 Jan 2003 13:33:42 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h04Ih3g01585 for rddp-archive@odin.ietf.org; Sat, 4 Jan 2003 13:43:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h04Ih3J01582 for ; Sat, 4 Jan 2003 13:43:03 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14311 for ; Sat, 4 Jan 2003 13:33:10 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h04IgIJ01540; Sat, 4 Jan 2003 13:42:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h04IfTJ01519 for ; Sat, 4 Jan 2003 13:41:29 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14280 for ; Sat, 4 Jan 2003 13:31:36 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4570560109 for ; Sat, 4 Jan 2003 13:34:48 -0500 (EST) Date: Sat, 4 Jan 2003 12:35:08 -0600 From: Caitlin Bestler Subject: Re: [rddp] Alignment and TCP modification To: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C6CA@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/2/03, Black_David@emc.com wrote: > >Beyond this, I saw a divergence of opinion on how much >attention implementers should pay to aligned vs. unaligned >traffic beyond agreement that it should not be treated as >an error case. This divergence ranged from "most traffic >will be aligned, but unaligned traffic should not run any >slower than ordinary TCP traffic" to "unaligned traffic >will be common enough that implementations need to >optimize for it, even if most/all senders send aligned" >(quoted text is my paraphrase in both cases). >Fortunately, the WG does not need to settle this issue - >implementers are always free to optimize whatever cases >they believe are important. Some of the emails also >raised other issues that I'll leave it to those involved >to decide whether to discuss further on this list. > I'm firmly in the camp that favors ensuring that unaligned operation is *possible* but expecting that it will be the exception. It is important to keep in mind that RDDP provides two basic benefits: direct placement of incoming data and eliminating false ordering requirements. Features such as alignment support implementation of direct placement in hardware. These are important benefits, but they are not necessary to justify the deployment of RDDP. A ULP can still benefit from the relaxed ordering rules and/or from merely having one end of a session with hardware accelerated data placement. The prime scenario for this to me is an RDMA enabled file server dealing with a large number of clients. The clients may not have enough RDMA specific traffic to justify hardware assist. But the server can still benefit by being able to use RDMA at its end, because it does have enough traffic to justify hardware placement. The question then becomes whether protocol enhancements should be done to support optimal hardware implementation when faced with high volume out-of-order delivery of non-alligned traffic. My estimate remains that the overwhelming majority of connections, even long-haul connections, will not break alignment. Therefore lower-cost hardware that has a limited capacity to optimize non-aligned packets will still be of considerable value. With those assumptions, I think most people would agree that addign complexity and overhead to the protocol (such as a second CRC for just the header) is not justified. Does anyone agree that the second CRC provides no benefit when alignment is maintained? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 6 14:39:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12065 for ; Mon, 6 Jan 2003 14:39:51 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h06JoCa08975 for rddp-archive@odin.ietf.org; Mon, 6 Jan 2003 14:50:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h06JoCJ08972 for ; Mon, 6 Jan 2003 14:50:12 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12048 for ; Mon, 6 Jan 2003 14:39:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h06JnSJ08907; Mon, 6 Jan 2003 14:49:28 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h06Jm8J08849 for ; Mon, 6 Jan 2003 14:48:08 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11963 for ; Mon, 6 Jan 2003 14:37:16 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id LAA15693 for ; Mon, 6 Jan 2003 11:40:29 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 6 Jan 2003 14:40:28 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8F9A@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Mon, 6 Jan 2003 14:40:27 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. I don't believe anyone has proposed or recommended otherwise. (MPA states an MPA-aware sender SHOULD align, but even that is not contradictory to the above.) I thought all alignment proposals were optional on the part of the sender. Alignment is potentially exploitable by the receiver if it exists, and some mechanism should allow the receiver to determine if alignment is present. Maybe I am missing the point here. > - This would imply that a sender using an unmodified stack > could send RDDP payloads that are directly placeable > by an RDDP receiver. By directly placeable, you mean placeable even if received out or order? If so, I don't understand how an unaligned RDDP PDU is directly placeable, and as you stated, unmodified TCP stacks can't do alignment. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 09:28:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA14851 for ; Tue, 7 Jan 2003 09:28:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07EcoQ24768 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 09:38:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07EcoJ24765 for ; Tue, 7 Jan 2003 09:38:50 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA14748 for ; Tue, 7 Jan 2003 09:24:17 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07EYCJ23740; Tue, 7 Jan 2003 09:34:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07EXjJ23693 for ; Tue, 7 Jan 2003 09:33:45 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA14708 for ; Tue, 7 Jan 2003 09:22:30 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 7 Jan 2003 09:25:25 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C6DF@corpmx14.us.dg.com> To: Jim.Williams@emulex.com, rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 7 Jan 2003 09:25:06 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, > > So, I believe the primary issue before the WG is whether to accept: > > - The RDDP protocols must allow implementations over > > unmodified TCP/IP stacks. > > as a requirement. > > I don't believe anyone has proposed or recommended otherwise. > (MPA states an MPA-aware sender SHOULD align, but even that > is not contradictory to the above.) > I thought all alignment proposals were optional on the > part of the sender. Alignment is potentially exploitable by > the receiver if it exists, and some mechanism should > allow the receiver to determine if alignment is present. > Maybe I am missing the point here. The point is that "optional" (MAY) is a considerably weaker requirement than SHOULD (see RFC 2119). I believe that an unmodified TCP stack cannot comply with a "SHOULD align" requirement solely on the basis that the an implementer does not want to or cannot modify the stack, in contrast to a "MAY align" requirement. In turn, a "MAY align" requirement for senders would make it incumbent upon receivers to do a "reasonable job" for unaligned traffic (and there are a range of views in the WG on what is "reasonable"), in contrast to a "SHOULD align" that would allow receivers to take the attitude that "the sender ignored a SHOULD, and hence deserves the lousy performance that it's going to get". This is not just a wordsmithing issue, as it is strongly related to your next concern: > > - This would imply that a sender using an unmodified stack > > could send RDDP payloads that are directly placeable > > by an RDDP receiver. > > By directly placeable, you mean placeable even if received out > or order? If so, I don't understand how an unaligned RDDP PDU is > directly placeable, and as you stated, unmodified TCP stacks > can't do alignment. I believe so. If we accept the requirement, I am concerned about problem magnification - essentially, transient network congestion causes TCP to start sending unaligned, which in turn causes serious performance problems on servers that relied on aligned transmission for performance, thus turning a network problem that TCP can readily ride out into a serious overall system performance glitch. While implementers cannot be prevented from doing this, one can use carefully drafted and well explained SHOULDs and SHOULD NOTs to warn of the perils lurking here. Markers and similar mechanisms are among the ways to obtain direct placeability of unaligned RDDP PDUs, although I don't see the need for every RDDP PDU to be directly placeable under all circumstances (cf. previous list discussion on why there's no such thing as a truly bufferless [R]NIC). Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 **NEW** FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 10:08:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16494 for ; Tue, 7 Jan 2003 10:08:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07FJhk27295 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 10:19:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07FJhJ27292 for ; Tue, 7 Jan 2003 10:19:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16482 for ; Tue, 7 Jan 2003 10:08:27 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07FJ2J27275; Tue, 7 Jan 2003 10:19:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07FIVJ27248 for ; Tue, 7 Jan 2003 10:18:31 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16466 for ; Tue, 7 Jan 2003 10:07:14 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id HAA20936 for ; Tue, 7 Jan 2003 07:10:27 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Tue, 7 Jan 2003 10:10:27 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8F9F@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 7 Jan 2003 10:10:16 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > So, I believe the primary issue before the WG is whether > > > to accept: > > > - The RDDP protocols must allow implementations over > > > unmodified TCP/IP stacks. > > > as a requirement. > > > > I don't believe anyone has proposed or recommended otherwise. > > (MPA states an MPA-aware sender SHOULD align, but even that > > is not contradictory to the above.) > > [...] > > The point is that "optional" (MAY) is a considerably weaker > requirement than SHOULD (see RFC 2119). I believe that an > unmodified TCP stack cannot comply with a "SHOULD align" > requirement solely on the basis that the an implementer does > not want to or cannot modify the stack, in contrast to a > "MAY align" requirement. If a sender MAY be MPA-aware, and an MPA-aware sender SHOULD align, then we have an overall MAY. This was how I interpreted the draft. However, your point is well taken, and the draft could be a bit clearer on this. > [...] > > > Markers and similar mechanisms are among the ways to obtain direct > placeability of unaligned RDDP PDUs [...]. The data can't be placed until the header is received. In the unaligned case, at best only part of the data can be placed, even with markers or similar mechanism. Doing partial packet placement, while theoretically possible, has some serious practical concerns. 1. To be of any value, it implies the ability to free partial buffers for re-use, while keeping the unplaced data. Or else copying the unplaced fragment elsewhere to allow freeing the buffer. Either causes serious implementation complexities that are avoided in the aligned case. 2. Since each supported connection may have an associated unplaced fragment, the total data buffering required now scales linearly with the number of active connections. This is fundamentally different than the aligned case. Worse, if there are multiple gaps, there may be multiple unplaced fragments on a single connection. I do not believe that direct placement of unaligned PDUs achieves the objectives that the proponents of direct placement are looking for. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 10:49:15 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA18877 for ; Tue, 7 Jan 2003 10:49:15 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07G01i30890 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 11:00:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07G00J30877 for ; Tue, 7 Jan 2003 11:00:00 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA18856 for ; Tue, 7 Jan 2003 10:48:35 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07Fx8J30845; Tue, 7 Jan 2003 10:59:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07FwQJ30813 for ; Tue, 7 Jan 2003 10:58:26 -0500 Received: from d12lmsgate-4.de.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA18823 for ; Tue, 7 Jan 2003 10:47:08 -0500 (EST) Received: from d12relay02.de.ibm.com (d12relay02.de.ibm.com [9.165.215.23]) by d12lmsgate-4.de.ibm.com (8.12.3/8.12.3) with ESMTP id h07Fnm11124640; Tue, 7 Jan 2003 16:49:48 +0100 Received: from d10ml001.telaviv.ibm.com (d10ml001.telaviv.ibm.com [9.148.216.55]) by d12relay02.de.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id h07FnijM127794; Tue, 7 Jan 2003 16:49:48 +0100 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8F9F@xbl.ma.emulex.com> Subject: RE: [rddp] Alignment and TCP modification To: "Williams, Jim" Cc: rddp@ietf.org X-Mailer: Lotus Notes Release 6.0 September 26, 2002 Message-ID: From: "Vadim Makhervaks" Date: Tue, 7 Jan 2003 17:49:41 +0200 X-MIMETrack: Serialize by Router on D10ML001/10/M/IBM(Release 5.0.9a |January 7, 2002) at 07/01/2003 17:49:47 MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , inline |---------+----------------------------> | | "Williams, Jim" | | | | | | Sent by: | | | rddp-admin@ietf.o| | | rg | | | | | | | | | 07/01/03 05:10 PM| |---------+----------------------------> >-------------------------------------------------------------------------------------------------------------------------------| | | | To: rddp@ietf.org | | cc: | | Subject: RE: [rddp] Alignment and TCP modification | >-------------------------------------------------------------------------------------------------------------------------------| > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > So, I believe the primary issue before the WG is whether > > > to accept: > > > - The RDDP protocols must allow implementations over > > > unmodified TCP/IP stacks. > > > as a requirement. > > > > I don't believe anyone has proposed or recommended otherwise. > > (MPA states an MPA-aware sender SHOULD align, but even that > > is not contradictory to the above.) > > [...] > > The point is that "optional" (MAY) is a considerably weaker > requirement than SHOULD (see RFC 2119). I believe that an > unmodified TCP stack cannot comply with a "SHOULD align" > requirement solely on the basis that the an implementer does > not want to or cannot modify the stack, in contrast to a > "MAY align" requirement. If a sender MAY be MPA-aware, and an MPA-aware sender SHOULD align, then we have an overall MAY. This was how I interpreted the draft. However, your point is well taken, and the draft could be a bit clearer on this. > [...] > > > Markers and similar mechanisms are among the ways to obtain direct > placeability of unaligned RDDP PDUs [...]. The data can't be placed until the header is received. In the unaligned case, at best only part of the data can be placed, even with markers or similar mechanism. I would say that the data cannot be placed until the entire MPA segment has been received. Even in-order reception of part of MPA segment would require to keep the received part in the intermediate buffers I don't think that purpose of Markers is to enable direct data placement. Markers just help to find the next DDP header. This feature is very valuable when TCP segment is lost. In aligned case Markers allow to ensure alignment and perform direct data placement. In not aligned case, Markers reduce the size of the intermediate buffers - placement of the arriving DDP segments can be done without waiting for the lost segment. Another feature of Markers, is their transparency to the resegmenting TCP boxes. Doing partial packet placement, while theoretically possible, has some serious practical concerns. 1. To be of any value, it implies the ability to free partial buffers for re-use, while keeping the unplaced data. Or else copying the unplaced fragment elsewhere to allow freeing the buffer. Either causes serious implementation complexities that are avoided in the aligned case. 2. Since each supported connection may have an associated unplaced fragment, the total data buffering required now scales linearly with the number of active connections. This is fundamentally different than the aligned case. Worse, if there are multiple gaps, there may be multiple unplaced fragments on a single connection. I do not believe that direct placement of unaligned PDUs achieves the objectives that the proponents of direct placement are looking for. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 16:28:30 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29287 for ; Tue, 7 Jan 2003 16:28:30 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07LdMh20131 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 16:39:22 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07LdMJ20128 for ; Tue, 7 Jan 2003 16:39:22 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29272 for ; Tue, 7 Jan 2003 16:27:50 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07LbLJ19666; Tue, 7 Jan 2003 16:37:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07LalJ19306 for ; Tue, 7 Jan 2003 16:36:47 -0500 Received: from sccrmhc01.attbi.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29217 for ; Tue, 7 Jan 2003 16:25:22 -0500 (EST) Message-Id: <200301072125.QAA29217@ietf.org> Received: from cs.uchicago.edu (h005018030f43.ne.client2.attbi.com[66.31.16.32]) by sccrmhc01.attbi.com (sccrmhc01) with SMTP id <2003010721283700100128ime>; Tue, 7 Jan 2003 21:28:37 +0000 To: Black_David@emc.com cc: rddp@ietf.org Subject: Re: [rddp] Alignment and TCP modification In-Reply-To: Message from Black_David@emc.com of "Thu, 02 Jan 2003 15:21:58 EST." <277DD60FB639D511AC0400B0D068B71E0564C6CA@corpmx14.us.dg.com> References: <277DD60FB639D511AC0400B0D068B71E0564C6CA@corpmx14.us.dg.com> Date: Tue, 07 Jan 2003 16:28:36 -0500 From: Stephen Bailey Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David, I'm too woozy right this second to completely understand the details of your consensus request, so I'll just charge ahead. Perhaps stating the obvious, it seems ridiculous to require RDDP run on a TCP that performs sender alignment. RDDP must be able to run on an unmodified TCP. It seems equally ridiculous to expect the receiver to hypothesize but verify sender alignment without any information about whether the sender is even trying to provide it. An unmodified sender is not likely to end up aligning by accident. That leaves the endpoints negotiating whether the sender will try, and if so, the receiver will verify, correct? Steph > In response to my request for explanations of opposition to > requiring sender alignment: > > So, let me ask those three people, plus anyone else > who is opposed to a general approach of requiring > sender alignment to send me a brief email off the list > summarizing their viewpoint and the reasoning behind it. > > I received a total of six responses, five of which expressed > varying degrees of opposition to a sender alignment requirement: > - Somesh Gupta, Caitlin Bestler, David Borman, Randy > Stewart and Shridhar Mukund > and one that indicated that alignment could be an acceptable > compromise if it could be shown to be of practical value: > - Jim Williams > I'll leave it to these people to explain their views in further > detail if they think it's warranted (in some cases, what was sent > to me was a summary of what had already been posted to the list). > > I'm struck by the fact that a single theme was in common among > the five individuals who expressed opposition, namely: > - The RDDP specifications need to allow implementations > over unmodified TCP/IP stacks. > > The primary rationales for this are: > (1) An unmodified stack will send unaligned in some circumstances > that cannot be avoided unless one resorts to truly draconian > and unacceptable measures such as a ULP-level stop-and-wait > protocol (ULP ACK between sends drains TCP send queue ensuring > that next ULP send is aligned by TCP). > (2) Even for a hardware-accelerated ULP server implementation, > software ULP clients will be common and important. Implementing > the RDDP functionality in software over an unmodified stack for > ULP clients in this scenario is important to reduce barriers > to adoption/use. For example, this enables all of RDDP to be > implemented in user mode over an existing stack. > (3) The ability to run over an unmodified stack makes the RDDP > protocols far more applicable and hence amenable to > widespread adoption, including the above ULP client scenario. > > Beyond this, I saw a divergence of opinion on how much attention > implementers should pay to aligned vs. unaligned traffic beyond > agreement that it should not be treated as an error case. This > divergence ranged from "most traffic will be aligned, but unaligned > traffic should not run any slower than ordinary TCP traffic" to > "unaligned traffic will be common enough that implementations > need to optimize for it, even if most/all senders send aligned" > (quoted text is my paraphrase in both cases). Fortunately, the WG > does not need to settle this issue - implementers are always free > to optimize whatever cases they believe are important. Some of > the emails also raised other issues that I'll leave it to those > involved to decide whether to discuss further on this list. > > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. If this should be accepted as a requirement, > then sender alignment cannot be required of all senders (this > wouldn't rule out negotiating it in some fashion). Here are a > few more implications: > - Unmodified means 100% unmodified, not even "minor" changes. > - This would imply that a sender using an unmodified stack > could send RDDP payloads that are directly placeable > by an RDDP receiver. > - There don't appear to be any receiver implications, as > although an optimized RDDP implementation will likely > use a modified stack, I haven't seen any proposal that > requires such modifications (e.g., disallows the brute > force technique of having TCP buffer as usual and then > running RDDP functionality on the in-order TCP bytestream). > - The practical meaning of "allow" would be that an > implementation is possible without violating any MUST, > MUST NOT, SHOULD, or SHOULD NOT requirement. > The latter bullet is based on my view that "I don't want to modify > an existing stack" is not a sufficient reason by itself to disregard > a SHOULD or SHOULD NOT requirement. > > All of this is open to discussion (including the immediately > preceding SHOULD/SHOULD NOT implication). That should be sufficient > to get further discussion going ;-). I would ask that those > opposed to accepting the requirement start from the above numbered > rationale and explain what they believe to be wrong/unconvincing/ > missing/etc. > > Thanks, > --David > > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 **NEW** FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 18:03:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02234 for ; Tue, 7 Jan 2003 18:03:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07NEZB26204 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 18:14:35 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07NEYJ26201 for ; Tue, 7 Jan 2003 18:14:34 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02219 for ; Tue, 7 Jan 2003 18:03:02 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07NDbJ26108; Tue, 7 Jan 2003 18:13:37 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07N8oJ25943 for ; Tue, 7 Jan 2003 18:08:50 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA02065 for ; Tue, 7 Jan 2003 17:57:24 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 7 Jan 2003 18:00:35 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C6F3@corpmx14.us.dg.com> To: Jim.Williams@emulex.com, rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 7 Jan 2003 17:59:54 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, > > > > So, I believe the primary issue before the WG is whether > > > > to accept: > > > > - The RDDP protocols must allow implementations over > > > > unmodified TCP/IP stacks. > > > > as a requirement. > > > > > > I don't believe anyone has proposed or recommended otherwise. > > > (MPA states an MPA-aware sender SHOULD align, but even that > > > is not contradictory to the above.) > > > [...] > > > > The point is that "optional" (MAY) is a considerably weaker > > requirement than SHOULD (see RFC 2119). I believe that an > > unmodified TCP stack cannot comply with a "SHOULD align" > > requirement solely on the basis that the an implementer does > > not want to or cannot modify the stack, in contrast to a > > "MAY align" requirement. > > If a sender MAY be MPA-aware, and an MPA-aware sender > SHOULD align, then we have an overall MAY. This was how > I interpreted the draft. However, your > point is well taken, and the draft could be a bit clearer on this. That's one side of it, the other side (ellipsed in your message) is whether the receiver is under some obligation do a "reasonable job" of RDDP on traffic that arrives unaligned and what that means. If the only way to avoid "MPA-aware sender SHOULD align" is to not be MPA-aware, and the sender not being MPA-aware disables RDDP at the receiver, then RDDP does not run over unmodified TCP stacks (oops). In defense of the MPA draft, I believe it intended to separate the MPA additions to the bytestream from MPA-aware TCP, but the MPA with unmodified TCP case doesn't get sufficient attention. > > Markers and similar mechanisms are among the ways to obtain direct > > placeability of unaligned RDDP PDUs [...]. > > The data can't be placed until the header is received. In the > unaligned case, at best only part of the data can be placed, even > with markers or similar mechanism. > > Doing partial packet placement, while theoretically possible, > has some serious practical concerns. > > 1. To be of any value, it implies the ability to free partial > buffers for re-use, while keeping the unplaced data. > Or else copying the unplaced fragment elsewhere to allow > freeing the buffer. Either causes serious implementation > complexities that are avoided in the aligned case. I think that's overstated. There are "complexities", but they don't entail either copying or partial freeing. For example, one could use reference counting to not free a buffer until all of its pieces have been placed. > 2. Since each supported connection may have an associated > unplaced fragment, the total data buffering required > now scales linearly with the number of active connections. > This is fundamentally different than the aligned case. > Worse, if there are multiple gaps, there may be > multiple unplaced fragments on a single connection. Yes, but that's still better than connection count x bandwidth x round trip time for the case in which a TCP retransmit has to be waited for before placement can occur. > I do not believe that direct placement of unaligned PDUs > achieves the objectives that the proponents of direct > placement are looking for. No comment, for further discussion. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 **NEW** FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 7 18:13:56 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02653 for ; Tue, 7 Jan 2003 18:13:56 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h07NOp326826 for rddp-archive@odin.ietf.org; Tue, 7 Jan 2003 18:24:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07NOpJ26823 for ; Tue, 7 Jan 2003 18:24:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02591 for ; Tue, 7 Jan 2003 18:13:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07NO2J26773; Tue, 7 Jan 2003 18:24:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h07NLrJ26592 for ; Tue, 7 Jan 2003 18:21:53 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA02462 for ; Tue, 7 Jan 2003 18:10:27 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 7 Jan 2003 18:13:42 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C6F4@corpmx14.us.dg.com> To: steph@cs.uchicago.edu Cc: rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 7 Jan 2003 18:13:02 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Steph, > I'm too woozy right this second to completely understand the details > of your consensus request, so I'll just charge ahead. By all means. This consensus request is a bit tricky as I'm trying to establish consensus on a requirement, some of whose implications are not immediately or plainly obvious. > Perhaps stating the obvious, it seems ridiculous to require RDDP run > on a TCP that performs sender alignment. RDDP must be able to run on > an unmodified TCP. I'll note you as being in favor of the requirement. > It seems equally ridiculous to expect the receiver > to hypothesize but verify sender alignment without any information > about whether the sender is even trying to provide it. An unmodified > sender is not likely to end up aligning by accident. I'm not sure about that. For a request/response ULP where there's not enough traffic to keep data flowing in both directions all the time, alignment by accident may well happen based on draining the send queue at various points (esp. if the Nagle optimization is disabled which seems to be getting an increasing level of recommendation these days). iSCSI, NFS, and CIFS come to mind as examples of such protocols for some reason [;-)], but HTTP can also behave this way. > That leaves the endpoints negotiating whether the sender will try, and > if so, the receiver will verify, correct? That's certainly one way to approach the problem. My impression of the MPA draft is that it went in the direction of always trying to optimize for alignment in the hope that modified senders will help out by aligning in situations where unmodified ones may fail to align, but I'll be happy to have the MPA authors clarify this. Some sort of negotiation-like mechanism could also help with the "avoidance of framed vs. non-framed traffic confusion" issue that the WG has to resolve. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 **NEW** FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 8 12:19:08 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26108 for ; Wed, 8 Jan 2003 12:19:08 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h08HUPW05546 for rddp-archive@odin.ietf.org; Wed, 8 Jan 2003 12:30:25 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HUOJ05543 for ; Wed, 8 Jan 2003 12:30:25 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26043 for ; Wed, 8 Jan 2003 12:18:36 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HTDJ05471; Wed, 8 Jan 2003 12:29:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HSSJ05418 for ; Wed, 8 Jan 2003 12:28:28 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25892 for ; Wed, 8 Jan 2003 12:16:40 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D1D1D2450CA; Wed, 8 Jan 2003 12:19:54 -0500 (EST) Date: Wed, 8 Jan 2003 11:20:14 -0600 From: Caitlin Bestler Subject: Re: [rddp] Alignment and TCP modification To: Stephen Bailey , rddp@ietf.org X-Priority: 3 In-Reply-To: <200301072125.QAA29217@ietf.org> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/7/03, Stephen Bailey wrote: > >Perhaps stating the obvious, it seems ridiculous to >require RDDP run on a TCP that performs sender alignment. >RDDP must be able to run on an unmodified TCP. It seems >equally ridiculous to expect the receiver to hypothesize >but verify sender alignment without any information about >whether the sender is even trying to provide it. An >unmodified sender is not likely to end up aligning by >accident. > A DDP/MPA stack implemented on top of a conventional sockets interface could be attempting to achieve alignment by flushing. This could succeed most of the time, but fail when the unaltered stack had reasons to ignore the user level hint (such as retransmissions or congestion). Similarly a TCP-spoofing middlebox (that terminates and re-orginates the TCP connection) might be deployed even without a severe MTU change. For example, the pacing characteristics of the two networks are radically different. Such a middlebox might pass *most* TCP segments through unmodified, but reserve the right to consolidate them on retransmits, etc. Administrative methods could be deployed to allow a receiver to *know* certain connections will be aligned. For example a list of "safe" netmasks could be applied. But I do not think there is anyway to *dynamically* determine that a given connection is "safe" and is guaranteed to maintain alignment from that point forward. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 8 12:28:54 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27017 for ; Wed, 8 Jan 2003 12:28:54 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h08HeCn06695 for rddp-archive@odin.ietf.org; Wed, 8 Jan 2003 12:40:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HeBJ06691 for ; Wed, 8 Jan 2003 12:40:11 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26996 for ; Wed, 8 Jan 2003 12:28:22 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08Hd2J06612; Wed, 8 Jan 2003 12:39:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HcuJ06582 for ; Wed, 8 Jan 2003 12:38:56 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26880 for ; Wed, 8 Jan 2003 12:27:07 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3853B245096; Wed, 8 Jan 2003 12:30:22 -0500 (EST) Date: Wed, 8 Jan 2003 11:30:46 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment and TCP modification To: rddp@ietf.org Cc: Vadim Makhervaks X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/7/03, Vadim Makhervaks replying to Jim Williams wrote: >The data can't be placed until the header is received. In >the unaligned case, at best only part of the data can be >placed, even with markers or similar mechanism. > > I would say that the data cannot be placed until the >entire MPA segment has been received. Even in-order >reception of part of MPA segment would require to keep the >received part in the intermediate buffers > > I don't think that purpose of Markers is to enable >direct data placement. Markers just help to find the next >DDP header. This feature is very valuable when TCP segment >is lost. In aligned case Markers allow to ensure alignment >and perform direct data placement. In not aligned case, >Markers reduce the size of the intermediate buffers - >placement of the arriving DDP segments can be done without >waiting for the lost segment. Another feature of Markers, >is their transparency to the resegmenting TCP boxes. Markers enable detection of the DDP segment even when delivered out-of-order. This is true even when alignment has not been preserved, its just that two (or possibly more) TCP segments will have to be received to get the DDP segment. But placement is still enabled even if a prior TCP segment holding portions of prior DDP segments is still missing. Consider the very common scenario where there are many DDP segments in-flight, and only a single segment has been dropped. Even without alignment, nearly every DDP segment can be found and placed. The worst case scenario without alignment does indeed require extensive buffering (theoretically up to half of what a non-marker solution would require). But as previosly discussed, it would be a totally valid implementation decision to handle such a worst case scenario on the host processor. The protocol works even without hardware optimization of worst case scenarios. Most importantly, the effort required to support non-alligned connections does not have to have any adverse effect on the alligned connections. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 8 12:45:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27891 for ; Wed, 8 Jan 2003 12:45:22 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h08HudM08207 for rddp-archive@odin.ietf.org; Wed, 8 Jan 2003 12:56:39 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HudJ08204 for ; Wed, 8 Jan 2003 12:56:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27766 for ; Wed, 8 Jan 2003 12:44:50 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08Hu2J07663; Wed, 8 Jan 2003 12:56:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h08HseJ07565 for ; Wed, 8 Jan 2003 12:54:40 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA27625 for ; Wed, 8 Jan 2003 12:42:52 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 177992450B0; Wed, 8 Jan 2003 12:46:07 -0500 (EST) Date: Wed, 8 Jan 2003 11:46:31 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment and TCP modification To: rddp@ietf.org Cc: "Williams, Jim" X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8F9F@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/7/03, Williams, Jim wrote: > > 2. Since each supported connection may have an associated > unplaced fragment, the total data buffering required > now scales linearly with the number of active > connections. It scales linearly with the number of active unaligned connections. No additional buffering is required for aligned connections. Whether the wording is MAY or SHOULD, the intent is clear that anyone intending to make extensive use of DDP/MPA/TCP really needs to do alignment, and to ensure that their traffic is not routinely trounced by a middlebox that will destroy that alignment. It is my opinion that the vast majority of connections in the overwhelming majority of deployments will originate and maintain alignment. Relatively low cost hardware can optimize an amazingly large number of these connections. This is of great value to the end user, whether or not we can guarantee that any specific connection will be aligned. An implementation can limit the amount of buffering it wishes to allocate to support unaligned reception. This can be in the form of a limit on RNIC buffering, or in simply passing them to the host to handle (and hence limiting support to the host's capacity). This means that an RNIC will have to rate itself based on the maximum number of concurrent aligned connections and unaligned connections that it can support. It is under no obligation to divert an unlimited portion of its resources to support unaligned connections. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 9 13:15:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19589 for ; Thu, 9 Jan 2003 13:15:22 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h09IRA111739 for rddp-archive@odin.ietf.org; Thu, 9 Jan 2003 13:27:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09IRAJ11736 for ; Thu, 9 Jan 2003 13:27:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19533 for ; Thu, 9 Jan 2003 13:11:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09INCJ11573; Thu, 9 Jan 2003 13:23:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09IMPJ11497 for ; Thu, 9 Jan 2003 13:22:25 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19489; Thu, 9 Jan 2003 13:10:06 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 5024024534D; Thu, 9 Jan 2003 13:13:21 -0500 (EST) Date: Thu, 9 Jan 2003 12:13:47 -0600 From: Caitlin Bestler To: "'tsvwg@ietf.org'" Cc: "'Randall Stewart'" , "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FA8@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] RE: [Tsvwg] Concerns about SCTP Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/9/03, Williams, Jim wrote: > >Interestingly, though, RDDP has the property that all >received data can be immediately processed. There is >never any need to stop the remote sender. The only >problem occurs when data comes in faster than it >can be processed. In this case the desired behavior >is to slow down the sender a bit, not stop it. >Therefore ECN might be a better functional fit >for this requirement than rwnd. > That property is intrinsic to RDDP no matter what transport it runs over. The ULP is responsible for pacing untagged messages, but only the LLP paces tagged messages. RDDP releases all LLP buffers "immediately", meaning as quickly as the placement of the tagged/untagged data can be completed. Given that characteristic it is highly unlikely that there would be buffering resources dedicated to specific assocation or connection. Therefore the challenge, whether over MPA/TCP and/or SCTP, is to ensure that the internal bandwidth required to support DDP placement will not be swamped by all incoming data. Specifically the windows SHOULD be set so that the risk of exhausting the placement bandwith is as low as possible. However those windoes CANNOT beset so that said risk is non-existent. LLP Credits are delivered asynchronously, uncertainty on placemnt speed because of potential host bus contention or retries, and more importantly in integral amounts. Those integral amounts, spread over hundreds of thousands of connections, either force a severe under-allocation or an over-allocation. And certainly the available to-host bandwidth over a conventional host bus is statistically reliable but defintely not guaranteed. Limiting your RNIC to the host bus *guaranteed* rates would be a severe limitation. I believe most implementers will properly choose to take a slight risk of over-allocation, knowing that in the worst case the result is that a packet is dropped. A properly credited LLP packet arrives and there is no available buffer for it. No LLP ack is generated, and it is eventually retransmitted. The same thing can happen at any number of routers and switches along the route for the very same reason. Also consider that in some cases the host memory placement capacity may actually exceed the capacity of the external link layer. In that case LLP flow control deals only with network congestion, RNIC and host buffering is irrelevant. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 9 14:40:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23887 for ; Thu, 9 Jan 2003 14:40:05 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h09Jpsc19383 for rddp-archive@odin.ietf.org; Thu, 9 Jan 2003 14:51:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09JpsJ19380 for ; Thu, 9 Jan 2003 14:51:54 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23881 for ; Thu, 9 Jan 2003 14:39:29 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09JpBJ19344; Thu, 9 Jan 2003 14:51:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h09Jo2J19271 for ; Thu, 9 Jan 2003 14:50:02 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23849 for ; Thu, 9 Jan 2003 14:37:40 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id LAA17548; Thu, 9 Jan 2003 11:36:53 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Thu, 9 Jan 2003 14:36:49 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FAB@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Caitlin Bestler'" Cc: "'Randall Stewart'" , rddp@ietf.org Date: Thu, 9 Jan 2003 14:36:49 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RE: [Tsvwg] Concerns about SCTP Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > > Therefore the challenge, whether over MPA/TCP and/or SCTP, > is to ensure that the internal bandwidth required to support > DDP placement will not be swamped by all incoming data. I am repeating previous stuff here, so I will be brief. But I think the above is not necessarily desirable. Implementations may not be able to keep up with the link when average size of incoming packet is small. For those that can, flow control is not an issue; simply drop packets in the rare instance of buffer overflow. For those that can't, flow control is an important problem since incoming packet rate must be throttled to below link rate, otherwise too many packets may need to be dropped. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 11:43:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03267 for ; Fri, 10 Jan 2003 11:43:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0AGuED12832 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 11:56:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AGuDJ12829 for ; Fri, 10 Jan 2003 11:56:13 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03218 for ; Fri, 10 Jan 2003 11:43:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AGs7J12750; Fri, 10 Jan 2003 11:54:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AGriJ12731 for ; Fri, 10 Jan 2003 11:53:44 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03117 for ; Fri, 10 Jan 2003 11:40:57 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA07827; Fri, 10 Jan 2003 08:44:06 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 10 Jan 2003 11:44:05 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FAD@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Fri, 10 Jan 2003 11:44:04 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. I would say yes to this. A few more thoughts follow. What is the required performance of an RDMA receiver when receiving data from a transmitter using an unmodified TCP/IP stack? 1. Only functional correctness matters, performance is not an issue. 2. Performance must be as good as conventional (non-RDMA) TCP. 3. Performance must be significantly better than conventional (non-RDMA) TCP otherwise RDMA should not be used in this case. A second question is: - Is the case of a large number of slow clients using unmodified TCP/IP stacks talking to a fast server using an RDMA accelerated NIC an important design point? If the answer to the second question is YES, then I believe that the answer to the first question is #3. If the answer to the second question is NO, then I believe the answer to the first question is #1, since in practice no one will use RDMA over an unmodified TCP/IP stack. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 12:03:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03854 for ; Fri, 10 Jan 2003 12:03:37 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0AHFsg14750 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 12:15:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHFrJ14747 for ; Fri, 10 Jan 2003 12:15:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03834 for ; Fri, 10 Jan 2003 12:03:06 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHF7J14729; Fri, 10 Jan 2003 12:15:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHF0J14706 for ; Fri, 10 Jan 2003 12:15:00 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03815 for ; Fri, 10 Jan 2003 12:02:12 -0500 (EST) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 6049D1DAC; Fri, 10 Jan 2003 11:05:29 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Fri, 10 Jan 2003 11:05:28 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] Alignment and TCP modification Date: Fri, 10 Jan 2003 11:05:25 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A1707628687FC7@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Alignment and TCP modification Thread-Index: AcK4x7bNxJmWtWvMQcOz6EhGp58yZgAAWI2A From: "Culley, Paul" To: "Williams, Jim" , , X-OriginalArrivalTime: 10 Jan 2003 17:05:28.0608 (UTC) FILETIME=[79CB8200:01C2B8CA] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h0AHF0J14707 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Of course all issues surrounding performance are really for the market to decide. That said, I can see very interesting scenarios where large numbers of slow clients using unmodified stacks is NOT a design point. Implementations that perform poorly in this case are perfectly acceptable and, if cheaper, preferred. Most of these scenarios involve a machine room or small campus environment, which also makes deployment of RDDP reasonable in the near term. I suspect that by the time we can get lots of small clients attached, they will probably have to deal with the WAN. This adds other considerations like security (can you say IPSEC) which may make more expensive NICs both necessary and acceptable. Paul Culley HP Sr. Fellow 281-514-5543 -----Original Message----- From: Williams, Jim [mailto:Jim.Williams@emulex.com] Sent: Friday, January 10, 2003 10:44 AM To: 'Black_David@emc.com'; rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. I would say yes to this. A few more thoughts follow. What is the required performance of an RDMA receiver when receiving data from a transmitter using an unmodified TCP/IP stack? 1. Only functional correctness matters, performance is not an issue. 2. Performance must be as good as conventional (non-RDMA) TCP. 3. Performance must be significantly better than conventional (non-RDMA) TCP otherwise RDMA should not be used in this case. A second question is: - Is the case of a large number of slow clients using unmodified TCP/IP stacks talking to a fast server using an RDMA accelerated NIC an important design point? If the answer to the second question is YES, then I believe that the answer to the first question is #3. If the answer to the second question is NO, then I believe the answer to the first question is #1, since in practice no one will use RDMA over an unmodified TCP/IP stack. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 12:30:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04790 for ; Fri, 10 Jan 2003 12:30:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0AHggh16442 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 12:42:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHggJ16439 for ; Fri, 10 Jan 2003 12:42:42 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04767 for ; Fri, 10 Jan 2003 12:29:55 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHg2J16404; Fri, 10 Jan 2003 12:42:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHfBJ16333 for ; Fri, 10 Jan 2003 12:41:11 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04689 for ; Fri, 10 Jan 2003 12:28:25 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0DA8C24562E; Fri, 10 Jan 2003 12:31:41 -0500 (EST) Date: Fri, 10 Jan 2003 11:32:10 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment and TCP modification To: "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FAD@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/10/03, Williams, Jim wrote: >> So, I believe the primary issue before the WG is whether to accept: >> - The RDDP protocols must allow implementations over >> unmodified TCP/IP stacks. >> as a requirement. > >I would say yes to this. A few more thoughts follow. > >What is the required performance of an RDMA receiver when >receiving data from a transmitter using an unmodified >TCP/IP stack? > > 1. Only functional correctness matters, performance is > not an issue. > > 2. Performance must be as good as conventional > (non-RDMA) TCP. > > 3. Performance must be significantly better than > conventional (non-RDMA) TCP otherwise RDMA should not be > used in this case. > I could see performance falling to 99% of TCP, where the TCP stack had pre-posted reads. In a worst case scenario the lack of allignment eliminates all optimizations for that one-directional stream. Then, no matter how valiant the DDP stack tried, it would be hard to overcome the inherent penalties of the MPA markers. That would especially be true if the connection were very reliable (in the statistical sense) and dropped packets were very rare. That's essentially agreeing with #2, with hints of #3 if compared to a non-optimized TCP stack. #1 is clearly unacceptable from any viewpoint other than it could be the result of absolute minimalism in drafting the standard. I would favor a SHOULD that unalligned performance be at least functionally equivalent of TCP performance. >A second question is: > > - Is the case of a large number of slow clients using > unmodified TCP/IP stacks talking to a fast server using > an RDMA accelerated NIC an important design point? > The fast server would be using an aligned stack. Maintaining alignment on reception over an unmodified TCP stack is far easier (there is less for the local stack to optimize). So the full benefits of alignment would be achieved in at least one direction. The server also recieves the benefit of choosing what order to deliver tagged buffers. As previously discussed, for a storage oriented server this can be a considerable benefit. BTW, am I correct that the only spec issue that depends upon these questions is that of the header CRC? A header CRC is indeed justified if we determine the need for higher performance on an unalligned stream. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 12:34:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04943 for ; Fri, 10 Jan 2003 12:34:22 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0AHkcZ16622 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 12:46:38 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHkcJ16619 for ; Fri, 10 Jan 2003 12:46:38 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04910 for ; Fri, 10 Jan 2003 12:33:51 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHk1J16576; Fri, 10 Jan 2003 12:46:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AHjiJ16558 for ; Fri, 10 Jan 2003 12:45:44 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04875 for ; Fri, 10 Jan 2003 12:32:53 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B5EBB24563C; Fri, 10 Jan 2003 12:36:09 -0500 (EST) Date: Fri, 10 Jan 2003 11:36:38 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment and TCP modification To: "Culley, Paul" , rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A1707628687FC7@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/10/03, Culley, Paul wrote: >Of course all issues surrounding performance are really for the market to decide. > >That said, I can see very interesting scenarios where >large numbers of slow clients using unmodified stacks is >NOT a design point. Implementations that perform poorly >in this case are perfectly acceptable and, if cheaper, >preferred. Most of these scenarios involve a machine room >or small campus environment, which also makes deployment >of RDDP reasonable in the near term. > >I suspect that by the time we can get lots of small >clients attached, they will probably have to deal with the >WAN. This adds other considerations like security (can >you say IPSEC) which may make more expensive NICs both >necessary and acceptable. > Security may be handled by a VPN router, or it might not be an issue for the specific ULP. Some examples of the latter: RDMA to support delivery of streaming media and/or web content, RDMA to support a remote sealed-storage SAN server (where the content is both encrypted and decrypted at the client end). Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 16:15:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16444 for ; Fri, 10 Jan 2003 16:15:44 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0ALS6n30252 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 16:28:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ALS5J30249 for ; Fri, 10 Jan 2003 16:28:05 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16395 for ; Fri, 10 Jan 2003 16:15:13 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ALRBJ30213; Fri, 10 Jan 2003 16:27:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ALQnJ30198 for ; Fri, 10 Jan 2003 16:26:49 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16385 for ; Fri, 10 Jan 2003 16:13:57 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id NAA15378; Fri, 10 Jan 2003 13:17:10 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 10 Jan 2003 16:17:10 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FAF@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Caitlin Bestler'" , rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Fri, 10 Jan 2003 16:17:07 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin Bestler wrote: > BTW, am I correct that the only spec issue that depends upon > these questions is that of the header CRC? CRC question is secondary. Primary goal was to move incrementally closer to consensus on alignment and markers. Answers to these questions may or may not help, but that was the goal. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 10 17:54:24 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19136 for ; Fri, 10 Jan 2003 17:54:24 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0AN6lQ04298 for rddp-archive@odin.ietf.org; Fri, 10 Jan 2003 18:06:47 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AN6lJ04295 for ; Fri, 10 Jan 2003 18:06:47 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19128 for ; Fri, 10 Jan 2003 17:53:52 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AN64J04273; Fri, 10 Jan 2003 18:06:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0AN5vJ04258 for ; Fri, 10 Jan 2003 18:05:57 -0500 Received: from magic.adaptec.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19119 for ; Fri, 10 Jan 2003 17:53:02 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.11.6+Sun/8.11.6) with ESMTP id h0AMuJj26479; Fri, 10 Jan 2003 14:56:19 -0800 (PST) Received: from aimexc02.adaptec.com (aimexc02.corp.adaptec.com [162.62.62.40]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id OAA12499; Fri, 10 Jan 2003 14:56:19 -0800 (PST) Received: by aimexc02.corp.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Fri, 10 Jan 2003 14:56:18 -0800 Message-ID: From: "Mukund, Shridhar" To: "'Williams, Jim'" , "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Fri, 10 Jan 2003 14:56:08 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The answer to the second question is a *loud* YES. Hence, the answer to the first question is #3 and possibly #3+, if there were one. Some related comments: * We seem to keep coming back to the *assumption* that RDMA is only for data center or machine room scenarios. The applicability of RDMA/RDDP is *far wider* than that. In the life time of this IETF standard, we are talking about convergent NICs that will make sense for any application that needs movement of ?significant? amounts of data. It may make sense to agree upon a list of compelling applications for the next 20+ years. * Ironically, it is the data center application that, *by-design*, has the luxury of negligible out-of-order-ness and/or packet drops. Since I may be delving in a few not-so-obvious and possibly debatable assumptions, I will take this topic up in a seperate thread. -Shridhar Mukund > -----Original Message----- > From: Williams, Jim [mailto:Jim.Williams@emulex.com] > Sent: Friday, January 10, 2003 8:44 AM > To: 'Black_David@emc.com'; rddp@ietf.org > Subject: RE: [rddp] Alignment and TCP modification > > > > So, I believe the primary issue before the WG is whether to accept: > > - The RDDP protocols must allow implementations over > > unmodified TCP/IP stacks. > > as a requirement. > > I would say yes to this. A few more thoughts follow. > > What is the required performance of an RDMA receiver when > receiving data from a transmitter using an unmodified TCP/IP stack? > > 1. Only functional correctness matters, performance is > not an issue. > > 2. Performance must be as good as conventional (non-RDMA) TCP. > > 3. Performance must be significantly better than conventional > (non-RDMA) TCP otherwise RDMA should not be used in this case. > > A second question is: > > - Is the case of a large number of slow clients using > unmodified TCP/IP stacks talking to a fast server > using an RDMA accelerated NIC an important design point? > > If the answer to the second question is YES, then I believe > that the answer to the first question is #3. > > If the answer to the second question is NO, then I believe > the answer to the first question is #1, since in practice > no one will use RDMA over an unmodified TCP/IP stack. > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jan 12 12:30:03 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12453 for ; Sun, 12 Jan 2003 12:30:03 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0CHhIo15710 for rddp-archive@odin.ietf.org; Sun, 12 Jan 2003 12:43:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0CHhIJ15707 for ; Sun, 12 Jan 2003 12:43:18 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12440 for ; Sun, 12 Jan 2003 12:29:03 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0CHg8J15676; Sun, 12 Jan 2003 12:42:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0CHf3J15642 for ; Sun, 12 Jan 2003 12:41:03 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA12427 for ; Sun, 12 Jan 2003 12:27:17 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id A0979244A70; Sun, 12 Jan 2003 12:30:35 -0500 (EST) Date: Sun, 12 Jan 2003 11:31:05 -0600 From: Caitlin Bestler To: "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FAB@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] LLP Flow Control (was Re: Concerns about SCTP) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/9/03, Williams, Jim wrote: > > >> -----Original Message----- >> From: Caitlin Bestler [mailto:cait@asomi.com] >> >> Therefore the challenge, whether over MPA/TCP and/or >> SCTP, is to ensure that the internal bandwidth required >> to support DDP placement will not be swamped by all >> incoming data. > >I am repeating previous stuff here, so I will be brief. >But I think the above is not necessarily desirable. >Implementations may not be able to keep up with the >link when average size of incoming packet is small. >For those that can, flow control is not an issue; simply >drop packets in the rare instance of buffer overflow. >For those that can't, flow control is an important >problem since incoming packet rate must be throttled to >below link rate, otherwise too many packets may need >to be dropped. > I believe the nature of an optimum push-based LLP / DDP interface changes the dynamics of handling many small packets. With a non-DDP interface, small packets can result in a temporary drop in the available resources. The implementation can choose to release the unused fragments, thereby consuming more CPU power and ultimately slowing down the rate at which the ULPs drain the queue. Or it can stick with a simpler allocation scheme, optimizing for faster release of the entire buffer rather than partial release of the unused portion. When a large number of small messages arrive, there is more fragmentation and less available CPU power or there is wasted space (i.e. the system's capacity is best measured by the number of packets, not number of bytes). With non-DDP interfaces, there is typically a hold time on these buffers. Data is not immediately transferred to the ULP. Therefore the small-packet-stress induced state will have a duration that could be long enough to warrant use of LLP flow control. An push-based LLP/DDP interface, by contrast, does not require hold time. The DDP can be required to place the data *immediately*, and to take note of completion status immediately. There can indeed be contention for the host memory bus. That can lead to delays and even an overload of the host bus capacity under worst case conditions. However, I cannot see how such a condition could ever sustain itself long enough to warrant use of LLP flow control. The *maximum* delay through the host bus should be magnitudes lower than the round-trip time over the network. Basically, LLP deliveries to the DDP layer simply cannot be held long enough to allow something as slow as LLP flow control to work. In the event that placement/completions cannot be delivered to ULP memory quickly enough then packets will have to be dropped. That is a given. But I cannot see how LLP flow control can be applied to a situation where the resource utilization duration is miniscule compared to the round-trip time. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 13 15:44:13 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23529 for ; Mon, 13 Jan 2003 15:44:13 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0DKw1U21727 for rddp-archive@odin.ietf.org; Mon, 13 Jan 2003 15:58:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DKw0J21724 for ; Mon, 13 Jan 2003 15:58:00 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23521 for ; Mon, 13 Jan 2003 15:43:42 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DKvGJ21679; Mon, 13 Jan 2003 15:57:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DKv0J21624 for ; Mon, 13 Jan 2003 15:57:00 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23484 for ; Mon, 13 Jan 2003 15:42:42 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id MAA29065 for ; Mon, 13 Jan 2003 12:46:00 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 13 Jan 2003 15:46:00 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FB8@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Mon, 13 Jan 2003 15:45:59 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , If software implementations are common and important, that suggests there is value in making the CRC optional. For hardware implementations, once the CRC is implemented, it might as well be used always. However for software there is significant overhead for the CRC, and some reasons why it might be unnecessary. 1. IPSec authentication is in use. 2. Connection is on a private network with good quality switches/routers and good link CRCs. So TCP checksum is enough. 3. ULP does its own end to end integrity check. > -----Original Message----- > From: Mukund, Shridhar [mailto:Shridhar_Mukund@adaptec.com] > Sent: Friday, January 10, 2003 5:56 PM > To: Williams, Jim; 'Black_David@emc.com'; rddp@ietf.org > Subject: RE: [rddp] Alignment and TCP modification > > > > > The answer to the second question is a *loud* YES. Hence, the > answer to the first question is #3 and possibly #3+, if there > were one. > > -Shridhar Mukund > > > What is the required performance of an RDMA receiver when > > receiving data from a transmitter using an unmodified TCP/IP stack? > > > > 1. Only functional correctness matters, performance is > > not an issue. > > > > 2. Performance must be as good as conventional (non-RDMA) TCP. > > > > 3. Performance must be significantly better than conventional > > (non-RDMA) TCP otherwise RDMA should not be used in this case. > > > > A second question is: > > > > - Is the case of a large number of slow clients using > > unmodified TCP/IP stacks talking to a fast server > > using an RDMA accelerated NIC an important design point? _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 13 17:13:59 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26041 for ; Mon, 13 Jan 2003 17:13:59 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0DMRn427493 for rddp-archive@odin.ietf.org; Mon, 13 Jan 2003 17:27:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMRnJ27490 for ; Mon, 13 Jan 2003 17:27:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26026 for ; Mon, 13 Jan 2003 17:13:28 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMR7J27440; Mon, 13 Jan 2003 17:27:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMQGJ27396 for ; Mon, 13 Jan 2003 17:26:16 -0500 Received: from magic.adaptec.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26002 for ; Mon, 13 Jan 2003 17:11:54 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.11.6+Sun/8.11.6) with ESMTP id h0DMFDj04978; Mon, 13 Jan 2003 14:15:13 -0800 (PST) Received: from aimexc02.adaptec.com (aimexc02.corp.adaptec.com [162.62.62.40]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id OAA12173; Mon, 13 Jan 2003 14:15:13 -0800 (PST) Received: by aimexc02.corp.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Mon, 13 Jan 2003 14:15:12 -0800 Message-ID: From: "Mukund, Shridhar" To: "'Williams, Jim'" , rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Mon, 13 Jan 2003 14:15:11 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , and, 4. Several applications can tolerate TCP chksm escape. ( I am assuming that you are talking only about payload CRC and not header CRC ) > -----Original Message----- > From: Williams, Jim [mailto:Jim.Williams@emulex.com] > Sent: Monday, January 13, 2003 12:46 PM > To: rddp@ietf.org > Subject: RE: [rddp] Alignment and TCP modification > > > If software implementations are common and important, > that suggests there is value in making the CRC optional. > For hardware implementations, once the CRC is implemented, > it might as well be used always. > > However for software there is significant overhead for the > CRC, and some reasons why it might be unnecessary. > > 1. IPSec authentication is in use. > > 2. Connection is on a private network with good quality > switches/routers and good link CRCs. So TCP checksum > is enough. > > 3. ULP does its own end to end integrity check. > > > > -----Original Message----- > > From: Mukund, Shridhar [mailto:Shridhar_Mukund@adaptec.com] > > Sent: Friday, January 10, 2003 5:56 PM > > To: Williams, Jim; 'Black_David@emc.com'; rddp@ietf.org > > Subject: RE: [rddp] Alignment and TCP modification > > > > > > > > > > The answer to the second question is a *loud* YES. Hence, the > > answer to the first question is #3 and possibly #3+, if there > > were one. > > > > -Shridhar Mukund > > > > > What is the required performance of an RDMA receiver when > > > receiving data from a transmitter using an unmodified > TCP/IP stack? > > > > > > 1. Only functional correctness matters, performance is > > > not an issue. > > > > > > 2. Performance must be as good as conventional > (non-RDMA) TCP. > > > > > > 3. Performance must be significantly better than conventional > > > (non-RDMA) TCP otherwise RDMA should not be used > in this case. > > > > > > A second question is: > > > > > > - Is the case of a large number of slow clients using > > > unmodified TCP/IP stacks talking to a fast server > > > using an RDMA accelerated NIC an important design point? > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 13 17:28:54 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26357 for ; Mon, 13 Jan 2003 17:28:54 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0DMgi228775 for rddp-archive@odin.ietf.org; Mon, 13 Jan 2003 17:42:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMgiJ28772 for ; Mon, 13 Jan 2003 17:42:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26349 for ; Mon, 13 Jan 2003 17:28:23 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMg2J28741; Mon, 13 Jan 2003 17:42:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMe5J28638 for ; Mon, 13 Jan 2003 17:40:05 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26292 for ; Mon, 13 Jan 2003 17:25:44 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id OAA01988; Mon, 13 Jan 2003 14:29:02 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 13 Jan 2003 17:29:02 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FB9@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Mukund, Shridhar'" , rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification Date: Mon, 13 Jan 2003 17:29:01 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Mukund, Shridhar [mailto:Shridhar_Mukund@adaptec.com] > Sent: Monday, January 13, 2003 5:15 PM > To: Williams, Jim; rddp@ietf.org > Subject: RE: [rddp] Alignment and TCP modification > > > However for software there is significant overhead for the > > CRC, and some reasons why it might be unnecessary. > > > > 1. IPSec authentication is in use. > > > > 2. Connection is on a private network with good quality > > switches/routers and good link CRCs. So TCP checksum > > is enough. > > > > 3. ULP does its own end to end integrity check. > > and, > > 4. Several applications can tolerate TCP chksm escape. > ( I am assuming that you are talking only about payload > CRC and not header CRC ) I was suggesting both header and payload CRCs be optional, however should be separate option, as you point out some may need header CRC but not payload CRC. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 13 17:44:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26870 for ; Mon, 13 Jan 2003 17:44:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0DMvvV29382 for rddp-archive@odin.ietf.org; Mon, 13 Jan 2003 17:57:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMvuJ29379 for ; Mon, 13 Jan 2003 17:57:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26860 for ; Mon, 13 Jan 2003 17:43:35 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMv2J29341; Mon, 13 Jan 2003 17:57:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0DMuYJ29322 for ; Mon, 13 Jan 2003 17:56:34 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26828 for ; Mon, 13 Jan 2003 17:42:12 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id A7727244C62; Mon, 13 Jan 2003 17:45:31 -0500 (EST) Date: Mon, 13 Jan 2003 16:46:04 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment and TCP modification To: "Williams, Jim" Cc: "'Mukund, Shridhar'" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FB9@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/13/03, Williams, Jim wrote: >> >> > However for software there is significant overhead for >> > the CRC, and some reasons why it might be unnecessary. >> > >> > 1. IPSec authentication is in use. >> > >> > 2. Connection is on a private network with good >> > quality switches/routers and good link CRCs. So TCP >> > checksum is enough. >> > >> > 3. ULP does its own end to end integrity check. >> >> and, >> >> 4. Several applications can tolerate TCP chksm escape. ( >> I am assuming that you are talking only about payload >> CRC and not header CRC ) > >I was suggesting both header and payload CRCs be optional, >however should be separate option, as you point out some >may need header CRC but not payload CRC. Deciding to rely upon ULP validation of the payload is valid. But if the DDP Header is received in error, the payload may have been placed anywhere that is write-enabled for the receiving ULP. Allowing omission of a CRC covering the DDP Header strikes me as very dangerous, and possibly a security hole. If there is a consensus to support splitting CRC coverage between header and payload, I could see making payload coverage optional. But the extra complexity is just one more argument for sticking with a single CRC. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 13 20:08:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00359 for ; Mon, 13 Jan 2003 20:08:51 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0E1Mja05623 for rddp-archive@odin.ietf.org; Mon, 13 Jan 2003 20:22:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0E1MjJ05620 for ; Mon, 13 Jan 2003 20:22:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00351 for ; Mon, 13 Jan 2003 20:08:20 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0E1M5J05609; Mon, 13 Jan 2003 20:22:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0E1LrJ05590 for ; Mon, 13 Jan 2003 20:21:53 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00341 for ; Mon, 13 Jan 2003 20:07:28 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 13 Jan 2003 20:10:48 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C75B@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 13 Jan 2003 20:10:01 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Alignment Commentary Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , A bit over a week ago, as WG chair, I posed the issue: > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. If this should be accepted as a requirement, > then sender alignment cannot be required of all senders (this > wouldn't rule out negotiating it in some fashion). Looking back over discussion since then, I've seen a fair amount of discussion of implications. I'm going to comment on some of that discussion, but leave the issue open for at least the rest of this week for further discussion. The key implications seem to involve receiver behavior when the sender is not modified (and in particular winds up sending unaligned due to factors beyond its control), as Jim Williams asked: > What is the required performance of an RDMA receiver when > receiving data from a transmitter using an unmodified TCP/IP stack? This is somewhat subtle, as I believe the bulk of the answer is to be found in what I originally wrote after commenting on the divergence of views on this topic: > Fortunately, the WG does not need to settle this issue - implementers > are always free to optimize whatever cases they believe are important. or in Paul Culley's words: > Of course all issues surrounding performance are really for the market > to decide. So, I would suggest that trying to write MUST or SHOULD requirements for receiver performance in this scenario is probably not a good idea. It's also unlikely to succeed, given the clear divergence of views on this subject - I believe I've seen support for all three possibilities that Jim listed: > 1. Only functional correctness matters, performance is not an issue. > > 2. Performance must be as good as conventional (non-RDMA) TCP. > > 3. Performance must be significantly better than conventional > (non-RDMA) TCP otherwise RDMA should not be used in this case. Given this, the most appropriate thing for the WG to do may be to ensure that implementers can make choices as they see fit and that the market has the opportunity to decide - this would be based on following the following guidance from Section 6 of RFC 2119 on usage of requirements imperatives such as MUST, SHOULD, and MAY: For example, they must not be used to try to impose a particular method on implementors where the method is not required for interoperability. For this specific issue, this guidance could be realized as a WG approach of avoiding design approaches and decisions that have the effect of ruling out any of the three alternative views listed above. I would ask people to consider this approach in further discussion. I also want to remind people of an important point related to the following comment that Paul Culley made: > That said, I can see very interesting scenarios where large numbers > of slow clients using unmodified stacks is NOT a design point. > Implementations that perform poorly in this case are perfectly > acceptable and, if cheaper, preferred. Most of these scenarios > involve a machine room or small campus environment, which also > makes deployment of RDDP reasonable in the near term. As written, Paul's comment is a reasonable rationale for why implementations matching view 1 above are important. I want to caution against moving beyond what Paul wrote to argue that this class of deployment scenario is the only one worth designing RDDP protocols for - that sort of approach runs counter the IETF principle of not designing protocols that are intended solely for operation on private networks, even though many IETF protocols are widely deployed on private networks. No criticism of Paul's remarks is intended - what he wrote is a nice example of staying on the correct side of this distinction while making his case for the class of implementation he believes is important. The alignment issue is still open. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 10:08:16 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26131 for ; Tue, 14 Jan 2003 10:08:16 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0EFMPB03396 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 10:22:25 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EFMPJ03393 for ; Tue, 14 Jan 2003 10:22:25 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26115 for ; Tue, 14 Jan 2003 10:07:44 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EFLQJ03318; Tue, 14 Jan 2003 10:21:27 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EFKAJ03257 for ; Tue, 14 Jan 2003 10:20:10 -0500 Received: from stewart.chicago.il.us (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26066 for ; Tue, 14 Jan 2003 10:05:28 -0500 (EST) Received: from stewart.chicago.il.us (stewart.chicago.il.us [127.0.0.1]) by stewart.chicago.il.us (8.12.6/8.12.3) with ESMTP id h0EF8ZDc002166; Tue, 14 Jan 2003 09:08:41 -0600 (CST) (envelope-from randall@stewart.chicago.il.us) Message-ID: <3E2427F1.3000108@stewart.chicago.il.us> Date: Tue, 14 Jan 2003 09:08:33 -0600 From: Randall Stewart User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.1) Gecko/20021005 X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Williams, Jim" CC: "'rddp@ietf.org'" Subject: Re: [rddp] RE: I-D ACTION:draft-stewart-rddp-sctp-01.txt References: <3356669BBE90C448AD4645C843E2BF289B8F7F@xbl.ma.emulex.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Williams, Jim wrote: > Some quick thoughts: > > > >> The DDP layer is responsible for submitting messages to the SCTP >> layer that conform to the PMTU. The SCTP layer MAY assume this has >> been done. The DDP layer MAY also wish to set the fragmentation >> point (usually termed SCTP_MAXSEG) to a low enough value to assure >> that PMTU changes will not cause IP fragmentation. > > > I don't have a problem with the "MAY" here, but usually that > won't be a good idea. Receiver performance will usually be optimized > by maximizing the fragment size. Keeping all the chunk headers > up front for locality of reference can only help, too. > > IP fragmentation on dynamic PMTU reductions (infrequent) seems > like a small price to pay. Probing for larger PMTUs seems like > a good application for multiple SCTP chunks. Is there a NOP > type chunk that could be used to pad out the PMTU to probe > for larger size without needing to bother the DDP layer > until the larger PMTU is actually observed (usually it won't > be)? > As a matter of fact, yes there is! And a couple of implementations at bakeoffs have it implemented (not my FreeBSD implementation however)... The Heartbeat chunk is a completely implementation defined chunk.. it can be as big or as small as the sending implementation wants but it MUST be sent back intact! So one could use it for PMTU discovery... interestingly enough it also does a PMTU discovery in both directions :> Do we want to add a hint in the document on this? Maybe we need a section titled: "Helpful SCTP implementation options" or something to that effect > >> This mapping places an entire SCTP association into a specific DDP >> mode: DDP or DDP+RDMA. It is presumed that the handling of incoming >> data chunks for DDP enabled associations is sufficiently different >> than for routine SCTP associations that it is undesirable to mix DDP >> and non-DDP streams in a single association. An application that >> needs to mix DDP and non-DDP traffic must use more than a single >> association. > > > This is not completely clear to me. Does this mean that if the DDP+RDMA > adaption indication is done for a specific association, then ALL traffic > over that association is DDP+RDMA, or does it mean that DDP+RDMA may be > negotiated and used by applications, with initial negotiation done in > non-DDP mode? The former seems much clearer and less ambiguous, but > not sure if that is how SCTP defines adoption indications. I think the Former is the proposal.. basically you get a Adatpion Indication at association setup.. and this places the ENTIRE association in the DDP/RDMA mode. > > > Using the 32 bit Payload Protocol Identifier to carry one bit of > information looks less than optimal. Also, with the current definition, > the DDP and RDMA headers end up being unaligned. I think some additional > thought should go into optimizing the combined header format. > I don't see how we can use more than a bit of it.. The PPI is meant to identify to network elements and middle boxes as well as the application what data is present in the chunk. To attempt to "harness" the entire PPI is NOT in accordance with how IANA will be assigning the PPI... using one or two of them, sort of like a port number, IS more in line with what they were designed for... The PPI is required to be assigned by IANA... I doubt you can get a block of PPI's to represent RDDP... a few yes a block no. > > Regarding ordering, I would strongly support the preferred option > of using unordered delivery. The argument can (and has) been made > that it is fully compliant with ordered transport semantics to > do out of order placement, however there will be counter arguments. > Some of those counter arguments actually seem to me to have > some merit. Regardless, even if those arguments could be won, > I don't see the down side of using unordered delivery. If nothing > else it should save a lot of time and reduce risk of rejection. > Agreed :> > I would suggest that using SCTP ordered delivery be considered > as a mechanism for providing IN ORDER placement and > IN ORDER delivery. This could be an optional mode, but there > may be applications where this is of significant value. > sounds viable... > Regarding the choice between option 5.1 and 5.2, I would > slightly favor 5.2 for its simplicity. However, only if > this can be done without any interoperability problems with > normal SCTP implementations. > There are currently socket options that are in the socketsapi document that allow one to disable SCTP fragmentation... so yep.. it is well within reason to do so... > Section 7.2 looks incomplete to me. Error shutdown and > resetting of a stream needs to end to end coordination and > requires a state machine on each end. Maybe it's simpler > than I think, but I think it should be documented more > clearly. Overall, I have some concern whether the > complexity is justified by the value. > > Section 7.1 table entry "DDP+RDMA" looks slightly off. > This is a XML issue and I will see if I can get it fixed in the next pass :> >>If the local endpoint only indicated >> RDMA, then ONLY the procedures in >> [RDMA-Draft] MUST be followed. > > > Should be DDP rather than RDMA. Formatting needs to be > tweaked also. I will get this one changed.. one of the items we missed last time... R > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > > -- Randall R. Stewart randall@stewart.chicago.il.us 815-342-5222 (cell phone) _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 11:07:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29999 for ; Tue, 14 Jan 2003 11:07:42 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0EGLrS08590 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 11:21:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EGLqJ08586 for ; Tue, 14 Jan 2003 11:21:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29974 for ; Tue, 14 Jan 2003 11:07:10 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EGL5J08528; Tue, 14 Jan 2003 11:21:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EGKfJ08474 for ; Tue, 14 Jan 2003 11:20:41 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29912 for ; Tue, 14 Jan 2003 11:05:59 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA17979; Tue, 14 Jan 2003 08:09:10 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Tue, 14 Jan 2003 11:09:10 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FBA@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Alignment Commentary Date: Tue, 14 Jan 2003 11:09:08 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Monday, January 13, 2003 8:10 PM > To: rddp@ietf.org > Subject: [rddp] Alignment Commentary > > [...] > > The key implications seem to involve receiver behavior when the > sender is not modified (and in particular winds up sending unaligned > due to factors beyond its control), as Jim Williams asked: > > > What is the required performance of an RDMA receiver when > > receiving data from a transmitter using an unmodified TCP/IP stack? > > This is somewhat subtle, as I believe the bulk of the answer > is to be found in what I originally wrote after commenting on the > divergence of views on this topic: > > > Fortunately, the WG does not need to > > settle this issue - implementers > > are always free to optimize whatever > > cases they believe are important. > > or in Paul Culley's words: > > > Of course all issues surrounding > > performance are really for > > the market to decide. This is a little tricky. I certainly agree that implementers are and should be free to make performance tradeoffs. It is not for this group to mandate performance. My use of lower case "must" and "required" were not intended to suggest the upper case equivalent. On the other hand, this group does need to make the protocol definition decisions that feed directly into these implementation performance issues. It's easy to say "all of the above; let the market sort it out." But there are a couple issues to keep in mind. It makes it a lot harder for an implementation to hit its own performance objective if it can't assume anything about the remote end of its connection (i.e. whether the remote end will align.) Also, it doesn't make sense to over generalize the protocol to cover cases that make no sense. I am quite skeptical about running RDDP in a mode that results in both ends having performance no greater than conventional TCP. Why use RDDP at all in that case? One also ought not to treat too lightly the issue of performance interoperability. Consider the case where Vendor A has a high performance product, and vendor B also has a high performance product. But each vendor's product is only high performance when connected to a like product. When vendor A talks to vendor B, the performance is terrible (but functionally correct). This may not be the worst thing in the world, and the market will ultimately sort it out. But if this is common, it constitutes a failure of this WG to do its job properly. The market will ultimately create a more restrictive de facto standard than the official IETF standard, but this is not a good thing to plan on. So I am left with the following thoughts: 1. RDDP over unmodified TCP is important, and will result in unaligned PDUs being common. 2. Senders may align PDUs, but there is no MUST or SHOULD associated with alignment. 3. Receivers may choose that only when talking to an aligning sender will high performance (better than non-RDDP TCP) be achieved. In this case the receiver should not even bother to negotiate use of RDDP when talking to RDDP over an unmodified TCP. There is no compelling reason to require a receiver that depends on sender alignment for performance to negotiate use of RDDP with a non-aligning sender. It should be acceptable for such a receiver to simply refuse to negotiate use of RDDP and use conventional non-RDDP connection in that case. 4. Other receiver implementations may choose to achieve high performance with either aligning or non-aligning senders. 5. Some method needs to be created for a sender to notify the receiver of ability to align. There are strong arguments that this should be done at the transport, not the RDDP layer. (i.e. a new TCP option on the SYN segment). ============ This line or reasoning quickly leads to the realization that three modes of RDDP would be supported: unaligned TCP, aligned TCP, and aligned SCTP. Not surprisingly, my own preference would be to limit this to two: unaligned TCP and aligned SCTP. However three modes seems like a reasonable compromise, if that is how the WG wants to proceed. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 13:37:53 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06170 for ; Tue, 14 Jan 2003 13:37:53 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0EIq8S21129 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 13:52:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EIq8J21126 for ; Tue, 14 Jan 2003 13:52:08 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06154 for ; Tue, 14 Jan 2003 13:37:21 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EIp8J21090; Tue, 14 Jan 2003 13:51:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EIoGJ21065 for ; Tue, 14 Jan 2003 13:50:16 -0500 Received: from mms1.broadcom.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06116 for ; Tue, 14 Jan 2003 13:35:30 -0500 (EST) Received: from 63.70.210.1 by mms1.broadcom.com with ESMTP (Broadcom MMS1 SMTP Relay (MMS v5.5.0)); Tue, 14 Jan 2003 10:38:17 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id KAA17646; Tue, 14 Jan 2003 10:38:35 -0800 (PST) content-class: urn:content-classes:message MIME-Version: 1.0 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 14 Jan 2003 10:38:26 -0800 Message-ID: Thread-Topic: [rddp] Alignment and TCP modification Thread-Index: AcK7V0I1QJlPU/yRQnixS+h6wHEE8AAhAHQQ From: "Uri Elzur" To: "Caitlin Bestler" , "Williams, Jim" cc: "Mukund, Shridhar" , rddp@ietf.org X-WSS-ID: 123A8693379352-01-01 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h0EIoGJ21066 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit I don't think we have consensus over having a split CRC. On the contrary, one CRC has many advantages (no intention to re-has here). It will become more evident after the direction for alignment is set. If alignment is assumed, no need for two CRCs. Uri -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Monday, January 13, 2003 2:46 PM To: Williams, Jim Cc: 'Mukund, Shridhar'; rddp@ietf.org Subject: RE: [rddp] Alignment and TCP modification On 1/13/03, Williams, Jim wrote: >> >> > However for software there is significant overhead for >> > the CRC, and some reasons why it might be unnecessary. >> > >> > 1. IPSec authentication is in use. >> > >> > 2. Connection is on a private network with good >> > quality switches/routers and good link CRCs. So TCP >> > checksum is enough. >> > >> > 3. ULP does its own end to end integrity check. >> >> and, >> >> 4. Several applications can tolerate TCP chksm escape. ( >> I am assuming that you are talking only about payload >> CRC and not header CRC ) > >I was suggesting both header and payload CRCs be optional, >however should be separate option, as you point out some >may need header CRC but not payload CRC. Deciding to rely upon ULP validation of the payload is valid. But if the DDP Header is received in error, the payload may have been placed anywhere that is write-enabled for the receiving ULP. Allowing omission of a CRC covering the DDP Header strikes me as very dangerous, and possibly a security hole. If there is a consensus to support splitting CRC coverage between header and payload, I could see making payload coverage optional. But the extra complexity is just one more argument for sticking with a single CRC. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 16:52:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13075 for ; Tue, 14 Jan 2003 16:52:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0EM6wl05619 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 17:06:58 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EM6wJ05616 for ; Tue, 14 Jan 2003 17:06:58 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13064 for ; Tue, 14 Jan 2003 16:52:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EM6BJ05567; Tue, 14 Jan 2003 17:06:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0EM56J05515 for ; Tue, 14 Jan 2003 17:05:06 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13003 for ; Tue, 14 Jan 2003 16:50:17 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 463BE244828; Tue, 14 Jan 2003 16:53:35 -0500 (EST) Date: Tue, 14 Jan 2003 15:54:10 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment Commentary To: "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FBA@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/14/03, Williams, Jim wrote: > >So I am left with the following thoughts: > >1. RDDP over unmodified TCP is important, > and will result in unaligned PDUs being > common. > >2. Senders may align PDUs, but there is no > MUST or SHOULD associated with alignment. > >3. Receivers may choose that only when talking > to an aligning sender will high performance > (better than non-RDDP TCP) be achieved. > In this case the receiver should not even > bother to negotiate use of RDDP when talking > to RDDP over an unmodified TCP. > > There is no compelling reason to require > a receiver that depends on sender alignment > for performance to negotiate use of RDDP > with a non-aligning sender. > It should be acceptable > for such a receiver to simply refuse > to negotiate use of RDDP and use conventional > non-RDDP connection in that case. > >4. Other receiver implementations may choose > to achieve high performance with either > aligning or non-aligning senders. > >5. Some method needs to be created for a sender > to notify the receiver of ability to align. > There are strong arguments that this should > be done at the transport, not the RDDP layer. > (i.e. a new TCP option on the SYN segment). > >============ > >This line or reasoning quickly leads to the >realization that three modes of RDDP would be >supported: unaligned TCP, aligned TCP, and >aligned SCTP. > >Not surprisingly, my own preference would be >to limit this to two: unaligned TCP and >aligned SCTP. However three modes >seems like a reasonable compromise, if >that is how the WG wants to proceed. > > DDP over unaligned MPA/TCP still has two major benefits over the ULP using raw TCP: 1) Content can be shipped/fetched in an order optimized for actual physical storage, rather than in a precise byte order dictated by its nominal ULP form. Storage servers being able to ship/fetch disk sectors in the order they are reached is a major advantage, even when both ends of the connection are unaligned. Examples: disk striping, file fragmentation, logical files, database joins. 2) An unaligned DDP/MPA/TCP receiver still needs fewer system buffers than a plain TCP receiver. With plain TCP, the number of buffers required can be N-1 (where N is the maximum # of buffers in flight). With alignment *totally* destroyed, on a persistent basis, an unaligned receiver will be able to start placement when at least half of the in-flight segment have been received, and probably well before that. It is also important to remember that "non aligned senders" are probably best described as "non-alignment guaranteed senders". A DDP/MPA stack working on top of a conventional unmodified TCP stack will still be able to produce aligned segments *most* of the time. This all leads to a handful of decisions for the MPA draft: 1) There are major benefits to RDDP for both alignment and prompt transmittal of untagged messages. Does this justify the MPA draft overriding some default TCP "SHOULDs"? For example:an MPA-aware sender MAY or SHOULD use alignment. I believe there is a strong consensus for yes on this point. 2) Should the current MPA draft be modified with enhancements, such as the split CRC, which would benefit unaligned connections even though it would produce no benefits to aligned connections and all connections would have to pay the cost of the extra feature? I do not know if this has reached consensus yet. Some of the stated opposition was based upon a presumption that the second CRC would be too complex. That is clearly not the case. It is feasible, and would be of benefit for unaligned connections. My estimation on the portion of connections that would benefit from the feature, however, is so low that I do not consider it worthwhile. Someone needs to present a scenario where a high performance server would see a substantial number of unaligned packets. I cannot come up with one. 3) Is there a need to configure a connection as being aligned or unaligned, or should all connections be prepared to deal with unaligned packets at any time? I can see allocating my resources based upon a statistical model on the number of aligned versus unaligned packets. If that model is wrong, my performance will suffer. The same sort of engineering tradeoff is done with any application that wants to predict its bandwidth over a media that can require retransmissions because of errors. But just as I would expect to be ready to detect an error in *any* packet, no matter how reliable a particular connection has been historically, I would expect any connection to be ready to handle unaligned packets. Does anyone have a justification on why a receiver would benefit from a "heads up" on whether a connection will be aligned or not? 4) Are MPA markers sufficiently complex for a pure software stack to implement that it is worth negotiation of a "no-marker" mode? IMHO, Markers are truly ugly but ultimately not that hard to implement. It is not worth the complexity of supporting multiple modes and negotiating their use. However, there may be some merit in trying to synergize interval markers with iSCSI. But I wouldn't make that an objective, unless the Area Directors jump in to the contrary. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 18:21:30 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15763 for ; Tue, 14 Jan 2003 18:21:30 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0ENZoq11710 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 18:35:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ENZoJ11707 for ; Tue, 14 Jan 2003 18:35:50 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15756 for ; Tue, 14 Jan 2003 18:20:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ENZ7J11668; Tue, 14 Jan 2003 18:35:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0ENYkJ11598 for ; Tue, 14 Jan 2003 18:34:46 -0500 Received: from magic.adaptec.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA15701 for ; Tue, 14 Jan 2003 18:19:54 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.11.6+Sun/8.11.6) with ESMTP id h0ENNEj29777; Tue, 14 Jan 2003 15:23:14 -0800 (PST) Received: from aimexc02.adaptec.com (aimexc02.corp.adaptec.com [162.62.62.40]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id PAA24691; Tue, 14 Jan 2003 15:23:13 -0800 (PST) Received: by aimexc02.corp.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Tue, 14 Jan 2003 15:23:13 -0800 Message-ID: From: "Mukund, Shridhar" To: "'Black_David@emc.com'" , rddp@ietf.org Cc: "Munnangi, Siva" , "Mukund, Shridhar" Subject: RE: [rddp] Alignment and TCP modification Date: Tue, 14 Jan 2003 15:23:03 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Dear colleagues, In the next several email sub-threads, I intend to make the following main arguments with hopefully ample support arguments - so help me god :-) A. Alignment breaks TCP flow control and *semantics* in very *fundamental* ways. The SAR portion of DDP is an *artifact* of the current MPA solution rather than being an *intrinsic* complexity of the defacto problem statement, say "the quest". THE QUEST: The desire to direct-place every byte of every packet even if it arrives out of order (and unaligned). B. Contrary to intuition, SAR+MPA complicates design. In fact, it increases the buffering requirement instead of reducing it! I will stay implementation neutral, when discussing this. C. Lack of alignment is and will be sufficiently common that it should not be treated as an error case. Fluid flow (byte stream) property is probably the most important *intrinsic nature* and strength of TCP. D. For the fear of being stoned to death:-), I am going to assume "the quest" to be sacrosanct. It is illustrative to evaluate the incremental value proposition compared to say iSCSI-like-marker-and-DDP (the baseline alternative). On the other hand, it is also not necessary to break TCP's back bone for efficient and high-performance realization of "the quest". The suggestion is to perturb only the sheet-metal part of ROI. THE GOAL: To take RDDP TCP mapping from a state of non-solution, with sufficient risk of infant mortality, to TCP-like longevity and ubiquitous ness. Call for action (pl. save it for later): D1. Move the SAR part from DDP spec to SCTP adaptation. D2. Merge IFT+MPA and upgrade draft not to require alignment. ?How? I have no intention to derail the monumental efforts that have gone into RDMAP, Verbs, and DDP. I am humbled by that effort, esp. since I personally did not make any contribution to it. -Shridhar Mukund _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jan 14 21:40:26 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19899 for ; Tue, 14 Jan 2003 21:40:26 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0F2spH24019 for rddp-archive@odin.ietf.org; Tue, 14 Jan 2003 21:54:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0F2spJ24016 for ; Tue, 14 Jan 2003 21:54:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19896 for ; Tue, 14 Jan 2003 21:39:55 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0F2s9J23999; Tue, 14 Jan 2003 21:54:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0F2rHJ23915 for ; Tue, 14 Jan 2003 21:53:17 -0500 Received: from magic.adaptec.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19886 for ; Tue, 14 Jan 2003 21:38:21 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.11.6+Sun/8.11.6) with ESMTP id h0F2ffj27679; Tue, 14 Jan 2003 18:41:42 -0800 (PST) Received: from aimexc02.adaptec.com (aimexc02.corp.adaptec.com [162.62.62.40]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id SAA25481; Tue, 14 Jan 2003 18:41:41 -0800 (PST) Received: by aimexc02.corp.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Tue, 14 Jan 2003 18:41:40 -0800 Message-ID: From: "Mukund, Shridhar" To: "'Black_David@emc.com'" , "'rddp@ietf.org'" Cc: "Munnangi, Siva" , "Mukund, Shridhar" Date: Tue, 14 Jan 2003 18:41:38 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP TCP mapping - constituent parts and the baseline solution Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Dear colleagues, To analyze the problem on hand, I want to attempt to separate the variables and suggest relative value that each component brings to the party. In particular, recognize 80-20's. P1. Steering: The 2-tuple . If the PKey corresponds to tagged model than PKey=STag and Offset=byte offset. If the PKey is for untagged model than PKey=Stream# and Offset=MSN. Also message/byte-stream delimiters. At this stage, we make no segmentation-and-reassembly (SAR) assumptions. If a DDP message is a Java stream of 2GB than so be it. ULP/RDMAP may choose to segment for reasons other than transport. P2. Integrity: *Optional* Payload Digest. P3. Synchronization: *Optional and variable* forward pointing marker. Granted, backward pointing marker also achieves synchronization, but forward pointer is better when the inter-marker spacing is larger than the packet size. P4a. SAR & Alignment: I intend to show later that the *only compelling* reason to SAR is alignment, be it for MPA or SCTP. P4b. An alternative to P4 that addresses "THE QUEST". Comments: 1. P1+P2+P3 is iSCSI. iSCSI has built-in DDP today! On a side note, due to my ignorance, I have for a long time been under the impression that RDMAC was adopting this baseline solution :-( 2. P1 is of course essential and along with P2, it addresses several of RDDP needs. We will look at latency-sensitive data center scenario in particular. 3. The primary reason behind P3 is to make re-assembly buffer *feasible* in the LFN case (10Gbps @300ms, 0.375 GB). With some simplifying assumptions, the buffer size is a fn. of: a. Number of PDUs in a window.(BWxDelay), PDU size. b. Conditional probability of packet loss and that the lost packet contains the header. a and b compensate for each other -> helps assumptions Once the header is received, data segments do not need to be buffered if a. Data integrity is turned off or b. Alg. to compute out-of-order CRC is used. P3 is compelling for bulk data over LFN. It adds some value to latency sensitive aps. 4. P4a is nice and useful. Lets look at the roadblocks and cost though ... -Shridhar Mukund _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 10:12:16 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29359 for ; Wed, 15 Jan 2003 10:12:15 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FFQuu17200 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 10:26:56 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFQuJ17197 for ; Wed, 15 Jan 2003 10:26:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29348 for ; Wed, 15 Jan 2003 10:11:44 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFQDJ17153; Wed, 15 Jan 2003 10:26:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFPJJ17086 for ; Wed, 15 Jan 2003 10:25:19 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29299 for ; Wed, 15 Jan 2003 10:10:07 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id HAA14178; Wed, 15 Jan 2003 07:13:27 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Wed, 15 Jan 2003 10:13:27 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FBC@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Caitlin Bestler'" , rddp@ietf.org Subject: RE: [rddp] Alignment Commentary Date: Wed, 15 Jan 2003 10:13:18 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin Bestler wrote: > 4) Are MPA markers sufficiently complex for > a pure software stack to implement that it is > worth negotiation of a "no-marker" mode? > > IMHO, Markers are truly ugly but ultimately > not that hard to implement. It is not worth > the complexity of supporting multiple modes > and negotiating their use. In my mind, the strongest argument for making markers optional (assuming, hypothetically, that they are supported at all) is that five years from now when people realize that they were a bad idea and nobody is using them, they can simply be opted out. If you make them manditory, then if they are subsequently discovered to be a mistake, it is a much more difficult mistake to correct. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 10:36:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA00234 for ; Wed, 15 Jan 2003 10:36:12 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FFor319324 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 10:50:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFoqJ19321 for ; Wed, 15 Jan 2003 10:50:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA00198 for ; Wed, 15 Jan 2003 10:35:40 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFoCJ19294; Wed, 15 Jan 2003 10:50:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FFmpJ19193 for ; Wed, 15 Jan 2003 10:48:51 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA00156 for ; Wed, 15 Jan 2003 10:33:38 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id HAA14583; Wed, 15 Jan 2003 07:33:08 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Wed, 15 Jan 2003 10:33:07 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FBD@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Randall Stewart'" Cc: "'rddp@ietf.org'" Subject: RE: [rddp] RE: I-D ACTION:draft-stewart-rddp-sctp-01.txt Date: Wed, 15 Jan 2003 10:33:03 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Randall Stewart wrote: > The PPI is meant > to identify to network elements and middle boxes as well as > the application what data is present in the chunk. To attempt > to "harness" the entire PPI is NOT in accordance with how IANA > will be assigning the PPI... using one or two of them, sort of > like a port number, IS more in line with what they were designed > for... The PPI is required to be assigned by IANA... I doubt > you can get a block of PPI's to represent RDDP... a few yes > a block no. RFC 2960 says: The upper layer, i.e., the SCTP user, SHOULD standardize any specific protocol identifier with IANA if it is so desired. The use of any specific payload protocol identifier is out of the scope of SCTP. The phrase "SHOULD do X if it is so desired" seems a bit self contradictory. But it certainly suggests that RDDP "if so desired" may choose NOT to register the PPI with IANA and use it in a different manner more along the lines I described. (I.e. use it to carry the first 16 bits of the DDP header in addition to 16 other bits. The first 16 bits of the DDP header actually describe the payload type, so this is in the spirit of the PPI.) It sounds like you are implying a requirement that two different ULPs should have a non-overlapping set of PPIs in use, but I don't see this documented in RFC-2960, and it seems unnecessary since the adaption indication can fully disambiguate this. However, I am at a huge disadvantage arguing author's intent with the author, so maybe you could elaborate a bit on this. Thanks, Jim _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 11:07:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01290 for ; Wed, 15 Jan 2003 11:07:31 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FGMBJ21844 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 11:22:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGMBJ21841 for ; Wed, 15 Jan 2003 11:22:11 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01273 for ; Wed, 15 Jan 2003 11:06:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGLTJ21767; Wed, 15 Jan 2003 11:21:29 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGHaJ21492 for ; Wed, 15 Jan 2003 11:17:36 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01082 for ; Wed, 15 Jan 2003 11:02:25 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id F19FE2455E9; Wed, 15 Jan 2003 11:05:43 -0500 (EST) Date: Wed, 15 Jan 2003 10:06:21 -0600 From: Caitlin Bestler Subject: RE: [rddp] Alignment Commentary To: "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FBC@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/15/03, Williams, Jim wrote: >Caitlin Bestler wrote: >> 4) Are MPA markers sufficiently complex for >> a pure software stack to implement that it is >> worth negotiation of a "no-marker" mode? >> >> IMHO, Markers are truly ugly but ultimately >> not that hard to implement. It is not worth >> the complexity of supporting multiple modes >> and negotiating their use. > >In my mind, the strongest argument for making >markers optional (assuming, hypothetically, that >they are supported at all) is that five years >from now when people realize that they were a >bad idea and nobody is using them, they can simply >be opted out. > >If you make them manditory, then if they are >subsequently discovered to be a mistake, >it is a much more difficult mistake to correct. > It is also possible that high-performance/low buffer deployments will migrate to SCTP, leaving TCP for long-haul connections to clients behind antiquated firewalls without full SCTP support. In that sense markers would cease to the extent that TCP would no longer be used. Actually, the strongest argument for optional markers would be pushing for marker compatability between iSCSI and MPA. Allowing two *different* marker schemes, both of which are optional, strikes me as a major QA headache. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 11:20:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01859 for ; Wed, 15 Jan 2003 11:20:14 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FGYtq22691 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 11:34:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGYsJ22688 for ; Wed, 15 Jan 2003 11:34:54 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01816 for ; Wed, 15 Jan 2003 11:19:43 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGYEJ22641; Wed, 15 Jan 2003 11:34:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGXeJ22545 for ; Wed, 15 Jan 2003 11:33:40 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01744 for ; Wed, 15 Jan 2003 11:18:28 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 843BF2455F7; Wed, 15 Jan 2003 11:21:48 -0500 (EST) Date: Wed, 15 Jan 2003 10:22:25 -0600 From: Caitlin Bestler Subject: Re: [rddp] A consensus call and a poll To: Black_David@emc.com Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C670@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 12/16/02, Black_David@emc.com wrote: >(1) Having seen no defenders of a prohibition on packing > multiple ULPDUs into a TCP segment, I believe the > rough consensus of the RDDP WG that packing of > multiple ULPDUs into a TCP segment MUST be supported. > Anyone who disagrees should post that fact to the list > along with a technical rationale. There is one detail here that still has to be specified. When there are multiple ULPDUs between two markers, the marker should refer to the *first* one. The extra ULPDUs must be found by adding the length of the prior ULPDU. There are at least two scenarios where you could get a burst of very short DDP segments: - A series of short tagged DDP puts where the Data Sinks' buffer space was being used to form a join image. [The Data Source does not have the records requested organized as such anywhere in its memory]. - A series of short untagged DDP messages, perhaps flushing a large number of pending requests due to an orderly shutdown of a session. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 11:40:02 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02764 for ; Wed, 15 Jan 2003 11:40:01 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FGsgf24712 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 11:54:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGsgJ24709 for ; Wed, 15 Jan 2003 11:54:42 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02748 for ; Wed, 15 Jan 2003 11:39:30 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGs2J24685; Wed, 15 Jan 2003 11:54:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FGrNJ24645 for ; Wed, 15 Jan 2003 11:53:23 -0500 Received: from ztxmail03.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02733 for ; Wed, 15 Jan 2003 11:38:11 -0500 (EST) Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id 796003B29; Wed, 15 Jan 2003 10:41:30 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Wed, 15 Jan 2003 10:41:25 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6344.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] Alignment Commentary Date: Wed, 15 Jan 2003 10:41:24 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430A0@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] Alignment Commentary Thread-Index: AcK8qPYGHDNTA3PPR8aAlsgyZMsuewACyMJw From: "Barron, Dwight" To: "Williams, Jim" , "Caitlin Bestler" , X-OriginalArrivalTime: 15 Jan 2003 16:41:25.0184 (UTC) FILETIME=[F1833C00:01C2BCB4] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h0FGrNJ24646 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit The impact of this form optionality is that every implementor has to implement both options (on and off) and QA four permutations (mine on, your's on; mine on, your's off, etc.) to insure true interoperability. This seems a rather high price to pay today for something that may happen five years hence. Regards, Dwight > -----Original Message----- > From: Williams, Jim [mailto:Jim.Williams@emulex.com] > Sent: Wednesday, January 15, 2003 9:13 AM > To: 'Caitlin Bestler'; rddp@ietf.org > Subject: RE: [rddp] Alignment Commentary > > > Caitlin Bestler wrote: > > 4) Are MPA markers sufficiently complex for > > a pure software stack to implement that it is > > worth negotiation of a "no-marker" mode? > > > > IMHO, Markers are truly ugly but ultimately > > not that hard to implement. It is not worth > > the complexity of supporting multiple modes > > and negotiating their use. > > In my mind, the strongest argument for making > markers optional (assuming, hypothetically, that > they are supported at all) is that five years > from now when people realize that they were a > bad idea and nobody is using them, they can simply > be opted out. > > If you make them manditory, then if they are > subsequently discovered to be a mistake, > it is a much more difficult mistake to correct. > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 13:06:10 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05321 for ; Wed, 15 Jan 2003 13:06:10 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FIKrr31167 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 13:20:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FIKrJ31164 for ; Wed, 15 Jan 2003 13:20:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05302 for ; Wed, 15 Jan 2003 13:05:38 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FIK8J31140; Wed, 15 Jan 2003 13:20:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FIJ6J31083 for ; Wed, 15 Jan 2003 13:19:06 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05258 for ; Wed, 15 Jan 2003 13:03:51 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 6BC1CD042; Wed, 15 Jan 2003 10:06:59 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 87A4A157A2; Wed, 15 Jan 2003 10:06:57 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="US-ASCII" Date: Wed, 15 Jan 2003 10:06:35 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 07285-280075FE; Wed, 15 Jan 2003 10:06:24 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 4EAF59E25; Wed, 15 Jan 2003 10:06:24 -0800 (PST) Subject: RE: [rddp] Alignment Commentary To: "Caitlin Bestler" , "Rddp" X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Caitlin Bestler > Sent: Tuesday, January 14, 2003 1:54 PM > To: Williams, Jim; rddp@ietf.org > Subject: RE: [rddp] Alignment Commentary > > > On 1/14/03, Williams, Jim wrote: > > > > > >So I am left with the following thoughts: > > > >1. RDDP over unmodified TCP is important, > > and will result in unaligned PDUs being > > common. > > > >2. Senders may align PDUs, but there is no > > MUST or SHOULD associated with alignment. > > > >3. Receivers may choose that only when talking > > to an aligning sender will high performance > > (better than non-RDDP TCP) be achieved. > > In this case the receiver should not even > > bother to negotiate use of RDDP when talking > > to RDDP over an unmodified TCP. > > > > There is no compelling reason to require > > a receiver that depends on sender alignment > > for performance to negotiate use of RDDP > > with a non-aligning sender. > > It should be acceptable > > for such a receiver to simply refuse > > to negotiate use of RDDP and use conventional > > non-RDDP connection in that case. > > > >4. Other receiver implementations may choose > > to achieve high performance with either > > aligning or non-aligning senders. > > > >5. Some method needs to be created for a sender > > to notify the receiver of ability to align. > > There are strong arguments that this should > > be done at the transport, not the RDDP layer. > > (i.e. a new TCP option on the SYN segment). > > > >============ > > > >This line or reasoning quickly leads to the > >realization that three modes of RDDP would be > >supported: unaligned TCP, aligned TCP, and > >aligned SCTP. > > > >Not surprisingly, my own preference would be > >to limit this to two: unaligned TCP and > >aligned SCTP. However three modes > >seems like a reasonable compromise, if > >that is how the WG wants to proceed. > > > > > > DDP over unaligned MPA/TCP still has two major > benefits over the ULP using raw TCP: > > 1) Content can be shipped/fetched in an order > optimized for actual physical storage, rather > than in a precise byte order dictated by its > nominal ULP form. > > Storage servers being able to ship/fetch > disk sectors in the order they are reached > is a major advantage, even when both ends > of the connection are unaligned. > > Examples: disk striping, file fragmentation, > logical files, database joins. This capability has always existed, but I wonder how many storage servers actually implement this. Last time I asked this question, the unanimous answer seemed to be that even though it is an interesting capability, it is not implemented (maybe too complex overall for the benefit ??). The real benefit may be that upper layer protocols (such as storage over RDDP or whatever) are common for aligned or unaligned systems -- rather than have having two different ULPs. > > 2) An unaligned DDP/MPA/TCP receiver still needs > fewer system buffers than a plain TCP receiver. > > With plain TCP, the number of buffers required > can be N-1 (where N is the maximum # of buffers > in flight). > > With alignment *totally* destroyed, on a > persistent basis, an unaligned receiver will > be able to start placement when at least half > of the in-flight segment have been received, > and probably well before that. > > > It is also important to remember that "non aligned > senders" are probably best described as > "non-alignment guaranteed senders". A DDP/MPA > stack working on top of a conventional unmodified > TCP stack will still be able to produce aligned > segments *most* of the time. > > > This all leads to a handful of decisions for the > MPA draft: > > 1) There are major benefits to RDDP for both > alignment and prompt transmittal of untagged > messages. Does this justify the MPA draft > overriding some default TCP "SHOULDs"? > For example:an MPA-aware sender MAY or SHOULD > use alignment. > > I believe there is a strong consensus for yes > on this point. > > 2) Should the current MPA draft be modified with > enhancements, such as the split CRC, which > would benefit unaligned connections even though > it would produce no benefits to aligned > connections and all connections would have to > pay the cost of the extra feature? > > I do not know if this has reached consensus > yet. Some of the stated opposition was based > upon a presumption that the second CRC would be > too complex. That is clearly not the case. > > It is feasible, and would be of benefit for > unaligned connections. My estimation on the > portion of connections that would benefit > from the feature, however, is so low that > I do not consider it worthwhile. > > Someone needs to present a scenario where a > high performance server would see a substantial > number of unaligned packets. I cannot come up > with one. > > 3) Is there a need to configure a connection as > being aligned or unaligned, or should all > connections be prepared to deal with unaligned > packets at any time? > > I can see allocating my resources based upon a > statistical model on the number of aligned > versus unaligned packets. If that model is > wrong, my performance will suffer. The same > sort of engineering tradeoff is done with any > application that wants to predict its bandwidth > over a media that can require retransmissions > because of errors. > > But just as I would expect to be ready to > detect an error in *any* packet, no matter how > reliable a particular connection has been > historically, I would expect any connection > to be ready to handle unaligned packets. > > Does anyone have a justification on why a > receiver would benefit from a "heads up" on > whether a connection will be aligned or not? > > 4) Are MPA markers sufficiently complex for > a pure software stack to implement that it is > worth negotiation of a "no-marker" mode? > > IMHO, Markers are truly ugly but ultimately > not that hard to implement. It is not worth > the complexity of supporting multiple modes > and negotiating their use. > > However, there may be some merit in trying to > synergize interval markers with iSCSI. But > I wouldn't make that an objective, unless > the Area Directors jump in to the contrary. > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 15 17:09:38 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11703 for ; Wed, 15 Jan 2003 17:09:38 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0FMOPr14541 for rddp-archive@odin.ietf.org; Wed, 15 Jan 2003 17:24:25 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FMOPJ14538 for ; Wed, 15 Jan 2003 17:24:25 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11683 for ; Wed, 15 Jan 2003 17:09:07 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FMNkJ14471; Wed, 15 Jan 2003 17:23:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0FMG9J14022 for ; Wed, 15 Jan 2003 17:16:09 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11394 for ; Wed, 15 Jan 2003 17:00:49 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id OAA26144 for ; Wed, 15 Jan 2003 14:04:10 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Wed, 15 Jan 2003 17:04:10 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FBF@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Alignment Commentary Date: Wed, 15 Jan 2003 17:04:00 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Barron, Dwight [mailto:Dwight.Barron@hp.com] > Sent: Wednesday, January 15, 2003 11:41 AM > To: Williams, Jim; Caitlin Bestler; rddp@ietf.org > Subject: RE: [rddp] Alignment Commentary > > > The impact of this form optionality is that every implementor > has to implement both options (on and off) and QA four > permutations (mine on, your's on; mine on, your's off, etc.) > to insure true interoperability. This seems a rather high > price to pay today for something that may happen five years hence. > > Regards, > Dwight Yes, it is true that options have cost. Of course if the iSCSI precedent is followed and markers are optional to implement, then only those that choose to implement markers pay the cost. Anyone have feedback on how iSCSI markers are doing with respect to market acceptance? Too soon, I know, but this may be an indicator of how RDDP markers will fare as there is a similarity in the cost/benefit (assuming that RDDP, like iSCSI, does not mandate alignment). iSCSI markers are potentially more valuable in that they can be used for error recovery, whereas RDDP provides no such error recovery mechanism. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 16 13:59:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20694 for ; Thu, 16 Jan 2003 13:59:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0GJErl15480 for rddp-archive@odin.ietf.org; Thu, 16 Jan 2003 14:14:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJErJ15477 for ; Thu, 16 Jan 2003 14:14:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20627 for ; Thu, 16 Jan 2003 13:59:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJEBJ15425; Thu, 16 Jan 2003 14:14:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJDHJ15377 for ; Thu, 16 Jan 2003 14:13:17 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20494 for ; Thu, 16 Jan 2003 13:57:33 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 7068BD034 for ; Thu, 16 Jan 2003 11:00:45 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 49DDF15A1E for ; Thu, 16 Jan 2003 11:00:43 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Date: Thu, 16 Jan 2003 11:00:34 -0800 From: "Somesh Gupta" Importance: Normal Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 19063-1B9BB385; Thu, 16 Jan 2003 11:00:22 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 85A9314B94 for ; Thu, 16 Jan 2003 11:00:22 -0800 (PST) To: "Rddp" X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Subject: [rddp] DDP Message Length Question Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit The DDP draft, section 7.4, requires that the DDP must provide the ULP message length to the ULP when an untagged message is delivered. It states that the ULP Message length may be calculated by adding the MO and the payload length in the last DDP segment for a message. However at the same time,, section 7.2 says that DDP SHOULD transmit DDP segments within a DDP message in increasing MO order. Does seem that the two sections are in conflict with one another. Anyone care to provide a clarification. Thanks, Somesh _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 16 14:11:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA21086 for ; Thu, 16 Jan 2003 14:11:31 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0GJQiY16122 for rddp-archive@odin.ietf.org; Thu, 16 Jan 2003 14:26:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJQiJ16119 for ; Thu, 16 Jan 2003 14:26:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA21045 for ; Thu, 16 Jan 2003 14:10:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJQ3J16054; Thu, 16 Jan 2003 14:26:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0GJPfJ16017 for ; Thu, 16 Jan 2003 14:25:41 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA21009 for ; Thu, 16 Jan 2003 14:09:56 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id F2307D034; Thu, 16 Jan 2003 11:13:07 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id D110814CE4; Thu, 16 Jan 2003 11:13:05 -0800 (PST) Cc: "Rddp" Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="US-ASCII" Date: Thu, 16 Jan 2003 11:12:39 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 19438-23826AA4; Thu, 16 Jan 2003 11:12:27 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id AF65E168B5; Thu, 16 Jan 2003 11:12:27 -0800 (PST) Subject: RE: [rddp] DDP Message Length Question To: "Caitlin Bestler" X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Thanks for the clarification. > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Thursday, January 16, 2003 11:10 AM > To: Somesh Gupta > Subject: Re: [rddp] DDP Message Length Question > > > On 1/16/03, Somesh Gupta wrote: > > >The DDP draft, section 7.4, requires that the DDP must > >provide the ULP message length to the ULP when an untagged > >message is delivered. It states that the ULP Message > >length may be calculated by adding the MO and the payload > >length in the last DDP segment for a message. > > > >However at the same time,, section 7.2 says that DDP > >SHOULD transmit DDP segments within a DDP message in > >increasing MO order. > > > >Does seem that the two sections are in conflict with one > >another. Anyone care to provide a clarification. > > > > Section 6.1: > > The DDP Segment with the L bit set to 1 MUST be posted > to the LLP after all other DDP Segments of the > associated DDP Message have been posted to the LLP. For > an Untagged DDP Message, the DDP Segment with the L bit > set to 1 MUST carry the highest MO. > > > An implementation may, for reasons worthy of overriding a > "SHOULD", reshuffle the order of all segments in an untagged > message *except* for the last one, which MUST be last. > > The reason that it MUST be last, is that it allows the > receiver to calculate the message size from the last segment > alone. There is no required message state that must be > accumulated to reassemble the untagged message. > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 22 16:23:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07589 for ; Wed, 22 Jan 2003 16:23:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0MLfHE21702 for rddp-archive@odin.ietf.org; Wed, 22 Jan 2003 16:41:17 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MLfHJ21699 for ; Wed, 22 Jan 2003 16:41:17 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07563 for ; Wed, 22 Jan 2003 16:22:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MLeRJ21663; Wed, 22 Jan 2003 16:40:27 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MLd1J21582 for ; Wed, 22 Jan 2003 16:39:01 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA07527 for ; Wed, 22 Jan 2003 16:20:17 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 22 Jan 2003 16:23:35 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C7E4@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 22 Jan 2003 16:22:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Some Marker Comments Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , With WG chair hat off: On optional markers, Dwight Barron says: > The impact of this form optionality is that every implementor > has to implement both options (on and off) and QA four permutations > (mine on, your's on; mine on, your's off, etc.) to insure true > interoperability. This seems a rather high price to pay today > for something that may happen five years hence. Two and a fraction should be closer to the mark because both the on-off and off-on scenarios should turn into off-off in short order as the connection gets going. Recall that we need a mechanism to ensure that the two TCP endpoints aren't confused about the content of the bytestream, and the on-off and off-on scenarios both look like confusion. The count would go back to four if markers were independently optional in each direction, but seems like something that one could "just say no" to, provided that we had markers that were reasonable for software to cope with (IMHO, MPA's current 512-byte fixed interval isn't). On iSCSI markers, Jim Williams asks: > Anyone have feedback on how iSCSI markers are > doing with respect to market acceptance? Too soon, > I know, but this may be an indicator of how > RDDP markers will fare as there is a > similarity in the cost/benefit (assuming that > RDDP, like iSCSI, does not mandate alignment). I believe "not widely implemented" is a summary, possibly optimistic. Also, a while back, Jim commented: > 5. Some method needs to be created for a sender > to notify the receiver of ability to align. > There are strong arguments that this should > be done at the transport, not the RDDP layer. > (i.e. a new TCP option on the SYN segment). IMHO (WG chair hat still off), the odds of a new TCP option for this purpose getting through tsvwg are close to zero (I've been involved in somewhat analogous attempts, and my ears are still ringing ...). It would be better to pursue an inband notification mechanism that could also satisfy the "ensure no confusion about bytestream contents" requirement. Getting that design and the underlying rationale, weaknesses, tradeoffs, etc. are worked out and well-described in an Internet-Draft ought to be done before considering whether the result makes a sufficient case for a new TCP option. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jan 22 17:28:47 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09917 for ; Wed, 22 Jan 2003 17:28:47 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0MMkxQ26359 for rddp-archive@odin.ietf.org; Wed, 22 Jan 2003 17:46:59 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MMkxJ26356 for ; Wed, 22 Jan 2003 17:46:59 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09891 for ; Wed, 22 Jan 2003 17:28:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MMkBJ26318; Wed, 22 Jan 2003 17:46:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0MMjOJ26292 for ; Wed, 22 Jan 2003 17:45:24 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09845 for ; Wed, 22 Jan 2003 17:26:40 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 22 Jan 2003 17:30:04 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C7E5@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 22 Jan 2003 17:29:08 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Alignment Consensus attempt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , With my WG chair hat (and Kevlar[tm] suit) on, here's an attempt to call consensus on the alignment issue. My original post on January 2nd phrased the primary issue as: So, I believe the primary issue before the WG is whether to accept: - The RDDP protocols must allow implementations over unmodified TCP/IP stacks. as a requirement. Based on discussion since then, I believe the rough consensus of the RDDP WG is to accept this as a requirement. Restating the implications from that original message, (I've rephrased the second bullet to discuss RDDP [DDP/RDMA] operations rather than placement), some important things that this implies are: - Unmodified means 100% unmodified, not even "minor" changes. - An RDDP sender using an unmodified TCP stack can send RDDP payloads that result in the correct execution of RDDP operations by an RDDP receiver even if the sends are not aligned. - There don't appear to be any receiver implications, as nothing has been proposed that would prevent a receiver from having TCP buffer as usual and then run RDDP functionality on the resulting in-order TCP bytestream. - The practical meaning of "allow" is that a sender implementation is possible without violating any MUST, MUST NOT, SHOULD, or SHOULD NOT requirement. OTOH, conditional requirements for stack modification are not ruled out. For example: "If a sender notifies a receiver that the sender is operating in alignment-modified-mode by , then the sender SHOULD ..., MUST ..., etc." If this sort of approach is followed, the second bullet still applies at the receiver, e.g., an unmodified sender would not use , but would still be able to send RDDP payloads that invoke RDDP operations at the receiver. Beyond this, I also believe there is rough consensus for the RDDP WG to *not* set requirements (MUST/SHOULD) for receiver performance when the sender does not align traffic. It is apparent to me from list discussion that different implementers favor rather different implementation approaches that have different performance consequences for this scenario. Repeating Jim Williams's spectrum of possibilities: > 1. Only functional correctness matters, performance is not an issue. > > 2. Performance must be as good as conventional (non-RDMA) TCP. > > 3. Performance must be significantly better than conventional > (non-RDMA) TCP otherwise RDMA should not be used in this case. If this second consensus holds, the WG chair intends to hold the WG to an approach of avoiding design approaches and decisions that have the effect of ruling out any of the three alternative views listed above. This should have the effect of broadening the applicability of the WG's protocols. Nonetheless, I think the WG's results should capture some of the list discussion about scenarios that involve unmodified stacks, including when loss of alignment in such stacks is unavoidable despite reasonable efforts by an application/ULP (not including stop-and-wait), as it'll provide some useful insight to implementers. Anyone who disagrees with either of these consensus calls should explain why on the list. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 23 11:15:07 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11876 for ; Thu, 23 Jan 2003 11:15:07 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0NGXgq04973 for rddp-archive@odin.ietf.org; Thu, 23 Jan 2003 11:33:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGXgJ04970 for ; Thu, 23 Jan 2003 11:33:42 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11832 for ; Thu, 23 Jan 2003 11:14:35 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGWLJ04865; Thu, 23 Jan 2003 11:32:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGVIJ04788 for ; Thu, 23 Jan 2003 11:31:18 -0500 Received: from ztxmail03.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11635 for ; Thu, 23 Jan 2003 11:12:11 -0500 (EST) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id 3E6575709; Thu, 23 Jan 2003 10:15:36 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Thu, 23 Jan 2003 10:15:35 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Alignment Consensus attempt Date: Thu, 23 Jan 2003 10:15:34 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A1707628FFC6E8@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Alignment Consensus attempt Thread-Index: AcLCaW/oTZossaj6QRellEEPsqsW+wAkJSOg From: "Culley, Paul" To: , X-OriginalArrivalTime: 23 Jan 2003 16:15:35.0683 (UTC) FILETIME=[A93E2530:01C2C2FA] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h0NGVIJ04789 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit I agree with your consensus calls: 1) RDDP Must work over unmodified TCP stacks 2) The WG Must Not set Performance requirements (for any operating mode) but SHOULD allow a variety of performance points. For what it is worth, the authors expect to shortly deliver an updated MPA draft that will allow packing and explain in greater detail how MPA can be used on unmodified TCP. Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Wednesday, January 22, 2003 4:29 PM To: rddp@ietf.org Subject: [rddp] Alignment Consensus attempt With my WG chair hat (and Kevlar[tm] suit) on, here's an attempt to call consensus on the alignment issue. My original post on January 2nd phrased the primary issue as: So, I believe the primary issue before the WG is whether to accept: - The RDDP protocols must allow implementations over unmodified TCP/IP stacks. as a requirement. Based on discussion since then, I believe the rough consensus of the RDDP WG is to accept this as a requirement. Restating the implications from that original message, (I've rephrased the second bullet to discuss RDDP [DDP/RDMA] operations rather than placement), some important things that this implies are: - Unmodified means 100% unmodified, not even "minor" changes. - An RDDP sender using an unmodified TCP stack can send RDDP payloads that result in the correct execution of RDDP operations by an RDDP receiver even if the sends are not aligned. - There don't appear to be any receiver implications, as nothing has been proposed that would prevent a receiver from having TCP buffer as usual and then run RDDP functionality on the resulting in-order TCP bytestream. - The practical meaning of "allow" is that a sender implementation is possible without violating any MUST, MUST NOT, SHOULD, or SHOULD NOT requirement. OTOH, conditional requirements for stack modification are not ruled out. For example: "If a sender notifies a receiver that the sender is operating in alignment-modified-mode by , then the sender SHOULD ..., MUST ..., etc." If this sort of approach is followed, the second bullet still applies at the receiver, e.g., an unmodified sender would not use , but would still be able to send RDDP payloads that invoke RDDP operations at the receiver. Beyond this, I also believe there is rough consensus for the RDDP WG to *not* set requirements (MUST/SHOULD) for receiver performance when the sender does not align traffic. It is apparent to me from list discussion that different implementers favor rather different implementation approaches that have different performance consequences for this scenario. Repeating Jim Williams's spectrum of possibilities: > 1. Only functional correctness matters, performance is not an > issue. > > 2. Performance must be as good as conventional (non-RDMA) TCP. > > 3. Performance must be significantly better than conventional > (non-RDMA) TCP otherwise RDMA should not be used in this case. If this second consensus holds, the WG chair intends to hold the WG to an approach of avoiding design approaches and decisions that have the effect of ruling out any of the three alternative views listed above. This should have the effect of broadening the applicability of the WG's protocols. Nonetheless, I think the WG's results should capture some of the list discussion about scenarios that involve unmodified stacks, including when loss of alignment in such stacks is unavoidable despite reasonable efforts by an application/ULP (not including stop-and-wait), as it'll provide some useful insight to implementers. Anyone who disagrees with either of these consensus calls should explain why on the list. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 23 11:37:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA12619 for ; Thu, 23 Jan 2003 11:37:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0NGuX406763 for rddp-archive@odin.ietf.org; Thu, 23 Jan 2003 11:56:33 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGuXJ06760 for ; Thu, 23 Jan 2003 11:56:33 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA12608 for ; Thu, 23 Jan 2003 11:37:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGt2J06666; Thu, 23 Jan 2003 11:55:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NGqMJ06528 for ; Thu, 23 Jan 2003 11:52:22 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA12424 for ; Thu, 23 Jan 2003 11:33:15 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B20D4244C59; Thu, 23 Jan 2003 11:36:42 -0500 (EST) Date: Thu, 23 Jan 2003 10:37:41 -0600 From: Caitlin Bestler Subject: Re: [rddp] Some Marker Comments To: rddp@ietf.org Cc: Black_David@emc.com X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C7E4@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/22/03, Black_David@emc.com wrote: >provided that we had markers that were reasonable for >software to cope with (IMHO, MPA's current 512-byte fixed >interval isn't). I do not see any undue burden on a software solution from markers, at whatever interval, unless the software makes the misktake of physically assembling the DDP Segment in a consecutive sequence of memory bytes. The first step is to identify the DDP segment. With markers embedded, the identified location consists of portions of one or more received IP datagrams. If you want to be even more nitty gritty, that is generally within one more received Ethernet Frames (but there are exceptions). The DDP implementation has to do two things with the received DDP segment: interpret the DDP header and place the DDP contents. Both can be accomplished with a scatter/gather list. Unless you attempt to interpret the DDP header while it is has a marker imbedded in the middle of it there really isnt' that much of a problem for the software. A hardware implementation could apply the same algorithm, but has the additional options, such as mapping around the markers or identifying an incoming Ethernet Frame as being part of an MPA connection "on the fly" and compressing the markers during the act of receiving the data. It cannot cut-thru place the data, because the port number and header will not be known to be correct until the entire TCP segment is recieved, but it can "pre-digest" the incoming TCP segment on the presumption that the only way it will be kept is if it is indeed part of an MPA connection. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 23 13:54:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16522 for ; Thu, 23 Jan 2003 13:54:34 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0NJDCj16987 for rddp-archive@odin.ietf.org; Thu, 23 Jan 2003 14:13:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NJDCJ16983 for ; Thu, 23 Jan 2003 14:13:12 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16517 for ; Thu, 23 Jan 2003 13:54:03 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NJCDJ16910; Thu, 23 Jan 2003 14:12:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0NJBKJ16887 for ; Thu, 23 Jan 2003 14:11:20 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16461 for ; Thu, 23 Jan 2003 13:52:11 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 270B9D053; Thu, 23 Jan 2003 10:55:26 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 076551512D; Thu, 23 Jan 2003 10:55:26 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Date: Thu, 23 Jan 2003 10:55:02 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C7E5@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 13622-58D040BA; Thu, 23 Jan 2003 10:54:51 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id BD9B2126D0; Thu, 23 Jan 2003 10:54:51 -0800 (PST) Subject: RE: [rddp] Alignment Consensus attempt To: , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit David, There are many standard track protocols over TCP that have framing/record delineation. NFS and iSCSI probably form the two ends in terms of complexity/features. I am sure many of us would like to see a standards track TCP framing. Is there consensus that the TCP framing protocol be such that it can be standards track? OR Can we have two TCP framing protocols, one standards track and one informational? Is there a process for requesting a standards track framing if MPA remains as an informational RFC? Thanks, Somesh _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 23 19:50:29 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26104 for ; Thu, 23 Jan 2003 19:50:29 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0O19ER08832 for rddp-archive@odin.ietf.org; Thu, 23 Jan 2003 20:09:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O19EJ08829 for ; Thu, 23 Jan 2003 20:09:14 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26095 for ; Thu, 23 Jan 2003 19:49:58 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O18IJ08756; Thu, 23 Jan 2003 20:08:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O17SJ08724 for ; Thu, 23 Jan 2003 20:07:28 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26069 for ; Thu, 23 Jan 2003 19:48:12 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 23 Jan 2003 19:51:39 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C808@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Date: Thu, 23 Jan 2003 19:50:50 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, (WG chair hat off) > >provided that we had markers that were reasonable for > >software to cope with (IMHO, MPA's current 512-byte fixed > >interval isn't). > > I do not see any undue burden on a software solution from > markers, at whatever interval, unless the software makes the > misktake of physically assembling the DDP Segment in a > consecutive sequence of memory bytes. A copy is a way of completely fouling up, but one can get into trouble long before then, although "undue" is in the eye of the beholder. > The first step is to identify the DDP segment. With markers > embedded, the identified location consists of portions of > one or more received IP datagrams. If you want to be even > more nitty gritty, that is generally within one more > received Ethernet Frames (but there are exceptions). > > The DDP implementation has to do two things with the > received DDP segment: interpret the DDP header and place the > DDP contents. Both can be accomplished with a scatter/gather > list. Unless you attempt to interpret the DDP header while > it is has a marker imbedded in the middle of it there really > isnt' that much of a problem for the software. The "scatter/gather list" is the opportunity to get into trouble. Suppose we're dealing with an 8k data block in a jumbo frame (can be made to work in the data center). A 512-byte marker interval yields over 30 entries in the scatter/gather list, whereas a larger marker interval could yield 3. That sort of order of magnitude overhead difference tends to have visible costs in both software and hardware, doubly so if the hardware can't cope with a 30-entry list and needs to be redriven by software to get the job done. For a user-mode implementation on an unmodified stack, those are syscalls rather than scatter/ gather list entries if the syscall interface is one that doesn't support scatter/gather. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jan 23 20:01:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA26358 for ; Thu, 23 Jan 2003 20:01:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0O1KEa09301 for rddp-archive@odin.ietf.org; Thu, 23 Jan 2003 20:20:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O1KDJ09298 for ; Thu, 23 Jan 2003 20:20:13 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA26340 for ; Thu, 23 Jan 2003 20:00:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O1J1J09232; Thu, 23 Jan 2003 20:19:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0O1IrJ09212 for ; Thu, 23 Jan 2003 20:18:53 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26310 for ; Thu, 23 Jan 2003 19:59:37 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 23 Jan 2003 20:03:04 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C809@corpmx14.us.dg.com> To: somesh_gupta@silverbacksystems.com, rddp@ietf.org Date: Thu, 23 Jan 2003 20:02:05 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Framing and Standards Track Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Somesh, These questions are rather speculative, and their answers have a strong dependency on the degree of presence or absence of TCP modifications (even OPTIONAL modifications) in the specific "TCP framing" mechanism. > There are many standard track protocols over TCP that > have framing/record delineation. NFS and iSCSI probably > form the two ends in terms of complexity/features. Indeed there are - such protocols contain no TCP modifications. > I am sure many of us would like to see a standards track > TCP framing. > > Is there consensus that the TCP framing protocol be such > that it can be standards track? Not that I can see. I believe the "no changes to TCP" aspect of IFT was vehemently objected to, and I don't believe a "MUST NOT change TCP in any fashion" consensus call would have had any chance of success. > Can we have two TCP framing protocols, one standards > track and one informational? > > Is there a process for requesting a standards track > framing if MPA remains as an informational RFC? Send Draft ... Such questions are far easier to deal with in the context of a specific concrete proposal. Keep in mind that Jim Williams's IFT draft was an initial attempt to do something in this space ... Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 11:20:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23935 for ; Fri, 24 Jan 2003 11:20:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OGe2011336 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 11:40:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OGe2J11333 for ; Fri, 24 Jan 2003 11:40:02 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23923 for ; Fri, 24 Jan 2003 11:20:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OGdIJ11270; Fri, 24 Jan 2003 11:39:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OGc3J11200 for ; Fri, 24 Jan 2003 11:38:03 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA23857 for ; Fri, 24 Jan 2003 11:18:26 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA09234; Fri, 24 Jan 2003 08:21:45 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 11:21:44 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FC7@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Alignment Consensus attempt Date: Fri, 24 Jan 2003 11:21:43 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David Black wrote: > So, I believe the primary issue before the WG is whether to accept: > - The RDDP protocols must allow implementations over > unmodified TCP/IP stacks. > as a requirement. > > Based on discussion since then, I believe the rough consensus > of the RDDP WG is to accept this as a requirement. > > [...] > > Beyond this, I also believe there is rough consensus for the RDDP WG > to *not* set requirements (MUST/SHOULD) for receiver performance > when the sender does not align traffic. I am a little bothered by the implication here. If vendor A has a product which performs super when receiving aligned PDUs, but performs worse than non-RDDP when receiving unaligned PDUs, we say that this is a valid implementation choice. Vendor B has a product which performs super when receiving aligned PDUs, but when receiving unaligned PDUs it simply refuses to negotiate use of RDDP and defaults back to non-RDDP version of ULP. Therefore, vendor B actually performs better than vendor A in the unaligned case. But we say that vendor B's implementation is non-compliant. So, I would suggest that it is reasonable that an RDDP endpoint ought to be allowed to negotiate not to use RDDP with a sender of unaligned PDUs. We could, however, push this back to the ULP, and say that the receiver should advise (as part of the ULP negotiation) that is doesn't want to do RDDP with an unaligned sender, and the sender should have sense enough in that case to just not use RDDP (unless there is some overriding consideration). So, I don't necessarily disagree with the above consensus attempt, but would hope that this case is handled in a sensible manner. If a particular connection is better off not being in RDDP mode, then the negotiation process should leave it in non-RDDP mode, and this should happen naturally and automatically without requiring undue ULP or operator intervention. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 12:24:50 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25414 for ; Fri, 24 Jan 2003 12:24:50 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OHhuk15807 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 12:43:56 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OHhuJ15804 for ; Fri, 24 Jan 2003 12:43:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25405 for ; Fri, 24 Jan 2003 12:24:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OHhAJ15767; Fri, 24 Jan 2003 12:43:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OHghJ15740 for ; Fri, 24 Jan 2003 12:42:43 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25378 for ; Fri, 24 Jan 2003 12:23:06 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id JAA11483; Fri, 24 Jan 2003 09:26:25 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 12:26:24 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FC9@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Date: Fri, 24 Jan 2003 12:26:24 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David Black wrote: > Also, a while back, Jim commented: > > > 5. Some method needs to be created for a sender > > to notify the receiver of ability to align. > > There are strong arguments that this should > > be done at the transport, not the RDDP layer. > > (i.e. a new TCP option on the SYN segment). > > IMHO (WG chair hat still off), the odds of a new TCP > option for this purpose getting through tsvwg are close > to zero (I've been involved in somewhat analogous attempts, > and my ears are still ringing ...). I hate to see important technical issues decided by fear of being yelled at by people with gray beards. :) On the other hand some valuable experience comes along with the gray beards, and that should be factored into the discussion. If the fundamental objection is to aligning TCP (as opposed to using a TCP option to do it) then trying to sneak it by tsvwg by negotiating in-band and pretending not to modify TCP seems like bad protocol design as well as questionable politics. > It would be better > to pursue an inband notification mechanism that could also > satisfy the "ensure no confusion about bytestream contents" > requirement. Getting that design and the underlying rationale, > weaknesses, tradeoffs, etc. are worked out and well-described > in an Internet-Draft ought to be done before considering > whether the result makes a sufficient case for a new TCP > option. The fundamental choice is between: A. "TCP is a byte stream protocol and thou shalt not attempt to use it otherwise." and B. "Let's treat TCP as a sequence of segments which the sender can construct, the network will preserve, and the receiver can exploit." If the answer is A, then it's a waste of time to talk about either TCP options or in-band notification of alignment. If the answer is B, then in-band notification versus TCP option is a legitimate debate. I do object to the notion of, "Let's do B, but be sneaky enough about it so that nobody looking at the wire can determine that we aren't doing A, therefore tsvwg has no right to object." It results in a bad and overly complex protocol motivated by political rather than technical considerations. > > Thanks, > --David _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 12:49:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26458 for ; Fri, 24 Jan 2003 12:49:37 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OI8g817879 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 13:08:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OI8gJ17876 for ; Fri, 24 Jan 2003 13:08:42 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26423 for ; Fri, 24 Jan 2003 12:49:06 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OI82J17836; Fri, 24 Jan 2003 13:08:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OI75J17287 for ; Fri, 24 Jan 2003 13:07:05 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26360 for ; Fri, 24 Jan 2003 12:47:28 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 22E7B244EE8; Fri, 24 Jan 2003 12:50:55 -0500 (EST) Date: Fri, 24 Jan 2003 11:51:56 -0600 From: Caitlin Bestler Subject: RE: [rddp] Some Marker Comments To: "Williams, Jim" Cc: "'Black_David@emc.com'" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FC9@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/24/03, Williams, Jim wrote: >I do object to the notion of, "Let's do B, but be sneaky >enough about it so that nobody looking at the wire can >determine that we aren't doing A, therefore tsvwg has no >right to object." It results in a bad and overly complex >protocol motivated by political rather than technical >considerations. On the other hand I think it is legitimate to set an objective of "Let's do B, but in such a way that no intermediate network equipment is required to anything that it would not have done for A." I can even live with taking excruciating pain in drafting standards so that they fully comply with A. But I do strongly agree that it is going overboard if we add actual complexity or overhead on the wire to meet a "cannot be detected" goal, rather than a "fully interoperable with" objective. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 13:02:15 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26893 for ; Fri, 24 Jan 2003 13:02:15 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OILLL18395 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 13:21:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OILKJ18392 for ; Fri, 24 Jan 2003 13:21:20 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26885 for ; Fri, 24 Jan 2003 13:01:44 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OIK4J18328; Fri, 24 Jan 2003 13:20:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OIJnJ18294 for ; Fri, 24 Jan 2003 13:19:49 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA26818 for ; Fri, 24 Jan 2003 13:00:13 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 93B41244D6C; Fri, 24 Jan 2003 13:03:39 -0500 (EST) Date: Fri, 24 Jan 2003 12:04:40 -0600 From: Caitlin Bestler Subject: Re: [rddp] Alignment Consensus attempt To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C7E5@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 1/22/03, Black_David@emc.com wrote: > >Beyond this, I also believe there is rough consensus for >the RDDP WG to *not* set requirements (MUST/SHOULD) for >receiver performance when the sender does not align >traffic. It is apparent to me from list discussion that >different implementers favor rather different >implementation approaches that have different performance >consequences for this scenario. > I believe it would be appropriate for the WG to document an expectation that use of RDDP over unaligned MPA/TCP will not result in a signifigant performance penalty when compared to the ULP simply using TCP itself. My understanding of "SHOULD" is compatible with that type of goal, but ultimately I do not care if it is a MAY/SHOULD or a "hey guys, its a real cool idea if". The point is not to define an IETF approved performance test suite, but rather to provide some simple guidance to application developers and those deploying networks. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 13:37:50 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA28082 for ; Fri, 24 Jan 2003 13:37:50 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OIuv020441 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 13:56:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OIuvJ20438 for ; Fri, 24 Jan 2003 13:56:57 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA28069 for ; Fri, 24 Jan 2003 13:37:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OIuFJ20394; Fri, 24 Jan 2003 13:56:15 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OIsbJ20312 for ; Fri, 24 Jan 2003 13:54:37 -0500 Received: from e32.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27953 for ; Fri, 24 Jan 2003 13:34:59 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e32.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id h0OIbmuu043820; Fri, 24 Jan 2003 13:37:48 -0500 Received: from d03nm014.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.4) with ESMTP id h0OIg3dr181110; Fri, 24 Jan 2003 11:42:04 -0700 X-Priority: 1 (High) Importance: Normal Subject: RE: [rddp] Some Marker Comments To: "Williams, Jim" Cc: "'Black_David@emc.com'" , rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: John Hufferd Date: Fri, 24 Jan 2003 10:34:48 -0800 X-MIMETrack: Serialize by Router on D03NM014/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 01/24/2003 11:37:47 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, I do not see the world the same as you do. "Political" considerations as you call it is process that one follows while dealing with various different belief structures. In this case, the gray beards, as you call them, have had years of work that has helped form their beliefs. It is silly to simply ignore them and useful to find a path that considers them. Doing this may be "political", but that is not necessary a bad thing, and is the usual way groups of people work together. You use the term as if "political" has the meaning of "wrong", and in so doing negate the years of experience that is represented by the beliefs that are involved. As I understand the issue, TCP has a lot of implementations and we would like to have a way for as many of them as possible be able to exploit RDMA. Also, in some ways, the complexity of TCP and the various implementations of TCP -- which may have pushed the envelop, but are never the less are in wide use -- cause additional caution. Therefore, as I understand it, the ADs have asked us to show how unmodified TCP implementations can operate as a sender or receiver of RDMA PDUs without having a significant problems. I think this is in our interest to do so. And yes, transparency on the links, also goes a long way to prevent unforeseen side effects. Clearly one way to do this is to negotiate a parameter or two in which both sides understand what the other is capable of doing and then do the right thing. The ULP is always able to make its own decision on how it wants to react to the network configuration, I think that is a normal response, and does not have a lot to do with the RDMA protocol itself. . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/SSG San Jose Ca Main Office (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office (408) 997-6136, Cell: (408) 499-9702 Internet address: hufferd@us.ibm.com "Williams, Jim" @ietf.org on 01/24/2003 09:26:24 AM Sent by: rddp-admin@ietf.org To: "'Black_David@emc.com'" , rddp@ietf.org cc: Subject: RE: [rddp] Some Marker Comments David Black wrote: > Also, a while back, Jim commented: > > > 5. Some method needs to be created for a sender > > to notify the receiver of ability to align. > > There are strong arguments that this should > > be done at the transport, not the RDDP layer. > > (i.e. a new TCP option on the SYN segment). > > IMHO (WG chair hat still off), the odds of a new TCP > option for this purpose getting through tsvwg are close > to zero (I've been involved in somewhat analogous attempts, > and my ears are still ringing ...). I hate to see important technical issues decided by fear of being yelled at by people with gray beards. :) On the other hand some valuable experience comes along with the gray beards, and that should be factored into the discussion. If the fundamental objection is to aligning TCP (as opposed to using a TCP option to do it) then trying to sneak it by tsvwg by negotiating in-band and pretending not to modify TCP seems like bad protocol design as well as questionable politics. > It would be better > to pursue an inband notification mechanism that could also > satisfy the "ensure no confusion about bytestream contents" > requirement. Getting that design and the underlying rationale, > weaknesses, tradeoffs, etc. are worked out and well-described > in an Internet-Draft ought to be done before considering > whether the result makes a sufficient case for a new TCP > option. The fundamental choice is between: A. "TCP is a byte stream protocol and thou shalt not attempt to use it otherwise." and B. "Let's treat TCP as a sequence of segments which the sender can construct, the network will preserve, and the receiver can exploit." If the answer is A, then it's a waste of time to talk about either TCP options or in-band notification of alignment. If the answer is B, then in-band notification versus TCP option is a legitimate debate. I do object to the notion of, "Let's do B, but be sneaky enough about it so that nobody looking at the wire can determine that we aren't doing A, therefore tsvwg has no right to object." It results in a bad and overly complex protocol motivated by political rather than technical considerations. > > Thanks, > --David _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 14:42:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00247 for ; Fri, 24 Jan 2003 14:42:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OK1sW24755 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 15:01:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OK1sJ24752 for ; Fri, 24 Jan 2003 15:01:54 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00232 for ; Fri, 24 Jan 2003 14:42:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OK17J24731; Fri, 24 Jan 2003 15:01:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OK0BJ24653 for ; Fri, 24 Jan 2003 15:00:11 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00209 for ; Fri, 24 Jan 2003 14:40:31 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id LAA15890; Fri, 24 Jan 2003 11:43:49 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 14:43:48 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FCB@xbl.ma.emulex.com> From: "Williams, Jim" To: "'John Hufferd'" Cc: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Date: Fri, 24 Jan 2003 14:43:47 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , John, > -----Original Message----- > From: John Hufferd [mailto:hufferd@us.ibm.com] > > Jim, > I do not see the world the same as you do. "Political" > considerations as > you call it is process that one follows while dealing with various > different belief structures. In this case, the gray beards, > as you call > them, have had years of work that has helped form their > beliefs. It is > silly to simply ignore them and useful to find a path that > considers them. > Doing this may be "political", but that is not necessary a > bad thing, and > is the usual way groups of people work together. You use the > term as if > "political" has the meaning of "wrong", and in so doing > negate the years of > experience that is represented by the beliefs that are involved. That's fair. I was perhaps being a bit too flip, which often doesn't work well over email, so I apologize. I should have substituted "misguided political" for "political". I don't mean to imply that all political considerations are misguided. But in some cases it is fair to argue that they are. If anything I meant to underscore the value of the beliefs that have come from years of experience, but that value comes from fully understanding the reasons behind those beliefs. The danger is in treating those beliefs too superficially, which constitutes "misguided politics" in my opinion. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 17:37:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA04403 for ; Fri, 24 Jan 2003 17:37:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0OMuqd02901 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 17:56:52 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OMuqJ02898 for ; Fri, 24 Jan 2003 17:56:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA04400 for ; Fri, 24 Jan 2003 17:37:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OMuEJ02884; Fri, 24 Jan 2003 17:56:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0OMtVJ02826 for ; Fri, 24 Jan 2003 17:55:31 -0500 Received: from msgbas1x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA04348 for ; Fri, 24 Jan 2003 17:35:48 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id ABE5916A87; Fri, 24 Jan 2003 15:39:16 -0700 (MST) Received: from axcsbh1.cos.agilent.com (axcsbh1.cos.agilent.com [130.29.152.143]) by relcos1.cos.agilent.com (Postfix) with SMTP id 65D244D3; Fri, 24 Jan 2003 15:39:16 -0700 (MST) Received: from 130.29.152.143 by axcsbh1.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 24 Jan 2003 15:39:15 -0700 Received: by axcsbh1.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 15:39:15 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0106F1825@axcs04.cos.agilent.com> To: Black_David@emc.com, cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Date: Fri, 24 Jan 2003 15:39:14 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David, One thing markers enable is placement of most of the data that arrives out of order even in the absence of alignment. The marker can be used to find the start of the frame. If markers are placed every 512 bytes, then any DDP segment starting more than 512 bytes after the start of a gap in the TCP stream will have a marker pointing to it. The larger the marker spacing, the more data one has to hold rather than place while waiting for a gap to be filled in. Pat -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Thursday, January 23, 2003 4:51 PM To: cait@asomi.com; rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Caitlin, (WG chair hat off) > >provided that we had markers that were reasonable for > >software to cope with (IMHO, MPA's current 512-byte fixed > >interval isn't). > > I do not see any undue burden on a software solution from > markers, at whatever interval, unless the software makes the > misktake of physically assembling the DDP Segment in a > consecutive sequence of memory bytes. A copy is a way of completely fouling up, but one can get into trouble long before then, although "undue" is in the eye of the beholder. > The first step is to identify the DDP segment. With markers > embedded, the identified location consists of portions of > one or more received IP datagrams. If you want to be even > more nitty gritty, that is generally within one more > received Ethernet Frames (but there are exceptions). > > The DDP implementation has to do two things with the > received DDP segment: interpret the DDP header and place the > DDP contents. Both can be accomplished with a scatter/gather > list. Unless you attempt to interpret the DDP header while > it is has a marker imbedded in the middle of it there really > isnt' that much of a problem for the software. The "scatter/gather list" is the opportunity to get into trouble. Suppose we're dealing with an 8k data block in a jumbo frame (can be made to work in the data center). A 512-byte marker interval yields over 30 entries in the scatter/gather list, whereas a larger marker interval could yield 3. That sort of order of magnitude overhead difference tends to have visible costs in both software and hardware, doubly so if the hardware can't cope with a 30-entry list and needs to be redriven by software to get the job done. For a user-mode implementation on an unmodified stack, those are syscalls rather than scatter/ gather list entries if the syscall interface is one that doesn't support scatter/gather. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 19:48:44 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA06482 for ; Fri, 24 Jan 2003 19:48:44 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0P17w210029 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 20:07:58 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P17wJ10026 for ; Fri, 24 Jan 2003 20:07:58 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA06478 for ; Fri, 24 Jan 2003 19:48:13 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P17AJ09648; Fri, 24 Jan 2003 20:07:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P16UJ09340 for ; Fri, 24 Jan 2003 20:06:30 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA06471 for ; Fri, 24 Jan 2003 19:46:45 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 19:50:13 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C81E@corpmx14.us.dg.com> To: Jim.Williams@Emulex.com, rddp@ietf.org Subject: RE: [rddp] Some Marker Comments Date: Fri, 24 Jan 2003 19:49:19 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, > > > 5. Some method needs to be created for a sender > > > to notify the receiver of ability to align. > > > There are strong arguments that this should > > > be done at the transport, not the RDDP layer. > > > (i.e. a new TCP option on the SYN segment). > > > > IMHO (WG chair hat still off), the odds of a new TCP > > option for this purpose getting through tsvwg are close > > to zero (I've been involved in somewhat analogous attempts, > > and my ears are still ringing ...). > > I hate to see important technical issues decided by > fear of being yelled at by people with gray beards. :) > On the other hand some valuable experience > comes along with the gray beards, and that should be factored > into the discussion. If the fundamental objection is to > aligning TCP (as opposed to using a TCP option to do it) > then trying to sneak it by tsvwg by negotiating in-band > and pretending not to modify TCP seems like bad protocol > design as well as questionable politics. I believe the "fundamental objection" in this case is to creation of a TCP option for something (alignment) whose benefits are yet to be convincingly shown. As I wrote: > > It would be better > > to pursue an inband notification mechanism that could also > > satisfy the "ensure no confusion about bytestream contents" > > requirement. Getting that design and the underlying rationale, > > weaknesses, tradeoffs, etc. worked out and well-described > > in an Internet-Draft ought to be done before considering > > whether the result makes a sufficient case for a new TCP > > option. > > The fundamental choice is between: > > A. "TCP is a byte stream protocol and thou shalt not attempt > to use it otherwise." > and > > B. "Let's treat TCP as a sequence of segments which the sender > can construct, the network will preserve, and the receiver > can exploit." > > If the answer is A, then it's a waste of time to talk about either > TCP options or in-band notification of alignment. > > If the answer is B, then in-band notification versus TCP option > is a legitimate debate. It's a bit more nuanced than that. Those who need 100% undiluted "choice B" need to go use SCTP. If the answer were 100% undiluted "choice A", there would not be a TCP work item on the RDDP charter. So, the answer is somewhere in between. Among the tasks of the RDDP WG is to figure out where that right place is and back that choice with solid design rationale. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 24 20:15:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06798 for ; Fri, 24 Jan 2003 20:15:35 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0P1YoS10913 for rddp-archive@odin.ietf.org; Fri, 24 Jan 2003 20:34:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P1YnJ10910 for ; Fri, 24 Jan 2003 20:34:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06793 for ; Fri, 24 Jan 2003 20:15:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P1Y2J10895; Fri, 24 Jan 2003 20:34:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0P1XZJ10867 for ; Fri, 24 Jan 2003 20:33:35 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06790 for ; Fri, 24 Jan 2003 20:13:50 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 24 Jan 2003 20:17:17 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C81F@corpmx14.us.dg.com> To: Jim.Williams@Emulex.com, rddp@ietf.org Subject: RE: [rddp] Alignment Consensus attempt Date: Fri, 24 Jan 2003 20:16:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, > Vendor B has a product which performs super when > receiving aligned PDUs, but when receiving unaligned PDUs > it simply refuses to negotiate use of RDDP and defaults > back to non-RDDP version of ULP. Therefore, vendor B > actually performs better than vendor A in the unaligned > case. But we say that vendor B's implementation is > non-compliant. This has walked across the line from "MUST implement" to "MUST use" requirements very quickly. The "allow [sender] implementation over unmodified TCP/IP stacks" requirement is a "MUST implement". If Vendor B has not implemented unaligned PDU reception support, and cannot receive from such stacks under any circumstances, then it's not compliant. What vendor B is or is not required to negotiate under what circumstances is a different matter. Unfortunately, if vendor B refuses to negotiate unaligned reception in all circumstances, the fact that it has implemented unaligned reception cannot be demonstrated, and that probably puts vendor B out of compliance with a "MUST implement" requirement. As long as vendor B can negotiate and perform unaligned reception, there's probably a good degree of administrator flexibility available in what to configure vendor B's device to do based on performance, resource usage, etc. > So, I would suggest that it is reasonable that an RDDP > endpoint ought to be allowed to negotiate not to use > RDDP with a sender of unaligned PDUs. We could, however, > push this back to the ULP, and say that the receiver > should advise (as part of the ULP negotiation) that > is doesn't want to do RDDP with an unaligned sender, > and the sender should have sense enough in that > case to just not use RDDP (unless there is some > overriding consideration). While this is probably reasonable, I think there's another important case to consider - the ULP has only implemented the RDDP support and can't cope with RDDP being turned off. Vendor B may deal with this case by passing the unaligned PDUs through to software that takes care of the RDDP [esp. RDMA] functionality so the ULP still works. One of the benefits often claimed for RDMA is that it simplifies protocol development, but that won't come to pass if every RDMA-using protocol has to be implemented in both an RDMA and a non-RDMA version. > So, I don't necessarily disagree with the above > consensus attempt, but would hope that this case > is handled in a sensible manner. If a > particular connection is better off not being > in RDDP mode, then the negotiation process should > leave it in non-RDDP mode, and this should happen > naturally and automatically without requiring > undue ULP or operator intervention. As noted above, the IETF distinguishes between "implementation" and "use" requirements. Anything involving "negotiation" tends to fall into the "use" category, which is not what the consensus attempt was about. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jan 27 19:15:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01543 for ; Mon, 27 Jan 2003 19:15:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h0S0aA029279 for rddp-archive@odin.ietf.org; Mon, 27 Jan 2003 19:36:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0S0aAJ29276 for ; Mon, 27 Jan 2003 19:36:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01538 for ; Mon, 27 Jan 2003 19:14:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0S0ZJJ29218; Mon, 27 Jan 2003 19:35:19 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h0S0YeJ29179 for ; Mon, 27 Jan 2003 19:34:40 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01501 for ; Mon, 27 Jan 2003 19:13:27 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 06051244B16 for ; Mon, 27 Jan 2003 19:16:58 -0500 (EST) Date: Mon, 27 Jan 2003 18:17:50 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] STag Invalidation and RDMAP vs DDP question Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit What are the requirements for the timing of STag invalidation by the RDMAP layer? There are two scenarios that could make this something requiring careful specification. a) The immediately following message attempts to use the same STag. If the DDP layer MUST reject it, then the DDP layer will have to wait synchronously for the RDMAP layer to complete processing of the prior invalidating send message. If you are actually considering making these real layers this would be a problem. b) The prior message was an RDMA Read request. Will the read have to remain valid until it completes, because it was queued before the invalidate? I believe a clean layering is possible without disrupting ULP expectations that badly, if the following caveats are added. 1) The RDMAP engine must ensure that it has extracted all information required from an STag to complete any pending RDMA Read before it processes the STag Invalidation. 2) The RDMAP engine must not report completion of an invalidating send to the ULP before the STag has in fact been invalidated. 3) The RDMAP engine will declare an error when it detects an RDMA Write to an invalidated STag, even if the DDP layer had allowed the placement to occur. The net effect is that the memory is exposed for a very brief additional period, but that the exposure must be eliminated before the consumer is told of the invalidation. If something did slip into the crack, the consumer is told about it as well. Without this caveat, I believe that the DDP and RDMAP "layers" would have to be so synchronous with each other that they would not in fact be layers. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jan 31 20:08:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA24167 for ; Fri, 31 Jan 2003 20:08:22 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h111C6n31544 for rddp-archive@odin.ietf.org; Fri, 31 Jan 2003 20:12:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h111C6J31541 for ; Fri, 31 Jan 2003 20:12:06 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA24148 for ; Fri, 31 Jan 2003 20:07:50 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h111BJJ31521; Fri, 31 Jan 2003 20:11:19 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h111AqJ31498 for ; Fri, 31 Jan 2003 20:10:52 -0500 Received: from msgbas2x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA24142 for ; Fri, 31 Jan 2003 20:06:37 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id B5B022527; Fri, 31 Jan 2003 18:10:08 -0700 (MST) Received: from axcsbh3.cos.agilent.com (axcsbh3.cos.agilent.com [130.29.152.190]) by relcos1.cos.agilent.com (Postfix) with SMTP id 117905AE; Fri, 31 Jan 2003 18:10:07 -0700 (MST) Received: from 130.29.152.190 by axcsbh3.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 31 Jan 2003 18:10:06 -0700 Received: by axcsbh3.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 31 Jan 2003 18:10:06 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0109BDD3D@axcs04.cos.agilent.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] STag Invalidation and RDMAP vs DDP question Date: Fri, 31 Jan 2003 18:10:04 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, The only requirement is that the RDMAP for STag invalidation is that STag is to be invalided before RDMAP delivers the Send with Invalidate to the ULP. The purpose of the Send with Invalidate is to invalidate the STag so that the ULP can reuse the STag either on the same connection or another connection. That protection is accomplished by ensuring that the STag will be invalidated before the completion of the Send with Invalidate Message is delivered to the ULP. For scenario a, a Send with Invalidate Message doesn't protect against a following message in the data stream changing the data because DDP is allowed to do out of order placement. If the remote peer was going to corrupt the data it was writing to the tagged buffer, it could write the wrong data in the first place or it could delay sending the Send with Invalidate. There is therefore no reason to protect against an RDMA Write to the tagged buffer in the stream right after the Send with Invalidate as long as the data is placed before the Send with Invalidate is delivered to the ULP. If scenario b occurs, something is broken. A ULP should not issue a Send with invalidate for an STag when there are uncompleted RDMA Read Requests. There is no ordering between completion of RDMA Read Requests and completion of Sends. If scenario b occurs the STag will be invalidated and DDP may get an Invalid STag error if it hasn't finished processing the RDMA Read Response. Regards, Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Monday, January 27, 2003 4:18 PM To: rddp@ietf.org Subject: [rddp] STag Invalidation and RDMAP vs DDP question What are the requirements for the timing of STag invalidation by the RDMAP layer? There are two scenarios that could make this something requiring careful specification. a) The immediately following message attempts to use the same STag. If the DDP layer MUST reject it, then the DDP layer will have to wait synchronously for the RDMAP layer to complete processing of the prior invalidating send message. If you are actually considering making these real layers this would be a problem. b) The prior message was an RDMA Read request. Will the read have to remain valid until it completes, because it was queued before the invalidate? I believe a clean layering is possible without disrupting ULP expectations that badly, if the following caveats are added. 1) The RDMAP engine must ensure that it has extracted all information required from an STag to complete any pending RDMA Read before it processes the STag Invalidation. 2) The RDMAP engine must not report completion of an invalidating send to the ULP before the STag has in fact been invalidated. 3) The RDMAP engine will declare an error when it detects an RDMA Write to an invalidated STag, even if the DDP layer had allowed the placement to occur. The net effect is that the memory is exposed for a very brief additional period, but that the exposure must be eliminated before the consumer is told of the invalidation. If something did slip into the crack, the consumer is told about it as well. Without this caveat, I believe that the DDP and RDMAP "layers" would have to be so synchronous with each other that they would not in fact be layers. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp