From mailnull@www1.ietf.org Sun Jun 1 10:51:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA14224 for ; Sun, 1 Jun 2003 10:51:28 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h51Ep4S16303 for rddp-archive@odin.ietf.org; Sun, 1 Jun 2003 10:51:04 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51Ep3B16300 for ; Sun, 1 Jun 2003 10:51:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA14187 for ; Sun, 1 Jun 2003 10:50:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MU94-00003t-00 for rddp-web-archive@ietf.org; Sun, 01 Jun 2003 10:49:14 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MU94-00003q-00 for rddp-web-archive@ietf.org; Sun, 01 Jun 2003 10:49:14 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51EoDB16264; Sun, 1 Jun 2003 10:50:13 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51EnlB16227 for ; Sun, 1 Jun 2003 10:49:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA14166 for ; Sun, 1 Jun 2003 10:49:41 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MU7q-000037-00 for rddp@ietf.org; Sun, 01 Jun 2003 10:47:58 -0400 Received: from sandmail.sandburst.com ([216.57.132.42]) by ietf-mx with esmtp (Exim 4.12) id 19MU7p-00002P-00 for rddp@ietf.org; Sun, 01 Jun 2003 10:47:57 -0400 To: Caitlin Bestler Cc: rddp@ietf.org Subject: Re: [rddp] CRC - rough consensus update In-Reply-To: Message from Caitlin Bestler of "Sat, 31 May 2003 12:12:57 CDT." References: Date: Sun, 01 Jun 2003 10:49:09 -0400 From: Stephen Bailey Message-Id: <20030601144910.C2DDE4EB6@sandmail.sandburst.com> Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > There will indeed be software stacks that are cleanly > layered on top of existing TCP stacks. But they will > mostly be used for remote clients at low speeds. I think cleanly layered software implementations will be 90% of the initial RDDP connection endpoints. I disagree that it's acceptable that they be `low speed'. Pure software TCP implemention run at high speeds today with a cleanly layered implementation. Hardware offload hardware will eventually take over the world. We piddled for a long time with PIO disk controllers (`nobody can use more than x bytes/s of disk data, and the CPU can handle it easily'), and eventually offload won. This is no different. The question is when. A hard line position that makes RDDP incrementally less attractive than the `incumbent' (straight TCP), will slow RDDP's adoption. Significantly? For mandated use of CRC, I think so. I am claiming that optional-to-use CRC is technically defensible (not technically superior), and will substantially ease adoption of the technology. There are vastly more communication users out there that understand straight TCP than that understand RDDP. If you require CRC, you will have to explain to them: 1) why their CPU overhead is noticably worse than straight TCP (`because you're getting much better data integrity, and the implementation isn't really optimized, mumble, mumble....'), 2) why TCP's data integrity, on which they have relied all these years is actually dangerously inadequate. Even in the very small community that understood the value of protocol offload, it was slow (took years) to accept the advantage of generic offload (RDMA) over application specific offload (SCSIs, ServerNets, etc..). RDDP is a `sell' to a much larger (and less informed) community. I agree 90% of the software users will not care about the cost of CRC. It will be on for them and they won't even know it. However, some users, especially in the early days, will notice the cost, and will not like the idea of being FORCED to pay it. For those situations, being able to say `you get stronger integrity with this protocol, but you can turn it off if you REALLY need the performance back' is a huge help. At that point, most of the 10% that notice probably won't even turn it off. Forcefeeding a complete, new dogma in one lump just doesn't seem like the quickest road to adoption to me. We should just stack the deck in favor of the dogma, but avoid having the argument. Steph _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jun 1 17:55:29 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA03167 for ; Sun, 1 Jun 2003 17:55:29 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h51Lt4v00675 for rddp-archive@odin.ietf.org; Sun, 1 Jun 2003 17:55:04 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51Lt4B00672 for ; Sun, 1 Jun 2003 17:55:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA03164 for ; Sun, 1 Jun 2003 17:54:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MalP-0005P6-00 for rddp-web-archive@ietf.org; Sun, 01 Jun 2003 17:53:15 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MalO-0005P2-00 for rddp-web-archive@ietf.org; Sun, 01 Jun 2003 17:53:14 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51LsGB00642; Sun, 1 Jun 2003 17:54:16 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h51LraB00624 for ; Sun, 1 Jun 2003 17:53:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA03155 for ; Sun, 1 Jun 2003 17:53:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Majz-0005Oz-00 for rddp@ietf.org; Sun, 01 Jun 2003 17:51:47 -0400 Received: from zcamail05.zca.compaq.com ([161.114.32.105]) by ietf-mx with esmtp (Exim 4.12) id 19Majy-0005Ov-00 for rddp@ietf.org; Sun, 01 Jun 2003 17:51:47 -0400 Received: from cacexg12.americas.cpqcorp.net (cacexg12.americas.cpqcorp.net [16.105.250.119]) by zcamail05.zca.compaq.com (Postfix) with ESMTP id 4D99FDE22; Sun, 1 Jun 2003 14:53:01 -0700 (PDT) Received: from txnexc01.americas.cpqcorp.net ([16.74.7.244]) by cacexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Sun, 1 Jun 2003 14:53:00 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] CRC - rough consensus update Date: Sun, 1 Jun 2003 16:52:59 -0500 Message-ID: <717252EC3E37AE4392E2614EA24E9F2B05044E17@txnexc01.americas.cpqcorp.net> Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMni1Hm/7U8dkMTSq6En9eVmEPV9wA+wPtA From: "Teisberg, Robert" To: "Stephen Bailey" , X-OriginalArrivalTime: 01 Jun 2003 21:53:00.0820 (UTC) FILETIME=[2B92A540:01C32888] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h51LraB00625 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit > Software implementations are most likely to be > `cleanly layered' which means TCP and MPA (and IPSec) will > probably all have different touches of the data. While not disagreeing with Steph's prediction, I'd like to point out that extra data touches are relatively cheap (though not free) in modern processor architectures. As Somesh has obliquely reminded us, the expensive touch is the first one. Subsequently, the data are "hot" in cache and cost much less to touch a second time. "Cleanly layered" software implementations are guaranteed to touch the data at least once. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 12:01:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13326 for ; Mon, 2 Jun 2003 12:01:52 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52G1OL27374 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 12:01:24 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52G1OB27370 for ; Mon, 2 Jun 2003 12:01:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13306 for ; Mon, 2 Jun 2003 12:01:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mrih-0004KT-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 11:59:36 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Mrih-0004KQ-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 11:59:35 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52G0EB27277; Mon, 2 Jun 2003 12:00:14 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52FxYB27215 for ; Mon, 2 Jun 2003 11:59:35 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13252 for ; Mon, 2 Jun 2003 11:59:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mrgw-0004Jm-00 for rddp@ietf.org; Mon, 02 Jun 2003 11:57:46 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Mrgw-0004J0-00 for rddp@ietf.org; Mon, 02 Jun 2003 11:57:46 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 98532244CA0; Mon, 2 Jun 2003 11:59:30 -0400 (EDT) Date: Mon, 2 Jun 2003 10:59:00 -0500 From: Caitlin Bestler Subject: Re: [rddp] CRC - rough consensus update To: Stephen Bailey Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <20030601144910.C2DDE4EB6@sandmail.sandburst.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/1/03, Stephen Bailey wrote: > The question is when. A hard line position that makes > RDDP incrementally less attractive than the `incumbent' > (straight TCP), will slow RDDP's adoption. Significantly? > For mandated use of CRC, I think so. > > I am claiming that optional-to-use CRC is technically > defensible (not technically superior), and will > substantially ease adoption of the technology. I think it is very important that consumers not be given the impression that CRC protection is just something that was thrown in to MPA as some sort of back door attempt to improve TCP. Without CRC protection, RDDP over MPA/TCP is *NOT* in any merely plain TCP on steroids. Without the CRC, there are distinct additional vunerabilities. There are indeed specialized circumstances where the CRC is redundant, but that is a determination that should be made with extreme care. It is not something that should be selected on the justification that "TCP has been good enough so far". The specific extra dangers are: -- RDDP will enable running far closer to wire capacity, even if implemented purely in software. More payload means more risk of payload with undetected errors. -- RDDP and RDMAP interprets portions of payload themselves. Those headers are just as vunerable to undetected TCP errors as the ULP is. But it is far harder for the ULP to protect itself from these errors. Very precise methods of enabling and disabling STags must be followed. These are safety patterns that someone who thinks of this as "TCP on steroids" are likely to deploy correctly. I sense a strong concensus for allowing CRC suppression when the network truly provides the equivalent of end-to-end CRC32c protection. The strongest scenario for this is when most of the path is over IPSEC. But I would be very opposed to any description of CRC suppression which would in any way suggest that this option should be turned on based on the justification that "TCP checksums seem to work OK for our existing applications." MPA/TCP without CRC can be dangerous. It should not be used without careful evaluation. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 12:35:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14465 for ; Mon, 2 Jun 2003 12:35:22 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52GYuc30649 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 12:34:56 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52GYuB30646 for ; Mon, 2 Jun 2003 12:34:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14444 for ; Mon, 2 Jun 2003 12:34:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MsF9-0004aW-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 12:33:07 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MsF8-0004aT-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 12:33:06 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52GY7B30605; Mon, 2 Jun 2003 12:34:07 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52GXgB30560 for ; Mon, 2 Jun 2003 12:33:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14368 for ; Mon, 2 Jun 2003 12:33:37 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MsDx-0004ZT-00 for rddp@ietf.org; Mon, 02 Jun 2003 12:31:53 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19MsDw-0004ZK-00 for rddp@ietf.org; Mon, 02 Jun 2003 12:31:52 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h52GX8LW007710 for ; Mon, 2 Jun 2003 09:33:08 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 2 Jun 2003 12:33:07 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF289B9097@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update Date: Mon, 2 Jun 2003 12:33:03 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin Bestler wrote: > But we have to be very careful to make certain that the > costs do not grow. Most importantly, ULP designers must > have no doubts that they are guaranteed end-to-end CRC32c > protection. I am a bit concerned with confusing service with mechanism. The service provided by the transport is reliable delivery. The mechanism is the number of bits of CRC or checksum. If one wants to quantify reliability, a good metric might be Mean Time Between Undetected Error. Should the ULP be allowed to specify MTBUE to the transport (or MPA) and have the transport adapt the checksum or CRC to this requirement? I think this is a bad idea, but I think having the ULP specify (explicitly or implicitly) the need for a CRC is even worse. > I sense a strong concensus for allowing CRC suppression when > the network truly provides the equivalent of end-to-end > CRC32c protection. The strongest scenario for this is when > most of the path is over IPSEC. That's surprising, my reaction is exactly the opposite. I would regard the risk of IPSEC hardware/software/firmware bugs causing data corruption as far greater than the risk of data corruption on a vanilla TCP connection. I would be much more inclined to keep the CRC on when IPSEC is being used. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 13:33:24 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16007 for ; Mon, 2 Jun 2003 13:33:24 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52HX0302818 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 13:33:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52HX0B02811 for ; Mon, 2 Jun 2003 13:33:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15976 for ; Mon, 2 Jun 2003 13:32:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mt9K-0004tK-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 13:31:10 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Mt9J-0004tH-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 13:31:09 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52HW2B02747; Mon, 2 Jun 2003 13:32:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52HPgB02390 for ; Mon, 2 Jun 2003 13:25:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15716 for ; Mon, 2 Jun 2003 13:25:36 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mt2G-0004qC-00 for rddp@ietf.org; Mon, 02 Jun 2003 13:23:52 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Mt2G-0004q3-00 for rddp@ietf.org; Mon, 02 Jun 2003 13:23:52 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4CBE02453D9; Mon, 2 Jun 2003 13:25:37 -0400 (EDT) Date: Mon, 2 Jun 2003 12:25:07 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: "Williams, Jim" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9097@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/2/03, Williams, Jim wrote: > > >Caitlin Bestler wrote: > >> But we have to be very careful to make certain that the >> costs do not grow. Most importantly, ULP designers must >> have no doubts that they are guaranteed end-to-end CRC32c >> protection. > >I am a bit concerned with confusing service with mechanism. >The service provided by the transport is reliable delivery. >The mechanism is the number of bits of CRC or checksum. > >If one wants to quantify reliability, a good metric might >be Mean Time Between Undetected Error. Should the ULP be >allowed to specify MTBUE to the transport (or MPA) and >have the transport adapt the checksum or CRC to this >requirement? I think this is a bad idea, but I think >having the ULP specify (explicitly or implicitly) the need >for a CRC is even worse. > Agreed. But I would simly state it as "or equivalent", rather than drafting a mathamatical definition. For one thing, equivalence has to be measured relative to the actual error charateristics of the underlying L2 networks and hop points. I wouldn't want to penalize an L2 error detection/correction scheme merely because it was well adapted to that link's actual characteristics. That is especially true when you consider that some links have error correction, not just error detection. Coming up with an equivalence formula that applied to *all* networks would be a monumental undertaking. >> I sense a strong concensus for allowing CRC suppression >> when the network truly provides the equivalent of >> end-to-end CRC32c protection. The strongest scenario for >> this is when most of the path is over IPSEC. > >That's surprising, my reaction is exactly the opposite. >I would regard the risk of IPSEC hardware/software/firmware >bugs causing data corruption as far greater than the risk >of data corruption on a vanilla TCP connection. I would >be much more inclined to keep the CRC on when IPSEC is >being used. > Interesting point. Because an "IPSEC hop" manipulates the payload there is admittedly a higher chance that it would mismanipulate it. But I believe that this is a highly theoretical risk. Any good cryptographic cipher is already engineered to make it nearly impossible to end up with anything that "almost" resembles the true payload. The goal is that even a one bit error in the key will continue to produce gibberish. If this is not true, the cipher is vunerable to differential attacks which allow probing for the correct key one bit at a time. A good cipher provides zero useful information when the incorrect key is supplied, even when the key is "close". The rationale that has been offered for viewing IPsec as "safe" is that it has its own end-to-end protection. That rationale would also apply to many forms of IP tunneling, including ATM and IPoverIB. But in the spirit of the proposal, it would be up to the network administrator to determine whether they viewed the IPsec tunnel to be a single reliable link or not. Ultimately, the real test is that there are a very small number of highly reliable links, each with their own highly reliable error detection (and/or correction). In that sense, IPsec is not really different, it is just that it can be viewed as a single link. Now if the cryptographic algorithm in use is in any way at risk, then its not a highly reliable link. But I would think that an unreliable IPsec link would have been tossed in the trash long before any RDMA applications showed up. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 15:41:24 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA21266 for ; Mon, 2 Jun 2003 15:41:24 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52JeuQ14502 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 15:40:56 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52JeuB14499 for ; Mon, 2 Jun 2003 15:40:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA21239 for ; Mon, 2 Jun 2003 15:40:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mv9A-0005lC-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 15:39:08 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Mv9A-0005l9-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 15:39:08 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52Jc5B14338; Mon, 2 Jun 2003 15:38:05 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52JbSB14046 for ; Mon, 2 Jun 2003 15:37:28 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA21181 for ; Mon, 2 Jun 2003 15:37:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mv5o-0005kV-00 for rddp@ietf.org; Mon, 02 Jun 2003 15:35:40 -0400 Received: from ztxmail03.ztx.compaq.com ([161.114.1.207]) by ietf-mx with esmtp (Exim 4.12) id 19Mv5n-0005kM-00 for rddp@ietf.org; Mon, 02 Jun 2003 15:35:39 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id EDCE9CE97; Mon, 2 Jun 2003 14:36:51 -0500 (CDT) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 2 Jun 2003 14:36:51 -0500 X-MIMEOLE: Produced By Microsoft Exchange V6.0.6375.0 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] CRC - rough consensus update Date: Mon, 2 Jun 2003 14:36:51 -0500 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031431A3@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMoU0LMTKtOIf2cQt+CgXGo8lLE3QA5sIGw From: "Barron, Dwight" To: "Stephen Bailey" , "Caitlin Bestler" Cc: X-OriginalArrivalTime: 02 Jun 2003 19:36:51.0723 (UTC) FILETIME=[50D31DB0:01C3293E] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h52JbSB14055 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit > > I agree 90% of the software users will not care about the cost of CRC. > It will be on for them and they won't even know it. However, some > users, especially in the early days, will notice the cost, and will > not like the idea of being FORCED to pay it. For those situations, > being able to say `you get stronger integrity with this protocol, but > you can turn it off if you REALLY need the performance back' is a huge > help. At that point, most of the 10% that notice probably won't even > turn it off. > > Forcefeeding a complete, new dogma in one lump just doesn't seem like > the quickest road to adoption to me. > > We should just stack the deck in favor of the dogma, but avoid having > the argument. Unfortunately, I don't believe we avoid having the argument. It will get re-argued in every ULP working group, with the same ratio of people thinking RDDP end-to-end data integrity needs to be better than TCP checksums to those thinking the performance issues overwhelm adoption. Then they too will want to have an optional CRC. The efficacy of TCP checksums vs. CRC has been well documented and quantifiable, yet the performance impact of CRC vs. TCP checksums or even a criteria for establishing an acceptable performance impact of CRC has been requested but none has been offered. Regards, Dwight > > Steph > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 16:13:29 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22408 for ; Mon, 2 Jun 2003 16:13:29 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52KD2G17221 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 16:13:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KD2B17218 for ; Mon, 2 Jun 2003 16:13:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22394 for ; Mon, 2 Jun 2003 16:12:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MveD-00063B-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:11:13 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MveC-000637-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:11:12 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KC7B17146; Mon, 2 Jun 2003 16:12:07 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KBgB17117 for ; Mon, 2 Jun 2003 16:11:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22247 for ; Mon, 2 Jun 2003 16:11:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mvcw-00060p-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:09:54 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Mvcv-0005zq-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:09:53 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3B4912455A1; Mon, 2 Jun 2003 16:11:37 -0400 (EDT) Date: Mon, 2 Jun 2003 15:11:07 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: "Barron, Dwight" Cc: Stephen Bailey , rddp@ietf.org X-Priority: 3 In-Reply-To: <01CA1B1F74A4EC40A9C887F7B58F8724031431A3@cceexc16.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/2/03, Barron, Dwight wrote: >Unfortunately, I don't believe we avoid having the >argument. It will get re-argued in every ULP working >group, with the same ratio of people thinking RDDP >end-to-end data integrity needs to be better than TCP >checksums to those thinking the performance issues >overwhelm adoption. Then they too will want to have an >optional CRC. It's worse than that. The farther up you go, the less efficient a CRC implementation *can* be. But as they are installed anyway you end up with *more* CRC implementations, *none* of which are as efficient as they could have been at the lowest layer with end-to-end semantics. That is why I thought we had a consensus that nothing done in the way of making CRCs optional should ever cause a ULP to consider adding its own. The only reason for a ULP to add its own integrity checking should be if they require something *more* than an end-to-end CRC32 check. Do we have agreement on that objective? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 16:29:15 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22746 for ; Mon, 2 Jun 2003 16:29:15 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52KSmc18053 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 16:28:48 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KSmB18050 for ; Mon, 2 Jun 2003 16:28:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22741 for ; Mon, 2 Jun 2003 16:28:44 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MvtT-00068N-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:26:59 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MvtT-00068K-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:26:59 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KS1B18016; Mon, 2 Jun 2003 16:28:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KR1B17963 for ; Mon, 2 Jun 2003 16:27:01 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA22700 for ; Mon, 2 Jun 2003 16:26:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Mvrk-000681-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:25:12 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19Mvrj-00067j-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:25:11 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 86968BCC2; Mon, 2 Jun 2003 15:26:27 -0500 (CDT) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 2 Jun 2003 15:26:27 -0500 X-MIMEOLE: Produced By Microsoft Exchange V6.0.6375.0 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] CRC - rough consensus update Date: Mon, 2 Jun 2003 15:26:23 -0500 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031431A4@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMpQzR4AqmYWlf/SEe2otSTgd0R/AAAVh+A From: "Barron, Dwight" To: "Caitlin Bestler" Cc: "Stephen Bailey" , X-OriginalArrivalTime: 02 Jun 2003 20:26:27.0141 (UTC) FILETIME=[3E4FDF50:01C32945] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h52KR1B17964 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit While I believe there is consensus on that point in this group, I don't know how this workgroup can make such a binding decision on other workgroups that would be defining the ULPs. We do hereby solemnly swear that RDDP will provide data integrity equivalent to end-to-end CRC-32, unless and until it is optionally turned off. Regards, Dwight > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Monday, June 02, 2003 3:11 PM > To: Barron, Dwight > Cc: Stephen Bailey; rddp@ietf.org > Subject: RE: [rddp] CRC - rough consensus update > > > On 6/2/03, Barron, Dwight wrote: > > >Unfortunately, I don't believe we avoid having the > >argument. It will get re-argued in every ULP working > >group, with the same ratio of people thinking RDDP > >end-to-end data integrity needs to be better than TCP > >checksums to those thinking the performance issues > >overwhelm adoption. Then they too will want to have an > >optional CRC. > > It's worse than that. > > The farther up you go, the less efficient a CRC > implementation *can* be. > > But as they are installed anyway you end up with > *more* CRC implementations, *none* of which are as > efficient as they could have been at the lowest layer > with end-to-end semantics. > > That is why I thought we had a consensus that nothing > done in the way of making CRCs optional should ever > cause a ULP to consider adding its own. > > The only reason for a ULP to add its own integrity checking > should be if they require something *more* than an > end-to-end CRC32 check. > > Do we have agreement on that objective? > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 16:49:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23391 for ; Mon, 2 Jun 2003 16:49:14 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52KmmI20096 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 16:48:48 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KmmB20093 for ; Mon, 2 Jun 2003 16:48:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23362 for ; Mon, 2 Jun 2003 16:48:44 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MwCp-0006HV-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:46:59 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MwCo-0006HS-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:46:58 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52Km3B20008; Mon, 2 Jun 2003 16:48:03 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KleB19945 for ; Mon, 2 Jun 2003 16:47:40 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23324 for ; Mon, 2 Jun 2003 16:47:36 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MwBj-0006HG-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:45:51 -0400 Received: from dmz1.silverbacksystems.com ([65.172.158.82]) by ietf-mx with esmtp (Exim 4.12) id 19MwBi-0006Gy-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:45:50 -0400 Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id F1901D610; Mon, 2 Jun 2003 13:46:46 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id C13C624A19; Mon, 2 Jun 2003 13:46:46 -0700 (PDT) Cc: Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 2 Jun 2003 13:46:29 -0700 From: "Somesh Gupta" Importance: Normal In-reply-to: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 28514-69492647; Mon, 02 Jun 2003 13:46:15 -0700 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id A6AF423A01; Mon, 2 Jun 2003 13:46:15 -0700 (PDT) Subject: RE: [rddp] CRC - rough consensus update To: "Caitlin Bestler" X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Caitlin Bestler > Sent: Monday, June 02, 2003 1:11 PM > To: Barron, Dwight > Cc: Stephen Bailey; rddp@ietf.org > Subject: RE: [rddp] CRC - rough consensus update > > > On 6/2/03, Barron, Dwight wrote: > > >Unfortunately, I don't believe we avoid having the > >argument. It will get re-argued in every ULP working > >group, with the same ratio of people thinking RDDP > >end-to-end data integrity needs to be better than TCP > >checksums to those thinking the performance issues > >overwhelm adoption. Then they too will want to have an > >optional CRC. > > It's worse than that. > > The farther up you go, the less efficient a CRC > implementation *can* be. > > But as they are installed anyway you end up with > *more* CRC implementations, *none* of which are as > efficient as they could have been at the lowest layer > with end-to-end semantics. > > That is why I thought we had a consensus that nothing > done in the way of making CRCs optional should ever > cause a ULP to consider adding its own. > > The only reason for a ULP to add its own integrity checking > should be if they require something *more* than an > end-to-end CRC32 check. I am sure that some ULPs will want to add their own integrity checks even if they are in fact weaker than CRC32. If you absolutely positively must not have data corruption under any circumstances, trusting a NIC that offloads integrity checks may not be what you want. The oft-quoted paper indicates that more often than not, corruption was introduced in the stack or in the NIC. > > Do we have agreement on that objective? > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 16:54:25 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23516 for ; Mon, 2 Jun 2003 16:54:25 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h52KrxK20411 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 16:53:59 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KrxB20408 for ; Mon, 2 Jun 2003 16:53:59 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23498 for ; Mon, 2 Jun 2003 16:53:55 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MwHq-0006JN-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:52:10 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MwHp-0006JK-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 16:52:09 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52Kr1B20366; Mon, 2 Jun 2003 16:53:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h52KpaB20264 for ; Mon, 2 Jun 2003 16:51:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA23452 for ; Mon, 2 Jun 2003 16:51:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MwFX-0006Ik-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:49:47 -0400 Received: from dmz1.silverbacksystems.com ([65.172.158.82]) by ietf-mx with esmtp (Exim 4.12) id 19MwFW-0006Id-00 for rddp@ietf.org; Mon, 02 Jun 2003 16:49:46 -0400 Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id D9150D610; Mon, 2 Jun 2003 13:50:47 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id B411524AE3; Mon, 2 Jun 2003 13:50:47 -0700 (PDT) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 2 Jun 2003 13:50:33 -0700 From: "Somesh Gupta" Importance: Normal In-reply-to: <717252EC3E37AE4392E2614EA24E9F2B05044E17@txnexc01.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 28657-3E38F003; Mon, 02 Jun 2003 13:50:20 -0700 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id E72EE24ADD; Mon, 2 Jun 2003 13:50:19 -0700 (PDT) Subject: RE: [rddp] CRC - rough consensus update To: "Teisberg, Robert" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Robert, Varying degrees of offload are quite common these days. So any additional touch may be the only one. As an example, a sendfile does not require any data touch even today. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Teisberg, Robert > Sent: Sunday, June 01, 2003 2:53 PM > To: Stephen Bailey; rddp@ietf.org > Subject: RE: [rddp] CRC - rough consensus update > > > > Software implementations are most likely to be > > `cleanly layered' which means TCP and MPA (and IPSec) will > > probably all have different touches of the data. > > While not disagreeing with Steph's prediction, I'd like to point out > that extra data touches are relatively cheap (though not free) in > modern processor architectures. As Somesh has obliquely reminded > us, the expensive touch is the first one. Subsequently, the data are > "hot" in cache and cost much less to touch a second time. "Cleanly > layered" software implementations are guaranteed to touch the data > at least once. > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 20:19:21 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01104 for ; Mon, 2 Jun 2003 20:19:21 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h530Isd05169 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 20:18:54 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530IsB05166 for ; Mon, 2 Jun 2003 20:18:54 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01095 for ; Mon, 2 Jun 2003 20:18:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MzU8-00004i-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 20:17:05 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MzU8-00004e-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 20:17:04 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530I6B05137; Mon, 2 Jun 2003 20:18:06 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530GqB05010 for ; Mon, 2 Jun 2003 20:16:52 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01059 for ; Mon, 2 Jun 2003 20:16:49 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MzSB-00003r-00 for rddp@ietf.org; Mon, 02 Jun 2003 20:15:03 -0400 Received: from lion.ninthwonder.com ([151.199.66.147]) by ietf-mx with esmtp (Exim 4.12) id 19MzSA-00003g-00 for rddp@ietf.org; Mon, 02 Jun 2003 20:15:02 -0400 Received: by lion.ninthwonder.com (Postfix, from userid 1021) id C0F4330706; Mon, 2 Jun 2003 20:16:37 -0400 (EDT) Date: Tue, 3 Jun 2003 00:15:41 +0000 (UTC) From: wrstuden@wasabisystems.com X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: "Williams, Jim" Cc: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9097@xbl.ma.emulex.com> Message-ID: References: <3356669BBE90C448AD4645C843E2BF289B9097@xbl.ma.emulex.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Mon, 2 Jun 2003, Williams, Jim wrote: > Caitlin Bestler wrote: > > > But we have to be very careful to make certain that the > > costs do not grow. Most importantly, ULP designers must > > have no doubts that they are guaranteed end-to-end CRC32c > > protection. > > I am a bit concerned with confusing service with mechanism. > The service provided by the transport is reliable delivery. > The mechanism is the number of bits of CRC or checksum. > > If one wants to quantify reliability, a good metric might be > Mean Time Between Undetected Error. Should the ULP be > allowed to specify MTBUE to the transport (or MPA) and > have the transport adapt the checksum or CRC to this > requirement? I think this is a bad idea, but I think > having the ULP specify (explicitly or implicitly) the > need for a CRC is even worse. The problem is that if we talk about MTBUE, we have to cast everything to a very strong mathematical basis. While the fact that we can is important, it's not easy to cast the status of a live system into the numbers the math would need, so it would be very hard to actually use such a metric in practice. So if no one will be likely to use it, we should try to describe things a different way. :-) > > I sense a strong concensus for allowing CRC suppression when > > the network truly provides the equivalent of end-to-end > > CRC32c protection. The strongest scenario for this is when > > most of the path is over IPSEC. > > That's surprising, my reaction is exactly the opposite. > I would regard the risk of IPSEC hardware/software/firmware > bugs causing data corruption as far greater than the risk > of data corruption on a vanilla TCP connection. I would > be much more inclined to keep the CRC on when IPSEC is > being used. Interesting. What IPsec stacks are you familiar with? The ones I've used haven't left me with that impression. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 20:42:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01764 for ; Mon, 2 Jun 2003 20:42:17 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h530fow07200 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 20:41:50 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530foB07197 for ; Mon, 2 Jun 2003 20:41:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01751 for ; Mon, 2 Jun 2003 20:41:47 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MzqK-0000EF-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 20:40:00 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19MzqK-0000EC-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 20:40:00 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530f2B07166; Mon, 2 Jun 2003 20:41:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h530e0B07122 for ; Mon, 2 Jun 2003 20:40:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01711 for ; Mon, 2 Jun 2003 20:39:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19MzoY-0000Dg-00 for rddp@ietf.org; Mon, 02 Jun 2003 20:38:10 -0400 Received: from lion.ninthwonder.com ([151.199.66.147]) by ietf-mx with esmtp (Exim 4.12) id 19MzoY-0000Dd-00 for rddp@ietf.org; Mon, 02 Jun 2003 20:38:10 -0400 Received: by lion.ninthwonder.com (Postfix, from userid 1021) id 16FEA30706; Mon, 2 Jun 2003 20:39:57 -0400 (EDT) Date: Tue, 3 Jun 2003 00:39:01 +0000 (UTC) From: wrstuden@wasabisystems.com X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: "Barron, Dwight" Cc: Stephen Bailey , Caitlin Bestler , rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: <01CA1B1F74A4EC40A9C887F7B58F8724031431A3@cceexc16.americas.cpqcorp.net> Message-ID: References: <01CA1B1F74A4EC40A9C887F7B58F8724031431A3@cceexc16.americas.cpqcorp.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Mon, 2 Jun 2003, Barron, Dwight wrote: > > I agree 90% of the software users will not care about the cost of CRC. > > It will be on for them and they won't even know it. However, some > > users, especially in the early days, will notice the cost, and will > > not like the idea of being FORCED to pay it. For those situations, > > being able to say `you get stronger integrity with this protocol, but > > you can turn it off if you REALLY need the performance back' is a huge > > help. At that point, most of the 10% that notice probably won't even > > turn it off. > > > > Forcefeeding a complete, new dogma in one lump just doesn't seem like > > the quickest road to adoption to me. > > > > We should just stack the deck in favor of the dogma, but avoid having > > the argument. > > Unfortunately, I don't believe we avoid having the argument. It will get > re-argued in every ULP working group, with the same ratio of people > thinking RDDP end-to-end data integrity needs to be better than TCP > checksums to those thinking the performance issues overwhelm adoption. > Then they too will want to have an optional CRC. I thought what we were doing to do was say you either have a CRC, or you have a link that the admin has promised is as reliable as a CRC. > The efficacy of TCP checksums vs. CRC has been well documented and > quantifiable, yet the performance impact of CRC vs. TCP checksums or > even a criteria for establishing an acceptable performance impact of CRC > has been requested but none has been offered. I think talking about "performance impact" is the wrong way to go about it. It's not when the "performance impact" of the CRC is ok, it's when the link is reliable-enough w/o the CRC that you can skip it; it's not when is it ok to turn on, it's when is it safe to turn off. As for guidelines, they aren't easy to come up with, which is why we haven't given them. And that's also why we're saying CRCs should be on by default - you should show when you can relax things, not when you should enable the CRC. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 2 21:21:10 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02635 for ; Mon, 2 Jun 2003 21:21:10 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h531KiI09793 for rddp-archive@odin.ietf.org; Mon, 2 Jun 2003 21:20:44 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h531KhB09790 for ; Mon, 2 Jun 2003 21:20:43 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02620 for ; Mon, 2 Jun 2003 21:20:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19N0Rx-0000Up-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 21:18:53 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19N0Rw-0000Um-00 for rddp-web-archive@ietf.org; Mon, 02 Jun 2003 21:18:52 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h531K5B09742; Mon, 2 Jun 2003 21:20:05 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h531JOB09652 for ; Mon, 2 Jun 2003 21:19:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02583 for ; Mon, 2 Jun 2003 21:19:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19N0Qg-0000Td-00 for rddp@ietf.org; Mon, 02 Jun 2003 21:17:34 -0400 Received: from lion.ninthwonder.com ([151.199.66.147]) by ietf-mx with esmtp (Exim 4.12) id 19N0Qf-0000Ta-00 for rddp@ietf.org; Mon, 02 Jun 2003 21:17:33 -0400 Received: by lion.ninthwonder.com (Postfix, from userid 1021) id 2F53230706; Mon, 2 Jun 2003 21:19:20 -0400 (EDT) Date: Tue, 3 Jun 2003 01:18:24 +0000 (UTC) From: wrstuden@wasabisystems.com X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: "Barron, Dwight" Cc: Caitlin Bestler , Stephen Bailey , rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: <01CA1B1F74A4EC40A9C887F7B58F8724031431A4@cceexc16.americas.cpqcorp.net> Message-ID: References: <01CA1B1F74A4EC40A9C887F7B58F8724031431A4@cceexc16.americas.cpqcorp.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Mon, 2 Jun 2003, Barron, Dwight wrote: > While I believe there is consensus on that point in this group, I don't > know how this workgroup can make such a binding decision on other > workgroups that would be defining the ULPs. We do hereby solemnly swear > that RDDP will provide data integrity equivalent to end-to-end CRC-32, > unless and until it is optionally turned off. And we further note that the CRCs being off indicates that the admin(s) governing the connection believe/promise that the connection has data integrity equivalent to end-to-end CRC-32 even without the CRC-32. It took me a while to get exactly what Caitlin was saying with it, but this latter part is important. If we don't do this, then ULPs will be likely to add their own CRCs, making a mess. True, we can't _force_ the admin to not play fast&loose, but at least we can indicate she or he is doing something wrong. The main thing is that such a promise will do is indicate that ULPs that they can depend on CRC- level protection. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 3 02:22:02 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA20999 for ; Tue, 3 Jun 2003 02:22:02 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h536LaM10034 for rddp-archive@odin.ietf.org; Tue, 3 Jun 2003 02:21:36 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h536LZB10031 for ; Tue, 3 Jun 2003 02:21:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA20988 for ; Tue, 3 Jun 2003 02:21:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19N595-0002CB-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 02:19:43 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19N595-0002C7-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 02:19:43 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h536KpB09874; Tue, 3 Jun 2003 02:20:51 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h536JLB09706 for ; Tue, 3 Jun 2003 02:19:21 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA20847 for ; Tue, 3 Jun 2003 02:19:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19N56v-0002BO-00 for rddp@ietf.org; Tue, 03 Jun 2003 02:17:29 -0400 Received: from mms2.broadcom.com ([63.70.210.59]) by ietf-mx with esmtp (Exim 4.12) id 19N56u-0002BL-00 for rddp@ietf.org; Tue, 03 Jun 2003 02:17:28 -0400 Received: from 63.70.210.1 by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Mon, 02 Jun 2003 23:15:55 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id XAA20754; Mon, 2 Jun 2003 23:18:49 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Mon, 2 Jun 2003 23:17:50 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMpaiz6UF1yGt4ETBWDUFpEg6WSmwAAXnwQ From: "Uri Elzur" To: wrstuden@wasabisystems.com, "Barron, Dwight" cc: "Stephen Bailey" , "Caitlin Bestler" , rddp@ietf.org X-WSS-ID: 12C2E2912107152-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h536JLB09707 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit No one suggests to turn off Ethernet CRC even if you can show the local short link from a host to the next switch is error free. Why? It is in the hardware and it is free, so why bother. It was probably as difficult 20 years ago running @10Mb/S. If we make it mandatory it will be implemented in products supporting the outcome of this group's work. After some time, the impact for those who do it in SW, will go away as more will use HW acceleration or simply have faster CPU. On the other hand, we need be careful not to impose a too strong of an impact for the transition period. That's why a performance argument is interesting. If the _option_ to turn it off is there, it will forever add a negotiation which slows down the initialization and adds complexity to implementations. It will be yet another knob the administrator has to deal with and no one showed there is a HUGE performance hit Uri -----Original Message----- From: wrstuden@wasabisystems.com [mailto:wrstuden@wasabisystems.com] Sent: Monday, June 02, 2003 5:39 PM To: Barron, Dwight Cc: Stephen Bailey; Caitlin Bestler; rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update On Mon, 2 Jun 2003, Barron, Dwight wrote: > > I agree 90% of the software users will not care about the cost of > > CRC. It will be on for them and they won't even know it. However, > > some users, especially in the early days, will notice the cost, and > > will not like the idea of being FORCED to pay it. For those > > situations, being able to say `you get stronger integrity with this > > protocol, but you can turn it off if you REALLY need the performance > > back' is a huge help. At that point, most of the 10% that notice > > probably won't even turn it off. > > > > Forcefeeding a complete, new dogma in one lump just doesn't seem > > like the quickest road to adoption to me. > > > > We should just stack the deck in favor of the dogma, but avoid > > having the argument. > > Unfortunately, I don't believe we avoid having the argument. It will > get re-argued in every ULP working group, with the same ratio of > people thinking RDDP end-to-end data integrity needs to be better than > TCP checksums to those thinking the performance issues overwhelm > adoption. Then they too will want to have an optional CRC. I thought what we were doing to do was say you either have a CRC, or you have a link that the admin has promised is as reliable as a CRC. > The efficacy of TCP checksums vs. CRC has been well documented and > quantifiable, yet the performance impact of CRC vs. TCP checksums or > even a criteria for establishing an acceptable performance impact of > CRC has been requested but none has been offered. I think talking about "performance impact" is the wrong way to go about it. It's not when the "performance impact" of the CRC is ok, it's when the link is reliable-enough w/o the CRC that you can skip it; it's not when is it ok to turn on, it's when is it safe to turn off. As for guidelines, they aren't easy to come up with, which is why we haven't given them. And that's also why we're saying CRCs should be on by default - you should show when you can relax things, not when you should enable the CRC. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 3 10:48:48 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA07626 for ; Tue, 3 Jun 2003 10:48:48 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h53EmOD21068 for rddp-archive@odin.ietf.org; Tue, 3 Jun 2003 10:48:24 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53EmOB21065 for ; Tue, 3 Jun 2003 10:48:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA07618 for ; Tue, 3 Jun 2003 10:48:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ND3X-0006em-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 10:46:31 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ND3W-0006ej-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 10:46:30 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53ElfB20986; Tue, 3 Jun 2003 10:47:41 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53EjNB20809 for ; Tue, 3 Jun 2003 10:45:23 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA07514 for ; Tue, 3 Jun 2003 10:45:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ND0b-0006dj-00 for rddp@ietf.org; Tue, 03 Jun 2003 10:43:29 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19ND0b-0006de-00 for rddp@ietf.org; Tue, 03 Jun 2003 10:43:29 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h53Ei1LW027919 for ; Tue, 3 Jun 2003 07:44:36 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Tue, 3 Jun 2003 10:43:36 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF289B9098@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update Date: Tue, 3 Jun 2003 10:43:28 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > If the _option_ to turn it off is there, it will forever add a > negotiation which slows down the initialization and adds complexity to > implementations. It will be yet another knob the administrator has to > deal with and no one showed there is a HUGE performance hit > > Uri The complexity and performance burden of the option should be somewhere between negligible and zero if done right. Implementations are free to always use CRC and always require remote end to use CRC if the implementers wish. The knob only exists if implementers feel there is demand for it. The administrator only needs to know about the knob if he wants to use it. The pragmatic issue is that a customer may come to us and say, "We haven't seen a TCP checksum error in years, but we are wasting 10% of our compute resources on CRCs. Can you turn off the CRCs for us?" We will of course say yes, regardless of whether the standard allows it or not. But I would rather have a mechanism to do this defined within the standard than have to invent an ad hoc non-standard mechanism. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 3 11:11:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08508 for ; Tue, 3 Jun 2003 11:11:06 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h53FAgq23433 for rddp-archive@odin.ietf.org; Tue, 3 Jun 2003 11:10:42 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53FAgB23430 for ; Tue, 3 Jun 2003 11:10:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08496 for ; Tue, 3 Jun 2003 11:10:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NDP7-0006sP-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 11:08:49 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NDP6-0006sM-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 11:08:48 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53FA2B23381; Tue, 3 Jun 2003 11:10:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53F96B23320 for ; Tue, 3 Jun 2003 11:09:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA08436 for ; Tue, 3 Jun 2003 11:08:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NDNY-0006rE-00 for rddp@ietf.org; Tue, 03 Jun 2003 11:07:12 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19NDNX-0006r5-00 for rddp@ietf.org; Tue, 03 Jun 2003 11:07:11 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0131D2456C7; Tue, 3 Jun 2003 11:08:57 -0400 (EDT) Date: Tue, 3 Jun 2003 10:08:28 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: "Williams, Jim" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9098@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/3/03, Williams, Jim wrote: > >> If the _option_ to turn it off is there, it will forever >> add a negotiation which slows down the initialization >> and adds complexity to implementations. It will be yet >> another knob the administrator has to deal with and no >> one showed there is a HUGE performance hit >> >> Uri > >The complexity and performance burden of the option >should be somewhere between negligible and zero if >done right. Implementations are free to >always use CRC and always require remote end >to use CRC if the implementers wish. The knob only >exists if implementers feel there is demand for it. >The administrator only needs to know about the knob >if he wants to use it. > Good points. What makes the proposal acceptable to me is precisely that it can largely be ignored, and that it does not even require any setup delay to negotiate because the active side can simply send the first FULPDU with a CRC before finding out whether the other side will consent to suppressing CRCs. But it would be best to view them as "knobs". If they exist, and a stack have no ability to suppress CRCs at all is clearly acceptable, they need to be destination dependent. For example: try to suppress CRCs with destinations that conform to this list of IP masks. >The pragmatic issue is that a customer may come to us >and say, "We haven't seen a TCP checksum error in years, >but we are wasting 10% of our compute resources on CRCs. >Can you turn off the CRCs for us?" > >We will of course say yes, regardless of whether the >standard allows it or not. But I would rather have >a mechanism to do this defined within the standard >than have to invent an ad hoc non-standard mechanism. All valid arguments on why standards SHOULD be as accomodating as possible. One small disagreement: any customer who is wasting 10% of their compute resources on CRCs is using the wrong algorithm or has an amazingly underpowered processor. But the degree of waste is only relevant if the cost of the option is signifigant. As proposed, the cost of this option is that some bits in the MPA Mode frame are allocated, and that it has to be documented properly. A "All CRCs All The Time" implementation does not even have to *examine* those bits, it merely has to set its bits to indicate that it will insist on CRCs. That's a compile-time constant initializing a byte within a packet. A packet that would have been sent anyway, and a byte that was part of the packet anyway. You can't get options must cheaper than that. Even for a *slight* advantage for *some* implementations, that is a very low cost. I see no reason not to allow it. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 3 13:33:25 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15731 for ; Tue, 3 Jun 2003 13:33:25 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h53HX0H05107 for rddp-archive@odin.ietf.org; Tue, 3 Jun 2003 13:33:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53HX0B05104 for ; Tue, 3 Jun 2003 13:33:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15721 for ; Tue, 3 Jun 2003 13:32:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NFcq-0000su-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 13:31:08 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NFcq-0000sr-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 13:31:08 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53HWCB05049; Tue, 3 Jun 2003 13:32:12 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53HV9B04949 for ; Tue, 3 Jun 2003 13:31:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15647 for ; Tue, 3 Jun 2003 13:31:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NFb3-0000rp-00 for rddp@ietf.org; Tue, 03 Jun 2003 13:29:17 -0400 Received: from lion.ninthwonder.com ([151.199.66.147]) by ietf-mx with esmtp (Exim 4.12) id 19NFb0-0000rh-00 for rddp@ietf.org; Tue, 03 Jun 2003 13:29:14 -0400 Received: by lion.ninthwonder.com (Postfix, from userid 1021) id C777B30706; Tue, 3 Jun 2003 13:30:58 -0400 (EDT) Date: Tue, 3 Jun 2003 17:29:57 +0000 (UTC) From: wrstuden@wasabisystems.com X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Uri Elzur Cc: "Barron, Dwight" , Stephen Bailey , Caitlin Bestler , rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Mon, 2 Jun 2003, Uri Elzur wrote: > No one suggests to turn off Ethernet CRC even if you can show the local > short link from a host to the next switch is error free. Why? It is in > the hardware and it is free, so why bother. It was probably as difficult > 20 years ago running @10Mb/S. So? I don't see how the fact that no one can nor would turn off the Ethernet CRC matters here. > If we make it mandatory it will be implemented in products supporting > the outcome of this group's work. After some time, the impact for those > who do it in SW, will go away as more will use HW acceleration or simply > have faster CPU. On the other hand, we need be careful not to impose a > too strong of an impact for the transition period. That's why a > performance argument is interesting. Note: everyone is talking about CRCs being MUST implement, so they will be implemented. I disagree that the "impact" of a software CRC will ever go away. Yes, CPUs will get faster & faster. But as they get faster & faster, people will want them to do more; expectations will scale with ability. So whatever impact the CRC has will still be there. Also, we shouldn't talk about turning off the CRCs because of a transition impact. We should talk about doing it because it's safe. RDMA is a lot more sensitive to errors than tcp or sctp, mainly since we use in-stream data to decide where to write the packet contents. And some ULPs are more sensitive than others; ones with multiple buffers have to worry about data going into the wrong buffer. ;-) That's why we think disabling CRCs should be a per-port (ULP) kinda thing. An admin can know, "Oh, this server only has one open buffer at a time, turning off CRCs will be ok given our current network. That server has multiple buffers open, better keep on CRCs," and set policy accordingly. > If the _option_ to turn it off is there, it will forever add a > negotiation which slows down the initialization and adds complexity to > implementations. It will be yet another knob the administrator has to > deal with and no one showed there is a HUGE performance hit Now I'm confused. You talk about not using CRCs above, yet you don't want an option in-band to turn them off? How else would you turn them off? Just arbitrarily flipping them off? That's scary... We want a way in-band for either side to say, "We MUST use markers", so we need a form of negotiation. We have it down to a statement-of-desire negotiation. It's very simple, and closes in one round-trip. Each side says, "I want this," in the first PDU, and then everyone knows what's happening. We also have it down to where this won't slow down initial data, either (to the extent data can flow before the client has heard from the server). Since we will need an initial PDU anyway, using one flag in it for CRCs-can-turn-off isn't a loss at all. So I don't see what we loose from your discussion above by permitting CRCs to be negotiated to off? Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 3 13:51:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16429 for ; Tue, 3 Jun 2003 13:51:16 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h53HopD07632 for rddp-archive@odin.ietf.org; Tue, 3 Jun 2003 13:50:51 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53HopB07629 for ; Tue, 3 Jun 2003 13:50:51 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16405 for ; Tue, 3 Jun 2003 13:50:46 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NFu6-00012W-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 13:48:58 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NFu5-00012T-00 for rddp-web-archive@ietf.org; Tue, 03 Jun 2003 13:48:57 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53Ho5B07558; Tue, 3 Jun 2003 13:50:05 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h53HnjB07519 for ; Tue, 3 Jun 2003 13:49:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16345 for ; Tue, 3 Jun 2003 13:49:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NFt2-00011n-00 for rddp@ietf.org; Tue, 03 Jun 2003 13:47:52 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19NFt2-00011e-00 for rddp@ietf.org; Tue, 03 Jun 2003 13:47:52 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4C44C2452B0 for ; Tue, 3 Jun 2003 13:49:38 -0400 (EDT) Date: Tue, 3 Jun 2003 12:49:08 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] MPA Mode Frame "Negotiation" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit There seems to be an impression that MPA options will require a round-trip. This is not the case. All of the MPA Options discussed to data can be implemented "for free" within the MPA Mode frame that was mandated to ensure that MPA/TCP connections could not be confused with TCP connections. Assuming that the MPA Mode frame will itself be an MPA frame, it would contain the following: -- Whatever "signature" is deemed necessary to state "yes, I really know this is going to be an MPA Mode connection". -- Marker Mode bits (previously outlined in other emails) -- CRC Suppression preferred: if set, this end would prefer to suppress CRCs for this connection. MPA allows multiple frames to be bundled in a single TCP segment. So it is possible for the first send to be packed with the MPA Mode frame and sent immediately, even without having seen the remote end's MPA Mode frame. This is totally save because: -- If the Marker Mode bits do not match the connection will be torn down anyway. -- Calculating a CRC does no harm if the other side, after reading your CRC Suppression flag, ignores the CRC field. Each sound is "announcing" their MPA mode, and proceeding. There is nothing to "negotiate". There is no round-trip required. Now of course, you SHOULD wait for the other ends MPA Mode frame before sending the first request if you have no way of predicting their Marker mode. I would expect that servers that have short sessions, such as web servers, to advertise their mode and/or work in either mode to avoid this problem. As for CRC suppression, well you *could* wait to find out whether calculating a CRC32c was necessary. But if waiting really makes more sense then you have an amazingly underpowered processor. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 12:13:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA29291 for ; Wed, 4 Jun 2003 12:13:28 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h54GD2c30819 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 12:13:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54GD2B30816 for ; Wed, 4 Jun 2003 12:13:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA29272 for ; Wed, 4 Jun 2003 12:12:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Naqz-00055v-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 12:11:09 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Naqz-00055r-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 12:11:09 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54GC9B30733; Wed, 4 Jun 2003 12:12:09 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54G7CB29669 for ; Wed, 4 Jun 2003 12:07:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA29114 for ; Wed, 4 Jun 2003 12:07:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NalM-00053I-00 for rddp@ietf.org; Wed, 04 Jun 2003 12:05:20 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19NalL-00052s-00 for rddp@ietf.org; Wed, 04 Jun 2003 12:05:19 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B45E32457F4 for ; Wed, 4 Jun 2003 12:07:08 -0400 (EDT) Date: Wed, 4 Jun 2003 11:06:37 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Transport Neutral "Connections" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit It is highly preferable if applications can be indifferent as to which underlying reliable transport is used as the LLP for RDMAP/DDP. Even beyond that, there is a desire by application developers to apply the same application logic for RDMA-aware applications no matter which RDMA transport is used. This has led to the development of the Direct Access Transport (DAT) Collaboriative's uDAPL and kDAPL, and to the Open Group's ICSC IT-API. Support of these goals can be enhanced by a relatively minor change to both the SCTP adaptation and in the definition of the MPA Mode Frame. With this change, ULPs would be able to engage in the same message exchanges prior to having a fully operational "connection" between two RDMA-enabled endpoints, whether using RDMAP/DDP over MPA/TCP, over SCTP or even a non-IP transport such as InfiniBand. This type of uniformity will enable more rapid development of RDMA-optimized ULPs. The first issue to be addressed is the need for pre-connection data exchange. An RDMA Endpoint (referred to as a Queue Pair, or QP, in InfiniBand and the draft-hilland proposed iWARP verbs) must be associated with a Protection Domain. No receive operations may be posted to the endpoint before it is associated with a Protection Domain. Indeed under both the InfiniBand and proposed iWARP verbs an endpoint/QP is created within a Protection Domain. There are some applications where the choice of Protection Domain is dependent upon the identity of the remote ULP client. For example, if a user session requires multiple connections, it is highly desireable for all of those connections to use a single Protection Domain. InfiniBand, the DAT APIs and the IT-API all provide for the active side ULP to provide Provide Data when requesting a connection. This data is passed to the ULP to allow it to determine whether to accept the connection, and if so with which endpoint (and implicitly which Protection Domain). The Private Data can also be used to ensure that both ends of the connection have configured their RDMA endpoints compatibly on such matters as the RDMA Read capacity. Further ULP-specific uses are also presumed, such as establishing the identity of the client. Private Data is also allowed for when accepting the connection, to allow completion of any negotiation on RDMA resources and for other ULP reasons. What is proposed is that iWARP standardize on allowing a single ULP message to be sent in each direction in the process of establishing an LLP Connection. For SCTP: the stream reset message would allow variable length Private Data at the end of the chunk. The local interface SHOULD report this data to the ULP when reporting the stream's state transition. An additional Adaptation Layer Control function code would be defined to explicitly terminate, but not re-enable, a stream. The Local interface SHOUD report this, and any included Private Data, to the ULP when reporting the stream's state transition. For MPA/TCP: the MPA Mode Frame would be defined to be an MPA Frame, which allowed the same variable length Private Data at the end of the frame. The local interface SHOULD report this data to the ULP when reporting the connections state transition. For SCTP, a "connection" could be established as follows: - The passive side would listen on the destination port, to indicate its readiness to accept an RDMAP/DDP mode association. - The active side ULP would request a connection from a RDMA endpoint to a destination address and port. - If there was no association already established to that SCTP endpoint, the SCTP association would be established as with normal SCTP associations, except that the Adaptation indication would specifcy RDMAP/DDP. - The active side would select an unused Stream, and send a Stream Reset Data Chunk on that stream, including any ULP supplied Private Data. - The passive side listener would receive the reset chunk. It would notify the ULP of the requested "connection", including any passed Private Data. - To accept the connection request, the ULP would bind the SCTP stream to an RDMA endpoint, and provide the Private Data Response. An Adaptatio Layer Control data chunk with a Function Code of DDP Stream Reset would be sent to re-initialize the reverse stream and accept the "connection" request. - To reject the connection request, the passive side would send an Adaptation Layer Control data chunk with a Function Code of DDP Stream Terminate, including any ULP supplied Private Data. For MPA/TCP, a connection could be established in virtually the same way: - The passive side would listen on the destination port, to indicate its readiness to accept a connection. - The active side ULP would request a connection from a RDMA endpoint to a destination address and port. - The active side would initiate a TCP connection to the destination port. Acceptance/rejection of the connection would proceed as per normal TCP connection establishment. This would allow normal TCP gatekeepers, such as INETD and TCPserver, to exercise their normal safeguard/logging functions. - Both sides would indicate their expectation to receive an MPA Mode frame, and thereby enter MPA mode. - The active side would send an MPA Mode Frame including any ULP supplied Private Data. - The passive side listener would receive the MPA Mode Frame. Notice of this state transisiton would be reported to the Consumer, including the supplied Private Data. - To accept the connection request, the ULP would bind the TCP connections to an RDMA endpoint, and provide the Private Data Response. An MPA Mode Frame would be sent to confirm that this was to be an MPA mode connection, along with any ULP supplied Private data. - To reject the connection request, the passive side would send an incompatible MPA Mode Frame response, including any ULP supplied Private Data. Note that under both protocols exchange of Private Data is strictly optional. Neither the Adaptation Layer Control data chunk or the MPA Mode Frame is required to contain Private Data if the ULP has no desire to send it. When the ULP has no desire to exchange Private Data, the ALC DDP Stream Reset or MPA Mode Frames may be sent immediately by both ends. There is no need to wait for the active side "request" to arrive first. The alternative to this approach is the conduct these negotiations using the LLP alone. SCTP, even in RDMAP/DDP adaptation mode, can still send and receive user messages. A TCP connection can still send and receive byte streams. This approach presents a challenge, however. The ULP using TCP must consume the correct number of "negotiation" bytes *before* transitioning to MPA and binding the TCP connection with an RDMA endpoint. If the ULP would prefer a variable length, or even optional, Private Data exchange this can become a very tricky process. More importantly it is a *different* process than the exchanges required under SCTP or any other RDMA transport. I believe the benefits of allowing uniform "connection" establishment across reliable IP transports justify the addition of variable length Private Data fields to the DDP Stream Reset ALC data chunk and MPA Mode Frame. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 17:23:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13169 for ; Wed, 4 Jun 2003 17:23:51 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h54LNQk22387 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 17:23:26 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54LNQB22384 for ; Wed, 4 Jun 2003 17:23:26 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13142 for ; Wed, 4 Jun 2003 17:23:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NfhL-0000YT-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 17:21:31 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NfhL-0000YQ-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 17:21:31 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54LMIB22327; Wed, 4 Jun 2003 17:22:18 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54LLOB22297 for ; Wed, 4 Jun 2003 17:21:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13110 for ; Wed, 4 Jun 2003 17:21:19 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NffO-0000Xo-00 for rddp@ietf.org; Wed, 04 Jun 2003 17:19:30 -0400 Received: from mms3.broadcom.com ([63.70.210.38]) by ietf-mx with esmtp (Exim 4.12) id 19NffN-0000Xl-00 for rddp@ietf.org; Wed, 04 Jun 2003 17:19:29 -0400 Received: from 63.70.210.1 by mms3.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Wed, 04 Jun 2003 14:21:24 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id OAA00969; Wed, 4 Jun 2003 14:20:55 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Wed, 4 Jun 2003 14:19:55 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMp9bpEqamz16t+Qf208Znm5JHxlwAb7U+Q From: "Uri Elzur" To: wrstuden@wasabisystems.com cc: "Barron, Dwight" , "Stephen Bailey" , "Caitlin Bestler" , rddp@ietf.org X-WSS-ID: 12C0BD5E508333-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h54LLOB22298 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Maybe most important comment is that, if CRC is optional in any way, there is no way (or it becomes very hard) for the ULP to know it is there. It takes a mandatory API to make this info avail on every OS to every ULP. RDDP has no control over the APIs. It means the ULP has to design its own error detection and correction scheme. This might be the most important motivation to not only have CRC as MUST IMPLEMENT but also MUST USE. More comments embedded [UE] -----Original Message----- From: wrstuden@wasabisystems.com [mailto:wrstuden@wasabisystems.com] Sent: Tuesday, June 03, 2003 10:30 AM To: Uri Elzur Cc: Barron, Dwight; Stephen Bailey; Caitlin Bestler; rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update On Mon, 2 Jun 2003, Uri Elzur wrote: > No one suggests to turn off Ethernet CRC even if you can show the > local short link from a host to the next switch is error free. Why? It > is in the hardware and it is free, so why bother. It was probably as > difficult 20 years ago running @10Mb/S. So? I don't see how the fact that no one can nor would turn off the Ethernet CRC matters here. [UE] There was a suggestion on this mailing list that you don't need CRC on local high qual links. My point is that no one really wants to worry about managing CRC on link by link basis because A. it is an additional headache for the admin B. Admin normally has no way to know link qualities C. Admin normally will leave default as-is Second point: much like Ethernet at the time, once the decision to mandate CRC is taken, one can expect HW to support it. No one said we will not mandate CRC on a local link because it will be too much to ask of a 8086 or 80186, before HW supports it... [UE] > If we make it mandatory it will be implemented in products supporting > the outcome of this group's work. After some time, the impact for > those who do it in SW, will go away as more will use HW acceleration > or simply have faster CPU. On the other hand, we need be careful not > to impose a too strong of an impact for the transition period. That's > why a performance argument is interesting. Note: everyone is talking about CRCs being MUST implement, so they will be implemented. [UE]See above Comment [UE] I disagree that the "impact" of a software CRC will ever go away. Yes, CPUs will get faster & faster. But as they get faster & faster, people will want them to do more; expectations will scale with ability. So whatever impact the CRC has will still be there. [UE] Most likely data is touched at least once by SW, i.e. middleware/applications will consume the data. At that point the additional cost for CRC compute goes down significantly. But, as you say, there will always be additional "impact". This is why it is important to deliver a clear direction to the HW community - it is mandatory, put CRC in HW and this is how the "impact" goes away eventually - no need to worry about it in SW anymore [UE] Also, we shouldn't talk about turning off the CRCs because of a transition impact. We should talk about doing it because it's safe. [UE] I guess my message wasn't clear enough. I'm not suggesting to turn it off. I strongly believe CRC needs to be on[UE] RDMA is a lot more sensitive to errors than tcp or sctp, mainly since we use in-stream data to decide where to write the packet contents. And some ULPs are more sensitive than others; ones with multiple buffers have to worry about data going into the wrong buffer. ;-) That's why we think disabling CRCs should be a per-port (ULP) kinda thing. An admin can know, "Oh, this server only has one open buffer at a time, turning off CRCs will be ok given our current network. That server has multiple buffers open, better keep on CRCs," and set policy accordingly. [UE] with tcp you still have the risk of corrupted data landing in the app buffer, so apps (e.g.. iSCSI) have their own CRC mechanism. With RDMA whether there is one of few buffers, data can only land in buffers belonging to that "flow". Again, I don't see an admin turning CRC off. If CRC is avail it will be always on. Admin will never be 100% sure whether the app posts just 1 buffer, mult buffers etc. It can change from one OS to another and from one release to another.[UE] > If the _option_ to turn it off is there, it will forever add a > negotiation which slows down the initialization and adds complexity to > implementations. It will be yet another knob the administrator has to > deal with and no one showed there is a HUGE performance hit Now I'm confused. You talk about not using CRCs above, yet you don't want an option in-band to turn them off? How else would you turn them off? Just arbitrarily flipping them off? That's scary... [UE] no intention to recommend NO CRC. On the contrary, I'm for CRC and always ON. [UE] We want a way in-band for either side to say, "We MUST use markers", so we need a form of negotiation. We have it down to a statement-of-desire negotiation. It's very simple, and closes in one round-trip. Each side says, "I want this," in the first PDU, and then everyone knows what's happening. We also have it down to where this won't slow down initial data, either (to the extent data can flow before the client has heard from the server). Since we will need an initial PDU anyway, using one flag in it for CRCs-can-turn-off isn't a loss at all. So I don't see what we loose from your discussion above by permitting CRCs to be negotiated to off? Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 17:53:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14425 for ; Wed, 4 Jun 2003 17:53:17 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h54Lqre24254 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 17:52:53 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54LqrB24251 for ; Wed, 4 Jun 2003 17:52:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14406 for ; Wed, 4 Jun 2003 17:52:47 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Ng9q-0000ol-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 17:50:58 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Ng9q-0000oi-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 17:50:58 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54Lq6B24236; Wed, 4 Jun 2003 17:52:06 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h54LptB24221 for ; Wed, 4 Jun 2003 17:51:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14279 for ; Wed, 4 Jun 2003 17:51:49 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Ng8u-0000nA-00 for rddp@ietf.org; Wed, 04 Jun 2003 17:50:00 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Ng8t-0000mK-00 for rddp@ietf.org; Wed, 04 Jun 2003 17:49:59 -0400 Received: from 192.168.0.101 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 25B902451AA; Wed, 4 Jun 2003 17:51:41 -0400 (EDT) Date: Wed, 4 Jun 2003 16:51:07 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: Uri Elzur , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > Maybe most important comment is that, if CRC is optional in any way, > there is no way (or it becomes very hard) for the ULP to know it is > there. It takes a mandatory API to make this info avail on every OS to > every ULP. RDDP has no control over the APIs. It means the ULP has to > design its own error detection and correction scheme. This might be the > most important motivation to not only have CRC as MUST IMPLEMENT but > also MUST USE. I agree and disagree. An API that the ULP MUST use to guarantee that CRCs are used is clearly unacceptable. However, if there MAY be an API, which IF used MAY suppress CRCs, but if not available or not used then CRCs MUST be used. That is acceptable to me. If you want CRCs used all of the time, simply set the bit so indicating. In fact we could even agree that "zero" indicates that CRCs are mandatory, so in fact you have to do nothing. The field must be set with the same zero that it would have been set with if it had been "reserved - must be zero". If we were talking about *anything* that an implementation would have to *actively* do I would be in full agreement with you. As it is though, even though I am confident that I would never turn on this option, the cost of allowing others to have it seems to be as close to zero as is humanly possible. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 20:02:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18440 for ; Wed, 4 Jun 2003 20:02:27 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55020n32098 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 20:02:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55020B32095 for ; Wed, 4 Jun 2003 20:02:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18412 for ; Wed, 4 Jun 2003 20:01:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NiAp-0001P9-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:00:07 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NiAo-0001P6-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:00:06 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55019B32048; Wed, 4 Jun 2003 20:01:09 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5500JB31973 for ; Wed, 4 Jun 2003 20:00:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA18365 for ; Wed, 4 Jun 2003 20:00:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Ni9C-0001OP-00 for rddp@ietf.org; Wed, 04 Jun 2003 19:58:26 -0400 Received: from mms3.broadcom.com ([63.70.210.38]) by ietf-mx with esmtp (Exim 4.12) id 19Ni9B-0001OM-00 for rddp@ietf.org; Wed, 04 Jun 2003 19:58:25 -0400 Received: from 63.70.210.1 by mms3.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Wed, 04 Jun 2003 17:00:19 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id QAA01837; Wed, 4 Jun 2003 16:59:49 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Wed, 4 Jun 2003 16:58:49 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMq40P6eJ+dEz8cT1SwkomiRl+1qgAD2W4g From: "Uri Elzur" To: "Caitlin Bestler" , rddp@ietf.org X-WSS-ID: 12C05799530461-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h5500JB31974 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Caitlin Strikes me we're on the same page. However, I don't see a way around using an API, IF CRC is OPTIONAL. Let me try to clarify it, using the following scenario: in the case MPA protocol makes the use of CRC OPTIONAL, in a particular case MPA may have negotiated the connection to NO CRC. Now the ULP cannot tell how MPA has negotiated CRC without a proper API. i.e. I'm not sure the ULP can query MPA without a proper API (that I also think can not be part of what RDDP requires or can be guaranteed to be available) Therefore what will the ULP do? How will a ULP that wants to RELY on MPA's CRC and NOT have to provide its own error recovery do it? Uri -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, June 04, 2003 2:51 PM To: Uri Elzur; rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update > Maybe most important comment is that, if CRC is optional in any way, > there is no way (or it becomes very hard) for the ULP to know it is > there. It takes a mandatory API to make this info avail on every OS to > every ULP. RDDP has no control over the APIs. It means the ULP has to > design its own error detection and correction scheme. This might be > the most important motivation to not only have CRC as MUST IMPLEMENT > but also MUST USE. I agree and disagree. An API that the ULP MUST use to guarantee that CRCs are used is clearly unacceptable. However, if there MAY be an API, which IF used MAY suppress CRCs, but if not available or not used then CRCs MUST be used. That is acceptable to me. If you want CRCs used all of the time, simply set the bit so indicating. In fact we could even agree that "zero" indicates that CRCs are mandatory, so in fact you have to do nothing. The field must be set with the same zero that it would have been set with if it had been "reserved - must be zero". If we were talking about *anything* that an implementation would have to *actively* do I would be in full agreement with you. As it is though, even though I am confident that I would never turn on this option, the cost of allowing others to have it seems to be as close to zero as is humanly possible. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 20:19:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19073 for ; Wed, 4 Jun 2003 20:19:01 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h550IXn01219 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 20:18:33 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550IXB01216 for ; Wed, 4 Jun 2003 20:18:33 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19049 for ; Wed, 4 Jun 2003 20:18:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NiQq-0001Wn-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:16:40 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NiQq-0001Wk-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:16:40 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550HmB01188; Wed, 4 Jun 2003 20:17:48 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550GuB01115 for ; Wed, 4 Jun 2003 20:16:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19026 for ; Wed, 4 Jun 2003 20:16:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NiPH-0001Wf-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:15:03 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19NiPH-0001Wc-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:15:03 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 2DC44244C9E; Wed, 4 Jun 2003 20:16:52 -0400 (EDT) Date: Wed, 4 Jun 2003 19:16:21 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: Uri Elzur Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/4/03, Uri Elzur wrote: >Caitlin > >Strikes me we're on the same page. However, I don't see a >way around using an API, IF CRC is OPTIONAL. > Agreed. By inclination I am against options. I ended up convincing myself that this one made sense while trying to compose a blistering email tearing the rationale for the option to shreds. I could come up with lots of reasons why it was far from essential, but not as to how it hurt anybody (beyond wasting an option bit that might theoretically have found a use sometime 30 years later). >Let me try to clarify it, using the following scenario: in >the case MPA protocol makes the use of CRC OPTIONAL, in a >particular case MPA may have negotiated the connection to >NO CRC. Now the ULP cannot tell how MPA has negotiated >CRC without a proper API. i.e. I'm not sure the ULP can >query MPA without a proper API (that I also think can not >be part of what RDDP requires or can be guaranteed to be >available) > >Therefore what will the ULP do? How will a ULP that wants >to RELY on MPA's CRC and NOT have to provide its own error >recovery do it? > There have been two criteria proposed: First, equivalence of result - not method. The ULP does not truly care *how* the payload is protected (well, at least it shouldn't). It cares *that* it is protected. An end-to-end CRC32c is defined to be a "safe harbor", that is one method is defined which will provide adequate protection without any further questions asked. Under the "equivalent protection" override, the CRC is suppressed for specific paths *only* if the network administrator can vouch that these paths truly have equivalent protections. Personally, I would anticipate that this feature would be used by all system administrators who are bored and have absolutely nothing else in their in-basket. ;-) Second, their MAY be an API that allows a ULP to volunteer for a lower level of protection. This lower level MUST NOT be applied unless the ULP actively invokes this API. Ignorance of such an API MUST leave the higher standard undisturbed. With these two rules, any ULP that requires the equivalence of end-to-end CRC32c protection need do nothing. Any application designed that does not understand how to lower the lower-protection API will receive the full end-to-end CRC32c (or equivalent) protection. So in the absence of an API to query MPA, the ULP can assume that it has CRC32c (or equivalent) protection. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 20:25:19 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19252 for ; Wed, 4 Jun 2003 20:25:19 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h550Oq201572 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 20:24:52 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550OqB01569 for ; Wed, 4 Jun 2003 20:24:52 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19235 for ; Wed, 4 Jun 2003 20:24:49 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NiWx-0001aS-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:22:59 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NiWw-0001aP-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:22:58 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550O2B01541; Wed, 4 Jun 2003 20:24:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550NmB01520 for ; Wed, 4 Jun 2003 20:23:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19222 for ; Wed, 4 Jun 2003 20:23:45 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NiVv-0001a8-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:21:55 -0400 Received: from saway.wasabisystems.com ([151.199.66.145] helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19NiVu-0001a5-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:21:54 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id 5E9BF40136; Wed, 4 Jun 2003 20:23:45 -0400 (EDT) Date: Thu, 5 Jun 2003 00:22:47 +0000 (UTC) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Uri Elzur Cc: "Barron, Dwight" , Stephen Bailey , Caitlin Bestler , rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Wed, 4 Jun 2003, Uri Elzur wrote: > Maybe most important comment is that, if CRC is optional in any way, > there is no way (or it becomes very hard) for the ULP to know it is > there. It takes a mandatory API to make this info avail on every OS to > every ULP. RDDP has no control over the APIs. It means the ULP has to > design its own error detection and correction scheme. This might be the > most important motivation to not only have CRC as MUST IMPLEMENT but > also MUST USE. No. We specifically don't want such an API as it's not needed. That's the whole point of setting guidelines as to when CRCs may be disabled. And that's why I suggested using the word, "promise," in there. Specifically so ULPs _don't_ feel the need to add their own error detection schemes. That's the reason for requiring that the ULP always have a CRC-32C- comparable protection level. Either it's the CRC, or, for some reason that the admin knows, there is a comparable level of protection. End-to-end IPsec is one obvious case. > More comments embedded [UE] > > -----Original Message----- > From: wrstuden@wasabisystems.com [mailto:wrstuden@wasabisystems.com] > Sent: Tuesday, June 03, 2003 10:30 AM > To: Uri Elzur > Cc: Barron, Dwight; Stephen Bailey; Caitlin Bestler; rddp@ietf.org > Subject: RE: [rddp] CRC - rough consensus update > > > On Mon, 2 Jun 2003, Uri Elzur wrote: > > > No one suggests to turn off Ethernet CRC even if you can show the > > local short link from a host to the next switch is error free. Why? It > > > is in the hardware and it is free, so why bother. It was probably as > > difficult 20 years ago running @10Mb/S. > > So? I don't see how the fact that no one can nor would turn off the > Ethernet CRC matters here. > > [UE] > There was a suggestion on this mailing list that you don't need CRC on > local high qual links. My point is that no one really wants to worry > about managing CRC on link by link basis because > A. it is an additional headache for the admin > B. Admin normally has no way to know link qualities > C. Admin normally will leave default as-is The suggestion wasn't that CRCs SHOULD be turned off on high-quality links, but that sites/ULPs will be content/safe if they are. An admin who leaves things as the default will be leaving CRCs enabled. I've heard a number of folks state CRCs should default to on, and I agree with them. > Second point: much like Ethernet at the time, once the decision to > mandate CRC is taken, one can expect HW to support it. No one said we > will not mandate CRC on a local link because it will be too much to ask > of a 8086 or 80186, before HW supports it... [UE] Comparing this CRC with Ethernet's is really clouding the issue. First off, Ethernet _needs_ the CRC. Ethernet was designed as a distributed party line where, among other things, you could have two transmitters ending up sending overlapping transmissions. If the packets were sized right, a receiver could see one entirely contained in the other. Without the CRC, there is no way to detect this error. It has nothing to do with line quality but how ethernet actually works. Second, Ethernet's CRC happens at the very front end of processing. Nothing flows farther up the processing chain if the CRC's bad. RDMA's CRC, though, hapens after a bit of processing. So even if you have hardware assist, chances are you'll have a noticable processing point where the CRC happens. It won't be invisable. > > If we make it mandatory it will be implemented in products supporting > > the outcome of this group's work. After some time, the impact for > > those who do it in SW, will go away as more will use HW acceleration > > or simply have faster CPU. On the other hand, we need be careful not > > to impose a too strong of an impact for the transition period. That's > > why a performance argument is interesting. > > Note: everyone is talking about CRCs being MUST implement, so they will > be implemented. > > [UE]See above Comment [UE] > > I disagree that the "impact" of a software CRC will ever go away. Yes, > CPUs will get faster & faster. But as they get faster & faster, people > will want them to do more; expectations will scale with ability. So > whatever impact the CRC has will still be there. > > [UE] Most likely data is touched at least once by SW, i.e. > middleware/applications will consume the data. At that point the > additional cost for CRC compute goes down significantly. But, as you > say, there will always be additional "impact". This is why it is > important to deliver a clear direction to the HW community - it is > mandatory, put CRC in HW and this is how the "impact" goes away > eventually - no need to worry about it in SW anymore [UE] > > Also, we shouldn't talk about turning off the CRCs because of a > transition impact. We should talk about doing it because it's safe. > > [UE] I guess my message wasn't clear enough. I'm not suggesting to turn > it off. I strongly believe CRC needs to be on[UE] > > RDMA is a lot more sensitive to errors than tcp or sctp, mainly since we > use in-stream data to decide where to write the packet contents. And > some ULPs are more sensitive than others; ones with multiple buffers > have to worry about data going into the wrong buffer. ;-) > > That's why we think disabling CRCs should be a per-port (ULP) kinda > thing. An admin can know, "Oh, this server only has one open buffer at a > time, turning off CRCs will be ok given our current network. That server > has multiple buffers open, better keep on CRCs," and set policy > accordingly. > > [UE] with tcp you still have the risk of corrupted data landing in the > app buffer, so apps (e.g.. iSCSI) have their own CRC mechanism. With > RDMA whether there is one of few buffers, data can only land in buffers > belonging to that "flow". Again, I don't see an admin turning CRC off. > If CRC is avail it will be always on. Admin will never be 100% sure > whether the app posts just 1 buffer, mult buffers etc. It can change > from one OS to another and from one release to another.[UE] I know admins who I think can get this question right. Yes, they are not the majority, but they do exist. I think the majority will just leave CRCs on (especially if we word the spec right), so ULPs will be safe. Also, the iSCSI CRC is a bit different. It has both header and data options, which are independant. The reason for that is so that the data CRC can be end-to-end, even through different iSCSI gateways. For instance, an iSCSI/SCPT <-> iSCSI/TCP gateway, or an iSCSI/TCP <-> iSCSI/RDMA gateway. For an iSCSI/RDMA connection that doesn't get translated, there's really no need for either the header or data CRCs (AFAICT). > > If the _option_ to turn it off is there, it will forever add a > > negotiation which slows down the initialization and adds complexity to > > > implementations. It will be yet another knob the administrator has to > > deal with and no one showed there is a HUGE performance hit > > Now I'm confused. You talk about not using CRCs above, yet you don't > want an option in-band to turn them off? How else would you turn them > off? Just arbitrarily flipping them off? That's scary... > [UE] no intention to recommend NO CRC. On the contrary, I'm for CRC and > always ON. [UE] Well, there are folks on the list who see cases where they want CRCs off. The WG has decided to accomidate them as long as we don't end up shooting ourselves in the foot. Making ULPs think they need to add their own CRC 'cause ours might be off is what we consider, "shooting ourselves in the foot." My understanding of the budding concensus is that that's ok to request disabling CRCs as long as the admin is able to determine that even w/o the CRC, the connection will be as reliable as if it had the CRC enabled. Thus ULPs are ASSURED that they have at least CRC-32C reliability. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 4 20:35:11 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19596 for ; Wed, 4 Jun 2003 20:35:11 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h550Yiw02025 for rddp-archive@odin.ietf.org; Wed, 4 Jun 2003 20:34:44 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550YiB02022 for ; Wed, 4 Jun 2003 20:34:44 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19565 for ; Wed, 4 Jun 2003 20:34:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NigV-0001dM-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:32:51 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NigU-0001dJ-00 for rddp-web-archive@ietf.org; Wed, 04 Jun 2003 20:32:50 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550Y1B01980; Wed, 4 Jun 2003 20:34:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h550XMB01953 for ; Wed, 4 Jun 2003 20:33:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19528 for ; Wed, 4 Jun 2003 20:33:19 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NifB-0001cx-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:31:29 -0400 Received: from saway.wasabisystems.com ([151.199.66.145] helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19NifA-0001cu-00 for rddp@ietf.org; Wed, 04 Jun 2003 20:31:28 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id DAB2640136; Wed, 4 Jun 2003 20:33:19 -0400 (EDT) Date: Thu, 5 Jun 2003 00:32:22 +0000 (UTC) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Uri Elzur Cc: Caitlin Bestler , rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Wed, 4 Jun 2003, Uri Elzur wrote: > Caitlin > > Strikes me we're on the same page. However, I don't see a way around > using an API, IF CRC is OPTIONAL. The point is we aren't making CRCs "OPTIONAL." Optional means you are free to turn it on or off. We've been talking about making it such that the admin MAY indicate that CRCs can be skipped IF the admin is CONFIDENT that the connection will be as reliable as if it had the CRC on. Thus there's a burden on the admin that wants to mark CRCs as ignorable, which is not there if we use the word, "OPTIONAL." > Let me try to clarify it, using the following scenario: in the case MPA > protocol makes the use of CRC OPTIONAL, in a particular case MPA may > have negotiated the connection to NO CRC. Now the ULP cannot tell how > MPA has negotiated CRC without a proper API. i.e. I'm not sure the ULP > can query MPA without a proper API (that I also think can not be part of > what RDDP requires or can be guaranteed to be available) > > Therefore what will the ULP do? How will a ULP that wants to RELY on > MPA's CRC and NOT have to provide its own error recovery do it? The ULP will do the only thing it can. It will know that regardless of the CRC setting, it will have a link as reliable as if it has the CRC-32C verification. Because the only time(s) it is valid for the CRCs to be neglectable is when other factors known to the admin(s) indicate that the connection is reliable. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 10:36:20 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA28596 for ; Thu, 5 Jun 2003 10:36:20 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55EZtJ08982 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 10:35:55 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55EZtB08979 for ; Thu, 5 Jun 2003 10:35:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA28585 for ; Thu, 5 Jun 2003 10:35:50 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NvoU-0007T7-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 10:33:58 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NvoU-0007T4-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 10:33:58 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55EZAB08957; Thu, 5 Jun 2003 10:35:10 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55EYqB08917 for ; Thu, 5 Jun 2003 10:34:52 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA28555 for ; Thu, 5 Jun 2003 10:34:46 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NvnT-0007Su-00 for rddp@ietf.org; Thu, 05 Jun 2003 10:32:55 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19NvnS-0007Sl-00 for rddp@ietf.org; Thu, 05 Jun 2003 10:32:54 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id D0954EC12; Thu, 5 Jun 2003 09:34:15 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Thu, 5 Jun 2003 09:34:15 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Transport Neutral "Connections" Date: Thu, 5 Jun 2003 09:33:24 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762A05103F@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMqtfEEcg/AUsXJQ4GLEHUTgyKfCQAtONVg From: "Culley, Paul" To: "Caitlin Bestler" , X-OriginalArrivalTime: 05 Jun 2003 14:34:15.0613 (UTC) FILETIME=[8A2DD6D0:01C32B6F] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h55EYqB08918 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Regarding "private data"; DDP/MPA are currently expected to be enabled together, after creating the QP (and associating it with a PD). The method we expected to use to deal with private data was to have DDP/MPA's ULP (perhaps a bit of code in the IT-API interface for example) be in charge of starting the TCP connections and then, after doing any negotiations needed between endpoints (using streaming mode TCP data), create the QP and transition the connection to DDP/MPA mode. The "private data" would be exchanged in TCP streaming mode (probably using standard sockets calls under the API). When the Application/API had any necessary stuff completed (like figuring out the correct PD), the QP can be created, buffers posted, and the TCP "connection context" passed to the iWARP stack to start DDP/MPA mode processing. The Verbs document you referred to supports this type of startup. I think that using the TCP streaming mode prior to MPA enablement is preferred to adding additional features to MPA as you suggest. The difference in flow over your connection establishment sequence would only be that MPA is not involved until the "bind" step. The IT-API group would have to publish a small draft describing the use of the streaming mode and expected data format prior to iWARP startup, but this is little different than other applications or APIs such as SDP etc. Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, June 04, 2003 11:07 AM To: rddp@ietf.org Subject: [rddp] Transport Neutral "Connections" It is highly preferable if applications can be indifferent as to which underlying reliable transport is used as the LLP for RDMAP/DDP. Even beyond that, there is a desire by application developers to apply the same application logic for RDMA-aware applications no matter which RDMA transport is used. This has led to the development of the Direct Access Transport (DAT) Collaboriative's uDAPL and kDAPL, and to the Open Group's ICSC IT-API. Support of these goals can be enhanced by a relatively minor change to both the SCTP adaptation and in the definition of the MPA Mode Frame. With this change, ULPs would be able to engage in the same message exchanges prior to having a fully operational "connection" between two RDMA-enabled endpoints, whether using RDMAP/DDP over MPA/TCP, over SCTP or even a non-IP transport such as InfiniBand. This type of uniformity will enable more rapid development of RDMA-optimized ULPs. The first issue to be addressed is the need for pre-connection data exchange. An RDMA Endpoint (referred to as a Queue Pair, or QP, in InfiniBand and the draft-hilland proposed iWARP verbs) must be associated with a Protection Domain. No receive operations may be posted to the endpoint before it is associated with a Protection Domain. Indeed under both the InfiniBand and proposed iWARP verbs an endpoint/QP is created within a Protection Domain. There are some applications where the choice of Protection Domain is dependent upon the identity of the remote ULP client. For example, if a user session requires multiple connections, it is highly desireable for all of those connections to use a single Protection Domain. InfiniBand, the DAT APIs and the IT-API all provide for the active side ULP to provide Provide Data when requesting a connection. This data is passed to the ULP to allow it to determine whether to accept the connection, and if so with which endpoint (and implicitly which Protection Domain). The Private Data can also be used to ensure that both ends of the connection have configured their RDMA endpoints compatibly on such matters as the RDMA Read capacity. Further ULP-specific uses are also presumed, such as establishing the identity of the client. Private Data is also allowed for when accepting the connection, to allow completion of any negotiation on RDMA resources and for other ULP reasons. What is proposed is that iWARP standardize on allowing a single ULP message to be sent in each direction in the process of establishing an LLP Connection. For SCTP: the stream reset message would allow variable length Private Data at the end of the chunk. The local interface SHOULD report this data to the ULP when reporting the stream's state transition. An additional Adaptation Layer Control function code would be defined to explicitly terminate, but not re-enable, a stream. The Local interface SHOUD report this, and any included Private Data, to the ULP when reporting the stream's state transition. For MPA/TCP: the MPA Mode Frame would be defined to be an MPA Frame, which allowed the same variable length Private Data at the end of the frame. The local interface SHOULD report this data to the ULP when reporting the connections state transition. For SCTP, a "connection" could be established as follows: - The passive side would listen on the destination port, to indicate its readiness to accept an RDMAP/DDP mode association. - The active side ULP would request a connection from a RDMA endpoint to a destination address and port. - If there was no association already established to that SCTP endpoint, the SCTP association would be established as with normal SCTP associations, except that the Adaptation indication would specifcy RDMAP/DDP. - The active side would select an unused Stream, and send a Stream Reset Data Chunk on that stream, including any ULP supplied Private Data. - The passive side listener would receive the reset chunk. It would notify the ULP of the requested "connection", including any passed Private Data. - To accept the connection request, the ULP would bind the SCTP stream to an RDMA endpoint, and provide the Private Data Response. An Adaptatio Layer Control data chunk with a Function Code of DDP Stream Reset would be sent to re-initialize the reverse stream and accept the "connection" request. - To reject the connection request, the passive side would send an Adaptation Layer Control data chunk with a Function Code of DDP Stream Terminate, including any ULP supplied Private Data. For MPA/TCP, a connection could be established in virtually the same way: - The passive side would listen on the destination port, to indicate its readiness to accept a connection. - The active side ULP would request a connection from a RDMA endpoint to a destination address and port. - The active side would initiate a TCP connection to the destination port. Acceptance/rejection of the connection would proceed as per normal TCP connection establishment. This would allow normal TCP gatekeepers, such as INETD and TCPserver, to exercise their normal safeguard/logging functions. - Both sides would indicate their expectation to receive an MPA Mode frame, and thereby enter MPA mode. - The active side would send an MPA Mode Frame including any ULP supplied Private Data. - The passive side listener would receive the MPA Mode Frame. Notice of this state transisiton would be reported to the Consumer, including the supplied Private Data. - To accept the connection request, the ULP would bind the TCP connections to an RDMA endpoint, and provide the Private Data Response. An MPA Mode Frame would be sent to confirm that this was to be an MPA mode connection, along with any ULP supplied Private data. - To reject the connection request, the passive side would send an incompatible MPA Mode Frame response, including any ULP supplied Private Data. Note that under both protocols exchange of Private Data is strictly optional. Neither the Adaptation Layer Control data chunk or the MPA Mode Frame is required to contain Private Data if the ULP has no desire to send it. When the ULP has no desire to exchange Private Data, the ALC DDP Stream Reset or MPA Mode Frames may be sent immediately by both ends. There is no need to wait for the active side "request" to arrive first. The alternative to this approach is the conduct these negotiations using the LLP alone. SCTP, even in RDMAP/DDP adaptation mode, can still send and receive user messages. A TCP connection can still send and receive byte streams. This approach presents a challenge, however. The ULP using TCP must consume the correct number of "negotiation" bytes *before* transitioning to MPA and binding the TCP connection with an RDMA endpoint. If the ULP would prefer a variable length, or even optional, Private Data exchange this can become a very tricky process. More importantly it is a *different* process than the exchanges required under SCTP or any other RDMA transport. I believe the benefits of allowing uniform "connection" establishment across reliable IP transports justify the addition of variable length Private Data fields to the DDP Stream Reset ALC data chunk and MPA Mode Frame. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 11:39:38 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03690 for ; Thu, 5 Jun 2003 11:39:37 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55Fd9215462 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 11:39:09 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55Fd9B15459 for ; Thu, 5 Jun 2003 11:39:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03618 for ; Thu, 5 Jun 2003 11:39:07 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Nwnk-0000TU-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 11:37:16 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Nwnj-0000TR-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 11:37:15 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55FcEB15407; Thu, 5 Jun 2003 11:38:14 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55FbYB15041 for ; Thu, 5 Jun 2003 11:37:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA03403 for ; Thu, 5 Jun 2003 11:37:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NwmC-0000Sg-00 for rddp@ietf.org; Thu, 05 Jun 2003 11:35:41 -0400 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 19NwmC-0000Rf-00 for rddp@ietf.org; Thu, 05 Jun 2003 11:35:40 -0400 Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.9/8.12.9/NTAP-1.4) with ESMTP id h55FZsJo000923; Thu, 5 Jun 2003 08:35:54 -0700 (PDT) Received: from svlexc01.hq.netapp.com (svlexc01.corp.netapp.com [10.10.22.171]) by frejya.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h55FZstf009917; Thu, 5 Jun 2003 08:35:54 -0700 (PDT) content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Subject: RE: [rddp] Transport Neutral "Connections" Date: Thu, 5 Jun 2003 08:35:47 -0700 Message-ID: Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMqtfEEcg/AUsXJQ4GLEHUTgyKfCQAtONVgAAJj9GA= From: "Kanevsky, Arkady" To: "Culley, Paul" , "Caitlin Bestler" , Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h55FbYB15050 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit But API can not define protocol only ULP can. So, ITWG or DAT can not define a protocol for connection management nor the one for private data. Arkady Arkady Kanevsky email: arkady@netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 3rd Floor http: www.netapp.com Waltham, MA 02451-2010 general phone: 781-768-5300 > -----Original Message----- > From: Culley, Paul [mailto:Paul.Culley@hp.com] > Sent: Thu, June 05, 2003 10:33 AM > To: Caitlin Bestler; rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > Regarding "private data"; > > DDP/MPA are currently expected to be enabled together, after creating > the QP (and associating it with a PD). The method we > expected to use to > deal with private data was to have DDP/MPA's ULP (perhaps a > bit of code > in the IT-API interface for example) be in charge of starting the TCP > connections and then, after doing any negotiations needed between > endpoints (using streaming mode TCP data), create the QP and > transition > the connection to DDP/MPA mode. > > The "private data" would be exchanged in TCP streaming mode (probably > using standard sockets calls under the API). When the Application/API > had any necessary stuff completed (like figuring out the correct PD), > the QP can be created, buffers posted, and the TCP > "connection context" > passed to the iWARP stack to start DDP/MPA mode processing. The Verbs > document you referred to supports this type of startup. > > I think that using the TCP streaming mode prior to MPA enablement is > preferred to adding additional features to MPA as you suggest. > The difference in flow over your connection establishment > sequence would > only be that MPA is not involved until the "bind" step. > > The IT-API group would have to publish a small draft > describing the use > of the streaming mode and expected data format prior to iWARP startup, > but this is little different than other applications or APIs > such as SDP > etc. > > Paul R. Culley > HP Senior Fellow > 281-514-5543 > > > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Wednesday, June 04, 2003 11:07 AM > To: rddp@ietf.org > Subject: [rddp] Transport Neutral "Connections" > > > It is highly preferable if applications can be indifferent > as to which underlying reliable transport is used as the LLP for > RDMAP/DDP. > > Even beyond that, there is a desire by application > developers to apply the same application logic for > RDMA-aware applications no matter which RDMA transport is > used. This has led to the development of the Direct Access Transport > (DAT) Collaboriative's uDAPL and kDAPL, and to the Open Group's ICSC > IT-API. > > Support of these goals can be enhanced by a relatively minor change to > both the SCTP adaptation and in the definition of the MPA Mode Frame. > > With this change, ULPs would be able to engage in the same message > exchanges prior to having a fully operational "connection" between two > RDMA-enabled endpoints, whether using RDMAP/DDP over MPA/TCP, > over SCTP > or even a non-IP transport such as InfiniBand. > > This type of uniformity will enable more rapid development > of RDMA-optimized ULPs. > > > The first issue to be addressed is the need for > pre-connection data exchange. > > An RDMA Endpoint (referred to as a Queue Pair, or QP, in > InfiniBand and > the draft-hilland proposed iWARP verbs) must be associated with a > Protection Domain. No receive operations may be posted to the endpoint > before it is associated with a Protection Domain. Indeed > under both the > InfiniBand and proposed iWARP verbs an endpoint/QP is created within a > Protection Domain. > > There are some applications where the choice of Protection Domain is > dependent upon the identity of the remote ULP client. For > example, if a > user session requires multiple connections, it is highly > desireable for > all of those connections to use a single Protection Domain. > > InfiniBand, the DAT APIs and the IT-API all provide for > the active side ULP to provide Provide Data when requesting > a connection. This data is passed to the ULP to allow it > to determine whether to accept the connection, and if so > with which endpoint (and implicitly which Protection > Domain). > > The Private Data can also be used to ensure that both ends > of the connection have configured their RDMA endpoints compatibly on > such matters as the RDMA Read capacity. Further ULP-specific uses are > also presumed, such as establishing the identity of the client. > > Private Data is also allowed for when accepting the > connection, to allow completion of any negotiation on > RDMA resources and for other ULP reasons. > > > What is proposed is that iWARP standardize on allowing a > single ULP message to be sent in each direction in the > process of establishing an LLP Connection. > > For SCTP: the stream reset message would allow variable > length Private Data at the end of the chunk. The local > interface SHOULD > report this data to the ULP when reporting the stream's state > transition. > > An additional Adaptation Layer Control function code would > be defined to explicitly terminate, but not re-enable, > a stream. The Local interface SHOUD report this, and any included > Private Data, to the ULP when reporting the stream's state transition. > > For MPA/TCP: the MPA Mode Frame would be defined to be an > MPA Frame, which allowed the same variable length Private > Data at the end of the frame. The local interface SHOULD > report this data to the ULP when reporting the connections state > transition. > > For SCTP, a "connection" could be established as follows: > > - The passive side would listen on the destination port, > to indicate its readiness to accept an RDMAP/DDP mode > association. > - The active side ULP would request a connection from a > RDMA endpoint to a destination address and port. > - If there was no association already established to > that SCTP endpoint, the SCTP association would be > established as with normal SCTP associations, except > that the Adaptation indication would specifcy RDMAP/DDP. > - The active side would select an unused Stream, and > send a Stream Reset Data Chunk on that stream, including > any ULP supplied Private Data. > - The passive side listener would receive the reset chunk. > It would notify the ULP of the requested "connection", > including any passed Private Data. > - To accept the connection request, the ULP would bind > the SCTP stream to an RDMA endpoint, and provide > the Private Data Response. An Adaptatio Layer Control > data chunk with a Function Code of DDP Stream Reset would > be sent to re-initialize the reverse stream and accept > the "connection" request. > - To reject the connection request, the passive side > would send an Adaptation Layer Control data chunk > with a Function Code of DDP Stream Terminate, including > any ULP supplied Private Data. > > For MPA/TCP, a connection could be established in virtually > the same way: > > - The passive side would listen on the destination port, > to indicate its readiness to accept a connection. > - The active side ULP would request a connection from a > RDMA endpoint to a destination address and port. > - The active side would initiate a TCP connection to the > destination port. Acceptance/rejection of the connection > would proceed as per normal TCP connection establishment. > This would allow normal TCP gatekeepers, such as INETD > and TCPserver, to exercise their normal safeguard/logging > functions. > - Both sides would indicate their expectation to receive > an MPA Mode frame, and thereby enter MPA mode. > - The active side would send an MPA Mode Frame including > any ULP supplied Private Data. > - The passive side listener would receive the MPA Mode > Frame. Notice of this state transisiton would be reported > to the Consumer, including the supplied Private Data. > - To accept the connection request, the ULP would bind > the TCP connections to an RDMA endpoint, and provide > the Private Data Response. An MPA Mode Frame would > be sent to confirm that this was to be an MPA mode > connection, along with any ULP supplied Private data. > - To reject the connection request, the passive side > would send an incompatible MPA Mode Frame response, > including any ULP supplied Private Data. > > Note that under both protocols exchange of Private Data is strictly > optional. Neither the Adaptation Layer Control data chunk or the MPA > Mode Frame is required to contain Private Data if the ULP has > no desire > to send it. When the ULP has no desire to exchange Private > Data, the ALC > DDP Stream Reset or MPA Mode Frames may be sent immediately by both > ends. There is no need to wait for the active side "request" to arrive > first. > > The alternative to this approach is the conduct these > negotiations using > the LLP alone. SCTP, even in RDMAP/DDP adaptation mode, can still send > and receive user messages. A TCP connection can still send and receive > byte streams. > > This approach presents a challenge, however. The ULP using > TCP must consume the correct number of "negotiation" bytes > *before* transitioning to MPA and binding the TCP connection with an > RDMA endpoint. If the ULP would prefer a variable length, or even > optional, Private Data exchange this can become a very tricky process. > > More importantly it is a *different* process than the > exchanges required under SCTP or any other RDMA transport. > > I believe the benefits of allowing uniform "connection" establishment > across reliable IP transports justify the addition of variable length > Private Data fields to the DDP Stream Reset ALC data chunk > and MPA Mode > Frame. > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 12:48:32 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05994 for ; Thu, 5 Jun 2003 12:48:32 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55Gm6k20994 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 12:48:06 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55Gm6B20991 for ; Thu, 5 Jun 2003 12:48:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05986 for ; Thu, 5 Jun 2003 12:48:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19NxsR-00010u-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 12:46:11 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19NxsR-00010q-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 12:46:11 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55GlLB20964; Thu, 5 Jun 2003 12:47:21 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55GkRB20918 for ; Thu, 5 Jun 2003 12:46:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05916 for ; Thu, 5 Jun 2003 12:46:22 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Nxqq-0000zt-00 for rddp@ietf.org; Thu, 05 Jun 2003 12:44:32 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Nxqi-0000zK-00 for rddp@ietf.org; Thu, 05 Jun 2003 12:44:24 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 6A01E24510B; Thu, 5 Jun 2003 12:46:15 -0400 (EDT) Date: Thu, 5 Jun 2003 11:45:44 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: "Culley, Paul" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762A05103F@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/5/03, Culley, Paul wrote: >Regarding "private data"; > >DDP/MPA are currently expected to be enabled together, >after creating the QP (and associating it with a PD). The >method we expected to use to deal with private data was to >have DDP/MPA's ULP (perhaps a bit of code in the IT-API >interface for example) be in charge of starting the TCP >connections and then, after doing any negotiations needed >between endpoints (using streaming mode TCP data), create >the QP and transition the connection to DDP/MPA mode. > >The "private data" would be exchanged in TCP streaming >mode (probably using standard sockets calls under the >API). When the Application/API had any necessary stuff >completed (like figuring out the correct PD), the QP can >be created, buffers posted, and the TCP "connection >context" passed to the iWARP stack to start DDP/MPA mode >processing. The Verbs document you referred to supports >this type of startup. > >I think that using the TCP streaming mode prior to MPA >enablement is preferred to adding additional features to >MPA as you suggest. The difference in flow over your >connection establishment sequence would only be that MPA >is not involved until the "bind" step. > >The IT-API group would have to publish a small draft >describing the use of the streaming mode and expected data >format prior to iWARP startup, but this is little >different than other applications or APIs such as SDP etc. > Requiring Private Data negotiations to be done with the LLP forces the ULP to be aware of what the ULP is. With the change proposed, the ULP could be indifferent as to whether the LLP was SCTP or TCP. Do you agree this would be of value? Secondly, TCP is not well suited to this purpose. In all of the current proposals for pre-RDMA-mode negotiations, the two sides have to agree *precisely* on how many bytes of Private Data will be exchanged in each direction. ULPs that need this data *frequently* have variable information needs, or even *optional* private data. Including Private Data in the MPA Mode frame solves this problem very easily. An MPA frame already has a length. A frame to introduce MPA Mode is already called for. Further, there is no need to use this Private Data. Applications that wanted to use LLP specific methods of exchanging private data would be free to do so. This proposal *enables* transport neutral connection establishment. The only benefit I can identify to use of LLP-specific streaming mode to negotiate connections is that such a negotiation could last several rounds. However, neither DAT or the IT-API found multi-round negotiations to be useful. The InfiniBand CM protocol does not support multiple-rounds of ULP negotiation either. The bottom line goal of pre-RDMA negotiations is to enable the correct Protection Domain to be selected. This has two important characteristics: -- Frequenty there is no need to constrain the selection of the Protection Domain. Meaning that many connections require zero Private Data. Incorporating the Private Data as an optional variable length field supports zero length Private Data very inexspensively. Testing the TCP stream to see *if* there is an MPA mode frame present would be difficult. I will point out that the current verbs proposed in draft-hilland do not support *checking* to see if the TCP stream is ready to enter MPA mode. -- One message exchange is all that is required to determine any and all Protection Domain and QP/endpoint needs. Other session details can be negotiated after the QP/endpoints are linked. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 15:29:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13424 for ; Thu, 5 Jun 2003 15:29:31 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55JT2701908 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 15:29:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55JT2B01905 for ; Thu, 5 Jun 2003 15:29:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13404 for ; Thu, 5 Jun 2003 15:29:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O0OD-0002Zn-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 15:27:09 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19O0OC-0002Zk-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 15:27:08 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55JSBB01839; Thu, 5 Jun 2003 15:28:11 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55JR1B01794 for ; Thu, 5 Jun 2003 15:27:01 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13341 for ; Thu, 5 Jun 2003 15:27:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O0MG-0002ZH-00 for rddp@ietf.org; Thu, 05 Jun 2003 15:25:08 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19O0MF-0002Ys-00 for rddp@ietf.org; Thu, 05 Jun 2003 15:25:07 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id C9995EE10; Thu, 5 Jun 2003 14:26:25 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Thu, 5 Jun 2003 14:26:25 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Transport Neutral "Connections" Date: Thu, 5 Jun 2003 14:26:24 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762A051058@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMrgf4JNqOCoTnKQL2i93GLh2geXgAEAG9A From: "Culley, Paul" To: "Caitlin Bestler" Cc: X-OriginalArrivalTime: 05 Jun 2003 19:26:25.0376 (UTC) FILETIME=[5ABB6600:01C32B98] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h55JR1B01795 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Embedded... Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Thursday, June 05, 2003 11:46 AM To: Culley, Paul Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Requiring Private Data negotiations to be done with the LLP forces the ULP to be aware of what the ULP is. With the change proposed, the ULP could be indifferent as to whether the LLP was SCTP or TCP. Do you agree this would be of value? I agree that the ultimate ULP (the Application) should not see any difference; that is the point of a good API (to hide differences in HW or SW). However, I believe that OS's generally use some "shim" code to mask differences in the various interfaces that are below it. I doubt very much that we can (or should) hide all of the differences between iWARP and InfiniBand (for instance) in the MPA layer, and am proposing that we should not even try to hide all of the differences between TCP and SCTP, at least not in MPA. I could see someone creating a rather simple ietf draft doing what you describe using TCP streaming, before converting to iWARP mode, if the IT-API guys don't want to write it. Secondly, TCP is not well suited to this purpose. In all of the current proposals for pre-RDMA-mode negotiations, the two sides have to agree *precisely* on how many bytes of Private Data will be exchanged in each direction. ULPs that need this data *frequently* have variable information needs, or even *optional* private data. Including Private Data in the MPA Mode frame solves this problem very easily. An MPA frame already has a length. A frame to introduce MPA Mode is already called for. The mode frame I am currently thinking about is simply declarative; it tells MPA how to operate (or whether to close the connection). What you describe requires that the ULP also get involved between the reception of the MPA mode frame, and the full activation of DDP/MPA. The ULP would have to see the frame's private data (if any), decide what to do, and then somehow migrate the "partial MPA" mode to a full iWARP connection and QP. It creates a new mode of MPA (not really related to full iWARP mode) just for this purpose. Further, there is no need to use this Private Data. Applications that wanted to use LLP specific methods of exchanging private data would be free to do so. This proposal *enables* transport neutral connection establishment. agree with the first part; but must point out that we will then have to deal with three modes (Streaming mode at start, "pre MPA", and full iWARP) and the fun of describing the methods of switching between these modes when apps will be using some combination of them. I prefer that for iWARP, we stick with just two modes (streaming, and iWARP) and let another draft describe the startup needed with the IT-API. The only benefit I can identify to use of LLP-specific streaming mode to negotiate connections is that such a negotiation could last several rounds. However, neither DAT or the IT-API found multi-round negotiations to be useful. The InfiniBand CM protocol does not support multiple-rounds of ULP negotiation either. There is at least one application in the works that does use a multi-round negotiation prior to iWARP startup... Of course it is written to use TCP streaming mode on a legacy port for this. The bottom line goal of pre-RDMA negotiations is to enable the correct Protection Domain to be selected. This has two important characteristics: -- Frequenty there is no need to constrain the selection of the Protection Domain. Meaning that many connections require zero Private Data. Incorporating the Private Data as an optional variable length field supports zero length Private Data very inexspensively. Testing the TCP stream to see *if* there is an MPA mode frame present would be difficult. I will point out that the current verbs proposed in draft-hilland do not support *checking* to see if the TCP stream is ready to enter MPA mode. The current verbs leaves it to the ULP to perform such checks; for some applications a streaming mode setup or negotiation is used, for others outside knowledge (port number, name service etc.) will be used to determine that DDP/MPA is to be used. I mostly agree with what you are saying, I just don't want to add this functionality to MPA; I believe it belongs in one of MPA's ULPs, for the APIs you describe that would be a thin shim between the Application and the Verbs. -- One message exchange is all that is required to determine any and all Protection Domain and QP/endpoint needs. Other session details can be negotiated after the QP/endpoints are linked. Caitlin Bestler http://asomi.com/CaitlinBestler/ On 6/5/03, Culley, Paul wrote: >Regarding "private data"; > >DDP/MPA are currently expected to be enabled together, >after creating the QP (and associating it with a PD). The method we >expected to use to deal with private data was to have DDP/MPA's ULP >(perhaps a bit of code in the IT-API interface for example) be in >charge of starting the TCP connections and then, after doing any >negotiations needed between endpoints (using streaming mode TCP data), >create the QP and transition the connection to DDP/MPA mode. > >The "private data" would be exchanged in TCP streaming >mode (probably using standard sockets calls under the >API). When the Application/API had any necessary stuff completed (like >figuring out the correct PD), the QP can be created, buffers posted, >and the TCP "connection context" passed to the iWARP stack to start >DDP/MPA mode processing. The Verbs document you referred to supports >this type of startup. > >I think that using the TCP streaming mode prior to MPA enablement is >preferred to adding additional features to MPA as you suggest. The >difference in flow over your connection establishment sequence would >only be that MPA is not involved until the "bind" step. > >The IT-API group would have to publish a small draft describing the use >of the streaming mode and expected data format prior to iWARP startup, >but this is little different than other applications or APIs such as >SDP etc. > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 19:43:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA22471 for ; Thu, 5 Jun 2003 19:43:22 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h55NgtR22307 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 19:42:55 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55NgtB22304 for ; Thu, 5 Jun 2003 19:42:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA22465 for ; Thu, 5 Jun 2003 19:42:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O4Ls-0004eV-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 19:41:00 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19O4Ls-0004eS-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 19:41:00 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55Ng9B22199; Thu, 5 Jun 2003 19:42:09 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h55NfWB22155 for ; Thu, 5 Jun 2003 19:41:32 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA22432 for ; Thu, 5 Jun 2003 19:41:28 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O4KX-0004e6-00 for rddp@ietf.org; Thu, 05 Jun 2003 19:39:37 -0400 Received: from magic-mail.adaptec.com ([208.236.45.100] helo=magic.adaptec.com) by ietf-mx with esmtp (Exim 4.12) id 19O4KW-0004dx-00 for rddp@ietf.org; Thu, 05 Jun 2003 19:39:36 -0400 Received: from kingdom.adaptec.com (kingdom.adaptec.com [162.62.50.10]) by magic.adaptec.com (8.11.6/8.11.6) with ESMTP id h55NZkP17971; Thu, 5 Jun 2003 16:35:47 -0700 Received: from aimexc07.adaptec.com (aimexc07.adaptec.com [162.62.62.47]) by kingdom.adaptec.com (8.11.7+Sun/8.11.7) with ESMTP id h55NerD08313; Thu, 5 Jun 2003 16:40:54 -0700 (PDT) Received: by aimexc07.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Thu, 5 Jun 2003 16:40:53 -0700 Message-ID: From: "Carrier, John" To: Caitlin Bestler Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Date: Thu, 5 Jun 2003 16:40:52 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Hi Caitlin, I do see the value of what you are suggesting, but don't see how the ULP can be involved with MPA directly. A mode frame can easily be inserted as the first MPA frame in an RDMA connection, but for the options discussed on this list so far, the information exchanged is strictly between MPA peers. I don't see any way for MPA to pass any data up to the ULP directly. As Paul points out, in the current RDMAP/DDP model there is streaming mode and then iWARP mode. What you are suggesting is a third, intermediate mode. The problem is that MPA talks directly to DDP and there is no provision for a ULP to receive data from DDP unless the ULP has pre-posted a receive buffer. Pre-posting a receive buffer implies that the RDMA endpoint has already been created for the ULP. InfiniBand sends its private data out-of-band via CM messages before the RDMA connection is established. Since there isn't a similar out-of-band mechanism for iWARP, I think the private data exchange should take place in-band using the LLP connection in streaming mode, before the transition to iWARP mode, regardless of the whether the LLP is SCTP or TCP. --jc > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Thursday, June 05, 2003 9:46 AM > To: Culley, Paul > Cc: rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > On 6/5/03, Culley, Paul wrote: > > >Regarding "private data"; > > > >DDP/MPA are currently expected to be enabled together, > >after creating the QP (and associating it with a PD). The > >method we expected to use to deal with private data was to > >have DDP/MPA's ULP (perhaps a bit of code in the IT-API > >interface for example) be in charge of starting the TCP > >connections and then, after doing any negotiations needed > >between endpoints (using streaming mode TCP data), create > >the QP and transition the connection to DDP/MPA mode. > > > >The "private data" would be exchanged in TCP streaming > >mode (probably using standard sockets calls under the > >API). When the Application/API had any necessary stuff > >completed (like figuring out the correct PD), the QP can > >be created, buffers posted, and the TCP "connection > >context" passed to the iWARP stack to start DDP/MPA mode > >processing. The Verbs document you referred to supports > >this type of startup. > > > >I think that using the TCP streaming mode prior to MPA > >enablement is preferred to adding additional features to > >MPA as you suggest. The difference in flow over your > >connection establishment sequence would only be that MPA > >is not involved until the "bind" step. > > > >The IT-API group would have to publish a small draft > >describing the use of the streaming mode and expected data > >format prior to iWARP startup, but this is little > >different than other applications or APIs such as SDP etc. > > > > Requiring Private Data negotiations to be done with the LLP > forces the ULP to be aware of what the ULP is. With the > change proposed, the ULP could be indifferent as to whether > the LLP was SCTP or TCP. > > Do you agree this would be of value? > > Secondly, TCP is not well suited to this purpose. In all of > the current proposals for pre-RDMA-mode negotiations, the > two sides have to agree *precisely* on how many bytes of > Private Data will be exchanged in each direction. > > ULPs that need this data *frequently* have variable > information needs, or even *optional* private data. > > Including Private Data in the MPA Mode frame solves this > problem very easily. An MPA frame already has a length. > A frame to introduce MPA Mode is already called for. > > Further, there is no need to use this Private Data. > Applications that wanted to use LLP specific methods > of exchanging private data would be free to do so. > This proposal *enables* transport neutral connection > establishment. > > The only benefit I can identify to use of LLP-specific > streaming mode to negotiate connections is that such > a negotiation could last several rounds. > > However, neither DAT or the IT-API found multi-round > negotiations to be useful. The InfiniBand CM protocol > does not support multiple-rounds of ULP negotiation either. > > The bottom line goal of pre-RDMA negotiations is to enable > the correct Protection Domain to be selected. This has > two important characteristics: > > -- Frequenty there is no need to constrain the selection > of the Protection Domain. Meaning that many connections > require zero Private Data. Incorporating the Private Data > as an optional variable length field supports zero length > Private Data very inexspensively. Testing the TCP stream > to see *if* there is an MPA mode frame present would be > difficult. I will point out that the current verbs > proposed in draft-hilland do not support *checking* to > see if the TCP stream is ready to enter MPA mode. > > -- One message exchange is all that is required to determine > any and all Protection Domain and QP/endpoint needs. > Other session details can be negotiated after the > QP/endpoints are linked. > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 22:14:18 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25449 for ; Thu, 5 Jun 2003 22:14:18 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h562Drk31990 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 22:13:53 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562DrB31987 for ; Thu, 5 Jun 2003 22:13:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25430 for ; Thu, 5 Jun 2003 22:13:48 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O6hv-0005PD-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 22:11:55 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19O6hv-0005PA-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 22:11:55 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562D9B31968; Thu, 5 Jun 2003 22:13:09 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562CeB31950 for ; Thu, 5 Jun 2003 22:12:40 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25409 for ; Thu, 5 Jun 2003 22:12:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O6gl-0005Op-00 for rddp@ietf.org; Thu, 05 Jun 2003 22:10:43 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19O6gk-0005Om-00 for rddp@ietf.org; Thu, 05 Jun 2003 22:10:42 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id EFD21245DA6; Thu, 5 Jun 2003 22:12:33 -0400 (EDT) Date: Thu, 5 Jun 2003 21:12:03 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: "Culley, Paul" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762A051058@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/5/03, Culley, Paul wrote: > >Requiring Private Data negotiations to be done with the >LLP forces the ULP to be aware of what the ULP is. With >the change proposed, the ULP could be indifferent as to >whether the LLP was SCTP or TCP. > >Do you agree this would be of value? > I agree that the ultimate ULP (the Application) >should not see any difference; that is the point of a good >API (to hide differences in HW or SW). However, I believe >that OS's generally use some "shim" code to mask >differences in the various interfaces that are below it. >I doubt very much that we can (or should) hide all of the >differences between iWARP and InfiniBand (for instance) in >the MPA layer, and am proposing that we should not even >try to hide all of the differences between TCP and SCTP, >at least not in MPA. I could see someone creating a >rather simple ietf draft doing what you describe using TCP >streaming, before converting to iWARP mode, if the IT-API >guys don't want to write it. > A middleware API cannot "hide the difference" without either creating fixed size Private Data messages (losing flexibility), defining a protocol to define the length (and running the risk that each ULP will adopt a different protocol), or dynamically involve the ULP. The natural, and existing, API is to exchange one message in each direction. The fuzziness of "message" under TCP is the core of the problem. >Secondly, TCP is not well suited to this purpose. In all >of the current proposals for pre-RDMA-mode negotiations, >the two sides have to agree *precisely* on how many bytes >of Private Data will be exchanged in each direction. > >ULPs that need this data *frequently* have variable >information needs, or even *optional* private data. > >Including Private Data in the MPA Mode frame solves this >problem very easily. An MPA frame already has a length. A >frame to introduce MPA Mode is already called for. > The mode frame I am currently thinking about is >simply declarative; it tells MPA how to operate (or >whether to close the connection). What you describe >requires that the ULP also get involved between the >reception of the MPA mode frame, and the full activation >of DDP/MPA. The ULP would have to see the frame's private >data (if any), decide what to do, and then somehow migrate >the "partial MPA" mode to a full iWARP connection and QP. >It creates a new mode of MPA (not really related to full >iWARP mode) just for this purpose. > It would stll be a declarative frame. The steps do not change that much either. The passive side does the following: 1) Listen for a TCP connection on the port. 2) When the socket is delivered, enable MPA receive. 3) Wait for a transition/event indicating that an MPA mode frame has been received (or an error if it was not, but then you teardown and quit). This notice/event includes any Private Data that had been within the MPA Mode frame. 4) Select/configure the endpoint/QP, supplying any response Private Data. This fully enters MPA mode and binds the TCP connection to the endpoint/QP. The MPA Mode Frame is sent with the Private Data. The active side is even simpler: 1) Initiate the TCP connection. 2) Once the TCP connection is established, bind it to endpoint/QP supplying the Private Data. The MPA Mode Frame is sent with any supplied Private Data. 3) Wait for the MPA Mode Frame to confirm. Report any Private Data received to the ULP. The currently proposed iWARP verbs already describe two stage entry to MPA mode. If a ULP does not require Private Data, then each side can simply send their MPA Mode Frames and their off and running. There is no penalty for allowing for Private Data. This proposal also deals with *optional* Private Data naturally. This is not something that can be dealt with using pre-MPA mode TCP exchanges. Actual bytes must be sent to say "sorry no Private Data supplied" or the ULP might accidentally consume the MPA Mode frame in normal streaming mode. Remember that stack cannot prevent the ULP from consuming the MPA Mode Frame in streaming mode. There is no way to know that this is not valid streaming mode data. > >However, neither DAT or the IT-API found multi-round >negotiations to be useful. The InfiniBand CM protocol does >not support multiple-rounds of ULP negotiation either. > There is at least one application in the works that >does use a multi-round negotiation prior to iWARP >startup... Of course it is written to use TCP streaming >mode on a legacy port for this. Which is *exactly* what we need to avoid. An iWARP application SHOULD NOT be aware of which LLP is in use. The need to configure an endpoint/QP properly is intrinsic to iWARP. iWARP should provide the means to negotiate it. Note that there are *NOT* three states for the TCP connection. There is a pre-MPA mode, and a post-MPA mode. In both cases a single MPA frame is the delimiter. The only question is whether Private Data that is directly related to enabling RDMA mode communication is communicated within that delimiting packet, or is communicated prior to MPA mode in a way that undermines LLP transparency. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 5 22:19:08 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25587 for ; Thu, 5 Jun 2003 22:19:08 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h562Ih932281 for rddp-archive@odin.ietf.org; Thu, 5 Jun 2003 22:18:43 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562IgB32278 for ; Thu, 5 Jun 2003 22:18:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25580 for ; Thu, 5 Jun 2003 22:18:37 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O6mb-0005R3-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 22:16:45 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19O6ma-0005R0-00 for rddp-web-archive@ietf.org; Thu, 05 Jun 2003 22:16:44 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562I1B32244; Thu, 5 Jun 2003 22:18:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h562HZB32214 for ; Thu, 5 Jun 2003 22:17:35 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA25547 for ; Thu, 5 Jun 2003 22:17:29 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19O6lV-0005Qn-00 for rddp@ietf.org; Thu, 05 Jun 2003 22:15:37 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19O6lV-0005QU-00 for rddp@ietf.org; Thu, 05 Jun 2003 22:15:37 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 12845244813; Thu, 5 Jun 2003 22:17:26 -0400 (EDT) Date: Thu, 5 Jun 2003 21:16:56 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: "Carrier, John" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/5/03, Carrier, John wrote: >As Paul points out, in the current RDMAP/DDP model there >is streaming mode and then iWARP mode. What you are >suggesting is a third, intermediate mode. The problem is >that MPA talks directly to DDP and there is no provision >for a ULP to receive data from DDP unless the ULP has >pre-posted a receive buffer. Pre-posting a receive buffer >implies that the RDMA endpoint has already been created >for the ULP. The ULP has authorizes entry to MPA mode and tells the MPA/stack to expect a confirming MPA Mode frame. A buffer can easily be provided at that time. With or without Private Data, any TCP byte stream that converts to MPA has a series of bytes before MPA mode, an MPA Mode Frame, and then an MPA mode series of bytes. That is not changed by the inclusion of Private Data within the MPA Mode Frame. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 10:59:23 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA27270 for ; Fri, 6 Jun 2003 10:59:23 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56Ex0k30437 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 10:59:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56Ex0B30434 for ; Fri, 6 Jun 2003 10:59:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA27238 for ; Fri, 6 Jun 2003 10:58:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OIeK-0002nA-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 10:57:00 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OIeK-0002n7-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 10:57:00 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56EuIB30264; Fri, 6 Jun 2003 10:56:18 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56ErWB30072 for ; Fri, 6 Jun 2003 10:53:32 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26904 for ; Fri, 6 Jun 2003 10:53:26 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OIZ3-0002k2-00 for rddp@ietf.org; Fri, 06 Jun 2003 10:51:33 -0400 Received: from fmr02.intel.com ([192.55.52.25] helo=caduceus.fm.intel.com) by ietf-mx with esmtp (Exim 4.12) id 19OIZ2-0002jy-00 for rddp@ietf.org; Fri, 06 Jun 2003 10:51:32 -0400 Received: from talaria.fm.intel.com (talaria.fm.intel.com [10.1.192.39]) by caduceus.fm.intel.com (8.11.6p2/8.11.6/d: outer.mc,v 1.66 2003/05/22 21:17:36 rfjohns1 Exp $) with ESMTP id h56EjVZ26075 for ; Fri, 6 Jun 2003 14:45:31 GMT Received: from fmsmsxvs043.fm.intel.com (fmsmsxvs043.fm.intel.com [132.233.42.129]) by talaria.fm.intel.com (8.11.6p2/8.11.6/d: inner.mc,v 1.35 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h56Es3M10302 for ; Fri, 6 Jun 2003 14:54:03 GMT Received: from fmsmsx331.amr.corp.intel.com ([132.233.42.135]) by fmsmsxvs043.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2003060607505226741 ; Fri, 06 Jun 2003 07:50:52 -0700 Received: from fmsmsx402.amr.corp.intel.com ([132.233.42.200]) by fmsmsx331.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Fri, 6 Jun 2003 07:53:23 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Subject: RE: [rddp] Transport Neutral "Connections" Date: Fri, 6 Jun 2003 07:53:23 -0700 Message-ID: <6315617889C99D4BA7C14687DEC8DB4E03D6D7C1@fmsmsx402.fm.intel.com> Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMr0/ISfr+/D9NZRhy7HigsHO0UjQAZSAng From: "Shah, Hemal" To: "Caitlin Bestler" , "Carrier, John" Cc: X-OriginalArrivalTime: 06 Jun 2003 14:53:23.0383 (UTC) FILETIME=[60B76470:01C32C3B] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h56ErWB30073 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit I think CRC on/off should be an administrative issue. I do not think we need to add wire protocol behavior for it. I do not think we need to define MPA mode frame negotiation for it. An MPA implementation MAY choose to provide the disabling of the MPA CRC verification for the incoming FPDUs and/or the MPA CRC computation for the outgoing FPDUs by the means of the local interface or configuration. If an MPA implementation does not provide this capability, then the MPA CRC computation/verification will always be performed by the MPA layer. In any case, the MPA CRC field is always present in the FPDU. I believe the above is a reasonable compromise for both the hardware and software implementations. I strongly prefer that we have to deal with only two modes: Stream mode and iWARP mode. How the Private Data is exchanged and CRC on/off capability is communicated is out of the scope of MPA specification. Hemal -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Thursday, June 05, 2003 9:17 PM To: Carrier, John Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" On 6/5/03, Carrier, John wrote: >As Paul points out, in the current RDMAP/DDP model there >is streaming mode and then iWARP mode. What you are >suggesting is a third, intermediate mode. The problem is >that MPA talks directly to DDP and there is no provision >for a ULP to receive data from DDP unless the ULP has >pre-posted a receive buffer. Pre-posting a receive buffer >implies that the RDMA endpoint has already been created >for the ULP. The ULP has authorizes entry to MPA mode and tells the MPA/stack to expect a confirming MPA Mode frame. A buffer can easily be provided at that time. With or without Private Data, any TCP byte stream that converts to MPA has a series of bytes before MPA mode, an MPA Mode Frame, and then an MPA mode series of bytes. That is not changed by the inclusion of Private Data within the MPA Mode Frame. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 11:17:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28101 for ; Fri, 6 Jun 2003 11:17:35 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56FH7932560 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 11:17:07 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FH7B32557 for ; Fri, 6 Jun 2003 11:17:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28065 for ; Fri, 6 Jun 2003 11:17:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OIvw-0002zw-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 11:15:12 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OIvv-0002zt-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 11:15:11 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FEJB32407; Fri, 6 Jun 2003 11:14:19 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FDBB32335 for ; Fri, 6 Jun 2003 11:13:11 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA27944 for ; Fri, 6 Jun 2003 11:13:09 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OIs8-0002xk-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:11:16 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19OIs8-0002xY-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:11:16 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id A101E244D23; Fri, 6 Jun 2003 11:13:10 -0400 (EDT) Date: Fri, 6 Jun 2003 10:12:39 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: "Shah, Hemal" Cc: "Carrier, John" , rddp@ietf.org X-Priority: 3 In-Reply-To: <6315617889C99D4BA7C14687DEC8DB4E03D6D7C1@fmsmsx402.fm.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/6/03, Shah, Hemal wrote: >I strongly prefer that we have to deal with only two >modes: Stream mode and iWARP mode. How the Private Data is >exchanged and CRC on/off capability is communicated is out >of the scope of MPA specification. So you are arguing that you cannot enter MPA mode without fully associating it with an RDMAP/DDP endpoint? Wouldn't that mean that MPA could *only* be used with RDMAP/DDP? The idea of shortening the CRC to cover only the header, as opposed to omitting it, was rejected because we considered it inappropriate for MPA to know how long the RDMAP/DDP Header was. But why wouldn't we if the only way to enable MPA was to associate it with an RDMAP/DDP endpoint? There is obviously a desire to preserve the draft RDMAC iWARP verbs. But at least as far as that draft deals with enabling an LLP Stream, it has failed to deal with SCTP and fails to cleanly layer DDP over the LLP. The proposal to allow Private Data in the MPA Frame mode was an attempt to *minimize* changes to those verbs, but merely requiring that MPA be supported *before* enabling RDMAP/DDP would also solve the problem. Either solution would work. I do not think it is appropriate to define MPA in a way which a) forces iWARP applications to directly use the LLP themselves for iWARP related functions, and b) assumes the local interface will horrendously rototill its *interface* to mandate that MPA, DDP and RDMAP all be enabled at the same time with a single user action. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 11:54:43 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29550 for ; Fri, 6 Jun 2003 11:54:43 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56FsG103409 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 11:54:16 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FsGB03402 for ; Fri, 6 Jun 2003 11:54:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29490 for ; Fri, 6 Jun 2003 11:54:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJVs-0003Nw-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 11:52:20 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OJVr-0003Nt-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 11:52:19 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FrVB03216; Fri, 6 Jun 2003 11:53:31 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FkOB02867 for ; Fri, 6 Jun 2003 11:46:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29153 for ; Fri, 6 Jun 2003 11:46:21 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJOG-0003Ho-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:44:28 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19OJOF-0003HZ-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:44:27 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 89CEEBBA4; Fri, 6 Jun 2003 10:45:49 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Fri, 6 Jun 2003 10:45:46 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Transport Neutral "Connections" Date: Fri, 6 Jun 2003 10:45:46 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762A051087@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMr0WBs1vksJeIxT+ubT+S2IiF1SQAYQ6FQ From: "Culley, Paul" To: "Caitlin Bestler" Cc: X-OriginalArrivalTime: 06 Jun 2003 15:45:46.0621 (UTC) FILETIME=[B23B72D0:01C32C42] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h56FkOB02868 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Below... Paul R. Culley HP Senior Fellow 281-514-5543 > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Thursday, June 05, 2003 9:12 PM > To: Culley, Paul > Cc: rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > On 6/5/03, Culley, Paul wrote: > > > > >Requiring Private Data negotiations to be done with the > >LLP forces the ULP to be aware of what the ULP is. With > >the change proposed, the ULP could be indifferent as to > whether the LLP > >was SCTP or TCP. > > > >Do you agree this would be of value? > > I agree that the ultimate ULP (the Application) > >should not see any difference; that is the point of a good > >API (to hide differences in HW or SW). However, I believe that OS's > >generally use some "shim" code to mask differences in the various > >interfaces that are below it. I doubt very much that we can > (or should) > >hide all of the differences between iWARP and InfiniBand > (for instance) > >in the MPA layer, and am proposing that we should not even > >try to hide all of the differences between TCP and SCTP, > >at least not in MPA. I could see someone creating a > >rather simple ietf draft doing what you describe using TCP > >streaming, before converting to iWARP mode, if the IT-API > >guys don't want to write it. > > > A middleware API cannot "hide the difference" without > either creating fixed size Private Data messages (losing > flexibility), defining a protocol to define the length (and > running the risk that each ULP will adopt a different > protocol), or dynamically involve the ULP. If the API (or another draft standard) "defines a protocol to define the length" then each ULP that uses that API will use the same protocol. Those that don't use the API may also use the standard (or not as desired). I just believe that MPA is not the right place to make that standard, because it adds functionality to MPA that is not always needed and adds extra stuff to the proposed "verbs". > > The natural, and existing, API is to exchange one message > in each direction. The fuzziness of "message" under TCP > is the core of the problem. > > > > >Secondly, TCP is not well suited to this purpose. In all > >of the current proposals for pre-RDMA-mode negotiations, > >the two sides have to agree *precisely* on how many bytes > >of Private Data will be exchanged in each direction. > > > >ULPs that need this data *frequently* have variable > information needs, > >or even *optional* private data. > > > >Including Private Data in the MPA Mode frame solves this > problem very > >easily. An MPA frame already has a length. A frame to introduce MPA > >Mode is already called for. > > > The mode frame I am currently thinking about is > >simply declarative; it tells MPA how to operate (or > >whether to close the connection). What you describe > >requires that the ULP also get involved between the > >reception of the MPA mode frame, and the full activation > >of DDP/MPA. The ULP would have to see the frame's private data (if > >any), decide what to do, and then somehow migrate the "partial MPA" > >mode to a full iWARP connection and QP. It creates a new mode of MPA > >(not really related to full iWARP mode) just for this purpose. > > > It would stll be a declarative frame. > > The steps do not change that much either. > > The passive side does the following: > > 1) Listen for a TCP connection on the port. > 2) When the socket is delivered, enable MPA receive. > 3) Wait for a transition/event indicating that an > MPA mode frame has been received (or an error if > it was not, but then you teardown and quit). > This notice/event includes any Private Data that > had been within the MPA Mode frame. > 4) Select/configure the endpoint/QP, supplying any > response Private Data. This fully enters MPA mode > and binds the TCP connection to the endpoint/QP. > The MPA Mode Frame is sent with the Private Data. > > The active side is even simpler: > > 1) Initiate the TCP connection. > 2) Once the TCP connection is established, bind it > to endpoint/QP supplying the Private Data. The > MPA Mode Frame is sent with any supplied Private Data. > 3) Wait for the MPA Mode Frame to confirm. Report any > Private Data received to the ULP. Does this mean that the active side cannot send its first messages until the private data is returned? Or that the active side does not actually need the private data to send the first message? In the former case, you are mandating a full two way exchange (a round trip) prior to the active side's first iWARP message. The passive side can send after receiving the mode frame. This is probably a new topic worthy of separate discussion... > > The currently proposed iWARP verbs already describe two > stage entry to MPA mode. > > If a ULP does not require Private Data, then each side can > simply send their MPA Mode Frames and their off and running. > There is no penalty for allowing for Private Data. > > This proposal also deals with *optional* Private Data > naturally. This is not something that can be dealt with using > pre-MPA mode TCP exchanges. Actual bytes must be sent to say > "sorry no Private Data supplied" or the ULP might > accidentally consume the MPA Mode frame in normal streaming mode. I disagree here; it does not matter if MPA describes how we deal with the private data length, or some other draft that moves bytes over streaming mode. Keep in mind that I like your basic idea, I just don't think it needs to be part of MPA... > > Remember that stack cannot prevent the ULP from consuming > the MPA Mode Frame in streaming mode. There is no way to > know that this is not valid streaming mode data. This is a problem in all cases where any streaming data is allowed prior to entering iWARP mode; that is why the verbs specification is so careful to describe the one way that MUST be supported by an interface. > > > > > > >However, neither DAT or the IT-API found multi-round > negotiations to be > >useful. The InfiniBand CM protocol does not support > multiple-rounds of > >ULP negotiation either. > > > There is at least one application in the works that > >does use a multi-round negotiation prior to iWARP > >startup... Of course it is written to use TCP streaming > >mode on a legacy port for this. > > Which is *exactly* what we need to avoid. An iWARP > application SHOULD NOT be aware of which LLP is in use. > > The need to configure an endpoint/QP properly is intrinsic > to iWARP. iWARP should provide the means to negotiate it. It does; a streaming mode, prior to iWARP, where you can do anything... > > Note that there are *NOT* three states for the TCP > connection. There is a pre-MPA mode, and a post-MPA mode. > In both cases a single MPA frame is the delimiter. The > only question is whether Private Data that is directly > related to enabling RDMA mode communication is communicated > within that delimiting packet, or is communicated prior to > MPA mode in a way that undermines LLP transparency. Semantics... The iWARP stack would have to have three modes; pre-MPA, MPA-Frame with private data reception, and full iWARP. > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 12:04:56 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00128 for ; Fri, 6 Jun 2003 12:04:55 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56G4TP04193 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 12:04:29 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56G4TB04190 for ; Fri, 6 Jun 2003 12:04:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00099 for ; Fri, 6 Jun 2003 12:04:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJfk-0003Vj-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 12:02:32 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OJfk-0003Vd-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 12:02:32 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56G3cB04104; Fri, 6 Jun 2003 12:03:38 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56FxwB03779 for ; Fri, 6 Jun 2003 11:59:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29797 for ; Fri, 6 Jun 2003 11:59:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJbN-0003SL-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:58:02 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19OJbN-0003RS-00 for rddp@ietf.org; Fri, 06 Jun 2003 11:58:01 -0400 Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 0BA41EEFF; Fri, 6 Jun 2003 10:59:25 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Fri, 6 Jun 2003 10:59:24 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Transport Neutral "Connections" Date: Fri, 6 Jun 2003 10:59:23 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762A05108B@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Transport Neutral "Connections" Thread-Index: AcMsP88bufgUXaakTwqTxoHItxByMAAAubIg From: "Culley, Paul" To: "Caitlin Bestler" , "Shah, Hemal" Cc: "Carrier, John" , X-OriginalArrivalTime: 06 Jun 2003 15:59:24.0569 (UTC) FILETIME=[99C46890:01C32C44] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h56FxwB03781 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Below... Paul R. Culley HP Senior Fellow 281-514-5543 > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Friday, June 06, 2003 10:13 AM > To: Shah, Hemal > Cc: Carrier, John; rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > On 6/6/03, Shah, Hemal wrote: > > >I strongly prefer that we have to deal with only two > >modes: Stream mode and iWARP mode. How the Private Data is exchanged > >and CRC on/off capability is communicated is out of the scope of MPA > >specification. > > So you are arguing that you cannot enter MPA mode without > fully associating it with an RDMAP/DDP endpoint? > > Wouldn't that mean that MPA could *only* be used with RDMAP/DDP? That is the way that the current MPA draft is written... > > The idea of shortening the CRC to cover only the header, > as opposed to omitting it, was rejected because we > considered it inappropriate for MPA to know how long > the RDMAP/DDP Header was. > > But why wouldn't we if the only way to enable MPA > was to associate it with an RDMAP/DDP endpoint? Quite another topic... > > There is obviously a desire to preserve the draft > RDMAC iWARP verbs. But at least as far as that draft > deals with enabling an LLP Stream, it has failed to > deal with SCTP and fails to cleanly layer DDP over > the LLP. We freely admit that SCTP interface has not been thoroughly thought out, mostly because there were no SCTP experts available to author those sections... > > The proposal to allow Private Data in the MPA Frame > mode was an attempt to *minimize* changes to those verbs, > but merely requiring that MPA be supported *before* > enabling RDMAP/DDP would also solve the problem. > > Either solution would work. I agree... > > I do not think it is appropriate to define MPA in a way > which a) forces iWARP applications to directly use the > LLP themselves for iWARP related functions, and > b) assumes the local interface will horrendously > rototill its *interface* to mandate that MPA, DDP > and RDMAP all be enabled at the same time with a > single user action. A) I don't expect that 'applications' using standard startup APIs that expect to use private data will have a problem; the API (and some other draft standard) will deal with it. B) For what it is worth, the vendors that helped in authoring the various specs seemed to prefer the "big switch" approach to starting iWARP; the idea of starting each layer at a different time was just not very interesting. > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 12:14:19 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00708 for ; Fri, 6 Jun 2003 12:14:18 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56GDqA06268 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 12:13:52 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56GDqB06265 for ; Fri, 6 Jun 2003 12:13:52 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00688 for ; Fri, 6 Jun 2003 12:13:48 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJoq-0003g8-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 12:11:56 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OJop-0003g2-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 12:11:55 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56GD1B06081; Fri, 6 Jun 2003 12:13:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56G9vB05562 for ; Fri, 6 Jun 2003 12:09:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00333 for ; Fri, 6 Jun 2003 12:09:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OJl3-0003Z8-00 for rddp@ietf.org; Fri, 06 Jun 2003 12:08:01 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19OJl2-0003Yw-00 for rddp@ietf.org; Fri, 06 Jun 2003 12:08:00 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4EE7F244AB2; Fri, 6 Jun 2003 12:09:55 -0400 (EDT) Date: Fri, 6 Jun 2003 11:09:23 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: "Culley, Paul" , rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762A051087@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/6/03, Culley, Paul wrote: >Does this mean that the active side cannot send its first >messages until the private data is returned? Or that the >active side does not actually need the private data to >send the first message? > That is the current definition for both the DAT and IT-API. This is of course based upon the need to be transport neutral. You cannot do your first send under InfiniBand until the CM response is recieved for the simple reason that you don't know the destination address. There is no reason for iWARP to impose that restriction however. Each side would be able to send immediately after sending their MPA Mode Frame if their ULP feels no need to receive the Private Data. I'm already working on how to evolve the DAT/IT-API model to improve the API interactions to be compatible with this iWARP capability while still working for InfiniBand. It will probably involve allowing the ULP to post writes in this stage, but not guaranteeing they will be sent before getting the response from the other side. [Actually, because of flow-control, there is never a guarantee on when things will be sent, so that will probably work fairly well.] >In the former case, you are mandating a full two way >exchange (a round trip) prior to the active side's first >iWARP message. The passive side can send after receiving >the mode frame. This is probably a new topic worthy of >separate discussion... Agreed. Applications that do not require a two-way Private Data exchange should not be forced into doing so merely because InfiniBand required a two-way handshake. As an objective, the recomended iWARP connection establishment procedures should work for ULPs that: -- require NO exchange of Private Data. -- require Private Data only from the Active side. -- require an exchange of Private Data. >I disagree here; it does not matter if MPA describes how >we deal with the private data length, or some other draft >that moves bytes over streaming mode. Keep in mind that I >like your basic idea, I just don't think it needs to be >part of MPA... Then where else would you recomend solving it? An additional mapping protocol for "Connection Establishment Messages" to be mapped to both SCTP and TCP? Wouldn't including a Private Data field in existing messages be much simpler? >> Remember that stack cannot prevent the ULP from consuming >> the MPA Mode Frame in streaming mode. There is no way to >> know that this is not valid streaming mode data. >This is a problem in all cases where any streaming data is >allowed prior to entering iWARP mode; that is why the >verbs specification is so careful to describe the one way >that MUST be supported by an interface. But it is simpler when Private Data is included in the existing adaptation layer messages. The ULPs merely have to ensure that they have reach a quiet state on the TCP connection before beginning work. Without Private Data Support, the ULPs must perform this shift while having messages in flight. Indeed, it is likely that the Private Data and the trailing MPA Mode Frame could end up within the very same TCP segment. >Semantics... The iWARP stack would have to have three >modes; pre-MPA, MPA-Frame with private data reception, and >full iWARP. Are you stating that MPA cannot be enabled distinctly from iWARP? Isn't that a severe layering violation? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 16:49:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12227 for ; Fri, 6 Jun 2003 16:49:30 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56Kn4g28820 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 16:49:04 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56Kn4B28817 for ; Fri, 6 Jun 2003 16:49:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12207 for ; Fri, 6 Jun 2003 16:49:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OO78-0006Lz-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 16:47:06 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OO77-0006Lv-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 16:47:05 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56KmLB28742; Fri, 6 Jun 2003 16:48:21 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56KkaB28655 for ; Fri, 6 Jun 2003 16:46:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA11993 for ; Fri, 6 Jun 2003 16:46:32 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OO4j-0006L1-00 for rddp@ietf.org; Fri, 06 Jun 2003 16:44:37 -0400 Received: from msgbas2x.cos.agilent.com ([192.25.240.37]) by ietf-mx with esmtp (Exim 4.12) id 19OO4j-0006Ks-00 for rddp@ietf.org; Fri, 06 Jun 2003 16:44:37 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id 7A6D421DD; Fri, 6 Jun 2003 14:46:31 -0600 (MDT) Received: from axcsbh3.cos.agilent.com (axcsbh3.cos.agilent.com [130.29.152.190]) by relcos1.cos.agilent.com (Postfix) with SMTP id 3B6EE5FF; Fri, 6 Jun 2003 14:46:31 -0600 (MDT) Received: from 130.29.152.190 by axcsbh3.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 06 Jun 2003 14:46:30 -0600 Received: by axcsbh3.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 6 Jun 2003 14:46:30 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00131CA4C@axcs04.cos.agilent.com> To: cait@asomi.com, Paul.Culley@hp.com Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Date: Fri, 6 Jun 2003 14:46:28 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, I do not understand why you say: Secondly, TCP is not well suited to this purpose. In all of the current proposals for pre-RDMA-mode negotiations, the two sides have to agree *precisely* on how many bytes of Private Data will be exchanged in each direction. There is no need for the two sides to agree on the size of the pre-RDMA mode message(s). It is entirely possible to define a pre-RDMA negotiation that allows for variable size Private Data if that is desired. All that is necessary is that one have some means of knowing when one has reached the end of the not MPA data so that one can start MPA/DDP/RDMAP. One already needs to support operation over TCP directly and operation over the iWARP stack over TCP. If ther has to be a point when MPA is operating but DDP/RDMAP is not, then we have to support and test an additional operating mode. I don't see what is gained by it. -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Thursday, June 05, 2003 9:46 AM To: Culley, Paul Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" On 6/5/03, Culley, Paul wrote: >Regarding "private data"; > >DDP/MPA are currently expected to be enabled together, >after creating the QP (and associating it with a PD). The >method we expected to use to deal with private data was to >have DDP/MPA's ULP (perhaps a bit of code in the IT-API >interface for example) be in charge of starting the TCP >connections and then, after doing any negotiations needed >between endpoints (using streaming mode TCP data), create >the QP and transition the connection to DDP/MPA mode. > >The "private data" would be exchanged in TCP streaming >mode (probably using standard sockets calls under the >API). When the Application/API had any necessary stuff >completed (like figuring out the correct PD), the QP can >be created, buffers posted, and the TCP "connection >context" passed to the iWARP stack to start DDP/MPA mode >processing. The Verbs document you referred to supports >this type of startup. > >I think that using the TCP streaming mode prior to MPA >enablement is preferred to adding additional features to >MPA as you suggest. The difference in flow over your >connection establishment sequence would only be that MPA >is not involved until the "bind" step. > >The IT-API group would have to publish a small draft >describing the use of the streaming mode and expected data >format prior to iWARP startup, but this is little >different than other applications or APIs such as SDP etc. > Requiring Private Data negotiations to be done with the LLP forces the ULP to be aware of what the ULP is. With the change proposed, the ULP could be indifferent as to whether the LLP was SCTP or TCP. Do you agree this would be of value? Secondly, TCP is not well suited to this purpose. In all of the current proposals for pre-RDMA-mode negotiations, the two sides have to agree *precisely* on how many bytes of Private Data will be exchanged in each direction. ULPs that need this data *frequently* have variable information needs, or even *optional* private data. Including Private Data in the MPA Mode frame solves this problem very easily. An MPA frame already has a length. A frame to introduce MPA Mode is already called for. Further, there is no need to use this Private Data. Applications that wanted to use LLP specific methods of exchanging private data would be free to do so. This proposal *enables* transport neutral connection establishment. The only benefit I can identify to use of LLP-specific streaming mode to negotiate connections is that such a negotiation could last several rounds. However, neither DAT or the IT-API found multi-round negotiations to be useful. The InfiniBand CM protocol does not support multiple-rounds of ULP negotiation either. The bottom line goal of pre-RDMA negotiations is to enable the correct Protection Domain to be selected. This has two important characteristics: -- Frequenty there is no need to constrain the selection of the Protection Domain. Meaning that many connections require zero Private Data. Incorporating the Private Data as an optional variable length field supports zero length Private Data very inexspensively. Testing the TCP stream to see *if* there is an MPA mode frame present would be difficult. I will point out that the current verbs proposed in draft-hilland do not support *checking* to see if the TCP stream is ready to enter MPA mode. -- One message exchange is all that is required to determine any and all Protection Domain and QP/endpoint needs. Other session details can be negotiated after the QP/endpoints are linked. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 17:27:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14080 for ; Fri, 6 Jun 2003 17:27:12 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56LQlK32191 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 17:26:47 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56LQlB32188 for ; Fri, 6 Jun 2003 17:26:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14048 for ; Fri, 6 Jun 2003 17:26:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OOhc-0006mN-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 17:24:48 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OOhb-0006mK-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 17:24:47 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56LQ2B32144; Fri, 6 Jun 2003 17:26:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56LPdB32125 for ; Fri, 6 Jun 2003 17:25:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14011 for ; Fri, 6 Jun 2003 17:25:34 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OOgW-0006lk-00 for rddp@ietf.org; Fri, 06 Jun 2003 17:23:40 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19OOgW-0006ji-00 for rddp@ietf.org; Fri, 06 Jun 2003 17:23:40 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 16E282458D2; Fri, 6 Jun 2003 17:25:36 -0400 (EDT) Date: Fri, 6 Jun 2003 16:25:04 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: pat_thaler@agilent.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A00131CA4C@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/6/03, pat_thaler@agilent.com wrote: >Caitlin, > >I do not understand why you say: >>Secondly, TCP is not well suited to this purpose. In all >>of the current proposals for pre-RDMA-mode negotiations, >>the two sides have to agree *precisely* on how many bytes >>of Private Data will be exchanged in each direction. MPA mode must be entered at a precise byte within the TCP stream. Further, this cannot be correctly done based upon timing, boundaries of TCP segments, or other artifacts no matter how likely they are. It is totally reasonable for one side to send its final pre-RDMA mode negotiation data, and then send its MPA Mode frame followed by its first DDP segment. They could even end up being in the same TCP segment. The proposed iWarp verbs in draft-hilland even provide a single verb to send a last streaming mode message and then enter MPA mode. So bundling of at least two of those operations is in fact likely. The ULP would have to use the plain-TCP stack to consume *exactly* the correct number of bytes. Then it has to tell the stack to shift to MPA mode, consuming the MPA Mode frame as confirmation, and *then* MPA processing is fully enabled. The various complaints about "three modes" all seem to ignore that it is the existence of the MPA Mode Frame that forces this, the presence of Private Data within the frame changes nothing. Anyway, how does the receiving end know how many bytes to consume? After extensive discussion within the DAT Collaborative and the Opengroup's ICSC it was agreed that the only method that worked with the current definitions was for the ULP to *read* the Private Data a TCP stream -- and that simulating this interface for SCTP and InfiniBand would be unnatural. More importantly, it probably increases the number of kernel/user transitions. The ULP is awoken with notice of the TCP connection. It then has to read the TCP connection, and *then* bind the TCP connection to an RDMA endpoint/QP. The procedure I proposed would provide a single notice to the ULP that an iWARP connection had been requested (whether on a pair of SCTP Streams or on a TCP connection). That notice would include any collected Private Data. The ULP would then either accept or reject the request. If accepted the endpoint/QP is specified, and optional response Private Data is supplied. This is *always?* two user/kernel interactions. Compare that with the "consume the TCP stream yourself" approach: -- If the ULP knows there is no Private Data, it's a breakeven. Two user/kernel interactions will work. -- If the ULP knows there will be a fixed amount of Private Data it can read that, then respond. That's three user/kernel interactions. An increase of one. -- If the Private Data has a variable length, the ULP must first read the minimum length, and *then* read the additional length. That's four user/kernel interactions, or an increase of two. A standard protocol, whether integrated into MPA or not, would allow the kernel/hardware to complete the entire process of accepting the the TCP connection, reading the Private Data, and presenting a notice to the ULP. Further, that same interface could be supported for SCTP. The ULP requires fewer interactions with the kernel *and* maintains transport neutrality. The current MPA solution only looks simple if you assume applications have nothing negotiate AND that they are written over TCP. That is *not* an acceptable assumption to make in designing iWARP. > >There is no need for the two sides to agree on the size of >the pre-RDMA mode message(s). It is entirely possible to >define a pre-RDMA negotiation that allows for variable >size Private Data if that is desired. All that is >necessary is that one have some means of knowing when one >has reached the end of the not MPA data so that one can >start MPA/DDP/RDMAP. > But suggest a negotiation would be a) ULP specific and b) LLP specific. Neither is desirable. Neither matches the procedures that a ULP would use to establish a connection between RDMA endpoints over other transports. The impact on MPA and the propoed verbs is actually quite slight. Variable length Private Data must be added to an MPA Mode Frame that will be required anyway. Secondly there is a difference between enabling for MPA receive and final binding to the iWARP endpoint/QP. Such a two-step enabling process is already described in draft-hilland. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 19:43:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA18701 for ; Fri, 6 Jun 2003 19:43:35 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h56Nh8d09093 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 19:43:08 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56Nh8B09090 for ; Fri, 6 Jun 2003 19:43:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA18697 for ; Fri, 6 Jun 2003 19:43:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OQpb-0007df-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 19:41:11 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OQpa-0007dc-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 19:41:10 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56NgGB09072; Fri, 6 Jun 2003 19:42:16 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h56NfpB09051 for ; Fri, 6 Jun 2003 19:41:51 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA18681 for ; Fri, 6 Jun 2003 19:41:48 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OQoM-0007dQ-00 for rddp@ietf.org; Fri, 06 Jun 2003 19:39:54 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=maho3msx2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19OQoL-0007dK-00 for rddp@ietf.org; Fri, 06 Jun 2003 19:39:53 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 6 Jun 2003 19:41:13 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0B747EE4@corpmx14.corp.emc.com> To: rddp@ietf.org Date: Fri, 6 Jun 2003 19:33:22 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h56NfpB09052 Subject: [rddp] CRC mandatory to use? Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , X-MIME-Autoconverted: from 8bit to quoted-printable by www1.ietf.org id h56NgGB09072 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h56Nh8B09090 Content-Transfer-Encoding: 8bit I've gone back over the past couple of weeks of email traffic on this issue. Let me first deal with a few red herrings: - The "negotiation causes delay" assertion has resurfaced. I hope Caitlin Bestler's explanation (Tuesday, 6/3) will finally lay this one to rest: There seems to be an impression that MPA options will require a round-trip. This is not the case. All of the MPA Options discussed to data can be implemented "for free" within the MPA Mode frame that was mandated to ensure that MPA/TCP connections could not be confused with TCP connections. The model the WG is heading for is more of an announcement (of capabilities/preferences/requirements) than a negotiation model - traffic can be sent immediately behind the initial announcement, and if the announcements are incompatible (and unrecoverably so), the connection closes. The WG has been headed for this sort of model (and not negotiation round trips) since last fall's Atlanta meeting - would the advocates of "MUST USE CRCs" kindly stop demolishing this strawman? - There's been some discussion about how decisions made here will constrain future actions of other WGs that use RDDP. While a MUST in a specification applies to use of that specification by reference, it does not constrain another WG that adapts RDDP for its own use based on its best technical judgment for that use. The IP Storage WG did this for IPsec for a number of reasons, and the final result was that some "SHOULD"s and "MUST"s from the underlying IPsec RFCs became "MAY"s and even "SHOULD NOT"s (e.g., DES usage). Based on this example, it would be wrong to assume that we can impose our technical judgment on all uses of RDDP for all time. We can explain clearly why we did what we did, and hope that future engineers will see the wisdom of our ways, but the notion that a MUST here will settle this issue for all time and in all circumstances is naïve - we surely do not have a permanent monopoly on serious interest in this area. - I saw an assertion that RDMA is just a remote extension of local DMA. While that may be valid for consideration of level of integrity protection, it's rather wrong in general over the Internet. The analogous assertion that RPC is just a remote extension of local procedure call has caused a lot of bad distributed systems design. - There seems to be some confusion over IPsec security properties. Encryption by itself provides no additional integrity properties. IPsec provides both encryption and cryptographic integrity (keyed HMAC), and the security community is now of the general opinion that encryption should in general not be used without some sort of cryptographic integrity check. When IPsec integrity is in use, the result is generally much stronger than a CRC (hint: ESP uses 96 bits of HMAC) - when it's in use with sufficient coverage (e.g., one is not worried about risks of corruption between end systems and their IPsec offload engines), there's little added value to a higher level CRC. Overall I'm struck by the almost absolute certainty that the advocates of "MUST use CRCs" seem to exhibit in predicting and scoping the usage of RDDP. Past experience with protocols of general utility indicates that it's hard to predict all the ways in which good new tools will get used. An example to think about is that RSVP was originally designed for setup of QoS-enhanced paths for IP traffic. I don't believe its designers anticipated RSVP's most common use today - MPLS label-switched path setup - but they designed enough flexibility into RSVP so that it could be adapted for this purpose (for which it is now the predominant mechanism). From a process standpoint, it's clear that my initial attempt to call consensus on (E)+(E') has failed: (E) CRCs are OPTIONAL to use, but (per (D) above), either end of the connection can force CRC usage in both directions. (E') CRC usage is an administrative configuration decision - it is not intended to be visible to ULPs. OTOH, it is also clear to me from the ensuing discussion that WG rough consensus for CRCs being mandatory to use will not be obtained. There have been some useful elaborations on (E') from the discussion, which I'm not going to try to capture here. It still appears to me that (E) and (E') are the only realistic way to make forward progress. This is based in part on my view of the technical discussion - from what I can see, stronger technical points are being made in favor of (E) and (E') by comparison to the MUST use CRCs position. A significant portion of the latter seems to be: "I'd never ever disable CRCs in any RDDP system I design", which is consistent with (E) and (E'), unless we're going to go back to the "all options are evil" nonsense (please don't ...). Everyone's entitled to their own design judgment, but let's not go overboard in constraining the ability of others to exercise their own judgment in a fashion that reaches different conclusions. For further discussion ... not that anyone needs that encouragement ... Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 6 21:01:16 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20028 for ; Fri, 6 Jun 2003 21:01:16 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5710n613482 for rddp-archive@odin.ietf.org; Fri, 6 Jun 2003 21:00:49 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5710nB13479 for ; Fri, 6 Jun 2003 21:00:49 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20012 for ; Fri, 6 Jun 2003 21:00:45 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OS2k-0000Fq-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 20:58:50 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OS2k-0000Fn-00 for rddp-web-archive@ietf.org; Fri, 06 Jun 2003 20:58:50 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5710BB13470; Fri, 6 Jun 2003 21:00:11 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h570xbB13410 for ; Fri, 6 Jun 2003 20:59:37 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA19970 for ; Fri, 6 Jun 2003 20:59:33 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OS1a-0000Ee-00 for rddp@ietf.org; Fri, 06 Jun 2003 20:57:38 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19OS1a-0000EM-00 for rddp@ietf.org; Fri, 06 Jun 2003 20:57:38 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id A07512453FF for ; Fri, 6 Jun 2003 20:59:34 -0400 (EDT) Date: Fri, 6 Jun 2003 19:59:02 -0500 From: Caitlin Bestler Subject: Re: [rddp] CRC mandatory to use? To: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0B747EE4@corpmx14.corp.emc.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/6/03, Black_David@emc.com wrote: >(E) CRCs are OPTIONAL to use, but (per (D) above), > either end of the connection can force CRC > usage in both directions. > >(E') CRC usage is an administrative configuration >decision - it is not intended to be visible to ULPs. I'd like to propose some refinements on (E') in order to clarify discussion of issues. (E1) The decision for one host to request CRC suppression MAY be made on an administrative basis for any path that provides equivalent protection from undetected errors as an end-to-end CRC32c. The process MUST be invisible to the ULP. (E2) The required standard to allow a request for CRC suppression MAY be lowered on a per ULP basis. If the ULP takes no explicit action to lower the standard it MUST remain being the equivalent of an end-to-end CRC32c. Even if non officially blessed a form of E1 could end up being implemented without official blessing under a "No Harm No Foul" line of thinking. I strongly agree with Jim Williams' point that we are better off having this occur in public, where the standard is clearly labeled. Without such a rule we may encourage vendors to create such a shortcut when interoperating with their own RNICs. Adopting a formal rule ensures interoperability and a even playing field. E2 is more expansive, and covers the widest range of suppression that has been suggested without undue risk. The key is that it assures that there is nothing the ULP has to check to ensure that the standard has not been lowered. And most importantly, we have given no ULP designers a reason to add their own protection based upon any perception that protection is "optional". Now, if they plan to add their own 256-bit protection, they might decide that its plain silly to keep the 32-bit MPA check. I don't have a problem with that. What I think we all want to avoid is a discussion that goes something like "Should we add our own CRC? I remember something about MPA having optional protection, and I can't find the API to make sure its turned on." Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 7 20:43:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27415 for ; Sat, 7 Jun 2003 20:43:31 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h580h4407832 for rddp-archive@odin.ietf.org; Sat, 7 Jun 2003 20:43:04 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h580h4B07829 for ; Sat, 7 Jun 2003 20:43:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27411 for ; Sat, 7 Jun 2003 20:43:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OoF6-0007FO-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 20:41:04 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OoF5-0007FK-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 20:41:03 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h580gOB07819; Sat, 7 Jun 2003 20:42:24 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h580fmB07796 for ; Sat, 7 Jun 2003 20:41:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27402 for ; Sat, 7 Jun 2003 20:41:45 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OoDs-0007FC-00 for rddp@ietf.org; Sat, 07 Jun 2003 20:39:48 -0400 Received: from saway.wasabisystems.com ([151.199.66.145] helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19OoDr-0007F8-00 for rddp@ietf.org; Sat, 07 Jun 2003 20:39:47 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id 6623B4013A; Sat, 7 Jun 2003 20:41:39 -0400 (EDT) Date: Sun, 8 Jun 2003 00:40:36 +0000 (UTC) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Caitlin Bestler Cc: rddp@ietf.org Subject: Re: [rddp] CRC mandatory to use? In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Fri, 6 Jun 2003, Caitlin Bestler wrote: > On 6/6/03, Black_David@emc.com wrote: > > >(E) CRCs are OPTIONAL to use, but (per (D) above), > > either end of the connection can force CRC > > usage in both directions. > > > >(E') CRC usage is an administrative configuration > >decision - it is not intended to be visible to ULPs. > > I'd like to propose some refinements on (E') in order > to clarify discussion of issues. > > > (E1) The decision for one host to request CRC suppression > MAY be made on an administrative basis for any path > that provides equivalent protection from undetected > errors as an end-to-end CRC32c. > > The process MUST be invisible to the ULP. Of the two, I like this one better. It keeps the idea that there SHOULD be the equivalent of CRC32C the whole way. > (E2) The required standard to allow a request for CRC > suppression MAY be lowered on a per ULP basis. > If the ULP takes no explicit action to lower the > standard it MUST remain being the equivalent of > an end-to-end CRC32c. > > Even if non officially blessed a form of E1 could end > up being implemented without official blessing under > a "No Harm No Foul" line of thinking. I strongly agree > with Jim Williams' point that we are better off having > this occur in public, where the standard is clearly labeled. > Without such a rule we may encourage vendors to create > such a shortcut when interoperating with their own RNICs. > Adopting a formal rule ensures interoperability and a > even playing field. > > E2 is more expansive, and covers the widest range of > suppression that has been suggested without undue risk. > The key is that it assures that there is nothing the > ULP has to check to ensure that the standard has not > been lowered. And most importantly, we have given no > ULP designers a reason to add their own protection > based upon any perception that protection is "optional". > > Now, if they plan to add their own 256-bit protection, > they might decide that its plain silly to keep the > 32-bit MPA check. I don't have a problem with that. > What I think we all want to avoid is a discussion > that goes something like "Should we add our own > CRC? I remember something about MPA having optional > protection, and I can't find the API to make sure > its turned on." I agree with wanting to avoid that discussion if possible. The guidelines on when to turn off CRCs should help avoid it. I hope. :-) Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 7 21:17:25 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA27988 for ; Sat, 7 Jun 2003 21:17:25 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h581GxX09439 for rddp-archive@odin.ietf.org; Sat, 7 Jun 2003 21:16:59 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h581GxB09436 for ; Sat, 7 Jun 2003 21:16:59 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA27964 for ; Sat, 7 Jun 2003 21:16:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Oolu-0007Pn-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 21:14:58 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Oolt-0007Pk-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 21:14:57 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h581GFB09421; Sat, 7 Jun 2003 21:16:15 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h581F0B09370 for ; Sat, 7 Jun 2003 21:15:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA27699; Sat, 7 Jun 2003 21:14:55 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Oojy-0007LF-00; Sat, 07 Jun 2003 21:12:58 -0400 Received: from e31.co.us.ibm.com ([32.97.110.129]) by ietf-mx with esmtp (Exim 4.12) id 19Oojx-0007LC-00; Sat, 07 Jun 2003 21:12:57 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e31.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h581EMll263044; Sat, 7 Jun 2003 21:14:22 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h581ELeV163416; Sat, 7 Jun 2003 19:14:21 -0600 To: Caitlin Bestler Cc: "Culley, Paul" , rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Sat, 7 Jun 2003 18:14:17 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 3, 2003) at 06/07/2003 19:14:21, Serialize complete at 06/07/2003 19:14:21 Content-Type: multipart/alternative; boundary="=_alternative 0006BD0A88256D3F_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 0006BD0A88256D3F_= Content-Type: text/plain; charset="us-ascii" Catlin, Paul said the following: > There is at least one application in the works that >does use a multi-round negotiation prior to iWARP >startup... Of course it is written to use TCP streaming >mode on a legacy port for this. And you replied with "Which is *exactly* what we need to avoid. An iWARP application SHOULD NOT be aware of which LLP is in use." I find that answer less than satisfying. I also know of a application that must have a number of multi-round negotiations before they go into RDMA mode. This has to do with understanding all the capabilities common between the sides, before actually deciding on RDMA mode, or continuing in TCP/IP mode. That application does NOT want to stop the connection if the other side does not want to enter RDMA mode. In other words it wants to fall back to a previously defined TCP/IP protocol and continue operations. Perhaps I misunderstood you, but if you do not want to support this multi-round negotiations, then I do not think we are covering all the requirements. Did I misunderstand you? . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com --=_alternative 0006BD0A88256D3F_= Content-Type: text/html; charset="us-ascii"
Catlin,
Paul said the following:

><prc> There is at least one application in the works that
>does use a multi-round negotiation prior to iWARP
>startup...  Of course it is written to use TCP streaming
>mode on a legacy port for this. <prc>
And you replied with

"Which is *exactly* what we need to avoid. An iWARP
application SHOULD NOT be aware of which LLP is in use."

I find that answer less than satisfying.  I also know of a application that must have a number of multi-round negotiations before they go into RDMA mode.  This has to do with understanding all the capabilities common between the sides, before actually deciding on RDMA mode, or continuing in TCP/IP mode.  That application does NOT want to stop the connection if the other side does not want to enter RDMA mode.  In other words it wants to fall back to a previously defined TCP/IP protocol and continue operations.

Perhaps I misunderstood you, but if you do not want to support this multi-round negotiations, then I do not think we are covering all the requirements.  

Did I misunderstand you?

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com --=_alternative 0006BD0A88256D3F_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 7 22:17:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28994 for ; Sat, 7 Jun 2003 22:17:39 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h582HEL12462 for rddp-archive@odin.ietf.org; Sat, 7 Jun 2003 22:17:14 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h582HEB12458 for ; Sat, 7 Jun 2003 22:17:14 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28987 for ; Sat, 7 Jun 2003 22:17:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OpiC-0007hw-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 22:15:12 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19OpiA-0007hr-00 for rddp-web-archive@ietf.org; Sat, 07 Jun 2003 22:15:10 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h582GKB12436; Sat, 7 Jun 2003 22:16:22 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h582FXB12416 for ; Sat, 7 Jun 2003 22:15:33 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA28934 for ; Sat, 7 Jun 2003 22:15:27 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19OpgZ-0007hD-00 for rddp@ietf.org; Sat, 07 Jun 2003 22:13:31 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19OpgY-0007gy-00 for rddp@ietf.org; Sat, 07 Jun 2003 22:13:30 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D4EFF245376; Sat, 7 Jun 2003 22:15:26 -0400 (EDT) Date: Sat, 7 Jun 2003 21:14:55 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: John Hufferd , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/7/03, John Hufferd wrote: >Catlin, >Paul said the following: > >> There is at least one application in the works that >>does use a multi-round negotiation prior to iWARP >>startup... Of course it is written to use TCP streaming >>mode on a legacy port for this. > >And you replied with > >"Which is *exactly* what we need to avoid. An iWARP >application SHOULD NOT be aware of which LLP is in use." > >I find that answer less than satisfying. I also know of a >application that must have a number of multi-round >negotiations before they go into RDMA mode. This has to >do with understanding all the capabilities common between >the sides, before actually deciding on RDMA mode, or >continuing in TCP/IP mode. That application does NOT want >to stop the connection if the other side does not want to >enter RDMA mode. In other words it wants to fall back to >a previously defined TCP/IP protocol and continue >operations. > >Perhaps I misunderstood you, but if you do not want to >support this multi-round negotiations, then I do not think >we are covering all the requirements. > >Did I misunderstand you? > No, I was dealing with a different set of applications. An application which forms a TCP connection, and *then* decides whether or not to enable RDMA mode, obviously must be aware that it is using TCP. Indeed, for reasons I'll go into in a moment, that model does not work well for SCTP. The model I was working from was one where the use of RDMA is presumed in advance. The ULP is RDMA specific. Their may be a non-RDMA equivalent, but it is running on a different port (or perhaps using a different IP address). Not surprisingly this is the model that has dominated discussions within both the DAT Collaborative and the OpenGroup's ICSC. Their objective, after all, is to define transport neutral APIs for RDMA-aware applications. So to rephrase more precisely, it should be possible for a transport neutral application to communicate the Private Date required to select a Protection Domain and configure the RDMA Read credits in a single message sent *before* the RDMA endpoint is selected -- and to do so in a transport neutral way. To achieve this goal, either InfiniBand and SCTP must simulate a "stream" of pre-RDMA endpoint data, or TCP must provide a method of sending "one message". I believe the best way to do so is to include a variable length Private Data field in the MPA Mode frame. The ability to send one Private Data message before selecting the RDMA endpoint is already present in InfiniBand and is easily defined for SCTP. Alternatively, a protocol could be defined to delimit such a message over TCP independently of the shift into MPA mode. Either method works. In either case, the procedure is not imposed on any applications. Those that want to establish a TCP connection and then dynamically decide to enter RDMA mode are free to do so. If a Private Data field is defined, there will be no requirement to use it, and no prohibition against negotiating in plain TCP before its use. But there are a larger group of ULPs that will be RDMA specific from the very beginning. Indeed, for some it would make no sense to have a non-RDMA version. Would you run SDP over plain TCP? Even with less extreme cases, optimum use of RDMA is best achieved when the ULP is fully aware of RDMA's capabilities. While there will undoubtedly be servers that have valid reasons for dynamically deciding whether to enable RDMA, there will be at least as large a group that prefer to run the two distinctly different sets of logic through different daemons running on different ports. The server advertises two different sevices (whether by SLP, in the URL, or by whatever method), and the client chooses which to invoke. There is an issue on the dynamic "in-session" switch to RDMA model as applied to SCTP. The current SCTP draft will not support such a model. The rationale is that sharing control of streams within an single SCTP association between a non-RDMA stack and a RDMA stack would be very difficult. A likely adoption strategy would have the RDMA stack implemented on an RNIC, with the non-RDMA stack remaining "as is" on the processor. Segregating SCTP associations between the RDMA-implementation and the current stack allows RDMA to be deployed without disrupting current stacks. This same issue exists with TCP connections. But TCP only requires a relatively clean handoff from the conventional stack to the MPA/iWARP stack (relatively clean, because responsiblity for retransmitting might be split right at the shift point). An SCTP implementation cannot shift just one Stream. Then handling of TSNs, SACKs and all other control chunks is common across all streams. So is the correct response: a) Don't worry about it. The "dynamic in-session" model is highly specialized and inherently related to existing servers that deal with long-haul connections where the clients have little a priori knowledge of the server. There is little need to support this model for SCTP. b) Require an iWARP capable SCTP stack to also support non-iWARP streams. The trick is that non-iWARP streams can have *ordered* Data Chunks. The handling of which is both more complex and more consuming of memory bandwidth than for an iWARP stream (which only uses unordered data chunks). If dynamic in-session switching is a general model that should be supported in an LLP fashion then the SCTP draft should find a way to accomodate it -- preferably in a way that doesn't make an implemetnation that much more difficult. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jun 8 15:33:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28909 for ; Sun, 8 Jun 2003 15:33:09 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h58JWfS18937 for rddp-archive@odin.ietf.org; Sun, 8 Jun 2003 15:32:41 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58JWfB18934 for ; Sun, 8 Jun 2003 15:32:41 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28886 for ; Sun, 8 Jun 2003 15:32:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P5rp-0004yy-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 15:30:13 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19P5rp-0004yv-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 15:30:13 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58JVPB18884; Sun, 8 Jun 2003 15:31:25 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58JUiB18839 for ; Sun, 8 Jun 2003 15:30:44 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA28847 for ; Sun, 8 Jun 2003 15:30:26 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P5q8-0004yS-00 for rddp@ietf.org; Sun, 08 Jun 2003 15:28:28 -0400 Received: from mms2.broadcom.com ([63.70.210.59]) by ietf-mx with esmtp (Exim 4.12) id 19P5q7-0004yP-00 for rddp@ietf.org; Sun, 08 Jun 2003 15:28:27 -0400 Received: from 63.70.210.1 by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Sun, 08 Jun 2003 12:27:02 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id MAA05386; Sun, 8 Jun 2003 12:29:56 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Sun, 8 Jun 2003 12:28:51 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMp4ezBfRNzf25yQUK2+DDZVdyK3gEEXrfg From: "Uri Elzur" To: "Williams, Jim" , rddp@ietf.org X-WSS-ID: 12FD518C3135016-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h58JUsB18846 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Jim The issue it that CRC as an option, prevents the App from knowing FOR SURE the CRC is in MPA (see other mail) The cost for CRC in SW goes away after some time (short) especially if we eliminate the option, as everyone will have it in the HW. CRC in HW is trivial and low cost. If we decide to keep CRC as an option, we consciously muddy the water by not providing a clear direction for implementations and then you will really get CRC in SW for long time which is undesirable Uri -----Original Message----- From: Williams, Jim [mailto:Jim.Williams@emulex.com] Sent: Tuesday, June 03, 2003 7:43 AM To: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update > If the _option_ to turn it off is there, it will forever add a > negotiation which slows down the initialization and adds complexity to > implementations. It will be yet another knob the administrator has to > deal with and no one showed there is a HUGE performance hit > > Uri The complexity and performance burden of the option should be somewhere between negligible and zero if done right. Implementations are free to always use CRC and always require remote end to use CRC if the implementers wish. The knob only exists if implementers feel there is demand for it. The administrator only needs to know about the knob if he wants to use it. The pragmatic issue is that a customer may come to us and say, "We haven't seen a TCP checksum error in years, but we are wasting 10% of our compute resources on CRCs. Can you turn off the CRCs for us?" We will of course say yes, regardless of whether the standard allows it or not. But I would rather have a mechanism to do this defined within the standard than have to invent an ad hoc non-standard mechanism. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jun 8 15:55:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29157 for ; Sun, 8 Jun 2003 15:55:42 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h58JtFh20152 for rddp-archive@odin.ietf.org; Sun, 8 Jun 2003 15:55:15 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58JtFB20149 for ; Sun, 8 Jun 2003 15:55:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29150 for ; Sun, 8 Jun 2003 15:54:43 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6Dc-00051l-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 15:52:44 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19P6Dc-00051i-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 15:52:44 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58Js1B20109; Sun, 8 Jun 2003 15:54:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58JquB20092 for ; Sun, 8 Jun 2003 15:52:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29135 for ; Sun, 8 Jun 2003 15:52:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6Be-00051C-00 for rddp@ietf.org; Sun, 08 Jun 2003 15:50:42 -0400 Received: from mms2.broadcom.com ([63.70.210.59]) by ietf-mx with esmtp (Exim 4.12) id 19P6Bd-000519-00 for rddp@ietf.org; Sun, 08 Jun 2003 15:50:41 -0400 Received: from 63.70.210.1 by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Sun, 08 Jun 2003 12:49:20 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id MAA05864; Sun, 8 Jun 2003 12:52:14 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Sun, 8 Jun 2003 12:51:13 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMp5K+QLbLVzXwwTgazUiErF1BlLQEEAj2Q From: "Uri Elzur" To: "Caitlin Bestler" , "Williams, Jim" cc: rddp@ietf.org X-WSS-ID: 12FD4CCA3137369-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h58Jr2B20095 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit A. If the proposal is that it takes one side that requires CRC to mandate CRC for the particular flow, it allows "knob' free implementations for the side where CRC is always on. It still requires a mechanism for the ULP to know this kind of HW is installed. How is that achieved? B. If one side can force CRC, how can the other side turn it off? Or is the idea that that side will not initiate comm with CRC on but if required by remote peer it will be supported??? So in any case the SW needs to have the capability to compute/check CRC. C. As mentioned below (is it really 10%..), I'm not sure the cost of CRC in SW justifies all of this lengthy discussion. If it this was a real heavy weight mechanism like line speed encryption, I can see the need for an option. D. not sure I followed the idea of destination masks, I guess this is just for the side with optional CRC. This clearly adds additional complexity (un-bound) to remember all destination and their preference. Is it the right tradeoff? Uri -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Tuesday, June 03, 2003 8:08 AM To: Williams, Jim Cc: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update On 6/3/03, Williams, Jim wrote: > >> If the _option_ to turn it off is there, it will forever >> add a negotiation which slows down the initialization >> and adds complexity to implementations. It will be yet another knob >> the administrator has to deal with and no one showed there is a HUGE >> performance hit >> >> Uri > >The complexity and performance burden of the option >should be somewhere between negligible and zero if >done right. Implementations are free to >always use CRC and always require remote end >to use CRC if the implementers wish. The knob only >exists if implementers feel there is demand for it. >The administrator only needs to know about the knob >if he wants to use it. > Good points. What makes the proposal acceptable to me is precisely that it can largely be ignored, and that it does not even require any setup delay to negotiate because the active side can simply send the first FULPDU with a CRC before finding out whether the other side will consent to suppressing CRCs. But it would be best to view them as "knobs". If they exist, and a stack have no ability to suppress CRCs at all is clearly acceptable, they need to be destination dependent. For example: try to suppress CRCs with destinations that conform to this list of IP masks. >The pragmatic issue is that a customer may come to us >and say, "We haven't seen a TCP checksum error in years, >but we are wasting 10% of our compute resources on CRCs. >Can you turn off the CRCs for us?" > >We will of course say yes, regardless of whether the >standard allows it or not. But I would rather have >a mechanism to do this defined within the standard >than have to invent an ad hoc non-standard mechanism. All valid arguments on why standards SHOULD be as accomodating as possible. One small disagreement: any customer who is wasting 10% of their compute resources on CRCs is using the wrong algorithm or has an amazingly underpowered processor. But the degree of waste is only relevant if the cost of the option is signifigant. As proposed, the cost of this option is that some bits in the MPA Mode frame are allocated, and that it has to be documented properly. A "All CRCs All The Time" implementation does not even have to *examine* those bits, it merely has to set its bits to indicate that it will insist on CRCs. That's a compile-time constant initializing a byte within a packet. A packet that would have been sent anyway, and a byte that was part of the packet anyway. You can't get options must cheaper than that. Even for a *slight* advantage for *some* implementations, that is a very low cost. I see no reason not to allow it. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jun 8 16:06:32 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29378 for ; Sun, 8 Jun 2003 16:06:32 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h58K64S20471 for rddp-archive@odin.ietf.org; Sun, 8 Jun 2003 16:06:04 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K64B20468 for ; Sun, 8 Jun 2003 16:06:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29369 for ; Sun, 8 Jun 2003 16:05:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6OD-00055J-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 16:03:41 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19P6OD-00055G-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 16:03:41 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K51B20433; Sun, 8 Jun 2003 16:05:03 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K49B20405 for ; Sun, 8 Jun 2003 16:04:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29336 for ; Sun, 8 Jun 2003 16:03:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6MT-00054Z-00 for rddp@ietf.org; Sun, 08 Jun 2003 16:01:53 -0400 Received: from mms2.broadcom.com ([63.70.210.59]) by ietf-mx with esmtp (Exim 4.12) id 19P6MS-00054W-00 for rddp@ietf.org; Sun, 08 Jun 2003 16:01:52 -0400 Received: from 63.70.210.1 by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Sun, 08 Jun 2003 13:00:25 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id NAA06145; Sun, 8 Jun 2003 13:03:19 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Sun, 8 Jun 2003 13:02:18 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMq94mb29PTly3LSlG/CBIJF0EMGADAUG3g From: "Uri Elzur" To: "Caitlin Bestler" cc: rddp@ietf.org X-WSS-ID: 12FD4A533138467-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h58K4BB20406 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Caitlin If indeed the end result is that ULP is _guaranteed_ to have same data protection with all transport and needs no API to verify it, I'm on-board. Still not sure what this additional complexity on the side with OPTIONAL CRC buys anyone. Uri -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, June 04, 2003 5:16 PM To: Uri Elzur Cc: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update On 6/4/03, Uri Elzur wrote: >Caitlin > >Strikes me we're on the same page. However, I don't see a >way around using an API, IF CRC is OPTIONAL. > Agreed. By inclination I am against options. I ended up convincing myself that this one made sense while trying to compose a blistering email tearing the rationale for the option to shreds. I could come up with lots of reasons why it was far from essential, but not as to how it hurt anybody (beyond wasting an option bit that might theoretically have found a use sometime 30 years later). >Let me try to clarify it, using the following scenario: in >the case MPA protocol makes the use of CRC OPTIONAL, in a particular >case MPA may have negotiated the connection to NO CRC. Now the ULP >cannot tell how MPA has negotiated CRC without a proper API. i.e. I'm >not sure the ULP can query MPA without a proper API (that I also think >can not be part of what RDDP requires or can be guaranteed to be >available) > >Therefore what will the ULP do? How will a ULP that wants >to RELY on MPA's CRC and NOT have to provide its own error recovery do >it? > There have been two criteria proposed: First, equivalence of result - not method. The ULP does not truly care *how* the payload is protected (well, at least it shouldn't). It cares *that* it is protected. An end-to-end CRC32c is defined to be a "safe harbor", that is one method is defined which will provide adequate protection without any further questions asked. Under the "equivalent protection" override, the CRC is suppressed for specific paths *only* if the network administrator can vouch that these paths truly have equivalent protections. Personally, I would anticipate that this feature would be used by all system administrators who are bored and have absolutely nothing else in their in-basket. ;-) Second, their MAY be an API that allows a ULP to volunteer for a lower level of protection. This lower level MUST NOT be applied unless the ULP actively invokes this API. Ignorance of such an API MUST leave the higher standard undisturbed. With these two rules, any ULP that requires the equivalence of end-to-end CRC32c protection need do nothing. Any application designed that does not understand how to lower the lower-protection API will receive the full end-to-end CRC32c (or equivalent) protection. So in the absence of an API to query MPA, the ULP can assume that it has CRC32c (or equivalent) protection. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Jun 8 16:09:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29452 for ; Sun, 8 Jun 2003 16:09:35 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h58K98G21425 for rddp-archive@odin.ietf.org; Sun, 8 Jun 2003 16:09:08 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K98B21422 for ; Sun, 8 Jun 2003 16:09:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29414 for ; Sun, 8 Jun 2003 16:08:36 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6R3-00055p-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 16:06:37 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19P6R3-00055m-00 for rddp-web-archive@ietf.org; Sun, 08 Jun 2003 16:06:37 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K81B21400; Sun, 8 Jun 2003 16:08:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h58K7YB21384 for ; Sun, 8 Jun 2003 16:07:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA29400 for ; Sun, 8 Jun 2003 16:07:18 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19P6Po-00055g-00 for rddp@ietf.org; Sun, 08 Jun 2003 16:05:20 -0400 Received: from mms2.broadcom.com ([63.70.210.59]) by ietf-mx with esmtp (Exim 4.12) id 19P6Pn-00055d-00 for rddp@ietf.org; Sun, 08 Jun 2003 16:05:19 -0400 Received: from 63.70.210.1 by mms2.broadcom.com with ESMTP (Broadcom SMTP Relay (MMS v5.5.2)); Sun, 08 Jun 2003 13:03:54 -0700 Received: from nt-irva-0740.brcm.ad.broadcom.com ([10.8.194.53]) by mon-irva-11.broadcom.com (8.9.1/8.9.1) with ESMTP id NAA06309; Sun, 8 Jun 2003 13:06:48 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Subject: RE: [rddp] CRC - rough consensus update Date: Sun, 8 Jun 2003 13:05:47 -0700 Message-ID: Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMq+eTJZ+qpAyklTPKGQQqAIZIGvAC/4p0w From: "Uri Elzur" To: wrstuden@wasabisystems.com cc: "Caitlin Bestler" , rddp@ietf.org X-WSS-ID: 12FD49203138838-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h58K7nB21386 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Ok. Seems like we're converging around the aspect of CRC (or equivalent) is always _guaranteed_ to the ULP. I'm also in support of it Uri -----Original Message----- From: wrstuden@wasabisystems.com [mailto:wrstuden@wasabisystems.com] Sent: Wednesday, June 04, 2003 5:32 PM To: Uri Elzur Cc: Caitlin Bestler; rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update On Wed, 4 Jun 2003, Uri Elzur wrote: > Caitlin > > Strikes me we're on the same page. However, I don't see a way around > using an API, IF CRC is OPTIONAL. The point is we aren't making CRCs "OPTIONAL." Optional means you are free to turn it on or off. We've been talking about making it such that the admin MAY indicate that CRCs can be skipped IF the admin is CONFIDENT that the connection will be as reliable as if it had the CRC on. Thus there's a burden on the admin that wants to mark CRCs as ignorable, which is not there if we use the word, "OPTIONAL." > Let me try to clarify it, using the following scenario: in the case > MPA protocol makes the use of CRC OPTIONAL, in a particular case MPA > may have negotiated the connection to NO CRC. Now the ULP cannot tell > how MPA has negotiated CRC without a proper API. i.e. I'm not sure > the ULP can query MPA without a proper API (that I also think can not > be part of what RDDP requires or can be guaranteed to be available) > > Therefore what will the ULP do? How will a ULP that wants to RELY on > MPA's CRC and NOT have to provide its own error recovery do it? The ULP will do the only thing it can. It will know that regardless of the CRC setting, it will have a link as reliable as if it has the CRC-32C verification. Because the only time(s) it is valid for the CRCs to be neglectable is when other factors known to the admin(s) indicate that the connection is reliable. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 9 12:10:03 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10059 for ; Mon, 9 Jun 2003 12:10:03 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h59G9ac14072 for rddp-archive@odin.ietf.org; Mon, 9 Jun 2003 12:09:36 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59G9aB14069 for ; Mon, 9 Jun 2003 12:09:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10039 for ; Mon, 9 Jun 2003 12:09:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PPBF-0004hs-00 for rddp-web-archive@ietf.org; Mon, 09 Jun 2003 12:07:33 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19PPBE-0004ho-00 for rddp-web-archive@ietf.org; Mon, 09 Jun 2003 12:07:32 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59G8sB14015; Mon, 9 Jun 2003 12:08:54 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59G3vB12688 for ; Mon, 9 Jun 2003 12:03:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA09873 for ; Mon, 9 Jun 2003 12:03:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PP5m-0004fs-00 for rddp@ietf.org; Mon, 09 Jun 2003 12:01:54 -0400 Received: from zcamail05.zca.compaq.com ([161.114.32.105]) by ietf-mx with esmtp (Exim 4.12) id 19PP5l-0004fm-00 for rddp@ietf.org; Mon, 09 Jun 2003 12:01:53 -0400 Received: from cacexg12.americas.cpqcorp.net (cacexg12.americas.cpqcorp.net [16.105.250.119]) by zcamail05.zca.compaq.com (Postfix) with ESMTP id 4E2C8DF82; Mon, 9 Jun 2003 09:03:16 -0700 (PDT) Received: from txnexc01.americas.cpqcorp.net ([16.74.7.244]) by cacexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 9 Jun 2003 09:03:15 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] CRC - rough consensus update Date: Mon, 9 Jun 2003 11:03:14 -0500 Message-ID: <717252EC3E37AE4392E2614EA24E9F2B051AC6A8@txnexc01.americas.cpqcorp.net> Thread-Topic: [rddp] CRC - rough consensus update Thread-Index: AcMp5K+QLbLVzXwwTgazUiErF1BlLQEEAj2QACke+wA= From: "Teisberg, Robert" To: "Uri Elzur" , "Caitlin Bestler" , "Williams, Jim" Cc: X-OriginalArrivalTime: 09 Jun 2003 16:03:15.0773 (UTC) FILETIME=[A2D096D0:01C32EA0] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h59G3wB12689 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit > -----Original Message----- > From: Uri Elzur [mailto:uri@broadcom.com] > A. If the proposal is that it takes one side that requires > CRC to mandate CRC for the particular flow, it allows "knob' > free implementations for the side where CRC is always on. It > still requires a mechanism for the ULP to know this kind of > HW is installed. How is that achieved? I don't see why the ULP needs to know. I'd bury the code to turn on the "I don't need CRCs" bit in the RNIC driver. Then it's the RNIC implementor's job to make sure the driver and RNIC agree. An RNIC (or its driver) that always insists on CRCs would hardwire the bit appropriately. An RNIC that doesn't would need an (out-of-scope for IETF purposes) administrative interface to control the bit. If the ULP gets a vote regarding turning off CRCs, that requires a simple API, but the mechanism is the same. > > B. If one side can force CRC, how can the other side turn it > off? Or is the idea that that side will not initiate comm > with CRC on but if required by remote peer it will be > supported??? So in any case the SW needs to have the > capability to compute/check CRC. I've been interpreting the "no CRCs" bit as a preference. Each end sends an MPA mode declaration indicating its CRC preference. If the preferences don't match, the endpoints conservatively agree to generate and check CRCs. Some suggested requirements: After receipt of an MPA mode declaration indicating that its peer requires CRCs, an MPA instance MUST continue generating and checking CRCs until the connection terminates. If an MPA instance has declared that it does not require CRCs, it MUST turn off CRC checking immediately after receipt of an MPA mode declaration indicating that its peer also does not require CRCs. It MAY continue generating CRCs. Until its partner's preference is confirmed, it MUST generate and check CRCs. The MPA mode declaration MUST contain a valid CRC. So, yes, I agree that a software MPA implementation needs to be able to compute and check CRCs. We may be able to get out of running the CRC code most of the time, but we can't escape implementing it. Apologies to lazy programmers everywhere :-) [snip] > > D. not sure I followed the idea of destination masks, I guess > this is just for the side with optional CRC. This clearly > adds additional complexity (un-bound) to remember all > destination and their preference. Is it the right tradeoff? The complexity in a hardware RNIC is trivial: just a one-bit register and an AND gate. The administrative and software complexity is another matter. Those who think it's worth the price will gladly pay it. Those who don't will mandate CRCs. The market will choose. > > Uri > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Tuesday, June 03, 2003 8:08 AM > To: Williams, Jim > Cc: rddp@ietf.org > Subject: RE: [rddp] CRC - rough consensus update > > > On 6/3/03, Williams, Jim wrote: > > > > >> If the _option_ to turn it off is there, it will forever > >> add a negotiation which slows down the initialization > >> and adds complexity to implementations. It will be yet > another knob > >> the administrator has to deal with and no one showed there > is a HUGE > >> performance hit > >> > >> Uri > > > >The complexity and performance burden of the option > >should be somewhere between negligible and zero if > >done right. Implementations are free to > >always use CRC and always require remote end > >to use CRC if the implementers wish. The knob only > >exists if implementers feel there is demand for it. > >The administrator only needs to know about the knob > >if he wants to use it. > > > > Good points. What makes the proposal acceptable to me > is precisely that it can largely be ignored, and that > it does not even require any setup delay to negotiate > because the active side can simply send the first FULPDU > with a CRC before finding out whether the other side will > consent to suppressing CRCs. > > But it would be best to view them as "knobs". If they exist, > and a stack have no ability to suppress CRCs at all is > clearly acceptable, they need to be destination dependent. > For example: try to suppress CRCs with destinations that > conform to this list of IP masks. > > >The pragmatic issue is that a customer may come to us > >and say, "We haven't seen a TCP checksum error in years, > >but we are wasting 10% of our compute resources on CRCs. > >Can you turn off the CRCs for us?" > > > >We will of course say yes, regardless of whether the > >standard allows it or not. But I would rather have > >a mechanism to do this defined within the standard > >than have to invent an ad hoc non-standard mechanism. > > All valid arguments on why standards SHOULD be as > accomodating as possible. > > One small disagreement: any customer who is wasting 10% of > their compute resources on CRCs is using the wrong algorithm > or has an amazingly underpowered processor. > > But the degree of waste is only relevant if the cost of the > option is signifigant. As proposed, the cost of this option > is that some bits in the MPA Mode frame are allocated, and > that it has to be documented properly. > > A "All CRCs All The Time" implementation does not even have > to *examine* those bits, it merely has to set its bits to > indicate that it will insist on CRCs. That's a compile-time > constant initializing a byte within a packet. A packet that > would have been sent anyway, and a byte that was part of the > packet anyway. You can't get options must cheaper than that. > > Even for a *slight* advantage for *some* implementations, > that is a very low cost. I see no reason not to allow it. > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Jun 9 12:23:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10425 for ; Mon, 9 Jun 2003 12:23:08 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h59GMg714547 for rddp-archive@odin.ietf.org; Mon, 9 Jun 2003 12:22:42 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59GMgB14544 for ; Mon, 9 Jun 2003 12:22:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10403 for ; Mon, 9 Jun 2003 12:22:38 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PPNv-0004lw-00 for rddp-web-archive@ietf.org; Mon, 09 Jun 2003 12:20:39 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19PPNu-0004lt-00 for rddp-web-archive@ietf.org; Mon, 09 Jun 2003 12:20:38 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59GM1B14523; Mon, 9 Jun 2003 12:22:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h59GL4B14496 for ; Mon, 9 Jun 2003 12:21:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA10364 for ; Mon, 9 Jun 2003 12:21:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PPML-0004ld-00 for rddp@ietf.org; Mon, 09 Jun 2003 12:19:01 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19PPMK-0004lP-00 for rddp@ietf.org; Mon, 09 Jun 2003 12:19:00 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B932C245150; Mon, 9 Jun 2003 12:21:02 -0400 (EDT) Date: Mon, 9 Jun 2003 11:20:30 -0500 From: Caitlin Bestler Subject: RE: [rddp] CRC - rough consensus update To: Uri Elzur Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/8/03, Uri Elzur wrote: >D. not sure I followed the idea of destination masks, I >guess this is just for the side with optional CRC. This >clearly adds additional complexity (un-bound) to remember >all destination and their preference. Is it the right >tradeoff? Destination masks are to indicate the set of Destination Addresses for which end-to-end CRC32c equivalence applies. If you claim that you have end-to-end CRC32c equivalence for *any* destination then either your network doesn't connect to much or you're lying. It's more likely that end-to-end CRC32c equivalence will only ever apply to the local network or a remote network connected via IPsec. Not that I expect folks to take much notice, but those are the very environments where use of MPA/TCP is not justified. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 10 10:37:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03429 for ; Tue, 10 Jun 2003 10:37:45 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5AEbKE27605 for rddp-archive@odin.ietf.org; Tue, 10 Jun 2003 10:37:20 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AEbKB27602 for ; Tue, 10 Jun 2003 10:37:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03381 for ; Tue, 10 Jun 2003 10:37:14 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PkDQ-0006y1-00 for rddp-web-archive@ietf.org; Tue, 10 Jun 2003 10:35:12 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19PkDQ-0006xy-00 for rddp-web-archive@ietf.org; Tue, 10 Jun 2003 10:35:12 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AEaLB26979; Tue, 10 Jun 2003 10:36:21 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AEV4B26678 for ; Tue, 10 Jun 2003 10:31:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02984 for ; Tue, 10 Jun 2003 10:30:58 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Pk7M-0006sb-00 for rddp@ietf.org; Tue, 10 Jun 2003 10:28:56 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=maho3msx2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19Pk7L-0006sM-00 for rddp@ietf.org; Tue, 10 Jun 2003 10:28:56 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 10 Jun 2003 10:30:23 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CD2E@corpmx14.corp.emc.com> To: rddp@ietf.org Date: Tue, 10 Jun 2003 10:22:18 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] CRC coverage (B) consensus call Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin Bestler recently wrote: The idea of shortening the CRC to cover only the header, as opposed to omitting it, was rejected because we considered it inappropriate for MPA to know how long the RDMAP/DDP Header was. That was a statement of the design rationale for MPA. However, it's strongly related to open issue (B) on which I could not call consensus previously: (B) CRC usage is all-or-nothing on header and data. There is no requirement for CRC on header but not on data or CRC on data but not on header. Based on subsequent email discussion, I believe that RDDP WG rough consensus now exists for (B). I'm calling this consensus over Somesh Gupta's objections; anyone else who objects should post to the mailing list. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 10 12:03:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06790 for ; Tue, 10 Jun 2003 12:03:35 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5AG38M02456 for rddp-archive@odin.ietf.org; Tue, 10 Jun 2003 12:03:08 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AG38B02453 for ; Tue, 10 Jun 2003 12:03:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06747 for ; Tue, 10 Jun 2003 12:03:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PlYV-0007id-00 for rddp-web-archive@ietf.org; Tue, 10 Jun 2003 12:01:03 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19PlYV-0007ia-00 for rddp-web-archive@ietf.org; Tue, 10 Jun 2003 12:01:03 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AG2FB02252; Tue, 10 Jun 2003 12:02:15 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5AFw6B01796 for ; Tue, 10 Jun 2003 11:58:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA06594 for ; Tue, 10 Jun 2003 11:58:03 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19PlTd-0007gc-00 for rddp@ietf.org; Tue, 10 Jun 2003 11:56:01 -0400 Received: from [168.159.2.8] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19PlTc-0007gZ-00 for rddp@ietf.org; Tue, 10 Jun 2003 11:56:01 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h5AFw0q24799 for ; Tue, 10 Jun 2003 11:58:00 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 10 Jun 2003 11:57:55 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CD34@corpmx14.corp.emc.com> To: rddp@ietf.org Date: Tue, 10 Jun 2003 11:49:47 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Vienna meetings Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Here are the current time slots scheduled for rddp WG meetings in Vienna. I want to remind everyone that these are SUBJECT TO CHANGE until the IETF agenda is finalized (should happen in the first part of the week of June 23rd). TUESDAY, July 15, 2003 1700-1800 Afternoon Sessions IV APP calsch Calendaring and Scheduling WG OPS v6ops IPv6 Operations WG RTG isis IS-IS for IP Internets WG SEC krb-wg Kerberos WG TSV rddp Remote Direct Data Placement WG TSV trigtran Triggers for Transport BOF THURSDAY, July 17, 2003 1530-1730 Afternoon Sessions II OPS v6ops IPv6 Operations WG RTG vrrp Virtual Router Redundancy Protocol WG SEC saag Open Security Area Directorate TSV rddp Remote Direct Data Placement WG As things currently stand, the rddp/saag conflict will result in taking up security issues during the Tuesday session. TCP mapping issues will probably be on Thursday, and other topics could be in either session. I expect to assemble the WB meeting agendas during the week of June 23rd. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Jun 11 16:52:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA02385 for ; Wed, 11 Jun 2003 16:52:06 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5BKpdP02870 for rddp-archive@odin.ietf.org; Wed, 11 Jun 2003 16:51:39 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5BKpdm02867 for ; Wed, 11 Jun 2003 16:51:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA02360 for ; Wed, 11 Jun 2003 16:51:36 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QCXE-0003lX-00 for rddp-web-archive@ietf.org; Wed, 11 Jun 2003 16:49:32 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19QCXD-0003lT-00 for rddp-web-archive@ietf.org; Wed, 11 Jun 2003 16:49:31 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5BKoIm02710; Wed, 11 Jun 2003 16:50:18 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5BKjmm02412 for ; Wed, 11 Jun 2003 16:45:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA02018 for ; Wed, 11 Jun 2003 16:45:44 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QCRZ-0003gr-00 for rddp@ietf.org; Wed, 11 Jun 2003 16:43:41 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19QCRY-0003gn-00 for rddp@ietf.org; Wed, 11 Jun 2003 16:43:40 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 2686116EFA; Wed, 11 Jun 2003 14:45:45 -0600 (MDT) Received: from axcsbh3.cos.agilent.com (axcsbh3.cos.agilent.com [130.29.152.190]) by relcos1.cos.agilent.com (Postfix) with SMTP id DD0655F2; Wed, 11 Jun 2003 14:45:44 -0600 (MDT) Received: from 130.29.152.190 by axcsbh3.cos.agilent.com (InterScan E-Mail VirusWall NT); Wed, 11 Jun 2003 14:45:44 -0600 Received: by axcsbh3.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Wed, 11 Jun 2003 14:45:44 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00131CA61@axcs04.cos.agilent.com> To: uri@broadcom.com, Jim.Williams@emulex.com, rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update Date: Wed, 11 Jun 2003 14:45:41 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, I can say the almost the same thing about CRC calculation (even in software). A number of algorithms that are out there are not highly optimized. With a good algorithm, it shouldn't be taking 10% of processing. For instance, we have found that we cut processing time in half if we went from the traditional byte at a time software CRC calculation to our algorithm which handles a 32-bit word at a time: Loop for each data word XOR current CRC result with next data word. Do a table lookup for each byte of the data word. (4 tables are used; one for each byte position) XOR the values from the tables to get the new CRC result. This gets rid of all the shifts and half the XORs that the other method uses. I can go into more detail, but it is a bit of a pain to show the math in flat ASCII. Regards, Pat -----Original Message----- From: Williams, Jim [mailto:Jim.Williams@emulex.com] Sent: Tuesday, June 03, 2003 7:43 AM To: rddp@ietf.org Subject: RE: [rddp] CRC - rough consensus update > If the _option_ to turn it off is there, it will forever add a > negotiation which slows down the initialization and adds complexity to > implementations. It will be yet another knob the administrator has to > deal with and no one showed there is a HUGE performance hit > > Uri The complexity and performance burden of the option should be somewhere between negligible and zero if done right. Implementations are free to always use CRC and always require remote end to use CRC if the implementers wish. The knob only exists if implementers feel there is demand for it. The administrator only needs to know about the knob if he wants to use it. The pragmatic issue is that a customer may come to us and say, "We haven't seen a TCP checksum error in years, but we are wasting 10% of our compute resources on CRCs. Can you turn off the CRCs for us?" We will of course say yes, regardless of whether the standard allows it or not. But I would rather have a mechanism to do this defined within the standard than have to invent an ad hoc non-standard mechanism. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Jun 12 14:52:27 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22805 for ; Thu, 12 Jun 2003 14:52:27 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5CIq0Z16311 for rddp-archive@odin.ietf.org; Thu, 12 Jun 2003 14:52:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5CIq0m16308 for ; Thu, 12 Jun 2003 14:52:00 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22780 for ; Thu, 12 Jun 2003 14:51:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QX8w-0005HL-00 for rddp-web-archive@ietf.org; Thu, 12 Jun 2003 14:49:50 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19QX8w-0005HI-00 for rddp-web-archive@ietf.org; Thu, 12 Jun 2003 14:49:50 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5CHmba11819; Thu, 12 Jun 2003 13:48:37 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5CHKvm08917 for ; Thu, 12 Jun 2003 13:20:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA19618 for ; Thu, 12 Jun 2003 13:20:53 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QViq-0004Rb-00 for rddp@ietf.org; Thu, 12 Jun 2003 13:18:48 -0400 Received: from [128.221.31.40] (helo=mxic2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19QViq-0004Qy-00 for rddp@ietf.org; Thu, 12 Jun 2003 13:18:48 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Thu, 12 Jun 2003 13:20:17 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CD6D@corpmx14.corp.emc.com> To: rddp@ietf.org Date: Thu, 12 Jun 2003 13:12:08 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] New Milestones Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The rechartering process for the RDDP WG has completed, and there is now a revised set of milestones on the WG web page: http://www.ietf.org/html.charters/rddp-charter.html To a first approximation, completion of major work items has moved out by about one IETF meeting. This is not a license to slack off - the ADs like WGs that deliver things in advance of their promised dates. Thanks, --David (RDDP WG chair) ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 13 08:27:32 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA00123 for ; Fri, 13 Jun 2003 08:27:31 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5DCR3704596 for rddp-archive@odin.ietf.org; Fri, 13 Jun 2003 08:27:03 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DCR3m04593 for ; Fri, 13 Jun 2003 08:27:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA00120 for ; Fri, 13 Jun 2003 08:27:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Qnbw-000420-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 08:24:52 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Qnbw-00041w-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 08:24:52 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5CJQ1a18354; Thu, 12 Jun 2003 15:26:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5CJP7m18324 for ; Thu, 12 Jun 2003 15:25:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA24777 for ; Thu, 12 Jun 2003 15:25:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QXf1-0005Tt-00 for rddp@ietf.org; Thu, 12 Jun 2003 15:22:59 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19QXf0-0005Tc-00 for rddp@ietf.org; Thu, 12 Jun 2003 15:22:58 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h5CJOTLW005823 for ; Thu, 12 Jun 2003 12:24:29 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Thu, 12 Jun 2003 15:24:28 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF289B90A9@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Date: Thu, 12 Jun 2003 15:24:27 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I strongly agree with the sentiments from Caitlin below. I have great concerns about that model that requires the ULP to set up an LLP connection, optionally negotiate RDDP, and then change connection mode to RDDP. My concern is about how the lines of responsibility are drawn between the host OS vendor, the NIC vendor, and the ULP developer. If the ULP developer ends up with a number of different APIs equal to the product of the number of NIC vendors supported and the number of OS's supported, this becomes unworkable. Likwise, the NIC vendors need to be able to treat host OS's as closed black boxes. Otherwise schedules become open ended and impossible to depend on. Also costs to port to new OS's become prohibitive. There may be a way to resolve these issues, but the complexity of the verbs draft, the close coupling with the host OS, and the need for ULP vendors to work with existing host OS networking stacks leaves me pessimistic. The idea of a transport neutral API that can be used for RDMA avoids most of these problems. NIC vendors provide the entire API, and ULP implementors write to it. The other important piece is a software implementation of RDMA that allows RDMA aware ULPs to use the RDMA API, but communicate through the host OS network stack, and do so with no loss (or minimal) loss of efficiency of what could be done if the ULP used the host OS network stack directly in non-RDMA mode. > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Saturday, June 07, 2003 10:15 PM > To: John Hufferd; rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > > An application which forms a TCP connection, and *then* > decides whether or not to enable RDMA mode, obviously > must be aware that it is using TCP. > > Indeed, for reasons I'll go into in a moment, that model > does not work well for SCTP. > > The model I was working from was one where the use of > RDMA is presumed in advance. The ULP is RDMA specific. > Their may be a non-RDMA equivalent, but it is running > on a different port (or perhaps using a different IP > address). > > Not surprisingly this is the model that has dominated > discussions within both the DAT Collaborative and > the OpenGroup's ICSC. Their objective, after all, is > to define transport neutral APIs for RDMA-aware > applications. > > So to rephrase more precisely, it should be possible > for a transport neutral application to communicate > the Private Date required to select a Protection Domain > and configure the RDMA Read credits in a single message > sent *before* the RDMA endpoint is selected -- and to > do so in a transport neutral way. > > To achieve this goal, either InfiniBand and SCTP must > simulate a "stream" of pre-RDMA endpoint data, or TCP > must provide a method of sending "one message". > > I believe the best way to do so is to include a variable > length Private Data field in the MPA Mode frame. The > ability to send one Private Data message before selecting > the RDMA endpoint is already present in InfiniBand and > is easily defined for SCTP. > > Alternatively, a protocol could be defined to delimit such > a message over TCP independently of the shift into MPA mode. > Either method works. > > In either case, the procedure is not imposed on any > applications. Those that want to establish a TCP connection > and then dynamically decide to enter RDMA mode are free to > do so. If a Private Data field is defined, there will be > no requirement to use it, and no prohibition against > negotiating in plain TCP before its use. > > But there are a larger group of ULPs that will be RDMA > specific from the very beginning. Indeed, for some it would > make no sense to have a non-RDMA version. Would you run > SDP over plain TCP? > > Even with less extreme cases, optimum use of RDMA is best > achieved when the ULP is fully aware of RDMA's capabilities. > While there will undoubtedly be servers that have valid > reasons for dynamically deciding whether to enable RDMA, > there will be at least as large a group that prefer to > run the two distinctly different sets of logic through > different daemons running on different ports. The server > advertises two different sevices (whether by SLP, in > the URL, or by whatever method), and the client chooses > which to invoke. > > > > There is an issue on the dynamic "in-session" switch to > RDMA model as applied to SCTP. The current SCTP draft > will not support such a model. > > The rationale is that sharing control of streams within an > single SCTP association between a non-RDMA stack and a RDMA > stack would be very difficult. A likely adoption strategy > would have the RDMA stack implemented on an RNIC, with the > non-RDMA stack remaining "as is" on the processor. > > Segregating SCTP associations between the > RDMA-implementation and the current stack allows RDMA to be > deployed without disrupting current stacks. > > This same issue exists with TCP connections. But TCP only > requires a relatively clean handoff from the conventional > stack to the MPA/iWARP stack (relatively clean, because > responsiblity for retransmitting might be split right at > the shift point). > > An SCTP implementation cannot shift just one Stream. > Then handling of TSNs, SACKs and all other control chunks > is common across all streams. > > So is the correct response: > > a) Don't worry about it. The "dynamic in-session" model is > highly specialized and inherently related to existing > servers that deal with long-haul connections where the > clients have little a priori knowledge of the server. > There is little need to support this model for SCTP. > b) Require an iWARP capable SCTP stack to also support > non-iWARP streams. The trick is that non-iWARP streams > can have *ordered* Data Chunks. The handling of which > is both more complex and more consuming of memory > bandwidth than for an iWARP stream (which only uses > unordered data chunks). > > If dynamic in-session switching is a general model that > should be supported in an LLP fashion then the SCTP draft > should find a way to accomodate it -- preferably in a way > that doesn't make an implemetnation that much more > difficult. > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 13 12:35:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA11362 for ; Fri, 13 Jun 2003 12:35:22 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5DGYtb27366 for rddp-archive@odin.ietf.org; Fri, 13 Jun 2003 12:34:55 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DGYsm27363 for ; Fri, 13 Jun 2003 12:34:54 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA11336 for ; Fri, 13 Jun 2003 12:34:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QrTo-0006XR-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 12:32:44 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19QrTn-0006XO-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 12:32:43 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DBq5a02005; Fri, 13 Jun 2003 07:52:05 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DBpVm01941 for ; Fri, 13 Jun 2003 07:51:31 -0400 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA29125; Fri, 13 Jun 2003 07:51:30 -0400 (EDT) Message-Id: <200306131151.HAA29125@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Fri, 13 Jun 2003 07:51:30 -0400 Subject: [rddp] I-D ACTION:draft-ietf-rddp-applicability-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : Applicability of Remote Direct Memory Access Protocol (RDMA)and Direct Data Placement (DDP) Author(s) : C. Bestler, L. Coene Filename : draft-ietf-rddp-applicability-00.txt Pages : 21 Date : 2003-6-12 This document describes the applicability of Remote Direct Memory Access Protocol (RDMAP) and the Direct Data Placement Protocol (DDP). It contrasts the different transport options over IP that DDP can use, compares use of DDP with direct use of the supporting transports, and compares DDP over IP transports with non-IP transports that support RDMA functionality. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-applicability-00.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-applicability-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-applicability-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-6-12134007.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-applicability-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-applicability-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-6-12134007.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 13 13:17:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13748 for ; Fri, 13 Jun 2003 13:17:39 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5DHHCH12157 for rddp-archive@odin.ietf.org; Fri, 13 Jun 2003 13:17:12 -0400 Received: from ietf.org (lists.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DHHBm11292 for ; Fri, 13 Jun 2003 13:17:11 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13637 for ; Fri, 13 Jun 2003 13:17:07 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Qs8h-00076n-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 13:15:00 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Qs8h-00076k-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 13:14:59 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DH31a07074; Fri, 13 Jun 2003 13:03:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DH2Gm06594 for ; Fri, 13 Jun 2003 13:02:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA12904 for ; Fri, 13 Jun 2003 13:02:13 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19QruH-0006sE-00 for rddp@ietf.org; Fri, 13 Jun 2003 13:00:05 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19QruG-0006rJ-00 for rddp@ietf.org; Fri, 13 Jun 2003 13:00:05 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0453B24536A for ; Fri, 13 Jun 2003 13:02:11 -0400 (EDT) Date: Fri, 13 Jun 2003 12:01:36 -0500 From: Caitlin Bestler Subject: Re: [rddp] I-D ACTION:draft-ietf-rddp-applicability-00.txt To: rddp@ietf.org X-Priority: 3 In-Reply-To: <200306131151.HAA29125@ietf.org> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I'd like to provide a quick summary of the changes. There are numerous scattered language clarifications that do not represent new content, but should express the original intent more clearly. An additional author, Lode Coene, who can be credited with most of the above clarifications. Some more specific substantive changes of note: - RDMAP and DDP are treated more distinctly, clarifying the role of each in providig services to the ULP. - A discussion of ULPs that could actually be hurt by RDMAPs consolidation of completion notifications. - Added comparison with "parse and place" where the network stack is aware of the ULP protocol. - Added discussion on header overhead in comparing transports. - Added discussion on data integrity, and that it is NOT something that requires the ULP to be LLP aware. - Added discussion of security implications of having multiple STags enabled for a given connection. - Discussed the inherent link between RDMAP and reliable transports. - Pointed out that both TCP and SCTP mappings allow existing connection/association establishment procedures to be followed for security purposes. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 13 20:42:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00069 for ; Fri, 13 Jun 2003 20:42:52 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5E0gOZ06699 for rddp-archive@odin.ietf.org; Fri, 13 Jun 2003 20:42:24 -0400 Received: from ietf.org (lists.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5E0gNm06696 for ; Fri, 13 Jun 2003 20:42:23 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00060 for ; Fri, 13 Jun 2003 20:42:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Qz5U-0002kc-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 20:40:08 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Qz5U-0002kZ-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 20:40:08 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DNI0a32751; Fri, 13 Jun 2003 19:18:00 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DNH5m32686 for ; Fri, 13 Jun 2003 19:17:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA28114 for ; Fri, 13 Jun 2003 19:17:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Qxl1-0002F9-00 for rddp@ietf.org; Fri, 13 Jun 2003 19:14:55 -0400 Received: from sandmail.sandburst.com ([216.57.132.42]) by ietf-mx with esmtp (Exim 4.12) id 19Qxl0-0002F5-00 for rddp@ietf.org; Fri, 13 Jun 2003 19:14:54 -0400 To: Caitlin Bestler Cc: John Hufferd , rddp@ietf.org Subject: Re: [rddp] Transport Neutral "Connections" In-Reply-To: Message from Caitlin Bestler of "Sat, 07 Jun 2003 21:14:55 CDT." References: Date: Fri, 13 Jun 2003 19:16:33 -0400 From: Stephen Bailey Message-Id: <20030613231632.A2EB64E82@sandmail.sandburst.com> Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > If dynamic in-session switching is a general model that > should be supported in an LLP fashion then the SCTP draft > should find a way to accomodate it -- preferably in a way > that doesn't make an implemetnation that much more > difficult. I believe that would be wise. SCTP will also have protocols that you may want to run with or without RDDP. It seems pretty clear (from other examples, e.g. SSL) that allowing in-band negotiation our best bet to make RDDP easy to use. It may cause a touch of additional implementation complexity, but that'll be a drop in the bucket compared to the total complexity of any RDDP implementation. Steph _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Jun 13 23:50:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04671 for ; Fri, 13 Jun 2003 23:50:46 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5E3oIi18184 for rddp-archive@odin.ietf.org; Fri, 13 Jun 2003 23:50:18 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5E3oCm18181 for ; Fri, 13 Jun 2003 23:50:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04666 for ; Fri, 13 Jun 2003 23:50:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19R21G-0003rR-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 23:47:58 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19R21F-0003rO-00 for rddp-web-archive@ietf.org; Fri, 13 Jun 2003 23:47:57 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DNh1a02371; Fri, 13 Jun 2003 19:43:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5DNgmm02352 for ; Fri, 13 Jun 2003 19:42:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA28584 for ; Fri, 13 Jun 2003 19:42:45 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Qy9u-0002Ml-00 for rddp@ietf.org; Fri, 13 Jun 2003 19:40:38 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Qy9t-0002MV-00 for rddp@ietf.org; Fri, 13 Jun 2003 19:40:37 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 90A54245050; Fri, 13 Jun 2003 19:42:49 -0400 (EDT) Date: Fri, 13 Jun 2003 18:42:15 -0500 From: Caitlin Bestler Subject: Re: [rddp] Transport Neutral "Connections" To: Stephen Bailey , rddp@ietf.org X-Priority: 3 In-Reply-To: <20030613231632.A2EB64E82@sandmail.sandburst.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/13/03, Stephen Bailey wrote: >> If dynamic in-session switching is a general model that >> should be supported in an LLP fashion then the SCTP draft >> should find a way to accomodate it -- preferably in a way >> that doesn't make an implemetnation that much more >> difficult. > >I believe that would be wise. > >SCTP will also have protocols that you may want to run >with or without RDDP. It seems pretty clear (from other >examples, e.g. SSL) that allowing in-band negotiation our >best bet to make RDDP easy to use. It may cause a touch of >additional implementation complexity, but that'll be a >drop in the bucket compared to the total complexity of any >RDDP implementation. > >Steph There's actually two approaches that are theoretically simple, but I fear are quite disruptive. a) Define a range of Payload Protocol Identifiers which will be passed through the DDP stack to a non-RDDP SCTP stack. This stack could also originate said Data Chunks with those Payload Protocol Identifiers. Very simple, conceptually, except for the fact that its an interface that no current SCTP stack has -- and otherwise the adaptation runs very cleanly on top of existing SCTP stacks (more so than for MPA/TCP). b) Simply mandate that an SCTP Stream pair be reassingable to a new DDP Endpoint at any time. The "pre" negotiations could be done with untagged messages and one Protection Domain (with no STags enabled). When full RDMA capabilities are enabled the stream can be handed off to a new Endpoint with the negotiated credentials. Again it is very simple conceptually, except that nobody makes endpoints like that. For InfiniBand, DAT, the IT API and RDMAC proposed iWARP verbs an endpoint (QP) is created *within* a Protection Domain. That's why I'm still favor the single-message model. Of course I'm more than willing to update the draft to match whatever the consensus favors. The ultimate goal is enable a transport neutral API that works over both MPA/TCP and SCTP (and InfiniBand would be nice too) that does not lower them all to the lowest common denominator. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 14 04:38:10 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA21714 for ; Sat, 14 Jun 2003 04:38:10 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5E8bgF16892 for rddp-archive@odin.ietf.org; Sat, 14 Jun 2003 04:37:42 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5E8bgm16889 for ; Sat, 14 Jun 2003 04:37:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id EAA21701 for ; Sat, 14 Jun 2003 04:37:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19R6VW-0005Oy-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 04:35:30 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19R6VW-0005Ov-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 04:35:30 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5E6f3a09328; Sat, 14 Jun 2003 02:41:03 -0400 Received: from ietf.org (lists.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5E6eMm09312 for ; Sat, 14 Jun 2003 02:40:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA20083; Sat, 14 Jun 2003 02:40:19 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19R4fx-00051o-00; Sat, 14 Jun 2003 02:38:09 -0400 Received: from e32.co.us.ibm.com ([32.97.110.130]) by ietf-mx with esmtp (Exim 4.12) id 19R4fv-000518-00; Sat, 14 Jun 2003 02:38:07 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e32.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h5E6d2r9265374; Sat, 14 Jun 2003 02:39:02 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h5E6d00l169256; Sat, 14 Jun 2003 00:39:01 -0600 To: "Williams, Jim" Cc: rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Fri, 13 Jun 2003 23:38:56 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 9, 2003) at 06/14/2003 00:39:00, Serialize complete at 06/14/2003 00:39:00 Content-Type: multipart/alternative; boundary="=_alternative 0024727E88256D45_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 0024727E88256D45_= Content-Type: text/plain; charset="us-ascii" Jim, I am not sure of what your statement would prevent, but as I mentioned to Caitlin, and she agreed, we have important applications that MUST first initiate in LLP mode, and after finishing with negotiations may then switch into RDMA mode. So I wonder if the thing you talk about in the first part of your note needs to be possible anyway, I am not sure that it is an issue. Caitlin's point, as I understand it, is not to prevent that type of operation, but instead, permit applications that KNOW they are going to enter RDMA mode, to be able to start out that way, and as a result be independent of the underpinning LLP. Now it is possible that I misunderstood your point, but if not, these are two different needs. Not one versus the other. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com "Williams, Jim" Sent by: rddp-admin@ietf.org 06/12/2003 12:24 PM To: rddp@ietf.org cc: Subject: RE: [rddp] Transport Neutral "Connections" I strongly agree with the sentiments from Caitlin below. I have great concerns about that model that requires the ULP to set up an LLP connection, optionally negotiate RDDP, and then change connection mode to RDDP. My concern is about how the lines of responsibility are drawn between the host OS vendor, the NIC vendor, and the ULP developer. If the ULP developer ends up with a number of different APIs equal to the product of the number of NIC vendors supported and the number of OS's supported, this becomes unworkable. Likwise, the NIC vendors need to be able to treat host OS's as closed black boxes. Otherwise schedules become open ended and impossible to depend on. Also costs to port to new OS's become prohibitive. There may be a way to resolve these issues, but the complexity of the verbs draft, the close coupling with the host OS, and the need for ULP vendors to work with existing host OS networking stacks leaves me pessimistic. The idea of a transport neutral API that can be used for RDMA avoids most of these problems. NIC vendors provide the entire API, and ULP implementors write to it. The other important piece is a software implementation of RDMA that allows RDMA aware ULPs to use the RDMA API, but communicate through the host OS network stack, and do so with no loss (or minimal) loss of efficiency of what could be done if the ULP used the host OS network stack directly in non-RDMA mode. > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Saturday, June 07, 2003 10:15 PM > To: John Hufferd; rddp@ietf.org > Subject: RE: [rddp] Transport Neutral "Connections" > > > > An application which forms a TCP connection, and *then* > decides whether or not to enable RDMA mode, obviously > must be aware that it is using TCP. > > Indeed, for reasons I'll go into in a moment, that model > does not work well for SCTP. > > The model I was working from was one where the use of > RDMA is presumed in advance. The ULP is RDMA specific. > Their may be a non-RDMA equivalent, but it is running > on a different port (or perhaps using a different IP > address). > > Not surprisingly this is the model that has dominated > discussions within both the DAT Collaborative and > the OpenGroup's ICSC. Their objective, after all, is > to define transport neutral APIs for RDMA-aware > applications. > > So to rephrase more precisely, it should be possible > for a transport neutral application to communicate > the Private Date required to select a Protection Domain > and configure the RDMA Read credits in a single message > sent *before* the RDMA endpoint is selected -- and to > do so in a transport neutral way. > > To achieve this goal, either InfiniBand and SCTP must > simulate a "stream" of pre-RDMA endpoint data, or TCP > must provide a method of sending "one message". > > I believe the best way to do so is to include a variable > length Private Data field in the MPA Mode frame. The > ability to send one Private Data message before selecting > the RDMA endpoint is already present in InfiniBand and > is easily defined for SCTP. > > Alternatively, a protocol could be defined to delimit such > a message over TCP independently of the shift into MPA mode. > Either method works. > > In either case, the procedure is not imposed on any > applications. Those that want to establish a TCP connection > and then dynamically decide to enter RDMA mode are free to > do so. If a Private Data field is defined, there will be > no requirement to use it, and no prohibition against > negotiating in plain TCP before its use. > > But there are a larger group of ULPs that will be RDMA > specific from the very beginning. Indeed, for some it would > make no sense to have a non-RDMA version. Would you run > SDP over plain TCP? > > Even with less extreme cases, optimum use of RDMA is best > achieved when the ULP is fully aware of RDMA's capabilities. > While there will undoubtedly be servers that have valid > reasons for dynamically deciding whether to enable RDMA, > there will be at least as large a group that prefer to > run the two distinctly different sets of logic through > different daemons running on different ports. The server > advertises two different sevices (whether by SLP, in > the URL, or by whatever method), and the client chooses > which to invoke. > > > > There is an issue on the dynamic "in-session" switch to > RDMA model as applied to SCTP. The current SCTP draft > will not support such a model. > > The rationale is that sharing control of streams within an > single SCTP association between a non-RDMA stack and a RDMA > stack would be very difficult. A likely adoption strategy > would have the RDMA stack implemented on an RNIC, with the > non-RDMA stack remaining "as is" on the processor. > > Segregating SCTP associations between the > RDMA-implementation and the current stack allows RDMA to be > deployed without disrupting current stacks. > > This same issue exists with TCP connections. But TCP only > requires a relatively clean handoff from the conventional > stack to the MPA/iWARP stack (relatively clean, because > responsiblity for retransmitting might be split right at > the shift point). > > An SCTP implementation cannot shift just one Stream. > Then handling of TSNs, SACKs and all other control chunks > is common across all streams. > > So is the correct response: > > a) Don't worry about it. The "dynamic in-session" model is > highly specialized and inherently related to existing > servers that deal with long-haul connections where the > clients have little a priori knowledge of the server. > There is little need to support this model for SCTP. > b) Require an iWARP capable SCTP stack to also support > non-iWARP streams. The trick is that non-iWARP streams > can have *ordered* Data Chunks. The handling of which > is both more complex and more consuming of memory > bandwidth than for an iWARP stream (which only uses > unordered data chunks). > > If dynamic in-session switching is a general model that > should be supported in an LLP fashion then the SCTP draft > should find a way to accomodate it -- preferably in a way > that doesn't make an implemetnation that much more > difficult. > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp --=_alternative 0024727E88256D45_= Content-Type: text/html; charset="us-ascii"
Jim,
I am not sure of what your statement would prevent, but as I mentioned to Caitlin, and she agreed, we have important applications that MUST first initiate in LLP mode, and after finishing with negotiations may then switch into RDMA mode.  So I wonder if the thing you talk about in the first part of your note needs to be possible anyway,  I am not sure that it is an issue.  

Caitlin's point, as I understand it, is not to prevent that type of operation, but instead, permit applications that KNOW they are going to enter RDMA mode, to be able to start out that way, and as a result be independent of the underpinning LLP.  

Now it is possible that I misunderstood your point, but if not,
these are two different needs.  Not one versus the other.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


"Williams, Jim" <Jim.Williams@emulex.com>
Sent by: rddp-admin@ietf.org

06/12/2003 12:24 PM

       
        To:        rddp@ietf.org
        cc:        
        Subject:        RE: [rddp] Transport Neutral "Connections"



I strongly agree with the sentiments
from Caitlin below.

I have great concerns about that model that requires
the ULP to set up an LLP connection, optionally
negotiate RDDP, and then change connection mode to
RDDP.

My concern is about how the lines of responsibility
are drawn between the host OS vendor,
the NIC vendor, and the ULP developer.

If the ULP developer ends up with a number of
different APIs equal to the product of the number
of NIC vendors supported and the number of OS's supported,
this becomes unworkable.

Likwise, the NIC vendors need to be able to treat
host OS's as closed black boxes.  Otherwise
schedules become open ended and impossible
to depend on.  Also costs to port to new
OS's become prohibitive.

There may be a way to resolve these issues, but
the complexity of the verbs draft, the close
coupling with the host OS, and the need for
ULP vendors to work with existing host OS
networking stacks leaves me pessimistic.

The idea of a transport neutral API that
can be used for RDMA avoids most of these problems.
NIC vendors provide the entire API, and ULP
implementors write to it.

The other important piece is a software implementation
of RDMA that allows RDMA aware ULPs to use the RDMA API,
but communicate through the host OS network stack,
and do so with no loss (or minimal) loss of efficiency
of what could be done if the ULP used the host
OS network stack directly in non-RDMA mode.


> -----Original Message-----
> From: Caitlin Bestler [mailto:cait@asomi.com]
> Sent: Saturday, June 07, 2003 10:15 PM
> To: John Hufferd; rddp@ietf.org
> Subject: RE: [rddp] Transport Neutral "Connections"
>
>
>
> An application which forms a TCP connection, and *then*
> decides whether or not to enable RDMA mode, obviously
> must be aware that it is using TCP.
>
> Indeed, for reasons I'll go into in a moment, that model
> does not work well for SCTP.
>
> The model I was working from was one where the use of
> RDMA is presumed in advance. The ULP is RDMA specific.
> Their may be a non-RDMA equivalent, but it is running
> on a different port (or perhaps using a different IP
> address).
>
> Not surprisingly this is the model that has dominated
> discussions within both the DAT Collaborative and
> the OpenGroup's ICSC. Their objective, after all, is
> to define transport neutral APIs for RDMA-aware
> applications.
>
> So to rephrase more precisely, it should be possible
> for a transport neutral application to communicate
> the Private Date required to select a Protection Domain
> and configure the RDMA Read credits in a single message
> sent *before* the RDMA endpoint is selected -- and to
> do so in a transport neutral way.
>
> To achieve this goal, either InfiniBand and SCTP must
> simulate a "stream" of pre-RDMA endpoint data, or TCP
> must provide a method of sending "one message".
>
> I believe the best way to do so is to include a variable
> length Private Data field in the MPA Mode frame. The
> ability to send one Private Data message before selecting
> the RDMA endpoint is already present in InfiniBand and
> is easily defined for SCTP.
>
> Alternatively, a protocol could be defined to delimit such
> a message over TCP independently of the shift into MPA mode.
> Either method  works.
>
> In either case, the procedure is not imposed on any
> applications. Those that want to establish a TCP connection
> and then dynamically decide to enter RDMA mode are free to
> do so. If a Private Data field is defined, there will be
> no requirement to use it, and no prohibition against
> negotiating in plain TCP before its use.
>
> But there are a larger group of ULPs that will be RDMA
> specific from the very beginning. Indeed, for some it would
> make no sense to have a non-RDMA version. Would you run
> SDP over plain TCP?
>
> Even with less extreme cases, optimum use of RDMA is best
> achieved when the ULP is fully aware of RDMA's capabilities.
> While there will undoubtedly be servers that have valid
> reasons for dynamically deciding whether to enable RDMA,
> there will be at least as large a group that prefer to
> run the two distinctly different sets of logic through
> different daemons running on different ports. The server
> advertises two different sevices (whether by SLP, in
> the URL, or by whatever method), and the client chooses
> which to invoke.
>
>
>
> There is an issue on the dynamic "in-session" switch to
> RDMA model as applied to SCTP. The current SCTP draft
> will not support such a model.
>
> The rationale is that sharing control of streams within an
> single SCTP association between a non-RDMA stack and a RDMA
> stack would be very difficult. A likely adoption strategy
> would have the RDMA stack implemented on an RNIC, with the
> non-RDMA stack remaining "as is" on the processor.
>
> Segregating SCTP associations between the
> RDMA-implementation and the current stack allows RDMA to be
> deployed without disrupting current stacks.
>
> This same issue exists with TCP connections. But TCP only
> requires a relatively clean handoff from the conventional
> stack to the MPA/iWARP stack (relatively clean, because
> responsiblity for retransmitting might be split right at
> the shift point).
>
> An SCTP implementation cannot shift just one Stream.
> Then handling of TSNs, SACKs and all other control chunks
> is common across all streams.
>
> So is the correct response:
>
> a) Don't worry about it. The "dynamic in-session" model is
>    highly specialized and inherently related to existing
>    servers that deal with long-haul connections where the
>    clients have little a priori knowledge of the server.
>    There is little need to support this model for SCTP.
> b) Require an iWARP capable SCTP stack to also support
>    non-iWARP streams. The trick is that non-iWARP streams
>    can have *ordered* Data Chunks. The handling of which
>    is both more complex and more consuming of memory
>    bandwidth than for an iWARP stream (which only uses
>    unordered data chunks).
>    
> If dynamic in-session switching is a general model that
> should be supported in an LLP fashion then the SCTP draft
> should find a way to accomodate it -- preferably in a way
> that doesn't make an implemetnation that much more
> difficult.
>
>
>
>
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> _______________________________________________
> rddp mailing list
> rddp@ietf.org
> https://www1.ietf.org/mailman/listinfo/rddp
>
_______________________________________________
rddp mailing list
rddp@ietf.org
https://www1.ietf.org/mailman/listinfo/rddp

--=_alternative 0024727E88256D45_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 14 12:14:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28357 for ; Sat, 14 Jun 2003 12:14:03 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5EGDaq10233 for rddp-archive@odin.ietf.org; Sat, 14 Jun 2003 12:13:36 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EGDam10230 for ; Sat, 14 Jun 2003 12:13:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA28340 for ; Sat, 14 Jun 2003 12:13:28 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RDcc-0006kt-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 12:11:18 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19RDcc-0006km-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 12:11:18 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EEZ1a03885; Sat, 14 Jun 2003 10:35:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EEYKm03836 for ; Sat, 14 Jun 2003 10:34:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26877; Sat, 14 Jun 2003 10:34:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RC4d-0006Pf-00; Sat, 14 Jun 2003 10:32:07 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19RC4c-0006PW-00; Sat, 14 Jun 2003 10:32:06 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id CBEBB244B16; Sat, 14 Jun 2003 10:33:52 -0400 (EDT) Date: Sat, 14 Jun 2003 09:33:19 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: John Hufferd Cc: "Williams, Jim" , rddp@ietf.org, rddp-admin@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/13/03, John Hufferd wrote: >Jim, >I am not sure of what your statement would prevent, but as >I mentioned to Caitlin, and she agreed, we have important >applications that MUST first initiate in LLP mode, and >after finishing with negotiations may then switch into >RDMA mode. So I wonder if the thing you talk about in the >first part of your note needs to be possible anyway, I am >not sure that it is an issue. > There is a great deal of difference between a model that allows *either* application model, and one that *requires* the the ULP to set up the LLP itself. I read Jim's statement as matching my concern -- the problem is that the current proposals are only friendly for LLP aware applications. Vague suggestions that some middleware can sweep everything under the rug ring very hollow with me. I have major doubts as to how a TCP stack can be an efficient TCP stack AND handle the last step of negotiating RDMA entry efficiently. Not only is the ULP using a byte *stream* for the final pre-RDMA negotiation message unnatural (and fail to match any other transport), it has the nasty effect of requiring a hybrid stack to parse a single TCP segment in two different stacks. (In particular pulling bytes from the NIC only after the ULP has actually read them is a very inefficient way to run a normal TCP stream. So if the bytes have already been flushed from the NIC "prematurely we'd have to "give them back". Neither slow-fetch or push-back are optimal or elegant solutions. And both get far uglier when you have to retransmit across a network that doesn't preserve TCP segment boundaries). That won't be reliable. What it will lead to is applications "playing it safe" by delaying MPA entry long enough for the TCP connection to have gone idle. Effectively creating the very round-trip delay that we agreed that the MPA mode frame was not supposed to require. Efficient RDMA Mode entry requires capture of the last pre-RDMA message *and* entry into MPA mode to be an atomic step that can be performed *without* the ULP being directly involved. We're supposed to be minimizing the number of required ULP interactions, right? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 14 15:17:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02753 for ; Sat, 14 Jun 2003 15:17:58 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5EJHTs20439 for rddp-archive@odin.ietf.org; Sat, 14 Jun 2003 15:17:29 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EJHTm20436 for ; Sat, 14 Jun 2003 15:17:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA02712 for ; Sat, 14 Jun 2003 15:17:27 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RGUe-0007Vl-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 15:15:16 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19RGUe-0007Vi-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 15:15:16 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EHx2a15355; Sat, 14 Jun 2003 13:59:02 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EHwhm15330 for ; Sat, 14 Jun 2003 13:58:43 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00262; Sat, 14 Jun 2003 13:58:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RFGP-0007Bk-00; Sat, 14 Jun 2003 13:56:29 -0400 Received: from e34.co.us.ibm.com ([32.97.110.132]) by ietf-mx with esmtp (Exim 4.12) id 19RFGO-0007BV-00; Sat, 14 Jun 2003 13:56:28 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e34.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h5EHw1AC260518; Sat, 14 Jun 2003 13:58:02 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h5EHw1rb120516; Sat, 14 Jun 2003 11:58:01 -0600 To: Caitlin Bestler Cc: "Williams, Jim" , rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Sat, 14 Jun 2003 10:57:55 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 9, 2003) at 06/14/2003 11:58:01, Serialize complete at 06/14/2003 11:58:01 Content-Type: multipart/alternative; boundary="=_alternative 00629DC788256D45_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 00629DC788256D45_= Content-Type: text/plain; charset="us-ascii" Caitlin, I can not determine from your note whether you are now objecting to the application being able to perform its own negotiation before entering RDMA mode (via an Atomic last message operation) as is currently defined; or are you agreeing with that, but ALSO want a LLP neutral method to enter the RDMA mode? . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler 06/14/2003 07:33 AM To: John Hufferd/San Jose/IBM@IBMUS cc: "Williams, Jim" , rddp@ietf.org, rddp-admin@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" On 6/13/03, John Hufferd wrote: >Jim, >I am not sure of what your statement would prevent, but as >I mentioned to Caitlin, and she agreed, we have important >applications that MUST first initiate in LLP mode, and >after finishing with negotiations may then switch into >RDMA mode. So I wonder if the thing you talk about in the >first part of your note needs to be possible anyway, I am >not sure that it is an issue. > There is a great deal of difference between a model that allows *either* application model, and one that *requires* the the ULP to set up the LLP itself. I read Jim's statement as matching my concern -- the problem is that the current proposals are only friendly for LLP aware applications. Vague suggestions that some middleware can sweep everything under the rug ring very hollow with me. I have major doubts as to how a TCP stack can be an efficient TCP stack AND handle the last step of negotiating RDMA entry efficiently. Not only is the ULP using a byte *stream* for the final pre-RDMA negotiation message unnatural (and fail to match any other transport), it has the nasty effect of requiring a hybrid stack to parse a single TCP segment in two different stacks. (In particular pulling bytes from the NIC only after the ULP has actually read them is a very inefficient way to run a normal TCP stream. So if the bytes have already been flushed from the NIC "prematurely we'd have to "give them back". Neither slow-fetch or push-back are optimal or elegant solutions. And both get far uglier when you have to retransmit across a network that doesn't preserve TCP segment boundaries). That won't be reliable. What it will lead to is applications "playing it safe" by delaying MPA entry long enough for the TCP connection to have gone idle. Effectively creating the very round-trip delay that we agreed that the MPA mode frame was not supposed to require. Efficient RDMA Mode entry requires capture of the last pre-RDMA message *and* entry into MPA mode to be an atomic step that can be performed *without* the ULP being directly involved. We're supposed to be minimizing the number of required ULP interactions, right? Caitlin Bestler http://asomi.com/CaitlinBestler/ --=_alternative 00629DC788256D45_= Content-Type: text/html; charset="us-ascii"
Caitlin,
I can not determine from your note whether you are now objecting to the application being able to perform its own negotiation before entering RDMA mode (via an Atomic last message operation) as is currently defined; or are you agreeing with that, but ALSO want a LLP neutral method to enter the RDMA mode?

 
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Caitlin Bestler <cait@asomi.com>

06/14/2003 07:33 AM

       
        To:        John Hufferd/San Jose/IBM@IBMUS
        cc:        "Williams, Jim" <Jim.Williams@emulex.com>, rddp@ietf.org, rddp-admin@ietf.org
        Subject:        RE: [rddp] Transport Neutral "Connections"



On 6/13/03, John Hufferd wrote:

>Jim,
>I am not sure of what your statement would prevent, but as
>I mentioned to Caitlin, and she agreed, we have important
>applications that MUST first initiate in LLP mode, and
>after finishing with negotiations may then switch into
>RDMA mode.  So I wonder if the thing you talk about in the
>first part of your note needs to be possible anyway,  I am
>not sure that it is an issue.
>

There is a great deal of difference between a model that
allows *either* application model, and one that *requires*
the the ULP to set up the LLP itself.

I read Jim's statement as matching my concern -- the
problem is that the current proposals are only friendly
for LLP aware applications.

Vague suggestions that some middleware can sweep everything
under the rug ring very hollow with me. I have major doubts
as to how a TCP stack can be an efficient TCP stack AND
handle the last step of negotiating RDMA entry efficiently.

Not only is the ULP using a byte *stream* for the
final pre-RDMA negotiation message unnatural (and fail to
match any other transport), it has the nasty effect of
requiring a hybrid stack to parse a single TCP
segment in two different stacks. (In particular pulling
bytes from the NIC only after the ULP has actually read them
is a very inefficient way to run a normal TCP stream. So if
the bytes have already been flushed from the NIC
"prematurely we'd have to "give them back". Neither
slow-fetch or push-back are optimal or elegant solutions.
And both get far uglier when you have to retransmit across a
network that doesn't preserve TCP segment boundaries).

That won't be reliable. What it will lead to is applications
"playing it safe" by delaying MPA entry long enough for
the TCP connection to have gone idle. Effectively
creating the very round-trip delay that we agreed that
the MPA mode frame was not supposed to require.

Efficient RDMA Mode entry requires capture of the last
pre-RDMA message *and* entry into MPA mode to be an
atomic step that can be performed *without* the ULP
being directly involved. We're supposed to be minimizing
the number of required ULP interactions, right?


Caitlin Bestler
http://asomi.com/CaitlinBestler/

--=_alternative 00629DC788256D45_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 14 18:38:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06862 for ; Sat, 14 Jun 2003 18:38:04 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5EMbbE00349 for rddp-archive@odin.ietf.org; Sat, 14 Jun 2003 18:37:37 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EMbbm00346 for ; Sat, 14 Jun 2003 18:37:37 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06858 for ; Sat, 14 Jun 2003 18:37:33 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RJcK-0000XU-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 18:35:24 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19RJcJ-0000XR-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 18:35:23 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EId1a18474; Sat, 14 Jun 2003 14:39:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5EIcdm18454 for ; Sat, 14 Jun 2003 14:38:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA01052 for ; Sat, 14 Jun 2003 14:38:37 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RFt4-0007N2-00 for rddp@ietf.org; Sat, 14 Jun 2003 14:36:26 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19RFt3-0007My-00 for rddp@ietf.org; Sat, 14 Jun 2003 14:36:25 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9987A244EE1; Sat, 14 Jun 2003 14:38:38 -0400 (EDT) Date: Sat, 14 Jun 2003 13:38:05 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: John Hufferd , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/14/03, John Hufferd wrote: >Caitlin, >I can not determine from your note whether you are now >objecting to the application being able to perform its own >negotiation before entering RDMA mode (via an Atomic last >message operation) as is currently defined; or are you >agreeing with that, but ALSO want a LLP neutral method to >enter the RDMA mode? The latter. But I don't view the LLP neutral as the "additional" method. It should be the primary method. Conditional negotiation is a special need for certain applications. The LLP SHOULD be invisible to the ULP. If the ULP wants to deal with the ULP, well then there certainly isn't any reason to forbid it. I just don't want to force the ULP to deal with the LLP - or even suggest that it ought to. I would prefer to state forthrightly that the ULP SHOULD NOT deal with the LLP. The current proposals for MPA do exactly that. Vague unspecified middleware to parse a "message" above the stack does not solve the problem. It is a) still the ULP, and b) unrealistically complex. This middleware would be expected to consume the *precise* number of bytes required, and leave the remaining byte sitting for transfer to the MPA stack *instantly* when the request to assign the TCP Stream to an RDMA Endpoint/QP is made. This would be true even if the MPA Mode frame had been included in the very same TCP segment as the final pre-RDMA "message". This creates a nasty problem for at least some valid implementation strategies. Assume that at least a few sites will not be eager to turn over all TCP connections to a new stack. Therefore there will be a non-MPA stack still running on the host CPU, regardless of whatever RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC. The RNIC is also the Ethernet NIC. I am going to take a bold guess here and venture that at least a few sites will wish to preserve the current IP stack. The existing IP stack is known to all of the other existing applications that don't need this "RDMA" feature yet. The same stack that has already been tested and audited with all of those applications. New Ethernet NICs are much simpler to test and validate than new IP stacks. So at least some sites will want the RNIC to present itself as a conventional NIC for the benefit of the existing IP stack. And Ethernet NICs handoff entire frames, not bytes from TCP frames as the ULP reads them from sockets. Now this handoff can be solved, but it is tricky. Involving the ULP in the middle of it will only make things far worse. "Atomic" may be too strong of a word. But I do believe that receipt of the final "negotiation" message and entry to MPA mode needs to be co-ordinated at the kernel and/or hardware layer. The ULP should only be involved for policy issues -- not the mechanics. The mechanics are messy *and* LLP specific. To encourage, not just allow, the ULP to be LLP-neutral we need to define a procedure where the ULP can exchange messages that lead to the assignment of RDMA endpoints (or QPs if you insist on callling them that, but if you think about it, they no longer have two queues). Further, certain of those messages are naturally trigger events directly related to session state changes (request, accept and terminate). Those semantic are best recognized as they are parsed, not after a bounce through the ULP that only delays things. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Jun 14 20:15:27 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA08217 for ; Sat, 14 Jun 2003 20:15:27 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5F0EwN05199 for rddp-archive@odin.ietf.org; Sat, 14 Jun 2003 20:14:58 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5F0Ewm05195 for ; Sat, 14 Jun 2003 20:14:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA08201 for ; Sat, 14 Jun 2003 20:14:56 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RL8W-0000qn-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 20:12:44 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19RL8V-0000qk-00 for rddp-web-archive@ietf.org; Sat, 14 Jun 2003 20:12:43 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5ELU1a28320; Sat, 14 Jun 2003 17:30:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5ELTNm28295 for ; Sat, 14 Jun 2003 17:29:23 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA04724 for ; Sat, 14 Jun 2003 17:29:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19RIYI-0000Hi-00 for rddp@ietf.org; Sat, 14 Jun 2003 17:27:10 -0400 Received: from e34.co.us.ibm.com ([32.97.110.132]) by ietf-mx with esmtp (Exim 4.12) id 19RIYH-0000HN-00 for rddp@ietf.org; Sat, 14 Jun 2003 17:27:09 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e34.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h5ELSnAC280648; Sat, 14 Jun 2003 17:28:49 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h5ELSmrb017564; Sat, 14 Jun 2003 15:28:48 -0600 To: Caitlin Bestler Cc: rddp@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Sat, 14 Jun 2003 14:28:45 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 9, 2003) at 06/14/2003 15:28:48, Serialize complete at 06/14/2003 15:28:48 Content-Type: multipart/alternative; boundary="=_alternative 0075E9C788256D45_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 0075E9C788256D45_= Content-Type: text/plain; charset="us-ascii" Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode. This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all). Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler 06/14/2003 11:38 AM To: John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org cc: Subject: RE: [rddp] Transport Neutral "Connections" On 6/14/03, John Hufferd wrote: >Caitlin, >I can not determine from your note whether you are now >objecting to the application being able to perform its own >negotiation before entering RDMA mode (via an Atomic last >message operation) as is currently defined; or are you >agreeing with that, but ALSO want a LLP neutral method to >enter the RDMA mode? The latter. But I don't view the LLP neutral as the "additional" method. It should be the primary method. Conditional negotiation is a special need for certain applications. The LLP SHOULD be invisible to the ULP. If the ULP wants to deal with the ULP, well then there certainly isn't any reason to forbid it. I just don't want to force the ULP to deal with the LLP - or even suggest that it ought to. I would prefer to state forthrightly that the ULP SHOULD NOT deal with the LLP. The current proposals for MPA do exactly that. Vague unspecified middleware to parse a "message" above the stack does not solve the problem. It is a) still the ULP, and b) unrealistically complex. This middleware would be expected to consume the *precise* number of bytes required, and leave the remaining byte sitting for transfer to the MPA stack *instantly* when the request to assign the TCP Stream to an RDMA Endpoint/QP is made. This would be true even if the MPA Mode frame had been included in the very same TCP segment as the final pre-RDMA "message". This creates a nasty problem for at least some valid implementation strategies. Assume that at least a few sites will not be eager to turn over all TCP connections to a new stack. Therefore there will be a non-MPA stack still running on the host CPU, regardless of whatever RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC. The RNIC is also the Ethernet NIC. I am going to take a bold guess here and venture that at least a few sites will wish to preserve the current IP stack. The existing IP stack is known to all of the other existing applications that don't need this "RDMA" feature yet. The same stack that has already been tested and audited with all of those applications. New Ethernet NICs are much simpler to test and validate than new IP stacks. So at least some sites will want the RNIC to present itself as a conventional NIC for the benefit of the existing IP stack. And Ethernet NICs handoff entire frames, not bytes from TCP frames as the ULP reads them from sockets. Now this handoff can be solved, but it is tricky. Involving the ULP in the middle of it will only make things far worse. "Atomic" may be too strong of a word. But I do believe that receipt of the final "negotiation" message and entry to MPA mode needs to be co-ordinated at the kernel and/or hardware layer. The ULP should only be involved for policy issues -- not the mechanics. The mechanics are messy *and* LLP specific. To encourage, not just allow, the ULP to be LLP-neutral we need to define a procedure where the ULP can exchange messages that lead to the assignment of RDMA endpoints (or QPs if you insist on callling them that, but if you think about it, they no longer have two queues). Further, certain of those messages are naturally trigger events directly related to session state changes (request, accept and terminate). Those semantic are best recognized as they are parsed, not after a bounce through the ULP that only delays things. Caitlin Bestler http://asomi.com/CaitlinBestler/ --=_alternative 0075E9C788256D45_= Content-Type: text/html; charset="us-ascii"
Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode.  This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all).  

Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Caitlin Bestler <cait@asomi.com>

06/14/2003 11:38 AM

       
        To:        John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org
        cc:        
        Subject:        RE: [rddp] Transport Neutral "Connections"



On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.


To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.


Caitlin Bestler
http://asomi.com/CaitlinBestler/

--=_alternative 0075E9C788256D45_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 17 02:59:25 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA01248 for ; Tue, 17 Jun 2003 02:59:25 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5H6wvT28467 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 02:58:57 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5H6wum28464 for ; Tue, 17 Jun 2003 02:58:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA01235 for ; Tue, 17 Jun 2003 02:58:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SAOT-0003Xh-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 02:56:37 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SAOT-0003Xd-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 02:56:37 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5GMj1a12114; Mon, 16 Jun 2003 18:45:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5GMiCm12079 for ; Mon, 16 Jun 2003 18:44:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA07289 for ; Mon, 16 Jun 2003 18:44:08 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19S2fi-0007mU-00 for rddp@ietf.org; Mon, 16 Jun 2003 18:41:54 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=maho3msx2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19S2fh-0007mO-00 for rddp@ietf.org; Mon, 16 Jun 2003 18:41:54 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 16 Jun 2003 18:43:38 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> To: hufferd@us.ibm.com, cait@asomi.com Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Date: Mon, 16 Jun 2003 18:35:11 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C33457.8BE20370" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C33457.8BE20370 Content-Type: text/plain; charset="iso-8859-1" Folks, I think there are three "models" of how to get into RDMA mode under discussion: (1) Enter immediately on startup, possibly with an initial frame containing private data required to setup the connection. (2) Enter at some subsequent point, "quiescing" the connection first. (3) Enter at some subsequent point, on the fly without a hiccup. I read Caitlin's posts as saying that (3) is hard support in hardware; in order to obtain the flexibility of entering RDMA mode at some subsequent point the result is likely to require something that looks like a handshake with the associated latency. This doesn't rule out (2), it just says that those who want that flexibility are going to wind up paying some synchronization latency (Aside: it may be that the structure of the ULP requires that this be paid anyway, but that depends on the ULP). Viewed from another angle, those who want to minimize elapsed time from startup to blasting away in RDMA mode will want to use model (1), and the proposed private data mechanism may help make that model more applicable. Those who wish the generality of switching into RDMA mode later can use (2), with the knowledge that some sort of handshake is needed. I don't think I see a problem here - the main point is that the flexibility of deferred startup may come with a price, i.e., ULP designers should be thinking about (2), not (3). Thanks, --David -----Original Message----- From: John Hufferd [mailto:hufferd@us.ibm.com] Sent: Saturday, June 14, 2003 5:29 PM To: Caitlin Bestler Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode. This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all). Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler 06/14/2003 11:38 AM To: John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org cc: Subject: RE: [rddp] Transport Neutral "Connections" On 6/14/03, John Hufferd wrote: >Caitlin, >I can not determine from your note whether you are now >objecting to the application being able to perform its own >negotiation before entering RDMA mode (via an Atomic last >message operation) as is currently defined; or are you >agreeing with that, but ALSO want a LLP neutral method to >enter the RDMA mode? The latter. But I don't view the LLP neutral as the "additional" method. It should be the primary method. Conditional negotiation is a special need for certain applications. The LLP SHOULD be invisible to the ULP. If the ULP wants to deal with the ULP, well then there certainly isn't any reason to forbid it. I just don't want to force the ULP to deal with the LLP - or even suggest that it ought to. I would prefer to state forthrightly that the ULP SHOULD NOT deal with the LLP. The current proposals for MPA do exactly that. Vague unspecified middleware to parse a "message" above the stack does not solve the problem. It is a) still the ULP, and b) unrealistically complex. This middleware would be expected to consume the *precise* number of bytes required, and leave the remaining byte sitting for transfer to the MPA stack *instantly* when the request to assign the TCP Stream to an RDMA Endpoint/QP is made. This would be true even if the MPA Mode frame had been included in the very same TCP segment as the final pre-RDMA "message". This creates a nasty problem for at least some valid implementation strategies. Assume that at least a few sites will not be eager to turn over all TCP connections to a new stack. Therefore there will be a non-MPA stack still running on the host CPU, regardless of whatever RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC. The RNIC is also the Ethernet NIC. I am going to take a bold guess here and venture that at least a few sites will wish to preserve the current IP stack. The existing IP stack is known to all of the other existing applications that don't need this "RDMA" feature yet. The same stack that has already been tested and audited with all of those applications. New Ethernet NICs are much simpler to test and validate than new IP stacks. So at least some sites will want the RNIC to present itself as a conventional NIC for the benefit of the existing IP stack. And Ethernet NICs handoff entire frames, not bytes from TCP frames as the ULP reads them from sockets. Now this handoff can be solved, but it is tricky. Involving the ULP in the middle of it will only make things far worse. "Atomic" may be too strong of a word. But I do believe that receipt of the final "negotiation" message and entry to MPA mode needs to be co-ordinated at the kernel and/or hardware layer. The ULP should only be involved for policy issues -- not the mechanics. The mechanics are messy *and* LLP specific. To encourage, not just allow, the ULP to be LLP-neutral we need to define a procedure where the ULP can exchange messages that lead to the assignment of RDMA endpoints (or QPs if you insist on callling them that, but if you think about it, they no longer have two queues). Further, certain of those messages are naturally trigger events directly related to session state changes (request, accept and terminate). Those semantic are best recognized as they are parsed, not after a bounce through the ULP that only delays things. Caitlin Bestler http://asomi.com/CaitlinBestler/ ------_=_NextPart_001_01C33457.8BE20370 Content-Type: text/html; charset="iso-8859-1" Message
Folks,
 
I think there are three "models" of how to get into RDMA mode
under discussion:
 
(1) Enter immediately on startup, possibly with an initial frame
    containing private data required to setup the connection.
(2) Enter at some subsequent point, "quiescing" the connection first.
(3) Enter at some subsequent point, on the fly without a hiccup.
 
I read Caitlin's posts as saying that (3) is hard support in
hardware; in order to obtain the flexibility of entering RDMA mode
at some subsequent point the result is likely to require something
that looks like a handshake with the associated latency.  This doesn't
rule out (2), it just says that those who want that flexibility are
going to wind up paying some synchronization latency (Aside: it may
be that the structure of the ULP requires that this be paid anyway,
but that depends on the ULP).  Viewed from another angle, those who
want to minimize elapsed time from startup to blasting away in RDMA
mode will want to use model (1), and the proposed private data
mechanism may help make that model more applicable.  Those who wish
the generality of switching into RDMA mode later can use (2), with
the knowledge that some sort of handshake is needed.
 
I don't think I see a problem here - the main point is that the
flexibility of deferred startup may come with a price, i.e., ULP
designers should be thinking about (2), not (3).
 
Thanks,
--David
 
-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Saturday, June 14, 2003 5:29 PM
To: Caitlin Bestler
Cc: rddp@ietf.org
Subject: RE: [rddp] Transport Neutral "Connections"


Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode.  This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all).  

Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Caitlin Bestler <cait@asomi.com>

06/14/2003 11:38 AM

       
        To:        John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org
        cc:        
        Subject:        RE: [rddp] Transport Neutral "Connections"



On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.


To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.


Caitlin Bestler
http://asomi.com/CaitlinBestler/


------_=_NextPart_001_01C33457.8BE20370-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Jun 17 12:29:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25520 for ; Tue, 17 Jun 2003 12:29:57 -0400 (EDT) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HGTUW11982 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 12:29:30 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5HGTUm11979 for ; Tue, 17 Jun 2003 12:29:30 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25500 for ; Tue, 17 Jun 2003 12:29:27 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SJId-0001Xk-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 12:27:11 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SJId-0001Xh-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 12:27:11 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5H803a02595; Tue, 17 Jun 2003 04:00:03 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5H7xDm02510 for ; Tue, 17 Jun 2003 03:59:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA02552; Tue, 17 Jun 2003 03:59:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SBKp-0003st-00; Tue, 17 Jun 2003 03:56:55 -0400 Received: from e35.co.us.ibm.com ([32.97.110.133]) by ietf-mx with esmtp (Exim 4.12) id 19SBKo-0003sL-00; Tue, 17 Jun 2003 03:56:54 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e35.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h5H7wT2R243626; Tue, 17 Jun 2003 03:58:29 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h5H7wTcH071722; Tue, 17 Jun 2003 01:58:29 -0600 To: Black_David@emc.com Cc: cait@asomi.com, rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Tue, 17 Jun 2003 00:58:25 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 06/17/2003 01:58:29, Serialize complete at 06/17/2003 01:58:29 Content-Type: multipart/alternative; boundary="=_alternative 002BB9E088256D48_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 002BB9E088256D48_= Content-Type: text/plain; charset="us-ascii" David, I think I agree with your statement, but I am not quite sure what on the fly means. Or exactly what "quiescing" means in this case. If for instance we are saying that everything is brought to a half duplex like mode, where side A is waiting for side B to send a finial message before entering RDMA mode? If so, then I think that would be OK. This might mean, for example that: 1. Side B will need to have an atomic method of sending the last message in LLP mode, and then transiting into RDMA mode, and 2. After side A has received the last LLP message from B, it may then transit into RDMA mode. Would you say that is quiescing and entering RDMA mode? . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Black_David@emc.com Sent by: rddp-admin@ietf.org 06/16/2003 03:35 PM To: John Hufferd/San Jose/IBM@IBMUS, cait@asomi.com cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Folks, I think there are three "models" of how to get into RDMA mode under discussion: (1) Enter immediately on startup, possibly with an initial frame containing private data required to setup the connection. (2) Enter at some subsequent point, "quiescing" the connection first. (3) Enter at some subsequent point, on the fly without a hiccup. I read Caitlin's posts as saying that (3) is hard support in hardware; in order to obtain the flexibility of entering RDMA mode at some subsequent point the result is likely to require something that looks like a handshake with the associated latency. This doesn't rule out (2), it just says that those who want that flexibility are going to wind up paying some synchronization latency (Aside: it may be that the structure of the ULP requires that this be paid anyway, but that depends on the ULP). Viewed from another angle, those who want to minimize elapsed time from startup to blasting away in RDMA mode will want to use model (1), and the proposed private data mechanism may help make that model more applicable. Those who wish the generality of switching into RDMA mode later can use (2), with the knowledge that some sort of handshake is needed. I don't think I see a problem here - the main point is that the flexibility of deferred startup may come with a price, i.e., ULP designers should be thinking about (2), not (3). Thanks, --David -----Original Message----- From: John Hufferd [mailto:hufferd@us.ibm.com] Sent: Saturday, June 14, 2003 5:29 PM To: Caitlin Bestler Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode. This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all). Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler 06/14/2003 11:38 AM To: John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org cc: Subject: RE: [rddp] Transport Neutral "Connections" On 6/14/03, John Hufferd wrote: >Caitlin, >I can not determine from your note whether you are now >objecting to the application being able to perform its own >negotiation before entering RDMA mode (via an Atomic last >message operation) as is currently defined; or are you >agreeing with that, but ALSO want a LLP neutral method to >enter the RDMA mode? The latter. But I don't view the LLP neutral as the "additional" method. It should be the primary method. Conditional negotiation is a special need for certain applications. The LLP SHOULD be invisible to the ULP. If the ULP wants to deal with the ULP, well then there certainly isn't any reason to forbid it. I just don't want to force the ULP to deal with the LLP - or even suggest that it ought to. I would prefer to state forthrightly that the ULP SHOULD NOT deal with the LLP. The current proposals for MPA do exactly that. Vague unspecified middleware to parse a "message" above the stack does not solve the problem. It is a) still the ULP, and b) unrealistically complex. This middleware would be expected to consume the *precise* number of bytes required, and leave the remaining byte sitting for transfer to the MPA stack *instantly* when the request to assign the TCP Stream to an RDMA Endpoint/QP is made. This would be true even if the MPA Mode frame had been included in the very same TCP segment as the final pre-RDMA "message". This creates a nasty problem for at least some valid implementation strategies. Assume that at least a few sites will not be eager to turn over all TCP connections to a new stack. Therefore there will be a non-MPA stack still running on the host CPU, regardless of whatever RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC. The RNIC is also the Ethernet NIC. I am going to take a bold guess here and venture that at least a few sites will wish to preserve the current IP stack. The existing IP stack is known to all of the other existing applications that don't need this "RDMA" feature yet. The same stack that has already been tested and audited with all of those applications. New Ethernet NICs are much simpler to test and validate than new IP stacks. So at least some sites will want the RNIC to present itself as a conventional NIC for the benefit of the existing IP stack. And Ethernet NICs handoff entire frames, not bytes from TCP frames as the ULP reads them from sockets. Now this handoff can be solved, but it is tricky. Involving the ULP in the middle of it will only make things far worse. "Atomic" may be too strong of a word. But I do believe that receipt of the final "negotiation" message and entry to MPA mode needs to be co-ordinated at the kernel and/or hardware layer. The ULP should only be involved for policy issues -- not the mechanics. The mechanics are messy *and* LLP specific. To encourage, not just allow, the ULP to be LLP-neutral we need to define a procedure where the ULP can exchange messages that lead to the assignment of RDMA endpoints (or QPs if you insist on callling them that, but if you think about it, they no longer have two queues). Further, certain of those messages are naturally trigger events directly related to session state changes (request, accept and terminate). Those semantic are best recognized as they are parsed, not after a bounce through the ULP that only delays things. Caitlin Bestler http://asomi.com/CaitlinBestler/ --=_alternative 002BB9E088256D48_= Content-Type: text/html; charset="us-ascii"
David,
I think I agree with your statement, but I am not quite sure what on the fly means.  Or exactly what "quiescing" means in this case.

If for instance we are saying that everything is brought to a half duplex like mode, where side A is waiting for side B to send a finial message before entering RDMA mode?  If so,  then I think that would be OK.  

This might mean, for example that:
1. Side B will need to have an atomic method of sending the last message in LLP mode, and then transiting into RDMA mode, and
2. After side A has received the last LLP message from B, it may then transit into RDMA mode.

Would you say that is quiescing and entering RDMA mode?
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Black_David@emc.com
Sent by: rddp-admin@ietf.org

06/16/2003 03:35 PM

       
        To:        John Hufferd/San Jose/IBM@IBMUS, cait@asomi.com
        cc:        rddp@ietf.org
        Subject:        RE: [rddp] Transport Neutral "Connections"



Folks,
 
I think there are three "models" of how to get into RDMA mode
under discussion:
 
(1) Enter immediately on startup, possibly with an initial frame
    containing private data required to setup the connection.
(2) Enter at some subsequent point, "quiescing" the connection first.
(3) Enter at some subsequent point, on the fly without a hiccup.
 
I read Caitlin's posts as saying that (3) is hard support in
hardware; in order to obtain the flexibility of entering RDMA mode
at some subsequent point the result is likely to require something
that looks like a handshake with the associated latency.  This doesn't
rule out (2), it just says that those who want that flexibility are
going to wind up paying some synchronization latency (Aside: it may
be that the structure of the ULP requires that this be paid anyway,
but that depends on the ULP).  Viewed from another angle, those who
want to minimize elapsed time from startup to blasting away in RDMA
mode will want to use model (1), and the proposed private data
mechanism may help make that model more applicable.  Those who wish
the generality of switching into RDMA mode later can use (2), with
the knowledge that some sort of handshake is needed.
 
I don't think I see a problem here - the main point is that the
flexibility of deferred startup may come with a price, i.e., ULP
designers should be thinking about (2), not (3).
 
Thanks,
--David
 
-----Original Message-----
From: John Hufferd [mailto:hufferd@us.ibm.com]
Sent: Saturday, June 14, 2003 5:29 PM
To: Caitlin Bestler
Cc: rddp@ietf.org
Subject: RE: [rddp] Transport Neutral "Connections"

Well I thought we were almost together, but I can not accept the statement that "ULP SHOULD NOT deal with the LLP " because it is much to restrictive for some very important applications that must start their functional negotiations in TCP/IP mode, and then when RDMA mode is agreed upon, switch into that mode.  This permits the application to gradually move into RDMA mode across a network of nodes that implement RDMA at different rates (or not at all).  

Now the above statement should not indicate that there should not be a way to go directly to RDMA mode, it just says that we need an Overt method to accommodate the above type of negotiation whether or not we come up with a direct mode for entering RDMA, independent of the LLP.

.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com

Caitlin Bestler <cait@asomi.com>

06/14/2003 11:38 AM

       
       To:        John Hufferd/San Jose/IBM@IBMUS, rddp@ietf.org
       cc:        
       Subject:        RE: [rddp] Transport Neutral "Connections"




On 6/14/03, John Hufferd wrote:

>Caitlin,
>I can not determine from your note whether you are now
>objecting to the application being able to perform its
own
>negotiation before entering RDMA mode (via an Atomic
last
>message operation) as is currently defined; or are you
>agreeing with that, but ALSO want a LLP neutral method
to
>enter the RDMA mode?

The latter.

But I don't view the LLP neutral as the "additional"
method. It should be the primary method. Conditional
negotiation is a special need for certain applications.
The LLP SHOULD be invisible to the ULP. If the ULP
wants to deal with the ULP, well then there certainly
isn't any reason to forbid it. I just don't want to
force the ULP to deal with the LLP - or even suggest
that it ought to. I would prefer to state forthrightly
that the ULP SHOULD NOT deal with the LLP.

The current proposals for MPA do exactly that. Vague
unspecified middleware to parse a "message" above the
stack does not solve the problem. It is a) still the
ULP, and b) unrealistically complex.

This middleware would be expected to consume the
*precise* number of bytes required, and leave the
remaining byte sitting for transfer to the MPA stack
*instantly* when the request to assign the TCP Stream
to an RDMA Endpoint/QP is made. This would be true even
if the MPA Mode frame had been included in the very
same TCP segment as the final pre-RDMA "message".

This creates a nasty problem for at least some valid
implementation strategies.

Assume that at least a few sites will not be eager to
turn over all TCP connections to a new stack.

Therefore there will be a non-MPA stack still running
on the host CPU, regardless of whatever
RDMAP/DDP/MPA/TCP/IP stack there is on the RNIC.

The RNIC is also the Ethernet NIC. I am going to take a
bold guess here and venture that at least a few sites
will wish to preserve the current IP stack. The
existing IP stack is known to all of the other existing
applications that don't need this "RDMA" feature yet.
The same stack that has already been tested and audited
with all of those applications. New Ethernet NICs are
much simpler to test and validate than new IP stacks.
So at least some sites will want the RNIC to present
itself as a conventional NIC for the benefit of the
existing IP stack.

And Ethernet NICs handoff entire frames, not bytes from
TCP frames as the ULP reads them from sockets.

Now this handoff can be solved, but it is tricky.
Involving the ULP in the middle of it will only make
things far worse.

"Atomic" may be too strong of a word. But I do believe
that receipt of the final "negotiation" message and
entry to MPA mode needs to be co-ordinated at the
kernel and/or hardware layer. The ULP should only be
involved for policy issues -- not the mechanics. The
mechanics are messy *and* LLP specific.


To encourage, not just allow, the ULP to be LLP-neutral
we need to define a procedure where the ULP can exchange
messages that lead to the assignment of RDMA endpoints
(or QPs if you insist on callling them that, but if you
think about it, they no longer have two queues).

Further, certain of those messages are naturally trigger
events directly related to session state changes
(request, accept and terminate). Those semantic are
best recognized as they are parsed, not after a bounce
through the ULP that only delays things.


Caitlin Bestler
http://asomi.com/CaitlinBestler/

--=_alternative 002BB9E088256D48_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 14:31:50 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA01578 for ; Tue, 17 Jun 2003 14:31:50 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HIVMw17154 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 14:31:22 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SLDy-0004Lw-Ep for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 14:30:30 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA01484 for ; Tue, 17 Jun 2003 14:30:27 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SLBi-0002zl-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 14:28:10 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SLBi-0002zi-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 14:28:10 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5HEh1a02030; Tue, 17 Jun 2003 10:43:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5HEg5m01989 for ; Tue, 17 Jun 2003 10:42:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA20061 for ; Tue, 17 Jun 2003 10:42:01 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SHcg-00007V-00 for rddp@ietf.org; Tue, 17 Jun 2003 10:39:46 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=maho3msx2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19SHcf-000079-00 for rddp@ietf.org; Tue, 17 Jun 2003 10:39:45 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 17 Jun 2003 10:41:11 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CDA7@corpmx14.us.dg.com> To: hufferd@us.ibm.com Cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" Date: Tue, 17 Jun 2003 10:32:39 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , John, >> I think there are three "models" of how to get into RDMA mode >> under discussion: >> >> (1) Enter immediately on startup, possibly with an initial frame >> containing private data required to setup the connection. >> (2) Enter at some subsequent point, "quiescing" the connection first. >> (3) Enter at some subsequent point, on the fly without a hiccup. >> >> I read Caitlin's posts as saying that (3) is hard to support in >> hardware; in order to obtain the flexibility of entering RDMA mode >> at some subsequent point the result is likely to require something >> that looks like a handshake with the associated latency. This doesn't >> rule out (2), it just says that those who want that flexibility are >> going to wind up paying some synchronization latency (Aside: it may >> be that the structure of the ULP requires that this be paid anyway, >> but that depends on the ULP). Viewed from another angle, those who >> want to minimize elapsed time from startup to blasting away in RDMA >> mode will want to use model (1), and the proposed private data >> mechanism may help make that model more applicable. Those who wish >> the generality of switching into RDMA mode later can use (2), with >> the knowledge that some sort of handshake is needed. >> >> I don't think I see a problem here - the main point is that the >> flexibility of deferred startup may come with a price, i.e., ULP >> designers should be thinking about (2), not (3). > I think I agree with your statement, but I am not quite sure > what on the fly means. Or exactly what "quiescing" means in this case. The problematic scenario is that one side sends a stream of data and expects the other side to switch into RDMA mode in the middle (that's "on-the-fly". The implementation problem is that by the time the receiver ULP figures out that it's supposed to switch into RDMA mode, some of the RDMA-mode traffic has already been processed in non-RDMA mode. Quiescing is needed to avoid this. > If for instance we are saying that everything is brought to a > half duplex like mode, where side A is waiting for side B to send > a finial message before entering RDMA mode? If so, then I think > that would be OK. > > This might mean, for example that: > 1. Side B will need to have an atomic method of sending the > last message in LLP mode, and then transiting into RDMA mode, and > 2. After side A has received the last LLP message from B, it may > then transit into RDMA mode. > > Would you say that is quiescing and entering RDMA mode? Only for that half of the connection, side A still has to cleanly enter RDMA mode, so there's another pair of steps involving a wait for B: 3. Side A transitions into RDMA mode and sends its first RDMA-mode message to B. 4. B waits for receipt of that message before sending any RDMA-mode traffic to A (i.e., B cannot send any traffic to A after sending its "last LLP message" until it receives an RDMA-mode message indicating that A has switched). Notice that both directions of traffic pause as part of this - A has to wait for the last non-RDMA message from B (step 2) before sending any RDMA traffic, and B has to wait for the first RDMA message from A (step 4) before sending RDMA traffic. Side B's method does not need to be atomic - B can: a) Transition receive side of connection into RDMA mode b) Send last message to A in non-RDMA mode c) Transition send side of connection into RDMA mode In contrast, A can transition both sides of the connection into RDMA mode at the same time because B's final non-RDMA message says that the response from A is to be in RDMA mode. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 15:23:44 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA05956 for ; Tue, 17 Jun 2003 15:23:44 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HJNFU10946 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 15:23:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SL12-0003dY-M2 for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 14:17:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00303 for ; Tue, 17 Jun 2003 14:17:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SKyn-0002gC-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 14:14:49 -0400 Received: from ietf.org ([132.151.1.19] helo=www1.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SKym-0002g9-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 14:14:48 -0400 Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5HFY1a05642; Tue, 17 Jun 2003 11:34:01 -0400 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h5HFXmm05632 for ; Tue, 17 Jun 2003 11:33:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22845 for ; Tue, 17 Jun 2003 11:33:46 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SIQk-0000qo-00 for rddp@ietf.org; Tue, 17 Jun 2003 11:31:30 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19SIQj-0000pI-00 for rddp@ietf.org; Tue, 17 Jun 2003 11:31:29 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D574A245643; Tue, 17 Jun 2003 11:33:44 -0400 (EDT) Date: Tue, 17 Jun 2003 10:33:10 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: Black_David@emc.com Cc: hufferd@us.ibm.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/16/03, Black_David@emc.com wrote: >Folks, > >I think there are three "models" of how to get into >RDMA mode under discussion: > >(1) Enter immediately on startup, possibly with an > initial frame containing private data required to > setup the connection. This is what I see as the main model, but it needs to allow the passive side to receive the Private Data in the initial frame, then select/configure its endpoint/QP. I favor making the initiat frame also serve as the MPA mode frame. However any frame that can be parsed without active ULP involvement, after the ULP selects this mode, is fine with me. I want the ULP to interpret the message, not be involved in determining when the entire message has arrived. The meaning of the message can be LLP independent, the mechanics of collecting it cannot. >(2) Enter at some subsequent point, "quiescing" the >connection first. >(3) Enter at some subsequent point, on the fly without >a hiccup. > My real concern here is how to define "quiescing". What I fear is that implemenetations will interpret "message" as being "all data that arrives in the first TCP segment or n microseconds thereafter." What is really scary about that definition is that it will almost always work. Bugs that hide well are always the hardest to kill. TCP segments are not guaranteed to be preserved. So there has to be some method of determining when the "message" has been collected that does not rely upon TCP segment boundaries or timing. The best implementation-independent definition of "quiescing" that I can come up with is that all prior bytes have been acked, and both sides are in full agreement that RDMA mode should be established. Each side could then send a single TCP segment any last ULP initialization message and the MPA initialization, while expecting the same. [The single TCP segment is a measure of efficiency and succintness, I'm not proposing that as a protocol requirement, just something that should be achievable by an optimized implementation]. In an active/active scenario these messages could pass in flight. For active/passive the active side would send, the passive side would receive, interpret, select/configure its endpoint and then respond. My assumption here is that the RNIC will be expected to support non-RDMA TCP connections, and may be required/ expected to do so with a conventional IP stack. So unless I'm intending to run BSD on my RNIC, the only way I'll satisfy a really thorough security concious network admin is to pass the TCP connections I am not using through to an existing IP stack. To do that efficiently, I want to "place" those received "non-RDMA" IP datagrams to the convention IP stack just as immediately as I place the data from RDMA connections. I can achieve that if the "MPA Entry Sequence" is something that can be entirely handled by the RNIC after being enabled by the ULP while the connection is quiescent. Most ULPs should figure out that the easiest time to have a quiescent connection is right when its established. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 15:51:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07818 for ; Tue, 17 Jun 2003 15:51:41 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HJpDo24508 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 15:51:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SMOO-0005p5-Iz for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 15:45:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07539 for ; Tue, 17 Jun 2003 15:45:18 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SMMA-0004CM-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 15:43:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SMM9-0004CJ-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 15:43:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SM3q-00037u-IL; Tue, 17 Jun 2003 15:24:06 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SLa6-0006LW-Qc for rddp@optimus.ietf.org; Tue, 17 Jun 2003 14:53:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA02679 for ; Tue, 17 Jun 2003 14:53:19 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SLXq-0003GE-00 for rddp@ietf.org; Tue, 17 Jun 2003 14:51:02 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19SLXq-0003FX-00 for rddp@ietf.org; Tue, 17 Jun 2003 14:51:02 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4788A244CD6; Tue, 17 Jun 2003 14:53:23 -0400 (EDT) Date: Tue, 17 Jun 2003 13:52:46 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: Black_David@emc.com Cc: hufferd@us.ibm.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564CDA7@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/17/03, Black_David@emc.com wrote: >> I think I agree with your statement, but I am not >> quite sure what on the fly means. Or exactly what >> "quiescing" means in this case. > >The problematic scenario is that one side sends a >stream of data and expects the other side to switch >into RDMA mode in the middle (that's "on-the-fly". >The implementation problem is that by the time the >receiver ULP figures out that it's supposed to switch >into RDMA mode, some of the RDMA-mode traffic has >already been processed in non-RDMA mode. Quiescing is >needed to avoid this. Exactly. The off-load stack wants to "place" TCP segments immediately upon receipt. This is true even when in "non-RDMA" mode. It's just that they are being "placed" to the conventional stack, and probably as Ethernet frames rather than as TCP segments. It wouldn't make sense for the off-load stack to have to buffer TCP segments recieved out-of-order when that is what it was designed *not* to do. Let the conventional stack handle putting conventional TCP segments back in order. But to do that we can't have any pre-mature placements to the conventional stack. If segment N is going to cause the receiver to shift the stream to RDMA mode, then we can't handle N+1. >> If for instance we are saying that everything is brought to a >> half duplex like mode, where side A is waiting for side B to send >> a finial message before entering RDMA mode? If so, then I think >> that would be OK. >> >> This might mean, for example that: >> 1. Side B will need to have an atomic method of >> sending the last message in LLP mode, and then >> transiting into RDMA mode, and >> 2. After side A has received the last LLP message >> from B, it may then transit into RDMA mode. >> >> Would you say that is quiescing and entering RDMA mode? > >Only for that half of the connection, side A still has >to cleanly enter RDMA mode, so there's another pair of >steps involving a wait for B: > If I understand John's example correctly, A is quiescent because B is in the passive mode and will not send the first RDMA-mode message until A does the first RDMA-mode transmit. But that does need to be explicitly stated. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 16:42:03 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07817 for ; Tue, 17 Jun 2003 15:51:41 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HJpDG24496 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 15:51:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SMOD-0005lH-Pk for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 15:45:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07519 for ; Tue, 17 Jun 2003 15:45:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SMLz-0004CG-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 15:42:51 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SMLy-0004CD-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 15:42:50 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SM3q-00037k-Bk; Tue, 17 Jun 2003 15:24:06 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SLXl-0006AN-EF for rddp@optimus.ietf.org; Tue, 17 Jun 2003 14:50:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA02557 for ; Tue, 17 Jun 2003 14:50:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SLVV-0003Do-00 for rddp@ietf.org; Tue, 17 Jun 2003 14:48:37 -0400 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 19SLVU-0003D0-00 for rddp@ietf.org; Tue, 17 Jun 2003 14:48:36 -0400 Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.9/8.12.9/NTAP-1.4) with ESMTP id h5HIoMJo023020 for ; Tue, 17 Jun 2003 11:50:22 -0700 (PDT) Received: from tmt.netapp.com (tmt [10.97.1.31] (may be forged)) by frejya.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h5HIoLtg000203 for ; Tue, 17 Jun 2003 11:50:22 -0700 (PDT) Message-Id: <5.2.1.1.2.20030617143422.029bd328@silver.nane.netapp.com> Date: Tue, 17 Jun 2003 14:49:45 -0400 To: rddp@ietf.org From: "Talpey, Thomas" Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Simple implementation of "RDMA mode model 1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Of the three options David Black listed, the first, >I think there are three "models" of how to get into RDMA mode >under discussion: > >(1) Enter immediately on startup, possibly with an initial frame > containing private data required to setup the connection. has a rather easy solution: Upon initial endpoint (socket) creation, each side limits its TCP receive window to a small value, say 512 bytes. This is supported by most systems through use of the setsockopt(SO_RCVBUF) api. (Some systems however do enforce larger minimum buffers, such as several KB.) Then, provided that the initial "frame" is exactly this many bytes, it is trivial for each peer to convey this payload before entering RDMA mode. After consuming the frame and using any contents to initialize the RDMA endpoint, the receive buffer is raised to a more reasonable value by a subsequent setsockopt() at each end. There is the issue of new data arriving between the time that the "frame" is consumed but before RDMA mode has been entered, however this is a local interface issue and in any case wouldn't be a problem for software RDMA implementations. Also, it requires prior knowledge of the peer's intention to send such a frame, which of course implies a well-known port or other prior contract. But, it will work and is a reasonable design point, in my opinion. By the way, on some systems, a tiny initial window of 1 or 2 bytes, or even 0, is permitted. It's within the realm of possibility that this technique could be used to initiate RDMA mode without any exchange whatsoever. I have a test program which demonstrates this over the sockets api if anyone wants to try it out. Tom. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 16:48:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA10816 for ; Tue, 17 Jun 2003 16:48:39 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HKmC104914 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 16:48:12 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SNNE-0001HB-4l for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 16:48:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA10779 for ; Tue, 17 Jun 2003 16:48:09 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SNKz-0004wB-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 16:45:53 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SNKy-0004w8-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 16:45:52 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SN4g-0008RH-FY; Tue, 17 Jun 2003 16:29:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SMhr-0006pb-Go for rddp@optimus.ietf.org; Tue, 17 Jun 2003 16:05:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08358 for ; Tue, 17 Jun 2003 16:05:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SMfd-0004Mc-00 for rddp@ietf.org; Tue, 17 Jun 2003 16:03:09 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19SMfc-0004LP-00 for rddp@ietf.org; Tue, 17 Jun 2003 16:03:08 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3A60F244AC1; Tue, 17 Jun 2003 16:04:59 -0400 (EDT) Date: Tue, 17 Jun 2003 15:04:23 -0500 From: Caitlin Bestler Subject: Re: [rddp] Simple implementation of "RDMA mode model 1" To: "Talpey, Thomas" , rddp@ietf.org X-Priority: 3 In-Reply-To: <5.2.1.1.2.20030617143422.029bd328@silver.nane.netapp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 6/17/03, Talpey, Thomas wrote: >Of the three options David Black listed, the first, > >>I think there are three "models" of how to get into RDMA mode >>under discussion: >> >>(1) Enter immediately on startup, possibly with an initial frame >> containing private data required to setup the connection. > >has a rather easy solution: > >Upon initial endpoint (socket) creation, each side >limits its TCP receive window to a small value, say >512 bytes. This is supported by most systems through >use of the setsockopt(SO_RCVBUF) api. (Some systems >however do enforce larger minimum buffers, such as >several KB.) > >Then, provided that the initial "frame" is exactly >this many bytes, it is trivial for each peer to convey >this payload before entering RDMA mode. After >consuming the frame and using any contents to >initialize the RDMA endpoint, the receive buffer is >raised to a more reasonable value by a subsequent >setsockopt() at each end. > >There is the issue of new data arriving between the >time that the "frame" is consumed but before RDMA mode >has been entered, however this is a local interface >issue and in any case wouldn't be a problem for >software RDMA implementations. > >Also, it requires prior knowledge of the peer's >intention to send such a frame, which of course >implies a well-known port or other prior contract. >But, it will work and is a reasonable design point, in >my opinion. > >By the way, on some systems, a tiny initial window of >1 or 2 bytes, or even 0, is permitted. It's within the >realm of possibility that this technique could be used >to initiate RDMA mode without any exchange whatsoever. > >I have a test program which demonstrates this over the >sockets api if anyone wants to try it out. > This is very close to what I was envisioning, except that it could be combined with the MPA Mode "proof" and it would be at most 512 bytes (or some other small size), but could be smaller when the ULP has nothing worth saying before establishing the full RDMA connection. Of course, we can take InfiniBand's CM protocol as proof that having a small fixed size is clearly acceptable to application developers. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 17 16:55:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA11091 for ; Tue, 17 Jun 2003 16:55:37 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5HKt9L06563 for rddp-archive@odin.ietf.org; Tue, 17 Jun 2003 16:55:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SNTx-0001hm-O9 for rddp-web-archive@optimus.ietf.org; Tue, 17 Jun 2003 16:55:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA11085 for ; Tue, 17 Jun 2003 16:55:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SNRi-00051z-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 16:52:50 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SNRi-00051w-00 for rddp-web-archive@ietf.org; Tue, 17 Jun 2003 16:52:50 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SN4h-0008Rt-HF; Tue, 17 Jun 2003 16:29:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SMwL-0007Mg-Jj for rddp@optimus.ietf.org; Tue, 17 Jun 2003 16:20:25 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09111; Tue, 17 Jun 2003 16:20:21 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SMu4-0004UI-00; Tue, 17 Jun 2003 16:18:04 -0400 Received: from e33.co.us.ibm.com ([32.97.110.131]) by ietf-mx with esmtp (Exim 4.12) id 19SMu3-0004U7-00; Tue, 17 Jun 2003 16:18:03 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e33.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h5HKJfJk270006; Tue, 17 Jun 2003 16:19:41 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h5HKJenE045218; Tue, 17 Jun 2003 14:19:40 -0600 To: Black_David@emc.com Cc: rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Transport Neutral "Connections" X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Tue, 17 Jun 2003 13:20:24 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 06/17/2003 14:19:40, Serialize complete at 06/17/2003 14:19:40 Content-Type: multipart/alternative; boundary="=_alternative 006FA85088256D48_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 006FA85088256D48_= Content-Type: text/plain; charset="us-ascii" David, I agree with your statement about the extra step that must be done, and that side B can not send an RDMA message to A until it receives an RDMA message from A. I think, however, more amplification is need with regards to your last statement ... "Side B's method does not need to be atomic - B can: a) Transition receive side of connection into RDMA mode b) Send last message to A in non-RDMA mode c) Transition send side of connection into RDMA mode In contrast, A can transition both sides of the connection into RDMA mode at the same time because B's final non-RDMA message says that the response from A is to be in RDMA mode." From a verbs/API interface, this is what I am calling Atomic, I think it is up to the RNIC (or RDMA Stack) implementation to determine how this is actually accomplished, but the point is, that the application should see this as Atomic. This means that the implementation must make sure that it can handle, ACKs, and errors from the last non RDMA mode message while being able to handle any new RDMA incoming messages, etc. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Black_David@emc.com Sent by: rddp-admin@ietf.org 06/17/2003 07:32 AM To: John Hufferd/San Jose/IBM@IBMUS cc: rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" John, >> I think there are three "models" of how to get into RDMA mode >> under discussion: >> >> (1) Enter immediately on startup, possibly with an initial frame >> containing private data required to setup the connection. >> (2) Enter at some subsequent point, "quiescing" the connection first. >> (3) Enter at some subsequent point, on the fly without a hiccup. >> >> I read Caitlin's posts as saying that (3) is hard to support in >> hardware; in order to obtain the flexibility of entering RDMA mode >> at some subsequent point the result is likely to require something >> that looks like a handshake with the associated latency. This doesn't >> rule out (2), it just says that those who want that flexibility are >> going to wind up paying some synchronization latency (Aside: it may >> be that the structure of the ULP requires that this be paid anyway, >> but that depends on the ULP). Viewed from another angle, those who >> want to minimize elapsed time from startup to blasting away in RDMA >> mode will want to use model (1), and the proposed private data >> mechanism may help make that model more applicable. Those who wish >> the generality of switching into RDMA mode later can use (2), with >> the knowledge that some sort of handshake is needed. >> >> I don't think I see a problem here - the main point is that the >> flexibility of deferred startup may come with a price, i.e., ULP >> designers should be thinking about (2), not (3). > I think I agree with your statement, but I am not quite sure > what on the fly means. Or exactly what "quiescing" means in this case. The problematic scenario is that one side sends a stream of data and expects the other side to switch into RDMA mode in the middle (that's "on-the-fly". The implementation problem is that by the time the receiver ULP figures out that it's supposed to switch into RDMA mode, some of the RDMA-mode traffic has already been processed in non-RDMA mode. Quiescing is needed to avoid this. > If for instance we are saying that everything is brought to a > half duplex like mode, where side A is waiting for side B to send > a finial message before entering RDMA mode? If so, then I think > that would be OK. > > This might mean, for example that: > 1. Side B will need to have an atomic method of sending the > last message in LLP mode, and then transiting into RDMA mode, and > 2. After side A has received the last LLP message from B, it may > then transit into RDMA mode. > > Would you say that is quiescing and entering RDMA mode? Only for that half of the connection, side A still has to cleanly enter RDMA mode, so there's another pair of steps involving a wait for B: 3. Side A transitions into RDMA mode and sends its first RDMA-mode message to B. 4. B waits for receipt of that message before sending any RDMA-mode traffic to A (i.e., B cannot send any traffic to A after sending its "last LLP message" until it receives an RDMA-mode message indicating that A has switched). Notice that both directions of traffic pause as part of this - A has to wait for the last non-RDMA message from B (step 2) before sending any RDMA traffic, and B has to wait for the first RDMA message from A (step 4) before sending RDMA traffic. Side B's method does not need to be atomic - B can: a) Transition receive side of connection into RDMA mode b) Send last message to A in non-RDMA mode c) Transition send side of connection into RDMA mode In contrast, A can transition both sides of the connection into RDMA mode at the same time because B's final non-RDMA message says that the response from A is to be in RDMA mode. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp --=_alternative 006FA85088256D48_= Content-Type: text/html; charset="us-ascii"
David,
I agree with your statement about the extra step that must be done, and that side B can not send an RDMA message to A until it receives an RDMA message from A.

I think, however, more amplification is need  with regards to your last statement ...

"Side B's method does not need to be atomic - B can:
a) Transition receive side of connection into RDMA mode
b) Send last message to A in non-RDMA mode
c) Transition send side of connection into RDMA mode
In contrast, A can transition both sides of the connection into
RDMA mode at the same time because B's final non-RDMA message says
that the response from A is to be in RDMA mode."

From a verbs/API interface, this is what I am calling Atomic, I think it is up to the RNIC (or RDMA Stack) implementation to determine how this is actually accomplished, but the point is, that the application should see this as Atomic. This means that the implementation must make sure that it can handle, ACKs, and errors from the last non RDMA mode message while being able to handle any new RDMA incoming messages, etc.


.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Home Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Black_David@emc.com
Sent by: rddp-admin@ietf.org

06/17/2003 07:32 AM

       
        To:        John Hufferd/San Jose/IBM@IBMUS
        cc:        rddp@ietf.org
        Subject:        RE: [rddp] Transport Neutral "Connections"



John,

>> I think there are three "models" of how to get into RDMA mode
>> under discussion:
>>  
>> (1) Enter immediately on startup, possibly with an initial frame
>>     containing private data required to setup the connection.
>> (2) Enter at some subsequent point, "quiescing" the connection first.
>> (3) Enter at some subsequent point, on the fly without a hiccup.
>>  
>> I read Caitlin's posts as saying that (3) is hard to support in
>> hardware; in order to obtain the flexibility of entering RDMA mode
>> at some subsequent point the result is likely to require something
>> that looks like a handshake with the associated latency.  This doesn't
>> rule out (2), it just says that those who want that flexibility are
>> going to wind up paying some synchronization latency (Aside: it may
>> be that the structure of the ULP requires that this be paid anyway,
>> but that depends on the ULP).  Viewed from another angle, those who
>> want to minimize elapsed time from startup to blasting away in RDMA
>> mode will want to use model (1), and the proposed private data
>> mechanism may help make that model more applicable.  Those who wish
>> the generality of switching into RDMA mode later can use (2), with
>> the knowledge that some sort of handshake is needed.
>>
>> I don't think I see a problem here - the main point is that the
>> flexibility of deferred startup may come with a price, i.e., ULP
>> designers should be thinking about (2), not (3).

> I think I agree with your statement, but I am not quite sure
> what on the fly means.  Or exactly what "quiescing" means in this case.

The problematic scenario is that one side sends a stream of data and
expects the other side to switch into RDMA mode in the middle (that's
"on-the-fly".  The implementation problem is that by the time the
receiver ULP figures out that it's supposed to switch into RDMA mode,
some of the RDMA-mode traffic has already been processed in non-RDMA
mode.  Quiescing is needed to avoid this.

> If for instance we are saying that everything is brought to a
> half duplex like mode, where side A is waiting for side B to send
> a finial message before entering RDMA mode?  If so, then I think
> that would be OK.
>
> This might mean, for example that:
> 1. Side B will need to have an atomic method of sending the
> last message in LLP mode, and then transiting into RDMA mode, and
> 2. After side A has received the last LLP message from B, it may
> then transit into RDMA mode.
>
> Would you say that is quiescing and entering RDMA mode?

Only for that half of the connection, side A still has to cleanly
enter RDMA mode, so there's another pair of steps involving a wait
for B:

3.  Side A transitions into RDMA mode and sends its first RDMA-mode
                message to B.
4.  B waits for receipt of that message before sending any RDMA-mode
                traffic to A (i.e., B cannot send any traffic to A after
                sending its "last LLP message" until it receives an RDMA-mode
                message indicating that A has switched).

Notice that both directions of traffic pause as part of this -
A has to wait for the last non-RDMA message from B (step 2) before
sending any RDMA traffic, and B has to wait for the first RDMA
message from A (step 4) before sending RDMA traffic.

Side B's method does not need to be atomic - B can:
a) Transition receive side of connection into RDMA mode
b) Send last message to A in non-RDMA mode
c) Transition send side of connection into RDMA mode
In contrast, A can transition both sides of the connection into
RDMA mode at the same time because B's final non-RDMA message says
that the response from A is to be in RDMA mode.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
black_david@emc.com        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
rddp@ietf.org
https://www1.ietf.org/mailman/listinfo/rddp

--=_alternative 006FA85088256D48_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 18 13:10:44 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA16649 for ; Wed, 18 Jun 2003 13:10:43 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5IHAHi13063 for rddp-archive@odin.ietf.org; Wed, 18 Jun 2003 13:10:17 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Sfdb-0001GO-C4 for rddp-web-archive@optimus.ietf.org; Wed, 18 Jun 2003 12:18:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA14880 for ; Wed, 18 Jun 2003 12:18:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SfbL-0006NQ-00 for rddp-web-archive@ietf.org; Wed, 18 Jun 2003 12:15:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19SfbK-0006NM-00 for rddp-web-archive@ietf.org; Wed, 18 Jun 2003 12:15:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Sezc-0007aE-WD; Wed, 18 Jun 2003 11:37:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19SeYD-0006NN-Bc for rddp@optimus.ietf.org; Wed, 18 Jun 2003 11:08:41 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11511 for ; Wed, 18 Jun 2003 11:08:37 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19SeVv-0005Zm-00 for rddp@ietf.org; Wed, 18 Jun 2003 11:06:19 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19SeVv-0005Yk-00 for rddp@ietf.org; Wed, 18 Jun 2003 11:06:19 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h5IF81LW022665; Wed, 18 Jun 2003 08:08:02 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Wed, 18 Jun 2003 11:08:01 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF289B90B0@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Talpey, Thomas'" , rddp@ietf.org Subject: RE: [rddp] Simple implementation of "RDMA mode model 1" Date: Wed, 18 Jun 2003 11:07:53 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I would add that if you are intending to transfer the connection mid-stream from the software TCP stack to a hardware accelerated implementation, it is likely you are already modifying the software TCP, so tweaking the windowing should not be much incremental effort. There is one annoying detail that could be a problem, though. If the TCP connection negotiates window scaling, which may be needed for high performance, the actual window size will have some quantization error resulting in a few extra (or too few) bytes being received. For instance, if you want 512 bytes and receive 312, you need to acknowledge the 312 and set the new window to 200. But if the window scale factor is 16, this needs to be rounded up to 208 which is a multiple of 16, or down to 192. > has a rather easy solution: > > Upon initial endpoint (socket) creation, each side limits its TCP > receive window to a small value, say 512 bytes. This is supported > by most systems through use of the setsockopt(SO_RCVBUF) api. > (Some systems however do enforce larger minimum buffers, such as > several KB.) > > Then, provided that the initial "frame" is exactly this many bytes, it > is trivial for each peer to convey this payload before entering RDMA > mode. After consuming the frame and using any contents to initialize > the RDMA endpoint, the receive buffer is raised to a more reasonable > value by a subsequent setsockopt() at each end. > > There is the issue of new data arriving between the time that > the "frame" > is consumed but before RDMA mode has been entered, however this is > a local interface issue and in any case wouldn't be a problem > for software > RDMA implementations. > > Also, it requires prior knowledge of the peer's intention to send such > a frame, which of course implies a well-known port or other > prior contract. > But, it will work and is a reasonable design point, in my opinion. > > By the way, on some systems, a tiny initial window of 1 or 2 bytes, or > even 0, is permitted. It's within the realm of possibility that this > technique could be used to initiate RDMA mode without any exchange > whatsoever. > > I have a test program which demonstrates this over the sockets api if > anyone wants to try it out. > > Tom. > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sun Jun 22 11:14:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13266 for ; Sun, 22 Jun 2003 11:14:41 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5MFECq13327 for rddp-archive@odin.ietf.org; Sun, 22 Jun 2003 11:14:12 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U6Xk-0003Ss-Sv for rddp-web-archive@optimus.ietf.org; Sun, 22 Jun 2003 11:14:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13259 for ; Sun, 22 Jun 2003 11:14:10 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19U6Xk-0004Za-00 for rddp-web-archive@ietf.org; Sun, 22 Jun 2003 11:14:12 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19U6Xe-0004ZW-00 for rddp-web-archive@ietf.org; Sun, 22 Jun 2003 11:14:06 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U6XY-0003SQ-T4; Sun, 22 Jun 2003 11:14:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U6WW-0003SC-8f for rddp@optimus.ietf.org; Sun, 22 Jun 2003 11:13:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA13244 for ; Sun, 22 Jun 2003 11:12:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19U6WG-0004ZO-00 for rddp@ietf.org; Sun, 22 Jun 2003 11:12:40 -0400 Received: from sccrmhc13.comcast.net ([204.127.202.64] helo=sccrmhc13.attbi.com) by ietf-mx with esmtp (Exim 4.12) id 19U6W5-0004ZE-00 for rddp@ietf.org; Sun, 22 Jun 2003 11:12:29 -0400 Received: from cs.uchicago.edu (h005018030f43.ne.client2.attbi.com[24.128.140.161](untrusted sender)) by attbi.com (sccrmhc13) with SMTP id <20030622151134016009s7cie>; Sun, 22 Jun 2003 15:11:34 +0000 To: "Williams, Jim" cc: "'Talpey, Thomas'" , rddp@ietf.org Subject: Re: [rddp] Simple implementation of "RDMA mode model 1" In-Reply-To: Message from "Williams, Jim" of "Wed, 18 Jun 2003 11:07:53 EDT." <3356669BBE90C448AD4645C843E2BF289B90B0@xbl.ma.emulex.com> References: <3356669BBE90C448AD4645C843E2BF289B90B0@xbl.ma.emulex.com> Date: Sun, 22 Jun 2003 11:11:34 -0400 From: Stephen Bailey Message-Id: Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > I would add that if you are intending to > transfer the connection mid-stream from > the software TCP stack to a hardware > accelerated implementation, it is likely > you are already modifying the > software TCP, so tweaking > the windowing should not be much incremental > effort. While I generally agree this may be possible, I still don't think we should be messing around with this type of stuff. A ULP that does not adopt the `turn on immediately' model probably already has some handshaking, so connection setup is already not `lightweight' without RDDP/TCP. It would be nice if the RDDP/TCP handshake could be specified in such a way that it piggybacks the ULP handshake. That's what we tried to do with the original framing drafts: define what must be exchanged, and how a peer controls exactly when the first RDDPed (framed) data will come to it. However, if y'all think it's problematic to allow this (I understand why), we shouldn't mess with it. Declare victory and move on for now. Steph _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sun Jun 22 12:17:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13755 for ; Sun, 22 Jun 2003 12:17:35 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5MGH7u19949 for rddp-archive@odin.ietf.org; Sun, 22 Jun 2003 12:17:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U7Wd-0005Bg-La for rddp-web-archive@optimus.ietf.org; Sun, 22 Jun 2003 12:17:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13752 for ; Sun, 22 Jun 2003 12:17:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19U7Wc-0004hZ-00 for rddp-web-archive@ietf.org; Sun, 22 Jun 2003 12:17:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19U7WW-0004hW-00 for rddp-web-archive@ietf.org; Sun, 22 Jun 2003 12:17:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U7WW-00058a-Oa; Sun, 22 Jun 2003 12:17:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19U7Vy-00057u-3Y for rddp@optimus.ietf.org; Sun, 22 Jun 2003 12:16:26 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13749 for ; Sun, 22 Jun 2003 12:16:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19U7Vh-0004hT-00 for rddp@ietf.org; Sun, 22 Jun 2003 12:16:09 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19U7VX-0004hO-00 for rddp@ietf.org; Sun, 22 Jun 2003 12:15:59 -0400 Received: from asomi.com (208-41-143-203.client.dsl.net [208.41.143.203]) by thebe.your-site.com (Postfix) with ESMTP id 7AC3724568C; Sun, 22 Jun 2003 12:15:06 -0400 (EDT) Date: Sun, 22 Jun 2003 11:14:29 -0500 Subject: Re: [rddp] Simple implementation of "RDMA mode model 1" Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) From: Caitlin Bestler To: Stephen Bailey , rddp@ietf.org Content-Transfer-Encoding: 7bit In-Reply-To: Message-Id: <99FAD4DE-A4CC-11D7-8F7D-0003939A7978@asomi.com> X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Sunday, June 22, 2003, at 10:11 AM, Stephen Bailey wrote: >> I would add that if you are intending to >> transfer the connection mid-stream from >> the software TCP stack to a hardware >> accelerated implementation, it is likely >> you are already modifying the >> software TCP, so tweaking >> the windowing should not be much incremental >> effort. > > While I generally agree this may be possible, I still don't think we > should be messing around with this type of stuff. > > A ULP that does not adopt the `turn on immediately' model probably > already has some handshaking, so connection setup is already not > `lightweight' without RDDP/TCP. > > It would be nice if the RDDP/TCP handshake could be specified in such > a way that it piggybacks the ULP handshake. That's what we tried to > do with the original framing drafts: define what must be exchanged, > and how a peer controls exactly when the first RDDPed (framed) data > will come to it. However, if y'all think it's problematic to allow > this (I understand why), we shouldn't mess with it. Declare victory > and move on for now. There are two categories of negotiation that are fairly simple to deal with. a) turn on immediately. Endpoint configuration is not session specific. This can be done simply, and presented to the ULP in a LLP-neutral way. b) turns on after extensive ULP specific negotiations that are required for ULP-specific reasons. Once we fully document the rules on the state of the connection at the delayed entry point (i.e., what can be in-flight, what must have already been acked, etc.) this will be fairly simple as well. The descriptions already floated on this type of procedure are fine, they just fail to state all of the simplifying conditions as flat rules. This can be made LLP-neutral if the sole criteria on whether to enter RDMA mode is capability. That is an endpoint can be non-RDMA, RDMA-only or support both. Everything works as expected, with the caveat that if both ends are RDMA capable then the association will be RDMA capable. There is a third category of program that is relevant though: c) Session-specific negotiations are required. Further, these negotiations are RDMA specific and/or this is a RDMA-only protocol. RDMA specific factors that have the potential of being session-specific include the number of initial send credits and the number of allowed RDMA Read operations. I believe that c) will be a common model. In fact it is the *only* model that was developed by the DAT Collaborative. It is also the model used in other RDMA protocols such as InfiniBand. Properly supporting c) under MPA/TCP requires either including Private Data in the MPA Mode Frame, or defining a pre-MPA "frame" that can be captured by the stack (rather than by the ULP). This reduces the ULP interaction to supplying and accepting a "private data" that accompanies a connection request or response. Caitlin Bestler http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 23 13:52:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA01081 for ; Mon, 23 Jun 2003 13:52:42 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5NHqET18867 for rddp-archive@odin.ietf.org; Mon, 23 Jun 2003 13:52:14 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UVUE-0004uE-9R for rddp-web-archive@optimus.ietf.org; Mon, 23 Jun 2003 13:52:14 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA01027 for ; Mon, 23 Jun 2003 13:52:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UVUB-0004HW-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 13:52:12 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19UVU6-0004HN-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 13:52:06 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UVU1-0004oX-59; Mon, 23 Jun 2003 13:52:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UVTC-0004nM-Bl for rddp@optimus.ietf.org; Mon, 23 Jun 2003 13:51:40 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00967 for ; Mon, 23 Jun 2003 13:50:52 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UVSv-0004GU-00 for rddp@ietf.org; Mon, 23 Jun 2003 13:50:53 -0400 Received: from saway.wasabisystems.com ([151.199.66.145] helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19UVSk-0004GF-00 for rddp@ietf.org; Mon, 23 Jun 2003 13:50:42 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id 789A64013B; Mon, 23 Jun 2003 13:50:03 -0400 (EDT) Date: Mon, 23 Jun 2003 10:49:47 -0700 (PDT) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Black_David@emc.com Cc: hufferd@us.ibm.com, cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] Transport Neutral "Connections" In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> Message-ID: References: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Sorry for the delay, I tried to send this last week, but it silently disapeared. On Mon, 16 Jun 2003 Black_David@emc.com wrote: > Folks, > > I think there are three "models" of how to get into RDMA mode > under discussion: > > (1) Enter immediately on startup, possibly with an initial frame > containing private data required to setup the connection. > (2) Enter at some subsequent point, "quiescing" the connection first. > (3) Enter at some subsequent point, on the fly without a hiccup. > > I read Caitlin's posts as saying that (3) is hard support in > hardware; in order to obtain the flexibility of entering RDMA mode While I agree that it's not easy to support now, I've been looking at some of the OS infrastructure proposals for adding full tcp offload support, and I think it won't be too hard. The offload proposals I've seen talk about having the host OS handle TCP connection setup, then hand off an existing connection. The way handoff works is that the host OS signals the NIC that it will be getting the connection, so it sets up filters to intercept packets. The host OS then finishes processing any pending packets that had been delivered to it, then uploads the connection state to the NIC. The NIC then starts handling the connection. Yes, that's TCP. But handling RDMA on top of that isn't hard. The same procedure can happen if the host OS has a software RDMA stack. The software stack processes any pending mis-labeled TCP packets (really RDMA but we didn't know we were doing RDMA at that point). That will advance RDMA connection state, then the host OS offloads the RDMA processing to the card. So if a software/host OS stack is available, (3) won't be too hard. It also won't require more complexity than TCP connection offload will require. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 23 14:36:00 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03278 for ; Mon, 23 Jun 2003 14:36:00 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5NIZW725122 for rddp-archive@odin.ietf.org; Mon, 23 Jun 2003 14:35:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UWA8-0006X7-N8 for rddp-web-archive@optimus.ietf.org; Mon, 23 Jun 2003 14:35:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03270 for ; Mon, 23 Jun 2003 14:35:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UW9d-0006VH-I2; Mon, 23 Jun 2003 14:35:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UW8s-0006UT-3s for rddp@optimus.ietf.org; Mon, 23 Jun 2003 14:34:14 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03220 for ; Mon, 23 Jun 2003 14:34:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UW8p-0004f7-00 for rddp@ietf.org; Mon, 23 Jun 2003 14:34:11 -0400 Received: from mail4.microsoft.com ([131.107.3.122]) by ietf-mx with esmtp (Exim 4.12) id 19UW8e-0004eg-00 for rddp@ietf.org; Mon, 23 Jun 2003 14:34:00 -0400 Received: from inet-vrs-04.redmond.corp.microsoft.com ([157.54.8.149]) by mail4.microsoft.com with Microsoft SMTPSVC(5.0.2195.6624); Mon, 23 Jun 2003 11:32:44 -0700 Received: from 157.54.8.109 by inet-vrs-04.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 23 Jun 2003 11:32:44 -0700 Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 23 Jun 2003 11:32:43 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 23 Jun 2003 11:32:43 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.82]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3788.0); Mon, 23 Jun 2003 11:32:43 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6940.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C339B5.D54FB0BE" Date: Mon, 23 Jun 2003 11:32:42 -0700 Message-ID: X-MS-Has-Attach: yes Thread-Topic: I-D ACTION:draft-pinkerton-rddp-security-00.txt Thread-Index: AcM3kOHoOC59km/ZQse5/LKeepp/swCG5irw From: "Jim Pinkerton" To: X-OriginalArrivalTime: 23 Jun 2003 18:32:43.0348 (UTC) FILETIME=[D5B07140:01C339B5] Subject: [rddp] FW: I-D ACTION:draft-pinkerton-rddp-security-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. ------_=_NextPart_001_01C339B5.D54FB0BE Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable FYI. Our initial draft at a security analysis was just posted to the IETF ID site (note that the URL wraps to a new line, so a simple cut-and-paste won't work). From the introduction: This specification is work in progress - the intent is to begin the=20 discussion with a well thought out framework to analyze the issues.=20 An area of particular concern is that there may be more attacks=20 possible than have been enumerated here. Feedback would be much appreciated. Jim > -----Original Message----- > From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] > Sent: Friday, June 20, 2003 11:27 AM > Subject: I-D ACTION:draft-pinkerton-rddp-security-00.txt >=20 >=20 > A New Internet-Draft is available from the on-line=20 > Internet-Drafts directories. >=20 >=20 > Title : DDP/RDMAP Security > Author(s) : J. Pinkerton > Filename : draft-pinkerton-rddp-security-00.txt > Pages : 37 > Date : 2003-6-20 > =09 > This document analyzes security issues around implementation and use=20 > of the Direct Data Placement Protocol(DDP) and Remote Direct Memory=20 > Access Protocol (RDMAP). It first defines an architectural model for=20 > an RDMA Network Interface Card (RNIC), which can implement DDP or=20 > RDMAP and DDP. The model includes a definition of resources that can=20 > be attacked. This document then introduces various Trust Models=20 > between a local peer and a remote peer and the tools that can be=20 > used to create countermeasures against attacks. Finally, the=20 > document reviews various attacks and the countermeasures to be used=20 > against them, grouping the attacks into spoofing, tampering,=20 > information disclosure, denial of service, and elevation of=20 > privilege. >=20 > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-pinkerton-rddp-secur ity-00.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-pinkerton-rddp-security-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-pinkerton-rddp-security-00.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_001_01C339B5.D54FB0BE Content-Type: application/octet-stream; name="ATT62020.TXT" Content-Description: ATT62020.TXT Content-Disposition: attachment; filename="ATT62020.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBNZXNzYWdlL0V4dGVybmFsLWJvZHk7DQoJYWNjZXNzLXR5cGU9Im1haWwt c2VydmVyIjsNCglzZXJ2ZXI9Im1haWxzZXJ2QGlldGYub3JnIg0KDQpDb250ZW50LVR5cGU6IHRl eHQvcGxhaW4NCkNvbnRlbnQtSUQ6CTwyMDAzLTYtMjAxNDAyMDcuSS1EQGlldGYub3JnPg0KDQpF TkNPRElORyBtaW1lDQpGSUxFIC9pbnRlcm5ldC1kcmFmdHMvZHJhZnQtcGlua2VydG9uLXJkZHAt c2VjdXJpdHktMDAudHh0DQo= ------_=_NextPart_001_01C339B5.D54FB0BE Content-Type: application/octet-stream; name="ATT6369676.TXT" Content-Description: ATT6369676.TXT Content-Disposition: attachment; filename="ATT6369676.TXT" Content-Transfer-Encoding: base64 Q29udGVudC1UeXBlOiBtZXNzYWdlL2V4dGVybmFsLWJvZHk7DQoJc2l0ZT0iaW50ZXJuZXQtZHJh ZnRzIjsNCglkaXI9ImRyYWZ0LXBpbmtlcnRvbi1yZGRwLXNlY3VyaXR5LTAwLnR4dCI7DQoJbW9k ZT0iZnRwLmlldGYub3JnIjsNCglhY2Nlc3MtdHlwZT0iYW5vbi1mdHAiDQoNCg== ------_=_NextPart_001_01C339B5.D54FB0BE-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 23 15:36:38 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07238 for ; Mon, 23 Jun 2003 15:36:38 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5NJa9I04746 for rddp-archive@odin.ietf.org; Mon, 23 Jun 2003 15:36:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UX6n-0001ET-R0 for rddp-web-archive@optimus.ietf.org; Mon, 23 Jun 2003 15:36:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07215 for ; Mon, 23 Jun 2003 15:36:07 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UX6m-0005BF-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 15:36:08 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19UX6g-0005BC-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 15:36:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UX6g-0001BR-3N; Mon, 23 Jun 2003 15:36:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UX50-00019N-Aa for rddp@optimus.ietf.org; Mon, 23 Jun 2003 15:35:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA07162 for ; Mon, 23 Jun 2003 15:34:16 -0400 (EDT) From: cait@asomi.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UX4y-0005A8-00 for rddp@ietf.org; Mon, 23 Jun 2003 15:34:17 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19UX4o-00059s-00 for rddp@ietf.org; Mon, 23 Jun 2003 15:34:06 -0400 Received: from asomi.com (unknown [140.186.45.27]) by thebe.your-site.com (Postfix) with ESMTP id CCD2D2450A3; Mon, 23 Jun 2003 15:11:30 -0400 (EDT) Received: from 208.41.143.203 (SquirrelMail authenticated user cait@asomi.com) by webmail with HTTP; Mon, 23 Jun 2003 15:10:13 -0400 (EDT) Message-ID: <34034.208.41.143.203.1056395413.squirrel@webmail> Date: Mon, 23 Jun 2003 15:10:13 -0400 (EDT) Subject: RE: [rddp] Transport Neutral 'Connections' To: In-Reply-To: References: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> X-Priority: 3 Importance: Normal Cc: , , , X-Mailer: SquirrelMail (version 1.2.9) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable > > The offload proposals I've seen talk about having the > host OS handle TCP connection setup, then hand off an > existing connection. The way handoff works is that the > host OS signals the NIC that it will be getting the > connection, so it sets up filters to intercept packets. > The host OS then finishes processing any pending packets > that had been delivered to it, then uploads the > connection state to the NIC. The NIC then starts > handling the connection. > I would be interested in hearing any explanations that can be made publicly as to how you allow the RNIC to accept TCP connections and still keep the RNIC current on anti-DoS attacks. It is easy when you're running BSD on the RNIC, but in the long run I presume the goal is to avoid requiring an RNIC to have a full processor of its own. I suppose you could view TCP-connection grooming as a feature of the RNIC, but then how do you keep it current? This approach also runs into some nasty problems with accepting SCTP INIT requests for multi-homed endpoints. How do the two RNICs avoid race conditions? My presumption has been that at least some designs will want the RNIC to take control of RDMA, not of connection grooming. Therefore they will want to exercise full control over accepting connections, including logging. How do you propose to perform the safeguard/logging functions currently performed by INETD, TCPSERVER and similiar connection management daemons? > Yes, that's TCP. But handling RDMA on top of that isn't > hard. The same procedure can happen if the host OS has a > software RDMA stack. The software stack processes any > pending mis-labeled TCP packets (really RDMA but we > didn't know we were doing RDMA at that point). That will > advance RDMA connection state, then the host OS offloads > the RDMA processing to the card. > > So if a software/host OS stack is available, (3) won't > be too hard. It also won't require more complexity than > TCP connection offload will require. > > Take care, > > Bill ------------ Caitlin Bestler cait@asomi.com http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 23 18:23:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13488 for ; Mon, 23 Jun 2003 18:23:34 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5NMN8i26816 for rddp-archive@odin.ietf.org; Mon, 23 Jun 2003 18:23:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UZiN-0006yR-W7 for rddp-web-archive@optimus.ietf.org; Mon, 23 Jun 2003 18:23:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13456 for ; Mon, 23 Jun 2003 18:23:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UZiL-0006Ad-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 18:23:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19UZiF-0006Aa-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 18:22:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UZiG-0006wj-S6; Mon, 23 Jun 2003 18:23:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UZhk-0006wE-N4 for rddp@optimus.ietf.org; Mon, 23 Jun 2003 18:22:43 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13434 for ; Mon, 23 Jun 2003 18:22:09 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UZhS-0006AA-00 for rddp@ietf.org; Mon, 23 Jun 2003 18:22:10 -0400 Received: from saway.wasabisystems.com ([151.199.66.145] helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19UZhI-00068p-00 for rddp@ietf.org; Mon, 23 Jun 2003 18:22:00 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id B80E04013D; Mon, 23 Jun 2003 18:19:46 -0400 (EDT) Date: Mon, 23 Jun 2003 15:19:26 -0700 (PDT) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: cait@asomi.com Cc: Black_David@emc.com, hufferd@us.ibm.com, rddp@ietf.org Subject: RE: [rddp] Transport Neutral 'Connections' In-Reply-To: <34034.208.41.143.203.1056395413.squirrel@webmail> Message-ID: References: <277DD60FB639D511AC0400B0D068B71E0564CDA1@corpmx14.us.dg.com> <34034.208.41.143.203.1056395413.squirrel@webmail> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Mon, 23 Jun 2003 cait@asomi.com wrote: > > The offload proposals I've seen talk about having the > > host OS handle TCP connection setup, then hand off an > > existing connection. The way handoff works is that the > > host OS signals the NIC that it will be getting the > > connection, so it sets up filters to intercept packets. > > The host OS then finishes processing any pending packets > > that had been delivered to it, then uploads the > > connection state to the NIC. The NIC then starts > > handling the connection. > > > > I would be interested in hearing any explanations that > can be made publicly as to how you allow the RNIC to > accept TCP connections and still keep the RNIC current > on anti-DoS attacks. You don't. You let the host OS accept the connection, and then the host tells the NIC to take it over. That way you keep using the host OS security policies; you don't try to map N different host OS policies to M different NIC infrastructures. Also, doing things that way (the host OS stays in charge) answers a lot of questions about how to handle policy; you just let the host do it. :-) Note, part of this idea is that the NIC looks like a simple NIC in addition to everything else. :-) > It is easy when you're running BSD on the RNIC, but in > the long run I presume the goal is to avoid requiring > an RNIC to have a full processor of its own. I think the above class of solution will help with that. All the RNIC needs is to be told all the TCP state for a connection, and it can do the TCP offload. Since you also have to offload routing and ARP info, there's a whole hierarchy. So getting back to RNICs, it's no biggie to add RDMA as another layer on top. ;-) > I suppose you could view TCP-connection grooming as a > feature of the RNIC, but then how do you keep it current? > > This approach also runs into some nasty problems with > accepting SCTP INIT requests for multi-homed endpoints. > How do the two RNICs avoid race conditions? As above, the RNICs would be reacting to host OS direction, so the host OS would have to have chosen one of them. :-) > My presumption has been that at least some designs will > want the RNIC to take control of RDMA, not of connection > grooming. Therefore they will want to exercise full control > over accepting connections, including logging. My gut feel is that ALL implementations should do this. By letting the host OS do it, you remove a lot of issues. :-) > How do you propose to perform the safeguard/logging > functions currently performed by INETD, TCPSERVER and > similar connection management daemons? Note: I'm not proposing this. I'm noting what I've seen (in a vague conceptual manner :-) Either the host OS would accept the connection, move it to the RNIC, then have inetd stop it, or the host OS would do whatever it does with inetd and say tcpwrappers, and only move it to the NIC if it passes. > > Yes, that's TCP. But handling RDMA on top of that isn't > > hard. The same procedure can happen if the host OS has a > > software RDMA stack. The software stack processes any > > pending mis-labeled TCP packets (really RDMA but we > > didn't know we were doing RDMA at that point). That will > > advance RDMA connection state, then the host OS offloads > > the RDMA processing to the card. > > > > So if a software/host OS stack is available, (3) won't > > be too hard. It also won't require more complexity than > > TCP connection offload will require. Just to be clear, I'm not saying folks shouldn't do David's (2), but that (3) will probably be feasible. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 23 19:56:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA15603 for ; Mon, 23 Jun 2003 19:56:45 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5NNuGC04016 for rddp-archive@odin.ietf.org; Mon, 23 Jun 2003 19:56:16 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UbAW-00012h-Ks for rddp-web-archive@optimus.ietf.org; Mon, 23 Jun 2003 19:56:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA15598 for ; Mon, 23 Jun 2003 19:56:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UbAM-0006gW-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 19:56:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19UbAG-0006gT-00 for rddp-web-archive@ietf.org; Mon, 23 Jun 2003 19:56:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UbAH-00010o-5I; Mon, 23 Jun 2003 19:56:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Ub9d-0000ze-32 for rddp@optimus.ietf.org; Mon, 23 Jun 2003 19:55:21 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA15573 for ; Mon, 23 Jun 2003 19:55:19 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Ub9b-0006g2-00 for rddp@ietf.org; Mon, 23 Jun 2003 19:55:19 -0400 Received: from sj-iport-3-in.cisco.com ([171.71.176.72] helo=sj-iport-3.cisco.com) by ietf-mx with esmtp (Exim 4.12) id 19Ub9Q-0006fc-00 for rddp@ietf.org; Mon, 23 Jun 2003 19:55:08 -0400 Received: from cisco.com (171.68.223.137) by sj-iport-3.cisco.com with ESMTP; 23 Jun 2003 16:57:04 -0800 Received: from mira-sjc5-b.cisco.com (IDENT:mirapoint@mira-sjc5-b.cisco.com [171.71.163.14]) by sj-core-3.cisco.com (8.12.6/8.12.6) with ESMTP id h5NNrtQx011611 for ; Mon, 23 Jun 2003 16:53:55 -0700 (PDT) Received: from allyn-w2k01.cisco.com (dhcp-171-69-59-9.cisco.com [171.69.59.9]) by mira-sjc5-b.cisco.com (Mirapoint Messaging Server MOS 3.3.3-GR) with ESMTP id AIK86878; Mon, 23 Jun 2003 16:47:07 -0700 (PDT) Message-Id: <4.3.2.7.2.20030623144714.05c4b828@mira-sjcm-3.cisco.com> X-Sender: allyn@mira-sjcm-3.cisco.com X-Mailer: QUALCOMM Windows Eudora Version 4.3.2 Date: Mon, 23 Jun 2003 16:53:40 -0700 To: rddp@ietf.org From: Allyn Romanow Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="=====================_22449260==_.ALT" Subject: [rddp] NICELI Workshop on Network-I/O Convergence, Call for Participation Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --=====================_22449260==_.ALT Content-Type: text/plain; charset="us-ascii"; format=flowed Call for Participation: Workshop on Network-I/O Convergence: Experience, Lessons, Implications (NICELI) in conjunction with SIGCOMM 2003, Karlsruhe, Germany August 27, 2003 http://www.acm.org/sigs/sigcomm/sigcomm2003/workshop/niceli/ The performance and commodity price advantages of modern LANs have created a convergence of networks and I/O, promising price efficiencies and true interoperability, for storage and for cluster interconnect. The NICELI workshop provides a forum for researchers and practitioners to discuss the merits, drawbacks, applications, and practical implications of protocol and implementation designs. Three sessions of papers cover the promises and realities of network-I/O convergence; storage-specific protocols; and novel approaches. NICELI also includes two invited talks, by David Cheriton (Stanford) and Wu-chun Feng (Los Alamos and Ohio State). We invite protocol, software andhardware designers, and application users, to attend. --=====================_22449260==_.ALT Content-Type: text/html; charset="us-ascii" Call for Participation:
Workshop on Network-I/O Convergence: Experience, Lessons, Implications (NICELI)
in conjunction with SIGCOMM 2003, Karlsruhe, Germany
August 27, 2003
http://www.acm.org/sigs/sigcomm/sigcomm2003/workshop/niceli/

The performance and commodity price advantages of modern
LANs have created a convergence of networks and I/O,
promising price efficiencies and true interoperability, for
storage and for cluster interconnect. The NICELI workshop
provides a forum for researchers and practitioners to
discuss the merits, drawbacks, applications, and practical
implications of protocol and implementation designs. Three
sessions of papers cover the promises and realities of
network-I/O convergence; storage-specific protocols; and
novel approaches. NICELI also includes two invited talks, by
David Cheriton (Stanford) and Wu-chun Feng (Los Alamos and
Ohio State). We invite protocol, software andhardware designers,
and application users, to attend.

--=====================_22449260==_.ALT-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 24 14:10:00 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28696 for ; Tue, 24 Jun 2003 14:10:00 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5OI9WD27085 for rddp-archive@odin.ietf.org; Tue, 24 Jun 2003 14:09:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UsEW-00072m-Et for rddp-web-archive@optimus.ietf.org; Tue, 24 Jun 2003 14:09:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28650 for ; Tue, 24 Jun 2003 14:09:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UsE1-00070U-1Y; Tue, 24 Jun 2003 14:09:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UsDM-0006zu-VO for rddp@optimus.ietf.org; Tue, 24 Jun 2003 14:08:20 -0400 Received: from mx01.netapp.com (mx01.netapp.com [198.95.226.53]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA28529 for ; Tue, 24 Jun 2003 14:08:17 -0400 (EDT) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.9/8.12.9/NTAP-1.4) with ESMTP id h5OI5RJo025847 for ; Tue, 24 Jun 2003 11:05:27 -0700 (PDT) Received: from tmt.netapp.com ([10.58.48.91]) by frejya.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h5OI5Rtg012497 for ; Tue, 24 Jun 2003 11:05:27 -0700 (PDT) Message-Id: <5.2.1.1.2.20030624140434.0333fe58@silver.nane.netapp.com> Date: Tue, 24 Jun 2003 14:04:53 -0400 To: rddp@ietf.org From: "Talpey, Thomas" Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Fwd: [nfsv4] I-D ACTION:draft-talpey-nfs-rdma-problem-statement-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Of potential interest to RDDP. Tom. > ---------- Forwarded Message ---------- >Subject: [nfsv4] I-D ACTION:draft-talpey-nfs-rdma-problem-statement-00.txt >Date: Tue, 24 Jun 2003 06:49:53 -0400 >List-Help: >List-Subscribe: >,g?subject=subscribe> >From: >Cc: >Reply-To: > >A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > Title : NFS RDMA Problem Statement > Author(s) : T. Talpey, C. Juszczak > Filename : draft-talpey-nfs-rdma-problem-statement-00.txt > Pages : 15 > Date : 2003-6-23 > >This draft addresses applying Remote Direct Memory Access to the >NFS protocols. NFS implementations historically incur significant >overhead due to data copies on end-host systems, as well as other >sources. The potential benefits of RDMA to these implementations >are explored, and the reasons why RDMA is especially well-suited to >NFS and network file protocols in general are evaluated. > >A URL for this Internet-Draft is: >http://www.ietf.org/internet-drafts/draft-talpey-nfs-rdma-problem-statement- >00.txt > >To remove yourself from the IETF Announcement list, send a message to >ietf-announce-request with the word unsubscribe in the body of the message. > >Internet-Drafts are also available by anonymous FTP. Login with the username >"anonymous" and a password of your e-mail address. After logging in, >type "cd internet-drafts" and then > "get draft-talpey-nfs-rdma-problem-statement-00.txt". > >A list of Internet-Drafts directories can be found in >http://www.ietf.org/shadow.html >or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > >Internet-Drafts can also be obtained by e-mail. > >Send a message to: > mailserv@ietf.org. >In the body type: > "FILE /internet-drafts/draft-talpey-nfs-rdma-problem-statement-00.txt". > >NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > >Below is the data which will enable a MIME compliant mail reader >implementation to automatically retrieve the ASCII version of the >Internet-Draft. > > > ---------- End of Forwarded Message ---------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jun 24 17:18:00 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11491 for ; Tue, 24 Jun 2003 17:17:59 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5OLHWX25878 for rddp-archive@odin.ietf.org; Tue, 24 Jun 2003 17:17:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19UvAS-0006jJ-Ou for rddp-web-archive@optimus.ietf.org; Tue, 24 Jun 2003 17:17:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11485 for ; Tue, 24 Jun 2003 17:17:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Uv9x-0006hW-5J; Tue, 24 Jun 2003 17:17:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Uv96-0006h0-SJ for rddp@optimus.ietf.org; Tue, 24 Jun 2003 17:16:38 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11459 for ; Tue, 24 Jun 2003 17:15:50 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Uv8p-0007PT-00 for rddp@ietf.org; Tue, 24 Jun 2003 17:15:51 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Uv8f-0007Ob-00 for rddp@ietf.org; Tue, 24 Jun 2003 17:15:41 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9B7F0244B8D; Tue, 24 Jun 2003 17:11:41 -0400 (EDT) Date: Tue, 24 Jun 2003 16:11:02 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: ellen.m.deleganes@intel.com, jpink@windows.microsoft.com X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] RE: Security Draft - Definition of Trust / relevance Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Section 3 has the following definition: > > Trust - When one party depends upon the other party to not subvert > the goals of the protocols, e.g., it will not attempt to perform the > following attacks: spoofing, repudiation, information disclosure, > denial of service, or elevation of privilege. I don't find that definition to be that useful. Of all the things that can go wrong, the *intent* of the peer is only relevant when dealing with inadvertently authorized RDMA Reads. The key security issue with RDMA is ensuring that the ULP understands that *it* controls what access it gives the RNIC and its peers. There are various strategies for doing so, ranging from very cautious to very trusting. Remote access to memory is *only* authorized by posting a read or by enabling an STag for a Protection Domain. Security issues may arise when that exposure is wider than the ULP understands. This may because the ULP is being cavalier, or that the local interface does not offer adequate controls. For example the upper-layer-protocol may define a very clear procedure for use of a circular buffer, but it is unlikely that the local interface will offer any direct support for dynamically adjusting the exposed portion of the STag to match the protocol specification. In particular, I doubt anyone will ever build an RNIC that offers anything more than enabling an STag for a single range of bytes. But there is nothing in the wire protocol that requires the set of valid addresses for an STag to be a single contiguous range of bytes. Anyway, the point is that for a variety of reasons, more bytes may be exposed for read and/or write than would be the absolute minimum required to support the ULP. When this is write access, the data sink has a choice to trust to rely on its peer fully conforming to the upper layer protocol as specified, or it may assume that *all* bytes exposed for write are volatile and undefined. For writes, the *intent* of the peer is not the critical factor. Indeed I would argue given that the peer has an already established and potentially authenticated LLP stream that a far more serious concern is *accidental* targeting of exposed memory. After all, software engineers capable of making a mistake are far more common than those that would deliberately attack you. Intent is a relevant issue when dealing with inadvertently exposed read access. I would characterize the over-exposure issue as follows: -- The application MUST be able to expose buffers to read/write as required by the upper-layer-protocol. -- The application SHOULD be able to use the local interface to provide only the required access. However the application MAY enable more access to simplify its own logic. -- The application MAY assume that its peer will be in full compliance with the upper layer protocol. However it SHOULD NOT do so if failure to comply places the work or information about any other session at risk. (In other words, it is perfectly OK to take a "you break the rules, you mess yourself up" attitude. But you should not allow peer X breaking the rules to have any adverse affect on peer Y). _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 12:27:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20153 for ; Wed, 25 Jun 2003 12:27:41 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PGRED09779 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 12:27:14 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VD73-0002Xe-V0 for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 12:27:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20126 for ; Wed, 25 Jun 2003 12:27:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VD73-0005BX-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:27:13 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VD6y-0005BO-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:27:08 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VD6q-0002T0-6p; Wed, 25 Jun 2003 12:27:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VD6d-0002SB-OL for rddp@optimus.ietf.org; Wed, 25 Jun 2003 12:26:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20072 for ; Wed, 25 Jun 2003 12:26:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VD6O-0005AV-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:26:32 -0400 Received: from palrel11.hp.com ([156.153.255.246]) by ietf-mx with esmtp (Exim 4.12) id 19VD6D-00059D-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:26:21 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id BB9031C011E6 for ; Wed, 25 Jun 2003 09:23:24 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.170.187]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h5PGIVT03511 for ; Wed, 25 Jun 2003 09:18:31 -0700 (PDT) Message-Id: <5.1.0.14.2.20030625091842.026f40d0@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Wed, 25 Jun 2003 09:22:07 -0700 To: rddp@ietf.org From: Michael Krause Subject: Re: [rddp] Re: security-draft : Stale Data In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 08:37 PM 6/24/2003 -0500, Caitlin Bestler wrote: > >From 7.4.2 "Using RDMA Read to Access Stale Data" > > > Because of this, it is RECOMMENDED that the Local Peer ensure that > > no stale data is contained in the buffer when remote read access > > rights are initially granted (this can be done by zeroing the > > contents of the memory, for example). > >This is only a concern if the buffer might contain data that originated >outside of the current session. That could be done by zeroing the >contents of memory at the beginning of the session, rather than >before advertising the buffer. > >If a single buffer pool is used across multiple sessions, however, >then the ULP should indeed obliterate any old contents before >enabling it for remote read access. While I agree in principle, it seems to me that anything enabled for remote read is nominally updated to contain new data that one desires the remote to read thus I would expect either the region / window to be updated prior to the advertisement either with new application data with anything that goes beyond the updated address space (one can always advertise more than the actual updated data such as array that is sequentially updated / read over time) to be zero / randomized prior to it initially being advertised. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 12:37:36 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20555 for ; Wed, 25 Jun 2003 12:37:36 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PGb9G10999 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 12:37:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDGf-0002rC-5f for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 12:37:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20548 for ; Wed, 25 Jun 2003 12:37:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VDGe-0005Fw-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:37:08 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VDGY-0005Ft-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:37:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDGX-0002qE-Gf; Wed, 25 Jun 2003 12:37:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDFR-0002oa-CN for rddp@optimus.ietf.org; Wed, 25 Jun 2003 12:36:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20497 for ; Wed, 25 Jun 2003 12:35:50 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VDAY-0005D0-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:30:50 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19VDAN-0005Cu-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:30:39 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id F30ED245804; Wed, 25 Jun 2003 12:00:24 -0400 (EDT) Date: Wed, 25 Jun 2003 10:59:45 -0500 From: Caitlin Bestler Subject: RE: [rddp] Transport Neutral "Connections" To: rddp@ietf.org Cc: wrstuden@wasabisystems.com X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, June 23, 2003, wrstuden@wasabisystems.com wrote: >The offload proposals I've seen talk about having the host OS >handle TCP connection setup, then hand off an existing >connection. The way handoff works is that the host OS signals >the NIC that it will be getting the connection, so it sets up >filters to intercept packets. The host OS then finishes >processing any pending packets that had been delivered to it, >then uploads the connection state to the NIC. The NIC then >starts handling the connection. My first read of what you were saying was wrong. Having read your clarification, I still don't accept that it is an easy solution. The "connection state" that you are suggesting simply be handed back to the offload engine would have to include: -- All prior portions of the TCP stream that have not been consumed. -- Notations on which portions of that have been acked. Keep in mind that portions may have been SACKed. -- All prior transmits that have not been acked. -- and of course the obvious items: window size, scaling, congestion... And all of the above has to work with varying scale options. I believe that the model of doing a TCP specific negotiation and *then* deciding to enter RDMA mode as a by-product can and should be supported. But we need to determine what the real requirements are, and explicitly state all of the simplifying assumptions. Examples would include "all prior TCP segments must have been acked before the final streaming mode message may be sent." I also believe we can support an equivalent capability for SCTP, *if* the sole criteria for entry to RDMA mode is capability. I have not seen any scenarios for optional-RDMA session establishment that call for not establishing it even when both sides are capable. So I do not anticipate that this would be an issue, particularly with SCTP where there is not even a marker-penalty for using RDMA mode. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 12:58:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA21211 for ; Wed, 25 Jun 2003 12:58:34 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PGw7d14833 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 12:58:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDax-0003rA-4M for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 12:58:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA21206 for ; Wed, 25 Jun 2003 12:58:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VDaw-0005Oe-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:58:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VDar-0005Ob-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 12:58:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDaq-0003pW-4r; Wed, 25 Jun 2003 12:58:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VDab-0003ox-85 for rddp@optimus.ietf.org; Wed, 25 Jun 2003 12:57:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA21197 for ; Wed, 25 Jun 2003 12:57:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VDaa-0005OO-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:57:44 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19VDaP-0005O1-00 for rddp@ietf.org; Wed, 25 Jun 2003 12:57:33 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 39950245967; Wed, 25 Jun 2003 12:56:57 -0400 (EDT) Date: Wed, 25 Jun 2003 11:56:56 -0500 From: Caitlin Bestler Subject: Re: [rddp] Re: security-draft : Stale Data To: Michael Krause , rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030625091842.026f40d0@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, June 25, 2003, krause@cup.hp.com wrote: >At 08:37 PM 6/24/2003 -0500, Caitlin Bestler wrote: >> >From 7.4.2 "Using RDMA Read to Access Stale Data" >> >> > Because of this, it is RECOMMENDED that the Local Peer ensure that >> > no stale data is contained in the buffer when remote read access >> > rights are initially granted (this can be done by zeroing the >> > contents of the memory, for example). >> >>This is only a concern if the buffer might contain data that originated >>outside of the current session. That could be done by zeroing the >>contents of memory at the beginning of the session, rather than >>before advertising the buffer. >> >>If a single buffer pool is used across multiple sessions, however, >>then the ULP should indeed obliterate any old contents before >>enabling it for remote read access. > >While I agree in principle, it seems to me that anything >enabled for remote read is nominally updated to contain new >data that one desires the remote to read thus I would expect >either the region / window to be updated prior to the >advertisement either with new application data with anything >that goes beyond the updated address space (one can always >advertise more than the actual updated data such as array that >is sequentially updated / read over time) to be zero / >randomized prior to it initially being advertised. > But it may be advertised once, and then updated multiple times. The ULP may have its own methods of knowing when the advertisement is logically void. For example B might write to a buffer held by A, then somehow let C and D know that it has been updated. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 18:14:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06538 for ; Wed, 25 Jun 2003 18:14:00 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PMDZq20589 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 18:13:35 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VIWF-0005M0-0n for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 18:13:35 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06500 for ; Wed, 25 Jun 2003 18:13:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VIVh-0005HS-6r; Wed, 25 Jun 2003 18:13:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VIUa-0005DG-OL for rddp@optimus.ietf.org; Wed, 25 Jun 2003 18:12:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06301 for ; Wed, 25 Jun 2003 18:11:33 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VIUI-00015R-00 for rddp@ietf.org; Wed, 25 Jun 2003 18:11:35 -0400 Received: from msgbas1tx.cos.agilent.com ([192.25.240.37] helo=msgbas2x.cos.agilent.com) by ietf-mx with esmtp (Exim 4.12) id 19VIU8-00013z-00 for rddp@ietf.org; Wed, 25 Jun 2003 18:11:24 -0400 Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id 46DE243B; Wed, 25 Jun 2003 16:09:14 -0600 (MDT) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by relcos2.cos.agilent.com (Postfix) with SMTP id 136EC19; Wed, 25 Jun 2003 16:09:14 -0600 (MDT) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Wed, 25 Jun 2003 16:09:08 -0600 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Wed, 25 Jun 2003 16:09:07 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E03FA@axcs04.cos.agilent.com> To: cait@asomi.com, krause@cup.hp.com, rddp@ietf.org Subject: RE: [rddp] Re: security-draft : Stale Data Date: Wed, 25 Jun 2003 16:09:06 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , If it is advertised once and updated multiple times, then the ULP can meet the requirement by ensuring that data for the first Read has been put into the buffer (e.g. any stale data from another session) before it enables remote read access. Even if this involves extra operations to separately enable remote read access, this may be less of a burden then zeroing out the memory. It would be reasonable to mention more than one alterntative for how the Local Peer can ensure that stale data is not exposed such as: >> > Because of this, it is RECOMMENDED that the Local Peer ensure that >> > no stale data is contained in the buffer when remote read access >> > rights are initially granted (this can be done by zeroing the >> > contents of the memory, or by ensuring that appropriate data has been written into the buffer before remote read access is enabled. >> > for example). Regards, Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, June 25, 2003 9:57 AM To: Michael Krause; rddp@ietf.org Subject: Re: [rddp] Re: security-draft : Stale Data > >While I agree in principle, it seems to me that anything >enabled for remote read is nominally updated to contain new >data that one desires the remote to read thus I would expect >either the region / window to be updated prior to the >advertisement either with new application data with anything >that goes beyond the updated address space (one can always >advertise more than the actual updated data such as array that >is sequentially updated / read over time) to be zero / >randomized prior to it initially being advertised. > But it may be advertised once, and then updated multiple times. The ULP may have its own methods of knowing when the advertisement is logically void. For example B might write to a buffer held by A, then somehow let C and D know that it has been updated. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 19:14:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA10569 for ; Wed, 25 Jun 2003 19:14:46 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PNEJD01667 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 19:14:19 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJT1-0000Qo-Ih for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 19:14:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA10526 for ; Wed, 25 Jun 2003 19:14:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VJSn-00021O-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:14:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VJSi-00021L-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:14:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJSi-0000P9-MY; Wed, 25 Jun 2003 19:14:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJSI-0000On-Vy for rddp@optimus.ietf.org; Wed, 25 Jun 2003 19:13:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA10489 for ; Wed, 25 Jun 2003 19:13:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VJSH-00020Z-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:13:33 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19VJS6-0001yJ-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:13:22 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9997F245459; Wed, 25 Jun 2003 19:10:14 -0400 (EDT) Date: Wed, 25 Jun 2003 18:10:13 -0500 From: Caitlin Bestler Subject: RE: [rddp] Re: security-draft : Stale Data To: pat_thaler@agilent.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A0132E03FA@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, June 25, 2003, pat_thaler@agilent.com wrote: > If it is advertised once and updated multiple times, then the > ULP can meet the requirement by ensuring that data for the > first Read has been put into the buffer (e.g. any stale data > from another session) before it enables remote read access. > Even if this involves extra operations to separately enable > remote read access, this may be less of a burden then zeroing > out the memory. How an application chooses to prepare a buffer that it has enabled for read is a ULP issue. The tradeoffs between enabling/ disabling in the ULP versus doing so with the local interface are really out of scope. But it is not a *security* issue. Obviously a ULP should ensure that a buffer is not read before the contents are actually there. But if the wrong contents are read that is a program error. It is no more a "security" issue than any program error is. The section in question was dealing with ensuring that the remote peer not gain access to unitialized bytes that might contain data that was not supposed to be known to them. If in fact the only risk is that the remote peer might read *old* data that was *previously* offered to them, then there really is no security issue. My position is that ULP developers are looking for advice that is specific to RDMA, not general tips on robust programming. Initializing the buffer *before* it is enabled the *first* time accomplishes this. There is no *security* reason why it cannot remain enabled from that point forward even if its contents are logically void. Just because one component in a distributed system MUST NOT do X does not mean that the system MUST, or even SHOULD, be actively configured to detect/prevent it from doing so. This is especially true if a violation put no other sessions at any risk. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 19:47:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12271 for ; Wed, 25 Jun 2003 19:47:39 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PNlDN09464 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 19:47:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJyr-0002SZ-4K for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 19:47:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12215 for ; Wed, 25 Jun 2003 19:47:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VJyj-0002QV-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:47:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VJyd-0002QS-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:47:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJyf-0002PY-1u; Wed, 25 Jun 2003 19:47:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VJxz-0002Ny-Pq for rddp@optimus.ietf.org; Wed, 25 Jun 2003 19:46:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12156 for ; Wed, 25 Jun 2003 19:46:15 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VJxx-0002Ph-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:46:18 -0400 Received: from msgbas1tx.cos.agilent.com ([192.25.240.37] helo=msgbas2x.cos.agilent.com) by ietf-mx with esmtp (Exim 4.12) id 19VJxn-0002Ol-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:46:07 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id 0BA30972; Wed, 25 Jun 2003 17:44:42 -0600 (MDT) Received: from wcosbh22.cos.agilent.com (wcosbh22.cos.agilent.com [130.29.152.178]) by relcos1.cos.agilent.com (Postfix) with ESMTP id EB1ADC8; Wed, 25 Jun 2003 17:44:41 -0600 (MDT) content-class: urn:content-classes:message Subject: RE: [rddp] Re: security-draft : Stale Data Date: Wed, 25 Jun 2003 17:44:33 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E03FC@axcs04.cos.agilent.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thread-Topic: [rddp] Re: security-draft : Stale Data X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Thread-Index: AcM7bvHfk29l1cx6Sfi8KPt9vxIYgwAA1KPw To: , , Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Caitlin, What I suggested isn't anything different then what you are saying - the = important thing is protecting from data left from a prior application or = sessions use of the memory (which I called "stale" data because that is = what the original text used). There are at least two ways to protect = against that security hole: a) initialize the memory to zero before advertising it, or b) ensure that the new data has been written to the buffer before = advertising it. For memory that is dynamically moved around between sessions, (a) can = entail a lot of unnecessary extra work while (b) just requires being = careful about the order of execution of the operations one has to do = anyway or that one may sometimes have to do a separate operation to = enable remote read access. Both (a) and (b) are good examples of how to = provide the protection and both should be included in the text.=20 My suggestion only addressed what should be done before enabling the = first access and not anything about how a ULP coordinates successive = modifications and accesses of the same memory for a session so I don't = understand your objection. Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, June 25, 2003 4:10 PM To: pat_thaler@agilent.com; rddp@ietf.org Subject: RE: [rddp] Re: security-draft : Stale Data On Wednesday, June 25, 2003, pat_thaler@agilent.com wrote: > If it is advertised once and updated multiple times, then the > ULP can meet the requirement by ensuring that data for the > first Read has been put into the buffer (e.g. any stale data > from another session) before it enables remote read access. > Even if this involves extra operations to separately enable > remote read access, this may be less of a burden then zeroing > out the memory. How an application chooses to prepare a buffer that it has enabled for read is a ULP issue. The tradeoffs between enabling/ disabling in the ULP versus doing so with the local interface are really out of scope. But it is not a *security* issue. Obviously a ULP should ensure that a buffer is not read before the contents are actually there. But if the wrong contents are read that is a program error. It is no more a "security" issue than any program error is. The section in question was dealing with ensuring that the remote peer not gain access to unitialized bytes that might contain data that was not supposed to be known to them. If in fact the only risk is that the remote peer might read *old* data that was *previously* offered to them, then there really is no security issue. My position is that ULP developers are looking for advice that is specific to RDMA, not general tips on robust programming. Initializing the buffer *before* it is enabled the *first* time accomplishes this. There is no *security* reason why it cannot remain enabled from that point forward even if its contents are logically void. Just because one component in a distributed system MUST NOT do X does not mean that the system MUST, or even SHOULD, be actively configured to detect/prevent it from doing so. This is especially true if a violation put no other sessions at any risk. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jun 25 19:53:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12582 for ; Wed, 25 Jun 2003 19:53:46 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PNrK110805 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 19:53:20 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VK4m-0002oC-9d for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 19:53:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12547 for ; Wed, 25 Jun 2003 19:53:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VK4X-0002UI-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:53:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19VK4R-0002UF-00 for rddp-web-archive@ietf.org; Wed, 25 Jun 2003 19:52:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VK4S-0002m3-SL; Wed, 25 Jun 2003 19:53:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VK46-0002kx-0k for rddp@optimus.ietf.org; Wed, 25 Jun 2003 19:52:38 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12507 for ; Wed, 25 Jun 2003 19:52:33 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19VK44-0002U2-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:52:36 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19VK3t-0002TF-00 for rddp@ietf.org; Wed, 25 Jun 2003 19:52:25 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id EFD9C24534B; Wed, 25 Jun 2003 19:51:49 -0400 (EDT) Date: Wed, 25 Jun 2003 18:51:48 -0500 From: Caitlin Bestler Subject: RE: [rddp] Re: security-draft : Stale Data To: pat_thaler@agilent.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A0132E03FC@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, June 25, 2003, pat_thaler@agilent.com wrote: >Caitlin, > <> > >My suggestion only addressed what should be done before enabling the >first access and not anything about how a ULP coordinates successive >modifications and accesses of the same memory for a session so I don't >understand your objection. > >Pat > OK, in the context of the original text I read your statement to imply that DDP-level enabling should be done *each* time that the buffer was advertised. As we both agree, this is not necessary. Rereading your comment, I agree that it is not implied in your answer, just in the originl draft text. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jun 26 14:20:53 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12795 for ; Thu, 26 Jun 2003 14:20:53 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PFhsP19063 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 11:43:54 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCR3-0004wd-Pl for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 11:43:49 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16913 for ; Wed, 25 Jun 2003 11:43:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCQE-0004gC-O5; Wed, 25 Jun 2003 11:42:58 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VBzM-0006YL-II for rddp@optimus.ietf.org; Wed, 25 Jun 2003 11:15:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02085 for ; Tue, 24 Jun 2003 21:38:10 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UzEg-0001KW-00 for rddp@ietf.org; Tue, 24 Jun 2003 21:38:10 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19UzEV-0001KP-00 for rddp@ietf.org; Tue, 24 Jun 2003 21:37:59 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 593812448B8; Tue, 24 Jun 2003 21:38:02 -0400 (EDT) Date: Tue, 24 Jun 2003 20:37:23 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: jpink@microsoft.com, Ellen M Deleganes X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: security-draft : Stale Data Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit >From 7.4.2 "Using RDMA Read to Access Stale Data" > Because of this, it is RECOMMENDED that the Local Peer ensure that > no stale data is contained in the buffer when remote read access > rights are initially granted (this can be done by zeroing the > contents of the memory, for example). This is only a concern if the buffer might contain data that originated outside of the current session. That could be done by zeroing the contents of memory at the beginning of the session, rather than before advertising the buffer. If a single buffer pool is used across multiple sessions, however, then the ULP should indeed obliterate any old contents before enabling it for remote read access. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jun 26 14:25:56 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA13182 for ; Thu, 26 Jun 2003 14:25:56 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PFmcb27908 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 11:48:38 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCVh-0007Dy-4j for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 11:48:37 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17720 for ; Wed, 25 Jun 2003 11:48:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCVB-0006tT-Rb; Wed, 25 Jun 2003 11:48:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VBzA-0007wP-TX for rddp@optimus.ietf.org; Wed, 25 Jun 2003 11:15:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA03898 for ; Tue, 24 Jun 2003 22:51:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19V0NA-0001cJ-00 for rddp@ietf.org; Tue, 24 Jun 2003 22:51:00 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19V0N0-0001cE-00 for rddp@ietf.org; Tue, 24 Jun 2003 22:50:50 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 21A8F245097; Tue, 24 Jun 2003 22:50:37 -0400 (EDT) Date: Tue, 24 Jun 2003 21:49:57 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: jpink@microsoft.com, Ellen M Deleganes X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: security-draft : 7.4.6 - Using Multiple STags Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit >7.4.6 Using Multiple Stags to access the same buffer > > Multiple STags accessing the same buffer at the same time can result > in unintentional information disclosure. When one STag has remote > read access enabled and a different STag has remote write access > enabled to the same buffer, it is possible for one connection to > view the contents that have been written by another Remote Peer. > This is not a problem when all of the STags accessing the buffer > have only remote read enabled. For Trust Models NS-NT, S-NT, S-LT it > is RECOMMENDED that multiple Remote Peers not be granted access to > the same buffer through different STags at the same time. A buffer > should be exposed to only one Untrusted Remote Peer at a time to > ensure that no information disclosure occurs between peers. This strikes me as highly redundant data that might dissuade some legitimate usage of multiple STags. The general rule is that the contents of *any* advertised buffer are logically void *until* completion. That applies to local acces, *or* any remotely enabled access. A distributed ULP that has chosen to be "trusting" (i.e. that it is only required to work when all participants follow the upper layer protocol) could easily have a buffer that is exposed to multiple participants. However, it would be the ULPs responsibility to ensure that at most one process was attempting to update the buffer at any time, and that nobody was trying to read it during an update. That is a general rule for *any* multi-threaded, let alone, multi-processor or distributed application. It has nothing to do with RDMA. I really don't see the value of including it. Use of unitialized pointers is a bad idea too. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jun 26 14:45:20 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14317 for ; Thu, 26 Jun 2003 14:45:19 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PFkK324745 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 11:46:20 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCTT-0006G7-CX for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 11:46:19 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17366 for ; Wed, 25 Jun 2003 11:45:53 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCSZ-0005ri-II; Wed, 25 Jun 2003 11:45:23 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VC0A-0007wP-DB for rddp@optimus.ietf.org; Wed, 25 Jun 2003 11:16:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA01361 for ; Tue, 24 Jun 2003 21:31:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UyaW-0000pK-00 for rddp@ietf.org; Tue, 24 Jun 2003 20:56:40 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19UyaM-0000pF-00 for rddp@ietf.org; Tue, 24 Jun 2003 20:56:30 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id DFB982449A6; Tue, 24 Jun 2003 20:56:17 -0400 (EDT) Date: Tue, 24 Jun 2003 19:55:38 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: jpink@microsoft.com, Ellen M Deleganes X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: security-draft : completions and ULP pacing Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit >7.1.4 Limiting the Scope of the Completion Queue > > Completions associated with sending and receiving data, or > setting up buffers for sending and receiving data, could be > accumulated in a shared Completion Queue for a group of > RDMAP/DDP Streams, or a specific RDMAP/DDP Stream could have a > dedicated Completion Queue. Limiting Completion Queue > association to one, or a small number of RDMAP/DDP Streams can > prevent several forms of Denial of Service attacks. There is an assumption implicit in this section that I believe is unwise. At the minimum, a vunerability that is a defect of one proposed local interface is being presented as a general security concern. As noted previously, a DDP stream relies upon ULP flow control more so than does a non-RDMA stream. Normally the connection/ association is flow controlled based upon LLP congestion control and LLP buffer capacity advertising. A fully optimized RDMA-enabled connection/association makes no use of LLP buffering. Therefore, reducing transport flow control to congestion control only. The assumption in the RDMA Architectural model is that the data sink must enable each receive. The presumption is that the Data Sink is able to communicate credits, most probably in its untagged messages, after enabling the matching number of receives. One of the strengths of the iWARP protocol suite is that by leaving credit management to the ULP, the ULP can easily dynamically adjust the credit level. Indeed, it can be an automatic feedback associated with processing. As the load in processing requests increases, new credits will be issued more slowly. The following loop is an example: post_receive() while (get_completion(recv_completion_queue)) process_message() post_receive() send_response() The longer that "process_message" takes, the slower the peer will receive its new credit. Note that the new receive is authorized just before sending the response. The key is that the ULP explicitly authorizes the reception of each untagged message. I will point out that there were no objections to this rule in the architecture draft. The only objection was the one that I raised, which stated that the local interface MAY allow the ULP to authorize reception of an untagged message without supplying a buffer -- but that the ULP MUST still authorize each receive. This is a rule that I believe the local interface SHOULD/MUST enforce. The local interface can allow a Receive Queue to be fed from a buffer pool. When dealing with a large number of clients this may be a wise policy decision. If this is combined with authorizing receptions on a per stream basis there is far less of a security concern than if the the presence of a buffer implies authorization. Under the latter scenario, a peer that is not in compliance with its ULP protocol rules will quickly exceed its alloted number of receives. The DDP stream would then be terminated. If instead it is the receive queues that are shared, one over-eager peer can take more than its fair share of buffers. Even without exhausting the buffer pool, it could thereby generate too many completions. If the each "receive must be enabled for an endpoint" rule is followed, then the maximum size of any completion queue can be easily determined. Given the relatively small size of completions, there would be no reason to favor multiple completion queues over a single completion queue. That should be left to the designer of the local ULP implementation. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jun 26 14:46:20 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14368 for ; Thu, 26 Jun 2003 14:46:20 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PFjdK23260 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 11:45:39 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCSp-00062s-I5 for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 11:45:39 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17278 for ; Wed, 25 Jun 2003 11:45:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCSH-0005hY-9P; Wed, 25 Jun 2003 11:45:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VC0K-00089b-Nn for rddp@optimus.ietf.org; Wed, 25 Jun 2003 11:16:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA12385 for ; Tue, 24 Jun 2003 17:45:43 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Uvbl-0007Yr-00 for rddp@ietf.org; Tue, 24 Jun 2003 17:45:45 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Uvba-0007Yb-00 for rddp@ietf.org; Tue, 24 Jun 2003 17:45:34 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 23284244C93; Tue, 24 Jun 2003 17:45:24 -0400 (EDT) Date: Tue, 24 Jun 2003 16:44:45 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: jpink@microsoft.com, Ellen M Deleganes X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Subject: [rddp] Re: security-draft : scattered points Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable page 6: > A Non-Privileged Application=C3=86s capabilities are a logical sub- > set of the Privileged Application=C3=86s. It is assumed by the > local host infrastructure that a Non-Privileged Application is > Untrusted. All Non-Privileged Application interactions with > the RNIC Engine that could affect other applications need to > be done through a Trusted intermediary that can verify the > Non- Privileged Application requests.=20 I believe the correct specification is that all interactions *that require verification* must be passed through a Trusted intermediary. As emphasized later in the document, there can be "safe" interactions, which are interpreted in the context of earlier validated commands (especially work requests). The general principle is that the RNIC Engine MUST NOT perform any access to memory that it was not authorized to access, and that said authorization must come from an entity that had the right to grant that permission. Different execution environments may have more than two classes of privilege. Whatever privileges a given software component has, it ought to be allowed to give to the RNIC Engine. (I avoid saying "should" because I recognize the value to RNIC developers to standardize their interface with the host system. However, as valuable as that may be, I believe it is out of scope for this document). Missing section - Definitions Especially "connection". Which can be an LLP communication link with a rmeote host which must be estabished, maintained, have acks, congestion windows, etc. Such as a TCP connection or an SCTP association. Or it could be the LLP Stream over which two RDMAP/DDP endpoints communicate DDP Segments. Such as a TCP connection or a pair of SCTP streams. If you review the group charter you'll realize that assumptions that the LLP is TCP are to be avoided. >4.2.3 Page Translation Tables=20 > > Page Translation Tables are the structures used by the RNIC to > be able to access application memory for data transfer > operations. Even though these structures are called "Page" > Translation Tables, they may not reference a page at all =C3=BB > conceptually they are used to map an application address space > representation of a buffer to the physical addresses that are > used by the RNIC Engine to move data. If on a specific system, > a mapping is not used, then a subset of the attacks examined > may be appropriate. STags must be translated from the STag-specific address space to an address space that is RNIC Engine accessible. It would be valid for a system to have a feature whereby memory maps could be defined for use by IO cards. An STag could simply be a valid window of such an IO Card accessible memory map. In most cases the translation will be to physical addresses, but that is merely a highly prevelant design choice. It has nothing to do with the protocol. There is an important aspect of Page Translation Tables that should be mentioned here -- namely that multiple STags SHOULD be able to share a single memory map. In IB terms, you have to be able to create a Memory Region and use it with multiple Memory Windows. It is important that the ULP actually be able to precisely tailor its memory exposure to match its actual requirements. The goal expressed in the DAT Collaborative's DAPL specifications is that window binding has to be sufficiently lightweight that it is reasonable to perform on a per-data-transfer basis. That all but requires a distinction between the mapping resource and the range-checking STag. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jun 26 14:50:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14638 for ; Thu, 26 Jun 2003 14:50:22 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5PFjvV24052 for rddp-archive@odin.ietf.org; Wed, 25 Jun 2003 11:45:57 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCSv-0006BP-GQ for rddp-web-archive@optimus.ietf.org; Wed, 25 Jun 2003 11:45:45 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17332 for ; Wed, 25 Jun 2003 11:45:42 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VCSQ-0005ml-SG; Wed, 25 Jun 2003 11:45:14 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19VC0B-0006YL-VX for rddp@optimus.ietf.org; Wed, 25 Jun 2003 11:16:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA01648 for ; Tue, 24 Jun 2003 21:32:49 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19UxTS-0000Gr-00 for rddp@ietf.org; Tue, 24 Jun 2003 19:45:18 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19UxTH-0000GJ-00 for rddp@ietf.org; Tue, 24 Jun 2003 19:45:07 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 806CE244A7D; Tue, 24 Jun 2003 19:19:25 -0400 (EDT) Date: Tue, 24 Jun 2003 18:18:46 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: jpink@microsoft.com, Ellen M Deleganes X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Subject: [rddp] Re: security-draft : more scattered points Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable >4.2.4 STag Namespace=20 > > The DDP specification defines a 32-bit namespace for the STag. > Implementations may vary in terms of the actual number of > STags that are supported. In any case, this is a bounded > resource that can come under attack. The DDP specification carefully avoids specifying the scope of the 32-bit namespace. It is *likely* to match the scope of the RNIC, but could be system-wide, per IP address or even per Protection Domain. The above paragraph may give a reader the incorrect impression that the scope is system wide. >4.2.5 Completion Queues=20 > > Completion Queues are used in this specification to > conceptually represent how the RNIC Engine notifies the > Application of the completion of the transmission of data, or > the completion of the reception of data through the Data > Transfer Interface. Because there could be many transmissions > or receptions in flight at any one time, completions are > modeled as a queue rather than a single event. An > implementation may also use the Completion Queue to notify the > application of other activities, for example, the completion > of a mapping of an STag to a specific application buffer. It needs to be clarified that there is only a signal completion for a *set* of tagged and untagged messages. This makes it clear that sizing of the completion queue is a solvable problem. The maximum number of completions is easily derived from the number of receives enabled and the number of work requests sent (plus an allowance for other errors). While the consolidation of ULP notifications is major benefit of RDMAP/DDP, it is a security concern. An application that over-exposes its buffers will not have an "audit trail" to examine to see what its peer actually sent. > 1. NS-NT - Non-Shared Local Resources, no Local Trust, no > Remote Trust =C3=BB typically a server application that wants to > run in a mode that has the least number of potential attacks. It should be observed that a server application may choose to go even *more* paranoid, and simply not advertise any of its own buffers. The benefits of RDMA can be achieved by requiring the client to expose its buffers. The DAFS protocol is the obvious example of this. > 4. S-LT - Shared Local Resources, Local Trust, no Remote > Trust =C3=BB typically an application, which provides a session > layer and uses multiple Streams, to provide additional > throughput or fail- over capabilities. All of the Streams > within the local application trust each other, but do not > trust the remote peer. Could you give an example? I hope you are not implying that the ULP should be solving problems that are the responsiblity of the LLP, such as multistreaming and multi-homing. >7.1.1 Protection Domain (PD)=20 > > Protection Domains are associated with two of the resources of > concern, connection context memory and STags associated with > data buffers. Protection Domains are used mainly to ensure > that an STag can only be used to access the associated data > buffer through connections in the same Protection Domain as > that STag.=20 > > For the Trust Models that are defined to have non-shared > resources (Trust Models NS-NT and NS-RT), it is recommended > that each Stream be associated with its own, unique Protection > Domain. For those Trust Models where resources are shared > (Trust Models S-NT, S-LT and S-T), it is recommended that > Protection Domain be limited to the number of Streams that > share the same Trust Model. I don't see the connection between Protection Domain scope and the Trust Models. My advice would simply be that a Protection Domain should be no wider than the ULP requires. Use of a common Protection Domain for multiple DDP Streams should only be done to meet an application requirement. It should not be done because the RNIC is being stingy and offering fewer PD resources than the number of DDP Streams it can support. >7.1.2 Limiting STag Scope=20 > > The key to protecting a local data buffer is to limit the > scope of its STag to the level appropriate for the Trust > Model. The scope of the STag can be measured in multiple ways.=20 > > * Number of Connections and/or Streams on which the STag is > valid. One way to limit the scope of the STag is to limit > the connections and/or Streams that are allowed to use the > STag. As noted in the previous section, use of Protection > Domains appropriately can limit the scope of the STag. > It is also possible to create an STag that is valid only on a > single connection, even in the case where several connections > are associated with the Protection Domain of the STag.=20 My reading of the DDP spec is that setting the Protection Domain is *the* way to control the scope of an STag. Specifically, each endpoint has an assigned Protection Domain. The validity of an STag is checked against that PD, or within the scope of it. That the only two checks that I am aware of that are consistent with the DDP specs are: packet.stag.pd =3D=3D endpoint.pd isValid(endpoint.pd+packet.stag) =20 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 07:55:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13048 for ; Mon, 30 Jun 2003 07:55:04 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5UBsYg08492 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 07:54:34 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxEw-0002Cj-R6 for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 07:54:34 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA12907 for ; Mon, 30 Jun 2003 07:54:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxEP-00027j-3w; Mon, 30 Jun 2003 07:54:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxEB-00024U-Nd for rddp@optimus.ietf.org; Mon, 30 Jun 2003 07:53:47 -0400 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA12572; Mon, 30 Jun 2003 07:53:46 -0400 (EDT) Message-Id: <200306301153.HAA12572@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Mon, 30 Jun 2003 07:53:46 -0400 Subject: [rddp] I-D ACTION:draft-ietf-rddp-arch-02.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : The Architecture of Direct Data Placement (DDP)And Remote Direct Memory Access (RDMA)On Internet Protocols Author(s) : S. Bailey, T. Talpey Filename : draft-ietf-rddp-arch-02.txt Pages : 17 Date : 2003-6-27 This document defines an abstract architecture for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to run on Internet Protocol-suite transports. This architecture does not necessarily reflect the proper way to implement such protocols, but is, rather, a descriptive tool for defining and understanding the protocols. DDP allows the efficient placement of data into buffers designated by Upper Layer Protocols (e.g. RDMA). RDMA provides the semantics to enable Remote Direct Memory Access between peers in a way consistent with application requirements. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-arch-02.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-arch-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-arch-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-6-27150825.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-arch-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-arch-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-6-27150825.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 10:02:24 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA13047 for ; Mon, 30 Jun 2003 07:55:04 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5UBsYI08475 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 07:54:34 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxEw-0002Cc-OV for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 07:54:34 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA12908 for ; Mon, 30 Jun 2003 07:54:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxEO-00027b-U7; Mon, 30 Jun 2003 07:54:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19WxE5-00024P-Mk for rddp@optimus.ietf.org; Mon, 30 Jun 2003 07:53:41 -0400 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA12540; Mon, 30 Jun 2003 07:53:40 -0400 (EDT) Message-Id: <200306301153.HAA12540@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Mon, 30 Jun 2003 07:53:40 -0400 Subject: [rddp] I-D ACTION:draft-ietf-rddp-problem-statement-02.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : RDMA over IP Problem Statement Author(s) : A. Romanow, J. Mogul, T. Talpey, S. Bailey Filename : draft-ietf-rddp-problem-statement-02.txt Pages : 18 Date : 2003-6-27 This draft addresses an IP-based solution to the problem of high system costs due to network I/O copying in end-hosts at high speeds. The problem is due to the high cost of memory bandwidth, and it can be substantially improved using 'copy avoidance.' The high overhead has limited the use of TCP/IP in interconnection networks especially where high bandwidth, low latency and/or low overhead of end-system data movement are required by the hosted application. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-problem-statement-02.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-problem-statement-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-problem-statement-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-6-27150816.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-problem-statement-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-problem-statement-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-6-27150816.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 18:46:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20365 for ; Mon, 30 Jun 2003 18:46:41 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5UMkDI26559 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 18:46:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7PZ-0006uI-17 for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 18:46:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20346 for ; Mon, 30 Jun 2003 18:46:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X7PW-0004vS-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 18:46:10 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19X7PQ-0004vK-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 18:46:04 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7PN-0006sc-Lg; Mon, 30 Jun 2003 18:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7PH-0006sD-0f for rddp@optimus.ietf.org; Mon, 30 Jun 2003 18:45:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA20335 for ; Mon, 30 Jun 2003 18:45:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X7PD-0004vG-00 for rddp@ietf.org; Mon, 30 Jun 2003 18:45:51 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19X7P3-0004v1-00 for rddp@ietf.org; Mon, 30 Jun 2003 18:45:41 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id C24552448E7 for ; Mon, 30 Jun 2003 18:44:50 -0400 (EDT) Date: Mon, 30 Jun 2003 17:44:48 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 In-Reply-To: <200306301153.HAA12572@ietf.org> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Shared Receive Queues (Re: I-D ACTION:draft-ietf-rddp-arch-02) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Comparing this draft to the prior, the main change I can see is the removal of the "ddp_post_recv" operation. This is a problem. First there is a procedural issue. The only issue raised as a result of the last call and post receive was mine. Specifically, I objected to the requirement that a *buffer* be provided with the ddp_post_recv operation, because an implementation might choose to allow the buffers to be taken from a buffer pool. The actual change is far greater than that however. It eliminates the requirement that reception of each untagged message be pre-enabled by the ULP. In my opinion this is a grave mistake. It creates robustness and security issues. With this change we are seeing a presumption of a solution being written into the draft, after last call, when that proposal has not been made on the reflector. Secondly, there is the substantive issue. There is a problem that has been identified, but not properly discussed within the WG. It needs to be discussed, and a concensus reached on the best solution. The label some will give to this is "Shared Receive Queues", but that is a label of *one* solution. I also believe it is the wrong solution. The problem would be better described as "Over Dedication of Server Buffers". Basically, with the presumed local interface described in the first two drafts of the architecture, a server that supports a large number of concurrent clients would have to dedicate for too much memory to receive untagged requests from clients. Basically, if each client is allowed to have 2 messages in flight, and there are 100,000 clients, then the server would be required to pre-post 200,000 buffers. However, it is quite possible that given the amount of time it takes to process one received request, and the maximum number of requests that can arrive through the server's physical links, that there is never a need for more than 50,000 buffers. I believe that there can be full agreement that this is a legitimate problem, and that neither the architecture or wire protocol should preclude a server from offering a local interface with a solution. However, I believe this can be solved without eliminating the rule that the Data Sink ULP must pre-authorize each untagged message. Further, eliminating that rule has an adverse affect on the robustness and security of RDMA protocols. The shared receive queue solution is described in draft-hilland. It basically allows an endpoint to pull receive buffers from a shared receive queue rather than from its own receive queue. Replacement buffers are pre-posted to the receive queue, rather than to the specific receive queue. What I propose instead is allowing a receive queue to be fed from a shared buffer pool. The Data Sink would still pre-authorize each receive for each endpoint. So there would be a trackable credit count for each endpoint/QP. If the endpoint/QP has a shared buffer pool, any buffer supplied would be placed in the buffer pull. Alternately the ULP could pre-authorize a receive without supplying a buffer (thereby under-provisioning the pool), or post a buffer directly to the shared buffer pool (to reduce the under-provisioning). Both the Shared Receive Queue and Shared Buffer Pool allow a local interface to optionally enable a server to under-provision the receive buffers supporting a large number of concurrent clients. Both methods accept the risk that a DDP Stream will have to be torn down should the Server ULP do a poor job of making its statistical predictions. However this is considered an acceptable risk for server applications to make. Local Interface features to support under-provisioning would never be mandatory-to-implement, and would never be configured as a default option that could be selected by a naive application developer. The difference between the two methods is in how the system will respond when one client exceeds it allotted credits. This difference has both robustness and security issues. RDMA-enabled applications are dependent on ULP pacing for flow control. Transport layer buffering is no longer effective. RDMA-servers typically pace their clients by controlling the maximum number of in-flight untagged messages. DAFS negotiates a limit per session. NFS allows for it to be dynamically adjusted. In either case, a typical processor loop is as follows: post_initial_receive_buffers() while (get_untagged_message_completion()) process_received_request() post_replacement_receive_buffer() -- variable under NFS post_untagged_message_response() There are some important characteristics of this loop. It is self-adaptive, the longer it takes to process a request the slower the new message is posted. It also works with either Shared Receive Queues or Shared Buffer Pools. The only difference is in where receive_buffers are posted. The difference comes when the remote client is over-eager and sends more than it is allowed to, but not enough more that it would drain the shared resources totally. With Shared Receive Queues The DDP Stream is NOT terminated. If the client exceeds a configured limit for the stream, a notice will be given to the ULP. But the DDP Stream will NOT be impaired in any way (i.e., the client can exceed the limit even more until the ULP *responds* to the limit). Even if detected, the limit is configured, i.e. static. It does not automatically adjust while a request is being processed. An over-eager client that had an "off by one" error in its favor could rely upon not being caught. If the ULP response is sufficiently slow, a different DDP Stream will fail to obtain a buffer, resulting in termination of that DDP Stream. It is one thing for the Server to take this risk based upon its prediction of statistical buffer usage. If limits are actually hit, the server is probably already in overload state anyway and not providing very good service as it is. This is a different vunerability to a form of DoS attack. This is listed in the security draft, but it is incorrectly identified as being a trait of any RDMA server that shares a completion queue across multiple clients. In fact the vunerability exists only when Shared Receive Queues are used. With a Shared Buffer Pool The DDP Stream is terminated as soon as it exceeds its allotment, whether or not there are buffers available. Enforcement of ULP flow control is vital to robust and secure RDMA applications. Further, it is not something that the ULP can naturally do for itself. Typically completed receives are queued in a way that is not visible to the ULP. by the time the ULP reaps these completions there is no way to know what the "high water mark" was. Further complicating the local interface to allow for alarms just complicates the local interface, and still leaves a vunerability because illegal receives will continue until the ULP responds. A DoS defense that is dependent upon the ULP reacting to an asynchronous notification has some serious limitations. I am not suggesting that Shared Buffer Pools be adopted as an official IETF blessed interface. Rather, I am suggesting that Shared Receive Queues NOT be recognized. Shared Buffer Pools have no implications on the definition of the ULP. Shared Receive Queues change the semantics on when a DDP Stream will be terminated, and hence undermine the enforcement of ULP flow control. The original architecture is preserved, with the exception of allowing receives to be enabled without providing a buffer. But buffers are a local issue, they should not have end-to-end implications. Authorization of receive operations is visible to the peer. I am particularly concerned that the failure of some implementations to enforce this rule will in fact create a "rule" where clients *expect* to be able to "cheat" a bit, and will consider those servers that enforce credit limits properly to be defective. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 19:21:44 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA21742 for ; Mon, 30 Jun 2003 19:21:43 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h5UNLDH03188 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 19:21:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7xR-0000pL-TN for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 19:21:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA21729 for ; Mon, 30 Jun 2003 19:21:13 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X7xP-0005B3-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 19:21:11 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19X7xF-0005B0-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 19:21:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7xF-0000ng-Dq; Mon, 30 Jun 2003 19:21:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X7x9-0000nV-1w for rddp@optimus.ietf.org; Mon, 30 Jun 2003 19:20:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA21709 for ; Mon, 30 Jun 2003 19:20:54 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X7x7-0005Ax-00 for rddp@ietf.org; Mon, 30 Jun 2003 19:20:53 -0400 Received: from [168.159.2.8] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19X7ww-0005As-00 for rddp@ietf.org; Mon, 30 Jun 2003 19:20:42 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h5UNKTt11242; Mon, 30 Jun 2003 19:20:29 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 30 Jun 2003 19:20:28 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CE63@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] Shared Receive Queues (Re: I-D ACTION:draft-ietf-rddp- arch-02) Date: Mon, 30 Jun 2003 19:11:22 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Comparing this draft to the prior, the main change I can see > is the removal of the "ddp_post_recv" operation. > > This is a problem. > > First there is a procedural issue. The only issue raised as a > result of the last call and post receive was mine. > Specifically, I objected to the requirement that a *buffer* > be provided with the ddp_post_recv operation, because an > implementation might choose to allow the buffers to be taken > from a buffer pool. That's enough to hold both drafts (problem statement and architecture) for a revision of the architecture draft after Vienna - Caitlin is correct that last call comments generally need to be resolved to the satisfaction of the person raising the comment. The reason for holding both drafts is that they need to go to the IESG and IETF Last Call as a pair. The issue raised is valid and needs to be discussed - it will be on the agenda for Vienna, but the final resolution will be on this list. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Monday, June 30, 2003 6:45 PM > To: rddp@ietf.org > Subject: [rddp] Shared Receive Queues (Re: I-D > ACTION:draft-ietf-rddp-arch-02) > > > Comparing this draft to the prior, the main change I can see > is the removal of the "ddp_post_recv" operation. > > This is a problem. > > First there is a procedural issue. The only issue raised as a > result of the last call and post receive was mine. > Specifically, I objected to the requirement that a *buffer* > be provided with the ddp_post_recv operation, because an > implementation might choose to allow the buffers to be taken > from a buffer pool. > > The actual change is far greater than that however. It > eliminates the requirement that reception of each untagged > message be pre-enabled by the ULP. > > In my opinion this is a grave mistake. It creates robustness > and security issues. > > With this change we are seeing a presumption of a solution > being written into the draft, after last call, when that > proposal has not been made on the reflector. > > > Secondly, there is the substantive issue. There is a problem > that has been identified, but not properly discussed within > the WG. It needs to be discussed, and a concensus reached on > the best solution. > > The label some will give to this is "Shared Receive Queues", > but that is a label of *one* solution. I also believe it is > the wrong solution. > > The problem would be better described as "Over Dedication of > Server Buffers". > > Basically, with the presumed local interface described in the > first two drafts of the architecture, a server that supports > a large number of concurrent clients would have to dedicate > for too much memory to receive untagged requests from clients. > > Basically, if each client is allowed to have 2 messages in > flight, and there are 100,000 clients, then the server would > be required to pre-post 200,000 buffers. > > However, it is quite possible that given the amount of time > it takes to process one received request, and the maximum > number of requests that can arrive through the server's > physical links, that there is never a need for more than > 50,000 buffers. > > I believe that there can be full agreement that this is a > legitimate problem, and that neither the architecture or wire > protocol should preclude a server from offering a local > interface with a solution. > > However, I believe this can be solved without eliminating the > rule that the Data Sink ULP must pre-authorize each untagged > message. Further, eliminating that rule has an adverse affect > on the robustness and security of RDMA protocols. > > The shared receive queue solution is described in > draft-hilland. It basically allows an endpoint to pull > receive buffers from a shared receive queue rather than from > its own receive queue. Replacement buffers are pre-posted to > the receive queue, rather than to the specific receive queue. > > What I propose instead is allowing a receive queue to be fed > from a shared buffer pool. The Data Sink would still > pre-authorize each receive for each endpoint. So there would > be a trackable credit count for each endpoint/QP. If the > endpoint/QP has a shared buffer pool, any buffer supplied > would be placed in the buffer pull. Alternately the ULP could > pre-authorize a receive without supplying a buffer (thereby > under-provisioning the pool), or post a buffer directly to > the shared buffer pool (to reduce the under-provisioning). > > Both the Shared Receive Queue and Shared Buffer Pool allow a > local interface to optionally enable a server to > under-provision the receive buffers supporting a large number > of concurrent clients. > > Both methods accept the risk that a DDP Stream will have to > be torn down should the Server ULP do a poor job of making > its statistical predictions. However this is considered an > acceptable risk for server applications to make. Local > Interface features to support under-provisioning would never > be mandatory-to-implement, and would never be configured as a > default option that could be selected by a naive application > developer. > > The difference between the two methods is in how the system > will respond when one client exceeds it allotted credits. > This difference has both robustness and security issues. > > RDMA-enabled applications are dependent on ULP pacing for > flow control. Transport layer buffering is no longer > effective. RDMA-servers typically pace their clients by > controlling the maximum number of in-flight untagged > messages. DAFS negotiates a limit per session. NFS allows for > it to be dynamically adjusted. > > In either case, a typical processor loop is as follows: > > post_initial_receive_buffers() > while (get_untagged_message_completion()) > process_received_request() > post_replacement_receive_buffer() -- variable under NFS > post_untagged_message_response() > > There are some important characteristics of this loop. It is > self-adaptive, the longer it takes to process a request the > slower the new message is posted. > > It also works with either Shared Receive Queues or Shared > Buffer Pools. The only difference is in where receive_buffers > are posted. > > The difference comes when the remote client is over-eager and > sends more than it is allowed to, but not enough more that it > would drain the shared resources totally. > > With Shared Receive Queues > > The DDP Stream is NOT terminated. > > If the client exceeds a configured limit for the stream, a notice > will be given to the ULP. But the DDP Stream will NOT be impaired > in any way (i.e., the client can exceed the limit even more until > the ULP *responds* to the limit). > > Even if detected, the limit is configured, i.e. static. It does > not automatically adjust while a request is being processed. > An over-eager client that had an "off by one" error in its favor > could rely upon not being caught. > > If the ULP response is sufficiently slow, a different DDP Stream > will fail to obtain a buffer, resulting in termination of that > DDP Stream. It is one thing for the Server to take this risk based > upon its prediction of statistical buffer usage. If limits are > actually hit, the server is probably already in overload state > anyway and not providing very good service as it is. This is > a different vunerability to a form of DoS attack. This is > listed in the security draft, but it is incorrectly identified > as being a trait of any RDMA server that shares a completion queue > across multiple clients. In fact the vunerability exists only when > Shared Receive Queues are used. > > With a Shared Buffer Pool > > The DDP Stream is terminated as soon as it exceeds its allotment, > whether or not there are buffers available. > > > Enforcement of ULP flow control is vital to robust and secure > RDMA applications. Further, it is not something that the ULP > can naturally do for itself. Typically completed receives are > queued in a way that is not visible to the ULP. by the time > the ULP reaps these completions there is no way to know what > the "high water mark" was. Further complicating the local > interface to allow for alarms just complicates the local > interface, and still leaves a vunerability because illegal > receives will continue until the ULP responds. A DoS defense > that is dependent upon the ULP reacting to an asynchronous > notification has some serious limitations. > > I am not suggesting that Shared Buffer Pools be adopted as an > official IETF blessed interface. Rather, I am suggesting that > Shared Receive Queues NOT be recognized. > > Shared Buffer Pools have no implications on the definition of > the ULP. Shared Receive Queues change the semantics on when a > DDP Stream will be terminated, and hence undermine the > enforcement of ULP flow control. The original architecture is > preserved, with the exception of allowing receives to be > enabled without providing a buffer. But buffers are a > local issue, they should not have end-to-end implications. > Authorization of receive operations is visible to the peer. I > am particularly concerned that the failure of some > implementations to enforce this rule will in fact create a > "rule" where clients *expect* to be able to "cheat" a bit, > and will consider those servers that enforce credit limits > properly to be defective. > > > > > > > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 21:17:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25264 for ; Mon, 30 Jun 2003 21:17:42 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h611HDH27028 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 21:17:13 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9lh-00071r-LP for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 21:17:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25253 for ; Mon, 30 Jun 2003 21:17:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X9le-0005ro-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:17:10 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19X9lX-0005rl-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:17:03 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9lV-00070C-Hc; Mon, 30 Jun 2003 21:17:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9kk-0006vc-Ic for rddp@optimus.ietf.org; Mon, 30 Jun 2003 21:16:14 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25225 for ; Mon, 30 Jun 2003 21:16:12 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X9ki-0005rL-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:16:12 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19X9kX-0005r0-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:16:01 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 30 Jun 2003 21:15:09 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CE66@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 30 Jun 2003 21:06:02 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Security Draft issues Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , With my "I think I know something about security" hat on, here are some comments on recent email concerning the security draft. All of these are for further discussion; nothing in this message should be construed as instructions from the WG chair. -- Definition of Trust Trust - When one party depends upon the other party to not subvert the goals of the protocols, e.g., it will not attempt to perform the following attacks: spoofing, repudiation, information disclosure, denial of service, or elevation of privilege. I think this definition might be usefully rewritten to be in terms of partial trust (i.e., not to engage in specific attacks that are listed each time the definition is used). Complete Trust is hard to achieve, and in some cases is essentially impossible (e.g., operating systems do not exhibit complete Trust of most applications) and the draft goes on to correctly say that Privileged Applications are only partially trusted. A useful consequence of this would be a need to be explicit about the threats of concern in the definitions of Local Trust and Remote Trust in Section 5, which would help clarify that section. It's also the case that Trust and Trusted (esp. capitalized) have a well-understood meaning in some portions of the security community that involves assurances about information leakage in multi-level systems. Most of the components we are talking about here have no hope or intention of achieving that level of Trust, and hence sticking to notions of partial trust is likely to be clearer (yes, there are IETF folks who will read this as applying to multi-level systems and ding us accordingly). -- Protection Domain and Trust Model For those Trust Models where resources are shared (Trust Models S-NT, S-LT and S-T), it is recommended that Protection Domain be limited to the number of Streams that share the same Trust Model. I don't think that's what was intended. Consider two applications on the same system that use the same trust model (e.g. S-NT), but don't trust each other (e.g., a web server and a file server). Extending the PD across all streams in both applications is clearly wrong. In addition to "the same Trust Model", another important criterion is "scope of mutual trust", which would prevent this. I also agree with the sense of the comment against this - if in doubt, use a smaller protection domain, as Caitlin Bestler noted: > I don't see the connection between Protection Domain scope > and the Trust Models. My advice would simply be that a Protection > Domain should be no wider than the ULP requires. -- STag scope limits Caitlin Bestler wrote: > My reading of the DDP spec is that setting the Protection Domain > is *the* way to control the scope of an STag. I don't think so. Section 10.3 of the DDP draft describes both the Protection Domain and stream association mechanisms and then says: DDP MUST support Protection Domain association and DDP stream association mechanisms for associating an STag and a DDP Stream. IMHO, that implies that the functionality of using stream association within a Protection Domain (as envisioned by the security draft) is REQUIRED. IIRC, this topic was discussed in Atlanta, but I'm not going to object too strenuously to reopening it in the context of the security draft; keep in mind that both models were felt to be useful at the time. -- Multiple STags, same buffer > A distributed ULP that has chosen to be "trusting" (i.e. that it is > only required to work when all participants follow the upper layer > protocol) could easily have a buffer that is exposed to multiple > participants. However, it would be the ULPs responsibility to ensure > that at most one process was attempting to update the buffer at any > time, and that nobody was trying to read it during an update. There are two related partial trust models/assumptions that need to be distinguished here: - Client-Server partial trust, but no inter-Client trust. A server accessed by independently authenticated users is an example. For example, Bob and Alice both trust the fileserver and v.v., but Bob does not trust Alice and v.v. - Client-Server + inter-Client partial trust. An example is when all the clients are part of a single application that engages in things like barrier synchronization. Sharing a buffer among multiple participants is only reasonable where the partial trust in the "ULPs responsibility" is not just Client-Server, but also Client-Client - the notion of "trusting" in the quoted text appears to be assuming Client-Client partial trust (e.g., Alice and Bob won't try to peek at each other's data, or there's no harm in them doing so), which is not the general case. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 21:31:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25565 for ; Mon, 30 Jun 2003 21:31:57 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h611VTx30592 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 21:31:29 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9zV-0007xL-6D for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 21:31:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25524 for ; Mon, 30 Jun 2003 21:31:26 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X9zS-0005vq-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:31:26 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19X9zB-0005vh-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:31:09 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9z3-0007sy-SM; Mon, 30 Jun 2003 21:31:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19X9yh-0007nu-5V for rddp@optimus.ietf.org; Mon, 30 Jun 2003 21:30:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25509 for ; Mon, 30 Jun 2003 21:30:37 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19X9yb-0005vZ-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:30:33 -0400 Received: from [168.159.2.8] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19X9yM-0005vS-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:30:18 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h611U6t00120 for ; Mon, 30 Jun 2003 21:30:06 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 30 Jun 2003 21:30:05 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CE67@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 30 Jun 2003 21:20:54 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP Agenda items Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , It's time to assemble the RDDP agenda for Vienna. Would draft authors please send me initial guesses of how much agenda time is needed for your draft and what the major open technical issues are? Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 21:55:02 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25970 for ; Mon, 30 Jun 2003 21:55:02 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h611sY503087 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 21:54:34 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XALq-0000ni-SO for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 21:54:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25956 for ; Mon, 30 Jun 2003 21:54:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XALn-00063J-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:54:31 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19XALR-00063E-00 for rddp-web-archive@ietf.org; Mon, 30 Jun 2003 21:54:09 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XALJ-0000m5-3W; Mon, 30 Jun 2003 21:54:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XAKr-0000ky-Nd for rddp@optimus.ietf.org; Mon, 30 Jun 2003 21:53:33 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25917 for ; Mon, 30 Jun 2003 21:53:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XAKo-000630-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:53:30 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19XAKP-00061N-00 for rddp@ietf.org; Mon, 30 Jun 2003 21:53:05 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 06B56244A1F; Mon, 30 Jun 2003 21:52:11 -0400 (EDT) Date: Mon, 30 Jun 2003 20:52:08 -0500 From: Caitlin Bestler Subject: Re: [rddp] Security Draft issues - STag / PDs. To: Black_David@emc.com Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564CE66@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, June 30, 2003, Black_David@emc.com wrote: >Caitlin Bestler wrote: > >> My reading of the DDP spec is that setting the Protection Domain >> is *the* way to control the scope of an STag. > >I don't think so. Section 10.3 of the DDP draft describes both >the Protection Domain and stream association mechanisms and then says: > > DDP MUST support > Protection Domain association and DDP stream association mechanisms > for associating an STag and a DDP Stream. > >IMHO, that implies that the functionality of using stream association >within a Protection Domain (as envisioned by the security draft) is >REQUIRED. IIRC, this topic was discussed in Atlanta, but I'm not >going to object too strenuously to reopening it in the context of >the security draft; keep in mind that both models were felt to be >useful at the time. Funny, I read that section as describing two different interfaces to the same mechanism. Specifically: >10.3 Association of an STag and a DDP Stream > > There are several mechanisms for associating an STag and a DDP > Stream. Two reasonable mechanisms for this association are a > Protection Domain (PD) association and a DDP Stream association. > > Under the Protection Domain (PD) association, a unique Protection > Domain Identifier (PD ID) is created and used locally to associate > an STag with a set of DDP Streams. Under this mechanism, the use of > the STag is only permitted on the DDP Streams that have the same PD > ID as the STag. For an incoming DDP Segment of a Tagged DDP Message > on a DDP Stream, if the PD ID of the DDP Stream is not the same as > the PD ID of the STag targeted by the Tagged DDP Message, then the > DDP Segment is not placed and the DDP layer MUST surface a local > error to the ULP. Note that the PD ID is locally defined, and cannot > be directly manipulated by the Remote Peer. > > Under the DDP Stream association, a DDP Stream is identified locally > by a unique DDP Stream identifier (ID). > Which is really the same thing as a Protection Domain Identifier, except that the Consumer does not have to generate it. > An STag is associated with a > DDP Stream by using a DDP Stream ID. In this case, for an incoming > DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP > Stream ID of the DDP Stream is not the same as the DDP Stream ID of > the STag targeted by the Tagged DDP Message, then the DDP Segment is > not placed and the DDP layer MUST surface a local error to the ULP. > Note that the DDP Stream ID is locally defined, and cannot be > directly manipulated by the Remote Peer. > So that in either case the "Protection Identifier" of the DDP Stream is compared with the STag's "Protection Identifier". > A ULP SHOULD associate an STag and a DDP Stream. DDP MUST support > Protection Domain association and DDP Stream association mechanisms > for associating an STag and a DDP Stream. And *nowhere* does it imply that a single DDP Stream needs to be associated with *two* "Protection Identifiers", or that an STag ever has to be associated with *two* "Protection Identifiers". In any event, I interpreted the security-draft text as specifically implying that you could have both methods active on the *same* stream or the same STag. My interpretation of the "DDP Stream ID" was to the effect of "each DDP Stream should have a default Protection ID that is unique to it and will be used if the ULP does not explicitly assign it to a Protection Domain." Personally, I'm working on a software implementation, so a quick OR that isn't going to make that much difference in performance, especially if optimizated tso that the matching id more often the first one. But if that was meant to imply checking two values then it clearly needs to be reworded. But my recomendation is for a single check. It's simple. It's quick, and its adequate. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jun 30 22:16:59 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA26750 for ; Mon, 30 Jun 2003 22:16:59 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h612GVm08877 for rddp-archive@odin.ietf.org; Mon, 30 Jun 2003 22:16:31 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XAh5-0002J6-Dq for rddp-web-archive@optimus.ietf.org; Mon, 30 Jun 2003 22:16:31 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA26732 for ; Mon, 30 Jun 2003 22:16:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XAga-0002HP-St; Mon, 30 Jun 2003 22:16:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XAfv-0002GO-PU for rddp@optimus.ietf.org; Mon, 30 Jun 2003 22:15:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA26705 for ; Mon, 30 Jun 2003 22:15:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XAfs-0006Cu-00 for rddp@ietf.org; Mon, 30 Jun 2003 22:15:16 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19XAfi-0006CK-00 for rddp@ietf.org; Mon, 30 Jun 2003 22:15:06 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 5F565244FEB; Mon, 30 Jun 2003 22:14:31 -0400 (EDT) Date: Mon, 30 Jun 2003 21:14:28 -0500 From: Caitlin Bestler Subject: Re: [rddp] Security Draft issues To: Black_David@emc.com Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564CE66@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, June 30, 2003, Black_David@emc.com wrote: >There are two related partial trust models/assumptions that need to be >distinguished here: > >- Client-Server partial trust, but no inter-Client trust. A server > accessed by independently authenticated users is an example. > For example, Bob and Alice both trust the fileserver and v.v., > but Bob does not trust Alice and v.v. >- Client-Server + inter-Client partial trust. An example is when > all the clients are part of a single application that engages > in things like barrier synchronization. > >Sharing a buffer among multiple participants is only reasonable where >the partial trust in the "ULPs responsibility" is not just >Client-Server, >but also Client-Client - the notion of "trusting" in the quoted text >appears to be assuming Client-Client partial trust (e.g., Alice and >Bob won't try to peek at each other's data, or there's no harm in them >doing so), which is not the general case. Agreed. This is a better wording of the point I was trying to raise about what impact there would be if the remote ULP did not follow the rules. Phrased as above it makes it clear that just because *you* trust the peer to follow the rules does not necessarily mean that you should make that decision on behalf of the *other* clients. One level out from that, you should not extend that trust on behalf of other programs running on the same host. So you should *never* be so trusting that you can be tricked into violating system security. Basically, if you're running with root privileges, you shouldn't trust anybody or anything external unless you are the *sole* application running on the machine. And even then I wouldn't recomend it. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp